SEARCHENGINES
Google To Work On Complementary Robots.txt Protocols
Google announced last night that it is looking to develop a complementary protocol to the 30-year-old robots.txt protocol. This is because of all the new generative AI technologies Google and other companies are releasing.
This announcement comes shortly after the news around Open AI accessing paywalled content for its ChatGPT service. But I know many of you are not surprised that Google and others are exploring alternatives to robots.txt with all this generative AI technology floating around the web.
Nothing is changing today, all Google announced was that in the “coming months” they will hold discussions with the “community” to come up with new ideas for a new solution.
Google wrote, “Today, we’re kicking off a public discussion, inviting members of the web and AI communities to weigh in on approaches to complementary protocols. We’d like a broad range of voices from across web publishers, civil society, academia and more fields from around the world to join the discussion, and we will be convening those interested in participating over the coming months.”
Google added that it believes “it’s time for the web and AI communities to explore additional machine-readable means for web publisher choice and control for emerging AI and research use cases.”
What this all means right now, is, I don’t know. But here are some responses to my tweet about it:
How about allowing regular expressions in robots.txt? I bet that would solve 75% of the crawl directive challenges SEOs run into.
— Eric Heiken (@EricHeiken) July 6, 2023
I think it works OK, although maybe after 30y it should become robots.xml or something since lots of stuff has been added, and structured file might be more prone to accidental errors
— Miloš Mileusnić (@mileusna) July 6, 2023
“Now that we’ve already trained our LLMs on all your proprietary and copyrighted content, we will finally start thinking about giving you a way to opt out of any of your future content for being used to make us rich.” https://t.co/dda8hHQPfq
— Barry Adams 📰 (@badams) July 6, 2023
Gary Illyes from Google, who worked on this protocol over the years, wrote on LinkedIn, “It’s time. Nearly 30 years ago robots.txt was born and it served the internet well all this time. With the emerging AI technologies, we need to complement it with new instructions (rules) that were designed for AI applications specifically.”
And John Mueller:
I’m excited to see this happening. https://t.co/UTdmeCVwhl
— John Mueller (official) · Not #30D (@JohnMu) July 6, 2023
Today, we’re kicking off a public discussion to explore a machine-readable means for web publisher choice and control for emerging AI & research use cases. Learn more on this effort, including how to join the discussion by signing up: https://t.co/iF9WNyhN3O
— Google SearchLiaison (@searchliaison) July 6, 2023
If you want to participate, fill out this form.
Do any of you have any ideas?
Forum discussion at Twitter.
SEARCHENGINES
Google Shopping Researched with AI
Google Shopping is testing a new AI Overview answer titled “Researched with AI – Experimental.” This seems to pull together reviews for specific products based on Google Gemini AI.
I am told this was displayed in the Google Shopping interface for the query [TV] – I cannot replicate this.
But Khushal Bherwani shared a couple screenshots on X – here are one of those screenshots:
Bordie Clark was then able to replicate this and post this video of it in action:
Whoa, Google is now testing out a researched with AI experiment within the Shopping tab. This was first discovered by @b4k_khushal. The experience involved an AIO-type summary appearing at the top with product suggestions included. cc @rustybrick Details: https://t.co/WupmipgeQg pic.twitter.com/VpPl9SBS0n
— SERP Alert (@SERPalerts) October 7, 2024
Here are more:
🆕 Google testing “Researched with AI · Experimental” in shopping search with big More button pic.twitter.com/CLajfEO0iM
— Khushal Bherwani (@b4k_khushal) October 7, 2024
This seems to replace the “AI Overview” with this “Researched with AI” specifically in Google Shopping. Although, it looks a lot like an AI Overview, just with a different title.
Forum discussion at X.
SEARCHENGINES
Google AI Overview Ads, New Link Format, AI Organized Search Results & Plus More
Google announced a bunch of new AI Search features including search ads finally rolling out to AI Overviews, the new link format rolling out, AI-organized search results, new AI shopping features, Google Lens and more.
Of course, Google announced this when I am offline, but at least I got a heads up a few hours prior. So I was able to write it up for Search Engine Land and here and schedule it.
Ads In Google AI Overviews
Google Search and Shopping Ads are now rolling out to AI Overviews. These ads will have the “Sponsored” label on them, like other search ads. Google said these ads now available for mobile users in the US after several months of testing.
“People have been fidning the ads within AI Overviews helpful because they can quickly connect with relevant businesses, products and services to take the next step at the exact moment they need them,” Google wrote.
Here is a screenshot of the ads in the AI Overviews, which look like they are under the AI Overviews:
Google is also rolling out Shopping Ads through Google Lens. Shopping ads will appear above and alongside visual search results by the end of the year.
Google wrote, “For example, let’s say you’re shopping at a mall and a backpack displayed in a store’s window catches your eye. You want to learn more — and check if there’s other colors available — but you’re in a hurry. Simply tap the Lens icon in the search bar, snap a photo or upload one from your gallery to learn more about the product. Lens will bring together our advanced AI models and Google’s Shopping Graph — which has information on more than 45 billion products — to identify the exact item in your photo.”
Links Updated In AI Overviews
As Google showed us in August, Google has a new link format for AI Overviews that are now rolling out “globally to all countries where AI Overviews are available,” Google told us.
Google also told us, “with AI Overviews, we’re seeing that people have been visiting a greater diversity of websites for help with more complex questions.” “And when people click from search result pages with AI Overviews, these clicks are higher quality for websites — meaning users are more likely to spend more time on the sites they visit,” Google added.
They said they have seen a positive reception during testing, saying, “we’ve seen that this improved experience has driven an increase in traffic to supporting websites compared to the previous design, and people are finding it easier to visit sites that interest them.”
Here is the illustration Google posted:
Here are the screenshots of this test we posted a few weeks ago:
AI-Organized Search Results
Also previously announced, Google is now rolling out AI-organized search results.
Google wrote, “we’re rolling out search results pages organized with AI in the U.S. — beginning with recipes and meal inspiration on mobile. Y ou’ll now see a full-page experience, with relevant results organized just for you. You can easily explore content and perspectives from across the web including articles, videos, forums and more — all in one place.”
Here is what it looks like:
Plus, as I mentioned at Search Engine Land, there is also more on Google Lens and Shopping:
- New multimodal Lens features. Google is launching the ability to search with video in Lens and ask complex questions about moving images (available through Search Labs) and voice input in Lens, so you can ask questions out loud while snapping a photo (or recording video) to more easily search what you see.
- Shop easily with Lens. Google is now showing more helpful product information when using Lens for shopping, including a product’s price across retailers, reviews, or where to buy.
SEARCHENGINES
Google Ranking Volatility Record, Forbes Advisor Slapped, Bing Generative Search Experience & More
We had more of the same Google ranking volatility but this time, we broke a record of the longest period of heated search ranking volatility ever. Google may have slapped Forbes Advisor with some sort of penalty. Bing generative search experience is now rolling out. I posted the big Google October 2024 webmaster report. Google Search Console released sticky performance filters. Google Search Console recommendations is now fully rolled out but more people see it. Danny Sullivan, Google’s Search Liaison, was interviewed again, this time by Aledya Solis. Google warned about using JavaScript generated product markup. Google spoke again about noindex versus canonical tags. Google said don’t use a generic coming soon page for new pages. Bing Search can get dates and times wrong for search snippets. Microsoft Advertising has a new ad console experience, and new diagnostics and performance snapshot. Bing is testing replacing the ad label with a sponsored label. Google Shopping is testing a new updated interface with expandable refinements. Google Local Service Ads has a new bidding option for target cost per lead. BingBot celebrated its 14th birthday. Google didn’t celebrate its 26th birthday for some reason. I am offline today and yesterday for Rosh Hashanah, so this video and post was pre-recorded and scheduled. The Google AI Overview news did not make it into the video, as I recorded it before I was given a heads-up on this news. That was the search news this week at the Search Engine Roundtable.
Sponsor: BruceClay one of the founding fathers of the SEO space, doing search marketing optimization since 1996. Bruce Clay is big into SEO training, check out seotraining.com to learn more and check them out at bruceclay.com. Also, check out their new product, Prewriter.ai – this tool empowers writers to write better and more efficiently, so check it out.
Make sure to subscribe to our video feed or subscribe directly on iTunes, Apple Podcasts, Spotify, Google Podcasts or your favorite podcast player to be notified of these updates and download the video in the background. Here is the YouTube version of the feed. For the original iTunes version, click here
Search Topics of Discussion:
Please do subscribe on YouTube or subscribe via iTunes or on your favorite RSS reader. Don’t forget to comment below with the right answer and good luck!
-
WORDPRESS5 days ago
WordPress biz Automattic details WP Engine deal demands • The Register
-
SEARCHENGINES6 days ago
Daily Search Forum Recap: October 1, 2024
-
SEARCHENGINES5 days ago
Programming Note: Rosh Hashanah 5785
-
SEO7 days ago
Bing Expands Generative Search Capabilities For Complex Queries
-
WORDPRESS7 days ago
How to Easily Add Snapchat Pixel for WooCommerce in WordPress
-
SEARCHENGINES4 days ago
Daily Search Forum Recap: October 3, 2024
-
WORDPRESS6 days ago
How to Create A Website to Sell Products In 8 Steps [+6 Expert Tips]
-
SEO5 days ago
How To Stop Filter Results From Eating Crawl Budget