SEARCHENGINES

Extended – New Google Crawler To Block Bard Or Google AI From Using Your Content

Published

7 months ago

September 30, 2023

Google has announced a new Googlebot, a new Google crawler, named Google-Extended that you can use to control if your content can help improve Bard and Vertex AI generative APIs or future Google AI products. So if you want to disallow Bard from using your content, you specify so in your robots.txt with the user agent Google-Extended.

Google won’t crawl from Google-Extended, Google will still crawl from its normal Googlebot or other bots. But using Google-Extended will communicate to Google not to use that content for Bard or other AI Google projects. A Google spokesperson told me, “Google-Extended will tell Google not to use the site’s content for Bard and Vertex AI generative APIs.” “For Search, website administrators should continue to use the Googlebot user agent through robots.txt and the NOINDEX meta tag to manage their content in search results, including experiments like Search Generative Experience,” Google added.

Essentially this allows you to allow Google Search to crawl, index and rank your website but disallow Bard or other Google AI projects from using your content.

This comes after Bing offered controls to block Bing Chat AI from using your site a week ago.

“Today we’re announcing Google-Extended, a new control that web publishers can use to manage whether their sites help improve Bard and Vertex AI generative APIs, including future generations of models that power those products. By using Google-Extended to control access to content on a site, a website administrator can choose whether to help these AI models become more accurate and capable over time,” Google wrote.

Google-Extended is a “standalone product token that web publishers can use to manage whether their sites help improve Bard and Vertex AI generative APIs, including future generations of models that power those products,” Google explained.

The User agent token is Google-Extended

“Google-Extended doesn’t have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity,” Google added.

I am not sure if this is the alternative approach for robots.txt for AI…

Big news on the AI front. You can implement via robots.txt -> Announcing Google-Extended, a new control that web publishers can use to manage whether their sites help improve Bard & Vertex AI generative APIs, including future generations of models https://t.co/L73rm6mwzM pic.twitter.com/BtcQ5kaATP
— Glenn Gabe (@glenngabe) September 28, 2023

Note, Google News bot also works a similar way, where it does not crawl but uses the directive for using that content in Google News:

Yes, like Googlebot-News.
— John, aka “a total bell cheese” (@JohnMu) September 29, 2023

Forum discussion at X.

Related Topics:Bard Block Content Crawler Extended Google searchengine

Up Next

Google Local Chat With A Live Agent Expanding

Don't Miss

Google September Helpful Content Update Done, SGE Updates, Links Not A Top Ranking Factor, Bing Hiding Ads & Google’s 25th Birthday

SEARCHENGINES

Daily Search Forum Recap: April 25, 2024

Published

8 hours ago

April 25, 2024

Entireweb News Bot

Here is a recap of what happened in the search forums today, through the eyes of the Search Engine Roundtable and other search forums on the web.

The Google March 2024 core update is still rolling out and the SEO chatter is super heated despite the tools calming. Google Ads API version 16.1 is now out. Google’s John Mueller says splitting and merging sites takes longer than normal site moves for Google to process. Google updated its favicon documentation. And a scathing report on how Google executive Prabhakar Raghavan killed Google Search.

Search Engine Roundtable Stories:

Google March Core Update Stilling Rolling Out & Heated SEO Chatter Continue

Over the past few days, while I was offline, the SEO chatter around the Google search ranking volatility continued to be super heated. The Google tracking tools seemed to calm down a bit, but the chatter is still very heated. This is all while the Google March 2024 core update is still rolling out 51 days later.

Report: How Prabhakar Raghavan Killed Google Search

Ed Zitron wrote a piece named The Man Who Killed Google Search. It goes through in detail how Prabhakar Raghavan, Google’s former head of ads – led a coup so that he could run Google Search, and how an email chain from 2019 began a cascade of events that would lead to him running it into the ground, he said.

Google Favicon Documentation Adds Rel Attribute Value Definitions

Google has updated its favicon documentation for Google Search to add definitions for each supported rel attribute value in the Google Search favicon documentation.

Google Ads API Version 16.1 Now Available

Google released version 16.1 of the Google Ads API yesterday. The update includes query assets for Demand Gen, more location service details, more support warnings, Target ROAS bid simulation and more.

Google: Splitting & Merging Sites Takes Longer Than Normal Site Migrations

Want to scare an SEO? Just tell them they need to manage a site migration. Want to make an SEO faint? Tell them they need to manage to split a site into two or more sites while merging content on those sites. John Mueller from Google said it takes Google longer to process site splits and merges than normal site migrations.

Google Chefs In Dublin

Here is a photo I found on Instagram of a bunch of chefs at the Google office in Dublin. I am not sure if this was for some event or if Googlers were doing some sort of cooking class but it was a photo that caught my eye.

Other Great Search Threads:

Interested in AI assistants within YouTube? -> The new experimental “Ask AI” feature in YouTube is pretty cool. Just tap the button and ask any question about the video you’re watching. Note, AI can’t control the video player as of n, Glenn Gabe on X

What skeleton do you have in your closet?, WebmasterWorld

Googlebot will crawl from one location (often the US), and if you redirect it based on its location, Googlebot would only see (and index) that country version. It’s better to use something like a banner., John Mueller on X

I don’t know your sites, but even if the content’s the same, they’re essentially different sites (especially with ccTLDs), so it would be normal for a migration to affect them differently (and this seems to be quite a way back in the meantime)., John Mueller on X

Search engines recrawl URLs at different rates, sometimes it’s multiple times a day, sometimes it’s once every few months. The verified removal tool is fastest, the public removal tool takes a few days because it needs to verify the URL properly., John Mueller on X

You are now a Google Search Engineer. How do you fix organic search?, Gareth Boyd on X

Search Engine Land Stories:

Feedback:

Have feedback on this daily recap; let me know on Twitter @rustybrick or @seroundtable, on Threads, Mastodon and Bluesky and you can follow us on Facebook and on Google News and make sure to subscribe to the YouTube channel, Apple Podcasts, Spotify, Google Podcasts or just contact us the old fashion way.

SEARCHENGINES

Google Won’t Change The 301 Signals For Ranking & SEO

Published

1 day ago

April 24, 2024

Entireweb News Bot

Gary Illyes from Google said on stage at the SERP conference last week that there is no way that Google would change how the 301 redirect signal works for SEO or search rankings. Gary added that it’s a very reliable signal.

Nikola Minkov quoted Gary Illyes as saying, “It is a very reliable signal, and there is no way we could change that signal,” when asked if a 301 redirect not working is a myth. Honestly, I am not sure the context of this question, as it is not clear from the post on X, but here it is:

More from @methode:
– 301 redirect not working is a myth. “It is a very reliable signal, and there is no way we could change that signal”.#SERPConf2024 #SERPConf2024International
— Nikola Minkov (@n_minkov) April 19, 2024

We’ve covered 301 redirects here countless times – but I never saw a myth that Google does not use 301 redirects as a signal for canonicalization or for passing signals from an old URL to the redirected URL.

Forum discussion at X.

Note: This was pre-written and scheduled to be posted today, I am currently offline for Passover.

SEARCHENGINES

Google Again Says Ignore Link Spam Especially To 404 Pages

Published

3 days ago

April 23, 2024

Entireweb News Bot

I am not sure how many times Google has said that you do not need to disavow spammy links, that you can ignore link spam attacks and that links pointing to pages that 404/410 are links that do not count – but John Mueller from Google said it again.

In a thread on X, John Mueller from Google wrote, “if the links are going to URLs that 404 on your site, they’re already dropped.” “They do nothing,” he added, “If there’s no indexable destination URL, there’s no link.”

John then added, “I’d generally ignore link-spam, and definitely ignore link-spam to 404s.”

Asking if it would hurt to disavow, after responding with the messages above, John wrote:

It will do absolutely nothing. I would take the time to rework a holistic & forward-looking strategy for the site overall instead of working on incremental tweaks (other tweaks might do something, but you probably need real change, not tweaks).

Earlier this year we had tons of SEOs notice spammy links to 404 error pages, John said ignore them. In 2021, Google said links to 404 pages do not count, Google also said that in 2012 and many other times.

Plus, outside of links to 404 pages, Google has said to ignore spammy links, time and time again – even the toxic links – ignore them. The messaging around this changed in 2016 when Penguin 4.0 was released and Google began devaluing links over demoting them.

Here are those new posts in context:

I’d say add both. Lol
— Jeremy Rivera (@JeremyRiveraSEO) April 11, 2024

Sure. But also, save yourself the work completely :-).
— John 🧀 … 🧀 (@JohnMu) April 11, 2024

Re-reading your initial post – if the links are going to URLs that 404 on your site, they’re already dropped. They do nothing. If there’s no indexable destination URL, there’s no link. I’d generally ignore link-spam, and definitely ignore link-spam to 404s.
— John 🧀 … 🧀 (@JohnMu) April 11, 2024

… but still… is this a dumb idea?
— Rebekah Edwards (@rebekah_creates) April 11, 2024
Advertisement

It will do absolutely nothing. I would take the time to rework a holistic & forward-looking strategy for the site overall instead of working on incremental tweaks (other tweaks might do something, but you probably need real change, not tweaks).
— John 🧀 … 🧀 (@JohnMu) April 11, 2024

And in general, Google says it ignores spammy links, so you should too (not new) but this post from John Mueller is:

I would just ignore them, Google ignores them too. Sometimes they’re just more visible in tools, but that doesn’t mean they’re a problem.
— John 🧀 … 🧀 (@JohnMu) April 18, 2024

And then also on Mastodon wrote about a similar situation, “Google has 2 decades of practice of ignoring spammy links. There’s no need to do anything for those links.”

Forum discussion at X.

Note: This was pre-written and scheduled to be posted today, I am currently offline for Passover.