Confirmed Google User Agent & Crawler

Google’s John Mueller confirmed that the GoogleProber and the GoogleProducer (which we covered here) are both legit Googlebot user agents and crawlers. “These are official Google requests with a user-agent specific to the Google News config in Publisher Center so if they are blocked the feeds you set up may not work properly,” John added.

John did admit there is an issue with one or both of these bots crawling too much (either for his site or in general). John said, “We have found an issue that is affecting you where we are making too frequent requests and this will be fixed shortly.” John also said “We’ll also work on fixing the broken info-link.”

Here are these tweets for more context:

Hi @searchliaison @dannysullivan @JohnMu we are seeing a lot of traffic to publisher’s sites with a new user-agent ‘GoogleProducer; (+https://t.co/5l7Fm3K043) ‘. Our WAF is currently blocking it, but it’s origin is actually Google Proxy hosts. Is this legit traffic? Thank you! pic.twitter.com/twVlVI5gqJ

— Hernán Marsili (@hmarsili) January 31, 2023

Thank you, also, from the same IPs we have identify increased activity from GoogleProber UA. pic.twitter.com/J8eN3pRK32

— Hernán Marsili (@hmarsili) February 1, 2023

Advertisement

… We have found an issue that is affecting you where we are making too frequent requests and this will be fixed shortly. We’ll also work on fixing the broken info-link.

— johnmu is a ranking factor and so are you 🐀 (@JohnMu) February 7, 2023

Forum discussion at Twitter.