NEWS

Can Being Crawled by Googlebot Desktop Cause Indexing Delay? via @sejournal, @martinibuster

Published

3 years ago

August 13, 2021

googlebot indexing delay 61164cad544c5 sej

Google’s John Mueller answered a question about whether it matters to indexing speed if Googlebot mobile or Googlebot desktop crawls a page. The person asking the question was worried about slow indexing and wanted to know if getting crawled by Googlebot desktop (instead of Googlebot mobile) was the reason.

Googlebot Desktop and Mobile

Google has a multitude of website crawlers that follow from link to link discovering new web pages to crawl and index.

If you look at your traffic logs you might see:

Googlebot Desktop
Googlebot Smartphone
Googlebot Image
Mediapartners-Google (Mobile AdSense)
FeedFetcher-Google
Google-Read-Aloud
AdsBot-Google
Googlebot-Video

Continue Reading Below

Those are just a few of the Google-related Google crawlers that can visit a website.

The question John Mueller answered was primarily concerned with Googlebot Desktop and Googlebot Smartphone and what impact on indexing the desktop version might have.

This is the question:

“Our website’s posts are being indexed after one week. The site is old and mobile-friendly.
When I submit URLs in Inspection Tools, it shows it’s crawled by Google smartphone.
But in the About section, the indexing crawler is shown as Googlebot Desktop.
Does this mean the Smartphone bot is crawling the URL and the Desktop bot is indexing this URL?
Is this the reason behind the indexing delay?”

Continue Reading Below

John Mueller’s answer:

“This would not be a reason for any kind of indexing issues.
I don’t know offhand why you might see this mix of desktop and mobile crawlers there.
It might just be that the Search Console is essentially saying well, by default, the Inspection Tool will just use the mobile Googlebot.
In practice, we always crawl with both of these crawlers.
It’s just a matter of how much we crawl and index with any of these crawlers.
So, usually you’ll see something like 80% is coming from mobile Googlebot and 20% maybe from desktop Googlebot.
Advertisement

And that kind of ratio, that is essentially normal and doesn’t mean that things are slower than anything else.”

Multiple Versions of Googlebot Crawl Websites

Google’s Googlebot support page states that both versions of Googlebot visit web pages.

According to Google’s official documentation:

“Your website will probably be crawled by both Googlebot Desktop and Googlebot Smartphone.
You can identify the subtype of Googlebot by looking at the user agent string in the request.”

According to Mueller and Google it’s totally normal to be crawled by both and there’s no indexing impact related to getting crawled by Googlebot Desktop.