Connect with us


Why GoogleBot Doesn’t Crawl Enough Pages on Some Sites via @sejournal, @martinibuster



In a Google SEO Office Hours hangout Google’s John Mueller was asked why Google did not crawl enough web pages. The person asking the question explained that Google was crawling at a pace that was insufficient to keep pace with an enormously large website.  John Mueller explained why Google might not be crawling enough pages.

What is the Google Crawl Budget?

GoogleBot is the name of Google’s crawler that goes to web page to web page indexing them for ranking purposes.

But because the web is large Google has a strategy of only indexing higher quality web pages and not indexing the low quality web pages.

According to Google’s developer page for huge websites (in the millions of web pages):

“The amount of time and resources that Google devotes to crawling a site is commonly called the site’s crawl budget.

Note that not everything crawled on your site will necessarily be indexed; each page must be evaluated, consolidated, and assessed to determine whether it will be indexed after it has been crawled.

Crawl budget is determined by two main elements: crawl capacity limit and crawl demand.”


Continue Reading Below

Related: Google SEO 101: Website Crawl Budget Explained


What Decides GoogleBot Crawl Budget?

The person asking the question had a site with hundreds of thousands of pages. But Google was only crawling about 2,000 web pages per day, a rate that is too slow for such a large site.

The person asking the question followed up with the following question:

“Do you have any other advice for getting insight into the current crawling budget?

Just because I feel like we’ve really been trying to make improvements but haven’t seen a jump in pages per day crawled.”

Google’s Mueller asked the person how big the site is.

The person asking the question answered:

“Our site is in the hundreds of thousands of pages.

And we’ve seen maybe around 2,000 pages per day being crawled even though there’s like a backlog of like 60,000 discovered but not yet indexed or crawled pages.”

Google’s John Mueller answered:

“So in practice, I see two main reasons why that happens.

On the one hand if the server is significantly slow, which is… the response time, I think you see that in the crawl stats report as well.


That’s one area where if… like if I had to give you a number, I’d say aim for something below 300, 400 milliseconds, something like that on average.

Because that allows us to crawl pretty much as much as we need.

It’s not the same as the page speed kind of thing.

So that’s… one thing to watch out for.”


Continue Reading Below

Related: Crawl Budget: Everything You Need to Know for SEO

Site Quality Can Impact GoogleBot Crawl Budget

Google’s John Mueller next mentioned the issue of site quality.


Poor site quality can cause the GoogleBot crawler to not crawl a website.

Google’s John Mueller explained:

“The other big reason why we don’t crawl a lot from websites is because we’re not convinced about the quality overall.

So that’s something where, especially with newer sites, I see us sometimes struggle with that.

And I also see sometimes people saying well, it’s technically possible to create a website with a million pages because we have a database and we just put it online.

And just by doing that, essentially from one day to the next we’ll find a lot of these pages but we’ll be like, we’re not sure about the quality of these pages yet.

And we’ll be a bit more cautious about crawling and indexing them until we’re sure that the quality is actually good.”

Factors that Affect How Many Pages Google Crawls

There are other factors that can affect how many pages Google crawls that weren’t mentioned.

For example, a website hosted on a shared server might be unable to serve pages quick enough to Google because there might be other sites on the server that are using excessive resources, slowing down the server for the other thousands of sites on that server.


Another reason may be that the server is getting slammed by rogue bots, causing the website to slow down.

John Mueller’s advice to note the speed that the server is serving web pages is good. Be sure to check it after hours at night because many crawlers like Google will crawl in the early morning hours because that’s generally a less disruptive time to crawl and there are less site visitors on sites at that hour.


Read the Google Developer Page on Crawl Budget for Big Sites:
Large Site Owner’s Guide to Managing Your Crawl Budget


Continue Reading Below

Watch Google’s John Mueller answer the question about GoogleBot not crawling enough web pages.

View it at approximately the 25:46 minute mark:

[embedded content]



Google December Product Reviews Update Affects More Than English Language Sites? via @sejournal, @martinibuster



Google’s Product Reviews update was announced to be rolling out to the English language. No mention was made as to if or when it would roll out to other languages. Mueller answered a question as to whether it is rolling out to other languages.

Google December 2021 Product Reviews Update

On December 1, 2021, Google announced on Twitter that a Product Review update would be rolling out that would focus on English language web pages.

The focus of the update was for improving the quality of reviews shown in Google search, specifically targeting review sites.

A Googler tweeted a description of the kinds of sites that would be targeted for demotion in the search rankings:

“Mainly relevant to sites that post articles reviewing products.

Think of sites like “best TVs under $200″.com.

Goal is to improve the quality and usefulness of reviews we show users.”



Continue Reading Below

Google also published a blog post with more guidance on the product review update that introduced two new best practices that Google’s algorithm would be looking for.

The first best practice was a requirement of evidence that a product was actually handled and reviewed.

The second best practice was to provide links to more than one place that a user could purchase the product.

The Twitter announcement stated that it was rolling out to English language websites. The blog post did not mention what languages it was rolling out to nor did the blog post specify that the product review update was limited to the English language.

Google’s Mueller Thinking About Product Reviews Update

Screenshot of Google's John Mueller trying to recall if December Product Review Update affects more than the English language

Screenshot of Google's John Mueller trying to recall if December Product Review Update affects more than the English language

Product Review Update Targets More Languages?

The person asking the question was rightly under the impression that the product review update only affected English language search results.



Continue Reading Below

But he asserted that he was seeing search volatility in the German language that appears to be related to Google’s December 2021 Product Review Update.

This is his question:

“I was seeing some movements in German search as well.

So I was wondering if there could also be an effect on websites in other languages by this product reviews update… because we had lots of movement and volatility in the last weeks.

…My question is, is it possible that the product reviews update affects other sites as well?”

John Mueller answered:

“I don’t know… like other languages?

My assumption was this was global and and across all languages.

But I don’t know what we announced in the blog post specifically.


But usually we try to push the engineering team to make a decision on that so that we can document it properly in the blog post.

I don’t know if that happened with the product reviews update. I don’t recall the complete blog post.

But it’s… from my point of view it seems like something that we could be doing in multiple languages and wouldn’t be tied to English.

And even if it were English initially, it feels like something that is relevant across the board, and we should try to find ways to roll that out to other languages over time as well.

So I’m not particularly surprised that you see changes in Germany.

But I also don’t know what we actually announced with regards to the locations and languages that are involved.”

Does Product Reviews Update Affect More Languages?

While the tweeted announcement specified that the product reviews update was limited to the English language the official blog post did not mention any such limitations.

Google’s John Mueller offered his opinion that the product reviews update is something that Google could do in multiple languages.


One must wonder if the tweet was meant to communicate that the update was rolling out first in English and subsequently to other languages.

It’s unclear if the product reviews update was rolled out globally to more languages. Hopefully Google will clarify this soon.


Google Blog Post About Product Reviews Update

Product reviews update and your site

Google’s New Product Reviews Guidelines

Write high quality product reviews

John Mueller Discusses If Product Reviews Update Is Global

Watch Mueller answer the question at the 14:00 Minute Mark

[embedded content]

Continue Reading

Subscribe To our Newsletter
We promise not to spam you. Unsubscribe at any time.
Invalid email address