Understanding Googlebot’s Byte Limits and Crawling System

Recently, Google’s Gary Illyes wrote a blog post explaining how Googlebot, Google’s web crawler, works. This article covers important topics such as byte limits, how Googlebot fetches information, and the structure of Google’s crawling system.

What is Googlebot?

Googlebot is a program that searches the web to find and gather information about websites. It is one part of a larger system that includes other tools like Google Shopping and AdSense. All of these tools use the same crawling platform but have different settings, such as user agent strings and byte limits.

When Googlebot is mentioned in server logs, it refers to Google Search. Other tools appear with different names, which you can find on Google’s official crawler documentation site.

How the 2 MB Limit Works

Googlebot can fetch up to 2 MB of data from a webpage. This limit does not apply to PDF files, which can be up to 64 MB. If a webpage is larger than 2 MB, Googlebot simply stops fetching at that point and sends the truncated content to Google’s indexing systems.

It’s important to know that HTTP request headers count towards the 2 MB limit. This means if your page has a lot of header information, it could reach the limit faster. Every other resource linked in the HTML, like CSS and JavaScript files, is fetched separately and does not count towards the 2 MB limit.

Rendering After Fetching

After Googlebot fetches the data, the Web Rendering Service (WRS) processes the content. It understands the page’s layout and functionality by executing JavaScript and pulling in CSS files. However, it does not request images or videos.

The WRS handles every request without storing any previous data, which means it treats each request separately. For more details on handling JavaScript, you can refer to Google’s JavaScript troubleshooting documentation.

Tips to Stay Under the Limit

To ensure your pages stay within the 2 MB limit, Google provides some recommendations:

1. Move Heavy Files: Keep large CSS and JavaScript files external, as these have separate limits.
2. Prioritize Important Tags: Place important elements higher in your HTML, like meta tags and structured data.
3. Check Inline Content: Be cautious with inline base64 images or large menus, as they may quickly add up to the limit.

Remember, this 2 MB limit can change as web pages evolve.

Why This Matters

Understanding these limits is crucial for website owners and developers. Although most pages fall below the 2 MB threshold, knowing how bytes count can help prevent issues. Sites with large HTTP headers or complex markup should be especially aware.

The distinction between Googlebot’s 2 MB limit and the default 15 MB limit for other crawlers helps clarify why different crawlers behave the way they do.

Looking Ahead

Google has actively updated its documentation regarding Googlebot’s crawling limits. While these figures may change over time, the current 2 MB limit is manageable for most sites. However, sites with heavy inline content should make sure their critical information is accessible within the first 2 MB.

In conclusion, by understanding how Googlebot operates and keeping these guidelines in mind, you can help ensure that your website is efficiently crawled and indexed by Google.