ChatGPT Now Crawls 3.6 Times More Than Googlebot

In the world of websites and search engines, it’s commonly thought that Googlebot is the biggest crawler sending requests to websites. However, new information shows this is no longer true!

A recent study analyzed over 24 million requests from 69 different websites over 55 days. The results are surprising: OpenAI’s ChatGPT-User crawler made 3.6 times more requests than Googlebot. This figure doesn’t even include requests from another OpenAI crawler called GPTBot.

Key Findings

1. AI Crawlers Are More Active:
   • ChatGPT-User, OpenAI’s retrieval crawler, made 133,361 requests.
   • Googlebot, by comparison, made only 37,426 requests.
   • In total, AI crawlers made 3.6 times more requests than traditional crawlers like Googlebot and Bingbot.
2. Two Different Crawlers by OpenAI:
   • OpenAI uses two crawlers:
     • ChatGPT-User: This crawler fetches websites to provide real-time answers when users ask questions.
     • GPTBot: This one gathers data to improve OpenAI’s models. Many website owners don’t realize the difference and might block one while allowing the other.
   • Together, both OpenAI crawlers made 142,225 requests, which means they are more active than Googlebot.
3. AI Crawlers Are Fast & Reliable:
   • AI crawlers fetch data quickly and efficiently. For example, ChatGPT-User has a response time of just 11 milliseconds.
   • In contrast, Googlebot takes 84 milliseconds. This means AI crawlers can handle requests faster, but the total number of requests can still add pressure on servers.
4. Googlebot Sees a Different Version of Your Site:
   • Googlebot faces challenges when it runs into errors like “page not found” (404) or “access denied” (403), affecting its success rate.
   • While Googlebot has a 96.3% success rate, ChatGPT-User achieves 99.99%. This is because Googlebot revisits old URLs that might not exist anymore.
5. A Clear Trend:
   • Reports show that AI crawling activity surged 15 times in 2025. Companies should adjust their strategies to accommodate this shift.

What Should Websites Do?

To adapt to this change in crawling behavior, website owners can take several steps:

1. Review Your Robots.txt:
   • Make sure your website allows both ChatGPT-User and GPTBot to access it. Block only what you absolutely don’t want included.
2. Clean Up Broken Links:
   • Regularly check for and fix any broken links. Proper redirects should be in place to avoid unnecessary errors.
3. Optimize for AI Crawlers:
   • Ensure your website is accessible to all kinds of crawlers. If you’re using complex technologies (like JavaScript), it might block AI crawlers.
4. Prepare for High Traffic:
   • Be aware that AI crawlers might send many requests. Plan your hosting and infrastructure to accommodate this increase in traffic.

Conclusion

The landscape of website crawling is changing quickly. AI crawlers like ChatGPT-User are now more dominant than traditional crawlers like Googlebot. Adapting to these changes can help website owners improve visibility and maintain smooth operation. Keeping your website accessible and optimizing for these new crawlers is key to staying relevant in this evolving digital space.