SECURITY
AI Crawlers Accessing Sites 50,000 Times Per Visitor according to Cloudflare
Summary: Cloudflare’s new tool helps website owners see how often AI bots visit their sites compared to real people. The data shows that some AI crawlers visit the same website up to 50,000 times for every person who actually looks at it. This is a big change because it can cost publishers money and make it harder for them to earn from ads. The tool gives important information so publishers can better understand and manage this AI traffic.
On July 1, 2026, Cloudflare introduced a new tool called Attribution Business Insights for its Bot Management customers. This tool helps website owners understand how many times AI crawlers visit their sites compared to the human readers that come from these crawlers.
What is Cloudflare’s New Tool?
The Attribution Business Insights dashboard was launched during Cloudflare’s second annual Content Independence Day event. The dashboard helps business leaders, rather than just security engineers, see which companies are visiting their websites using automated systems, also known as bots. While the dashboard doesn’t block any traffic by itself, it provides valuable insights into how much traffic comes from AI crawlers compared to real human visitors.
Understanding the Problem
Traditionally, internet traffic was balanced. Search engines would crawl websites a few times and then send visitors who might buy products or view ads. This helped pay for content creators. However, that balance has changed. Cloudflare’s data shows that AI crawlers can visit a web page from 118 times to nearly 50,000 times for every human visitor they send.
This means that some AI crawlers could hit the same page tens of thousands of times before sending even one actual reader. This change is significant because it disrupts the way website owners earn money from ads and engagements.
What Can the Dashboard Show?
The dashboard provides four main categories of information:
- Total Bot Traffic: Shows the amount of bot traffic compared to human traffic.
- Crawl-to-Referral Ratio: Displays the number of times a company’s bots crawl a webpage compared to the number of actual visitors.
- Ranked List of Bots: Lists which bots generate the most traffic, along with their country of origin and whether they are blocked or allowed on the site.
- Crawler Classification: Categorizes crawlers based on their activities, such as Training (gathering data for AI), Search (updating databases), or Agent (acting on behalf of a user).
Why Is This Important?
Cloudflare believes that the imbalance in crawls versus referrals creates “double costs” for publishers. First, they lose the referral traffic that supports their content. Second, they still incur costs for serving requests from bots, especially the Training crawlers that bring no value.
The dashboard aims to invite business leaders into the conversation, highlighting that AI traffic affects more than just the tech side of operations. It allows publishers to make better decisions regarding licensing and interactions with companies that use AI crawlers.
Cloudflare suggests that publishers can use the data to negotiate better terms with AI companies regarding content access based on their crawling activity.
Conclusion
Cloudflare’s Attribution Business Insights is a crucial step for website owners, especially in the age of AI. The data helps them navigate the challenges brought on by an overwhelming number of AI crawlers. With ratios of up to 50,000 crawls per referral, understanding this traffic is essential for maintaining a healthy online business and ensuring that content creators are fairly compensated for their work.
This new tool, launched on July 1, 2026, represents Cloudflare’s commitment to adapting to the evolving web landscape and helps equip website publishers with the insights they need for their online strategies.
