SEO
“It’s Impossible To Crawl The Whole Web”

In response to a question about why SEO tools don’t show all backlinks, Google’s Search Advocate John Mueller says it’s impossible to crawl the whole web.
This is stated in a comment on Reddit in a thread started by a frustrated SEO professional.
They ask why all links pointing to a site aren’t getting found by an SEO tool they’re using.
Which tool the person is using isn’t important. As we learn from Mueller, it’s not possible for any tool to discover 100% of a website’s inbound links.
Here’s why.
There’s No Way To Crawl The Web “Properly”
Mueller says there’s no objectively correct way to crawl the web because it has an infinite number of URLs.
No one has the resources to keep an endless amount of URLs in a database, so web crawlers try to determine what’s worth crawling
As Mueller explains, that inevitably leads to URLs getting crawled infrequently or not at all.
“There’s no objective way to crawl the web properly.
It’s theoretically impossible to crawl it all, since the number of actual URLs is effectively infinite. Since nobody can afford to keep an infinite number of URLs in a database, all web crawlers make assumptions, simplifications, and guesses about what is realistically worth crawling.
And even then, for practical purposes, you can’t crawl all of that all the time, the internet doesn’t have enough connectivity & bandwidth for that, and it costs a lot of money if you want to access a lot of pages regularly (for the crawler, and for the site’s owner).
Past that, some pages change quickly, others haven’t changed for 10 years – so crawlers try to save effort by focusing more on the pages that they expect to change, rather than those that they expect not to change.”
How Web Crawlers Determine What’s Worth Crawling
Mueller goes on to explain how web crawlers, including search engines and SEO tools, figure out which URLs are worth crawling.
“And then, we touch on the part where crawlers try to figure out which pages are actually useful.
The web is filled with junk that nobody cares about, pages that have been spammed into uselessness. These pages may still regularly change, they may have reasonable URLs, but they’re just destined for the landfill, and any search engine that cares about their users will ignore them.
Sometimes it’s not just obvious junk either. More & more, sites are technically ok, but just don’t reach “the bar” from a quality point of view to merit being crawled more.”
Web Crawlers Work With A Limited Set Of URLs
Mueller concludes his response saying all web crawlers work on a “simplified” set of URLs.
Since there’s no correct way to crawl the web, as mentioned previously, every SEO tool has its own way of deciding which URLs are worth crawling.
That’s why one tool may discover backlinks that another tool didn’t find.
“Therefore, all crawlers (including SEO tools) work on a very simplified set of URLs, they have to work out how often to crawl, which URLs to crawl more often, and which parts of the web to ignore. There are no fixed rules for any of this, so every tool will have to make their own decisions along the way. That’s why search engines have different content indexed, why SEO tools list different links, why any metrics built on top of these are so different.”
Source: Reddit
Featured Image: rangizzz/Shutterstock
fbq('track', 'PageView');
fbq('trackSingle', '1321385257908563', 'ViewContent', { content_name: 'googles-john-mueller-its-impossible-to-crawl-the-whole-web', content_category: 'news digital-marketing-tools ' });
SEO
Surfer SEO Unveils New Semrush Integration

Surfer SEO has a new growth management tool that includes backlink data provided by industry leader, Semrush.
Here’s what the integration means for users.
Surfer SEO
Surfer SEO is a machine learning tool used by SEOs to conduct keyword research, create content strategies, and generate AI outlines.
Companies such as Square, ClickUp, and Shopify use the tool, in addition to over 15,000 other brands and agencies.
Semrush
Semrush is an enterprise SEO tool marketers use to audit websites, conduct competitor analysis and develop content strategies.
IBM, TESLA, and Amazon are just a few companies using it, in addition to more than 10 million marketing professionals.
Starting today, Surfer SEO will use Semrush’s backlink data to expand Grow Flow task recommendations.
Grow Flow
Grow Flow is like your friendly SEO AI assistant.
Described on the Surfer website as an “AI growth management platform,” the tool provides a few SEO tasks each week to help you stay on track — and not get lost in the overwhelming amount of information that comes with learning SEO.
For example, it may recommend adding keywords (GSC) that a website is ranking for but not explicitly speaking to in an article.
Or, the tool may recommend where to add internal links.




It can also recommend new content topics that users can open in a content editor at the push of a button.




The integration with Semrush comes into play in the Grow Flow recommendations.
Once you connect Surfer to the freemium Semrush account, you unlock new tasks.




Voila – a list of referring domains every week pulled from your competitors!




Manually researching a competitor’s backlinks is a considerable drain on internal resources. Thanks to this integration, you’ll discover new opportunities automatically.
An Industry First
Until now, Semrush has not integrated with an SEO company.
Historically, Semrush has only integrated with Google Products like Search Console and Analytics, and social media networks like Facebook and Twitter.
It’s also worked with task management tools like Trello and Monday.com – but never a direct SEO competitor.
Semrush’s “SEO writing assistant” feature is a direct competitor to Surfer’s “SEO Content Editor”: one of its most popular features.
So, why partner and offer this fantastic AI assistant to search marketers – for free?
I asked Tomasz Niezgoda, Surfer SEO’s Marketing Executive and Partner, how this partnership came to be.
“Pretty straightforward,” he said. “Semrush reached out to us with a proposition to integrate Surfer inside their marketplace [and] after some time, we decided to give it a shot and started working on this integration.”
From the beginning, Semrush felt like this would be a successful integration because it was applying its backlink data in a customer-oriented way.
“For Semrush, it’s a very meaningful integration. It allows Surfer SEO users to gain valuable link building insights and knowledge, which is crucial for ranking their content,” said Eugene Levin, President, Semrush.
What This Means
Two major SEO competitors are working together to create a free tool for small business owners and entry-level marketers to develop weekly best practices.
Matt Diggity, CEO and Founder at Diggity Marketing, calls the tool “Simple. Efficient. Automated.”
“You get the list every week and gain immediate insights [into] which referrals your competition is getting. Then, you can start working on your own link-building strategy straight away.”
The two giants hint that combining Semrush’s backlink data with the machine learning power of Surfer is only the beginning.
“We like each other,” said Niezgoda. “Maybe it’s just the first integration that’s coming.”
Featured Image: Screenshot from Grow Flow, June 2022
fbq('track', 'PageView');
fbq('trackSingle', '1321385257908563', 'ViewContent', { content_name: 'sej-exclusive-surfer-seo-unveils-semrush-integration', content_category: 'news digital-marketing-tools' }); }