Connect with us

SEO

“It’s Impossible To Crawl The Whole Web”

Published

on

“It’s Impossible To Crawl The Whole Web”


In response to a question about why SEO tools don’t show all backlinks, Google’s Search Advocate John Mueller says it’s impossible to crawl the whole web.

This is stated in a comment on Reddit in a thread started by a frustrated SEO professional.

They ask why all links pointing to a site aren’t getting found by an SEO tool they’re using.

Which tool the person is using isn’t important. As we learn from Mueller, it’s not possible for any tool to discover 100% of a website’s inbound links.

Here’s why.

There’s No Way To Crawl The Web “Properly”

Mueller says there’s no objectively correct way to crawl the web because it has an infinite number of URLs.

No one has the resources to keep an endless amount of URLs in a database, so web crawlers try to determine what’s worth crawling

Advertisement

As Mueller explains, that inevitably leads to URLs getting crawled infrequently or not at all.

“There’s no objective way to crawl the web properly.

It’s theoretically impossible to crawl it all, since the number of actual URLs is effectively infinite. Since nobody can afford to keep an infinite number of URLs in a database, all web crawlers make assumptions, simplifications, and guesses about what is realistically worth crawling.

And even then, for practical purposes, you can’t crawl all of that all the time, the internet doesn’t have enough connectivity & bandwidth for that, and it costs a lot of money if you want to access a lot of pages regularly (for the crawler, and for the site’s owner).

Past that, some pages change quickly, others haven’t changed for 10 years – so crawlers try to save effort by focusing more on the pages that they expect to change, rather than those that they expect not to change.”

How Web Crawlers Determine What’s Worth Crawling

Mueller goes on to explain how web crawlers, including search engines and SEO tools, figure out which URLs are worth crawling.

“And then, we touch on the part where crawlers try to figure out which pages are actually useful.

The web is filled with junk that nobody cares about, pages that have been spammed into uselessness. These pages may still regularly change, they may have reasonable URLs, but they’re just destined for the landfill, and any search engine that cares about their users will ignore them.

Sometimes it’s not just obvious junk either. More & more, sites are technically ok, but just don’t reach “the bar” from a quality point of view to merit being crawled more.”

Web Crawlers Work With A Limited Set Of URLs

Mueller concludes his response saying all web crawlers work on a “simplified” set of URLs.

Advertisement

Since there’s no correct way to crawl the web, as mentioned previously, every SEO tool has its own way of deciding which URLs are worth crawling.

That’s why one tool may discover backlinks that another tool didn’t find.

“Therefore, all crawlers (including SEO tools) work on a very simplified set of URLs, they have to work out how often to crawl, which URLs to crawl more often, and which parts of the web to ignore. There are no fixed rules for any of this, so every tool will have to make their own decisions along the way. That’s why search engines have different content indexed, why SEO tools list different links, why any metrics built on top of these are so different.”


Source: Reddit

Featured Image: rangizzz/Shutterstock

fbq('track', 'PageView');

fbq('trackSingle', '1321385257908563', 'ViewContent', { content_name: 'googles-john-mueller-its-impossible-to-crawl-the-whole-web', content_category: 'news digital-marketing-tools ' });





Source link

See also  Does Google Crawl URLs In Structured Data?

SEO

Surfer SEO Unveils New Semrush Integration

Published

on

Surfer SEO Unveils New Semrush Integration

Surfer SEO has a new growth management tool that includes backlink data provided by industry leader, Semrush.

Here’s what the integration means for users.

Surfer SEO

Surfer SEO is a machine learning tool used by SEOs to conduct keyword research, create content strategies, and generate AI outlines.

Companies such as Square, ClickUp, and Shopify use the tool, in addition to over 15,000 other brands and agencies.

Semrush

Semrush is an enterprise SEO tool marketers use to audit websites, conduct competitor analysis and develop content strategies.

IBM, TESLA, and Amazon are just a few companies using it, in addition to more than 10 million marketing professionals.

Starting today, Surfer SEO will use Semrush’s backlink data to expand Grow Flow task recommendations.

Advertisement

Grow Flow

Grow Flow is like your friendly SEO AI assistant.

Described on the Surfer website as an “AI growth management platform,” the tool provides a few SEO tasks each week to help you stay on track — and not get lost in the overwhelming amount of information that comes with learning SEO.

For example, it may recommend adding keywords (GSC) that a website is ranking for but not explicitly speaking to in an article.

Screenshot from Grow Flow, June 2022

Or, the tool may recommend where to add internal links.

internal links_growflow_surfer seoScreenshot from Grow Flow, June 2022

It can also recommend new content topics that users can open in a content editor at the push of a button.

new content topics_content editor_growflowScreenshot from Grow Flow, June 2022

The integration with Semrush comes into play in the Grow Flow recommendations.

See also  Why Do Core Web Vitals Scores Always Change?

Once you connect Surfer to the freemium Semrush account, you unlock new tasks.

connect semrush to surfer_growflowScreenshot from Grow Flow, June 2022

Voila – a list of referring domains every week pulled from your competitors!

Backlink research automated_Growflow_Surfer SEOScreenshot from Grow Flow, June 2022

Manually researching a competitor’s backlinks is a considerable drain on internal resources. Thanks to this integration, you’ll discover new opportunities automatically.

An Industry First

Until now, Semrush has not integrated with an SEO company.

Historically, Semrush has only integrated with Google Products like Search Console and Analytics, and social media networks like Facebook and Twitter.

It’s also worked with task management tools like Trello and Monday.com – but never a direct SEO competitor.

Semrush’s “SEO writing assistant” feature is a direct competitor to Surfer’s “SEO Content Editor”: one of its most popular features.

Advertisement

So, why partner and offer this fantastic AI assistant to search marketers – for free?

I asked Tomasz Niezgoda, Surfer SEO’s Marketing Executive and Partner, how this partnership came to be.

“Pretty straightforward,” he said. “Semrush reached out to us with a proposition to integrate Surfer inside their marketplace [and] after some time, we decided to give it a shot and started working on this integration.”

From the beginning, Semrush felt like this would be a successful integration because it was applying its backlink data in a customer-oriented way.

“For Semrush, it’s a very meaningful integration. It allows Surfer SEO users to gain valuable link building insights and knowledge, which is crucial for ranking their content,” said Eugene Levin, President, Semrush.

What This Means

Two major SEO competitors are working together to create a free tool for small business owners and entry-level marketers to develop weekly best practices.

See also  25 Best About Us Page Examples On The Web

Matt Diggity, CEO and Founder at Diggity Marketing, calls the tool “Simple. Efficient. Automated.”

“You get the list every week and gain immediate insights [into] which referrals your competition is getting. Then, you can start working on your own link-building strategy straight away.”

Advertisement

The two giants hint that combining Semrush’s backlink data with the machine learning power of Surfer is only the beginning.

“We like each other,” said Niezgoda. “Maybe it’s just the first integration that’s coming.”


Featured Image: Screenshot from Grow Flow, June 2022

fbq('track', 'PageView');

fbq('trackSingle', '1321385257908563', 'ViewContent', { content_name: 'sej-exclusive-surfer-seo-unveils-semrush-integration', content_category: 'news digital-marketing-tools' }); }



Source link

Continue Reading

DON'T MISS ANY IMPORTANT NEWS!
Subscribe To our Newsletter
We promise not to spam you. Unsubscribe at any time.
Invalid email address

Trending

en_USEnglish