Connect with us


Everything We Know About IndexNow So Far



Everything We Know About IndexNow So Far

Hopefully, you’ve heard about IndexNow at this point.

We covered its announcement in October and in December, I had Fabrice Canel, Principal Program Manager for Microsoft Bing, on the SEJ Show to talk about this revolution in how search engines discover new and updated content.

But what is it, exactly, and what do you need to know about it for SEO?

Here’s what we know so far.

Why We Need IndexNow

Search engines have been discovering content the same way for a long time.

Way back in 1993, JumpStation became the first internet discovery tool to send out bots to follow (or crawl) links from website to website and create a content map, or index.

Within the next few years, other search engines adopted this process as a way to find new content.


This process was included in an algorithm that also considered content, relevancy, and other factors – and search engine optimization as we know it was born.

It didn’t take long before SEO-savvy web publishers recognized the importance of indexing in securing high search engine rankings. In fact, without indexed content, a website won’t generate organic traffic.

But therein lies the rub: Unless you’re a large, highly trafficked website, it may be days or even weeks before a search engine crawls your site and indexes new pages. Heck, even if you are a large ecommerce site or marketplace, it may take engines weeks to pick up new arrivals or out of stock products as well – especially when depending on discoverability with basic crawls.

Meanwhile, you may have lost important traffic to competitors.

To streamline this process, Microsoft Bing released a new initiative in October 2021 called IndexNow.

So What Is IndexNow?

IndexNow is an open-source protocol that allows website publishers to instantly index across participating search engines, updating results based upon the latest content changes.

Simply put, it’s a simple ping that notifies search engines that a URL and its content have been added, updated, or deleted.

By eliminating the need for exploratory crawls, IndexNow seeks to improve overall internet efficiency.


How Does IndexNow Work?

There are two different ways search engines obtain index data: Pull and push.

See also  IndexNow Now Sharing URLs Between Search Engines & IndexNow API

Pull indexing occurs when a search engine visits your site to request web pages and gathers data from the server.

This is how search engines traditionally operate.

Push indexing is when the web publisher or content management system notifies the search engine that pages have been added or deleted or other changes have been made.

That’s what’s truly groundbreaking about the IndexNow – it allows all URLs submitted to any IndexNow-enabled search engine to be simultaneously submitted to all the other search engines using the protocol.

You can read more about the technical aspects of how IndexNow works here.

Which Search Engines Use IndexNow?

At the time of publication, Bing and Yandex are using IndexNow.

A handful of others are reportedly testing the protocol, according to an unnamed expert.


So far, DuckDuckGo has not announced any plans to adopt IndexNow.

However, it’s worth noting that the privacy-focused search engine draws from up to 400 sources in addition to using its own DuckDuckBot web crawler in the compilation of its search results.

Given that Bing and Yandex are among those sources, you could see updated content appearing in DuckDuckGo quicker due to IndexNow, as well.

Even Bing’s largest competitor, Google, the foremost of all search engines, have confirmed they will be testing the IndexNow protocol.

And while they haven’t gone so far as announce whether or not they will adopt it, the fact they are experimenting with it shows a recognition of its capabilities.

Google’s top concerns about IndexNow seem to be related to sustainability and efficiency.

Google is currently using HTTP/2, an efficient fundamental data transfer protocol for more than half of all crawls, and may ultimately decide this is a better fit for their needs.

Or they may decide to build their own alternative if they determine an API-based approach is needed – and it’s not a foregone conclusion that they will.


But not everyone is on board.

There are plenty of people in the SEO community who don’t see the need for this protocol, instead claiming XML sitemaps already accomplish what IndexNow is trying to do.

See also  How To Increase Agency Revenue With New Services & Partnerships

You can read more about the debate here.

How Do You Use IndexNow?

Notifying search engines of updates to your web content using IndexNow is a straightforward process:

  1. Generate an API Key – This is submitted alongside URLs to ensure ownership of the domain. You can do this using an online key generator tool like the one found here.
  2. Host API Key – Your API key is hosted on the root directory in txt format.
  3. Submit URLs with Parameters – You can either submit URLs individually or in bulk. Submit your key location as a parameter.
  4. Check Your Submissions – Using the Bing Webmaster Tools portal, you can verify which URLs have been submitted and discovered.

For examples and more in-depth instructions, you can visit the IndexNow site.

More advanced users can use Python for bulk indexation and URL submission automation.

This requires a basic understanding of Python syntax and familiarity with Python libraries and packages.

We have a handy step-by-step guide to walk you through this process here.

Which Tools Sync With IndexNow?

Any website developer can use the process outlined here to take advantage of IndexNow, but what’s really exciting is that you don’t have to know HTML from HDMI to reap the benefits.


A number of large companies, including LinkedIn, MSN, and GitHub are all planning migrations to IndexNow.

Additionally, other search solution providers like Botify, OnCrawl, Onely and Yext are in the process of adopting this protocol.

Content management systems (CMS) are also jumping on board.

WordPress already offers an IndexNow plugin. Duda supports this proactive approach to web crawling, and Wix plans to integrate soon.

Bing recently announced Rank Math and All In One SEO integration for IndexNow, as well.

Content delivery networks (CDN) supporting IndexNow include Cloudflare and Akamai.

Through their proxy servers, this will speed up data delivery and bring it closer to users.

What Are The Benefits Of Using IndexNow?

In addition to emphasizing the evolution of indexing from pull to push, the primary benefit of IndexNow for publishers is the elimination of the time between updates being made and search engines discovering them.


By allowing webmasters to notify all participating URLs with a single API call, search engine content discovery is streamlined, which in turn makes the web more efficient.

See also  How To Use IndexNow API With Python For Bulk Indexing

This is advantageous to site owners, as growing requirements are reduced on the server.

Search engine spiders no longer have to perform exploratory crawls to determine whether a page has been updated.

This enables servers to work more efficiently without the need for a redundant site load, which in turn reduces energy usage.

And unlike an XML sitemap, which only redirects SEO crawlers to pages within your website, IndexNow allows you to inform search engines of non-200 status code pages.

IndexNow Frequently Asked Questions

What Happens After I Submit A URL?

IndexNow conveys your changes to search engines, which in turn apply them to page rankings.

Using IndexNow doesn’t guarantee web pages will be crawled or index and it may take time for changes to be reflected.

How Many URLs Can I submit Using IndexNow?

You can provide search engines with a list of up to 10,000 URLs with a single API call.


Will I Rank Better In Search Results If I Index My Content Faster?

Yes, the sooner your pages are indexed, the more quickly they can start competing for high-ranking search engine spots.

What Does All This Mean?

The march of technology is inexorable.

And as such a vital part of the modern world, it’s only natural that changes to the internet, and in particular search engines, would be part of them.

By shifting the process of indexing sites away from a crawling process to something faster and more efficient, IndexNow seems to be on the cusp of a revolution.

And the fact that companies like MSN, GitHub, and WordPress are all joining in seems to hint that this protocol will soon become the norm.

More resources:

Featured Image: ra2 studio/Shutterstock

!function(f,b,e,v,n,t,s) {if(f.fbq)return;n=f.fbq=function(){n.callMethod? n.callMethod.apply(n,arguments):n.queue.push(arguments)}; if(!f._fbq)f._fbq=n;n.push=n;n.loaded=!0;n.version='2.0'; n.queue=[];t=b.createElement(e);t.async=!0; t.src=v;s=b.getElementsByTagName(e)[0]; s.parentNode.insertBefore(t,s)}(window,document,'script', '');

if( typeof sopp !== "undefined" && sopp === 'yes' ){ fbq('dataProcessingOptions', ['LDU'], 1, 1000); }else{ fbq('dataProcessingOptions', []); }

fbq('init', '1321385257908563');

fbq('track', 'PageView');

fbq('trackSingle', '1321385257908563', 'ViewContent', { content_name: 'indexnow-what-we-know', content_category: 'seo ' });

Source link


Surfer SEO Unveils New Semrush Integration



Surfer SEO Unveils New Semrush Integration

Surfer SEO has a new growth management tool that includes backlink data provided by industry leader, Semrush.

Here’s what the integration means for users.

Surfer SEO

Surfer SEO is a machine learning tool used by SEOs to conduct keyword research, create content strategies, and generate AI outlines.

Companies such as Square, ClickUp, and Shopify use the tool, in addition to over 15,000 other brands and agencies.


Semrush is an enterprise SEO tool marketers use to audit websites, conduct competitor analysis and develop content strategies.

IBM, TESLA, and Amazon are just a few companies using it, in addition to more than 10 million marketing professionals.

Starting today, Surfer SEO will use Semrush’s backlink data to expand Grow Flow task recommendations.


Grow Flow

Grow Flow is like your friendly SEO AI assistant.

Described on the Surfer website as an “AI growth management platform,” the tool provides a few SEO tasks each week to help you stay on track — and not get lost in the overwhelming amount of information that comes with learning SEO.

For example, it may recommend adding keywords (GSC) that a website is ranking for but not explicitly speaking to in an article.

Screenshot from Grow Flow, June 2022

Or, the tool may recommend where to add internal links.

internal links_growflow_surfer seoScreenshot from Grow Flow, June 2022

It can also recommend new content topics that users can open in a content editor at the push of a button.

new content topics_content editor_growflowScreenshot from Grow Flow, June 2022

The integration with Semrush comes into play in the Grow Flow recommendations.

See also  11 Ways to Build a Google Algorithm Update Resistant SEO Strategy

Once you connect Surfer to the freemium Semrush account, you unlock new tasks.

connect semrush to surfer_growflowScreenshot from Grow Flow, June 2022

Voila – a list of referring domains every week pulled from your competitors!

Backlink research automated_Growflow_Surfer SEOScreenshot from Grow Flow, June 2022

Manually researching a competitor’s backlinks is a considerable drain on internal resources. Thanks to this integration, you’ll discover new opportunities automatically.

An Industry First

Until now, Semrush has not integrated with an SEO company.

Historically, Semrush has only integrated with Google Products like Search Console and Analytics, and social media networks like Facebook and Twitter.

It’s also worked with task management tools like Trello and – but never a direct SEO competitor.

Semrush’s “SEO writing assistant” feature is a direct competitor to Surfer’s “SEO Content Editor”: one of its most popular features.


So, why partner and offer this fantastic AI assistant to search marketers – for free?

I asked Tomasz Niezgoda, Surfer SEO’s Marketing Executive and Partner, how this partnership came to be.

“Pretty straightforward,” he said. “Semrush reached out to us with a proposition to integrate Surfer inside their marketplace [and] after some time, we decided to give it a shot and started working on this integration.”

From the beginning, Semrush felt like this would be a successful integration because it was applying its backlink data in a customer-oriented way.

“For Semrush, it’s a very meaningful integration. It allows Surfer SEO users to gain valuable link building insights and knowledge, which is crucial for ranking their content,” said Eugene Levin, President, Semrush.

What This Means

Two major SEO competitors are working together to create a free tool for small business owners and entry-level marketers to develop weekly best practices.

See also  Tips For Avoiding Misinformation In SEO Resources & Conversations

Matt Diggity, CEO and Founder at Diggity Marketing, calls the tool “Simple. Efficient. Automated.”

“You get the list every week and gain immediate insights [into] which referrals your competition is getting. Then, you can start working on your own link-building strategy straight away.”


The two giants hint that combining Semrush’s backlink data with the machine learning power of Surfer is only the beginning.

“We like each other,” said Niezgoda. “Maybe it’s just the first integration that’s coming.”

Featured Image: Screenshot from Grow Flow, June 2022

fbq('track', 'PageView');

fbq('trackSingle', '1321385257908563', 'ViewContent', { content_name: 'sej-exclusive-surfer-seo-unveils-semrush-integration', content_category: 'news digital-marketing-tools' }); }

Source link

Continue Reading

Subscribe To our Newsletter
We promise not to spam you. Unsubscribe at any time.
Invalid email address