Connect with us


What It Is & How to Use It



What It Is & How to Use It

Diagnosing technical issues on your website can be one of the most time-consuming but important aspects of running a website.

To make things worse, Google only allows you to inspect one URL at a time to diagnose potential issues on your website (this is done within Google Search Console).

Luckily, there is now a faster way to test your website: enter the Google Search Console URL Inspection API…

What is the Google Search Console URL Inspection API?

The Google Search Console URL Inspection API is a way to bulk-check the data that Google Search Console has on URLs. Its purpose is to help developers and SEOs more efficiently debug and optimize their pages using Google’s own data.

Here’s an example of me using the API to check whether a few URLs are indexed and submitted in my sitemap:

In table form, API showing Chris' URLs are indexed but not submitted in sitemap

What type of data can you get from the Google Search Console URL Inspection API?

The Google Search Console URL Inspection API allows you to pull a wide range of data. Below is a list of some of the most discussed features:


With this field, you can understand exactly when Googlebot last crawled your website. This is extremely useful for SEOs and developers to measure the frequency of Google’s crawling of their sites. Previously, you could only get access to this type of data through log file analysis or spot-checking individual URLs with Google Search Console.


With this field, you can understand whether you have any robots.txt rules that will block Googlebot. This is something you can check manually, but being able to test it at scale with Google’s own data is a fantastic step forward.


googleCanonical and userCanonical

In some situations, Google has been known to select a different canonical from the one that has been specified in the code. In this situation, having the ability to compare both (side by side and at scale) using the API is useful for enabling you to make the appropriate changes.


This field allows you to understand which user agent is used for a crawl of your site: Mobile/Desktop. The response codes are below for reference:

  • DESKTOP – Desktop user agent
  • MOBILE – Mobile user agent


Understanding the pageFetchState can help you diagnose server errors, not found 4xxs, soft 404s, redirection errors, pages blocked by robots.txt, and invalid URLs. A list of responses is below for reference.

Field What it means
SUCCESSFUL Successful fetch
SOFT_404 Soft 404
BLOCKED_ROBOTS_TXT Blocked by robots.txt
NOT_FOUND Not found (404)
ACCESS_DENIED Blocked due to unauthorized request (401)
SERVER_ERROR Server error (5xx)
REDIRECT_ERROR Redirection error
ACCESS_FORBIDDEN Blocked due to access forbidden (403)
BLOCKED_4XX Blocked due to other 4xx issue (not 403, 404)


The indexing state tells you the current status of indexation for the URLs. Apart from the more obvious Pass and Fail responses, there are other responses:

  • NEUTRAL is equivalent to the “Excluded” message in Search Console.
  • PARTIAL is equivalent to the “Valid with warnings” message in Search Console.
  • VERDICT_UNSPECIFIED means that Google is unable to come to a conclusion about the URL(s) in question.


This gives you detail on whether a URL has been submitted in your sitemap and indexed.


This allows you to see where each page is linked from, according to Google.


This enables you to understand which URLs are included in the sitemap(s).

Other uses for the API

You can also use the API to inspect your AMP site—if you have one.

How to use the Google Search Console URL Inspection API step by step

Using the Google Search Console URL Inspection API involves making a request to Google. The request parameters you need to define are the URL that you want to inspect and also the URL of the property in Google Search Console.

The request body contains data with the following structure:

Structure of data

If you are curious to learn more about how to use the API, Google has extensive documentation about this.

Below is an example of the type of response you can get from the API:

Code response from API

If you’re not comfortable with code or just want to give it a go straight away, you can use’s free Google Bulk Inspect URLs tool. The tool provides a quick way to query the API without any coding skills!

Here’s how to use it. You can:

  1. Go to, authorize access to your Google account, and select the Search Console property you want to test. Then paste your URLs into the box below. (The data will be processed in your browser and not uploaded to a server or shared with anyone.)
  2. Click the “Inspect URLs” button. The data will start to pull from the API.
  3. Export the data as a CSV or Excel file by clicking the button.
  4. Analyze the data and check for any potential issues.

How can you use the Google Search Console URL Inspection API in practice?

In theory, the Google Search Console URL Inspection API seems like a great way to understand more about your website. However, you can pull so much data that it’s difficult to know where to start. So let’s look at a few examples of use cases.

1. Site migration – diagnosing any technical issues

Site migrations can cause all kinds of issues. For example, developers can accidentally block Google from crawling your site or certain pages via robots.txt.

Luckily, the Google Search Console URL Inspection API makes auditing for these issues a doddle.

For example, you can check whether you’re blocking Googlebot from crawling URLs in bulk by calling robotsTxtState.

Here is an example of me using the Google Search Console URL Inspection API (via to call robotsTxtState to see the current status of my URLs.

In table form, API showing status of Chris' URLs as "allowed"

As you can see, these pages are not blocked by robots.txt, and there are no issues here.


 Site migrations can sometimes lead to unforeseen technical SEO issues. After the migration, we recommend using a tool, such as Ahrefs’ Site Audit, to check your website for over 100 pre-defined SEO issues.

2. Understand if Google has respected your declared canonicals

If you make a change to the canonical tags across your site, you will want to know whether or not Google is respecting them.


You may be wondering why Google ignores the canonical that you declared. Google can do this for a number of reasons, for example:

  • Your declared canonical is not https. (Google prefers https for canonicals.)
  • Google has chosen a page that it believes is a better canonical page than your declared canonical.
  • Your declared canonical is a noindex page.

Below is an example of me using the Google Search Console URL Inspection API to see whether Google has respected my declared canonicals:

In table form, API showing Google respects Chris' canonicals

As we can see from the above screenshot, there are no issues with these particular pages and Google is respecting the canonicals.


To quickly see if the googleCanonical matches the userCanonical, export the data from the Google Bulk Inspect URLs tool to CSV and use an IF formula in Excel. For example, assuming your googleCanonical data is in Column A and your userCanonical is in Column B, you can use the formula =IF(A2=B2, “Self referencing”,”Non Self Referencing”) to check for non-matching canonicals.

3. Understand when Google recrawls after you make changes to your site

When you update many pages on your website, you will want to know the impact of your efforts. This can only happen after Google has recrawled your site.

With the Google Search Console URL Inspection API, you can see the precise time Google crawled your pages by using lastCrawlTime.

If you can’t get access to the log files for your website, then this is a great alternative to understand how Google crawls your site.

Here’s an example of me checking this:

In table form, API showing "last crawl" date and time for each URL

As you can see in the screenshot above, lastCrawlTime shows the date and time my website was crawled. In this example, the most recent crawl by Google is the homepage.

Understanding when Google recrawls your website following any changes will allow you to link whether or not the changes you made have any positive or negative impact following Google’s crawl.



How to get around the Google Search Console URL Inspection API limits?

Although the Google Search Console URL Inspection API is limited to 2,000 queries per day, this query limit is determined by Google Property.

This means you can have multiple properties within one website if they are verified separately in Google Search Console, effectively allowing you to bypass the limit of 2,000 queries per day.

Google Search Console allows you to have 1,000 properties in your Google Search Console account, so this should be more than enough for most users.

Can I use the Google Search Console URL Inspection API on any website?

Another potential limiting factor is you can only run the Google Search Console URL Inspection API on a property that you own in Google Search Console. If you don’t have access to the property, then you cannot audit it using the Google Search Console URL Inspection API.

So this means auditing a site that you don’t have access to can be problematic.

How accurate is the data?

Accuracy of the data itself has been an issue for Google over the last few years. This API gives you access to that data. So arguably, the Google Search Console URL Inspection API is only as good as the data within it.

Tim Guillot shares tweet to John Mu about data issues; Tim adds the issues are "not fixed"

As we have previously shown in our study of Google Keyword Planner’s accuracy, data from Google is often not as accurate as people assume it to be.

Final thoughts

The Google Search Console URL Inspection API is a great way for site owners to get bulk data directly from Google on a larger scale than what was previously possible from Google Search Console.


Daniel Waisberg and the team behind the Google Search Console URL Inspection API have definitely done a great job of getting this released into the wild.

But one of the criticisms of the Google Search Console URL Inspection API from the SEO community is that the query rate limit is too low for larger sites. (It is capped at 2,000 queries per day, per property.)

For larger sites, this is not enough. Also, despite the possible workarounds, this number still seems to be on the low side.

What’s your experience of using the Google Search Console URL Inspection API? Got more questions? Ping me on Twitter. 🙂

Source link



A Guide To Social Media Algorithms & How They Work



A Guide To Social Media Algorithms & How They Work

Why do so many marketers keep asking, “How do social media algorithms work?” Because the algorithms for the major platforms can change quickly.

But, marketers should also keep asking, “Which social media platforms have the most users?” Because that data can change frequently, as well.

So, here are the latest answers to the first question about the algorithms for the eight platforms that you should be considering today.

Spoiler alert: This update contains some surprising shifts in the latest data on monthly unique visitors, monthly visits, and monthly average visit duration from SimilarWeb.

How Does The YouTube Algorithm Work?

YouTube got 1.953 billion unique visitors worldwide in May 2022. The platform received 35.083 billion monthly visits that month with an average visit duration of 21:41.

Now, some social media marketers may be shocked, shocked to find YouTube ranking ahead of Facebook.

But, SimilarWeb’s data above is only for desktop and mobile web channels. It doesn’t include data for connected TVs, which became the fastest-growing screen among YouTube viewers in 2020.


This makes it imperative to know how YouTube’s algorithm works.

YouTube’s algorithm tries to match each viewer to the videos they’re most likely to watch and enjoy. But, with over 500 hours of video content uploaded every minute, this is a Herculean task.

YouTube’s search and discovery systems tackle this challenge by paying close attention to:

  • What viewers watch.
  • What they don’t watch.
  • How much time do they spend watching?
  • What do they share and like?

Next, you need to learn that YouTube has multiple algorithms, including ones for:

  • YouTube Search: Videos are ranked based on how well titles, descriptions, and video content match the viewer’s search and which videos get the most engagement for a search.
  • Up Next: The ranking of suggested videos is based on machine learning’s understanding of which ones viewers are most likely to watch next. These videos are often related to the video a viewer is watching, but they can also be personalized based on the viewer’s watch history.
  • Your homepage: Videos are selected based on how often viewers watch a channel or topic, how well similar videos have interested and satisfied similar viewers, and how many times YouTube has already shown each video to a viewer.
  • YouTube Shorts: YouTube wants both short and long videos to succeed. So, relative watch time is generally more important for short videos, while absolute watch time is generally more important for longer videos.

So, what should you do next?

First, read my column, How To Optimize YouTube Videos To Help Ukraine, which provides tips on keyword research, title optimization, writing descriptions, custom thumbnails, and other video SEO best practices.

Next, read Jon Clark’s article, 13 Key Elements Of Successful YouTube Videos. He focuses on how to make a great video.

Why is that important? Because YouTube’s search and discovery system “finds” videos for each viewer and their varying interests in order to get them to watch more videos that they’ll enjoy so they’ll come back to YouTube regularly.

How Does The Facebook Algorithm Work?

Facebook got only 1.620 billion unique visitors worldwide in May. The platform received 19.739 billion visits that month with an average session duration of 10:05.

Now, Facebook’s unique visitors started dipping worldwide in February 2022.


But, as you can see in the chart below, there was a substantial drop in unique visitors in Russia in early March, after Russia blocked Facebook in an effort to control the spread of information on the invasion of Ukraine.

Screenshot courtesy of Similarweb, June 2022

This had a negative impact on Facebook’s total unique visitors worldwide, which were already losing momentum. Nevertheless, the platform is still too big to ignore.

So, how does Facebook’s algorithm work today?

Well, we knew how Facebook’s News Feed ranking process worked in December 2021 when Anna Stepanov, Head of Facebook App Integrity, wrote a post that said:

“News Feed uses personalized ranking, which takes into account thousands of unique signals to understand what’s most meaningful to you. Our aim isn’t to keep you scrolling on Facebook for hours on end, but to give you an enjoyable experience that you want to return to.”

And she summarized half a dozen of the biggest changes Facebook had made in 2021 to give users more control over, and insight into, how content appears in their News Feed.

This included publishing a new series of Widely Viewed Content Reports to share what content is seen by the most people in News Feed in the U.S.

Ironically, Facebook’s latest Widely Viewed Content Report showed the top four domains in Q4 2021:

  • (168.1 million content viewers).
  • (118.4 million).
  • (112.4 million).
  • (105.0 million).

But, then in February 2022, Matt G. Southern reported Facebook Shifts Focus To Short-Form Video After Stock Plunge. And on June 16, 2022, Southern reported Facebook To Restructure Main Feed Around Video Content.

So, what should you do next? First, read Southern’s stories and learn why Tom Alison, head of Facebook, plans to turn its main feed into a “discovery engine” for video content.

According to Alison, the main tab in the Facebook app will become a mix of Stories and Reels at the top, followed by posts that its discovery engine will recommend from across both Facebook and Instagram.


Next, follow Southern’s expert, authoritative, and trustworthy advice:

“The best way to prepare for this change, if Facebook is a priority for you and your business, is to get comfortable with creating and publishing more short form video. While Facebook will continue to surface text and photo posts, they’ll be ancillary to the main attractions of Reels and Stories.”

How Does The Instagram Algorithm Work?

Instagram got 1.050 billion unique visitors worldwide in May. The platform received 6.497 billion visits that month with an average session duration of 07:51.

Russia has also banned Instagram, but the growth in unique visitors from other countries around the world has offset that.

So, you still need to know how Instagram’s algorithms work.

In June 2021, Adam Mosseri, the head of Instagram, wrote a post entitled, Shedding More Light On How Instagram Works. He revealed:

“Instagram doesn’t have one algorithm that oversees what people do and don’t see on the app. We use a variety of algorithms, classifiers, and processes, each with its own purpose.”

For the Feed and Stories, the key ranking signals are:

  • Information about the post: How popular a post is, when it was posted, how long it is, if it’s a video, and if it’s attached to a location.
  • Information about the person who posted: How many times users have interacted with that person in the past few weeks.
  • User activity: What a user might be interested in and how many posts they’ve liked.
  • User history of interacting with someone: How interested a user is in seeing posts from a particular person.

For Explore, the key ranking signals are:

  • Information about the post: How popular a post seems to be as well as how many and how quickly other people are liking, commenting, sharing, and saving a post.
  • User history of interacting with someone: (See above.)
  • User activity: What posts a user has liked, saved, or commented on as well as how they’ve interacted with posts in Explore in the past.
  • Information about the person who posted: (See above.)

For Reels, the key ranking signals are:

  • User activity: Which Reels a user has liked, commented on, and engaged with recently.
  • User history of interacting with someone: (See above.)
  • Information about the reel: The audio track, video data such as pixels and whole frames, as well as popularity.
  • Information about the person who posted: (See above.)

So, each part of the app uses similar ranking signals, but their order of importance varies. Mosseri explained:

“People tend to look for their closest friends in Stories, but they want to discover something entirely new in Explore. We rank things differently in different parts of the app, based on how people use them.”

For more tips and advice, read the article by Shelley Walsh entitled, 22 Ways To Get More Instagram Followers Right Now. Then, read Amanda DiSilvestro’s article, How To Use Instagram Reels For Business.

How Does The Twitter Algorithm Work?

Twitter got 979 million unique visitors worldwide in May. The platform received 7.056 billion visits that month with an average session duration of 10.39.


This data does not screen for fake or spam accounts. Nevertheless, it’s worth investing the time and effort to keep up with how Twitter’s algorithm works.

Like most social media platforms, Twitter has multiple algorithms.

Twitter says its “algorithmic Home timeline displays a stream of Tweets from accounts you have chosen to follow on Twitter, as well as recommendations of other content we think you might be interested in based on accounts you interact with frequently, Tweets you engage with, and more.”

If users want to, they can click on the star symbol to see the latest Tweets as they happen. But, few people choose to drink water from a firehose.

If they want to, users can click on “Explore” and see Trending tweets or ones about COVID-19, News, Sports, and Entertainment.

If users want to, they can click on “More” to see the Topics that Twitter thinks they’re interested in.

Like most social media platforms, Twitter’s algorithms use machine learning to sort content based on different ranking signals.

And it’s worth noting that Twitter is currently involved in analyzing the results of its algorithms as part of its “responsible machine learning initiative.”


Here’s what Twitter has said publicly about its Home timeline, Trends, and Topics ranking signals:


  • ​​Users’ previous actions on Twitter, like their own Tweets and Tweets they’ve engaged with.
  • Accounts they often engage with.
  • Topics they follow and engage with most.
  • The number of Tweets related to a topic.
  • For Trends: their location.


  • For Tweets: “How popular it is and how people in your network are interacting with [the Tweet].”
  • For Trends: “The number of Tweets related to the Trend.”
  • For Topics: “How much people are Tweeting, Retweeting, replying, and liking Tweets about that Topic.”


  • For Trends: “Topics that are popular now, rather than topics that have been popular for a while or on a daily basis.”

Rich Media:

  • The type of media the Tweet includes like an image, video, GIF, and polls.

For more advice and tips, read Lisa Buyer’s article, 8 Terrific Tips To Optimize A Twitter Business Or Brand Profile. Then, read the article by Julia McCoy entitled, How To Be A Top Tweeter: 10 Tips That Will Get Your Tweets Noticed.

How Does The TikTok Algorithm Work?

TikTok got 690 million monthly visitors worldwide in May. The platform received 1.766 billion visits that month with an average session duration of 03:48.

This data doesn’t include, which is counted separately. But, as the chart below illustrates, gets about 98% of the unique visitors worldwide for both of the ByteDance apps. gets about 98% of the unique visitors worldwideScreenshot courtesy of Similarweb, June 2022

So, you should probably learn how TikTok’s algorithm works ASAP.

In June 2020, TikTok revealed how its recommendation system selected videos in a post entitled, How TikTok recommends videos #ForYou.

Little has fundamentally changed since then, except the U.S. government is no longer trying to ban the social media platform.

TikTok’s For You feed presents a stream of videos curated to each user’s interests, making it easy for a user to find content and creators they love.

In other words, there isn’t one For You feed for over one billion monthly active TikTok users. There are a billion For You feeds tailored to what each user watches, likes, and shares.


TikTok added, “This feed is powered by a recommendation system that delivers content to each user that is likely to be of interest to that particular user.”

And recommendations are based on a number of factors, including:

  • User interactions such as the videos they like or share, accounts they follow, comments they post, and content they create.
  • Video information, which might include details like captions, sounds, and hashtags.
  • Device and account settings like their language preference, country setting, and device type.

TikTok also revealed:

“All these factors are processed by our recommendation system and weighted based on their value to a user. A strong indicator of interest, such as whether a user finishes watching a longer video from beginning to end, would receive greater weight than a weak indicator, such as whether the video’s viewer and creator are both in the same country.

Videos are then ranked to determine the likelihood of a user’s interest in a piece of content, and delivered to each unique For You feed.”

On the other hand, TikTok said:

“While a video is likely to receive more views if posted by an account that has more followers, by virtue of that account having built up a larger follower base, neither follower count nor whether the account has had previous high-performing videos are direct factors in the recommendation system.”

So, what should you do next? First, read Miranda Miller’s article, 40+ TikTok Stats Digital Marketers Need To Know. Then, read my column, How TikTok’s Search Algorithms Power Content Discovery.

How Does The Pinterest Algorithm Work?

Pinterest got 409 million unique visitors worldwide in May. The platform received 945 million visits that month with an average session duration of 05:29.

With Instagram declaring it is “no longer just a square photo-sharing app,” this is the time to learn how Pinterest’s algorithm works.

The ranking factors on Pinterest relate more to engagement metrics and social shares, but it also involves keywords.


And Pinterest autocomplete provides ideas by automatically suggesting semantically related modifiers to a core keyword.

Pinterest’s search feature then curates a user’s “feed” based on what they’re searching for and how those key terms are used in the Pins being shared by content creators.

Pinterest also categorizes and sub-categorizes topics to make it easy to find keywords for your particular niche.

To optimize your Pins:

  • Use long images: The optimal Pin size is 1,000 by 1,500 px or a ratio of 2:3.
  • Use eye-catching colors: Catch users’ attention and stand out with high-contrast colors.
  • Use enticing, keyword-rich titles: Entice users to click through to your content.
  • Use detailed descriptions: Include your target keywords in your descriptions.

Then, optimize your boards. Boards provide a great opportunity to tell Pinterest’s search engine how you categorize your products and/or organize your content, which will only aid visibility.

Finally, aim for engagement, which can increase your Pin’s (and your profile’s) visibility in search, increasing your traffic.

For additional information and advice, read Southern’s story, Pinterest Updates Algorithm To Surface More Content Types. Then, read Jessica Foster’s article, 12 Pinterest SEO Tips For High-Traffic Success.

How Does The LinkedIn Algorithm Work?

LinkedIn got 306 million unique visitors worldwide in May. The platform received 1.479 billion visits that month with an average session duration of 07:32.

So, social media marketers – especially ones at B2B organizations – need to know how LinkedIn’s algorithm works.


In June 2019, Pete Davies, Senior Director of Product Management at LinkedIn, wrote a post entitled, What’s in your LinkedIn Feed: People You Know, Talking about Things You Care About. He explained, “The more valuable the conversation, the higher in your feed the post will be.”

How does LinkedIn’s algorithm know if a conversation is valuable? It uses the following framework:

  • People you know: LinkedIn’s algorithm looks at a user’s connections and prioritizes who they’ve interacted with directly through comments and reactions; the user’s implicit interests and experiences based on information in their profile; explicit signals, such as who a user works with; as well as who would benefit from hearing from the user.
  • Talking about: A lot of sophistication goes into understanding a good conversation. As a rule of thumb, better conversations are authentic and have a constructive back and forth.
  • Things you care about: LinkedIn’s algorithm also looks at whether the content and the conversation are relevant and interesting to a user. It considers a number of signals, including joining groups and following hashtags, people, and pages.

So, what should you do next? First, read Jessica Foster’s article, How The LinkedIn Algorithm Works & Optimizing For It. Then, read Matt G. Southern’s article, LinkedIn Debunks Algorithm Myths In New Video Series.

How Does The Reddit Algorithm Work?

Reddit got 237 million unique visitors worldwide in May. The platform received 1.669 billion visits that month with an average session duration of 09:59.

With Facebook setting its sights on video to regain its momentum, this is a good time to learn how Reddit’s algorithm works.

In June 2021, the official blog for Reddit posted Evolving the Best Sort for Reddit’s Home Feed. It provided insights into how Reddit determines which relevant posts to show users.

The post revealed that:

“Reddit’s systems build a list of potential candidate posts from multiple sources, pass the posts through multiple filtering steps, then rank the posts according to the specified sorting method. Over the years, we’ve built many options to choose from when it comes to sorting your Home feed.”

Here’s how each sort option recommends content:

  • “Hot” ranks using votes and post age.
  • “New” displays the most recently published posts.
  • “Top” shows users the highest vote count posts from a specified time range.
  • “Controversial” shows posts with both high count upvotes and downvotes.
  • “Rising” populates posts with lots of recent votes and comments.
  • ‘Best” uses machine learning algorithms to personalize the order in which users see posts.

For more tips and information, read the article by Brent Csutoras entitled, A Beginner’s Guide To Reddit: How To Get Started & Be Successful. Then, read Southern’s story, Reddit Makes Comments Searchable.

Why Should You Keep Asking Questions?

The latest data from SimilarWeb indicates that you should continue asking “Which social media platforms have the most users?” as well as “How do social media algorithms work?”


Things change too quickly and frequently in this particular arena for anyone to think that past performance is even remotely indicative of future results.

More Resources:

Featured Image:

fbq('track', 'PageView');

fbq('trackSingle', '1321385257908563', 'ViewContent', { content_name: 'how-social-media-algorithms-work', content_category: 'social social-strategy' }); }

Source link

Continue Reading

Subscribe To our Newsletter
We promise not to spam you. Unsubscribe at any time.
Invalid email address