Connect with us

SEO

What It Is & How to Use It

Published

on

What It Is & How to Use It

Diagnosing technical issues on your website can be one of the most time-consuming but important aspects of running a website.

To make things worse, Google only allows you to inspect one URL at a time to diagnose potential issues on your website (this is done within Google Search Console).

Luckily, there is now a faster way to test your website: enter the Google Search Console URL Inspection API…

What is the Google Search Console URL Inspection API?

The Google Search Console URL Inspection API is a way to bulk-check the data that Google Search Console has on URLs. Its purpose is to help developers and SEOs more efficiently debug and optimize their pages using Google’s own data.

Here’s an example of me using the API to check whether a few URLs are indexed and submitted in my sitemap:

In table form, API showing Chris' URLs are indexed but not submitted in sitemap

What type of data can you get from the Google Search Console URL Inspection API?

The Google Search Console URL Inspection API allows you to pull a wide range of data. Below is a list of some of the most discussed features:

lastCrawlTime

With this field, you can understand exactly when Googlebot last crawled your website. This is extremely useful for SEOs and developers to measure the frequency of Google’s crawling of their sites. Previously, you could only get access to this type of data through log file analysis or spot-checking individual URLs with Google Search Console.

robotsTxtState

With this field, you can understand whether you have any robots.txt rules that will block Googlebot. This is something you can check manually, but being able to test it at scale with Google’s own data is a fantastic step forward.

googleCanonical and userCanonical

In some situations, Google has been known to select a different canonical from the one that has been specified in the code. In this situation, having the ability to compare both (side by side and at scale) using the API is useful for enabling you to make the appropriate changes.

crawledAs

This field allows you to understand which user agent is used for a crawl of your site: Mobile/Desktop. The response codes are below for reference:

  • DESKTOP – Desktop user agent
  • MOBILE – Mobile user agent

pageFetchState

Understanding the pageFetchState can help you diagnose server errors, not found 4xxs, soft 404s, redirection errors, pages blocked by robots.txt, and invalid URLs. A list of responses is below for reference.

Field What it means
PAGE_FETCH_STATE_UNSPECIFIED Unknown fetch state
SUCCESSFUL Successful fetch
SOFT_404 Soft 404
BLOCKED_ROBOTS_TXT Blocked by robots.txt
NOT_FOUND Not found (404)
ACCESS_DENIED Blocked due to unauthorized request (401)
SERVER_ERROR Server error (5xx)
REDIRECT_ERROR Redirection error
ACCESS_FORBIDDEN Blocked due to access forbidden (403)
BLOCKED_4XX Blocked due to other 4xx issue (not 403, 404)
INTERNAL_CRAWL_ERROR Internal error
INVALID_URL Invalid URL

indexingState

The indexing state tells you the current status of indexation for the URLs. Apart from the more obvious Pass and Fail responses, there are other responses:

  • NEUTRAL is equivalent to the “Excluded” message in Search Console.
  • PARTIAL is equivalent to the “Valid with warnings” message in Search Console.
  • VERDICT_UNSPECIFIED means that Google is unable to come to a conclusion about the URL(s) in question.

coverageState 

This gives you detail on whether a URL has been submitted in your sitemap and indexed.

referringUrls

This allows you to see where each page is linked from, according to Google.

Sitemap 

This enables you to understand which URLs are included in the sitemap(s).

Other uses for the API

You can also use the API to inspect your AMP site—if you have one.

How to use the Google Search Console URL Inspection API step by step

Using the Google Search Console URL Inspection API involves making a request to Google. The request parameters you need to define are the URL that you want to inspect and also the URL of the property in Google Search Console.

The request body contains data with the following structure:

Structure of data

If you are curious to learn more about how to use the API, Google has extensive documentation about this.

Below is an example of the type of response you can get from the API:

Code response from API

If you’re not comfortable with code or just want to give it a go straight away, you can use valentin.app’s free Google Bulk Inspect URLs tool. The tool provides a quick way to query the API without any coding skills!

Here’s how to use it. You can:

  1. Go to https://valentin.app/inspect.html, authorize access to your Google account, and select the Search Console property you want to test. Then paste your URLs into the box below. (The data will be processed in your browser and not uploaded to a server or shared with anyone.)
  2. Click the “Inspect URLs” button. The data will start to pull from the API.
  3. Export the data as a CSV or Excel file by clicking the button.
  4. Analyze the data and check for any potential issues.

How can you use the Google Search Console URL Inspection API in practice?

In theory, the Google Search Console URL Inspection API seems like a great way to understand more about your website. However, you can pull so much data that it’s difficult to know where to start. So let’s look at a few examples of use cases.

1. Site migration – diagnosing any technical issues

Site migrations can cause all kinds of issues. For example, developers can accidentally block Google from crawling your site or certain pages via robots.txt.

Luckily, the Google Search Console URL Inspection API makes auditing for these issues a doddle.

For example, you can check whether you’re blocking Googlebot from crawling URLs in bulk by calling robotsTxtState.

Here is an example of me using the Google Search Console URL Inspection API (via valentin.app) to call robotsTxtState to see the current status of my URLs.

In table form, API showing status of Chris' URLs as "allowed"

As you can see, these pages are not blocked by robots.txt, and there are no issues here.

TIP

 Site migrations can sometimes lead to unforeseen technical SEO issues. After the migration, we recommend using a tool, such as Ahrefs’ Site Audit, to check your website for over 100 pre-defined SEO issues.

2. Understand if Google has respected your declared canonicals

If you make a change to the canonical tags across your site, you will want to know whether or not Google is respecting them.

You may be wondering why Google ignores the canonical that you declared. Google can do this for a number of reasons, for example:

  • Your declared canonical is not https. (Google prefers https for canonicals.)
  • Google has chosen a page that it believes is a better canonical page than your declared canonical.
  • Your declared canonical is a noindex page.

Below is an example of me using the Google Search Console URL Inspection API to see whether Google has respected my declared canonicals:

In table form, API showing Google respects Chris' canonicals

As we can see from the above screenshot, there are no issues with these particular pages and Google is respecting the canonicals.

TIP

To quickly see if the googleCanonical matches the userCanonical, export the data from the Google Bulk Inspect URLs tool to CSV and use an IF formula in Excel. For example, assuming your googleCanonical data is in Column A and your userCanonical is in Column B, you can use the formula =IF(A2=B2, “Self referencing”,”Non Self Referencing”) to check for non-matching canonicals.

3. Understand when Google recrawls after you make changes to your site

When you update many pages on your website, you will want to know the impact of your efforts. This can only happen after Google has recrawled your site.

With the Google Search Console URL Inspection API, you can see the precise time Google crawled your pages by using lastCrawlTime.

If you can’t get access to the log files for your website, then this is a great alternative to understand how Google crawls your site.

Here’s an example of me checking this:

In table form, API showing "last crawl" date and time for each URL

As you can see in the screenshot above, lastCrawlTime shows the date and time my website was crawled. In this example, the most recent crawl by Google is the homepage.

Understanding when Google recrawls your website following any changes will allow you to link whether or not the changes you made have any positive or negative impact following Google’s crawl.

FAQs

How to get around the Google Search Console URL Inspection API limits?

Although the Google Search Console URL Inspection API is limited to 2,000 queries per day, this query limit is determined by Google Property.

This means you can have multiple properties within one website if they are verified separately in Google Search Console, effectively allowing you to bypass the limit of 2,000 queries per day.

Google Search Console allows you to have 1,000 properties in your Google Search Console account, so this should be more than enough for most users.

Can I use the Google Search Console URL Inspection API on any website?

Another potential limiting factor is you can only run the Google Search Console URL Inspection API on a property that you own in Google Search Console. If you don’t have access to the property, then you cannot audit it using the Google Search Console URL Inspection API.

So this means auditing a site that you don’t have access to can be problematic.

How accurate is the data?

Accuracy of the data itself has been an issue for Google over the last few years. This API gives you access to that data. So arguably, the Google Search Console URL Inspection API is only as good as the data within it.

Tim Guillot shares tweet to John Mu about data issues; Tim adds the issues are "not fixed"

As we have previously shown in our study of Google Keyword Planner’s accuracy, data from Google is often not as accurate as people assume it to be.

Final thoughts

The Google Search Console URL Inspection API is a great way for site owners to get bulk data directly from Google on a larger scale than what was previously possible from Google Search Console.

Daniel Waisberg and the team behind the Google Search Console URL Inspection API have definitely done a great job of getting this released into the wild.

But one of the criticisms of the Google Search Console URL Inspection API from the SEO community is that the query rate limit is too low for larger sites. (It is capped at 2,000 queries per day, per property.)

For larger sites, this is not enough. Also, despite the possible workarounds, this number still seems to be on the low side.

What’s your experience of using the Google Search Console URL Inspection API? Got more questions? Ping me on Twitter. 🙂




Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Click to comment

You must be logged in to post a comment Login

Leave a Reply

SEO

Google Quietly Ends Covid-Era Rich Results

Published

on

By

Google Quietly Ends Covid-Era Rich Results

Google removed the Covid-era structured data associated with the Home Activities rich results that allowed online events to be surfaced in search since August 2020, publishing a mention of the removal in the search documentation changelog.

Home Activities Rich Results

The structured data for the Home Activities rich results allowed providers of online livestreams, pre-recorded events and online events to be findable in Google Search.

The original documentation has been completely removed from the Google Search Central webpages and now redirects to a changelog notation that explains that the Home Activity rich results is no longer available for display.

The original purpose was to allow people to discover things to do from home while in quarantine, particularly online classes and events. Google’s rich results surfaced details of how to watch, description of the activities and registration information.

Providers of online events were required to use Event or Video structured data. Publishers and businesses who have this kind of structured data should be aware that this kind of rich result is no longer surfaced but it’s not necessary to remove the structured data if it’s a burden, it’s not going to hurt anything to publish structured data that isn’t used for rich results.

The changelog for Google’s official documentation explains:

“Removing home activity documentation
What: Removed documentation on home activity structured data.

Why: The home activity feature no longer appears in Google Search results.”

Read more about Google’s Home Activities rich results:

Google Announces Home Activities Rich Results

Read the Wayback Machine’s archive of Google’s original announcement from 2020:

Home activities

Featured Image by Shutterstock/Olga Strel

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

SEO

Google’s Gary Illyes: Lastmod Signal Is Binary

Published

on

By

Google's Gary Illyes: Lastmod Signal Is Binary

In a recent LinkedIn discussion, Gary Illyes, Analyst at Google, revealed that the search engine takes a binary approach when assessing a website’s lastmod signal from sitemaps.

The revelation came as Illyes encouraged website owners to upgrade to WordPress 6.5, which now natively supports the lastmod element in sitemaps.

When Mark Williams-Cook asked if Google has a “reputation system” to gauge how much to trust a site’s reported lastmod dates, Illyes stated, “It’s binary: we either trust it or we don’t.”

No Shades Of Gray For Lastmod

The lastmod tag indicates the date of the most recent significant update to a webpage, helping search engines prioritize crawling and indexing.

Illyes’ response suggests Google doesn’t factor in a website’s history or gradually build trust in the lastmod values being reported.

Google either accepts the lastmod dates provided in a site’s sitemap as accurate, or it disregards them.

This binary approach reinforces the need to implement the lastmod tag correctly and only specify dates when making meaningful changes.

Illyes commends the WordPress developer community for their work on version 6.5, which automatically populates the lastmod field without extra configuration.

Accurate Lastmod Essential For Crawl Prioritization

While convenient for WordPress users, the native lastmod support is only beneficial if Google trusts you’re using it correctly.

Inaccurate lastmod tags could lead to Google ignoring the signal when scheduling crawls.

With Illyes confirming Google’s stance, it shows there’s no room for error when using this tag.

Why SEJ Cares

Understanding how Google acts on lastmod can help ensure Google displays new publish dates in search results when you update your content.

It’s an all-or-nothing situation – if the dates are deemed untrustworthy, the signal could be disregarded sitewide.

With the information revealed by Illyes, you can ensure your implementation follows best practices to the letter.


Featured Image: Danishch/Shutterstock

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

SEO

How to Persuade Your Boss to Send You to Ahrefs Evolve

Published

on

How to Persuade Your Boss to Send You to Ahrefs Evolve

There’s one thing standing between you and several days of SEO, socializing, and Singaporean sunshine: your boss (and their Q4 budget 😅).

But don’t worry—we’ve got your back. Here are 5 arguments (and an example message) you can use to persuade your boss to send you to Ahrefs Evolve.

About Ahrefs Evolve

  • 2 days in sunny Singapore (Oct 24–25)
  • 500 digital marketing enthusiasts
  • 18 top speakers from around the world

Learn more and buy tickets.

SEO is changing at a breakneck pace. Between AI Overviews, Google’s rolling update schedule, their huge API leak, and all the documents released during their antitrust trial, it’s hard to keep up. What works in SEO today?

You could watch a YouTube video or two, maybe even attend an hour-long webinar. Or, much more effective: you could spend two full days learning from a panel of 18 international SEO experts, discussing your takeaways live with other attendees.

How to Persuade Your Boss to Send You to AhrefsHow to Persuade Your Boss to Send You to Ahrefs
Evolve speakers from around the world.

Our world-class speakers are tackling the hardest problems and best opportunities in SEO today. The talk agenda covers topics like:

  • Responding to AI Overviews: Amanda King will teach you how to respond to AI Overviews, Google Gemini, and other AI search functions.
  • Surviving (and thriving) Google’s algo updates: Lily Ray will talk through Google’s recent updates, and share data-driven recommendations for what’s working in search today.
  • Planning for the future of SEO: Bernard Huang will talk through the failures of AI content and the path to better results.

(And attendees will get video recordings of each session, so you can share the knowledge with your teammates too.)

View the full talk agenda here.

There’s no substitute for meeting with influencers, peers, and partners in real life. 

Conferences create serendipity: chance encounters and conversations that can have a huge positive impact on you and your business. By way of example, these are some of the real benefits that have come my way from attending conferences:

  • Conversations that lead to new customers for our business,
  • Invitations to speak at events,
  • New business partnerships and co-marketing opportunities, and
  • Meeting people that we went on to hire.

There’s a “halo” effect that lingers long after the event is over: the people you meet will remember you for longer, think more highly of you, and be more likely to help you out, should you ask.

(And let’s not forget: there’s a lot of information, particularly in SEO, that only gets shared in person.)

The “international” part of Evolve matters too. Evolve is a different crowd to your local run-of-the-mill conference. It’s a chance to meet with people from markets you wouldn’t normally meet—from Australia to Indonesia and beyond.

How to Persuade Your Boss to Send You to AhrefsHow to Persuade Your Boss to Send You to Ahrefs
Evolve attendees by home country.

If you’re an Ahrefs customer (thank you!), you’ll learn tons of tips, tricks and workflow improvements from attending Evolve. You’ll have opportunities to:

  • Attend talks from the Ahrefs team, showcasing advanced features and strategies that you can use in your own business.
  • Pick our brains at the Ahrefs booth, where we’ll offer informal 1:1 coaching sessions and previews of up-coming releases (like our new content optimization tool 🤫).
  • Join dedicated Ahrefs training workshops, hosted by the Ahrefs team and Ahrefs power users (tickets for these workshops will sold separately).

As a manager myself, there are two questions I need answered when approving expenses:

  • Is this a reasonable cost?
  • Will we see a return on this investment?

To answer those questions: early bird tickets for Evolve start at $570. For context, “super early bird” tickets for MozCon (another popular SEO conference) this year were almost twice as much: $999.

There’s a lot included in the ticket price too:

  • World-class international speakers,
  • 5-star hotel venue,
  • 5-star hotel food (two tea breaks with snacks & lunch),
  • Networking afterparty, and
  • Full talk recordings to later share with your team.

SEO is a crucial growth channel for most businesses. If you can improve your company’s SEO performance after attending Evolve (and we think you will), you’ll very easily see a positive return on the investment.

Traveling to tropical Singapore (and eating tons of satay) is great for you, but it’s also great for your team. Attending Evolve is a chance to break with routine, reignite your passion for marketing, and come back to your job reinvigorated.

This would be true for any international conference, but it goes double for Singapore. It’s a truly unique place: an ultra-safe, high-tech city that brings together dozens of different cultures.

1718123166 301 How to Persuade Your Boss to Send You to Ahrefs1718123166 301 How to Persuade Your Boss to Send You to Ahrefs
Little India in Singapore

You’ll discover different beliefs, working practices, and ways of business—and if you’re anything like me, come back a richer, wiser person for the experience.

If you’re nervous about pitching your boss on attending Evolve, remember: the worst that can happen is a polite “not this time”, and you’ll find yourself in the same position you are now.

So here goes: take this message template, tweak it to your liking, and send it to your boss over email or Slack… and I’ll see you in Singapore 😉

Email template

Hi [your boss’ name],

Our SEO tool provider, Ahrefs, is holding an SEO and digital marketing conference in Singapore in October. I’d like to attend, and I think it’s in the company’s interest:

  • The talks will help us respond to all the changes happening in SEO today. I’m particularly interested in the talks about AI and recent Google updates. 
  • I can network with my peers. I can discover what’s working at other companies, and explore opportunities for partnerships and co-marketing.
  • I can learn how we can use Ahrefs better across the organization.
  • I’ll come back reinvigorated with new ideas and motivation, and I can share my top takeaways and talk recordings with my team after the event.

Early bird tickets are $570. Given how important SEO is to the growth of our business, I think we’ll easily see a return from the spend.

Can we set up time to chat in more detail? Thanks!

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

Trending