SEO
What It Is & How to Use It
Diagnosing technical issues on your website can be one of the most time-consuming but important aspects of running a website.
To make things worse, Google only allows you to inspect one URL at a time to diagnose potential issues on your website (this is done within Google Search Console).
Luckily, there is now a faster way to test your website: enter the Google Search Console URL Inspection API…
What is the Google Search Console URL Inspection API?
The Google Search Console URL Inspection API is a way to bulk-check the data that Google Search Console has on URLs. Its purpose is to help developers and SEOs more efficiently debug and optimize their pages using Google’s own data.
Here’s an example of me using the API to check whether a few URLs are indexed and submitted in my sitemap:
What type of data can you get from the Google Search Console URL Inspection API?
The Google Search Console URL Inspection API allows you to pull a wide range of data. Below is a list of some of the most discussed features:
lastCrawlTime
With this field, you can understand exactly when Googlebot last crawled your website. This is extremely useful for SEOs and developers to measure the frequency of Google’s crawling of their sites. Previously, you could only get access to this type of data through log file analysis or spot-checking individual URLs with Google Search Console.
robotsTxtState
With this field, you can understand whether you have any robots.txt rules that will block Googlebot. This is something you can check manually, but being able to test it at scale with Google’s own data is a fantastic step forward.
googleCanonical and userCanonical
In some situations, Google has been known to select a different canonical from the one that has been specified in the code. In this situation, having the ability to compare both (side by side and at scale) using the API is useful for enabling you to make the appropriate changes.
crawledAs
This field allows you to understand which user agent is used for a crawl of your site: Mobile/Desktop. The response codes are below for reference:
- DESKTOP – Desktop user agent
- MOBILE – Mobile user agent
pageFetchState
Understanding the pageFetchState can help you diagnose server errors, not found 4xxs, soft 404s, redirection errors, pages blocked by robots.txt, and invalid URLs. A list of responses is below for reference.
| Field | What it means |
|---|---|
| PAGE_FETCH_STATE_UNSPECIFIED | Unknown fetch state |
| SUCCESSFUL | Successful fetch |
| SOFT_404 | Soft 404 |
| BLOCKED_ROBOTS_TXT | Blocked by robots.txt |
| NOT_FOUND | Not found (404) |
| ACCESS_DENIED | Blocked due to unauthorized request (401) |
| SERVER_ERROR | Server error (5xx) |
| REDIRECT_ERROR | Redirection error |
| ACCESS_FORBIDDEN | Blocked due to access forbidden (403) |
| BLOCKED_4XX | Blocked due to other 4xx issue (not 403, 404) |
| INTERNAL_CRAWL_ERROR | Internal error |
| INVALID_URL | Invalid URL |
indexingState
The indexing state tells you the current status of indexation for the URLs. Apart from the more obvious Pass and Fail responses, there are other responses:
- NEUTRAL is equivalent to the “Excluded” message in Search Console.
- PARTIAL is equivalent to the “Valid with warnings” message in Search Console.
- VERDICT_UNSPECIFIED means that Google is unable to come to a conclusion about the URL(s) in question.
coverageState
This gives you detail on whether a URL has been submitted in your sitemap and indexed.
referringUrls
This allows you to see where each page is linked from, according to Google.
Sitemap
This enables you to understand which URLs are included in the sitemap(s).
Other uses for the API
You can also use the API to inspect your AMP site—if you have one.
How to use the Google Search Console URL Inspection API step by step
Using the Google Search Console URL Inspection API involves making a request to Google. The request parameters you need to define are the URL that you want to inspect and also the URL of the property in Google Search Console.
The request body contains data with the following structure:
If you are curious to learn more about how to use the API, Google has extensive documentation about this.
Below is an example of the type of response you can get from the API:
If you’re not comfortable with code or just want to give it a go straight away, you can use valentin.app’s free Google Bulk Inspect URLs tool. The tool provides a quick way to query the API without any coding skills!
Here’s how to use it. You can:
- Go to https://valentin.app/inspect.html, authorize access to your Google account, and select the Search Console property you want to test. Then paste your URLs into the box below. (The data will be processed in your browser and not uploaded to a server or shared with anyone.)
- Click the “Inspect URLs” button. The data will start to pull from the API.
- Export the data as a CSV or Excel file by clicking the button.
- Analyze the data and check for any potential issues.
How can you use the Google Search Console URL Inspection API in practice?
In theory, the Google Search Console URL Inspection API seems like a great way to understand more about your website. However, you can pull so much data that it’s difficult to know where to start. So let’s look at a few examples of use cases.
1. Site migration – diagnosing any technical issues
Site migrations can cause all kinds of issues. For example, developers can accidentally block Google from crawling your site or certain pages via robots.txt.
Luckily, the Google Search Console URL Inspection API makes auditing for these issues a doddle.
For example, you can check whether you’re blocking Googlebot from crawling URLs in bulk by calling robotsTxtState.
Here is an example of me using the Google Search Console URL Inspection API (via valentin.app) to call robotsTxtState to see the current status of my URLs.
As you can see, these pages are not blocked by robots.txt, and there are no issues here.
TIP
2. Understand if Google has respected your declared canonicals
If you make a change to the canonical tags across your site, you will want to know whether or not Google is respecting them.
You may be wondering why Google ignores the canonical that you declared. Google can do this for a number of reasons, for example:
- Your declared canonical is not https. (Google prefers https for canonicals.)
- Google has chosen a page that it believes is a better canonical page than your declared canonical.
- Your declared canonical is a noindex page.
Below is an example of me using the Google Search Console URL Inspection API to see whether Google has respected my declared canonicals:
As we can see from the above screenshot, there are no issues with these particular pages and Google is respecting the canonicals.
TIP
To quickly see if the googleCanonical matches the userCanonical, export the data from the Google Bulk Inspect URLs tool to CSV and use an IF formula in Excel. For example, assuming your googleCanonical data is in Column A and your userCanonical is in Column B, you can use the formula =IF(A2=B2, “Self referencing”,”Non Self Referencing”) to check for non-matching canonicals.
3. Understand when Google recrawls after you make changes to your site
When you update many pages on your website, you will want to know the impact of your efforts. This can only happen after Google has recrawled your site.
With the Google Search Console URL Inspection API, you can see the precise time Google crawled your pages by using lastCrawlTime.
If you can’t get access to the log files for your website, then this is a great alternative to understand how Google crawls your site.
Here’s an example of me checking this:
As you can see in the screenshot above, lastCrawlTime shows the date and time my website was crawled. In this example, the most recent crawl by Google is the homepage.
Understanding when Google recrawls your website following any changes will allow you to link whether or not the changes you made have any positive or negative impact following Google’s crawl.
FAQs
How to get around the Google Search Console URL Inspection API limits?
Although the Google Search Console URL Inspection API is limited to 2,000 queries per day, this query limit is determined by Google Property.
This means you can have multiple properties within one website if they are verified separately in Google Search Console, effectively allowing you to bypass the limit of 2,000 queries per day.
Google Search Console allows you to have 1,000 properties in your Google Search Console account, so this should be more than enough for most users.
Can I use the Google Search Console URL Inspection API on any website?
Another potential limiting factor is you can only run the Google Search Console URL Inspection API on a property that you own in Google Search Console. If you don’t have access to the property, then you cannot audit it using the Google Search Console URL Inspection API.
So this means auditing a site that you don’t have access to can be problematic.
How accurate is the data?
Accuracy of the data itself has been an issue for Google over the last few years. This API gives you access to that data. So arguably, the Google Search Console URL Inspection API is only as good as the data within it.
As we have previously shown in our study of Google Keyword Planner’s accuracy, data from Google is often not as accurate as people assume it to be.
Final thoughts
The Google Search Console URL Inspection API is a great way for site owners to get bulk data directly from Google on a larger scale than what was previously possible from Google Search Console.
Daniel Waisberg and the team behind the Google Search Console URL Inspection API have definitely done a great job of getting this released into the wild.
But one of the criticisms of the Google Search Console URL Inspection API from the SEO community is that the query rate limit is too low for larger sites. (It is capped at 2,000 queries per day, per property.)
For larger sites, this is not enough. Also, despite the possible workarounds, this number still seems to be on the low side.
What’s your experience of using the Google Search Console URL Inspection API? Got more questions? Ping me on Twitter. 🙂
SEO
WordPress Insiders Discuss WordPress Stagnation
A recent webinar featuring WordPress executives from Automattic and Elementor, along with developers and Joost de Valk, discussed the stagnation in WordPress growth, exploring the causes and potential solutions.
Stagnation Was The Webinar Topic
The webinar, “Is WordPress’ Market share Declining? And What Should Product Businesses Do About it?” was a frank discussion about what can be done to increase the market share of new users that are choosing a web publishing platform.
Yet something that came up is that there are some areas that WordPress is doing exceptionally well so it’s not all doom and gloom. As will be seen later on, the fact that the WordPress core isn’t progressing in terms of specific technological adoption isn’t necessarily a sign that WordPress is falling behind, it’s actually a feature.
Yet there is a stagnation as mentioned at the 17:07 minute mark:
“…Basically you’re saying it’s not necessarily declining, but it’s not increasing and the energy is lagging. “
The response to the above statement acknowledged that while there are areas of growth like in the education and government sectors, the rest was “up for grabs.”
Joost de Valk spoke directly and unambiguously acknowledged the stagnation at the 18:09 minute mark:
“I agree with Noel. I think it’s stagnant.”
That said, Joost also saw opportunities with ecommerce, with the performance of WooCommerce. WooCommerce, by the way, outperformed WordPress as a whole with a 6.80% year over year growth rate, so there’s a good reason that Joost was optimistic of the ecommerce sector.
A general sense that WordPress was entering a stall however was not in dispute, as shown in remarks at the 31:45 minute mark:
“… the WordPress product market share is not decreasing, but it is stagnating…”
Facing Reality Is Productive
Humans have two ways to deal with a problem:
- Acknowledge the problem and seek solutions
- Pretend it’s not there and proceed as if everything is okay
WordPress is a publishing platform that’s loved around the world and has literally created countless jobs, careers, powered online commerce as well as helped establish new industries in developing applications that extend WordPress.
Many people have a stake in WordPress’ continued survival so any talk about WordPress entering a stall and descent phase like an airplane that reached the maximum altitude is frightening and some people would prefer to shout it down to make it go away.
Acknowledging facts and not brushing them aside is what this webinar achieved as a step toward identifying solutions. Everyone in the discussion has a stake in the continued growth of WordPress and their goal was to put it out there for the community to also get involved.
The live webinar featured:
- Miriam Schwab, Elementor’s Head of WP Relations
- Rich Tabor, Automattic Product Manager
- Joost de Valk, founder of Yoast SEO
- Co-hosts Matt Cromwell and Amber Hinds, both members of the WordPress developer community moderated the discussion.
WordPress Market Share Stagnation
The webinar acknowledged that WordPress market share, the percentage of websites online that use WordPress, was stagnating. Stagnation is a state at which something is neither moving forward nor backwards, it is simply stuck at an in between point. And that’s what was openly acknowledged and the main point of the discussion was understanding the reasons why and what could be done about it.
Statistics gathered by the HTTPArchive and published on Joost de Valk’s blog show that WordPress experienced a year over year growth of 1.85%, having spent the year growing and contracting its market share. For example, over the latest month over month period the market share dropped by -0.28%.
Crowing about the WordPress 1.85% growth rate as evidence that everything is fine is to ignore that a large percentage of new businesses and websites coming online are increasingly going to other platforms, with year over year growth rates of other platforms outpacing the rate of growth of WordPress.
Out of the top 10 Content Management Systems, only six experienced year over year (YoY) growth.
CMS YoY Growth
- Webflow: 25.00%
- Shopify: 15.61%
- Wix: 10.71%
- Squarespace: 9.04%
- Duda: 8.89%
- WordPress: 1.85%
Why Stagnation Is A Problem
An important point made in the webinar is that stagnation can have a negative trickle-down effect on the business ecosystem by reducing growth opportunities and customer acquisition. If fewer of the new businesses coming online are opting in for WordPress are clients that will never come looking for a theme, plugin, development or SEO service.
It was noted at the 4:18 minute mark by Joost de Valk:
“…when you’re investing and when you’re building a product in the WordPress space, the market share or whether WordPress is growing or not has a deep impact on how easy it is to well to get people to, to buy the software that you want to sell them.”
Perception Of Innovation
One of the potential reasons for the struggle to achieve significant growth is the perception of a lack of innovation, pointed out at the 16:51 minute mark that there’s still no integration with popular technologies like Next JS, an open-source web development platform that is optimized for fast rollout of scalable and search-friendly websites.
It was observed at the 16:51 minute mark:
“…and still today we have no integration with next JS or anything like that…”
Someone else agreed but also expressed at the 41:52 minute mark, that the lack of innovation in the WordPress core can also be seen as a deliberate effort to make WordPress extensible so that if users find a gap a developer can step in and make a plugin to make WordPress be whatever users and developers want it to be.
“It’s not trying to be everything for everyone because it’s extensible. So if WordPress has a… let’s say a weakness for a particular segment or could be doing better in some way. Then you can come along and develop a plug in for it and that is one of the beautiful things about WordPress.”
Is Improved Marketing A Solution
One of the things that was identified as an area of improvement is marketing. They didn’t say it would solve all problems. It was simply noted that competitors are actively advertising and promoting but WordPress is by comparison not really proactively there. I think to extend that idea, which wasn’t expressed in the webinar, is to consider that if WordPress isn’t out there putting out a positive marketing message then the only thing consumers might be exposed to is the daily news of another vulnerability.
Someone commented in the 16:21 minute mark:
“I’m missing the excitement of WordPress and I’m not feeling that in the market. …I think a lot of that is around the product marketing and how we repackage WordPress for certain verticals because this one-size-fits-all means that in every single vertical we’re being displaced by campaigns that have paid or, you know, have received a a certain amount of funding and can go after us, right?”
This idea of marketing being a shortcoming of WordPress was raised earlier in the webinar at the 18:27 minute mark where it was acknowledged that growth was in some respects driven by the WordPress ecosystem with associated products like Elementor driving the growth in adoption of WordPress by new businesses.
They said:
“…the only logical conclusion is that the fact that marketing of WordPress itself is has actually always been a pain point, is now starting to actually hurt us.”
Future Of WordPress
This webinar is important because it features the voices of people who are actively involved at every level of WordPress, from development, marketing, accessibility, WordPress security, to plugin development. These are insiders with a deep interest in the continued evolution of WordPress as a viable platform for getting online.
The fact that they’re talking about the stagnation of WordPress should be of concern to everybody and that they are talking about solutions shows that the WordPress community is not in denial but is directly confronting situations, which is how a thriving ecosystem should be responding.
Watch the webinar:
Is WordPress’ Market share Declining? And What Should Product Businesses Do About it?
Featured Image by Shutterstock/Krakenimages.com
SEO
Google’s New Support For AVIF Images May Boost SEO
Google announced that images in the AVIF file format will now be eligible to be shown in Google Search and Google Images, including all platforms that surface Google Search data. AVIF will dramatically lower image sizes and improve Core Web Vitals scores, particularly Largest Contentful Paint.
How AVIF Can Improve SEO
Getting pages crawled and indexed are the first step of effective SEO. Anything that lowers file size and speeds up web page rendering will help search crawlers get to the content faster and improve the amount of pages crawled.
Google’s crawl budget documentation recommends increasing the speeds of page loading and rendering as a way to avoid receiving “Hostload exceeded” warnings.
It also says that faster loading times enables Googlebot to crawl more pages:
Improve your site’s crawl efficiency
Increase your page loading speed
Google’s crawling is limited by bandwidth, time, and availability of Googlebot instances. If your server responds to requests quicker, we might be able to crawl more pages on your site.
What Is AVIF?
AVIF (AVI Image File Format) is a next generation open source image file format that combines the best of JPEG, PNG, and GIF image file formats but in a more compressed format for smaller image files (by 50% for JPEG format).
AVIF supports transparency like PNG and photographic images like JPEG does but does but with a higher level of dynamic range, deeper blacks, and better compression (meaning smaller file sizes). AVIF even supports animation like GIF does.
AVIF Versus WebP
AVIF is generally a better file format than WebP in terms of smaller files size (compression) and image quality. WebP is better for lossless images, where maintaining high quality regardless of file size is more important. But for everyday web usage, AVIF is the better choice.
See also: 12 Important Image SEO Tips You Need To Know
Is AVIF Supported?
AVIF is currently supported by Chrome, Edge, Firefox, Opera, and Safari browsers. Not all content management systems support AVIF. However, both WordPress and Joomla support AVIF. In terms of CDN, Cloudflare also already supports AVIF.
I couldn’t at this time ascertain whether Bing supports AVIF files and will update this article once I find out.
Current website usage of AVIF stands at 0.2% but now that it’s available to surfaced in Google Search, expect that percentage to grow. AVIF images will probably become a standard image format because of its high compression will help sites perform far better than they currently do with JPEG and PNG formats.
Research conducted in July 2024 by Joost de Valk (founder of Yoast, ) discovered that social media platforms don’t all support AVIF files. He found that LinkedIn, Mastodon, Slack, and Twitter/X do not currently support AVIF but that Facebook, Pinterest, Threads and WhatsApp do support it.
AVIF Images Are Automatically Indexable By Google
According to Google’s announcement there is nothing special that needs to be done to make AVIF image files indexable.
“Over the recent years, AVIF has become one of the most commonly used image formats on the web. We’re happy to announce that AVIF is now a supported file type in Google Search, for Google Images as well as any place that uses images in Google Search. You don’t need to do anything special to have your AVIF files indexed by Google.”
Read Google’s announcement:
Supporting AVIF in Google Search
Featured Image by Shutterstock/Cast Of Thousands
SEO
CMOs Called Out For Reliance On AI Content For SEO
Eli Schwartz, Author of Product-Led SEO, started a discussion on LinkedIn about there being too many CMOs (Chief Marketing Officers) who believe that AI written content is an SEO strategy. He predicted that there will be reckoning on the way after their strategies end in failure.
This is what Eli had to say:
“Too many CMOs think that AI-written content is an SEO strategy that will replace actual SEO.
This mistake is going to lead to an explosion in demand for SEO strategists to help them fix their traffic when they find out they might have been wrong.”
Everyone in the discussion, which received 54 comments, strongly agreed with Eli, except for one guy.
What Is Google’s Policy On AI Generated Content?
Google’s policy hasn’t changed although they did update their guidance and spam policies on March 5, 2024 at the same time as the rollout of the March 2024 Core Algorithm Update. Many publishers who used AI to create content subsequently reported losing rankings.
Yet it’s not said that using AI is enough to merit poor rankings, it’s content that is created for ranking purposes.
Google wrote these guidelines specifically for autogenerated content, including AI generated content (Wayback machine copy dated March 6, 2024)
“Our long-standing spam policy has been that use of automation, including generative AI, is spam if the primary purpose is manipulating ranking in Search results. The updated policy is in the same spirit of our previous policy and based on the same principle. It’s been expanded to account for more sophisticated scaled content creation methods where it isn’t always clear whether low quality content was created purely through automation.
Our new policy is meant to help people focus more clearly on the idea that producing content at scale is abusive if done for the purpose of manipulating search rankings and that this applies whether automation or humans are involved.”
Many in Eli’s discussion were in agreement that reliance on AI by some organizations may come to haunt them, except for that one guy in the discussion
Read the discussion on LinkedIn:
Too many CMOs think that AI-written content is an SEO strategy that will replace actual SEO
Featured Image by Shutterstock/Cast Of Thousands
Google’s Revamped Documentation Shows 4 Reasons To Refresh Content
Daily Search Forum Recap: August 26, 2024
Google Ranking Bug Fixed, August Core Update Swings, AI Overviews, Google Ads Bug & More
How to Secure Your WordPress Store
Daily Search Forum Recap: August 27, 2024
Business Owners are Batting 1,000 With This All-in-One Management Hub
Google Migrating All To Google Merchant Center Next By September
10 Best StudioPress Alternatives (Genesis Framework)
9 Best Elementor Alternatives 2024 (Faster Page Builders)
What Every Entrepreneur Can Learn from NCAA Coach Dawn Staley’s Leadership Style
-
SEO7 days agoGoogle’s Revamped Documentation Shows 4 Reasons To Refresh Content
-
SEARCHENGINES5 days ago
Daily Search Forum Recap: August 26, 2024
-
SEARCHENGINES7 days agoGoogle Ranking Bug Fixed, August Core Update Swings, AI Overviews, Google Ads Bug & More
-
WORDPRESS7 days agoHow to Secure Your WordPress Store
-
SEARCHENGINES4 days ago
Daily Search Forum Recap: August 27, 2024
-
AFFILIATE MARKETING7 days agoBusiness Owners are Batting 1,000 With This All-in-One Management Hub
-
SEARCHENGINES6 days agoGoogle Migrating All To Google Merchant Center Next By September
-
WORDPRESS5 days ago10 Best StudioPress Alternatives (Genesis Framework)
You must be logged in to post a comment Login