Screaming Frog Adds Google URL Inspection API

Screaming Frog didn’t waste any time integrating Google’s new URL inspection API that allows access to current indexing data. The new API allows Screaming Frog to include seven brand new filters that identify website problems in need of attention.

In addition to the expanded usefulness of Screaming Frog the new update, code named Romeo, also patches a log4j vulnerability issue, adds new keyboard shortcuts and fixes some bugs to make it more stable.

Seven More Ways to Fix Websites

Google’s new URL Inspection API allows software tools like Screaming Frog to gain access to Google Search Console and import data into the tool.

API stands for Application Programming Interface. It’s like a bridge between two software programs that allows both of them to connect to share data and achieve a certain level of integration between the two programs.

Screaming Frog’s integration of the URL Inspection API allows the popular site auditing program to provide the following brand new troubleshooting filters.

Seven New Auditing Filters:

1. “URL Is Not on Google
   The URL is not indexed by Google and won’t appear in the search results. This filter can include non-indexable URLs (such as those that are ‘noindex’) as well as Indexable URLs that are able to be indexed. It’s a catch all filter for anything not on Google according to the API.
2. Indexable URL Not Indexed
   Indexable URLs found in the crawl that are not indexed by Google and won’t appear in the search results. This can include URLs that are unknown to Google, or those that have been discovered but not indexed, and more.
3. URL is on Google, But Has Issues
   The URL has been indexed and can appear in Google Search results, but there are some problems with mobile usability, AMP or Rich results that might mean it doesn’t appear in an optimal way.
4. User-Declared Canonical Not Selected
   Google has chosen to index a different URL to the one declared by the user in the HTML.
5. Page Is Not Mobile Friendly
   The page has issues on mobile devices.
6. AMP URL Is Invalid
   The AMP has an error that will prevent it from being indexed.
7. Rich Result Invalid
   The URL has an error with one or more rich result enhancements that will prevent the rich result from showing in the Google search results.”

The value of the URL Inspection Tool API integration is that now you can overlay the Search Console indexing data onto the Screaming Frog crawl data, which allows you to see all the on-page data points extracted by Screaming Frog juxtaposed with the indexing status.

Screaming Frog now gains a Rich Results inspection tool that allows bulk export of the data from the Bulk Export tab at the top menu.

Additional Updates

Other updates, in addition to the aforementioned log4j vulnerability patch are the addition of keyboard shortcuts (ctrl + W to close crawling projects), they made the Word Cloud Visualization tool language-aware in order to improve word-filtering, fixed a Page Speed Insights bug, an issue with JavaScript crawls that would cause a stall, fixed CSS Path extraction to make it support leading and trailing slashes and various other fixes that make Screaming Frog run smoother.

Screaming Frog URL Inspection API Integration Part 1?

The update announcement offered a statement that noted that there is still some more integration work to be done to make it even better.

Because of practical limitations in how data can be fit into the Screaming Frog interface not all of the URL inspection data could be fit into it. But they promised to listen to user feedback to include additional data if users feel it’s important.

The Screaming Frog announcement noted:

“With use and user feedback, the way we integrate the data to gain additional insights will be improved as well.

If you spot any issues, or would like to see additional data or reporting from the URL Inspection API – just let us know via support.

Advertisement

We already have a few items planned, but wanted to get the basic feature including the most important items out there for users to enjoy quickly.”

So it looks like even more good things are coming our way from Screaming Frog.

Citation

Read the Official Screaming Frog Announcement

Screaming Frog SEO Spider Update – Version 16.6