Google revealed details of two new crawlers that are optimized for scraping image and video content for “research and development” purposes. Although the documentation doesn’t explicitly say so, it’s presumed that there is no impact in ranking should publishers decide to block the new crawlers.

It should be noted that the data scraped by these crawlers are not explicitly for AI training data, that’s what the Google-Extended crawler is for.

GoogleOther Crawlers

The two new crawlers are versions of Google’s GoogleOther crawler that was launched in April 2023. The original GoogleOther crawler was also designated for use by Google product teams for research and development in what is described as one-off crawls, the description of which offers clues about what the new GoogleOther variants will be used for.

The purpose of the original GoogleOther crawler is officially described as:

“GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.”

Two GoogleOther Variants

There are two new GoogleOther crawlers:

• GoogleOther-Image
• GoogleOther-Video

The new variants are for crawling binary data, which is data that’s not text. HTML data is generally referred to as text files, ASCII or Unicode files. If it can be viewed in a text file then it’s a text file/ASCII/Unicode file. Binary files are files that can’t be open in a text viewer app, files like image, audio, and video.

The new GoogleOther variants are for image and video content. Google lists user agent tokens for both of the new crawlers which can be used in a robots.txt for blocking the new crawlers.

1. GoogleOther-Image

User agent tokens:

• GoogleOther-Image
• GoogleOther

Full user agent string:

GoogleOther-Image/1.0

2. GoogleOther-Video

User agent tokens:

• GoogleOther-Video
• GoogleOther

Full user agent string:

GoogleOther-Video/1.0

Newly Updated GoogleOther User Agent Strings

Google also updated the GoogleOther user agent strings for the regular GoogleOther crawler. For blocking purposes you can continue using the same user agent token as before (GoogleOther). The new Users Agent Strings are just the data sent to servers to identify the full description of the crawlers, in particular the technology used. In this case the technology used is Chrome, with the model number periodically updated to reflect which version is used (W.X.Y.Z is a Chrome version number placeholder in the example listed below)

The full list of GoogleOther user agent strings:

• Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; GoogleOther)
• Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36

GoogleOther Family Of Bots

These new bots may from time to time show up in your server logs and this information will help in identifying them as genuine Google crawlers and will help publishers who may want to opt out of having their images and videos scraped for research and development purposes.

Read the updated Google crawler documentation

GoogleOther-Image

GoogleOther-Video

Featured Image by Shutterstock/ColorMaker

Reddit partners with OpenAI to integrate content into ChatGPT.

• Reddit and OpenAI announce a partnership.
• Reddit content will be used in ChatGPT.
• Concerns about accuracy of Reddit user-generated content.

SEO tracking involves regularly checking a set of metrics to evaluate a website’s performance in search engine results. Some of the most widely adopted metrics include keyword rankings, organic traffic, conversions, and referring domain growth.

Tracking the right metrics is crucial for SEO (search engine optimization) success. You need them to analyze your SEO performance, report to stakeholders, and take the right kind of action to improve your site’s visibility (such as improving content or building more backlinks).

Besides keeping an eye on your own website’s key metrics, it’s also smart to check out how your competitors are doing on the same metrics as you. If you notice they’re getting good results, you can figure out what tactics they’re using and consider using them too.

You can track SEO for your site to a fair degree using free tools like Google Search Console or Ahrefs Webmaster Tools. If you want deeper insights, better data, and the ability to analyze your competitors’ websites, you’ll need a tool like Ahrefs.

This guide is aimed at getting you started with tracking your SEO progress the right way. We’ll cover:

• What metrics are worth tracking in SEO.
• How to set up the tools to get the data you need.
• How to track your competitors.
• How to go a step further and build an SEO report.

While there are numerous metrics and KPIs you could track, it’s not necessary to monitor all of them continuously. You really just need these seven key metrics to effectively gauge whether your SEO efforts are working.

1. Keyword rankings

Keyword ranking refers to where your page shows up on the search engine results page (SERP) for a specific keyword. It’s like a spot on a list, and you want your page to be as high up on that list as possible — the higher the spot, the more visitors you can attract.

It’s important to keep an eye on where your keywords are ranking because if they drop lower on the list, your website might get fewer visitors. But you don’t have to watch the rankings for every single keyword, just the main ones that matter most for your key pages.

Also, if you notice your rankings are climbing higher, that’s a good sign. It means that your SEO efforts are paying off.

How to track keyword rankings

To track your keyword rankings, it’s best to use a rank tracker tool like Ahrefs’ Rank Tracker; a tool that allows you to create a list of keywords and automatically monitor their positions in the SERPs for different locations, both for mobile and desktop.

Rank Tracker will suggest keywords for tracking when you set up a new project. Just make sure you’re tracking them in the locations you want to rank (that is, countries where you can serve clients and languages in which you create content).

No need to add each and every keyword from that list. Just add the ones that are important to you and you’ll likely want to track and improve. Typically, you’ll want to track target keywords — the main topic of the page and the main keyword you optimize for.

Once added, you can see your keywords in Rank Tracker’s Overview report.

Another way to start tracking keywords is to hit Add keywords in the top right corner — best for adding single keywords or importing a list from a document.

And once data starts rolling in, you will be able to see your ranking progress in time. In the screenshot below, the Ranking history report with a quick insight into recent ranking history and a full ranking history graph.

2. Share of voice

Share of voice (SOV) is a measure of how many clicks your website gets from search engines compared to the total number of clicks available for the keywords you’re tracking.

The higher your rankings, the higher your Share of Voice, and the larger your slice of the market pie.

SOV is a one-of-a-kind metric because of two things:

• It considers your performance in context to your competitors, giving you a more accurate picture of where you stand in your industry.
• It doesn’t take into account the search volume of keywords with all of their fluctuations. If you see that your traffic has gone down but your Share of Voice (SOV) remains high, it suggests that the lower traffic is because the keywords you’re targeting have become less popular overall, rather than a decrease in the effectiveness of your SEO strategies.

How to track share of voice

The share of voice metric is another reason to get a rank tracking tool. If the feature is supported, these kinds of tools calculate the metric automatically, so there’s no need to keep a spreadsheet with manually tracked numbers.

In Ahrefs’s Rank Tracker, you’ll find SOV under the Competitors tab.

SOV is calculated by taking all of the tracked keywords into account, yet some of your keywords might be more important than others. If that’s the case, you can track SOV only for a certain topic, SEO campaigns, specific authors, etc. Just select a set of keywords and assign a tag for them.

Then, simply select that tag in the Competitors report.

3. Organic traffic

Organic traffic is basically the number of clicks that come to your website from people finding it through Google. If your website shows up higher on the SERPs, usually more people will click on it and visit your site.

Keeping track of how many visitors come to your site from search engines helps you understand if what you’re doing with SEO is actually working. If you see more visitors over time, your SEO efforts are paying off.

Organic traffic is the pinnacle of SEO, but it’s also important to understand which keywords drive that traffic. So although it’s arguably the most important metric, it’s never a good idea to track this metric alone.

How to track organic traffic

In terms of raw organic traffic from Google Search, the most accurate data will likely come from their Search Console (for Bing, that would be Webmaster Tools). You can view this data right inside the tool or integrate it with analytics tools like Google Analytics, Hubspot, and Ahrefs for more convenience.

Raw traffic data is useful for getting a quick snapshot of your current performance, tracking growth trends, and calculating traffic growth for your reports.

But to dive a bit deeper into your organic traffic data, you might want to use a tool like Ahrefs’ Site Explorer because it makes it easier to analyze performance. Here are a few ways you can use the Overview and Top pages report in that tool.

Overlay competitor data on top for a quick performance analysis.

Overlay organic pages to see how adding new content correlates with traffic.

See performance in a year-over-year comparison to gauge the impact of long-term projects.

Use daily traffic chart to pinpoint the exact day when a traffic increase or decline happened (for instance, due to a Google core update).

Identify pages that account for the biggest traffic losses and improve them. You’ll find this in the Top pages report inside Site Explorer.

4. Conversions

Conversions from organic visits to paid customers are typically hard to measure since this comes down to measuring the ROI of content, which is complicated in itself. However, when we asked marketers about this metric, we found a few interesting ways to solve that problem. For your inspiration, here’s what they measure:

• Conversion as revenue/signups correlation with traffic. This metric assumes that more website visitors increase your chances of turning them into subscribers or buyers.
• Conversion growth from bottom-funnel content. Content aimed at users who are on the brink of purchasing can greatly boost sales because it provides that last bit of persuasion they need to complete a purchase.
• Conversion from first page to paying customer. If the first page a visitor lands on leads to a sale, it’s a clear sign that your content is doing its job effectively.

How to track conversions

For example, in GA4, conversions are called “key events” and are based on tracking user interaction. If a specific event takes place, such as a purchase, a file download, or a form completion, the tool records this as a conversion.

To set up conversion tracking in GA4 you first need to create an event that will be counted as conversion and mark it a key event in the Admin panel of your site (aka property).

Then, to see conversion from the organic traffic channel (the channel you’re optimizing with SEO), go to the Advertising panel.

Here are a few ideas to use this report:

• See how many and which key events were driven by organic search in the last month or quarter.
• See how organic traffic stacks up to other acquisition channels.
• See the share of organic traffic for events with longer conversion paths (the attribution paths tab).

For more information about how to properly set up GA4 for conversion tracking, see this guide.

5. Referring domain growth

Referring domains are essentially the individual websites that link back to your website. By monitoring these, you get a clear picture of how your link profile is expanding over time.

As your link profile grows with more quality links from diverse domains, it helps to build your site’s authority. This authority is crucial because search engines use it as one of the key factors to determine where your pages should rank in search results.

Essentially, the more authoritative your site becomes, the higher your pages are likely to rank and the harder it becomes for others to outrank you.

How to track referring domain growth

• Set up a project (if you haven’t done so yet) and go to your Dashboard.
• Click on the Backlinks card, which gives you a quick insight into backlinks growth.
• Click on the card to get more data (if you need it).

Aim to build as many or ideally more links from unique domains than your competitors to increase your chances for ranking. Read our link-building guide to learn how:

6. Technical SEO issues

• Indexability issues. This means search engines might have trouble finding and adding your website’s pages to their index. If they can’t find your pages, people searching online won’t either.
• Broken pages. These are pages on your website that don’t load properly and might show errors instead (the 404 and 5XX error pagers we all see every now and then). It’s like trying to open a door that won’t budge.
• Few or no internal links. Internal links are connections between one page on your website and another. If your site doesn’t have many, it’s like having a map with missing paths, making it hard for both users and search engines to navigate.
• Mobile experience issues. This refers to problems when your website is accessed on mobile devices. If your site is hard to use on a phone or tablet, visitors might give up and go elsewhere.
• HTTPS issues. HTTPS is a security measure for websites. Ideally, Google wants to show only secure websites.
• Performance and stability issues. This is about how fast your website loads and how smoothly it runs. Typically measured through Core Web Vitals.
• Lack of relevant schema markup. Schema markup is code that helps search engines understand the content on your pages and display it attractively in search results. Without it, your site might not look as inviting or clear when it appears in search and it might not rank in these special spots that Google holds for pages with applied schema.
• Duplicate content. This is when the same or very similar content appears in multiple places on your website. Google will likely show only one of those pages and it might not be the page you want.

Besides these issues, there are more than 100 other possible issues related to less important technical SEO factors and on-page SEO. I won’t cover all of them here since you can learn what they are and how to fix them right inside Ahrefs.

How to track technical SEO issues (aka SEO health)

Use Ahrefs’ Site Audit (free in Ahrefs Webmaster Tools) to monitor for serious technical issues, marked in the tool as “errors”.

• Open Site Audit tool inside Ahrefs.

• Click on Errors in the “Issues distribution” card.

• Go to the issue list, then click on the question mark next to the error and follow the instructions.

To keep your site in good SEO health, schedule regular crawls in Site Audit and fix the most pressing issues.

How to track competitors’ keyword rankings

To track your competitors’ rankings, use a rank tracking tool that allows you to automatically monitor their positions on the keywords you target yourself. So whenever you add keywords you want to target in your strategy, the tool will track both your and your competitors’ rank for that keyword.

In Ahrefs’ Rank Tracker all you need to do is set add your competitors’ URLs (you can track entire domains or specific directories). You can do it as soon as setting up your project or add them later on in the Competitors section.

You can use competitor ranking data to:

• Improve the pages where your competitors outrank you to gain more SOV.
• Set goals and benchmarks.
• Compare historical rankings to your performance over time.
• Quickly see the competitive landscape; see how well you’re doing compared to competitors.
• See how much more traffic you could gain if you outranked competitors.

How to track multiple metrics for a portfolio of pages

You can also track more than just rankings. Using the Portfolios feature in Ahrefs, you can monitor key metrics such as traffic growth and the increase in referring domains for multiple competitors all at once to analyze their overall SEO performance.

You can use this feature to monitor specific pages on your competitors’ sites (such as topics on a blog) or combine all your competitors’ sites to see how your entire niche performs in organic search.

To create a portfolio in Ahrefs, go to the Dashboard and click New > Portfolio, then fill in the URLs you want to track.

Tip

This feature is especially useful if you’re managing SEO for multiple clients — you can track their entire portfolio as one.

It’s also handy if you have multiple authors on your content team; for example, you can track all articles written by a particular author or keep tabs on all guest and freelance posts.

How to track competitors’ new keywords, backlinks, and web mentions

The final method of tracking your competitors allows you to get email alerts when a competitor:

• Ranks for a new keyword. Useful for getting content ideas from your competitors’ new content.
• Rise and fall in keyword rankings. For example, if you see an important keyword suddenly climbing into the top 3, that means your competitor is doing something right, and it’s worth investigating. It’s worth noting that this feature scans, all of the keywords, the site ranks for and not only the ones you track, so it gives you a much wider scope.
• Gain or lose backlinks. Both situations are potential link building opportunities.
• Their brand or product is mentioned online. So, when a competitor gets featured in a review, ranking, or digital PR, you can add that site to your list of link building/PR prospects.

To set it up:

1. Go to Ahrefs Alerts (in the More dropdown menu)
2. Choose the type of Alert you want to set up.
3. Click New alert or choose from one of the projects and fill out the details. In case of the mentions alerts, see our documentation to take advantage of advanced queries.

If you’re doing SEO for someone else, at some point, you will need to put all of those metrics in a report.

In some cases, it may be enough to show the raw data with a few sentences of commentary. This is true in in-house environments when you’re reporting to someone who can interpret the data themselves, especially if you’ve worked with them for a long time.

But if you’re reporting for a client, raw numbers won’t be enough. Additionally, you will need at least these three elements:

• Executive summary: Summarizes the entire report, focusing on major points and outcomes for quick reading by senior stakeholders.
• Opportunities for improvement: Identifies potential areas for SEO enhancements.
• Roadmap: Outlines past achievements and future steps in the SEO strategy.

It’s also important how to report data for your and your stakeholders understanding and convenience. For instance, many clients require a live interactive dashboard with all the data available at all times (similar to these Ahrefs templates for Looker Studio).

Others prefer a document where everything is laid out in layman’s terms — they appreciate the data but they don’t really want to deal with it.

We’ve put together some resources, including a template, to help you quickly and efficiently create a solid report:

Final thoughts

• Set up a routine to check performance. In SEO, consistency is crucial. Conduct weekly checks to assess immediate impacts, like how quickly Google recognizes recent content updates. For long-term planning, such as strategizing quarterly content, review your SEO performance monthly or quarterly. This routine ensures you’re always informed and ready to adapt your strategies as needed.
• Don’t neglect competitor tracking. Failing to track your competitors’ actions could make it challenging to reverse-engineer their strategies later, potentially putting you at a disadvantage.
• Learn about the data before choosing your tools. The reliability and depth of the data provided by an SEO tool are crucial for making accurate assessments and decisions. Before committing to an SEO tool, research its data sources, update frequency, historical data accuracy, and the breadth of its index. You can read about Ahref’s data here and in data studies like this one on search data accuracy.

Got questions or comments? Let me know on X or LinkedIn.