Connect with us

SEO

Google Considers Reducing Webpage Crawl Rate

Published

on

Google Considers Reducing Webpage Crawl Rate

Google may reduce the frequency of crawling webpages as it grows more conscious of the sustainability of crawling and indexing.

This topic is discussed by Google’s Search Relations team, which is made up of John Mueller, Martin Splitt, and Gary Illyes.

Together, in the latest episode of the Search Off the Record podcast, they discuss what to expect from Google in 2022 and beyond.

Among the topics they address is crawling and indexing, which SEO professionals and website owners say they’ve seen less of over the past year.

That’s going to be a key focus for Google this year as it aims to make crawling more sustainable by conserving computing resources.

Advertisement

Here’s what that will mean for your website and its performance in search results.

Sustainability Of Crawling & Indexing

Since Googlebot crawling and indexing happens virtually, it’s not something you may think has an impact on the environment.

Illyes brings this issue to attention when he says computing isn’t sustainable in general:

“… what I mean is that computing, in general, is not really sustainable. And if you think of Bitcoin, for example, Bitcoin mining has real impact on the environment that you can actually measure, especially if the electricity is coming from coal plants or other less sustainable plants.

We are carbon-free, since I don’t even know, 2007 or something, 2009, but it doesn’t mean that we can’t reduce even more our footprint on the environment. And crawling is one of those things that early on, we could chop off some low-hanging fruits.”

The low-hanging fruits, in this instance, refers to unnecessary web crawling. Such as crawling webpages that haven’t had any recent updates.

How Will Google Make Crawling More Sustainable?

Illyes goes on to explain that web crawling can be made more sustainable by cutting down on refresh crawls.

Advertisement

There are two types of Googlebot crawling: crawling to discover new content and crawling to refresh existing content.

Google is considering scaling back on crawling to refresh content.

Illyes continues:

“… one thing that we do, and we might not need to do that much, is refresh crawls. Which means that once we discovered a document, a URL, then we go, we crawl it, and then, eventually, we are going to go back and revisit that URL. That is a refresh crawl.

And then every single time we go back to that one URL, that will always be a refresh crawl. Now, how often do we need to go back to that URL?”

He goes on to give an example of certain websites that warrant a significant number of refresh crawls for some parts of the site but not others.

A website like Wall Street Journal is constantly updating its homepage with new content, so it deserves a lot of refresh crawls.

Advertisement

However, WSJ is not likely updating its About page as frequently, so Google doesn’t need to keep doing refresh crawls on those types of pages.

“So you don’t have to go back there that much. And often, we can’t estimate this well, and we definitely have room for improvement there on refresh crawls. Because sometimes it just seems wasteful that we are hitting the same URL over and over again.

Sometimes we are hitting 404 pages, for example, for no good reason or no apparent reason. And all these things are basically stuff that we could improve on and then reduce our footprint even more.”

If Google were to cut down on refresh crawls, which is not 100% confirmed, here’s the impact that could have on your website.

What Does A Reduction In Crawl Rate Mean For Your Website?

There’s a belief out there that a high crawl rate is a positive SEO signal, even if you’re not updating your content as often as Google is crawling it.

That’s a misconception, Illyes says, as content will not necessarily rank better because it gets crawled more.

Mueller:

Advertisement

“So I guess that’s kind of also a misconception that people have inthat they think if a page gets crawled more, it’ll get ranked more. Is that correct that that’s a misconception, or is that actually true?”

Illyes:

“It’s a misconception.”

Mueller:

“OK, so no need to try to force something to be re-crawled if it doesn’t actually change. It’s not going to rank better.”

Again, it’s not confirmed that Google will reduce refresh crawls, but it’s an idea the team is actively considering.

If Google follows through on this idea it won’t be a bad thing for your website. More crawling does not mean better rankings.

Moreover, the idea is to learn which pages need refresh crawls and which pages do not. That means the pages you change more often will continue to be refreshed and updated in search results.

For more details on how Google plans to pull this off, listen to the full discussion in the podcast below (starting at the 2:40 mark):

Advertisement


Featured Image: Alena Veasey/Shutterstock




Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address

SEO

Content Pruning: Why It Works, and How to Do It

Published

on

Content Pruning: Why It Works, and How to Do It

Content pruning sounds pretty appealing: delete a ton of content and see your organic traffic improve. But pruning has risks (like deleting useful pages and useful backlinks), and benefits are not guaranteed: So how does pruning actually work? And when

Read more ›



Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

SEO

8 Free SEO Reporting Tools

Published

on

8 Free SEO Reporting Tools

There’s no shortage of SEO reporting tools to choose from—but what are the core tools you need to put together an SEO report?

In this article, I’ll share eight of my favorite SEO reporting tools to help you create a comprehensive SEO report for free.

Price: Free

Google Search Console, often called GSC, is one of the most widely used tools to track important SEO metrics from Google Search.

Most common reporting use case

GSC has a ton of data to dive into, but the main performance indicator SEOs look at first in GSC is Clicks on the main Overview dashboard.

Advertisement

As the data is from Google, SEOs consider it to be a good barometer for tracking organic search performance. As well as clicks data, you can also track the following from the Performance report:

  • Total Impressions
  • Average CTR
  • Average Position
gsc-performance-overviewgsc-performance-overview

Tip

If you’ve signed up for AWT using Google Search Console, you can view your GSC performance data in Ahrefs by clicking “GSC Performance” from the main dashboard.

But for most SEO reporting, GSC clicks data is exported into a spreadsheet and turned into a chart to visualize year-over-year performance.

organic-traffic-graph-showing-clicks-year-over-yearorganic-traffic-graph-showing-clicks-year-over-year

Favorite feature

One of my favorite reports in GSC is the Indexing report. It’s useful for SEO reporting because you can share the indexed to non-indexed pages ratio in your SEO report.

google-search-console-indexed-pages-reportgoogle-search-console-indexed-pages-report

If the website has a lot of non-indexed pages, then it’s worth reviewing the pages to understand why they haven’t been indexed.

Price: Free

Google Looker Studio (GLS), previously known as Google Data Studio (GDS), is a free tool that helps visualize data in shareable dashboards.

Most common reporting use case

Dashboards are an important part of SEO reporting, and GLS allows you to get a total view of search performance from multiple sources through its integrations.

Advertisement

Out of the box, GLS allows you to connect to many different data sources.

Such as:

  • Marketing products – Google Ads, Google Analytics, Display & Video 360, Search Ads 360
  • Consumer products – Google Sheets, YouTube, and Google Search Console
  • Databases – BigQuery, MySQL, and PostgreSQL
  • Social media platformsFacebook, Reddit, and Twitter
  • Files – CSV file upload and Google Cloud Storage

Sidenote.

If you don’t have the time to create your own report manually, Ahrefs has three Google Looker Studio connectors that can help you create automated SEO reporting for any website in a few clicks

google-looker-studio-partner-connectorsgoogle-looker-studio-partner-connectors

Here’s what a dashboard in GLS looks like:

ahrefs-seo-audit-dashboardahrefs-seo-audit-dashboard
Ahrefs Google Looker Studio integration

With this type of dashboard, you share reports that are easy to understand with clients or other stakeholders.

Favorite feature

The ability to blend and filter data from different sources, like GA and GSC, means you can get a customized overview of your total search performance, tailored to your website.

Price: Free for 500 URLs

Screaming Frog is a website crawler that helps you audit your website.

Advertisement

Screaming Frog’s free version of its crawler is perfect if you want to run a quick audit on a bunch of URLs. The free version is limited to 500 URLs—making it ideal for crawling smaller websites.

screaming-frog-user-interface-screenshotscreaming-frog-user-interface-screenshot

Most common reporting use case

When it comes to reporting, the Reports menu in Screaming Frog SEO Spider has a wealth of information you can look over that covers all the technical aspects of your website, such as analyzing, redirects, canonicals, pagination, hreflang, structured data, and more.

Once you’ve crawled your site, it’s just a matter of downloading the reports you need and working out the main issues to summarize in your SEO report.

Favorite feature

Screaming Frog can pull in data from other tools, including Ahrefs, using APIs. 

If you already had access to a few SEO tools’ APIs, you could pull data from all of them directly into Screaming Frog. This is useful if you want to combine crawl data with performance data or other 3rd party tools.

screaming-frog-api-accessscreaming-frog-api-access

Even if you’ve never configured an API, connecting other tools to Screaming Frog is straightforward.

Price: Free

Ahrefs has a large selection of free SEO tools to help you at every stage of your SEO campaign, and many of these can be used to provide insights for your SEO reporting.

Advertisement
when-to-use-ahrefs-free-tools-across-the-seo-process-illustrationwhen-to-use-ahrefs-free-tools-across-the-seo-process-illustration

For example, you could use our:

Most common reporting use case

One of our most popular free SEO tools is Ahrefs Webmaster Tools (AWT), which you can use for your SEO reporting.

With AWT, you can:

  • Monitor your SEO health over time by setting up scheduled SEO audits
  • See the performance of your website
  • Check all known backlinks for your website
ahrefs-overviewahrefs-overview

Favorite feature

Of all the Ahrefs free tools, my favorite is AWT. Within it, site auditing is my favorite feature—once you’ve set it up, it’s a completely hands-free way to keep track of your website’s technical performance and monitor its health.

If you already have access to Google Search Console, it’s a no-brainer to set up a free AWT account and schedule a technical crawl of your website(s).

Price: Free

Ahrefs’ SEO Toolbar is a free Chrome and Firefox extension useful for diagnosing on-page technical issues and performing quick spot checks on your website’s pages.

Most common reporting use case

For SEO reporting, it’s useful to run an on-page check on your website’s top pages to ensure there aren’t any serious on-page issues.

Advertisement
ahrefs-seo-toolbar-overviewahrefs-seo-toolbar-overview

With the free version, you get the following features:

  • On-page SEO report
  • Redirect tracer with HTTP Headers
  • Outgoing links report with link highlighter and broken link checker
  • SERP positions
  • Country changer for SERP

The SEO toolbar is excellent for spot-checking issues with pages on your website. If you are not confident with inspecting the code, it can also give you valuable pointers on what elements you need to include on your pages to make them search-friendly.

If anything is wrong with the page, the toolbar highlights it, with red indicating a critical issue.

severity-highlight-ahrefs-seo-toolbarseverity-highlight-ahrefs-seo-toolbar

Favorite feature

The section I use the most frequently in the SEO toolbar is the Indexability tab. In this section, you can see whether the page can be crawled and indexed by Google.

indexability-tab-ahrefs-seo-toolbarindexability-tab-ahrefs-seo-toolbar

Although you can do this by inspecting the code manually, using the toolbar is much faster.

Price: Free

Like GSC, Google Analytics is another tool you can use to track the performance of your website, tracking sessions and conversions and much more on your website.

google-analytics-screenshotgoogle-analytics-screenshot

Most common reporting use case

GA gives you a total view of website traffic from several different sources, such as direct, social, organic, paid traffic, and more.

Favorite feature

You can create and track up to 300 events and 30 conversions with GA4. Previously, with universal analytics, you could only track 20 conversions. This makes conversion and event tracking easier within GA4.

Price: Free

Google Slides is Google’s version of Microsoft PowerPoint. If you don’t have a dashboard set up to report on your SEO performance, the next best thing is to assemble a slide deck.

Advertisement

Many SEO agencies present their report through dashboard insights and PowerPoint presentations. However, if you don’t have access to PowerPoint, then Google Slides is an excellent (free) alternative.

google-slides-screenshotgoogle-slides-screenshot

Most common reporting use cases

The most common use of Google Slides is to create a monthly SEO report. If you don’t know what to include in a monthly report, use our SEO report template.

Favorite feature

One of my favorite features is the ability to share your presentation on a video chat directly from Google Slides. You can do this by clicking the camera icon in the top right.

share-video-chat-google-slidesshare-video-chat-google-slides

This is useful if you are working with remote clients and makes sharing your reports easy.

Price: Free

Google Trends allows you to view a keyword’s popularity over time in any country. The data shown is the relative popularity ratio scaled from 0-100, not the direct volume of search queries.

Most common reporting use cases

Google Trends is useful for showing how the popularity of certain searches can increase or decrease over time. If you work with a website that often has trending products, services, or news, it can be useful to illustrate this visually in your SEO report.

Google Trends makes it easy to spot seasonal trends for product categories. For example, people want to buy BBQs when the weather is sunny.

Advertisement

Using Google Trends, we can see that peak demand for BBQs usually happens in June-July every year.

bbq-google-trends-graphbbq-google-trends-graph

Using this data across the last five years, we could be fairly sure when the BBQ season would start and end.

Favorite feature

Comparing two or more search terms against each other over time is one of my favorite uses of Google Trends, as it can be used to tell its own story.

google-trends-comparison-examplegoogle-trends-comparison-example

Embellishing your report with trends data allows you to gain further insights into market trends.

You can even dig into trends at a regional level if you need to.

regional-trends-via-google-trendsregional-trends-via-google-trends

Final thoughts

These free tools will help you put together the foundations for a well-rounded SEO report.

The tools you use for SEO reporting don’t always have to be expensive—even large companies use many of the free tools mentioned to create insights for their client’s SEO reports.

Got more questions? Ping me on X 🙂

Advertisement



Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

SEO

Study Reveals Potential Disruption For Brands & SEO

Published

on

By

Study Reveals Potential Disruption For Brands & SEO

A new study by Authoritas suggests that Google’s AI-powered Search Generative Experience (SGE), currently being tested with a limited group of users, could adversely impact brand visibility and organic search traffic.

These findings include:

  • When an SGE box is expanded, the top organic result drops by over 1,200 pixels on average, significantly reducing visibility.
  • 62% of SGE links come from domains outside the top 10 organic results.
  • Ecommerce, electronics, and fashion-related searches saw the greatest disruption, though all verticals were somewhat impacted.

Adapting to generative search may require a shift in SEO strategies, focusing more on long-form content, expert insights, and multimedia formats.

As Google continues to invest in AI-powered search, the Authoritas study provides an early look at the potential challenges and opportunities ahead.

High Penetration Rate & Industry-Wide Effects

The study analyzed 2,900 brand and product-related keywords across 15 industry verticals and found that Google displays SGE results for 91.4% of all search queries.

The prevalence of SGE results indicates they impact a majority of websites across various industries.

Advertisement

The research analyzed the typical composition of SGE results. On average, each SGE element contained between 10-11 links sourced from an average of four different domains.

This indicates brands may need to earn multiple links and listings within these AI-curated results to maintain visibility and traffic.

The research also suggests that larger, well-established websites like Quora and Reddit will likely perform better in SGE results than smaller websites and lesser-known brands.

Shifting Dynamics In Organic Search Results

With SGE results occupying the entire first page, websites that currently hold the top positions may experience a significant decrease in traffic and click-through rates.

When a user clicks to expand the SGE element, the study found that, on average, the #1 ranked organic result drops a sizeable 1,255 pixels down the page.

Even if a website ranks number one in organic search, it may effectively be pushed down to the second page due to the prominence of SGE results.

Advertisement

New Competition From Unexpected Sources

The study revealed that SGE frequently surfaces links and content from websites that didn’t appear in the top organic rankings.

On average, only 20.1% of SGE links exactly matched a URL from the first page of Google search results.

An additional 17.9% of SGE links were from the same domains as page one results but linked to different pages. The remaining 62% of SGE links came from sources outside the top organic results.

Challenges For Brand Term Optimization & Local Search

The study reveals that SGE results for branded terms may include competitors’ websites alongside the brand’s own site, potentially leading to increased competition for brand visibility.

Laurence O’Toole, CEO and founder of Authoritas, states:

“Brands are not immune. These new types of generative results introduce more opportunities for third-party sites and even competitors to rank for your brand terms and related brand and product terms that you care about.”

Additionally, local businesses may face similar challenges, as SGE results could feature competing local brands even when users search for a specific brand in a regional context.

Advertisement

Methodology & Limitations

To arrive at these insights, Authoritas analyzed a robust dataset of 2,900 search keywords across a spectrum of query types, including specific brand names, brand + generic terms, brand + product names, generic terms, and specific product names. The keywords were distributed across 15 industry verticals.

The study utilized a consistent desktop browser viewport to quantify pixel-based changes in the search results. Authoritas also developed proprietary “alignment scores” to measure the degree of overlap between traditional organic search results and the new SGE links.

While acknowledging some limitations, such as the keyword set needing to be fully representative of each vertical and the still-evolving nature of SGE, Authoritas maintains that the insights hold value in preparing brands for the new realities of an AI-powered search ecosystem.

Why We Care

The findings of the Authoritas study have implications for businesses, marketers, and SEO professionals. As Google’s SGE becomes more prevalent, it could disrupt traditional organic search rankings and traffic patterns.

Brands that have invested heavily in SEO and have achieved top rankings for key terms may find their visibility and click-through rates diminished by the prominence of SGE results.

SGE introduces new competition from unexpected sources, as most SGE links come from domains outside the top 10 organic results. This means businesses may need to compete not only with their traditional rivals but also with a broader range of websites that gain visibility through SGE.

Advertisement

As Google is a primary source of traffic and leads for many businesses, any changes to its search results can impact visibility, brand awareness, and revenue.

How This Could Help You

While the rise of SGE presents challenges, it also offers opportunities.

Taking into account what we’ve learned from the Authoritas study, here are some actionable takeaways:

  • As SGE favors in-depth, informative content, businesses may benefit from investing in comprehensive, well-researched articles and guides that provide value to users.
  • Incorporating expert quotes, interviews, and authoritative sources within your content could increase the likelihood of being featured in SGE results.
  • Enriching your content with images, videos, and other multimedia elements may help capture the attention of both users and the SGE algorithm.
  • Building a strong brand presence across multiple channels, including social media, industry forums, and relevant websites, can increase your chances of appearing in SGE.
  • Creating a trustworthy brand and managing your online reputation will be crucial, as SGE may feature competitors alongside your website.

Looking Ahead

While the long-term impact of SGE will depend on user adoption and the perceived usefulness of results, this study’s findings serve as a valuable starting point for businesses and SEO professionals.

By proactively addressing the challenges and opportunities SGE presents, you can increase your chances of success in the new search environment.


Featured Image: BestForBest/Shutterstock

Source link

Advertisement
Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

Trending

Follow by Email
RSS