SEO
Why Your Rankings Are in Flux
![Why Your Rankings Are in Flux Why Your Rankings Are in Flux](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113571_Why-Your-Rankings-Are-in-Flux.jpg)
Search traffic isn’t just a metric — it’s a business model. Companies like HubSpot, Nerdwallet, and Zapier have built multimillion dollar empires on it.
I’ve personally witnessed a gaming brand use search share of voice data to justify a $60M acquisition and market expansion.
The bottom line is: search rankings and revenue are intrinsically linked. That’s why SERP volatility, even down to the keyword level, can make a huge dent in business value.
Think about it: a law firm ranking for a high-value keyword ($75 CPC) would have to shell out $186K a month on ads if their organic visibility halved, just to keep revenue steady.
Sidenote.
Ahrefs calculates a unique CTR for every single keyword, to give the most accurate page-level traffic figures. For the purpose of this example, however, we’ve used a generic position 2 CTR to work out traffic and value on the keyword level.
Now imagine that multiplied across tens, if not hundreds of equally valuable keywords.
That’s the cost of SERP volatility.
SERP volatility refers to the degree of fluctuation in Google’s search results. In volatile SERPs, different pages shift in and out of the top 10, while stable SERPs stay relatively steady over time.
SERP flux is an inevitability. You’ll experience it across all pages on your site to various extents, and, yes, even “stable” SERPs experience a level of volatility.
In the words of Google’s Senior Search Analyst, John Mueller
…there is no one-shot secret to long-lasting online success. Even if you find something that works now, the web, user desires, and how they engage with websites changes. It’s really hard to make good, popular, persistent things.
The reality is, Google is always testing and indexing to better meet searcher needs.
When the SERPs are volatile, it’s a sign Google is looking for opportunities to serve more relevant content.
What a “stable” SERP looks like
This chart may still look a little shaky, but it was just about the most stable SERP I could find after checking through what felt like hundreds of queries.
Only two sites (highlighted in yellow) dropped out of the top 10 over the course of the year, and the remaining eight pages saw relatively minor flux or position flipping.
In most other SERPs, sites flitted in and out on a near-constant basis.
It’s clear from this example that individual SERPs are intrinsically volatile. Tackling volatility at the keyword-group level is likely a more efficient way to improve your traffic and rankings.
What a “volatile” SERP looks like
![Why Your Rankings Are in Flux An example of a volatile SERP](https://articles.entireweb.com/wp-content/uploads/2024/08/Why-Your-Rankings-Are-in-Flux.jpg)
![Why Your Rankings Are in Flux An example of a volatile SERP](https://articles.entireweb.com/wp-content/uploads/2024/08/Why-Your-Rankings-Are-in-Flux.jpg)
Hopefully you can see the difference here.
In a volatile SERP, pages move in and out of the top 10, experiencing flips and tremors on an almost daily basis.
I’ve highlighted astrology.com because it’s the only result that, despite considerable flux, manages to maintain top 10 visibility all the way throughout the year.
There are two ways to think about SERP volatility
Multi-keyword flux
Multi-keyword flux is when the traffic and rankings of multiple keywords oscillate in the SERPs.
If you’re affected by this kind of volatility, you’ll feel an impact at the page, site, or industry level.
Multi-keyword flux can be triggered by search engine algorithm updates and shifting search intent, amongst other things.
Single keyword flux
Single keyword flux is when your content is experiencing traffic or ranking volatility for one keyword only.
This type of volatility impacts just a single page, but it’s worth investigating if the associated keyword holds significant value – whether that’s monetary value (e.g. CPC) or brand value (e.g. a branded keyword).
When your rankings fluctuate, it’s not by chance. Here’s a look at the main causes of SERP volatility – also known as SEO volatility – including algorithm updates, content issues, and changing search intent.
Algorithm updates
Search engines are always updating their algorithms to a greater or lesser extent, to improve results for users.
As a Google spokesperson told the BBC
“Our recent updates aim to connect people with content that is helpful, satisfying and original, from a diverse range of sites across the web,”
These improvements can cause long-lasting Google volatility across entire industries.
Take the latest August core update, for example. Google made algorithmic adjustments to promote useful content by small and independent publishers, and issued a notice to say that it would take a full month to roll out.
Similarly, the March 2024 core update targeted hundreds of sites with manipulative content, and took 45 long days to complete.
Search engines usually announce major updates in advance, giving the SEO community a chance to address issues before they take root.
![Why Your Rankings Are in Flux Google announcing March 2024 core update](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113566_971_Why-Your-Rankings-Are-in-Flux.png)
![Why Your Rankings Are in Flux Google announcing March 2024 core update](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113566_971_Why-Your-Rankings-Are-in-Flux.png)
I say “usually”, because Google also has a habit of pushing smaller updates live without warning – like this one, where it deindexed a large number of URLs due to a “shift in perception” (in the words of Google Analyst, Gary Illyes).
Just like official updates, unconfirmed updates can lead to wide scale fluctuation.
Content needs refreshing
Search engines want to serve the freshest information possible to keep searchers coming back.
When the SERPs evolve but your content doesn’t, you’ll tend to see some SERP volatility.
If your page is littered with broken links or redundant information, pages delivering a better experience will inevitably outrank you.
Sites ranking for freshness related keywords are more susceptible to movement because searchers are on the lookout for new information. Google refers to these keywords as “Queries that deserve freshness”.
Examples of QDF keywords with volatile SERPs:
Crawling/indexing issues
If Google doesn’t crawl and index your pages properly, your content may appear only fleetingly in search results.
Equally, when resources like JavaScript or CSS are blocked from crawling, Google can misinterpret your page, which leads to peaks and troughs in ranking.
Cannibalization
Ever since the 2019 diversity update, Google has preferred to rank one site per SERP – making exceptions only for highly relevant content.
As a result, when you have two or more pieces of content fulfilling the same search intent, your rankings will flip and traffic can be inconsistent.
You’re essentially muddying the waters by giving search engines too many options; on top of competing with your rivals, you end up competing with yourself.
Irrelevant/low-quality content
There’s a goal behind every query, and it’s reflected in user search behavior; the keywords they choose, the results they click – even the time they spend on site.
Thanks to some leaked Google search documentation shared by SparkToro’s Rand Fishkin and iPullRank’s Mike King, we can now say with quiet confidence that Google processes all of this user activity when it ranks content.
If user behavior indicates that your content is underperforming, your page is more likely to drop in and out of Google.
Low quality content looks like:
- Thin content
- Misinformation
- A lack of E-E-A-T
- Slow load times
- Intrusive ads
- Keyword stuffing
Competitor content improvements
Competitors creating, updating, and ultimately improving their content can cause a certain level of volatility for your site – especially if they have greater brand authority.
Brand authority is the trust and credibility your brand commands in its industry. It’s shaped by factors like the quality of your content, the strength of your backlinks, and how much your brand gets talked about online.
Search intent is changing
Search intent is the thread connecting everything you’ve read so far. When intent is changing or ambiguous, the SERPs become volatile.
Google is ultimately trying to understand what the user is expecting to find when they search a keyword, so it tests and reshuffles results, causing rankings to flip.
Search volatility is even more pronounced when intent shifts and keywords take on a new meaning.
Here’s an example.
Before OpenAI announced ChatGPT, the dominant intent behind the keyword “LLM” was about “Understanding LLM Programs” – in other words, 79% of the top 10 search results catered to users interested in learning about “Master of Law” degrees.
Before
A year on, and by the time ChatGPT had become firmly embedded in our tech stack, the SERP had seen 16 changes in the top 10 and intent had shifted almost entirely to “Understanding Large Language Models” — the technology foundational to modern AI.
After
![Why Your Rankings Are in Flux SERP for LLM keyword after intent switch](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113567_726_Why-Your-Rankings-Are-in-Flux.png)
![Why Your Rankings Are in Flux SERP for LLM keyword after intent switch](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113567_726_Why-Your-Rankings-Are-in-Flux.png)
Sometimes, SERP volatility settles as the search engine better understands the intent. Other times, results stay in flux as a result of constantly changing intent (e.g. queries that deserve freshness).
But at what point does volatility turn into a full-on SERP switch? Can it be measured by the passing of time or the “SERP differential” – the degree to which the results change?
The only way to find out is to study the SERP.
Search results can change slowly
Search intent can shift gradually. Recently, while looking for content to update on the Ahrefs blog, I noticed a piece on “Website traffic” that had once driven impressive amounts of traffic.
But, when I did more digging into the SERPs (using the Identify Intents tool in Keywords Explorer), I noticed a slow shift in intent from informational guides in 2021, to free tools and tool compilation lists in 2024.
![Why Your Rankings Are in Flux Example of slow search intent shift](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113567_481_Why-Your-Rankings-Are-in-Flux.png)
![Why Your Rankings Are in Flux Example of slow search intent shift](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113567_481_Why-Your-Rankings-Are-in-Flux.png)
Looking at the SERP comparison metrics, I noticed the top 10 positions had changed 17 times, and both pages had received a SERP similarity score of just 10/100.
![Why Your Rankings Are in Flux SERP similarity score and top SERP changes in Ahrefs Identify Intents tool](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113567_398_Why-Your-Rankings-Are-in-Flux.png)
![Why Your Rankings Are in Flux SERP similarity score and top SERP changes in Ahrefs Identify Intents tool](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113567_398_Why-Your-Rankings-Are-in-Flux.png)
All signs pointed to slow-burning volatility and a near-total SERP switch.
Search intent can also change quickly
For queries deserving freshness, like the trending “Willy Wonka experience” example I mentioned earlier, the SERPs flipped within months as interest in the viral story waned.
![Why Your Rankings Are in Flux Example of rapid search intent shift](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113567_409_Why-Your-Rankings-Are-in-Flux.jpg)
![Why Your Rankings Are in Flux Example of rapid search intent shift](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113567_409_Why-Your-Rankings-Are-in-Flux.jpg)
This rapid shift was once again reflected in the SERP Similarity score, which came in at just 2/100.
![Why Your Rankings Are in Flux SERP similarity score and top SERP changes in Ahrefs Identify Intents tool](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113568_843_Why-Your-Rankings-Are-in-Flux.png)
![Why Your Rankings Are in Flux SERP similarity score and top SERP changes in Ahrefs Identify Intents tool](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113568_843_Why-Your-Rankings-Are-in-Flux.png)
If you want to better understand how the SERPs are evolving – whether they’re experiencing volatility or undergoing a complete transformation – it’s useful to analyze results in this way.
Volatility can manifest at the page, site, or industry level; to identify each type you’ll need to take a slightly different approach.
Below I run you through the tools you’ll need to spot SERP volatility in all its incarnations.
Check on an industry level
To anticipate when the SERPs are about to shift, many SEOs monitor top-level SERP volatility using algorithm “weather” tools like Algoroo.
This particular tool tracks macro-level SERP volatility by measuring both positive and negative ranking changes. The output is a simple, traffic-light-coded chart showing SERP flux over time.
![Why Your Rankings Are in Flux Algoroo SERP volatility weather chart](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113568_426_Why-Your-Rankings-Are-in-Flux.png)
![Why Your Rankings Are in Flux Algoroo SERP volatility weather chart](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113568_426_Why-Your-Rankings-Are-in-Flux.png)
If you want to analyze your own industry-wide volatility, head to Ahrefs Site Explorer and search your site or subfolder…
![Why Your Rankings Are in Flux Searching in Ahrefs Site Explorer](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113568_535_Why-Your-Rankings-Are-in-Flux.png)
![Why Your Rankings Are in Flux Searching in Ahrefs Site Explorer](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113568_535_Why-Your-Rankings-Are-in-Flux.png)
Then add your competitors in the organic search view.
![Why Your Rankings Are in Flux A Google update show in Ahrefs Share of Voice chart](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113568_812_Why-Your-Rankings-Are-in-Flux.png)
![Why Your Rankings Are in Flux A Google update show in Ahrefs Share of Voice chart](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113568_812_Why-Your-Rankings-Are-in-Flux.png)
In this example, I’m monitoring “Share of Voice” for a group of YMYL ( “Your Money or Your Life”) sites.
Sidenote.
These sites handle content that directly affects a user’s health, finances, and safety. Any incorrect information can cause real-life harm, so search engines have much stricter standards to ensure accuracy and trust. As a result, these sites tend to feel the impact of updates more acutely – meaning we’re able to see the impact of volatility more clearly.
Hovering over the Ⓖ symbols in Site Explorer reveals detailed information about official, macro-level algorithm updates. This helps us tie the obvious SERP volatility we can see to the March 2023 core update.
Share of voice, being a percentage, offers a more accurate way to compare sites than total traffic figures. While traffic can vary widely between sites, share of voice allows you to look past those discrepancies and zero in on relative performance and shared volatility.
Check on a site level
You can easily check site volatility in Google Search Console, based on metrics like clicks and impressions.
![Why Your Rankings Are in Flux Google search console clicks and impressions data](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113568_287_Why-Your-Rankings-Are-in-Flux.png)
![Why Your Rankings Are in Flux Google search console clicks and impressions data](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113568_287_Why-Your-Rankings-Are-in-Flux.png)
And if you want to track site performance across a specific group of keywords, you can set up a Project in Ahrefs Rank Tracker.
![Why Your Rankings Are in Flux Ahrefs Rank Tracker showing share of voice volatility](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113568_874_Why-Your-Rankings-Are-in-Flux.png)
![Why Your Rankings Are in Flux Ahrefs Rank Tracker showing share of voice volatility](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113568_874_Why-Your-Rankings-Are-in-Flux.png)
Check on a page level
To analyze the fluctuation of a single keyword, you can search a keyword in Ahrefs Keywords Explorer and view SERP Position History Charts.
![Why Your Rankings Are in Flux Checking single keyword SERP volatility in Ahrefs Position History](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113568_452_Why-Your-Rankings-Are-in-Flux.png)
![Why Your Rankings Are in Flux Checking single keyword SERP volatility in Ahrefs Position History](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113568_452_Why-Your-Rankings-Are-in-Flux.png)
Focusing on the top 10 results will help you filter out the noise. You can also view Position History Charts whenever you see this chart icon next to a keyword…
![Why Your Rankings Are in Flux Arrow pointing out Position History Charts icon in Ahrefs](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113568_615_Why-Your-Rankings-Are-in-Flux.png)
![Why Your Rankings Are in Flux Arrow pointing out Position History Charts icon in Ahrefs](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113568_615_Why-Your-Rankings-Are-in-Flux.png)
You don’t have to just accept volatility. There are tons of things you can do to stabilize rankings – from improving your content’s E-E-A-T, to diversifying your traffic sources.
It’s beyond the scope of this article to go through every tactic, but I’ve included the most important ones below.
Regularly monitor intent
Search intent changes directly track to volatility. Use the Identifying Intents tool I mentioned earlier will help you assess how well your content hits the mark for the SERP’s dominant intents.
![Why Your Rankings Are in Flux Arrow pointing to Identify Intents tool in Ahrefs](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113568_749_Why-Your-Rankings-Are-in-Flux.png)
![Why Your Rankings Are in Flux Arrow pointing to Identify Intents tool in Ahrefs](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113568_749_Why-Your-Rankings-Are-in-Flux.png)
Keep content updated
Some topics crave freshness more than others. For these, regular updates aren’t just nice to have — they’re essential. Neglect them, and you risk dropping off the SERP entirely.
Here’s what to do:
- Create a “freshness portfolio” of your key content using Ahrefs Portfolios
- Prioritize your top-performing pages
- Update regularly to maintain rankings and prevent traffic loss
![Why Your Rankings Are in Flux Arrow pointing to Portfolios in Ahrefs](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113569_3_Why-Your-Rankings-Are-in-Flux.png)
![Why Your Rankings Are in Flux Arrow pointing to Portfolios in Ahrefs](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113569_3_Why-Your-Rankings-Are-in-Flux.png)
Remember: stale content can cost you traffic and conversions. Stay on top of updates to keep your SERP positions (and your revenue) healthy.
Content you might want to include in your “freshness portfolio”:
- Date specific content/research (e.g. statistics blogs, headlines featuring years)
- News and trend content
- Pages that mention deals and offers
- Pages that mention pricing or plan details
Fix cannibalization
To avoid duplicating content and intent, make sure you’re creating new content in the context of your existing back catalog.
A simple Google site operator search will help you find all existing pages on your site related to a topic.
![Why Your Rankings Are in Flux An example of a Google site operator search](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113569_538_Why-Your-Rankings-Are-in-Flux.png)
![Why Your Rankings Are in Flux An example of a Google site operator search](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113569_538_Why-Your-Rankings-Are-in-Flux.png)
Ahrefs has approximately 19 pages that mention “SERP volatility” in passing, but zero that cover the topic in any real detail.
That means this article shouldn’t lead to any cannibalization (at least, I hope not!).
Toggling on the “Multiple URLs” filter in Ahrefs Organic Keywords report can also show you when you’re ranking more than once for a single keyword.
![Why Your Rankings Are in Flux Arrow pointing to Multiple URLs filter in Ahrefs](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113569_967_Why-Your-Rankings-Are-in-Flux.png)
![Why Your Rankings Are in Flux Arrow pointing to Multiple URLs filter in Ahrefs](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113569_967_Why-Your-Rankings-Are-in-Flux.png)
Use this workflow to spot pages that are at greater risk of cannibalization and SERP volatility.
Study competitor content
Assess rivals who rank consistently or often in volatile SERPs. What are they doing differently? How does their content satisfy intent? What topics are they covering that you’re missing?
Run your page through our Content Grader tool to spot content gaps and get actionable advice on how you can improve.
![Why Your Rankings Are in Flux Example of content gaps in Ahrefs Content Grader](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113569_377_Why-Your-Rankings-Are-in-Flux.jpg)
![Why Your Rankings Are in Flux Example of content gaps in Ahrefs Content Grader](https://articles.entireweb.com/wp-content/uploads/2024/08/1724113569_377_Why-Your-Rankings-Are-in-Flux.jpg)
Audit and troubleshoot technical issues
If your site is experiencing SERP flux off the back of an algorithm update, it’s essential to run a thorough site audit.
When Google’s Site Reputation Abuse update specifically targeted sites with manipulative link practices, SEOs would have had to run an audit to clean up their external link profiles and earn back visibility.
Diagnosing and fixing technical issues can improve your site’s ability to get indexed consistently in the SERPs. Chris Haines has written a great 11 step site audit guide to get you started.
Final thoughts
SERP volatility can hit your traffic and revenue hard.
But stabilizing your rankings isn’t just about patching up a leaky bucket. It’s also about retaining a competitive edge and taking traffic away from your biggest competition.
To keep your site’s performance steady, you need to make a habit of monitoring search intent, regularly updating content, and troubleshooting any technical issues that could be dragging you down.
Do this right, and you might even be able to turn SERP volatility into an opportunity.
SEO
How to Become an SEO Conference Speaker
![How to Become an SEO Conference Speaker How to Become an SEO Conference Speaker](https://articles.entireweb.com/wp-content/uploads/2024/08/1724090168_How-to-Become-an-SEO-Conference-Speaker.jpg)
I asked 11 of SEO’s own headline acts to share their best advice for aspiring speakers.
Here’s their personal roadmap for speaking on the biggest stages, building your personal brand, and even getting flown around the world on someone else’s dollar.
Big dreams often start with small first steps—in this case, hosting webinars and running presentations within your company.
Start little by little with online shorter events first, to longer and then eventually in-person meetups, to later bigger conferences.
Start from in-house presentations where you can present something to your team. It doesn’t even have to be in person. You can just ask people to join a call where you will share your experience with some kind of topic. You will create slides, you will create a presentation, and that will be that. The other thing you can do is give a webinar, which is also public speaking, it’s just not in person—it’s online.
These virtual events allow you to collect social proof for your speaking abilities, making it easier for you to persuade event organizers that you’re a good fit:
Organizers are already going to be familiar with you or they are going to research you online. So, having a strong social media presence, along with articles you’ve written and videos you’ve recorded—including podcasts, webinars, and interviews—goes a long way. Video is key because organizers want to see how you speak and if you can engage an audience.
If you have recordings of yourself speaking, this helps a lot to pitch yourself to organizers because they understand that you did this before. You have experience creating slides, creating coherent presentations, you don’t have stage fright, so you’re not going to fail miserably.
Once you can run a webinar without breaking into too much of a sweat, local meetups are a great next step:
Speaking at local events is a good way to build up your profile in an easier environment. Having a event like SMX as your first speaking gig is more daunting.
As Lily Ray explains, earning your first dose of live audience feedback can be a real confidence booster:
I started with a small meetup in NYC run by Botify, where I spoke on a panel with a few amazing SEOs. I was so nervous! But after hearing some feedback from the audience, that was the moment I realized that I could actually add value to the discussion and people in the room benefitted from hearing what I had to say.
Our CMO Tim views these local meetups as a great way to get reps in and prepare for big stages:
In many cities, there are meetups for 20, 30, or 40 people. Sometimes you can even organize a meetup, invite others, and speak there. It will take you a few years to get there, but once you’ve done some in-house speaking and smaller meetups, and you’ve developed a lot of interesting, unique ideas of your own—not just things you’ve learned from someone else—you can apply to speak at bigger conferences.
When you’re ready for the big time, there’s no real secret to success. You just need to pitch organizers:
Basically, to become a conference speaker, all you need to do is to apply.
I got more gigs by simply pitching event organizers and then sending the organizer a custom note on LinkedIn or via email if I’m connected with them. Don’t be afraid to pitch, get rejected, and then pitch again.
Apply to as many conferences as possible. Research the ones you’re interested in and find out if they have an application process or if they work on an invite-only basis.
Some events are more welcoming of newbie speakers than others, like our beloved brightonSEO:
Some conferences, like brightonSEO, intentionally recruit a certain percentage of new speakers. They have an application process on their website, which I took advantage of when I was starting out.
And speaking of brightonSEO, here’s conference founder Kelvin’s tips for pitching:
Be actionable. Our audience loves practical how-to information. Aim for takeaways like tasks, tools, and books.
Be specific. Specific topics are more likely to be chosen. Detailed titles and descriptions win us over.
Avoid the basics. Our audience is not new to digital marketing. Go beyond general tips and theories.
No self-promotion. Avoid pitches that focus on your tool or client results.
Use research. Relevant and timely data boosts your chances.
Be authoritative. Know your subject inside and out. Show your expertise.
Pitch a tight topic. Our talks are 20 minutes. Narrow topics allow for detailed, focused presentations.
Most of the speakers I asked find their talk topics in the same simple way: their own personal experience.
Speak on topics you know inside and out, where you have a lot of knowledge and opinions, as opposed to trying to speak about things that might be outside your wheelhouse but you feel are important to cover. The more you share things you actually know, the more natural and engaging the talk will come across to the audience.
I also recommend sticking to areas of SEO/ digital you are most comfortable with and have deep subject matter experience. Don’t try and speak about something you’re not an expert in.
And if you’re worried that you don’t have any useful experience—you’d be wrong.
My journey into speaking started with the simple thought that if I could teach people about what I knew surrounding digital marketing, companies and people would eventually pay me to do the things that I was talking about.
Speaking is about sharing what you see to be useful from your own day to day, whatever your experience is! There will be always someone who will find it useful, since it will be another perspective from real experience.
Public speaking is a skill like any other, something that can be improved and developed. Many of SEO’s biggest names still make time to practice, via training, studying famous speakers, or just simply workshopping their presentations out loud:
I highly recommend doing speaker training. You can pick up some great tactics to make your speaking better. I paid for speaker training from my team with the organiser of Ted X, and it was extremely valuable.
Limit the desire to study marketing speakers for inspiration. Instead, study the greatest orators and comedians of all time to better understand how to tell a story that captivates and hooks an audience.
Practice your full talk out loud at least once before you present. This is probably the number one most important rule I’ve learned as a speaker – you need to formulate the sentences out loud a few times for the talk to become muscle memory.
Want to learn from the best speakers in the world?
There’s a Ryan Holiday quote about writing that I regularly share with my team, and it applies equally here: if you want to be a good speaker, “go do interesting things.”
You don’t have to be a world-class orator if you share an interesting idea and jump straight to the good stuff.
Having something interesting to say is critical, in my opinion. I go to lots of conferences where speakers talk about topics that have already been covered many times or that the audience could look up on the internet. But presenting something fresh and interesting is hard. That’s where the value is!
Typically, people want to hear unique information. This is not something you’ve heard before that you want to share on stage, but something you figured out yourself. So, years of experience definitely contribute to your ability to become a speaker.
Never waste time on an introduction. So many speakers waste 5 minutes at the start of their presentation explaining who they are and why they’re important. No one cares. Everyone can Google that information or read your bio in the pamphlet for the event. Get to the good stuff. Deliver value immediately or capture the audience’s attention with something bold.
With a few talks under your belt, you want to think about “niching down”, and focusing your topics on a particular topic or style. As Cyrus explains:
Remember that organizers need to cover a diverse set of topics, so having a “niche” can work in your favor.
Lazarina Stoy has great ideas for finding a niche that feels right for you:
Choose a topic or even talk format that is unique to you. You could do this by doing an ‘audit’ of your processes and compare/contrasting to others.
You could do this by analysis, research, and insights from events (think Lily Ray’s Winners and Losers from Algorithm Updates series).
You could do this by giving an old process a revamp by incorporating new technology or new data to enhance the insights.
You could do this by showing how a process is different in your niche (e.g. on-page SEO for Healthcare).
The opportunities are endless!
The more people you connect with and talk to, the greater the odds that lucky, serendipitous opportunities will present themselves to you. As Andy Chadwick explains:
I landed my first conference speaking opportunity through effective networking at the conferences I attended. I made sure to let the right people know that I was interested in public speaking. When other conferences began seeking recommendations for speakers, those connections recommended me.
Bernard Huang is proactive about understanding and addressing the needs of everyone involved in speaking—organizers, other speakers, and your audience:
“If you do become a conference speaker… you now have 3 bosses to please:
1. The conference organizer—your ability to communicate and coordinate with them on deadlines, topics, reimbursements will be remembered.
2. The audience—what will lead to more potential presentations is your ability to deliver meaningful and relevant information to the audience. Present what you know and understand, but make sure you present in a relatable manner, depending on attendees.
3. Other speakers—a hidden benefit of becoming a speaker is the VIP events that you get to attend. You may initially feel imposter syndrome (I know I still do) at these networking events but make sure to play it cool and see how you can help your fellow speakers. This will go a long way since speakers oftentimes get asked to recommend other speakers for future events.
Final thoughts
This should provide a decent roadmap for working your way onto the biggest stages in the SEO industry. With the practical stuff out of the way, I’ll turn the inspirational final thoughts over to Lazarina:
Even if you just motivate someone else by stepping on the podium or by saying something they needed to hear, that’s a win! So, don’t be afraid to give it a go—we’re all rooting for you.
SEO
13 Essential On-Page SEO Factors You Need To Know
![13 Essential On-Page SEO Factors You Need To Know 13 Essential On-Page SEO Factors You Need To Know](https://articles.entireweb.com/wp-content/uploads/2024/08/13-Essential-On-Page-SEO-Factors-You-Need-To-Know.png)
On-page SEO is fine-tuning various website components to help search engines crawl, understand, and rank pages for relevant queries.
While off-page factors like backlinks and brand signals are critical, optimizing on-page elements lays the groundwork for maximizing search visibility.
Beyond the content itself, on-page factors signal a page’s relevance and quality. The website architecture, including site speed, mobile-friendliness, and URL structures, impacts on-page SEO.
On-page SEO matters because:
- It helps search engines find and show your pages to users.
- Higher-ranked pages get more clicks and visitors.
- Good rankings boost your brand’s trustworthiness.
- It enables you to create content that meets your audience’s needs.
- It’s the foundation for other SEO efforts like building links.
This guide explores 13 essential on-page SEO elements, from E-E-A-T and keyword semantics to HTML tags and site architecture.
13 Essential On-Page SEO Factors
On-page SEO can be divided into content, HTML, and website architecture. We’ll look at each individually.
Content
You’ve heard it before: Content is king.
SEO without it is like a beautiful new sports car without an engine; it might look nice, but it’s going nowhere. But not all content is created equal.
Here are the content factors you need to consider to maximize your on-site SEO:
1. E-E-A-T
One way Google weights your site is based on E-E-A-T, or experience, expertise, authoritativeness, and trustworthiness.
As highlighted in Google’s Search Quality Rater Guidelines, E-E-A-T evaluates the first-hand experience, subject matter expertise, authority, and trustworthiness demonstrated by a website and its content creators.
Google added experience as a new component, signaling the increasing value placed on content created by those with relevant credentials and direct, real-world experience with the topic. This is especially critical for YMYL (Your Money or Your Life) topics like health, finance, safety, etc.
While Google has only confirmed a few E-E-A-T elements like PageRank and links, it’s generally accepted that factors like author expertise, topical authority, transparency, and hands-on experience play a significant role in E-E-A-T evaluations.
2. Keywords
Creating content that includes the words and phrases your target customers are searching for is essential.
However, with advancements in AI and natural language processing, you’ll need to think beyond individual keywords.
Optimize for:
- Semantically related phrases and topics (entities): For example, if you offer cloud data storage services, related entities could include backup solutions, disaster recovery, data management, etc.
- Contextual meaning and intent: A search for “cloud migration” could have different intents, such as technical how-to guides, pricing/cost info, migration strategies, etc.
- Providing comprehensive answers: Cover related subtopics to address customer journeys fully.
Use keyword research tools to identify relevant entities and related queries around your main topics.
Get started by downloading our ebook on keyword research.
3. SEO Writing
Creating content that prioritizes search engines and converts human visitors to your site is an art.
Writing copy that reads well and adheres to SEO best practices can be challenging unless you’ve done it before.
We have an entire piece dedicated to helping you master the art, but some of the key takeaways include:
- Emphasize readability: Your content should be easily scannable so users can quickly find the information they want.
- Don’t overuse keywords: Keyword stuffing is a technique used by unscrupulous SEO professionals to game the system. Google looks down on sites that overuse keywords. If caught, your page could be demoted in SERPs or removed altogether.
- Keep sentences and paragraphs brief: If you’ve ever clicked on a webpage only to be assaulted by an unbroken wall of text, you know how hard it is to read lengthy pieces of copy. Avoid driving users away by keeping your sentences and paragraphs short.
- Use subheadings: Subheads stand out because of their size, attracting attention from people scanning your page. Use an ample amount of content to guide readers down the page.
- Use bulleted lists: This may feel very meta, but bulleted lists are an excellent way to break information into easily digestible chunks. Use them whenever they make sense.
- Add personal experience: Where relevant, discuss the author’s experience, background, and hands-on knowledge related to the topic to demonstrate experience credentials.
4. Freshness
For rapidly evolving topics, keeping your content fresh and providing new value as you learn more about your audience’s needs is critical.
Google rewards sites that maintain their content rather than letting it become stale or outdated.
Some tips:
- Update content regularly with new information, insights, or angles.
- Fix inaccuracies or outdated information promptly.
- Expand content to cover newly discovered areas of audience interest.
- Consider content exports or opt-in offers for frequently updated content.
5. Visual Assets
Adding pictures, videos, charts, and other eye-catching visuals makes it more attractive for visitors and improves its appearance in search results.
Optimizing images can also help you to gain more visibility through image search and in the SERP image carousel.
To make your content easy to find in text searches and image-based searches, here are some tips:
- Provide contextual information and relevant details in image captions.
- Implement schema markup for images, videos, products, etc., to enhance search visibility.
- Ensure visual assets are high-quality, original, and relevant to the page content.
- For ecommerce sites, provide multiple clear product images from various angles.
As computer vision models advance, search engines will better understand and surface relevant images and videos.
Optimizing for visual search now can help future-proof your content.
Read More:
HTML
HyperText Markup Language or HTML is the standard markup language used to structure your webpage and content. It tells the user’s browser where to display what on the page and it also tells search engines what your page is about.
Here are the on-page SEO HTML factors you need to consider:
6. Title Tags
This is one of those areas where it’s essential to focus on the details.
On its own, this snippet of code probably isn’t going to have you shooting up SERP rankings.
However, when combined with other on-page elements (like the ones discussed here), title tags can help you provide context and demonstrate your site’s relevancy.
For a more thorough look at how to optimize your title tags, read this.
7. Meta Description
A veteran SEO professional is throwing up their hands at the screen. “Oh, come on,” they’re saying, “Everyone knows meta descriptions aren’t an SEO ranking factor.”
They’re only partly correct. While it’s true there’s a lot of evidence against meta descriptions as a ranking factor, they’re wrong about everyone knowing that.
But, don’t let them dissuade you from adding meta descriptions to your site.
Despite their relative lack of SEO use, descriptions offer two key benefits:
- They can help Google understand what your webpage is all about.
- They have an outsized influence on your CTRs.
Better meta descriptions give searchers a better understanding of your page, leading to more click-throughs. So, don’t neglect them.
8. Image Optimization
We discussed the importance of visual on-page assets on your page, now it’s time to examine their technical aspects more closely.
Here are some tips to help optimize yours:
- Include SEO-friendly alt tags.
- Choose the proper format and file size for fast loading.
- Customize file names instead of using something like IMG_08759.
- Ensure your images are mobile-friendly.
Once again, we have an excellent resource for more in-depth information on HTML image optimization. Read it here.
9. Geotagging (For Local Search)
It may be a global economy, but most business is still done at a local level. Connect with the people in your neighborhood by optimizing your on-page local SEO.
There are three main SEO tactics to consider when focusing on local traffic:
- Optimizing listings and citations, including name, address, phone number (NAP), website URL, business descriptions, and getting reviews.
- Optimizing local content, including accommodating “near me” searches, providing location-based content, or buying a local website or blog.
- Building links with other local businesses and organizations.
Some additional local SEO tactics to incorporate:
- Implement localized schema markup for local business listings, events, special offers, etc.
- Optimize Google Business Profile with up-to-date info, photos, posts, Q&A, and locally relevant content.
- Leverage proximity and geolocation data for mobile search.
- Create location-specific pages, content hubs, or microsites.
Examples of effective local SEO could look like:
- A restaurant featuring locally sourced food specialties on dedicated pages.
- A service provider’s site with geo-pages for all service areas.
- An ecommerce store highlighting inventory available for local pickup.
For more information on building your geotagging SEO strategy, read this.
Read More:
Website Architecture
Having a well-structured website is essential for two reasons: First, a website laid out logically will be crawled more effectively by search engines, and second, it will create richer user experiences.
Here are the factors to consider when optimizing your site’s architecture:
10. Site Speed
A clunky, slow-loading site does more than frustrate and drive away visitors – it hurts your search ranking, too.
Search Engine Journal investigated the effect of a page’s loading time on SEO and confirmed that page speed is a ranking factor in search results.
However, the minimum speed your site needs to meet is constantly changing.
This can be achieved by meeting Google’s Core Web Vitals minimum threshold. If your site isn’t currently meeting these standards, there are several steps you can take, including:
- Enabling compression.
- Reducing redirects.
- Optimizing images.
- Leveraging browser caches.
11. Responsive Design
Mobile search volume surpassed desktop in 2016 and has only grown since then.
Because more users are on mobile devices, Google followed the logical path and began prioritizing sites with designs that adapt to mobile screens.
While ranking in search results without a responsive design is still possible, Google strongly recommends having one.
You can read more about the effect site responsiveness has on search results here.
12. URL Structure
There was a time when URLs played a prominent role in SEO. Businesses would include keywords in domain names to help them rank higher.
But Google, doing what Google does, changed the algorithm. What was once so important to rankings now plays a much smaller role.
That’s not to say it doesn’t matter. Search engines still include your URLs in your overall score – they just don’t hold the same prominence they once did.
However, there is evidence they play a role in a site’s initial ranking, and some professionals believe they’re used to group pages. While they shouldn’t be your top SEO priority, you don’t want to ignore them.
Read more about how URLs factor into Google rankings here.
13. Links
Remember E-E-A-T from way back at the beginning of this article?
One of the best ways your website can establish it is through links from other reputable websites.
Think of it this way: Who would you rather trust your 401(k) to – a financial advisor who manages Warren Buffet’s portfolio or your cousin Jimmy, who lives in your aunt’s basement? Jimmy might do a fine job, potentially even outperforming Buffet’s guy. But he doesn’t have the credibility that comes with a strong co-sign.
Links work in the same way.
There are three main types you need to know about for SEO:
Of the three, inbound links are the most important for boosting E-E-A-T signals. High-quality, relevant inbound links, especially from authoritative and experienced sources, can help demonstrate your site’s expertise, authoritativeness, and trustworthiness.
SEO professionals use various methods to generate quality incoming links, including social media, creating sharable infographics, and even asking for backlinks.
But beware: Not all inbound links are helpful. Some, especially those from link farms, forum posts, and guestbooks, can be fake links that cheat the rankings system. If you don’t disavow these, it can hurt your ranking.
Here’s information on how and when to disavow links.
Read More:
On-Page SEO Vs. Off-Page SEO
We’ve talked a lot about on-page SEO, but there’s also something known as off-page SEO. The difference, as you could probably tell by the names, is where it happens.
On-page SEO is everything you can do internally to boost your rankings, including keyword optimization, meta descriptions, title tags, alt text, and website structure.
Off-page SEO refers to all external factors that impact your site’s rankings. This includes backlinks, E-E-A-T, local SEO, social media mentions, and pay-per-click.
You have much more control over your on-page SEO, but it’s also important to consider off-page SEO – you need both to achieve your goals.
However, it would be best to first focus on building a good, relevant webpage that’s fully optimized for search engines before you begin investing a lot of resources into building links and promoting your site.
Conclusion
As search algorithms evolve, the need to create high-quality, relevant content and optimize technical elements persists.
Key takeaways to remember:
- Focus on creating valuable, user-centric content that demonstrates E-E-A-T.
- Optimize technical elements like HTML tags, site speed, and mobile responsiveness.
- Maintain a logical site structure and use internal linking effectively.
- Regularly update and refresh content to maintain relevance.
- Remember that on-page SEO works with off-page factors for overall SEO success.
Approach this as an ongoing process rather than a one-time fix.
Consistently implementing these tactics will considerably improve your chances of ranking well in search results.
More resources:
Featured Image: BestForBest/Shutterstock
SEO
Crawl Me Maybe? How Website Crawlers Work
![Crawl Me Maybe? How Website Crawlers Work Crawl Me Maybe? How Website Crawlers Work](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066770_Crawl-Me-Maybe-How-Website-Crawlers-Work.jpg)
You might have heard of website crawling before — you may even have a vague idea of what it’s about — but do you know why it’s important, or what differentiates it from web crawling? (yes, there is a difference!)
Search engines are increasingly ruthless when it comes to the quality of the sites they allow into the search results.
If you don’t grasp the basics of optimizing for web crawlers (and eventual users), your organic traffic may well pay the price.
A good website crawler can show you how to protect and even enhance your site’s visibility.
Here’s what you need to know about both web crawlers and site crawlers.
A web crawler is a software program or script that automatically scours the internet, analyzing and indexing web pages.
Also known as a web spider or spiderbot, web crawlers assess a page’s content to decide how to prioritize it in their indexes.
Googlebot, Google’s web crawler, meticulously browses the web, following links from page to page, gathering data, and processing content for inclusion in Google’s search engine.
How do web crawlers impact SEO?
Web crawlers analyze your page and decide how indexable or rankable it is, which ultimately determines your ability to drive organic traffic.
If you want to be discovered in search results, then it’s important you ready your content for crawling and indexing.
Did you know?
- Visits over 8 billion web pages every 24 hours
- Updates every 15–30 minutes
- Is the #1 most active SEO crawler (and 4th most active crawler worldwide)
There are roughly seven stages to web crawling:
1. URL Discovery
When you publish your page (e.g. to your sitemap), the web crawler discovers it and uses it as a ‘seed’ URL. Just like seeds in the cycle of germination, these starter URLs allow the crawl and subsequent crawling loops to begin.
2. Crawling
After URL discovery, your page is scheduled and then crawled. Content like meta tags, images, links, and structured data are downloaded to the search engine’s servers, where they await parsing and indexing.
3. Parsing
Parsing essentially means analysis. The crawler bot extracts the data it’s just crawled to determine how to index and rank the page.
3a. The URL Discovery Loop
Also during the parsing phase, but worthy of its own subsection, is the URL discovery loop. This is when newly discovered links (including links discovered via redirects) are added to a queue of URLs for the crawler to visit. These are effectively new ‘seed’ URLs, and steps 1–3 get repeated as part of the ‘URL discovery loop’.
4. Indexing
While new URLs are being discovered, the original URL gets indexed. Indexing is when search engines store the data collected from web pages. It enables them to quickly retrieve relevant results for user queries.
5. Ranking
Indexed pages get ranked in search engines based on quality, relevance to search queries, and ability to meet certain other ranking factors. These pages are then served to users when they perform a search.
6. Crawl ends
Eventually the entire crawl (including the URL rediscovery loop) ends based on factors like time allocated, number of pages crawled, depth of links followed etc.
7. Revisiting
Crawlers periodically revisit the page to check for updates, new content, or changes in structure.
![Crawl Me Maybe? How Website Crawlers Work Graphic showing a 7 step flow diagram of how web crawlers work](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066765_364_Crawl-Me-Maybe-How-Website-Crawlers-Work.jpg)
![Crawl Me Maybe? How Website Crawlers Work Graphic showing a 7 step flow diagram of how web crawlers work](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066765_364_Crawl-Me-Maybe-How-Website-Crawlers-Work.jpg)
As you can probably guess, the number of URLs discovered and crawled in this process grows exponentially in just a few hops.
![Crawl Me Maybe? How Website Crawlers Work A graphic visualizing website crawlers following links exponentially](https://articles.entireweb.com/wp-content/uploads/2024/08/Crawl-Me-Maybe-How-Website-Crawlers-Work.png)
![Crawl Me Maybe? How Website Crawlers Work A graphic visualizing website crawlers following links exponentially](https://articles.entireweb.com/wp-content/uploads/2024/08/Crawl-Me-Maybe-How-Website-Crawlers-Work.png)
Search engine web crawlers are autonomous, meaning you can’t trigger them to crawl or switch them on/off at will.
You can, however, notify crawlers of site updates via:
XML sitemaps
An XML sitemap is a file that lists all the important pages on your website to help search engines accurately discover and index your content.
Google’s URL inspection tool
You can ask Google to consider recrawling your site content via its URL inspection tool in Google Search Console. You may get a message in GSC if Google knows about your URL but hasn’t yet crawled or indexed it. If so, find out how to fix “Discovered — currently not indexed”.
IndexNow
Instead of waiting for bots to re-crawl and index your content, you can use IndexNow to automatically ping search engines like Bing, Yandex, Naver, Seznam.cz, and Yep, whenever you:
- Add new pages
- Update existing content
- Remove outdated pages
- Implement redirects
You can set up automatic IndexNow submissions via Ahrefs Site Audit.
![Crawl Me Maybe? How Website Crawlers Work screenshot of IndexNow API key in Ahrefs Site Audit](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_507_Crawl-Me-Maybe-How-Website-Crawlers-Work.jpg)
![Crawl Me Maybe? How Website Crawlers Work screenshot of IndexNow API key in Ahrefs Site Audit](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_507_Crawl-Me-Maybe-How-Website-Crawlers-Work.jpg)
Search engine crawling decisions are dynamic and a little obscure.
Although we don’t know the definitive criteria Google uses to determine when or how often to crawl content, we’ve deduced three of the most important areas.
This is based on breadcrumbs dropped by Google, both in support documentation and during rep interviews.
1. Prioritize quality
Google PageRank evaluates the number and quality of links to a page, considering them as “votes” of importance.
Pages earning quality links are deemed more important and are ranked higher in search results.
PageRank is a foundational part of Google’s algorithm. It makes sense then that the quality of your links and content plays a big part in how your site is crawled and indexed.
To judge your site’s quality, Google looks at factors such as:
To assess the pages on your site with the most links, check out the Best by Links report.
Pay attention to the “First seen”, “Last check” column, which reveals which pages have been crawled most often, and when.
![Crawl Me Maybe? How Website Crawlers Work Ahrefs Best by Links report highlighting first seen last check column](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_903_Crawl-Me-Maybe-How-Website-Crawlers-Work.jpg)
![Crawl Me Maybe? How Website Crawlers Work Ahrefs Best by Links report highlighting first seen last check column](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_903_Crawl-Me-Maybe-How-Website-Crawlers-Work.jpg)
2. Keep things fresh
According to Google’s Senior Search Analyst, John Mueller…
Search engines recrawl URLs at different rates, sometimes it’s multiple times a day, sometimes it’s once every few months.
But if you regularly update your content, you’ll see crawlers dropping by more often.
Search engines like Google want to deliver accurate and up-to-date information to remain competitive and relevant, so updating your content is like dangling a carrot on a stick.
You can examine just how quickly Google processes your updates by checking your crawl stats in Google Search Console.
While you’re there, look at the breakdown of crawling “By purpose” (i.e. percent split of pages refreshed vs pages newly discovered). This will also help you work out just how often you’re encouraging web crawlers to revisit your site.
![Crawl Me Maybe? How Website Crawlers Work 1724066766 671 Crawl Me Maybe How Website Crawlers Work](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_671_Crawl-Me-Maybe-How-Website-Crawlers-Work.png)
![Crawl Me Maybe? How Website Crawlers Work 1724066766 671 Crawl Me Maybe How Website Crawlers Work](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_671_Crawl-Me-Maybe-How-Website-Crawlers-Work.png)
To find specific pages that need updating on your site, head to the Top Pages report in Ahrefs Site Explorer, then:
- Set the traffic filter to “Declined”
- Set the comparison date to the last year or two
- Look at Content Changes status and update pages with only minor changes
![Crawl Me Maybe? How Website Crawlers Work 3 part process of updating pages based on content changes in Ahrefs](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_222_Crawl-Me-Maybe-How-Website-Crawlers-Work.png)
![Crawl Me Maybe? How Website Crawlers Work 3 part process of updating pages based on content changes in Ahrefs](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_222_Crawl-Me-Maybe-How-Website-Crawlers-Work.png)
Top Pages shows you the content on your site driving the most organic traffic. Pushing updates to these pages will encourage crawlers to visit your best content more often, and (hopefully) boost any declining traffic.
3. Refine your site structure
Offering a clear site structure via a logical sitemap, and backing that up with relevant internal links will help crawlers:
- Better navigate your site
- Understand its hierarchy
- Index and rank your most valuable content
Combined, these factors will also please users, since they support easy navigation, reduced bounce rates, and increased engagement.
Below are some more elements that can potentially influence how your site gets discovered and prioritized in crawling:
![Crawl Me Maybe? How Website Crawlers Work Graphic showing the factors that can affect web crawl discoverability](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_393_Crawl-Me-Maybe-How-Website-Crawlers-Work.png)
![Crawl Me Maybe? How Website Crawlers Work Graphic showing the factors that can affect web crawl discoverability](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_393_Crawl-Me-Maybe-How-Website-Crawlers-Work.png)
What is crawl budget?
For this reason, each site has a crawl budget, which is the number of URLs a crawler can and wants to crawl. Factors like site speed, mobile-friendliness, and a logical site structure impact the efficacy of crawl budget.
For a deeper dive into crawl budgets, check out Patrick Stox’s guide: When Should You Worry About Crawl Budget?
Web crawlers like Google crawl the entire internet, and you can’t control which sites they visit, or how often.
But you can use website crawlers, which are like your own private bots.
Ask them to crawl your website to find and fix important SEO problems, or study your competitors’ site, turning their biggest weaknesses into your opportunities.
Site crawlers essentially simulate search performance. They help you understand how a search engine’s web crawlers might interpret your pages, based on their:
- Structure
- Content
- Meta data
- Page load speed
- Errors
- Etc
Example: Ahrefs Site Audit
The Ahrefs Site Audit crawler powers the tools: RankTracker, Projects, and Ahrefs’ main website crawling tool: Site Audit.
Site Audit helps SEOs to:
- Analyze 170+ technical SEO issues
- Conduct on-demand crawls, with live site performance data
- Assess up to 170k URLs a minute
- Troubleshoot, maintain, and improve their visibility in search engines
From URL discovery to revisiting, website crawlers operate very similarly to web crawlers – only instead of indexing and ranking your page in the SERPs, they store and analyze it in their own database.
You can crawl your site either locally or remotely. Desktop crawlers like ScreamingFrog let you download and customize your site crawl, while cloud-based tools like Ahrefs Site Audit perform the crawl without using your computer’s resources – helping you work collaboratively on fixes and site optimization.
If you want to scan entire websites in real time to detect technical SEO problems, configure a crawl in Site Audit.
It will give you visual data breakdowns, site health scores, and detailed fix recommendations to help you understand how a search engine interprets your site.
1. Set up your crawl
Navigate to the Site Audit tab and choose an existing project, or set one up.
![Crawl Me Maybe? How Website Crawlers Work Screenshot of import/add project page in Ahrefs Site Audit](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_528_Crawl-Me-Maybe-How-Website-Crawlers-Work.png)
![Crawl Me Maybe? How Website Crawlers Work Screenshot of import/add project page in Ahrefs Site Audit](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_528_Crawl-Me-Maybe-How-Website-Crawlers-Work.png)
A project is any domain, subdomain, or URL you want to track over time.
Once you’ve configured your crawl settings – including your crawl schedule and URL sources – you can start your audit and you’ll be notified as soon as it’s complete.
Here are some things you can do right away.
2. Diagnose top errors
The Top Issues overview in Site Audit shows you your most pressing errors, warnings, and notices, based on the number of URLs affected.
![Crawl Me Maybe? How Website Crawlers Work 1724066766 700 Crawl Me Maybe How Website Crawlers Work](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_700_Crawl-Me-Maybe-How-Website-Crawlers-Work.png)
![Crawl Me Maybe? How Website Crawlers Work 1724066766 700 Crawl Me Maybe How Website Crawlers Work](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_700_Crawl-Me-Maybe-How-Website-Crawlers-Work.png)
Working through these as part of your SEO roadmap will help you:
1. Spot errors (red icons) impacting crawling – e.g.
- HTTP status code/client errors
- Broken links
- Canonical issues
2. Optimize your content and rankings based on warnings (yellow) – e.g.
- Missing alt text
- Links to redirects
- Overly long meta descriptions
3. Maintain steady visibility with notices (blue icon) – e.g.
- Organic traffic drops
- Multiple H1s
- Indexable pages not in sitemap
Filter issues
You can also prioritize fixes using filters.
Say you have thousands of pages with missing meta descriptions. Make the task more manageable and impactful by targeting high traffic pages first.
- Head to the Page Explorer report in Site Audit
- Select the advanced filter dropdown
- Set an internal pages filter
- Select an ‘And’ operator
- Select ‘Meta description’ and ‘Not exists’
- Select ‘Organic traffic > 100’
![Crawl Me Maybe? How Website Crawlers Work Screenshot of how to find pages with missing meta descriptions, over 100 organic traffic, in Ahrefs Page Explorer](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_682_Crawl-Me-Maybe-How-Website-Crawlers-Work.png)
![Crawl Me Maybe? How Website Crawlers Work Screenshot of how to find pages with missing meta descriptions, over 100 organic traffic, in Ahrefs Page Explorer](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_682_Crawl-Me-Maybe-How-Website-Crawlers-Work.png)
Crawl the most important parts of your site
Segment and zero-in on the most important pages on your site (e.g. subfolders or subdomains) using Site Audit’s 200+ filters – whether that’s your blog, ecommerce store, or even pages that earn over a certain traffic threshold.
![Crawl Me Maybe? How Website Crawlers Work Screenshot of Ahrefs Site Audit pointing out configure segment option](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_901_Crawl-Me-Maybe-How-Website-Crawlers-Work.jpg)
![Crawl Me Maybe? How Website Crawlers Work Screenshot of Ahrefs Site Audit pointing out configure segment option](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_901_Crawl-Me-Maybe-How-Website-Crawlers-Work.jpg)
3. Expedite fixes
If you don’t have coding experience, then the prospect of crawling your site and implementing fixes can be intimidating.
If you do have dev support, issues are easier to remedy, but then it becomes a matter of bargaining for another person’s time.
We’ve got a new feature on the way to help you solve for these kinds of headaches.
Coming soon, Patches are fixes you can make autonomously in Site Audit.
![Crawl Me Maybe? How Website Crawlers Work Screenshot of Ahrefs Patches tool calling out the Patch It feature](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_653_Crawl-Me-Maybe-How-Website-Crawlers-Work.png)
![Crawl Me Maybe? How Website Crawlers Work Screenshot of Ahrefs Patches tool calling out the Patch It feature](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_653_Crawl-Me-Maybe-How-Website-Crawlers-Work.png)
Title changes, missing meta descriptions, site-wide broken links – when you face these kinds of errors you can hit “Patch it” to publish a fix directly to your website, without having to pester a dev.
And if you’re unsure of anything, you can roll-back your patches at any point.
![Crawl Me Maybe? How Website Crawlers Work Screenshot of Ahrefs Patches tool calling out drafts, published, and unpublished statuses](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_602_Crawl-Me-Maybe-How-Website-Crawlers-Work.png)
![Crawl Me Maybe? How Website Crawlers Work Screenshot of Ahrefs Patches tool calling out drafts, published, and unpublished statuses](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_602_Crawl-Me-Maybe-How-Website-Crawlers-Work.png)
4. Spot optimization opportunities
Auditing your site with a website crawler is as much about spotting opportunities as it is about fixing bugs.
Improve internal linking
The Internal Link Opportunities report in Site Audit shows you relevant internal linking suggestions, by taking the top 10 keywords (by traffic) for each crawled page, then looking for mentions of them on your other crawled pages.
‘Source’ pages are the ones you should link from, and ‘Target’ pages are the ones you should link to.
![Crawl Me Maybe? How Website Crawlers Work Screenshot of Internal Link Opportunities report in Ahrefs Site Audit highlighting source page and target page](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_940_Crawl-Me-Maybe-How-Website-Crawlers-Work.png)
![Crawl Me Maybe? How Website Crawlers Work Screenshot of Internal Link Opportunities report in Ahrefs Site Audit highlighting source page and target page](https://articles.entireweb.com/wp-content/uploads/2024/08/1724066766_940_Crawl-Me-Maybe-How-Website-Crawlers-Work.png)
The more high quality connections you make between your content, the easier it will be for Googlebot to crawl your site.
Final thoughts
Understanding website crawling is more than just an SEO hack – it’s foundational knowledge that directly impacts your traffic and ROI.
Knowing how crawlers work means knowing how search engines “see” your site, and that’s half the battle when it comes to ranking.
-
SEARCHENGINES5 days ago
Daily Search Forum Recap: August 14, 2024
-
SEARCHENGINES6 days ago
Daily Search Forum Recap: August 13, 2024
-
AFFILIATE MARKETING6 days ago
Musk, Trump Hit With Federal Labor Charges Over Livestream
-
SEO6 days ago
ChatGPT Study Finds Training Data Doesn’t Match Real-World Use
-
SEARCHENGINES2 days ago
Google August Core Update, Big Ranking Bug, DOJ Google Breakup, AI Overview Links, Gemini Research & More
-
SEO5 days ago
The Long-Term Strategy of Building A Personal Brand Through Content And Value With Eric Enge
-
SEARCHENGINES4 days ago
Daily Search Forum Recap: August 15, 2024
-
SEO5 days ago
Anthropic Announces Prompt Caching With Claude