I asked 11 of SEO’s own headline acts to share their best advice for aspiring speakers.

Here’s their personal roadmap for speaking on the biggest stages, building your personal brand, and even getting flown around the world on someone else’s dollar.

Big dreams often start with small first steps—in this case, hosting webinars and running presentations within your company.

Start little by little with online shorter events first, to longer and then eventually in-person meetups, to later bigger conferences.

Start from in-house presentations where you can present something to your team. It doesn’t even have to be in person. You can just ask people to join a call where you will share your experience with some kind of topic. You will create slides, you will create a presentation, and that will be that. The other thing you can do is give a webinar, which is also public speaking, it’s just not in person—it’s online.

These virtual events allow you to collect social proof for your speaking abilities, making it easier for you to persuade event organizers that you’re a good fit:

Organizers are already going to be familiar with you or they are going to research you online. So, having a strong social media presence, along with articles you’ve written and videos you’ve recorded—including podcasts, webinars, and interviews—goes a long way. Video is key because organizers want to see how you speak and if you can engage an audience.

If you have recordings of yourself speaking, this helps a lot to pitch yourself to organizers because they understand that you did this before. You have experience creating slides, creating coherent presentations, you don’t have stage fright, so you’re not going to fail miserably.

Once you can run a webinar without breaking into too much of a sweat, local meetups are a great next step:

Speaking at local events is a good way to build up your profile in an easier environment. Having a event like SMX as your first speaking gig is more daunting. 

As Lily Ray explains, earning your first dose of live audience feedback can be a real confidence booster:

I started with a small meetup in NYC run by Botify, where I spoke on a panel with a few amazing SEOs. I was so nervous! But after hearing some feedback from the audience, that was the moment I realized that I could actually add value to the discussion and people in the room benefitted from hearing what I had to say.

Our CMO Tim views these local meetups as a great way to get reps in and prepare for big stages:

In many cities, there are meetups for 20, 30, or 40 people. Sometimes you can even organize a meetup, invite others, and speak there. It will take you a few years to get there, but once you’ve done some in-house speaking and smaller meetups, and you’ve developed a lot of interesting, unique ideas of your own—not just things you’ve learned from someone else—you can apply to speak at bigger conferences.

When you’re ready for the big time, there’s no real secret to success. You just need to pitch organizers:

Basically, to become a conference speaker, all you need to do is to apply.

I got more gigs by simply pitching event organizers and then sending the organizer a custom note on LinkedIn or via email if I’m connected with them. Don’t be afraid to pitch, get rejected, and then pitch again.

Apply to as many conferences as possible. Research the ones you’re interested in and find out if they have an application process or if they work on an invite-only basis.

Some events are more welcoming of newbie speakers than others, like our beloved brightonSEO:

Some conferences, like brightonSEO, intentionally recruit a certain percentage of new speakers. They have an application process on their website, which I took advantage of when I was starting out.

And speaking of brightonSEO, here’s conference founder Kelvin’s tips for pitching:

Be actionable. Our audience loves practical how-to information. Aim for takeaways like tasks, tools, and books.

Be specific. Specific topics are more likely to be chosen. Detailed titles and descriptions win us over.

Avoid the basics. Our audience is not new to digital marketing. Go beyond general tips and theories.

No self-promotion. Avoid pitches that focus on your tool or client results.

Use research. Relevant and timely data boosts your chances.

Be authoritative. Know your subject inside and out. Show your expertise.

Pitch a tight topic. Our talks are 20 minutes. Narrow topics allow for detailed, focused presentations.

Most of the speakers I asked find their talk topics in the same simple way: their own personal experience.

Speak on topics you know inside and out, where you have a lot of knowledge and opinions, as opposed to trying to speak about things that might be outside your wheelhouse but you feel are important to cover. The more you share things you actually know, the more natural and engaging the talk will come across to the audience.

I also recommend sticking to areas of SEO/ digital you are most comfortable with and have deep subject matter experience. Don’t try and speak about something you’re not an expert in.

And if you’re worried that you don’t have any useful experience—you’d be wrong.

My journey into speaking started with the simple thought that if I could teach people about what I knew surrounding digital marketing, companies and people would eventually pay me to do the things that I was talking about. 

Speaking is about sharing what you see to be useful from your own day to day, whatever your experience is! There will be always someone who will find it useful, since it will be another perspective from real experience.

Public speaking is a skill like any other, something that can be improved and developed. Many of SEO’s biggest names still make time to practice, via training, studying famous speakers, or just simply workshopping their presentations out loud:

I highly recommend doing speaker training. You can pick up some great tactics to make your speaking better. I paid for speaker training from my team with the organiser of Ted X, and it was extremely valuable. 

Limit the desire to study marketing speakers for inspiration. Instead, study the greatest orators and comedians of all time to better understand how to tell a story that captivates and hooks an audience. 

Practice your full talk out loud at least once before you present. This is probably the number one most important rule I’ve learned as a speaker – you need to formulate the sentences out loud a few times for the talk to become muscle memory.

There’s a Ryan Holiday quote about writing that I regularly share with my team, and it applies equally here: if you want to be a good speaker, “go do interesting things.”

You don’t have to be a world-class orator if you share an interesting idea and jump straight to the good stuff.

Having something interesting to say is critical, in my opinion. I go to lots of conferences where speakers talk about topics that have already been covered many times or that the audience could look up on the internet. But presenting something fresh and interesting is hard. That’s where the value is!

Typically, people want to hear unique information. This is not something you’ve heard before that you want to share on stage, but something you figured out yourself. So, years of experience definitely contribute to your ability to become a speaker.

Never waste time on an introduction. So many speakers waste 5 minutes at the start of their presentation explaining who they are and why they’re important. No one cares. Everyone can Google that information or read your bio in the pamphlet for the event. Get to the good stuff. Deliver value immediately or capture the audience’s attention with something bold. 

With a few talks under your belt, you want to think about “niching down”, and focusing your topics on a particular topic or style. As Cyrus explains:

Remember that organizers need to cover a diverse set of topics, so having a “niche” can work in your favor.

Lazarina Stoy has great ideas for finding a niche that feels right for you:

Choose a topic or even talk format that is unique to you. You could do this by doing an ‘audit’ of your processes and compare/contrasting to others.

You could do this by analysis, research, and insights from events (think Lily Ray’s Winners and Losers from Algorithm Updates series).

You could do this by giving an old process a revamp by incorporating new technology or new data to enhance the insights.

You could do this by showing how a process is different in your niche (e.g. on-page SEO for Healthcare).

The opportunities are endless!

The more people you connect with and talk to, the greater the odds that lucky, serendipitous opportunities will present themselves to you. As Andy Chadwick explains:

I landed my first conference speaking opportunity through effective networking at the conferences I attended. I made sure to let the right people know that I was interested in public speaking. When other conferences began seeking recommendations for speakers, those connections recommended me.

Bernard Huang is proactive about understanding and addressing the needs of everyone involved in speaking—organizers, other speakers, and your audience:

“If you do become a conference speaker… you now have 3 bosses to please:

1. The conference organizer—your ability to communicate and coordinate with them on deadlines, topics, reimbursements will be remembered.

2. The audience—what will lead to more potential presentations is your ability to deliver meaningful and relevant information to the audience. Present what you know and understand, but make sure you present in a relatable manner, depending on attendees.

3. Other speakers—a hidden benefit of becoming a speaker is the VIP events that you get to attend. You may initially feel imposter syndrome (I know I still do) at these networking events but make sure to play it cool and see how you can help your fellow speakers. This will go a long way since speakers oftentimes get asked to recommend other speakers for future events.

Final thoughts

This should provide a decent roadmap for working your way onto the biggest stages in the SEO industry. With the practical stuff out of the way, I’ll turn the inspirational final thoughts over to Lazarina:

Even if you just motivate someone else by stepping on the podium or by saying something they needed to hear, that’s a win! So, don’t be afraid to give it a go—we’re all rooting for you.

On-page SEO is fine-tuning various website components to help search engines crawl, understand, and rank pages for relevant queries.

While off-page factors like backlinks and brand signals are critical, optimizing on-page elements lays the groundwork for maximizing search visibility.

Beyond the content itself, on-page factors signal a page’s relevance and quality. The website architecture, including site speed, mobile-friendliness, and URL structures, impacts on-page SEO.

On-page SEO matters because:

• It helps search engines find and show your pages to users.
• Higher-ranked pages get more clicks and visitors.
• Good rankings boost your brand’s trustworthiness.
• It enables you to create content that meets your audience’s needs.
• It’s the foundation for other SEO efforts like building links.

This guide explores 13 essential on-page SEO elements, from E-E-A-T and keyword semantics to HTML tags and site architecture.

13 Essential On-Page SEO Factors

On-page SEO can be divided into content, HTML, and website architecture. We’ll look at each individually.

Content

You’ve heard it before: Content is king.

SEO without it is like a beautiful new sports car without an engine; it might look nice, but it’s going nowhere. But not all content is created equal.

Here are the content factors you need to consider to maximize your on-site SEO:

1. E-E-A-T

One way Google weights your site is based on E-E-A-T, or experience, expertise, authoritativeness, and trustworthiness.

As highlighted in Google’s Search Quality Rater Guidelines, E-E-A-T evaluates the first-hand experience, subject matter expertise, authority, and trustworthiness demonstrated by a website and its content creators.

Google added experience as a new component, signaling the increasing value placed on content created by those with relevant credentials and direct, real-world experience with the topic. This is especially critical for YMYL (Your Money or Your Life) topics like health, finance, safety, etc.

While Google has only confirmed a few E-E-A-T elements like PageRank and links, it’s generally accepted that factors like author expertise, topical authority, transparency, and hands-on experience play a significant role in E-E-A-T evaluations.

2. Keywords

Creating content that includes the words and phrases your target customers are searching for is essential.

However, with advancements in AI and natural language processing, you’ll need to think beyond individual keywords.

Optimize for:

• Semantically related phrases and topics (entities): For example, if you offer cloud data storage services, related entities could include backup solutions, disaster recovery, data management, etc.
• Contextual meaning and intent: A search for “cloud migration” could have different intents, such as technical how-to guides, pricing/cost info, migration strategies, etc.
• Providing comprehensive answers: Cover related subtopics to address customer journeys fully.

Use keyword research tools to identify relevant entities and related queries around your main topics.

Get started by downloading our ebook on keyword research.

3. SEO Writing

Creating content that prioritizes search engines and converts human visitors to your site is an art.

Writing copy that reads well and adheres to SEO best practices can be challenging unless you’ve done it before.

We have an entire piece dedicated to helping you master the art, but some of the key takeaways include:

• Emphasize readability: Your content should be easily scannable so users can quickly find the information they want.
• Don’t overuse keywords: Keyword stuffing is a technique used by unscrupulous SEO professionals to game the system. Google looks down on sites that overuse keywords. If caught, your page could be demoted in SERPs or removed altogether.
• Keep sentences and paragraphs brief: If you’ve ever clicked on a webpage only to be assaulted by an unbroken wall of text, you know how hard it is to read lengthy pieces of copy. Avoid driving users away by keeping your sentences and paragraphs short.
• Use subheadings: Subheads stand out because of their size, attracting attention from people scanning your page. Use an ample amount of content to guide readers down the page.
• Use bulleted lists: This may feel very meta, but bulleted lists are an excellent way to break information into easily digestible chunks. Use them whenever they make sense.
• Add personal experience: Where relevant, discuss the author’s experience, background, and hands-on knowledge related to the topic to demonstrate experience credentials.

4. Freshness

For rapidly evolving topics, keeping your content fresh and providing new value as you learn more about your audience’s needs is critical.

Google rewards sites that maintain their content rather than letting it become stale or outdated.

Some tips:

• Update content regularly with new information, insights, or angles.
• Fix inaccuracies or outdated information promptly.
• Expand content to cover newly discovered areas of audience interest.
• Consider content exports or opt-in offers for frequently updated content.

5. Visual Assets

Adding pictures, videos, charts, and other eye-catching visuals makes it more attractive for visitors and improves its appearance in search results.

Optimizing images can also help you to gain more visibility through image search and in the SERP image carousel.

To make your content easy to find in text searches and image-based searches, here are some tips:

• Provide contextual information and relevant details in image captions.
• Implement schema markup for images, videos, products, etc., to enhance search visibility.
• Ensure visual assets are high-quality, original, and relevant to the page content.
• For ecommerce sites, provide multiple clear product images from various angles.

As computer vision models advance, search engines will better understand and surface relevant images and videos.

Optimizing for visual search now can help future-proof your content.

Read More:

HTML

HyperText Markup Language or HTML is the standard markup language used to structure your webpage and content. It tells the user’s browser where to display what on the page and it also tells search engines what your page is about.

Here are the on-page SEO HTML factors you need to consider:

6. Title Tags

This is one of those areas where it’s essential to focus on the details.

On its own, this snippet of code probably isn’t going to have you shooting up SERP rankings.

However, when combined with other on-page elements (like the ones discussed here), title tags can help you provide context and demonstrate your site’s relevancy.

For a more thorough look at how to optimize your title tags, read this.

7. Meta Description

A veteran SEO professional is throwing up their hands at the screen. “Oh, come on,” they’re saying, “Everyone knows meta descriptions aren’t an SEO ranking factor.”

They’re only partly correct. While it’s true there’s a lot of evidence against meta descriptions as a ranking factor, they’re wrong about everyone knowing that.

But, don’t let them dissuade you from adding meta descriptions to your site.

Despite their relative lack of SEO use, descriptions offer two key benefits:

• They can help Google understand what your webpage is all about.
• They have an outsized influence on your CTRs.

Better meta descriptions give searchers a better understanding of your page, leading to more click-throughs. So, don’t neglect them.

8. Image Optimization

We discussed the importance of visual on-page assets on your page, now it’s time to examine their technical aspects more closely.

Here are some tips to help optimize yours:

• Include SEO-friendly alt tags.
• Choose the proper format and file size for fast loading.
• Customize file names instead of using something like IMG_08759.
• Ensure your images are mobile-friendly.

Once again, we have an excellent resource for more in-depth information on HTML image optimization. Read it here.

9. Geotagging (For Local Search)

It may be a global economy, but most business is still done at a local level. Connect with the people in your neighborhood by optimizing your on-page local SEO.

There are three main SEO tactics to consider when focusing on local traffic:

• Optimizing listings and citations, including name, address, phone number (NAP), website URL, business descriptions, and getting reviews.
• Optimizing local content, including accommodating “near me” searches, providing location-based content, or buying a local website or blog.
•  Building links with other local businesses and organizations.

Some additional local SEO tactics to incorporate:

• Implement localized schema markup for local business listings, events, special offers, etc.
• Optimize Google Business Profile with up-to-date info, photos, posts, Q&A, and locally relevant content.
• Leverage proximity and geolocation data for mobile search.
• Create location-specific pages, content hubs, or microsites.

Examples of effective local SEO could look like:

• A restaurant featuring locally sourced food specialties on dedicated pages.
• A service provider’s site with geo-pages for all service areas.
• An ecommerce store highlighting inventory available for local pickup.

For more information on building your geotagging SEO strategy, read this.

Read More:

Website Architecture

Having a well-structured website is essential for two reasons: First, a website laid out logically will be crawled more effectively by search engines, and second, it will create richer user experiences.

Here are the factors to consider when optimizing your site’s architecture:

10. Site Speed

A clunky, slow-loading site does more than frustrate and drive away visitors – it hurts your search ranking, too.

Search Engine Journal investigated the effect of a page’s loading time on SEO and confirmed that page speed is a ranking factor in search results.

However, the minimum speed your site needs to meet is constantly changing.

This can be achieved by meeting Google’s Core Web Vitals minimum threshold. If your site isn’t currently meeting these standards, there are several steps you can take, including:

• Enabling compression.
• Reducing redirects.
• Optimizing images.
• Leveraging browser caches.

11. Responsive Design

Mobile search volume surpassed desktop in 2016 and has only grown since then.

Because more users are on mobile devices, Google followed the logical path and began prioritizing sites with designs that adapt to mobile screens.

While ranking in search results without a responsive design is still possible, Google strongly recommends having one.

You can read more about the effect site responsiveness has on search results here.

12. URL Structure

There was a time when URLs played a prominent role in SEO. Businesses would include keywords in domain names to help them rank higher.

But Google, doing what Google does, changed the algorithm. What was once so important to rankings now plays a much smaller role.

That’s not to say it doesn’t matter. Search engines still include your URLs in your overall score – they just don’t hold the same prominence they once did.

However, there is evidence they play a role in a site’s initial ranking, and some professionals believe they’re used to group pages. While they shouldn’t be your top SEO priority, you don’t want to ignore them.

Read more about how URLs factor into Google rankings here.

13. Links

Remember E-E-A-T from way back at the beginning of this article?

One of the best ways your website can establish it is through links from other reputable websites.

Think of it this way: Who would you rather trust your 401(k) to – a financial advisor who manages Warren Buffet’s portfolio or your cousin Jimmy, who lives in your aunt’s basement? Jimmy might do a fine job, potentially even outperforming Buffet’s guy. But he doesn’t have the credibility that comes with a strong co-sign.

Links work in the same way.

There are three main types you need to know about for SEO:

Of the three, inbound links are the most important for boosting E-E-A-T signals. High-quality, relevant inbound links, especially from authoritative and experienced sources, can help demonstrate your site’s expertise, authoritativeness, and trustworthiness.

SEO professionals use various methods to generate quality incoming links, including social media, creating sharable infographics, and even asking for backlinks.

But beware: Not all inbound links are helpful. Some, especially those from link farms, forum posts, and guestbooks, can be fake links that cheat the rankings system. If you don’t disavow these, it can hurt your ranking.

Here’s information on how and when to disavow links.

Read More:

On-Page SEO Vs. Off-Page SEO

We’ve talked a lot about on-page SEO, but there’s also something known as off-page SEO. The difference, as you could probably tell by the names, is where it happens.

On-page SEO is everything you can do internally to boost your rankings, including keyword optimization, meta descriptions, title tags, alt text, and website structure.

Off-page SEO refers to all external factors that impact your site’s rankings. This includes backlinks, E-E-A-T, local SEO, social media mentions, and pay-per-click.

You have much more control over your on-page SEO, but it’s also important to consider off-page SEO – you need both to achieve your goals.

However, it would be best to first focus on building a good, relevant webpage that’s fully optimized for search engines before you begin investing a lot of resources into building links and promoting your site.

Conclusion

As search algorithms evolve, the need to create high-quality, relevant content and optimize technical elements persists.

Key takeaways to remember:

1. Focus on creating valuable, user-centric content that demonstrates E-E-A-T.
2. Optimize technical elements like HTML tags, site speed, and mobile responsiveness.
3. Maintain a logical site structure and use internal linking effectively.
4. Regularly update and refresh content to maintain relevance.
5. Remember that on-page SEO works with off-page factors for overall SEO success.

Approach this as an ongoing process rather than a one-time fix.

Consistently implementing these tactics will considerably improve your chances of ranking well in search results.

More resources: 

---

Featured Image: BestForBest/Shutterstock

You might have heard of website crawling before — you may even have a vague idea of what it’s about — but do you know why it’s important, or what differentiates it from web crawling? (yes, there is a difference!) 

Search engines are increasingly ruthless when it comes to the quality of the sites they allow into the search results.

If you don’t grasp the basics of optimizing for web crawlers (and eventual users), your organic traffic may well pay the price.

A good website crawler can show you how to protect and even enhance your site’s visibility.

Here’s what you need to know about both web crawlers and site crawlers.

A web crawler is a software program or script that automatically scours the internet, analyzing and indexing web pages.

Also known as a web spider or spiderbot, web crawlers assess a page’s content to decide how to prioritize it in their indexes.

Googlebot, Google’s web crawler, meticulously browses the web, following links from page to page, gathering data, and processing content for inclusion in Google’s search engine.

How do web crawlers impact SEO?

Web crawlers analyze your page and decide how indexable or rankable it is, which ultimately determines your ability to drive organic traffic.

If you want to be discovered in search results, then it’s important you ready your content for crawling and indexing.

There are roughly seven stages to web crawling:

1. URL Discovery

When you publish your page (e.g. to your sitemap), the web crawler discovers it and uses it as a ‘seed’ URL. Just like seeds in the cycle of germination, these starter URLs allow the crawl and subsequent crawling loops to begin.

2. Crawling

After URL discovery, your page is scheduled and then crawled. Content like meta tags, images, links, and structured data are downloaded to the search engine’s servers, where they await parsing and indexing.

3. Parsing

Parsing essentially means analysis. The crawler bot extracts the data it’s just crawled to determine how to index and rank the page.

3a. The URL Discovery Loop

Also during the parsing phase, but worthy of its own subsection, is the URL discovery loop. This is when newly discovered links (including links discovered via redirects) are added to a queue of URLs for the crawler to visit. These are effectively new ‘seed’ URLs, and steps 1–3 get repeated as part of the ‘URL discovery loop’.

4. Indexing

While new URLs are being discovered, the original URL gets indexed. Indexing is when search engines store the data collected from web pages. It enables them to quickly retrieve relevant results for user queries.

5. Ranking

Indexed pages get ranked in search engines based on quality, relevance to search queries, and ability to meet certain other ranking factors. These pages are then served to users when they perform a search.

6. Crawl ends

Eventually the entire crawl (including the URL rediscovery loop) ends based on factors like time allocated, number of pages crawled, depth of links followed etc.

7. Revisiting

Crawlers periodically revisit the page to check for updates, new content, or changes in structure.

As you can probably guess, the number of URLs discovered and crawled in this process grows exponentially in just a few hops.

Search engine web crawlers are autonomous, meaning you can’t trigger them to crawl or switch them on/off at will.

You can, however, notify crawlers of site updates via:

XML sitemaps

An XML sitemap is a file that lists all the important pages on your website to help search engines accurately discover and index your content.

Google’s URL inspection tool

You can ask Google to consider recrawling your site content via its URL inspection tool in Google Search Console. You may get a message in GSC if Google knows about your URL but hasn’t yet crawled or indexed it. If so, find out how to fix “Discovered — currently not indexed”.

IndexNow

Instead of waiting for bots to re-crawl and index your content, you can use IndexNow to automatically ping search engines like Bing, Yandex, Naver, Seznam.cz, and Yep, whenever you:

• Add new pages
• Update existing content
• Remove outdated pages
• Implement redirects

You can set up automatic IndexNow submissions via Ahrefs Site Audit.

Search engine crawling decisions are dynamic and a little obscure.

Although we don’t know the definitive criteria Google uses to determine when or how often to crawl content, we’ve deduced three of the most important areas.

This is based on breadcrumbs dropped by Google, both in support documentation and during rep interviews.

1. Prioritize quality

Google PageRank evaluates the number and quality of links to a page, considering them as “votes” of importance.

Pages earning quality links are deemed more important and are ranked higher in search results.

PageRank is a foundational part of Google’s algorithm. It makes sense then that the quality of your links and content plays a big part in how your site is crawled and indexed.

To judge your site’s quality, Google looks at factors such as:

To assess the pages on your site with the most links, check out the Best by Links report.

Pay attention to the “First seen”, “Last check” column, which reveals which pages have been crawled most often, and when.

2. Keep things fresh

According to Google’s Senior Search Analyst, John Mueller…

Search engines recrawl URLs at different rates, sometimes it’s multiple times a day, sometimes it’s once every few months.

But if you regularly update your content, you’ll see crawlers dropping by more often.

Search engines like Google want to deliver accurate and up-to-date information to remain competitive and relevant, so updating your content is like dangling a carrot on a stick.

You can examine just how quickly Google processes your updates by checking your crawl stats in Google Search Console.

While you’re there, look at the breakdown of crawling “By purpose” (i.e. percent split of pages refreshed vs pages newly discovered). This will also help you work out just how often you’re encouraging web crawlers to revisit your site.

To find specific pages that need updating on your site, head to the Top Pages report in Ahrefs Site Explorer, then:

1. Set the traffic filter to “Declined”
2. Set the comparison date to the last year or two
3. Look at Content Changes status and update pages with only minor changes

Top Pages shows you the content on your site driving the most organic traffic. Pushing updates to these pages will encourage crawlers to visit your best content more often, and (hopefully) boost any declining traffic.

3. Refine your site structure

Offering a clear site structure via a logical sitemap, and backing that up with relevant internal links will help crawlers:

• Better navigate your site
• Understand its hierarchy
• Index and rank your most valuable content

Combined, these factors will also please users, since they support easy navigation, reduced bounce rates, and increased engagement.

Below are some more elements that can potentially influence how your site gets discovered and prioritized in crawling:

Web crawlers like Google crawl the entire internet, and you can’t control which sites they visit, or how often.

But you can use website crawlers, which are like your own private bots.

Ask them to crawl your website to find and fix important SEO problems, or study your competitors’ site, turning their biggest weaknesses into your opportunities.

Site crawlers essentially simulate search performance. They help you understand how a search engine’s web crawlers might interpret your pages, based on their:

• Structure
• Content
• Meta data
• Page load speed
• Errors
• Etc

Example: Ahrefs Site Audit

The Ahrefs Site Audit crawler powers the tools: RankTracker, Projects, and Ahrefs’ main website crawling tool: Site Audit.

Site Audit helps SEOs to:

• Analyze 170+ technical SEO issues
• Conduct on-demand crawls, with live site performance data
• Assess up to 170k URLs a minute
• Troubleshoot, maintain, and improve their visibility in search engines

From URL discovery to revisiting, website crawlers operate very similarly to web crawlers – only instead of indexing and ranking your page in the SERPs, they store and analyze it in their own database.

You can crawl your site either locally or remotely. Desktop crawlers like ScreamingFrog let you download and customize your site crawl, while cloud-based tools like Ahrefs Site Audit perform the crawl without using your computer’s resources – helping you work collaboratively on fixes and site optimization.

If you want to scan entire websites in real time to detect technical SEO problems, configure a crawl in Site Audit.

It will give you visual data breakdowns, site health scores, and detailed fix recommendations to help you understand how a search engine interprets your site.

1. Set up your crawl

Navigate to the Site Audit tab and choose an existing project, or set one up.

A project is any domain, subdomain, or URL you want to track over time.

Once you’ve configured your crawl settings – including your crawl schedule and URL sources – you can start your audit and you’ll be notified as soon as it’s complete.

Here are some things you can do right away.

2. Diagnose top errors

The Top Issues overview in Site Audit shows you your most pressing errors, warnings, and notices, based on the number of URLs affected.

Working through these as part of your SEO roadmap will help you:

1. Spot errors (red icons) impacting crawling – e.g.

• HTTP status code/client errors
• Broken links
• Canonical issues

2. Optimize your content and rankings based on warnings (yellow) – e.g.

• Missing alt text
• Links to redirects
• Overly long meta descriptions

3. Maintain steady visibility with notices (blue icon) – e.g.

• Organic traffic drops
• Multiple H1s
• Indexable pages not in sitemap

Filter issues

You can also prioritize fixes using filters.

Say you have thousands of pages with missing meta descriptions. Make the task more manageable and impactful by targeting high traffic pages first.

1. Head to the Page Explorer report in Site Audit
2. Select the advanced filter dropdown
3. Set an internal pages filter
4. Select an ‘And’ operator
5. Select ‘Meta description’ and ‘Not exists’
6. Select ‘Organic traffic > 100’

Crawl the most important parts of your site

Segment and zero-in on the most important pages on your site (e.g. subfolders or subdomains) using Site Audit’s 200+ filters – whether that’s your blog, ecommerce store, or even pages that earn over a certain traffic threshold.

3. Expedite fixes

If you don’t have coding experience, then the prospect of crawling your site and implementing fixes can be intimidating.

If you do have dev support, issues are easier to remedy, but then it becomes a matter of bargaining for another person’s time.

We’ve got a new feature on the way to help you solve for these kinds of headaches.

Coming soon, Patches are fixes you can make autonomously in Site Audit.

Title changes, missing meta descriptions, site-wide broken links – when you face these kinds of errors you can hit “Patch it” to publish a fix directly to your website, without having to pester a dev.

And if you’re unsure of anything, you can roll-back your patches at any point.

4. Spot optimization opportunities

Auditing your site with a website crawler is as much about spotting opportunities as it is about fixing bugs.

Improve internal linking

The Internal Link Opportunities report in Site Audit shows you relevant internal linking suggestions, by taking the top 10 keywords (by traffic) for each crawled page, then looking for mentions of them on your other crawled pages.

‘Source’ pages are the ones you should link from, and ‘Target’ pages are the ones you should link to.

The more high quality connections you make between your content, the easier it will be for Googlebot to crawl your site.

Final thoughts

Understanding website crawling is more than just an SEO hack – it’s foundational knowledge that directly impacts your traffic and ROI.

Knowing how crawlers work means knowing how search engines “see” your site, and that’s half the battle when it comes to ranking.