The SEO game has so many moving parts that it often seems like, as soon as we’re done optimizing one part of a website, we have to move back to the part we were just working on.
Once you’re out of the “I’m new here” stage and feel that you have some real SEO experience under your belt, you might start to feel that there are some things you can devote less time to correcting.
I always like to say that a website with indexability issues is a site that’s in its own way; that website is inadvertently telling Google not to rank its pages because they don’t load correctly or they redirect too many times.
If you think you can’t or shouldn’t be devoting time to the decidedly not-so-glamorous task of fixing your site’s indexability, think again.
Indexability problems can cause your rankings to plummet and your site traffic to dry up quickly.
So, your crawl budget has to be top of mind.
In this post, I’ll present you with 11 tips to consider as you go about improving your website’s indexability.
1. Track Crawl Status With Google Search Console
Errors in your crawl status could be indicative of a deeper issue on your site.
Checking your crawl status every 30-60 days is important to identify potential errors that are impacting your site’s overall marketing performance.
It’s literally the first step of SEO; without it, all other efforts are null.
Right there on the sidebar, you’ll be able to check your crawl status under the index tab.
Now, if you want to remove access to a certain webpage, you can tell Search Console directly. This is useful if a page is temporarily redirected or has a 404 error.
A 410 parameter will permanently remove a page from the index, so beware of using the nuclear option.
Common Crawl Errors & Solutions
If your website is unfortunate enough to be experiencing a crawl error, it may require an easy solution or be indicative of a much larger technical problem on your site.
The most common crawl errors I see are:
To diagnose some of these errors, you can leverage the URL Inspection tool to see how Google views your site.
Failure to properly fetch and render a page could be indicative of a deeper DNS error that will need to be resolved by your DNS provider.
Resolving a server error requires diagnosing a specific error. The most common errors include:
- Connection refused.
- Connect failed.
- Connect timeout.
- No response.
Most of the time, a server error is usually temporary, although a persistent problem could require you to contact your hosting provider directly.
Robots.txt errors, on the other hand, could be more problematic for your site. If your robots.txt file is returning a 200 or 404 error, it means search engines are having difficulty retrieving this file.
You could submit a robots.txt sitemap or avoid the protocol altogether, opting to manually noindex pages that could be problematic for your crawl.
Resolving these errors quickly will ensure that all of your target pages are crawled and indexed the next time search engines crawl your site.
2. Create Mobile-Friendly Webpages
With the arrival of the mobile-first index, we must also optimize our pages to display mobile-friendly copies on the mobile index.
The good news is that a desktop copy will still be indexed and displayed under the mobile index if a mobile-friendly copy does not exist. The bad news is that your rankings may suffer as a result.
There are many technical tweaks that can instantly make your website more mobile-friendly including:
- Implementing responsive web design.
- Inserting the viewpoint meta tag in content.
- Minifying on-page resources (CSS and JS).
- Tagging pages with the AMP cache.
- Optimizing and compressing images for faster load times.
- Reducing the size of on-page UI elements.
Be sure to test your website on a mobile platform and run it through Google PageSpeed Insights. Page speed is an important ranking factor and can affect the speed at which search engines can crawl your site.
3. Update Content Regularly
Search engines will crawl your site more regularly if you produce new content on a regular basis.
This is especially useful for publishers who need new stories published and indexed on a regular basis.
Producing content on a regular basis signal to search engines that your site is constantly improving and publishing new content, and therefore needs to be crawled more often to reach its intended audience.
4. Submit A Sitemap To Each Search Engine
One of the best tips for indexation to this day remains to submit a sitemap to Google Search Console and Bing Webmaster Tools.
You can create an XML version using a sitemap generator or manually create one in Google Search Console by tagging the canonical version of each page that contains duplicate content.
5. Optimize Your Interlinking Scheme
Establishing a consistent information architecture is crucial to ensuring that your website is not only properly indexed, but also properly organized.
Creating main service categories where related webpages can sit can further help search engines properly index webpage content under certain categories when the intent may not be clear.
6. Deep Link To Isolated Webpages
If a webpage on your site or a subdomain is created in isolation or an error preventing it from being crawled, you can get it indexed by acquiring a link on an external domain.
This is an especially useful strategy for promoting new pieces of content on your website and getting it indexed quicker.
Beware of syndicating content to accomplish this as search engines may ignore syndicated pages, and it could create duplicate errors if not properly canonicalized.
7. Minify On-Page Resources & Increase Load Times
Forcing search engines to crawl large and unoptimized images will eat up your crawl budget and prevent your site from being indexed as often.
Even certain resources like Flash and CSS can perform poorly over mobile devices and eat up your crawl budget.
In a sense, it’s a lose-lose scenario where page speed and crawl budget are sacrificed for obtrusive on-page elements.
Be sure to optimize your webpage for speed, especially over mobile, by minifying on-page resources, such as CSS. You can also enable caching and compression to help spiders crawl your site faster.
8. Fix Pages With Noindex Tags
Over the course of your website’s development, it may make sense to implement a noindex tag on pages that may be duplicated or only meant for users who take a certain action.
Regardless, you can identify webpages with noindex tags that are preventing them from being crawled by using a free online tool like Screaming Frog.
The Yoast plugin for WordPress allows you to easily switch a page from index to noindex. You could also do this manually in the backend of pages on your site.
9. Set A Custom Crawl Rate
In the old version of Google Search Console, you can actually slow or customize the speed of your crawl rates if Google’s spiders are negatively impacting your site.
This also gives your website time to make necessary changes if it is going through a significant redesign or migration.
10. Eliminate Duplicate Content
Having massive amounts of duplicate content can significantly slow down your crawl rate and eat up your crawl budget.
You can eliminate these problems by either blocking these pages from being indexed or placing a canonical tag on the page you wish to be indexed.
Along the same lines, it pays to optimize the meta tags of each individual page to prevent search engines from mistaking similar pages as duplicate content in their crawl.
11. Block Pages You Don’t Want Spiders To Crawl
There may be instances where you want to prevent search engines from crawling a specific page. You can accomplish this by the following methods:
- Placing a noindex tag.
- Placing the URL in a robots.txt file.
- Deleting the page altogether.
This can also help your crawls run more efficiently, instead of forcing search engines to pour through duplicate content.
The state of your website’s crawlability problems will more or less depend on how much you’ve been staying current with your own SEO.
If you’re tinkering in the back end all the time, you may have identified these issues before they got out of hand and started affecting your rankings.
If you’re not sure, though, run a quick scan in Google Search Console to see how you’re doing.
The results can really be educational!
Featured Image: Ernie Janes/Shutterstock
How We Used a Video Course to Promote Ahrefs (And Got 500K+ Views)
Creating and selling educational courses can be a lucrative business. But if you already have a product to sell, you can actually use courses as a marketing tool.
Back in 2017, about two years after joining Ahrefs, I decided to create a course on content marketing.
I had a very clear understanding of how an educational course would help me promote Ahrefs.
- People like courses – Folks like Brian Dean and Glen Allsopp were selling theirs for $500 to $2,000 a pop (and rather successfully). So a free course of comparable quality was sure to get attention.
- Courses allow for a deeper connection – You would basically be spending a few hours one on one with your students. And if you managed to win their trust, you’d get an opportunity to promote your product to them.
That was my raw thought process going into this venture.
And I absolutely didn’t expect that the lifespan of my course would be as interesting and nuanced as it turned out to be.
The lessons of my course have generated over 500K+ in total views, brought in mid-five-figures in revenue (without even trying), and turned out to be a very helpful resource for our various marketing purposes.
So here goes the story of my “Blogging for Business” course.
1. The creation
I won’t give you any tips on how to create a successful course (well, maybe just one). There are plenty of resources (courses?) on that topic already.
All I want to say is that my own experience was quite grueling.
The 10 lessons of my course span some 40K words. I have never attempted the feat of writing a book, but I imagine creating such a lengthy course is as close as it gets.
I spent a tremendous amount of time polishing each lesson. The course was going to be free, so it was critical that my content was riveting. If not, people would just bounce from it.
Paid courses are quite different in that sense. You pay money to watch them. So even if the content is boring at times, you’ll persevere anyway to ensure a return on your investment.
When I showed the draft version of the course to my friend, Ali Mese, he gave me a simple yet invaluable tip: “Break your lessons into smaller ones. Make each just three to four minutes long.”
How did I not think of this myself?
Short, “snackable” lessons provide a better sense of completion and progress. You’re also more likely to finish a short lesson without getting distracted by something.
I’m pretty sure that it is because of this simple tip that my course landed this Netflix comparison (i.e., best compliment ever):
2. The strategy
With the prices of similar courses ranging from $500 to $2,000, it was really tempting to make some profit with ours.
I think we had around 15,000 paying customers at Ahrefs at that time (and many more on the free plan). So if just 1% of them bought that course for $1K, that would be an easy $150K to pocket. And then we could keep upselling it to our future customers.
Alternatively, we thought about giving access to the course to our paying customers only.
This might have boosted our sales, since the course was a cool addition to the Ahrefs subscription.
And it could also improve user retention. The course was a great training resource for new employees, which our customers would lose access to if they canceled their Ahrefs subscription.
And yet, releasing it for free as a lead acquisition and lead nurturing play seemed to make a lot more sense than the other two options. So we stuck to that.
3. The waitlist
Teasing something to people before you let them get it seems like one of the fundamental rules of marketing.
- Apple announces new products way before they’re available in stores.
- Movie studios publish trailers of upcoming movies months (sometimes years) before they hit the theaters.
- When you have a surprise for your significant other (or your kids), you can’t help but give them some hints before the reveal.
There’s something about “the wait” and the anticipation that we humans just love to experience.
So while I was toiling away and putting lessons of my course together, we launched a landing page to announce it and collect people’s emails.
In case someone hesitated to leave their email, we had two cool bonuses to nudge them:
- Access to the private Slack community
- Free two-week trial of Ahrefs
4. The promotion
I don’t remember our exact promotion strategy. But I’m pretty sure it went something like this:
I also added a little “sharing loop” to the welcome email. I asked people to tell their friends about the course, justifying it with the fact that taking the course with others was more fun than doing it alone.
I have no idea how effective that “growth hack” was, but there was no reason not to encourage sharing.
In total, we managed to get some 16,000 people on our waitlist by the day of the course launch.
5. The launch
On a set date, the following email went out to our waitlist:
Did you notice the “note” saying that the videos were only available for free for 30 days? We did that to nudge people to watch them as soon as possible and not save them to the “Watch later” folder.
In retrospect, I wish we had used this angle from the very beginning: “FREE for 30 days. Then $799.”
This would’ve killed two birds with one stone:
- Added an urgency to complete the course as soon as possible
- Made the course more desirable by assigning a specific (and rather high) monetary value to it
(If only we could be as smart about predicting the future as we are about reflecting on the past.)
Once it was live, the course started to promote itself. I was seeing many super flattering tweets:
We then took the most prominent of those tweets and featured them on the course landing page for some social proof. (They’re still there, by the way.)
6. The paywall
Once the 30 days of free access ran out, we added a $799 paywall. And it didn’t take long for the first sale to arrive:
This early luck didn’t push us to focus on selling this course, though. We didn’t invest any effort into promoting it. It was just sitting passively in our Academy with a $799 price tag, and that was it.
And yet, despite the lack of promotion, that course was generating 8-10 sales every month—which were mostly coming from word of mouth.
Thanks to its hefty price, my course soon appeared on some popular websites with pirated courses. And we were actually glad that it did. Because that meant more people would learn about our content and product.
Then some people who were “late to the party” started asking me if I was ever going to reopen the course for free again. This actually seemed like a perfectly reasonable strategy at the time:
7. The giveaways
And whenever we partnered with someone, they were super happy to get a few licenses of the course, which they could give out to their audience.
8. The relaunch
Despite my original plan to update and relaunch this course once a year, I got buried under other work and didn’t manage to find time for it.
And then the pandemic hit.
That’s when we noticed a cool trend. Many companies were providing free access to their premium educational materials. This was done to support the “stay at home” narrative and help people learn new skills.
I think it was SQ who suggested that we should jump on that train with my “Blogging for Business” course. And so we did:
We couldn’t have hoped for a better timing for that relaunch. The buzz was absolutely insane. The announcement tweet alone has generated a staggering 278K+ impressions (not without some paid boosts, of course).
We also went ahead and reposted that course on ProductHunt once again (because why not?).
All in all, that relaunch turned out to be even more successful than the original launch itself.
In the course of their lifespan on Wistia, the 40 video lessons of my course generated a total of 372K plays.
And this isn’t even the end of it.
9. The launch on YouTube
Because the course was now free, it no longer made sense to host it at Wistia. So we uploaded all lessons to YouTube and made them public.
To date, the 41 videos of my course have generated about 187K views on YouTube.
It’s fair to mention that we had around 200,000 subscribers on our channel at the time of publishing my course there. A brand-new channel with no existing subscribers will likely generate fewer views.
10. The relaunch on YouTube [coming soon]
Here’s an interesting observation that both Sam and I made at around the same time.
Many people were publishing their courses on YouTube as a single video spanning a few hours rather than cutting them into individual lessons like we did. And those long videos were generating millions of views!
Like these two, ranking at the top for “learn Python course,” which have 33M and 27M views, respectively:
Well, the “single video” version of that same course has blown it out of the water with over 1M views as of today.
I’m sure you can already tell where I’m going with this.
We’re soon going to republish my “Blogging for Business” course on YouTube as a single video. And hopefully, it will perform just as well.
So that’s the story of my “Blogging for Business” course. From the very beginning, it was planned as a promotional tool for Ahrefs. And judging by its performance, I guess it fulfilled its purpose rather successfully.
Don’t get me wrong, though.
The fact that my course was conceived as a promotional tool doesn’t mean that I didn’t pour my heart and soul into it. It was a perfectly genuine and honest attempt to create a super useful educational resource for content marketing newbies.
And I’m still hoping to work on the 2.0 version of it someday. In the past four years, I have accrued quite a bit more content marketing knowledge that I’m keen to share with everyone. So follow me on Twitter, and stay tuned.
Follow This Purpose-Driven Path to Greater SEO Success
Instagram Expands Access to Reels Templates, Adds New Music Recommendations for Reels Clips
Twitter Resumes Nofollow Attributes To Links
How We Used a Video Course to Promote Ahrefs (And Got 500K+ Views)
Twitter challenges Indian orders to block content: reports
Coming Soon to Xbox Game Pass: PowerWash Simulator, Last Call BBS, Escape Academy, and More
To Be Eligible For Google Course Rich Results, You Need At Least Three Courses
Twitter Launches New ‘Brand Connect Playbook’ to Provide Guidance for Marketers
PowerWash Simulator is Ready to Clean Up with Game Pass
TLD Experimentation & Authority Building with Publishing and SEO [Podcast]
Why Google Doesn’t Like Some SEO Metrics
How Software Systems Enhance the Performance of Gym Business?
9 Creative Company Profile Examples to Inspire You [Templates]
Strategizing Your Instagram Marketing – DigitalMarketer
How to Calculate Your Lead Generation Goals [Free Calculator]
Google Single URL Inspection Tool Dog
Google Bar & Pool Table Room
24 questions to ask identity resolution vendors during a demo
Good Web Sites Are Good For SEO, Says Google
Alcides Aguasvivas On Proper Infrastructure For Sites To Perform Well In Search
SEARCHENGINES5 days ago
MARKETING7 days ago
Using Google Analytics 4 integrations for insights and media activations
TECHNOLOGY19 hours ago
How Businesses Can Automate Root Cause Analysis (RCA) With Machine Learning
SEO7 days ago
Top 10 Content Management Systems In 2022