Connect with us

SEO

10 Steps To Boost Your Site’s Crawlability And Indexability

Published

on

7 Steps To Boost Your Site’s Crawlability And Indexability

Keywords and content may be the twin pillars upon which most search engine optimization strategies are built, but they’re far from the only ones that matter.

Less commonly discussed but equally important – not just to users but to search bots – is your website’s discoverability.

There are roughly 50 billion webpages on 1.93 billion websites on the internet. This is far too many for any human team to explore, so these bots, also called spiders, perform a significant role.

These bots determine each page’s content by following links from website to website and page to page. This information is compiled into a vast database, or index, of URLs, which are then put through the search engine’s algorithm for ranking.

This two-step process of navigating and understanding your site is called crawling and indexing.

As an SEO professional, you’ve undoubtedly heard these terms before, but let’s define them just for clarity’s sake:

  • Crawlability refers to how well these search engine bots can scan and index your webpages.
  • Indexability measures the search engine’s ability to analyze your webpages and add them to its index.

As you can probably imagine, these are both essential parts of SEO.

If your site suffers from poor crawlability, for example, many broken links and dead ends, search engine crawlers won’t be able to access all your content, which will exclude it from the index.

Indexability, on the other hand, is vital because pages that are not indexed will not appear in search results. How can Google rank a page it hasn’t included in its database?

The crawling and indexing process is a bit more complicated than we’ve discussed here, but that’s the basic overview.

If you’re looking for a more in-depth discussion of how they work, Dave Davies has an excellent piece on crawling and indexing.

How To Improve Crawling And Indexing

Now that we’ve covered just how important these two processes are let’s look at some elements of your website that affect crawling and indexing – and discuss ways to optimize your site for them.

1. Improve Page Loading Speed

With billions of webpages to catalog, web spiders don’t have all day to wait for your links to load. This is sometimes referred to as a crawl budget.

If your site doesn’t load within the specified time frame, they’ll leave your site, which means you’ll remain uncrawled and unindexed. And as you can imagine, this is not good for SEO purposes.

Thus, it’s a good idea to regularly evaluate your page speed and improve it wherever you can.

You can use Google Search Console or tools like Screaming Frog to check your website’s speed.

If your site is running slow, take steps to alleviate the problem. This could include upgrading your server or hosting platform, enabling compression, minifying CSS, JavaScript, and HTML, and eliminating or reducing redirects.

Figure out what’s slowing down your load time by checking your Core Web Vitals report. If you want more refined information about your goals, particularly from a user-centric view, Google Lighthouse is an open-source tool you may find very useful.

2. Strengthen Internal Link Structure

A good site structure and internal linking are foundational elements of a successful SEO strategy. A disorganized website is difficult for search engines to crawl, which makes internal linking one of the most important things a website can do.

But don’t just take our word for it. Here’s what Google’s search advocate John Mueller had to say about it:

“Internal linking is super critical for SEO. I think it’s one of the biggest things that you can do on a website to kind of guide Google and guide visitors to the pages that you think are important.”

If your internal linking is poor, you also risk orphaned pages or those pages that don’t link to any other part of your website. Because nothing is directed to these pages, the only way for search engines to find them is from your sitemap.

To eliminate this problem and others caused by poor structure, create a logical internal structure for your site.

Your homepage should link to subpages supported by pages further down the pyramid. These subpages should then have contextual links where it feels natural.

Another thing to keep an eye on is broken links, including those with typos in the URL. This, of course, leads to a broken link, which will lead to the dreaded 404 error. In other words, page not found.

The problem with this is that broken links are not helping and are harming your crawlability.

Double-check your URLs, particularly if you’ve recently undergone a site migration, bulk delete, or structure change. And make sure you’re not linking to old or deleted URLs.

Other best practices for internal linking include having a good amount of linkable content (content is always king), using anchor text instead of linked images, and using a “reasonable number” of links on a page (whatever that means).

Oh yeah, and ensure you’re using follow links for internal links.

3. Submit Your Sitemap To Google

Given enough time, and assuming you haven’t told it not to, Google will crawl your site. And that’s great, but it’s not helping your search ranking while you’re waiting.

If you’ve recently made changes to your content and want Google to know about it immediately, it’s a good idea to submit a sitemap to Google Search Console.

A sitemap is another file that lives in your root directory. It serves as a roadmap for search engines with direct links to every page on your site.

This is beneficial for indexability because it allows Google to learn about multiple pages simultaneously. Whereas a crawler may have to follow five internal links to discover a deep page, by submitting an XML sitemap, it can find all of your pages with a single visit to your sitemap file.

Submitting your sitemap to Google is particularly useful if you have a deep website, frequently add new pages or content, or your site does not have good internal linking.

4. Update Robots.txt Files

You probably want to have a robots.txt file for your website. While it’s not required, 99% of websites use it as a rule of thumb. If you’re unfamiliar with this is, it’s a plain text file in your website’s root directory.

It tells search engine crawlers how you would like them to crawl your site. Its primary use is to manage bot traffic and keep your site from being overloaded with requests.

Where this comes in handy in terms of crawlability is limiting which pages Google crawls and indexes. For example, you probably don’t want pages like directories, shopping carts, and tags in Google’s directory.

Of course, this helpful text file can also negatively impact your crawlability. It’s well worth looking at your robots.txt file (or having an expert do it if you’re not confident in your abilities) to see if you’re inadvertently blocking crawler access to your pages.

Some common mistakes in robots.text files include:

  • Robots.txt is not in the root directory.
  • Poor use of wildcards.
  • Noindex in robots.txt.
  • Blocked scripts, stylesheets and images.
  • No sitemap URL.

For an in-depth examination of each of these issues – and tips for resolving them, read this article.

5. Check Your Canonicalization

Canonical tags consolidate signals from multiple URLs into a single canonical URL. This can be a helpful way to tell Google to index the pages you want while skipping duplicates and outdated versions.

But this opens the door for rogue canonical tags. These refer to older versions of a page that no longer exists, leading to search engines indexing the wrong pages and leaving your preferred pages invisible.

To eliminate this problem, use a URL inspection tool to scan for rogue tags and remove them.

If your website is geared towards international traffic, i.e., if you direct users in different countries to different canonical pages, you need to have canonical tags for each language. This ensures your pages are being indexed in each language your site is using.

6. Perform A Site Audit

Now that you’ve performed all these other steps, there’s still one final thing you need to do to ensure your site is optimized for crawling and indexing: a site audit. And that starts with checking the percentage of pages Google has indexed for your site.

Check Your Indexability Rate

Your indexability rate is the number of pages in Google’s index divided by the number of pages on our website.

You can find out how many pages are in the google index from Google Search Console Index  by going to the “Pages” tab and checking the number of pages on the website from the CMS admin panel.

There’s a good chance your site will have some pages you don’t want indexed, so this number likely won’t be 100%. But if the indexability rate is below 90%, then you have issues that need to be investigated.

You can get your no-indexed URLs from Search Console and run an audit for them. This could help you understand what is causing the issue.

Another useful site auditing tool included in Google Search Console is the URL Inspection Tool. This allows you to see what Google spiders see, which you can then compare to real webpages to understand what Google is unable to render.

Audit Newly Published Pages

Any time you publish new pages to your website or update your most important pages, you should make sure they’re being indexed. Go into Google Search Console and make sure they’re all showing up.

If you’re still having issues, an audit can also give you insight into which other parts of your SEO strategy are falling short, so it’s a double win. Scale your audit process with tools like:

  1. Screaming Frog
  2. Semrush
  3. Ziptie
  4. Oncrawl
  5. Lumar

7. Check For Low-Quality Or Duplicate Content

If Google doesn’t view your content as valuable to searchers, it may decide it’s not worthy to index. This thin content, as it’s known could be poorly written content (e.g., filled with grammar mistakes and spelling errors), boilerplate content that’s not unique to your site, or content with no external signals about its value and authority.

To find this, determine which pages on your site are not being indexed, and then review the target queries for them. Are they providing high-quality answers to the questions of searchers? If not, replace or refresh them.

Duplicate content is another reason bots can get hung up while crawling your site. Basically, what happens is that your coding structure has confused it and it doesn’t know which version to index. This could be caused by things like session IDs, redundant content elements and pagination issues.

Sometimes, this will trigger an alert in Google Search Console, telling you Google is encountering more URLs than it thinks it should. If you haven’t received one, check your crawl results for things like duplicate or missing tags, or URLs with extra characters that could be creating extra work for bots.

Correct these issues by fixing tags, removing pages or adjusting Google’s access.

8. Eliminate Redirect Chains And Internal Redirects

As websites evolve, redirects are a natural byproduct, directing visitors from one page to a newer or more relevant one. But while they’re common on most sites, if you’re mishandling them, you could be inadvertently sabotaging your own indexing.

There are several mistakes you can make when creating redirects, but one of the most common is redirect chains. These occur when there’s more than one redirect between the link clicked on and the destination. Google doesn’t look on this as a positive signal.

In more extreme cases, you may initiate a redirect loop, in which a page redirects to another page, which directs to another page, and so on, until it eventually links back to the very first page. In other words, you’ve created a never-ending loop that goes nowhere.

Check your site’s redirects using Screaming Frog, Redirect-Checker.org or a similar tool.

9. Fix Broken Links

In a similar vein, broken links can wreak havoc on your site’s crawlability. You should regularly be checking your site to ensure you don’t have broken links, as this will not only hurt your SEO results, but will frustrate human users.

There are a number of ways you can find broken links on your site, including manually evaluating each and every link on your site (header, footer, navigation, in-text, etc.), or you can use Google Search Console, Analytics or Screaming Frog to find 404 errors.

Once you’ve found broken links, you have three options for fixing them: redirecting them (see the section above for caveats), updating them or removing them.

10. IndexNow

IndexNow is a relatively new protocol that allows URLs to be submitted simultaneously between search engines via an API. It works like a super-charged version of submitting an XML sitemap by alerting search engines about new URLs and changes to your website.

Basically, what it does is provides crawlers with a roadmap to your site upfront. They enter your site with information they need, so there’s no need to constantly recheck the sitemap. And unlike XML sitemaps, it allows you to inform search engines about non-200 status code pages.

Implementing it is easy, and only requires you to generate an API key, host it in your directory or another location, and submit your URLs in the recommended format.

Wrapping Up

By now, you should have a good understanding of your website’s indexability and crawlability. You should also understand just how important these two factors are to your search rankings.

If Google’s spiders can crawl and index your site, it doesn’t matter how many keywords, backlinks, and tags you use – you won’t appear in search results.

And that’s why it’s essential to regularly check your site for anything that could be waylaying, misleading, or misdirecting bots.

So, get yourself a good set of tools and get started. Be diligent and mindful of the details, and you’ll soon have Google spiders swarming your site like spiders.

More Resources:


Featured Image: Roman Samborskyi/Shutterstock



Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address

SEO

Google Updating Cryptocurrency Advertising Policy For 2024

Published

on

By

Google Updating Cryptocurrency Advertising Policy For 2024

Google published an announcement of upcoming changes to their cryptocurrency advertising policies and advises advertisers to make themselves aware of the changes and prepare to be in compliance with the new requirements.

The upcoming updates are to Google’s Cryptocurrencies and related products policy for the advertisement of Cryptocurrency Coin Trusts. The changes are set to take effect on January 29th, 2024.

Cryptocurrency Coin Trusts are financial products that enable investors to trade shares in trusts holding substantial amounts of digital currency. These trusts provide investors with equity in cryptocurrencies without having direct ownership. They are also an option for creating a more diversified portfolio.

The policy updates by Google that are coming in 2024 aim to describe the scope and requirements for the advertisement of Cryptocurrency Coin Trusts. Advertisers targeting the United States will be able to promote these products and services as long as they abide by specific policies outlined in the updated requirements and that they also obtain certification from Google.

The updated policy changes are not limited to the United States. They will apply globally to all accounts advertising Cryptocurrency Coin Trusts.

Google’s announcement also reminded advertisers of their obligation for compliance to local laws in the areas where the ads are targeted.

Google’s approach for violations of the new policy will be to first give a warning before imposing an account suspension.

Advertisers that fail to comply with the updated policy will receive a warning at least seven days before a potential account suspension. This time period provides advertisers with an opportunity to fix non-compliance issues and to get back into compliance with the revised guidelines.

Advertisers are encouraged to refer to Google’s documentation on “About restricted financial products certification.”

The deadline for the change in policy is January 29th, 2024. Cryptocurrency Coin Trusts advertisers will need to pay close attention to the updated policies in order to ensure compliance.

Read Google’s announcement:

Updates to Cryptocurrencies and related products policy (December 2023)

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

SEO

SEO Trends You Can’t Ignore In 2024

Published

on

SEO Trends You Can’t Ignore In 2024

Most SEO trends fade quickly. But some of them stick and deserve your attention.

Let’s explore what those are and how to take advantage of them.

If you give ChatGPT a title and ask it to write a blog post, it will—in seconds.

This is super impressive, but there are a couple of issues:

  • Everyone else using ChatGPT is creating the same content. It’s the same for users of other GPT-powered AI writing tools, too—which is basically all of them.
  • The content is extremely dull. Sure, you can ask ChatGPT to “make it more entertaining,” but it usually overcompensates and hands back a cringe version of the same boring content.

In the words of Gael Breton:

How to take advantage of this SEO trend

Don’t use AI to write entire articles. They’ll be boring as heck. Instead, use it as a creative sparring partner to help you write better content and automate monotonous tasks.

For example, you can ask ChatGPT To write an outline from a working title and a list of keywords (which you can pull from Ahrefs)—and it does a pretty decent job.

Prompt:

Create an outline for a post entitled “[working title]” based on these keywords: [list]

Result:

ChatGPT's outline for a blog post. Pretty good!ChatGPT's outline for a blog post. Pretty good!

When you’ve written your draft, you can ask to polish it in seconds by asking ChatGPT to proofread it.

ChatGPT proofreading my content and making it betterChatGPT proofreading my content and making it better

Then you can automate the boring stuff, like creating more enticing title tags…

ChatGPT writing enticing title tagsChatGPT writing enticing title tags

… and writing a meta description:

ChatGPT writing a meta descriptionChatGPT writing a meta description

If you notice a few months down the line that your content ranks well but hasn’t won the featured snippet, ChatGPT can help with that, too.

For example, Ahrefs tells us we rank in position 3 for “affiliate marketing” but don’t own the snippet.

Ahrefs showing featured snippets that we don't own, despite ranking in the top 3Ahrefs showing featured snippets that we don't own, despite ranking in the top 3

If we check Google, the snippet is a definition. Asking ChatGPT to simplify our definition may solve this problem.

ChatGPT rewriting a definition and making it betterChatGPT rewriting a definition and making it better

In short, there are a near-infinite number of ways to use ChatGPT (and other AI writing tools) to create better content. And all of them buck the trend of asking it to write boring, boilerplate articles from scratch.

Programmatic SEO refers to the creation of keyword-targeted pages in an automatic (or near automatic) way.

Nomadlist’s location pages are a perfect example:

Example of a page from NomadListExample of a page from NomadList

Each page focuses on a specific city and shares the same core information—internet speeds, cost, temperature, etc. All of this information is pulled programmatically from a database and the site gets an estimated 46k monthly search visits in total.

Estimated monthly search traffic to NomadListEstimated monthly search traffic to NomadList

Programmatic SEO is nothing new. It’s been around forever. It’s just the hot thing right now because AI tools like ChatGPT make it easier and more accessible than ever before.

The problem? As John Mueller pointed out on Twitter X, much of it is spam:

How to take advantage of this SEO trend

Don’t use programmatic SEO to publish insane amounts of spam that’ll probably get hit in the next Google update. Use it to scale valuable content that will stand the test of time.

For example, Wise’s currency conversion pages currently get an estimated 31.7M monthly search visits:

Estimated monthly search traffic to Wise's currently conversion pages (insane!)Estimated monthly search traffic to Wise's currently conversion pages (insane!)

This is because the content is actually useful. Each page features an interactive tool showing the live exchange rate for any amount…

The interactive currently conversion tool on Wise's pagesThe interactive currently conversion tool on Wise's pages

… the exchange rate over time…

The exchange rate over time graph on Wise's pagesThe exchange rate over time graph on Wise's pages

… a handy email notification option when the exchange rates exceed a certain amount…

The email notification option on Wise's pagesThe email notification option on Wise's pages

… handy conversion charts for popular amounts…

The handy conversion charts on Wise's pagesThe handy conversion charts on Wise's pages

… and a comparison of the cheapest ways to send money abroad in your chosen currency:

The useful comparison table on Wise's pagesThe useful comparison table on Wise's pages

It doesn’t matter that all of these pages use the same template. The data is exactly what you want to see when you search [currency 1] to [currency 2].

That’s probably why Wise ranks in the top 10 for over 66,000 of these keywords:

Wise's keyword rankings for currency conversion pagesWise's keyword rankings for currency conversion pages

Looking to take advantage of programmatic content in 2024 like Wise? Check out the guide below.

People love ChatGPT because it answers questions fast and succinctly, so it’s no surprise that generative AI is already making its way into search.

For example, if you ask Bing for a definition or how to do something basic, AI will generate an answer on the fly right there in the search results.

Bing's search results for "definition of mental health"Bing's search results for "definition of mental health"
Bing's search results for "how to add drop down list in google sheets"Bing's search results for "how to add drop down list in google sheets"

In other words, thanks to AI, users no longer have to click on a search result for answers to simple questions. It’s like featured snippets on steroids.

This might not be a huge deal right now, but when Google’s version of this (Search Generative Experience) comes out of beta, many websites will see clicks fall off a cliff.

How to take advantage of this SEO trend

Don’t invest too much in topics that generative AI can easily answer. You’ll only lose clicks like crazy to AI in the long run. Instead, start prioritizing topics that AI will struggle to answer.

How do you know which topics it will struggle to answer? Try asking ChatGPT. If it gives a good and concise answer, it’s clearly an easy question.

For example, there are hundreds of searches for how to calculate a percentage in Google Sheets every month in the US:

Estimated monthly search volume for "google sheets percentage formula" via Ahrefs' Keywords ExplorerEstimated monthly search volume for "google sheets percentage formula" via Ahrefs' Keywords Explorer

If you ask ChatGPT for the solution, it gives you a perfect answer in about fifty words.

ChatGPT's answer to the Google Sheets percentage calculation formulaChatGPT's answer to the Google Sheets percentage calculation formula

This is the perfect example of a topic where generative AI will remove the need to click on a search result for many.

That’s probably not going to be the case for a topic like this:

Example of a topic that AI shouldn't impact too muchExample of a topic that AI shouldn't impact too much

Sure. Generative AI might be able to tell you how to create a template—but it can’t make one for you. And even if it can in the future, it will never be a personal finance expert with experience. You’ll always have to click on a search result for a template created by that person.

These are the kinds of topics to prioritize in 2024 and beyond.

Sidenote.

None of this means you should stop targeting “simple” topics altogether. You’ll always be able to get some traffic from them. My point is not to be obsessed with ranking for keywords whose days are numbered. Prioritize topics with long-term value instead.

Bonus: 3 SEO trends to ignore in 2024

Not all SEO trends move the needle. Here are just a few of those trends and why you should ignore them.

People are using voice search more than ever

In 2014, Google revealed that 41% of Americans use voice search daily. According to research by UpCity, that number was up to 50% as of 2022. I haven’t seen any data for 2023 yet, but I’d imagine it’s above 50%.

Why you should ignore this SEO trend

75% of voice search results come from a page ranking in the top 3, and 40.7% come from a featured snippet. If you’re already optimizing for those things, there’s not much more you can do.

People are using visual search for shopping more than ever

In 2022, Insider Intelligence reported that 22% of US adults have shopped with visual search (Google Lens, Bing Visual Search, etc.). That number is up from just 15% in 2021.

Why you should ignore this SEO trend

Much like voice search, there’s no real way to optimize for visual search. Sure, it helps to have good quality product images, optimized filenames and alt text, and product schema markup on your pages—but you should be doing this stuff anyway as it’s been a best practice since forever.

People are using Bing more than ever before

Bing’s Yusuf Mehdi announced in March 2023 that the search engine had surpassed 100M daily active users for the first time ever. This came just one month after the launch of AI-powered Bing.

Why you should ignore this SEO trend

Bing might be more popular than ever, but its market share still only stands at around ~3% according to estimates by Statcounter. Google’s market share stands at roughly 92%, so that’s the one you should be optimizing for.

Plus, it’s often the case that if you rank in Google, you also rank in Bing—so it really doesn’t deserve any focus.

Final thoughts

Keeping your finger on the pulse and taking advantage of trends makes sense, but don’t let them distract you from the boring stuff that’s always worked: find what people are searching for > create content about it > build backlinks > repeat.

Got questions? Ping me on Twitter X.



Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

SEO

Mozilla VPN Security Risks Discovered

Published

on

By

Mozilla VPN Security Risks Discovered

Mozilla published the results of a recent third-party security audit of its VPN services as part of it’s commitment to user privacy and security. The survey revealed security issues which were presented to Mozilla to be addressed with fixes to ensure user privacy and security.

Many search marketers use VPNs during the course of their business especially when using a Wi-Fi connection in order to protect sensitive data, so the  trustworthiness of a VNP is essential.

Mozilla VPN

A Virtual Private Network (VPN), is a service that hides (encrypts) a user’s Internet traffic so that no third party (like an ISP) can snoop and see what sites a user is visiting.

VPNs also add a layer of security from malicious activities such as session hijacking which can give an attacker full access to the websites a user is visiting.

There is a high expectation from users that the VPN will protect their privacy when they are browsing on the Internet.

Mozilla thus employs the services of a third party to conduct a security audit to make sure their VPN is thoroughly locked down.

Security Risks Discovered

The audit revealed vulnerabilities of medium or higher severity, ranging from Denial of Service (DoS). risks to keychain access leaks (related to encryption) and the lack of access controls.

Cure53, the third party security firm, discovered and addressed several risks. Among the issues were potential VPN leaks to the vulnerability of a rogue extension that disabled the VPN.

The scope of the audit encompassed the following products:

  • Mozilla VPN Qt6 App for macOS
  • Mozilla VPN Qt6 App for Linux
  • Mozilla VPN Qt6 App for Windows
  • Mozilla VPN Qt6 App for iOS
  • Mozilla VPN Qt6 App for Androi

These are the risks identified by the security audit:

  • FVP-03-003: DoS via serialized intent
  • FVP-03-008: Keychain access level leaks WG private key to iCloud
  • VP-03-010: VPN leak via captive portal detection
  • FVP-03-011: Lack of local TCP server access controls
  • FVP-03-012: Rogue extension can disable VPN using mozillavpnnp (High)

The rogue extension issue was rated as high severity. Each risk was subsequently addressed by Mozilla.

Mozilla presented the results of the security audit as part of their commitment to transparency and to maintain the trust and security of their users. Conducting a third party security audit is a best practice for a VPN provider that helps assure that the VPN is trustworthy and reliable.

Read Mozilla’s announcement:
Mozilla VPN Security Audit 2023

Featured Image by Shutterstock/Meilun

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

Trending