Connect with us

SEO

500 Response on Robots.txt Fetch Can Impact Rich Results

Published

on

500 Response on Robots.txt Fetch Can Impact Rich Results


Google’s John Mueller received feedback about a bug in how Search Console validates rich results. Google will drop images from rich results because of an error in how a CDN that hosts the images handles a request for a non-existent robots.txt. The bug that was discovered was in how search console and Google’s rich results test will fail to alert the publisher of the error and subsequently give the structured data a successful validation.

A bug in the context of programming is when a software program behaves in an unexpected manner. A bug isn’t always a problem in the coding but as in this case, it could be a failure to anticipate an issue which in turn leads to unintended results, like this one.

The publisher asking the question tried to use Google’s tools to diagnose the reason why their rich results were disappearing and was surprised to find that they were of no use for this particular error.

While this issue was affecting the recipe rich results image preview in Google’s recipe rich results, this problem could also be an issue for other situations as well.

So it’s good to be aware of this problem as it might surface in other ways.

Recipe Rich Results Image Previews Disappeared

The person asking the question provided a background of what happened.

He related what happened:

“We ran into a bit of a tiger trap, I would say, in terms of rich recipe results.

We have hundreds of thousands of recipes which are indexed and there’s lots of traffic coming through from the recipe gallery.

And then… over a period of time it stopped.

And all of the meta data checked out and Google search Console was saying …this is all rich recipe content, it’s all good, it can be shown.

We finally noticed that in the preview, when you preview the result, the image was missing.

And it seems that there was a change at Google and that if a robots.txt was required in order for images to be retrieved, then nothing we could see in the tools was actually saying anything was invalid.

And so it’s a bit awkward right, when you check something to say “is this a valid rich recipe result?” and it says yea, it’s great, it’s absolutely great, we’ve got all the metadata.

And you check all the URLs and all the images are right, but it turns out behind the scenes, there was a new requirement that you have a robots.txt.”

John Mueller asked:

“How do you mean that you had to have a robots.txt?”

The person asking the question responded:

“What we found is, if you requested the robots.txt from our CDN, it gave you like a 500.

When we put a robots.txt there, immediately the previews started appearing correctly.

And that involves crawling and putting it onto a static site, I think.

So we operationally, we found adding that robots.txt did the job.”

John Mueller nodded his head and said:

“Yeah, okay.

So from our point of view, it’s not that a robots.txt file is required. But it has to have a proper result code.

So if you don’t have on, it should return 404.

If you do have one, then we can obviously read that.

But if you return a server error for the robots.txt file, then our systems will assume that maybe there is an issue with the server and we won’t crawl.

And that’s kind of something that’s been like that since the beginning.

But these kinds of issues where especially when you are on a CDN and it’s on a separate hostname, sometimes that’s really hard to spot.

And I imaging the rich results test, at least as far as I know, it focuses on the content that is on the HTML page.

So the JSON-LD markup that you have there, it probably doesn’t check to see if the images are actually fetchable.

And then if they can’t be fetched then, of course, we can’t use them in the carousel, too.

So that might be something that we need to figure out how to highlight better.”

500 Error Response for CDN Robots.txt Can Cause Issues

This is one of those show stopping SEO problems that are hard to diagnose but can cause a lot of negative issues as the person asking the question noted.

Normally a crawl for a robots.txt that is non-existent should result in a server response code of 404, which means that the robots.txt does not exist.

So if the request for a robots.txt file is generating a 500 response code then that’s an indication that something on the server or the CMS is misconfigured.

The short term solution is to upload a robots.txt file.

But it might be a good idea to dive into the CMS or server to check what the underlying issue is.

500 Response Code for a Robots.txt Fetch

The negative consequences for the recipes rich results preview because of a CDN that returns a 500 error response might be a rare issue.

A 500 server error response code sometimes happens when there is something unexpected or missing in the code and the server responds by ending the code processing and throwing the 500 response code.

For example, if you edit a PHP file and forget to indicate the end of a section of  code then that might cause the server to give up processing the code and throw a 500 response.

Whatever the reason for the error response when Google tried to fetch the robots.txt, this is a good issue to keep in mind for that rare situation when it happens to you.

Citation

CDN for Images and Recipe Rich Results Bug

Watch at the 51:45 Minute Mark





Source link

SEO

A Simple (But Complete) SEO Tutorial for Beginners in 7 Steps

Published

on

A Simple (But Complete) SEO Tutorial for Beginners in 7 Steps

Starting with SEO can be overwhelming—many technical terms, checklists of tens of tasks to do, learning resources contradicting each other, and the list goes on. I remember when I got into the game.

This SEO tutorial should get you on the right track. We’ll go through seven essential steps to help you increase organic traffic and ensure you have the right foundation to advance your SEO skills further.

Let’s dive into it.

1. Understand what keywords people are searching for

People look up information around your brand, product, or service in tons of different ways.

Our job is to find these keywords and choose the best ones to target with relevant content. This process is known as keyword research.

You’ll need a keyword research tool to do that. There are a bunch of free ones, such as our free keyword generator, so you can get started right away. Simply plug in keywords that describe your business, products, or problems your audience might need to solve.

Let’s say we have a coffee equipment shop, so discovering what people look up regarding espresso machines and how frequently is a good starting point:

Keyword ideas from Ahrefs' free keyword generator tool

Are you used to saying “espresso maker” instead? Well, this is exactly why we do keyword research because “espresso machine” has 15 times the monthly search volume:

Monthly search volumes of "espresso machine" and "espresso maker," respectively
Screenshot from Ahrefs’ Keywords Explorer, which allows you to get data for many keywords at once.

To me, these two keywords are synonyms and can be used interchangeably. But maybe others imagine two different things? Let’s use Google as the best verification method here since its goal is to deliver the most relevant results to any search query.

If there’s a big overlap of the search results, then Google sees these two keywords as synonymous, meaning that most people think that way too.

You can either open up two Google search tabs or use the SERP overview comparison feature in Ahrefs’ Keywords Explorer. As you can see, the overlap is huge and there’s not a single page targeting the keyword “espresso maker”:

SERP overview comparison of two synonymous keywords in Ahrefs' Keywords Explorer

So clearly, “espresso machine” is the right keyword to target here. And we also just touched on the topic of the next step.

Learn more: How to Do Keyword Research for SEO

2. Create content that searchers want to see

Chose a keyword that you want to rank for? Then you need to create a piece of content that aligns with search intent. In other words, figure out what people searching that keyword are looking for and deliver it to them.

We just went through the example of people searching for “coffee makers” actually meaning “coffee machines,” which is more commonly used. In general, we can assume that what already ranks at the top for any given keyword is something most searchers want to see.

Let’s keep up with the coffee equipment theme and target the keyword “burr coffee grinder.” Here’s what Google’s top results in the U.S. look like:

SERP overview for "burr coffee grinder"

We’ve got two types of pages here: product reviews with recommendations (in blue) and e-commerce category pages (in yellow). Since the category pages are in the minority and only the biggest players like Amazon or Target rank with them, I’d go with the listicle kind of post recommending multiple coffee grinders.

If you look closely, some of those review pages don’t even specifically target the “burr” grinder type but focus on the best coffee grinders in general. Those naturally mention a few burr grinders, so they’re relevant too.

Both ways here are fine, although focusing on “burr” specifically will likely result in a higher chance of ranking well for this keyword, especially for websites that aren’t huge authorities in the niche yet.

Last but not least, it should go without saying that searchers also want to see valuable and trustworthy information. A good rule of thumb is not to cover topics that you’re not very familiar with and can’t provide value. There’s too much regurgitated information on the internet already, so don’t just create content for the sake of it.

3. Get more clicks with compelling titles

Want to squeeze the most clicks out of your SERP impressions? You need to catch searchers’ attention.

Now, the truth is if your page’s title is plain bad to begin with, you’ll have a lower chance of ranking well. It’s hardly a surprise that search engines assess title tags for ranking the pages since the title is the most visible SERP component:

Google's SERP for "grinding coffee for moka pot" search query

Which result would you click on here? The second one caught my attention immediately because the title promises I’ll find the answer quickly in some sort of chart. The rest of the titles are good too, but they don’t stand out to me despite higher relevance to my search query.

Yes, you should absolutely use your primary keyword in your title. Also, the example above just proves that there are other variables at play. Here’s a short list of the best practices for writing great titles:

  • Include your primary keyword in a natural way – This means you can transform “grinding coffee for moka pot” into something like “How to Grind Coffee for a Delicious Moka Pot.”
  • Make it descriptive – It should perfectly sum up the core of your content.
  • Try to stand out – Always check pages that already rank for the keyword and think how you can differentiate your own.
  • Avoid overpromises and clickbaits – Try your best to deliver what you promise in the SERP snippet.
  • Fit into Google’s pixel limits – Titles truncate (or even change) when you go over a certain pixel length in your title tag. Use a SERP snippet preview tool like this.

4. Provide a good user experience (UX)

Ever came across a promising search snippet that you clicked through and then a messy website that you didn’t really want to engage with appeared? Yeah, me too.

That’s why Google is looking at multiple UX factors when it comes to ranking its results. Visitor satisfaction isn’t only dependent on the provided information but on the whole experience of your page and website.

Think about these UX factors as a solid foundation for your other SEO efforts, not a silver bullet to skyrocket your rankings:

Mobile-friendly website 

How your website looks and behaves on mobile matters more than its desktop version. Most searches take place on our small devices, and that’s why Google predominantly uses mobile versions of pages in its index.

Yet this factor is easy to overlook since we use our desktop devices to create websites. Here are a few tips to make both human and robot visitors on mobile happy:

  • Use a modern CMS that can easily make both desktop and mobile versions of your pages look good. Most popular CMS choices should be fine here.
  • Keep the important content visible on both mobile and desktop. Small variations are fine.
  • Check and troubleshoot mobile reports in the Experience section of your Google Search Console:
Page Experience report in GSC

Learn more: Mobile-First Indexing: What You Need to Know

Solid website structure and navigation

Website structure is how your site is organized and webpages interlinked. Having a logical site structure helps visitors and search engines easily find and navigate content while improving conversions and supporting your SEO efforts.

We could go really deep into creating an optimal website structure. But listen—don’t go deep here. Instead, choose a flat website structure like this:

Flat vs. deep website structure

The top-level pages should be your most important ones, and they should be linked from your header navigational elements that are accessible at all times. Since we just talked about mobile versions, here’s an example of the popular “hamburger” menu at the top-right corner:

Hamburger navigation element on mobile

Learn more: Website Structure: How to Build Your SEO Foundation

Easy-to-read content

Would you get this far if this article was all just a block of text? Probably not.

Notice how we’re making use of headings and subheadings (from H1 to H3s) to add proper hierarchy to the content. But that will still make for a rather dry-looking page. Here are the three most important tips to make your content easy and enjoyable to read:

  1. Add relevant visual elements to cut through the text.
  2. Use multiple text formatting options (like I just did with this list and bolded words).
  3. Write like your audience talks to avoid trying to sound smart with big words and overcomplicated terminology.

By the way, even your URLs could be considered part of your content. It’s something people see on the SERP, along with the title and description. Making your URLs short and descriptive is the best way to ensure they’re SEO-friendly.

Ads and pop-ups in moderation

The best way to ensure your visitors leave your pages ASAP is to overwhelm them with stuff they didn’t come for. Like here:

Example of a page with too many ads and pop-ups

By all means, monetize your website with ads and grow your email list with pop-ups—but do them in moderation. Potentially higher short-term gains are not worth losing in the long run.

HTTPS

Providing a secure connection between your server and visitors has been a must for many years already.

New websites should come HTTPS-enabled by default with good hosting providers. Just double-check that there’s a “lock” icon next to your domain name in your browser signaling a valid TLS certificate that’s needed to run on this secure protocol:

Secure connection via HTTPS

Internal links are links within the same website. Their main roles in SEO are to provide paths for search bots to crawl websites and pass link equity from linking pages.

Think about the link equity as votes that pages get in the form of links from other pages throughout the whole internet. Since internal links are fully within your control, it’s something we should tackle first.

Simply find suitable internal link opportunities every time you publish a new piece of content. Then add the links.

The easiest way to find these opportunities is to use Google. Surprised? With the help of the site: search operator, Google will show you all your website’s pages that contain a certain keyword (or similar ones). Like this:

Looking for internal link opportunities with Google

I even narrowed down the search to our blog section of the website and used the quotation marks as another search operator to return exact matches on that keyword. Voilá, we get a list of pages that talk about “internal links” to internally link our articles on this topic. OK, we’re getting too meta here.

Replicate the process for each new piece of content and its target keyword, and you’ll get ahead of many competitors.

Learn more: Internal Links for SEO: An Actionable Guide

6. Get high-quality backlinks

Now we get to the part of links you don’t control because they come from external websites: backlinks.

Backlinks are one of the most important ranking factors, meaning that there’s a huge demand for them. Getting worthy backlinks isn’t a piece of cake.

There are tons of ways to build links. The overarching rule of thumb is to provide enough (non-monetary) value to motivate the other side to link to you.

Let me explain the “non-monetary value.”

If you’re trying to get backlinks to subpar content or products, then you’ll most likely have to buy the link or provide something else in exchange for it. Not only would you be unlikely to even get responses to such pitches, but Google could also penalize you for this unnatural link acquisition that violates its webmaster guidelines.

Your biggest asset for building high-quality backlinks is stellar content others find valuable and interesting. Not only can asking for links become quite effective, but you’ll also have a chance to grow your link profile passively without having to write a single email.

Take a look at our five most popular linked-to pages (excluding the homepage):

Best by links report in Ahrefs' Site Explorer
Screenshot taken from the Best by links report in Ahrefs’ Site Explorer.

Two of them are free tools providing valuable data, and one is a data study with important insights for our industry. This type of content is called link bait, and it’s arguably the best way to land high-quality links—as those often come naturally, not from you asking for them.

So how do we qualify backlinks as high-quality?

Generally speaking, the best links you can get are “followed” links placed within the main content that’s published on a highly authoritative website in your niche.

To sum this up: Building links is an important part of SEO, and you’ll do well if people find your pages so valuable they’re naturally inclined to link to it on their own.

Since link building is a whole SEO discipline, I recommend you also check these two guides to learn more about it:

7. Ensure your website has a good technical foundation

Perfecting all the steps above won’t matter if search engines can’t access your content. You need to make sure a search engine can discover, crawl, and store the pages in its index.

Luckily, severe technical errors that could cause your content to not rank at all are rather rare. Unless you click on random things in your CMS, you’re probably fine.

But there are quite a few issues that can hinder your success in search engines.

Well, the good thing is that you don’t need to have any deep technical knowledge to discover and troubleshoot technical SEO issues yourself. All you have to do is to use a tool like Ahrefs Webmaster Tools, set up a regular SEO audit, and keep an eye on the issues it finds.

You could do this for free right now. Ahrefs’ Site Audit tool will crawl your website (like search engines) and check it for more than 130 SEO issues. It will then give you a comprehensive overview showing all the issues with explanations on how to fix them:

Issue example in Ahrefs Webmaster Tools

This will guide you on the spot. If you first want to learn the theory, we have a guide for you too.

Learn more: The Beginner’s Guide to Technical SEO

Final thoughts

There are multiple guides referenced throughout the tutorial, so come back to each step whenever you want to learn more. But don’t fall into the “learning” rabbit hole, as nothing can beat hands-on experience.

Go and apply what you just learned. Then monitor your progress, learn from your mistakes, and double down on what’s working. Once you get stuck, try to perform an SEO audit.

We could go on and on. So here’s my last important piece of advice: be patient. SEO takes time, and there are no guarantees.

Got any questions? Ping me on Twitter.  



Source link

Continue Reading




DON'T MISS ANY IMPORTANT NEWS!
Subscribe To our Newsletter
We promise not to spam you. Unsubscribe at any time.
Invalid email address

Trending

en_USEnglish