Connect with us

SEO

Everything You Need To Know About The X-Robots-Tag HTTP Header

Published

on

Everything You Need To Know About The X-Robots-Tag HTTP Header

Search engine optimization, in its most basic sense, relies upon one thing above all others: Search engine spiders crawling and indexing your site.

But nearly every website is going to have pages that you don’t want to include in this exploration.

For example, do you really want your privacy policy or internal search pages showing up in Google results?

In a best-case scenario, these are doing nothing to drive traffic to your site actively, and in a worst-case, they could be diverting traffic from more important pages.

Luckily, Google allows webmasters to tell search engine bots what pages and content to crawl and what to ignore. There are several ways to do this, the most common being using a robots.txt file or the meta robots tag.

We have an excellent and detailed explanation of the ins and outs of robots.txt, which you should definitely read.

But in high-level terms, it’s a plain text file that lives in your website’s root and follows the Robots Exclusion Protocol (REP).

Robots.txt provides crawlers with instructions about the site as a whole, while meta robots tags include directions for specific pages.

Some meta robots tags you might employ include index, which tells search engines to add the page to their index; noindex, which tells it not to add a page to the index or include it in search results; follow, which instructs a search engine to follow the links on a page; nofollow, which tells it not to follow links, and a whole host of others.

Both robots.txt and meta robots tags are useful tools to keep in your toolbox, but there’s also another way to instruct search engine bots to noindex or nofollow: the X-Robots-Tag.

What Is The X-Robots-Tag?

The X-Robots-Tag is another way for you to control how your webpages are crawled and indexed by spiders. As part of the HTTP header response to a URL, it controls indexing for an entire page, as well as the specific elements on that page.

And whereas using meta robots tags is fairly straightforward, the X-Robots-Tag is a bit more complicated.

But this, of course, raises the question:

When Should You Use The X-Robots-Tag?

According to Google, “Any directive that can be used in a robots meta tag can also be specified as an X-Robots-Tag.”

While you can set robots.txt-related directives in the headers of an HTTP response with both the meta robots tag and X-Robots Tag, there are certain situations where you would want to use the X-Robots-Tag – the two most common being when:

  • You want to control how your non-HTML files are being crawled and indexed.
  • You want to serve directives site-wide instead of on a page level.

For example, if you want to block a specific image or video from being crawled – the HTTP response method makes this easy.

The X-Robots-Tag header is also useful because it allows you to combine multiple tags within an HTTP response or use a comma-separated list of directives to specify directives.

Maybe you don’t want a certain page to be cached and want it to be unavailable after a certain date. You can use a combination of “noarchive” and “unavailable_after” tags to instruct search engine bots to follow these instructions.

Essentially, the power of the X-Robots-Tag is that it is much more flexible than the meta robots tag.

The advantage of using an X-Robots-Tag with HTTP responses is that it allows you to use regular expressions to execute crawl directives on non-HTML, as well as apply parameters on a larger, global level.

To help you understand the difference between these directives, it’s helpful to categorize them by type. That is, are they crawler directives or indexer directives?

Here’s a handy cheat sheet to explain:

Crawler Directives Indexer Directives
Robots.txt – uses the user agent, allow, disallow, and sitemap directives to specify where on-site search engine bots are allowed to crawl and not allowed to crawl. Meta Robots tag – allows you to specify and prevent search engines from showing particular pages on a site in search results.

Nofollow – allows you to specify links that should not pass on authority or PageRank.

X-Robots-tag – allows you to control how specified file types are indexed.

Where Do You Put The X-Robots-Tag?

Let’s say you want to block specific file types. An ideal approach would be to add the X-Robots-Tag to an Apache configuration or a .htaccess file.

The X-Robots-Tag can be added to a site’s HTTP responses in an Apache server configuration via .htaccess file.

Real-World Examples And Uses Of The X-Robots-Tag

So that sounds great in theory, but what does it look like in the real world? Let’s take a look.

Let’s say we wanted search engines not to index .pdf file types. This configuration on Apache servers would look something like the below:

<Files ~ ".pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

In Nginx, it would look like the below:

location ~* .pdf$ {
  add_header X-Robots-Tag "noindex, nofollow";
}

Now, let’s look at a different scenario. Let’s say we want to use the X-Robots-Tag to block image files, such as .jpg, .gif, .png, etc., from being indexed. You could do this with an X-Robots-Tag that would look like the below:

<Files ~ ".(png|jpe?g|gif)$">
Header set X-Robots-Tag "noindex"
</Files>

Please note that understanding how these directives work and the impact they have on one another is crucial.

For example, what happens if both the X-Robots-Tag and a meta robots tag are located when crawler bots discover a URL?

If that URL is blocked from robots.txt, then certain indexing and serving directives cannot be discovered and will not be followed.

If directives are to be followed, then the URLs containing those cannot be disallowed from crawling.

Check For An X-Robots-Tag

There are a few different methods that can be used to check for an X-Robots-Tag on the site.

The easiest way to check is to install a browser extension that will tell you X-Robots-Tag information about the URL.

Screenshot of Robots Exclusion Checker, December 2022Robots Exclusion Checker

Another plugin you can use to determine whether an X-Robots-Tag is being used, for example, is the Web Developer plugin.

By clicking on the plugin in your browser and navigating to “View Response Headers,” you can see the various HTTP headers being used.

web developer plugin

web developer plugin

Another method that can be used for scaling in order to pinpoint issues on websites with a million pages is Screaming Frog.

After running a site through Screaming Frog, you can navigate to the “X-Robots-Tag” column.

This will show you which sections of the site are using the tag, along with which specific directives.

Screaming Frog Report. X-Robot-TagScreenshot of Screaming Frog Report. X-Robot-Tag, December 2022Screaming Frog Report. X-Robot-Tag

Using X-Robots-Tags On Your Site

Understanding and controlling how search engines interact with your website is the cornerstone of search engine optimization. And the X-Robots-Tag is a powerful tool you can use to do just that.

Just be aware: It’s not without its dangers. It is very easy to make a mistake and deindex your entire site.

That said, if you’re reading this piece, you’re probably not an SEO beginner. So long as you use it wisely, take your time and check your work, you’ll find the X-Robots-Tag to be a useful addition to your arsenal.

More Resources:


Featured Image: Song_about_summer/Shutterstock

window.addEventListener( ‘load’, function() {
setTimeout(function(){ striggerEvent( ‘load2’ ); }, 2000);
});

window.addEventListener( ‘load2’, function() {

if( sopp != ‘yes’ && addtl_consent != ‘1~’ && !ss_u ){

!function(f,b,e,v,n,t,s)
{if(f.fbq)return;n=f.fbq=function(){n.callMethod?
n.callMethod.apply(n,arguments):n.queue.push(arguments)};
if(!f._fbq)f._fbq=n;n.push=n;n.loaded=!0;n.version=’2.0′;
n.queue=[];t=b.createElement(e);t.async=!0;
t.src=v;s=b.getElementsByTagName(e)[0];
s.parentNode.insertBefore(t,s)}(window,document,’script’,
‘https://connect.facebook.net/en_US/fbevents.js’);

if( typeof sopp !== “undefined” && sopp === ‘yes’ ){
fbq(‘dataProcessingOptions’, [‘LDU’], 1, 1000);
}else{
fbq(‘dataProcessingOptions’, []);
}

fbq(‘init’, ‘1321385257908563’);

fbq(‘track’, ‘PageView’);

fbq(‘trackSingle’, ‘1321385257908563’, ‘ViewContent’, {
content_name: ‘everything-x-robots-tag’,
content_category: ‘seo technical-seo’
});
}
});

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address

SEO

Why Every Marketer Should Be On Reddit

Published

on

By

Why Every Marketer Should Be On Reddit

In its nearly 20 years of existence, Reddit has built a robust, dynamic, and highly engaged community.

It has cemented itself as not only one of the key influencers for the internet culture we have come to know today, but also as one of the top 10 social media sites in the world, with more than 500,000 monthly visitors across more than 100,000 active communities.

Despite its size and influence, marketers have long avoided, overlooked, and failed to establish a presence on what is arguably one of the most influential social media sites today.

To be fair, Reddit has always been known as one of the hardest communities to have marketing success with, requiring months (if not years) of dedicated commitment to actually being a part of the community and engaging with your audience through meaningful, genuine, and honest interactions.

To many, that is just too much risk and too much work, but let’s be honest – most amazing things take risk and hard work. And for all of you who put in that work to be successful on Reddit, you already see the payoff it brings.

Regardless of the reasons, marketers who continue to overlook Reddit’s potential are missing out on an extremely influential community with ideal targeted demographics, where authenticity and genuine connections lead to valuable information, positive branding, and marketing success.

Not convinced? Let’s take a closer look at Reddit!

Reddit Stats You Cannot Ignore

Reddit users are dedicated to their use of the platform, with a reported 82 million (16.4%) of its 500 million users visiting the site daily, sharing over 1 billion posts, with over 16 billion comments across 100,000+ active communities, making it easily one of the top visited and most active sites in the United States today.

The user base is so loyal that they spend, on average, 20 minutes on the platform each day (and over 45 minutes a day for long-time users of more than 7 years) and, in many cases, are inactive on other social platforms.

This makes Reddit one of the only places to actually get in front of this massive audience, with 32% being inactive on Facebook, 37% on Instagram, 41% on TikTok, and 53% on X.

When it comes to demographics, a 2019 survey concluded that the overall Reddit audience was majority male (59%), ranged in age (36% aged 18-29; 25% aged 30-49), and well educated (42% college degree; 31% some college) making them an ideal audience many businesses looking for success.

They also helped Reddit improve revenues by 21% to $804 million in 2023, with a global addressable market for advertising estimated at $1.4 trillion by 2027.

Thought by many to be mostly a North American audience, Reddit claims more than 50% of its traffic comes from international users, a rather impressive climb from its earlier years.

It is important to remember, though, that Reddit is a platform that allows communities to grow and thrive, so those demographics change from subreddit to subreddit.

If that isn’t enough to grab your attention, surveys have found that 75% of users find Reddit a trustworthy source for making purchasing decisions.

For years now, it has been included heavily and prominently in Google search results, which have become even more visible due to reports following their inclusion in SGE results and with Google’s recent $60 million a year deal with Reddit to have real-time access to Reddit content and to use its content to train Google’s future AI models.

Speaking of Reddit’s visibility in Google’s search results, let’s take a closer look at why that deal is so important, especially to search marketers.

Reddit In Google Search, SGE, And Google’s Reddit Deal

For years, Reddit results have been prominently displayed in Google’s search results – so much so that users have even started adding the term “reddit” to the end of their search queries.

So often, you will see Google suggest search lines with the term “reddit” prefilled at the end.

Screenshot by author from search for [are purple mattresses good], Google, May 2024

Even without selecting the query that includes “reddit,” you will notice Reddit’s prominent placement in two places: discussions and forums, and Reddit sitelinks, which are both just below the ads and the first organic result.

Reddit resultsScreenshot by author from search for [are purple mattresses good], Google, May 2024

This was amplified by Google’s recent Helpful Content Update (HCU), which allowed platforms like Reddit and other forums to show up more frequently in search results,

Reddit’s ability to be helpful in satisfying a user’s search for information is so successful – in combination with reported shortages in new content for AI models from all major platforms – it is not surprising Google struck a deal with Reddit to show its content faster in search results and to use the content in training future AI models.

Since news of the deal between Reddit and Google was reported, there has been a lot of additional focus on Reddit’s visibility in Google’s search results.

Experts throughout the search industry report an increase in not only the total visibility of Reddit going up within search but also the speed at which a Reddit post is indexed and shown to users.

Google responded to concerns about Reddit showing up more in search results, saying that “some of the SEO folks who tend to be vocal on this platform (X) really dislike seeing more forum content in our search results. But actual searchers seem to like it. They proactively seek it out. It makes sense for us to be showing it to keep the search results relevant and satisfying for everyone.”

Google responded to concerns of Reddit showing up more in search resultsScreenshot by author from X (Twitter), May 2024

Although disputed by Google, Roger Montii wrote about one report of Reddit content being indexed within five minutes.

ChatGPT to Surface Reddit Content

Adding to the deal Reddit made with Google, it was recently reported that OpenAI has partnered with Reddit to surface their content in ChatGPT, adding to the clear value that both OpenAI and Google see in Reddit’s content today and in the future.

Of course, this has sparked a lot of interest in companies of all sizes. They want to figure out how their brands can participate and succeed on Reddit in hopes of increasing their visibility in Google’s search results, SGE, and ChatGPT and building their brand’s visibility amongst one of the most influential audiences on the internet today.

**Quick warning: Reddit requires genuine, long-term engagement and a strategy for success. So, I would definitely advise individuals and businesses to avoid trying to game, spam, or blindly jump into marketing on Reddit and take the time to really understand the platform, its audience, and your place of value within its communities.

Let’s look at how some major brands have been successful on Reddit in the past.

noosa Yoghurt

Noosa Yoghurt’s 2021 back-to-school campaign on Reddit was a great example of tapping into niche audiences to create engaging, user-driven content.

To capitalize on Reddit’s community of vocal yogurt enthusiasts, noosa launched a Flavor Poll that invited Redditors to upvote their favorite noosa flavors in the comments. This created a sense of community and interactivity around what could have been a standard poll.

Noosa Yoghurt’s 2021 back-to-school campaign on Reddit.Screenshot by author from Reddit, May 2024

After tallying the upvotes, noosa published a Promoted Post that showcased the results using a sleek custom infographic, highlighting the community’s favorites while also giving the brand important insight into consumer preferences.

Noosa published a Promoted Post that showcased the results.Screenshot by author from Reddit, May 2024

The results:

  • 49% increase in brand recall among Reddit users who were exposed to the campaign.
  • 30% increase in brand favorability.
  • 39% increase in purchase intent (15X the typical lift observed in the food vertical).
  • 50+ billion monthly views.

Sony Pictures Germany

To drum up excitement around the release of “Spider-Man: Across the Spider-Verse” in Germany, Sony Pictures Germany crafted a Reddit campaign that combined organic and promoted content to engage the platform’s robust fan communities.

The brand created a Reddit account under the protagonist’s name, u/MilesMorales, and used it to share exclusive content directly within Spiderman-related subreddits.

It also organized an AMA with the German voice actor Miles Morales to create a more personal connection with fans and promoted video posts featuring the movie trailer to amplify the excitement.

Sony Pictures Germany crafted a Reddit campaign.Screenshot by author from Reddit, May 2024

The results:

  • Sony Pictures Germany achieved a click-through rate (CTR) that was 344% higher than that of its standard paid media.
  • 831 upvotes.
  • 618,000 impressions.
  • The film remained No. 1 on the German movie charts for three consecutive weeks.

It is important to note that promoting your content and your brand on Reddit is definitely not the only value Reddit can bring you and your business.

Reddit is an amazing place to get honest feedback.

Whether you are trying to figure out what your target audience wants from your brand, test out a new feature or concept before investing and launching it openly, or even just to have continued open communication with your customers to support them and keep them connected to your brand, Reddit can help you succeed in an amazing way.

Transamerica Helping Reddit Make Dollars Make Sense

Transamerica’s campaign on Reddit is a great example of a company identifying an opportunity to connect with its target audience on the subreddit /r/finance.

This allows the company to better understand its customers’ needs and focus on content creation and future marketing plans.

Transamerica’s campaign on RedditScreenshot by author from Reddit, May 2024

Aligning itself with its audience by demonstrating awareness of the topics around personal finance happening in the subreddit, as well as acknowledging the Redditors working at the company, it asked what topics it could create “specifically for the reddit community” around the topic of finance.

It stayed involved and engaged throughout the campaign, responding to threads and maintaining a lighthearted and sometimes humorous tone.

Transamerica’s campaign on RedditScreenshot by author from Reddit, May 2024

After getting all the feedback it needed, it created content around the more popular requests, going as far as branding and designing them similar to Reddit.

IRA vs 401K venn diagramScreenshot by author from Reddit, May 2024

It even took the time to individually comment a link to the published content on its site, bringing attention back to the overall campaign and the content it created.

This campaign was 10 years ago, and it is still referenced as a core example of engaging on Reddit as a brand in a respectful, thoughtful, and meaningful way.

Why You Should Be On Reddit Wrapup

TL;DR: If your brand has something meaningful to say and is interested in truly connecting with your audience, then yes, you should be on Reddit.

These successful brand campaign examples showcase Reddit’s power when used correctly. With research to understand the specific communities you want to reach, Reddit can have exceptional performance compared to other channels.

The stakes are high to get it right because Reddit communities can be highly negative toward self-serving promotion. But if you put in the effort and solve people’s needs and problems, Reddit has the potential to be a high-performance channel.

Edit: Updated daily active users figure based on Reddit’s Q1 Earnings Report

More resources:


Featured Image: Julia Tim/Shutterstock



Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

SEO

No Algorithmic Actions For Site Reputation Abuse Yet

Published

on

By

Looking up at an angle at the Google sign on the Head Office for Canada

Google’s Search Liaison, Danny Sullivan, has confirmed that the search engine hasn’t launched algorithmic actions targeting site reputation abuse.

This clarification addresses speculation within the SEO community that recent traffic drops are related to Google’s previously announced policy update.

Sullivan Says No Update Rolled Out

Lily Ray, an SEO professional, shared a screenshot on Twitter showing a significant drop in traffic for the website Groupon starting on May 6.

Ray suggested this was evidence that Google had begun rolling out algorithmic penalties for sites violating the company’s site reputation abuse policy.

However, Sullivan quickly stepped in, stating:

“We have not gone live with algorithmic actions on site reputation abuse. I well imagine when we do, we’ll be very clear about that. Publishers seeing changes and thinking it’s this — it’s not — results change all the time for all types of reasons.”

Sullivan added that when the actions are rolled out, they will only impact specific content, not entire websites.

This is an important distinction, as it suggests that even if a site has some pages manually penalized, the rest of the domain can rank normally.

Background On Google’s Site Reputation Abuse Policy

Earlier this year, Google announced a new policy to combat what it calls “site reputation abuse.”

This refers to situations where third-party content is published on authoritative domains with little oversight or involvement from the host site.

Examples include sponsored posts, advertorials, and partner content that is loosely related to or unrelated to a site’s primary purpose.

Under the new policy, Google is taking manual action against offending pages and plans to incorporate algorithmic detection.

What This Means For Publishers & SEOs

While Google hasn’t launched any algorithmic updates related to site reputation abuse, the manual actions have publishers on high alert.

Those who rely heavily on sponsored content or partner posts to drive traffic should audit their sites and remove any potential policy violations.

Sullivan’s confirmation that algorithmic changes haven’t occurred may provide temporary relief.

Additionally, his statements also serve as a reminder that significant ranking fluctuations can happen at any time due to various factors, not just specific policy rollouts.


FAQ

Will Google’s future algorithmic actions impact entire websites or specific content?

When Google eventually rolls out algorithmic actions for site reputation abuse, these actions will target specific content rather than the entire website.

This means that if certain pages are found to be in violation, only those pages will be affected, allowing other parts of the site to continue ranking normally.

What should publishers and SEOs do in light of Google’s site reputation abuse policy?

Publishers and SEO professionals should audit their sites to identify and remove any content that may violate Google’s site reputation abuse policy.

This includes sponsored posts and partner content that doesn’t align with the site’s primary purpose. Taking these steps can mitigate the risk of manual penalties from Google.

What is the context of the recent traffic drops seen in the SEO community?

Google claims the recent drops for coupon sites aren’t linked to any algorithmic actions for site reputation abuse. Traffic fluctuations can occur for various reasons and aren’t always linked to a specific algorithm update.


Featured Image: sockagphoto/Shutterstock



Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

SEO

WP Rocket WordPress Plugin Now Optimizes LCP Core Web Vitals Metric

Published

on

By

WP Rocket WordPress Plugin Now Optimizes LCP Core Web Vitals Metric

WP Rocket, the WordPress page speed performance plugin, just announced the release of a new version that will help publishers optimize for Largest Contentful Paint (LCP), an important Core Web Vitals metric.

Large Contentful Paint (LCP)

LCP is a page speed metric that’s designed to show how fast it takes for a user to perceive that the page is loaded and read to be interacted with. This metric measures the time it takes for the main content elements has fully loaded. This gives an idea of how usable a webpage is. The faster the LCP the better the user experience will be.

WP Rocket 3.16

WP Rocket is a caching plugin that helps a site perform faster. The way page caching generally works is that the website will store frequently accessed webpages and resources so that when someone visits the page the website doesn’t have to fetch the data from the database, which takes time, but instead will serve the webpage from the cache. This is super important when a website has a lot of site visitors because that can use a lot of server resources to fetch and build the same website over and over for every visitor.

The lastest version of WP Rocket (3.16) now contains Automatic LCP optimization, which means that it will optimize the on-page elements from the main content so that they are served first thereby raising the LCP scores and providing a better user experience.

Because it’s automatic there’s really nothing to fiddle around with or fine tune.

According to WP Rocket:

  • Automatic LCP Optimization: Optimizes the Largest Contentful Paint, a critical metric for website speed, automatically enhancing overall PageSpeed scores.
  • Smart Management of Above-the-Fold Images: Automatically detects and prioritizes critical above-the-fold images, loading them immediately to improve user experience and performance metrics.

All new functionalities operate seamlessly in the background, requiring no direct intervention from the user. Upon installing or upgrading to WP Rocket 3.16, these optimizations are automatically enabled, though customization options remain accessible for those who prefer manual control.”

Read the official announcement:

WP Rocket 3.16: Improving LCP and PageSpeed Score Automatically

Featured Image by Shutterstock/ICONMAN66

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

Trending