Connect with us

SEO

2 Ways To Crush Scrapers & Hackers With Wordfence

Published

on

2 Ways To Crush Scrapers & Hackers With Wordfence


Wordfence is a popular WordPress security plugin. Among the features are scanner that monitors for hacked files and a firewall with regularly updated rules that proactively blocks malicious bots.

There’s also a useful feature tucked away in the tool that makes user-configurable firewall rules available that can supercharge your ability to block hackers, scrapers and spammers.

For some reason this tool is not immediately visible and you have to click through several menus to find it.

But once you find it you’ll discover an easy and effective way to block scrapers, hackers and spammers from attacking your site.

Scrapers are especially troublesome because they plagiarize your content and publish it elsewhere.

Now, with the tool provided by Wordfence you can do something about those scrapers.

Using a tool like Wordfence can help reduce the amount of content that scrapers can plagiarize.

Advertisement

There are many WordPress security plugins and SaaS solutions to choose from that are highly recommended, including Sucuri Security and Cloudflare. Wordfence is one of many security solutions available and it’s up to you to figure out which feels more comfortable within your workflow.

Wordfence and other solutions function fine as a set it and forget it solution.

However, in my experience I have found that the user configurable firewall in Wordfence gives one an opportunity to dial up the bot hammering power and really stick it to the hackers and scrapers.

But before you dial up the firewall it’s important to know how far these firewall rules can be taken and we’ll take a look at that, too.

Wordfence WordPress Security

Wordfence is trusted by over 4 million users for protecting their WordPress sites.

The default Firewall behavior is to block bots that grab too many pages too fast or bots and humans that display activities that signal an intent to hack the site.

The firewall will block the IP address of the rogue bot for a set period of time, after which Wordfence drops the block.

The default settings on the firewall works great.

Advertisement

But sometimes bots still get through and are able to scrape a site or probe it for vulnerabilities by scraping the site slowly.

A common approach by hackers is to set a bot to hit the site quickly and when it gets blocked it will rotate to other IP addresses and user agents, which causes a firewall to start the detection process all over again.

But these bots aren’t always programmed very well which makes it easy to block them more efficiently than with the default Wordfence settings.

Background Information About Wordfence Firewall Rules

It’s possible to accomplish efficient bot blocking with server level tools, multiple plugins and even by the use of an .htaccess file.

But editing an .htaccess file can be tricky because there are strict rules to follow and a mistake in the .htaccess file can cause the entire site to fail.

Using firewall rules is simply an easier way to block bots.

See also  Vulnerability Found in WordPress Anti-Malware Firewall

What Can You Block With Wordfence?

Wordfence allows you to create rules to block according to each of the following reasons:

Advertisement
  • IP Address Range
  • Hostname
  • Browser User Agent
  • Referrer

IP Address Range

IP address means the IP address of the server or ISP that the bot or human is coming from.

Hostname

Hostname means the name of the host. The host isn’t always declared, sometimes the bot/human visitor displays just an IP address.

Browser User Agent

Every site visitor generally tells the server what browser it is using. Browser User Agent means the browser that the visitor says it’s using.  A bot can say it’s virtually any browser, which they sometimes do in order to evade detection.

Referrer

This is a page that a bot or human supposedly clicked a link from.

Wordfence Custom Pattern Blocking

The way to block bad bots using any of the above four variables is by adding a custom rule in the Custom Pattern Blocking tool.

Here’s how to reach it.

Step 1

Click the link to the Firewall from the left side admin menu in WordPress

Step 2

Choose the tab labeled Blocking

Advertisement

Wordfence step 2

Step 3

Choose the “Custom Pattern” tab and create a firewall rule in the appropriate field. One of the fields is labeled “Block Reason.” Use that field to add a descriptive phrase like Hostname, User Agent or whatever. It will help you to review all rules you create by being able to sort by what kind of block it is.

Wordfence step 3

Step 4

Wordfence step 4

Step 5

Make your rule by clicking the “Block Visitors Matching This Pattern” button and you’re done.

Wordfence step 5

Wordfence rules can use the asterisk (*) as a wild card.

Should You Block IP Addresses with Wordfence?

Wordfence makes it easy for a publisher to set up firewall rules that efficiently blocks bots.

That’s a blessing but it can also be a curse. For example, permanently blocking thousands of IP addresses using Wordfence firewall is not efficient and probably not a proper use of Wordfence.

Temporarily blocking IP addresses is fine. Permanently blocking IP addresses probably not fine because, as I understand it, going by memory, this can bloat or slow down your WordPress installation.

In general, permanently blocking thousands or even millions of IP addresses is best accomplished with an .htaccess file.

Hostname Blocking with Wordfence

Blocking a hostname with Wordfence can be a way to block hackers, spammers and scrapers. By clicking Wordfence > Tools you can view the Wordfence Live Traffic log.

Advertisement

That shows you bot and human visitors, including bots that were blocked automatically by Wordfence.

Not all site visitors display their hostname. However in some cases they do display their hostname and that makes it easy to block an entire web host.

For example, one site, for whatever reason, attracts DDOS levels of bot traffic from a single host. None of my other sites attracts that much attention from this host, just this one site.

See also  What Is the Google Knowledge Graph & How it Works

Between March 2020 and December 2021 that one site received over 250,000 attacks and every single one of them was blocked by Wordfence.

Clearly, blocking bots by hostname can be useful if you want to block a cloud host that sends nothing but hackers and scrapers.

However some hosts, like Amazon Web Services (AWS) send both bad bots and good bots. Blocking AWS servers can also inadvertently block good bots.

So it’s important to monitor you’re traffic and be absolutely certain that blocking a hostname will not backfire.

On the other hand, if you have no use for traffic from Russia or China, then it’s easy to block hackers, scrapers and spammers from those two countries by creating a firewall rule using the hostname field.

Advertisement

All you have to do is create a rule that blocks all hostnames that end in .ru and .cn. That will block all Russian and Chinese hostnames that end in .ru and .cn.

This is what you enter into the Hostname field:

*.ru
*.cn

This is not meant to encourage anyone to use Wordfence to block Russian and Chinese bots via the hostname. It’s just an example to show how it’s done.

Block Hackers and Scrapers By User Agent

Many rogue bots use old and out of date browser user agents.

After Russia invaded Ukraine I noticed an increase in hacking bots using the Chrome 90 user agent (UA) from the same group of web hosts. Normally bot traffic is different across the different websites. So this stood out when they all looked the same across all of my sites.

Whenever Wordfence automatically blocked these bots for hitting my site too fast the bots would switch IP address and begin hitting the sites over and over again.

So I decided to block these bots by their Browser User Agent (often referred to as simply, UA).

Advertisement

First I checked the StatCounter website to determine how many users around the world are using Chrome 90. According to the StatCounter statistics, Chrome 90 browser share as of January 2022 stood at 0.09% market share in the USA.

At the time of this writing the Chrome browser is at version 100. Considering that Chrome automatically updates browser versions for the vast majority of users it’s not surprising that the usage of Chrome 90 is virtually nothing, so it’s very  unlikely that blocking all visitors using a Chrome 90 browser user agent will not block an actual and legit person visiting your site.

So I determined that it’s safe to block anything that shows up to my site with the Chrome 90 user agent.

However, there are online tools, like GTMetrix and a security server header checker, that use the Chrome 90 user agent.

So if I blocked all versions of Chrome 90 (by using this rule: *Chrome/90.*), I would also block those two online tools.

See also  YouTube Testing ‘Search Chips’ On Desktop

Another way to do is to look at the specific Chrome 90 variants used by the hackers and the online tools.

GTMetrix and the other tool use this Chrome UA:

Chrome/90.0.4430.212

Hackers and scrapers use these Chrome UAs:

Advertisement
Chrome/90.0.4400.8
Chrome/90.0.4427.0
Chrome/90.0.4430.72
Chrome/90.0.4430.85
Chrome/90.0.4430.86
Chrome/90.0.4430.93

So, if you want to allow the online tools to still scan your site but also block the bad bots, this is an example of how to do it:

*Chrome/90.0.4400.8*
*Chrome/90.0.4427.0*
*Chrome/90.0.4430.72*
*Chrome/90.0.4430.85*
*Chrome/90.0.4430.86*
*Chrome/90.0.4430.93*

This is how to block Chrome/90.0.4430.93:

How to block Chrome 90 with Wordfence

Caveat About Blocking User Agents

Before blocking Chrome 90 I kept checking the Wordfence traffic log (accessible at Wordfence > Tools) in order to be sure that no legit bots, like GTMetrix, are using Chrome 90 was using that user agent.

For example, you might not want to block Chrome 96 because some of Google’s tools use Chrome 96 as a user agent.

Always research whether legitimate bots are using a particular user agent or hostname.

And easy way to research that is by using the Wordfence Traffic Log.

Wordfence Traffic Log

The Wordfence traffic log shows you at a glance all user agents accessing your site in near real-time. The traffic log shows information such as user agent, indicates whether the visitor is a bot or a human, provides the IP address, hostname, the page being accessed and other information that helps determine if a visitor is legit or not.

The way to access the traffic log is by clicking Wordfence > Tools.

Advertisement

Blocking old browser versions is an easy way to block a lot of bad bots.  Chrome versions from the 80, 70, 60, 50, 30 and 40 series are particularly numerous on some sites.

Here’s an example of how to block old Chrome UAs that are  used by bad bots:

*Chrome/8*.*
*Chrome/7*.*
*Chrome/6*.*
*Chrome/5.0*
*Chrome/95.*
*Chrome/5*.*
*Chrome/3*.*
*Chrome/4*.*

Again, the above is not an encouragement to block the above bots.

The reason I would use *Chrome/6*.* is because with a single rule I can block the entire Chrome 60 series of user agents, Chrome 60, 61, 63, etc., without having to write all ten user agents.

I can block the entire 60 series with a single rule.

Do not block the ten and up series like this *Chrome/1*.* because that will also block the most current version of Chrome, Chrome 100.

The above is an example of how to block bad bots using the described Chrome user agents.

Bad bots also use old and retired Firefox browser user agents and some even display python-requests/ as a user agent.

Advertisement

Be Careful When Creating Firewall Rules

Always do your research first to determine what bad bots are using on your own sites and make sure that no legitimate bots or site visitors are using those old and retired browser user agents.

The way to do your research is by inspecting your traffic log files or the Wordfence traffic logs to determine which user agents (or hostnames) are from malicious traffic that you don’t want.





Source link

SEO

Surfer SEO Unveils New Semrush Integration

Published

on

Surfer SEO Unveils New Semrush Integration

Surfer SEO has a new growth management tool that includes backlink data provided by industry leader, Semrush.

Here’s what the integration means for users.

Surfer SEO

Surfer SEO is a machine learning tool used by SEOs to conduct keyword research, create content strategies, and generate AI outlines.

Companies such as Square, ClickUp, and Shopify use the tool, in addition to over 15,000 other brands and agencies.

Semrush

Semrush is an enterprise SEO tool marketers use to audit websites, conduct competitor analysis and develop content strategies.

IBM, TESLA, and Amazon are just a few companies using it, in addition to more than 10 million marketing professionals.

Starting today, Surfer SEO will use Semrush’s backlink data to expand Grow Flow task recommendations.

Advertisement

Grow Flow

Grow Flow is like your friendly SEO AI assistant.

Described on the Surfer website as an “AI growth management platform,” the tool provides a few SEO tasks each week to help you stay on track — and not get lost in the overwhelming amount of information that comes with learning SEO.

For example, it may recommend adding keywords (GSC) that a website is ranking for but not explicitly speaking to in an article.

Screenshot from Grow Flow, June 2022

Or, the tool may recommend where to add internal links.

internal links_growflow_surfer seoScreenshot from Grow Flow, June 2022

It can also recommend new content topics that users can open in a content editor at the push of a button.

new content topics_content editor_growflowScreenshot from Grow Flow, June 2022

The integration with Semrush comes into play in the Grow Flow recommendations.

See also  3 Ways Technology Can Simplify the Life of Firstline Workers

Once you connect Surfer to the freemium Semrush account, you unlock new tasks.

connect semrush to surfer_growflowScreenshot from Grow Flow, June 2022

Voila – a list of referring domains every week pulled from your competitors!

Backlink research automated_Growflow_Surfer SEOScreenshot from Grow Flow, June 2022

Manually researching a competitor’s backlinks is a considerable drain on internal resources. Thanks to this integration, you’ll discover new opportunities automatically.

An Industry First

Until now, Semrush has not integrated with an SEO company.

Historically, Semrush has only integrated with Google Products like Search Console and Analytics, and social media networks like Facebook and Twitter.

It’s also worked with task management tools like Trello and Monday.com – but never a direct SEO competitor.

Semrush’s “SEO writing assistant” feature is a direct competitor to Surfer’s “SEO Content Editor”: one of its most popular features.

Advertisement

So, why partner and offer this fantastic AI assistant to search marketers – for free?

I asked Tomasz Niezgoda, Surfer SEO’s Marketing Executive and Partner, how this partnership came to be.

“Pretty straightforward,” he said. “Semrush reached out to us with a proposition to integrate Surfer inside their marketplace [and] after some time, we decided to give it a shot and started working on this integration.”

From the beginning, Semrush felt like this would be a successful integration because it was applying its backlink data in a customer-oriented way.

“For Semrush, it’s a very meaningful integration. It allows Surfer SEO users to gain valuable link building insights and knowledge, which is crucial for ranking their content,” said Eugene Levin, President, Semrush.

What This Means

Two major SEO competitors are working together to create a free tool for small business owners and entry-level marketers to develop weekly best practices.

See also  6 Top Tips From Experts

Matt Diggity, CEO and Founder at Diggity Marketing, calls the tool “Simple. Efficient. Automated.”

“You get the list every week and gain immediate insights [into] which referrals your competition is getting. Then, you can start working on your own link-building strategy straight away.”

Advertisement

The two giants hint that combining Semrush’s backlink data with the machine learning power of Surfer is only the beginning.

“We like each other,” said Niezgoda. “Maybe it’s just the first integration that’s coming.”


Featured Image: Screenshot from Grow Flow, June 2022

fbq('track', 'PageView');

fbq('trackSingle', '1321385257908563', 'ViewContent', { content_name: 'sej-exclusive-surfer-seo-unveils-semrush-integration', content_category: 'news digital-marketing-tools' }); }



Source link

Continue Reading

DON'T MISS ANY IMPORTANT NEWS!
Subscribe To our Newsletter
We promise not to spam you. Unsubscribe at any time.
Invalid email address

Trending

en_USEnglish