SEO
Using Python + Streamlit To Find Striking Distance Keyword Opportunities
Python is an excellent tool to automate repetitive tasks as well as gain additional insights into data.
In this article, you’ll learn how to build a tool to check which keywords are close to ranking in positions one to three and advises whether there is an opportunity to naturally work those keywords into the page.
It’s perfect for Python beginners and pros alike and is a great introduction to using Python for SEO.
If you’d just like to get stuck in there’s a handy Streamlit app available for the code. This is simple to use and requires no coding experience.
There’s also a Google Colaboratory Sheet if you’d like to poke around with the code. If you can crawl a website, you can use this script!
Here’s an example of what we’ll be making today:
These keywords are found in the page title and H1, but not in the copy. Adding these keywords naturally to the existing copy would be an easy way to increase relevancy for these keywords.
By taking the hint from search engines and naturally including any missing keywords a site already ranks for, we increase the confidence of search engines to rank those keywords higher in the SERPs.
This report can be created manually, but it’s pretty time-consuming.
So, we’re going to automate the process using a Python SEO script.
Preview Of The Output
This is a sample of what the final output will look like after running the report:
Screenshot from Microsoft Excel, October 2021The final output takes the top five opportunities by search volume for each page and neatly lays each one horizontally along with the estimated search volume.
It also shows the total search volume of all keywords a page has within striking distance, as well as the total number of keywords within reach.
The top five keywords by search volume are then checked to see if they are found in the title, H1, or copy, then flagged TRUE or FALSE.
This is great for finding quick wins! Just add the missing keyword naturally into the page copy, title, or H1.
Getting Started
The setup is fairly straightforward. We just need a crawl of the site (ideally with a custom extraction for the copy you’d like to check), and an exported file of all keywords a site ranks for.
This post will walk you through the setup, the code, and will link to a Google Colaboratory sheet if you just want to get stuck in without coding it yourself.
To get started you will need:
We’ve named this the Striking Distance Report as it flags keywords that are easily within striking distance.
(We have defined striking distance as keywords that rank in positions four to 20, but have made this a configurable option in case you would like to define your own parameters.)
Striking Distance SEO Report: Getting Started
1. Crawl The Target Website
- Set a custom extractor for the page copy (optional, but recommended).
- Filter out pagination pages from the crawl.
2. Export All Keywords The Site Ranks For Using Your Favorite Provider
- Filter keywords that trigger as a site link.
- Remove keywords that trigger as an image.
- Filter branded keywords.
- Use both exports to create an actionable Striking Distance report from the keyword and crawl data with Python.
Crawling The Site
I’ve opted to use Screaming Frog to get the initial crawl. Any crawler will work, so long as the CSV export uses the same column names or they’re renamed to match.
The script expects to find the following columns in the crawl CSV export:
"Address", "Title 1", "H1-1", "Copy 1", "Indexability"
Crawl Settings
The first thing to do is to head over to the main configuration settings within Screaming Frog:
Configuration > Spider > Crawl
The main settings to use are:
Crawl Internal Links, Canonicals, and the Pagination (Rel Next/Prev) setting.
(The script will work with everything else selected, but the crawl will take longer to complete!)
Screenshot from Screaming Frog, October 2021Next, it’s on to the Extraction tab.
Configuration > Spider > Extraction
Screenshot from Screaming Frog, October 2021At a bare minimum, we need to extract the page title, H1, and calculate whether the page is indexable as shown below.
Indexability is useful because it’s an easy way for the script to identify which URLs to drop in one go, leaving only keywords that are eligible to rank in the SERPs.
If the script cannot find the indexability column, it’ll still work as normal but won’t differentiate between pages that can and cannot rank.
Setting A Custom Extractor For Page Copy
In order to check whether a keyword is found within the page copy, we need to set a custom extractor in Screaming Frog.
Configuration > Custom > Extraction
Name the extractor “Copy” as seen below.
Screenshot from Screaming Frog, October 2021Important: The script expects the extractor to be named “Copy” as above, so please double check!
Lastly, make sure Extract Text is selected to export the copy as text, rather than HTML.
There are many guides on using custom extractors online if you need help setting one up, so I won’t go over it again here.
Once the extraction has been set it’s time to crawl the site and export the HTML file in CSV format.
Exporting The CSV File
Exporting the CSV file is as easy as changing the drop-down menu displayed underneath Internal to HTML and pressing the Export button.
Internal > HTML > Export
Screenshot from Screaming Frog, October 2021After clicking Export, It’s important to make sure the type is set to CSV format.
The export screen should look like the below:
Screenshot from Screaming Frog, October 2021Tip 1: Filtering Out Pagination Pages
I recommend filtering out pagination pages from your crawl either by selecting Respect Next/Prev under the Advanced settings (or just deleting them from the CSV file, if you prefer).
Screenshot from Screaming Frog, October 2021Tip 2: Saving The Crawl Settings
Once you have set the crawl up, it’s worth just saving the crawl settings (which will also remember the custom extraction).
This will save a lot of time if you want to use the script again in the future.
File > Configuration > Save As
Screenshot from Screaming Frog, October 2021Exporting Keywords
Once we have the crawl file, the next step is to load your favorite keyword research tool and export all of the keywords a site ranks for.
The goal here is to export all the keywords a site ranks for, filtering out branded keywords and any which triggered as a sitelink or image.
For this example, I’m using the Organic Keyword Report in Ahrefs, but it will work just as well with Semrush if that’s your preferred tool.
In Ahrefs, enter the domain you’d like to check in Site Explorer and choose Organic Keywords.
Screenshot from Ahrefs.com, October 2021Site Explorer > Organic Keywords
Screenshot from Ahrefs.com, October 2021This will bring up all keywords the site is ranking for.
Filtering Out Sitelinks And Image links
The next step is to filter out any keywords triggered as a sitelink or an image pack.
The reason we need to filter out sitelinks is that they have no influence on the parent URL ranking. This is because only the parent page technically ranks for the keyword, not the sitelink URLs displayed under it.
Filtering out sitelinks will ensure that we are optimizing the correct page.
Screenshot from Ahrefs.com, October 2021Here’s how to do it in Ahrefs.
Screenshot from Ahrefs.com, October 2021Lastly, I recommend filtering out any branded keywords. You can do this by filtering the CSV output directly, or by pre-filtering in the keyword tool of your choice before the export.
Finally, when exporting make sure to choose Full Export and the UTF-8 format as shown below.
Screenshot from Ahrefs.com, October 2021By default, the script works with Ahrefs (v1/v2) and Semrush keyword exports. It can work with any keyword CSV file as long as the column names the script expects are present.
Processing
The following instructions pertain to running a Google Colaboratory sheet to execute the code.
There is now a simpler option for those that prefer it in the form of a Streamlit app. Simply follow the instructions provided to upload your crawl and keyword file.
Now that we have our exported files, all that’s left to be done is to upload them to the Google Colaboratory sheet for processing.
Select Runtime > Run all from the top navigation to run all cells in the sheet.
Screenshot from Colab.research.google.com, October 2021The script will prompt you to upload the keyword CSV from Ahrefs or Semrush first and the crawl file afterward.
Screenshot from Colab.research.google.com, October 2021That’s it! The script will automatically download an actionable CSV file you can use to optimize your site.
Screenshot from Microsoft Excel, October 2021Once you’re familiar with the whole process, using the script is really straightforward.
Code Breakdown And Explanation
If you’re learning Python for SEO and interested in what the code is doing to produce the report, stick around for the code walkthrough!
Install The Libraries
Let’s install pandas to get the ball rolling.
!pip install pandas
Import The Modules
Next, we need to import the required modules.
import pandas as pd from pandas import DataFrame, Series from typing import Union from google.colab import files
Set The Variables
Now it’s time to set the variables.
The script considers any keywords between positions four and 20 as within striking distance.
Changing the variables here will let you define your own range if desired. It’s worth experimenting with the settings to get the best possible output for your needs.
# set all variables here min_volume = 10 # set the minimum search volume min_position = 4 # set the minimum position / default = 4 max_position = 20 # set the maximum position / default = 20 drop_all_true = True # If all checks (h1/title/copy) are true, remove the recommendation (Nothing to do) pagination_filters = "filterby|page|p=" # filter patterns used to detect and drop paginated pages
Upload The Keyword Export CSV File
The next step is to read in the list of keywords from the CSV file.
It is set up to accept an Ahrefs report (V1 and V2) as well as a Semrush export.
This code reads in the CSV file into a Pandas DataFrame.
upload = files.upload()
upload = list(upload.keys())[0]
df_keywords = pd.read_csv(
(upload),
error_bad_lines=False,
low_memory=False,
encoding="utf8",
dtype={
"URL": "str",
"Keyword": "str",
"Volume": "str",
"Position": int,
"Current URL": "str",
"Search Volume": int,
},
)
print("Uploaded Keyword CSV File Successfully!")
If everything went to plan, you’ll see a preview of the DataFrame created from the keyword CSV export.
Screenshot from Colab.research.google.com, October 2021Upload The Crawl Export CSV File
Once the keywords have been imported, it’s time to upload the crawl file.
This fairly simple piece of code reads in the crawl with some error handling option and creates a Pandas DataFrame named df_crawl.
upload = files.upload()
upload = list(upload.keys())[0]
df_crawl = pd.read_csv(
(upload),
error_bad_lines=False,
low_memory=False,
encoding="utf8",
dtype="str",
)
print("Uploaded Crawl Dataframe Successfully!")
Once the CSV file has finished uploading, you’ll see a preview of the DataFrame.
Screenshot from Colab.research.google.com, October 2021Clean And Standardize The Keyword Data
The next step is to rename the column names to ensure standardization between the most common types of file exports.
Essentially, we’re getting the keyword DataFrame into a good state and filtering using cutoffs defined by the variables.
df_keywords.rename(
columns={
"Current position": "Position",
"Current URL": "URL",
"Search Volume": "Volume",
},
inplace=True,
)
# keep only the following columns from the keyword dataframe
cols = "URL", "Keyword", "Volume", "Position"
df_keywords = df_keywords.reindex(columns=cols)
try:
# clean the data. (v1 of the ahrefs keyword export combines strings and ints in the volume column)
df_keywords["Volume"] = df_keywords["Volume"].str.replace("0-10", "0")
except AttributeError:
pass
# clean the keyword data
df_keywords = df_keywords[df_keywords["URL"].notna()] # remove any missing values
df_keywords = df_keywords[df_keywords["Volume"].notna()] # remove any missing values
df_keywords = df_keywords.astype({"Volume": int}) # change data type to int
df_keywords = df_keywords.sort_values(by="Volume", ascending=False) # sort by highest vol to keep the top opportunity
# make new dataframe to merge search volume back in later
df_keyword_vol = df_keywords[["Keyword", "Volume"]]
# drop rows if minimum search volume doesn't match specified criteria
df_keywords.loc[df_keywords["Volume"] < min_volume, "Volume_Too_Low"] = "drop"
df_keywords = df_keywords[~df_keywords["Volume_Too_Low"].isin(["drop"])]
# drop rows if minimum search position doesn't match specified criteria
df_keywords.loc[df_keywords["Position"] <= min_position, "Position_Too_High"] = "drop"
df_keywords = df_keywords[~df_keywords["Position_Too_High"].isin(["drop"])]
# drop rows if maximum search position doesn't match specified criteria
df_keywords.loc[df_keywords["Position"] >= max_position, "Position_Too_Low"] = "drop"
df_keywords = df_keywords[~df_keywords["Position_Too_Low"].isin(["drop"])]
Clean And Standardize The Crawl Data
Next, we need to clean and standardize the crawl data.
Essentially, we use reindex to only keep the “Address,” “Indexability,” “Page Title,” “H1-1,” and “Copy 1” columns, discarding the rest.
We use the handy “Indexability” column to only keep rows that are indexable. This will drop canonicalized URLs, redirects, and so on. I recommend enabling this option in the crawl.
Lastly, we standardize the column names so they’re a little nicer to work with.
# keep only the following columns from the crawl dataframe
cols = "Address", "Indexability", "Title 1", "H1-1", "Copy 1"
df_crawl = df_crawl.reindex(columns=cols)
# drop non-indexable rows
df_crawl = df_crawl[~df_crawl["Indexability"].isin(["Non-Indexable"])]
# standardise the column names
df_crawl.rename(columns={"Address": "URL", "Title 1": "Title", "H1-1": "H1", "Copy 1": "Copy"}, inplace=True)
df_crawl.head()
Group The Keywords
As we approach the final output, it’s necessary to group our keywords together to calculate the total opportunity for each page.
Here, we’re calculating how many keywords are within striking distance for each page, along with the combined search volume.
# groups the URLs (remove the dupes and combines stats)
# make a copy of the keywords dataframe for grouping - this ensures stats can be merged back in later from the OG df
df_keywords_group = df_keywords.copy()
df_keywords_group["KWs in Striking Dist."] = 1 # used to count the number of keywords in striking distance
df_keywords_group = (
df_keywords_group.groupby("URL")
.agg({"Volume": "sum", "KWs in Striking Dist.": "count"})
.reset_index()
)
df_keywords_group.head()
Screenshot from Colab.research.google.com, October 2021Once complete, you’ll see a preview of the DataFrame.
Display Keywords In Adjacent Rows
We use the grouped data as the basis for the final output. We use Pandas.unstack to reshape the DataFrame to display the keywords in the style of a GrepWords export.
Screenshot from Colab.research.google.com, October 2021# create a new df, combine the merged data with the original data. display in adjacent rows ala grepwords
df_merged_all_kws = df_keywords_group.merge(
df_keywords.groupby("URL")["Keyword"]
.apply(lambda x: x.reset_index(drop=True))
.unstack()
.reset_index()
)
# sort by biggest opportunity
df_merged_all_kws = df_merged_all_kws.sort_values(
by="KWs in Striking Dist.", ascending=False
)
# reindex the columns to keep just the top five keywords
cols = "URL", "Volume", "KWs in Striking Dist.", 0, 1, 2, 3, 4
df_merged_all_kws = df_merged_all_kws.reindex(columns=cols)
# create union and rename the columns
df_striking: Union[Series, DataFrame, None] = df_merged_all_kws.rename(
columns={
"Volume": "Striking Dist. Vol",
0: "KW1",
1: "KW2",
2: "KW3",
3: "KW4",
4: "KW5",
}
)
# merges striking distance df with crawl df to merge in the title, h1 and category description
df_striking = pd.merge(df_striking, df_crawl, on="URL", how="inner")
Set The Final Column Order And Insert Placeholder Columns
Lastly, we set the final column order and merge in the original keyword data.
There are a lot of columns to sort and create!
# set the final column order and merge the keyword data in
cols = [
"URL",
"Title",
"H1",
"Copy",
"Striking Dist. Vol",
"KWs in Striking Dist.",
"KW1",
"KW1 Vol",
"KW1 in Title",
"KW1 in H1",
"KW1 in Copy",
"KW2",
"KW2 Vol",
"KW2 in Title",
"KW2 in H1",
"KW2 in Copy",
"KW3",
"KW3 Vol",
"KW3 in Title",
"KW3 in H1",
"KW3 in Copy",
"KW4",
"KW4 Vol",
"KW4 in Title",
"KW4 in H1",
"KW4 in Copy",
"KW5",
"KW5 Vol",
"KW5 in Title",
"KW5 in H1",
"KW5 in Copy",
]
# re-index the columns to place them in a logical order + inserts new blank columns for kw checks.
df_striking = df_striking.reindex(columns=cols)
Merge In The Keyword Data For Each Column
This code merges the keyword volume data back into the DataFrame. It’s more or less the equivalent of an Excel VLOOKUP function.
# merge in keyword data for each keyword column (KW1 - KW5) df_striking = pd.merge(df_striking, df_keyword_vol, left_on="KW1", right_on="Keyword", how="left") df_striking['KW1 Vol'] = df_striking['Volume'] df_striking.drop(['Keyword', 'Volume'], axis=1, inplace=True) df_striking = pd.merge(df_striking, df_keyword_vol, left_on="KW2", right_on="Keyword", how="left") df_striking['KW2 Vol'] = df_striking['Volume'] df_striking.drop(['Keyword', 'Volume'], axis=1, inplace=True) df_striking = pd.merge(df_striking, df_keyword_vol, left_on="KW3", right_on="Keyword", how="left") df_striking['KW3 Vol'] = df_striking['Volume'] df_striking.drop(['Keyword', 'Volume'], axis=1, inplace=True) df_striking = pd.merge(df_striking, df_keyword_vol, left_on="KW4", right_on="Keyword", how="left") df_striking['KW4 Vol'] = df_striking['Volume'] df_striking.drop(['Keyword', 'Volume'], axis=1, inplace=True) df_striking = pd.merge(df_striking, df_keyword_vol, left_on="KW5", right_on="Keyword", how="left") df_striking['KW5 Vol'] = df_striking['Volume'] df_striking.drop(['Keyword', 'Volume'], axis=1, inplace=True)
Clean The Data Some More
The data requires additional cleaning to populate empty values, (NaNs), as empty strings. This improves the readability of the final output by creating blank cells, instead of cells populated with NaN string values.
Next, we convert the columns to lowercase so that they match when checking whether a target keyword is featured in a specific column.
# replace nan values with empty strings
df_striking = df_striking.fillna("")
# drop the title, h1 and category description to lower case so kws can be matched to them
df_striking["Title"] = df_striking["Title"].str.lower()
df_striking["H1"] = df_striking["H1"].str.lower()
df_striking["Copy"] = df_striking["Copy"].str.lower()
Check Whether The Keyword Appears In The Title/H1/Copy and Return True Or False
This code checks if the target keyword is found in the page title/H1 or copy.
It’ll flag true or false depending on whether a keyword was found within the on-page elements.
df_striking["KW1 in Title"] = df_striking.apply(lambda row: row["KW1"] in row["Title"], axis=1) df_striking["KW1 in H1"] = df_striking.apply(lambda row: row["KW1"] in row["H1"], axis=1) df_striking["KW1 in Copy"] = df_striking.apply(lambda row: row["KW1"] in row["Copy"], axis=1) df_striking["KW2 in Title"] = df_striking.apply(lambda row: row["KW2"] in row["Title"], axis=1) df_striking["KW2 in H1"] = df_striking.apply(lambda row: row["KW2"] in row["H1"], axis=1) df_striking["KW2 in Copy"] = df_striking.apply(lambda row: row["KW2"] in row["Copy"], axis=1) df_striking["KW3 in Title"] = df_striking.apply(lambda row: row["KW3"] in row["Title"], axis=1) df_striking["KW3 in H1"] = df_striking.apply(lambda row: row["KW3"] in row["H1"], axis=1) df_striking["KW3 in Copy"] = df_striking.apply(lambda row: row["KW3"] in row["Copy"], axis=1) df_striking["KW4 in Title"] = df_striking.apply(lambda row: row["KW4"] in row["Title"], axis=1) df_striking["KW4 in H1"] = df_striking.apply(lambda row: row["KW4"] in row["H1"], axis=1) df_striking["KW4 in Copy"] = df_striking.apply(lambda row: row["KW4"] in row["Copy"], axis=1) df_striking["KW5 in Title"] = df_striking.apply(lambda row: row["KW5"] in row["Title"], axis=1) df_striking["KW5 in H1"] = df_striking.apply(lambda row: row["KW5"] in row["H1"], axis=1) df_striking["KW5 in Copy"] = df_striking.apply(lambda row: row["KW5"] in row["Copy"], axis=1)
Delete True/False Values If There Is No Keyword
This will delete true/false values when there is no keyword adjacent.
# delete true / false values if there is no keyword df_striking.loc[df_striking["KW1"] == "", ["KW1 in Title", "KW1 in H1", "KW1 in Copy"]] = "" df_striking.loc[df_striking["KW2"] == "", ["KW2 in Title", "KW2 in H1", "KW2 in Copy"]] = "" df_striking.loc[df_striking["KW3"] == "", ["KW3 in Title", "KW3 in H1", "KW3 in Copy"]] = "" df_striking.loc[df_striking["KW4"] == "", ["KW4 in Title", "KW4 in H1", "KW4 in Copy"]] = "" df_striking.loc[df_striking["KW5"] == "", ["KW5 in Title", "KW5 in H1", "KW5 in Copy"]] = "" df_striking.head()
Drop Rows If All Values == True
This configurable option is really useful for reducing the amount of QA time required for the final output by dropping the keyword opportunity from the final output if it is found in all three columns.
def true_dropper(col1, col2, col3):
drop = df_striking.drop(
df_striking[
(df_striking[col1] == True)
& (df_striking[col2] == True)
& (df_striking[col3] == True)
].index
)
return drop
if drop_all_true == True:
df_striking = true_dropper("KW1 in Title", "KW1 in H1", "KW1 in Copy")
df_striking = true_dropper("KW2 in Title", "KW2 in H1", "KW2 in Copy")
df_striking = true_dropper("KW3 in Title", "KW3 in H1", "KW3 in Copy")
df_striking = true_dropper("KW4 in Title", "KW4 in H1", "KW4 in Copy")
df_striking = true_dropper("KW5 in Title", "KW5 in H1", "KW5 in Copy")
Download The CSV File
The last step is to download the CSV file and start the optimization process.
df_striking.to_csv('Keywords in Striking Distance.csv', index=False)
files.download("Keywords in Striking Distance.csv")
Conclusion
If you are looking for quick wins for any website, the striking distance report is a really easy way to find them.
Don’t let the number of steps fool you. It’s not as complex as it seems. It’s as simple as uploading a crawl and keyword export to the supplied Google Colab sheet or using the Streamlit app.
The results are definitely worth it!
More Resources:
Featured Image: aurielaki/Shutterstock
!function(f,b,e,v,n,t,s)
{if(f.fbq)return;n=f.fbq=function(){n.callMethod?
n.callMethod.apply(n,arguments):n.queue.push(arguments)};
if(!f._fbq)f._fbq=n;n.push=n;n.loaded=!0;n.version=’2.0′;
n.queue=[];t=b.createElement(e);t.async=!0;
t.src=v;s=b.getElementsByTagName(e)[0];
s.parentNode.insertBefore(t,s)}(window,document,’script’,
‘https://connect.facebook.net/en_US/fbevents.js’);
if( typeof sopp !== “undefined” && sopp === ‘yes’ ){
fbq(‘dataProcessingOptions’, [‘LDU’], 1, 1000);
}else{
fbq(‘dataProcessingOptions’, []);
}
fbq(‘init’, ‘1321385257908563’);
fbq(‘track’, ‘PageView’);
fbq(‘trackSingle’, ‘1321385257908563’, ‘ViewContent’, {
content_name: ‘striking-distance-keywords-python’,
content_category: ‘seo-strategy technical-seo ‘
});
SEO
2024 WordPress Vulnerability Report Shows Errors Sites Keep Making
WordPress security scanner WPScan’s 2024 WordPress vulnerability report calls attention to WordPress vulnerability trends and suggests the kinds of things website publishers (and SEOs) should be looking out for.
Some of the key findings from the report were that just over 20% of vulnerabilities were rated as high or critical level threats, with medium severity threats, at 67% of reported vulnerabilities, making up the majority. Many regard medium level vulnerabilities as if they are low-level threats and that’s a mistake because they’re not low level and should be regarded as deserving attention.
The WPScan report advised:
“While severity doesn’t translate directly to the risk of exploitation, it’s an important guideline for website owners to make an educated decision about when to disable or update the extension.”
WordPress Vulnerability Severity Distribution
Critical level vulnerabilities, the highest level of threat, represented only 2.38% of vulnerabilities, which is essentially good news for WordPress publishers. Yet as mentioned earlier, when combined with the percentages of high level threats (17.68%) the number or concerning vulnerabilities rises to almost 20%.
Here are the percentages by severity ratings:
- Critical 2.38%
- Low 12.83%
- High 17.68%
- Medium 67.12%
Authenticated Versus Unauthenticated
Authenticated vulnerabilities are those that require an attacker to first attain user credentials and their accompanying permission levels in order to exploit a particular vulnerability. Exploits that require subscriber-level authentication are the most exploitable of the authenticated exploits and those that require administrator level access present the least risk (although not always a low risk for a variety of reasons).
Unauthenticated attacks are generally the easiest to exploit because anyone can launch an attack without having to first acquire a user credential.
The WPScan vulnerability report found that about 22% of reported vulnerabilities required subscriber level or no authentication at all, representing the most exploitable vulnerabilities. On the other end of the scale of the exploitability are vulnerabilities requiring admin permission levels representing a total of 30.71% of reported vulnerabilities.
Permission Levels Required For Exploits
Vulnerabilities requiring administrator level credentials represented the highest percentage of exploits, followed by Cross Site Request Forgery (CSRF) with 24.74% of vulnerabilities. This is interesting because CSRF is an attack that uses social engineering to get a victim to click a link from which the user’s permission levels are acquired. This is a mistake that WordPress publishers should be aware of because all it takes is for an admin level user to follow a link which then enables the hacker to assume admin level privileges to the WordPress website.
The following is the percentages of exploits ordered by roles necessary to launch an attack.
Ascending Order Of User Roles For Vulnerabilities
- Author 2.19%
- Subscriber 10.4%
- Unauthenticated 12.35%
- Contributor 19.62%
- CSRF 24.74%
- Admin 30.71%
Most Common Vulnerability Types Requiring Minimal Authentication
Broken Access Control in the context of WordPress refers to a security failure that can allow an attacker without necessary permission credentials to gain access to higher credential permissions.
In the section of the report that looks at the occurrences and vulnerabilities underlying unauthenticated or subscriber level vulnerabilities reported (Occurrence vs Vulnerability on Unauthenticated or Subscriber+ reports), WPScan breaks down the percentages for each vulnerability type that is most common for exploits that are the easiest to launch (because they require minimal to no user credential authentication).
The WPScan threat report noted that Broken Access Control represents a whopping 84.99% followed by SQL injection (20.64%).
The Open Worldwide Application Security Project (OWASP) defines Broken Access Control as:
“Access control, sometimes called authorization, is how a web application grants access to content and functions to some users and not others. These checks are performed after authentication, and govern what ‘authorized’ users are allowed to do.
Access control sounds like a simple problem but is insidiously difficult to implement correctly. A web application’s access control model is closely tied to the content and functions that the site provides. In addition, the users may fall into a number of groups or roles with different abilities or privileges.”
SQL injection, at 20.64% represents the second most prevalent type of vulnerability, which WPScan referred to as both “high severity and risk” in the context of vulnerabilities requiring minimal authentication levels because attackers can access and/or tamper with the database which is the heart of every WordPress website.
These are the percentages:
- Broken Access Control 84.99%
- SQL Injection 20.64%
- Cross-Site Scripting 9.4%
- Unauthenticated Arbitrary File Upload 5.28%
- Sensitive Data Disclosure 4.59%
- Insecure Direct Object Reference (IDOR) 3.67%
- Remote Code Execution 2.52%
- Other 14.45%
Vulnerabilities In The WordPress Core Itself
The overwhelming majority of vulnerability issues were reported in third-party plugins and themes. However, there were in 2023 a total of 13 vulnerabilities reported in the WordPress core itself. Out of the thirteen vulnerabilities only one of them was rated as a high severity threat, which is the second highest level, with Critical being the highest level vulnerability threat, a rating scoring system maintained by the Common Vulnerability Scoring System (CVSS).
The WordPress core platform itself is held to the highest standards and benefits from a worldwide community that is vigilant in discovering and patching vulnerabilities.
Website Security Should Be Considered As Technical SEO
Site audits don’t normally cover website security but in my opinion every responsible audit should at least talk about security headers. As I’ve been saying for years, website security quickly becomes an SEO issue once a website’s ranking start disappearing from the search engine results pages (SERPs) due to being compromised by a vulnerability. That’s why it’s critical to be proactive about website security.
According to the WPScan report, the main point of entry for hacked websites were leaked credentials and weak passwords. Ensuring strong password standards plus two-factor authentication is an important part of every website’s security stance.
Using security headers is another way to help protect against Cross-Site Scripting and other kinds of vulnerabilities.
Lastly, a WordPress firewall and website hardening are also useful proactive approaches to website security. I once added a forum to a brand new website I created and it was immediately under attack within minutes. Believe it or not, virtually every website worldwide is under attack 24 hours a day by bots scanning for vulnerabilities.
Read the WPScan Report:
WPScan 2024 Website Threat Report
Featured Image by Shutterstock/Ljupco Smokovski
SEO
An In-Depth Guide And Best Practices For Mobile SEO
Over the years, search engines have encouraged businesses to improve mobile experience on their websites. More than 60% of web traffic comes from mobile, and in some cases based on the industry, mobile traffic can reach up to 90%.
Since Google has completed its switch to mobile-first indexing, the question is no longer “if” your website should be optimized for mobile, but how well it is adapted to meet these criteria. A new challenge has emerged for SEO professionals with the introduction of Interaction to Next Paint (INP), which replaced First Input Delay (FID) starting March, 12 2024.
Thus, understanding mobile SEO’s latest advancements, especially with the shift to INP, is crucial. This guide offers practical steps to optimize your site effectively for today’s mobile-focused SEO requirements.
What Is Mobile SEO And Why Is It Important?
The goal of mobile SEO is to optimize your website to attain better visibility in search engine results specifically tailored for mobile devices.
This form of SEO not only aims to boost search engine rankings, but also prioritizes enhancing mobile user experience through both content and technology.
While, in many ways, mobile SEO and traditional SEO share similar practices, additional steps related to site rendering and content are required to meet the needs of mobile users and the speed requirements of mobile devices.
Does this need to be a priority for your website? How urgent is it?
Consider this: 58% of the world’s web traffic comes from mobile devices.
If you aren’t focused on mobile users, there is a good chance you’re missing out on a tremendous amount of traffic.
Mobile-First Indexing
Additionally, as of 2023, Google has switched its crawlers to a mobile-first indexing priority.
This means that the mobile experience of your site is critical to maintaining efficient indexing, which is the step before ranking algorithms come into play.
Read more: Where We Are Today With Google’s Mobile-First Index
How Much Of Your Traffic Is From Mobile?
How much traffic potential you have with mobile users can depend on various factors, including your industry (B2B sites might attract primarily desktop users, for example) and the search intent your content addresses (users might prefer desktop for larger purchases, for example).
Regardless of where your industry and the search intent of your users might be, the future will demand that you optimize your site experience for mobile devices.
How can you assess your current mix of mobile vs. desktop users?
An easy way to see what percentage of your users is on mobile is to go into Google Analytics 4.
- Click Reports in the left column.
- Click on the Insights icon on the right side of the screen.
- Scroll down to Suggested Questions and click on it.
- Click on Technology.
- Click on Top Device model by Users.
- Then click on Top Device category by Users under Related Results.
- The breakdown of Top Device category will match the date range selected at the top of GA4.
You can also set up a report in Looker Studio.
- Add your site to the Data source.
- Add Device category to the Dimension field.
- Add 30-day active users to the Metric field.
- Click on Chart to select the view that works best for you.
Screenshot from Looker Studio, March 2024You can add more Dimensions to really dig into the data to see which pages attract which type of users, what the mobile-to-desktop mix is by country, which search engines send the most mobile users, and so much more.
Read more: Why Mobile And Desktop Rankings Are Different
How To Check If Your Site Is Mobile-Friendly
Now that you know how to build a report on mobile and desktop usage, you need to figure out if your site is optimized for mobile traffic.
While Google removed the mobile-friendly testing tool from Google Search Console in December 2023, there are still a number of useful tools for evaluating your site for mobile users.
Bing still has a mobile-friendly testing tool that will tell you the following:
- Viewport is configured correctly.
- Page content fits device width.
- Text on the page is readable.
- Links and tap targets are sufficiently large and touch-friendly.
- Any other issues detected.
Google’s Lighthouse Chrome extension provides you with an evaluation of your site’s performance across several factors, including load times, accessibility, and SEO.
To use, install the Lighthouse Chrome extension.
- Go to your website in your browser.
- Click on the orange lighthouse icon in your browser’s address bar.
- Click Generate Report.
- A new tab will open and display your scores once the evaluation is complete.
Screenshot from Lighthouse, March 2024You can also use the Lighthouse report in Developer Tools in Chrome.
- Simply click on the three dots next to the address bar.
- Select “More Tools.”
- Select Developer Tools.
- Click on the Lighthouse tab.
- Choose “Mobile” and click the “Analyze page load” button.
Screenshot from Lighthouse, March 2024Another option that Google offers is the PageSpeed Insights (PSI) tool. Simply add your URL into the field and click Analyze.
PSI will integrate any Core Web Vitals scores into the resulting view so you can see what your users are experiencing when they come to your site.
Screenshot from PageSpeed Insights, March 2024Other tools, like WebPageTest.org, will graphically display the processes and load times for everything it takes to display your webpages.
With this information, you can see which processes block the loading of your pages, which ones take the longest to load, and how this affects your overall page load times.
You can also emulate the mobile experience by using Developer Tools in Chrome, which allows you to switch back and forth between a desktop and mobile experience.
Screenshot from Google Chrome Developer Tools, March 2024Lastly, use your own mobile device to load and navigate your website:
- Does it take forever to load?
- Are you able to navigate your site to find the most important information?
- Is it easy to add something to cart?
- Can you read the text?
Read more: Google PageSpeed Insights Reports: A Technical Guide
How To Optimize Your Site Mobile-First
With all these tools, keep an eye on the Performance and Accessibility scores, as these directly affect mobile users.
Expand each section within the PageSpeed Insights report to see what elements are affecting your score.
These sections can give your developers their marching orders for optimizing the mobile experience.
While mobile speeds for cellular networks have steadily improved around the world (the average speed in the U.S. has jumped to 27.06 Mbps from 11.14 Mbps in just eight years), speed and usability for mobile users are at a premium.
Read more: Top 7 SEO Benefits Of Responsive Web Design
Best Practices For Mobile Optimization
Unlike traditional SEO, which can focus heavily on ensuring that you are using the language of your users as it relates to the intersection of your products/services and their needs, optimizing for mobile SEO can seem very technical SEO-heavy.
While you still need to be focused on matching your content with the needs of the user, mobile search optimization will require the aid of your developers and designers to be fully effective.
Below are several key factors in mobile SEO to keep in mind as you’re optimizing your site.
Site Rendering
How your site responds to different devices is one of the most important elements in mobile SEO.
The two most common approaches to this are responsive design and dynamic serving.
Responsive design is the most common of the two options.
Using your site’s cascading style sheets (CSS) and flexible layouts, as well as responsive content delivery networks (CDN) and modern image file types, responsive design allows your site to adjust to a variety of screen sizes, orientations, and resolutions.
With the responsive design, elements on the page adjust in size and location based on the size of the screen.
You can simply resize the window of your desktop browser and see how this works.
Screenshot from web.dev, March 2024This is the approach that Google recommends.
Adaptive design, also known as dynamic serving, consists of multiple fixed layouts that are dynamically served to the user based on their device.
Sites can have a separate layout for desktop, smartphone, and tablet users. Each design can be modified to remove functionality that may not make sense for certain device types.
This is a less efficient approach, but it does give sites more control over what each device sees.
While these will not be covered here, two other options:
- Progressive Web Apps (PWA), which can seamlessly integrate into a mobile app.
- Separate mobile site/URL (which is no longer recommended).
Read more: An Introduction To Rendering For SEO
Interaction to Next Paint (INP)
Google has introduced Interaction to Next Paint (INP) as a more comprehensive measure of user experience, succeeding First Input Delay. While FID measures the time from when a user first interacts with your page (e.g., clicking a link, tapping a button) to the time when the browser is actually able to begin processing event handlers in response to that interaction. INP, on the other hand, broadens the scope by measuring the responsiveness of a website throughout the entire lifespan of a page, not just first interaction.
Note that actions such as hovering and scrolling do not influence INP, however, keyboard-driven scrolling or navigational actions are considered keystrokes that may activate events measured by INP but not scrolling which is happeing due to interaction.
Scrolling may indirectly affect INP, for example in scenarios where users scroll through content, and additional content is lazy-loaded from the API. While the act of scrolling itself isn’t included in the INP calculation, the processing, necessary for loading additional content, can create contention on the main thread, thereby increasing interaction latency and adversely affecting the INP score.
What qualifies as an optimal INP score?
- An INP under 200ms indicates good responsiveness.
- Between 200ms and 500ms needs improvement.
- Over 500ms means page has poor responsiveness.
and these are common issues causing poor INP scores:
- Long JavaScript Tasks: Heavy JavaScript execution can block the main thread, delaying the browser’s ability to respond to user interactions. Thus break long JS tasks into smaller chunks by using scheduler API.
- Large DOM (HTML) Size: A large DOM ( starting from 1500 elements) can severely impact a website’s interactive performance. Every additional DOM element increases the work required to render pages and respond to user interactions.
- Inefficient Event Callbacks: Event handlers that execute lengthy or complex operations can significantly affect INP scores. Poorly optimized callbacks attached to user interactions, like clicks, keypress or taps, can block the main thread, delaying the browser’s ability to render visual feedback promptly. For example when handlers perform heavy computations or initiate synchronous network requests such on clicks.
and you can troubleshoot INP issues using free and paid tools.
As a good starting point I would recommend to check your INP scores by geos via treo.sh which will give you a great high level insights where you struggle with most.
INP scores by GeosRead more: How To Improve Interaction To Next Paint (INP)
Image Optimization
Images add a lot of value to the content on your site and can greatly affect the user experience.
From page speeds to image quality, you could adversely affect the user experience if you haven’t optimized your images.
This is especially true for the mobile experience. Images need to adjust to smaller screens, varying resolutions, and screen orientation.
- Use responsive images
- Implement lazy loading
- Compress your images (use WebP)
- Add your images into sitemap
Optimizing images is an entire science, and I advise you to read our comprehensive guide on image SEO how to implement the mentioned recommendations.
Avoid Intrusive Interstitials
Google rarely uses concrete language to state that something is a ranking factor or will result in a penalty, so you know it means business about intrusive interstitials in the mobile experience.
Intrusive interstitials are basically pop-ups on a page that prevent the user from seeing content on the page.
John Mueller, Google’s Senior Search Analyst, stated that they are specifically interested in the first interaction a user has after clicking on a search result.
Not all pop-ups are considered bad. Interstitial types that are considered “intrusive” by Google include:
- Pop-ups that cover most or all of the page content.
- Non-responsive interstitials or pop-ups that are impossible for mobile users to close.
- Pop-ups that are not triggered by a user action, such as a scroll or a click.
Read more: 7 Tips To Keep Pop-Ups From Harming Your SEO
Structured Data
Most of the tips provided in this guide so far are focused on usability and speed and have an additive effect, but there are changes that can directly influence how your site appears in mobile search results.
Search engine results pages (SERPs) haven’t been the “10 blue links” in a very long time.
They now reflect the diversity of search intent, showing a variety of different sections to meet the needs of users. Local Pack, shopping listing ads, video content, and more dominate the mobile search experience.
As a result, it’s more important than ever to provide structured data markup to the search engines, so they can display rich results for users.
In this example, you can see that both Zojirushi and Amazon have included structured data for their rice cookers, and Google is displaying rich results for both.
Screenshot from search for [Japanese rice cookers], Google, March 2024Adding structured data markup to your site can influence how well your site shows up for local searches and product-related searches.
Using JSON-LD, you can mark up the business, product, and services data on your pages in Schema markup.
If you use WordPress as the content management system for your site, there are several plugins available that will automatically mark up your content with structured data.
Read more: What Structured Data To Use And Where To Use It?
Content Style
When you think about your mobile users and the screens on their devices, this can greatly influence how you write your content.
Rather than long, detailed paragraphs, mobile users prefer concise writing styles for mobile reading.
Each key point in your content should be a single line of text that easily fits on a mobile screen.
Your font sizes should adjust to the screen’s resolution to avoid eye strain for your users.
If possible, allow for a dark or dim mode for your site to further reduce eye strain.
Headers should be concise and address the searcher’s intent. Rather than lengthy section headers, keep it simple.
Finally, make sure that your text renders in a font size that’s readable.
Read more: 10 Tips For Creating Mobile-Friendly Content
Tap Targets
As important as text size, the tap targets on your pages should be sized and laid out appropriately.
Tap targets include navigation elements, links, form fields, and buttons like “Add to Cart” buttons.
Targets smaller than 48 pixels by 48 pixels and targets that overlap or are overlapped by other page elements will be called out in the Lighthouse report.
Tap targets are essential to the mobile user experience, especially for ecommerce websites, so optimizing them is vital to the health of your online business.
Read more: Google’s Lighthouse SEO Audit Tool Now Measures Tap Target Spacing
Prioritizing These Tips
If you have delayed making your site mobile-friendly until now, this guide may feel overwhelming. As a result, you may not know what to prioritize first.
As with so many other optimizations in SEO, it’s important to understand which changes will have the greatest impact, and this is just as true for mobile SEO.
Think of SEO as a framework in which your site’s technical aspects are the foundation of your content. Without a solid foundation, even the best content may struggle to rank.
- Responsive or Dynamic Rendering: If your site requires the user to zoom and scroll right or left to read the content on your pages, no number of other optimizations can help you. This should be first on your list.
- Content Style: Rethink how your users will consume your content online. Avoid very long paragraphs. “Brevity is the soul of wit,” to quote Shakespeare.
- Image Optimization: Begin migrating your images to next-gen image formats and optimize your content display network for speed and responsiveness.
- Tap Targets: A site that prevents users from navigating or converting into sales won’t be in business long. Make navigation, links, and buttons usable for them.
- Structured Data: While this element ranks last in priority on this list, rich results can improve your chances of receiving traffic from a search engine, so add this to your to-do list once you’ve completed the other optimizations.
Summary
From How Search Works, “Google’s mission is to organize the world’s information and make it universally accessible and useful.”
If Google’s primary mission is focused on making all the world’s information accessible and useful, then you know they will prefer surfacing sites that align with that vision.
Since a growing percentage of users are on mobile devices, you may want to infer the word “everywhere” added to the end of the mission statement.
Are you missing out on traffic from mobile devices because of a poor mobile experience?
If you hope to remain relevant, make mobile SEO a priority now.
Featured Image: Paulo Bobita/Search Engine Journal
SEO
HARO Has Been Dead for a While
I know nothing about the new tool. I haven’t tried it. But after trying to use HARO recently, I can’t say I’m surprised or saddened by its death. It’s been a walking corpse for a while.
I used HARO way back in the day to build links. It worked. But a couple of months ago, I experienced the platform from the other side when I decided to try to source some “expert” insights for our posts.
After just a few minutes of work, I got hundreds of pitches:
So, I grabbed a cup of coffee and began to work through them. It didn’t take long before I lost the will to live. Every other pitch seemed like nothing more than lazy AI-generated nonsense from someone who definitely wasn’t an expert.
Here’s one of them:
Seriously. Who writes like that? I’m a self-confessed dullard (any fellow Dull Men’s Club members here?), and even I’m not that dull…
I don’t think I looked through more than 30-40 of the responses. I just couldn’t bring myself to do it. It felt like having a conversation with ChatGPT… and not a very good one!
Despite only reviewing a few dozen of the many pitches I received, one stood out to me:
Believe it or not, this response came from a past client of mine who runs an SEO agency in the UK. Given how knowledgeable and experienced he is (he actually taught me a lot about SEO back in the day when I used to hassle him with questions on Skype), this pitch rang alarm bells for two reasons:
- I truly doubt he spends his time replying to HARO queries
- I know for a fact he’s no fan of Neil Patel (sorry, Neil, but I’m sure you’re aware of your reputation at this point!)
So… I decided to confront him 😉
Here’s what he said:
Shocker.
I pressed him for more details:
I’m getting a really good deal and paying per link rather than the typical £xxxx per month for X number of pitches. […] The responses as you’ve seen are not ideal but that’s a risk I’m prepared to take as realistically I dont have the time to do it myself. He’s not native english, but I have had to have a word with him a few times about clearly using AI. On the low cost ones I don’t care but on authority sites it needs to be more refined.
I think this pretty much sums up the state of HARO before its death. Most “pitches” were just AI answers from SEOs trying to build links for their clients.
Don’t get me wrong. I’m not throwing shade here. I know that good links are hard to come by, so you have to do what works. And the reality is that HARO did work. Just look at the example below. You can tell from the anchor and surrounding text in Ahrefs that these links were almost certainly built with HARO:
But this was the problem. HARO worked so well back in the day that it was only a matter of time before spammers and the #scale crew ruined it for everyone. That’s what happened, and now HARO is no more. So…
If you’re a link builder, I think it’s time to admit that HARO link building is dead and move on.
No tactic works well forever. It’s the law of sh**ty clickthroughs. This is why you don’t see SEOs having huge success with tactics like broken link building anymore. They’ve moved on to more innovative tactics or, dare I say it, are just buying links.
Sidenote.
Talking of buying links, here’s something to ponder: if Connectively charges for pitches, are links built through those pitches technically paid? If so, do they violate Google’s spam policies? It’s a murky old world this SEO lark, eh?
If you’re a journalist, Connectively might be worth a shot. But with experts being charged for pitches, you probably won’t get as many responses. That might be a good thing. You might get less spam. Or you might just get spammed by SEOs with deep pockets. The jury’s out for now.
My advice? Look for alternative methods like finding and reaching out to experts directly. You can easily use tools like Content Explorer to find folks who’ve written lots of content about the topic and are likely to be experts.
For example, if you look for content with “backlinks” in the title and go to the Authors tab, you might see a familiar name. 😉
I don’t know if I’d call myself an expert, but I’d be happy to give you a quote if you reached out on social media or emailed me (here’s how to find my email address).
Alternatively, you can bait your audience into giving you their insights on social media. I did this recently with a poll on X and included many of the responses in my guide to toxic backlinks.
Either of these options is quicker than using HARO because you don’t have to sift through hundreds of responses looking for a needle in a haystack. If you disagree with me and still love HARO, feel free to tell me why on X 😉
Roundel Media Studio: What to Expect From Target’s New Self-Service Platform
Google Limits News Links In California Over Proposed ‘Link Tax’ Law
Daily Search Forum Recap: April 12, 2024
10 Paid Search & PPC Planning Best Practices
Google Core Update Volatility, Helpful Content Update Gone, Dangerous Google Search Results & Google Ads Confusion
Google Unplugs “Notes on Search” Experiment
2 Ways to Take Back the Power in Your Business: Part 2
Critical Display Error in Brand Safety Metrics On Twitter/X Corrected
WordPress Releases A Performance Plugin For “Near-Instant Load Times”
Weekend Google Core Ranking Volatility
-
MARKETING6 days agoRoundel Media Studio: What to Expect From Target’s New Self-Service Platform
-
SEO6 days agoGoogle Limits News Links In California Over Proposed ‘Link Tax’ Law
-
SEARCHENGINES6 days ago
Daily Search Forum Recap: April 12, 2024
-
SEO5 days ago10 Paid Search & PPC Planning Best Practices
-
SEARCHENGINES5 days agoGoogle Core Update Volatility, Helpful Content Update Gone, Dangerous Google Search Results & Google Ads Confusion
-
SEO7 days agoGoogle Unplugs “Notes on Search” Experiment
-
MARKETING6 days ago2 Ways to Take Back the Power in Your Business: Part 2
-
PPC6 days agoCritical Display Error in Brand Safety Metrics On Twitter/X Corrected
You must be logged in to post a comment Login