Chances are, you’ve used one of the more popular tools such as Ahrefs or Semrush to analyze your site’s backlinks.
These tools trawl the web to get a list of sites linking to your website with a domain rating and other data describing the quality of your backlinks.
It’s no secret that backlinks play a big part in Google’s algorithm, so it makes sense as a minimum to understand your own site before comparing it with the competition.
While using tools gives you insight into specific metrics, learning to analyze backlinks on your own gives you more flexibility into what it is you’re measuring and how it’s presented.
And although you could do most of the analysis on a spreadsheet, Python has certain advantages.
Other than the sheer number of rows it can handle, it can also more readily look at the statistical side, such as distributions.
In this column, you’ll find step-by-step instructions on how to visualize basic backlink analysis and customize your reports by considering different link attributes using Python.
Not Taking A Seat
We’re going to pick a small website from the U.K. furniture sector as an example and walk through some basic analysis using Python.
So what is the value of a site’s backlinks for SEO?
At its simplest, I’d say quality and quantity.
Quality is subjective to the expert yet definitive to Google by way of metrics such as authority and content relevance.
We’ll start by evaluating the link quality with the available data before evaluating the quantity.
Time to code.
import re import time import random import pandas as pd import numpy as np import datetime from datetime import timedelta from plotnine import * import matplotlib.pyplot as plt from pandas.api.types import is_string_dtype from pandas.api.types import is_numeric_dtype import uritools pd.set_option('display.max_colwidth', None) %matplotlib inline root_domain = 'johnsankey.co.uk' hostdomain = 'www.johnsankey.co.uk' hostname="johnsankey" full_domain = 'https://www.johnsankey.co.uk' target_name="John Sankey"
We start by importing the data and cleaning up the column names to make it easier to handle and quicker to type for the later stages.
target_ahrefs_raw = pd.read_csv( 'data/johnsankey.co.uk-refdomains-subdomains__2022-03-18_15-15-47.csv')
List comprehensions are a powerful and less intensive way to clean up the column names.
target_ahrefs_raw.columns = [col.lower() for col in target_ahrefs_raw.columns]
The list comprehension instructs Python to convert the column name to lower case for each column (‘col’) in the dataframe’s columns.
target_ahrefs_raw.columns = [col.replace(' ','_') for col in target_ahrefs_raw.columns] target_ahrefs_raw.columns = [col.replace('.','_') for col in target_ahrefs_raw.columns] target_ahrefs_raw.columns = [col.replace('__','_') for col in target_ahrefs_raw.columns] target_ahrefs_raw.columns = [col.replace('(','') for col in target_ahrefs_raw.columns] target_ahrefs_raw.columns = [col.replace(')','') for col in target_ahrefs_raw.columns] target_ahrefs_raw.columns = [col.replace('%','') for col in target_ahrefs_raw.columns]
Though not strictly necessary, I like having a count column as standard for aggregations and a single value column “project” should I need to group the entire table.
target_ahrefs_raw['rd_count'] = 1 target_ahrefs_raw['project'] = target_name target_ahrefs_raw
Now we have a dataframe with clean column names.
The next step is to clean the actual table values and make them more useful for analysis.
Make a copy of the previous dataframe and give it a new name.
target_ahrefs_clean_dtypes = target_ahrefs_raw
Clean the dofollow_ref_domains column, which tells us how many ref domains the site linking has.
In this case, we’ll convert the dashes to zeroes and then cast the whole column as a whole number.
# referring_domains target_ahrefs_clean_dtypes['dofollow_ref_domains'] = np.where(target_ahrefs_clean_dtypes['dofollow_ref_domains'] == '-', 0, target_ahrefs_clean_dtypes['dofollow_ref_domains']) target_ahrefs_clean_dtypes['dofollow_ref_domains'] = target_ahrefs_clean_dtypes['dofollow_ref_domains'].astype(int) # linked_domains target_ahrefs_clean_dtypes['dofollow_linked_domains'] = np.where(target_ahrefs_clean_dtypes['dofollow_linked_domains'] == '-', 0, target_ahrefs_clean_dtypes['dofollow_linked_domains']) target_ahrefs_clean_dtypes['dofollow_linked_domains'] = target_ahrefs_clean_dtypes['dofollow_linked_domains'].astype(int)
First_seen tells us the date the link was first found.
We’ll convert the string to a date format that Python can process and then use this to derive the age of the links later on.
# first_seen target_ahrefs_clean_dtypes['first_seen'] = pd.to_datetime(target_ahrefs_clean_dtypes['first_seen'], format="%d/%m/%Y %H:%M")
Converting first_seen to a date also means we can perform time aggregations by month and year.
This is useful as it’s not always the case that links for a site will get acquired daily, although it would be nice for my own site if it did!
target_ahrefs_clean_dtypes['month_year'] = target_ahrefs_clean_dtypes['first_seen'].dt.to_period('M')
The link age is calculated by taking today’s date and subtracting the first_seen date.
Then it’s converted to a number format and divided by a huge number to get the number of days.
# link age target_ahrefs_clean_dtypes['link_age'] = datetime.datetime.now() - target_ahrefs_clean_dtypes['first_seen'] target_ahrefs_clean_dtypes['link_age'] = target_ahrefs_clean_dtypes['link_age'] target_ahrefs_clean_dtypes['link_age'] = target_ahrefs_clean_dtypes['link_age'].astype(int) target_ahrefs_clean_dtypes['link_age'] = (target_ahrefs_clean_dtypes['link_age']/(3600 * 24 * 1000000000)).round(0) target_ahrefs_clean_dtypes
With the data types cleaned, and some new data features created, the fun can begin!
The first part of our analysis evaluates link quality, which summarizes the whole dataframe using the describe function to get descriptive statistics of all the columns.
target_ahrefs_analysis = target_ahrefs_clean_dtypes target_ahrefs_analysis.describe()
So from the above table, we can see the average (mean), the number of referring domains (107), and the variation (the 25th percentile and so on).
The average Domain Rating (equivalent to Moz’s Domain Authority) of referring domains is 27.
Is that a good thing?
In the absence of competitor data to compare in this market sector, it’s hard to know. This is where your experience as an SEO practitioner comes in.
However, I’m certain we could all agree that it could be higher.
How much higher to make a shift is another question.
The table above can be a bit dry and hard to visualize, so we’ll plot a histogram to get an intuitive understanding of the referring domain’s authority.
dr_dist_plt = ( ggplot(target_ahrefs_analysis, aes(x = 'dr')) + geom_histogram(alpha = 0.6, fill="blue", bins = 100) + scale_y_continuous() + theme(legend_position = 'right')) dr_dist_plt
The distribution is heavily skewed, showing that most of the referring domains have an authority rating of zero.
Beyond zero, the distribution looks fairly uniform, with an equal amount of domains across different levels of authority.
Let’s check out the distribution below.
linkage_dist_plt = ( ggplot(target_ahrefs_analysis, aes(x = 'link_age')) + geom_histogram(alpha = 0.6, fill="blue", bins = 100) + scale_y_continuous() + theme(legend_position = 'right')) linkage_dist_plt
The distribution looks more normal even if it is still skewed with the majority of the links being new.
The most common link age appears to be around 200 days, which is less than a year, suggesting most of the links were acquired recently.
Out of interest, let’s see how this correlates with domain authority.
dr_linkage_plt = ( ggplot(target_ahrefs_analysis, aes(x = 'dr', y = 'link_age')) + geom_point(alpha = 0.4, colour="blue", size = 2) + geom_smooth(method = 'lm', se = False, colour="red", size = 3, alpha = 0.4) ) print(target_ahrefs_analysis['dr'].corr(target_ahrefs_analysis['link_age'])) dr_linkage_plt 0.1941101232345909
The plot (along with the 0.19 figure printed above) shows no correlation between the two.
And why should there be?
A correlation would only imply that the higher authority links were acquired in the early phase of the site’s history.
The reason for the non-correlation will become more apparent later on.
We’ll now look at the link quality throughout time.
If we were to literally plot the number of links by date, the time series would look rather messy and less useful as shown below (no code supplied to render the chart).
To achieve this, we will calculate a running average of the Domain Rating by month of the year.
Note the expanding( ) function, which instructs Pandas to include all previous rows with each new row.
target_rd_cummean_df = target_ahrefs_analysis target_rd_mean_df = target_rd_cummean_df.groupby(['month_year'])['dr'].sum().reset_index() target_rd_mean_df['dr_runavg'] = target_rd_mean_df['dr'].expanding().mean() target_rd_mean_df
We now have a table that we can use to feed the graph and visualize it.
dr_cummean_smooth_plt = ( ggplot(target_rd_mean_df, aes(x = 'month_year', y = 'dr_runavg', group = 1)) + geom_line(alpha = 0.6, colour="blue", size = 2) + scale_y_continuous() + scale_x_date() + theme(legend_position = 'right', axis_text_x=element_text(rotation=90, hjust=1) ))
This is quite interesting as it seems the site started off attracting high authority links at the beginning of its time (probably a PR campaign launching the business).
It then faded for four years before reprising with a new link acquisition of high authority links again.
Volume Of Links
It sounds good just writing that heading!
Who wouldn’t want a large volume of (good) links to their site?
Quality is one thing; volume is another, which is what we’ll analyze next.
Much like the previous operation, we’ll use the expanding function to calculate a cumulative sum of the links acquired to date.
target_count_cumsum_df = target_ahrefs_analysis target_count_cumsum_df = target_count_cumsum_df.groupby(['month_year'])['rd_count'].sum().reset_index() target_count_cumsum_df['count_runsum'] = target_count_cumsum_df['rd_count'].expanding().sum() target_count_cumsum_df
That’s the data, now the graph.
target_count_cumsum_plt = ( ggplot(target_count_cumsum_df, aes(x = 'month_year', y = 'count_runsum', group = 1)) + geom_line(alpha = 0.6, colour="blue", size = 2) + scale_y_continuous() + scale_x_date() + theme(legend_position = 'right', axis_text_x=element_text(rotation=90, hjust=1) )) target_count_cumsum_plt
We see that links acquired at the beginning of 2017 slowed down but steadily added over the next four years before accelerating again around March 2021.
Again, it would be good to correlate that with performance.
Taking It Further
Of course, the above is just the tip of the iceberg, as it’s a simple exploration of one site. It’s difficult to infer anything useful for improving rankings in competitive search spaces.
Below are some areas for further data exploration and analysis.
- Adding social media share data to both the destination URLs.
- Correlating overall site visibility with the running average DR over time.
- Plotting the distribution of DR over time.
- Adding search volume data on the host names to see how many brand searches the referring domains receive as a measure of true authority.
- Joining with crawl data to the destination URLs to test for content relevance.
- Link velocity – the rate at which new links from new sites are acquired.
- Integrating all of the above ideas into your analysis to compare to your competitors.
I’m certain there are plenty of ideas not listed above, feel free to share below.
Featured Image: metamorworks/Shutterstock
Upgrade Your SEO Content Strategy With These 3 Steps [Webinar]
With Google constantly releasing algorithm updates, how do you keep up?
Gone are the days when businesses could rank on Google simply by pumping out generic, keyword-stuffed material, which wasn’t necessarily user-friendly.
Now, you can stay ahead of the curve with a forward-thinking approach to search marketing.
In today’s ever-evolving digital landscape, it’s crucial that your SEO tactics stay up-to-date. And, your content should be one of the first places you assess.
A refreshed content strategy can increase organic traffic and drive more conversions.
The key to consistently high rankings is creating high-quality content that offers maximum value to searchers.
It’s all about matching intent to keep users on your site longer and take action.
Join me and Carlos Meza, President and CEO at Crowd Content, as we talk about ways you can upgrade your SEO content strategy in just three steps. This webinar will detail how you can execute a winning content strategy that delivers the best ROI for your efforts.
- Drop Outdated Tactics: Learn about and let go of content strategies that used to work but don’t anymore due to Google algorithm updates.
- Think Ahead: Discover how to future-proof your content strategy to stay ahead of market changes.
- Level Up: Get tips to scale your content strategy, increase organic traffic, and improve search rankings.
In this live session, you’ll learn how to optimize your content for user intent and relevancy by using a meaningful content structure.
If you struggle with adapting to fast-changing SEO standards, this webinar will help keep your content strategies up-to-date.
Don’t miss out! Register now and discover how quality content can increase your online presence.
Can’t make the live session? That’s ok. Sign up for this sponsored webinar, and we’ll send you a recording after the event. Hope to see you there! Be sure to bring your questions for the live Q&A!
BCN Group strengthens Microsoft Cloud Services presence with Evo-Soft acquisition
Upgrade Your SEO Content Strategy With These 3 Steps [Webinar]
Here’s How Much You Can Really Make From Affiliate Marketing
WhatsApp Launches ‘Call Links’ to Better Facilitate Group Audio and Video Chats
Google Product Review Updates Still Get Periodic Updates
How We Increased a Client’s Leads by 384% in Six Months by Focusing on One Topic Cluster [Case Study]
Google Core & Product Review Updates Finish Rolling Out
Update for Capcom Fighting Collection with More Features Available Now
UK eyes big TikTok fine over child privacy lapse
Why B2B Companies are Entering the Editorial Space [& What You Can Learn From Them]
How to Create UTM Tracking URLs on Google Analytics
Google Is Not Yet Done Rolling Out The Helpful Content Update
How to Target Keywords With Blog Posts
Google On Why Helpful Content Update Seems Quiet
If You Love Escape Rooms, You’ll Love the Elaborate Puzzles of Zero Escape: Zero Time Dilemma
Why & How Machine Learning Took Over Paid Advertising
Google Updates Documentation On Meta Descriptions
The Ultimate SEO Checklist For Boosting Organic Traffic: 6 Highlights
Google Learning Video Structured Data Docs Breaks Out educationalLevel
How to limit your reliance on canonicals and boost crawl efficiency