Connect with us


Bulk Loading Performance Tests With PageSpeed Insights API & Python



Bulk Loading Performance Tests With PageSpeed Insights API & Python

Google offers PageSpeed Insights API to help SEO pros and developers by mixing real-world data with simulation data,  providing load performance timing data related to web pages.

The difference between the Google PageSpeed Insights (PSI) and Lighthouse is that PSI involves both real-world and lab data, while Lighthouse performs a page loading simulation by modifying the connection and user-agent of the device.

Another point of difference is that PSI doesn’t supply any information related to web accessibility, SEO, or progressive web apps (PWAs), while Lighthouse provides all of the above.

Thus, when we use PageSpeed Insights API for the bulk URL loading performance test, we won’t have any data for accessibility.

However, PSI provides more information related to the page speed performance, such as “DOM Size,” “Deepest DOM Child Element,” “Total Task Count,” and “DOM Content Loaded” timing.

One more advantage of the PageSpeed Insights API is that it gives the “observed metrics” and “actual metrics” different names.

In this guide, you will learn:

  • How to create a production-level Python Script.
  • How to use APIs with Python.
  • How to construct data frames from API responses.
  • How to analyze the API responses.
  • How to parse URLs and process URL requests’ responses.
  • How to store the API responses with proper structure.

An example output of the Page Speed Insights API call with Python is below.

Screenshot from author, June 2022

Libraries For Using PageSpeed Insights API With Python

The necessary libraries to use PSI API with Python are below.

  • Advertools retrieves testing URLs from the sitemap of a website.
  • Pandas is to construct the data frame and flatten the JSON output of the API.
  • Requests are to make a request to the specific API endpoint.
  • JSON is to take the API response and put it into the specifically related dictionary point.
  • Datetime is to modify the specific output file’s name with the date of the moment.
  • URLlib is to parse the test subject website URL.

How To Use PSI API With Python?

To use the PSI API with Python, follow the steps below.

  • Get a PageSpeed Insights API key.
  • Import the necessary libraries.
  • Parse the URL for the test subject website.
  • Take the Date of Moment for file name.
  • Take URLs into a list from a sitemap.
  • Choose the metrics that you want from PSI API.
  • Create a For Loop for taking the API Response for all URLs.
  • Construct the data frame with chosen PSI API metrics.
  • Output the results in the form of XLSX.

1. Get PageSpeed Insights API Key

Use the PageSpeed Insights API Documentation to get the API Key.

Click the “Get a Key” button below.

psi api key Image from, June 2022

Choose a project that you have created in Google Developer Console.

google developer console api projectImage from, June 2022

Enable the PageSpeed Insights API on that specific project.

page speed insights api enableImage from, June 2022

You will need to use the specific API Key in your API Requests.

2. Import The Necessary Libraries

Use the lines below to import the fundamental libraries.

    import advertools as adv
    import pandas as pd
    import requests
    import json
    from datetime import datetime
    from urllib.parse import urlparse

3. Parse The URL For The Test Subject Website

To parse the URL of the subject website, use the code structure below.

  domain = urlparse(sitemap_url)
  domain = domain.netloc.split(".")[1]

The “domain” variable is the parsed version of the sitemap URL.

The “netloc” represents the specific URL’s domain section. When we split it with the “.” it takes the “middle section” which represents the domain name.

Here, “0” is for “www,” “1” for “domain name,” and “2” is for “domain extension,” if we split it with “.”

4. Take The Date Of Moment For File Name

To take the date of the specific function call moment, use the “” method. provides the specific time of the specific moment. Use the “strftime” with the “%Y”, “”%m”, and “%d” values. “%Y” is for the year. The “%m” and “%d” are numeric values for the specific month and the day.

 date ="%Y_%m_%d")

5. Take URLs Into A List From A Sitemap

To take the URLs into a list form from a sitemap file, use the code block below.

   sitemap = adv.sitemap_to_df(sitemap_url)
   sitemap_urls = sitemap["loc"].to_list()

If you read the Python Sitemap Health Audit, you can learn further information about the sitemaps.

6. Choose The Metrics That You Want From PSI API

To choose the PSI API response JSON properties, you should see the JSON file itself.

It is highly relevant to the reading, parsing, and flattening of JSON objects.

It is even related to Semantic SEO, thanks to the concept of “directed graph,” and “JSON-LD” structured data.

In this article, we won’t focus on examining the specific PSI API Response’s JSON hierarchies.

You can see the metrics that I have chosen to gather from PSI API. It is richer than the basic default output of PSI API, which only gives the Core Web Vitals Metrics, or Speed Index-Interaction to Next Paint, Time to First Byte, and First Contentful Paint.

Of course, it also gives “suggestions” by saying “Avoid Chaining Critical Requests,” but there is no need to put a sentence into a data frame.


In the future, these suggestions, or even every individual chain event, their KB and MS values can be taken into a single column with the name “psi_suggestions.”

For a start, you can check the metrics that I have chosen, and an important amount of them will be first for you.

PSI API Metrics, the first section is below.

    fid = []
    lcp = []
    cls_ = []
    url = []
    fcp = []
    performance_score = []
    total_tasks = []
    total_tasks_time = []
    long_tasks = []
    dom_size = []
    maximum_dom_depth = []
    maximum_child_element = []
    observed_fcp  = []
    observed_fid = []
    observed_lcp = []
    observed_cls = []
    observed_fp = []
    observed_fmp = []
    observed_dom_content_loaded = []
    observed_speed_index = []
    observed_total_blocking_time = []
    observed_first_visual_change = []
    observed_last_visual_change = []
    observed_tti = []
    observed_max_potential_fid = []

This section includes all the observed and simulated fundamental page speed metrics, along with some non-fundamental ones, like “DOM Content Loaded,” or “First Meaningful Paint.”

The second section of PSI Metrics focuses on possible byte and time savings from the unused code amount.

    render_blocking_resources_ms_save = []
    unused_javascript_ms_save = []
    unused_javascript_byte_save = []
    unused_css_rules_ms_save = []
    unused_css_rules_bytes_save = []

A third section of the PSI metrics focuses on server response time, responsive image usage benefits, or not, using harms.

    possible_server_response_time_saving = []
    possible_responsive_image_ms_save = []

Note: Overall Performance Score comes from “performance_score.”

7. Create A For Loop For Taking The API Response For All URLs

The for loop is to take all of the URLs from the sitemap file and use the PSI API for all of them one by one. The for loop for PSI API automation has several sections.


The first section of the PSI API for loop starts with duplicate URL prevention.

In the sitemaps, you can see a URL that appears multiple times. This section prevents it.

for i in sitemap_urls[:9]:
         # Prevent the duplicate "/" trailing slash URL requests to override the information.
         if i.endswith("/"):
               r = requests.get(f"{i}&strategy=mobile&locale=en&key={api_key}")
               r = requests.get(f"{i}/&strategy=mobile&locale=en&key={api_key}")

Remember to check the “api_key” at the end of the endpoint for PageSpeed Insights API.

Check the status code. In the sitemaps, there might be non-200 status code URLs; these should be cleaned.

         if r.status_code == 200:
               data_ = json.loads(r.text)

The next section appends the specific metrics to the specific dictionary that we have created before “_data.”

               performance_score.append(data_["lighthouseResult"]["categories"]["performance"]["score"] * 100)

Next section focuses on “total task” count, and DOM Size.


The next section takes the “DOM Depth” and “Deepest DOM Element.”


The next section takes the specific observed test results during our Page Speed Insights API.


The next section takes the Unused Code amount and the wasted bytes, in milliseconds along with the render-blocking resources.


The next section is to provide responsive image benefits and server response timing.


The next section is to make the function continue to work in case there is an error.


Example Usage Of Page Speed Insights API With Python For Bulk Testing

To use the specific code blocks, put them into a Python function.

Run the script, and you will get 29 page speed-related metrics in the columns below.

pagespeed insights apiScreenshot from author, June 2022


PageSpeed Insights API provides different types of page loading performance metrics.

It demonstrates how Google engineers perceive the concept of page loading performance, and possibly use these metrics as a ranking, UX, and quality-understanding point of view.

Using Python for bulk page speed tests gives you a snapshot of the entire website to help analyze the possible user experience, crawl efficiency, conversion rate, and ranking improvements.

More resources:


Featured Image: Dundanim/Shutterstock

Source link


Decoded Headless CMS & SEO



Decoded Headless CMS & SEO

What is headless CMS? Why are more companies looking into it nowadays?

Powered by greater flexibility and scalability, it might be the next generation of content management systems.

Lidia Infante, Senior SEO Manager at, joined me on the SEJ Show to explain how headless CMS works and how it can be particularly beneficial for ecommerce businesses.

I don’t have to build every page. I create the content in a core database, a central source of truth. And then, I deploy it following a template.–Lidia Infante, [16:00]

I think there’s a little bit of a misconception about headless. A lot of people out there, the conversations I’m having with people that are not tech SEOs is they feel that just utilizing a headless means that it’s automatically going to be faster, quicker, and a better experience. –Loren Baker, [25:15]

For most ecommerce sites, I would recommend going with a headless solution, especially if you’re on a direct-to-consumer side with a big brand.–Lidia Infante, [45:09]

[00:00] – Lidia’s background.
[10:36] – What is headless CMS & how is it helpful?
[17:20] – Do you build from scratch in a headless environment?
[22:00] – Are headless CMS quicker?
[26:00] – SEO optimizations in headless.
[31:30] – Other benefits of using a headless CMS system.
[36:00] – How well does AI content rank in it?
[39:52] – Where does Lidia see headless going in the next four to five years?
[44:45] – What kind of business should look into headless CMS?
[49:59] – How important is JavaScript?


Resources mentioned: – Free boosted Sanity plan for SEJ readers

Regarding SEO and other stuff you can do, I think the main pain point that Headless solves is content velocity.–Lidia Infante, [28:00]

Headless CMS can guide you a little bit more when it comes to optimizing web performance. Different headless CMSs will help you achieve amazing core web vitals differently.–Lidia Infante, [26:19]

B2C brands need to ensure that the experience they provide on their website is not the typical ecommerce experience of title, product, picture, price, and description. It’s more similar to the experience they can provide in a shop. –Lidia Infante

For more content like this, subscribe to our YouTube channel:

Connect with Lidia Infante:

Lidia Infante is a consultant, writer, and speaker who has been working to help businesses grow their reach through SEO in European markets, the US, and Australia.

She got her start on the business side of things through psychology, allowing for organic growth strategies that are now driving ecommerce sites’ success!


As part of her passion for SEO, she enjoys regularly participating in podcasts and webinars in the SEO community. She writes about international strategy, digital business, and women’s rights on her website.

Connect with Lidia on LinkedIn:
Follow her on Twitter:
Visit her website:

Connect with Loren Baker, Founder of Search Engine Journal:

Follow him on Twitter:
Connect with him on LinkedIn:

!function(f,b,e,v,n,t,s) {if(f.fbq)return;n=f.fbq=function(){n.callMethod? n.callMethod.apply(n,arguments):n.queue.push(arguments)}; if(!f._fbq)f._fbq=n;n.push=n;n.loaded=!0;n.version='2.0'; n.queue=[];t=b.createElement(e);t.async=!0; t.src=v;s=b.getElementsByTagName(e)[0]; s.parentNode.insertBefore(t,s)}(window,document,'script', '');

if( typeof sopp !== "undefined" && sopp === 'yes' ){ fbq('dataProcessingOptions', ['LDU'], 1, 1000); }else{ fbq('dataProcessingOptions', []); }

fbq('init', '1321385257908563');

fbq('init', '164237177383067'); // custom pixel

fbq('track', 'PageView');

fbq('trackSingle', '1321385257908563', 'ViewContent', { content_name: 'headless-cms-seo', content_category: 'cms search-engine-journal-show wp' }); } });

Source link

Continue Reading

Subscribe To our Newsletter
We promise not to spam you. Unsubscribe at any time.
Invalid email address