Connect with us


6 Core Web Vitals Extraction Methods For CrUX With Pros & Cons



6 Core Web Vitals Extraction Methods For CrUX With Pros & Cons

Since the announcement of the Page Experience update and its full rollout last September 2021, many SEO professionals worldwide have turned their attention to improving Core Web Vitals for the websites they manage.

Making sure that you have a good user experience across all browsers and devices is important from a business standpoint.

However, as SEO experts, we need to understand not only how users experience our site, but how Google is measuring Core Web Vitals and whether there is a way to get access to this data, as well.

That way, we not only benefit our users but know how Google judges our websites – and our competitors’ – within this specific area. This ultimately enables us to prioritize fixes based on this information.

With that in mind, in this article we’re exploring:

  • What data Google uses to measure Core Web Vitals.
  • What sources are available to extract this data and their limitations.
  • Which are the best sources for SEO purposes (from my point of view).
  • How to access these data sources, with examples.


Continue Reading Below

What Data Is Google Using To Measure Core Web Vitals?

Based on the information Google has provided, they are using the data collected in the Chrome User Experience Report to measure Core Web Vitals for Search.

They have announced this on multiple occasions, including John Mueller’s “Core Web Vitals & SEO” session at the Chrome Dev Summit in 2020 and most recently during the Web Vitals AMA session at Google I/O 2021.

Core Web Vitals and SEO, Google Chrome Developers, December 2020

The Chrome User Experience Report, or CrUX for short, gathers loading performance information from URLs visited by real Chrome users that meet specific criteria.


Continue Reading Below

To put this in context, when looking at Core Web Vitals measurement purely from Google’s point of view, they are looking at a segmented subset of your whole user base.

Browsers Split ExampleImage created by author, December 2021

Obviously, we can’t know what percentage of Chrome users are part of the CrUX report for any given website, as this is not disclosed by Google. Also, how big or small this subset is will depend entirely on your users.

In an ideal world, you should track Core Web Vitals on your site for all users with a third-party tool or using Google’s own web vitals library. However, the data in CrUX is the best information we have that it’s publicly available.

What Sources Are Available To Extract Core Web Vitals From The CrUX Database?

Knowing that Google is using CrUX data for Search, the next step is understanding how to get your hands on this data.

There are six ways of extracting Core Web Vitals from CrUX directly from Google:

  • CrUX API.
  • PageSpeed Insights API.
  • CrUX Data Studio Dashboard.
  • PageSpeed Insights Tool.
  • CrUX BigQuery project.
  • Google Search Console.
extraction methods core web vitals cruxImage created by author, December 2021

Each of these sources has its benefits and drawbacks. I’ve created a small framework to classify which one is best for the type of analysis you would normally do for SEO.


Continue Reading Below

The metrics included in this framework are:

  • URL Extraction: Can we extract CWV data for a specific URL (if available)?
  • Domain/Origin: Can we extract CWV data for a specific domain (if available)?
  • Devices: Can we segment the data by Mobile, Desktop, or Tablet?
  • Network Connection: Can we segment the data by the users’ network speed?
  • Fresh data: Do we get the most recent available data (last 28-days from the day of extraction)?
  • Historic data: Can we access data from previous months/years?
  • Cost-free: Can we access the data without paying?
  • Scalability: Can we extract this data easily for 1000s of URLs or domains?
  • UI Access: Does this data source have an easy-to-use user interface?

Ranking Of The Best Sources To Extract CrUX Data For SEO

Although this list might be a bit biased because I like to use programmatic solutions for my day-to-day work, I have tried all these methods before.

Hence, all the information here is based on my experience working on solving and monitoring Core Web Vitals issues for real clients.

Here is the list of methods to extract Core Web Vitals from Google and how they compare against each other based on my comparison framework.

Core Web Vitals field data extraction Comparison TableImage created by author, December 2021

1. The CrUX API

The CrUX API is, in my opinion, the easiest and most complete API to extract Core Web Vitals from CrUX overall.


Continue Reading Below

It is fairly straightforward to use and it contains all the information you might need to understand, report and monitor CWV issues for your websites.

Chrome UX report API screenshotScreenshot by author, December 2021


  • Both URL and Origin-level data are accessible through the API when these are available.
  • You can segment all three devices (Mobile, Desktop, and Tablet).
  • Network connection information is available. You can extract data for 4G, 3G, 2G, slow-2G, and offline.
  • You can extract the freshest available data which is the average aggregated data from the previous 28-days from the last complete day. This is (in theory) what Google Search uses to assess Core Web Vitals for a website.
  • It is completely free to use and easily scalable. The only quota limit is on the number of queries per minute which is 150. Additionally, it has a really fast response time in comparison to other APIs like the PageSpeed Insights API.


  • At the moment, there is no available access to historic data. Hence, you can only access the aggregation of the previous 28-days. However, this can be circumvented by storing the data daily for future access.
  • There is no easily accessible user interface for the API for now.

How To Access CWV Data With The CrUX API

My weapon of choice when it comes to API extraction is JavaScript, specifically Node.js. Therefore, the examples I’ll show you will mostly be in JavaScript.

If you don’t know how to run it, this short post explains how to install Node.js on your laptop so you can try these examples in your own time.


Continue Reading Below

// Create an index.js file, paste the code below & run `npm install axios` in your terminal

/* Modules */
const axios = require('axios');

/* Script Variables */
const apiKey = 'YOUR-API-KEY-HERE' // Get your key here
const cruxEndpoint = `${apiKey}`;

// Custom function to call the CruX API
const getApiData = async (type, url) => {
  // Create request body
  const req = {}
  req[type] = url

  // Send API Request
  const { data } = await axios(cruxEndpoint, {
    method: 'POST',
    headers: {
      'Content-Type': `application/json`,
    data: JSON.stringify(req)
  return data

// Run script (IIFE) - Change 'type' & 'URL'
(async () => {
  const testOrigin = await getApiData('origin', '')
  const testURL = await getApiData('url', '')
  console.log(testOrigin, testURL);

If you would like me to share a fully-fetched version that loops through all possible devices & connections for a list of URLs, let me know on Twitter.

2. The PageSpeed Insights API

The PageSpeed Insights API is a close second when it comes to extracting field data from CrUX.

It gives us very useful information but there are a few missing dimensions compared to the CrUX API that could be helpful when diagnosing CWV issues for your sites.

PageSpeed Insights API screenshotScreenshot by author, December 2021



Continue Reading Below

  • Both URL and Origin-level data is accessible through the API when these are available.
  • You can segment the data by Mobile & Desktop.
  • Same as the CrUX API, you can extract the freshest available data which is the average aggregated data from the previous 28-days from the last complete day.
  • It is completely free to use and easily scalable. There is a quota limit of 240 requests per minute and 25,000 per day.
  • You can access this API through an easy-to-use user interface with the PageSpeed Insights Tool from Google (although it’s not that scalable).


  • You can’t segment the data by Tablet users.
  • No network connection information is available. Hence, all the different connections are aggregated when extracting CWV data.
  • At the moment, there is no available access to historic data. Hence, you can only access the aggregation of the previous 28-days. This can be solved by storing the data daily for future access.
  • This service runs Lighthouse in the background to get lab metrics in the same requests. Hence, the API response is a bit slower than the CrUX API.

How To Access CWV Data With The PageSpeed Insights API

Here is a small example of how you can extract CWV data from the PageSpeed Insights API. If you want a plug-and-play script to run you can download my repository from Github.

// Create an index.js file, paste the code below & run `npm install axios` in your terminal

/* Modules */
const axios = require('axios');

/* Script Variables */
const apiKey = 'YOUR-API-KEY-HERE' // Get your key here

// Custom function to extract data from PageSpeed API
const getApiData = async (url) => {
  const endpoint="";
  const apiResponse = await axios(`${endpoint}?url=${url}&key=${apiKey}`); // Create HTTP call
  const urlCWV =; // Extract URL field
  const domainCWV =; // Extract domain field data

  console.log(urlCWV, domainCWV); // Log URL field data and Domain Field data if available
  return { urlCWV, domainCWV };

// Call your custom function

3. The CrUX BigQuery Project

The CrUX BigQuery project is a huge database of real user metrics records that dates back to October 2017. This huge project is full of great information. But like any other source, it has pros and cons.


Continue Reading Below


  • You can access origin-level data.
  • You can segment all three devices (Mobile, Desktop, and Tablet).
  • You can extract data for all types of network connections when available (4G, 3G, 2G, slow-2G and offline).
  • You can extract historic data beyond the last available month up until October 2017.
  • You can scale this for as many projects as you want and the data is very flexible with the potential to create your own custom tables if you wish.
  • There are additional metrics and dimensions that could be useful for your analysis but are not available in the CrUX API or PageSpeed Insights APIs like “Time To First Byte” or country-level segmentation.


  • You cannot access URL-level data.
  • This dataset is updated every second Tuesday of the month for the previous month. Hence, if you want to monitor CWV more regularly this wouldn’t be the right source.
  • You need a working understanding of SQL to dig into the data.
  • It costs money to run. Although there is a free usage tier on BigQuery, you will need to add billing details within Google Cloud Platform in order to use it. Don’t get discouraged by this. For small to medium-scale reporting, you should be within the free tier.

How To Access CWV Data With The CrUX BigQuery Project

If you already have a Google Cloud Platform account, you can access the project using this link.

You’ll need to enable the BigQuery API. Once enabled you can access the data straight from your SQL editor.

SQL example Big Query CrUX ExtractionScreenshot by author, December 2021

You don’t have to be an expert on SQL but a bit of familiarity would take you a long way.


Continue Reading Below

Here are two fantastic resources that will help you kick-start your journey when analyzing this data: Rick Viscomi’s CrUX Cookbook and Paul Calvano’s Biguery for CrUX tutorial.

Until now, I’ve only explained data sources that require a bit of programming knowledge. But you don’t need to know how to code in order to get your hands on Core Web Vitals data from the CrUX report.

The next three methods will allow you to access Core Web Vitals data from CrUX without programming knowledge.

4. The PageSpeed Insights Tool

The PageSpeed Insights Tool from Google is a perfectly good “no-code” alternative to get access to CruX data.

PageSpeed Insights ToolScreenshot by author, December 2021

The benefits are exactly the same as the ones for the PageSpeed Insight API but the only drawback is that this method is not very scalable.


Continue Reading Below

In order to get data from multiple URLs, you will need to manually input each URL into the tool.

How To Access CWV Data With The PageSpeed Insights Tool

Input the URL/domain that you would like to get data from in the PageSpeed Insights Tool.

If there is available information for both the URL or the domain (origin), you will find it at the top of the results shown after the tool has finished running.

core web vitals PSI toolScreenshot by author, December 2021

5. The CrUX Data Studio Dashboard

The CrUX Data Studio Dashboard is a fantastic tool built by Rick Viscomi to access the CrUX Big Query project very easily with a nice user interface.

CrUX Data Studio DasboardScreenshot by author, December 2021



Continue Reading Below

  • You can access Origin-level data.
  • You can segment all three devices (Mobile, Desktop, and Tablet).
  • You can extract data for all types of network connections when available (4G, 3G, 2G, slow-2G and offline).
  • You can extract historic data beyond the last available month.
  • It is completely free to build.
  • It is very easy to set up and has a simple user interface.
  • There are additional metrics that could be useful for your analysis but are not available in the other APIs like Time To First Byte.


  • You cannot access URL-level data.
  • This report is tied to the data available in the Big Query project which is updated every second Tuesday of the month for the previous month. Hence, if you want to monitor CWV data more regularly it’s not possible through this data source.
  • This method is not really scalable if you are planning to monitor more than a few domains.

How To Access CWV Data With The CrUX Data Studio Dashboard

Create a copy of the template directly on Data Studio through

You can then add the domain you are interested in, hit “Create report” and you will get the report from the latest available month.

CrUX Data Studio Dashboard setupScreenshot by author, December 2021

If you receive an error, make sure you have added the domain name correctly.

Alternatively, it might be that your domain isn’t included in the BigQuery dataset. You can find more information about how this report works in Rick Viscomi’s post on


Continue Reading Below

6. Search Console’s Core Web Vitals Report

Search Console’s Core Web Vitals Report is a relatively new addition to the GSC platform. It is useful but quite unique in terms of the metrics it displays per property.

Search Console’s Core Web Vitals ReportScreenshot by author, December 2021


  • You can access data at template-level which is a unique approach to the rest of the methodologies. This is a really good idea and in many cases, the aggregation works as expected.
  • You can segment the data by Mobile & Desktop.
  • In principle, you can extract the freshest available data which is the average aggregated data from the previous 28-days from the last complete day.
  • There is 90-days worth of data but only by the number of affected URLs per group (good metric, needs improvement metric, poor metric)
  • The GSC user interface is very easy to use.


  • You cannot access URL-level data or origin-level data. The data is aggregated by “similar URLs” and “Aggregated metric” value which is good but it’s harder to track individual URL progress.
  • You cannot download the individual URLs matched as “similar”, only the total number.
  • You can’t segment the data by Tablet users.
  • No network connection information is available.
  • At the moment, there is no available access to historic data beyond the previous 90 days.

How To Access CWV Data With Search Console

The only way to extract the data is through the user interface for now. The report is divided between “Mobile” and “Desktop”.


Continue Reading Below

Each device cateogry contains individual “Poor”, “Needs Improvement” and “Good” reports for each of the Core Web Vital metrics (LCP, CLS, FID).

GSC CWV report example for LCPScreenshot by author, December 2021

Each report has an export function (CSV, Excel, or Google Sheet) that will allow you to download a table with the “base URL,” the number of “Similar URLs” and the “aggregated metric” value per group.

It also allows the number of URLs affected within that subsection for the last 90 days.


Continue Reading Below

Final Thoughts

Regardless of your coding skills, there are many ways to extract Core Web Vitals data from CrUX to monitor your websites and competitors.

If you are comfortable with a little bit of programming and looking to monitor Core Web Vitals on a regular basis at scale, the CrUX API will be your best solution.

Alternatively, if you are more focused on general domain trends and don’t need the data that often without needing to track lots of different domains, the CrUX Data Studio Dashboard would be the most comfortable solution.

Remember that measuring how our websites are performing against Google’s CWV benchmarks is the first step towards improving them. Because without a goal, you can’t score.

Keep in mind though, that as our Technical Director William Nye always tells us, “Strategy is important but execution is everything.”

More resources:


Continue Reading Below

Featured Image: FFFLOW/Shutterstock

Source link


Sustaining A SaaS Brand & Organic Channel During A Recession



Sustaining A SaaS Brand & Organic Channel During A Recession

During an economic recession, marketing budgets and ROAS typically comes under much more scrutiny.

You should read this article for reasons you should not cut your SEO spending during a recession.

The next question will be about ROI and what you can do to mitigate the oncoming issues.

During an economic downturn, the objectives of reducing churn are amplified. Your sales pipelines may see less activity, and the C-suite may focus more on MRR (monthly recurring revenue) and ARR (annual recurring revenue).

In this article, I will look at subscription-model-based businesses and some methods and strategies that can pivot their SEO efforts toward maintaining performance and SEO ROI (return on investment).

Understanding Why Accounts Cancel

Customers cancel their subscriptions for myriad reasons, but during an economic downturn, reasons tend to gravitate toward costs and perceived value.

Other reasons include not receiving enough value from the subscription, difficulty canceling their subscription, or feeling that customer support is unresponsive or unhelpful.

You can identify these issues before customers provide feedback on an exit survey. Create opportunities for conversations and feedback loops with the sales and customer service teams. This lets customers address concerns before they cancel.

Targeting Disengagement & Value Shortfalls

To show this value, we can pivot our content and messaging to demonstrate opportunity costs and how the upfront cost prevents a more significant shortfall in the long run.

Encountering usage friction with the software is an identifiable problem.

Within the organization, teams should be able to provide you access to DAU (daily active user) and MAU (monthly active user) data.

Companies often boast about having high numbers of each, but the data can also be used to identify accounts with below-average or spare login frequency, and these can then be collated and reached out to.

  • Put accounts on low and mid-tier subscriptions into an email gauntlet and reach out. Offer a consultation with an accounts person. You could also ask them to fill out a feedback form to identify pain points to help build a content strategy.
  • Reach out to accounts on high-tier subscriptions with existing account managers.

Addressing customer issues could be as simple as rewording elements of commercial product pages, adding additional sections, or reinforcing the value proposition with case studies.

You can also address these issues with traditional blog content. Add more support articles to your support center and build out existing ones with media such as video to address common friction points.

Developing Content Against Competitor Value Pitfalls

Price is likely the most challenging reason for leaving to predict and manage. Price is informed and dictated by other business needs and costs. While it might make sense to offer deals to high-value accounts, reducing the price on a wide scale likely isn’t an option.

Price and cost are subjective to the value your solution provides. So Demonstrating your benefits can help customers justify the expenditure.

Any solution’s cost must, at minimum, balance out the problem or provide additional value.

This is known as a cost-benefit analysis. A vital part of a cost-benefit analysis is comparing the costs of the solution versus the benefits and determining a net present value.

During this assessment, your messaging can leverage and demonstrate additional benefits, or benefit enhancements, against your competitors.

In SaaS, you could break this down as comparisons between both product elements and overall “package” elements:

  • Direct product features and performance of those features.
  • Indirect product features and “add ons” that supplement the core product.
  • The bandwidth of the solution on a monthly or annual basis.
  • The number of user seats/sub-accounts per main account.
  • Speed of customer support response (and level of customer support).

A typical approach to highlighting competitor pitfalls is with comparison tables and our-brand-v-competitor-brand URLs and blogs.

These pages will then compete with your competitors’ versions and independent websites, affiliates, and other reviews for clicks and to sway consumer opinion.

You must also explain these benefits and competitive advantages on the product pages themselves.

Bullet listing the product features is commonplace. But make sure the benefits are explained directly against your competitors. This can help these competitive advantages better resonate with your target audience.

Reinforcing Brand Solution Compounds

A brand compound search term is a term made up of two or more words and refers to a specific brand.

For example, the brand compound search term “Decathlon waterproofs” would highlight users wanting to find waterproofs specifically from the brand Decathlon.

Users performing searches like this also reaffirms the connection between topics and brands, helping Google further understand relationships and relevancy.

To optimize brand compound search terms, you need to understand the concept of semantic marketing. This means knowing how different words, phrases, and ideas relate in terms of meaning.

You should research how your target audience searches for information related to your product or service and use those search terms in your content.

Another strategy you can use is to add modifiers to your search terms.

These can be words like “best,” “how,” or any other qualifier that will make the search more specific. This will help you get more targeted traffic that will likely convert better than generic search terms.


While these are uncertain times and competition for users and recurring revenue becoming more fierce, pivoting your SEO and content strategy to focus on value propositions and addressing consumer friction points can help better qualify leads and provide objection questions that consumers will take to competitors.

In this strategy, the keyword search volumes and other values might not be high. When you’re addressing user friction points and concerns, the value is qualitative, not quantitative.

More resources:

Featured Image: VectorMine/Shutterstock

Source link

Continue Reading


Where Are The Advertisers Leaving Twitter Going For The Super Bowl?



Where Are The Advertisers Leaving Twitter Going For The Super Bowl?

Since Elon Musk’s takeover of Twitter last October 27, 2022, things at the social media company have gone from bad to worse.

You probably saw this coming from a mile away – especially if you had read about a study by Media Matters that was published on November 22, 2022, entitled, “In less than a month, Elon Musk has driven away half of Twitter’s top 100 advertisers.”

If you missed that, then you’ve probably read Matt G. Southern’s article in Search Engine Journal, which was entitled, “Twitter’s Revenue Down 40% As 500 Top Advertisers Pull Out.”

This mass exodus creates a challenge for digital advertising executives and their agencies. Where should they go long term?

And what should they do in the short term – with Super Bowl LVII coming up on Sunday, February 12, 2023?

Ideally, these advertisers would follow their audience. If they knew where Twitter users were going, their ad budgets could follow them.

But it isn’t clear where Twitter users are going – or if they’ve even left yet.

Fake Followers On Twitter And Brand Safety

According to the latest data from Similarweb, a digital intelligence platform, there were 6.9 billion monthly visits to Twitter worldwide during December 2022 – up slightly from 6.8 billion in November, and down slightly from 7.0 billion in October.

So, if a high-profile user like Boston Mayor Michelle Wu has taken a step back from the frequent posts on her Twitter account, @wutrain, which has more than 152,000 followers, then it appears that other users have stepped up their monthly visits.

This includes several accounts that had been banned previously for spreading disinformation, which Musk unbanned.

(Disinformation is defined as “deliberately misleading or biased information,” while misinformation may be spread without the sender having harmful intentions.)

It’s also worth noting that SparkToro, which provides audience research software, also has a free tool called Fake Follower Audit, which analyzes Twitter accounts.

This tool defines “fake followers” as ones that are unreachable and will not see the account’s tweets either because they’re spam, bots, and propaganda, or because they’re no longer active on Twitter.

On Jan. 24, 2023, I used this tool and found that 70.2% of the 126.5 million followers of the @elonmusk account were fake.

According to the tool, accounts with a similar-sized following to @elonmusk have a median of 41% fake followers. So, Elon Musk’s account has more fake followers than most.

Screenshot from SparkToro, January 2023

By comparison, 20.6% of the followers of the @wutreain account were fake. So, Michelle Wu’s account has fewer fake followers than accounts with a similar-sized following.

Sparktoro results for fake followersScreenshot from SparkToro, January 2023

In fact, most Twitter accounts have significant numbers of fake followers.

This underlines the brand safety concerns that many advertisers and media buyers have, but it doesn’t give them any guidance on where they should move their ad dollars.

Who Are Twitter’s Top Competitors And What Are Their Monthly Visits?

So, I asked Similarweb if they had more data that might help. And they sent me the monthly visits from desktop and mobile devices worldwide for Twitter and its top competitors:

  • 34.6 billion in December 2022, down 2.8% from 35.6 billion in December 2021.
  • 18.1 billion in December 2022, down 14.2% from 21.1 billion in December 2021.
  • 6.9 billion in December 2022, up 1.5% from 6.8 billion in December 2021.
  • 6.3 billion in December 2022, down 3.1% from 6.5 billion in December 2021.
  • 1.9 billion in December 2022, up 26.7% from 1.5 billion in December 2021.
  • 1.8 billion in December 2022, down 5.3% from 1.9 billion in December 2021.
  • 1.5 billion in December 2022, up 7.1% from 1.4 billion in December 2021.
  • 1.0 billion in December 2022, up 11.1% from 0.9 billion in December 2021.

The most significant trends worth noting are monthly visits to TikTok are up 26.7% year over year from a smaller base, while monthly visits to Facebook are down 14.2% from a bigger base.

So, the short-term events at Twitter over the past 90 days may have taken the spotlight off the long-term trends at TikTok and Facebook over the past year for some industry observers.

But based on Southern’s article in Search Engine Journal, “Facebook Shifts Focus To Short-Form Video After Stock Plunge,” which was published on February 6, 2022, Facebook CEO Mark Zuckerberg is focused on these trends.

In a call with investors, Zuckerberg said back then:

“People have a lot of choices for how they want to spend their time, and apps like TikTok are growing very quickly. And this is why our focus on Reels is so important over the long term.”

Meanwhile, there were 91% more monthly visits to YouTube in December 2022 than there were to Facebook. And that only counts the visits that Similarweb tracks from mobile and desktop devices.

Similarweb doesn’t track visits from connected TVs (CTVs).

Measuring Data From Connected TVs (CTVs) And Co-Viewing

Why would I wish to draw your attention to CTVs?

First, global viewers watched a daily average of over 700 million hours of YouTube content on TV devices, according to YouTube internal data from January 2022.

And Insider Intelligence reported in 2022 that 36.4% of the U.S. share of average time spent per day with YouTube came from connected devices, including Apple TV, Google Chromecast, Roku, and Xfinity Flex, while 49.3% came from mobile devices, and 14.3% came from desktops or laptops.

Second, when people watch YouTube on a connected TV, they often watch it together with their friends, family, and colleagues – just like they did at Super Bowl parties before the pandemic.

There’s even a term for this behavior: Co-viewing.

And advertisers can now measure their total YouTube CTV audience using real-time and census-level surveys in over 100 countries and 70 languages.

This means Heineken and Marvel Studios can measure the co-viewing of their Super Bowl ad in more than 100 markets around the globe where Heineken 0.0 non-alcoholic beer is sold, and/or 26 countries where “Ant-Man and The Wasp: Quantumania” is scheduled to be released three to five days after the Big Game.

It also enables Apple Music to measure the co-viewing of their Super Bowl LVII Halftime Show during Big Game parties worldwide (except Mainland China, Iran, North Korea, and Turkmenistan, where access to YouTube is currently blocked).

And, if FanDuel has already migrated to Google Analytics 4 (GA4), then the innovative sports-tech entertainment company can not only measure the co-viewing of their Big Game teasers on YouTube AdBlitz in 16 states where sports betting is legal, but also measure engaged-view conversions (EVCs) from YouTube within 3 days of viewing Rob Gronkowski’s attempt to kick a live field goal.


Advertisers couldn’t do that in 2022. But they could in a couple of weeks.

If advertisers want to follow their audience, then they should be moving some of their ad budgets out of Facebook, testing new tactics, and experimenting with new initiatives on YouTube in 2023.

Where should the advertisers leaving Twitter shift their budgets long term? And how will that change their Super Bowl strategies in the short term?

According to Similarweb, monthly visits to, the platform’s ad-buying portal dropped 15% worldwide from 2.5 million in December 2021 to 2.1 million in December 2022.

So, advertisers were heading for the exit weeks before they learned that 500 top advertisers had left the platform.

Where Did Their Ad Budgets Go?

Well, it’s hard to track YouTube advertising, which is buried in Google’s sprawling ad business.

And we can’t use as a proxy for interest in advertising on that platform because it’s used by businesses for other purposes, such as managing organic content on their Facebook pages.

But monthly visits to, that platform’s ad-buying portal, jumped 88.3% from 1.6 million in December 2021 to 3.0 million in December 2022.

Monthly visits to are up 36.6% from 5.1 million in December 2021 to 7.0 million in December 2022.

Monthly visits to are up 23.3% from 1.1 million in December 2021 to 1.4 million in December 2022.

And monthly visits to are up 14.6% from 5.7 million in December 2021 to 6.5 million in December 2022.

It appears that lots of advertisers are hedging their bets by spreading their money around.

Now, most of them should probably continue to move their ad budgets into Snapchat, TikTok, Pinterest, and LinkedIn – unless the “Chief Twit” can find a way to keep his microblogging service from becoming “a free-for-all hellscape, where anything can be said with no consequences!

How will advertisers leaving Twitter change their Super Bowl plan this year?

To double-check my analysis, I interviewed Joaquim Salguerio, who is the Paid Media Director at LINK Agency. He’s managed media budgets of over eight figures at multiple advertising agencies.

Below are my questions and his answers.

Greg Jarboe: “Which brands feel that Twitter has broken their trust since Musk bought the platform?”

Joaquim Salguerio: “I would say that several brands will have different reasonings for this break of trust.

First, if you’re an automaker, there’s suddenly a very tight relationship between Twitter and one of your competitors.

Second, advertisers that are quite averse to taking risks with their communications because of brand safety concerns might feel that they still need to be addressed.

Most of all, in a year where we’re seeing mass layoffs from several corporations, the Twitter troubles have given marketing teams a reason to re-evaluate its effectiveness during a time of budget cuts. That would be a more important factor than trust for most brands.

Obviously, there are some famous cases, such as the Lou Paskalis case, but it’s difficult to pinpoint a brand list that would have trust as their only concern.”

GJ: “Do you think it will be hard for Twitter to regain their trust before this year’s Super Bowl?”

JS: “It’s highly unlikely that any brand that has lost trust in Twitter will change its mind in the near future, and definitely not in time for the Super Bowl. Most marketing plans for the event will be finalized by now and recent communications by Twitter leadership haven’t signaled any change in direction.

If anything, from industry comments within my own network, I can say that comments from Musk recently (“Ads are too frequent on Twitter and too big. Taking steps to address both in coming weeks.”) were quite badly received. For any marketers that believe Twitter advertising isn’t sufficiently effective, this pushes them further away.

Brand communications should still occur on Twitter during Super Bowl though – it will have a peak in usage. And advertising verticals that should dominate the advertising space on Twitter are not the ones crossing the platform from their plans.”

GJ: “How do you think advertisers will change their Super Bowl plans around Twitter this year?”

JS: “The main change for advertising plans will likely be for brand comms amplification. As an example, the betting industry will likely be heavily present on Twitter during the game and I would expect little to no change in plans.”

In the FCMG category, though, time sensitivity won’t be as important, which means that social media teams will likely be making an attempt at virality without relying as much on paid dollars.

If budgets are to diverge, they will likely be moved within the social space and toward platforms that will have user discussion/engagement from the Super Bowl (TikTok, Reddit, etc.)”

GJ: “What trends will we see in advertising budget allocation for this year’s Super Bowl?”

Joaquim Salguerio: “We should see budget planning much in line with previous years in all honesty. TV is still the most important media channel on Super Bowl day.

Digital spend will likely go towards social platforms, we predict a growth in TikTok and Reddit advertising around the big day for most brands.

Twitter should still have a strong advertising budget allocated to the platform by the verticals aiming to get actions from users during the game (food delivery/betting/etc.).”

GJ: “Which platforms will benefit from this shift?”

JS: “Likely, we will see TikTok as the biggest winner from a shift in advertising dollars, as the growth numbers are making it harder to ignore the platform as a placement that needs to be in the plan.

Reddit can also capture some of this budget as it has the right characteristics marketers are looking for around the Super Bowl – it’s relevant to what’s happening at the moment and similar demographics.”

GJ: “Do you think advertisers that step away from Twitter for this year’s Big Game will stay away long term?”

JS: “That is impossible to know, as it’s completely dependent on how the platform evolves and the advertising solutions it will provide. Twitter’s proposition was always centered around brand marketing (their performance offering was always known to be sub-par).

Unless brand safety concerns are addressed by brands that decided to step away, it’s hard to foresee a change.

I would say that overall, Super Bowl ad spend on Twitter should not be as affected as it’s been portrayed – it makes sense to reach audiences where audiences are.

Especially if you know the mindset. The bigger issue is what happens when there isn’t a Super Bowl or a World Cup.”

More resources:

Featured Image: Brocreative/Shutterstock

Source link

Continue Reading


Is ChatGPT Use Of Web Content Fair?



Is ChatGPT Use Of Web Content Fair?

Large Language Models (LLMs) like ChatGPT train using multiple sources of information, including web content. This data forms the basis of summaries of that content in the form of articles that are produced without attribution or benefit to those who published the original content used for training ChatGPT.

Search engines download website content (called crawling and indexing) to provide answers in the form of links to the websites.

Website publishers have the ability to opt-out of having their content crawled and indexed by search engines through the Robots Exclusion Protocol, commonly referred to as Robots.txt.

The Robots Exclusions Protocol is not an official Internet standard but it’s one that legitimate web crawlers obey.

Should web publishers be able to use the Robots.txt protocol to prevent large language models from using their website content?

Large Language Models Use Website Content Without Attribution

Some who are involved with search marketing are uncomfortable with how website data is used to train machines without giving anything back, like an acknowledgement or traffic.

Hans Petter Blindheim (LinkedIn profile), Senior Expert at Curamando shared his opinions with me.

Hans commented:

“When an author writes something after having learned something from an article on your site, they will more often than not link to your original work because it offers credibility and as a professional courtesy.

It’s called a citation.

But the scale at which ChatGPT assimilates content and does not grant anything back differentiates it from both Google and people.

A website is generally created with a business directive in mind.

Google helps people find the content, providing traffic, which has a mutual benefit to it.

But it’s not like large language models asked your permission to use your content, they just use it in a broader sense than what was expected when your content was published.

And if the AI language models do not offer value in return – why should publishers allow them to crawl and use the content?

Does their use of your content meet the standards of fair use?

When ChatGPT and Google’s own ML/AI models trains on your content without permission, spins what it learns there and uses that while keeping people away from your websites – shouldn’t the industry and also lawmakers try to take back control over the Internet by forcing them to transition to an “opt-in” model?”

The concerns that Hans expresses are reasonable.

In light of how fast technology is evolving, should laws concerning fair use be reconsidered and updated?

I asked John Rizvi, a Registered Patent Attorney (LinkedIn profile) who is board certified in Intellectual Property Law, if Internet copyright laws are outdated.

John answered:

“Yes, without a doubt.

One major bone of contention in cases like this is the fact that the law inevitably evolves far more slowly than technology does.

In the 1800s, this maybe didn’t matter so much because advances were relatively slow and so legal machinery was more or less tooled to match.

Today, however, runaway technological advances have far outstripped the ability of the law to keep up.

There are simply too many advances and too many moving parts for the law to keep up.

As it is currently constituted and administered, largely by people who are hardly experts in the areas of technology we’re discussing here, the law is poorly equipped or structured to keep pace with technology…and we must consider that this isn’t an entirely bad thing.

So, in one regard, yes, Intellectual Property law does need to evolve if it even purports, let alone hopes, to keep pace with technological advances.

The primary problem is striking a balance between keeping up with the ways various forms of tech can be used while holding back from blatant overreach or outright censorship for political gain cloaked in benevolent intentions.

The law also has to take care not to legislate against possible uses of tech so broadly as to strangle any potential benefit that may derive from them.

You could easily run afoul of the First Amendment and any number of settled cases that circumscribe how, why, and to what degree intellectual property can be used and by whom.

And attempting to envision every conceivable usage of technology years or decades before the framework exists to make it viable or even possible would be an exceedingly dangerous fool’s errand.

In situations like this, the law really cannot help but be reactive to how technology is used…not necessarily how it was intended.

That’s not likely to change anytime soon, unless we hit a massive and unanticipated tech plateau that allows the law time to catch up to current events.”

So it appears that the issue of copyright laws has many considerations to balance when it comes to how AI is trained, there is no simple answer.

OpenAI and Microsoft Sued

An interesting case that was recently filed is one in which OpenAI and Microsoft used open source code to create their CoPilot product.

The problem with using open source code is that the Creative Commons license requires attribution.

According to an article published in a scholarly journal:

“Plaintiffs allege that OpenAI and GitHub assembled and distributed a commercial product called Copilot to create generative code using publicly accessible code originally made available under various “open source”-style licenses, many of which include an attribution requirement.

As GitHub states, ‘…[t]rained on billions of lines of code, GitHub Copilot turns natural language prompts into coding suggestions across dozens of languages.’

The resulting product allegedly omitted any credit to the original creators.”

The author of that article, who is a legal expert on the subject of copyrights, wrote that many view open source Creative Commons licenses as a “free-for-all.”

Some may also consider the phrase free-for-all a fair description of the datasets comprised of Internet content are scraped and used to generate AI products like ChatGPT.

Background on LLMs and Datasets

Large language models train on multiple data sets of content. Datasets can consist of emails, books, government data, Wikipedia articles, and even datasets created of websites linked from posts on Reddit that have at least three upvotes.

Many of the datasets related to the content of the Internet have their origins in the crawl created by a non-profit organization called Common Crawl.

Their dataset, the Common Crawl dataset, is available free for download and use.

The Common Crawl dataset is the starting point for many other datasets that created from it.

For example, GPT-3 used a filtered version of Common Crawl (Language Models are Few-Shot Learners PDF).

This is how  GPT-3 researchers used the website data contained within the Common Crawl dataset:

“Datasets for language models have rapidly expanded, culminating in the Common Crawl dataset… constituting nearly a trillion words.

This size of dataset is sufficient to train our largest models without ever updating on the same sequence twice.

However, we have found that unfiltered or lightly filtered versions of Common Crawl tend to have lower quality than more curated datasets.

Therefore, we took 3 steps to improve the average quality of our datasets:

(1) we downloaded and filtered a version of CommonCrawl based on similarity to a range of high-quality reference corpora,

(2) we performed fuzzy deduplication at the document level, within and across datasets, to prevent redundancy and preserve the integrity of our held-out validation set as an accurate measure of overfitting, and

(3) we also added known high-quality reference corpora to the training mix to augment CommonCrawl and increase its diversity.”

Google’s C4 dataset (Colossal, Cleaned Crawl Corpus), which was used to create the Text-to-Text Transfer Transformer (T5), has its roots in the Common Crawl dataset, too.

Their research paper (Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer PDF) explains:

“Before presenting the results from our large-scale empirical study, we review the necessary background topics required to understand our results, including the Transformer model architecture and the downstream tasks we evaluate on.

We also introduce our approach for treating every problem as a text-to-text task and describe our “Colossal Clean Crawled Corpus” (C4), the Common Crawl-based data set we created as a source of unlabeled text data.

We refer to our model and framework as the ‘Text-to-Text Transfer Transformer’ (T5).”

Google published an article on their AI blog that further explains how Common Crawl data (which contains content scraped from the Internet) was used to create C4.

They wrote:

“An important ingredient for transfer learning is the unlabeled dataset used for pre-training.

To accurately measure the effect of scaling up the amount of pre-training, one needs a dataset that is not only high quality and diverse, but also massive.

Existing pre-training datasets don’t meet all three of these criteria — for example, text from Wikipedia is high quality, but uniform in style and relatively small for our purposes, while the Common Crawl web scrapes are enormous and highly diverse, but fairly low quality.

To satisfy these requirements, we developed the Colossal Clean Crawled Corpus (C4), a cleaned version of Common Crawl that is two orders of magnitude larger than Wikipedia.

Our cleaning process involved deduplication, discarding incomplete sentences, and removing offensive or noisy content.

This filtering led to better results on downstream tasks, while the additional size allowed the model size to increase without overfitting during pre-training.”

Google, OpenAI, even Oracle’s Open Data are using Internet content, your content, to create datasets that are then used to create AI applications like ChatGPT.

Common Crawl Can Be Blocked

It is possible to block Common Crawl and subsequently opt-out of all the datasets that are based on Common Crawl.

But if the site has already been crawled then the website data is already in datasets. There is no way to remove your content from the Common Crawl dataset and any of the other derivative datasets like C4 and .

Using the Robots.txt protocol will only block future crawls by Common Crawl, it won’t stop researchers from using content already in the dataset.

How to Block Common Crawl From Your Data

Blocking Common Crawl is possible through the use of the Robots.txt protocol, within the above discussed limitations.

The Common Crawl bot is called, CCBot.

It is identified using the most up to date CCBot User-Agent string: CCBot/2.0

Blocking CCBot with Robots.txt is accomplished the same as with any other bot.

Here is the code for blocking CCBot with Robots.txt.

User-agent: CCBot
Disallow: /

CCBot crawls from Amazon AWS IP addresses.

CCBot also follows the nofollow Robots meta tag:

<meta name="robots" content="nofollow">

What If You’re Not Blocking Common Crawl?

Web content can be downloaded without permission, which is how browsers work, they download content.

Google or anybody else does not need permission to download and use content that is published publicly.

Website Publishers Have Limited Options

The consideration of whether it is ethical to train AI on web content doesn’t seem to be a part of any conversation about the ethics of how AI technology is developed.

It seems to be taken for granted that Internet content can be downloaded, summarized and transformed into a product called ChatGPT.

Does that seem fair? The answer is complicated.

Featured image by Shutterstock/

Source link

Continue Reading