SEO
How To Read Google Patents In 5 Easy Steps
Reading and understanding patents filed by Google can be challenging but this guide will help you to understand what the patents are about and to as avoid the many common mistakes that lead to misunderstandings.
How To Understand Google Patents
Before starting to read a patent it’s important to understand how to read the patents. The following rules will form the foundation upon which you can build a solid understanding of what patents mean.
Step #1 Do Not Scan Patents
One of the biggest mistakes I see people make when reading patents is to approach the task as if it’s a treasure hunt. They scan the patents looking for tidbits and secrets about Google’s algorithms.
I know people do this because I’ve seen so many wrong conclusions made by SEOs who I can tell didn’t read the patent because they only speak about the one or two sentences that jump out at them.
Had they read the entire patent they would have understood that the passage they got excited about had nothing to do with ranking websites.
Reading a patent is not like a treasure hunt with a metal detector where the treasure hunter scans an entire field and then stops in one spot to dig up a cache of gold coins.
Don’t scan a patent. Read it.
Step #2 Understand The Context Of The Patent
A patent is like an elephant. An elephant has a trunk, big ears, a little tail and legs thick as trees. Similarly, a patent is made up of multiple sections that are each very important because they create the context of what the patent is about. Each section of a patent is important.
And just like how each part of an elephant in the context of the entire animal helps to better understand the creature, so it is with patents that every section only makes sense within the context of the entire patent.
In order to understand the patent it’s important to read the entire patent several times in order to be able step back and see the entire patent, not just one part of the patent.
Reading the entire patent reveals what the context of the entire patent is, which is the most important thing about the patent, what the entire thing means.
Step #3 Not Every Patent Is About Ranking
If there’s any one thing I wish the reader to take away from this article is this rule. When I read tweets or articles by people who don’t know how to read patents, this is the rule that they haven’t understood. Consequently, the interpretation of the patent is wrong.
Google Search is not just one ranking algorithm. There are many algorithms that comprise different parts of Search. The Ranking Engine and the Indexing Engine are just two parts of Search.
Other elements of search that may be referred to are:
- Ranking engine
- Modification engine
- Indexing engine
- Query reviser engine
Those are just a few of the kinds of software engines that are a part of a typical search engine. While the different software engines are not necessarily a part of the ranking part of Google’s algorithms, that does not minimize their importance.
Back in 2020 Gary Illyes of Google tweeted that Search consists of thousands of different systems working together.
He tweeted about the indexing engine:
“The indexing system, Caffeine, does multiple things:
1. ingests fetchlogs,
2. renders and converts fetched data,
3. extracts links, meta and structured data,
4. extracts and computes some signals,
5. schedules new crawls,
6. and builds the index that is pushed to serving.”
He followed up with another tweet about the thousands of systems in search:
“Don’t oversimplify search for it’s not simple at all: thousands of interconnected systems working together to provide users high quality and relevant results…
…the last time i did this exercise I counted off the top of my head about 150 different systems from crawling to ranking, so thousands is likely not an exaggeration. Yes, some things are micro services”
Here’s The Important Takeaway:
There are many parts of Search. But not all parts of Search are a part of the ranking systems.
A very important habit to cultivate when reading a patent is to let the patent tell you what it’s about.
Equally important is to not make assumptions or assume that something is implied. Patents don’t generally imply. They may be broad and and they may seem to be so repetitive that it almost feels like a deliberate attempt obfuscate (make it hard to understand) and they consistently describe the inventions in extremely broad terms, but they don’t really imply what they are describing.
Patents, for legal purposes, are actually quite specific about what the patents are about.
If something is used for ranking then it will not be implied, the patent will say so because that’s an important quality to describe in a patent application.
Step #4 Entity & Entities: Understand The Use Of Abstraction
One of the biggest mistakes that happens to people who read patents is to overlook the context of where the invention can be used. For example, let’s review a specific patent called “Identifying subjective attributes by analysis of curation signals.”
This patent mentions entities 52 times and the word “entity” is mentioned in the patent itself 124 times. One can easily guess that this patent is probably about entities, right? It makes sense that if the patent mentions the words “entities” and “entity” nearly 200 times that the patent is about entities.
But that would be an unfortunate assumption because the patent is not about entities at all because the context of the use of the words “entity” and “entities” in this patent is to refer to a broad and inclusive range of items, subjects, or objects to which the invention can be applied.
Patents often cast a wide net in terms of how the invention can be used, which helps to ensure that the patent’s claims aren’t limited to one type of use but can be applied in many ways.
The word “entity” in this patent is used as a catch-all term that allows the patent to cover a wide range of different types of content or objects. It is used in the sense of an abstraction so that it can be applied to multiple objects or forms of content. This frees the patent to focus on the functionality of the invention and how it can be applied.
The use of abstraction keeps a patent from being tied down to the specifics of what it is being applied to because in most cases the patent is trying to communicate how it can be applied in many different ways.
In fact, the patent places the invention in the context of different forms of content entities such as videos, images, and audio clips. The patent also refers to text-based content (like articles, blog posts), as well as more tangible entities (like products, services, organizations, or even individuals).
Here is an example from the patent where it explicitly refers to video clips as one of the entities that the patent is concerned with:
“In one implementation, the above procedure is performed for each entity in a given set of entities (e.g., video clips in a video clip repository, etc.), and an inverse mapping from subjective attributes to entities in the set is generated based on the subjective attributes and relevancy scores.”
In this context, “video clips” are explicitly mentioned as an example of the entities to which the invention can be applied. The passage indicates that the procedure described in the patent (identifying and scoring subjective attributes of entities) is applicable to video clips.”
Here is another passage where the word entity is used to denote a type of content:
“Entity store 120 is a persistent storage that is capable of storing entities such as media clips (e.g., video clips, audio clips, clips containing both video and audio, images, etc.) and other types of content items (e.g., webpages, text-based documents, restaurant reviews, movie reviews, etc.), as well as data structures to tag, organize, and index the entities.”
That part of the patent describes “content items” as entities and gives examples like webpages, text-based documents, restaurant reviews, and movie reviews, alongside media clips such as video and audio clips. This and other similar passages show that the term “entity” within the context of this patent broadly encompasses multiple forms of digital content.
That patent, titled Identifying subjective attributes by analysis of curation signals, is actually related to a recommender system or search that leverages User Generated Content like comments for the purpose of tagging digital content with the subjective opinions of those users.
The patent specifically uses the example of users describing an entity (like an image or a video) as funny, which can then be used to surface a video that has the subjective quality of funny as a part of a recommender system.
The most obvious application of this patent is for finding videos on YouTube that users and authors have described as funny. The use of this patent isn’t limited to just YouTube videos, it can also be used in other scenarios that intersect with user generated content.
The patent explicitly mentions the application of the invention in the context of a recommender system in the following passage:
“In one implementation, the above procedure is performed for each entity in a given set of entities (e.g., video clips in a video clip repository, etc.), and an inverse mapping from subjective attributes to entities in the set is generated based on the subjective attributes and relevancy scores.
The inverse mapping can then be used to efficiently identify all entities in the set that match a given subjective attribute (e.g., all entities that have been associated with the subjective attribute ‘funny’, etc.), thereby enabling rapid retrieval of relevant entities for processing keyword searches, populating playlists, delivering advertisements, generating training sets for the classifier, and so forth.”
Some SEOs, because the patent mentions authors three times have claimed that this patent has something to do with ranking content authors and because of that they also associate the patent it with E-A-T.
Others, because the patent mentions the words “entity” and “entities” so many times have come to believe it has something to do with natural language processing and semantic understanding of webpages.
But neither of those are true and now that I’ve explained some of this patent it should be apparent how a lack of understanding of how to read a patent plus approaching patents with the mindset of treasure hunting for spicy algorithm clues can lead to unfortunate and misleading errors in understanding what the patents are actually about.
In a future article I will walk through different patents and I think doing that will help readers understand how to read a patent. If that’s something you are interested in then please share this article on social media and let me know!
I’m going to end this article with a description of the different parts of a patent, which should go some way to building an understanding of patents.
Step #5 Know The Parts Of A Patent
Every patent is comprised of multiple parts, a beginning, a middle and an end that each have a specific purpose. Many patents are also accompanied by illustrations that are helpful for understanding what the patent is about.
Patents typically follow this pattern:
Abstract:
A concise summary of the patent, giving a quick overview of what the invention is and what it does. It’s provides a brief explanation. This part is actually important because it tells what the patent is about. Do not be one of those SEOs who skip this part to go treasure hunting in the middle parts for clues about the algorithm. Pay attention to the Abstract.
Background:
This section offers context for the invention. It typically gives an overview of the field related to the invention and in a direct or indirect way explains how the invention fits into the context. This is another important part of the patent. It doesn’t give up clues about the algorithm but it tells what part of the system it belongs to and what it’s trying to do.
Summary:
The Summary provides a more detailed overview of the invention than the Abstract. We often say you can step back and view the forest, can step closer and see the trees. The Summary can be said to be stepping forward to see the leaves and just like a tree has a lot of leaves, a Summary can contain a lot of details.
The Summary outlines the invention’s primary objectives, features, and the minutiae of how it does it and all the variations of how it does it. It is almost always an eye-wateringly comprehensive description.
The very first paragraph though can often be the most descriptive and understandable part, after which the summary deep-dives into fine detail. One can feel lost in the seemingly redundant descriptions of the invention. It can be boring but read it at least twice, more if you need to.
Don’t be dismayed if you can’t understand it all because this part isn’t about finding the spicy bits that make for good tweets. This part of reading a patent is sometimes more about kind of absorbing the ideas and getting a feel for it.
Brief Description Of The Drawings:
In patents where drawings are included, this section explains what each drawing represents, sometimes with just a single sentence. It can be as brief as this:
“FIG. 1 is a diagram that illustrates obtaining an authoritative search result.
FIG. 2 is a diagram that illustrates resources visited during an example viewing session.
FIG. 3 is a flow chart of an example process for adjusting search result scores.”
The descriptions provide valuable information and are just as important as the illustrations themselves. They both can communicate a sharper understanding of the function of the patent invention.
What may seem like an invention about choosing authoritative sites for search results might in the illustrations turn out to be about finding the right files on a mobile phone and not have anything to do with information retrieval.
This where my advice to let the patent tell you what it’s about pays off. People too often skip these parts because they don’t contain spicy details. What happens next is that they miss the context for the entire patent and reach completely mistaken conclusions.
Detailed Description Of The Patent:
This is an in-depth description of the invention that uses the illustrations (figure 1, figure 2, etc.) as the organizing factor. This section may include technical information, how the invention works, how it is organized in relation to other parts, and how it can be used.
This section is intended to be thorough enough that someone skilled in the field could replicate the invention but also general enough so that it can be broadly applied in different ways.
Embodiment Examples:
Here is where specific examples of the invention are provided. The word “embodiment” refers to a particular implementation or an example of the invention. It is a way for the inventor to describe specific ways the invention can be used.
There are different contexts of the word embodiment that make it clear what the inventor considers a part of the invention, it is used in the context of illustrating the real-world use of the invention, define technical aspects and to show different ways the invention can be made or used.
That last one you’ll see a lot of paragraphs describing “in another embodiment the invention can bla bla bla…”
So when you see that word “embodiment” try to think of the word “body” and then “embody” in the sense of making something tangible and that will help you to better understand the “Embodiment” section of a patent.
Claims:
The Claims are the legal part of the patent. This section defines the scope of protection that the patent is looking for and it also offers insights into what the patent is about because this section often talks about what’s new and different about the invention. So don’t skip this part.
Citations:
This part lists other patents that are relevant to the invention. It’s used to acknowledge similar inventions but also to show how this invention is different from them and how it improves on what came before.
Firm Starting Point For Reading Patents
You should by this point have a foundation for practicing how to read a patent. Don’t be discouraged if the patent seems opaque and hard to understand. That’s normal.
I asked Jeff Coyle (Google Scholar Webpage), cofounder of MarketMuse (LinkedIn) for tips about reading patents because he’s filed some patent applications, authored a research paper, is a 20+ year SEO and has read thousands of patents, an expert.
Jeff offered this advice:
“Use Google Patent’s optional ‘non-patent literature’ Google Scholar search to find articles that may reference or support your knowledge of a patent.
Also understand that sometimes understanding a patent in isolation is nearly impossible, which is why it’s important to build context by collecting and reviewing connected patent and non-patent citations, child/priority patents/applications.
Another way that helps me to understand patents is to research other patents filed by the same authors. These are my core methods for understanding patents.”
That last tip is super important because some inventors tend to invent one kind of thing. So if you’re in doubt about whether a patent is about a certain thing, take a look at other patents that the inventor has filed to see if they tend to file patents on what you think a patent is about.
Patents have their own kind of language, with a formal structure and purpose to each section. Anyone who has learned a second language knows how important it is to look up words and to understand the structure that’s inherent in what’s written.
So don’t be discouraged because with practice you will be able to read patents better than many in the SEO industry are currently able to.
I intend at some point to walk through several patents with the hope that this will help you improve on reading patents. And remember to let me know on social media if this is something you want me to write!
Understanding the Impact of Google’s November 2024 Core Update on Global Search Rankings
Introduction
In November 2024, Google launched its latest core algorithm update, a broad refinement designed to enhance the quality of its search engine results. Rolling out over approximately two weeks, the update continues Google’s ongoing commitment to delivering more relevant, useful, and high-quality search experiences for users worldwide. This article explores the nature of the November 2024 Core Update, its potential impact on websites, and strategies for site owners to adapt and thrive in its aftermath.
1. What Is a Google Core Update?
Core updates are large-scale changes to Google’s search algorithms. Unlike targeted updates aimed at specific sectors or issues, core updates broadly impact all regions and languages. They reflect Google’s effort to re-evaluate how content is assessed and ranked based on relevance, usefulness, and reliability. Previous updates include significant releases like the March and August 2024 updates, illustrating the frequency and scope of these changes.
2. Goals of the November 2024 Core Update
The November update focuses on refining the quality of search results. According to Google’s official statements, it seeks to amplify genuinely useful content while reducing the visibility of content primarily designed to manipulate rankings without meeting user needs. This effort emphasizes Google’s consistent push for “people-first” content—engaging and useful information that serves users, not search engines.
3. Key Features and Characteristics of the Update
- Global Impact: The update affects search rankings on a global scale and is not confined to any particular industry or niche.
- Rollout Duration: Spanning about two weeks, the rollout’s timing allows Google to fully implement algorithmic changes and assess their effects.
- Broad Adjustments: The update doesn’t target specific sites but involves systemic reassessment across Google’s ranking systems.
- Dynamic Search Environment: This core update follows in the footsteps of the August and March 2024 updates, representing a year of significant search result refinement.
4. What This Means for Site Owners
- Traffic Fluctuations: Websites may observe shifts in rankings and traffic during the update’s rollout and subsequent completion. These changes highlight the dynamic nature of Google search and require continuous monitoring and adaptation.
- Recommended Actions:
- Wait and Analyze: Site owners experiencing changes should wait until the rollout’s completion before making significant adjustments.
- Utilize Google Search Console: Compare traffic and ranking data from before and after the update to identify potential areas of improvement.
- Focus on High-Impact Pages: Pages with notable drops in ranking should undergo thorough content evaluation using Google’s guidelines
5. Recovery and Adaptation Strategies
Recovering from a negative impact due to a core update may take weeks or months as Google’s systems adjust and validate content changes. Site owners should prioritize delivering high-quality, reliable, and user-focused content. Specific steps include:
- Content Evaluation: Assess content against Google’s guidelines, focusing on readability, user satisfaction, and factual accuracy.
- No Quick Fixes: Avoid superficial changes aimed solely at improving rankings. Sustainable improvements are more valuable and impactful(November 2024 core upda…).
- People-First Content: Ensure content serves real user needs, as opposed to purely SEO-driven objectives. This aligns with Google’s long-term priorities for search quality
6. Comparative Analysis with Previous Updates
The November 2024 Core Update continues trends observed in previous updates like March and August 2024. While each update has its nuances, their collective goal remains consistent: bettering search quality and delivering relevant results. Comparing data from these updates can reveal patterns and offer insights into Google’s evolving criteria
7. Broader Implications for the SEO Industry
Google’s ongoing core updates underscore the critical importance of a user-centric approach to SEO. For digital marketers and SEO specialists, adapting strategies to these updates involves staying informed, using reliable analytics tools, and keeping content fresh and engaging. The need for adaptability is paramount, as Google continually shifts the parameters of what defines quality content
Conclusion
The November 2024 Core Update serves as a reminder that Google’s algorithmic changes are not designed to punish but to reward helpful, authentic, and user-focused content. Site owners and marketers who embrace this philosophy are better positioned to weather core updates and even benefit from improved rankings and traffic over time. By maintaining a focus on user experience, transparency, and relevance, creators can align with Google’s evolving standards and thrive in the ever-changing digital landscape
SEO
How to Revive an Old Blog Article for SEO
Quick question: What do you typically do with your old blog posts? Most likely, the answer is: Not much.
If that’s the case, you’re not alone. Many of us in SEO and content marketing tend to focus on continuously creating new content, rather than leveraging our existing blog posts.
However, here’s the reality—Google is becoming increasingly sophisticated in evaluating content quality, and we need to adapt accordingly. Just as it’s easier to encourage existing customers to make repeat purchases, updating old content on your website is a more efficient and sustainable strategy in the long run.
Ways to Optimize Older Content
Some of your old content might not be optimized for SEO very well, rank for irrelevant keywords, or drive no traffic at all. If the quality is still decent, however, you should be able to optimize it properly with little effort.
Refresh Content
If your blog post contains a specific year or mentions current events, it may become outdated over time. If the rest of the content is still relevant (like if it’s targeting an evergreen topic), simply updating the date might be all you need to do.
Rewrite Old Blog Posts
When the content quality is low (you might have greatly improved your writing skills since you’ve written the post) but the potential is still there, there’s not much you can do apart from rewriting an old blog post completely.
This is not a waste—you’re saving time on brainstorming since the basic structure is already in place. Now, focus on improving the quality.
Delete Old Blog Posts
You might find a blog post that just seems unusable. Should you delete your old content? It depends. If it’s completely outdated, of low quality, and irrelevant to any valuable keywords for your website, it’s better to remove it.
Once you decide to delete the post, don’t forget to set up a 301 redirect to a related post or page, or to your homepage.
Promote Old Blog Posts
Sometimes all your content needs is a bit of promotion to start ranking and getting traffic again. Share it on your social media, link to it from a new post – do something to get it discoverable again to your audience. This can give it the boost it needs to attract organic links too.
Which Blog Posts Should You Update?
Deciding when to update or rewrite blog posts is a decision that relies on one important thing: a content audit.
Use your Google Analytics to find out which blog posts used to drive tons of traffic, but no longer have the same reach. You can also use Google Search Console to find out which of your blog posts have lost visibility in comparison to previous months. I have a guide on website analysis using Google Analytics and Google Search Console you can follow.
If you use keyword tracking tools like SE Ranking, you can also use the data it provides to come up with a list of blog posts that have dropped in the rankings.
Make data-driven decisions to identify which blog posts would benefit from these updates – i.e., which ones still have the chance to recover their keyword rankings and organic traffic.
With Google’s helpful content update, which emphasizes better user experiences, it’s crucial to ensure your content remains relevant, valuable, and up-to-date.
How To Update Old Blog Posts for SEO
Updating articles can be an involved process. Here are some tips and tactics to help you get it right.
Author’s Note: I have a Comprehensive On-Page SEO Checklist you might also be interested in following while you’re doing your content audit.
Conduct New Keyword Research
Updating your post without any guide won’t get you far. Always do your keyword research to understand how users are searching for your given topic.
Proper research can also show you relevant questions and sections that can be added to the blog post you’re updating or rewriting. Make sure to take a look at the People Also Ask (PAA) section that shows up when you search for your target keyword. Check out other websites like Answer The Public, Reddit, and Quora to see what users are looking for too.
Look for New Ranking Opportunities
When trying to revive an old blog post for SEO, keep an eye out for new SEO opportunities (e.g., AI Overview, featured snippets, and related search terms) that didn’t exist when you first wrote your blog post. Some of these features can be targeted by the new content you will add to your post, if you write with the aim to be eligible for it.
Rewrite Headlines and Meta Tags
If you want to attract new readers, consider updating your headlines and meta tags.
Your headlines and meta tags should fulfill these three things:
- Reflect the rewritten and new content you’ve added to the blog post.
- Be optimized for the new keywords it’s targeting (if any).
- Appeal to your target audience – who may have changed tastes from when the blog post was originally made.
Remember that your meta tags in particular act like a brief advertisement for your blog post, since this is what the user first sees when your blog post is shown in the search results page.
Take a look at your blog post’s click-through rate on Google Search Console – if it falls below 2%, it’s definitely time for new meta tags.
Replace Outdated Information and Statistics
Updating blog content with current studies and statistics enhances the relevance and credibility of your post. By providing up-to-date information, you help your audience make better, well-informed decisions, while also showing that your content is trustworthy.
Tighten or Expand Ideas
Your old content might be too short to provide real value to users – or you might have rambled on and on in your post. It’s important to evaluate whether you need to make your content more concise, or if you need to elaborate more.
Keep the following tips in mind as you refine your blog post’s ideas:
- Evaluate Helpfulness: Measure how well your content addresses your readers’ pain points. Aim to follow the E-E-A-T model (Experience, Expertise, Authoritativeness, Trustworthiness).
- Identify Missing Context: Consider whether your content needs more detail or clarification. View it from your audience’s perspective and ask if the information is complete, or if more information is needed.
- Interview Experts: Speak with industry experts or thought leaders to get fresh insights. This will help support your writing, and provide unique points that enhance the value of your content.
- Use Better Examples: Examples help simplify complex concepts. Add new examples or improve existing ones to strengthen your points.
- Add New Sections if Needed: If your content lacks depth or misses a key point, add new sections to cover these areas more thoroughly.
- Remove Fluff: Every sentence should contribute to the overall narrative. Eliminate unnecessary content to make your post more concise.
- Revise Listicles: Update listicle items based on SEO recommendations and content quality. Add or remove headings to stay competitive with higher-ranking posts.
Improve Visuals and Other Media
No doubt that there are tons of old graphics and photos in your blog posts that can be improved with the tools we have today. Make sure all of the visuals used in your content are appealing and high quality.
Update Internal and External Links
Are your internal and external links up to date? They need to be for your SEO and user experience. Outdated links can lead to broken pages or irrelevant content, frustrating readers and hurting your site’s performance.
You need to check for any broken links on your old blog posts, and update them ASAP. Updating your old blog posts can also lead to new opportunities to link internally to other blog posts and pages, which may not have been available when the post was originally published.
Optimize for Conversions
When updating content, the ultimate goal is often to increase conversions. However, your conversion goals may have changed over the years.
So here’s what you need to check in your updated blog post. First, does the call-to-action (CTA) still link to the products or services you want to promote? If not, update it to direct readers to the current solution or offer.
Second, consider where you can use different conversion strategies. Don’t just add a CTA at the end of the post.
Last, make sure that the blog post leverages product-led content. It’s going to help you mention your products and services in a way that feels natural, without being too pushy. Being subtle can be a high ROI tactic for updated posts.
Key Takeaway
Reviving old blog articles for SEO is a powerful strategy that can breathe new life into your content and boost your website’s visibility. Instead of solely focusing on creating new posts, taking the time to refresh existing content can yield impressive results, both in terms of traffic and conversions.
By implementing these strategies, you can transform old blog posts into valuable resources that attract new readers and retain existing ones. So, roll up your sleeves, dive into your archives, and start updating your content today—your audience and search rankings will thank you!
SEO
How Compression Can Be Used To Detect Low Quality Pages
The concept of Compressibility as a quality signal is not widely known, but SEOs should be aware of it. Search engines can use web page compressibility to identify duplicate pages, doorway pages with similar content, and pages with repetitive keywords, making it useful knowledge for SEO.
Although the following research paper demonstrates a successful use of on-page features for detecting spam, the deliberate lack of transparency by search engines makes it difficult to say with certainty if search engines are applying this or similar techniques.
What Is Compressibility?
In computing, compressibility refers to how much a file (data) can be reduced in size while retaining essential information, typically to maximize storage space or to allow more data to be transmitted over the Internet.
TL/DR Of Compression
Compression replaces repeated words and phrases with shorter references, reducing the file size by significant margins. Search engines typically compress indexed web pages to maximize storage space, reduce bandwidth, and improve retrieval speed, among other reasons.
This is a simplified explanation of how compression works:
- Identify Patterns:
A compression algorithm scans the text to find repeated words, patterns and phrases - Shorter Codes Take Up Less Space:
The codes and symbols use less storage space then the original words and phrases, which results in a smaller file size. - Shorter References Use Less Bits:
The “code” that essentially symbolizes the replaced words and phrases uses less data than the originals.
A bonus effect of using compression is that it can also be used to identify duplicate pages, doorway pages with similar content, and pages with repetitive keywords.
Research Paper About Detecting Spam
This research paper is significant because it was authored by distinguished computer scientists known for breakthroughs in AI, distributed computing, information retrieval, and other fields.
Marc Najork
One of the co-authors of the research paper is Marc Najork, a prominent research scientist who currently holds the title of Distinguished Research Scientist at Google DeepMind. He’s a co-author of the papers for TW-BERT, has contributed research for increasing the accuracy of using implicit user feedback like clicks, and worked on creating improved AI-based information retrieval (DSI++: Updating Transformer Memory with New Documents), among many other major breakthroughs in information retrieval.
Dennis Fetterly
Another of the co-authors is Dennis Fetterly, currently a software engineer at Google. He is listed as a co-inventor in a patent for a ranking algorithm that uses links, and is known for his research in distributed computing and information retrieval.
Those are just two of the distinguished researchers listed as co-authors of the 2006 Microsoft research paper about identifying spam through on-page content features. Among the several on-page content features the research paper analyzes is compressibility, which they discovered can be used as a classifier for indicating that a web page is spammy.
Detecting Spam Web Pages Through Content Analysis
Although the research paper was authored in 2006, its findings remain relevant to today.
Then, as now, people attempted to rank hundreds or thousands of location-based web pages that were essentially duplicate content aside from city, region, or state names. Then, as now, SEOs often created web pages for search engines by excessively repeating keywords within titles, meta descriptions, headings, internal anchor text, and within the content to improve rankings.
Section 4.6 of the research paper explains:
“Some search engines give higher weight to pages containing the query keywords several times. For example, for a given query term, a page that contains it ten times may be higher ranked than a page that contains it only once. To take advantage of such engines, some spam pages replicate their content several times in an attempt to rank higher.”
The research paper explains that search engines compress web pages and use the compressed version to reference the original web page. They note that excessive amounts of redundant words results in a higher level of compressibility. So they set about testing if there’s a correlation between a high level of compressibility and spam.
They write:
“Our approach in this section to locating redundant content within a page is to compress the page; to save space and disk time, search engines often compress web pages after indexing them, but before adding them to a page cache.
…We measure the redundancy of web pages by the compression ratio, the size of the uncompressed page divided by the size of the compressed page. We used GZIP …to compress pages, a fast and effective compression algorithm.”
High Compressibility Correlates To Spam
The results of the research showed that web pages with at least a compression ratio of 4.0 tended to be low quality web pages, spam. However, the highest rates of compressibility became less consistent because there were fewer data points, making it harder to interpret.
Figure 9: Prevalence of spam relative to compressibility of page.
The researchers concluded:
“70% of all sampled pages with a compression ratio of at least 4.0 were judged to be spam.”
But they also discovered that using the compression ratio by itself still resulted in false positives, where non-spam pages were incorrectly identified as spam:
“The compression ratio heuristic described in Section 4.6 fared best, correctly identifying 660 (27.9%) of the spam pages in our collection, while misidentifying 2, 068 (12.0%) of all judged pages.
Using all of the aforementioned features, the classification accuracy after the ten-fold cross validation process is encouraging:
95.4% of our judged pages were classified correctly, while 4.6% were classified incorrectly.
More specifically, for the spam class 1, 940 out of the 2, 364 pages, were classified correctly. For the non-spam class, 14, 440 out of the 14,804 pages were classified correctly. Consequently, 788 pages were classified incorrectly.”
The next section describes an interesting discovery about how to increase the accuracy of using on-page signals for identifying spam.
Insight Into Quality Rankings
The research paper examined multiple on-page signals, including compressibility. They discovered that each individual signal (classifier) was able to find some spam but that relying on any one signal on its own resulted in flagging non-spam pages for spam, which are commonly referred to as false positive.
The researchers made an important discovery that everyone interested in SEO should know, which is that using multiple classifiers increased the accuracy of detecting spam and decreased the likelihood of false positives. Just as important, the compressibility signal only identifies one kind of spam but not the full range of spam.
The takeaway is that compressibility is a good way to identify one kind of spam but there are other kinds of spam that aren’t caught with this one signal. Other kinds of spam were not caught with the compressibility signal.
This is the part that every SEO and publisher should be aware of:
“In the previous section, we presented a number of heuristics for assaying spam web pages. That is, we measured several characteristics of web pages, and found ranges of those characteristics which correlated with a page being spam. Nevertheless, when used individually, no technique uncovers most of the spam in our data set without flagging many non-spam pages as spam.
For example, considering the compression ratio heuristic described in Section 4.6, one of our most promising methods, the average probability of spam for ratios of 4.2 and higher is 72%. But only about 1.5% of all pages fall in this range. This number is far below the 13.8% of spam pages that we identified in our data set.”
So, even though compressibility was one of the better signals for identifying spam, it still was unable to uncover the full range of spam within the dataset the researchers used to test the signals.
Combining Multiple Signals
The above results indicated that individual signals of low quality are less accurate. So they tested using multiple signals. What they discovered was that combining multiple on-page signals for detecting spam resulted in a better accuracy rate with less pages misclassified as spam.
The researchers explained that they tested the use of multiple signals:
“One way of combining our heuristic methods is to view the spam detection problem as a classification problem. In this case, we want to create a classification model (or classifier) which, given a web page, will use the page’s features jointly in order to (correctly, we hope) classify it in one of two classes: spam and non-spam.”
These are their conclusions about using multiple signals:
“We have studied various aspects of content-based spam on the web using a real-world data set from the MSNSearch crawler. We have presented a number of heuristic methods for detecting content based spam. Some of our spam detection methods are more effective than others, however when used in isolation our methods may not identify all of the spam pages. For this reason, we combined our spam-detection methods to create a highly accurate C4.5 classifier. Our classifier can correctly identify 86.2% of all spam pages, while flagging very few legitimate pages as spam.”
Key Insight:
Misidentifying “very few legitimate pages as spam” was a significant breakthrough. The important insight that everyone involved with SEO should take away from this is that one signal by itself can result in false positives. Using multiple signals increases the accuracy.
What this means is that SEO tests of isolated ranking or quality signals will not yield reliable results that can be trusted for making strategy or business decisions.
Takeaways
We don’t know for certain if compressibility is used at the search engines but it’s an easy to use signal that combined with others could be used to catch simple kinds of spam like thousands of city name doorway pages with similar content. Yet even if the search engines don’t use this signal, it does show how easy it is to catch that kind of search engine manipulation and that it’s something search engines are well able to handle today.
Here are the key points of this article to keep in mind:
- Doorway pages with duplicate content is easy to catch because they compress at a higher ratio than normal web pages.
- Groups of web pages with a compression ratio above 4.0 were predominantly spam.
- Negative quality signals used by themselves to catch spam can lead to false positives.
- In this particular test, they discovered that on-page negative quality signals only catch specific types of spam.
- When used alone, the compressibility signal only catches redundancy-type spam, fails to detect other forms of spam, and leads to false positives.
- Combing quality signals improves spam detection accuracy and reduces false positives.
- Search engines today have a higher accuracy of spam detection with the use of AI like Spam Brain.
Read the research paper, which is linked from the Google Scholar page of Marc Najork:
Detecting spam web pages through content analysis
Featured Image by Shutterstock/pathdoc