SEO
The Downfall of Google & How to Survive in the New Era of Search
The relationship we used to have with Google was simple. It was a fair-use argument: we give Google content to index, and we get traffic in return. We do a good and ethical job at creating that content, we get rewarded with better rankings than those who do not.
AI has essentially turned all that on its head. Google I/O 2024 announced several AI-powered applications, the most notable being AI Overviews, which has since been rolled out in Google US.
It’s the beginning of a disruptive search landscape, and we have very little idea how that’s going to go. And, considering how Google plans to use AI Overviews, one has to wonder whether or not this is still fair to websites.
Worse, it brings us to this question: “What will happen to Google — and is it still valuable for me to work towards ranking in Google’s search results?”
The State of Google Search in 2024
Google search results today are almost unrecognizable from what it was a few years ago. Gone is the era of 10 blue links, replaced now with dynamic, “enriched” search features, displayed in such a way that, theoretically, makes it easier for a user to gain more and more information right on the results page.
In essence, Google is becoming less of a search engine and more of a portal for users.
The biggest step towards that was the launch of the Search Generative Experience (SGE). Though all their documentation and PR say that this phase is an experiment, it’s clear Google is trying to lay a foundational long-term change to the way people search on their platform.
To create this new search experience, SGE was powered by generative AI, which added another layer of input and output, enabling Google to not only understand more complex questions but answer them and guide users to new ones.
But that’s all in theory, and most technology sounds really, really good when it’s just in theory. But the reality of this shift in search is more consequential than most people think.
Moving towards an AI-powered search experience means reshaping how billions and billions of people find information online — and how Google makes money.
What Are AI Overviews
Using Google’s AI models, search queries are analyzed, and an answer is formulated. This then provides an AI card with links to publishers beneath the overview section.
They differ from featured snippets because they don’t directly show and link relevant content from websites. Instead, content is scraped, analyzed, and mashed together to provide what the AI thinks is the most comprehensive and accurate answer to your query.
If you’ve ever tried to teach yourself something complicated and reword those super technical concepts into what you think are layman’s terms, you know this is a difficult task, and one that can easily be done wrong.
And yes, even AI trained on countless datasets can be wrong. Other articles have covered how bad the results of Google’s AI Overviews are so far, so I won’t even bother. But it’s bad, and not very reassuring to anyone using it.
Why it Matters for Websites
Ok, so Google’s newest tool isn’t doing great, why should you care? It’s not like Google hasn’t been changing things up in the last decade, and this isn’t the first time its newest products have flopped. It’s not the first time that Google’s decentered organic results, either.
But this change won’t just push organic search results further below the fold — it’ll completely change the traditional search journey of a Google user.
By taking steps towards becoming a search portal, Google enables users to get answers to their questions without ever having to visit a website. And, it’s not unlikely that in the next few years, Google can evolve to allow users to fulfill transactions within the search interface itself.
How it Affects Independent Websites
Where is AI Overview getting its answers? From us, from SEOs, content creators, and webmasters. Google is doing this to millions and millions of websites — taking their content, without permission, for profit. All while taking that profit from our websites.
It’s unscrupulous, and is zero comfort for those who have already been negatively affected by algorithm updates in the past year.
And Google CEO Sundar Pichai doesn’t seem to worry about how it affects the very websites their search engine indexes and uses for profit. In a recent interview with the Verge, Pichai is optimistic that Google AI Overviews and Search will drive more traffic and engagement to websites — but then also does not commit to giving us a way to definitively see that ourselves just yet.
The reason? “The more we spec it out, then the more people will design for that,” he says. Pichai also says whether or not that data will be eventually given to websites is up to the Search team, not him.
I want to emphasize that Google definitely has that data, but they just won’t give us access to that in the Google Search Console performance reports. So we have no idea if our links in the AI Overviews have a higher or lower CTR than the normal search results
So, we have to trust Google. But after the recent antitrust case, the Search API document leak, and the validity of their statements being called into question, can we really trust them?
The Downfall of Google Search…
The trust in Google has been waning for several months now, if not years. Many SEOs and affiliates even say that Google search has died — an article from Edward Zitron, called “The Man Who Killed Google Search,” details how Prabhakar Raghavan ousted people like Ben Gomes from the helm of search and made its quality less of a priority, all for the short term bottom line.
And now we’re moving into what many call a “zero-click search” era, where users are becoming less and less inclined to click on organic search results. And Google is trying to capitalize on that with its AI Overviews, and failing miserably. Worse, it’s late to the game.
AI chatbots like ChatGPT, Microsoft Copilot, and Perplexity did it much earlier, and arguably, much better than Google is doing now.
…and Rise of AI Chatbots
Google’s whole point is that it’s supposed to provide the best search experience to users. So when it fails, what do people do? They abandon ship and go to what serves them best.
And right now, that’s AI chatbots like Perplexity.
I’ve been using this for a couple of months now, and the experience is great. Sources are provided and put at the priority — always placed on top of the generated answer. Answers were factual and included citations, and so far I haven’t seen any AI-generated nonsense.
It’s a step above other chatbots like ChatGPT as well, since it does have access to the internet and a proprietary search database, so answers are, generally, more accurate and up-to-date. I also found the generated answers to be much more detailed in Perplexity than when I tried out Google Bard (now known as Gemini) a while back.
One of the biggest complaints about Google Search is that the results are flooded with YouTube videos and Reddit forums. Perplexity avoids this completely by allowing users to opt out of those kinds of results and providing specific focus settings for those who are looking for forum answers and videos.
Using this, it’s not hard to see why many users have switched. After all, Perplexity does what AI overviews should be doing, and it does it better. And if Perplexity can give you the answers you want, why bother with Google and other websites at all?
The Next Step for SEO
This brings me back to the first question I posed at the beginning: If AI chatbots and AI-powered search results are summarizing answers for you, why would you bother with organic search results? And if users shift towards that mentality, what’s the incentive for websites to continue their work? Why put new content on the web?
Is this new era fair to independent publishers? And critically, what does this mean for the future of SEO and website traffic?
It might not happen this year, or the next, but soon enough websites will be fighting for what little organic traffic is left from Google.
So how do you stay at the top? How do you keep getting more people to visit your website?
There are really only a few things I can recommend for this new era.
Focus on Your Customers
Chatbots like Perplexity, Gemini, and Copilot all need sources to cite. Providing detailed and accurate answers increases your chances of becoming that source. We may not be able to stop chatbots from using our content, but at least you can still gain some clicks whenever they do.
Aside from that, I’m still of the mindset that there’s plenty of value in providing insightful, useful answers to the people who are looking for them. It helps you connect with your audience, proves your authority in your niche, and builds trust. Plus, high-quality content for nuanced answers is something that chatbots and AI writing tools have a hard time replicating anyway.
The recent Google search document leak, in my opinion, also proves just how important producing customer-centric content is. Their algorithm wouldn’t factor in so much click-related data for nothing.
Utilize Structured Data Markup
Making sure your content is easily understood by machines is the next step. Applying schema or structured data helps them understand and categorize web content, and increases the odds of you being featured in AI-generated results.
Diversify Your Traffic Sources
It’s becoming increasingly clear that relying solely on Google for traffic will become riskier as time goes on — even if you do end up building a strong brand presence, authority, and readership.
Diversifying traffic sources and including social media and direct engagement strategies will be essential for keeping your website alive. It’s how you’ll get featured on publications, listicles, directories, and other platforms for users to find out about you.
Key Takeaway
Did Google just effectively kill independent publishers and websites? Are we entering the “zero-click search” era? Maybe, but what’s sure is that Google search will push organic search results down in favor of what they believe is an enhanced search experience.
As Google transforms into an AI-driven search portal, what websites like mine and yours can do is shift SEO tactics, expand to other platforms, and adopt a holistic approach that emphasizes content quality, relevance, and user engagement.
SEO
How Compression Can Be Used To Detect Low Quality Pages
The concept of Compressibility as a quality signal is not widely known, but SEOs should be aware of it. Search engines can use web page compressibility to identify duplicate pages, doorway pages with similar content, and pages with repetitive keywords, making it useful knowledge for SEO.
Although the following research paper demonstrates a successful use of on-page features for detecting spam, the deliberate lack of transparency by search engines makes it difficult to say with certainty if search engines are applying this or similar techniques.
What Is Compressibility?
In computing, compressibility refers to how much a file (data) can be reduced in size while retaining essential information, typically to maximize storage space or to allow more data to be transmitted over the Internet.
TL/DR Of Compression
Compression replaces repeated words and phrases with shorter references, reducing the file size by significant margins. Search engines typically compress indexed web pages to maximize storage space, reduce bandwidth, and improve retrieval speed, among other reasons.
This is a simplified explanation of how compression works:
- Identify Patterns:
A compression algorithm scans the text to find repeated words, patterns and phrases - Shorter Codes Take Up Less Space:
The codes and symbols use less storage space then the original words and phrases, which results in a smaller file size. - Shorter References Use Less Bits:
The “code” that essentially symbolizes the replaced words and phrases uses less data than the originals.
A bonus effect of using compression is that it can also be used to identify duplicate pages, doorway pages with similar content, and pages with repetitive keywords.
Research Paper About Detecting Spam
This research paper is significant because it was authored by distinguished computer scientists known for breakthroughs in AI, distributed computing, information retrieval, and other fields.
Marc Najork
One of the co-authors of the research paper is Marc Najork, a prominent research scientist who currently holds the title of Distinguished Research Scientist at Google DeepMind. He’s a co-author of the papers for TW-BERT, has contributed research for increasing the accuracy of using implicit user feedback like clicks, and worked on creating improved AI-based information retrieval (DSI++: Updating Transformer Memory with New Documents), among many other major breakthroughs in information retrieval.
Dennis Fetterly
Another of the co-authors is Dennis Fetterly, currently a software engineer at Google. He is listed as a co-inventor in a patent for a ranking algorithm that uses links, and is known for his research in distributed computing and information retrieval.
Those are just two of the distinguished researchers listed as co-authors of the 2006 Microsoft research paper about identifying spam through on-page content features. Among the several on-page content features the research paper analyzes is compressibility, which they discovered can be used as a classifier for indicating that a web page is spammy.
Detecting Spam Web Pages Through Content Analysis
Although the research paper was authored in 2006, its findings remain relevant to today.
Then, as now, people attempted to rank hundreds or thousands of location-based web pages that were essentially duplicate content aside from city, region, or state names. Then, as now, SEOs often created web pages for search engines by excessively repeating keywords within titles, meta descriptions, headings, internal anchor text, and within the content to improve rankings.
Section 4.6 of the research paper explains:
“Some search engines give higher weight to pages containing the query keywords several times. For example, for a given query term, a page that contains it ten times may be higher ranked than a page that contains it only once. To take advantage of such engines, some spam pages replicate their content several times in an attempt to rank higher.”
The research paper explains that search engines compress web pages and use the compressed version to reference the original web page. They note that excessive amounts of redundant words results in a higher level of compressibility. So they set about testing if there’s a correlation between a high level of compressibility and spam.
They write:
“Our approach in this section to locating redundant content within a page is to compress the page; to save space and disk time, search engines often compress web pages after indexing them, but before adding them to a page cache.
…We measure the redundancy of web pages by the compression ratio, the size of the uncompressed page divided by the size of the compressed page. We used GZIP …to compress pages, a fast and effective compression algorithm.”
High Compressibility Correlates To Spam
The results of the research showed that web pages with at least a compression ratio of 4.0 tended to be low quality web pages, spam. However, the highest rates of compressibility became less consistent because there were fewer data points, making it harder to interpret.
Figure 9: Prevalence of spam relative to compressibility of page.
The researchers concluded:
“70% of all sampled pages with a compression ratio of at least 4.0 were judged to be spam.”
But they also discovered that using the compression ratio by itself still resulted in false positives, where non-spam pages were incorrectly identified as spam:
“The compression ratio heuristic described in Section 4.6 fared best, correctly identifying 660 (27.9%) of the spam pages in our collection, while misidentifying 2, 068 (12.0%) of all judged pages.
Using all of the aforementioned features, the classification accuracy after the ten-fold cross validation process is encouraging:
95.4% of our judged pages were classified correctly, while 4.6% were classified incorrectly.
More specifically, for the spam class 1, 940 out of the 2, 364 pages, were classified correctly. For the non-spam class, 14, 440 out of the 14,804 pages were classified correctly. Consequently, 788 pages were classified incorrectly.”
The next section describes an interesting discovery about how to increase the accuracy of using on-page signals for identifying spam.
Insight Into Quality Rankings
The research paper examined multiple on-page signals, including compressibility. They discovered that each individual signal (classifier) was able to find some spam but that relying on any one signal on its own resulted in flagging non-spam pages for spam, which are commonly referred to as false positive.
The researchers made an important discovery that everyone interested in SEO should know, which is that using multiple classifiers increased the accuracy of detecting spam and decreased the likelihood of false positives. Just as important, the compressibility signal only identifies one kind of spam but not the full range of spam.
The takeaway is that compressibility is a good way to identify one kind of spam but there are other kinds of spam that aren’t caught with this one signal. Other kinds of spam were not caught with the compressibility signal.
This is the part that every SEO and publisher should be aware of:
“In the previous section, we presented a number of heuristics for assaying spam web pages. That is, we measured several characteristics of web pages, and found ranges of those characteristics which correlated with a page being spam. Nevertheless, when used individually, no technique uncovers most of the spam in our data set without flagging many non-spam pages as spam.
For example, considering the compression ratio heuristic described in Section 4.6, one of our most promising methods, the average probability of spam for ratios of 4.2 and higher is 72%. But only about 1.5% of all pages fall in this range. This number is far below the 13.8% of spam pages that we identified in our data set.”
So, even though compressibility was one of the better signals for identifying spam, it still was unable to uncover the full range of spam within the dataset the researchers used to test the signals.
Combining Multiple Signals
The above results indicated that individual signals of low quality are less accurate. So they tested using multiple signals. What they discovered was that combining multiple on-page signals for detecting spam resulted in a better accuracy rate with less pages misclassified as spam.
The researchers explained that they tested the use of multiple signals:
“One way of combining our heuristic methods is to view the spam detection problem as a classification problem. In this case, we want to create a classification model (or classifier) which, given a web page, will use the page’s features jointly in order to (correctly, we hope) classify it in one of two classes: spam and non-spam.”
These are their conclusions about using multiple signals:
“We have studied various aspects of content-based spam on the web using a real-world data set from the MSNSearch crawler. We have presented a number of heuristic methods for detecting content based spam. Some of our spam detection methods are more effective than others, however when used in isolation our methods may not identify all of the spam pages. For this reason, we combined our spam-detection methods to create a highly accurate C4.5 classifier. Our classifier can correctly identify 86.2% of all spam pages, while flagging very few legitimate pages as spam.”
Key Insight:
Misidentifying “very few legitimate pages as spam” was a significant breakthrough. The important insight that everyone involved with SEO should take away from this is that one signal by itself can result in false positives. Using multiple signals increases the accuracy.
What this means is that SEO tests of isolated ranking or quality signals will not yield reliable results that can be trusted for making strategy or business decisions.
Takeaways
We don’t know for certain if compressibility is used at the search engines but it’s an easy to use signal that combined with others could be used to catch simple kinds of spam like thousands of city name doorway pages with similar content. Yet even if the search engines don’t use this signal, it does show how easy it is to catch that kind of search engine manipulation and that it’s something search engines are well able to handle today.
Here are the key points of this article to keep in mind:
- Doorway pages with duplicate content is easy to catch because they compress at a higher ratio than normal web pages.
- Groups of web pages with a compression ratio above 4.0 were predominantly spam.
- Negative quality signals used by themselves to catch spam can lead to false positives.
- In this particular test, they discovered that on-page negative quality signals only catch specific types of spam.
- When used alone, the compressibility signal only catches redundancy-type spam, fails to detect other forms of spam, and leads to false positives.
- Combing quality signals improves spam detection accuracy and reduces false positives.
- Search engines today have a higher accuracy of spam detection with the use of AI like Spam Brain.
Read the research paper, which is linked from the Google Scholar page of Marc Najork:
Detecting spam web pages through content analysis
Featured Image by Shutterstock/pathdoc
SEO
New Google Trends SEO Documentation
Google Search Central published new documentation on Google Trends, explaining how to use it for search marketing. This guide serves as an easy to understand introduction for newcomers and a helpful refresher for experienced search marketers and publishers.
The new guide has six sections:
- About Google Trends
- Tutorial on monitoring trends
- How to do keyword research with the tool
- How to prioritize content with Trends data
- How to use Google Trends for competitor research
- How to use Google Trends for analyzing brand awareness and sentiment
The section about monitoring trends advises there are two kinds of rising trends, general and specific trends, which can be useful for developing content to publish on a site.
Using the Explore tool, you can leave the search box empty and view the current rising trends worldwide or use a drop down menu to focus on trends in a specific country. Users can further filter rising trends by time periods, categories and the type of search. The results show rising trends by topic and by keywords.
To search for specific trends users just need to enter the specific queries and then filter them by country, time, categories and type of search.
The section called Content Calendar describes how to use Google Trends to understand which content topics to prioritize.
Google explains:
“Google Trends can be helpful not only to get ideas on what to write, but also to prioritize when to publish it. To help you better prioritize which topics to focus on, try to find seasonal trends in the data. With that information, you can plan ahead to have high quality content available on your site a little before people are searching for it, so that when they do, your content is ready for them.”
Read the new Google Trends documentation:
Get started with Google Trends
Featured Image by Shutterstock/Luis Molinero
SEO
All the best things about Ahrefs Evolve 2024
Hey all, I’m Rebekah and I am your Chosen One to “do a blog post for Ahrefs Evolve 2024”.
What does that entail exactly? I don’t know. In fact, Sam Oh asked me yesterday what the title of this post would be. “Is it like…Ahrefs Evolve 2024: Recap of day 1 and day 2…?”
Even as I nodded, I couldn’t get over how absolutely boring that sounded. So I’m going to do THIS instead: a curation of all the best things YOU loved about Ahrefs’ first conference, lifted directly from X.
Let’s go!
OUR HUGE SCREEN
The largest presentation screen I’ve ever seen! #ahrefsevolve pic.twitter.com/oboiMFW1TN
— Patrick Stox (@patrickstox) October 24, 2024
This is the biggest presentation screen I ever seen in my life. It’s like iMax for SEO presentations. #ahrefsevolve pic.twitter.com/sAfZ1rtePx
— Suganthan Mohanadasan (@Suganthanmn) October 24, 2024
CONFERENCE VENUE ITSELF
It was recently named the best new skyscraper in the world, by the way.
The Ahrefs conference venue feels like being in inception. #AhrefsEvolve pic.twitter.com/18Yjai1Cej
— Suganthan Mohanadasan (@Suganthanmn) October 24, 2024
I’m in Singapore for @ahrefs Evolve this week. Keen to connect with people doing interesting work on the future of search / AI #ahrefsevolve pic.twitter.com/s00UkIbxpf
— Alex Denning (@AlexDenning) October 23, 2024
OUR AMAZING SPEAKER LINEUP – SUPER INFORMATIVE, USEFUL TALKS!
A super insightful explanation of how Google Search Ranking works #ahrefsevolve pic.twitter.com/Cd1VSET2Aj
— Amanda Walls (@amandajwalls) October 24, 2024
“would I even do this if Google didn’t exist?” – what a great question to assess if you actually have the right focus when creating content amazing presentation from @amandaecking at #AhrefsEvolve pic.twitter.com/a6OKbKxwiS
— Aleyda Solis ️ (@aleyda) October 24, 2024
Attending @CyrusShepard ‘s talk on WTF is Helpful Content in Google’s algorithm at #AhrefsEvolve
“Focus on people first content”
Super relevant for content creators who want to stay ahead of the ever evolving Google search curve! #SEOTalk #SEO pic.twitter.com/KRTL13SB0g
This is the first time I am listening to @aleyda and it is really amazing. Lot of insights and actionable information.
Thank you #aleyda for power packed presentation.#AhrefsEvolve @ahrefs #seo pic.twitter.com/Xe3A9MGfrr
— Jignesh Gohel (@jigneshgohel) October 25, 2024
— Parth Suba (@parthsuba77) October 24, 2024
@thinking_slows thoughts on AI content – “it’s very good if you want to be average”.
We can do a lot better and Ryan explains how. Love it @ahrefs #AhrefsEvolve pic.twitter.com/qFqWs6QBH5
— Andy Chadwick (@digitalquokka) October 24, 2024
A super insightful explanation of how Google Search Ranking works #ahrefsevolve pic.twitter.com/Cd1VSET2Aj
— Amanda Walls (@amandajwalls) October 24, 2024
This is the first time I am listening to @aleyda and it is really amazing. Lot of insights and actionable information.
Thank you #aleyda for power packed presentation.#AhrefsEvolve @ahrefs #seo pic.twitter.com/Xe3A9MGfrr
— Jignesh Gohel (@jigneshgohel) October 25, 2024
GREAT MUSIC
First time I’ve ever Shazam’d a track during SEO conference ambience…. and the track wasn’t even Shazamable! #AhrefsEvolve @ahrefs pic.twitter.com/ZDzJOZMILt
— Lily Ray (@lilyraynyc) October 24, 2024
AMAZING GOODIES
Ahrefs Evolveきました！@ahrefs @AhrefsJP #AhrefsEvolve pic.twitter.com/33EiejQPdX
— さくらぎ (@sakuragi_ksy) October 24, 2024
Aside from the very interesting topics, what makes this conference even cooler are the ton of awesome freebies
Kudos for making all of these happen for #AhrefsEvolve @ahrefs team pic.twitter.com/DGzk5FSTN8
— Krista Melgarejo (@kimelgarejo) October 24, 2024
Content Goblin and SEO alligator party stickers are definitely going on my laptop. @ahrefs #ahrefsevolve pic.twitter.com/QBsBuY5Yix
— Patrick Stox (@patrickstox) October 24, 2024
This is one of the best swag bags I’ve received at any conference!
Either @ahrefs actually cares or the other conference swag bags aren’t up to par w Ahrefs!#AhrefsEvolve pic.twitter.com/Yc9e6wZPHn— Moses Sanchez (@SanchezMoses) October 25, 2024
SELFIE BATTLE
Some background: Tim and Sam have a challenge going on to see who can take the most number of selfies with all of you. Last I heard, Sam was winning – but there is room for a comeback yet!
Got the rare selfie with both @timsoulo and @samsgoh #AhrefsEvolve
— Bernard Huang (@bernardjhuang) October 24, 2024
THAT BELL
Everybody’s just waiting for this one.
@timsoulo @ahrefs #AhrefsEvolve pic.twitter.com/6ypWaTGDDP
— Jinbo Liang (@JinboLiang) October 24, 2024
STICKER WALL
Viva la vida, viva Seo!
Awante Argentina loco!#AhrefsEvolve pic.twitter.com/sfhbI2kWSH
— Gaston Riera. (@GastonRiera) October 24, 2024
AND, OF COURSE…ALL OF YOU!
#AhrefsEvolve let’s goooooooooooo!!! pic.twitter.com/THtdvdtUyB
— Tim Soulo (@timsoulo) October 24, 2024
–
There’s a TON more content on LinkedIn – click here – but I have limited time to get this post up and can’t quite figure out how to embed LinkedIn posts so…let’s stop here for now. I’ll keep updating as we go along!
How AI is Transforming SEO and What Website Owners Need to Know
Automattic demanded web host pay $32M annually for using WordPress trademark
5 Most Profitable Online Businesses You Can Start Today for Free!
Google Search Ranking Volatility October 26th & 27th & 23rd & 24th
Cut Costs, Not Features with This Microsoft Bundle Deal
New Google Trends SEO Documentation
WP Engine sues WordPress co-creator Mullenweg and Automattic, alleging abuse of power
How Compression Can Be Used To Detect Low Quality Pages
3 Trends That Will Change the Future of Entrepreneurship
-
AI3 days agoHow AI is Transforming SEO and What Website Owners Need to Know
-
WORDPRESS6 days agoAutomattic demanded web host pay $32M annually for using WordPress trademark
-
WORDPRESS7 days ago5 Most Profitable Online Businesses You Can Start Today for Free!
-
SEARCHENGINES6 days agoGoogle Search Ranking Volatility October 26th & 27th & 23rd & 24th
-
AFFILIATE MARKETING6 days agoCut Costs, Not Features with This Microsoft Bundle Deal
-
SEO7 days agoNew Google Trends SEO Documentation
-
WORDPRESS6 days agoWP Engine sues WordPress co-creator Mullenweg and Automattic, alleging abuse of power
-
SEO6 days agoHow Compression Can Be Used To Detect Low Quality Pages