SEARCHENGINES

Google Deduplication On Top Stories From Web Search Results

Published

2 years ago

January 20, 2022

Google Deduplication On Top Stories From Web Search Results

We know Google has deduplication efforts for when a featured snippet is displayed so that it might remove the snippet from the main web results. Google may do the same, in some cases, for when a URL is listed in the top stories section and then not show it in the main web results.

Google in some cases may be removing the web search result snippet when it shows that same result in the top stories section of the Google search results.

Update: Here is how deduping works for top stories:

Just to cap off with the further clarification I promised, we deduplicate a link from web results if a link appears as the first link in Top Stories and if the Top Stories box appears before web results. If it comes after, we don’t. And again, it’s something we’re reviewing.
— Danny Sullivan (@dannysullivan) January 19, 2022

Rest of the story continued but the update is in the tweet above.

Danny Sullivan of Google said this on Twitter after a complaint from The Verge. Danny said “it’s showing in Top Stories, it is getting deduplicated from the rest of the page. Deduplication can often be useful. Doing this search in the way that user might by using solution-seeking terms rather than unusual terms in the headline, there you are at the top in Top Stories plus deduplicating means there’s more variety from other publications. In searches like that, our systems also are going to generally seek to show the most helpful, reliable info they can. That’s why you don’t see a lot of duplicates of your article showing. Duplicates certainly exist, but it isn’t that helpful to show them. That leads to headline-oriented searches. As I said before, that’s super common among authors. I used to do it all the time, myself. But headline searches contain typically contain a lot of terms, so our systems shift to return pages that have those terms. his means authors are more likely to find duplicates, even though for typical searches that readers would do, these are unlikely to appear. But our deduplication feature may still kick in even for these, as was happening in this case. As I said, deduplication can be helpful. But we also understand the concern this might be raising. We’ve been doing this with Top Stories since last May, but we’re going to revisit this to see if we should continue or perhaps make other changes. Also, I’m still checking, but I believe this deduplication is especially unique in that it only happens with Top Stories if there’s a single story shown or perhaps only for the very first story shown.”

So you can see, Danny is giving this as an excuse for why other publications are in the web results and not The Verge for that query. But as you can also see, he seems to explain that sometimes it does not work this way.

Deduplication can often be useful. Doing this search in the way that user might by using solution-seeking terms rather than unusual terms in the headline, there you are at the top in Top Stories plus deduplicating means there’s more variety from other publications…. pic.twitter.com/638IAZLWIV
— Danny Sullivan (@dannysullivan) January 18, 2022

The interesting part is it does seem that Google is indeed deduping here when it comes to top stories.

The funny thing, hours before this, News SEO Barry Adams did a whole Twitter thread on this:

This is not the case. It can be easily disproven; look for recent articles published on a major news outlet (BBC, The Guardian, NYT, etc). Search for relevant keywords relating to that article.
Chances are, you’ll see the article in the news box as well as on the regular SERP. pic.twitter.com/3u7HH7jppq
Advertisement

— Barry Adams 📰 (@badams) January 18, 2022

So a newly published article has had very little opportunity to rank in regular SERPs. It may show in Top Stories boxes, but ranking in regular SERPs could take days or weeks or even months.
Plus, authoritative publishers have an edge here, because, well, they’re authoritative.
— Barry Adams 📰 (@badams) January 18, 2022

In summary; there is no ‘news’ filter on Google SERPs. Articles can and do show up in both Top Stories and regular SERPs.
But often this doesn’t happen due to issues relating to speed, intent, and competition.
/end
— Barry Adams 📰 (@badams) January 18, 2022
Advertisement

So I pointed out to Barry Adams the tweet from Danny and his response:

Colour me shocked, Danny being wrong on something. 🤪
— Barry Adams 📰 (@badams) January 18, 2022

So I did some sample searches and Google does seem to remove a recent story from this site from the top stories and show it in web results, and at the same time, remove those stories in top stories from the web results. Notice SER is not in top stories here but is in the web results and at the same time SEL and SEJ is in the top stories but not in the web results (click to enlarge):

But yes, the story does show in the news tab:

But sometimes Google is not deduplicating these top stories, a query for [the hidden resignation] shows the Business Insider story in top stories in some browsers twice and some once (thanks Glenn Gabe for the query and discussing it with me):

Deduped:

Not Deduped:

Here is another example from Glenn:

Another good example of a url showing up in Top Stories that also ranks in the 10 blue links. But from what Danny explained yesterday, Google’s deduplication system for Top Stories is nuanced. So it’s not going to happen all of the time (which makes sense based on other queries). pic.twitter.com/ywiwiQnEfw
— Glenn Gabe (@glenngabe) January 19, 2022

So it is not clear when and why Google might deduplicate a URL from showing in web search when it also shows in top stories. It might be a timing issue or something else.

Danny Sullivan did say “Also, I’m still checking, but I believe this deduplication is especially unique in that it only happens with Top Stories if there’s a single story shown or perhaps only for the very first story shown.”

Also, I’m still checking, but I believe this deduplication is especially unique in that it only happens with Top Stories if there’s a single story shown or perhaps only for the very first story shown.
— Danny Sullivan (@dannysullivan) January 18, 2022
Advertisement

We hope to get some clarity soon.

Forum discussion at Twitter.

Related Topics:Deduplication Google Results Search stories top Web

Up Next

Recipe Schema Markup Drops Ranges For Cook, Prep & Total Time

Don't Miss

Daily Search Forum Recap: January 19, 2022

Click to comment

You must be logged in to post a comment Login

SEARCHENGINES

Daily Search Forum Recap: April 19, 2024

Published

6 hours ago

April 20, 2024

Entireweb News Bot

Here is a recap of what happened in the search forums today, through the eyes of the Search Engine Roundtable and other search forums on the web.

Google is testing more ad formats for its search ads. BingBot is testing new compression methods. Google said it is unlikely your rankings would drop just because you have two or more sites. Google says the indexing API might work for unsupported formats. Google said hyphenated domains are not bad by themselves. And I posted the weekly SEO video recap.

Search Engine Roundtable Stories:

Search News Buzz Video Recap: Google Core Update Flux, AdSense Ad Intent, California Link Tax & More

The Google March 2024 core update is still rolling out, almost 6 weeks now, and we saw two shifts of ranking volatility, both mid-week and the weekend before. Google’s Danny Sullivan went on the defensive on search quality and forum listings…

Google Tests More Google Ad Card Formats

Google is testing another ad format, another card like format, for its search ads. We previously saw the vertical and tall ad cards, then the carousel horizontal card format and now here are just plain horizontal ad cards.

Google: It’s Unlikely Your Rankings Dropped Because You Have Two Websites

Google was asked if having two websites and Google finding out about them would cause the websites to drop in rankings. John Mueller from Google said that is “not likely,” adding that many people have several websites and that is fine.

Google: Indexing API May Work For Unsupported Content But…

Gary Illyes from Google said in the recent Google SEO office hours the Google Indexing API may work for unsupported content formats, but that you shouldn’t be “surprised if suddenly it stopped working for unsupported verticals overnight.”

Google: Are Hyphenated Domains Are Bad For Google Ranking?

The topic of using hyphens in domain names is one of the oldest SEO topics we have in this space. Heck, in 2004, SEOs said there was a specific Google ranking algorithm that filtered out domains with hyphens in them (Google did not by the way).

Bingbot To Test Zstd Compression After Fully Gaining Full Brotli Compression

Fabrice Canel from Microsoft announced that BingBot now fully supports Brotli compression and will soon be testing zstd Zstandard compression, a lossless data compression, for its crawler.

Google Cloud Socks

Google loves to print socks for its various swag and here are two variations of socks from the Google Cloud team. I like the white socks because they look more retro but here are nicer versions as well.

Other Great Search Threads:

From a practical POV, you can specify your prices using structured data on the page, with a merchant center feed (which many ecommerce systems can make for you) or manually in merchant center., John Mueller on X

I passed this on. I don’t think there’s anything special going on here, but the feedback on this being confusing is good to pass on., John Mueller on X

If you have older reports that are still helpful, you’ll just need to click the View “button” and access ones you want to remain active., AdsLiaison on X

The way I see this is more that there are things which can be recognized & ignored, but a site might still be appropriate to show (I don’t know this site/query). Folks get bad SEO advice, and if that can be neutralized granularly without broader effects, , John Mueller on X

Search Engine Land Stories:

Feedback:

Have feedback on this daily recap; let me know on Twitter @rustybrick or @seroundtable, on Threads, Mastodon and Bluesky and you can follow us on Facebook and on Google News and make sure to subscribe to the YouTube channel, Apple Podcasts, Spotify, Google Podcasts or just contact us the old fashion way.

SEARCHENGINES

Daily Search Forum Recap: April 18, 2024

Published

1 day ago

April 19, 2024

Entireweb News Bot

Here is a recap of what happened in the search forums today, through the eyes of the Search Engine Roundtable and other search forums on the web.

Google Ads notified advertisers that some ad customizers will stop working soon. Google discontinued the limited use video carousel markup. Google Maps has new directions and travel features, plus a bunch of new EV features. Google Notes on Search might not end in May 2024 despite what people suggest. Google Business Profiles asked businesses to register their defibrillators, their AEDs.

Search Engine Roundtable Stories:

Google Notes On Search Won’t Necessarily Go Away In May

There have been some people noticing that the Google Notes On Search labs experiment has an end date of May 2024 and thus they are expecting Notes on Search to be turned off by then. Just because it has that end date listed, it does not mean the labs experiment will end on that date.

Google Maps Releases New Directions, Travel & EV Features

Google has announced a number of new travel features with Google Maps and Travel around driving alternatives, trains and buses, travel impact mode and then some new EV driving direction features. Google says these features are to give you more “sustainable choices.”

Google Ads Reminds Advertisers Some Ad Customizers Will Go Away May 31st

On August 31, 2021, Google notified advertisers it would drop support for expanded text ads for responsive search ads. Google is now reminding advertisers that Ad customizers for text ads, expanded text ads and Dynamic Search Ads will stop serving after May 31, 2024.

Google Drops Video Carousel Markup

Google has discontinued its support for video carousel markup and has thus removed it from its video structured data help documentation. Initially, Google tested video carousel on a limited number of sites and is now saying that it “ultimately found that it wasn’t useful for the ecosystem at scale.”

Google Business Profiles Register Your Defibrillator (AED)

Google sent out emails a couple of days ago to businesses asking them to register their Automated External Defibrillator (AED) with Google Business Profiles. Google said, “you could save lives” if you do it.

Google NYC Earthquake Reaction Video

A couple of weeks ago we had an earthquake in the New York region and I finally found a video from the Google NYC office of the reaction of some Googlers while the earthquake happened. These Googlers were recording a video and felt it but were unsure if that was a real earthquake.

Other Great Search Threads:

Search Engine Land Stories:

Feedback:

SEARCHENGINES

Daily Search Forum Recap: April 17, 2024

Published

2 days ago

April 18, 2024

Entireweb News Bot

Here is a recap of what happened in the search forums today, through the eyes of the Search Engine Roundtable and other search forums on the web.

The Google March core update still is causing volatility and it is deepening. Google may make changes to the crawl stats report in Search Console. Google Search Console now shows unused ownership tokens. Google Maps will drop the review drafts. Google Add added generative AI for image generate for Demand Gen campaigns. Google Search is testing more sitelinks designs.

Search Engine Roundtable Stories:

Deepening Google Core Ranking Volatility Hits Yesterday & Today

I know we just reported on more volatility this Monday, possibly related to the Google March 2024 core update, but we are seeing even more volatility over the past 24 hours related to this update. This volatility shows deepening movement for those who were hit, which is incredibly sad in many cases.

Google Search Console Adds Unused Ownership Tokens

Google Search Console has a new security feature under user and permission management to help you manage your unused ownership tokens. Basically, it helps you manage who has permissions to your Search Console profiles and who should not, maybe some legacy verifications.

Google: Should We Explain The Crawl Stats Report in Search Console

Martin Splitt from Google posted a one-line question asking, “Should we do something to explain the crawl stats report in GSC?” This was posted on Mastodon, which has a small audience, but the responses were pretty good.

New Google Ads AI Generated Image Tool For Demand Gen Campaigns

Google Ads announced that Demand Gen campaigns are getting generative AI tools to create image assets. This is rolling out to advertisers around the world in English with more languages to come later this year, Google said.

Google Tests Horizontal Lines For Sitelinks & People Also Ask

Google Search is testing horizontal lines under the sitelinks and the people also ask elements in the search results. This is a long horizontal line that goes across the while search result snippet.

Google Maps No Longer Will Support Draft Reviews

Google Maps will soon stop supporting drafting reviews for local business listings and Google Business Profiles. Google said that starting July 16, 2024, review drafts will no longer be supported.

Doogler In Google Play Area

Some of you love photos of dogs, so here is another one of a Doogler, a Google dog, in a play area, at the Google office. This was shared by the Life at Google Twitter account.