Google’s Gary Illyes said on the last Search Off The Record Podcast that Google in 2022 is looking to make crawling more efficient and environmentally friendly. And while Google is investigating ways to do that with IndexNow, Gary said it wont be done in a way “that people expect” with that.
Here is the podcast embed, this starts at about 2:30 into the podcast:
He said one way of doing this is for Google to look at the refresh crawls and see about a way of crawling some pages and URLs less often for those refresh crawls. Discovery calls are for new URLs that Google has yet to index but refresh crawls are recrawls of URLs Google has already crawled before but to see if the page has been updated and has new signals. So Google might crawl old URLs less often or maybe more efficiently.
Gary said “how can we reduce even more Googlebots and other crawlers, Google crawlers’ footprint on the Internet, on the environment. And then, if you think about it, one thing that we do and we might not need to do that much is refresh crawls. Which means that once we discovered a document, a URL, then we go, we crawl it, and then, eventually, we are going to go back and revisit that URL. That is a refresh crawl. And then every single time we go back to that one URL, that will always be a refresh crawl. Now, how often do we need to go back to that URL?”
Gary added, “you could say that, for example, if you take the CNN or Wall Street Journal homepage, which is changing every five seconds, then we do need to go back very often. But then the About page of either of these news outlets, they don’t change too often. So you don’t have to go back there that much. And often, we can’t estimate this well, and we definitely have room for improvement there on refresh crawls, because sometimes, it just seems wasteful that we are hitting the same URL over and over again. Sometimes we are hitting 404 pages, for example, for no good reason or no apparent reason. And all these things are basically stuff that we could improve on and then reduce our footprint even more.”
Gary then commented on IndexNow saying they are experimenting with it but if Google does do something with it, it won’t be in the format we are thinking about. He said “but IndexNow could be something that might be useful, and we are running some experiments to see if that’s the case. Probably, it’s not going to be in the form that people expect, but we’ll see. But I can definitely see that it might prove useful in some cases at least.”
They spoke about XML sitemaps and the issues with it for improving discover and crawling. Basically people change the last mod date in the XML sitemap when the URL has not changed. So Gary said “we are just not going to use it” because people don’t generate these sitemaps correctly and accurately.
Crawling and Ranking
And there is a misconception that crawling more leads to more and higher rankings, that is not true Gary and John said. John said “So I guess that’s kind of also a misconception that people have in that they think if a page gets crawled more, it’ll get ranked more. Is that correct that that’s a misconception, or is that actually true?” Gary said “it’s a misconception.”
Google might be more transparent about how crawling works in 2022, so be ready for that.
Forum discussion at Twitter.
Offline Wednesday For Yom Kippur 5783
This is just a programming note that I will be 100% offline starting Tuesday night (October 4th) through Wednesday night (October 5th) for Yom Kippur. I am not scheduling any stories on Yom Kippur, nothing will be posted. We have a white theme going up today around 3pm ET through all of Yom Kippur (as a reminder, so check it out).
Feel free to check out the archives or catch up with the weekly videos, check out the new search vlogs or just browse the search pics. Oh, and if you have nothing to do, feel free to subscribe to my YouTube channel. 🙂
Yom Kippur is one of the most significant days of the year in the Jewish religion. We basically pray and fast (and sleep) the whole 25-hour period. You can learn more about the day on Wikipedia if you are interested. This year it will be more interesting as it might rain and the outside, socially distant set up might be a bit wet…
For my fellow Jews – have an easy and meaningful fast. If I have done anything to insult, upset, or embarrass anyone – I am sincerely sorry. I ask for your forgiveness.
We will resume on Thursday, after Yom Kippur.
Meta Announces New Privacy-Focused Ad Targeting Solutions, Improvements in Automated Targeting
Daily SEO Fix: Competitive Link Research
Let’s Sing ABBA: When Karaoke Meets Disco
Google’s Helpful Content & Core Updates [Webinar]
Rights holders got Google to remove 6 billion links from Search over 10 years
Google to pay $85 million to settle claims it deceptively tracked users
Instagram Tests Out New Ad Options, Including Explore Placement and Interactive AR Displays
Remaking Dead Space, Available for Pre-Order Today
Workforce Trends You Need To Know About in 2022 [New Data]
Offline Wednesday For Yom Kippur 5783
Explore the Path to Digital Future: Interconnect, Integrate and Innovate
Google Updates Documentation On Meta Descriptions
Daily Search Forum Recap: September 5, 2022
7 Tips For Creating Instagram Story Ads That Convert
How To Launch Your First Google Ads Remarketing Campaign
Microsoft Advertising Gains Pinterest Import, More Google Imports, & More
Google Again Says Spikes In Crawling Activity Not A Sign Of The Helpful Content Update Rollout
Daily Search Forum Recap: September 30, 2022
The Ultimate Timeline of Google Algorithm Updates (+ Recommendations)
Confusion Over Google Search Console’s HTTPS Is Invalid And Might Prevent It From Being Indexed