Connect with us


Google Looking To Make Crawling More Efficient & Environmental Friendly



Google Looking To Make Crawling More Efficient & Environmental Friendly

Google’s Gary Illyes said on the last Search Off The Record Podcast that Google in 2022 is looking to make crawling more efficient and environmentally friendly. And while Google is investigating ways to do that with IndexNow, Gary said it wont be done in a way “that people expect” with that.

Here is the podcast embed, this starts at about 2:30 into the podcast:

Refresh Crawls

He said one way of doing this is for Google to look at the refresh crawls and see about a way of crawling some pages and URLs less often for those refresh crawls. Discovery calls are for new URLs that Google has yet to index but refresh crawls are recrawls of URLs Google has already crawled before but to see if the page has been updated and has new signals. So Google might crawl old URLs less often or maybe more efficiently.

Gary said “how can we reduce even more Googlebots and other crawlers, Google crawlers’ footprint on the Internet, on the environment. And then, if you think about it, one thing that we do and we might not need to do that much is refresh crawls. Which means that once we discovered a document, a URL, then we go, we crawl it, and then, eventually, we are going to go back and revisit that URL. That is a refresh crawl. And then every single time we go back to that one URL, that will always be a refresh crawl. Now, how often do we need to go back to that URL?”

Gary added, “you could say that, for example, if you take the CNN or Wall Street Journal homepage, which is changing every five seconds, then we do need to go back very often. But then the About page of either of these news outlets, they don’t change too often. So you don’t have to go back there that much. And often, we can’t estimate this well, and we definitely have room for improvement there on refresh crawls, because sometimes, it just seems wasteful that we are hitting the same URL over and over again. Sometimes we are hitting 404 pages, for example, for no good reason or no apparent reason. And all these things are basically stuff that we could improve on and then reduce our footprint even more.”


Gary then commented on IndexNow saying they are experimenting with it but if Google does do something with it, it won’t be in the format we are thinking about. He said “but IndexNow could be something that might be useful, and we are running some experiments to see if that’s the case. Probably, it’s not going to be in the form that people expect, but we’ll see. But I can definitely see that it might prove useful in some cases at least.”


XML Sitemaps

They spoke about XML sitemaps and the issues with it for improving discover and crawling. Basically people change the last mod date in the XML sitemap when the URL has not changed. So Gary said “we are just not going to use it” because people don’t generate these sitemaps correctly and accurately.

Crawling and Ranking

And there is a misconception that crawling more leads to more and higher rankings, that is not true Gary and John said. John said “So I guess that’s kind of also a misconception that people have in that they think if a page gets crawled more, it’ll get ranked more. Is that correct that that’s a misconception, or is that actually true?” Gary said “it’s a misconception.”

Google might be more transparent about how crawling works in 2022, so be ready for that.

Forum discussion at Twitter.

Source link


Offline Wednesday For Yom Kippur 5783



Yom Kippur 5783

This is just a programming note that I will be 100% offline starting Tuesday night (October 4th) through Wednesday night (October 5th) for Yom Kippur. I am not scheduling any stories on Yom Kippur, nothing will be posted. We have a white theme going up today around 3pm ET through all of Yom Kippur (as a reminder, so check it out).

Feel free to check out the archives or catch up with the weekly videos, check out the new search vlogs or just browse the search pics. Oh, and if you have nothing to do, feel free to subscribe to my YouTube channel. 🙂

Yom Kippur is one of the most significant days of the year in the Jewish religion. We basically pray and fast (and sleep) the whole 25-hour period. You can learn more about the day on Wikipedia if you are interested. This year it will be more interesting as it might rain and the outside, socially distant set up might be a bit wet…

For my fellow Jews – have an easy and meaningful fast. If I have done anything to insult, upset, or embarrass anyone – I am sincerely sorry. I ask for your forgiveness.

We will resume on Thursday, after Yom Kippur.


Continue Reading

Subscribe To our Newsletter
We promise not to spam you. Unsubscribe at any time.
Invalid email address