SEO
Google Considers Reducing Webpage Crawl Rate

Google may reduce the frequency of crawling webpages as it grows more conscious of the sustainability of crawling and indexing.
This topic is discussed by Google’s Search Relations team, which is made up of John Mueller, Martin Splitt, and Gary Illyes.
Together, in the latest episode of the Search Off the Record podcast, they discuss what to expect from Google in 2022 and beyond.
Among the topics they address is crawling and indexing, which SEO professionals and website owners say they’ve seen less of over the past year.
That’s going to be a key focus for Google this year as it aims to make crawling more sustainable by conserving computing resources.
Here’s what that will mean for your website and its performance in search results.
Sustainability Of Crawling & Indexing
Since Googlebot crawling and indexing happens virtually, it’s not something you may think has an impact on the environment.
Illyes brings this issue to attention when he says computing isn’t sustainable in general:
“… what I mean is that computing, in general, is not really sustainable. And if you think of Bitcoin, for example, Bitcoin mining has real impact on the environment that you can actually measure, especially if the electricity is coming from coal plants or other less sustainable plants.
We are carbon-free, since I don’t even know, 2007 or something, 2009, but it doesn’t mean that we can’t reduce even more our footprint on the environment. And crawling is one of those things that early on, we could chop off some low-hanging fruits.”
The low-hanging fruits, in this instance, refers to unnecessary web crawling. Such as crawling webpages that haven’t had any recent updates.
How Will Google Make Crawling More Sustainable?
Illyes goes on to explain that web crawling can be made more sustainable by cutting down on refresh crawls.
There are two types of Googlebot crawling: crawling to discover new content and crawling to refresh existing content.
Google is considering scaling back on crawling to refresh content.
Illyes continues:
“… one thing that we do, and we might not need to do that much, is refresh crawls. Which means that once we discovered a document, a URL, then we go, we crawl it, and then, eventually, we are going to go back and revisit that URL. That is a refresh crawl.
And then every single time we go back to that one URL, that will always be a refresh crawl. Now, how often do we need to go back to that URL?”
He goes on to give an example of certain websites that warrant a significant number of refresh crawls for some parts of the site but not others.
A website like Wall Street Journal is constantly updating its homepage with new content, so it deserves a lot of refresh crawls.
However, WSJ is not likely updating its About page as frequently, so Google doesn’t need to keep doing refresh crawls on those types of pages.
“So you don’t have to go back there that much. And often, we can’t estimate this well, and we definitely have room for improvement there on refresh crawls. Because sometimes it just seems wasteful that we are hitting the same URL over and over again.
Sometimes we are hitting 404 pages, for example, for no good reason or no apparent reason. And all these things are basically stuff that we could improve on and then reduce our footprint even more.”
If Google were to cut down on refresh crawls, which is not 100% confirmed, here’s the impact that could have on your website.
What Does A Reduction In Crawl Rate Mean For Your Website?
There’s a belief out there that a high crawl rate is a positive SEO signal, even if you’re not updating your content as often as Google is crawling it.
That’s a misconception, Illyes says, as content will not necessarily rank better because it gets crawled more.
Mueller:
“So I guess that’s kind of also a misconception that people have inthat they think if a page gets crawled more, it’ll get ranked more. Is that correct that that’s a misconception, or is that actually true?”
Illyes:
“It’s a misconception.”
Mueller:
“OK, so no need to try to force something to be re-crawled if it doesn’t actually change. It’s not going to rank better.”
Again, it’s not confirmed that Google will reduce refresh crawls, but it’s an idea the team is actively considering.
If Google follows through on this idea it won’t be a bad thing for your website. More crawling does not mean better rankings.
Moreover, the idea is to learn which pages need refresh crawls and which pages do not. That means the pages you change more often will continue to be refreshed and updated in search results.
For more details on how Google plans to pull this off, listen to the full discussion in the podcast below (starting at the 2:40 mark):
Featured Image: Alena Veasey/Shutterstock
SEO
Google To Curb Microtargeting In Consumer Finance Ads

Google is updating its policy limiting personalized advertising to include more restrictions on ads related to consumer financial products and services.
Google’s personalized ads policy prohibits targeting users based on sensitive categories like race, religion, or sexual orientation.
Over the years, Google has continued updating the policy to introduce new limitations. The latest update to restrict consumer finance ads is part of Google’s ongoing efforts to refine its ad targeting practices.
What’s Changing?
Google will update its personalized ads policy in February 2024 to prevent advertisers from targeting audiences for credit and banking ads based on sensitive factors like gender, age, parental status, marital status, or zip code.
Google’s current policy prohibiting “Credit in personalized ads” will be renamed “Consumer finance in personalized ads” under the changes.
Google’s new policy will state:
“In the United States and Canada, the following sensitive interest categories cannot be targeted to audiences based on gender, age, parental status, marital status, or ZIP code.
Offers relating to credit or products or services related to credit lending, banking products and services, or certain financial planning and management services.”
Google provided examples, including “credit cards and loans including home loans, car loans, appliance loans, short-term loans,” as well as “banking and checking accounts” and “debt management products.”
When Does The New Policy Take Effect?
The updated limitations on personalized advertising will take effect on February 28, 2024, with full enforcement expected within six weeks.
Google said advertisers in violation will receive a warning at least seven days before any account suspension.
According to Google, the policy change aims to protect users’ privacy better and prevent discrimination in financial services advertising.
However, the company will still allow generalized ads for credit and banking products that do not use sensitive personal data for targeting.
What Do Advertisers Need To Do?
Google will begin enforcing the updated restrictions in late February 2024 but advises advertisers to review their campaigns for compliance issues sooner.
Advertisers should carefully check their ad targeting settings, remove improper personalization based on sensitive categories, and adhere to the revised policy requirements.
Failure to follow the rules could lead to account suspension after an initial warning. Google will work with advertisers to ensure a smooth transition during the ramp-up period over the next six months.
Featured Image: SurfsUp/Shutterstock
SEO
Google Discusses Fixing 404 Errors From Inbound Links

Google’s John Mueller responded to a thread in Reddit about finding and fixing inbound broken links, offering a nuanced insight that some broken links are worth finding and fixing and others are not.
Reddit Question About Inbound Broken Links
Someone asked on Reddit if there’s a way to find broken links for free.
This is the question:
“Is it possible to locate broken links in a similar manner to identifying expired domain names?”
The person asking the question clarified if this was a question about an inbound broken link from an external site.
John Mueller Explains How To Find 404 Errors To Fix
John Mueller responded:
“If you want to see which links to your website are broken & “relevant”, you can look at the analytics of your 404 page and check the referrers there, filtering out your domain.
This brings up those which actually get traffic, which is probably a good proxy.
If you have access to your server logs, you could get it in a bit more detail + see which ones search engine bots crawl.
It’s a bit of technical work, but no external tools needed, and likely a better estimation of what’s useful to fix/redirect.”
In his response, John Mueller answers the question on how to find 404 responses caused by broken inbound links and identify what’s “useful to fix” or to “redirect.”
Mueller Advises On When Not To “Fix” 404 Pages
John Mueller next offered advice on when it doesn’t make sense to not fix a 404 page.
Mueller explained:
“Keep in mind that you don’t have to fix 404 pages, having things go away is normal & fine.
The SEO ‘value’ of bringing a 404 back is probably less than the work you put into it.”
Some 404s Should Be Fixed And Some Don’t Need Fixing
John Mueller said that there are situations where a 404 error generated from an inbound link is easy to fix and suggested ways to find those errors and fix them.
Mueller also said that there are some cases where it’s basically a waste of time.
What wasn’t mentioned was what the difference was between the two and this may have caused some confusion.
Inbound Broken Links To Existing Webpages
There are times when another sites links into your site but uses the wrong URL. Traffic from the broken link on the outside site will generate a 404 response code on your site.
These kinds of links are easy to find and useful to fix.
There are other situations when an outside site will link to the correct webpage but the webpage URL changed and the 301 redirect is missing.
Those kinds of inbound broken links are also easy to find and useful to fix. If in doubt, read our guide on when to redirect URLs.
In both of those cases the inbound broken links to the existing webpages will generate a 404 response and this will show up in server logs, Google Search Console and in plugins like the Redirection WordPress plugin.
If the site is on WordPress and it’s using the Redirection plugin, identifying the problem is easy because the Redirection plugin offers a report of all 404 responses with all the necessary information for diagnosing and fixing the problem.
In the case where the Redirection plugin isn’t used one can also hand code an .htaccess rule for handling the redirect.
Lastly, one can contact the other website that’s generating the broken link and ask them to fix it. There’s always a small chance that the other site might decide to remove the link altogether. So it might be easier and faster to just fix it on your side.
Whichever approach is taken to fix the external inbound broken link, finding and fixing these issues is relatively simple.
Inbound Broken Links To Removed Pages
There are other situations where an old webpage was removed for a legitimate reason, like an event passed or a service is no longer offered.
In that case it makes sense to just show a 404 response code because that’s one of the reasons why a 404 response should be shown. It’s not a bad thing to show a 404 response.
Some people might want to get some value from the inbound link and create a new webpage to stand in for the missing page.
But that might not be useful because the link is for something that is irrelevant and of no use because the reason for the page no longer exists.
Even if you create a new reason, it’s possible that some of that link equity might flow to the page but it’s useless because the topic of that inbound link is totally irrelevant to anyting but the expired reason.
Redirecting the missing page to the home page is a strategy that some people use to benefit from the link to a page that no longer exists. But Google treats those links as Soft 404s, which then passes no benefit.
These are the cases that John Mueller was probably referring to when he said:
“…you don’t have to fix 404 pages, having things go away is normal & fine.
The SEO ‘value’ of bringing a 404 back is probably less than the work you put into it.”
Mueller is right, there are some pages that should be gone and totally removed from a website and the proper server response for those pages should be a 404 error response.
SEO
Site Quality Is Simpler Than People Think

Google’s John Mueller, Martin Splitt and Gary Illyes discussed site quality in a recent podcast, explaining the different ways of thinking about site quality and at one point saying it’s not rocket science. The discussion suggests that site quality could be simpler than most people know.
Site Quality Is Not Rocket Science
The first point they touched on is to recommend reading site quality documentation, insisting that site quality is not especially difficult to understand.
Gary Illyes said:
“So I would go to a search engine’s documentation.
Most of them have some documentation about how they function and just try to figure out where your content might be failing or where your page might be failing because honestly, okay, this is patronizing, but it’s not rocket science.”
No Tools For Site Quality – What To Do?
Gary acknowledged that there’s no tool for diagnosing site quality, not in the same way there are tools for objectively detecting technical issues.
The traffic metrics that show a downward movement don’t explain why, they just show that something changed.
Gary Illyes:
“I found the up-down metric completely useless because you still have to figure out what’s wrong with it or why people didn’t like it.
And then you’re like, “This is a perfectly good page. I wrote it, I know that it’s perfect.”
And then people, or I don’t know, like 99.7% of people are downvoting it. And you’re like, ‘Why?’”
Martin Splitt
“And I think that’s another thing.
How do I spot, I wrote the page, so clearly it is perfect and helpful and useful and amazing, but then people disagree, as you say.
How do you think about that? What do you do then?
How can I make my content more helpful, better, more useful? I don’t know.
…There’s all these tools that I can just look at and I see that something’s good or something’s bad.
But for quality, how do I go about that?”
Gary Illyes
“What if quality is actually simpler than at least most people think?
…What if it’s about writing the thing that will help people achieve whatever they need to achieve when they come to the page? And that’s it.”
Martin Splitt asked if Gary was talking about reviewing the page from the perspective of the user.
Illyes answered:
“No, we are reframing.”
Reframing generally means to think about the problem differently.
Gary’s example is to reframe the problem as whether the page delivers what it says it’s going to deliver (like helping users achieve X,Y,Z).
Something I see a lot with content is that the topic being targeted (for example, queries about how to catch a trout) isn’t matched by the content (which might actually be about tools for catching trout) which is not what the site visitor wants to achieve.
Quality In Terms Of Adding Value
There are different kinds of things that relate to site and page quality and in the next part of the podcast John Mueller and Gary Illyes discuss the issue about adding something of value.
Adding something of value came up in the context of where the SERPs offer good answers from websites that people not only enjoy but they expect to see those sites as answers for those queries.
You can tell when users expect specific sites for individual search queries when Google Suggests shows the brand name and the keyword.
That’s a clue that probably a lot of people are turning keywords into branded searches, which signals to Google what people want to see.
So, the problem of quality in those situations isn’t about being relevant for a query with the perfect answer.
For these situations, like for competitive queries, it’s not enough to be relevant or have the perfect answer.
John Mueller explains:
“The one thing I sometimes run into when talking with people is that they’ll be like, “Well, I feel I need to make this page.”
And I made this page for users in air quotes…
But then when I look at the search results, it’s like 9,000 other people also made this page.
It’s like, is this really adding value to the Internet?
And that’s sometimes kind of a weird discussion to have.
It’s like, ‘Well, it’s a good page, but who needs it?’
There are so many other versions of this page already, and people are happy with those.”
This is the type of situation where competitive analysis to “reverse engineer” the SERPs works against the SEO.
It’s stale because using what’s in the SERPs as a template for what to do rank is feeding Google what it already has.
It’s like, as an example, let’s represent the site ranked in Google with a baseline of the number zero.
Let’s imagine everything in the SERPs has a baseline of zero. Less than zero is poor quality. Higher than zero is higher quality.
Zero is not better than zero, it’s just zero.
The SEOs who think they’re reverse engineering Google by copying entities, copying topics, they’re really just achieving an imperfect score of zero.
So, according to Mueller, Google responds with, “it’s a good page, but who needs it?”
What Google is looking for in this situation is not the baseline of what’s already in the SERPs, zero.
According to Mueller, they’re looking for something that’s not the same as the baseline.
So in my analogy, Google is looking for something above the baseline of what is already in the SERPs, a number greater than zero, which is a one.
You can’t add value by feeding Google back what’s already there. And you can’t add value by doing the same thing ten times bigger. It’s still the same thing.
Breaking Into The SERPs By The Side Door
Gary Illyes next discusses a way to break into a tough SERP, saying the way to do it is indirectly.
This is an old strategy but a good one that still works today.
So, rather than bringing a knife to a gunfight, Gary Illyes suggests choosing more realistic battles to compete in.
Gary continued the conversation about competing in tough SERPs.
He said:
“…this also is kind of related to the age-old topic that if you are a new site, then how can you break into your niche?
I think on today’s Internet, like back when I was doing ‘SEO’, it was already hard.
For certain topics or niches, it was absolutely a nightmare, like ….mesothelioma….
That was just impossible to break into. Legal topics, it was impossible to break into.
And I think by now, we have so much content on the Internet that there’s a very large number of topics where it is like 15 years ago or 20 years ago, that mesothelioma topic, where it was impossible to break into.
…I remember Matt Cutts, former head of Web Spam, …he was doing these videos.
And in one of the videos, he said try to offer something unique or your own perspective to the thing that you are writing about.
Then the number of perspective or available perspectives, free perspectives, is probably already gone.
But if you find a niche where people are not talking too much about, then suddenly, it’s much easier to break into.
So basically, this is me saying that you can break into most niches if you know what you are doing and if you are actually trying to help people.”
What Illyes is suggesting as a direction is to “know what you are doing and if you are actually trying to help people.”
That’s one of my secrets to staying one step ahead in SEO.
For example, before the reviews update, before Google added Experience to E-A-T, I was telling clients privately to do that for their review pages and I told them to keep it a secret, because I knew I had it dialed in.
I’m not psychic, I was just looking at what Google wants to rank and I figured it out several years before the reviews update that you need to have original photos, you need to have hands-on experience with the reviewed product, etc.
Gary’s right when he advises to look at the problem from the perspective of “trying to help people.”
He next followed up with this idea about choosing which battles to fight.
He said:
“…and I think the other big motivator is, as always, money. People are trying to break into niches that make the most money. I mean, duh, I would do the same thing probably.
But if you write about these topics that most people don’t write about, let’s say just three people wrote about it on the Internet, then maybe you can capture some traffic.
And then if you have many of those, then maybe you can even outdo those high-traffic niches.”
Barriers To Entry
What Gary is talking about is how to get around the barrier to entry, which are the established sites. His suggestion is to stay away from offering what everyone else is offering (which is a quality thing).
Creating content that the bigger sites can’t or don’t know to create is an approach I’ve used with a new site.
Weaknesses can be things that the big site does poorly, like their inability to resonate with a younger or older audience and so on.
Those are examples of offering something different that makes the site stand out from a quality perspective.
Gary is talking about picking the battles that can be won, planting a flag, then moving on to the next hill.
That’s a far better strategies than walking up toe to toe with the bigger opponent.
Analyzing For Quality Issues
It’s a lot easier to analyze a site for technical issues than it is for quality issues.
But a few of the takeaways are:
- Be aware that the people closest to the content are not always the best judges of content is quality.
- Read Google’s search documentation (for on-page factors, content, and quality guidelines).
- Content quality is simpler than it seems. Just think about knowing the topic well and being helpful to people.
- Being original is about looking at the SERPs for things that you can do differently, not about copying what the competitors are doing.
In my experience, it’s super important to keep an open mind, to not get locked into one way of thinking, especially when it comes to site quality. This will help one keep from getting locked into a point of view that can keep one from seeing the true cause of ranking issues.
Featured Image by Shutterstock/Stone36
-
FACEBOOK4 days ago
Indian Government Warns Facebook, YouTube About Deepfakes, Misinformation Violations
-
MARKETING3 days ago
Whiteboard Friday Recap 2023: AI Edition
-
MARKETING6 days ago
“Undercover” Case Studies: Why the Future of Marketing Is Proving Yourself in the Field
-
SEARCHENGINES6 days ago
Follower Count Is Not A Google Search Ranking Factor
-
MARKETING7 days ago
Sam’s Club Member Access Platform (MAP) Advertiser’s Guide
-
SOCIAL6 days ago
17-Year-Old Claims To Make 6 Figures A Year
-
SOCIAL5 days ago
Meta Stock: Still Room For Upside In A Maturing Market (NASDAQ:META)
-
SOCIAL7 days ago
U.S. Senators Accuse X of Profiting From Terrorist Propaganda in the App
You must be logged in to post a comment Login