SEO
Google Confirms 3 Ways To Make Googlebot Crawl More
Google’s Gary Illyes and Lizzi Sassman discussed three factors that trigger increased Googlebot crawling. While they downplayed the need for constant crawling, they acknowledged there a ways to encourage Googlebot to revisit a website.
1. Impact of High-Quality Content on Crawling Frequency
One of the things they talked about was the quality of a website. A lot of people suffer from the discovered not indexed issue and that’s sometimes caused by certain SEO practices that people have learned and believe are a good practice. I’ve been doing SEO for 25 years and one thing that’s always stayed the same is that industry defined best practices are generally years behind what Google is doing. Yet, it’s hard to see what’s wrong if a person is convinced that they’re doing everything right.
Gary Illyes shared a reason for an elevated crawl frequency at the 4:42 minute mark, explaining that one of triggers for a high level of crawling is signals of high quality that Google’s algorithms detect.
Gary said it at the 4:42 minute mark:
“…generally if the content of a site is of high quality and it’s helpful and people like it in general, then Googlebot–well, Google–tends to crawl more from that site…”
There’s a lot of nuance to the above statement that’s missing, like what are the signals of high quality and helpfulness that will trigger Google to decide to crawl more frequently?
Well, Google never says. But we can speculate and the following are some of my educated guesses.
We know that there are patents about branded search that count branded searches made by users as implied links. Some people think that “implied links” are brand mentions, but “brand mentions” are absolutely not what the patent talks about.
Then there’s the Navboost patent that’s been around since 2004. Some people equate the Navboost patent with clicks but if you read the actual patent from 2004 you’ll see that it never mentions click through rates (CTR). It talks about user interaction signals. Clicks was a topic of intense research in the early 2000s but if you read the research papers and the patents it’s easy to understand what I mean when it’s not so simple as “monkey clicks the website in the SERPs, Google ranks it higher, monkey gets banana.”
In general, I think that signals that indicate people perceive a site as helpful, I think that can help a website rank better. And sometimes that can be giving people what they expect to see, giving people what they expect to see.
Site owners will tell me that Google is ranking garbage and when I take a look I can see what they mean, the sites are kind of garbagey. But on the other hand the content is giving people what they want because they don’t really know how to tell the difference between what they expect to see and actual good quality content (I call that the Froot Loops algorithm).
What’s the Froot Loops algorithm? It’s an effect from Google’s reliance on user satisfaction signals to judge whether their search results are making users happy. Here’s what I previously published about Google’s Froot Loops algorithm:
“Ever walk down a supermarket cereal aisle and note how many sugar-laden kinds of cereal line the shelves? That’s user satisfaction in action. People expect to see sugar bomb cereals in their cereal aisle and supermarkets satisfy that user intent.
I often look at the Froot Loops on the cereal aisle and think, “Who eats that stuff?” Apparently, a lot of people do, that’s why the box is on the supermarket shelf – because people expect to see it there.
Google is doing the same thing as the supermarket. Google is showing the results that are most likely to satisfy users, just like that cereal aisle.”
An example of a garbagey site that satisfies users is a popular recipe site (that I won’t name) that publishes easy to cook recipes that are inauthentic and uses shortcuts like cream of mushroom soup out of the can as an ingredient. I’m fairly experienced in the kitchen and those recipes make me cringe. But people I know love that site because they really don’t know better, they just want an easy recipe.
What the helpfulness conversation is really about is understanding the online audience and giving them what they want, which is different from giving them what they should want. Understanding what people want and giving it to them is, in my opinion, what searchers will find helpful and ring Google’s helpfulness signal bells.
2. Increased Publishing Activity
Another thing that Illyes and Sassman said could trigger Googlebot to crawl more is an increased frequency of publishing, like if a site suddenly increased the amount of pages it is publishing. But Illyes said that in the context of a hacked site that all of a sudden started publishing more web pages. A hacked site that’s publishing a lot of pages would cause Googlebot to crawl more.
If we zoom out to examine that statement from the perspective of the forest then it’s pretty evident that he’s implying that an increase in publication activity may trigger an increase in crawl activity. It’s not that the site was hacked that is causing Googlebot to crawl more, it’s the increase in publishing that’s causing it.
Here is where Gary cites a burst of publishing activity as a Googlebot trigger:
“…but it can also mean that, I don’t know, the site was hacked. And then there’s a bunch of new URLs that Googlebot gets excited about, and then it goes out and then it’s crawling like crazy.”
A lot of new pages makes Googlebot get excited and crawl a site “like crazy” is the takeaway there. No further elaboration is needed, let’s move on.
3. Consistency Of Content Quality
Gary Illyes goes on to mention that Google may reconsider the overall site quality and that may cause a drop in crawl frequency.
Here’s what Gary said:
“…if we are not crawling much or we are gradually slowing down with crawling, that might be a sign of low-quality content or that we rethought the quality of the site.”
What does Gary mean when he says that Google “rethought the quality of the site?” My take on it is that sometimes the overall site quality of a site can go down if there’s parts of the site that aren’t to the same standard as the original site quality. In my opinion, based on things I’ve seen over the years, at some point the low quality content may begin to outweigh the good content and drag the rest of the site down with it.
When people come to me saying that they have a “content cannibalism” issue, when I take a look at it, what they’re really suffering from is a low quality content issue in another part of the site.
Lizzi Sassman goes on to ask at around the 6 minute mark if there’s an impact if the website content was static, neither improving or getting worse, but simply not changing. Gary resisted giving an answer, simply saying that Googlebot returns to check on the site to see if it has changed and says that “probably” Googlebot might slow down the crawling if there is no changes but qualified that statement by saying that he didn’t know.
Something that went unsaid but is related to the Consistency of Content Quality is that sometimes the topic changes and if the content is static then it may automatically lose relevance and begin to lose rankings. So it’s a good idea to do a regular Content Audit to see if the topic has changed and if so to update the content so that it continues to be relevant to users, readers and consumers when they have conversations about a topic.
Three Ways To Improve Relations With Googlebot
As Gary and Lizzi made clear, it’s not really about poking Googlebot to get it to come around just for the sake of getting it to crawl. The point is to think about your content and its relationship to the users.
1. Is the content high quality?
Does the content address a topic or does it address a keyword? Sites that use a keyword-based content strategy are the ones that I see suffering in the 2024 core algorithm updates. Strategies that are based on topics tend to produce better content and sailed through the algorithm updates.
2. Increased Publishing Activity
An increase in publishing activity can cause Googlebot to come around more often. Regardless of whether it’s because a site is hacked or a site is putting more vigor into their content publishing strategy, a regular content publishing schedule is a good thing and has always been a good thing. There is no “set it and forget it” when it comes to content publishing.
3. Consistency Of Content Quality
Content quality, topicality, and relevance to users over time is an important consideration and will assure that Googlebot will continue to come around to say hello. A drop in any of those factors (quality, topicality, and relevance) could affect Googlebot crawling which itself is a symptom of the more importat factor, which is how Google’s algorithm itself regards the content.
Listen to the Google Search Off The Record Podcast beginning at about the 4 minute mark:
Featured Image by Shutterstock/Cast Of Thousands
SEO
Mediavine Bans Publisher For Overuse Of AI-Generated Content
According to details surfacing online, ad management firm Mediavine is terminating publishers’ accounts for overusing AI.
Mediavine is a leading ad management company providing products and services to help website publishers monetize their content.
The company holds elite status as a Google Certified Publishing Partner, which indicates that it meets Google’s highest standards and requirements for ad networks and exchanges.
AI Content Triggers Account Terminations
The terminations came to light in a post on the Reddit forum r/Blogging, where a user shared an email they received from Mediavine citing “overuse of artificially created content.”
Trista Jensen, Mediavine’s Director of Ad Operations & Market Quality, states in the email:
“Our third party content quality tools have flagged your sites for overuse of artificially created content. Further internal investigation has confirmed those findings.”
Jensen stated that due to the overuse of AI content, “our top partners will stop spending on your sites, which will negatively affect future monetization efforts.”
Consequently, Mediavine terminated the publisher’s account “effective immediately.”
The Risks Of Low-Quality AI Content
This strict enforcement aligns with Mediavine’s publicly stated policy prohibiting websites from using “low-quality, mass-produced, unedited or undisclosed AI content that is scraped from other websites.”
In a March 7 blog post titled “AI and Our Commitment to a Creator-First Future,” the company declared opposition to low-value AI content that could “devalue the contributions of legitimate content creators.”
Mediavine warned in the post:
“Without publishers, there is no open web. There is no content to train the models that power AI. There is no internet.”
The company says it’s using its platform to “advocate for publishers” and uphold quality standards in the face of AI’s disruptive potential.
Mediavine states:
“We’re also developing faster, automated tools to help us identify low-quality, mass-produced AI content across the web.”
Targeting ‘AI Clickbait Kingpin’ Tactics
While the Reddit user’s identity wasn’t disclosed, the incident has drawn connections to the tactics of Nebojša Vujinović Vujo, who was dubbed an “AI Clickbait Kingpin” in a recent Wired exposé.
According to Wired, Vujo acquired over 2,000 dormant domains and populated them with AI-generated, search-optimized content designed purely to capture ad revenue.
His strategies represent the low-quality, artificial content Mediavine has vowed to prohibit.
Potential Implications
Lost Revenue
Mediavine’s terminations highlight potential implications for publishers that rely on artificial intelligence to generate website content at scale.
Perhaps the most immediate and tangible implication is the risk of losing ad revenue.
For publishers that depend heavily on programmatic advertising or sponsored content deals as key revenue drivers, being blocked from major ad networks could devastate their business models.
Devalued Domains
Another potential impact is the devaluation of domains and websites built primarily on AI-generated content.
If this pattern of AI content overuse triggers account terminations from companies like Mediavine, it could drastically diminish the value proposition of scooping up these domains.
Damaged Reputations & Brands
Beyond the lost monetization opportunities, publishers leaning too heavily into automated AI content also risk permanent reputational damage to their brands.
Once a determining authority flags a website for AI overuse, it could impact how that site is perceived by readers, other industry partners, and search engines.
In Summary
AI has value as an assistive tool for publishers, but relying heavily on automated content creation poses significant risks.
These include monetization challenges, potential reputation damage, and increasing regulatory scrutiny. Mediavine’s strict policy illustrates the possible consequences for publishers.
It’s important to note that Mediavine’s move to terminate publisher accounts over AI content overuse represents an independent policy stance taken by the ad management firm itself.
The action doesn’t directly reflect the content policies or enforcement positions of Google, whose publishing partner program Mediavine is certified under.
We have reached out to Mediavine requesting a comment on this story. We’ll update this article with more information when it’s provided.
Featured Image: Simple Line/Shutterstock
SEO
Google’s Guidance About The Recent Ranking Update
Google’s Danny Sullivan explained the recent update, addressing site recoveries and cautioning against making radical changes to improve rankings. He also offered advice for publishes whose rankings didn’t improve after the last update.
Google’s Still Improving The Algorithm
Danny said that Google is still working on their ranking algorithm, indicating that more changes (for the positive) are likely on the way. The main idea he was getting across is that they’re still trying to fill the gaps in surfacing high quality content from independent sites. Which is good because big brand sites don’t necessarily have the best answers.
He wrote:
“…the work to connect people with “a range of high quality sites, including small or independent sites that are creating useful, original content” is not done with this latest update. We’re continuing to look at this area and how to improve further with future updates.”
A Message To Those Who Were Left Behind
There was a message to those publishers whose work failed to recover with the latest update, to let them know that Google is still working to surface more of the independent content and that there may be relief on the next go.
Danny advised:
“…if you’re feeling confused about what to do in terms of rankings…if you know you’re producing great content for your readers…If you know you’re producing it, keep doing that…it’s to us to keep working on our systems to better reward it.”
Google Cautions Against “Improving” Sites
Something really interesting that he mentioned was a caution against trying to improve rankings of something that’s already on page one in order to rank even higher. Tweaking a site to get from position six or whatever to something higher has always been a risky thing to do for many reasons I won’t elaborate on here. But Danny’s warning increases the pressure to not just think twice before trying to optimize a page for search engines but to think three times and then some more.
Danny cautioned that sites that make it to the top of the SERPs should consider that a win and to let it ride instead of making changes right now in order to improve their rankings. The reason for that caution is that the search results continue to change and the implication is that changing a site now may negatively impact the rankings in a newly updated search index.
He wrote:
“If you’re showing in the top results for queries, that’s generally a sign that we really view your content well. Sometimes people then wonder how to move up a place or two. Rankings can and do change naturally over time. We recommend against making radical changes to try and move up a spot or two”
How Google Handled Feedback
There was also some light shed on what Google did with all the feedback they received from publishers who lost rankings. Danny wrote that the feedback and site examples he received was summarized, with examples, and sent to the search engineers for review. They continue to use that feedback for the next round of improvements.
He explained:
“I went through it all, by hand, to ensure all the sites who submitted were indeed heard. You were, and you continue to be. …I summarized all that feedback, pulling out some of the compelling examples of where our systems could do a better job, especially in terms of rewarding open web creators. Our search engineers have reviewed it and continue to review it, along with other feedback we receive, to see how we can make search better for everyone, including creators.”
Feedback Itself Didn’t Lead To Recovery
Danny also pointed out that sites that recovered their rankings did not do so because of they submitted feedback to Google. Danny wasn’t specific about this point but it conforms with previous statements about Google’s algorithms that they implement fixes at scale. So instead of saying, “Hey let’s fix the rankings of this one site” it’s more about figuring out if the problem is symptomatic of something widescale and how to change things for everybody with the same problem.
Danny wrote:
“No one who submitted, by the way, got some type of recovery in Search because they submitted. Our systems don’t work that way.”
That feedback didn’t lead to recovery but was used as data shouldn’t be surprising. Even as far back as the 2004 Florida Update Matt Cutts collected feedback from people, including myself, and I didn’t see a recovery for a false positive until everyone else also got back their rankings.
Takeaways
Google’s work on their algorithm is ongoing:
Google is continuing to tune its algorithms to improve its ability to rank high quality content, especially from smaller publishers. Danny Sullivan emphasized that this is an ongoing process.
What content creators should focus on:
Danny’s statement encouraged publishers to focus on consistently creating high quality content and not to focus on optimizing for algorithms. Focusing on quality should be the priority.
What should publishers do if their high-quality content isn’t yet rewarded with better rankings?
Publishers who are certain of the quality of their content are encouraged to hold steady and keep it coming because Google’s algorithms are still being refined.
Featured Image by Shutterstock/Cast Of Thousands
SEO
Plot Up To Five Metrics At Once
Google has rolled out changes to Analytics, adding features to help you make more sense of your data.
The update brings several key improvements:
- You can now compare up to five different metrics side by side.
- A new tool automatically spots unusual trends in your data.
- A more detailed report on transactions gives a closer look at revenue.
- The acquisition reports now separate user and session data more clearly.
- It’s easier to understand what each report does with new descriptions.
Here’s an overview of these new features, why they matter, and how they might help improve your data analysis and decision-making.
▶ ️We’ve introduced plot rows in detailed reports. You can now visualize up to 5 rows of data directly within your detailed reports to measure their changes over time.
We’ve also launched these new report features:
🔎: Anomaly detection to flag unusual data fluctuations
📊:… pic.twitter.com/VDPXe2Q9wQ— Google Analytics (@googleanalytics) September 5, 2024
Plot Rows: Enhanced Data Visualization
The most prominent addition is the “Plot Rows” feature.
You can now visualize up to five rows of data simultaneously within your reports, allowing for quick comparisons and trend analysis.
This feature is accessible by selecting the desired rows and clicking the “Plot Rows” option.
Anomaly Detection: Spotting Unusual Patterns
Google Analytics has implemented an anomaly detection system to help you identify potential issues or opportunities.
This new tool automatically flags unusual data fluctuations, making it easier to spot unexpected traffic spikes, sudden drops, or other noteworthy trends.
Improved Report Navigation & Understanding
Google Analytics has added hover-over descriptions for report titles.
These brief explanations provide context and include links to more detailed information about each report’s purpose and metrics.
Key Event Marking In Events Report
The Events report allows you to mark significant events for easy reference.
This feature, accessed through a three-dot menu at the end of each event row, helps you prioritize and track important data points.
New Transactions Report For Revenue Insights
For ecommerce businesses, the new Transactions report offers granular insights into revenue streams.
This feature provides information about each transaction, utilizing the transaction_id parameter to give you a comprehensive view of sales data.
Scope Changes In Acquisition Reports
Google has refined its acquisition reports to offer more targeted metrics.
The User Acquisition report now includes user-related metrics such as Total Users, New Users, and Returning Users.
Meanwhile, the Traffic Acquisition report focuses on session-related metrics like Sessions, Engaged Sessions, and Sessions per Event.
What To Do Next
As you explore these new features, keep in mind:
- Familiarize yourself with the new Plot Rows function to make the most of comparative data analysis.
- Pay attention to the anomaly detection alerts, but always investigate the context behind flagged data points.
- Take advantage of the more detailed Transactions report to understand your revenue patterns better.
- Experiment with the refined acquisition reports to see which metrics are most valuable for your needs.
As with any new tool, there will likely be a learning curve as you incorporate these features into your workflow.
FAQ
What is the “Plot Rows” feature in Google Analytics?
The “Plot Rows” feature allows you to visualize up to five rows of data at the same time. This makes it easier to compare different metrics side by side within your reports, facilitating quick comparisons and trend analysis. To use this feature, select the desired rows and click the “Plot Rows” option.
How does the new anomaly detection system work in Google Analytics?
Google Analytics’ new anomaly detection system automatically flags unusual data patterns. This tool helps identify potential issues or opportunities by spotting unexpected traffic spikes, sudden drops, or other notable trends, making it easier for users to focus on significant data fluctuations.
What improvements have been made to the Transactions report in Google Analytics?
The enhanced Transactions report provides detailed insights into revenue for ecommerce businesses. It utilizes the transaction_id parameter to offer granular information about each transaction, helping businesses get a better understanding of their revenue streams.
Featured Image: Vladimka production/Shutterstock
How to Market When Information is Dirt Cheap
Daily Search Forum Recap: September 2, 2024
Early Analysis & User Feedback
What Is Largest Contentful Paint: An Easy Explanation
Google Trends Subscriptions Quietly Canceled
Daily Search Forum Recap: September 3, 2024
Google August 2024 Core Update Impact Survey Results
Tento Launches as Exclusive Lending Partner for WooCommerce
How To Run A Live Youtube Show To An International Audience And Gain Traction
14+ Best Email Automation Tools For Your Business (Expert Pick)
-
SEO6 days agoHow to Market When Information is Dirt Cheap
-
SEARCHENGINES6 days ago
Daily Search Forum Recap: September 2, 2024
-
SEO3 days agoEarly Analysis & User Feedback
-
SEO6 days agoWhat Is Largest Contentful Paint: An Easy Explanation
-
SEO5 days agoGoogle Trends Subscriptions Quietly Canceled
-
SEARCHENGINES5 days ago
Daily Search Forum Recap: September 3, 2024
-
SEARCHENGINES7 days agoGoogle August 2024 Core Update Impact Survey Results
-
WORDPRESS7 days agoTento Launches as Exclusive Lending Partner for WooCommerce