Google: It’s Normal for 20% of a Site to Not Be Indexed

Google’s John Mueller answered a question about indexing, offering insights into how overall site quality influences indexing patterns. He also offered the insight that it’s within the bounds of normal that 20% of a site’s content is not indexed.
Pages Discovered But Not Crawled
The person asking the question offered background information about their site.
Of particular concern was the stated fact that the server was overloaded and if that might affect how many pages Google indexes.
When a server is overloaded the request for a web page may result in a 500 error response. This is because when a server cannot serve a web page the standard response is a 500 Internal Server Error message.
The person asking the question did not mention that Google Search Console was reporting that Googlebot was receiving 500 error response codes.
So if it’s the case that Googlebot did not receive a 500 error response then the server overload issue is probably not the reason why 20% of the pages are not getting indexed.
The person asked the following question:
“20% of my pages are not getting indexed.
It says they’re discovered but not crawled.
Does this have anything to do with the fact that it’s not crawled because of potential overload of my server?
Or does it have to do with the quality of the page?”
Crawl Budget Not Generally Why Small Sites Have Non-indexed Pages
Google’s John Mueller offered an interesting explanation of how overall site quality is an important factor that determines whether Googlebot will index more web pages.
But first he discussed how the crawl budget isn’t usually a reason why pages remain non-indexed for a small site.
John Mueller answered:
“Probably a little of both.
So usually if we’re talking about a smaller site then it’s mostly not a case that we’re limited by the crawling capacity, which is the crawl budget side of things.
If we’re talking about a site that has millions of pages, then that’s something where I would consider looking at the crawl budget side of things.
But smaller sites probably less so.”
Overall Site Quality Determines Indexing
John next went into detail about how overall site quality can affect how much of a website is crawled and indexed.
This part is especially interesting because it gives a peek at how Google evaluates a site in terms of quality and how the overall impression influences indexing.
Mueller continued his answer:
“With regards to the quality, when it comes to understanding the quality of the website, that is something that we take into account quite strongly with regards to crawling and indexing of the rest of the website.
But that’s not something that’s necessarily related to the individual URL.
So if you have five pages that are not indexed at the moment, it’s not that those five pages are the ones we would consider low quality.
It’s more that …overall, we consider this website maybe to be a little bit lower quality. And therefore we won’t go off and index everything on this site.
Because if we don’t have that page indexed, then we’re not really going to know if that’s high quality or low quality.
So that’s the direction I would head there …if you have a smaller site and you’re seeing a significant part of your pages are not being indexed, then I would take a step back and try to reconsider the overall quality of the website and not focus so much on technical issues for those pages.”
Technical Factors and Indexing
Mueller next mentions technical factors and how easy it is for modern sites to get that part right so that it doesn’t get in the way of indexing.
Mueller observed:
“Because I think, for the most part, sites nowadays are technically reasonable.
If you’re using a common CMS then it’s really hard to do something really wrong.
And it’s often more a matter of the overall quality.”
It’s Normal for 20% of a Site to Not Be Indexed
This next part is also interesting in that Mueller downplays 20% of a site not indexed as something that is within the bounds of normal.
Mueller has more access to information about how much of sites are typically not indexed so I take him at his word because he speaking from the perspective of Google.
Mueller explains why it’s normal for pages to not be indexed:
“The other thing to keep in mind with regards to indexing, is it’s completely normal that we don’t index everything off of the website.
So if you look at any larger website or any even midsize or smaller website, you’ll see fluctuations in indexing.
It’ll go up and down and it’s never going to be the case that we index 100% of everything that’s on a website.
So if you have a hundred pages and (I don’t know) 80 of them are being indexed, then I wouldn’t see that as being a problem that you need to fix.
That’s sometimes just how it is for the moment.
And over time, when you get to like 200 pages on your website and we index 180 of them, then that percentage gets a little bit smaller.
But it’s always going to be the case that we don’t index 100% of everything that we know about.”
Don’t Panic if Pages Aren’t Indexed
There’s quite a lot of information Mueller shared about indexing to take in.
- It’s within the bounds of normal for 20% of a site to not be indexed.
- Technical issues probably won’t impeded indexing.
- Overall site quality can determine how much of a site gets indexed.
- How much of a site gets indexed fluctuates.
- Small sites generally don’t have to worry about crawl budget.
Citation
It’s Normal for 20% of a Site to be Non-indexed
Watch Mueller discussing what is normal indexing from about the 27:26 minute mark.
Google Brings Bard Students Math and Coding Education in the Summer

Google is stepping up its AI efforts this summer by sending Bard, its high-profile chatbot, to summer school. The aim? To boost the bot’s math and coding smarts. These developments are excellent news— when Bard first debuted, it was admittedly not a finished product. But Google is steadily plugging away at it, and have now implemented implicit code execution for logical prompts, and handy Google Sheets’ integration to take it to the next level.
Thanks to implicit code execution, Bard can respond to inquiries requiring calculation or computation with Python code snippets running in the background. What’s even more amazing is that coders can take this generated code and modify it for their projects. Though Google is still apprehensive about guaranteeing the accuracy of Bard’s answers, this feature is said to improve the accuracy of math and word problems by an impressive 30%.
In addition to this, Bard can now export directly to Sheets when asked about tables. So, you don’t need to worry about copying and pasting, which comes with the risk of losing formatting or data.
From the company’s I/O keynote address, it is clear that they are focused on making the most of what Bard can offer. As they continue to speak highly of the chatbot, we’re sure to expect more features and capabilities when the summer comes around.
Google Bard vs. ChatGPT: which is the better AI chatbot?

Google Bard and ChatGPT are two of the most prominent artificial intelligence (AI) chatbots available in 2023. But which is better? Both offer natural language responses to natural language inputs, using machine learning and millions of data points to craft useful, informative responses. Most of the time. These AI tools aren’t perfect yet, but they point to an exciting future of AI assistant search and learning tools that will make information all the more readily available.
As similar as these chatbots are, they also have some distinct differences. Here’s how ChatGPT and Google Bard measure up against one another.
Which is better, Google Bard or ChatGPT?
This is a tricky question to answer, as at the time of writing, you can only use Google Bard if you’re part of a select group of early beta testers. As for its competition, you can use ChatGPT right now, completely for free. You may have to contend with a waitlist, but if you want to skip that, there’s a paid-for Plus version offering those interested in a more complete tool the option of paying for the privilege.
Still, when Google Bard becomes more widely available, it should offer credible competition for ChatGPT. Both use natural language models — Google Bard uses Google’s internal LaMDA (Language Model for Dialogue Applications), whereas ChatGPT uses an older GPT-3 language model. Google Bard bases its responses to questions on more recent data, with ChatGPT mainly trained on data that was available prior to 2021. This is similar to how Microsoft’s Bing Chat works.
We’ll have to reserve judgment on which is the more capable AI chatbot until we get time to play with Google Bard ourselves, but it looks set to be a close contest when it is more readily available.
Are Google Bard and ChatGPT available yet?
As mentioned, ChatGPT is available in free and paid-for tiers. You might have to sit in a queue for the free version for a while, but anyone can play around with its capabilities.
Google Bard is currently only available to limited beta testers and is not available to the wider public.

What’s the difference between Google Bard and ChatGPT?
ChatGPT and Google Bard are very similar natural language AI chatbots, but they have some differences, and are designed to be used in slightly different ways — at least for now. ChatGPT has been used for answering direct questions with direct answers, mostly correctly, but it’s caused a lot of consternation among white collar workers, like writers, SEO advisors, and copy editors, since it has also demonstrated an impressive ability to write creatively — even if it has faced a few problems with accuracy and plagiarism.
Still, Microsoft has integrated ChatGPT into its Bing search engine to give users the ability to ask direct questions of the search engine, rather than searching for terms of keywords to find the best results. It has also built it into its Teams communications tool, and it’s coming to the Edge browser in a limited form. The Opera browser has also pledged to integrate ChatGPT in the future.
ChatGPT | Google Bard |
Accessible through ChatGPT site. Only text responses are returned via queries. | Integrated with Google Search. You only need to change a Google setting to get your regular search results when using Google Bard AI, and vice versa. |
ChatGPT produces answers from its trained database from 2021 and before. | Google Apprentice Bard AI will be able to answer real-time questions. |
Based on GPT (Generative Pre-trained Transformer). | Based on LaMDA (Language Model for Dialogue Applications). |
Service has a free and paid plan option (called ChatGPT Plus). | Service is free. |
Has built-in plagiarism tool called GPT-2 Output Detector. | No built-in plagiarism detection tool. |
Available now | Still in beta test phase |
Google Bard was mainly designed around augmenting Google’s own search tool, however it is also destined to become an automated support tool for businesses without the funds to pay for human support teams. It will be offered to customers through a trained AI responder. It is likely to be integrated into the Chrome browser and its Chromium derivatives before long. Google is also expected to open up Google Bard to third-party developers in the future.
Under the hood, Google Bard uses Google’s LaMDA language model, while ChatGPT uses its own GPT3 model. ChatGPT is based on slightly older data, restricted in its current GPT3 model to data collected prior to 2022, while Google Bard is built on data provided on recent years too. However, that doesn’t necessarily make it more accurate, as Google Bard has faced problems with incorrect answers to questions, even in its initial unveiling.
ChatGPT also has a built-in plagiarism checker, while Google Bard does not, but Google Bard doesn’t have the creative applications of ChatGPT just yet.
Google to pay $391.5 million settlement over location tracking, state AGs say

Google has agreed to pay a $391.5 million settlement to 40 states to resolve accusations that it tracked people’s locations in violation of state laws, including snooping on consumers’ whereabouts even after they told the tech behemoth to bug off.
Louisiana Attorney General Jeff Landry said it is time for Big Tech to recognize state laws that limit data collection efforts.
“I have been ringing the alarm bell on big tech for years, and this is why,” Mr. Landry, a Republican, said in a statement Monday. “Citizens must be able to make informed decisions about what information they release to big tech.”
The attorneys general said the investigation resulted in the largest-ever multistate privacy settlement. Connecticut Attorney General William Tong, a Democrat, said Google’s penalty is a “historic win for consumers.”
“Location data is among the most sensitive and valuable personal information Google collects, and there are so many reasons why a consumer may opt out of tracking,” Mr. Tong said. “Our investigation found that Google continued to collect this personal information even after consumers told them not to. That is an unacceptable invasion of consumer privacy, and a violation of state law.”
Location tracking can help tech companies sell digital ads to marketers looking to connect with consumers within their vicinity. It’s another tool in a data-gathering toolkit that generates more than $200 billion in annual ad revenue for Google, accounting for most of the profits pouring into the coffers of its corporate parent, Alphabet, which has a market value of $1.2 trillion.
The settlement is part of a series of legal challenges to Big Tech in the U.S. and around the world, which include consumer protection and antitrust lawsuits.
Though Google, based in Mountain View, California, said it fixed the problems several years ago, the company’s critics remained skeptical. State attorneys general who also have tussled with Google have questioned whether the tech company will follow through on its commitments.
The states aren’t dialing back their scrutiny of Google’s empire.
Last month, Texas Attorney General Ken Paxton said he was filing a lawsuit over reports that Google unlawfully collected millions of Texans’ biometric data such as “voiceprints and records of face geometry.”
The states began investigating Google’s location tracking after The Associated Press reported in 2018 that Android devices and iPhones were storing location data despite the activation of privacy settings intended to prevent the company from following along.
Arizona Attorney General Mark Brnovich went after the company in May 2020. The state’s lawsuit charged that the company had defrauded its users by misleading them into believing they could keep their whereabouts private by turning off location tracking in the settings of their software.
Arizona settled its case with Google for $85 million last month. By then, attorneys general in several other states and the District of Columbia had pounced with their own lawsuits seeking to hold Google accountable.
Along with the hefty penalty, the state attorneys general said, Google must not hide key information about location tracking, must give users detailed information about the types of location tracking information Google collects, and must show additional information to people when users turn location-related account settings to “off.”
States will receive differing sums from the settlement. Mr. Landry’s office said Louisiana would receive more than $12.7 million, and Mr. Tong’s office said Connecticut would collect more than $6.5 million.
The financial penalty will not cripple Google’s business. The company raked in $69 billion in revenue for the third quarter of 2022, according to reports, yielding about $13.9 billion in profit.
Google downplayed its location-tracking tools Monday and said it changed the products at issue long ago.
“Consistent with improvements we’ve made in recent years, we have settled this investigation which was based on outdated product policies that we changed years ago,” Google spokesman Jose Castaneda said in a statement.
Google product managers Marlo McGriff and David Monsees defended their company’s Search and Maps products’ usage of location information.
“Location information lets us offer you a more helpful experience when you use our products,” the two men wrote on Google’s blog. “From Google Maps’ driving directions that show you how to avoid traffic to Google Search surfacing local restaurants and letting you know how busy they are, location information helps connect experiences across Google to what’s most relevant and useful.”
The blog post touted transparency tools and auto-delete controls that Google has developed in recent years and said the private browsing Incognito mode prevents Google Maps from saving an account’s search history.
Mr. McGriff and Mr. Monsees said Google would make changes to its products as part of the settlement. The changes include simplifying the process for deleting location data, updating the method to set up an account and revamping information hubs.
“We’ll provide a new control that allows users to easily turn off their Location History and Web & App Activity settings and delete their past data in one simple flow,” Mr. McGriff and Mr. Monsees wrote. “We’ll also continue deleting Location History data for users who have not recently contributed new Location History data to their account.”
• This article is based in part on wire service reports.
-
SEO6 days ago
Google Discusses Fixing 404 Errors From Inbound Links
-
SOCIAL4 days ago
Musk regrets controversial post but won’t bow to advertiser ‘blackmail’
-
SEARCHENGINES6 days ago
Google Search Console Was Down Today
-
MARKETING7 days ago
10 Advanced Tips for Crafting Engaging Social Content Strategies
-
SEO4 days ago
A Year Of AI Developments From OpenAI
-
MARKETING6 days ago
How to Schedule Ad Customizers for Google RSAs [2024]
-
SEO5 days ago
SEO Salary Survey 2023 [Industry Research]
-
PPC4 days ago
5 Quick Tips to Increase Referral Traffic