GOOGLE

How 500 Error Codes Can Impact Google Indexing

Published

3 years ago

August 16, 2021

How 500 Error Codes Can Impact Google Indexing

Google’s John Mueller answered a question about at what point will 500 error response codes cause Googlebot to crawl content less often. Mueller explained scenarios where 500 response codes won’t affect crawling and when they may cause pages to drop from Google’s index.

Accidentally Blocked Googlebot and Caused 500 Errors

The person asking the question explained that their content delivery system was set up to block bad bots while allowing actual human users and legit bots to browse the site.

Everything was working fine until they updated their “server monitoring suite” and apparently the previous bot blocking filters and settings weren’t carried over to the new system.

Afterward they noticed that Google Search Console began listing 500 error response codes in response to Googlebot crawling.

The person finished explaining what happened then asked the question:

“The question came up from our technology team whether this represented real user impact and why we would look specifically at Googlebot and not real user metrics to prove that there’s an issue here.

So, given that context, I have a few questions.

The first is just to get the technology concern out …of the way.

From your perspective, how does Googlebot view 500 series errors and could you give any clarity on established …thresholds at which point Googlebot will …crawl source content …less, based on those errors?”

What is a 500 Error Code?

Servers communicate a variety of status response codes to browsers and bots. These response codes communicate, for example, whether the request for a page was successful, redirected to another page, blocked or unsuccessful.

The 200 response code communicates that the request for a web page was successful.
The 500 response code means that the request for a web page was unsuccessful.

The formal name of the 500 response code is:

"500 Internal Server Error"

There are other versions of the 500 response code that communicate a specific reason why the request for a web page was unsuccessful.

The 500 response code is somewhat generic in that it simply means that the page request was unsuccessful. But it often means that the server was down but not always.

Google Search Console and 500 Error Responses

It’s a good idea to monitor the Search Console for 500 Error Responses because they are a strong indicator that something is wrong.

Sometimes it could mean that the server is under heavy load, for example from too many bots and because of that it is unable to serve web pages.

In this case it appears that the problem was due to a misconfiguration in the Content Delivery Network (CDN) that inadvertently blocked Google from accessing web pages.

How Googlebot Indexing is Affected by 500 Errors

Mueller provided a helpful explanation of the steps Google takes in response to 500 error codes and how eventually continued errors could lead to those web pages being dropped from Google’s search index.

Google’s John Mueller responded:

“We don’t have any strong thresholds on that.

But essentially what happens with 500 errors is we’ll try to retry them.

And if we continue to see …the 500 errors then we will …slow down crawling.

And if we continue to see that there are 500 errors then we will drop those URLs from the index.

So that’s something if every now and then individual pages have a 500 error, it’s like no big deal.

We will retry them, they’ll remain indexed and the next time we retry them that’s fine.”

Next, Mueller explains that the reason Googlebot slows down the crawling is in case Googlebot itself is causing the server to overload.

“But if a large part of a site consistently has 500 errors and we might assume that maybe we’re causing the problem and we’ll slow down crawling of the whole site and at some point we’ll say well, it looks like these pages are really gone, we’re going to drop them.

So that’s essentially the effects that you would see there and if you’re talking about a large site and wondering like what percentage of 500 errors is okay… I don’t know.

My feeling is if you’re seeing something more than one percent then that sounds like something is kind of broken and probably would be something where we would start to slow down.

But I don’t think we have any hard thresholds where we’d say, this many requests and this many errors means this much slowing down.”

Monitor Search Console for 500 Errors

It’s up to every publisher to decide what their daily routine is. For some people checking the search console once a week is enough. Some may find it reassuring to check Search Console once a day in order to respond to unforeseen circumstances.

In any case, 500 error codes is something to monitor Search Console for. As John Mueller mentioned, large-scale 500 errors could be a flag that something is wrong and in need of immediate attention.

Citations

World Wide Web Consortium (W3C) List of Server Response Codes

How Does Google Respond to 500 Response Codes?
Watch the discussion at the 13:48 minute mark

Searchenginejournal.com

GOOGLE

Google Warns About Misuse of Its Indexing API

Published

3 weeks ago

September 12, 2024

Max

Google Warns About Misuse of Its Indexing API

Google has updated its Indexing API documentation with a clear warning about spam detection and the possible consequences of misuse.

Warning Against API Misuse The new message in the guide says:

“All submissions through the Indexing API are checked for spam. Any misuse, like using multiple accounts or going over the usage limits, could lead to access being taken away.”

This warning is aimed at people trying to abuse the system by exceeding the API’s limits or breaking Google’s rules.

What Is the Indexing API? The Indexing API allows websites to tell Google when job posting or livestream video pages are added or removed. It helps websites with fast-changing content get their pages crawled and indexed quickly.

But it seems some users have been trying to abuse this by using multiple accounts to get more access.

Impact of the Update Google is now closely watching how people use the Indexing API. If someone breaks the rules, they might lose access to the tool, which could make it harder for them to keep their search results updated for time-sensitive content.

How To Stay Compliant To use the Indexing API properly, follow these rules:

Don’t go over the usage limits, and if you need more, ask Google instead of using multiple accounts.
Use the API only for job postings or livestream videos, and make sure your data is correct.
Follow all of Google’s API guidelines and spam policies.
Use sitemaps along with the API, not as a replacement.

Remember, the Indexing API isn’t a shortcut to faster indexing. Follow the rules to keep your access.

GOOGLE

This Week in Search News: Simple and Easy-to-Read Update

Published

3 months ago

June 30, 2024

Entireweb News Bot

This Week in Search News: Simple and Easy-to-Read Update

Here’s what happened in the world of Google and search engines this week:

1. Google’s June 2024 Spam Update

Google finished rolling out its June 2024 spam update over a period of seven days. This update aims to reduce spammy content in search results.

2. Changes to Google Search Interface

Google has removed the continuous scroll feature for search results. Instead, it’s back to the old system of pages.

3. New Features and Tests

Link Cards: Google is testing link cards at the top of AI-generated overviews.
Health Overviews: There are more AI-generated health overviews showing up in search results.
Local Panels: Google is testing AI overviews in local information panels.

4. Search Rankings and Quality

Improving Rankings: Google said it can improve its search ranking system but will only do so on a large scale.
Measuring Quality: Google’s Elizabeth Tucker shared how they measure search quality.

5. Advice for Content Creators

Brand Names in Reviews: Google advises not to avoid mentioning brand names in review content.
Fixing 404 Pages: Google explained when it’s important to fix 404 error pages.

6. New Search Features in Google Chrome

Google Chrome for mobile devices has added several new search features to enhance user experience.

7. New Tests and Features in Google Search

Credit Card Widget: Google is testing a new widget for credit card information in search results.
Sliding Search Results: When making a new search query, the results might slide to the right.

8. Bing’s New Feature

Bing is now using AI to write “People Also Ask” questions in search results.

9. Local Search Ranking Factors

Menu items and popular times might be factors that influence local search rankings on Google.

10. Google Ads Updates

Query Matching and Brand Controls: Google Ads updated its query matching and brand controls, and advertisers are happy with these changes.
Lead Credits: Google will automate lead credits for Local Service Ads. Google says this is a good change, but some advertisers are worried.
tROAS Insights Box: Google Ads is testing a new insights box for tROAS (Target Return on Ad Spend) in Performance Max and Standard Shopping campaigns.
WordPress Tag Code: There is a new conversion code for Google Ads on WordPress sites.

These updates highlight how Google and other search engines are continuously evolving to improve user experience and provide better advertising tools.

AI

Exploring the Evolution of Language Translation: A Comparative Analysis of AI Chatbots and Google Translate

Published

7 months ago

February 26, 2024

Max

A Comparative Analysis of AI Chatbots and Google Translate

According to an article on PCMag, while Google Translate makes translating sentences into over 100 languages easy, regular users acknowledge that there’s still room for improvement.

In theory, large language models (LLMs) such as ChatGPT are expected to bring about a new era in language translation. These models consume vast amounts of text-based training data and real-time feedback from users worldwide, enabling them to quickly learn to generate coherent, human-like sentences in a wide range of languages.

However, despite the anticipation that ChatGPT would revolutionize translation, previous experiences have shown that such expectations are often inaccurate, posing challenges for translation accuracy. To put these claims to the test, PCMag conducted a blind test, asking fluent speakers of eight non-English languages to evaluate the translation results from various AI services.

The test compared ChatGPT (both the free and paid versions) to Google Translate, as well as to other competing chatbots such as Microsoft Copilot and Google Gemini. The evaluation involved comparing the translation quality for two test paragraphs across different languages, including Polish, French, Korean, Spanish, Arabic, Tagalog, and Amharic.

In the first test conducted in June 2023, participants consistently favored AI chatbots over Google Translate. ChatGPT, Google Bard (now Gemini), and Microsoft Bing outperformed Google Translate, with ChatGPT receiving the highest praise. ChatGPT demonstrated superior performance in converting colloquialisms, while Google Translate often provided literal translations that lacked cultural nuance.

For instance, ChatGPT accurately translated colloquial expressions like “blow off steam,” whereas Google Translate produced more literal translations that failed to resonate across cultures. Participants appreciated ChatGPT’s ability to maintain consistent levels of formality and its consideration of gender options in translations.

The success of AI chatbots like ChatGPT can be attributed to reinforcement learning with human feedback (RLHF), which allows these models to learn from human preferences and produce culturally appropriate translations, particularly for non-native speakers. However, it’s essential to note that while AI chatbots outperformed Google Translate, they still had limitations and occasional inaccuracies.

In a subsequent test, PCMag evaluated different versions of ChatGPT, including the free and paid versions, as well as language-specific AI agents from OpenAI’s GPTStore. The paid version of ChatGPT, known as ChatGPT Plus, consistently delivered the best translations across various languages. However, Google Translate also showed improvement, performing surprisingly well compared to previous tests.

Overall, while ChatGPT Plus emerged as the preferred choice for translation, Google Translate demonstrated notable improvement, challenging the notion that AI chatbots are always superior to traditional translation tools.

Source: https://www.pcmag.com/articles/google-translate-vs-chatgpt-which-is-the-best-language-translator