Connect with us

SEARCHENGINES

Google Has Three Categories Of Web Crawlers

Published

7 months ago

on

Google Bot Lineup

Google has updated its Verifying Googlebot and other Google crawlers help document to add a new section describing the three categories or types of crawlers they have. They have their Googlebot crawler, special-case crawlers and user-triggered crawlers.

I believe this was done after we, including me, were obsessed a bit over the new GoogleOther crawler. Then Gary Illyes from Google added, “Please don’t overthink it, it’s really that boring.” But I do what I do and I overthink’ed it. So Gary did what he does and had a help document to explain this in more detail.

The help document says, “Google’s crawlers fall into three categories.”

(1) Googlebot: The main crawler for Google’s search products, it alays respects robots.txt rules. Its reverse DNS mask is “crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com” and the list of IP ranges are in this googlebot.json file.

(2) Special-case crawlers: Crawlers that perform specific functions (such as AdsBot), which may or may not respect robots.txt rules. Its reverse DNS mask is “rate-limited-proxy-***-***-***-***.google.com” and the list of IP ranges are in this special-crawlers.json file.

(3) User-triggered fetchers: Tools and product functions where the end user triggers a fetch. For example, Google Site Verifier acts on the request of a user. Because the fetch was requested by a user, these fetchers ignore robots.txt rules. Its reverse DNS mask is “***-***-***-***.gae.googleusercontent.com” and the list of IP ranges are in this user-triggered-fetchers.json file.

Here is a screenshot of the new section in this help document:

click for full size

Also, with this, Google restructured the order of the Googlebots on the Google crawlers page:

Old New Googlebot Structured Order

Forum discussion at Twitter.



Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Related Topics:

SEARCHENGINES

Most SEOs Believe Google’s November Core & Reviews Updates Will Complete In December

Published

10 hours ago

on

November 28, 2023

By

Google Robot Waiting By Computer

The Google November 2023 core update and Google November 2023 reviews update are still both rolling out, and we have no estimated time of when they will finish rolling out. Both are well past their estimated two-week rollout period. I ran a poll yesterday asking SEOs when they think it will be done, and most said in December and not in the next few days left of November.

The poll had just under 600 votes in 24-hours and 79% said the update will be completed in December and 21% said in the last days remaining in November. I posted the poll on X – here it is:

Personally, I thought it would be done by now and I still think it will be done in the next day or so – but I am wrong a lot. 🙂

I do hope it is done today so they can roll out the Christmas holiday update (kidding…).

Forum discussion at X.

Update: I’ll have a story tomorrow but at least one of the updates is done in November:



Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

SEARCHENGINES

Google Search Console Was Down Today

Published

1 day ago

on

November 28, 2023

By

Google Robot Hurt Tired

Google Search Console seems to be down right now. If you try to access it, you will get a 500 server error. I suspect Search Console will be back up shortly but right now, if it is down for you, you are not alone.

Update: At 8:52 am ET Google Search Console is back, the tool was offline for about 50 minutes.

Google’s Daniel Waisberg is 100% aware of the issue. He responded to some posts about the issue, saying, “we’re working on it.” John Mueller from Google is also reposting Daniel’s posts on X.

Here is what it looked like when you tried to access it while it was down:

Gsc 500 Error Down

Search Console went down at about 8 am ET and returned at 8:52 am ET.

Here are the posts on X from Daniel Waisberg of Google on this issue:

Who knows, maybe we will have a new Search Console feature when it returns?

Here are some of the complaints:

Update: At 8:52 am ET Google Search Console is back, the tool was offline for about 50 minutes.



Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

SEARCHENGINES

Google Testing New Flight Prices Rich Result?

Published

2 days ago

on

November 27, 2023

By

Woman Booking Flight Airport Google Logo

Google may be testing a new type of rich result for flight prices. This test shows a section under the flight search snippet that contains the dates, trip details and price.

This was spotted by Antoine Eripret who was able to trigger this numerous times, over numerous days, while in Spain. Here is his screenshot he posted on X:

Google Flight Prices Rich Result Test

He also posted this video of this in action for me:

Google Flight Prices Rich Result Test

Recently Google told us not to put flight prices in titles because those prices change too quickly. Maybe Google is testing updating those prices in more real time through the use of structured data and is thus more confident to show them as rich results and not title elements?

I don’t think Google officially supports flight schema but maybe they will soon?

Forum discussion at X.



Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

Trending