Google may be crawling the web with a new crawler, a new Googlebot, named GoogleProducer. This useragent is not listed on the official Google crawlers page but maybe it is too new to be listed yet?

Hernán Marsili spotted this and asked Google about this on Twitter. He said, “we are seeing a lot of traffic to publisher’s sites with a new user-agent ‘GoogleProducer; (+http://goo.gl/7y4SX) ‘. Our WAF is currently blocking it, but it’s origin is actually Google Proxy hosts. Is this legit traffic?”

That link goes to this page that 404s within the Google News Producer section.

As an FYI, Google Producer is part of Google News and Google Currents, I believe. This help document from Google says, “You will use Google Currents producer to manage your issues (e.g., pricing, description, etc.) for Google Play Magazines. If you’re participating in Google Currents, Producer will look familiar to you. You can manage both your edition(s) for Google Currents and your issues for Google Play Magazines through Producer. See our article on publisher account setup for the steps you need to take to fully set up your publisher account and magazines for Play. If you have any questions about using Producer for Google Currents, please see the Currents Producer Help Center.” Note, the Currents Producer Help Center redirects to the Google Publisher Center help center.

John Mueller of Google said he will look into it:

I’ll check – thanks for pinging.

— John Mueller is watching out for Google+ 🐀 (@JohnMu) February 1, 2023

I guess we will see what he says but until then, I do suspect this is a legit Googlebot crawler.

Update: This is old, an older one:

2011 on this thread?https://t.co/NTqpOhQYSl

— Pedro Dias (@pedrodias) February 2, 2023

Forum discussion at Twitter.

Microsoft Bing posted a new blog post saying “for XML sitemaps, one of the most critical tags you can include in your sitemap is the “lastmod” tag.” And it will become even more critical as Bing is reworking its crawl scheduling stack to rely more on this lastmod field.

Yes, by June, the way Bing decides what to crawl will be more dependent on the lastmod tag. Fabrice Canel from Microsoft wrote, “we are revamping our crawl scheduling stack to better utilize the information provided by the “lastmod” tag in sitemaps.” This is being done so it can “enhance” the “crawl efficiency by reducing unnecessary crawling of unchanged content and prioritizing recently updated content.”

“We have already begun implementing these changes on a limited scale and plan to fully roll them out by June,” he added.

So making sure your lastmod date is accurate is now even more important. It should be the last time you modified the URL, not the time the URL was first published and not the time the XML sitemap file was generated. In fact, that is the biggest issue Bing found with the field, that it often just shows the date the XML sitemap file was generated and not the date the page of the URL was last modified.

Here are some data points Bing put together on XML sitemaps:

• 58% of hosts have at least one XML sitemap.
• 84% of these sitemaps have a lastmod attribute set.
• 79% have lastmod values correct.
• 18% have lastmod values not correctly set.
• 3% has lastmod values for only some of the URLs.
• 16% of these sitemaps don’t have a lastmod attribute set.
• 42% of hosts don’t have one XML sitemap

Oh, Bing still wants you to use the IndexNow protocol for the most efficient crawl solution but if you don’t – make sure your lastmod date is accurate.

In terms of Google, in 2015 Google said they don’t really use the lastmod date but then changed that in 2020 they said they do. The current Google documentation says, “Google uses the lastmod value if it’s consistently and verifiably (for example by comparing to the last modification of the page) accurate.”

Forum discussion at Twitter.

Google has updated the video indexing report within Google Search Console to add impression data and a way to filter the report by your available sitemaps.

As a reminder, the video indexing report went fully live in August 2022 after Google started to slowly roll out the video index report within Google Search Console earlier in 2022.

Google added two new features to the report; impression data and a sitemap filter. Here is a GIF of these two features:

You can now overlay the impressions your indexed videos saw directly in this report. Google said, “the impressions are aggregated by page which means that if the same page appears multiple times in a single search result page (or a single Discover session), then we consider each appearance as an impression.”

Here is what it looks like:

Google added, “The Search performance report groups video search appearances by property, not by URL, which means that if multiple pages show in a search results, we’ll count only one impression. As a result, the Search performance report can show lower impression counts than the Video page indexing report.”

The sitemap filter is a nice addition also, so you can see what videos you submitted via your sitemaps compared to what Google really indexed. Google said, “To help you focus on the video pages that matter most to you, you can now filter the Video indexing report to show only video pages that are present in a selected sitemap. The filter applies to all the report features: the chart, chart totals, issue list, and exports.”

Here is a screenshot of that:

You also see a section in the sitemaps location for discovered videos:

Hi @JohnMu @rustybrick
“Discovered Video” Tab in Google search console is it something new? pic.twitter.com/hu0rcEGqKW

— Jaydip Pancholi (@jaydip_ahir_333) February 2, 2023

Google Search Console Video indexing report has been also added to the Indexing>Sitemaps section.

now we can easily navigate & monitor page indexing & video indexing from the sitemap.

CC:- @glenngabe @rustybrick #SEO #GoogleSearch pic.twitter.com/NT18IE8O2f

— Vijay Chauhan 📈 (@VijayChauhanSEO) February 2, 2023

Again, if Google sees videos on your site, Google will display the new “Video indexing report” on the left navigation bar in the coverage section of Google Search Console. The report shows the status of video indexing on your site. It helps you answer the following questions:

• In how many pages has Google identified a video?
• Which videos were indexed successfully?
• What are the issues preventing videos from being indexed?

Forum discussion at Twitter.