CNBC reported the other day that Google is working on its own AI chatbot named Apprentice Bard. Apprentice Bard is reportedly built on Google’s AI LaMDA language model stack and while Google is being more cautious with this rollout, Google is working on testing an AI bot in search.

CNBC wrote, “As a result of ChatGPT, the LaMDA team has been asked to prioritize working on a response to ChatGPT,” read one internal memo viewed by CNBC. “In the short term, it takes precedence over other projects,” the email continued, warning that some employees stop attending certain unrelated meetings.

“Apprentice Bard looks similar to ChatGPT: Employees can enter a question in a dialog box and get a text answer, then give feedback on the response. Based on several responses viewed by CNBC, Apprentice Bard’s answers can include recent events, a feature ChatGPT doesn’t have yet,” CNBC said. This makes sense, as Google can crawl the web in almost real-time and process that information faster than any other company.

The examples given by CNBC show that Google even picked up on the Google layoff news and was able to respond to questions about this. Whereas ChatGPT only has content from 2021 or earlier.

Also, CNBC said Google is working on designing an alternative search interface to support this chat feature. CNBC said, “One view showed the home search page offering five different prompts for potential questions placed directly under the main search bar, replacing the current “I’m feeling lucky” bar. It also showed a small chat logo inside the far right end of the search bar.” “When a question is entered, the search results show a grey bubble directly under the search bar, offering more human-like responses than typical search results. Directly beneath that, the page suggests several follow-up questions related to the first one. Under that, it shows typical search results, including links and headlines.”

Super interesting stuff and I suspect that if Google does release something, it will be a lot better than what we’ve been seeing so far, if that is even imaginable…

More: “One view showed the search page offering 5 different prompts for potential questions placed directly under the main search bar. It also showed a small chat logo inside the far right end of the search bar. The page suggests several follow-up questions related to the first.” pic.twitter.com/JOFWxlMTho

— Glenn Gabe (@glenngabe) February 1, 2023

Forum discussion at Twitter.

Google may be crawling the web with a new crawler, a new Googlebot, named GoogleProducer. This useragent is not listed on the official Google crawlers page but maybe it is too new to be listed yet?

Hernán Marsili spotted this and asked Google about this on Twitter. He said, “we are seeing a lot of traffic to publisher’s sites with a new user-agent ‘GoogleProducer; (+http://goo.gl/7y4SX) ‘. Our WAF is currently blocking it, but it’s origin is actually Google Proxy hosts. Is this legit traffic?”

That link goes to this page that 404s within the Google News Producer section.

As an FYI, Google Producer is part of Google News and Google Currents, I believe. This help document from Google says, “You will use Google Currents producer to manage your issues (e.g., pricing, description, etc.) for Google Play Magazines. If you’re participating in Google Currents, Producer will look familiar to you. You can manage both your edition(s) for Google Currents and your issues for Google Play Magazines through Producer. See our article on publisher account setup for the steps you need to take to fully set up your publisher account and magazines for Play. If you have any questions about using Producer for Google Currents, please see the Currents Producer Help Center.” Note, the Currents Producer Help Center redirects to the Google Publisher Center help center.

John Mueller of Google said he will look into it:

I’ll check – thanks for pinging.

— John Mueller is watching out for Google+ 🐀 (@JohnMu) February 1, 2023

I guess we will see what he says but until then, I do suspect this is a legit Googlebot crawler.

Update: This is old, an older one:

2011 on this thread?https://t.co/NTqpOhQYSl

— Pedro Dias (@pedrodias) February 2, 2023

Forum discussion at Twitter.

Microsoft Bing posted a new blog post saying “for XML sitemaps, one of the most critical tags you can include in your sitemap is the “lastmod” tag.” And it will become even more critical as Bing is reworking its crawl scheduling stack to rely more on this lastmod field.

Yes, by June, the way Bing decides what to crawl will be more dependent on the lastmod tag. Fabrice Canel from Microsoft wrote, “we are revamping our crawl scheduling stack to better utilize the information provided by the “lastmod” tag in sitemaps.” This is being done so it can “enhance” the “crawl efficiency by reducing unnecessary crawling of unchanged content and prioritizing recently updated content.”

“We have already begun implementing these changes on a limited scale and plan to fully roll them out by June,” he added.

So making sure your lastmod date is accurate is now even more important. It should be the last time you modified the URL, not the time the URL was first published and not the time the XML sitemap file was generated. In fact, that is the biggest issue Bing found with the field, that it often just shows the date the XML sitemap file was generated and not the date the page of the URL was last modified.

Here are some data points Bing put together on XML sitemaps:

• 58% of hosts have at least one XML sitemap.
• 84% of these sitemaps have a lastmod attribute set.
• 79% have lastmod values correct.
• 18% have lastmod values not correctly set.
• 3% has lastmod values for only some of the URLs.
• 16% of these sitemaps don’t have a lastmod attribute set.
• 42% of hosts don’t have one XML sitemap

Oh, Bing still wants you to use the IndexNow protocol for the most efficient crawl solution but if you don’t – make sure your lastmod date is accurate.

In terms of Google, in 2015 Google said they don’t really use the lastmod date but then changed that in 2020 they said they do. The current Google documentation says, “Google uses the lastmod value if it’s consistently and verifiably (for example by comparing to the last modification of the page) accurate.”

Forum discussion at Twitter.