Bing Revamps Crawl System To Enhance Efficiency

According to a recent study by Bing, most websites have XML sitemaps, with the “lastmod” tag being the most critical component of these sitemaps.

The “lastmod” tag indicates the last time the webpages linked by the sitemap were modified and is used by search engines to determine how often to crawl a site and which pages to index.

However, the study also revealed that a significant number of “lastmod” values in XML sitemaps were set incorrectly, with the most prevalent issue being identical dates on all sitemaps.

Upon consulting with web admins, Microsoft discovered that the dates were set to the date of sitemap generation rather than content modification.

To address this issue, Bing is revamping its crawl scheduling stack to better utilize the information provided by the “lastmod” tag in sitemaps.

This will improve crawl efficiency by reducing unnecessary crawling of unchanged content and prioritizing recently updated content.

The improvements have already begun on a limited scale and are expected to roll out by June fully.

Additionally, Microsoft has updated sitemap.org for improved clarity by adding the following line:

“Note that the date must be set to the date the linked page was last modified, not when the sitemap is generated.”

How To Use The Lastmod Tag Correctly

To correctly set the “lastmod” tag in a sitemap, you should include it in the <url> tag for each page in the sitemap.

The date should be in W3C Datetime format, with the most commonly used formats being YYYY-MM-DD or YYYY-MM-DDThh:mm:ssTZD.

The date should reflect the last time the page was modified and should be updated regularly to ensure that search engines understand the relevance and frequency of updates.

Here’s an example code snippet:

<?xml version=”1.0″ encoding=”UTF-8″?>

<urlset xmlns=”http://www.sitemaps.org/schemas/sitemap/0.9″>

   <url>

      <loc>http://www.example.com/</loc>

      <lastmod>2023-01-23</lastmod>      

   </url>

Google’s Advice: Use Lastmod Tag After Significant Changes Only

Google’s crawlers also utilize the “lastmod” tag, and the suggestions on using it by both major search engines are similar.

Google Search Advocate John Mueller recently discussed the lastmod tag in the January edition of Google’s office-hours Q&A sessions.

It’s worth noting that Google recommends only using the “lastmod” tag for substantial modifications, which was not mentioned in Microsoft’s blog post.

Changing the date in the lastmod tag after minor edits can be viewed as an attempt to manipulate search snippets.

In Summary

Microsoft’s recent study and efforts to improve the utilization of the “lastmod” tag in sitemaps will result in more efficient and effective webpage crawling.

Publishers are encouraged to regularly update their sitemaps and lastmod tags to ensure that their pages are correctly indexed and easily accessible by search engines.

Featured Image: mundissima/Shutterstock

Source: Microsoft