On-page SEO is fine-tuning various website components to help search engines crawl, understand, and rank pages for relevant queries.

While off-page factors like backlinks and brand signals are critical, optimizing on-page elements lays the groundwork for maximizing search visibility.

Beyond the content itself, on-page factors signal a page’s relevance and quality. The website architecture, including site speed, mobile-friendliness, and URL structures, impacts on-page SEO.

On-page SEO matters because:

• It helps search engines find and show your pages to users.
• Higher-ranked pages get more clicks and visitors.
• Good rankings boost your brand’s trustworthiness.
• It enables you to create content that meets your audience’s needs.
• It’s the foundation for other SEO efforts like building links.

This guide explores 13 essential on-page SEO elements, from E-E-A-T and keyword semantics to HTML tags and site architecture.

13 Essential On-Page SEO Factors

On-page SEO can be divided into content, HTML, and website architecture. We’ll look at each individually.

Content

You’ve heard it before: Content is king.

SEO without it is like a beautiful new sports car without an engine; it might look nice, but it’s going nowhere. But not all content is created equal.

Here are the content factors you need to consider to maximize your on-site SEO:

1. E-E-A-T

One way Google weights your site is based on E-E-A-T, or experience, expertise, authoritativeness, and trustworthiness.

As highlighted in Google’s Search Quality Rater Guidelines, E-E-A-T evaluates the first-hand experience, subject matter expertise, authority, and trustworthiness demonstrated by a website and its content creators.

Google added experience as a new component, signaling the increasing value placed on content created by those with relevant credentials and direct, real-world experience with the topic. This is especially critical for YMYL (Your Money or Your Life) topics like health, finance, safety, etc.

While Google has only confirmed a few E-E-A-T elements like PageRank and links, it’s generally accepted that factors like author expertise, topical authority, transparency, and hands-on experience play a significant role in E-E-A-T evaluations.

2. Keywords

Creating content that includes the words and phrases your target customers are searching for is essential.

However, with advancements in AI and natural language processing, you’ll need to think beyond individual keywords.

Optimize for:

• Semantically related phrases and topics (entities): For example, if you offer cloud data storage services, related entities could include backup solutions, disaster recovery, data management, etc.
• Contextual meaning and intent: A search for “cloud migration” could have different intents, such as technical how-to guides, pricing/cost info, migration strategies, etc.
• Providing comprehensive answers: Cover related subtopics to address customer journeys fully.

Use keyword research tools to identify relevant entities and related queries around your main topics.

Get started by downloading our ebook on keyword research.

3. SEO Writing

Creating content that prioritizes search engines and converts human visitors to your site is an art.

Writing copy that reads well and adheres to SEO best practices can be challenging unless you’ve done it before.

We have an entire piece dedicated to helping you master the art, but some of the key takeaways include:

• Emphasize readability: Your content should be easily scannable so users can quickly find the information they want.
• Don’t overuse keywords: Keyword stuffing is a technique used by unscrupulous SEO professionals to game the system. Google looks down on sites that overuse keywords. If caught, your page could be demoted in SERPs or removed altogether.
• Keep sentences and paragraphs brief: If you’ve ever clicked on a webpage only to be assaulted by an unbroken wall of text, you know how hard it is to read lengthy pieces of copy. Avoid driving users away by keeping your sentences and paragraphs short.
• Use subheadings: Subheads stand out because of their size, attracting attention from people scanning your page. Use an ample amount of content to guide readers down the page.
• Use bulleted lists: This may feel very meta, but bulleted lists are an excellent way to break information into easily digestible chunks. Use them whenever they make sense.
• Add personal experience: Where relevant, discuss the author’s experience, background, and hands-on knowledge related to the topic to demonstrate experience credentials.

4. Freshness

For rapidly evolving topics, keeping your content fresh and providing new value as you learn more about your audience’s needs is critical.

Google rewards sites that maintain their content rather than letting it become stale or outdated.

Some tips:

• Update content regularly with new information, insights, or angles.
• Fix inaccuracies or outdated information promptly.
• Expand content to cover newly discovered areas of audience interest.
• Consider content exports or opt-in offers for frequently updated content.

5. Visual Assets

Adding pictures, videos, charts, and other eye-catching visuals makes it more attractive for visitors and improves its appearance in search results.

Optimizing images can also help you to gain more visibility through image search and in the SERP image carousel.

To make your content easy to find in text searches and image-based searches, here are some tips:

• Provide contextual information and relevant details in image captions.
• Implement schema markup for images, videos, products, etc., to enhance search visibility.
• Ensure visual assets are high-quality, original, and relevant to the page content.
• For ecommerce sites, provide multiple clear product images from various angles.

As computer vision models advance, search engines will better understand and surface relevant images and videos.

Optimizing for visual search now can help future-proof your content.

Read More:

HTML

HyperText Markup Language or HTML is the standard markup language used to structure your webpage and content. It tells the user’s browser where to display what on the page and it also tells search engines what your page is about.

Here are the on-page SEO HTML factors you need to consider:

6. Title Tags

This is one of those areas where it’s essential to focus on the details.

On its own, this snippet of code probably isn’t going to have you shooting up SERP rankings.

However, when combined with other on-page elements (like the ones discussed here), title tags can help you provide context and demonstrate your site’s relevancy.

For a more thorough look at how to optimize your title tags, read this.

7. Meta Description

A veteran SEO professional is throwing up their hands at the screen. “Oh, come on,” they’re saying, “Everyone knows meta descriptions aren’t an SEO ranking factor.”

They’re only partly correct. While it’s true there’s a lot of evidence against meta descriptions as a ranking factor, they’re wrong about everyone knowing that.

But, don’t let them dissuade you from adding meta descriptions to your site.

Despite their relative lack of SEO use, descriptions offer two key benefits:

• They can help Google understand what your webpage is all about.
• They have an outsized influence on your CTRs.

Better meta descriptions give searchers a better understanding of your page, leading to more click-throughs. So, don’t neglect them.

8. Image Optimization

We discussed the importance of visual on-page assets on your page, now it’s time to examine their technical aspects more closely.

Here are some tips to help optimize yours:

• Include SEO-friendly alt tags.
• Choose the proper format and file size for fast loading.
• Customize file names instead of using something like IMG_08759.
• Ensure your images are mobile-friendly.

Once again, we have an excellent resource for more in-depth information on HTML image optimization. Read it here.

9. Geotagging (For Local Search)

It may be a global economy, but most business is still done at a local level. Connect with the people in your neighborhood by optimizing your on-page local SEO.

There are three main SEO tactics to consider when focusing on local traffic:

• Optimizing listings and citations, including name, address, phone number (NAP), website URL, business descriptions, and getting reviews.
• Optimizing local content, including accommodating “near me” searches, providing location-based content, or buying a local website or blog.
•  Building links with other local businesses and organizations.

Some additional local SEO tactics to incorporate:

• Implement localized schema markup for local business listings, events, special offers, etc.
• Optimize Google Business Profile with up-to-date info, photos, posts, Q&A, and locally relevant content.
• Leverage proximity and geolocation data for mobile search.
• Create location-specific pages, content hubs, or microsites.

Examples of effective local SEO could look like:

• A restaurant featuring locally sourced food specialties on dedicated pages.
• A service provider’s site with geo-pages for all service areas.
• An ecommerce store highlighting inventory available for local pickup.

For more information on building your geotagging SEO strategy, read this.

Read More:

Website Architecture

Having a well-structured website is essential for two reasons: First, a website laid out logically will be crawled more effectively by search engines, and second, it will create richer user experiences.

Here are the factors to consider when optimizing your site’s architecture:

10. Site Speed

A clunky, slow-loading site does more than frustrate and drive away visitors – it hurts your search ranking, too.

Search Engine Journal investigated the effect of a page’s loading time on SEO and confirmed that page speed is a ranking factor in search results.

However, the minimum speed your site needs to meet is constantly changing.

This can be achieved by meeting Google’s Core Web Vitals minimum threshold. If your site isn’t currently meeting these standards, there are several steps you can take, including:

• Enabling compression.
• Reducing redirects.
• Optimizing images.
• Leveraging browser caches.

11. Responsive Design

Mobile search volume surpassed desktop in 2016 and has only grown since then.

Because more users are on mobile devices, Google followed the logical path and began prioritizing sites with designs that adapt to mobile screens.

While ranking in search results without a responsive design is still possible, Google strongly recommends having one.

You can read more about the effect site responsiveness has on search results here.

12. URL Structure

There was a time when URLs played a prominent role in SEO. Businesses would include keywords in domain names to help them rank higher.

But Google, doing what Google does, changed the algorithm. What was once so important to rankings now plays a much smaller role.

That’s not to say it doesn’t matter. Search engines still include your URLs in your overall score – they just don’t hold the same prominence they once did.

However, there is evidence they play a role in a site’s initial ranking, and some professionals believe they’re used to group pages. While they shouldn’t be your top SEO priority, you don’t want to ignore them.

Read more about how URLs factor into Google rankings here.

13. Links

Remember E-E-A-T from way back at the beginning of this article?

One of the best ways your website can establish it is through links from other reputable websites.

Think of it this way: Who would you rather trust your 401(k) to – a financial advisor who manages Warren Buffet’s portfolio or your cousin Jimmy, who lives in your aunt’s basement? Jimmy might do a fine job, potentially even outperforming Buffet’s guy. But he doesn’t have the credibility that comes with a strong co-sign.

Links work in the same way.

There are three main types you need to know about for SEO:

Of the three, inbound links are the most important for boosting E-E-A-T signals. High-quality, relevant inbound links, especially from authoritative and experienced sources, can help demonstrate your site’s expertise, authoritativeness, and trustworthiness.

SEO professionals use various methods to generate quality incoming links, including social media, creating sharable infographics, and even asking for backlinks.

But beware: Not all inbound links are helpful. Some, especially those from link farms, forum posts, and guestbooks, can be fake links that cheat the rankings system. If you don’t disavow these, it can hurt your ranking.

Here’s information on how and when to disavow links.

Read More:

On-Page SEO Vs. Off-Page SEO

We’ve talked a lot about on-page SEO, but there’s also something known as off-page SEO. The difference, as you could probably tell by the names, is where it happens.

On-page SEO is everything you can do internally to boost your rankings, including keyword optimization, meta descriptions, title tags, alt text, and website structure.

Off-page SEO refers to all external factors that impact your site’s rankings. This includes backlinks, E-E-A-T, local SEO, social media mentions, and pay-per-click.

You have much more control over your on-page SEO, but it’s also important to consider off-page SEO – you need both to achieve your goals.

However, it would be best to first focus on building a good, relevant webpage that’s fully optimized for search engines before you begin investing a lot of resources into building links and promoting your site.

Conclusion

As search algorithms evolve, the need to create high-quality, relevant content and optimize technical elements persists.

Key takeaways to remember:

1. Focus on creating valuable, user-centric content that demonstrates E-E-A-T.
2. Optimize technical elements like HTML tags, site speed, and mobile responsiveness.
3. Maintain a logical site structure and use internal linking effectively.
4. Regularly update and refresh content to maintain relevance.
5. Remember that on-page SEO works with off-page factors for overall SEO success.

Approach this as an ongoing process rather than a one-time fix.

Consistently implementing these tactics will considerably improve your chances of ranking well in search results.

More resources: 

---

Featured Image: BestForBest/Shutterstock

You might have heard of website crawling before — you may even have a vague idea of what it’s about — but do you know why it’s important, or what differentiates it from web crawling? (yes, there is a difference!) 

Search engines are increasingly ruthless when it comes to the quality of the sites they allow into the search results.

If you don’t grasp the basics of optimizing for web crawlers (and eventual users), your organic traffic may well pay the price.

A good website crawler can show you how to protect and even enhance your site’s visibility.

Here’s what you need to know about both web crawlers and site crawlers.

A web crawler is a software program or script that automatically scours the internet, analyzing and indexing web pages.

Also known as a web spider or spiderbot, web crawlers assess a page’s content to decide how to prioritize it in their indexes.

Googlebot, Google’s web crawler, meticulously browses the web, following links from page to page, gathering data, and processing content for inclusion in Google’s search engine.

How do web crawlers impact SEO?

Web crawlers analyze your page and decide how indexable or rankable it is, which ultimately determines your ability to drive organic traffic.

If you want to be discovered in search results, then it’s important you ready your content for crawling and indexing.

There are roughly seven stages to web crawling:

1. URL Discovery

When you publish your page (e.g. to your sitemap), the web crawler discovers it and uses it as a ‘seed’ URL. Just like seeds in the cycle of germination, these starter URLs allow the crawl and subsequent crawling loops to begin.

2. Crawling

After URL discovery, your page is scheduled and then crawled. Content like meta tags, images, links, and structured data are downloaded to the search engine’s servers, where they await parsing and indexing.

3. Parsing

Parsing essentially means analysis. The crawler bot extracts the data it’s just crawled to determine how to index and rank the page.

3a. The URL Discovery Loop

Also during the parsing phase, but worthy of its own subsection, is the URL discovery loop. This is when newly discovered links (including links discovered via redirects) are added to a queue of URLs for the crawler to visit. These are effectively new ‘seed’ URLs, and steps 1–3 get repeated as part of the ‘URL discovery loop’.

4. Indexing

While new URLs are being discovered, the original URL gets indexed. Indexing is when search engines store the data collected from web pages. It enables them to quickly retrieve relevant results for user queries.

5. Ranking

Indexed pages get ranked in search engines based on quality, relevance to search queries, and ability to meet certain other ranking factors. These pages are then served to users when they perform a search.

6. Crawl ends

Eventually the entire crawl (including the URL rediscovery loop) ends based on factors like time allocated, number of pages crawled, depth of links followed etc.

7. Revisiting

Crawlers periodically revisit the page to check for updates, new content, or changes in structure.

As you can probably guess, the number of URLs discovered and crawled in this process grows exponentially in just a few hops.

Search engine web crawlers are autonomous, meaning you can’t trigger them to crawl or switch them on/off at will.

You can, however, notify crawlers of site updates via:

XML sitemaps

An XML sitemap is a file that lists all the important pages on your website to help search engines accurately discover and index your content.

Google’s URL inspection tool

You can ask Google to consider recrawling your site content via its URL inspection tool in Google Search Console. You may get a message in GSC if Google knows about your URL but hasn’t yet crawled or indexed it. If so, find out how to fix “Discovered — currently not indexed”.

IndexNow

Instead of waiting for bots to re-crawl and index your content, you can use IndexNow to automatically ping search engines like Bing, Yandex, Naver, Seznam.cz, and Yep, whenever you:

• Add new pages
• Update existing content
• Remove outdated pages
• Implement redirects

You can set up automatic IndexNow submissions via Ahrefs Site Audit.

Search engine crawling decisions are dynamic and a little obscure.

Although we don’t know the definitive criteria Google uses to determine when or how often to crawl content, we’ve deduced three of the most important areas.

This is based on breadcrumbs dropped by Google, both in support documentation and during rep interviews.

1. Prioritize quality

Google PageRank evaluates the number and quality of links to a page, considering them as “votes” of importance.

Pages earning quality links are deemed more important and are ranked higher in search results.

PageRank is a foundational part of Google’s algorithm. It makes sense then that the quality of your links and content plays a big part in how your site is crawled and indexed.

To judge your site’s quality, Google looks at factors such as:

To assess the pages on your site with the most links, check out the Best by Links report.

Pay attention to the “First seen”, “Last check” column, which reveals which pages have been crawled most often, and when.

2. Keep things fresh

According to Google’s Senior Search Analyst, John Mueller…

Search engines recrawl URLs at different rates, sometimes it’s multiple times a day, sometimes it’s once every few months.

But if you regularly update your content, you’ll see crawlers dropping by more often.

Search engines like Google want to deliver accurate and up-to-date information to remain competitive and relevant, so updating your content is like dangling a carrot on a stick.

You can examine just how quickly Google processes your updates by checking your crawl stats in Google Search Console.

While you’re there, look at the breakdown of crawling “By purpose” (i.e. percent split of pages refreshed vs pages newly discovered). This will also help you work out just how often you’re encouraging web crawlers to revisit your site.

To find specific pages that need updating on your site, head to the Top Pages report in Ahrefs Site Explorer, then:

1. Set the traffic filter to “Declined”
2. Set the comparison date to the last year or two
3. Look at Content Changes status and update pages with only minor changes

Top Pages shows you the content on your site driving the most organic traffic. Pushing updates to these pages will encourage crawlers to visit your best content more often, and (hopefully) boost any declining traffic.

3. Refine your site structure

Offering a clear site structure via a logical sitemap, and backing that up with relevant internal links will help crawlers:

• Better navigate your site
• Understand its hierarchy
• Index and rank your most valuable content

Combined, these factors will also please users, since they support easy navigation, reduced bounce rates, and increased engagement.

Below are some more elements that can potentially influence how your site gets discovered and prioritized in crawling:

Web crawlers like Google crawl the entire internet, and you can’t control which sites they visit, or how often.

But you can use website crawlers, which are like your own private bots.

Ask them to crawl your website to find and fix important SEO problems, or study your competitors’ site, turning their biggest weaknesses into your opportunities.

Site crawlers essentially simulate search performance. They help you understand how a search engine’s web crawlers might interpret your pages, based on their:

• Structure
• Content
• Meta data
• Page load speed
• Errors
• Etc

Example: Ahrefs Site Audit

The Ahrefs Site Audit crawler powers the tools: RankTracker, Projects, and Ahrefs’ main website crawling tool: Site Audit.

Site Audit helps SEOs to:

• Analyze 170+ technical SEO issues
• Conduct on-demand crawls, with live site performance data
• Assess up to 170k URLs a minute
• Troubleshoot, maintain, and improve their visibility in search engines

From URL discovery to revisiting, website crawlers operate very similarly to web crawlers – only instead of indexing and ranking your page in the SERPs, they store and analyze it in their own database.

You can crawl your site either locally or remotely. Desktop crawlers like ScreamingFrog let you download and customize your site crawl, while cloud-based tools like Ahrefs Site Audit perform the crawl without using your computer’s resources – helping you work collaboratively on fixes and site optimization.

If you want to scan entire websites in real time to detect technical SEO problems, configure a crawl in Site Audit.

It will give you visual data breakdowns, site health scores, and detailed fix recommendations to help you understand how a search engine interprets your site.

1. Set up your crawl

Navigate to the Site Audit tab and choose an existing project, or set one up.

A project is any domain, subdomain, or URL you want to track over time.

Once you’ve configured your crawl settings – including your crawl schedule and URL sources – you can start your audit and you’ll be notified as soon as it’s complete.

Here are some things you can do right away.

2. Diagnose top errors

The Top Issues overview in Site Audit shows you your most pressing errors, warnings, and notices, based on the number of URLs affected.

Working through these as part of your SEO roadmap will help you:

1. Spot errors (red icons) impacting crawling – e.g.

• HTTP status code/client errors
• Broken links
• Canonical issues

2. Optimize your content and rankings based on warnings (yellow) – e.g.

• Missing alt text
• Links to redirects
• Overly long meta descriptions

3. Maintain steady visibility with notices (blue icon) – e.g.

• Organic traffic drops
• Multiple H1s
• Indexable pages not in sitemap

Filter issues

You can also prioritize fixes using filters.

Say you have thousands of pages with missing meta descriptions. Make the task more manageable and impactful by targeting high traffic pages first.

1. Head to the Page Explorer report in Site Audit
2. Select the advanced filter dropdown
3. Set an internal pages filter
4. Select an ‘And’ operator
5. Select ‘Meta description’ and ‘Not exists’
6. Select ‘Organic traffic > 100’

Crawl the most important parts of your site

Segment and zero-in on the most important pages on your site (e.g. subfolders or subdomains) using Site Audit’s 200+ filters – whether that’s your blog, ecommerce store, or even pages that earn over a certain traffic threshold.

3. Expedite fixes

If you don’t have coding experience, then the prospect of crawling your site and implementing fixes can be intimidating.

If you do have dev support, issues are easier to remedy, but then it becomes a matter of bargaining for another person’s time.

We’ve got a new feature on the way to help you solve for these kinds of headaches.

Coming soon, Patches are fixes you can make autonomously in Site Audit.

Title changes, missing meta descriptions, site-wide broken links – when you face these kinds of errors you can hit “Patch it” to publish a fix directly to your website, without having to pester a dev.

And if you’re unsure of anything, you can roll-back your patches at any point.

4. Spot optimization opportunities

Auditing your site with a website crawler is as much about spotting opportunities as it is about fixing bugs.

Improve internal linking

The Internal Link Opportunities report in Site Audit shows you relevant internal linking suggestions, by taking the top 10 keywords (by traffic) for each crawled page, then looking for mentions of them on your other crawled pages.

‘Source’ pages are the ones you should link from, and ‘Target’ pages are the ones you should link to.

The more high quality connections you make between your content, the easier it will be for Googlebot to crawl your site.

Final thoughts

Understanding website crawling is more than just an SEO hack – it’s foundational knowledge that directly impacts your traffic and ROI.

Knowing how crawlers work means knowing how search engines “see” your site, and that’s half the battle when it comes to ranking.

Google was recently granted a patent on ranking web pages, which may offer insights into how AI Overviews ranks content. The patent describes a method for ranking pages based on what a user might be interested in next.

Contextual Estimation Of Link Information Gain

The name of the patent is Contextual Estimation Of Link Information Gain, it was filed in 2018 and granted in June 2024. It’s about calculating a ranking score called Information Gain that is used to rank a second set of web pages that are likely to be of interest to a user as a slightly different follow-up topic related to a previous question.

The patent starts with general descriptions then adds layers of specifics over the course of paragraphs.  An analogy can be that it’s like a pizza. It starts out as a mozzarella pizza, then they add mushrooms, so now it’s a mushroom pizza. Then they add onions, so now it’s a mushroom and onion pizza. There are layers of specifics that build up to the entire context.

So if you read just one section of it, it’s easy to say, “It’s clearly a mushroom pizza” and be completely mistaken about what it really is.

There are layers of context but what it’s building up to is:

• Ranking a web page that is relevant for what a user might be interested in next.
• The context of the invention is an automated assistant or chatbot
• A search engine plays a role in a way that seems similar to Google’s AI Overviews

Information Gain And SEO: What’s Really Going On?

A couple of months ago I read a comment on social media asserting that “Information Gain” was a significant factor in a recent Google core algorithm update.  That mention surprised me because I’d never heard of information gain before. I asked some SEO friends about it and they’d never heard of it either.

What the person on social media had asserted was something like Google was using an “Information Gain” score to boost the ranking of web pages that had more information than other web pages. So the idea was that it was important to create pages that have more information than other pages, something along those lines.

So I read the patent and discovered that “Information Gain” is not about ranking pages with more information than other pages. It’s really about something that is more profound for SEO because it might help to understand one dimension of how AI Overviews might rank web pages.

TL/DR Of The Information Gain Patent

What the information gain patent is really about is even more interesting because it may give an indication of how AI Overviews (AIO) ranks web pages that a user might be interested next.  It’s sort of like introducing personalization by anticipating what a user will be interested in next.

The patent describes a scenario where a user makes a search query and the automated assistant or chatbot provides an answer that’s relevant to the question. The information gain scoring system works in the background to rank a second set of web pages that are relevant to a what the user might be interested in next. It’s a new dimension in how web pages are ranked.

The Patent’s Emphasis on Automated Assistants

There are multiple versions of the Information Gain patent dating from 2018 to 2024. The first version is similar to the last version with the most significant difference being the addition of chatbots as a context for where the information gain invention is used.

The patent uses the phrase “automated assistant” 69 times and uses the phrase “search engine” only 25 times.  Like with AI Overviews, search engines do play a role in this patent but it’s generally in the context of automated assistants.

As will become evident, there is nothing to suggest that a web page containing more information than the competition is likelier to be ranked higher in the organic search results. That’s not what this patent talks about.

General Description Of Context

All versions of the patent describe the presentation of search results within the context of an automated assistant and natural language question answering. The patent starts with a general description and progressively becomes more specific. This is a feature of patents in that they apply for protection for the widest contexts in which the invention can be used and become progressively specific.

The entire first section (the Abstract) doesn’t even mention web pages or links. It’s just about the information gain score within a very general context:

“An information gain score for a given document is indicative of additional information that is included in the document beyond information contained in documents that were previously viewed by the user.”

That is a nutshell description of the patent, with the key insight being that the information gain scoring happens on pages after the user has seen the first search results.

More Specific Context: Automated Assistants

The second paragraph in the section titled “Background” is slightly more specific and adds an additional layer of context for the invention because it mentions  links. Specifically, it’s about a user that makes a search query and receives links to search results – no information gain score calculated yet.

The Background section says:

“For example, a user may submit a search request and be provided with a set of documents and/or links to documents that are responsive to the submitted search request.”

The next part builds on top of a user having made a search query:

“Also, for example, a user may be provided with a document based on identified interests of the user, previously viewed documents of the user, and/or other criteria that may be utilized to identify and provide a document of interest. Information from the documents may be provided via, for example, an automated assistant and/or as results to a search engine. Further, information from the documents may be provided to the user in response to a search request and/or may be automatically served to the user based on continued searching after the user has ended a search session.”

That last sentence is poorly worded.

Here’s the original sentence:

“Further, information from the documents may be provided to the user in response to a search request and/or may be automatically served to the user based on continued searching after the user has ended a search session.”

Here’s how it makes more sense:

“Further, information from the documents may be provided to the user… based on continued searching after the user has ended a search session.”

The information provided to the user is “in response to a search request and/or may be automatically served to the user”

It’s a little clearer if you put parentheses around it:

Further, information from the documents may be provided to the user (in response to a search request and/or may be automatically served to the user) based on continued searching after the user has ended a search session.

Takeaways:

• The patent describes identifying documents that are relevant to the “interests of the user” based on “previously viewed documents” “and/or other criteria.”
• It sets a general context of an automated assistant “and/or” a search engine
• Information from the documents that are based on “previously viewed documents” “and/or other criteria” may be shown after the user continues searching.

More Specific Context: Chatbot

The patent next adds an additional layer of context and specificity by mentioning how chatbots can “extract” an answer from a web page (“document”) and show that as an answer. This is about showing a summary that contains the answer, kind of like featured snippets, but within the context of a chatbot.

The patent explains:

“In some cases, a subset of information may be extracted from the document for presentation to the user. For example, when a user engages in a spoken human-to-computer dialog with an automated assistant software process (also referred to as “chatbots,” “interactive personal assistants,” “intelligent personal assistants,” “personal voice assistants,” “conversational agents,” “virtual assistants,” etc.), the automated assistant may perform various types of processing to extract salient information from a document, so that the automated assistant can present the information in an abbreviated form.

As another example, some search engines will provide summary information from one or more responsive and/or relevant documents, in addition to or instead of links to responsive and/or relevant documents, in response to a user’s search query.”

The last sentence sounds like it’s describing something that’s like a featured snippet or like AI Overviews where it provides a summary. The sentence is very general and ambiguous because it uses “and/or” and “in addition to or instead of” and isn’t as specific as the preceding sentences. It’s an example of a patent being general for legal reasons.

Ranking The Next Set Of Search Results

The next section is called the Summary and it goes into more details about how the Information Gain score represents how likely the user will be interested in the next set of documents. It’s not about ranking search results, it’s about ranking the next set of search results (based on a related topic).

It states:

“An information gain score for a given document is indicative of additional information that is included in the given document beyond information contained in other documents that were already presented to the user.”

Ranking Based On Topic Of Web Pages

It then talks about presenting the web page in a browser, audibly reading the relevant part of the document or audibly/visually presenting a summary of the document (“audibly/visually presenting salient information extracted from the document to the user, etc.”)

But the part that’s really interesting is when it next explains using a topic of the web page as a representation of the the content, which is used to calculate the information gain score.

It describes many different ways of extracting the representation of what the page is about. But what’s important is that it’s describes calculating the Information Gain score based on a representation of what the content is about, like the topic.

“In some implementations, information gain scores may be determined for one or more documents by applying data indicative of the documents, such as their entire contents, salient extracted information, a semantic representation (e.g., an embedding, a feature vector, a bag-of-words representation, a histogram generated from words/phrases in the document, etc.) across a machine learning model to generate an information gain score.”

The patent goes on to describe ranking a first set of documents and using the Information Gain scores to rank additional sets of documents that anticipate follow up questions or a progression within a dialog of what the user is interested in.

The automated assistant can in some implementations query a search engine and then apply the Information Gain rankings to the multiple sets of search results (that are relevant to related search queries).

There are multiple variations of doing the same thing but in general terms this is what it describes:

“Based on the information gain scores, information contained in one or more of the new documents may be selectively provided to the user in a manner that reflects the likely information gain that can be attained by the user if the user were to be presented information from the selected documents.”

What All Versions Of The Patent Have In Common

All versions of the patent share general similarities over which more specifics are layered in over time (like adding onions to a mushroom pizza). The following are the baseline of what all the versions have in common.

Application Of Information Gain Score

All versions of the patent describe applying the information gain score to a second set of documents that have additional information beyond the first set of documents. Obviously, there is no criteria or information to guess what the user is going search for when they start a search session. So information gain scores are not applied to the first search results.

Examples of passages that are the same for all versions:

• A second set of documents is identified that is also related to the topic of the first set of documents but that have not yet been viewed by the user.
• For each new document in the second set of documents, an information gain score is determined that is indicative of, for the new document, whether the new document includes information that was not contained in the documents of the first set of documents…

Automated Assistants

All four versions of the patent refer to automated assistants that show search results in response to natural language queries.

The 2018 and 2023 versions of the patent both mention search engines 25 times. The 2o18 version mentions “automated assistant” 74 times and the latest version mentions it 69 times.

They all make references to “conversational agents,” “interactive personal assistants,” “intelligent personal assistants,” “personal voice assistants,” and “virtual assistants.”

It’s clear that the emphasis of the patent is on automated assistants, not the organic search results.

Dialog Turns

Note: In everyday language we use the word dialogue. In computing they the spell it dialog.

All versions of the patents refer to a way of interacting with the system in the form of a dialog, specifically a dialog turn. A dialog turn is the back and forth that happens when a user asks a question using natural language, receives an answer and then asks a follow up question or another question altogether. This can be natural language in text, text to speech (TTS), or audible.

The main aspect the patents have in common is the back and forth in what is called a “dialog turn.” All versions of the patent have this as a context.

Here’s an example of how the dialog turn works:

“Automated assistant client 106 and remote automated assistant 115 can process natural language input of a user and provide responses in the form of a dialog that includes one or more dialog turns. A dialog turn may include, for instance, user-provided natural language input and a response to natural language input by the automated assistant.

Thus, a dialog between the user and the automated assistant can be generated that allows the user to interact with the automated assistant …in a conversational manner.”

Problems That Information Gain Scores Solve

The main feature of the patent is to improve the user experience by understanding the additional value that a new document provides compared to documents that a user has already seen. This additional value is what is meant by the phrase Information Gain.

There are multiple ways that information gain is useful and one of the ways that all versions of the patent describes is in the context of an audio response and how a long-winded audio response is not good, including in a TTS (text to speech) context).

The patent explains the problem of a long-winded response:

“…and so the user may wait for substantially all of the response to be output before proceeding. In comparison with reading, the user is able to receive the audio information passively, however, the time taken to output is longer and there is a reduced ability to scan or scroll/skip through the information.”

The patent then explains how information gain can speed up answers by eliminating redundant (repetitive) answers or if the answer isn’t enough and forces the user into another dialog turn.

This part of the patent refers to the information density of a section in a web page, a section that answers the question with the least amount of words. Information density is about how “accurate,” “concise,” and “relevant”‘ the answer is for relevance and avoiding repetitiveness. Information density is important for audio/spoken answers.

This is what the patent says:

“As such, it is important in the context of an audio output that the output information is relevant, accurate and concise, in order to avoid an unnecessarily long output, a redundant output, or an extra dialog turn.

The information density of the output information becomes particularly important in improving the efficiency of a dialog session. Techniques described herein address these issues by reducing and/or eliminating presentation of information a user has already been provided, including in the audio human-to-computer dialog context.”

The idea of “information density” is important in a general sense because it communicates better for users but it’s probably extra important in the context of being shown in chatbot search results, whether it’s spoken or not. Google AI Overviews shows snippets from a web page but maybe more importantly, communicating in a concise manner is the best way to be on topic and make it easy for a search engine to understand content.

Search Results Interface

All versions of the Information Gain patent are clear that the invention is not in the context of organic search results. It’s explicitly within the context of ranking web pages within a natural language interface of an automated assistant and an AI chatbot.

However, there is a part of the patent that describes a way of showing users with the second set of results within a “search results interface.” The scenario is that the user sees an answer and then is interested in a related topic. The second set of ranked web pages are shown in a “search results interface.”

The patent explains:

“In some implementations, one or more of the new documents of the second set may be presented in a manner that is selected based on the information gain stores. For example, one or more of the new documents can be rendered as part of a search results interface that is presented to the user in response to a query that includes the topic of the documents, such as references to one or more documents. In some implementations, these search results may be ranked at least in part based on their respective information gain scores.”

…The user can then select one of the references and information contained in the particular document can be presented to the user. Subsequently, the user may return to the search results and the references to the document may again be provided to the user but updated based on new information gain scores for the documents that are referenced.

In some implementations, the references may be reranked and/or one or more documents may be excluded (or significantly demoted) from the search results based on the new information gain scores that were determined based on the document that was already viewed by the user.”

What is a search results interface? I think it’s just an interface that shows search results.

Let’s pause here to underline that it should be clear at this point that the patent is not about ranking web pages that are comprehensive about a topic. The overall context of the invention is showing documents within an automated assistant.

A search results interface is just an interface, it’s never described as being organic search results, it’s just an interface.

There’s more that is the same across all versions of the patent but the above are the important general outlines and context of it.

Claims Of The Patent

The claims section is where the scope of the actual invention is described and for which they are seeking legal protection over. It is mainly focused on the invention and less so on the context. Thus, there is no mention of a search engines, automated assistants, audible responses, or TTS (text to speech) within the Claims section. What remains is the context of search results interface which presumably covers all of the contexts.

Context: First Set Of Documents

It starts out by outlining the context of the invention. This context is receiving a query, identifying the topic, and ranking a first group of relevant web pages (documents) and selecting at least one of them as being relevant and either showing the document or communicating the information from the document (like a summary).

“1. A method implemented using one or more processors, comprising: receiving a query from a user, wherein the query includes a topic; identifying a first set of documents that are responsive to the query, wherein the documents of the set of documents are ranked, and wherein a ranking of a given document of the first set of documents is indicative of relevancy of information included in the given document to the topic; selecting, based on the rankings and from the documents of the first set of documents, a most relevant document providing at least a portion of the information from the most relevant document to the user;”

Context: Second Set Of Documents

Then what immediately follows is the part about ranking a second set of documents that contain additional information. This second set of documents is ranked using the information gain scores to show more information after showing a relevant document from the first group.

This is how it explains it:

“…in response to providing the most relevant document to the user, receiving a request from the user for additional information related to the topic; identifying a second set of documents, wherein the second set of documents includes at one or more of the documents of the first set of documents and does not include the most relevant document; determining, for each document of the second set, an information gain score, wherein the information gain score for a respective document of the second set is based on a quantity of new information included in the respective document of the second set that differs from information included in the most relevant document; ranking the second set of documents based on the information gain scores; and causing at least a portion of the information from one or more of the documents of the second set of documents to be presented to the user, wherein the information is presented based on the information gain scores.”

Granular Details

The rest of the claims section contains granular details about the concept of Information Gain, which is a ranking of documents based on what the user already has seen and represents a related topic that the user may be interested in. The purpose of these details is to lock them in for legal protection as part of the invention.

Here’s an example:

The method of claim 1, wherein identifying the first set comprises:
causing to be rendered, as part of a search results interface that is presented to the user in response to a previous query that includes the topic, references to one or more documents of the first set;
receiving user input that that indicates selection of one of the references to a particular document of the first set from the search results interface, wherein at least part of the particular document is provided to the user in response to the selection;

To make an analogy, it’s describing how to make the pizza dough, clean and cut the mushrooms, etc. It’s not important for our purposes to understand it as much as the general view of what the patent is about.

Information Gain Patent

An opinion was shared on social media that this patent has something to do with ranking web pages in the organic search results, I saw it, read the patent and discovered that’s not how the patent works. It’s a good patent and it’s important to correctly understand it. I analyzed multiple versions of the patent to see what they  had in common and what was different.

A careful reading of the patent shows that it is clearly focused on anticipating what the user may want to see based on what they have already seen. To accomplish this the patent describes the use of an Information Gain score for ranking web pages that are on topics that are related to the first search query but not specifically relevant to that first query.

The context of the invention is generally automated assistants, including chatbots. A search engine could be used as part of finding relevant documents but the context is not solely an organic search engine.

This patent could be applicable to the context of AI Overviews. I would not limit the context to AI Overviews as there are additional contexts such as spoken language in which Information Gain scoring could apply. Could it apply in additional contexts like Featured Snippets? The patent itself is not explicit about that.

Read the latest version of Information Gain patent:

Contextual estimation of link information gain

Featured Image by Shutterstock/Khosro