Connect with us


Google LIMoE – A Step Towards Goal Of A Single AI



Google LIMoE - A Step Towards Goal Of A Single AI

Google announced a new technology called LIMoE that it says represents a step toward reaching Google’s goal of an AI architecture called Pathways.

Pathways is an AI architecture that is a single model that can learn to do multiple tasks that are currently accomplished by employing multiple algorithms.

LIMoE is an acronym that stands for Learning Multiple Modalities with One Sparse Mixture-of-Experts Model. It’s a model that processes vision and text together.

While there are other architectures that to do similar things, the breakthrough is in the way the new model accomplishes these tasks, using a neural network technique called a Sparse Model.

The sparse model is described in a research paper from 2017 that introduced the Mixture-of-Experts layer (MoE) approach, in a research paper titled, Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.


In 2021 Google announced a MoE model called GLaM: Efficient Scaling of Language Models with Mixture-of-Experts that was trained just on text.

The difference with LIMoE is that it works on text and images simultaneously.

The sparse model is different from the the “dense” models in that instead of devoting every part of the model to accomplishing a task, the sparse model assigns the task to various “experts” that specialize in a part of the task.

What this does is to lower the computational cost, making the model more efficient.

So, similar to how a brain sees a dog and know it’s a dog, that it’s a pug and that the pug displays a silver fawn color coat, this model can also view an image and accomplish the task in a similar way, by assigning computational tasks to different experts that specialize in the task of recognizing a dog, its breed, its color, etc.

The LIMoE model routes the problems to the “experts” specializing in a particular task, achieving similar or better results than current approaches to solving problems.


An interesting feature of the model is how some of the experts specialize mostly in processing images, others specialize mostly in processing text and some experts specialize in doing both.

Google’s description of how LIMoE works shows how there’s an expert on eyes, another for wheels, an expert for striped textures, solid textures, words, door handles, food & fruits, sea & sky, and an expert for plant images.

The announcement about the new algorithm describes these experts:

“There are also some clear qualitative patterns among the image experts — e.g., in most LIMoE models, there is an expert that processes all image patches that contain text. …one expert processes fauna and greenery, and another processes human hands.”

Experts that specialize in different parts of the problems provide the ability to scale and to accurately accomplish many different tasks but at a lower computational cost.

The research paper summarizes their findings:

  • “We propose LIMoE, the first large-scale multimodal mixture of experts models.
  • We demonstrate in detail how prior approaches to regularising mixture of experts models fall short for multimodal learning, and propose a new entropy-based regularisation scheme to stabilise training.
  • We show that LIMoE generalises across architecture scales, with relative improvements in zero-shot ImageNet accuracy ranging from 7% to 13% over equivalent dense models.
  • Scaled further, LIMoE-H/14 achieves 84.1% zeroshot ImageNet accuracy, comparable to SOTA contrastive models with per-modality backbones and pre-training.”

Matches State of the Art

There are many research papers published every month. But only a few are highlighted by Google.

Typically Google spotlights research because it accomplishes something new, in addition to attaining a state of the art.


LIMoE accomplishes this feat of attaining comparable results to today’s best algorithms but does it more efficiently.

The researchers highlight this advantage:

“On zero-shot image classification, LIMoE outperforms both comparable dense multimodal models and two-tower approaches.

The largest LIMoE achieves 84.1% zero-shot ImageNet accuracy, comparable to more expensive state-of-the-art models.

Sparsity enables LIMoE to scale up gracefully and learn to handle very different inputs, addressing the tension between being a jack-of-all-trades generalist and a master-of-one specialist.”

The successful outcomes of LIMoE led the researchers to observe that LIMoE could be a way forward for achieving a multimodal generalist model.

The researchers observed:


“We believe the ability to build a generalist model with specialist components, which can decide how different modalities or tasks should interact, will be key to creating truly multimodal multitask models which excel at everything they do.

LIMoE is a promising first step in that direction.”

Potential Shortcomings, Biases & Other Ethical Problems

There are shortcomings to this architecture that are not discussed in Google’s announcement but are mentioned in the research paper itself.

The research paper notes that, similar to other large-scale models, LIMoE may also introduce biases into the results.

The researchers state that they have not yet “explicitly” addressed the problems inherent in large scale models.

They write:

“The potential harms of large scale models…, contrastive models… and web-scale multimodal data… also carry over here, as LIMoE does not explicitly address them.”

The above statement makes a reference (in a footnote link) to a 2021 research paper called, On the Opportunities and Risks of Foundation Models (PDF here).


That research paper from 2021 warns how emergent AI technologies can cause negative societal impact such as:

“…inequity, misuse, economic and environmental impact, legal and ethical considerations.”

According to the cited paper, ethical problems can also arise from the tendency toward the homogenization of tasks, which can then introduce a point of failure that is then reproduced to other tasks that follow downstream.

The cautionary research paper states:

“The significance of foundation models can be summarized with two words: emergence and homogenization.

Emergence means that the behavior of a system is implicitly induced rather than explicitly constructed; it is both the source of scientific excitement and anxiety about unanticipated consequences.

Homogenization indicates the consolidation of methodologies for building machine learning systems across a wide range of applications; it provides strong leverage towards many tasks but also creates single points of failure.”

One area of caution is in vision related AI.


The 2021 paper states that the ubiquity of cameras means that any advances in AI related to vision could carry a concomitant risk toward the technology being applied in an unanticipated manner which can have a “disruptive impact,” including with regard to privacy and surveillance.

Another cautionary warning related to advances in vision related AI is problems with accuracy and bias.

They note:

“There is a well-documented history of learned bias in computer vision models, resulting in lower accuracies and correlated errors for underrepresented groups, with consequently inappropriate and premature deployment to some real-world settings.”

The rest of the paper documents how AI technologies can learn existing biases and perpetuate inequities.

“Foundation models have the potential to yield inequitable outcomes: the treatment of people that is unjust, especially due to unequal distribution along lines that compound historical discrimination…. Like any AI system, foundation models can compound existing inequities by producing unfair outcomes, entrenching systems of power, and disproportionately distributing negative consequences of technology to those already marginalized…”

The LIMoE researchers noted that this particular model may be able to work around some of the biases against underrepresented groups because of the nature of how the experts specialize in certain things.

These kinds of negative outcomes are not theories, they are realities and have already negatively impacted lives in real-world applications such as unfair racial-based biases introduced by employment recruitment algorithms.


The authors of the LIMoE paper acknowledge those potential shortcomings in a short paragraph that serves as a cautionary caveat.

But they also note that there may be a potential to address some of the biases with this new approach.

They wrote:

“…the ability to scale models with experts that can specialize deeply may result in better performance on underrepresented groups.”

Lastly, a key attribute of this new technology that should be noted is that there is no explicit use stated for it.

It’s simply a technology that can process images and text in an efficient manner.

How it can be applied, if it ever is applied in this form or a future form, is never addressed.


And that’s an important factor that is raised by the cautionary paper (Opportunities and Risks of Foundation Models), calls attention to in that researchers create capabilities for AI without consideration for how they can be used and the impact they may have on issues like privacy and security.

“Foundation models are intermediary assets with no specified purpose before they are adapted; understanding their harms requires reasoning about both their properties and the role they play in building task-specific models.”

All of those caveats are left out of Google’s announcement article but are referenced in the PDF version of the research paper itself.

Pathways AI Architecture & LIMoE

Text, images, audio data are referred to as modalities, different kinds of data or task specialization, so to speak. Modalities can also mean spoken language and symbols.

So when you see the phrase “multimodal” or “modalities” in scientific articles and research papers, what they’re generally talking about is different kinds of data.

Google’s ultimate goal for AI is what it calls the Pathways Next-Generation AI Architecture.

Pathways represents a move away from machine learning models that do one thing really well (thus requiring thousands of them) to a single model that does everything really well.


Pathways (and LIMoE) is a multimodal approach to solving problems.

It’s described like this:

“People rely on multiple senses to perceive the world. That’s very different from how contemporary AI systems digest information.

Most of today’s models process just one modality of information at a time. They can take in text, or images or speech — but typically not all three at once.

Pathways could enable multimodal models that encompass vision, auditory, and language understanding simultaneously.”

What makes LIMoE important is that it is a multimodal architecture that is referred to by the researchers as an “…important step towards the Pathways vision…

The researchers describe LIMoE a “step” because there is more work to be done, which includes exploring how this approach can work with modalities beyond just images and text.


This research paper and the accompanying summary article shows what direction Google’s AI research is going and how it is getting there.


Read Google’s Summary Article About LIMoE

LIMoE: Learning Multiple Modalities with One Sparse Mixture-of-Experts Model

Download and Read the LIMoE Research Paper

Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts (PDF)

Image by Shutterstock/SvetaZi

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address


Measuring Content Impact Across The Customer Journey




Measuring Content Impact Across The Customer Journey

Understanding the impact of your content at every touchpoint of the customer journey is essential – but that’s easier said than done. From attracting potential leads to nurturing them into loyal customers, there are many touchpoints to look into.

So how do you identify and take advantage of these opportunities for growth?

Watch this on-demand webinar and learn a comprehensive approach for measuring the value of your content initiatives, so you can optimize resource allocation for maximum impact.

You’ll learn:

  • Fresh methods for measuring your content’s impact.
  • Fascinating insights using first-touch attribution, and how it differs from the usual last-touch perspective.
  • Ways to persuade decision-makers to invest in more content by showcasing its value convincingly.

With Bill Franklin and Oliver Tani of DAC Group, we unravel the nuances of attribution modeling, emphasizing the significance of layering first-touch and last-touch attribution within your measurement strategy. 

Check out these insights to help you craft compelling content tailored to each stage, using an approach rooted in first-hand experience to ensure your content resonates.


Whether you’re a seasoned marketer or new to content measurement, this webinar promises valuable insights and actionable tactics to elevate your SEO game and optimize your content initiatives for success. 

View the slides below or check out the full webinar for all the details.

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading


How to Find and Use Competitor Keywords



How to Find and Use Competitor Keywords

Competitor keywords are the keywords your rivals rank for in Google’s search results. They may rank organically or pay for Google Ads to rank in the paid results.

Knowing your competitors’ keywords is the easiest form of keyword research. If your competitors rank for or target particular keywords, it might be worth it for you to target them, too.

There is no way to see your competitors’ keywords without a tool like Ahrefs, which has a database of keywords and the sites that rank for them. As far as we know, Ahrefs has the biggest database of these keywords.

How to find all the keywords your competitor ranks for

  1. Go to Ahrefs’ Site Explorer
  2. Enter your competitor’s domain
  3. Go to the Organic keywords report

The report is sorted by traffic to show you the keywords sending your competitor the most visits. For example, Mailchimp gets most of its organic traffic from the keyword “mailchimp.”

Mailchimp gets most of its organic traffic from the keyword, “mailchimp”.Mailchimp gets most of its organic traffic from the keyword, “mailchimp”.

Since you’re unlikely to rank for your competitor’s brand, you might want to exclude branded keywords from the report. You can do this by adding a Keyword > Doesn’t contain filter. In this example, we’ll filter out keywords containing “mailchimp” or any potential misspellings:

Filtering out branded keywords in Organic keywords reportFiltering out branded keywords in Organic keywords report

If you’re a new brand competing with one that’s established, you might also want to look for popular low-difficulty keywords. You can do this by setting the Volume filter to a minimum of 500 and the KD filter to a maximum of 10.

Finding popular, low-difficulty keywords in Organic keywordsFinding popular, low-difficulty keywords in Organic keywords

How to find keywords your competitor ranks for, but you don’t

  1. Go to Competitive Analysis
  2. Enter your domain in the This target doesn’t rank for section
  3. Enter your competitor’s domain in the But these competitors do section
Competitive analysis reportCompetitive analysis report

Hit “Show keyword opportunities,” and you’ll see all the keywords your competitor ranks for, but you don’t.

Content gap reportContent gap report

You can also add a Volume and KD filter to find popular, low-difficulty keywords in this report.

Volume and KD filter in Content gapVolume and KD filter in Content gap

How to find keywords multiple competitors rank for, but you don’t

  1. Go to Competitive Analysis
  2. Enter your domain in the This target doesn’t rank for section
  3. Enter the domains of multiple competitors in the But these competitors do section
Competitive analysis report with multiple competitorsCompetitive analysis report with multiple competitors

You’ll see all the keywords that at least one of these competitors ranks for, but you don’t.

Content gap report with multiple competitorsContent gap report with multiple competitors

You can also narrow the list down to keywords that all competitors rank for. Click on the Competitors’ positions filter and choose All 3 competitors:

Selecting all 3 competitors to see keywords all 3 competitors rank forSelecting all 3 competitors to see keywords all 3 competitors rank for
  1. Go to Ahrefs’ Site Explorer
  2. Enter your competitor’s domain
  3. Go to the Paid keywords report
Paid keywords reportPaid keywords report

This report shows you the keywords your competitors are targeting via Google Ads.

Since your competitor is paying for traffic from these keywords, it may indicate that they’re profitable for them—and could be for you, too.


You know what keywords your competitors are ranking for or bidding on. But what do you do with them? There are basically three options.

1. Create pages to target these keywords

You can only rank for keywords if you have content about them. So, the most straightforward thing you can do for competitors’ keywords you want to rank for is to create pages to target them.

However, before you do this, it’s worth clustering your competitor’s keywords by Parent Topic. This will group keywords that mean the same or similar things so you can target them all with one page.

Here’s how to do that:

  1. Export your competitor’s keywords, either from the Organic Keywords or Content Gap report
  2. Paste them into Keywords Explorer
  3. Click the “Clusters by Parent Topic” tab
Clustering keywords by Parent TopicClustering keywords by Parent Topic

For example, MailChimp ranks for keywords like “what is digital marketing” and “digital marketing definition.” These and many others get clustered under the Parent Topic of “digital marketing” because people searching for them are all looking for the same thing: a definition of digital marketing. You only need to create one page to potentially rank for all these keywords.

Keywords under the cluster of "digital marketing"Keywords under the cluster of "digital marketing"

2. Optimize existing content by filling subtopics

You don’t always need to create new content to rank for competitors’ keywords. Sometimes, you can optimize the content you already have to rank for them.

How do you know which keywords you can do this for? Try this:

  1. Export your competitor’s keywords
  2. Paste them into Keywords Explorer
  3. Click the “Clusters by Parent Topic” tab
  4. Look for Parent Topics you already have content about

For example, if we analyze our competitor, we can see that seven keywords they rank for fall under the Parent Topic of “press release template.”

Our competitor ranks for seven keywords that fall under the "press release template" clusterOur competitor ranks for seven keywords that fall under the "press release template" cluster

If we search our site, we see that we already have a page about this topic.

Site search finds that we already have a blog post on press release templatesSite search finds that we already have a blog post on press release templates

If we click the caret and check the keywords in the cluster, we see keywords like “press release example” and “press release format.”

Keywords under the cluster of "press release template"Keywords under the cluster of "press release template"

To rank for the keywords in the cluster, we can probably optimize the page we already have by adding sections about the subtopics of “press release examples” and “press release format.”

3. Target these keywords with Google Ads

Paid keywords are the simplest—look through the report and see if there are any relevant keywords you might want to target, too.

For example, Mailchimp is bidding for the keyword “how to create a newsletter.”

Mailchimp is bidding for the keyword “how to create a newsletter”Mailchimp is bidding for the keyword “how to create a newsletter”

If you’re ConvertKit, you may also want to target this keyword since it’s relevant.

If you decide to target the same keyword via Google Ads, you can hover over the magnifying glass to see the ads your competitor is using.

Mailchimp's Google Ad for the keyword “how to create a newsletter”Mailchimp's Google Ad for the keyword “how to create a newsletter”

You can also see the landing page your competitor directs ad traffic to under the URL column.

The landing page Mailchimp is directing traffic to for “how to create a newsletter”The landing page Mailchimp is directing traffic to for “how to create a newsletter”

Learn more

Check out more tutorials on how to do competitor keyword analysis:

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading


Google Confirms Links Are Not That Important




Google confirms that links are not that important anymore

Google’s Gary Illyes confirmed at a recent search marketing conference that Google needs very few links, adding to the growing body of evidence that publishers need to focus on other factors. Gary tweeted confirmation that he indeed say those words.

Background Of Links For Ranking

Links were discovered in the late 1990’s to be a good signal for search engines to use for validating how authoritative a website is and then Google discovered soon after that anchor text could be used to provide semantic signals about what a webpage was about.

One of the most important research papers was Authoritative Sources in a Hyperlinked Environment by Jon M. Kleinberg, published around 1998 (link to research paper at the end of the article). The main discovery of this research paper is that there is too many web pages and there was no objective way to filter search results for quality in order to rank web pages for a subjective idea of relevance.

The author of the research paper discovered that links could be used as an objective filter for authoritativeness.

Kleinberg wrote:


“To provide effective search methods under these conditions, one needs a way to filter, from among a huge collection of relevant pages, a small set of the most “authoritative” or ‘definitive’ ones.”

This is the most influential research paper on links because it kick-started more research on ways to use links beyond as an authority metric but as a subjective metric for relevance.

Objective is something factual. Subjective is something that’s closer to an opinion. The founders of Google discovered how to use the subjective opinions of the Internet as a relevance metric for what to rank in the search results.

What Larry Page and Sergey Brin discovered and shared in their research paper (The Anatomy of a Large-Scale Hypertextual Web Search Engine – link at end of this article) was that it was possible to harness the power of anchor text to determine the subjective opinion of relevance from actual humans. It was essentially crowdsourcing the opinions of millions of website expressed through the link structure between each webpage.

What Did Gary Illyes Say About Links In 2024?

At a recent search conference in Bulgaria, Google’s Gary Illyes made a comment about how Google doesn’t really need that many links and how Google has made links less important.

Patrick Stox tweeted about what he heard at the search conference:

” ‘We need very few links to rank pages… Over the years we’ve made links less important.’ @methode #serpconf2024″

Google’s Gary Illyes tweeted a confirmation of that statement:


“I shouldn’t have said that… I definitely shouldn’t have said that”

Why Links Matter Less

The initial state of anchor text when Google first used links for ranking purposes was absolutely non-spammy, which is why it was so useful. Hyperlinks were primarily used as a way to send traffic from one website to another website.

But by 2004 or 2005 Google was using statistical analysis to detect manipulated links, then around 2004 “powered-by” links in website footers stopped passing anchor text value, and by 2006 links close to the words “advertising” stopped passing link value, links from directories stopped passing ranking value and by 2012 Google deployed a massive link algorithm called Penguin that destroyed the rankings of likely millions of websites, many of which were using guest posting.

The link signal eventually became so bad that Google decided in 2019 to selectively use nofollow links for ranking purposes. Google’s Gary Illyes confirmed that the change to nofollow was made because of the link signal.

Google Explicitly Confirms That Links Matter Less

In 2023 Google’s Gary Illyes shared at a PubCon Austin that links were not even in the top 3 of ranking factors. Then in March 2024, coinciding with the March 2024 Core Algorithm Update, Google updated their spam policies documentation to downplay the importance of links for ranking purposes.

Google March 2024 Core Update: 4 Changes To Link Signal

The documentation previously said:


“Google uses links as an important factor in determining the relevancy of web pages.”

The update to the documentation that mentioned links was updated to remove the word important.

Links are not just listed as just another factor:

“Google uses links as a factor in determining the relevancy of web pages.”

At the beginning of April Google’s John Mueller advised that there are more useful SEO activities to engage on than links.

Mueller explained:

“There are more important things for websites nowadays, and over-focusing on links will often result in you wasting your time doing things that don’t make your website better overall”

Finally, Gary Illyes explicitly said that Google needs very few links to rank webpages and confirmed it.

Why Google Doesn’t Need Links

The reason why Google doesn’t need many links is likely because of the extent of AI and natural language undertanding that Google uses in their algorithms. Google must be highly confident in its algorithm to be able to explicitly say that they don’t need it.

Way back when Google implemented the nofollow into the algorithm there were many link builders who sold comment spam links who continued to lie that comment spam still worked. As someone who started link building at the very beginning of modern SEO (I was the moderator of the link building forum at the #1 SEO forum of that time), I can say with confidence that links have stopped playing much of a role in rankings beginning several years ago, which is why I stopped about five or six years ago.

Read the research papers

Authoritative Sources in a Hyperlinked Environment – Jon M. Kleinberg (PDF)

The Anatomy of a Large-Scale Hypertextual Web Search Engine

Featured Image by Shutterstock/RYO Alexandre


Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading


Follow by Email