Connect with us


What would it take for new search engines to succeed?



Neeva and seek to pick up segments of users overlooked by Google, and with shifting user preferences and increased governmental scrutiny, there might be an opportunity.

For over half of its 22-year history, Google has been the most prevalent search engine in the United States. Over that term, its perception has gone from quintessential Silicon Valley startup and underdog to the gatekeeper of the internet, presiding over algorithms that have massive business implications and developing a reputation for expanding its business into different sectors in the name of providing a better experience to its users.

More recently, increased scrutiny over its business practices has led government regulators to crack down on perceived improprieties and some users have shown a slight sway towards a more privacy-oriented search experience. Pushback has also come from other search engines, such as DuckDuckGo and Ecosia, which have been vocal critics of how Google is presenting search engine alternatives to Android users in the EU.

The groundswell of resistance to the market leader may create the right circumstances for alternative search engines to assert themselves. Founded by former Google ad boss Sridhar Ramaswamy, Neeva was announced in June, and was announced this month by former Salesforce Chief Scientist Richard Socher. While taking a significant slice of search market share away from Google might be part of their overall goal, and what many marketers would like to see, success as a new search engine is contingent upon many factors and may come in a more modest form.

Regulators want to see more competition in search

Over the last few years, Google has been facing increased scrutiny over alleged anticompetitive practices both in the US and abroad. In 2018, it was penalized €4.3 billion (roughly $5 billion), the largest antitrust fine ever imposed by the European Commission (EC) — that’s on top of the €2.4 billion ($2.7 billion) fine it levied on Google the year before for favoring its own content in search results.

Last year, 48 state attorneys general joined in an antitrust investigation of the company. On the federal level, the Department of Justice filed an antitrust suit against Google in October, alleging that it uses contracts and its market power to neutralize rivals.

If Google is found to have engaged in anticompetitive tactics, the question then turns to remedies. A report issued by the House Judiciary Subcommittee on Antitrust recommended a number of potential remedies, including “structural separation” to restore competition, but Google is projecting confidence and may take the fight to court.

If that happens, it could be two years before an initial judgment, and even then, the company may pursue an appeal. However, with scrutiny over the company’s dominant position coming to a head, potential competitors have been gradually coming out of the thicket and attempting to distinguish themselves from the market leader.

RELATED: Does the Google antitrust case make an Apple search engine more likely?

Can anyone actually go head-to-head with Google?

As Google exists today, creating a search engine that could meaningfully compete with it would necessitate “[Building] a product that shows results at least as relevant, useful, fast, and cognitively low-load as Google themselves, then build a brand that at least tens to hundreds of millions of people rapidly trust and prefer to Google,” according to Rand Fishkin, CEO and co-founder of SparkToro, adding that the latter half is a more realistic possibility due to Google’s “continuing tack away from ‘beloved startup’ and toward ‘evil empire’ over the last ~8 years.”

“To be honest, I don’t see how that will happen,” Eric Enge, principal at Perficient, told Search Engine Land, “Quality of search results has a lot to do with the data you have access to, and [the search engine] with the most data wins — I don’t see how anyone can catch Google in that regard.”

Fishkin shared a similar sentiment: “I don’t think, realistically, anyone can build a search engine close to Google’s quality without their years of data on what people searched for, clicked, and found valuable vs. not (via measuring things like bounces-back-to-the-SERPs, query modifications, regularly choosing result #8 over #1, etc). That’s Google’s real secret sauce — the ace no one can touch. Tragically, I don’t think many folks in search (including these two new companies [Neeva and]) realize how impossible a hurdle that is to overcome.”

While the bar is high, these two potential rivals presumably believe there are areas of opportunity that Google has yet to claim, and have been able to attract investment towards that cause. As of June, 2020, Neeva had already raised $37.5 million and employed 25 individuals, and is being backed by Salesforce CEO Marc Benioff and venture capitalist Jim Breyer. Funding, however, is unlikely to level the playing field — even with Microsoft’s massive resources, Bing has largely been unable to sway users or digital marketers away from Google.

Catching Google does not need to be the goal

Instead of building a search engine that could take Google head on, “the objective could be to build a strong cadre of users interested in building out their ‘own corner of the web,’” Enge said with regards to Neeva, adding that this strategy would not require billions of users to be a successful business.

DuckDuckGo employs a similar strategy to differentiate itself from others and appeal to privacy-minded users. In November 2020, the number of queries DuckDuckGo handles per month increased to an all-time high of nearly 2.4 billion. Nevertheless, it still only accounts for 2.3% search market share in the US, compared to Google’s 87.7%.

DuckDuckGo claims 2.3% of the domestic search market and Bing accounts for 6.5%. Google’s share is 87.7%. Source: Statcounter.

Neeva’s subscription-based service, which will reportedly cost less than $10 per month when it launches, seeks to deliver a personalized yet ad-free experience. And, it may not have as many technological hurdles to overcome as it will leverage existing content and data sources, including Bing search results, Apple Maps, and This may help Neeva save on its development budget coming out of the gate, and if it’s able to attract enough subscribers, the company aims to lower its monthly fee, which may make it a more attractive alternative for new users.

Although exact details haven’t been announced, the website makes multiple references to helping users with their online purchasing decisions. The company’s early access survey also asks numerous questions related to e-commerce.

Many of the questions on’s early access survey pertain to how users conduct e-commerce.

If wants to be a player in the e-commerce space, it may also have to compete with Amazon. If it decides to challenge Amazon directly, then “the biggest challenges … are scale of operators, especially on the shipping+delivery side, then the two-sided marketplace-building challenge of growing potential customers and stores/sellers, and finally, luring enough investment to withstand the sustained price wars Amazon’s willing to engage in,” Fishkin said.

“So many people are wired to start their shopping with Amazon that it seems highly improbable that anyone will accomplish that,” said Enge, caveating that could still build a strong user base through its differentiating features, such as a custom URL with the user’s name to facilitate sharing.

Distinguishing itself in the e-commerce sector may be a more realistic path, as companies such as Etsy, eBay, Shopify and, to some extent, Google Shopping have been able to carve out their own niches in the space. “A site like, if they do integrate in some level of e-commerce, only need to start producing millions of dollars of revenue to be off to a great start,” Enge reiterated, “While this is still a non-trivial accomplishment, it’s a lot simpler than trying to unseat Amazon.”

What success might look like for new search engines

The long-term viability of potential Google or Amazon competitors will partially depend on their ability to attract an initial audience, which is difficult to forecast at this point since neither Neeva or have released more details or an expected launch date.

“In my opinion, no,” Fishkin said when asked about whether he thinks Neeva or’s differentiating business models and features are enough to attract new users during their launch period. “If I were in their shoes, I’m not sure I’d announce my ‘secret sauce’ way of competing against the tech monopolies until after I launched — so hopefully they’ve got more up their sleeves,” he said. 

A component of the “secret sauce” may hinge on identifying users who have needs that Google isn’t currently addressing. “These users will likely still use Google quite a bit, but they might use Neeva or for specific scenarios,” Enge said, “If these companies can maintain this level of focus they could have a chance of success.”

If these companies take traditional venture capital, success may look more familiar: “Becoming a unicorn with $1B+ returned in shareholder value,” Fishkin said, adding, “That could be through an acquisition, of course, but hopefully some of these folks would aim to actually compete long-term against Google.”

The potential impact on Google

The existence of more viable competitors provides marketers and users with more options, but it may also, to some extent, influence Google’s strategy. That may force Google to respond by catering more to the preferences of users and search professionals, which might improve the landscape for everyone.

“Certain specific innovations might spark new innovations, or adoption by, Google,” Enge said, noting that even a large degree of success for potential rivals would still represent small numbers for Google, and that the impact is unlikely to be evident in the near term. 

“Google’s never had a real competitor before, but when they had a perceived one (in Facebook), they took some fairly clumsy steps,” Fishkin said, referring to the company’s now-defunct social networking platform Google Plus, “So maybe we could hope that would happen again and give rise to an actual market for search instead of just a monopoly.”

Author: George Nguyen


Google’s Answer To OpenAI’s ChatGPT?



Google Chat Bot Ai

CNBC reported the other day that Google is working on its own AI chatbot named Apprentice Bard. Apprentice Bard is reportedly built on Google’s AI LaMDA language model stack and while Google is being more cautious with this rollout, Google is working on testing an AI bot in search.

CNBC wrote, “As a result of ChatGPT, the LaMDA team has been asked to prioritize working on a response to ChatGPT,” read one internal memo viewed by CNBC. “In the short term, it takes precedence over other projects,” the email continued, warning that some employees stop attending certain unrelated meetings.

“Apprentice Bard looks similar to ChatGPT: Employees can enter a question in a dialog box and get a text answer, then give feedback on the response. Based on several responses viewed by CNBC, Apprentice Bard’s answers can include recent events, a feature ChatGPT doesn’t have yet,” CNBC said. This makes sense, as Google can crawl the web in almost real-time and process that information faster than any other company.

The examples given by CNBC show that Google even picked up on the Google layoff news and was able to respond to questions about this. Whereas ChatGPT only has content from 2021 or earlier.

Also, CNBC said Google is working on designing an alternative search interface to support this chat feature. CNBC said, “One view showed the home search page offering five different prompts for potential questions placed directly under the main search bar, replacing the current “I’m feeling lucky” bar. It also showed a small chat logo inside the far right end of the search bar.” “When a question is entered, the search results show a grey bubble directly under the search bar, offering more human-like responses than typical search results. Directly beneath that, the page suggests several follow-up questions related to the first one. Under that, it shows typical search results, including links and headlines.”

Super interesting stuff and I suspect that if Google does release something, it will be a lot better than what we’ve been seeing so far, if that is even imaginable…

Forum discussion at Twitter.

Source link

Continue Reading


A New Googlebot Crawler From Google Named GoogleProducer



Crochet Google Bot Spider

Google may be crawling the web with a new crawler, a new Googlebot, named GoogleProducer. This useragent is not listed on the official Google crawlers page but maybe it is too new to be listed yet?

Hernán Marsili spotted this and asked Google about this on Twitter. He said, “we are seeing a lot of traffic to publisher’s sites with a new user-agent ‘GoogleProducer; (+ ‘. Our WAF is currently blocking it, but it’s origin is actually Google Proxy hosts. Is this legit traffic?”

That link goes to this page that 404s within the Google News Producer section.

As an FYI, Google Producer is part of Google News and Google Currents, I believe. This help document from Google says, “You will use Google Currents producer to manage your issues (e.g., pricing, description, etc.) for Google Play Magazines. If you’re participating in Google Currents, Producer will look familiar to you. You can manage both your edition(s) for Google Currents and your issues for Google Play Magazines through Producer. See our article on publisher account setup for the steps you need to take to fully set up your publisher account and magazines for Play. If you have any questions about using Producer for Google Currents, please see the Currents Producer Help Center.” Note, the Currents Producer Help Center redirects to the Google Publisher Center help center.

John Mueller of Google said he will look into it:

I guess we will see what he says but until then, I do suspect this is a legit Googlebot crawler.

Update: This is old, an older one:

Forum discussion at Twitter.

Source link

Continue Reading


Microsoft Bing Says The lastmod Tag In XML Sitemap File Is Critical




Microsoft Bing posted a new blog post saying “for XML sitemaps, one of the most critical tags you can include in your sitemap is the “lastmod” tag.” And it will become even more critical as Bing is reworking its crawl scheduling stack to rely more on this lastmod field.

Yes, by June, the way Bing decides what to crawl will be more dependent on the lastmod tag. Fabrice Canel from Microsoft wrote, “we are revamping our crawl scheduling stack to better utilize the information provided by the “lastmod” tag in sitemaps.” This is being done so it can “enhance” the “crawl efficiency by reducing unnecessary crawling of unchanged content and prioritizing recently updated content.”

“We have already begun implementing these changes on a limited scale and plan to fully roll them out by June,” he added.

So making sure your lastmod date is accurate is now even more important. It should be the last time you modified the URL, not the time the URL was first published and not the time the XML sitemap file was generated. In fact, that is the biggest issue Bing found with the field, that it often just shows the date the XML sitemap file was generated and not the date the page of the URL was last modified.

Here are some data points Bing put together on XML sitemaps:

  • 58% of hosts have at least one XML sitemap.
  • 84% of these sitemaps have a lastmod attribute set.
  • 79% have lastmod values correct.
  • 18% have lastmod values not correctly set.
  • 3% has lastmod values for only some of the URLs.
  • 16% of these sitemaps don’t have a lastmod attribute set.
  • 42% of hosts don’t have one XML sitemap

Oh, Bing still wants you to use the IndexNow protocol for the most efficient crawl solution but if you don’t – make sure your lastmod date is accurate.

In terms of Google, in 2015 Google said they don’t really use the lastmod date but then changed that in 2020 they said they do. The current Google documentation says, “Google uses the lastmod value if it’s consistently and verifiably (for example by comparing to the last modification of the page) accurate.”

Forum discussion at Twitter.

Source link

Continue Reading