A recent Google research paper on Long Form Question Answering illustrates how difficult it is to answer questions that need longer and nuanced answers. While the researchers were able to improve the state of the art of this kind of question answering, they also admitted that their results needed significant improvements.
I read this research paper last month when it was published and have been wanting to share it because it focuses on solving a shortcoming in search that isn’t discussed much at all.
I hope you find it as fascinating as I did!
What Search Engines Get Right
This research centers on Long Form Open-Domain Question Answering, an area that Natural Language processing continues to see improvements.
What search engines are good at is called, Factoid Open-domain Question Answering or simply Open-domain Question Answering.
Open Domain Question Answering is a task wherein an algorithm responds with an answer to a question in natural language.
What color is the sky? The sky is blue.
Long Form Question Answering (LFQA)
The research paper states that Long-form Question Answering (LFQA) is important but a challenge and that progress in being able to achieve this kind of question answering is not as far along as Open-domain Question Answering.
According to the research paper:
“Open-domain long-form question answering (LFQA) is a fundamental challenge in natural language processing (NLP) that involves retrieving documents relevant to a given question and using them to generate an elaborate paragraph-length answer.
While there has been remarkable recent progress in factoid open-domain question answering (QA), where a short phrase or entity is enough to answer a question, much less work has been done in the area of long-form question answering.
LFQA is nevertheless an important task, especially because it provides a testbed to measure the factuality of generative text models. But, are current benchmarks and evaluation metrics really suitable for making progress on LFQA?”
Search Engine Question Answering
Question answering by search engines typically consists of a searcher asking a question and the search engine returning a relatively short text of information.
Questions like “What’s the phone number of XYZ store?” is an example of a typical question that search engines are good at answering, especially because the answer is objective and not subjective.
Long Form Question Answering is harder because the questions demand answers in the form of paragraphs, not short texts.
Facebook is also working on long form question answering and came up with interesting solutions like using a question and answer subreddit called Explain Like I’m 5 (a dataset called ELI5). Facebook also admits that there more work to do. (Introducing Long-form Question Answering)
Examples of Long Form Questions
Once you read these examples of long form questions it’s going to be clearer how we’ve been trained by search engines to ask a limited set of queries. It might even seem shocking how almost infantile our questions are compared to long form questions.
The Google research paper offers these examples of long form questions:
- What goes on in those tall tower buildings owned by major banks?
- What exactly is fire, in detail? How can light and heat come from something we can’t really touch?
- Why do Britain and other English empire countries still bow to monarchs? What real purpose does the queen serve?
Facebook offers these examples of long form questions:
- Why are some restaurants better than others if they serve basically the same food?
- What are the differences between bodies of water like lakes, rivers, and seas?
- Why do we feel more jet lagged when traveling east?
Are Searchers Trained to Ask Short Questions for Factoids?
Google (and Bing) have a difficult time answering these long form types of questions. This may impact their ability to surface content that provides complex answers for complex questions.
Maybe people don’t ask these questions because they’ve been trained not to because of the poor responses. But if search engines were able to answer these kinds of questions then people would begin to ask them.
It’s a whole wide world of questions and answers that are missing from our search experience.
If I shorten the phrase “Why are some restaurants better than others if they serve basically the same food?” to “Why are some restaurants better than others?” Google and Bing still fail to provide an adequate answer.
The top Google search result for that question comes from the (HTTP insecure) blog of a Canadian Indian.
Google cites this section of the Indian restaurant in the SERP:
“People pay for the overall experience and not just the food and that is why some restaurants charge much more than others. Restaurant customers expect the prices to reflect the type of food, level of service and the overall atmosphere of the restaurant.”
What if the person had Popeye’s Fried Chicken versus KFC in mind when they asked that question?
There’s a certain amount of subjectivity that can creep into answering these kinds of questions that demands a long and coherent answer.
I can’t help thinking that there’s a better answer out there somewhere. But Google and Bing are unable to surface that kind of content.
Google Uses Signals to Identify High Quality Content
In a How Search Works explainer that Google published in September 2020, Google admits that it does not use the content itself to identify if it is reliable or trustworthy.
Google explains that it uses signals in a blog post titled, “How Google Delivers Reliable Information in Search.”
“…when it comes to high-quality, trustworthy information… We often can’t tell from the words or images alone if something is exaggerated, incorrect, low-quality or otherwise unhelpful.
Instead, search engines largely understand the quality of content through what are commonly called “signals.” You can think of these as clues about the characteristics of a page that align with what humans might interpret as high quality or reliable.
For example, the number of quality pages that link to a particular page is a signal that a page may be a trusted source of information on a topic.”
Unfortunately, that part of Google’s algorithm is unable to provide a correct answer to these kinds of long form questions.
And that’s an interesting and important fact to understand because it helps to be aware of what the limits are to search technology today.
What About Passage Ranking?
Passage Ranking is about ranking long web pages that contain the short answers for normal short queries needing an objective answer.
Martin Splitt used the example of finding a relevant answer about tomatoes in a web page that is mostly about gardening in general.
Passage ranking cannot solve the hard questions that Google currently cannot answer.
Both Google and Bing generally fail to answer LFQA type queries because this is an area that search engines still need to improve.
Hurdles to Progress
The research paper itself acknowledges that shortcoming in the title:
"Hurdles to Progress in Long-form Question Answering"
The research paper concludes by stating that its approach to solving this task “achieves state of the art performance” but that there are still issues to resolve and more research that needs to be done.
This is how the paper concludes:
“We present a “retrieval augmented” generation system that achieves state of the art performance on the ELI5 long-form question answering dataset. However, an in-depth analysis reveals several issues not only with our model, but also with the ELI5 dataset & evaluation metrics. We hope that the community works towards solving these issues so that we can climb the right hills and make meaningful progress.”
Questions and Speculation
It’s not possible to provide a definitive answer but one has to wonder if there are web pages out there that are missing out on traffic because both Google and Bing are not able to surface their long form content in answer to long form questions.
Also, some publisher mistakenly overwrite their articles in a quest to be authoritative. Is it possible that those publishers are over-writing themselves out of search traffic from queries that demand shorter answers since search engines can’t deliver nuanced answers available in longer documents?
There’s no way of knowing these answers for certain.
But one thing this research paper makes clear is that long-form question answering is a shortcoming in search engines today.
Google AI Blog Post
Progress and Challenges in Long-Form Open-Domain Question Answering
PDF Version of Research Paper
Hurdles to Progress in Long-form Question Answering
Facebook Web Page About LFQA
Introducing Long-form Question Answering
Google ska betala $391,5 miljoner för uppgörelse över platsspårning, säger statliga AG:er
Google has agreed to pay a $391.5 million settlement to 40 states to resolve accusations that it tracked people’s locations in violation of state laws, including snooping on consumers’ whereabouts even after they told the tech behemoth to bug off.
Louisiana Attorney General Jeff Landry said it is time for Big Tech to recognize state laws that limit data collection efforts.
“I have been ringing the alarm bell on big tech for years, and this is why,” Mr. Landry, a Republican, said in a statement Monday. “Citizens must be able to make informed decisions about what information they release to big tech.”
The attorneys general said the investigation resulted in the largest-ever multistate privacy settlement. Connecticut Attorney General William Tong, a Democrat, said Google’s penalty is a “historic win for consumers.”
“Location data is among the most sensitive and valuable personal information Google collects, and there are so many reasons why a consumer may opt out of tracking,” Mr. Tong said. “Our investigation found that Google continued to collect this personal information even after consumers told them not to. That is an unacceptable invasion of consumer privacy, and a violation of state law.”
Location tracking can help tech companies sell digital ads to marketers looking to connect with consumers within their vicinity. It’s another tool in a data-gathering toolkit that generates more than $200 billion in annual ad revenue for Google, accounting for most of the profits pouring into the coffers of its corporate parent, Alphabet, which has a market value of $1.2 trillion.
The settlement is part of a series of legal challenges to Big Tech in the U.S. and around the world, which include consumer protection and antitrust lawsuits.
Though Google, based in Mountain View, California, said it fixed the problems several years ago, the company’s critics remained skeptical. State attorneys general who also have tussled with Google have questioned whether the tech company will follow through on its commitments.
The states aren’t dialing back their scrutiny of Google’s empire.
Last month, Texas Attorney General Ken Paxton said he was filing a lawsuit over reports that Google unlawfully collected millions of Texans’ biometric data such as “voiceprints and records of face geometry.”
The states began investigating Google’s location tracking after The Associated Press reported in 2018 that Android devices and iPhones were storing location data despite the activation of privacy settings intended to prevent the company from following along.
Arizona Attorney General Mark Brnovich went after the company in May 2020. The state’s lawsuit charged that the company had defrauded its users by misleading them into believing they could keep their whereabouts private by turning off location tracking in the settings of their software.
Arizona settled its case with Google for $85 million last month. By then, attorneys general in several other states and the District of Columbia had pounced with their own lawsuits seeking to hold Google accountable.
Along with the hefty penalty, the state attorneys general said, Google must not hide key information about location tracking, must give users detailed information about the types of location tracking information Google collects, and must show additional information to people when users turn location-related account settings to “off.”
States will receive differing sums from the settlement. Mr. Landry’s office said Louisiana would receive more than $12.7 million, and Mr. Tong’s office said Connecticut would collect more than $6.5 million.
The financial penalty will not cripple Google’s business. The company raked in $69 billion in revenue for the third quarter of 2022, according to reports, yielding about $13.9 billion in profit.
Google downplayed its location-tracking tools Monday and said it changed the products at issue long ago.
“Consistent with improvements we’ve made in recent years, we have settled this investigation which was based on outdated product policies that we changed years ago,” Google spokesman Jose Castaneda said in a statement.
Google product managers Marlo McGriff and David Monsees defended their company’s Search and Maps products’ usage of location information.
“Location information lets us offer you a more helpful experience when you use our products,” the two men wrote on Google’s blog. “From Google Maps’ driving directions that show you how to avoid traffic to Google Search surfacing local restaurants and letting you know how busy they are, location information helps connect experiences across Google to what’s most relevant and useful.”
The blog post touted transparency tools and auto-delete controls that Google has developed in recent years and said the private browsing Incognito mode prevents Google Maps from saving an account’s search history.
Mr. McGriff and Mr. Monsees said Google would make changes to its products as part of the settlement. The changes include simplifying the process for deleting location data, updating the method to set up an account and revamping information hubs.
“We’ll provide a new control that allows users to easily turn off their Location History and Web & App Activity settings and delete their past data in one simple flow,” Mr. McGriff and Mr. Monsees wrote. “We’ll also continue deleting Location History data for users who have not recently contributed new Location History data to their account.”
• This article is based in part on wire service reports.