Connect with us

MARKETING

How to Increase Survey Completion Rate With 5 Top Tips

Published

on

How to Increase Survey Completion Rate With 5 Top Tips

Collecting high-quality data is crucial to making strategic observations about your customers. Researchers have to consider the best ways to design their surveys and then how to increase survey completion, because it makes the data more reliable.

→ Free Download: 5 Customer Survey Templates [Access Now]

I’m going to explain how survey completion plays into the reliability of data. Then, we’ll get into how to calculate your survey completion rate versus the number of questions you ask. Finally, I’ll offer some tips to help you increase survey completion rates.

My goal is to make your data-driven decisions more accurate and effective. And just for fun, I’ll use cats in the examples because mine won’t stop walking across my keyboard.

Why Measure Survey Completion

Let’s set the scene: We’re inside a laboratory with a group of cat researchers. They’re wearing little white coats and goggles — and they desperately want to know what other cats think of various fish.

They’ve written up a 10-question survey and invited 100 cats from all socioeconomic rungs — rough and hungry alley cats all the way up to the ones that thrice daily enjoy their Fancy Feast from a crystal dish.

Advertisement

Now, survey completion rates are measured with two metrics: response rate and completion rate. Combining those metrics determines what percentage, out of all 100 cats, finished the entire survey. If all 100 give their full report on how delicious fish is, you’d achieve 100% survey completion and know that your information is as accurate as possible.

But the truth is, nobody achieves 100% survey completion, not even golden retrievers.

With this in mind, here’s how it plays out:

  • Let’s say 10 cats never show up for the survey because they were sleeping.
  • Of the 90 cats that started the survey, only 25 got through a few questions. Then, they wandered off to knock over drinks.
  • Thus, 90 cats gave some level of response, and 65 completed the survey (90 – 25 = 65).
  • Unfortunately, those 25 cats who only partially completed the survey had important opinions — they like salmon way more than any other fish.

The cat researchers achieved 72% survey completion (65 divided by 90), but their survey will not reflect the 25% of cats — a full quarter! — that vastly prefer salmon. (The other 65 cats had no statistically significant preference, by the way. They just wanted to eat whatever fish they saw.)

Now, the Kitty Committee reviews the research and decides, well, if they like any old fish they see, then offer the least expensive ones so they get the highest profit margin.

CatCorp, their competitors, ran the same survey; however, they offered all 100 participants their own glass of water to knock over — with a fish inside, even!

Only 10 of their 100 cats started, but did not finish the survey. And the same 10 lazy cats from the other survey didn’t show up to this one, either.

Advertisement

So, there were 90 respondents and 80 completed surveys. CatCorp achieved an 88% completion rate (80 divided by 90), which recorded that most cats don’t care, but some really want salmon. CatCorp made salmon available and enjoyed higher profits than the Kitty Committee.

So you see, the higher your survey completion rates, the more reliable your data is. From there, you can make solid, data-driven decisions that are more accurate and effective. That’s the goal.

We measure the completion rates to be able to say, “Here’s how sure we can feel that this information is accurate.”

And if there’s a Maine Coon tycoon looking to invest, will they be more likely to do business with a cat food company whose decision-making metrics are 72% accurate or 88%? I suppose it could depend on who’s serving salmon.

While math was not my strongest subject in school, I had the great opportunity to take several college-level research and statistics classes, and the software we used did the math for us. That’s why I used 100 cats — to keep the math easy so we could focus on the importance of building reliable data.

Now, we’re going to talk equations and use more realistic numbers. Here’s the formula:

Completion rate equals the # of completed surveys divided by the # of survey respondents.

So, we need to take the number of completed surveys and divide that by the number of people who responded to at least one of your survey questions. Even just one question answered qualifies them as a respondent (versus nonrespondent, i.e., the 10 lazy cats who never show up).

Now, you’re running an email survey for, let’s say, Patton Avenue Pet Company. We’ll guess that the email list has 5,000 unique addresses to contact. You send out your survey to all of them.

Advertisement

Your analytics data reports that 3,000 people responded to one or more of your survey questions. Then, 1,200 of those respondents actually completed the entire survey.

3,000/5000 = 0.6 = 60% — that’s your pool of survey respondents who answered at least one question. That sounds pretty good! But some of them didn’t finish the survey. You need to know the percentage of people who completed the entire survey. So here we go:

Completion rate equals the # of completed surveys divided by the # of survey respondents.

Completion rate = (1,200/3,000) = 0.40 = 40%

Voila, 40% of your respondents did the entire survey.

Response Rate vs. Completion Rate

Okay, so we know why the completion rate matters and how we find the right number. But did you also hear the term response rate? They are completely different figures based on separate equations, and I’ll show them side by side to highlight the differences.

Advertisement
  • Completion Rate = # of Completed Surveys divided by # of Respondents
  • Response Rate = # of Respondents divided by Total # of surveys sent out

Here are examples using the same numbers from above:

Completion Rate = (1200/3,000) = 0.40 = 40%

Response Rate = (3,000/5000) = 0.60 = 60%

So, they are different figures that describe different things:

  • Completion rate: The percentage of your respondents that completed the entire survey. As a result, it indicates how sure we are that the information we have is accurate.
  • Response rate: The percentage of people who responded in any way to our survey questions.

The follow-up question is: How can we make this number as high as possible in order to be closer to a truer and more complete data set from the population we surveyed?

There’s more to learn about response rates and how to bump them up as high as you can, but we’re going to keep trucking with completion rates!

What’s a good survey completion rate?

That is a heavily loaded question. People in our industry have to say, “It depends,” far more than anybody wants to hear it, but it depends. Sorry about that.

There are lots of factors at play, such as what kind of survey you’re doing, what industry you’re doing it in, if it’s an internal or external survey, the population or sample size, the confidence level you’d like to hit, the margin of error you’re willing to accept, etc.

Advertisement

But you can’t really get a high completion rate unless you increase response rates first.

So instead of focusing on what’s a good completion rate, I think it’s more important to understand what makes a good response rate. Aim high enough, and survey completions should follow.

I checked in with the Qualtrics community and found this discussion about survey response rates:

“Just wondering what are the average response rates we see for online B2B CX surveys? […]

Current response rates: 6%–8%… We are looking at boosting the response rates but would first like to understand what is the average.”

The best answer came from a government service provider that works with businesses. The poster notes that their service is free to use, so they get very high response rates.

Advertisement

“I would say around 30–40% response rates to transactional surveys,” they write. “Our annual pulse survey usually sits closer to 12%. I think the type of survey and how long it has been since you rendered services is a huge factor.”

Since this conversation, “Delighted” (the Qualtrics blog) reported some fresher data:

survey completion rate vs number of questions new data, qualtrics data

Image Source

The takeaway here is that response rates vary widely depending on the channel you use to reach respondents. On the upper end, the Qualtrics blog reports that customers had 85% response rates for employee email NPS surveys and 33% for email NPS surveys.

A good response rate, the blog writes, “ranges between 5% and 30%. An excellent response rate is 50% or higher.”

This echoes reports from Customer Thermometer, which marks a response rate of 50% or higher as excellent. Response rates between 5%-30% are much more typical, the report notes. High response rates are driven by a strong motivation to complete the survey or a personal relationship between the brand and the customer.

Advertisement

If your business does little person-to-person contact, you’re out of luck. Customer Thermometer says you should expect responses on the lower end of the scale. The same goes for surveys distributed from unknown senders, which typically yield the lowest level of responses.

According to SurveyMonkey, surveys where the sender has no prior relationship have response rates of 20% to 30% on the high end.

Whatever numbers you do get, keep making those efforts to bring response rates up. That way, you have a better chance of increasing your survey completion rate. How, you ask?

Tips to Increase Survey Completion

If you want to boost survey completions among your customers, try the following tips.

1. Keep your survey brief.

We shouldn’t cram lots of questions into one survey, even if it’s tempting. Sure, it’d be nice to have more data points, but random people will probably not hunker down for 100 questions when we catch them during their half-hour lunch break.

Keep it short. Pare it down in any way you can.

Advertisement

Survey completion rate versus number of questions is a correlative relationship — the more questions you ask, the fewer people will answer them all. If you have the budget to pay the respondents, it’s a different story — to a degree.

“If you’re paying for survey responses, you’re more likely to get completions of a decently-sized survey. You’ll just want to avoid survey lengths that might tire, confuse, or frustrate the user. You’ll want to aim for quality over quantity,” says Pamela Bump, Head of Content Growth at HubSpot.

2. Give your customers an incentive.

For instance, if they’re cats, you could give them a glass of water with a fish inside.

Offer incentives that make sense for your target audience. If they feel like they are being rewarded for giving their time, they will have more motivation to complete the survey.

This can even accomplish two things at once — if you offer promo codes, discounts on products, or free shipping, it encourages them to shop with you again.

3. Keep it smooth and easy.

Keep your survey easy to read. Simplifying your questions has at least two benefits: People will understand the question better and give you the information you need, and people won’t get confused or frustrated and just leave the survey.

Advertisement

4. Know your customers and how to meet them where they are.

Here’s an anecdote about understanding your customers and learning how best to meet them where they are.

Early on in her role, Pamela Bump, HubSpot’s Head of Content Growth, conducted a survey of HubSpot Blog readers to learn more about their expertise levels, interests, challenges, and opportunities. Once published, she shared the survey with the blog’s email subscribers and a top reader list she had developed, aiming to receive 150+ responses.

“When the 20-question survey was getting a low response rate, I realized that blog readers were on the blog to read — not to give feedback. I removed questions that wouldn’t serve actionable insights. When I reshared a shorter, 10-question survey, it passed 200 responses in one week,” Bump shares.

Tip 5. Gamify your survey.

Make it fun! Brands have started turning surveys into eye candy with entertaining interfaces so they’re enjoyable to interact with.

Your respondents could unlock micro incentives as they answer more questions. You can word your questions in a fun and exciting way so it feels more like a BuzzFeed quiz. Someone saw the opportunity to make surveys into entertainment, and your imagination — well, and your budget — is the limit!

Your Turn to Boost Survey Completion Rates

Now, it’s time to start surveying. Remember to keep your user at the heart of the experience. Value your respondents’ time, and they’re more likely to give you compelling information. Creating short, fun-to-take surveys can also boost your completion rates.

Advertisement

Editor’s note: This post was originally published in December 2010 and has been updated for comprehensiveness.

Click me

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address

MARKETING

Crafting Effortless Sales Through ‘Wow’ Moments in Experience Marketing

Published

on

Crafting Effortless Sales Through 'Wow' Moments in Experience Marketing

Crafting Effortless Sales Through Wow Moments in Experience Marketing

In an era where consumers are bombarded with endless choices and digital noise, standing out as a brand is more challenging than ever. Enter experience marketing – a strategy that transcends traditional advertising by focusing on creating immersive, memorable interactions. This innovative approach leverages the elements of surprise, delight, and reciprocity to forge strong emotional connections with customers, making the sale of your core product feel effortless. But how can businesses implement this strategy effectively? This guide delves into the art of crafting ‘wow’ moments that captivate audiences and transform customer engagement.

The Basics of Experience Marketing

Experience marketing is an evolved form of marketing that focuses on creating meaningful interactions with customers, aiming to elicit strong emotional responses that lead to brand loyalty and advocacy. Unlike conventional marketing, which often prioritizes product promotion, experience marketing centers on the customer’s holistic journey with the brand, creating a narrative that resonates on a personal level.

In today’s competitive market, experience marketing is not just beneficial; it’s essential. It differentiates your brand in a crowded marketplace, elevating your offerings beyond mere commodities to become integral parts of your customers’ lives. Through memorable experiences, you not only attract attention but also foster a community of loyal customers who are more likely to return and recommend your brand to others.

Principles of Experience Marketing

At the heart of experience marketing lie several key principles:

  • Emotional Connection: Crafting campaigns that touch on human emotions, from joy to surprise, creating memorable moments that customers are eager to share.
  • Customer-Centricity: Putting the customer’s needs and desires at the forefront of every marketing strategy, ensuring that each interaction adds value and enhances their experience with the brand.
  • Immersive Experiences: Utilizing technology and storytelling to create immersive experiences that captivate customers, making your brand a living part of their world.
  • Engagement Across Touchpoints: Ensuring consistent, engaging experiences across all customer touchpoints, from digital platforms to physical stores.

Understanding Your Audience

Before diving into the intricacies of crafting ‘wow’ moments, it’s crucial to understand who you’re creating these moments for. Identifying your audience’s pain points and desires is the first step in tailoring experiences that truly resonate.

1709033181 544 Crafting Effortless Sales Through Wow Moments in Experience Marketing1709033181 544 Crafting Effortless Sales Through Wow Moments in Experience Marketing

This involves deep market research, customer interviews, and leveraging data analytics to paint a comprehensive picture of your target demographic. By understanding the journey your customers are on, you can design touchpoints that not only meet but exceed their expectations.

  • Identifying Pain Points and Desires: Use surveys, social media listening, and customer feedback to gather insights. What frustrates your customers about your industry? What do they wish for more than anything else? These insights will guide your efforts to create experiences that truly resonate.
  • Mapping the Customer Journey: Visualize every step a customer takes from discovering your brand to making a purchase and beyond. This map will highlight critical touchpoints where you can introduce ‘wow’ moments that transform the customer experience.

Developing Your Experience Marketing Strategy

With a clear understanding of your audience, it’s time to build the framework of your experience marketing strategy. This involves setting clear objectives, identifying key customer touchpoints, and conceptualizing the experiences you want to create.

  • Setting Objectives: Define what you aim to achieve with your experience marketing efforts. Whether it’s increasing brand awareness, boosting sales, or improving customer retention, having clear goals will shape your approach and help measure success.
  • Strategic Touchpoint Identification: List all the potential touchpoints where customers interact with your brand, from social media to in-store experiences. Consider every stage of the customer journey and look for opportunities to enhance these interactions.

Enhancing Customer Experiences with Surprise, Delight, and Reciprocity

This section is where the magic happens. By integrating the elements of surprise, delight, and reciprocity, you can elevate ordinary customer interactions into unforgettable experiences.

1709033181 790 Crafting Effortless Sales Through Wow Moments in Experience Marketing1709033181 790 Crafting Effortless Sales Through Wow Moments in Experience Marketing
  • Incorporating Surprise and Delight: Go beyond what’s expected. This could be as simple as a personalized thank-you note with each purchase or as elaborate as a surprise gift for loyal customers. The key is to create moments that feel special and unexpected.
  • Applying the Principle of Reciprocity: When customers receive something of value, they’re naturally inclined to give something back. This can be leveraged by offering helpful resources, exceptional service, or customer appreciation events. Such gestures encourage loyalty and positive word-of-mouth.
  • Examples and Case Studies: Highlight real-world examples of brands that have successfully implemented these strategies. Analyze what they did, why it worked, and how it impacted their relationship with customers.

Best Practices for Experience Marketing

To ensure your experience marketing strategy is as effective as possible, it’s important to adhere to some best practices.

  • Personalization at Scale: Leverage data and technology to personalize experiences without losing efficiency. Tailored experiences make customers feel valued and understood.
  • Using Technology to Enhance Experiences: From augmented reality (AR) to mobile apps, technology offers myriad ways to create immersive experiences that surprise and engage customers.
  • Measuring Success: Utilize analytics tools to track the success of your experience marketing initiatives. Key performance indicators (KPIs) could include engagement rates, conversion rates, and customer satisfaction scores.

Section 5: Overcoming Common Challenges

Even the best-laid plans can encounter obstacles. This section addresses common challenges in experience marketing and how to overcome them.

1709033181 656 Crafting Effortless Sales Through Wow Moments in Experience Marketing1709033181 656 Crafting Effortless Sales Through Wow Moments in Experience Marketing
  • Budget Constraints: Learn how to create impactful experiences without breaking the bank. It’s about creativity, not just expenditure.
  • Maintaining Consistency: Ensuring a consistent brand experience across all touchpoints can be daunting. Develop a comprehensive brand guideline and train your team accordingly.
  • Staying Ahead of Trends: The digital landscape is ever-changing. Stay informed about the latest trends in experience marketing and be ready to adapt your strategy as necessary.

The Path to Effortless Sales

By creating memorable experiences that resonate on a personal level, you make the path to purchase not just easy but natural. When customers feel connected to your brand, appreciated, and valued, making a sale becomes a byproduct of your relationship with them. Experience marketing, when done right, transforms transactions into interactions, customers into advocates, and products into passions.

Now is the time to reassess your marketing strategy. Are you just selling a product, or are you providing an unforgettable experience? Dive into the world of experience marketing and start creating those ‘wow’ moments that will not only distinguish your brand but also make sales feel effortless.


Disruptive Design Raising the Bar of Content Marketing with Graphic

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

MARKETING

The Current State of Google’s Search Generative Experience [What It Means for SEO in 2024]

Published

on

The Current State of Google's Search Generative Experience [What It Means for SEO in 2024]

SEO enthusiasts, known for naming algorithm updates after animals and embracing melodrama, find themselves in a landscape where the “adapt or die” mantra prevails. So when Google announced the launch of its Search Generative Experience (SGE) in May of 2023 at Google/IO, you can imagine the reaction was immense.

Although SGE has the potential to be a truly transformative force in the landscape, we’re still waiting for SGE to move out of the Google Labs Sandbox and integrate into standard search results. 

Curious about our current take on SGE and its potential impact on SEO in the future? Read on for more.

Decoding Google’s Defensive Move

In response to potential threats from competitors like ChatGPT, Bing, TikTok, Reddit, and Amazon, Google introduced SGE as a defensive maneuver. However, its initial beta release raised questions about its readiness and global deployment.

ChatGPT provided an existential threat that had the potential to eat into Google’s market share. When Bing started incorporating it into its search results, it was one of the most significant wins for Bing in a decade. In combination with threats from TikTok, Reddit, and Amazon, we see a more fractured search landscape less dominated by Google. Upon its launch, the expectation was that Google would push its SGE solution globally, impact most queries, and massively shake up organic search results and strategies to improve organic visibility.

Advertisement

Now, industry leaders are starting to question if Google is better off leaving SGE in the testing ground in Google labs. According to Google’s recent update, it appears that SGE will remain an opt-in experience in Google Labs (for at least the short term). If SGE was released, there could be a fundamental reset in understanding SEO. Everything from organic traffic to optimization tactics to tracking tools would need adjustments for the new experience. Therefore, the prospect of SGE staying in Google Labs is comforting if not entirely reliable. 

The ever-present option is that Google can change its mind at any point and push SGE out broadly as part of its standard search experience. For this reason, we see value in learning from our observations with SGE and continuing to stay on top of the experience.

SGE User Experience and Operational Challenges

If you’ve signed up for search labs and have been experimenting with SGE for a while, you know firsthand there are various issues that Google should address before rolling it out broadly to the public.

At a high level, these issues fall into two broad categories including user experience issues and operational issues.

Below are some significant issues we’ve come across, with Google making notable progress in addressing certain ones, while others still require improvement:

  • Load time – Too many AI-generated answers take longer to load than a user is willing to wait. Google recommends less than a 3-second load time to meet expectations. They’ll need to figure out how to consistently return results quickly if they want to see a higher adoption rate.
  • Layout – The SGE layout is massive. We believe any major rollout will be more streamlined to make it a less intrusive experience for users and allow more visibility for ads, and if we’re lucky, organic results. Unfortunately, there is still a decent chance that organic results will move below the fold, especially on mobile devices. Recently, Google has incorporated more results where users are prompted to generate the AI result if they’d like to see it. The hope is Google makes this the default in the event of a broad rollout where users can generate an AI result if they want one instead of assuming that’s what a user would like to see. 
  • Redundancy – The AI result duplicates features from the map pack and quick answer results. 
  • Attribution – Due to user feedback, Google includes sources on several of their AI-powered overviews where you can see relevant web pages if there is an arrow next to the result. Currently, the best way to appear as one of these relevant pages is to be one of the top-ranked results, which is convenient from an optimization standpoint. Changes to how attribution and sourcing are handled could heavily impact organic strategies. 

 

On the operational side, Google also faces significant hurdles to making SGE a viable product for its traditional search product. The biggest obstacle appears to be making the cost associated with the technology worth the business outcomes it provides. If this was a necessary investment to maintain market share, Google might be willing to eat the cost, but if their current position is relatively stable, Google doesn’t have much of an incentive to take on the additional cost burden of heavily leveraging generative AI while also presumably taking a hit to their ad revenue. Especially since slow user adoption doesn’t indicate this is something users are demanding at the moment.

Advertisement

While the current experience of SGE is including ads above the generative results now, the earliest iterations didn’t heavily feature sponsored ads. While they are now included, the current SGE layout would still significantly disrupt the ad experience we’re used to. During the Google I/O announcement, they made a statement to reassure advertisers they would be mindful of maintaining a distinct ad experience in search.  

“In this new generative experience, Search ads will continue to appear in dedicated ad slots throughout the page. And we’ll continue to uphold our commitment to ads transparency and making sure ads are distinguishable from organic search results” – Elizabeth Reid, VP, Search at Google

Google is trying to thread a delicate needle here of staying on the cutting edge with their search features, while trying not to upset their advertisers and needlessly hinder their own revenue stream. Roger Montti details more of the operational issues in a recent article digging into the surprising reasons SGE is stuck in Google Labs.

He lists three big problems that need to be solved before SGE will be integrated into the foreground of search:

  1. Large Language Models being inadequate as an information retrieval system
  2. The inefficiency and cost of transformer architecture
  3. Hallucinating (providing inaccurate answers)

 

Until SGE provides more user value and checks more boxes on the business sense side, the traditional search experience is here to stay. Unfortunately, we don’t know when or if Google will ever feel confident they’ve addressed all of these concerns, so we’ll need to stay prepared for change.

Experts Chime in on Search Generative Experience

Our team has been actively engaging with SGE, here’s a closer look at their thoughts and opinions on the experience so far:

Advertisement

“With SGE still in its early stages, I’ve noticed consistent changes in how the generative results are produced and weaved naturally into the SERPs. Because of this, I feel it is imperative to stay on top of these on-going changes to ensure we can continue to educate our clients on what to expect when SGE is officially incorporated into our everyday lives. Although an official launch date is currently unknown, I believe proactively testing various prompt types and recording our learnings is important to prepare our clients for this next evolution of Google search.” – Jon Pagano, SEO Sr. Specialist at Tinuiti

“It’s been exciting to watch SGE grow through different variations over the last year, but like other AI solutions its potential still outweighs its functionality and usefulness. What’s interesting to see is that SGE doesn’t just cite its sources of information, but also provides an enhanced preview of each webpage referenced. This presents a unique organic opportunity where previously untouchable top 10 rankings are far more accessible to the average website. Time will tell what the top ranking factors for SGE are, but verifiable content with strong E-E-A-T signals will be imperative. –Kate Fischer, SEO Specialist at Tinuiti

“Traditionally, AI tools were very good at analytical tasks. With the rise of ChatGPT, users can have long-form, multi-question conversations not yet available in search results. When, not if, released, Google’s Generative Experience will transform how we view AI and search. Because there are so many unknowns, some of the most impactful ways we prepare our clients are to discover and develop SEO strategies that AI tools can’t directly disrupt, like mid to low funnel content.” – Brandon Miller, SEO Specialist at Tinuiti

“SGE is going to make a huge impact on the ecommerce industry by changing the way users interact with the search results. Improved shopping experience will allow users to compare products, price match, and read reviews in order to make it quicker and easier for a user to find the best deals and purchase. Although this leads to more competitive results, it also improves organic visibility and expands our product reach. It is more important than ever to ensure all elements of a page are uniquely and specifically optimized for search. With the SGE updates expected to continue to impact search results, the best way to stay ahead is by focusing on strong user focused content and detailed product page optimizations.”  – Kellie Daley, SEO Sr. Specialist at Tinuiti

Navigating the Clash of Trends

One of the most interesting aspects of the generative AI trend in search is that it appears to be in direct opposition to other recent trends.

Advertisement

One of the ways Google has historically evaluated the efficacy of its search ranking systems is through the manual review of quality raters. In their quality rater guidelines, raters were instructed to review for things like expertise, authority, and trustworthiness (EAT) in results to determine if Google results are providing users the information they deserve. 

In 2022, Google updated their search guidelines to include another ‘e’ in the form of experience (EEAT). In their words, Google wanted to better assess if the content a user was consuming was created by someone with, “a degree of experience, such as with actual use of a product, having actually visited a place or communicating what a person has experienced. There are some situations where really what you value most is content produced by someone who has firsthand, life experience on the topic at hand.” 

Generative AI results, while cutting-edge technology and wildly impressive in some cases, stand in direct opposition to the principles of E-E-A-T. That’s not to say that there’s no room for both in search, but Google will have to determine what it thinks users value more between these competing trends. The slow adoption of SGE could be an indication that a preference for human experience, expertise, authority, and trust is winning round one in this fight. 

Along these lines, Google is also diversifying its search results to cater to the format in which users get their information. This takes the form of their Perspectives Filter. Also announced at Google I/O 2023, the perspectives filter incorporates more video, image, and discussion board posts from places like TikTok, YouTube, Reddit, and Quora. Once again, this trend shows the emphasis and value searchers place on experience and perspective. Users value individual experience over the impersonal conveyance of information. AI will never have these two things, even if it can provide a convincing imitation.

The current iteration of SGE seems to go too far in dismissing these trends in favor of generative AI. It’s an interesting challenge Google faces. If they don’t determine the prevailing trend correctly, veering too far in one direction can push more market share to ChatGPT or platforms like YouTube and TikTok.

Final Thoughts

The range of outcomes remains broad and fascinating for SGE. We can see this developing in different ways, and prognostication offers little value, but it’s invaluable to know the potential outcomes and prepare for as many of them as possible.

Advertisement

It’s critical that you or your search agency be interacting and experimenting with SGE because:

  • The format and results will most likely continue to see significant changes
  • This space moves quickly and it’s easy to fall behind
  • Google may fix all of the issues with SGE and decide to push it live, changing the landscape of search overnight
  • SGE experiments could inform other AI elements incorporated into the search experience

 

Ultimately, optimizing for the specific SGE experience we see now is less important because we know it will inevitably continue changing. We see more value in recognizing the trends and problems Google is trying to solve with this technology. With how quickly this space moves, any specifics mentioned in this article could be outdated in a week. That’s why focusing on intention and process is important at this stage of the game.

By understanding the future needs and wants SGE is attempting to address, we can help you future-proof your search strategies as much as possible. To some extent we’re always at the whims of the algorithm, but by maintaining a user-centric approach, you can make your customers happy, regardless of how they find you.

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

MARKETING

Here’s Optimizely’s Automatic Sample Ratio Mismatch Detection

Published

on

Here's Optimizely’s Automatic Sample Ratio Mismatch Detection

Optimizely Experiment’s automatic sample ratio mismatch (SRM) detection delivers peace of mind to experimenters. It reduces a user’s exposure time to bad experiences by rapidly detecting any experiment deterioration.

This deterioration is caused by unexpected imbalances of visitors to a variation in an experiment. Most importantly, this auto SRM detection empowers product managers, marketers, engineers, and experimentation teams to confidently launch more experiments. 

How Optimizely Experiment’s stats engine and automatic sample rate mismatch detection work together

The sample ratio mismatch actslike the bouncer at the door who has a mechanical counter, checking guests’ tickets (users) and telling them which room they get to party in.

Stats engine is like the party host who is always checking the vibes (behavior) of the guests as people come into the room.

Advertisement

If SRM does its job right, then stats engine can confidently tell which party room is better and direct more traffic to the winning variation (the better party) sooner.

Why would I want Optimizely Experiment’s SRM detection?

It’s equally important to ensure Optimizely Experiment users know their experiment results are trustworthy and have the tools to understand what an imbalance can mean for their results and how to prevent it.

Uniquely, Optimizely Experiment goes further by combining the power of automatic visitor imbalance detection with an insightful experiment health indicator. This experiment health indicator plays double duty by letting our customers know when all is well and there is no imbalance present.

Then, when just-in-time insight is needed to protect your business decisions, Optimizely also delivers just-in-time alerts that help our customers recognize the severity of, diagnose, and recover from errors.

Why should I care about sample ratio mismatch (SRM)?

Just like a fever is a symptom of many illnesses, a SRM is a symptom of a variety of data quality issues. Ignoring a SRM without knowing the root cause may result in a bad feature appearing to be good and being shipped out to users, or vice versa. Finding an experiment with an unknown source of traffic imbalance lets you turn it off quickly and reduce the blast radius.

Then what is the connection between a “mismatch” and “sample ratio”?

When we get ready to launch an experiment, we assign a traffic split of users for Optimizely Experiment to distribute to each variation. We expect the assigned traffic split to reasonably match up with the actual traffic split in a live experiment. An experiment is exposed to an SRM imbalance when there is a statistically significant difference between the expected and the actual assigned traffic splits of visitors to an experiment’s variations.

Advertisement
1. A mismatch doesn’t mean an imperfect match

Remember: A bonified imbalance requires a statistically significant result of the difference in visitors. Don’t expect a picture-perfect, identical, exact match of the launch-day traffic split to your in-production traffic split. There will always be some ever-so-slight deviation.

Not every traffic disparity automatically signifies that an experiment is useless. Because Optimizely deeply values our customers’ time and energy, we developed a new statistical test that continuously monitors experiment results and detects harmful SRMs as early as possible. All while still controlling for crying wolf over false positives (AKA when we conclude there is a surprising difference between a test variation and the baseline when there is no real difference). 

2. Going under the hood of Optimizely Experiment’s SRM detection algorithm

Optimizely Experiment’s automatic SRM detection feature employs a sequential Bayesian multinomial test (say that 5 times fast!), named sequential sample ratio mismatch. Optimizely statisticians Michael Lindon and Alen Malek pioneered this method, and it is a new contribution to the field of Sequential Statistics. Optimizely Experiment’s sample ratio mismatch detection harmonizes sequential and Bayesian methodologies by continuously checking traffic counts and testing for any significant imbalance in a variation’s visitor counts. The algorithm’s construction is Bayesian inspired to account for an experiment’s optional stopping and continuation while delivering sequential guarantees of Type-I error probabilities.

3. Beware of chi-eap alternatives!

The most popular freely available SRM calculators employ the chi-square test. We highly recommend a careful review of the mechanics of chi-square testing. The main issue with the chi-squared method is that problems are discovered only after collecting all the data. This is arguably far too late and goes against why most clients want SRM detention in the first place. In our blog post “A better way to test for sample ratio mismatches (or why I don’t use a chi-squared test)”, we go deeper into chi-square mechanics and how what we built accounts for the gaps left behind by the alternatives.

Common causes of an SRM  

1. Redirects & Delays

A SRM usually results from some visitors closing out and leaving the page before the redirect finishes executing. Because we only send the decision events once they arrive on the page and Optimizely Experiment loads, we can’t count these visitors in our results page unless they return at some point and send an event to Optimizely Experiment.

A SRM can emerge in the case of anything that would cause Optimizely Experiment’s event calls to delay or not fire, such as variation code changes. It also occurs when redirect experiments shuttle visitors to a different domain. This occurrence is exacerbated by slow connection times.

Advertisement
2. Force-bucketing

If a user first gets bucketed in the experiment and then that decision is used to force-bucket them in a subsequent experiment, then the results of that subsequent experiment will become imbalanced.

Here’s an example:

Variation A provides a wildly different user experience than Variation B.

Visitors bucketed into Variation A have a great experience, and many of them continue to log in and land into the subsequent experiment where they’re force-bucketed into Variation A.

But, visitors who were bucketed into Variation B aren’t having a good experience. Only a few users log in and land into a subsequent experiment where they will be force-bucketed into Variation B.

Well, now you have many more visitors in Variation A than in Variation B.

Advertisement
3. Site has its own redirects

Some sites have their own redirects (for example, 301s) that, combined with our redirects, can result in a visitor landing on a page without the snippet. This causes pending decision events to get locked in localStorage and Optimizely Experiment never receives or counts them.

4. Hold/send events API calls are housed outside of the snippet

Some users include hold/send events in project JS. However, others include it in other scripts on the page, such as in vendor bundles or analytics tracking scripts. This represents another script that must be properly loaded for the decisions to fire appropriately. Implementation or loading rates may differ across variations, particularly in the case of redirects.

Interested?  

If you’re already an Optimizely Experiment customer and you’d like to learn more about how automatic SRM detection benefits your A/B tests, check out our knowledge base documentation:

For further details you can always reach out to your customer success manager but do take a moment to review our documentation first!

If you’re not a customer, get started with us here! 

And if you’d like to dig deeper into the engine that powers Optimizely experimentation, you can check out our page faster decisions you can trust for digital experimentation. 

Advertisement

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

Trending

Follow by Email
RSS