MARKETING

How to Determine Your A/B Testing Sample Size & Time Frame

Published

1 year ago

November 3, 2022

How to Determine Your A/B Testing Sample Size & Time Frame

Do you remember your first A/B test you ran? I do. (Nerdy, I know.)

I felt simultaneously thrilled and terrified because I knew I had to actually use some of what I learned in college for my job.

There were some aspects of A/B testing I still remembered — for instance, I knew you need a big enough sample size to run the test on, and you need to run the test long enough to get statistically significant results.

But … that’s pretty much it. I wasn’t sure how big was “big enough” for sample sizes and how long was “long enough” for test durations — and Googling it gave me a variety of answers my college statistics courses definitely didn’t prepare me for.

Turns out I wasn’t alone: Those are two of the most common A/B testing questions we get from customers. And the reason the typical answers from a Google search aren’t that helpful is because they’re talking about A/B testing in an ideal, theoretical, non-marketing world.

So, I figured I’d do the research to help answer this question for you in a practical way. At the end of this post, you should be able to know how to determine the right sample size and time frame for your next A/B test. Let’s dive in.

A/B Testing Sample Size & Time Frame

In theory, to determine a winner between Variation A and Variation B, you need to wait until you have enough results to see if there is a statistically significant difference between the two.

Depending on your company, sample size, and how you execute the A/B test, getting statistically significant results could happen in hours or days or weeks — and you’ve just got to stick it out until you get those results. In theory, you should not restrict the time in which you’re gathering results.

For many A/B tests, waiting is no problem. Testing headline copy on a landing page? It’s cool to wait a month for results. Same goes with blog CTA creative — you’d be going for the long-term lead generation play, anyway.

But certain aspects of marketing demand shorter timelines when it comes to A/B testing. Take email as an example. With email, waiting for an A/B test to conclude can be a problem, for several practical reasons:

1. Each email send has a finite audience.

Unlike a landing page (where you can continue to gather new audience members over time), once you send an email A/B test off, that’s it — you can’t “add” more people to that A/B test. So you’ve got to figure out how squeeze the most juice out of your emails.

This will usually require you to send an A/B test to the smallest portion of your list needed to get statistically significant results, pick a winner, and then send the winning variation on to the rest of the list.

2. Running an email marketing program means you’re juggling at least a few email sends per week. (In reality, probably way more than that.)

If you spend too much time collecting results, you could miss out on sending your next email — which could have worse effects than if you sent a non-statistically-significant winner email on to one segment of your database.

3. Email sends are often designed to be timely.

Your marketing emails are optimized to deliver at a certain time of day, whether your emails are supporting the timing of a new campaign launch and/or landing in your recipient’s inboxes at a time they’d love to receive it. So if you wait for your email to be fully statistically significant, you might miss out on being timely and relevant — which could defeat the purpose of your email send in the first place.

That’s why email A/B testing programs have a “timing” setting built in: At the end of that time frame, if neither result is statistically significant, one variation (which you choose ahead of time) will be sent to the rest of your list. That way, you can still run A/B tests in email, but you can also work around your email marketing scheduling demands and ensure people are always getting timely content.

So to run A/B tests in email while still optimizing your sends for the best results, you’ve got to take both sample size and timing into account.

Next up — how to actually figure out your sample size and timing using data.

How to Determine Sample Size for an A/B Test

Now, let’s dive into how to actually calculate the sample size and timing you need for your next A/B test.

For our purposes, we’re going to use email as our example to demonstrate how you’ll determine sample size and timing for an A/B test. However, it’s important to note — the steps in this list can be used for any A/B test, not just email.

Let’s dive in.

Like mentioned above, each A/B test you send can only be sent to a finite audience — so you need to figure out how to maximize the results from that A/B test. To do that, you need to figure out the smallest portion of your total list needed to get statistically significant results. Here’s how you calculate it.

1. Assess whether you have enough contacts in your list to A/B test a sample in the first place.

To A/B test a sample of your list, you need to have a decently large list size — at least 1,000 contacts. If you have fewer than that in your list, the proportion of your list that you need to A/B test to get statistically significant results gets larger and larger.

For example, to get statistically significant results from a small list, you might have to test 85% or 95% of your list. And the results of the people on your list who haven’t been tested yet will be so small that you might as well have just sent half of your list one email version, and the other half another, and then measured the difference.

Your results might not be statistically significant at the end of it all, but at least you’re gathering learnings while you grow your lists to have more than 1,000 contacts. (If you want more tips on growing your email list so you can hit that 1,000 contact threshold, check out this blog post.)

Note for HubSpot customers: 1,000 contacts is also our benchmark for running A/B tests on samples of email sends — if you have fewer than 1,000 contacts in your selected list, the A version of your test will automatically be sent to half of your list and the B will be sent to the other half.

2. Use a sample size calculator.

Next, you’ll want to find a sample size calculator — HubSpot’s A/B Testing Kit offers a good, free sample size calculator.

Here’s what it looks like when you download it:

Download for Free

3. Put in your email’s Confidence Level, Confidence Interval, and Population into the tool.

Yep, that’s a lot of statistics jargon. Here’s what these terms translate to in your email:

Population: Your sample represents a larger group of people. This larger group is called your population.

In email, your population is the typical number of people in your list who get emails delivered to them — not the number of people you sent emails to. To calculate population, I’d look at the past three to five emails you’ve sent to this list, and average the total number of delivered emails. (Use the average when calculating sample size, as the total number of delivered emails will fluctuate.)

Confidence Interval: You might have heard this called “margin of error.” Lots of surveys use this, including political polls. This is the range of results you can expect this A/B test to explain once it’s run with the full population.

For example, in your emails, if you have an interval of 5, and 60% of your sample opens your Variation, you can be sure that between 55% (60 minus 5) and 65% (60 plus 5) would have also opened that email. The bigger the interval you choose, the more certain you can be that the populations true actions have been accounted for in that interval. At the same time, large intervals will give you less definitive results. It’s a trade-off you’ll have to make in your emails.

For our purposes, it’s not worth getting too caught up in confidence intervals. When you’re just getting started with A/B tests, I’d recommend choosing a smaller interval (ex: around 5).

Confidence Level: This tells you how sure you can be that your sample results lie within the above confidence interval. The lower the percentage, the less sure you can be about the results. The higher the percentage, the more people you’ll need in your sample, too.

Note for HubSpot customers: The HubSpot Email A/B tool automatically uses the 85% confidence level to determine a winner. Since that option isn’t available in this tool, I’d suggest choosing 95%.

Email A/B Test Example:

Let’s pretend we’re sending our first A/B test. Our list has 1,000 people in it and has a 95% deliverability rate. We want to be 95% confident our winning email metrics fall within a 5-point interval of our population metrics.

Here’s what we’d put in the tool:

Population: 950
Confidence Level: 95%
Confidence Interval: 5

sample_size_calculations

4. Click “Calculate” and your sample size will spit out.

Ta-da! The calculator will spit out your sample size.

In our example, our sample size is: 274.

This is the size one your variations needs to be. So for your email send, if you have one control and one variation, you’ll need to double this number. If you had a control and two variations, you’d triple it. (And so on.)

5. Depending on your email program, you may need to calculate the sample size’s percentage of the whole email.

HubSpot customers, I’m looking at you for this section. When you’re running an email A/B test, you’ll need to select the percentage of contacts to send the list to — not just the raw sample size.

To do that, you need to divide the number in your sample by the total number of contacts in your list. Here’s what that math looks like, using the example numbers above:

274 / 1,000 = 27.4%

This means that each sample (both your control AND your variation) needs to be sent to 27-28% of your audience — in other words, roughly a total of 55% of your total list.

email_ab_test_send

And that’s it! You should be ready to select your sending time.

How to Choose the Right Timeframe for Your A/B Test

Again, for figuring out the right timeframe for your A/B test, we’ll use the example of email sends – but this information should still apply regardless of the type of A/B test you’re conducting.

However, your timeframe will vary depending on your business’ goals, as well. If you’d like to design a new landing page by Q2 2021 and it’s Q4 2020, you’ll likely want to finish your A/B test by January or February so you can use those results to build the winning page.

But, for our purposes, let’s return to the email send example: You have to figure out how long to run your email A/B test before sending a (winning) version on to the rest of your list.

Figuring out the timing aspect is a little less statistically driven, but you should definitely use past data to help you make better decisions. Here’s how you can do that.

If you don’t have timing restrictions on when to send the winning email to the rest of the list, head over to your analytics.

Figure out when your email opens/clicks (or whatever your success metrics are) starts to drop off. Look your past email sends to figure this out.

For example, what percentage of total clicks did you get in your first day? If you found that you get 70% of your clicks in the first 24 hours, and then 5% each day after that, it’d make sense to cap your email A/B testing timing window for 24 hours because it wouldn’t be worth delaying your results just to gather a little bit of extra data.

In this scenario, you would probably want to keep your timing window to 24 hours, and at the end of 24 hours, your email program should let you know if they can determine a statistically significant winner.

Then, it’s up to you what to do next. If you have a large enough sample size and found a statistically significant winner at the end of the testing time frame, many email marketing programs will automatically and immediately send the winning variation.

If you have a large enough sample size and there’s no statistically significant winner at the end of the testing time frame, email marketing tools might also allow you to automatically send a variation of your choice.

If you have a smaller sample size or are running a 50/50 A/B test, when to send the next email based on the initial email’s results is entirely up to you.

If you have time restrictions on when to send the winning email to the rest of the list, figure out how late you can send the winner without it being untimely or affecting other email sends.

For example, if you’ve sent an email out at 3 p.m. EST for a flash sale that ends at midnight EST, you wouldn’t want to determine an A/B test winner at 11 p.m. Instead, you’d want to send the email closer to 6 or 7 p.m. — that’ll give the people not involved in the A/B test enough time to act on your email.

And that’s pretty much it, folks. After doing these calculations and examining your data, you should be in a much better state to conduct successful A/B tests — ones that are statistically valid and help you move the needle on your goals.

Related Topics:Determine Frame Sample Size TESTING Time

Up Next

Announcing the Keyword Research Certification: Create a Personalized Keyword Strategy

Don't Miss

It’s Scary That You’re Missing These 5 Marketing Skills

Click to comment

You must be logged in to post a comment Login

MARKETING

Tinuiti Marketing Analytics Recognized by Forrester

Published

3 hours ago

April 19, 2024

Entireweb News Bot

Tinuiti Marketing Analytics Recognized by Forrester

News

By Tinuiti Team

Apr 16 2024

Rapid Media Mix Modeling and Proprietary Tech Transform Brand Performance

Tinuiti, the largest independent full-funnel performance marketing agency, has been included in a recent Forrester Research report titled, “The Marketing Analytics Landscape, Q2 2024.” This report comprehensively overviews marketing analytics markets, use cases, and capabilities. B2C marketing leaders can use this research by Principal Analyst Tina Moffett to understand the intersection of marketing analytics capabilities and use cases to determine the vendor or service provider best positioned for their analytics and insights needs. Moffett describes the top marketing analytics markets as advertising agencies, marketing dashboards and business intelligence tools, marketing measurement and optimization platforms and service providers, and media analytics tools.

As an advertising agency, we believe Tinuiti is uniquely positioned to manage advertising campaigns for brands including buying, targeting, and measurement. Our proprietary measurement technology, Bliss Point by Tinuiti, allows us to measure the optimal level of investment to maximize impact and efficiency. According to the Forrester report, “only 30% of B2C marketing decision-makers say their organization uses marketing or media mix modeling (MMM),” so having a partner that knows, embraces, and utilizes MMM is important. As Tina astutely explains, data-driven agencies have amplified their marketing analytics competencies with data science expertise; and proprietary tools; and tailored their marketing analytics techniques based on industry, business, and data challenges.

Our Rapid Media Mix Modeling sets a new standard in the market with its exceptional speed, precision, and transparency. Our patented tech includes Rapid Media Mix Modeling, Always-on Incrementality, Brand Equity, Creative Insights, and Forecasting – it will get you to your Marketing Bliss Point in each channel, across your entire media mix, and your overall brand performance.

As a marketing leader you may ask yourself:

How much of our marketing budget should we allocate to driving store traffic versus e-commerce traffic?
How should we allocate our budget by channel to generate the most traffic and revenue possible?
How many customers did we acquire in a specific region with our media spend?
What is the impact of seasonality on our media mix?
How should we adjust our budget accordingly?
What is the optimal marketing channel mix to maximize brand awareness?

These are just a few of the questions that Bliss Point by Tinuiti can help you answer.

Learn more about our customer-obsessed, product-enabled, and fully integrated approach and how we’ve helped fuel full-funnel outcomes for the world’s most digital-forward brands like Poppi & Toms.

The Landscape report is available online to Forrester customers or for purchase here.

MARKETING

Ecommerce evolution: Blurring the lines between B2B and B2C

Published

1 day ago

April 18, 2024

Entireweb News Bot

Ecommerce evolution: Blurring the lines between B2B and B2C

Understanding convergence

B2B and B2C ecommerce are two distinct models of online selling. B2B ecommerce is between businesses, such as wholesalers, distributors, and manufacturers. B2C ecommerce refers to transactions between businesses like retailers and consumer brands, directly to individual shoppers.

However, in recent years, the boundaries between these two models have started to fade. This is known as the convergence between B2B and B2C ecommerce and how they are becoming more similar and integrated.

Source: White Paper: The evolution of the B2B Consumer Buyer (ClientPoint, Jan 2024)

What’s driving this change?

Ever increasing customer expectations

Customers today expect the same level of convenience, speed, and personalization in their B2B transactions as they do in their B2C interactions. B2B buyers are increasingly influenced by their B2C experiences. They want research, compare, and purchase products online, seamlessly transitioning between devices and channels. They also prefer to research and purchase online, using multiple devices and channels.

Forrester, 68% of buyers prefer to research on their own, online . Customers today expect the same level of convenience, speed, and personalization in their B2B transactions as they do in their B2C interactions. B2B buyers are increasingly influenced by their B2C experiences. They want research, compare, and purchase products online, seamlessly transitioning between devices and channels. They also prefer to research and purchase online, using multiple devices and channels

Technology and omnichannel strategies

Technology enables B2B and B2C ecommerce platforms to offer more features and functionalities, such as mobile optimization, chatbots, AI, and augmented reality. Omnichannel strategies allow B2B and B2C ecommerce businesses to provide a seamless and consistent customer experience across different touchpoints, such as websites, social media, email, and physical stores.

However, with every great leap forward comes its own set of challenges. The convergence of B2B and B2C markets means increased competition. Businesses now not only have to compete with their traditional rivals, but also with new entrants and disruptors from different sectors. For example, Amazon Business, a B2B ecommerce platform, has become a major threat to many B2B ecommerce businesses, as it offers a wide range of products, low prices, and fast delivery

“Amazon Business has proven that B2B ecommerce can leverage popular B2C-like functionality” argues Joe Albrecht, CEO / Managing Partner, Xngage. . With features like Subscribe-and-Save (auto-replenishment), one-click buying, and curated assortments by job role or work location, they make it easy for B2B buyers to go to their website and never leave. Plus, with exceptional customer service and promotional incentives like Amazon Business Prime Days, they have created a reinforcing loyalty loop.

And yet, according to Barron’s, Amazon Business is only expected to capture 1.5% of the $5.7 Trillion addressable business market by 2025. If other B2B companies can truly become digital-first organizations, they can compete and win in this fragmented space, too.”

If other B2B companies can truly become digital-first organizations, they can also compete and win in this fragmented space

Joe Albrecht
CEO/Managing Partner, XNGAGE

Increasing complexity

Another challenge is the increased complexity and cost of managing a converging ecommerce business. Businesses have to deal with different customer segments, requirements, and expectations, which may require different strategies, processes, and systems. For instance, B2B ecommerce businesses may have to handle more complex transactions, such as bulk orders, contract negotiations, and invoicing, while B2C ecommerce businesses may have to handle more customer service, returns, and loyalty programs. Moreover, B2B and B2C ecommerce businesses must invest in technology and infrastructure to support their convergence efforts, which may increase their operational and maintenance costs.

How to win

Here are a few ways companies can get ahead of the game:

Adopt B2C-like features in B2B platforms

User-friendly design, easy navigation, product reviews, personalization, recommendations, and ratings can help B2B ecommerce businesses to attract and retain more customers, as well as to increase their conversion and retention rates.

According to McKinsey, ecommerce businesses that offer B2C-like features like personalization can increase their revenues by 15% and reduce their costs by 20%. You can do this through personalization of your website with tools like Product Recommendations that help suggest related products to increase sales.

Focus on personalization and customer experience

B2B and B2C ecommerce businesses need to understand their customers’ needs, preferences, and behaviors, and tailor their offerings and interactions accordingly. Personalization and customer experience can help B2B and B2C ecommerce businesses to increase customer satisfaction, loyalty, and advocacy, as well as to improve their brand reputation and competitive advantage. According to a Salesforce report, 88% of customers say that the experience a company provides is as important as its products or services.

Market based on customer insights

Data and analytics can help B2B and B2C ecommerce businesses to gain insights into their customers, markets, competitors, and performance, and to optimize their strategies and operations accordingly. Data and analytics can also help B2B and B2C ecommerce businesses to identify new opportunities, trends, and innovations, and to anticipate and respond to customer needs and expectations. According to McKinsey, data-driven organizations are 23 times more likely to acquire customers, six times more likely to retain customers, and 19 times more likely to be profitable.

What’s next?

The convergence of B2B and B2C ecommerce is not a temporary phenomenon, but a long-term trend that will continue to shape the future of ecommerce. According to Statista, the global B2B ecommerce market is expected to reach $20.9 trillion by 2027, surpassing the B2C ecommerce market, which is expected to reach $10.5 trillion by 2027. Moreover, the report predicts that the convergence of B2B and B2C ecommerce will create new business models, such as B2B2C, B2A (business to anyone), and C2B (consumer to business).

Therefore, B2B and B2C ecommerce businesses need to prepare for the converging ecommerce landscape and take advantage of the opportunities and challenges it presents. Here are some recommendations for B2B and B2C ecommerce businesses to navigate the converging landscape:

Conduct a thorough analysis of your customers, competitors, and market, and identify the gaps and opportunities for convergence.
Develop a clear vision and strategy for convergence, and align your goals, objectives, and metrics with it.
Invest in technology and infrastructure that can support your convergence efforts, such as cloud, mobile, AI, and omnichannel platforms.
Implement B2C-like features in your B2B platforms, and vice versa, to enhance your customer experience and satisfaction.
Personalize your offerings and interactions with your customers, and provide them with relevant and valuable content and solutions.
Leverage data and analytics to optimize your performance and decision making, and to innovate and differentiate your business.
Collaborate and partner with other B2B and B2C ecommerce businesses, as well as with other stakeholders, such as suppliers, distributors, and customers, to create value and synergy.
Monitor and evaluate your convergence efforts, and adapt and improve them as needed.

By following these recommendations, B2B and B2C ecommerce businesses can bridge the gap between their models and create a more integrated and seamless ecommerce experience for their customers and themselves.

MARKETING

Streamlining Processes for Increased Efficiency and Results

Published

2 days ago

April 18, 2024

Entireweb News Bot

Streamlining Processes for Increased Efficiency and Results

How can businesses succeed nowadays when technology rules? With competition getting tougher and customers changing their preferences often, it’s a challenge. But using marketing automation can help make things easier and get better results. And in the future, it’s going to be even more important for all kinds of businesses.

So, let’s discuss how businesses can leverage marketing automation to stay ahead and thrive.

Benefits of automation marketing automation to boost your efforts

First, let’s explore the benefits of marketing automation to supercharge your efforts:

Marketing automation simplifies repetitive tasks, saving time and effort.

With automated workflows, processes become more efficient, leading to better productivity. For instance, automation not only streamlines tasks like email campaigns but also optimizes website speed, ensuring a seamless user experience. A faster website not only enhances customer satisfaction but also positively impacts search engine rankings, driving more organic traffic and ultimately boosting conversions.

Automation allows for precise targeting, reaching the right audience with personalized messages.

With automated workflows, processes become more efficient, leading to better productivity. A great example of automated workflow is Pipedrive & WhatsApp Integration in which an automated welcome message pops up on their WhatsApp

within seconds once a potential customer expresses interest in your business.

Increases ROI

By optimizing campaigns and reducing manual labor, automation can significantly improve return on investment.

Leveraging automation enables businesses to scale their marketing efforts effectively, driving growth and success. Additionally, incorporating lead scoring into automated marketing processes can streamline the identification of high-potential prospects, further optimizing resource allocation and maximizing conversion rates.

Harnessing the power of marketing automation can revolutionize your marketing strategy, leading to increased efficiency, higher returns, and sustainable growth in today’s competitive market. So, why wait? Start automating your marketing efforts today and propel your business to new heights, moreover if you have just learned ways on how to create an online business

How marketing automation can simplify operations and increase efficiency

Understanding the Change

Marketing automation has evolved significantly over time, from basic email marketing campaigns to sophisticated platforms that can manage entire marketing strategies. This progress has been fueled by advances in technology, particularly artificial intelligence (AI) and machine learning, making automation smarter and more adaptable.

One of the main reasons for this shift is the vast amount of data available to marketers today. From understanding customer demographics to analyzing behavior, the sheer volume of data is staggering. Marketing automation platforms use this data to create highly personalized and targeted campaigns, allowing businesses to connect with their audience on a deeper level.

The Emergence of AI-Powered Automation

In the future, AI-powered automation will play an even bigger role in marketing strategies. AI algorithms can analyze huge amounts of data in real-time, helping marketers identify trends, predict consumer behavior, and optimize campaigns as they go. This agility and responsiveness are crucial in today’s fast-moving digital world, where opportunities come and go in the blink of an eye. For example, we’re witnessing the rise of AI-based tools from AI website builders, to AI logo generators and even more, showing that we’re competing with time and efficiency.

Combining AI-powered automation with WordPress management services streamlines marketing efforts, enabling quick adaptation to changing trends and efficient management of online presence.

Moreover, AI can take care of routine tasks like content creation, scheduling, and testing, giving marketers more time to focus on strategic activities. By automating these repetitive tasks, businesses can work more efficiently, leading to better outcomes. AI can create social media ads tailored to specific demographics and preferences, ensuring that the content resonates with the target audience. With the help of an AI ad maker tool, businesses can efficiently produce high-quality advertisements that drive engagement and conversions across various social media platforms.

Personalization on a Large Scale

Personalization has always been important in marketing, and automation is making it possible on a larger scale. By using AI and machine learning, marketers can create tailored experiences for each customer based on their preferences, behaviors, and past interactions with the brand.

This level of personalization not only boosts customer satisfaction but also increases engagement and loyalty. When consumers feel understood and valued, they are more likely to become loyal customers and brand advocates. As automation technology continues to evolve, we can expect personalization to become even more advanced, enabling businesses to forge deeper connections with their audience. As your company has tiny homes for sale California, personalized experiences will ensure each customer finds their perfect fit, fostering lasting connections.

Integration Across Channels

Another trend shaping the future of marketing automation is the integration of multiple channels into a cohesive strategy. Today’s consumers interact with brands across various touchpoints, from social media and email to websites and mobile apps. Marketing automation platforms that can seamlessly integrate these channels and deliver consistent messaging will have a competitive edge. When creating a comparison website it’s important to ensure that the platform effectively aggregates data from diverse sources and presents it in a user-friendly manner, empowering consumers to make informed decisions.

Omni-channel integration not only betters the customer experience but also provides marketers with a comprehensive view of the customer journey. By tracking interactions across channels, businesses can gain valuable insights into how consumers engage with their brand, allowing them to refine their marketing strategies for maximum impact. Lastly, integrating SEO services into omni-channel strategies boosts visibility and helps businesses better understand and engage with their customers across different platforms.

The Human Element

While automation offers many benefits, it’s crucial not to overlook the human aspect of marketing. Despite advances in AI and machine learning, there are still elements of marketing that require human creativity, empathy, and strategic thinking.

Successful marketing automation strikes a balance between technology and human expertise. By using automation to handle routine tasks and data analysis, marketers can focus on what they do best – storytelling, building relationships, and driving innovation.

Conclusion

The future of marketing automation looks promising, offering improved efficiency and results for businesses of all sizes.

As AI continues to advance and consumer expectations change, automation will play an increasingly vital role in keeping businesses competitive.

By embracing automation technologies, marketers can simplify processes, deliver more personalized experiences, and ultimately, achieve their business goals more effectively than ever before.