Connect with us

MARKETING

How to Determine Your A/B Testing Sample Size & Time Frame

Published

on

How to Determine Your A/B Testing Sample Size & Time Frame

Do you remember your first A/B test you ran? I do. (Nerdy, I know.)

I felt simultaneously thrilled and terrified because I knew I had to actually use some of what I learned in college for my job.

There were some aspects of A/B testing I still remembered — for instance, I knew you need a big enough sample size to run the test on, and you need to run the test long enough to get statistically significant results.

But … that’s pretty much it. I wasn’t sure how big was “big enough” for sample sizes and how long was “long enough” for test durations — and Googling it gave me a variety of answers my college statistics courses definitely didn’t prepare me for.

Turns out I wasn’t alone: Those are two of the most common A/B testing questions we get from customers. And the reason the typical answers from a Google search aren’t that helpful is because they’re talking about A/B testing in an ideal, theoretical, non-marketing world.

So, I figured I’d do the research to help answer this question for you in a practical way. At the end of this post, you should be able to know how to determine the right sample size and time frame for your next A/B test. Let’s dive in.

Free Download: A/B Testing Guide and Kit

A/B Testing Sample Size & Time Frame

In theory, to determine a winner between Variation A and Variation B, you need to wait until you have enough results to see if there is a statistically significant difference between the two.

Depending on your company, sample size, and how you execute the A/B test, getting statistically significant results could happen in hours or days or weeks — and you’ve just got to stick it out until you get those results. In theory, you should not restrict the time in which you’re gathering results.

For many A/B tests, waiting is no problem. Testing headline copy on a landing page? It’s cool to wait a month for results. Same goes with blog CTA creative — you’d be going for the long-term lead generation play, anyway.

But certain aspects of marketing demand shorter timelines when it comes to A/B testing. Take email as an example. With email, waiting for an A/B test to conclude can be a problem, for several practical reasons:

1. Each email send has a finite audience.

Unlike a landing page (where you can continue to gather new audience members over time), once you send an email A/B test off, that’s it — you can’t “add” more people to that A/B test. So you’ve got to figure out how squeeze the most juice out of your emails.

This will usually require you to send an A/B test to the smallest portion of your list needed to get statistically significant results, pick a winner, and then send the winning variation on to the rest of the list.

2. Running an email marketing program means you’re juggling at least a few email sends per week. (In reality, probably way more than that.)

If you spend too much time collecting results, you could miss out on sending your next email — which could have worse effects than if you sent a non-statistically-significant winner email on to one segment of your database.

3. Email sends are often designed to be timely.

Your marketing emails are optimized to deliver at a certain time of day, whether your emails are supporting the timing of a new campaign launch and/or landing in your recipient’s inboxes at a time they’d love to receive it. So if you wait for your email to be fully statistically significant, you might miss out on being timely and relevant — which could defeat the purpose of your email send in the first place.

That’s why email A/B testing programs have a “timing” setting built in: At the end of that time frame, if neither result is statistically significant, one variation (which you choose ahead of time) will be sent to the rest of your list. That way, you can still run A/B tests in email, but you can also work around your email marketing scheduling demands and ensure people are always getting timely content.

So to run A/B tests in email while still optimizing your sends for the best results, you’ve got to take both sample size and timing into account.

Next up — how to actually figure out your sample size and timing using data.

How to Determine Sample Size for an A/B Test

Now, let’s dive into how to actually calculate the sample size and timing you need for your next A/B test.

For our purposes, we’re going to use email as our example to demonstrate how you’ll determine sample size and timing for an A/B test. However, it’s important to note — the steps in this list can be used for any A/B test, not just email.

Let’s dive in.

Like mentioned above, each A/B test you send can only be sent to a finite audience — so you need to figure out how to maximize the results from that A/B test. To do that, you need to figure out the smallest portion of your total list needed to get statistically significant results. Here’s how you calculate it.

1. Assess whether you have enough contacts in your list to A/B test a sample in the first place.

To A/B test a sample of your list, you need to have a decently large list size — at least 1,000 contacts. If you have fewer than that in your list, the proportion of your list that you need to A/B test to get statistically significant results gets larger and larger.

For example, to get statistically significant results from a small list, you might have to test 85% or 95% of your list. And the results of the people on your list who haven’t been tested yet will be so small that you might as well have just sent half of your list one email version, and the other half another, and then measured the difference.

Your results might not be statistically significant at the end of it all, but at least you’re gathering learnings while you grow your lists to have more than 1,000 contacts. (If you want more tips on growing your email list so you can hit that 1,000 contact threshold, check out this blog post.)

Note for HubSpot customers: 1,000 contacts is also our benchmark for running A/B tests on samples of email sends — if you have fewer than 1,000 contacts in your selected list, the A version of your test will automatically be sent to half of your list and the B will be sent to the other half.

2. Use a sample size calculator.

Next, you’ll want to find a sample size calculator — HubSpot’s A/B Testing Kit offers a good, free sample size calculator.

Here’s what it looks like when you download it:

ab significance calculatorDownload for Free

3. Put in your email’s Confidence Level, Confidence Interval, and Population into the tool.

Yep, that’s a lot of statistics jargon. Here’s what these terms translate to in your email:

Population: Your sample represents a larger group of people. This larger group is called your population.

In email, your population is the typical number of people in your list who get emails delivered to them — not the number of people you sent emails to. To calculate population, I’d look at the past three to five emails you’ve sent to this list, and average the total number of delivered emails. (Use the average when calculating sample size, as the total number of delivered emails will fluctuate.)

Confidence Interval: You might have heard this called “margin of error.” Lots of surveys use this, including political polls. This is the range of results you can expect this A/B test to explain once it’s run with the full population.

For example, in your emails, if you have an interval of 5, and 60% of your sample opens your Variation, you can be sure that between 55% (60 minus 5) and 65% (60 plus 5) would have also opened that email. The bigger the interval you choose, the more certain you can be that the populations true actions have been accounted for in that interval. At the same time, large intervals will give you less definitive results. It’s a trade-off you’ll have to make in your emails.

For our purposes, it’s not worth getting too caught up in confidence intervals. When you’re just getting started with A/B tests, I’d recommend choosing a smaller interval (ex: around 5).

Confidence Level: This tells you how sure you can be that your sample results lie within the above confidence interval. The lower the percentage, the less sure you can be about the results. The higher the percentage, the more people you’ll need in your sample, too.

Note for HubSpot customers: The HubSpot Email A/B tool automatically uses the 85% confidence level to determine a winner. Since that option isn’t available in this tool, I’d suggest choosing 95%.

Email A/B Test Example:

Let’s pretend we’re sending our first A/B test. Our list has 1,000 people in it and has a 95% deliverability rate. We want to be 95% confident our winning email metrics fall within a 5-point interval of our population metrics.

Here’s what we’d put in the tool:

  • Population: 950
  • Confidence Level: 95%
  • Confidence Interval: 5

sample_size_calculations

4. Click “Calculate” and your sample size will spit out.

Ta-da! The calculator will spit out your sample size.

In our example, our sample size is: 274.

This is the size one your variations needs to be. So for your email send, if you have one control and one variation, you’ll need to double this number. If you had a control and two variations, you’d triple it. (And so on.)

5. Depending on your email program, you may need to calculate the sample size’s percentage of the whole email.

HubSpot customers, I’m looking at you for this section. When you’re running an email A/B test, you’ll need to select the percentage of contacts to send the list to — not just the raw sample size.

To do that, you need to divide the number in your sample by the total number of contacts in your list. Here’s what that math looks like, using the example numbers above:

274 / 1,000 = 27.4%

This means that each sample (both your control AND your variation) needs to be sent to 27-28% of your audience — in other words, roughly a total of 55% of your total list.

email_ab_test_send

And that’s it! You should be ready to select your sending time.

How to Choose the Right Timeframe for Your A/B Test

Again, for figuring out the right timeframe for your A/B test, we’ll use the example of email sends – but this information should still apply regardless of the type of A/B test you’re conducting.

However, your timeframe will vary depending on your business’ goals, as well. If you’d like to design a new landing page by Q2 2021 and it’s Q4 2020, you’ll likely want to finish your A/B test by January or February so you can use those results to build the winning page.

But, for our purposes, let’s return to the email send example: You have to figure out how long to run your email A/B test before sending a (winning) version on to the rest of your list.

Figuring out the timing aspect is a little less statistically driven, but you should definitely use past data to help you make better decisions. Here’s how you can do that.

If you don’t have timing restrictions on when to send the winning email to the rest of the list, head over to your analytics.

Figure out when your email opens/clicks (or whatever your success metrics are) starts to drop off. Look your past email sends to figure this out.

For example, what percentage of total clicks did you get in your first day? If you found that you get 70% of your clicks in the first 24 hours, and then 5% each day after that, it’d make sense to cap your email A/B testing timing window for 24 hours because it wouldn’t be worth delaying your results just to gather a little bit of extra data.

In this scenario, you would probably want to keep your timing window to 24 hours, and at the end of 24 hours, your email program should let you know if they can determine a statistically significant winner.

Then, it’s up to you what to do next. If you have a large enough sample size and found a statistically significant winner at the end of the testing time frame, many email marketing programs will automatically and immediately send the winning variation.

If you have a large enough sample size and there’s no statistically significant winner at the end of the testing time frame, email marketing tools might also allow you to automatically send a variation of your choice.

If you have a smaller sample size or are running a 50/50 A/B test, when to send the next email based on the initial email’s results is entirely up to you.

If you have time restrictions on when to send the winning email to the rest of the list, figure out how late you can send the winner without it being untimely or affecting other email sends.

For example, if you’ve sent an email out at 3 p.m. EST for a flash sale that ends at midnight EST, you wouldn’t want to determine an A/B test winner at 11 p.m. Instead, you’d want to send the email closer to 6 or 7 p.m. — that’ll give the people not involved in the A/B test enough time to act on your email.

And that’s pretty much it, folks. After doing these calculations and examining your data, you should be in a much better state to conduct successful A/B tests — ones that are statistically valid and help you move the needle on your goals.

The Ultimate A/B Testing Kit

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address

MARKETING

How Does Success of Your Business Depend on Choosing Type of Native Advertising?

Published

on

How Does Success of Your Business Depend on Choosing Type of Native Advertising?

The very first commercial advertisement was shown on TV in 1941. It was only 10 seconds long and had an audience of 4,000 people. However, it became a strong trigger for rapid advertising development. The second half of the 20th century is known as the golden age of advertising until the Internet came to the forefront and entirely transformed the advertising landscape. The first commercial banner appeared in the mid-90s, then it was followed by pop-ups, pay-by-placement and paid-pay-click ads. Companies also started advertising their brands and adding their business logo designs, which contributes to consumer trust and trustworthiness.

The rise of social media in the mid-2000s opened a new dimension for advertising content to be integrated. The marketers were forced to make the ads less intrusive and more organic to attract younger users. This is how native advertising was born. This approach remains a perfect medium for goods and services promotion. Let’s see why and how native ads can become a win-win strategy for your business.

What is native advertising?

When it comes to digital marketing, every marketer talks about native advertising. What is the difference between traditional and native ones? You will not miss basic ads as they are typically promotional and gimmicky, while native advertising naturally blends into the content. The primary purpose of native ads is to create content that resonates with audience expectations and encourages users to perceive it seamlessly and harmoniously.

Simply put, native advertising is a paid media ad that organically aligns with the visual and operational features of the media format in which it appears. The concept is quite straightforward: while people just look through banner ads, they genuinely engage with native ads and read them. You may find a lot of native ads on Facebook, Twitter and Instagram – they appear in the form of “in-feed” posts that engage users in search for more stories, opinions, goods and services. This unobtrusive approach turns native ads into a powerful booster for any brand.

How does native advertising benefit your business?

An average Internet user comes across around 10,000 ads a day. But even physically, it is impossible to perceive this amount of information in 24 hours. So, most of them use adblockers, nullifying all efforts of markers. Native ads successfully overcome this digital challenge thanks to their authenticity. And this is not the only advantage of native advertising. How else does your business benefit? Here are just a few major benefits that prove the value of native ads:

Better brand awareness. Native ads contribute to the brand’s visibility. They seamlessly blend into educational, emotional, and visual types of content that can easily become viral. While promotional content typically receives limited shares, users readily share valuable or entertaining content. Consequently, while you incur expenses only for the display of native ads, your audience may go the extra mile by sharing your content and organically promoting your brand or SaaS product at no additional cost.

Increased click-through rates. Native ads can generate a thrilling click-through rate (CTR) primarily because they are meticulously content-adaptable. Thus, native ads become an integral part of the user’s journey without disrupting their browsing experience. Regardless of whether your native advertising campaign is designed to build an audience or drive specific actions, compelling content will always entice users to click through.

Cost-efficient campaign performance. Native advertising proves to be cheaper compared to a traditional ad format. It mainly stems from a higher CTR. Thanks to precise targeting and less customer resistance, native ads allow to bring down cost-per-click.

Native ads are continuously evolving, enabling marketers to experiment with different formats and use them for successful multi-channel campaigns and global reach.

Types of native advertising

Any content can become native advertising as there are no strict format restrictions. For example, it can be an article rating the best fitness applications, an equipment review, or a post by an influencer on a microblog. The same refers to the channels – native ads can be placed on regular websites and social media feeds. Still, some forms tend to be most frequently used.

  • In-feed ads. This type of ad appears within the content feed. You have definitely seen such posts on Facebook and Instagram or such videos on TikTok. They look like regular content but are tagged with an advertising label. The user sees these native ads when scrolling the feed on social media platforms.
  • Paid search ads. These are native ads that are displayed on the top and bottom of the search engine results page. They always match user’s queries and aim to capture their attention at the moment of a particular search and generate leads and conversions. This type of ad is effective for big search platforms with substantial traffic.
  • Recommendation widgets. These come in the form of either texts or images and can be found at the end of the page or on a website’s sidebar. Widgets offer related or intriguing content from either the same publisher or similar sources. This type of native ads is great for retargeting campaigns.
  • Sponsored content. This is one of the most popular types of native advertising. Within this format, an advertiser sponsors the creation of an article or content that aligns with the interests and values of the platform’s audience. They can be marked as “sponsored” or “recommended” to help users differentiate them from organic content.
  • Influencer Advertising. In this case, advertisers partner with popular bloggers or celebrities to gain the attention and trust of the audience. Influencers integrate a product, service, or event into their content or create custom content that matches their style and topic.

Each of these formats can bring stunning results if your native ads are relevant and provide value to users. Use a creative automation platform like Creatopy to design effective ads for your business.

How to create a workable native ad?

Consider these 5 steps for creating a successful native advertising campaign:

  • Define your target audienceUsers will always ignore all ads that are not relevant to them. Unwanted ads are frustrating and can even harm your brand. If you run a store for pets, make sure your ads show content that will be interesting for pet owners. Otherwise, the whole campaign will be undermined. Regular market research and data analysis will help you refine your audience and its demographics.
  • Set your goals. Each advertising campaign should have a clear-cut objective. Without well-defined goals, it is a waste of money. It is a must to know what you want to achieve – introduce your brand, boost sales or increase your audience.
  • Select the proper channels. Now, you need to determine how you will reach out to your customers. Consider displaying ads on social media platforms, targeting search engine result pages (SERPs), distributing paid articles, or utilizing in-ad units on different websites. You may even be able to get creative and use email or SMS in a less salesy and more “native”-feeling way—you can find samples of texts online to help give you ideas. Exploring demand side platforms (DSP) can also bring good results.
  • Offer compelling content. Do not underestimate the quality of the content for your native ads. Besides being expertly written, it must ideally match the style and language of the chosen channel,whether you’re promoting professional headshots, pet products, or anything else. The main distinctive feature of native advertising is that it should fit naturally within the natural content.
  • Track your campaign. After the launch of native ads, it is crucial to monitor the progress, evaluating the costs spent and results. Use tools that help you gain insights beyond standard KPIs like CTR and CPC. You should get engagement metrics, customer data, campaign data, and third-party activity data for further campaign management.

Key takeaway

Summing up the above, it is time to embrace native advertising if you haven’t done it yet. Native ads seamlessly blend with organic content across various platforms, yielding superior engagement and conversion rates compared to traditional display ads. Marketers are allocating higher budgets to native ads because this format proves to be more and more effective – content that adds value can successfully deal with ad fatigue. Native advertising is experiencing a surge in popularity, and it is to reach its peak. So, do not miss a chance to grow your business with the power of native ads.or you can do digital marketing course from Digital Vidya.

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

MARKETING

OpenAI’s Drama Should Teach Marketers These 2 Lessons

Published

on

OpenAI’s Drama Should Teach Marketers These 2 Lessons

A week or so ago, the extraordinary drama happening at OpenAI filled news feeds.

No need to get into all the saga’s details, as every publication seems to have covered it. We’re just waiting for someone to put together a video montage scored to the Game of Thrones music.

But as Sam Altman takes back the reigns of the company he helped to found, the existing board begins to disintegrate before your very eyes, and everyone agrees something spooked everybody, a question arises: Should you care?

Does OpenAI’s drama have any demonstrable implications for marketers integrating generative AI into their marketing strategies?

Watch CMI’s chief strategy advisor Robert Rose explain (and give a shoutout to Sutton’s pants rage on The Real Housewives of Beverly Hills), or keep reading his thoughts:

For those who spent last week figuring out what to put on your holiday table and missed every AI headline, here’s a brief version of what happened. OpenAI – the huge startup and creator of ChatGPT – went through dramatic events. Its board fired the mercurial CEO Sam Altman. Then, the 38-year-old entrepreneur accepted a job at Microsoft but returned to OpenAI a day later.

We won’t give a hot take on what it means for the startup world, board governance, or the tension between AI safety and Silicon Valley capitalism. Rather, we see some interesting things for marketers to put into perspective about how AI should fit into your overall content and marketing plans in the new year.

Robert highlights two takeaways from the OpenAI debacle – a drama that has yet to reach its final chapter: 1. The right structure and governance matters, and 2. Big platforms don’t become antifragile just because they’re big.

Let’s have Robert explain.

The right structure and governance matters

OpenAI’s structure may be key to the drama. OpenAI has a bizarre corporate governance framework. The board of directors controls a nonprofit called OpenAI. That nonprofit created a capped for-profit subsidiary – OpenAI GP LLC. The majority owner of that for-profit is OpenAI Global LLC, another for-profit company. The nonprofit works for the benefit of the world with a for-profit arm.

That seems like an earnest approach, given AI tech’s big and disruptive power. But it provides so many weird governance issues, including that the nonprofit board, which controls everything, has no duty to maximize profit. What could go wrong?

That’s why marketers should know more about the organizations behind the generative AI tools they use or are considering.

First, know your providers of generative AI software and services are all exploring the topics of governance and safety. Microsoft, Google, Anthropic, and others won’t have their internal debates erupt in public fireworks. Still, governance and management of safety over profits remains a big topic for them. You should be aware of how they approach those topics as you license solutions from them.

Second, recognize the productive use of generative AI is a content strategy and governance challenge, not a technology challenge. If you don’t solve the governance and cross-functional uses of the generative AI platforms you buy, you will run into big problems with its cross-functional, cross-siloed use. 

Big platforms do not become antifragile just because they’re big

Nicholas Taleb wrote a wonderful book, Antifragile: Things That Gain From Disorder. It explores how an antifragile structure doesn’t just withstand a shock; it actually improves because of a disruption or shock. It doesn’t just survive a big disruptive event; it gets stronger because of it.

It’s hard to imagine a company the size and scale of OpenAI could self-correct or even disappear tomorrow. But it can and does happen. And unfortunately, too many businesses build their strategies on that rented land.

In OpenAI’s recent case, the for-profit software won the day. But make no bones about that victory; the event wasn’t good for the company. If it bounces back, it won’t be stronger because of the debacle.

With that win on the for-profit side, hundreds, if not thousands, of generative AI startups breathed an audible sigh of relief. But a few moments later, they screamed “pivot” (in their best imitation of Ross from Friends instructing Chandler and Rachel to move a couch.)

They now realize the fragility of their software because it relies on OpenAI’s existence or willingness to provide the software. Imagine what could have happened if the OpenAI board had won their fight and, in the name of safety, simply killed any paid access to the API or the ability to build business models on top of it.

The last two weeks have done nothing to clear the already muddy waters encountered by companies and their plans to integrate generative AI solutions. Going forward, though, think about the issues when acquiring new generative AI software. Ask about how the vendor’s infrastructure is housed and identify the risks involved. And, if OpenAI expands its enterprise capabilities, consider the implications. What extra features will the off-the-shelf solutions provide? Do you need them? Will OpenAI become the Microsoft Office of your AI infrastructure?

Why you should care

With the voluminous media coverage of Open AI’s drama, you likely will see pushback on generative AI. In my social feeds, many marketers say they’re tired of the corporate soap opera that is irrelevant to their work.

They are half right. What Sam said and how Ilya responded, heart emojis, and how much the Twitch guy got for three days of work are fodder for the Netflix series sure to emerge. (Robert’s money is on Michael Cera starring.)

They’re wrong about its relevance to marketing. They must be experiencing attentional bias – paying more attention to some elements of the big event and ignoring others. OpenAI’s struggle is entertaining, no doubt. You’re glued to the drama. But understanding what happened with the events directly relates to your ability to manage similar ones successfully. That’s the part you need to get right.

Want more content marketing tips, insights, and examples? Subscribe to workday or weekly emails from CMI.

HANDPICKED RELATED CONTENT:

Cover image by Joseph Kalinowski/Content Marketing Institute

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

MARKETING

The Complete Guide to Becoming an Authentic Thought Leader

Published

on

The Complete Guide to Becoming an Authentic Thought Leader

Introduce your processes: If you’ve streamlined a particular process, share it. It could be the solution someone else is looking for.

Jump on trends and news: If there’s a hot topic or emerging trend, offer your unique perspective.

Share industry insights: Attended a webinar or podcast that offered valuable insights. Summarize the key takeaways and how they can be applied.

Share your successes: Write about strategies that have worked exceptionally well for you. Your audience will appreciate the proven advice. For example, I shared the process I used to help a former client rank for a keyword with over 2.2 million monthly searches.

Question outdated strategies: If you see a strategy that’s losing steam, suggest alternatives based on your experience and data.

5. Establish communication channels (How)

Once you know who your audience is and what they want to hear, the next step is figuring out how to reach them. Here’s how:

Choose the right platforms: You don’t need to have a presence on every social media platform. Pick two platforms where your audience hangs out and create content for that platform. For example, I’m active on LinkedIn and X because my target audience (SEOs, B2B SaaS, and marketers) is active on these platforms.

Repurpose content: Don’t limit yourself to just one type of content. Consider repurposing your content on Quora, Reddit, or even in webinars and podcasts. This increases your reach and reinforces your message.

Follow Your audience: Go where your audience goes. If they’re active on X, that’s where you should be posting. If they frequent industry webinars, consider becoming a guest on these webinars.

Daily vs. In-depth content: Balance is key. Use social media for daily tips and insights, and reserve your blog for more comprehensive guides and articles.

Network with influencers: Your audience is likely following other experts in the field. Engaging with these influencers puts your content in front of a like-minded audience. I try to spend 30 minutes to an hour daily engaging with content on X and LinkedIn. This is the best way to build a relationship so you’re not a complete stranger when you DM privately.

6. Think of thought leadership as part of your content marketing efforts

As with other content efforts, thought leadership doesn’t exist in a vacuum. It thrives when woven into a cohesive content marketing strategy. By aligning individual authority with your brand, you amplify the credibility of both.

Think of it as top-of-the-funnel content to:

  • Build awareness about your brand

  • Highlight the problems you solve

  • Demonstrate expertise by platforming experts within the company who deliver solutions

Consider the user journey. An individual enters at the top through a social media post, podcast, or blog post. Intrigued, they want to learn more about you and either search your name on Google or social media. If they like what they see, they might visit your website, and if the information fits their needs, they move from passive readers to active prospects in your sales pipeline.

Source link

Keep an eye on what we are doing
Be the first to get latest updates and exclusive content straight to your email inbox.
We promise not to spam you. You can unsubscribe at any time.
Invalid email address
Continue Reading

Trending