Connect with us

MARKETING

How to Determine Your A/B Testing Sample Size & Time Frame

Published

on

How to Determine Your A/B Testing Sample Size & Time Frame

Do you remember your first A/B test you ran? I do. (Nerdy, I know.)

I felt simultaneously thrilled and terrified because I knew I had to actually use some of what I learned in college for my job.

There were some aspects of A/B testing I still remembered — for instance, I knew you need a big enough sample size to run the test on, and you need to run the test long enough to get statistically significant results.

But … that’s pretty much it. I wasn’t sure how big was “big enough” for sample sizes and how long was “long enough” for test durations — and Googling it gave me a variety of answers my college statistics courses definitely didn’t prepare me for.

Turns out I wasn’t alone: Those are two of the most common A/B testing questions we get from customers. And the reason the typical answers from a Google search aren’t that helpful is because they’re talking about A/B testing in an ideal, theoretical, non-marketing world.

So, I figured I’d do the research to help answer this question for you in a practical way. At the end of this post, you should be able to know how to determine the right sample size and time frame for your next A/B test. Let’s dive in.

Free Download: A/B Testing Guide and Kit

A/B Testing Sample Size & Time Frame

In theory, to determine a winner between Variation A and Variation B, you need to wait until you have enough results to see if there is a statistically significant difference between the two.

Depending on your company, sample size, and how you execute the A/B test, getting statistically significant results could happen in hours or days or weeks — and you’ve just got to stick it out until you get those results. In theory, you should not restrict the time in which you’re gathering results.

For many A/B tests, waiting is no problem. Testing headline copy on a landing page? It’s cool to wait a month for results. Same goes with blog CTA creative — you’d be going for the long-term lead generation play, anyway.

But certain aspects of marketing demand shorter timelines when it comes to A/B testing. Take email as an example. With email, waiting for an A/B test to conclude can be a problem, for several practical reasons:

1. Each email send has a finite audience.

Unlike a landing page (where you can continue to gather new audience members over time), once you send an email A/B test off, that’s it — you can’t “add” more people to that A/B test. So you’ve got to figure out how squeeze the most juice out of your emails.

This will usually require you to send an A/B test to the smallest portion of your list needed to get statistically significant results, pick a winner, and then send the winning variation on to the rest of the list.

2. Running an email marketing program means you’re juggling at least a few email sends per week. (In reality, probably way more than that.)

If you spend too much time collecting results, you could miss out on sending your next email — which could have worse effects than if you sent a non-statistically-significant winner email on to one segment of your database.

3. Email sends are often designed to be timely.

Your marketing emails are optimized to deliver at a certain time of day, whether your emails are supporting the timing of a new campaign launch and/or landing in your recipient’s inboxes at a time they’d love to receive it. So if you wait for your email to be fully statistically significant, you might miss out on being timely and relevant — which could defeat the purpose of your email send in the first place.

That’s why email A/B testing programs have a “timing” setting built in: At the end of that time frame, if neither result is statistically significant, one variation (which you choose ahead of time) will be sent to the rest of your list. That way, you can still run A/B tests in email, but you can also work around your email marketing scheduling demands and ensure people are always getting timely content.

So to run A/B tests in email while still optimizing your sends for the best results, you’ve got to take both sample size and timing into account.

Next up — how to actually figure out your sample size and timing using data.

How to Determine Sample Size for an A/B Test

Now, let’s dive into how to actually calculate the sample size and timing you need for your next A/B test.

For our purposes, we’re going to use email as our example to demonstrate how you’ll determine sample size and timing for an A/B test. However, it’s important to note — the steps in this list can be used for any A/B test, not just email.

Let’s dive in.

Like mentioned above, each A/B test you send can only be sent to a finite audience — so you need to figure out how to maximize the results from that A/B test. To do that, you need to figure out the smallest portion of your total list needed to get statistically significant results. Here’s how you calculate it.

1. Assess whether you have enough contacts in your list to A/B test a sample in the first place.

To A/B test a sample of your list, you need to have a decently large list size — at least 1,000 contacts. If you have fewer than that in your list, the proportion of your list that you need to A/B test to get statistically significant results gets larger and larger.

For example, to get statistically significant results from a small list, you might have to test 85% or 95% of your list. And the results of the people on your list who haven’t been tested yet will be so small that you might as well have just sent half of your list one email version, and the other half another, and then measured the difference.

Your results might not be statistically significant at the end of it all, but at least you’re gathering learnings while you grow your lists to have more than 1,000 contacts. (If you want more tips on growing your email list so you can hit that 1,000 contact threshold, check out this blog post.)

Note for HubSpot customers: 1,000 contacts is also our benchmark for running A/B tests on samples of email sends — if you have fewer than 1,000 contacts in your selected list, the A version of your test will automatically be sent to half of your list and the B will be sent to the other half.

2. Use a sample size calculator.

Next, you’ll want to find a sample size calculator — HubSpot’s A/B Testing Kit offers a good, free sample size calculator.

Here’s what it looks like when you download it:

ab significance calculatorDownload for Free

3. Put in your email’s Confidence Level, Confidence Interval, and Population into the tool.

Yep, that’s a lot of statistics jargon. Here’s what these terms translate to in your email:

Population: Your sample represents a larger group of people. This larger group is called your population.

In email, your population is the typical number of people in your list who get emails delivered to them — not the number of people you sent emails to. To calculate population, I’d look at the past three to five emails you’ve sent to this list, and average the total number of delivered emails. (Use the average when calculating sample size, as the total number of delivered emails will fluctuate.)

Confidence Interval: You might have heard this called “margin of error.” Lots of surveys use this, including political polls. This is the range of results you can expect this A/B test to explain once it’s run with the full population.

For example, in your emails, if you have an interval of 5, and 60% of your sample opens your Variation, you can be sure that between 55% (60 minus 5) and 65% (60 plus 5) would have also opened that email. The bigger the interval you choose, the more certain you can be that the populations true actions have been accounted for in that interval. At the same time, large intervals will give you less definitive results. It’s a trade-off you’ll have to make in your emails.

For our purposes, it’s not worth getting too caught up in confidence intervals. When you’re just getting started with A/B tests, I’d recommend choosing a smaller interval (ex: around 5).

Confidence Level: This tells you how sure you can be that your sample results lie within the above confidence interval. The lower the percentage, the less sure you can be about the results. The higher the percentage, the more people you’ll need in your sample, too.

Note for HubSpot customers: The HubSpot Email A/B tool automatically uses the 85% confidence level to determine a winner. Since that option isn’t available in this tool, I’d suggest choosing 95%.

Email A/B Test Example:

Let’s pretend we’re sending our first A/B test. Our list has 1,000 people in it and has a 95% deliverability rate. We want to be 95% confident our winning email metrics fall within a 5-point interval of our population metrics.

Here’s what we’d put in the tool:

  • Population: 950
  • Confidence Level: 95%
  • Confidence Interval: 5

sample_size_calculations

4. Click “Calculate” and your sample size will spit out.

Ta-da! The calculator will spit out your sample size.

In our example, our sample size is: 274.

This is the size one your variations needs to be. So for your email send, if you have one control and one variation, you’ll need to double this number. If you had a control and two variations, you’d triple it. (And so on.)

5. Depending on your email program, you may need to calculate the sample size’s percentage of the whole email.

HubSpot customers, I’m looking at you for this section. When you’re running an email A/B test, you’ll need to select the percentage of contacts to send the list to — not just the raw sample size.

To do that, you need to divide the number in your sample by the total number of contacts in your list. Here’s what that math looks like, using the example numbers above:

274 / 1,000 = 27.4%

This means that each sample (both your control AND your variation) needs to be sent to 27-28% of your audience — in other words, roughly a total of 55% of your total list.

email_ab_test_send

And that’s it! You should be ready to select your sending time.

How to Choose the Right Timeframe for Your A/B Test

Again, for figuring out the right timeframe for your A/B test, we’ll use the example of email sends – but this information should still apply regardless of the type of A/B test you’re conducting.

However, your timeframe will vary depending on your business’ goals, as well. If you’d like to design a new landing page by Q2 2021 and it’s Q4 2020, you’ll likely want to finish your A/B test by January or February so you can use those results to build the winning page.

But, for our purposes, let’s return to the email send example: You have to figure out how long to run your email A/B test before sending a (winning) version on to the rest of your list.

Figuring out the timing aspect is a little less statistically driven, but you should definitely use past data to help you make better decisions. Here’s how you can do that.

If you don’t have timing restrictions on when to send the winning email to the rest of the list, head over to your analytics.

Figure out when your email opens/clicks (or whatever your success metrics are) starts to drop off. Look your past email sends to figure this out.

For example, what percentage of total clicks did you get in your first day? If you found that you get 70% of your clicks in the first 24 hours, and then 5% each day after that, it’d make sense to cap your email A/B testing timing window for 24 hours because it wouldn’t be worth delaying your results just to gather a little bit of extra data.

In this scenario, you would probably want to keep your timing window to 24 hours, and at the end of 24 hours, your email program should let you know if they can determine a statistically significant winner.

Then, it’s up to you what to do next. If you have a large enough sample size and found a statistically significant winner at the end of the testing time frame, many email marketing programs will automatically and immediately send the winning variation.

If you have a large enough sample size and there’s no statistically significant winner at the end of the testing time frame, email marketing tools might also allow you to automatically send a variation of your choice.

If you have a smaller sample size or are running a 50/50 A/B test, when to send the next email based on the initial email’s results is entirely up to you.

If you have time restrictions on when to send the winning email to the rest of the list, figure out how late you can send the winner without it being untimely or affecting other email sends.

For example, if you’ve sent an email out at 3 p.m. EST for a flash sale that ends at midnight EST, you wouldn’t want to determine an A/B test winner at 11 p.m. Instead, you’d want to send the email closer to 6 or 7 p.m. — that’ll give the people not involved in the A/B test enough time to act on your email.

And that’s pretty much it, folks. After doing these calculations and examining your data, you should be in a much better state to conduct successful A/B tests — ones that are statistically valid and help you move the needle on your goals.

The Ultimate A/B Testing Kit

Source link

MARKETING

How to Create Evergreen Content That Generates Traffic to Your Site

Published

on

How to Create Evergreen Content That Generates Traffic to Your Site

Every digital marketer understands that publishing articles or blog posts on your website is an excellent way to create content for your SEO strategy and build awareness. On top of that, you also get to connect to your audience on a deeper level.

However, many marketers focus on publishing seasonal content or articles on trending topics, which can drive traffic, but only for a limited time.

According to HubSpot’s State of Marketing Trends Report (2022), 83% of marketers say it’s better to focus on quality rather than quantity of content, even if it means publishing fewer blog posts.

But if your content depends on trending topics, how can you create quality content that lasts? Enter evergreen content.

First, let’s be honest; creating evergreen content isn’t a walk in the park. Many marketers have tried creating this type of content on their websites but can’t seem to get the results they need.

So how do you create something that doesn’t age?

In this article, I’ll discuss what evergreen content is, why it is important to have one, and how you can create one that will give you lasting results.

Let’s get started.

Evergreen Content Explained

I’m pretty sure many of you have heard this term many times already, and some of you might have an idea of what it is.

But for those still unfamiliar with it, evergreen content is search engine-optimized content that is relevant all year round and can stay fresh over an even longer period. Some may argue that any content is sustainable, given that it remains published once you upload it to your site.

But evergreen content isn’t just any other content. It continues to be of interest to your target audience even after its publication date, attracting more visitors over time.

The topics used in evergreen content are those that many users will search for at any time of the year, making it an excellent strategy for SEO and content marketing.

This type of content comes in many shapes and sizes. Depending on your products or services, evergreen formats can range from listicles, how-to guides, tips, reviews, and videos. Something to note is that producing these formats doesn’t automatically make your content evergreen.

The main ingredient to making evergreen content is the topic, which should stand the test of time.

To understand this better, let’s look at seasonal content. This type of content contains topics tied to a specific time of the year, a trending item, or even in the news or social media. The hype on these topics usually dies down once enough time is passed.

For instance, marketers often produce content for various holidays like Thanksgiving or Christmas.

Content around these times can often generate traffic, but once the holiday is over, it’s unlikely that level of traffic will continue.

Content Marketing Certification

Want to get certified in Content Marketing?

Leverage the tools and channels to predictably and profitably drive awareness, leads, sales, and referrals—EVERYTHING you need to know to become a true master of digital marketing.​ Click Here

Why Is It Important to Have Them on Your Website?

It’s no secret that seasonal content can quickly generate a lot of traffic to your site, but it can become irrelevant even quicker as new trends emerge.

Evergreen content can give you lasting results and so much more. Here are some reasons why you should start creating one:

  • Great for SEO. Evergreen topics are usually searched, which means your content will likely rank on search engines. On top of that, it will always drive traffic to your site and keep on doing so, especially if it’s high quality.
  • Lead magnet. Since you’re showing your expertise through evergreen content, it can encourage visitors to check your product or services. And because it’s relevant all year round, you can use it to attract new leads every time.
  • Excellent marketing material. This is one of my favorite benefits of evergreen content. You can promote it repeatedly, making it a versatile marketing tool.
  • Add value to you and your customers. Since it answers a common problem that your target audience has, you can help them address their issue. At the same time, the steady traffic it provides can help your website grow.
  • Impactful investment. Creating evergreen content takes time. This ensures that you provide high-quality content that will always be interesting to your audience. That’s why it’s an excellent investment by providing significant returns in the future.
  • Lasts for years. The idea that evergreen content can last for years ultimately means less work for you. A single high-quality post can give you a real bang for your buck, compared to seasonal ones that only last for a week or, if you’re lucky, a couple of months.

Step-by-Step Guide: Creating Evergreen Content

Now that we’ve established how critical evergreen content is for your digital marketing strategy, it’s time to learn the ropes of creating one that can be impactful for years. Follow these steps to begin:

1. Find a Problem to Solve

For evergreen content to succeed, it should solve a problem your customers face. It doesn’t have to be a major problem, but it’s something that many people may encounter in their lives.

For example, if you sell car parts like tires, one problem many car owners can encounter is changing their tires. You can create an article or a video showing your customers how they can change their tires and how often to rotate them to avoid wear and tear.

Now you’re not only solving their problem but providing extra value!

But I get it that you just don’t magically stumble upon a problem, and you need to put in some work to find the right fit for your website. So here are some steps you can take to find out the best problem you can address in your evergreen content:

  • Ask yourself. You know your business well, and you can use this first-hand knowledge to look for pain points your customers are facing.
  • Ask your customers. Don’t be afraid to send surveys to your customers. The worst thing that could happen is that they won’t answer it. But the benefits? Learning about their problems and what they expect from your business.
  • Check your competition. While some may tell you to copy what your competitors are doing, I think many other opportunities are available to avoid doing this. Instead, check to see what they’re doing so you can understand what works and what doesn’t. From there, it’ll be easier to come up with an idea of what problem you want to solve for your customers.
  • Check the internet. The internet is an excellent tool, especially if you know how to take advantage of it. One way is to check frequently asked questions about a particular industry through platforms like Quora and Reddit. You’ll be surprised by what information is waiting for you.

2. Conduct Keyword Research

Since evergreen content is a search-engine-optimized piece, it requires some keyword research to make it as effective as possible. This will also allow you to determine whether the problem you want to solve has merit, and people are actually searching the internet for answers to it.

So how do you go about this?

First, you need to invest in the right tools. While there are numerous tools, like Google Trends and Keyword Surfer, that can provide you with basic keyword research features for free. You might miss out on opportunities that only paid tools can provide. But if you’re not ready to shed some cash, the free versions can also give you valuable insights.

Next, you need to understand what to look for. The monthly search volume is the most important metric you have to check when conducting keyword research. This can tell the potential amount of viewers you can reach with a particular keyword or phrase.

However, note that there is yet to be an actual benchmark on what a healthy monthly search volume is, especially since niches and industries vary in size.

For instance, if you are working on a popular niche like fashion or beauty, you can see that the search volume is high. But a very specific niche, like bass fishing, may have a lower volume. Be sure to only compare different keywords in your niche.

Lastly, use long-tail keyword phrases instead of just keywords. Almost everyone we know uses the internet for answers to just about anything. So, when you optimize your content for longer keyword phrases, you can get their attention easier since it usually focuses on a more specific topic.

Going back to our example on the car parts business, your keyword phrases for tire changing content can include “how long does it take to change tires” or “how often should you change tires.” They also have high search volumes, meaning many people are interested to know the answers to those questions.

3. Create a Topic Using Your Chosen Keywords

Now, let’s go to the fun part. Once you have all the details you need and have collected all the keywords you want to use, it’s time to think of a topic for your evergreen content. To help you out, here are some questions you can ask yourself when coming up with a topic:

  • Will it be relevant to your customers after a year?
  • Will your customer and potential customers always search for this topic?
  • Will it be easy to reshare this topic on various platforms over and over without seeming outdated?

If your answer is all yes, then you found your golden ticket. You can frame this as your working title to see how to structure and position the content and think of any other related topics you might want to touch on along the way.

Important Things to Remember

Creating evergreen content may sound easy, but there are other things you need to consider to make it successful. You already have a topic in mind, and that’s a good start. However, there are other things that you need to remember when you’re putting it all together.

1. Make It Informative

Always remember quality over quantity. While longer blog posts can rank higher in search engines, it’s better to provide value to your audience than the search algorithm. As an example, if you were writing about the benefits of Google My Business software or social media marketing trends, your articles should still give some value to the readers. With this in mind, ensure your evergreen content is as long as it’s supposed to be—no fluff to achieve a particular word count. Your content should be made for humans, not robots.

Tiktok Ad Set Up Guide Free

DOWNLOAD: The TikTok Ad Set Up Guide. Learn how to quickly create 16 TikTok ads in the least amount of time possible. Click Here

2. Don’t Forget the Layout

Even if you have the best evergreen content piece in the world, if it doesn’t have good design and proper formatting, your efforts are put to waste. So, apart from ensuring your content can rank, you must deliver a memorable reading experience.

This will not only build customer trust but can potentially convert visitors to customers.

When it comes to optimizing the layout and design of your evergreen content, make sure that there is enough white space between the paragraphs. This makes it easy for your visitors to consume the content.

Next, don’t put a lot of distractions. If you do, they won’t be able to finish the content and will jump to another page. Lastly, make it visually appealing. You can add relevant images or videos to break walls of text. Just make sure you can get your readers at the end.

Remember, the longer readers stay and read your content, the higher your chances of turning them into qualified leads.

3. Regularly Update Your Content

Just because evergreen content can last for years doesn’t necessarily mean you can just upload it and never touch it again. Since your main goal is to provide information to your customers, you must ensure that those details are updated every time. It doesn’t have to be every week; you just need to ensure that the information is not outdated so you won’t mislead your readers.

You can also use this opportunity to link other website pages from your content. This can help with your overall SEO strategy, especially if the piece generates a lot of traffic. If you have already created evergreen content, here are some things you need to update:

  • Data and Statistics
  • Quotes
  • Screenshots and images
  • Titles
  • Examples

I don’t recommend adding dates to your evergreen content URL if you can help it(e.g., Best-Apps-for-Video-Editing-2022), as it will quickly become outdated, and your SEO efforts are going to waste every time you change the link.

But if you’re using one in your title, make sure you update that too. You can also use the help of AI writing software to add extra paragraphs and beef up your post a little.

4. Write for Beginners

This may sound counterintuitive, especially if you want to assert your expertise in the field. But people who search for broad topics are usually not experts, so ensure your content can cater to a large audience.

That’s why it’s best to avoid technical jargon. It’s okay to mention it when necessary. Just ensure that you give a brief explanation about it. At the same time, keep your tone conversational. Read your sentences and paragraphs to check if they sound natural.

5. Promote Your Content

Now that you’ve put together your winning content piece, it’s time to put it to work to reach your audience—it just doesn’t automatically give results after you publish it. This is where content promotion comes in.

There are various ways you can promote your content. This can be through social media sites like Facebook or Instagram. You can also promote it via email, especially if you have an extensive mailing list. Or even via guest blogging.

Since this can stand the test of time, you can promote it repeatedly, which is perfect for times when you just don’t know what to post.

Conclusion

Before evergreen content can give you the desired results, you must ensure that Google knows your site exists. If not, then you should index your site first and then work from there.

Creating evergreen content that lasts requires a lot of work, and you should be willing to do it. By following these practices I’ve shared, you can make an effective piece to drive ongoing growth to your site.

The results may not be quick, but if you do things right, you’ll reap the benefits far longer than you think.


Source link

Continue Reading

DON'T MISS ANY IMPORTANT NEWS!
Subscribe To our Newsletter
We promise not to spam you. Unsubscribe at any time.
Invalid email address

Trending

en_USEnglish