Guide to Analyzing A/B Testing Results

What is A/B testing?

A/B testing (aka split testing) is comparing two slightly different versions of the same element (e.g., an email CTA button) to determine which one is more effective at getting the results you want. You can aim to improve the conversion rate, average order value, or any other metric you think is dipping and that’s affecting your business goals.

In A/B testing, you tinker with just one thing: a subject line, the placement of a CTA button, sending time, etc. If you change several elements at a time and look at the difference, it’s called multivariate testing.

In short, you create a version you think might perform best, roll it out to half of your target audience (or to another significant number of users), and measure it against the version you currently have. We’ll look at the process of A/B testing in detail below — for now, let’s have a quick look at potential elements you can put to an A/B test.

What you can A/B test

For the purposes of this guide, we’ll talk about marketing elements you can test — as opposed to, say, usability elements. A/B testing in marketing deals with user behavior and preferences and directly affects your sales. Some of the questions A/B testing in marketing can answer are: Do people respond better to shorter subject lines? Does changing the color of a CTA button make it more noticeable?

For A/B testing in marketing, there are three broad areas you can experiment with.

Website pages

Pretty much anything you see on your website is fair game. You can test different headings, CTA placement, design elements, and content itself. Is a different heading more intriguing and makes the reader curious to scroll further? Should you add user-generated content that shows your product in action? Can a higher CTA placement improve its discoverability? Do shorter/friendlier sentences make your brand more relatable?

Below we have an example of a website homepage where we are sure some elements can be A/B tested to see whether there’s room for improvement.

There are several elements to A/B test on this landing page, as shown in the screenshot above: 

• CTA color. There are off-pink and white CTAs, so there can be a variation of this page with a unified color palette. 
• Content. This text can be simplified for testing to see which variant performs best.
• Partners’ block. Placing this block higher may bring more conversions, which is a good theory to put to split-test. 

Landing pages

Landing pages have everything your customer needs to know on a single page, with a simple goal in mind — usually to convince page visitors that your product/service can address their issues the best.

So why not run several versions of a landing page? Much of what we wrote about testing website pages applies here as well: different content formats, design elements, CTAs, headings can lead to different results.

This Calm landing page is great, but it could be even more effective. Here are two elements worth exploring via an A/B test:

• CTA colors. Do green CTAs really work better than white/purple ones? 
• Social proof block. Perhaps moving the reviews higher up might have a positive impact. 

Social media pages

The way your business handles social media presence makes a difference as well. You can experiment with content formats, publishing timing, promotional posts frequency, and graphic elements accompanying your posts, such as photos, videos, banners, even memes. And of course you should stick to a consistent tone of voice across your channels, so that your brand is instantly recognizable.

Email marketing campaigns

Unlike web and social you can only work with your existing customer base, of course, not any prospecting subscribers. You already have a subscriber base, so you are just checking out what resonates with them.

The most popular elements to split-test in email marketing are subject lines, preview text, CTA placement and copy, and sending times. You should also mind the different audience segments you have. It’s unlikely you send the same emails to everyone, so work within one segment when A/B testing for better results.

If you are looking for an A/B testing email tool, we have your back. Split-test subject lines, preview text, images, text formatting, CTAs, and much more with our email builder.

As an example, here are some elements that can be split-tested in this Semrush email:

• Subject line. It simply reads “Super Bowl Ad Spend.” Something more intriguing could increase the open rate.
• CTA. This email could have had more of an impact with a simple CTA — but there’s no CTA at all.

Advertising campaigns

Paid advertising remains one of the most effective methods to bring customers in. You can have sponsored links in organic search results, rely on social media to spread the word about your brand, and, of course, use offline ads to draw attention.

Your offline efforts would be harder to measure, but most social media channels allow you to create several variations of your ads and present different versions to different people, giving you a chance to test out different design and copy elements. There are ways to A/B test organic search result ads as well.

How to carry out an A/B test

Once you’ve decided which part of your marketing approach to split-test (social media, email, etc.), it’s time to actually carry out the test. 

Decide which metric you want to test

If you are in B2C, you might want to boost your sales, average order value, the conversion rate, or maybe the number of returning customers if you offer a subscription-based service. 

B2B companies might focus more on the number of leads generated because once other companies show their interest in your solution, purchases and subscriptions will follow — if the lead is nurtured correctly. It’s unlikely that a company representative browsing your website will hit “Choose a plan” straight away.

Brainstorm ideas and formulate your hypothesis

Once you’ve chosen the metric you want to test, the next important step is planning what to do. Will you alter a heading, subject line, design element, or CTA?

Review where your visitors or leads fall off and brainstorm ideas on why that might be and which change can turn things around. After you have a list of suggestions, choose one that has the best potential and make a specific prediction. For example, “Moving a CTA one block higher will increase the click rate by 10%.” That is both a hypothesis and how you are going to measure success.

Implement your hypothesis and create two variations

Technically, it’s just one variation. You already have another one in place — the underperforming one. Your existing one is called “the control,” and the one you think should outperform it is called “the challenger.” You are going to stack them up one against the other and see what happens.

Run the A/B test

Roll out your “challenger” version and track how it performs compared to “control.” There are two things to keep in mind here: the audience size and the split test duration.

You have to run the test on large enough audience segments to get statistically significant results, and to exclude the possibility of a false positive or a false negative. Anything can happen on a small sample size that won’t maintain its statistical significance level on a larger sample size. So, you can split your audience in half: show one half the control variation and the other half the challenger variation.

The test also has to go on long enough. A couple of days is just the same as a small sample size, only you’ll have a short time frame instead of a small audience. The potential for an inconclusive result remains. Aim for at least two weeks when running a split test and consider external factors (such as holidays) that can skew results during that time frame.

Analyzing A/B testing results

Now that you’ve run your A/B test on a large enough audience sample size and for a sufficient time period, it’s time to interpret the results you got. There are three possible ways for A/B testing analysis: comparing the primary metric, comparing the secondary metrics, and comparing how your variations performed with different audience segments. We’ll talk about audience segmentation a bit later, for now let’s focus on primary and secondary key metrics — as well as what exactly you are looking for when comparing them.

Check Probability to be Best and Uplift

Probability to be Best answers the question “Which variation has a higher chance of performing better?” This metric has to be in the region of 90%-95% for the test results to be considered conclusive and for you to roll out the winning variation for all of your customers. 

Disclaimer: It’s probably a bad idea to roll out one variation for everyone, even if the winner is clear. Why? The winning variation might only be winning for a certain audience segment (e.g., only for desktop users), but the bulk of your visitors rely on smartphones. We’ll get into more detail below.

Uplift answers the question “By how much the winning variation outperformed the losing one?” In short, how much better are things going to get once you apply the winning variation across the board? By how much will your conversion rate or AOV grow?

To calculate both of these metrics, you need to use dedicated calculators.

Check your primary metric

That’s why you are doing the test after all — to get better in one particular department. So that’s your most important indicator of success — how much your conversions grew, or how many more leads you’ve generated with another approach.

In the case of CampaignMonitor, their primary goal was conversion rate optimization. They put forward a theory that conversions would grow if the CTA text on their landing page dynamically changed to reflect the users’ search terms. The improved CTA lifted conversions by 31%.

Check the secondary metrics

It’s a good idea even if your challenger variation is on the losing side. For example, your conversion rates might take a dip with a different approach, but your average order value might go up, which, in turn, will make the conversion dip irrelevant.

Conversely, your primary metric (such as click-through rate) might be up, but purchases per user or average order value are down — which affects your business negatively. Analyzing secondary metrics might save you from making a mistake where you base success purely on the primary metric’s performance.

What to keep in mind when analyzing A/B testing results

Assess the winner using different approaches

For example, your winning variation might be the overwhelming favorite for desktop users, but not for mobile users. Or the winning variation works wonders for new visitors but flops among returning ones.

That’s why it’s important to consider factors that might compromise your customers’ experience when doing a side-by-side comparison after an A/B test. As opposed to simply rolling out the winning variation for everyone once you establish the results are conclusive.

Segment A/B testing data based on your audience

This point is a bit more nuanced than the previous one. Instead of just dividing your customers into new and returning or desktop vs mobile, try to dig deeper and work out how different variations perform for different audience segments.

You likely have a deep understanding of your audience already: who responds best to emails, who makes rare purchases with a large AOV, who prefers online shopping to in-store purchases. Think about your audience’s behavioral traits before declaring the winning variation is good for every user in a wider group (e.g. all mobile users).

One of the possible ways to segment your audience is by location. It’s a good place to start, especially if you don’t have more advanced behavioral data just yet. That’s exactly how First Midwest Bank increased their conversion rate 1.5x. The company tested different landing page hero images in different states and found the winning variation for each.

Don’t fret about neutral results

Sometimes you won’t get a clear-cut winner in a split test… and that’s alright! You have still learned more about your customers by doing an A/B test. You have also excluded one possible option of what might be wrong. Now you know for certain that pesky heading was not the problem. Or that coloring your CTA green instead of yellow makes no difference to your click-through rate.

This is exactly what happened with HubSpot when they ran an A/B test on their weekly newsletter. They were looking for ways to improve click-through rates — and wondered whether text alignment could impact it. The standard was center-aligned text — it was their control variation. The challenger variation featured left-aligned text — in the hope it would draw more attention to the CTA.

The result? Click-through rates actually dropped, with only 1 in 4 left-aligned emails outperforming center-aligned ones. But a negative result still taught them something about their audience’s preferences.

Wrapping up

A/B testing allows you to compare two slightly different versions of the same page, email, or social media approach. It is useful when you are trying to figure out why some of your metrics are dipping — or when you simply want to understand how to boost performance.

To carry out an A/B test, you should first decide which metric you want to improve, think about how it can be done, create a variation with one notable difference from your current version, and then run the test on a large enough sample size and for a sufficiently long time period.

Then it’s time to analyze A/B testing results by looking at the primary and secondary metrics, checking the Probability to be Best and Uplift, and examining your target audience. Don’t forget that the winning and losing variations can be deceptive because of secondary metrics and different audience segments. And don’t lose your cool if your analysis provides no insights on the surface: you’ve still gleaned valuable info about your customers.