A/B Tests

Quick Intro

A/B testing involves creating two variations of a digital asset to identify which version resonates better with users. These assets can include landing pages, display ads, marketing emails, and social media posts. During an A/B test, your audience is divided automatically, with half receiving “version A” and the other half “version B.” The effectiveness of each version is measured against conversion rate objectives, such as the percentage of users who click on a link, complete a form, or make a purchase.

The concept of A/B testing isn’t new with the rise of digital marketing. In the past, direct mail was the champion of “splitting” or “bucketing” offers to determine which approach worked best. Digital tools have refined this method, allowing for more precise, reliable, and faster results.

If you’re looking to grow your business, it can be challenging to determine which marketing assets truly engage your audience. A/B testing, along with other conversion optimisation techniques, provides an opportunity to experiment, refine your content, enhance customer experiences, and achieve your conversion goals more quickly. This guide will introduce you to the fundamentals of A/B testing, helping you to harness its full potential.

A/B Resting definition: What is A/B testing?

A/B tests, also referred to as split tests, enable you to compare two versions of a given asset to determine which performs better. Put simply, it’s about discovering whether your users prefer version A or version B.

The concept mirrors the scientific method. To understand the impact of changing a single variable, you must create a controlled environment where only that one variable is altered.

Consider the simple experiments you might have conducted in primary school. For instance, if you place two seeds in separate cups of soil, leaving one in a dark closet and the other by a sunny window, you’ll observe differing outcomes. This approach is the essence of A/B testing: isolating a change to see its effect.

The History of A/B testing

In the 1960s, marketers began to recognise how this type of testing could reveal the effectiveness of their advertising efforts. For instance, they explored whether a television commercial or a radio spot would attract more customers, or if direct mail campaigns were more successful using letters versus postcards.

With the rise of the internet in the 1990s, A/B testing transitioned to the digital realm. As digital marketing became more prominent, teams with access to technical resources could test their strategies in real time—and on a much larger scale than ever before.

What does A/B testing entail?

A/B testing uses digital tools to evaluate different components of a marketing campaign. To get started, you’ll need the following:

1. A campaign to test: Whether it’s an email, newsletter, ad, landing page, or another marketing medium, you must have an existing campaign ready to be tested.
2. Elements to test: Review the various elements within your campaign to identify what changes might encourage customers to take action. Be sure to test each element individually to ensure accurate measurement of its impact.
3. Clear goals: Your A/B testing objectives should focus on determining which version delivers better results for your business. Consider which metrics you’ll monitor, such as clicks, sign-ups, or purchases, to gauge success.

How does A/B testing work in the digital age?

At its core, A/B testing in marketing remains fundamentally the same as it has always been. You identify the variable you want to test—such as a blog post with images versus the same post without images. You then randomly show one version to visitors while controlling for all other factors, collecting data on metrics like bounce rates and time spent on the page.

It’s also possible to test multiple variables simultaneously. For instance, if you’re interested in assessing both font type and the use of images, you could create four variations of your blog post:

1. Arial with images
2. Arial without images
3. Times New Roman with images
4. Times New Roman without images

A/B testing software will gather data from experiments like these. Then, a member of your team analyses the results to determine whether implementing changes would be beneficial for the business—and, if so, what adjustments to make.

Why is A/B testing important?

A/B tests provide you with the data needed to maximise your marketing budget. Imagine your boss allocates a budget for driving traffic to your website using Google AdWords. You set up an A/B test to compare the number of clicks on three different article titles. For a week, you run the test, ensuring that at any given time, an equal number of ads are displayed for each title option.

The results from this test reveal which title attracts the most click-throughs. Armed with this data, you can optimise your marketing campaign for a higher return on investment (ROI), rather than relying on guesswork.

Minor Changes, Major Gains

A/B tests allow you to measure the impact of small changes that are relatively inexpensive to implement. Running an AdWords campaign can be costly, so it’s crucial to ensure every element performs optimally.

For instance, you could run A/B tests on your homepage by tweaking the font, text size, menu labels, links, and the positioning of your custom signup form. Testing two or three elements at a time helps you avoid overlapping variables that might skew results.

Once the testing is complete, you might discover that adjusting the last three elements boosts your conversion rate by 6% each. Your web designer can implement these changes in under an hour, potentially increasing your revenue by up to 18%.

Low Risks, High Rewards

A/B testing is not only cost-effective but also time-efficient. By testing just a couple of elements at a time, you quickly gather results and make informed decisions. If the real-world outcomes don’t align with your test results, you can easily revert to the previous version.

Maximising Traffic Value

A/B testing helps optimise your website’s effectiveness, leading to higher conversion rates per visitor. The higher your conversion rate, the less you need to spend on marketing, since more visitors are likely to take action.

Improvements to your website can enhance conversion rates for both paid and organic traffic, ensuring that every visitor has a greater chance of converting into a customer.

What Can You Use A/B Testing For?

When it comes to customer-facing content, A/B testing can be applied to a wide range of areas.

Common targets for A/B testing include:

• Email campaigns
• Individual email messages
• Multimedia marketing strategies
• Paid online advertisements
• Newsletters
• Website design

Within each of these categories, numerous variables can be tested. For instance, if you’re optimising your website design, you could experiment with factors such as:

• Colour schemes
• Page layouts
• Quantity and type of images
• Headings and subheadings
• Product pricing displays
• Special offers
• Call-to-action button designs
• Video emails vs. non-video emails

Almost any style or content element in customer-facing materials can be tested to optimise performance.

How to Conduct A/B Tests

At its core, the A/B testing process is much like the scientific method. To achieve the best results, it’s essential to approach it systematically. Just like in a laboratory experiment, A/B testing begins with selecting what to test. The entire process involves several key steps:

1. Identify a Specific Problem
   Start by pinpointing a particular issue. A vague statement like “not enough conversions” is too broad; there are too many variables that influence whether a visitor becomes a customer or whether an email recipient clicks through to your site. The goal is to identify why your content isn’t converting effectively.Example: Let’s say you work for a women’s clothing retailer that generates many online sales, yet very few of those sales are coming from email campaigns. By reviewing your analytics data, you discover that a significant number of users are opening your emails and reading them, but few actually proceed to make a purchase.
2. Analyse User Data
   While it’s possible to A/B test every aspect of your emails, it’s not practical to do so. Instead, focus on identifying the specific element that might be causing the issue.Example: Since people are opening your emails, your subject lines are clearly effective. Additionally, users are spending time reading the content, so it’s not driving them away. You also know that customers who arrive at your site from other channels tend to convert. This suggests that, although your emails are engaging, there may be an issue with the call-to-action that guides users to your website.
3. Develop a Hypothesis
   Now, narrow down your focus to the specific element you want to test. Define one or two unknowns to start with, and consider how changing that element might solve the problem.Example: You notice that the button leading to your online store is placed at the bottom of the email, below the fold. You hypothesise that moving it to a more prominent position at the top could drive more visitors to your site.
4. Test the Hypothesis
   Create a new version of the email that implements your idea and run an A/B test with your target audience, comparing it to the original version.Example: You design a new version of the email with the call-to-action button positioned above the fold but keep its design unchanged. You decide to run the test for 24 hours and measure the results.
5. Analyse the Results
   Once the test is complete, examine whether the new version led to any meaningful changes. If the results are inconclusive, consider testing another element.Example: The updated email version led to a slight increase in conversions, but your boss is curious if other changes could yield better results. Since you tested only the button’s position, you decide to experiment with placing it in two additional locations.
6. Challenge Your Champion
   In A/B testing terminology, a “champion” is the current best-performing option, while “challengers” are new variations you test against it. Once you have a winning option, you can continue testing it against other challengers to see if a new champion emerges.Example: After testing two versions of a landing page, you’ve identified a champion. However, there’s a third version you’d like to evaluate. This third version becomes the new challenger, and you test it against your original champion to see if it performs better.

After completing all six steps, you can decide whether the improvements are substantial enough to implement permanently. If not, you can continue running A/B tests on other elements, such as adjusting the size of the button or experimenting with different colour schemes to optimise further.

Tips for Successful A/B Testing

To maximise the effectiveness of your A/B tests, keep the following pointers in mind:

Use Representative Samples of Your Audience

To ensure reliable results, it’s crucial to have as similar participant groups as possible. Automated testing tools can help randomise which users see each version of your website.

However, if you send content directly to clients or leads (such as emails), you’ll need to manually create comparable lists. Aim for groups that are equal in size, and—if you have access to demographic data—distribute recipients evenly by factors like gender, age, and location. This minimises the influence of external factors on your test results.

Maximise Your Sample Size

Larger sample sizes lead to more reliable and statistically significant results. Any differences you observe are less likely to be due to chance.

For instance, if you send two versions of an email to groups of just 50 people each, a 5% increase in click-through rates translates to only 5 additional clicks. Such a small difference could easily occur by chance. However, if you run the same test with groups of 500, a 5% increase means 50 additional clicks, making the result much more likely to be genuinely significant.

Avoid Testing Too Many Variables at Once

It can be tempting to change multiple elements in one test, such as font style, button size, text colour, and layout. However, this approach can make your results difficult to interpret.

For example, redesigning a pop-up with several changes simultaneously might make a positive impact. However, you won’t know which specific change caused it. Was it the button size, colour, or text? You can pinpoint what makes a difference by testing one variable at a time. Remember, you can always run additional tests later to evaluate other elements.

Let the Test Run Its Full Course

A/B tests can be tempting to cut short when you see promising results early on. However, stopping a test prematurely can result in incomplete or misleading data. Allow the test to run for the full duration to ensure the results are statistically significant. External factors like time of day or day of the week can affect user behaviour, so completing the test period ensures your data benefits from proper randomisation.

Retest to Confirm Your Findings

Even with the best A/B testing tools, false positives can occur due to the variability in user behaviour. To ensure accuracy, consider rerunning tests with the same parameters, especially if the initial improvement was modest.

Retesting is particularly important if your results show only a slight increase in performance. Conducting occasional retests helps reduce the chance of acting on misleading data, especially if you frequently run A/B tests. Although retesting every experiment may not be feasible, periodic retesting can help validate your findings and reduce errors.

Following these best practices can make your A/B tests more reliable, actionable, and ultimately more valuable for optimising your marketing strategies.