[Guide]: AB Testing for eCommerce

This guide details how to maximize AB testing results.  While you can use any AB testing framework with Barilliance, we've found a few best practices that generate the best customer experiences.


We will start from the beginning.  First we define testing concepts such as AB testing, muti-armed bandit, and audience sizes.  Second, we will detail the benefits AB testing promises. Finally, we will break down AB testing best practices, and illustrate with examples.

If you'd like to skip straight to the AB testing best practices, click here.

What is AB testing? (in an eCommerce environment)

AB testing is a specific type of controlled experiment.  AB tests are used to determine the best option out of a possible range.  In an eCommerce context, this could be determining which offers to show, what products to recommend, or even which product images or layouts work best for a given success criteria.


Like other multi-arm bandit tests, there are two distinct parts.  First, there is an exploration phase where multiple options are considered.  Once a winner is determined, the second phase of exploitation starts. In this phase, the winner is chosen and the loser is discarded.

Properly constructed AB tests are comprised of the following parts:

  • Hypothesis - a testable statement predicting what will happen in the test. 
  • Independent Variable - The independent variable is what we are changing to see what effect it has. In the context of eCommerce AB tests, it might be changing which bundles we offer, or different incentives (such as free shipping vs discounts).
  • Dependent Variable (Success Criteria) - The dependent variable is how we are measuring success. For an eCommerce AB test the dependent variable is often a KPI you are trying to improve such as conversion rate, increasing average order value, or repeat orders.
  • Test and Control Groups - Test and control groups are the visitors, prospects, or customers that experience the distinct independent variables and whose actions represent the dependent variable.

“AB tests are used to determine the best option out of a possible range. ” 

While defining an AB test is nice, it is helpful to give a few illustrative examples for how AB testing can benefit eCommerce stores.

AB Testing Benefits: How To Use Testing to Improve eCommerce Stores

Recall that AB tests determine the best option out of a possible range. With this in mind, we can quickly go through a number of AB testing opportunities.

We could test what offer resonates best with returning customers. 

We could test which products recommendations convert the best for a given product sku. 

Customer experiences can also be tested. For example, we could test to see if a welcome pop-up performs better than a message bar. 

AB Testing Benefit 1: Improve ROI from Current Traffic

As a fundamental tool in conversion rate optimization, testing maximizes your ability to monetize existing traffic. 


Using AB testing, you can: 

  • Directly multiply profit
  • Lower your cost per customer acquisition
  • Unlock channels for lead generation
  • Increase affiliate benefits & pay without impacting bottom line

2016 homepage

2018 homepage

AB Testing Benefit 2: Improve customer experiences

As a tool for personalization software, testing unlocks data informed improvements to your customer experiences.

While Barilliance utilizes machine learning to automatically optimize personalized experiences, you also have the choice to override machine based experiences with rule based variants.

This is especially helpful when you have upcoming events, holidays, new products, categories, or campaigns. 

In these instances, it is important to leverage testing to get the best results.

AB Testing Benefit 3: Improve off-site customer interactions

Finally, AB testing can, and should, be applied to all customer experiences.  To effectively execute omnichannel strategies, testing capabilities are absolutely necessary.

A few off site AB testing benefits include

Which AB Tests should I perform first?

Every business has constrained resources.

In testing, the quality and number of tests you can run will be limited by:

  • Audience Size - Audience size will determine how many tests you can run and still arrive at conclusive results.
  • Time - You must balance AB testing efforts with ongoing responsibilities such as campaign building, holiday preparation, and other operations.
  • Skillsets - likewise, AB testing requires various skillsets which may be limited on your team. These include design, copywriting, and product sourcing.
  • Funding- Ultimately, all of the other constraints are further limited by funding considerations.

Given these constraints, the next question is what tests should you perform? 


While your business context will be unique, here are a few guideposts to help you identify the best AB tests to run. 

1. High traffic touch points

What parts of your customer experience do the most customers experience?

For example, if one product category accounts for 75% of sales, you should prioritize AB tests in that area first. On the other hand, if other categories currently receive marginal traffic, wait to optimize these parts until you've secured wins in higher trafficked areas. 

2. Important customer segments 

Second, what customer segments are important to your business? 

We recommend using RFM Analysis to determine the most important customer segments for your business

3. Identify bottlenecks

Finally, consider what bottlenecks exist in your business. 


Bottlenecks are places where every customer must bass through. For most eCommerce stores, this includes your checkout process, product pages, and possibly ads. 

AB Testing Best Practices: How to conduct an AB Test the right way

There are many ways to be successful with eCommerce optimization. However, we've found these guidelines lead to more consistent, faster results.

Define your success criteria

Ask yourself, "What will make this test successful?"  You should identify the single best KPI to evaluate a test.

Typically, your evaluation metric will be a composite metric, like return on marketing investment. It should be predictive of long-term outcomes.

The Harvard Business Review has an excellent writeup detailing a number of challenges Bing faced in determining which metric to prioritize.

While at first it might seem that revenue generated would be a good metric, they found that they could manipulate revenue by adding more ads. Unfortunately, the metric didn't fully capture the costs of these improvements in revenue - namely a worse customer experience and ultimately less people using Bing.

The team came up with an alternative. They wanted to focus on the customer experience first, and so the primary metric was to minimize the number of queries needed to complete any one session, yet maximize the number of sessions per user.

Balance explore vs optimization - consider big levers.

There is a constant tension between exploring new possibilities and exploiting previous findings. 

With AB testing, it can be tempting to shy away from big redesigns and instead focus only on smaller optimizations. However, the huge breakthroughs are more likely to come from big levers -  large changes in your offer or design. 

Make sure you leave room for these larger experiments. 

Optimize your sample sizes

One common misconception is that your control and test audience sizes need to be the same. 

The truth is, your control represents the winner of all previous AB tests. It is a tried and true effective experience that results in sales. 

Because of this, it is most common to dedicate more traffic to the control group, while looking for improvements with a smaller percentage of traffic.

On the other hand, if you want to increase the throughput of tests you can conduct, you can increase the traffic dedicated to test groups.

Don't trust averages. Segment your results

Your AB tests will have mixed results. It will positively impact some visitors, and negatively impact others.

If you just trust the aggregate numbers, you run the risk of alienating whole customer segments.

The answer is to break down the results by customer segment. If you don't have established segments defined in your personalization tool, I suggest running through a RFM Analysis to identify high priority segments.

Ultimately, you want to isolate a defined customer group that the experiment works on, and then make the winner show to that segment and that segment only.

Automate Your RFM Analysis:

Barilliance connects your offline and online customer data. You can define as many segments as you like, automatically enroll customers based on their actions, and trigger any number of marketing campaigns. Learn more here.

Successful AB Test Examples

We have a running collection of successful eCommerce AB test examples here

In that guide, we break down in detail the example tests. Here, we will briefly showcase tests from larger eCommerce stores. 

How MOO uses AB testing to improve conversions

Moo is a leading eCommerce store in the business card and stationary space. 


Above is an annotated screenshot of their current home page above the fold. Below we have the same page, just one year ago. 

Moo 2019

Comparing the two pages, there are six major improvements the team at Moo have implemented. For the full break down on what these improvements are and why they work, click here.

Next Steps...

To properly perform AB testing, you need a personalization partner. 

We wrote a detailed guide on how to select a personalization partner for your eCommerce store here.

However, in short, you want a technology that is able to connect your data in one place, identify profitable segments, and provide personalized experiences and AB tests targeted towards those segments.

If you'd like to see why hundreds of enterprise eCommerce stores have chosen Barilliance as their AB testing solution, request a demo here.