Introducing Enhanced Rule Variations, Multivariate and A/B Testing

Multivariate testing produces results. 


You can quickly iterate approaches, rapidly improving revenues, conversion rates, average order value, engagement, and more. 


While clients have always had the ability to perform a/b testing, Barilliance has released improved multivariate testing capabilities for our personalization suite. 


These updates can be used across our onsite personalization, product recommendation, and Live! products. 


Now, you can now test as many variations as you would like. Further, you can opt to have our machine learning capabilities automatically optimize traffic distribution to maximize results.

Introducing Rule Variations for Personalization, Product Recommendations, and Live!

Rule variations is a breakthrough update that allows easy multivariate testing.


While variations can be used across our suite, this post will walk through an onsite personalization example. 


In Barilliance, personalization capabilities are defined by rules.


When you create a rule, you are able to define a number of characteristics.

Now, once you complete a rule you have the option to add a variation.


To do so, simply click the "add variation" button located in the top right of the view. 

Just like when you create a rule, you can select any action for the variation. You also have complete control over the design. 


Below is a screenshot of the types of personalization actions you can select. 

Rule variations give you unlimited control on what to test. 


You can compare different offers while keeping the action constant. For example, comparing a popup that offers free shipping to a popup that offers a limited time bundle. 


Alternatively, you can test across action types. 

You could see how a traditional popup performs compared to a more aggressive screen overlay. 


Additionally, you can test beyond the actual content being displayed. You can target different segments, place you actions in different locations, or test different time slots. 


In other words, you can test all kinds of dynamic content simultaneously, adding as many variations as you would like. 

Optimize Variations with Testing & Control Groups

Creating variations is not enough to produce results. 


You need to be able to easily compare and optimize variations. We've added an additional tab when you create a personalization rule called "Testing & Optimization" as shown above. 

Creating Control Groups and Allocating Traffic

When you click the "Testing & Optimization" tab you will be presented with an easy to use traffic distribution bar and a few other options.

As you can see, you are able to control how much traffic each variation receives. 


You can divert any percentage of traffic to the original action, any of the variations you've created, or the control group.

Note that Barilliance automatically creates a control group for you.  All you need to do is allocate some traffic to the control group by adjusting the bar, and the control will be generated automatically. 


The control group is simply visitors who match the segment your rule is targeting. They are not shown the rule, or any variation you created, allowing you to see the direct impact of implementing the rule compared to doing nothing.

 

You can readjust the traffic weight at any time.

Setting Up Rule Goals and Machine Learning Optimization

Personalization software tools can be used to increase engagement, revenues, or any number of benefits. 


The truth is, not every rule will have the same criteria of success. Because of this, we’ve built in a number of ways to determine if a rule is working.

With Barilliance, you can optimize for any of the following criteria:

  • Conversion Rate -  Defined by the number of successful purchases divided by the total number of site sessions.
  • Revenue - The absolute revenue generated.
  • AOV - The average order value, or the revenue per successful transaction.
  • Time on Site - How long visitors stay on your site. Note that this is measured from the time the Barilliance pixel is first loaded. 
  • Add to Cart - The percent of visitors who add an item to their cart divided by the total number of site sessions.
  • Bounce Rate - The percentage of sessions in which the user leave the session immediately after the rule triggered for the first time.
  • URL Goal - The number of sessions that reach a defined URL divided by the number of total sessions. You can define a specific URL, or a substring. When using a substring, anytime a visitor visits a page with the substring in the URL will count as a success.  

Automatic, Machine Learning Optimization

While you can manually weight traffic, you also have the option to use machine learning to find the most effective variation.


To do so, simply check mark "Automatic Optimization". Barilliance will then automatically divert more traffic to the best variations. Fit is defined by the rule goals you specify above. 

Viewing Performance in Barilliance

Managing variations and rules is simplified in Barilliance's internal dashboard. 


You can view realtime performance of all rules in a single view, and drill down for more detail.


In the dashboard, you can click on a rule name. When you do, a breakdown of the rule and it's variations are laid out in table format. You can compare how they effect bounce rate, click through rate, conversions, revenue, and goal completion. 


We also graphically represent the rule and variation to quickly see what is working and what is not. 

The above example is of a simple pop-up offering a discount.


As you can see in the top left, the metric chosen as the goal is Conversion Rate. The dashboard dynamically updates to illustrate how the conversion rates compare between the personalization rule and control group.


We can see that the personalization rule has a significantly higher conversion rate, 20% higher than the control group.

You can also easily compare revenue and AOV numbers in the same pane.

In this case, while the conversion rate went up by 20%, the AOV decreased by 7%. 

Next Steps...

Multivalent testing can produce amazing results. 


We've made it easy to create distinct variations, and automatically optimize your personalization strategy to multiply revenues and engagement. 


If you'd like to see a demo of Barilliance's personalization suite in action, click here