Factorial A/B testing

A/B testing is the practice of making a change to your website and measuring the effects on your conversion rates or other core business metrics. Common solutions prevent you from doing what is most important: running tests for all your ideas and getting results your decision-makers can easily understand. Allow us to bring your A/B testing to the next level with virtually unlimited simultaneously running tests and actually interpretable results.

Business case

You are an eCommerce company with a large customer base and know the value of A/B testing. You perform A/B testing in the usual way, by allocating your traffic to only a few tests running in parallel but want to test dozens of ideas to win fast and fail fast. The time you need to wait for results is way too long, and then you get a statement of statistical non/significance, but that doesn’t really tell you how much you stand to gain or lose and with what probability.

Our solution

Our solution allows you to run each test on your full traffic and get results much faster. We help you set up proper randomisation and allocation of users into tests. You can run dozens of overlapping tests without having to worry about them influencing each other. You will get easily interpretable results, for example, “Variant B has a 93% chance to improve revenue per user, and the most likely uplift is 3 %. Most likely gain is 400k per month, and risked loss is 10k per month.” Such results can be produced thanks to computing a so-called posterior distribution of all metrics, using a Bayesian or bootstrapping approach. In addition, we can process gigabytes of your data thanks to distributed computing by Apache Spark.

Benefits

Results can be obtained faster thanks to running each test on 100% of your traffic.
You can run an “unlimited” number of concurrent tests.
Understandable outputs, supplying not only the expected value but also the probability of positive effect.

Reference

The A/B test evaluation solution from DataSentics is the definitive tool to assess product development. Thanks to smarter design and evaluation of A/B tests we can conduct twice as many of them compared to our previous solution. The outputs are much more interpretable from the business perspective: We know how much we can gain or risk losing and with what probability. The tool is an absolute necessity for our efforts to have a data-driven product team and we are extremely reliant on its results.