Janus

Janus is a simple and easy A/B Test Calculator that allows you to run A/B tests and get reports in a few lines of code.

It's developed mainly for website experiments. While many calculators only account for conversion analysis and consider only two variants, Janus allows multivariant tests and measure Conversion, Averge Ticket and Average Revenue Per User (ARPU) over variants, i.e, typical metrics for tests in marketplaces.

The engine name is an analogy to Janus, the god of changes and transitions.

Installation

Janus is distributed in pypi. You can install it by using:

pip install janus-web-ab-testing

You can now run a quick example of a A/B test:

from janus.variant import Variant
from janus.experiment import WebsiteExperiment

variant_A = Variant(
    name="A",
    impressions=1000, 
    conversions=100, 
    revenue=10000
)
variant_B = Variant(
    name="B",
    impressions=1000, 
    conversions=120, 
    revenue=9000
)
variants = [variant_A, variant_B]

experiment = WebsiteExperiment(variants, baseline_variant="A")
experiment.run()
experiment.print_reports()

And you should see:

---------------- Summary ----------------
+-------------+-------------+------------+
|             |           A |          B |
+=============+=============+============+
| impressions |        1000 |       1000 |
+-------------+-------------+------------+
| conversions |         100 |        120 |
+-------------+-------------+------------+
|     revenue | R$10,000.00 | R$9,000.00 |
+-------------+-------------+------------+
|  conversion |         0.1 |       0.12 |
+-------------+-------------+------------+
|  avg_ticket |    R$100.00 |    R$75.00 |
+-------------+-------------+------------+
|        arpu |     R$10.00 |     R$9.00 |
+-------------+-------------+------------+
----------- Conversion Stats -----------
+-----------------+--------+--------+
|                 |      A |      B |
+=================+========+========+
|   expected_loss | 0.0206 | 0.0005 |
+-----------------+--------+--------+
| prob_being_best | 0.0786 | 0.9214 |
+-----------------+--------+--------+
|            lift |    0.0 |    0.2 |
+-----------------+--------+--------+
-------------- ARPU Stats --------------
+-----------------+--------+--------+
|                 |      A |      B |
+=================+========+========+
|   expected_loss | R$0.14 | R$1.15 |
+-----------------+--------+--------+
| prob_being_best | 0.7936 | 0.2064 |
+-----------------+--------+--------+
|            lift |    0.0 |   -0.1 |
+-----------------+--------+--------+

See more in the quickstart notebook. The Average Ticket Statistics is still missing, but will be added soon (it's usually modelled with exponential distribution with gamma as prior)

The Janus Website

My plan is to build a website to showcase the Janus library and to provide a simple interface to run A/B tests. However, only the library is almost ready, and the next steps would be:

• build a backend with FastAPI using the library
• build a frontend (JavaScript maybe)
• deploy the website

Warning The distributed package is from this branch, which will be merged to main in the first release of this project. The main branch uses another previous approach for the calculations (yet correct), and has a streamlit app to showcase the library. In this branch, however, we'll change the whole project towards not only a library, but also for a full stack application, with frontend and backend.

References

• What is A/B Testing
• VWO paper about bayesian A/B testing
• Agile A/B testing with Bayesian Statistics and Python
• Understanding Bayesian A/B testing (using baseball statistics)
• It’s time to re-think A/B testing
• Conjugate Priors
• Curso de Teste A/B Bayesiano do Lazy Programmer
• Binomial Distributions
• Bayes theorem
• The quick proof of Bayes Theorem