django-bandits

This package enables optimization of your Django site via multi-armed bandits.

This package extends Django-Waffle by wrapping the feature flipper functions with bandit algorithms. This allows automatic testing and optimization as users interact with your site.

Installation

pip install django-bandits

or

poetry add django-bandits

Additionally, you can install it directly from GitHub:

Pip

pip install git+https://github.com/hubbs5/django-bandits.git

Poetry

poetry add git+https://github.com/hubbs5/django-bandits.git

Django Settings

Go to your settings.py file and add waffle to your installed apps list:

INSTALLED_APPS = [
    ...
    "waffle",
    "django_bandits",
    ...
]

Additionally, add django_bandits.middleware.UserActivityMiddleware to MIDDLEWARE after CSRF and authentication middleware, e.g.:

MIDDLEWARE = [
    ...
    "django.middleware.common.CommonMiddleware",
    "django.middleware.csrf.CsrfViewMiddleware",
    "django.contrib.auth.middleware.AuthenticationMiddleware",
    "django_bandits.middleware.UserActivityMiddleware", # It goes here!
    ...
]

Update settings.py to exclude any particular URL from conversion tracking, e.g.:

EXCLUDE_FROM_TRACKING = [
    ADMIN_URL,
]

Additionally, you can add a REGEX option to filter URLs. Helps to ensure you aren't picking up bot traffic that's trying to break into a Wordpress site or something of the sort.

EXCLUDE_FROM_TRACKING_REGEX = r"\.php|wordpress|\/wp-"

Finally add the following line to your settings.py file to ensure the bandit model is overriding Waffle's Flags.

WAFFLE_FLAG_MODEL = "django_bandits.BanditFlag"

Migrations

Create migrations by running python manage.py makemigrations followed by python manage.py migrate.

Enabling Bandits

With migrations complete, go to the Django site admin page. You should see something like this:

Select Flags under DJANGO_BANDITS and click "Add."

Make a name for the flag - typically the feature you want to test. Set it to test for whatever group you'd like to test it on. Refer to the Waffle documentation for details.

The bandit needs a FLAG URL to be set - this is the URL where the flag will be shown to users and is needed to track conversions and call the bandit to flip the feature when users reach that URL. Enter this as the Source URL.

To track a conversion, then you need to add a Target URL, which is where you want the user to end up.

In the example below, we're tracking the homepage (/) and want to see how many users click on a link to go to the contact page (/contact/).

This will consider any user who views your source URL and target URL to be a conversion during the session and will update the count shown in the image above.

To add a bandit, you'll need to select one of the bandit options from the list below:

Some bandits have customizable parameters (e.g. how frequently a random action is taken such as $\epsilon$). All allow you to set a minimum number of views and confidence interval before a winning version is selected (see more details below).

Now that a bandit is enabled, you need to update your templates or views in as done with Waffle to enable the feature flipping flag.

In a template, it may be something like this:

{% extends "base.html" %}
{% load waffle_tags %}

{% block content %}
{% flag "headline-flag" %}
    <h1>This is my test headline: Waffle Flag is <strong>active</strong></h1>
{% else %}
    <h1>This is my base headline: Waffle Flag is <strong>inactive</strong></h1>
{% endflag %}
    <a href="{% url 'ab_test:target' %}">Click here to see the target page</a>
{% endblock %}

Where "headline-flag" is the name of the flag you defined in the admin page.

Performance Tracking

The bandits will automatically select a winning option when the given criteria are met. When setting up a bandit, you can select the SIGNIFICANCE LEVEL(default is 0.05) and the MIN VIEWS (default is 100).

After the MIN VIEWS threshold is reached, the model will perform a two-sided t-test to determine if there's significant difference in the two options (i.e. p-value < SIGNIFICANCE LEVEL). If this criteria is met, the bandit will then default to the winning option for all future visits to avoid potential conversion losses from testing.

As a user, you can also view the conversion rate and the upper and lower bounds of the confidence intervals at any time within the admin page.