Skip to main content

Simple WSGI A/B testing.

Usage

  1. Define an experiment:

        from swab import Swab
        s = Swab('/tmp/.swab-test-data')
        s.addexperiment('button-size', ['default', 'larger'], 'order-completed')
    
    Here we have an experiment called 'button-text', for testing two sizes of
    button on a form.
    
    The two variants the experment are called 'small' and 'large'. The first
    listed is always taken as the control variant, and results will be
    presented in terms of how much better or worse the other variants have
    performed compared with the control
    
    The goal associated with the test is called 'order-completed'.
  2. Take a WSGI application:

    from swab import record_goal, show_variant
    
    def get_button_style(environ):
        if show_variant('button-size', environ) == 'small':
            return 'font-size: 100%'
        else:
            return 'font-size: 150%'
    
    def myapp(environ, start_response):
    
        if environ['REQUEST_METHOD'] == 'GET':
            start_response('200 OK', [])
            return ['''
                <html>
                    <body>
                        <form method="POST">
                            <button style='%s' type="submit">Complete your order</button>
                        </form>
                    </body>
                </html>''' % (get_button_style(environ),)
            ]
    
        if environ['REQUEST_METHOD'] == 'POST':
            swab.record_goal('order-completed', environ)
  3. Wrap it in the swab middleware:

    app = s.middleware(myapp)

4. Let it run, and after a while browse to http://<server>/swab/results to see which variant is winning.

Testing

To test your competing pages, start Swab in debug mode:

swab = Swab('/tmp/.swab-test-data', debug=True)

Then append ‘?swab.<experiment-name>=<variant-name>’ to URLs to force one variant or another to be shown.

Disable debug mode in production to avoid people messing about with your figures.

Basic design

Each visitor is assigned an identity which is persisted by means of a cookie. The identity is a base64 encoded randomly generated byte sequence. This identity is used as a seed for a RNG, which is used to switch visitors into test groups.

Every time a test is shown (triggered by the show_variant function), a line is entered into a file at <datadir>/<experiment>/<variant>/__all__

Every time a goal is recorded (triggered by the record_goal function), a line is entered into a file at <datadir>/<experiment>/<variant>/<goal>

Each log line has the format <timestamp>:<identity>\n.

No file locking is used: it is assumed that this will be run on a system where each line is smaller than the fs blocksize, allowing us to avoid this overhead. The lines may become interleaved, but there should be no risk of corruption even with multiple simultaneous writes. See http://www.perlmonks.org/?node_id=486488 for a discussion of the issue.

Statistical analysis

The further reading section contains links to the various articles that helped me implement the algorithms and statistical formulae used in Swab. However I don’t have a background in maths or stats and I would welcome contributions or corrections from anyone with a deeper understanding of these tests.

TODO

  • Allow a set percentage of users to be tipped into each test version

  • Filter search engine spiders out from tests

  • Investigate multivariate testing

  • Allow users to be excluded from tests after being shown a test variant (this would js-based exclusion of bots, eg http://www.kalzumeus.com/2010/06/07/detecting-bots-in-javascrip/)

Further reading

  1. 2010 Oliver Cope. For licensing information please see LICENSE.txt

Version 0.1.2

  • Minor bugfixes

Version 0.1.1

  • Bugfix for ZeroDivisionErrors when no data has been collected

Version 0.1

  • Initial release

Metadata

Release files for swab 0.1.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for swab 0.1.2
File Size Uploaded Provenance Actions
swab-0.1.2.tar.gz 13.1 kB Without provenance Details for swab-0.1.2.tar.gz

Release files / swab-0.1.2.tar.gz

Download URL swab-0.1.2.tar.gz
Size 13.1 kB
Tags Source
SHA-256 checksum
How to use checksums
e073e46098fa81d8e252b7cb2f82efdad510bb6b18fa56f193c241910689f010
BLAKE2b-256 checksum
How to use checksums
a13f34ef088f5ce4aa02817a628078b531f76614105f488c45b396048d3b9a07
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Security for swab 0.1.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page