UnittestRandGenState 0.1.4

Smart random-generation state persistence for unittest

====================
UnittestRandGenState
====================


------------
Introduction
------------

Randomized unit tests are great. The inputs fed to your algorithm are not consistent across executions, and so each time the tests run successfully, your confidence in the algorithm grows. Unfortunately, when a test fails, this inconsistency makes it hard to duplicate the input until the problem is solved.

This library provides a simple metaclass for dropping into `unittest.TestCase`_ that addresses this. As long as a test fails, it will continue making the same random choices (via `random`_ or `numpy.random`_) each execution. Any test that passed the previous execution, will get a fresh set of random choices.

.. _`unittest.TestCase`: http://docs.python.org/3/library/unittest.html#test-cases
.. _`random`: http://docs.python.org/3/library/random.html
.. _`numpy.random`: http://docs.scipy.org/doc/numpy-1.7.0/reference/routines.random.html


-------
Example
-------

Consider the following simplified example of a test case with three functions testing some algorithms. The first and third algorithms are OK, but the second has a bug when given a number that divides by 20 (the simplified example asserts this directly).


::

import random


class TestMyAlgs(unittest.TestCase):
# This "algorithm" is OK.
def test_ok(self):
i = random.randint(100, 200)
self.assertLess(i, i + 1)

# This "algorithm" fails for inputs that divide by 20.
def test_bad(self):
i = random.randint(100, 200)
self.assertNotEqual(i % 20, 0, i)

# This "algorithm" is OK too.
def test_another_ok(self):
i = random.randint(100, 200)
self.assertLess(i - 1, i)

The second function will fail approximately 5% of the executions. When it does so, it might be hard (especially in a real-life case) to reproduce the problem, and eventually solve it.

Using this library, a simple change makes things easier. Simply use
``unittest_rand_gen_state.Saver`` as the metaclass:

::

import random

import unittest_rand_gen_state


class TestMyAlgs(unittest.TestCase, metaclass = unittest_rand_gen_state.Saver):
...


or, using an older Python version:


::

import random

import unittest_rand_gen_state


class TestMyAlgs(unittest.TestCase):
__metaclass__ = unittest_rand_gen_state.Saver
...

Nothing else needs to be changed.

Once ``TestMyAlgs.test_bad`` first fails in some execution, then it will consistently fail in following executions. That is, say it fails the first time with
::

======================================================================
FAIL: test_bad (__main__.TestMyAlgs)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/unittest_rand_gen_state/__init__.py", line 75, in wrapped
fn(*args, **kwargs)
File "__init__.py", line 21, in test_bad
self.assertNotEqual(i % 20, 0, i)
AssertionError: 140

----------------------------------------------------------------------

then every following execution will fail exactly the same way, until it is fixed.


The two other tests, meanwhile, will run afresh all the while.


---------------------------------------
Download, Installation, And Bugtracking
---------------------------------------

* The package is at PyPI_.

.. _PyPI: http://pypi.python.org/pypi/UnittestRandGenState

* The usual setup for Python libraries is used. Type:

``$ pip install UnittestRandGenState``

or

``$ sudo pip install UnittestRandGenState``

* Bugtracking is on `Google Code`_.

.. _`Google Code`: http://code.google.com/p/unittest-rand-state-bugtracker/issues/list?can=1&q=


---------
Extending
---------

The metaclass supplied by this library deals with the states of `random`_ and `numpy.random`_, which should be sufficient for almost all tests. However, the library may be extended by dropping as a metaclass, a subclass of ``unittest_rand_gen_state.Saver`` supplying three class methods:

* ``reset_state``: A method specifying the meaning of resetting the random-generation state (e.g., ``random.seed``)
* ``get_state``: A method specifying the query to call for getting the current random-generation state (e.g., ``random.getstate``)
* ``set_state``: A method specifying the function to call to restore the random-generation state using the value returned by ``get_state`` (e.g., ``random.setstate``)

E.g., suppose a random-generation library, ``groovy_rng`` has the functions ``groovy_rng.seed`` (setting the state to some function determined by the current time), ``groovy_rng.get_state``, returning an int, and ``groovy_rng.set_state`` taking an int, then here is a metaclass for working with ``groovy_rng``:
::

class GroovyRNGSaver(unittest_rand_gen_state.Saver):
reset_state = groovy_rng.seed
get_state = groovy_rng.get_state
set_state = groovy_rng.set_state

This metaclass can be used just like in the previous example.


-------
Changes
-------

.. csv-table:: Changes
:header: "Version", "Date", "Description"
:widths: 15, 15, 70
:delim: $

0.1.4.$ 15/03/2013 $ Duplicate class-name bugfix, Python 2/3 coexistance bugfix
0.1.3 $ 23/02/2013 $ Better test hashes, better support for extension
0.1.2 $ 22/02/2013 $ Doc bugfix
0.1.1 $ 20/02/2013 $ Less stateful design
0.1.0 $ 18/02/2013 $ Initial Release