Smart random-generation state persistence for unittest
Randomized unit tests are great. The inputs fed to your algorithm are not consistent across executions, and so each time the tests run successfully, your confidence in the algorithm grows. Unfortunately, when a test fails, this inconsistency makes it hard to duplicate the input until the problem is solved.
This library provides a simple metaclass for dropping into unittest.TestCase that addresses this. As long as a test fails, it will continue making the same random choices (via random or numpy.random) each execution. Any test that passed the previous execution, will get a fresh set of random choices.
Consider the following simplified example of a test case with three functions testing some algorithms. The first and third algorithms are OK, but the second has a bug when given a number that divides by 20 (the simplified example asserts this directly).
import random class TestMyAlgs(unittest.TestCase): # This "algorithm" is OK. def test_ok(self): i = random.randint(100, 200) self.assertLess(i, i + 1) # This "algorithm" fails for inputs that divide by 20. def test_bad(self): i = random.randint(100, 200) self.assertNotEqual(i % 20, 0, i) # This "algorithm" is OK too. def test_another_ok(self): i = random.randint(100, 200) self.assertLess(i - 1, i)
The second function will fail approximately 5% of the executions. When it does so, it might be hard (especially in a real-life case) to reproduce the problem, and eventually solve it.
Using this library, a simple change makes things easier. Simply use unittest_rand_gen_state.Saver as the metaclass:
import random import unittest_rand_gen_state class TestMyAlgs(unittest.TestCase, metaclass = unittest_rand_gen_state.Saver): ...
or, using an older Python version:
import random import unittest_rand_gen_state class TestMyAlgs(unittest.TestCase): __metaclass__ = unittest_rand_gen_state.Saver ...
Nothing else needs to be changed.
Once TestMyAlgs.test_bad first fails in some execution, then it will consistently fail in following executions. That is, say it fails the first time with
====================================================================== FAIL: test_bad (__main__.TestMyAlgs) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/unittest_rand_gen_state/__init__.py", line 75, in wrapped fn(*args, **kwargs) File "__init__.py", line 21, in test_bad self.assertNotEqual(i % 20, 0, i) AssertionError: 140 ----------------------------------------------------------------------
then every following execution will fail exactly the same way, until it is fixed.
The two other tests, meanwhile, will run afresh all the while.
The metaclass supplied by this library deals with the states of random and numpy.random, which should be sufficient for almost all tests. However, the library may be extended by dropping as a metaclass, a subclass of unittest_rand_gen_state.Saver supplying three class methods:
- reset_state: A method specifying the meaning of resetting the random-generation state (e.g., random.seed)
- get_state: A method specifying the query to call for getting the current random-generation state (e.g., random.getstate)
- set_state: A method specifying the function to call to restore the random-generation state using the value returned by get_state (e.g., random.setstate)
E.g., suppose a random-generation library, groovy_rng has the functions groovy_rng.seed (setting the state to some function determined by the current time), groovy_rng.get_state, returning an int, and groovy_rng.set_state taking an int, then here is a metaclass for working with groovy_rng:
class GroovyRNGSaver(unittest_rand_gen_state.Saver): reset_state = groovy_rng.seed get_state = groovy_rng.get_state set_state = groovy_rng.set_state
This metaclass can be used just like in the previous example.
|0.1.4.||15/03/2013||Duplicate class-name bugfix, Python 2/3 coexistance bugfix|
|0.1.3||23/02/2013||Better test hashes, better support for extension|
|0.1.1||20/02/2013||Less stateful design|
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|File Name & Checksum SHA256 Checksum Help||Version||File Type||Upload Date|
|UnittestRandGenState-0.1.4.tar.gz (6.6 kB) Copy SHA256 Checksum SHA256||–||Source||Mar 15, 2013|
|UnittestRandGenState-0.1.4.zip (12.7 kB) Copy SHA256 Checksum SHA256||–||Source||Mar 15, 2013|