Pytest plugin to randomly order tests and control random.seed. (Also available for nose).
Features
All of these features are on by default but can be disabled with flags.
Randomly shuffles the order of test items. This is done first at the level of modules, then at the level of test classes (if you have them), then at the order of functions. This also works with things like doctests.
Resets random.seed() at the start of every test case and test to a fixed number - this defaults to time.time() from the start of your test run, but you can pass in --randomly-seed to repeat a randomness-induced failure.
If factory boy is installed, its random state is reset at the start of every test. This allows for repeatable use of its random ‘fuzzy’ features.
If faker is installed, its random state is reset at the start of every test. This is also for repeatable fuzzy data in tests - factory boy uses faker for lots of data.
If numpy is installed, its random state is reset at the start of every test.
If additional random generators are used, they can be registered under the pytest_randomly.random_seeder entry point and will have their seed reset at the start of every test. Register a function that takes the current seed value.
Works with pytest-xdist.
About
Randomness in testing can be quite powerful to discover hidden flaws in the tests themselves, as well as giving a little more coverage to your system.
By randomly ordering the tests, the risk of surprising inter-test dependencies is reduced - a technique used in many places, for example Google’s C++ test runner googletest.
By resetting the random seed to a repeatable number for each test, tests can create data based on random numbers and yet remain repeatable, for example factory boy’s fuzzy values. This is good for ensuring that tests specify the data they need and that the tested system is not affected by any data that is filled in randomly due to not being specified.
This plugin is a Pytest port of my plugin for nose, nose-randomly. I’ve written a blog post on its history.
Usage
Install from pip with:
python -m pip install pytest-randomlyPython 3.5 to 3.8 supported.
Pytest will automatically find the plugin and use it when you run pytest. The output will start with an extra line that tells you the random seed that is being used:
$ pytest
...
platform darwin -- Python 3.7.2, pytest-4.3.1, py-1.8.0, pluggy-0.9.0
Using --randomly-seed=1553614239
...If the tests fail due to ordering or randomly created data, you can restart them with that seed using the flag as suggested:
pytest --randomly-seed=1234Or more conveniently, use the special value last:
pytest --randomly-seed=lastSince the ordering is by module, then by class, you can debug inter-test pollution failures by narrowing down which tests are being run to find the bad interaction by rerunning just the module/class:
pytest --randomly-seed=1234 tests/module_that_failed/You can disable behaviours you don’t like with the following flags:
--randomly-dont-reset-seed - turn off the reset of random.seed() at the start of every test
--randomly-dont-reorganize - turn off the shuffling of the order of tests
The plugin appears to Pytest with the name ‘randomly’. To disable it altogether, you can use the -p argument, for example:
pytest -p no:randomlyEntry Point
If you’re using a different randomness generator in your third party package, you can register an entrypoint to be called every time pytest-randomly reseeds. Implement the entrypoint pytest_randomly.random_seeder, referring to a function/callable that takes one argument, the new seed (int).
For example in your setup.cfg:
[options.entry_points]
pytest_randomly.random_seeder =
mypackage = mypackage.reseedThen implement reseed(new_seed).
History
3.3.1 (2020-04-15)
Fix to work when pytest-xdist is not installed or active (PluginValidationError: unknown hook 'pytest_configure_node').
3.3.0 (2020-04-15)
Add pytest-xdist support. Previously it only worked reliably when setting --randomly-seed explicitly. When not provided, the default seed generated in workers could differ and collection would fail. Now when it is not provided, all xdist worker processes shared the same default seed generated in the master process.
3.2.1 (2020-01-13)
Update MANIFEST.in so tests are included in the sdist tarball again.
3.2.0 (2019-12-19)
Converted setuptools metadata to configuration file. This meant removing the __version__ attribute from the package. If you want to inspect the installed version, use importlib.metadata.version("pytest-randomly") (docs / backport).
Convert reading entrypoints to use importlib.metadata. Depend on importlib-metadata on Python < 3.8.
Update Python support to 3.5-3.8.
3.1.0 (2019-08-25)
Add plugins via entry points pytest_randomly.random_seeder to allow outside packages to register additional random generators to seed. This has added a dependency on the entrypoints package.
3.0.0 (2019-04-05)
Update Python support to 3.5-3.7, as 3.4 has reached its end of life.
Handle CollectErrors and ImportErrors during collection when accessing item.module.
2.1.1 (2019-03-26)
Fix including tests in sdist after re-arrangement in 2.1.0.
2.1.0 (2019-03-01)
Add the option --randomly-seed=last to reuse the last used value for the seed.
2.0.0 (2019-02-28)
Drop Python 2 support, only Python 3.4+ is supported now.
1.2.3 (2017-12-06)
Fix DeprecationWarning with recent versions of factory_boy.
1.2.2 (2017-11-03)
Fix collection to not sometimes crash when encoutering pytest Items without a module.
1.2.1 (2017-06-17)
Fix collection to be deterministically shuffled again, regression in 1.2.0.
1.2.0 (2017-06-16)
Dropped Python 2.6 compatibility, as upstream dependency NumPy did.
Reset and output the seed at the start of the test run when --randomly-dont-reset-seed is set, to allow the reorganization of tests to be reproducible.
1.1.2 (2016-10-27)
Reset the random state for NumPy too.
1.1.1 (2016-09-16)
Add Python 2.6 compatibility
1.1.0 (2016-09-12)
Offset the random seed during test setup and teardown. This is to avoid the awkward situation where test setup generates a random object, then the test generates a second one, but due to the re-seeding, they end up being always the same object. Thanks @rouge8 for the report.
1.0.0 (2016-04-15)
First release on PyPI.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file pytest-randomly-3.3.1.tar.gz.
File metadata
- Download URL: pytest-randomly-3.3.1.tar.gz
- Upload date:
- Size: 31.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.8.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
568b6a3bd251b566e8d99fc7bcb15c3c39561babd07e16412563b7dab1b0290a
|
|
| MD5 |
62a99a0f4681a2dd3d4066e3444665f4
|
|
| BLAKE2b-256 |
469fe43f7cfca572d89c2c39e8db3f68121a4d4ae4c233e0b5cc959764bb541f
|
File details
Details for the file pytest_randomly-3.3.1-py3-none-any.whl.
File metadata
- Download URL: pytest_randomly-3.3.1-py3-none-any.whl
- Upload date:
- Size: 7.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.8.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
9b06429c877b3aab360c7cf698f2a65dbe75db1b1ed0ab02dc3bb47f3487e469
|
|
| MD5 |
e2a85c19ebf136f7b66ae16053cf9c4c
|
|
| BLAKE2b-256 |
994fd97fdccd7d347251f9c8a1396e8fab09f99f0aa2e3dccff42eb63863400e
|
