pytest-stochastics

A pytest plugin for running stochastic tests with configurable policies.

Definition: Stochastic Test - A test that may occasionally fail due to the non-deterministic character of the test subject. Evaluated by an at_least of out out_of model.

Features

• Run stochastic tests multiple times with customizable pass/fail criteria
• Configure different test plans with fallback options

Installation

You can install pytest-stochastics using pip:

pip install pytest-stochastics

Usage

Configuration

Create a pytest_stochastics.json file in your project root with your test configuration:

{
    "test_plans": [
        {
            "plan": "weak",
            "policy_tests": [
                {
                    "policy": "always",
                    "tests": [
                        "tests/test_abc/test_1", 
                    ]
                },
                {
                    "policy": "mostly",
                    "tests": [
                        "tests/test_abc/test_2",
                        "tests/test_abc/test_3"
                    ]
                }
            ]
        },
        {
            "plan": "strong",
            "policy_tests": [
                {
                    "policy": "always",
                    "tests": [
                        "tests/test_abc/test_2"
                    ]
                }
            ]
        }
    ],
    "policies": [
        {
            "policy": "always",
            "at_least": 3,
            "out_of": 3
        },
        {
            "policy": "mostly",
            "at_least": 2,
            "out_of": 3
        }
    ],
    "plan_fallbacks": [
        {
            "plan": "strong",
            "overrides": "weak"
        }
    ]
}

Running Tests

Run your tests as usual with pytest:

pytest

You may override the default behaviour by defining a custom plan named default.

To specify a plan:

pytest --plan="name of plan"