Flaky-Repro

An empirical, frequency-based flaky test diagnosis and reproduction tool for pytest. Flaky-Repro investigates flaky tests by repeatedly executing them under different execution conditions, measuring failure frequencies, comparing PASS/FAIL executions, correlating conditions with captured evidence, confirming stronger candidates through repeated experiments, and attempting to reproduce the observed behavior.

 Why Flaky-Repro?

Flaky tests are difficult to diagnose because they can pass and fail under seemingly identical conditions. A simple approach is to repeatedly run a test:

pytest test_example.py::test_something

But repeated execution alone does not tell you which execution condition is associated with the failures. Flaky-Repro takes an empirical approach. Instead of only asking:

Does this test fail sometimes? Flaky-Repro investigates: Under which execution conditions does this test fail more frequently, and can that behavior be reproduced?

---

 Installation

Install Flaky-Repro directly from PyPI:

pip install flaky-repro

Verify the installation:

flaky-repro --version

Expected:

flaky-repro 0.1.2

---

 Quick Start

Run Flaky-Repro against a pytest target:

flaky-repro <pytest-target>

For example:

flaky-repro flaky_repro/examples/functional_validation/test_worker_validation.py::test_worker_sensitive

Flaky-Repro will automatically run the target through its diagnosis and reproduction pipeline.

 How It Works

Flaky-Repro follows a multi-stage empirical diagnosis process:

                         Pytest Target
                              │
                              ▼
                    ┌──────────────────┐
                    │     Preflight    │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │ Code Analysis    │
                    │  → experiment    │
                    │    shortlisting  │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │     Baseline     │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │   Investigation  │
                    │ workers / mode   │
                    │ timing / timeout │
                    │ order / state    │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │   Correlation    │
                    │ condition ↔      │
                    │ captured evidence│
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │ PASS ↔ FAIL      │
                    │   Comparison     │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │    Structured    │
                    │     Findings     │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │   Confirmation   │
                    └────────┬─────────┘
                             │
                             ▼
                    ┌──────────────────┐
                    │   Reproduction   │
                    └────────┬─────────┘
                             │
                             ▼
                       Diagnostic Result

The goal is not simply to rerun a test. The goal is to determine whether specific execution conditions are associated with increased failure frequency and whether those conditions can be reproduced consistently.

 What It Investigates

Flaky-Repro can investigate execution conditions and signals such as:

• Worker/concurrency levels
• Sequential vs parallel execution
• Timing delays
• Timeout conditions
• Test order and state-related signals
• Failure frequency under different conditions
• PASS ↔ FAIL differences
• Captured exceptions, tracebacks, stderr, and assertion evidence
• Candidate strength and consistency
• Reproduction consistency Before running experiments, Flaky-Repro can analyze the target test source and shortlist relevant experiment types from code-level signals such as async code, mutable state, timing calls, concurrency primitives, file writes, and randomness. The investigation remains empirical: code analysis selects hypotheses, while execution validates or rejects those hypotheses.

---

 Example

The following output is representative of the included functional validation pipeline for a timing/concurrency-sensitive pytest target. Result

                    FLAKY-REPRO
============================================================
TEST
Target : flaky_repro/examples/functional_validation/test_mode_validation.py::test_mode_sensitive
RESULT
Status : FLAKY

Code Analysis

CODE ANALYSIS
Signals detected:
- concurrency
- timing
- timeout
Shortlisted experiments:
- worker
- timing
- mode
- timeout

Code analysis is used to prioritize experiments; it does not claim that a detected code pattern is the root cause. Baseline

BASELINE
Runs         : 20
Passed       : 20
Failed       : 0
Failure Rate : 0%

Investigation Each selected condition is evaluated repeatedly. In the validated V0.1.2 run, 100 repetitions were used per investigation condition:

Type       Condition                 Failure Rate
Worker     2 workers                 37%
Worker     4 workers                 29%
Worker     8 workers                 20%
Timing     10 ms                     17%
Timing     20 ms                     11%
Timeout    30 s                       9%
Timeout    60 s                      12%

Condition Correlation Flaky-Repro groups executions by condition fingerprint and associates observed failures with the evidence captured in those runs.

CONDITION CORRELATION
Total known runs : 1260
Passed           : 1008
Failed           : 252
Unknown          : 0
Top observed condition:
Parallel / 4 workers / 30 s timeout
Observed failure rate : 27.14%
Signal strength       : MODERATE

The strength classification describes the observed signal in the collected runs. It is not statistical confidence and does not establish causation. PASS ↔ FAIL Comparison Flaky-Repro also compares passing and failing executions:

PASS ↔ FAIL COMPARISON
Passed runs : 1008
Failed runs : 252
Unknown     : 0
Average duration
PASS : 1.02 s
FAIL : 1.17 s
Delta: +0.15 s
Failing exception : TestFailure

This helps preserve the distinction between what conditions were associated with failures and what differed between successful and failing executions. Structured Findings The final output turns the collected evidence into structured findings containing:

• condition and condition fingerprint
• known/pass/fail/unknown counts
• observed failure rate and signal strength
• baseline comparison
• captured failure evidence
• PASS ↔ FAIL differences
• confirmation and reproduction results
• replay context where available
• an observational interpretation Findings explicitly avoid claiming that an observed condition is the definitive root cause. Confirmation and Reproduction Candidates are repeatedly evaluated before being treated as stronger findings, followed by higher-volume reproduction runs.

CONFIRMATION
Candidate conditions are re-tested to determine
whether the observed signal persists.
REPRODUCTION
Stronger candidates are subjected to additional runs
to determine whether the observed behavior can be reproduced.

This demonstrates the V0.1.2 workflow: preflight → code analysis → baseline → investigation → condition/evidence correlation → PASS/FAIL comparison → structured findings → confirmation → reproduction

Target : flaky_repro\examples\functional_validation\test_worker_validation.py::test_worker_sensitive RESULT

Status               : FLAKY Observed Pattern     : Parallel Execution Strongest Signal     : Worker Count Condition            : 8 workers Reproduction         : REPRODUCED

### Baseline
The test passed consistently under the sequential baseline:
```text
Baseline
  Runs         : 20
  Passed       : 20
  Failed       : 0
  Failure Rate : 0%

Candidate

Under the strongest candidate condition:

Candidate
  Condition    : Worker (8 workers)
  Runs         : 20
  Passed       : 13
  Failed       : 7
  Failure Rate : 35%

The observed failure-rate difference was:

Effect
  Failure delta: +35 pp
  Direction    : Increased

Investigation

Type           Condition                 Failure Rate
------------------------------------------------------------
Worker         2 workers                 5%
Worker         4 workers                 30%
Worker         8 workers                 35%
Timing         10 ms                     5%
Timing         20 ms                     10%
Mode           Sequential                0%
Mode           Parallel / 4              30%

Candidate Ranking

Rank   Candidate                    Failure Rate   Signal
------------------------------------------------------------
#1     Worker (8 workers)           35%            STRONG
#2     Worker (4 workers)           30%            STRONG
#3     Timing (20 ms)               10%            STRONG

Confirmation

Candidates are repeatedly evaluated to determine whether the observed effect persists:

Candidate                      Classification        Consistency
-----------------------------------------------------------------
#1 Worker (8 workers)          REPEATEDLY OBSERVED   100.0%
#2 Worker (4 workers)          REPEATEDLY OBSERVED   100.0%
#3 Timing (20 ms)              REPEATEDLY OBSERVED   100.0%

Reproduction

The strongest candidates are then subjected to higher-volume reproduction runs:

Rank   Candidate                  Repetitions   Consistency    Result
-----------------------------------------------------------------------
#1     Worker (8 workers)         3             100.0%         REPRODUCED
#2     Worker (4 workers)         3             100.0%         REPRODUCED
#3     Timing (20 ms)             3             100.0%         REPRODUCED

Interpretation

In this example:

• The test passed consistently under the sequential baseline.
• Parallel execution increased the observed failure rate.
• The 8-worker configuration produced the strongest observed signal.
• The candidate remained consistent during confirmation.
• The behavior was successfully reproduced during the reproduction stage. This demonstrates how Flaky-Repro moves from observation → investigation → candidate detection → confirmation → reproduction.

---

 CLI Usage

Show help

flaky-repro --help

or:

flaky-repro -h

Show version

flaky-repro --version

or:

flaky-repro -V

Run a test

flaky-repro <pytest-target>

Example:

flaky-repro tests/test_concurrency.py::test_sensitive

---

 Included Validation Examples

The repository includes functional validation tests designed to exercise different flaky-test behaviors, including:

• Worker/concurrency behavior
• Timing behavior
• Sequential vs parallel execution
• Stable tests
• Random/flaky behavior
• Multifactor conditions These examples are useful for understanding and validating the diagnostic pipeline.

---

 Safety & Limitations

Flaky-Repro repeatedly executes the target test under multiple configurations. Only run it against tests and environments where repeated execution is safe and side effects are acceptable. Be especially careful with tests that:

• Modify production data
• Send emails or messages
• Perform financial transactions
• Create irreversible external side effects
• Depend on shared external state
• Interact with rate-limited services

Important

Flaky-Repro is an empirical diagnostic tool. Its conclusions are based on observed execution frequencies and repeated experiments. A strong signal means that a condition was associated with an increased failure rate in the observed experiments. It should not automatically be interpreted as definitive proof of the underlying root cause.

 Project Status

Current version: 0.1.2 The 0.1.2 release provides:

• PyPI packaging
• Command-line interface
• Preflight target validation
• Structured execution runs
• Deterministic condition fingerprints
• Code analysis and experiment shortlisting
• Baseline execution
• Multi-condition investigation
• Candidate detection
• Condition ↔ evidence correlation
• PASS ↔ FAIL comparison
• Structured findings
• Candidate confirmation
• Reproduction testing
• Human-readable diagnostic output
• Functional validation examples Flaky-Repro is still an early release. Diagnostic behavior, internal APIs, configuration options, and output formats may change in future versions. Current limitations V0.1.2 is intentionally focused on empirical investigation rather than full root-cause diagnosis.
• Observed correlations are not causal proof.
• Exact replay of a prior execution is not yet implemented.
• Rich passing-side evidence and more advanced statistical analysis are planned for later work.
• Environment-specific CI reproduction is not guaranteed.
• Baseline and experiment sample sizes affect the stability of observed failure rates.
• Some experiment types, such as order/state investigation, are not automatically wired into every pipeline path yet.
• Tests with unsafe or irreversible side effects should not be investigated casually.

---

 Development

Clone the repository and enter the project directory:

git clone https://github.com/MrigankKumawat/Flaky-Repro.git
cd Flaky-Repro

Create a virtual environment:

python -m venv .venv

Activate it on Windows:

.venv\Scripts\activate

Install the project in editable mode:

pip install -e .

---

 License

Flaky-Repro is released under the MIT License. See the LICENSE file for the complete license text.

 Contributing

Issues, bug reports, suggestions, and improvements are welcome. If you discover a flaky-test pattern that Flaky-Repro handles incorrectly, opening an issue with a reproducible example can help improve the project.

 Flaky-Repro

Investigate the conditions behind flaky tests.