pytest-flakemark 1.1.0

Differential execution tracer that finds the exact file, line, and root cause of any flaky test.

FLAKEMARK

pytest-flakemark — Find the exact line where your flaky test breaks.

Not "your test is flaky." The actual file. The actual line. The actual fix.

Built by Khushdeep Sharma.

---

The Problem

FAILED tests/test_login.py::test_user_session
[Flaky — rerunning]
PASSED tests/test_login.py::test_user_session

Every existing tool gives you this. It tells you nothing new.

What FLAKEMARK Gives You

--------------------------------------------------------
FLAKEMARK - Flaky Test Root Cause Found
--------------------------------------------------------
File:       tests/test_login.py
Line:       47
Function:   test_user_session
Type:       timing_delta
Cause:      Race condition or timing dependency

Detail:     Line 47: 1.2ms (pass) vs 148.3ms (fail) — 124x timing difference.

Fix:        Replace time.sleep(N) with threading.Event().wait()
            or asyncio.wait_for(). Never hardcode sleep durations in tests.

Confidence: 85%  |  Total divergences: 1
--------------------------------------------------------

---

How FLAKEMARK Works

FLAKEMARK instruments your test at the AST level, runs it twice simultaneously, records every operation in both runs, then finds the exact line where the two executions diverged. That divergence is your bug.

Your test
  ├── Run 1 (instrumented) → ExecutionTrace A  [op, op, op ...]
  └── Run 2 (instrumented) → ExecutionTrace B  [op, op, op ...]
                                                     ↓
                                       DifferentialAnalyser
                                       Two-pointer trace walk
                                                     ↓
                                   "Line 47: 124x timing difference"

---

Install

pip install pytest-flakemark

Requires Python 3.10+. The only external dependency is pytest.

---

Usage — 4 Ways

1. Source string

from flakemark import FlakeMark

source = """
import random
def test_flaky():
    result = random.randint(0, 1)
    assert result == 1
"""

report = FlakeMark.diagnose_source(source, "test_flaky", runs=10)
print(report)

2. Real test file

from flakemark import FlakeMark

report = FlakeMark.diagnose_file(
    filepath       = "tests/test_api.py",
    test_func_name = "test_user_session",
    runs           = 6,
    project_root   = "/path/to/your/project",
)
print(report)

3. Batch scan entire folder

from flakemark import FlakeMark

results = FlakeMark.diagnose_batch("tests/", runs=4)

flaky = {k: v for k, v in results.items() if v.is_found()}
print(f"FLAKEMARK found {len(flaky)} flaky tests:\n")
for name, report in flaky.items():
    print(f"  {name}")
    print(f"  Line {report.primary.line} — {report.primary.divergence_type.value}")
    print(f"  Fix: {report.primary.fix[:60]}")

4. pytest CLI (after pip install)

pytest --flakemark-diagnose tests/
pytest --flakemark-diagnose --flakemark-runs=8 tests/test_api.py

---

What FLAKEMARK Detects

Type	What it means	Root cause
value_mismatch	Same line, different value	random, shared state
timing_delta	Same op, 3x+ slower	time.sleep(), race condition
thread_race	Same op, different thread	Missing Lock()
sequence_break	Different execution path	Test order dependency
missing_event	One run skipped an operation	Conditional on external state
early_termination	One run ended much sooner	Timeout, unhandled exception

---

Parameters

Parameter	Default	Meaning
runs	4	Times to run. Use 10+ for low-frequency flakes
timeout	30	Seconds before a run is killed
project_root	os.getcwd()	Project root so imports work

---

Accuracy

python tests/test_accuracy.py

Score: 15/15 = 100%
  ✓  VALUE_MISMATCH detected   (78% confidence)
  ✓  TIMING_DELTA detected     (85% confidence)
  ✓  THREAD_RACE detected      (93% confidence)
  ✓  EARLY_TERMINATION detected (55% confidence)
  ✓  No false positive on stable test (0%)
  ✓  Async test handled correctly

---

Comparison to Other Tools

	FLAKEMARK	FlakyGuard	pytest-randomly	CANNIER	Divergent
Finds exact root cause line	Yes	No	No	No	Yes (JS only)
Python / pytest	Yes	No (Java)	Yes	Yes	No (JS)
AST instrumentation	Yes	No	No	No	Partial
Subprocess isolation	Yes	No	No	No	No
Async test support	Yes	No	Yes	No	No
Zero dependencies	Yes	No	No	No	No

---

Project Structure

flakemark/
├── flakemark/
│   ├── __init__.py              FlakeMark, DivergenceReport, DivergenceType
│   ├── engine.py                FlakeMark class — diagnose_file/source/batch
│   ├── pytest_plugin.py         pytest plugin — --flakemark-diagnose flag
│   ├── tracer/
│   │   └── instrument.py        TraceInserter + run_instrumented()   ← CORE IP
│   └── differ/
│       └── divergence.py        DifferentialAnalyser + DivergenceType ← CORE IP
├── tests/
│   └── test_accuracy.py         Accuracy test suite
├── pyproject.toml               PyPI config
├── setup.py                     PyPI config
└── README.md

---

What to Copyright

File	What is original
tracer/instrument.py — TraceInserter	AST NodeTransformer for execution recording
tracer/instrument.py — run_instrumented()	Subprocess isolation runner with JSON protocol
differ/divergence.py — DifferentialAnalyser._walk()	Two-pointer differential trace walk
differ/divergence.py — DivergenceType	Six-category flakiness classification schema
differ/divergence.py — _score()	Confidence scoring formula
engine.py — FlakeMark._collect_traces()	Concurrent dual-run orchestration

---

License

MIT License — Copyright (c) 2026 Khushdeep Sharma. All rights reserved.

See LICENSE for details.

---

FLAKEMARK — Find the line. Fix the test.