FLAKEMARK

pytest-flakemark — Find the exact line where your flaky test breaks.

Not "your test is flaky." The actual file. The actual line. The actual fix.

Built by Khushdeep Sharma.

---

The Problem

FAILED tests/test_login.py::test_user_session
[Flaky — rerunning]
PASSED tests/test_login.py::test_user_session

Every existing tool gives you this. It tells you nothing new.

What FLAKEMARK Gives You

--------------------------------------------------------
FLAKEMARK - Flaky Test Root Cause Found
--------------------------------------------------------
File:       tests/test_login.py
Line:       47
Function:   test_user_session
Type:       timing_delta
Cause:      Race condition or timing dependency

Detail:     Line 47: 1.2ms (pass) vs 148.3ms (fail) — 124x timing difference.

Fix:        Replace time.sleep(N) with threading.Event().wait()
            or asyncio.wait_for(). Never hardcode sleep durations in tests.

Confidence: 85%  |  Total divergences: 1
--------------------------------------------------------

---

How FLAKEMARK Works

FLAKEMARK instruments your test at the AST level, runs it twice simultaneously, records every operation in both runs, then finds the exact line where the two executions diverged. That divergence is your bug.

Your test
  ├── Run 1 (instrumented) → ExecutionTrace A  [op, op, op ...]
  └── Run 2 (instrumented) → ExecutionTrace B  [op, op, op ...]
                                                     ↓
                                       DifferentialAnalyser
                                       Two-pointer trace walk
                                                     ↓
                                   "Line 47: 124x timing difference"

---

Install

pip install pytest-flakemark

Requires Python 3.10+. The only external dependency is pytest.

---

Usage — 4 Ways

1. Source string

from flakemark import FlakeMark

source = """
import random
def test_flaky():
    result = random.randint(0, 1)
    assert result == 1
"""

report = FlakeMark.diagnose_source(source, "test_flaky", runs=10)
print(report)

2. Real test file

from flakemark import FlakeMark

report = FlakeMark.diagnose_file(
    filepath       = "tests/test_api.py",
    test_func_name = "test_user_session",
    runs           = 6,
    project_root   = "/path/to/your/project",
)
print(report)

3. Batch scan entire folder

from flakemark import FlakeMark

results = FlakeMark.diagnose_batch("tests/", runs=4)

flaky = {k: v for k, v in results.items() if v.is_found()}
print(f"FLAKEMARK found {len(flaky)} flaky tests:\n")
for name, report in flaky.items():
    print(f"  {name}")
    print(f"  Line {report.primary.line} — {report.primary.divergence_type.value}")
    print(f"  Fix: {report.primary.fix[:60]}")

4. pytest CLI (after pip install)

pytest --flakemark-diagnose tests/
pytest --flakemark-diagnose --flakemark-runs=8 tests/test_api.py

---

What FLAKEMARK Detects

Type	What it means	Root cause
value_mismatch	Same line, different value	random, shared state
timing_delta	Same op, 3x+ slower	time.sleep(), race condition
thread_race	Same op, different thread	Missing Lock()
sequence_break	Different execution path	Test order dependency
missing_event	One run skipped an operation	Conditional on external state
early_termination	One run ended much sooner	Timeout, unhandled exception

---

Parameters

Parameter	Default	Meaning
runs	4	Times to run. Use 10+ for low-frequency flakes
timeout	30	Seconds before a run is killed
project_root	os.getcwd()	Project root so imports work

---

Comparison to Other Tools

	FLAKEMARK	FlakyGuard	pytest-randomly	CANNIER	Divergent
Finds exact root cause line	Yes	No	No	No	Yes (JS only)
Python / pytest	Yes	No (Java)	Yes	Yes	No (JS)
AST instrumentation	Yes	No	No	No	Partial
Subprocess isolation	Yes	No	No	No	No
Async test support	Yes	No	Yes	No	No
Zero dependencies	Yes	No	No	No	No

---

License

MIT License — Copyright (c) 2026 Khushdeep Sharma. All rights reserved.

See LICENSE for details.

---

FLAKEMARK — Find the line. Fix the test.