flakytest

See where your pytest suite spends its time, why, and what to fix first.

flakytest runs your tests under a sampling profiler and tells you, for every test, class and file, how much time goes to setup, the test itself and teardown, and inside those to the database, password hashing, network calls, sleeps, fixtures, factories, imports… down to the line of your code that causes it. Then it ranks what to fix, using the perf tricks for pytest, Django and SQLAlchemy that usually take someone an afternoon of profiling to find.

It works locally, needs no account, and is built to be used by coding agents as much as by people. flakytest.dev is optional: it collects the same profiles from every machine and CI job, so you can see why CI is slower than your laptop, and it still tracks flaky tests.

Documentation: how profiling works, every command and option, the report, the cloud and what it uploads.

$ pip install flakytest
$ flakytest run
...
============================== flakytest profile ===============================
812 tests in 1.4 min (test time 5.0 min) · local:anze@mbp · Python 3.15.0 · tachyon @ 1000 Hz
Time goes to: password hashing 41% · database 30% · test code 9% · sleep 6%
Slowest tests:
    3.21s  tests/test_orders.py::TestCheckout::test_checkout  (database 70%, setUp (call) 2.1s, 412 queries)
Fix next:
  1. [high] Use a fast password hasher in tests (~2.0 min per run)
  2. [high] Create shared test data in setUpTestData instead of setUp (~41s per run)
  3. [high] Run tests in parallel on all 10 CPUs (~3.9 min per run)
Report: .flakytest/runs/20261016-210311-8c3f2a/index.html

Install

pip install flakytest

On Python 3.15+ flakytest uses tachyon, the sampling profiler in the standard library. On older Pythons it uses py-spy: pip install 'flakytest[py-spy]'. Without either you still get exact durations, fixture times and query counts, just no breakdown inside tests.

Profile your tests

flakytest run                 # runs pytest with profiling and opens the report
flakytest run -- tests/api -x # anything after -- goes to pytest
pytest --flakytest-profile    # the same, from pytest itself (or FLAKYTEST_PROFILE=1 pytest)

Every run writes .flakytest/runs/<run id>/ with:

• index.html — the report. A list of every test grouped by file and class; click a test or a group to see where its time goes: setup/call/teardown, time by category, the pieces of work (fixtures, setUp, the test body, Django's transaction setup…), the hot lines in your code with their source, a call tree (pytest internals hidden), hot functions, fixtures, SQL queries and repeated queries (N+1), and the suggestions that apply. flakytest serve serves it on localhost.
• profile.json — everything in the report, documented in docs/profile-format.md.

.flakytest/ ignores itself in git.

Memory

Every test's memory is measured too, in whichever process runs it (each pytest-xdist worker measures its own tests): how much it needed above what its process already had (its peak), and how much it kept. The report lists the tests that needed the most (the Most memory filter in the HTML report), each worker's peak, and on Linux all workers added up over time, which is what a CI machine needs to avoid running out of memory. A process that keeps growing gets a memory_growth warning naming the tests it grew in.

Measuring costs well under a millisecond per test. Limitations:

• Exact per-test peaks need Linux, where the kernel's peak counter can be reset per test. On macOS and Windows a test's peak is only known when it raised its process's peak, and pip install psutil adds the memory at the start and end of every test.
• Only the test processes are measured: a database server, a browser or subprocesses aren't included.
• "Kept" is noisy, since Python and the C allocator often keep freed memory for reuse; peaks are reliable.

What it suggests

Each suggestion comes with the time it costs per run, the lines of your code that trigger it, the tests it affects, and how to make the change without making the tests less trustworthy. Among them:

With coding agents

The report is written so an agent can act on it:

flakytest report                      # Markdown: where time goes, ranked fixes with file:line, slowest tests
flakytest report --test tests/test_orders.py::TestCheckout   # one test or a group, with its call tree
flakytest report --format json        # the same as JSON
flakytest compare                     # previous vs latest run: faster/slower tests, and correctness

flakytest compare flags tests whose outcome changed and tests that stopped running, so a speed-up that deletes, skips or breaks tests doesn't go unnoticed. The report's Copy for agent button copies what you're looking at as Markdown.

flakytest mcp is an MCP server (stdio) over your local runs, with get_report, get_test, list_slow_tests, compare_runs and list_runs tools:

claude mcp add flakytest -- flakytest mcp

Options

If tests are too short for the sampling rate, the report says so and suggests a rate: rerun with --flakytest-rate 5000 (tachyon reads the process from the outside, so high rates are cheap).

How it works

pytest runs every test phase from CallInfo.from_call. flakytest wraps it so each phase runs inside a tiny generated function named after the test and phase (__flakytest_call_1f3a9c0d2b4e__). A sampling profiler attached to the pytest process (from a separate process, so it barely slows the tests down) captures that frame in every stack it samples, which attributes each sample to a test and phase exactly, with no clock synchronisation, for tachyon and py-spy alike, and per pytest-xdist worker. pytest's own durations stay the source of truth; samples only decide how a phase's time is split.

Fixture setup times are measured exactly with a pytest_fixture_setup wrapper, and Django (execute_wrapper) and SQLAlchemy (engine events) queries are counted per test.

Permissions. Sampling profilers read another process's memory. On Linux flakytest allows its own profiler to attach (prctl(PR_SET_PTRACER)), which is enough with the default ptrace_scope=1 (including GitHub Actions). In Docker you may need --cap-add SYS_PTRACE. On macOS profilers need root: sudo -E flakytest run. flakytest doctor checks your setup.

flakytest.dev

Set FLAKYTEST_SECRET_TOKEN (create a project on flakytest.dev) and every profiled run is uploaded, from developer machines and CI alike, so you can compare environments, see trends and regressions, and query it all from agents via flakytest.dev's MCP server. The token also turns on flaky test tracking for every run (profiled or not): failures are recorded per environment, and muted tests on flakytest.dev don't fail your build.

export FLAKYTEST_SECRET_TOKEN=****************************
pytest --flakytest-profile

What's uploaded is profile.json: test names, outcomes, durations, file paths relative to the project, function names, source lines of the hot spots in your code, normalized SQL of repeated queries, the environment name (local:<user>@<host> unless you set FLAKYTEST_ENV), git branch/commit and changed file names. Set FLAKYTEST_UPLOAD=0 to keep profiles local while still tracking flaky tests.

Learn more

The documentation covers everything above in more detail: the report, every CLI command, how samples are attributed, the categories and suggestions, CI setup, flaky test classification, the cloud's MCP server, uploads and data retention.

License

flakytest is distributed under the terms of the MIT license.