pytest-horizon

An experimental general-purpose parallel execution plugin for pytest. Python runs pytest, fixtures, hooks, plugins and conftest.py. A separate Rust process owns worker lifecycle, scheduling, deadlines, crash recovery and execution accounting.

The Playwright suite in this repository is a comparison workload. The execution engine does not import or require Playwright or xdist.

Install

The 0.1.0a1 release artifacts bundle the Python plugin and Rust controller. Windows x86-64 and Linux x86-64 wheels install without a Rust compiler. The Linux wheel requires glibc 2.28 or newer. Python 3.11+ and pytest 8.4–9.x are required; the initial release was validated on Python 3.14 with pytest 9.1.1.

Once 0.1.0a1 is published to PyPI:

python -m pip install pytest-horizon==0.1.0a1
python -m pytest your_tests --horizon=4

With uv, use uv add --dev pytest-horizon==0.1.0a1, then uv run pytest your_tests --horizon=4. A local wheel can also be installed by passing its .whl path to pip or uv. The alpha is experimental; see the execution contract and limits below before using it in an existing suite.

Release maintainers: see the release guide for the GitHub Actions workflow and tag instructions.

Build from source

Source installations require the Rust toolchain. Maturin compiles and installs the controller as part of the Python package build.

python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -e ".[dev]"
.\.venv\Scripts\python.exe -m pytest your_tests --horizon=4

The plugin locates the controller installed beside its Python interpreter, even without activating the environment. It also supports local Cargo builds and PATH as fallbacks. Set HORIZON_BINARY or use --horizon-binary to provide an explicit executable. After changing Rust code, reinstall the editable package or point HORIZON_BINARY at the rebuilt Cargo executable.

Run tests in parallel

# Preserve test order and worker affinity within each file (default).
.\.venv\Scripts\python.exe -m pytest your_tests --horizon=4 --horizon-schedule=file

# Distribute individual tests for finer balancing.
.\.venv\Scripts\python.exe -m pytest your_tests --horizon=4 --horizon-schedule=test

# Phase deadlines apply independently to setup, call and teardown.
.\.venv\Scripts\python.exe -m pytest your_tests --horizon=4 --horizon-timeout=30

# Explicitly opt into rerunning attempts interrupted by a worker crash or timeout.
.\.venv\Scripts\python.exe -m pytest your_tests --horizon=4 --horizon-retries=1

With no --horizon option, pytest runs normally. --collect-only uses native pytest collection. Selecting both --horizon and xdist -n is rejected.

Option	Default	Purpose
--horizon=N	Disabled	Start N local pytest workers.
--horizon-schedule=file|test	file	Keep files together or balance individual tests.
--horizon-history=on|off	on	Use previous durations to schedule longer work first.
--horizon-timeout=SECONDS	60	Deadline for each setup, call and teardown phase.
--horizon-startup-timeout=SECONDS	60	Deadline for worker startup and collection.
--horizon-shutdown-timeout=SECONDS	10	Deadline for worker shutdown.
--horizon-max-restarts=N	4	Maximum worker replacements per run.
--horizon-retries=N	0	Retries for attempts interrupted by a crash or deadline.
--horizon-binary=PATH	Automatic	Override the bundled Rust executable.

The horizon_worker_id fixture identifies a worker (hz0, hz1, ...). Use it to give workers separate external resources when needed. Without parallel mode it returns main. Session fixtures run once per worker, so an existing suite must tolerate multiple sessions and isolate shared files, accounts or databases.

Current execution contract

• Every worker runs a real pytest session on its main thread and collects the suite independently. Ordered collection manifests must match before dispatch.
• Scheduling owns metadata and indices, never serialized live pytest objects. File scheduling preserves collection order inside each file. Optional duration history orders scheduling units by longest estimated duration first, with stable ties. History is a performance hint, never a reason to skip a test.
• Each command names the current test and an already-reserved next test. The worker passes that actual next item into pytest's protocol, retaining fixture scopes across commands. Session fixtures belong to each worker session.
• Setup, call and teardown have independent hard deadlines in Rust. Startup / collection and shutdown have separate deadlines. A Python faulthandler dump provides best-effort stack evidence before a phase deadline expires.
• An interrupted attempt fails by default. The worker's unstarted assignments return to the scheduler, and a replacement worker collects and validates the manifest before executing more work. Replacement count is bounded.
• Opt-in retries apply only to interrupted attempts, not assertion failures. Earlier attempt events remain in the event log and retries are printed. Retrying can repeat external side effects; execution is not exactly-once.
• Pytest setup/call/teardown reports are reconstructed in the coordinator. Basic reporting, JUnit XML, skips, xfail, fixture errors and pytest-asyncio have subprocess integration coverage. Reports are buffered until attempt completion so an interrupted retry does not masquerade as an additional final result.
• Global --maxfail stops new dispatch; already running tests may finish.
• EOF on the coordinator's control pipe cancels the supervisor. Ctrl+C requests cancellation. Windows workers use Job Objects with kill-on-close semantics; Linux uses process groups. Both Windows and Linux (Ubuntu under WSL2) have subprocess integration coverage. Unix descendants that deliberately leave the process group are outside that containment boundary.

Diagnostics

Each execution writes .horizon/runs/<run-id>/:

• spec.json: execution settings and pytest arguments.
• events.jsonl: timestamped worker and coordinator events.
• summary.json: test states, attempt counts, totals, and supervisor exit status.
• worker-<number>.log: worker output, native output and best-effort stack dumps.
• controller.log: supervisor errors.

IPC uses dedicated handles and bounded event messages (8 MiB). Native writes to stdout are redirected to worker logs so test output cannot corrupt the protocol. Rust serializes each outgoing event once and writes a complete JSON frame to the log and reporting pipe, avoiding a filesystem write for each formatted field. Test output under -s is retained in those logs; live multiplexed output is not implemented. Duration history is stored in .horizon/durations.json.

Compare with xdist

The repository includes 48 local Playwright tests across six files: forms, delayed DOM updates, intercepted API requests, isolated storage, navigation and a deliberately slower file. No remote website, account or server is required.

$env:PLAYWRIGHT_BROWSERS_PATH = Join-Path (Get-Location) '.tools\browsers'
.\.venv\Scripts\python.exe -m playwright install chromium --only-shell
.\.venv\Scripts\python.exe scripts/compare.py --workers 4 --repeats 3 --faults

The comparison alternates engines, uses fresh processes and browsers, records complete command wall time, and checks per-test phase outcomes. It compares file scheduling with xdist loadfile, and individual scheduling with xdist load. Horizon runs both with and without populated historical durations. Every variant gets an unmeasured priming run. Results include every command and raw report.

The exact dependency versions for the recorded comparison are pinned in requirements-benchmark.txt. A tracked validation summary records the tested environments and outcomes. Each comparison generates a local artifacts/comparison-*/REPORT.md and results.json; raw logs and generated artifacts are excluded from Git.

Separate fault runs deliberately hang or kill a worker after a browser starts. Horizon gets a 3-second phase deadline; a common 12-second external watchdog bounds both engines. This tests default xdist without an additional timeout plugin. It does not reproduce or explain the original reported xdist freeze.

Verification

cargo test --locked
cargo build --release --locked
.\.venv\Scripts\python.exe -m pip install -r requirements-compatibility.txt
$env:HORIZON_BINARY = Join-Path (Get-Location) 'target\release\horizon-controller.exe'
$env:PYTEST_DISABLE_PLUGIN_AUTOLOAD = '1'
.\.venv\Scripts\python.exe -m pytest tests -p pytest_asyncio.plugin -q
Remove-Item Env:PYTEST_DISABLE_PLUGIN_AUTOLOAD

The tests start real Rust supervisors and Python workers. Plugin dependencies are loaded explicitly in each subprocess; a separate case tests installed plugins together with normal auto-loading. Missing optional plugin dependencies produce skips, so install the compatibility requirements before claiming matrix coverage.

The expanded suite exercises pytest-cov (branch aggregation, test contexts, early conftest imports and failure thresholds), pytest-randomly (shared seeds), pytest-mock, pytest-html (worker extras), pytest-rerunfailures (including --fail-on-flaky), pytest-timeout, Hypothesis, AnyIO and pytest-asyncio. Adapters in python/pytest_horizon/compatibility.py use some plugin internals; the pinned versions are the verified contract, not all past or future releases. Coverage from a forcibly killed worker can be incomplete. For --cov-context=test, use COVERAGE_CORE=ctrace or [run] core=ctrace in your coverage configuration. With the tested coverage version on Python 3.14, the default sys.monitoring tracer produced missing/mislabelled contexts in plain pytest, xdist and Horizon. Horizon rejects that unsupported combination before starting workers instead of publishing misleading per-test attribution.

For longer browser validation:

.\.venv\Scripts\python.exe scripts/stress.py --output artifacts/stress-local

This starts 12 paired cycles using one to eight workers, a 960-test session per engine, repeated Horizon browser crash/hang recovery, and bounded xdist failure cases with and without pytest-timeout. Each run retains commands, reports, measurements and Horizon state summaries. This is a correctness stress check; the original comparison harness is better suited to controlled timing runs.

For Linux, install Python venv support, a C linker and Cargo, then run bash scripts/setup_linux.sh. If required, install Chromium's system libraries using Playwright's install-deps chromium command with system permissions.

export HORIZON_BINARY="$PWD/target/linux/release/horizon-controller"
export PLAYWRIGHT_BROWSERS_PATH="$PWD/.tools/linux/browsers"
export PYTHONPYCACHEPREFIX="$PWD/.tools/linux/pycache"
PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 .tools/linux/venv/bin/python -m pytest tests \
  -p pytest_asyncio.plugin -q -o cache_dir=.tools/linux/pytest-cache
.tools/linux/venv/bin/python scripts/stress.py --output artifacts/stress-linux

Use separate bytecode, pytest cache and browser output paths when Windows and WSL share this checkout. WSL uses a Linux binary and Linux browser; measurements from a mounted Windows filesystem should not be treated as native Linux performance. The recorded hardening results are summarized in docs/VALIDATION.md. Detailed reports and machine-readable data are generated locally under artifacts/.

Prototype limits

This is not a full xdist replacement or a promise of compatibility with every pytest plugin. There is no remote execution, live debugging, resource-lock API, custom scheduler hook API, warning forwarding, or full compatibility with xdist-specific hooks. Arbitrary plugin-defined report payloads may require additional JSON normalization.

The coordinator is still a Python pytest process: blocking coordinator hooks or blocked local storage can stall reporting. The independent Rust phase watchdog does not make every possible coordinator or operating-system failure bounded. Forced termination cannot guarantee fixture finalizers or finalized browser traces. Different valid schedules can expose shared-state and order-dependent tests. Historical scheduling makes decisions consistent for a given manifest and history; it does not promise identical concurrent execution order.

Broader plugin and platform compatibility needs further work. The current evidence is a local synthetic comparison and deliberately injected failures, not a general performance or reliability claim.

Report an issue

Open an issue with your OS, Python, pytest and plugin versions, the command used, and a minimal reproducer. For a hang or crash, include the relevant Horizon summary and worker log after removing credentials and application data. Logs can contain test output and pytest arguments.

See the changelog for release notes. Licensed under the MIT license.