pytest-sources

pytest-sources is a plugin for pytest that allows tests to be run on multiple sources.

This is handy for autograding coursework, take-home interview screening, and conformance suites across implementation variations.

The full guide lives in docs/GUIDE.md; the internals are described in docs/DESIGN.md.

Features

• Run one test suite against many source directories, each test parameterised per source.
• One process per source isolation enabled when using -n.
• Per-source results table, with grid views of source × test.
• Per-source failure budget with --sources-maxfail.
• Tests run with the source on sys.path and as the working directory, so submissions can import and read their own files.
• Sources per test via @pytest.mark.sources, opt-outs via no_sources and no_chdir.

Installation

pip install pytest-sources

Quick start

pytest --sources "sources/*" -n auto

def test_add(source):
    solution = source.import_module("solution")
    assert solution.add(2, 3) == 5

tests/test_add.py::test_add[sources/alice] PASSED
tests/test_add.py::test_add[sources/bob]   FAILED

See source's full API for import_module, chdir, and the rest.

Reference

Options

Markers

Todo

• Support for --dist each and --dist worksteal.
• More efficient test collection. All tests are collected again per source spin-up.