pl-e2e-test

pl-e2e-test fuzzes PrairieLearn v3 questions through their real Python generate, prepare, render, parse, grade, test, and file phases. It was extracted from the pl-oer-lv102 course so that courses can install one uv-managed tool instead of copying the harness and its support scripts.

The package installs the reusable prairielearn_e2e Python API and a single pl-e2e-test command:

• pl-e2e-test (no command, or any pytest/harness options) runs the installed lifecycle pytest suite.
• pl-e2e-test diff fuzzes questions selected from a Git diff.
• pl-e2e-test parallel runs local shards and merges their results.
• pl-e2e-test add-seed records a failing concrete variant seed.
• pl-e2e-test reports merges parallel shard results into terminal and JUnit reports.
• pl-e2e-test version prints the installed version.
• pl-e2e-test help [COMMAND] prints top-level help, or one command's help.

Requirements and installation

• Python 3.13 or newer
• Git 2.30 or newer

The harness depends on pytest, Pyright, Ruff, Chevron, JSON Schema, lxml, NumPy, and PrairieLearn. PrairieLearn's Python package is currently consumed from its Git repository, so the course project must tell uv where to resolve that dependency:

[dependency-groups]
dev = ["pl-e2e-test"]

[tool.uv.sources.prairielearn]
git = "https://github.com/PrairieLearn/PrairieLearn.git"
subdirectory = "apps/prairielearn/python"

Then lock and install the environment:

uv lock
uv sync

For local development of this package, use a path source in the consuming project:

[tool.uv.sources.pl-e2e-test]
path = "../pl-e2e-test"
editable = true

Running lifecycle tests

Pass a question, a directory of questions, or an info.json file. Repeat --question-path to select multiple roots:

uv run pl-e2e-test \
  --question-path questions \
  --seed-count 3

The installed suite exposes separate generate, prepare, render, and grade pytest markers. Any remaining arguments are passed to pytest:

uv run pl-e2e-test \
  --question-path questions/chapter/example \
  --fuzz-seeds -m render -x -vv

--fuzz-seeds creates and reports a random 64-bit master seed. Reproduce the same variant set with --fuzz-seed N. Without either option, deterministic seeds start at zero. Question metadata with singleVariant: true always uses seed zero.

Local parallel runs

Run multiple shards in one machine or CI job, with six concurrent processes by default:

uv run pl-e2e-test parallel --shard-count 6 run -- \
  --question-path questions --fuzz-seeds

uv run pl-e2e-test parallel --shard-count 6 --junitxml results.xml diff -- \
  --base main --fuzz-seed 42

Launcher options go before the required run or diff selector; arguments after -- are forwarded to that command. Normal runs remain deterministic. For --fuzz-seeds and diff runs, the launcher generates one shared master seed unless --fuzz-seed is supplied, and prints a command for replaying the run without sharding.

Each shard runs in a separate Python process using the current environment and working directory. Terminals show Rich progress; redirected output prints completion lines. All shards finish before reports are merged, even when a test fails. Captured failure output and successful stderr are shown before the combined summary. Interrupting the launcher stops and reaps its children.

Intermediate reports are temporary. Pass the launcher's --junitxml option to retain merged JUnit output. Forwarded --shard, --shard-report, --junitxml, and --junit-xml options are rejected because the launcher manages those outputs. Choose the course with the working directory, or course_root in the Python API.

The same orchestration is available to Python callers:

from prairielearn_e2e import run_parallel

status = run_parallel(
    ["--question-path", "questions", "--fuzz-seeds"],
    command="run",
    shard_count=6,
    course_root="/path/to/course",
    junitxml="results.xml",
)

run_parallel returns an integer exit status, including 130 on interruption. Relative JUnit paths are resolved against the course root. With junitxml=None, merged output is temporary. Invalid launcher settings raise ValueError.

Courses using scripts/run_e2e_shards.py can replace its invocation with pl-e2e-test parallel, move --fuzz-seed into the forwarded arguments, and explicitly pass --fuzz-seeds for full-suite runs to preserve the script's fuzzing behavior. For example:

test-e2e-sharded:
	uv run --active pl-e2e-test parallel --shard-count "$(E2E_SHARD_COUNT)" run -- \
		$(E2E_TEST_ARGS) $(E2E_PRAIRIELEARN_ARG) --fuzz-seeds $(E2E_FUZZ_SEED_ARG) $(PYTEST_ARGS)

test-e2e-fuzz-sharded:
	uv run --active pl-e2e-test parallel --shard-count "$(E2E_SHARD_COUNT)" diff -- \
		--base "$(E2E_DIFF_BASE)" --head "$(E2E_DIFF_HEAD)" \
		--seed-count "$(E2E_FUZZ_SEED_COUNT)" $(E2E_FUZZ_SEED_ARG) $(E2E_PRAIRIELEARN_ARG) $(PYTEST_ARGS)

CI sharding

Use --shard INDEX/COUNT to divide the selected question variants between parallel jobs. The index is one-based. All selected lifecycle checks for one question and seed stay together, and variants are assigned round-robin so shard sizes differ by at most one. Pytest filters such as -m and -k are applied before sharding.

Add --shard-report PATH to write a mergeable result blob. Every shard must use the same checkout and arguments. Sharded fuzz runs must also use one explicit shared --fuzz-seed; automatic random fuzz seeds are rejected. After downloading all blobs, merge them into a combined terminal summary and JUnit report:

uv run pl-e2e-test reports e2e-reports \
  --junitxml pl-e2e-results.xml

The merger rejects missing or duplicate shards, mismatched test collections, mixed fuzz seeds, corrupt reports, and tests left unexecuted by interruption or -x. It exits unsuccessfully when the aggregate suite failed or was incomplete.

For example, a course can use this GitHub Actions matrix:

jobs:
  e2e:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v9
        with:
          enable-cache: true
      - run: uv python install && uv sync --locked
      - name: Run shard
        run: |
          uv run pl-e2e-test \
            --question-path questions \
            --seed-count 10 \
            --fuzz-seed "${{ github.run_id }}" \
            --shard "${{ matrix.shard }}/4" \
            --shard-report "e2e-reports/shard-${{ matrix.shard }}.pl-e2e-report.json"
      - name: Upload shard report
        if: ${{ !cancelled() }}
        uses: actions/upload-artifact@v4
        with:
          name: e2e-report-${{ matrix.shard }}
          path: e2e-reports/
          if-no-files-found: error
          retention-days: 1

  e2e-report:
    if: ${{ !cancelled() }}
    needs: e2e
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v9
      - run: uv python install && uv sync --locked
      - uses: actions/download-artifact@v4
        with:
          pattern: e2e-report-*
          path: e2e-reports
          merge-multiple: true
      - name: Merge reports
        run: |
          uv run pl-e2e-test reports e2e-reports \
            --junitxml pl-e2e-results.xml
      - name: Upload merged JUnit report
        if: ${{ always() }}
        uses: actions/upload-artifact@v4
        with:
          name: pl-e2e-results
          path: pl-e2e-results.xml
          if-no-files-found: warn

The course must provide .prairielearn/schemas/infoQuestion.json. The harness checks question metadata against that schema and uses the exact PrairieLearn revision installed in the uv environment. It stores a bare, partial PrairieLearn repository in the platform-standard user cache directory. The first use of a new Git source or revision requires network access; later runs reuse that exact revision offline. The uv lockfile, rather than the cache, decides which revision is tested.

To use an existing PrairieLearn worktree or bare repository instead, pass --prairielearn-path /path/to/PrairieLearn. The repository must already contain the installed revision. An explicit path is authoritative: the harness does not fetch into it or fall back to the managed cache. In either mode, only the pinned revision's Python element controllers are archived into the test run's temporary directory. Standard cache environment settings such as XDG_CACHE_HOME can relocate the managed cache; CI systems can persist that directory between runs if desired.

Harness configuration

Selector settings are read automatically from the course's pyproject.toml:

[tool.pl-e2e-test]
elementSolutionSelectors = ["pl-sketch"]
nonGradableSelectors = ['pl-sketch[read-only="true"]', 'pl-sketch[read-only="1"]']

These are the defaults. elementSolutionSelectors identifies elements that provide solutions without a correct_answers entry. nonGradableSelectors excludes display-only answers from grading checks. Each supplied list replaces its default; omitted settings keep their defaults, and [] disables a list.

Use --config path/to/settings.toml to select another TOML file with the same [tool.pl-e2e-test] table. This works with normal runs, diff, and arguments after parallel run or parallel diff. Explicit paths are relative to the working directory.

Question-local configuration

Store saved concrete seeds and narrow phase skips in .question-e2e.json beside the question's info.json:

{
  "regression_seeds": [1843927501],
  "skip_methods": {
    "grade": "Temporarily blocked by an upstream issue"
  }
}

Saved regression seeds run before newly generated seeds. Skips require a nonempty reason and may name only generate, prepare, render, or grade. Manual-grading questions automatically skip only the grade check.

Record the concrete seed shown in a failing pytest case with:

uv run pl-e2e-test add-seed \
  questions/chapter/example 1843927501

The command atomically creates or updates .question-e2e.json, preserves other settings, and sorts and deduplicates the seed list.

Diff-based fuzzing

The diff runner tests changed question directories. A non-ignored changed file outside questions/ selects every question:

uv run pl-e2e-test diff \
  --base origin/main \
  --head HEAD \
  --seed-count 10

Configure optional repository-relative ignore rules in the course's pyproject.toml:

[tool.pl-e2e-test]
diffignore = [
  ".github",
  "README.md",
  "*lock.yaml",
]

Rules use Git's native ignore syntax and are evaluated in order, so later rules may negate earlier ones with !. A missing pyproject.toml, tool table, or diffignore entry means that no paths are ignored. Additional arguments are forwarded to pytest.

To migrate from .question-e2e.diffignore, copy its non-comment rules into the diffignore array. The former file and --e2ediffignore option are no longer used.

In pull-request CI, fetch the base commit before running the command and pass the appropriate base and head SHAs. Set --fuzz-seed to replay a reported run. pl-e2e-test diff accepts the same --shard and --shard-report options. If a diff selects no questions, it writes a successful empty shard report so the merge job can still verify that every matrix job completed.

Python API

The original API remains available:

from prairielearn_e2e import PrairieLearnBackend, QuestionCase, QuestionHarness

QuestionHarness.run_generate, run_prepare, run_render, and run_grade execute increasingly complete portions of the lifecycle. run_variant remains an alias for the complete grade path.

Pass config=load_config(course_root=course_root) to QuestionHarness to load the course's TOML settings, or config=load_config("settings.toml") for an explicit file. load_config and HarnessConfig are exported from prairielearn_e2e. Omitting config uses the built-in defaults.

Development

This project uses uv 0.9 or newer:

uv sync
make test
make format
make smoke-dist

make test runs pytest, Pyright, Ruff linting, and Ruff's formatting check. make smoke-dist builds both distribution formats, installs each without dependencies into an isolated target, and imports it using the locked development dependencies.

Publishing

There are three supported publishing paths. Choose one for a release; do not run the local and GitHub paths for the same version because PyPI versions are immutable.

GitHub-hosted trusted publishing

Configure the PyPI project to trust the publish.yml workflow in the GitHub pypi environment. From a clean branch, run:

make publish-version VERSION=0.2.0

This runs the full checks, updates pyproject.toml and uv.lock, builds the distributions, creates a release commit and annotated v0.2.0 tag, and atomically pushes the branch and tag. The tag starts the workflow on ubuntu-latest; the workflow rechecks the tag/version match, tests, builds, generates attestations, and publishes with PyPI trusted-publishing OIDC credentials. Use REMOTE=name for a different Git remote.

If an unpushed commit already contains the requested version, use the guarded bypass:

make publish-version VERSION=0.2.0 ALLOW_PREBUMPED_VERSION=1

Direct local publishing

To publish the version already present in pyproject.toml directly from this machine, configure a uv-supported PyPI credential such as UV_PUBLISH_TOKEN, then run:

make publish-local

This runs all checks, builds and smoke-tests the current wheel and source archive, and passes only those two version-matched files to uv publish. It does not create a Git commit or tag.

Local self-hosted GitHub runner

The same trusted-publishing workflow can run on a repository-scoped Linux ARM64 runner inside Docker. The container has no host mounts and does not receive the Docker socket. Authenticate gh with repository administration permission, then run:

make runner-start
make publish-self-hosted RELEASE_TAG=v0.2.0

The tag must already exist. This dispatches publish.yml from main with the self-hosted runner choice. make runner-ensure starts Docker Desktop on macOS when needed and repairs the runner before dispatch. Other lifecycle commands are runner-status, runner-stop, runner-remove, runner-logs, and clean-runner.