pl-e2e-test

pl-e2e-test fuzzes PrairieLearn v3 questions through their real Python generate, prepare, render, parse, grade, test, and file phases. It was extracted from the pl-oer-lv102 course so that courses can install one uv-managed tool instead of copying the harness and its support scripts.

The package installs the reusable prairielearn_e2e Python API and these commands:

• pl-e2e-test runs the installed lifecycle pytest suite.
• pl-e2e-test-diff fuzzes questions selected from a Git diff.
• pl-e2e-add-regression-seed records a failing concrete variant seed.
• pl-e2e-merge-reports merges parallel shard results into terminal and JUnit reports.

Requirements and installation

• Python 3.13 or newer
• Git 2.30 or newer

The harness depends on pytest, Pyright, Ruff, Chevron, JSON Schema, lxml, NumPy, and PrairieLearn. PrairieLearn's Python package is currently consumed from its Git repository, so the course project must tell uv where to resolve that dependency:

[dependency-groups]
dev = ["pl-e2e-test"]

[tool.uv.sources.prairielearn]
git = "https://github.com/PrairieLearn/PrairieLearn.git"
subdirectory = "apps/prairielearn/python"

Then lock and install the environment:

uv lock
uv sync

For local development of this package, use a path source in the consuming project:

[tool.uv.sources.pl-e2e-test]
path = "../pl-e2e-test"
editable = true

Running lifecycle tests

Pass a question, a directory of questions, or an info.json file. Repeat --question-path to select multiple roots:

uv run pl-e2e-test \
  --question-path questions \
  --seed-count 3

The installed suite exposes separate generate, prepare, render, and grade pytest markers. Any remaining arguments are passed to pytest:

uv run pl-e2e-test \
  --question-path questions/chapter/example \
  --fuzz-seeds -m render -x -vv

--fuzz-seeds creates and reports a random 64-bit master seed. Reproduce the same variant set with --fuzz-seed N. Without either option, deterministic seeds start at zero. Question metadata with singleVariant: true always uses seed zero.

CI sharding

Use --shard INDEX/COUNT to divide the selected question variants between parallel jobs. The index is one-based. All selected lifecycle checks for one question and seed stay together, and variants are assigned round-robin so shard sizes differ by at most one. Pytest filters such as -m and -k are applied before sharding.

Add --shard-report PATH to write a mergeable result blob. Every shard must use the same checkout and arguments. Sharded fuzz runs must also use one explicit shared --fuzz-seed; automatic random fuzz seeds are rejected. After downloading all blobs, merge them into a combined terminal summary and JUnit report:

uv run pl-e2e-merge-reports e2e-reports \
  --junitxml pl-e2e-results.xml

The merger rejects missing or duplicate shards, mismatched test collections, mixed fuzz seeds, corrupt reports, and tests left unexecuted by interruption or -x. It exits unsuccessfully when the aggregate suite failed or was incomplete.

For example, a course can use this GitHub Actions matrix:

jobs:
  e2e:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v9
        with:
          enable-cache: true
      - run: uv python install && uv sync --locked
      - name: Run shard
        run: |
          uv run pl-e2e-test \
            --question-path questions \
            --seed-count 10 \
            --fuzz-seed "${{ github.run_id }}" \
            --shard "${{ matrix.shard }}/4" \
            --shard-report "e2e-reports/shard-${{ matrix.shard }}.pl-e2e-report.json"
      - name: Upload shard report
        if: ${{ !cancelled() }}
        uses: actions/upload-artifact@v4
        with:
          name: e2e-report-${{ matrix.shard }}
          path: e2e-reports/
          if-no-files-found: error
          retention-days: 1

  e2e-report:
    if: ${{ !cancelled() }}
    needs: e2e
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v9
      - run: uv python install && uv sync --locked
      - uses: actions/download-artifact@v4
        with:
          pattern: e2e-report-*
          path: e2e-reports
          merge-multiple: true
      - name: Merge reports
        run: |
          uv run pl-e2e-merge-reports e2e-reports \
            --junitxml pl-e2e-results.xml
      - name: Upload merged JUnit report
        if: ${{ always() }}
        uses: actions/upload-artifact@v4
        with:
          name: pl-e2e-results
          path: pl-e2e-results.xml
          if-no-files-found: warn

The course must provide .prairielearn/schemas/infoQuestion.json. The harness checks question metadata against that schema and uses the exact PrairieLearn revision installed in the uv environment. It stores a bare, partial PrairieLearn repository in the platform-standard user cache directory. The first use of a new Git source or revision requires network access; later runs reuse that exact revision offline. The uv lockfile, rather than the cache, decides which revision is tested.

To use an existing PrairieLearn worktree or bare repository instead, pass --prairielearn-path /path/to/PrairieLearn. The repository must already contain the installed revision. An explicit path is authoritative: the harness does not fetch into it or fall back to the managed cache. In either mode, only the pinned revision's Python element controllers are archived into the test run's temporary directory. Standard cache environment settings such as XDG_CACHE_HOME can relocate the managed cache; CI systems can persist that directory between runs if desired.

Question-local configuration

Store saved concrete seeds and narrow phase skips in .question-e2e.json beside the question's info.json:

{
  "regression_seeds": [1843927501],
  "skip_methods": {
    "grade": "Temporarily blocked by an upstream issue"
  }
}

Saved regression seeds run before newly generated seeds. Skips require a nonempty reason and may name only generate, prepare, render, or grade. Manual-grading questions automatically skip only the grade check.

Record the concrete seed shown in a failing pytest case with:

uv run pl-e2e-add-regression-seed \
  questions/chapter/example 1843927501

The command atomically creates or updates .question-e2e.json, preserves other settings, and sorts and deduplicates the seed list.

Diff-based fuzzing

The diff runner tests changed question directories. A non-ignored changed file outside questions/ selects every question:

uv run pl-e2e-test-diff \
  --base origin/main \
  --head HEAD \
  --seed-count 10

Optional repository-relative ignore rules live in .question-e2e.diffignore and use Git's native ignore syntax. Missing default configuration means that no paths are ignored. Pass --e2ediffignore PATH to require and use another file. Additional arguments are forwarded to pytest.

In pull-request CI, fetch the base commit before running the command and pass the appropriate base and head SHAs. Set --fuzz-seed to replay a reported run. pl-e2e-test-diff accepts the same --shard and --shard-report options. If a diff selects no questions, it writes a successful empty shard report so the merge job can still verify that every matrix job completed.

Python API

The original API remains available:

from prairielearn_e2e import PrairieLearnBackend, QuestionCase, QuestionHarness

QuestionHarness.run_generate, run_prepare, run_render, and run_grade execute increasingly complete portions of the lifecycle. run_variant remains an alias for the complete grade path.

Development

This project uses uv 0.9 or newer:

uv sync
make test
make format
make smoke-dist

make test runs pytest, Pyright, Ruff linting, and Ruff's formatting check. make smoke-dist builds both distribution formats, installs each without dependencies into an isolated target, and imports it using the locked development dependencies.

Publishing

There are three supported publishing paths. Choose one for a release; do not run the local and GitHub paths for the same version because PyPI versions are immutable.

GitHub-hosted trusted publishing

Configure the PyPI project to trust the publish.yml workflow in the GitHub pypi environment. From a clean branch, run:

make publish-version VERSION=0.2.0

This runs the full checks, updates pyproject.toml and uv.lock, builds the distributions, creates a release commit and annotated v0.2.0 tag, and atomically pushes the branch and tag. The tag starts the workflow on ubuntu-latest; the workflow rechecks the tag/version match, tests, builds, generates attestations, and publishes with PyPI trusted-publishing OIDC credentials. Use REMOTE=name for a different Git remote.

If an unpushed commit already contains the requested version, use the guarded bypass:

make publish-version VERSION=0.2.0 ALLOW_PREBUMPED_VERSION=1

Direct local publishing

To publish the version already present in pyproject.toml directly from this machine, configure a uv-supported PyPI credential such as UV_PUBLISH_TOKEN, then run:

make publish-local

This runs all checks, builds and smoke-tests the current wheel and source archive, and passes only those two version-matched files to uv publish. It does not create a Git commit or tag.

Local self-hosted GitHub runner

The same trusted-publishing workflow can run on a repository-scoped Linux ARM64 runner inside Docker. The container has no host mounts and does not receive the Docker socket. Authenticate gh with repository administration permission, then run:

make runner-start
make publish-self-hosted RELEASE_TAG=v0.2.0

The tag must already exist. This dispatches publish.yml from main with the self-hosted runner choice. make runner-ensure starts Docker Desktop on macOS when needed and repairs the runner before dispatch. Other lifecycle commands are runner-status, runner-stop, runner-remove, runner-logs, and clean-runner.