Skip to main content

pytest-grader

A pytest plugin for testing and scoring programming assignments.

Features

  • Assignment Scoring
    • Add point values to test functions using the @points(n) decorator
    • Show a score summary when running pytest --score
  • Test Locking as described in Basu et al., Automated Problem Clarification at Scale (abstract, pdf)
    • Lock doctests using the # LOCK comment before the function.
    • pytest-grader lock [src] [dst] will generate a copy of src with doctests locked.
    • pytest --unlock provides an interactive interface for unlocking locked doctests. Unlocking does not run any tests: a student unlocks before writing the code, so running the tests then would only show failures. Instead, every test is reported as skipped, and --score shows no score report. Run pytest again to test the code.
    • Doctests are ordinary doctests that pass under python3 -m doctest: an expected exception is written as its traceback, and a function value as its repr with ellipsis matching for the address, e.g. >>> make_adder(2) # doctest: +ELLIPSIS / <function make_adder.<locals>.adder at 0x...>.
    • Locking asks for what a student can predict: a traceback of any length is one answer, ERROR, and each function value is FUNCTION. When unlocking, type those (in any case). Directive comments are not shown. expected_outputs(example) gives the answers a locked example asks for, so tooling can show one blank per answer.
    • When unlocking, a string answer may be quoted with either single or double quotes (e.g. "hello" unlocks an expected 'hello'); the canonical form Python displays is recorded. An answer wrong only in its presence or absence of quotes is not accepted, but earns a hint saying so.
    • Unlocked outputs are saved in .unlocked.json (see --unlock-file) so that tests stay unlocked across pytest runs.
  • Test Isolation
    • Modules listed under reload_modules in grader.json are reloaded before each test, so a test that mutates a module (e.g. by monkeypatching one of its functions) does not affect later tests.
    • Globals injected by pytest's assertion rewriting (@py_builtins, @pytest_ar) are removed from doctest namespaces.
  • Test Timeouts
    • Each test (including each doctest) is limited to 10 seconds, so an infinite loop fails that test with a clear message instead of hanging the run. The remaining tests still run and are scored.
    • Adjust the limit with --timeout SECONDS; --timeout 0 disables it. The timeout is also disabled under --pdb.
    • Code blocked outside the Python interpreter (e.g. waiting on input() or a hung C call) cannot be interrupted; pure-Python loops always time out.

Usage

Include a conftest.py file in the distribution of your assignment that contains pytest_plugins = ["pytest_grader"].

Optionally describe the assignment in a grader.json file next to it:

{
  "reload_modules": ["hog"]
}

reload_modules lists modules reloaded before each test for isolation.

See the examples directory for more usage info.

License

MIT

Updating versions

  • Change version in pyproject.toml
  • uv build
  • uv publish If your pypi credentials are in ~/.pypirc, then instead run uvx uv-publish.

Metadata

Release files for pytest-grader 0.4.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pytest-grader 0.4.1
File Size Uploaded Provenance Actions
pytest_grader-0.4.1.tar.gz 22.8 kB Without provenance Details for pytest_grader-0.4.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pytest-grader 0.4.1
File Interpreter ABI Platform Actions
pytest_grader-0.4.1-py3-none-any.whl Python 3 none any Details for pytest_grader-0.4.1-py3-none-any.whl

Total release size: 38.1 kB

Release files / pytest_grader-0.4.1.tar.gz

Download URL pytest_grader-0.4.1.tar.gz
Size 22.8 kB
Tags Source
SHA-256 checksum
How to use checksums
eeefdf057e913405d141d1296295f7ce641dd6e2db496eb27c7a2d81c73f7b37
BLAKE2b-256 checksum
How to use checksums
6e21b6cb6aa46f3177fe4bfc568c65ca2a17226bf4f59692176ac34c39b4a4be
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.7.13

Release files / pytest_grader-0.4.1-py3-none-any.whl

Download URL pytest_grader-0.4.1-py3-none-any.whl
Size 15.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
5fad5019fcb4f3efa81c2017cc983cbfff2b9177e296fc59ad7d2933b1cffbac
BLAKE2b-256 checksum
How to use checksums
26550d26af1ed1f4cbae1bc78382982a544765dfd9ddf950e459a8e3c2074871
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.7.13

Security for pytest-grader 0.4.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page