pytest-grader
A pytest plugin for testing and scoring programming assignments.
Features
- Assignment Scoring
- Add point values to test functions using the
@points(n)decorator - Show a score summary when running
pytest --score
- Add point values to test functions using the
- Test Locking as described in Basu et al., Automated Problem Clarification at Scale (abstract, pdf)
- Lock doctests using the
# LOCKcomment before the function. pytest-grader lock [src] [dst]will generate a copy of src with doctests locked.pytest --unlockprovides an interactive interface for unlocking locked doctests. Unlocking does not run any tests: a student unlocks before writing the code, so running the tests then would only show failures. Instead, every test is reported as skipped, and--scoreshows no score report. Runpytestagain to test the code.- Doctests are ordinary doctests that pass under
python3 -m doctest: an expected exception is written as its traceback, and a function value as its repr with ellipsis matching for the address, e.g.>>> make_adder(2) # doctest: +ELLIPSIS/<function make_adder.<locals>.adder at 0x...>. - Locking asks for what a student can predict: a traceback of any length is one
answer,
ERROR, and each function value isFUNCTION. When unlocking, type those (in any case). Directive comments are not shown.expected_outputs(example)gives the answers a locked example asks for, so tooling can show one blank per answer. - When unlocking, a string answer may be quoted with either single or double quotes
(e.g.
"hello"unlocks an expected'hello'); the canonical form Python displays is recorded. An answer wrong only in its presence or absence of quotes is not accepted, but earns a hint saying so. - Unlocked outputs are saved in
.unlocked.json(see--unlock-file) so that tests stay unlocked across pytest runs.
- Lock doctests using the
- Test Isolation
- Modules listed under
reload_modulesingrader.jsonare reloaded before each test, so a test that mutates a module (e.g. by monkeypatching one of its functions) does not affect later tests. - Globals injected by pytest's assertion rewriting (
@py_builtins,@pytest_ar) are removed from doctest namespaces.
- Modules listed under
- Test Timeouts
- Each test (including each doctest) is limited to 10 seconds, so an infinite loop fails that test with a clear message instead of hanging the run. The remaining tests still run and are scored.
- Adjust the limit with
--timeout SECONDS;--timeout 0disables it. The timeout is also disabled under--pdb. - Code blocked outside the Python interpreter (e.g. waiting on
input()or a hung C call) cannot be interrupted; pure-Python loops always time out.
Usage
Include a conftest.py file in the distribution of your assignment that contains pytest_plugins = ["pytest_grader"].
Optionally describe the assignment in a grader.json file next to it:
{
"reload_modules": ["hog"]
}
reload_modules lists modules reloaded before each test for isolation.
See the examples directory for more usage info.
License
Updating versions
- Change version in
pyproject.toml uv builduv publishIf your pypi credentials are in~/.pypirc, then instead runuvx uv-publish.
Metadata
Release files for pytest-grader 0.4.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pytest_grader-0.4.1.tar.gz | 22.8 kB | Without provenance | Details for pytest_grader-0.4.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pytest_grader-0.4.1-py3-none-any.whl | Python 3 | none | any | Details for pytest_grader-0.4.1-py3-none-any.whl |
Total release size: 38.1 kB
Release files / pytest_grader-0.4.1.tar.gz
| Download URL | pytest_grader-0.4.1.tar.gz |
|---|---|
| Size | 22.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
eeefdf057e913405d141d1296295f7ce641dd6e2db496eb27c7a2d81c73f7b37
|
|
BLAKE2b-256 checksum How to use checksums |
6e21b6cb6aa46f3177fe4bfc568c65ca2a17226bf4f59692176ac34c39b4a4be
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.7.13
|
Release files / pytest_grader-0.4.1-py3-none-any.whl
| Download URL | pytest_grader-0.4.1-py3-none-any.whl |
|---|---|
| Size | 15.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
5fad5019fcb4f3efa81c2017cc983cbfff2b9177e296fc59ad7d2933b1cffbac
|
|
BLAKE2b-256 checksum How to use checksums |
26550d26af1ed1f4cbae1bc78382982a544765dfd9ddf950e459a8e3c2074871
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.7.13
|
Security for pytest-grader 0.4.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.