pytest-prairielearn-grader

A pytest plugin for autograding Python code in PrairieLearn. Student code runs in isolated subprocess sandboxes with configurable security restrictions, timeouts, and detailed feedback.

Installation

pip install pytest-prairielearn-grader

For notebook grading support:

pip install pytest-prairielearn-grader[notebook]

Quick Start

See quick_start.md for a full tutorial with PrairieLearn integration examples.

Key Features

Sandboxed Execution

Student code runs in a separate subprocess via Unix sockets, providing:

• Security isolation from the grading harness
• Import whitelist/blacklist enforcement
• Builtin function restrictions
• Timeout enforcement
• Privilege dropping (Unix)

Student-Friendly Feedback Mode

Control how much information students see when their code fails. Use @pytest.mark.output(level=...) per test, or set output_level globally in your ConfigObject:

from pytest_prairielearn_grader import ConfigObject

autograder_config = ConfigObject(
    output_level="friendly",  # Global default for all tests
)

Output levels:

Per-test markers override the global setting:

@pytest.mark.output(level="friendly")
@pytest.mark.grading_data(name="Test add", points=5)
def test_add(sandbox: StudentFixture) -> None:
    assert_fn_equal(sandbox, "add", args=(2, 3), expected=5)

Assertion Helpers

Student-friendly assertion functions that produce clean, readable failure messages:

from pytest_prairielearn_grader.assertions import assert_equal, assert_fn_equal

@pytest.mark.output(level="friendly")
@pytest.mark.grading_data(name="Test calculation", points=5)
def test_calc(sandbox: StudentFixture) -> None:
    # Produces: "Checking: add(2, 3)\nExpected output: 5\nYour code output: 4\n..."
    assert_fn_equal(sandbox, "add", args=(2, 3), expected=5)

Available helpers:

• assert_equal(actual, expected) — compare any values
• assert_approx_equal(actual, expected, rtol=1e-5, atol=1e-8) — numeric comparison
• assert_fn_equal(sandbox, func_name, args=..., expected=...) — call + compare
• assert_fn_approx_equal(sandbox, func_name, args=..., expected=...) — call + approx compare
• assert_true(condition) / assert_false(condition) — boolean checks

Ungradable Submission Detection

When student code has a SyntaxError, the submission is automatically marked as ungradable ("gradable": false in results). This means the student does not lose a grading attempt and sees a clear error message.

# Opt out if you want syntax errors to score 0 instead:
autograder_config = ConfigObject(
    syntax_errors_ungradable=False,
)

The grader also detects test collection failures (grader-side errors) and marks those as ungradable with a message for course staff.

ConfigObject

Type-safe, immutable configuration for all autograder settings:

from pytest_prairielearn_grader import ConfigObject

autograder_config = ConfigObject(
    sandbox_timeout=2.0,
    import_whitelist=["numpy", "math"],
    builtin_whitelist=["len", "range", "sum"],
    output_level="friendly",
    syntax_errors_ungradable=True,
    starting_vars={"coefficient": 10},
    names_for_user=["coefficient"],
)

See ConfigObject documentation for all options.

Results JSON Format

The autograder produces autograder_results.json with the following structure:

{
    "gradable": true,
    "score": 0.85,
    "tests": [
        {
            "test_id": "test_student.py::test_function[student_code]",
            "name": "Test Function",
            "max_points": 5,
            "points": 5.0,
            "points_frac": 1.0,
            "outcome": "passed",
            "message": ""
        }
    ]
}

When a submission is ungradable:

{
    "gradable": false,
    "format_errors": ["SyntaxError: invalid syntax (line 3)"],
    "message": "Your code could not be parsed. Please fix the syntax errors and resubmit.",
    "tests": []
}

Development

# Install dev dependencies
pip install -e ".[dev]"

# Run tests
uv run pytest tests -p no:prairielearn-grader

# Run specific scenario
uv run pytest tests/test_autograder_scenarios.py -k "test_friendly_feedback" -v

# Format
ruff format src/ tests/

# Lint
ruff check src/ tests/ --fix

# Type check
mypy src/

Docker Images

• eliotwrobson/grader-python-pytest:latest — full image with numpy, pandas, matplotlib, sympy
• eliotwrobson/grader-python-pytest:lite — minimal image with core dependencies only

Related Issues

• PrairieLearn #11137 — Redesign of Python Autograder
• PrairieLearn #13739 — New Python autograder tracking issue
• PrairieLearn #12113 — Customizable external grader test case visibility
• PrairieLearn #9636 — Mark submission as ungradable
• PrairieLearn #14143 — OOM/SIGKILL handling