Otter-Autograder 0.8.0

A collection of tools for grading homework automatically

Otter-Autograder

An autograding system for teaching, primarily focused on Canvas LMS integration. Supports automated grading of programming assignments (via Docker) and text submissions (like learning logs).

Installation

pip install Otter-Autograder

Quick Start

1. Set up Canvas API credentials

Create a .env file (by default this tool reads ~/.env):

CANVAS_API_KEY=your_canvas_api_key_here
CANVAS_API_URL=https://your-institution.instructure.com

2. Create a grading configuration

Create a YAML file (e.g., assignments.yaml) defining your courses and assignments:

privacy_mode: id_only  # none | id_only | blind
reveal_identity: false
idempotency_key: null  # Optional: set to skip re-pushing already pushed feedback
idempotency_state_dir: "~/.autograder/idempotency"  # Optional override

assignment_types:
  programming:
    kind: ProgrammingAssignment
    grader: template-grader
    settings:
      base_image_name: "your-docker-image"
      # Optional: mount extra repositories into specific container paths
      # additional_repos:
      #   - source_repo: "https://github.com/your-org/shared-tests"
      #     container_path: "/repo/shared-tests"
      container_repo_path: "/repo/programming-assignments"  # optional override; default shown
      record_retention: true
      records_dir: "~/autograder-records/your-course"  # required when record_retention=true

courses:
  - name: "Your Course"
    id: 12345
    assignment_groups:
      - type: programming
        assignments:
          - id: 67890
            repo_path: "PA1"

3. Run the grader

grade-assignments --yaml assignments.yaml

Use a specific env file:

grade-assignments --yaml assignments.yaml --env /path/to/.env

Temporarily include Canvas numeric IDs in logs (break-glass):

AUTOGRADER_BREAK_GLASS=1 grade-assignments --yaml assignments.yaml --reveal-identity

Idempotent push mode (safe rerun key):

grade-assignments --yaml assignments.yaml --idempotency-key spring26-ll2

Path safety defaults:

• record_retention: true requires an explicit absolute records_dir (or ~/...).
• records_dir is blocked if it points inside this git repo unless AUTOGRADER_ALLOW_IN_REPO_RECORDS=1.
• Idempotency state defaults to ~/.autograder/idempotency.

Features

Supported Assignment Types

• Programming Assignments: Docker-based grading with template matching and test execution
• Text Submissions: AI-powered grading with rubric generation and clustering analysis

Key Capabilities

• Parallel execution with configurable worker threads
• Privacy modes: none, id_only, blind
• Optional idempotent feedback push via idempotency_key
• Automatic score scaling to Canvas points
• Slack notifications for grading errors
• Record retention for audit trails
• Regrade support for existing submissions
• Test mode for validation before full grading runs

Usage Examples

Grade with limited submissions (testing)

grade-assignments --yaml config.yaml --limit 5

Regrade existing submissions

grade-assignments --yaml config.yaml --regrade

Test submissions without pushing grades

grade-assignments --yaml config.yaml --test

Control parallelism

grade-assignments --yaml config.yaml --max_workers 2

Show stage timings and push aggregates

grade-assignments --yaml config.yaml --show-stage-timings

Dry-run preflight (no grading)

grade-assignments --yaml config.yaml --dry-run

Dump effective merged assignment config

grade-assignments --yaml config.yaml --dump-config

Write a run report JSON

grade-assignments --yaml config.yaml --report ./run-report.json

Override Slack channel for run-level failure summaries

grade-assignments --yaml config.yaml --error-slack-channel C0123456789

Set custom idempotency state directory

grade-assignments --yaml config.yaml --idempotency-key spring26-ll2 --idempotency-state-dir ~/.autograder/state

Enable debug logging

grade-assignments --yaml config.yaml --debug

Configuration

See the example_files/ directory for complete configuration examples:

• workhorse.yaml: Recommended combined programming + text setup
• programming_assignments.yaml: Programming-only setup
• learning-logs.yaml: Text submission grading
• minimal-programming.yaml: Simplest programming assignment setup
• minimal-text.yaml: Simplest text assignment setup
• example-template.yaml: All available options

Requirements

• Python >= 3.12
• Docker (for programming assignment grading)
• Canvas API access
• Optional: OpenAI or Anthropic API keys for AI-powered features

Docker Security Boundaries

Programming submissions run in ephemeral Docker containers with baseline hardening:

• no-new-privileges:true
• explicit seccomp profile (Autograder/seccomp/autograder-seccomp.json by default)
• resource limits (mem_limit, nano_cpus, pids_limit)

Optional hardened mode:

• set AUTOGRADER_DOCKER_READ_ONLY_ROOT_FS=1 to use a read-only root filesystem (with writable tmpfs at /tmp and /var/tmp).

Override knobs (when needed for compatibility/performance):

• AUTOGRADER_DOCKER_SECCOMP_PROFILE (path to seccomp profile)
• AUTOGRADER_DOCKER_MEMORY_LIMIT (example: 1g)
• AUTOGRADER_DOCKER_NANO_CPUS (example: 2000000000 for 2 CPUs)
• AUTOGRADER_DOCKER_PIDS_LIMIT (example: 256)

Security note: this reduces risk but is not a complete sandbox against all kernel/container escape classes. Keep Docker and host OS patched.

Local Git Hygiene Hook (Recommended)

Install the repository-managed pre-commit hook so hygiene checks run before each commit:

bash scripts/install_git_hooks.sh

This hook runs scripts/check_version_bump_vendoring.sh and scripts/check_repo_hygiene.sh. On a staged pyproject.toml version bump, it auto-runs scripts/vendor_lms_interface.py, stages vendored updates, and asks for a second commit attempt after review. The installer also adds a repo-local alias so you can run git bump patch (or minor/major) to bump, vendor, stage, and commit without pathspec commits. git bump defaults to short vendoring summaries; use git bump patch --verbose for full vendoring logs.

Documentation

For detailed documentation, see:

• documentation/instructor_onboarding.md (minimal setup + common customizations)
• documentation/operations_runbook.md (failure autopsy + rerun procedures)
• documentation/troubleshooting.md (common runtime failures and recovery steps)
• documentation/architecture.md (system data flow and component relationships)
• documentation/customization.md (adding graders/kinds + common recipes)
• documentation/configuration_schema.md (field-by-field config reference)
• documentation/privacy_audit.md (PII surfaces and privacy controls)
• documentation/archives/step_by_step_grader_reference.md (archived grader concept for future redesign)
• documentation directory on GitHub

License

This project is licensed under the GPL-3.0-or-later license. See the LICENSE file for details.

Contributing

Contributions are welcome! Please open an issue or pull request on GitHub.