Otter-Autograder

An autograding system for teaching, primarily focused on Canvas LMS integration. Supports automated grading of programming assignments (via Docker), text submissions (like learning logs), and external-tool sourced grades such as Panopto watch analytics.

Installation

pip install Otter-Autograder

Quick Start

1. Set up Canvas API credentials

Create ~/.tokens/autograder.env:

CANVAS_API_KEY=your_canvas_api_key_here
CANVAS_API_URL=https://your-institution.instructure.com
PANOPTO_CLIENT_ID=your_panopto_client_id_here
PANOPTO_CLIENT_SECRET=your_panopto_client_secret_here

To bootstrap or refresh the Panopto token, run this from a machine with a browser:

grade-assignments refresh-panopto-token

It will print an authorization URL, capture the redirect on 127.0.0.1:8765, and write the resulting refresh token to ~/.tokens/autograder.panopto.json by default.

2. Create a grading configuration

Create a YAML file (e.g., assignments.yaml) defining your courses and assignments:

privacy_mode: id_only  # none | id_only | blind
reveal_identity: false
idempotency_key: null  # Optional: set to skip re-pushing already pushed feedback
idempotency_state_dir: "~/.autograder/idempotency"  # Optional override

assignment_types:
  programming:
    kind: ProgrammingAssignment
    grader: template-grader
    schedule:
      timezone: America/New_York
      rrule: "FREQ=DAILY;BYHOUR=0,12;BYMINUTE=0;BYSECOND=0"
    settings:
      base_image_name: "your-docker-image"
      # Optional: mount extra repositories into specific container paths
      # additional_repos:
      #   - source_repo: "https://github.com/your-org/shared-tests"
      #     container_path: "/repo/shared-tests"
      container_repo_path: "/repo/programming-assignments"  # optional override; default shown
      record_retention: true
      records_dir: "~/autograder-records/your-course"  # required when record_retention=true

courses:
  - name: "Your Course"
    id: 12345
    assignment_groups:
      - type: programming
        assignments:
          - id: 67890
            repo_path: "PA1"

For Panopto watch grading, put the shared Panopto base URL in the assignment type settings and the per-video session id on each assignment:

assignment_types:
  panopto_watch:
    kind: ExternalToolAssignment
    grader: panopto-watch-grader
    settings:
      panopto_base: "https://csumb.hosted.panopto.com/Panopto/"
      panopto_refresh_token_path: "~/.tokens/autograder.panopto.json"
      canvas_user_attribute: "sis_user_id"
      external_user_attribute: "username"
      # Set true to delete all prior Canvas submission comments before posting.
      clobber_feedback: false

courses:
  - name: "Course 35631"
    id: 35631
    assignment_groups:
      - type: panopto_watch
        assignments:
          - id: 601506
            panopto_id: "4992fd5c-fb07-4ddd-8fb2-b30400158378"

3. Run the grader

grade-assignments --yaml assignments.yaml

Use a specific env file:

grade-assignments --yaml assignments.yaml --env /path/to/credentials.env

Temporarily include Canvas numeric IDs in logs (break-glass):

AUTOGRADER_BREAK_GLASS=1 grade-assignments --yaml assignments.yaml --reveal-identity

Idempotent push mode (safe rerun key):

grade-assignments --yaml assignments.yaml --idempotency-key spring26-ll2

Path safety defaults:

• record_retention: true requires an explicit absolute records_dir (or ~/...).
• records_dir is blocked if it points inside this git repo unless AUTOGRADER_ALLOW_IN_REPO_RECORDS=1.
• Idempotency state defaults to ~/.autograder/idempotency.
• Schedule state defaults to ~/.autograder/schedule_state.yaml. The directory is created automatically. Set AUTOGRADER_SCHEDULE_STATE_PATH to override it.

Features

Supported Assignment Types

• Programming Assignments: Docker-based grading with template matching and test execution
• Text Submissions: AI-powered grading with rubric generation and clustering analysis
• External Tool Assignments: synthesize grades from external systems such as Panopto watch progress

Key Capabilities

• Parallel execution with configurable worker threads
• Privacy modes: none, id_only, blind
• Optional idempotent feedback push via idempotency_key
• Automatic score scaling to Canvas points
• Slack notifications for grading errors
• Record retention for audit trails
• Regrade support for existing submissions
• Test mode for validation before full grading runs

Usage Examples

Grade with limited submissions (testing)

grade-assignments --yaml config.yaml --limit 5

Regrade existing submissions

grade-assignments --yaml config.yaml --regrade

Regrade a single student submission

grade-assignments --yaml config.yaml --regrade --student-id 123456

Test submissions without pushing grades

grade-assignments --yaml config.yaml --test

Control parallelism

grade-assignments --yaml config.yaml --max_workers 2

Show stage timings and push aggregates

grade-assignments --yaml config.yaml --show-stage-timings

Dry-run preflight (no grading)

grade-assignments --yaml config.yaml --dry-run

Dump effective merged assignment config

grade-assignments --yaml config.yaml --dump-config

Write a run report JSON

grade-assignments --yaml config.yaml --report ./run-report.json

Override Slack channel for run-level failure summaries

grade-assignments --yaml config.yaml --error-slack-channel C0123456789

Test Slack notifications

Send a test message using the configured run-summary channel without accessing Canvas or grading submissions. --error-slack-channel can be used to test a different channel.

grade-assignments --yaml config.yaml --test-slack

Set custom idempotency state directory

grade-assignments --yaml config.yaml --idempotency-key spring26-ll2 --idempotency-state-dir ~/.autograder/state

Enable debug logging

grade-assignments --yaml config.yaml --debug

Configuration

See the example_files/ directory for complete configuration examples:

• workhorse.yaml: Recommended combined programming + text setup
• programming_assignments.yaml: Programming-only setup
• learning-logs.yaml: Text submission grading
• minimal-external.yaml: Simplest Panopto-backed external assignment setup
• minimal-programming.yaml: Simplest programming assignment setup
• minimal-text.yaml: Simplest text assignment setup
• example-template.yaml: All available options

Requirements

• Python >= 3.12
• Docker (for programming assignment grading)
• Canvas API access
• Optional: OpenAI or Anthropic API keys for AI-powered features

Docker Security Boundaries

Programming submissions run in ephemeral Docker containers with baseline hardening:

• no-new-privileges:true
• explicit seccomp profile (Autograder/seccomp/autograder-seccomp.json by default)
• resource limits (mem_limit, nano_cpus, pids_limit)

Optional hardened mode:

• set AUTOGRADER_DOCKER_READ_ONLY_ROOT_FS=1 to use a read-only root filesystem (with writable tmpfs at /tmp and /var/tmp).

Override knobs (when needed for compatibility/performance):

• AUTOGRADER_DOCKER_SECCOMP_PROFILE (path to seccomp profile)
• AUTOGRADER_DOCKER_MEMORY_LIMIT (example: 1g)
• AUTOGRADER_DOCKER_NANO_CPUS (example: 2000000000 for 2 CPUs)
• AUTOGRADER_DOCKER_PIDS_LIMIT (example: 256)

Security note: this reduces risk but is not a complete sandbox against all kernel/container escape classes. Keep Docker and host OS patched.

Local Git Hygiene Hook (Recommended)

Install the repository-managed pre-commit hook so hygiene checks run before each commit:

bash scripts/install_git_hooks.sh

This hook runs scripts/check_repo_hygiene.sh. The installer also adds a repo-local alias so you can run git bump patch (or minor/major) to bump, stage, commit, tag, and push. LMSInterface is installed from PyPI as the otterden-lms-interface dependency declared in pyproject.toml.

Documentation

For detailed documentation, see:

• documentation/instructor_onboarding.md (minimal setup + common customizations)
• documentation/operations_runbook.md (failure autopsy + rerun procedures)
• documentation/troubleshooting.md (common runtime failures and recovery steps)
• documentation/architecture.md (system data flow and component relationships)
• documentation/customization.md (adding graders/kinds + common recipes)
• documentation/configuration_schema.md (field-by-field config reference)
• documentation/privacy_audit.md (PII surfaces and privacy controls)
• documentation/archives/step_by_step_grader_reference.md (archived grader concept for future redesign)
• documentation directory on GitHub

License

This project is licensed under the GPL-3.0-or-later license. See the LICENSE file for details.

Contributing

Contributions are welcome! Please open an issue or pull request on GitHub.