CodEval

A Python utility to download student submissions to programming assignments from Canvas and GitHub and evaluate them using codeval scripts.

Team

• Sabira Abdolcader (sabdolc)
• Chelsie Chen (cChe1z)
• Aisha Syed (aisha-syed)
• Zarah Taufique (zarahtau)

Prerequisites

• Python 3.x
• Docker (for running evaluations in containers)
• A Canvas account with API access
• A GitHub account (for GitHub-based assignments)
• Optional AI provider packages: anthropic, openai, google-generativeai

Installation

# Install locally for development
pip install -e .

# Or install from PyPI
pip install assignment-codeval

# Install with AI provider support
pip install assignment-codeval[ai]

Configuration

Create a codeval.ini file with your Canvas and run settings:

[SERVER]
url=<canvas API>
token=<canvas token>
[RUN]
precommand=
command=

For distributed assignments, add:

[RUN]
dist_command=
host_ip=
[MONGO]
url=
db=

For AI benchmarking, add:

[AI]
anthropic_key=sk-ant-...
openai_key=sk-...
google_key=...

AI keys can also be set via environment variables: ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_API_KEY.

Refer to a sample config file here

Running the Application

# List all available commands
assignment-codeval --help

# Download submissions from Canvas/GitHub
assignment-codeval download-submissions <course_name>

# Evaluate downloaded submissions
assignment-codeval evaluate-submissions <course_name>

# Upload grading comments back to Canvas
assignment-codeval upload-submission-comments <course_name>

# Create an assignment on Canvas
assignment-codeval create-assignment <course_name> <specification_file>

# Check which submissions are missing grading
assignment-codeval check-grading <course_name>

# List recent codeval comments
assignment-codeval recent-comments <course_name>

# Test an assignment with AI models
assignment-codeval test-with-ai <codeval_file> [OPTIONS]

Usage

Codeval Specification Files

Codeval files (.codeval extension) define how to build, run, and test student submissions.

Assignment Description Tags

• ASSIGNMENT START <Assignment_name> — begins the Canvas assignment description in markdown (CRT_HW START also accepted, but deprecated)
• ASSIGNMENT END — ends the assignment description (CRT_HW END also accepted, but deprecated)

Specification Tags

Assignment Description Macros

Example Specification File

ASSIGNMENT START Hello World
# Hello World

## Problem Statement
Write a C program that reads a name from stdin and prints `Hello, <name>!`

## Submission
Submit your code to GITHUB_DIRECTORY

## Sample Examples
EXMPLS 2

## Discussion
DISCSN_URL

ASSIGNMENT END

# Compile the submission
C gcc -o hello hello.c

# Test case 1: greet "World"
T ./hello
I World
O Hello, World!

# Test case 2: greet "Alice"
T ./hello
I Alice
O Hello, Alice!

AI Benchmarking

Test assignments against multiple AI models:

# Test with all Anthropic models
assignment-codeval test-with-ai my_assignment.codeval -p anthropic

# Test with a specific model, 3 attempts
assignment-codeval test-with-ai my_assignment.codeval -m "Claude Sonnet 4" -n 3

Project Structure

src/assignment_codeval/
├── cli.py              # Click CLI entry point, registers all subcommands
├── submissions.py      # Download/upload submissions, list assignments
├── evaluate.py         # Run evaluation on submissions (run-evaluation)
├── create_assignment.py # Create assignments on Canvas
├── github_connect.py   # GitHub repository setup and integration
├── canvas_utils.py     # Canvas API utilities
├── ai_benchmark.py     # AI model testing (test-with-ai)
├── install_assignment.py # Install codeval files to local/remote destinations
├── recent_comments.py  # List recent codeval comments on Canvas
├── check_grading.py    # Check which submissions are missing grading
├── export_tests.py     # Export test cases from codeval files
├── convertMD2Html.py   # Markdown to HTML conversion
├── commons.py          # Shared utilities
├── file_utils.py       # File handling utilities
└── test_template.html  # HTML template for test results

tests/
├── unit/               # Unit tests and sample programs/codeval fixtures
├── integration/
│   └── test_codeval.py # Integration test suite
└── e2e/
    └── test_e2e.py     # End-to-end tests

samples/
├── codeval.ini               # Sample configuration file
└── assignment-name.codeval   # Sample specification file