CodEval
A Python utility to download student submissions to programming assignments from Canvas and GitHub and evaluate them using codeval scripts.
Team
- Sabira Abdolcader (sabdolc)
- Chelsie Chen (cChe1z)
- Aisha Syed (aisha-syed)
- Zarah Taufique (zarahtau)
Prerequisites
- Python 3.x
- Docker (for running evaluations in containers)
- A Canvas account with API access
- A GitHub account (for GitHub-based assignments)
- Optional AI provider packages:
anthropic,openai,google-generativeai
Installation
# Install locally for development
pip install -e .
# Or install from PyPI
pip install assignment-codeval
# Install with AI provider support
pip install assignment-codeval[ai]
Configuration
Create a codeval.ini file with your Canvas and run settings:
[SERVER]
url=<canvas API>
token=<canvas token>
[RUN]
precommand=
command=
For distributed assignments, add:
[RUN]
dist_command=
host_ip=
[MONGO]
url=
db=
For AI benchmarking, add:
[AI]
anthropic_key=sk-ant-...
openai_key=sk-...
google_key=...
AI keys can also be set via environment variables: ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_API_KEY.
Refer to a sample config file here
Running the Application
# List all available commands
assignment-codeval --help
# Download submissions from Canvas/GitHub
assignment-codeval download-submissions <course_name>
# Evaluate downloaded submissions
assignment-codeval evaluate-submissions <course_name>
# Upload grading comments back to Canvas
assignment-codeval upload-submission-comments <course_name>
# Create an assignment on Canvas
assignment-codeval create-assignment <course_name> <specification_file>
# Check which submissions are missing grading
assignment-codeval check-grading <course_name>
# List recent codeval comments
assignment-codeval recent-comments <course_name>
# Test an assignment with AI models
assignment-codeval test-with-ai <codeval_file> [OPTIONS]
Usage
Codeval Specification Files
Codeval files (.codeval extension) define how to build, run, and test student submissions.
Assignment Description Tags
ASSIGNMENT START <Assignment_name>— begins the Canvas assignment description in markdown (CRT_HW STARTalso accepted, but deprecated)ASSIGNMENT END— ends the assignment description (CRT_HW ENDalso accepted, but deprecated)
Specification Tags
| Tag | Meaning | Function |
|---|---|---|
| C | Compile Code | Specifies the command to compile the submission code |
| CTO | Compile Timeout | Timeout in seconds for the whole evaluation (compile and all test cases) to run |
| T/HT | Test Case | Command to run to test the submission (HT = hidden test) |
| I/IB/IF | Supply Input | Input for a test case. I adds a newline, IB does not, IF reads from a file |
| O/OB/OF | Check Output | Expected output. O adds a newline, OB does not, OF reads from a file |
| E/EB | Check Error | Expected error output. E adds a newline, EB does not |
| TO | Timeout | Time limit in seconds for a test case (default: 10) |
| X | Exit Code | Expected exit code for a test case (not checked unless X is specified) |
| TEMP | Temp File | Registers a file to be deleted before and after the next test run |
| CF | Check Function | Checks that a function is used in the submission |
| NCF | Check Not Function | Checks that a function is not used |
| CMD/TCMD | Run Command | Runs a command. TCMD fails evaluation if the command errors |
| Print Label | Prints a label/message to stdout | |
| CMP | Compare | Compares two files |
| Z | Download Zip | Zip files to download from Canvas for test cases |
Assignment Description Macros
| Macro | Replacement |
|---|---|
DISCSN_URL |
URL of the discussion created for the assignment |
EXMPLS <n> |
First n non-hidden test cases formatted for display |
FILE[file_name] |
Link to the specified file in the Codeval folder |
COMPILE |
Compile command from the C tag |
GITHUB_DIRECTORY |
GitHub directory for submission |
Example Specification File
ASSIGNMENT START Hello World
# Hello World
## Problem Statement
Write a C program that reads a name from stdin and prints `Hello, <name>!`
## Submission
Submit your code to GITHUB_DIRECTORY
## Sample Examples
EXMPLS 2
## Discussion
DISCSN_URL
ASSIGNMENT END
# Compile the submission
C gcc -o hello hello.c
# Test case 1: greet "World"
T ./hello
I World
O Hello, World!
# Test case 2: greet "Alice"
T ./hello
I Alice
O Hello, Alice!
AI Benchmarking
Test assignments against multiple AI models:
# Test with all Anthropic models
assignment-codeval test-with-ai my_assignment.codeval -p anthropic
# Test with a specific model, 3 attempts
assignment-codeval test-with-ai my_assignment.codeval -m "Claude Sonnet 4" -n 3
| Option | Description |
|---|---|
-o, --output-dir |
Directory to store solutions and results (default: ai_test_results) |
-n, --attempts |
Number of attempts per model (default: 1) |
-m, --models |
Specific models to test (repeatable) |
-p, --providers |
Filter by provider: anthropic, openai, google |
Project Structure
src/assignment_codeval/
├── cli.py # Click CLI entry point, registers all subcommands
├── submissions.py # Download/upload submissions, list assignments
├── evaluate.py # Run evaluation on submissions (run-evaluation)
├── create_assignment.py # Create assignments on Canvas
├── github_connect.py # GitHub repository setup and integration
├── canvas_utils.py # Canvas API utilities
├── ai_benchmark.py # AI model testing (test-with-ai)
├── install_assignment.py # Install codeval files to local/remote destinations
├── recent_comments.py # List recent codeval comments on Canvas
├── check_grading.py # Check which submissions are missing grading
├── export_tests.py # Export test cases from codeval files
├── convertMD2Html.py # Markdown to HTML conversion
├── commons.py # Shared utilities
├── file_utils.py # File handling utilities
└── test_template.html # HTML template for test results
tests/
├── unit/ # Unit tests and sample programs/codeval fixtures
├── integration/
│ └── test_codeval.py # Integration test suite
└── e2e/
└── test_e2e.py # End-to-end tests
samples/
├── codeval.ini # Sample configuration file
└── assignment-name.codeval # Sample specification file
Metadata
Release files for assignment-codeval 0.0.35
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| assignment_codeval-0.0.35.tar.gz | 52.4 kB | Without provenance | Details for assignment_codeval-0.0.35.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| assignment_codeval-0.0.35-py3-none-any.whl | Python 3 | none | any | Details for assignment_codeval-0.0.35-py3-none-any.whl |
Total release size: 107.9 kB
Release files / assignment_codeval-0.0.35.tar.gz
| Download URL | assignment_codeval-0.0.35.tar.gz |
|---|---|
| Size | 52.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
dcf02638686497d220a6c6a0f0f763ce2d75cbf670d36e2487c50d33638d5c62
|
|
BLAKE2b-256 checksum How to use checksums |
6b7357a311cb0ef62427f3b9b927deb3f49aee6ba69527e3b01c300d015cf352
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.3
|
Release files / assignment_codeval-0.0.35-py3-none-any.whl
| Download URL | assignment_codeval-0.0.35-py3-none-any.whl |
|---|---|
| Size | 55.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e35c2dc5b2a411237a09d577636dd9b989b13a3100800f095566f43a10a644c5
|
|
BLAKE2b-256 checksum How to use checksums |
f6d1ab87aae151d9820dca358de032c0449ce656830afe88f2dd0699e500216d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.3
|
Security for assignment-codeval 0.0.35
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.