autograder-py 1.0.1

The Python interface for the autograding server.

Python Interface for Lynx Grader

The canonical Python interface for EduLinq's Lynx Grader server.

Quick Links

• Resources
• API Reference
• CLI Reference
• Installation / Requirements
• Quickstart
• The CLI
  • Configuration
  • Commands for Students
    • Submitting an Assignment
      • Submitting an Assignment Late
    • Checking Your Last Submission
    • Getting a History of All Past Submissions
    • Managing your Password
  • Commands for TAs and Instructors
    • Proxy Submissions
      • Submitting Source Files for a Student
      • Resubmitting a Student's Code by ID
  • Commands for Course Builders

Resources

• API Reference
• CLI Reference
• Autograder Server
• Autograder Python Interface (this repo)
• Autograder Sample Course

Installation / Requirements

This project requires Python >= 3.9.

The project can be installed from PyPi with:

pip3 install autograder-py

Standard Python requirements are listed in pyproject.toml. The project and Python dependencies can be installed from source with:

pip3 install .

Cloning

This repository includes submodules. To fetch these submodules on clone, add the --recurse-submodules flag. For example:

git clone --recurse-submodules [e-mail]:edulinq/autograder-py.git

To fetch the submodules after cloning, you can use:

git submodule update --init --recursive

Quickstart

To provide easy access to a limited number of commands, we provide the autograder.run suite of shortcuts. These are just a small number of the most commonly used commands. For a more in-depth look at the available commands, see the cli section of this document.

autograder.run.submit

This command sends an assignment submission to the server. This is a shortcut for autograder.cli.courses.assignments.submissions.submit.

python3 -m autograder.run.submit my_file.py

To submit an assignment late, use the following command. For more information and examples, see the late submission section of this document.

python3 -m autograder.run.submit --allow-late my_file.py

autograder.run.history

This command gets a summary of all your past submissions for an assignment. This is a shortcut for autograder.cli.courses.assignments.submissions.user.history.

python3 -m autograder.run.history

autograder.run.peek

This command shows you your most recent (or a specific) submission for an assignment. This is a shortcut for autograder.cli.courses.assignments.submissions.user.peek.

python3 -m autograder.run.peek

To get a specific submission, just pass the submission ID (as shown in autograder.run.history).

python3 -m autograder.run.peek 123456789

autograder.run.auth

You can use this command to quickly check if your password/config is correct. This is a shortcut for autograder.cli.users.auth.

python3 -m autograder.run.auth

autograder.run.change-pass

This command lets your change your password to whatever you want. This is a shortcut for autograder.cli.users.pass.change.

python3 -m autograder.run.change-pass

You will then be prompted to enter (and re-enter) your new password.

autograder.run.reset-pass

This command will reset your password by sending an email to your registered email address. This is a shortcut for autograder.cli.users.pass.reset.

python3 -m autograder.run.reset-pass

The CLI

This project contains several tools for interacting with an autograding server and working with autograder assignments via the autograder.cli package. All tools will show their usage if given the --help options.

You can get a list of the package for each set of tools by invoking autograder.cli directly:

python3 -m autograder.cli

If you want to see every tool a package (and all its subpackages) have available in one output, you can use the -r/--recursive flag:

python3 -m autograder.cli --recursive

There are many available tools and instead of discussing each one here, this document will highlight the tools each type of user (student, TA, course developer) will generally use.

Configuration

Before discussing specific tools, you should know some general information about configuring and sending options to each tool.

To know who you are and what you are working on the autograder needs a few configuration options:

• server -- The autograding server to connect to.
• course -- The ID for the course you are enrolled in.
• assignment -- The current assignment you are working on (does not always apply)..
• user -- Your username (which is also your email).
• pass -- Your password (probably sent to you by the autograding server in an email).

All these options can be set on the command line when invoking on of these tools, e.g.,:

python3 -m autograder.run.submit --user [e-mail] --pass pass123 my_file.py

However, it will generally be more convenient to hold these common options in a more reusable location. There are several files where you can place your options, the three main ones are:

1. Local Config -- A file named config.json that lives in the directory you are running the command in. Useful for setting assignment-specific options (like an assignment name/id).
2. Project Config -- A file named autograder.json that lives in your project/repo root directory. Useful for course-specific options (like your course name/id).
3. Global Config -- A file named autograder.json that lives somewhere in your user's home directory. Useful for options that you share between courses (like your server, username, and password).

To see the exact location of each file, the order that options are loaded, and details about it, run any command with --help and see the "CONFIGURATION" section:

python3 -m autograder.run.auth --help

A config file may look something like:

{
    "course": "my-course",
    "assignment": "assignment-01",
    "server": "http://fake.autograder.edulinq.org",
    "user": "[e-mail]",
    "pass": "1234567890"
}

For brevity, all future commands in this document will assume that all standard config options are in the default config files (and thus will not need to be specified).

Commands for Students

Students will mainly be concerned with submitting assignments and checking on the status of their submission. Therefore, the autograder.run package will be their primary resource. This package contains tools for making, managing, and querying submissions.

Submitting an Assignment

Submitting an assignment to an autograder is done using the autograder.run.submit command. This command takes the standard config options as well as an optional message to attach to the submission (like a commit message) as well as all files to be included in the submission.

python3 -m autograder.run.submit --message "This is my submit message!" my_file.py

As many files as you need can be submitted (directories cannot be submitted):

python3 -m autograder.run.submit my_first_file.py my_second_file.java some_dir/*

The autograder will attempt to grade your assignment and will return some message about the result of grading. For example, a successful grading may look like:

The autograder successfully graded your assignment.
Autograder transcript for assignment: HO0.
Grading started at 2023-09-26 08:35 and ended at 2023-09-26 08:35.
Task 1.A (my_function): 40 / 40
Task 2.A (test_my_function_value): 30 / 30
Task 2.B (TestMyFunction): 30 / 30
Style: 0 / 0
   Style is clean!

Total: 100 / 100

On any successful grading (even if you got a zero), your result has been saved by the autograder and is in the system. On a submission failure, the autograder will tell you and you will not receive any grade for your submission. A failure may look like:

The autograder failed to grade your assignment.
Message from the autograder: Request could not be authenticated. Ensure that your username, password, and course are properly set.

Submitting an Assignment Late

If you are submitting an assignment late, the autograder requires confirmation in order to grade your submission. This helps users avoid situations where they accidentally submit an assignment late or submit to the wrong assignment. Users must add the --allow-late flag to the normal submission command when they want to submit an assignment past the due date.

For example, your output when submitting a late assignment may look like:

--- Message from Autograder ---
Attempting to submit assignment (HO0) late without the 'allow late' option.
It was due on 2024-12-13 16:00 (which was 48h34m57.178s ago).
Use the 'allow late' option to submit an assignment late.
See your interface's documentation for more information.
-------------------------------
Submission was rejected by the autograder.

When you see this message, be sure to double check the assignment name and due date. If those details look correct and you want to submit that assignment late, then run the following command:

python3 -m autograder.run.submit --allow-late my_file.py

Now, the server will grade your late submission like normal!

Checking Your Last Submission

You can ask the autograder to show you the grade report for your last submission using the autograder.run.peek command.

python3 -m autograder.run.peek

The output may look like:

Found a past submission for this assignment.
Autograder transcript for assignment: HO0.
Grading started at 2023-09-26 08:35 and ended at 2023-09-26 08:35.
Task 1.A (my_function): 40 / 40
Task 2.A (test_my_function_value): 30 / 30
Task 2.B (TestMyFunction): 30 / 30
Style: 0 / 0
   Style is clean!

Total: 100 / 100

If you have made no past (successful) submissions, then your output may look like:

No matching submission found.

Getting a History of All Past Submissions

You can use the autograder.run.history command to get a summary of all your past submissions for an assignment.

python3 -m autograder.run.history

The output may look like:

Found 2 submissions.
    Submission ID: 1695682455, Score: 24 / 100, Time: 2023-09-25 17:54.
    Submission ID: 1695735313, Score: 100 / 100, Time: 2023-09-26 08:35, Message: 'I did it!'

If you have made no past (successful) submissions, then your output may look like:

No matching submission found.

Managing your Password

Your password is the same throughout a single instance of the autograding server. This means that multiple courses that run on the same server will all use your same account (and therefore password).

Your initial password should have been emailed to the email associated with your account (typically your school email address).

To reset your password, use the autograder.run.reset-pass (aka autograder.cli.users.pass.reset) command:

python3 -m autograder.run.reset-pass

This will email you a new random password to your account's email. Once your password is reset, it is recommended to change it to whatever you want.

To change your password, you can use the autograder.run.change-pass (aka autograder.cli.users.pass.change) command:

python3 -m autograder.run.change-pass

You will then be prompted to enter (and re-enter) your new password. See the command's help prompt (--help) for additional ways you can supply your password.

Commands for TAs and Instructors

For those that are managing a course and students, most commands will be useful to you. So you should have a look through all commands via:

python3 -m autograder.cli -r

This will list all available packages and commands. You can omit the -r if you want to look at one package at a time. For example, to inspect the autograder.run package, you can do:

python -m autograder.run

Below is a list of commands you may want to look into. The help prompt of each command (accessible using the --help option) will give a more in-depth description of the command and available options.

• autograder.cli.courses.assignments.submissions.fetch.course.scores -- Get all the most recent scores for an assignment.
• autograder.cli.courses.assignments.submissions.fetch.user.attempt -- Get a student's submission (code) and grading output.
• autograder.cli.courses.assignments.submissions.fetch.course.attempts -- Get all the most recent submissions (code and grading output) for an assignment.
• autograder.cli.courses.users.list -- List all the users in a course.

LMS Commands

To interact directly with your LMS (e.g. Blackboard, Canvas, Moodle), we recommend the LMS Toolkit. This library provides a unified CLI and Python interface for interacting with different LMSs. This is what the autograder uses when connecting with LMSs.

Proxy Submissions

The autograder allows course staff to submit on behalf of students using "proxy" submissions. The suite of proxy commands use the following two new concepts:

• --proxy-email -- The student that the course staff is submitting on behalf of.
• --proxy-time -- The "fudged" submission time that is automatically set to the earlier time between now and one minute before the due date.
  • If you want to set manually, it is the number of milliseconds from the Unix Epoch. You can use a Unix Epoch time converter to help with the conversion.

As the commands are issued by course staff, the submissions are not subject to submission restrictions.

Submitting Source Files for a Student

To proxy submit source files for a student, use the autograder.cli.courses.assignments.submissions.proxy.submit command. Here is an example where course staff submits code (my_file.py) on behalf of [e-mail]. The example does not set a --proxy-time so the command automatically sets a proxy time that does not mark the submission as late.

python3 -m autograder.cli.courses.assignments.submissions.proxy.submit --proxy-email [e-mail] my_file.py

The output may look like:

Autograder transcript for assignment: HO0.
Grading started at 2025-04-02 12:45 and ended at 2025-04-02 12:45.
Task 1.A (my_function): 40 / 40
Task 2.A (test_my_function_value): 30 / 30
Task 2.B (TestMyFunction): 30 / 30
Style: 0 / 0
   Style is clean!

Total: 100 / 100

Extending the previous example, we can manually set the time of submission through --proxy-time. We will use the time 1741118400000 (March 4th, 2025 at 12:00 PM Pacific Time).

python3 -m autograder.cli.courses.assignments.submissions.proxy.submit --proxy-time 1741118400000 --proxy-email [e-mail] my_file.py

Resubmitting a Student's Code by ID

To proxy resubmit a student's code using their previous submission ID, use the autograder.cli.courses.assignments.submissions.proxy.resubmit command. By default, proxy resubmit targets the student's most recent submission:

python3 -m autograder.cli.courses.assignments.submissions.proxy.resubmit --proxy-email [e-mail]

The output may look like:

Autograder transcript for assignment: HO0.
Grading started at 2025-04-02 12:45 and ended at 2025-04-02 12:45.
Task 1.A (my_function): 40 / 40
Task 2.A (test_my_function_value): 30 / 30
Task 2.B (TestMyFunction): 30 / 30
Style: 0 / 0
   Style is clean!

Total: 100 / 100

We can target a specific submission ID using --target-submission. You can get a history of all past submissions to see a student's previous submission IDs and scores. For example, we can extend the previous example with a target submission ID of 123456789:

python3 -m autograder.cli.courses.assignments.submissions.proxy.resubmit --target-submission 123456789 --proxy-email [e-mail]

Commands for Course Builders

Users who are building courses should generally be aware of all the available tools, but most of your time will probably be spent in the autograder.cli.testing and autograder.cli.grading packages. autograder.cli.testing is for running tests and checks (usually locally) on assignments. autograder.cli.grading lets you grade assignments locally (without using an autograding server). Because the autograding server runs this package inside a Docker container to do grading, it can be much faster and more convenient to build assignments fully locally before using an autograding server.