lograder 0.2.0

A general-purpose autograder written primarily in Python 3.

lograder

This repository contains the source code for a general-purpose autograder. The idea from a high-level view is everything belongs to a pipeline, which can be found in src/lograder/pipeline/pipeline.py. The main type that is of note is Step, found in src/lograder/pipeline/step.py. The steps that make up the pipeline have a __call__ which is a generator. The generator may yield values (both errors and info) which are non-fatal, while return values are either fatal errors or the data to pass onto the next step.

The central mantra to this library is validation, validation, validation (because repetition legitimizes). All calls to processes should be through a TypedExecutable, which contains validated input models via CLIArgs. Additionally, any output needs to be a validated model and needs to be parsed programmatically and completely. Either the program must have an API, or the text parser must be bullet-proof.

Additionally, passing errors as values rather than exception based programming (except for developer bugs, staff errors, or validation errors) is preferred.

Finally, all the output is generated by the steps and will be handled by the loggers, which are responsible for producing logging output (obviously), but also the .json or .html or whatever is used by the underlying application.

With that out of the way, we can get a little more into the details.

---

Installation

pip install -e ".[dev]"

Run tests with pytest.

---

"The Pipeline"

The autograder must be able to take student files of any type, such as source files for C, C++, Python, Rust, Assembly, and so forth; build the project in as configurable manner (with a natural default, ONLY IF there is an intuitive default); create any sort of artifact, whether that be text output, file output, etc.; and then "test" the artifacts whatever that may mean.

The primary subpackage for this is located in src/lograder/pipeline.

Step generics

Every concrete Step subclass must declare exactly five type parameters:

class MyStep(Step[InputT, OkOutputT, ErrOutputT, OkDisplayT, ErrDisplayT]): ...

OkOutputT becomes the input for the next step. ErrOutputT and ErrDisplayT are fatal and non-fatal error types respectively, sent to the packet logger. __init_subclass__ enforces this at class definition time.

Result type

Result[T, E] (see src/lograder/common/result.py) is a Rust-inspired error-as-value container. Use Ok(value) and Err(value) constructors. danger_ok / danger_err are assertion-guarded unwrappers. Do not raise exceptions for expected failure modes — return Err(...) instead. DeveloperException and StaffException are reserved for internal bugs and course staff configuration errors respectively.

Manifest

Manifest (see src/lograder/pipeline/types/parcels.py) represents a project's file/directory tree and is the primary type flowing between Input and Check steps. It can be constructed from a directory scan (from_directory), a TOML file (from_toml), or a flat list of paths (from_flat). == checks exact equality; <= checks subset (received contains at least the expected files).

Input

These will be the very beginning of the pipeline, taking "no input" and producing some output (if no error, of course)! For example, the Gradescope autograder puts uploaded files into a fixed directory, so a natural input is to enumerate a local directory into a Manifest. However, there are other examples of inputs, such as a single file upload.

Check

These are mostly validation that happens before build. An example could be to make sure the project matches a specified manifest. Another example is to do a source file parse to ensure that an illegal library or operator is not used or is used under a certain number of times. The input and output type of these should be able to be chained together.

The simple_project.py check module generates named manifest-check classes (CMakeManifestCheck, MakefileManifestCheck, PyProjectManifestCheck) at import time, so they can be used directly.

Mixin

These will be used to inject staff code (such as specified header files) into the students code, or if the student is expected to write some file but not the surrounding build system, then it will inject the necessary files.

Build

These steps actually build the library, such as via cmake or make or pip install and produce artifacts that are testable or usable in the next steps.

Implemented builders: CMakeBuild (configure + build + File API artifact parsing), MakefileBuild (runs make; artifact parsing is not yet implemented).

Test

These are mostly post-build testing. Examples could be running a number of instances of the student's executable, passing different arguments and seeing if the output matches. Another example is checking the bytes of a binary file to see if they match or running valgrind to check for memory leaks.

---

Ideas / Future Work

• Implement Mixin.__call__ (currently a stub).
• Add Makefile artifact parsing in MakefileBuild (returns empty list today).
• Create Layout for lograder.pipeline.build.makefile.MakefileBuildOutput.
• Implement Pipeline.validate_step_types() call and validation in Pipeline.__call__.