lograder 0.0.4

An API for easy Gradescope Autograder assignment creation.

lograder: A Gradescope Autograder API

---

This project just serves to standard different kinds of tests that can be run on student code for the Gradescope autograder. Additionally, this project was developed for the University of Florida's Fall 2025 COP3504C (Advanced Programming Fundamentals), taught by Michael Link. However, you are completely free to use, remix, refactor, and abuse this code as much as you like.

---

Project Builders

---

C++ Complete Project with I/O Comparison

Build from C++ Source (WIP)

To build from source, you will need to import the C++ CxxSourceBuilder. The executable will be randomly named and put in either a build directory, if the student has one (./build) or the project root directory (./).

from lograder.dispatch import CxxSourceDispatcher
from lograder.output import AssignmentSummary

# Note that when you make a test, it's automatically
# registered with the `lograder.tests.registry.TestRegistry`

assignment = CxxSourceDispatcher(project_root="/autograder/submission")
preprocessor_results = assignment.preprocess()
build_results = assignment.build()
runtime_results = assignment.run_tests()

summary = AssignmentSummary(
    preprocessor_output=preprocessor_results.get_output(),
    build_output=build_results.get_output(),
    runtime_summary=runtime_results.get_summary(),
    test_cases=runtime_results.get_test_cases()
)

Build using CMake (WIP)

To build from a CMakeLists.txt, you will need to import the C++ CMakeBuilder. This method will automatically run a breadth-first search starting in the project root directory (./) and "lock on" the first (i.e. the file in the highest-level) CMakeLists.txt that it finds. If it can't find a CMakeLists.txt, it will raise an error.

Additionally, the program will look for the following targets first: main, build, and demo. Afterward, it will search for any target that doesn't match: all, install, test, package, package_source, edit_cache, rebuild_cache, clean, help, ALL_BUILD, ZERO_CHECK, INSTALL, RUN_TESTS, and PACKAGE, and run the first target that it finds. If it can't find a valid target, it will raise an error.

from lograder.dispatch import CMakeDispatcher
from lograder.output import AssignmentSummary

# Note that when you make a test, it's automatically
# registered with the `lograder.tests.registry.TestRegistry`

assignment = CMakeDispatcher(project_root="/autograder/submission")
preprocessor_results = assignment.preprocess()
build_results = assignment.build()
runtime_results = assignment.run_tests()

summary = AssignmentSummary(
    preprocessor_output=preprocessor_results.get_output(),
    build_output=build_results.get_output(),
    runtime_summary=runtime_results.get_summary(),
    test_cases=runtime_results.get_test_cases()
)

---

C++ Catch2 Unit Testing (WIP)

---

Python Complete Project with I/O Comparison

Run project from main.py (WIP)

Run project from pyproject.toml (WIP)

---

Python pytest Unit Testing (WIP)

---

Makefile Complete Project with I/O Comparison (WIP)

To build from a Makefile, you will need a MakefileBuilder. It follows the same general idea as the CMakeBuilder except that it searches for Makefile instead of CMakeLists.txt. Additionally, MakefileBuilder will just run the default make.

from lograder.dispatch import MakefileDispatcher
from lograder.output import AssignmentSummary

# Note that when you make a test, it's automatically
# registered with the `lograder.tests.registry.TestRegistry`

assignment = MakefileDispatcher(project_root="/autograder/submission")
preprocessor_results = assignment.preprocess()
build_results = assignment.build()
runtime_results = assignment.run_tests()

summary = AssignmentSummary(
    preprocessor_output=preprocessor_results.get_output(),
    build_output=build_results.get_output(),
    runtime_summary=runtime_results.get_summary(),
    test_cases=runtime_results.get_test_cases()
)

---

Test Generation

---

Output Comparison

Compare Simple Strings

For the smallest number of tiny test cases, there's no reason to have an over-bloated mess. You can just use:

from typing import Sequence, Optional, List
from pathlib import Path
from lograder.tests import make_tests_from_strs, ExecutableOutputComparisonTest


def make_test_from_strs(
        *,  # kwargs-only; to avoid confusion with argument sequence.
        names: Sequence[str],
        inputs: Sequence[str],
        expected_outputs: Sequence[str],
        flag_sets: Optional[Sequence[List[str | Path]]] = None,
        # Pass flags like ["--option-1", "--option-2"] to student programs
        weights: Optional[Sequence[float]] = None,  # Defaults to equal-weight.
) -> List[ExecutableOutputComparisonTest]: ...


# Here's an example of how you'd use the above method:
make_tests_from_strs(
    names=["Test Case 1", "Test Case 2"],
    inputs=["stdin-1", "stdin-2"],
    expected_outputs=["stdout-1", "stdout-2"]
)

Compare from Files

If you have a larger test, it would be very convenient to read files for input and output. Luckily, there's just the method to do so:

from typing import Sequence, Optional, List
from pathlib import Path
from lograder.tests import make_tests_from_files, FilePath, ExecutableOutputComparisonTest


# `make_tests_from_files` has the following signature.
def make_tests_from_files(
        *,  # kwargs-only; to avoid confusion with argument sequence.
        names: Sequence[str],
        input_files: Optional[Sequence[FilePath]] = None,  # `input_files` and `input_strs` mutually exclusive.
        input_strs: Optional[Sequence[str]] = None,
        expected_output_files: Optional[Sequence[FilePath]] = None,
        # same with `expected_output_files` and `expected_output_strs`
        expected_output_strs: Optional[Sequence[str]] = None,
        flag_sets: Optional[Sequence[List[str | Path]]] = None,
        # Pass flags like ["--option-1", "--option-2"] to student programs
        weights: Optional[Sequence[float]] = None,  # Defaults to equal-weight.
) -> List[ExecutableOutputComparisonTest]: ...


# Here's an example of how you'd use the above method:
make_tests_from_files(
    names=["Test Case 1", "Test Case 2"],
    input_files=["test/inputs/input1.txt", "test/inputs/input2.txt"],
    expected_output_files=["test/inputs/output1.txt", "test/inputs/output2.txt"]
)

Compare from Template

Finally, sometimes the test-cases might be very long but very repetitive. You can use make_tests_from_template and pass a TestCaseTemplate object and ...

from typing import Sequence, Optional, List
from pathlib import Path
from lograder.tests import make_tests_from_template, TestCaseTemplate, FilePath


# Here's the signature of a `TemplateSubstitution`
class TemplateSubstitution:
    def __init__(self, *args, **kwargs):
        # Stores args and kwargs to pass to str.format(...) later.
        ...


TSub = TemplateSubstitution  # Here's an alias that's quicker to type.


# Here's the signature of a `TestCaseTemplate`
class TestCaseTemplate:
    def __init__(self, *,
                 inputs: Optional[Sequence[str]] = None,
                 input_template_file: Optional[FilePath] = None,
                 input_template_str: Optional[str] = None,
                 input_substitutions: Optional[Sequence[TemplateSubstitution]] = None,
                 expected_outputs: Optional[Sequence[str]] = None,
                 expected_output_template_file: Optional[FilePath] = None,
                 expected_output_template_str: Optional[str] = None,
                 expected_output_substitutions: Optional[Sequence[TemplateSubstitution]] = None,
                 flag_sets: Optional[Sequence[List[str | Path]]] = None,  # Pass flags like ["--option-1", "--option-2"] to student programs
                 ):
        # +=====================================================================================+
        # | Validation Rules                                                                    |
        # +=====================================================================================+
        #   * If `inputs` is specified, all other `input_*` parameters must be left unspecified.
        #   * Same thing with `expected_outputs`.
        #   * If `inputs` is not specified, you must specify either (mutually exclusive) 
        #     `input_template_file` or `input_template_str` that follows a typical python
        #     format string, and you must specify `input_substitutions`.
        #   * Same thing with `expected_output_template_file`, `expected_output_template_str`, 
        #     and `expected_output_substitutions`
        ...


# Here's an example of how you would use TestCaseTemplate
test_suite_1 = TestCaseTemplate(
    inputs=["A", "B", "C"],  # Three (3) Total Cases
    expected_output_template_str="{}, {kwarged}, {}",
    expected_output_substitutions=[
        TSub(1.0, 2.0, kwarged="middle-arg-1"),  # Case 1 Substitutions
        TSub(2.0, 5.0, kwarged="middle-arg-2"),  # Case 2 Substitutions
        TSub(7.0, 6.0, kwarged="middle-arg-3"),  # Case 3 Substitutions
    ]
)
make_tests_from_template(
    ["Test 1", "Test 2", "Test 3"],
    test_suite_1
)  # remember to construct the tests!

Compare from Python Generator/Iterable

Sometimes, you want to generate a ton of test-cases (especially small test-cases), and it would be incredibly waste to have thousands of single-line files. You can create a python generator function that follows either the following Protocol or TypedDict.

from typing import Protocol, TypedDict, Generator, NotRequired, List
from pathlib import Path
from lograder.tests import make_tests_from_generator


# Your generator may return objects following the protocol...
class TestCaseProtocol(Protocol):
    def get_name(self): ...

    def get_input(self): ...

    def get_expected_output(self): ...

class FlaggedTestCaseProtocol(TestCaseProtocol, Protocol):
    def get_flags(self) -> List[str | Path]: ...
    
# Notice that TestCaseProtocol defaults to equal-weights
class WeightedTestCaseProtocol(TestCaseProtocol, Protocol):
    def get_weight(self): ...

# ... or you can directly return a dict with the following keys.
class TestCaseDict(TypedDict):
    name: str
    input: str
    expected_output: str
    weight: NotRequired[float]  # Defaults to 1.0, a.k.a. equal-weight.
    flags: NotRequired[List[str | Path]]


# Here's an example of the syntax as well as the required 
# signature of such a method:
@make_tests_from_generator
def test_suite_1() -> Generator[TestCaseProtocol | WeightedTestCaseProtocol | TestCaseDict, None, None]:
    pass

# You'll have to query the `TestRegistry` from `lograder.tests` to access these tests directly, though.