CLI tool for robust autograder chaff testing
Project description
brunotest
A coursework templating and testing engine to consolidate assignment code and ensure robust assignment quality.
The idea behind this framework is that it woul allow us to write only one solution code, and then carefully work out stencil around it (in fact, have the stencil generated automatically from the solution code).
The idea on how to do this is to use Region tags in the code, and
All of the templating will be done in a __brunotest__
hidden folder (that could be gitignored). The general use case is as follows (we'll say with the fibonacci
example given in examples
):
- Define our stencil/chaff/solution code
- Run
brunotest
to run the solution code against thetests
- Run
brunotest stencil
to run the compiled stencil code against thetests
- Run
brunotest [chaff name]
to run the compiled chaff code against thetests
. - Run
brunotest all
to run all of the code (solution, stencil, chaffs) against thetests
.
Ideally, this would also be enhanced by a VSCode extension that could allow you to more easily see the different chaffs/stencil alongside the current document you are editing (maybe similar to how Markdown preview works).
Some other considerations/definitions for now:
- The chaff implementations will be stored in
.chaff
files, and:- Do not require all of the regions defined in the solution to be overwritten, but regions that are not specified will remain the same.
- Can (somehow?) define which tests should fail and with what values, likely at the top?
- This may be somewhat challenging because we have to incorporate failing partial tests, test names, etc.
- The stencil code will be in a single
.stencil
file in the root directory (error on multiple found).- The stencil also does not have to specify every distinct region to overwrite, but those which are not specified will be left blank.
Gradescope Autograder Specifications
The gradescope autograder, with the configuration that we use it in, has the following directory structure:
/autograder/ # Base Directory
/autograder/student # Student submission
/autograder/solution # Solution code
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for brunotest-0.1.1.1687008168.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 890eee842fbd24719711ea2c5cb7aabe4b3bf17461dc01d907739a9cfbbbe61c |
|
MD5 | ee6b9c1f0a3f23f8c6c0b2afb2be32ba |
|
BLAKE2b-256 | 360db70dfc46abd5ad31d7b49519d7a533ae9bf2a9b88b555103c697074d6603 |
Hashes for brunotest-0.1.1.1687008168-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f6e216fe8f1a51fb8ef54dade116c3f2b82f7ea07f01b81e351bbb5496c9808a |
|
MD5 | 1473d892bc011e3d9241cb5fdf7fa927 |
|
BLAKE2b-256 | bbc8d9795a66692a5e0affbceb93636268f955330f599fa4eef1690f2f93bcbb |