Unittesting & Grading of Jupyter Notebooks
autograde is a tool for testing Jupyter notebooks. Its features include execution of notebooks (optionally isolated via docker/podman) with consecutive unit testing of the final notebook state. On top of that, an audit mode allows for refining results (e.g. grading plots by hand). Eventually, autograde can summarize these results in human and machine readable formats.
Now, in order to install autograde, run
pip install jupyter-autograde.
Alternatively, you can install autograde from source by cloning this repository and runing
poetry install within it.
This requires poetry to be installed on your system!
Eventually, build the respective container image:
python -m autograde build.
Note: in order to build a container image, autograde must not be installed via PyPI but from source code!
Once installed, autograde can be invoked via the autograde command. If you are using a virtual environment (which poetry does implicitly) you may have to activate it first. Alternative methods:
- `path/to/python -m autograde` runs autograde with a specific python binary, e.g. the one of your virtual environment.
- `poetry run autograde` if you’ve installed autograde from source, you
autograde comes with some example files located in the
demo/ subdirectory that we will use for now to illustrate the workflow. Run:
python -m autograde test demo/test.py demo/notebook.ipynb --target /tmp --context demo/context
What happened? Let’s first have a look at the arguments of autograde:
demo/test.pya script with test cases we want apply
demo/notebook.ipynbis the a notebook to be tested (here you may also specify a directory to be recursively searched for notebooks)
- The optional flag
--targettells autograde where to store results,
/tmpin our case, and the current working directory by default.
- The optional flag
--contextspecifies a directory that is mounted into the sandbox and may contain arbitrary files or subdirectories. This is useful when the notebook expects some external files to be present such as data sets.
The output is a compressed archive that is named something like
results_[Lastname1,Lastname2,...]_XXXXXXXX.zip and which has the following contents:
artifacts/: directory with all files that where created or modified by the tested notebook as well as rendered matplotlib plots.
code.py: code extracted from the notebook including
notebook.ipynb: an identical copy of the tested notebook
test_restults.json: test results
report sub command creates human readable HTML reports from test results:
python -m autograde report path/to/result(s)
The respective report is added to the results archive inplace.
Results from multiple test runs can be merged via the
patch sub command:
python -m autograde patch path/to/result(s) /path/to/patch/result(s)
In a typical scenario, test cases are not just applied to one notebook but many at a time. Therefore, autograde comes with a summary feature, that aggregates results, shows you a score distribution and has some very basic fraud detection. To create a summary, simply run:
python -m autograde summary path/to/results
Two new files will appear in the result directory:
summary.csv: aggregated results
summary.html: human readable summary report
To get an overview of all available commands and their usage, run
python -m autograde [sub command] --help
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size jupyter-autograde-0.2.9.tar.gz (35.8 kB)||File type Source||Python version None||Upload date||Hashes View|