benchie 0.0.2

A tool for automating benchmarks of programming assignments.

benchie

A tool for automating benchmarks of programming assignments.

• Github repository: https://github.com/berombau/benchie/
• Documentation https://berombau.github.io/benchie/

Installation

Install uv.

You can run the tool without installation using the following command:

uvx benchie --help

To install the tool in an environment, you can run:

uv venv
uv pip install benchie

Run as a student

Example data

With the example data in this repository at solutions/ and data/ you can run the following command:

benchie run -S -e "example_sleep" --disable_pretest -b hyperfine -b memray_tracker

The output will be stored in output/example_sleep/.

Custom data

Put different implementations at solutions/{exercise_1}/{implementation_1}.py.

Run the benchmark with disabled fetching of Dodona submissions and only the first 3 datasets:

benchie run -S -e "{exercise_1}" --subset_data 3

Output will be stored in output/{exercise_1}.

Run as a teacher

Fetch token from Dodona user profile and store as token file.

Setup recurrent job

• Use deploy keys
• Use git instead of https as git origin remote URL
• setup cron job
• copy conda init lines from ~/.bashrc to ~/.bashrc_conda

Use crontab -e to add a cron job (edit the repo path):

SHELL=/bin/bash
BASH_ENV=~/.bashrc_conda
0 * * * * conda activate combio_benchmark_2024; timeout --signal=KILL 1h benchie run -e "global_alignment" -fC >> ~/combio_benchmark_job.log 2>&1

---

Repository initiated with fpgmaas/cookiecutter-uv.