STREP - Sustainable and Trustworthy Reporting for ML and AI

Software for sustainable and trustworthy reporting in machine learning and artificial intelligence, as proposed in my PhD thesis and original research paper. With the publicly hosted Exploration tool, you can investigate evaluation databases - no code needs to run on your machine!

Note that this software is under active development - it reflects work in progress and is subject to change, so you might encounter delays, off-times, and differences to earlier publications. Check out the paper branch and dissertation branch for frozen states at the respective time of publication.

News and Release History

• 23/01/2026 - Now on PyPI, so you can finally run pip install strep (use strep[frontend] for labeling and exploration)
• 22/09/2025 - Published my PhD thesis based on STREP
• 17/04/2025 - Some changes and lots of new figures, soon to be found in my PhD thesis
• 13/01/2025 - Many fixes, updated Papers with Code and EdgeAccUSB databases
• 02/10/2024 - Improved scaling code (x15 speed), updated MetaQuRe and AutoXPCR databases
• 11/09/2024 - Improved functionality and presented the work at ECML-PKDD '24
• 30/04/2024 - Paper published in Data Mining and Knowledge Discovery, alongside initial repository

Explore your own databases

Instead of exploring the pre-assembled databases, you can also investigate your own custom evaluations by following these steps:

1. Prepare an evaluation database as a pandas DataFrame OR use mlflow experiments csv export to create a csv file with your ML experiment runs (each row should list one model performance evaluation on some data set)
2. Store the pickled database or csv summary in any directory, and optionally add some JSON meta information (check our databases for examples).
3. Install the complete STREP functionaility by running pip install strep[frontend] (Python 3.8 - 3.12).
4. Run the following code snippet:

from strep.index_scale import load_database, scale_and_rate
from strep.elex.app import Visualization

fname = 'path/to/your/database.pkl'
# load database and meta information (if available)
database, meta = load_database(fname)
# index-scale and rate database
rated_database = scale_and_rate(database, meta)
# start the interactive exploration tool
app = Visualization(rated_database)
app.run()

Contributing

I firmaly believe that sustainable and trustworthy reporting is a community effort. If you perform large-scale benchmark experiments, stress-test models, or have any other important evaluations to report - please get in touch! I would love to include other resource-aware evaluation databases and highlight your work.

Current available databases:

• ImageNetEff22 (Fischer et al. 2022): Efficiency information of popular ImageNet models
• EdgeAccUSB (Staay et al. 2024): Efficiency results of stress-tested USB accelerators for edge inference with computer vision models
• XPCR / Forecasting (Fischer et al. 2024): Efficiency information of DNNs for time series forecasting tasks
• MetaQuRe (Fischer et al. 2024): Resource and quality information of ML algorithm performance on tabular data
• RobustBench (Croce et al. 2020): Robustness and quality information of image classification models
• Papers With Code: The most popular benchmarks from this public database (code for re-assembling can be found here)

Citing

If you appreciate our work and code, please cite my PhD thesis and original research paper:

Fischer, R. Advancing the Sustainability of Machine Learning and Artificial Intelligence via Labeling and Meta-Learning.
Ph.D. Dissertation, TU Dortmund University (2025). https://doi.org/10.17877/DE290R-25716

Fischer, R., Liebig, T. & Morik, K. Towards More Sustainable and Trustworthy Reporting in Machine Learning. Data Mining and Knowledge Discovery 38, 1909–1928 (2024). https://doi.org/10.1007/s10618-024-01020-3

You can use the following bibtext entries:

@phdthesis{fischer_diss,
  title={Advancing the Sustainability of Machine Learning and Artificial Intelligence via Labeling and Meta-Learning},
  author={Fischer, Raphael},
  school={TU Dortmund University},
  url={http://doi.org/10.17877/DE290R-25716},
  doi={10.17877/DE290R-25716},
  year={2025}
}

@article{fischer_dami,
	title = {Towards More Sustainable and Trustworthy Reporting in Machine Learning},
	volume = {38},
	issn = {1573-756X},
	url = {https://doi.org/10.1007/s10618-024-01020-3},
	doi = {10.1007/s10618-024-01020-3},
	number = {4},
	journal = {Data Mining and Knowledge Discovery},
	author = {Fischer, Raphael and Liebig, Thomas and Morik, Katharina},
	year = {2024},
	pages = {1909--1928},
}

Repository Structure

• databases contains different gathered evaluation databases of ML reports, including scripts to assemble some of them.
• src/strep contains software that processes the databases, calculates index values and compound scores, and visualizes them.
• materials contains some additional data, scripts, and figures used in papers and my thesis.
• The top level files and scripts are used for building and deploying the software.

Terms of Use

Copyright (c) 2026 Raphael Fischer