mlstatreport 0.1.1

Statistical reporting engine for machine-learning benchmark experiments.

MLStatReport

MLStatReport turns machine-learning benchmark tables into traceable statistical reports. It provides a reusable Python library, a command-line interface, and a REST API that all execute the same validated analysis workflow.

The current release provides a focused frequentist workflow for multiple algorithms evaluated on multiple datasets. It produces deterministic JSON, CSV, Markdown, LaTeX, figures, a local bibliography, and an optional PDF when a LaTeX compiler is available.

Installation

MLStatReport requires Python 3.10 or newer.

python -m pip install mlstatreport

For a development checkout:

python -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

For a runtime-only installation, use python -m pip install . instead.

Dev Container

The recommended development environment is Visual Studio Code Dev Containers.

1. Open the repository in VS Code.
2. Run Dev Containers: Reopen in Container.
3. In the container terminal, run make install.

The container uses bash with Python 3.12 as the default development interpreter and includes the tools needed for formatting, type checking, tests, and the API service.

CLI quickstart

The example table is a wide benchmark matrix: one dataset column followed by one score column per algorithm.

mlstatreport --version

mlstatreport analyze examples/results.csv \
    --control Proposed \
    --metric accuracy \
    --higher-is-better \
    --posthoc holm \
    --out report/

Use mlstatreport analyze --help to see supported input formats and all options. Invalid configurations, such as --posthoc holm without --control, produce clear validation errors and do not write a report.

Expected output

The command creates a self-contained report directory:

report/
├── analysis.json
├── bibliography.bib
├── input_summary.md
├── interpretation.md
├── references.md
├── report.md
├── report.tex
├── report.pdf                  # only when LaTeX compilation succeeds
├── report_manifest.json
├── statistical_summary.md
├── figures/
│   ├── average_ranks.png
│   ├── critical_difference.png # only with more than two algorithms
│   ├── performance_heatmap.png
│   └── pvalue_heatmap.png      # only with control post-hoc comparisons
└── tables/
    ├── average_ranks.csv
    ├── descriptive_statistics.csv
    ├── effect_sizes.csv        # when effect sizes are present
    ├── posthoc_holm.csv        # only with control post-hoc comparisons
    └── posthoc_nemenyi.csv     # only without a control, k > 2

analysis.json is the deterministic machine-readable output. The report manifest records generated and skipped formats. No-control multi-algorithm critical-difference visual groups are exposed as statistical_analysis.critical_difference.cd_groups; control/Holm workflows leave that list empty because formal post-hoc decisions are Holm control-vs-rest comparisons. See docs/EXAMPLE_OUTPUT.md for the purpose of each file. Generated report/ directories are intentionally ignored by Git and excluded from Docker build contexts.

REST API quickstart

Start the local API in development mode:

make api

In another terminal:

curl http://localhost:8000/health

curl -X POST http://localhost:8000/api/v1/analyze \
    -F "file=@examples/results.csv" \
    -F "control_algorithm=Proposed" \
    -F "metric=accuracy" \
    -F "higher_is_better=true" \
    -F "posthoc=holm" \
    --output analysis.zip

The analysis endpoint accepts CSV, XLSX, and JSON benchmark files and returns the report directory as a ZIP archive. Uploads and generated files use request-scoped temporary storage. Validation failures return structured JSON errors; unexpected failures return a safe HTTP 500 response without a traceback. See docs/API.md for endpoint fields and response formats.

Docker and Docker Compose

Build and run the production image:

docker build -t mlstatreport .
docker run --rm -p 8000:8000 mlstatreport

make docker is a convenience alias for the build command. For a live-reload development API using the checked-out source, run:

docker compose up --build

Stop the Compose service with docker compose down.

Statistical methods currently implemented

• Per-algorithm descriptive statistics: mean, standard deviation, median, minimum, maximum, and sample count.
• Per-dataset average ranks with average-rank tie handling for both higher-is-better and lower-is-better metrics.
• Wilcoxon signed-rank test for exactly two algorithms, using zero_method="wilcox" to omit exact zero paired differences and a two-sided alternative by default.
• Paired row-wise win rate for the two-algorithm Wilcoxon workflow, computed over matched dataset rows with metric direction respected.
• Friedman omnibus test and Iman-Davenport correction for more than two algorithms.
• Kendall's W global effect size for the Friedman workflow, using the same tie-correction convention selected for the Friedman statistic.
• Nemenyi critical-difference value for more than two algorithms, used to render a Demšar-style CD diagram. No-control Nemenyi workflows include Nemenyi-style all-pairwise CD group bars when available; control/Holm workflows use the CD diagram as a rank-scale reference only.
• Control-versus-rest average-rank comparisons with two-sided normal p-values and Holm step-down adjustment for the multi-algorithm workflow.
• Nemenyi all-pairwise average-rank comparisons for the multi-algorithm workflow when no control algorithm is supplied.
• Pairwise paired win-rate values in Holm and Nemenyi post-hoc tables, computed over matched dataset rows and reported separately from significance decisions.
• Deterministic matplotlib average-rank, critical-difference, performance, and post-hoc p-value heatmap figures.
• Rule-based, evidence-linked interpretation and local BibTeX/Markdown references.

For exactly two algorithms, configure the pairwise workflow with --pairwise-test wilcoxon and --pairwise-alternative two-sided|less|greater. For more than two algorithms, pairwise settings are accepted as provenance but ignored by the Friedman/Iman-Davenport workflow. Multi-algorithm runs use Holm control-versus-rest comparisons when --control is provided, and Nemenyi all-pairwise comparisons when no control is provided.

Not implemented yet

• Finner, Bergmann-Hommel, Quade, aligned-rank, Bayesian methods, effect-size confidence intervals, or effect-size families beyond Kendall's W and paired win rate.
• Full journal-template support, bibliography retrieval, authentication, background jobs, report upload storage, and a web frontend.

Citations

Each generated report selects and writes the methodological citations it uses to bibliography.bib and references.md. The built-in registry includes the Wilcoxon (1945), Friedman (1937), Iman-Davenport (1980), Holm (1979), Nemenyi (1963), Kendall and Babington Smith (1939), Demšar (2006), García et al. (2010), García and Herrera (2008), and Santafé et al. (2015) references. Verify the generated bibliography against the original sources before publication, particularly entries intentionally stored with limited metadata.

Development commands

make install      # install editable package with development tools
make format       # apply Ruff fixes and Black formatting
make lint         # run Ruff and Black checks
make typecheck    # run mypy
make test         # run pytest
make ci           # lint, type-check, test, build, and check distributions
make build        # build source and wheel distributions
make check-dist   # build and validate distributions with twine
make clean        # remove local build and test artifacts
make api          # run the live-reload API
make docker       # build the production Docker image

Project layout

mlstatreport/      Reusable library, CLI, API, and rendering packages
examples/          Small benchmark inputs used by the quickstart
tests/             Automated tests
docs/              Architecture and output documentation
.devcontainer/     Development container configuration

See docs/ARCHITECTURE.md for package boundaries and extension guidance, and RELEASE.md for the release checklist. Release history is tracked in CHANGELOG.md, and contribution guidance is in CONTRIBUTING.md.

License

MLStatReport is released under the MIT License.