Evaluatio

Statistically rigorous evaluation for NLP with fast metrics and built-in inference tools.

Why Evaluatio?

Most libraries make it easy to compute metrics. Few make it easy to evaluate models correctly.

Evaluatio is designed to fix that.

It provides:

• Fast metric implementations (e.g. WER, CER)
• Uncertainty estimation (e.g. bootstrap confidence intervals)
• Model comparison tools (e.g. paired bootstrap tests)
• Multiple testing correction (e.g. Holm-Bonferroni)
• High performance (Rust-backed implementations)

What's included

Metrics

Statistical inference

Current task guides cover ASR. The inference tools are general-purpose and apply across tasks (especially for machine translation).

Quick example

import pandas as pd
from evaluatio.metrics.wer import (
    word_error_rate,
    word_error_rate_ci,
    word_error_rate_per_pair,
)
from evaluatio.inference.hypothesis import paired_bootstrap_test

df = pd.read_csv("inferences.csv")

# Corpus-level WER
wer = word_error_rate(df["references"], df["predictions"])

# Confidence interval
ci = word_error_rate_ci(df["references"], df["predictions"], 5000, 0.05)

# Model comparison
wer_a = word_error_rate_per_pair(df["references"], df["model_a"])
wer_b = word_error_rate_per_pair(df["references"], df["model_b"])

pvalue = paired_bootstrap_test(wer_a, wer_b, iterations=5000)

print(f"WER: {ci.mean:.3f} ± {ci.upper - ci.mean:.3f}")
print(f"P-value: {pvalue}")

This workflow reflects the recommended approach: compute a corpus-level metric, estimate uncertainty, and compare models using paired statistical tests.

Key features

Correct by default

• Explicit, strongly typed APIs that reduce common evaluation mistakes.
• Documentation is designed as a reference for statistically rigorous evaluation, not just API usage.

Built-in statistical inference

• Bootstrap confidence intervals
• Paired bootstrap significance testing
• Multiple testing correction

High performance

• Rust-backed core for efficient computation
• Faster WER/CER than common alternatives

Installation

pip install evaluatio

Documentation

Full documentation (including evaluation guides and statistical background): https://prvinspace.github.io/evaluatio/.

Status

ASR metrics (WER, CER) and the statistical inference tools are stable. APIs elsewhere may change.

Contribute to the project

There is always room for improvements, new metrics, new functionality, etc. If you have any suggestions or requests please feel free to add an issue! The main repository for the project can be found at codeberg.org/prvinspace/evaluatio.

Maintainer

The project is maintained by Preben Vangberg <prv21fgt@bangor.ac.uk>.

Etymology

The name evaluatio is a Latin noun and means "evaluation". It also doubles as the English verb "to evaluate" with the Welsh verbal derivational suffix -io, so it could also be Welsh slang for "to evaluate".