baseltest

Statistically honest testing for stochastic services, in Python.

Services built on LLMs, ML models, or randomised algorithms do not pass or fail a single invocation — they succeed at a rate. baseltest treats that rate as the thing under test: run the service repeatedly, judge each response against declared criteria, and render a verdict backed by exact statistics (the Fisher exact test against a measured baseline, the exact binomial test against a stated requirement, feasibility-checked sample sizes) rather than a green tick over a lucky sample.

baseltest is the Python member of the mavai framework family, alongside punit (Java) and feotest (Rust). It shares their statistical methodology — every formula is validated against the family's statistical oracle — and expresses it in Python idioms rather than porting either framework.

Where the project stands

The statistics core (baseltest.statistics) is implemented and conformance-validated against the oracle's published reference cases: the decision rules of Statistical Companion 1.7.1 (regression/fisher, compliance/exact-binomial, latency/precedence, latency/compliance-exact-binomial, and the smoke-intent compliance/smoke-exact-binomial and latency/smoke-compliance-exact-binomial) with the exact-boundary convention, feasibility checking, design and resolved power and sizing, and the descriptive Wilson interval, built on scipy/statsmodels.

The framework around it is in active development.

Try it in two minutes

The repository ships a ready-to-run example that needs no API key, no network, and no Python of your own — the service under test is simulated:

baseltest requires Python 3.11 or newer. Check what you have first:

python3 --version

If that prints 3.11+ you can use python3 below. Otherwise install a newer interpreter — it will live alongside your system Python, nothing is replaced:

• macOS (Homebrew): brew install python@3.12 → the interpreter is python3.12. If the command isn't found afterwards, run brew link python@3.12 or use the full path $(brew --prefix python@3.12)/bin/python3.12.
• Linux (Debian/Ubuntu): sudo apt install python3.12 python3.12-venv (on older releases, via the deadsnakes PPA).
• Any platform, pyenv: pyenv install 3.12 && pyenv local 3.12.
• Any platform, uv: uv venv --python 3.12 creates the venv below in one step.

Then — creating the venv with the new interpreter is the step that matters; installing 3.12 alone changes nothing until a venv is built from it:

git clone https://github.com/mavai-org/baseltest.git
cd baseltest

python3.12 -m venv venv          # or python3, if yours is already 3.11+
source venv/bin/activate
python --version                 # verify: must print 3.11+ — if not, stop and re-check the venv line
pip install -e ".[declarative]"   # the baseltest package ships the `basel` command

cd examples/simulated-service
basel test fortune-teller.yaml      # judge it against its declared bar
basel measure fortune-teller.yaml --samples 200   # or: record everything, persist a baseline

(If pip ever answers with Package 'baseltest' requires a different Python, that is this issue: the active pip still belongs to an older interpreter. deactivate, delete venv/, and recreate it with the 3.11+ interpreter as above.)

You'll see a verdict with its uncertainty stated — run the test a few times and watch the observed rate move while the conclusion stays statistically honest. The measure verb is the other posture over the same file: every criterion recorded, a baseline artefact persisted. A third verb, explore, sweeps a grid of service configurations declared in the services file and writes one descriptive artefact per configuration — triage before you measure. A fourth, optimize, searches the configuration space iteratively — a declared stepper proposes each next configuration, a scorer judges each iteration, and the full history is persisted as one artefact. Everything a run generates lands under _baseltest/ in the working directory. When you have a model credential to hand, examples/language-model/ runs a real language-model service from two small files — its grid pits two models against each other (GPT-4o-mini and Claude Haiku 4.5, with an optional entry for the fully open Swiss Apertus), so one basel explore run and one diff compare them on the same job; the examples README has the step-by-step, and the getting-started guide walks through all the verbs.

The command line

The baseltest package ships one command, basel, with five verbs. The contract file carries the claim; the verb carries the posture; the invocation carries the budget.

HTML reports are rendered by the mavai tool — mavai verdict|measure|explore|optimize <dir> [-o report.html] over the persisted artefacts. It is developed in the mavai-report project; obtain it from there, and see that project's README.md for installation instructions.

Frequently reached-for flags: --html-report <path> on any run-producing verb hands that run's artefacts to mavai, the family's report renderer, which the platform wheels carry (so pip install baseltest puts mavai on your path too, and basel --version names the one it carries); --baseline-dir, --verdict-dir, --explorations-dir, and --optimizations-dir relocate the artefact directories. Everything a run generates lands under _baseltest/.

Exit codes are contractual, made for CI: 0 success · 1 judgement failure (a declared bar or latency bound was breached) · 2 refusal (the service was never invoked: malformed file, unsupportable configuration) · 3 unsupportable (the evidence cannot carry the assertion in either direction). The getting-started guide walks through all of it, and the user guide is the complete reference — every verb, every file, every option.

The declarative core

baseltest is declarative-first. The primary way to author a test is a small, language-agnostic contract file — inputs, expectations, a service binding, a threshold — which baseltest turns into a full service contract evaluated by the statistical machinery. No statistical vocabulary is required to get a first honest result:

format: mavai-contract/1
contract: greeting-service-is-polite
service: greeting-service
inputs:
  - "Alice"
  - "Bob"
criteria:
  - threshold: 0.95
    contains: "hello"

What that core gives you today:

• Honest output: a declared threshold yields a statistical verdict with its uncertainty stated; no threshold yields a measurement explicitly labelled as an observation, never dressed up as a pass.
• Multiple criteria per contract: a service examined through several Bernoulli streams in one run — relevance at one bar, well-formedness at another.
• Structured-response checks: JSON, XML, and YAML transforms with standards-pinned path expressions (RFC 9535 JSONPath, XPath 1.0) — and the same path: expressions address the structured value of any transform you register in code.
• Measurement runs that persist a baseline artefact — the empirical record future regression tests verify against.

On the roadmap:

• An lm-eval bridge (separate package): mavai-grade statistics over lm-evaluation-harness runs, with baseltest as the statistical engine underneath.

Status

Pre-release (0.3.0.dev0). APIs and the contract-file surface are settling; nothing here is stable yet. If the approach interests you, mavai.org explains the methodology, and punit's user guide shows the mature end of the same ideas.

Licence

See LICENSE.