baseltest

Statistically honest testing for stochastic services, in Python.

Services built on LLMs, ML models, or randomised algorithms do not pass or fail a single invocation — they succeed at a rate. baseltest treats that rate as the thing under test: run the service repeatedly, judge each response against declared criteria, and render a verdict backed by real statistics (Wilson confidence bounds, feasibility-checked sample sizes) rather than a green tick over a lucky sample.

baseltest is the Python member of the mavai framework family, alongside punit (Java) and feotest (Rust). It shares their statistical methodology — every formula is validated against the family's statistical oracle — and expresses it in Python idioms rather than porting either framework.

Where the project stands

The statistics core (baseltest.statistics) is implemented and conformance-validated against the oracle's published reference cases: Wilson score construction, threshold derivation, feasibility checking, and sampling-power arithmetic, built on scipy/statsmodels.

The framework around it is in active development.

Try it in two minutes

The repository ships a ready-to-run example that needs no API key, no network, and no Python of your own — the service under test is simulated:

baseltest requires Python 3.11 or newer. Check what you have first:

python3 --version

If that prints 3.11+ you can use python3 below. Otherwise install a newer interpreter — it will live alongside your system Python, nothing is replaced:

  • macOS (Homebrew): brew install python@3.12 → the interpreter is python3.12. If the command isn't found afterwards, run brew link python@3.12 or use the full path $(brew --prefix python@3.12)/bin/python3.12.
  • Linux (Debian/Ubuntu): sudo apt install python3.12 python3.12-venv (on older releases, via the deadsnakes PPA).
  • Any platform, pyenv: pyenv install 3.12 && pyenv local 3.12.
  • Any platform, uv: uv venv --python 3.12 creates the venv below in one step.

Then — creating the venv with the new interpreter is the step that matters; installing 3.12 alone changes nothing until a venv is built from it:

git clone https://github.com/mavai-org/baseltest.git
cd baseltest

python3.12 -m venv venv          # or python3, if yours is already 3.11+
source venv/bin/activate
python --version                 # verify: must print 3.11+ — if not, stop and re-check the venv line
pip install -e ".[declarative]"   # the baseltest package ships the `basel` command

cd examples/simulated-service
basel test fortune-teller.yaml      # judge it against its declared bar
basel measure fortune-teller.yaml --samples 200   # or: record everything, persist a baseline

(If pip ever answers with Package 'baseltest' requires a different Python, that is this issue: the active pip still belongs to an older interpreter. deactivate, delete venv/, and recreate it with the 3.11+ interpreter as above.)

You'll see a verdict with its uncertainty stated — run the test a few times and watch the observed rate move while the conclusion stays statistically honest. The measure verb is the other posture over the same file: every criterion recorded, a baseline artefact persisted. A third verb, explore, sweeps a grid of service configurations declared in the services file and writes one descriptive artefact per configuration — triage before you measure. A fourth, optimize, searches the configuration space iteratively — a declared stepper proposes each next configuration, a scorer judges each iteration, and the full history is persisted as one artefact. Everything a run generates lands under _baseltest/ in the working directory. When you have a model credential to hand, examples/language-model/ runs a real language-model service from two small files — its grid pits two models against each other (GPT-4o-mini and Claude Haiku 4.5, with an optional entry for the fully open Swiss Apertus), so one basel explore run and one diff compare them on the same job; the examples README has the step-by-step, and the getting-started guide walks through all the verbs.

The command line

The baseltest package ships one command, basel, with five verbs. The contract file carries the claim; the verb carries the posture; the invocation carries the budget.

Verb What it does Sizing
basel check <contract.yaml> Validates the contract against its services file, bindings, and path: expressions — every load-time join, zero samples: the authoring loop's compile step. No sampling.
basel explore <contract.yaml> Runs every configuration in the service's grid and writes one descriptive artefact per configuration — triage, no verdicts. --samples-per-config (default 5; no count is ever refused as too small).
basel optimize <contract.yaml> [id] Runs one declared optimization: an iterative configuration search driven by its stepper, scored per iteration, the full history persisted as one artefact — descriptive, no verdicts. With several entries declared the id is required (or --all); never guessed. --samples-per-iteration (default 20).
basel measure <contract.yaml> --samples N Records every criterion and persists the baseline artefact — the durable record future empirical bars derive from, latency profile included. --samples is required: a measurement's budget is an experimental-design decision.
basel test <contract.yaml> Judges the thresholded criteria (and any declared latency bounds): a statistical verdict with its uncertainty stated, persisted as a verdict record. Empirical criteria are sized from your stated risk: --tolerate (or the criterion's tolerate: key) and --confidence compute the required n, prompted for on a terminal when unclaimed. Declared bars default to their feasibility minimum (a silently derived n above 100 is refused); --samples N sizes it yourself — explained, and confirmed when weak (--accept-weak-design for automation).

HTML reports are rendered by the mavai tool — mavai verdict|measure|explore|optimize <dir> [-o report.html] over the persisted artefacts. It is developed in the mavai-report project; obtain it from there, and see that project's README.md for installation instructions.

Frequently reached-for flags: --html-report <path> on any run-producing verb hands that run's artefacts to mavai, the family's report renderer, which the platform wheels carry (so pip install baseltest puts mavai on your path too, and basel --version names the one it carries); --baseline-dir, --verdict-dir, --explorations-dir, and --optimizations-dir relocate the artefact directories. Everything a run generates lands under _baseltest/.

Exit codes are contractual, made for CI: 0 success · 1 judgement failure (a declared bar or latency bound was breached) · 2 refusal (the service was never invoked: malformed file, unsupportable configuration) · 3 unsupportable (the evidence cannot carry the assertion in either direction). The getting-started guide walks through all of it, and the user guide is the complete reference — every verb, every file, every option.

The declarative core

baseltest is declarative-first. The primary way to author a test is a small, language-agnostic contract file — inputs, expectations, a service binding, a threshold — which baseltest turns into a full service contract evaluated by the statistical machinery. No statistical vocabulary is required to get a first honest result:

format: mavai-contract/1
contract: greeting-service-is-polite
service: greeting-service
inputs:
  - "Alice"
  - "Bob"
criteria:
  - threshold: 0.95
    contains: "hello"

What that core gives you today:

  • Honest output: a declared threshold yields a statistical verdict with its uncertainty stated; no threshold yields a measurement explicitly labelled as an observation, never dressed up as a pass.
  • Multiple criteria per contract: a service examined through several Bernoulli streams in one run — relevance at one bar, well-formedness at another.
  • Structured-response checks: JSON, XML, and YAML transforms with standards-pinned path expressions (RFC 9535 JSONPath, XPath 1.0) — and the same path: expressions address the structured value of any transform you register in code.
  • Measurement runs that persist a baseline artefact — the empirical record future regression tests verify against.

On the roadmap:

  • An lm-eval bridge (separate package): mavai-grade statistics over lm-evaluation-harness runs, with baseltest as the statistical engine underneath.

Status

Pre-release (0.3.0.dev0). APIs and the contract-file surface are settling; nothing here is stable yet. If the approach interests you, mavai.org explains the methodology, and punit's user guide shows the mature end of the same ideas.

Licence

See LICENSE.

Project links

Data verified by PyPI on Aug 11, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.24.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for michaelfranz from gravatar.com michaelfranz

Credits

Author: Mike Mannion

GitHub Statistics

Data verified by PyPI on Aug 11, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

Apache-2.0
View SPDX License List

Requires

Python >=3.11

Provides Extra

declarative dev

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

baseltest-0.22.2.tar.gz (515.6 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

baseltest-0.22.2-py3-none-win_amd64.whl (1.1 MB view details)

Uploaded Python 3Windows x86-64

baseltest-0.22.2-py3-none-manylinux_2_17_x86_64.whl (1.3 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ x86-64

baseltest-0.22.2-py3-none-manylinux_2_17_aarch64.whl (1.3 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ ARM64

baseltest-0.22.2-py3-none-macosx_11_0_arm64.whl (1.2 MB view details)

Uploaded Python 3macOS 11.0+ ARM64

baseltest-0.22.2-py3-none-macosx_10_12_x86_64.whl (1.2 MB view details)

Uploaded Python 3macOS 10.12+ x86-64

baseltest-0.22.2-py3-none-any.whl (280.3 kB view details)

Uploaded Python 3

File details

Details for the file baseltest-0.22.2.tar.gz.

File metadata

  • Download URL: baseltest-0.22.2.tar.gz
  • Upload date:
  • Size: 515.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for baseltest-0.22.2.tar.gz
Algorithm Hash digest
SHA256 9d07ddbacddeb34188363c6848d400f8d56d0fefa8b3ad7957dbb81d09045855
MD5 daabaaae55feef202c164285080c4a43
BLAKE2b-256 85cd3e433b93702baf74c9e2a9d506dc575a13d9d5e118ba163801b703cb4e99

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.22.2.tar.gz:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.22.2-py3-none-win_amd64.whl.

File metadata

  • Download URL: baseltest-0.22.2-py3-none-win_amd64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: Python 3, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for baseltest-0.22.2-py3-none-win_amd64.whl
Algorithm Hash digest
SHA256 49843a6ccc1fc97d6d3db6bda5447f21b98cacd5c9aff9326ac0d9c159a0edb1
MD5 b727014d34757bc9c8ed9a0acb2e8ba8
BLAKE2b-256 fbe5091b962d0b05d4ce3b40c544e21900e9fb8dcd6d895471cbe2a557cf61bd

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.22.2-py3-none-win_amd64.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.22.2-py3-none-manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for baseltest-0.22.2-py3-none-manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 e409f63c6b0990ab81c0115aba681f4eadcc99181986c22b827d9c88877cbf07
MD5 d9f02e8b127ec1609f73eb88b9ccf478
BLAKE2b-256 c8e8aa2e8822d74b727ae28fdca51bbd75a8364e4213ab605f91503eee780e66

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.22.2-py3-none-manylinux_2_17_x86_64.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.22.2-py3-none-manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for baseltest-0.22.2-py3-none-manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 bdc2ba93c5d65572fcded6cd2830b24894fb6308ac0875135820540d6092eba2
MD5 278b925b8ba7f62fbf2db6c4040ff8fe
BLAKE2b-256 603dca0f2eda340f0090d3785bd014c8c4a65a4b3851936c3f9485d6428d7501

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.22.2-py3-none-manylinux_2_17_aarch64.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.22.2-py3-none-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for baseltest-0.22.2-py3-none-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 83fd9e9541a0a3990c48cc602775779644fdd2095724865ba8e91c071a9498bb
MD5 924c5583a2ff33afcb9fdf85d9a5959e
BLAKE2b-256 69b3e1289977f8a89751576bebef0f885c340f87ea25682658b121d2000a6237

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.22.2-py3-none-macosx_11_0_arm64.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.22.2-py3-none-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for baseltest-0.22.2-py3-none-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 adeaba6d4c3e0fb5e75afcfd7da872b35f87276bef5f65c50f20727ff7ff98b5
MD5 f6c919292065df472d1aa41955849f71
BLAKE2b-256 d481190f46335fbe481eb06e15e5cac69cd6f7e4fa6c9fc1fffe720ecd9bf7bf

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.22.2-py3-none-macosx_10_12_x86_64.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.22.2-py3-none-any.whl.

File metadata

  • Download URL: baseltest-0.22.2-py3-none-any.whl
  • Upload date:
  • Size: 280.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for baseltest-0.22.2-py3-none-any.whl
Algorithm Hash digest
SHA256 cf8c55aa49818651e9e69969a1ca166e3fc10f505149343af660c0db1d511f08
MD5 2753f5bef9551537b955bee807ee2916
BLAKE2b-256 749ef132b081135a044d979ad790129971ca2ee4442acb6c04404ebecc752063

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.22.2-py3-none-any.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.24.0

7 files

0.23.0

7 files

This release

0.22.2 This release

7 files

0.22.1

7 files

0.22.0

7 files

0.21.0

2 files

0.20.0

2 files

0.19.0

2 files

0.18.0

2 files

0.17.0

2 files

0.16.1

2 files

0.16.0

2 files

0.15.0

2 files

0.14.0

2 files

0.13.0

2 files

0.12.0

2 files

0.11.0

2 files

0.10.0

2 files

0.9.0

2 files

0.8.0

2 files

0.7.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page