baseltest

Statistically honest testing for stochastic services, in Python.

Services built on LLMs, ML models, or randomised algorithms do not pass or fail a single invocation — they succeed at a rate. baseltest treats that rate as the thing under test: run the service repeatedly, judge each response against declared criteria, and render a verdict backed by real statistics (Wilson confidence bounds, feasibility-checked sample sizes) rather than a green tick over a lucky sample.

baseltest is the Python member of the mavai framework family, alongside punit (Java) and feotest (Rust). It shares their statistical methodology — every formula is validated against the family's statistical oracle — and expresses it in Python idioms rather than porting either framework.

Where the project stands

The statistics core (baseltest.statistics) is implemented and conformance-validated against the oracle's published reference cases: Wilson score construction, threshold derivation, feasibility checking, and sampling-power arithmetic, built on scipy/statsmodels.

The framework around it is in active development.

Try it in two minutes

The repository ships a ready-to-run example that needs no API key, no network, and no Python of your own — the service under test is simulated:

baseltest requires Python 3.11 or newer. Check what you have first:

python3 --version

If that prints 3.11+ you can use python3 below. Otherwise install a newer interpreter — it will live alongside your system Python, nothing is replaced:

  • macOS (Homebrew): brew install python@3.12 → the interpreter is python3.12. If the command isn't found afterwards, run brew link python@3.12 or use the full path $(brew --prefix python@3.12)/bin/python3.12.
  • Linux (Debian/Ubuntu): sudo apt install python3.12 python3.12-venv (on older releases, via the deadsnakes PPA).
  • Any platform, pyenv: pyenv install 3.12 && pyenv local 3.12.
  • Any platform, uv: uv venv --python 3.12 creates the venv below in one step.

Then — creating the venv with the new interpreter is the step that matters; installing 3.12 alone changes nothing until a venv is built from it:

git clone https://github.com/mavai-org/baseltest.git
cd baseltest

python3.12 -m venv venv          # or python3, if yours is already 3.11+
source venv/bin/activate
python --version                 # verify: must print 3.11+ — if not, stop and re-check the venv line
pip install -e ".[declarative]"   # the baseltest package ships the `basel` command

cd examples/simulated-service
basel test fortune-teller.yaml      # judge it against its declared bar
basel measure fortune-teller.yaml --samples 200   # or: record everything, persist a baseline

(If pip ever answers with Package 'baseltest' requires a different Python, that is this issue: the active pip still belongs to an older interpreter. deactivate, delete venv/, and recreate it with the 3.11+ interpreter as above.)

You'll see a verdict with its uncertainty stated — run the test a few times and watch the observed rate move while the conclusion stays statistically honest. The measure verb is the other posture over the same file: every criterion recorded, a baseline artefact persisted. A third verb, explore, sweeps a grid of service configurations declared in the services file and writes one descriptive artefact per configuration — triage before you measure. A fourth, optimize, searches the configuration space iteratively — a declared stepper proposes each next configuration, a scorer judges each iteration, and the full history is persisted as one artefact. Everything a run generates lands under _baseltest/ in the working directory. When you have a model credential to hand, examples/language-model/ runs a real language-model service from two small files — its grid pits two models against each other (GPT-4o-mini and Claude Haiku 4.5, with an optional entry for the fully open Swiss Apertus), so one basel explore run and one diff compare them on the same job; the examples README has the step-by-step, and the getting-started guide walks through all the verbs.

The command line

The baseltest package ships one command, basel, with five verbs. The contract file carries the claim; the verb carries the posture; the invocation carries the budget.

Verb What it does Sizing
basel check <contract.yaml> Validates the contract against its services file, bindings, and path: expressions — every load-time join, zero samples: the authoring loop's compile step. No sampling.
basel explore <contract.yaml> Runs every configuration in the service's grid and writes one descriptive artefact per configuration — triage, no verdicts. --samples-per-config (default 5; no count is ever refused as too small).
basel optimize <contract.yaml> [id] Runs one declared optimization: an iterative configuration search driven by its stepper, scored per iteration, the full history persisted as one artefact — descriptive, no verdicts. With several entries declared the id is required (or --all); never guessed. --samples-per-iteration (default 20).
basel measure <contract.yaml> --samples N Records every criterion and persists the baseline artefact — the durable record future empirical bars derive from, latency profile included. --samples is required: a measurement's budget is an experimental-design decision.
basel test <contract.yaml> Judges the thresholded criteria (and any declared latency bounds): a statistical verdict with its uncertainty stated, persisted as a verdict record. Empirical criteria are sized from your stated risk: --tolerate (or the criterion's tolerate: key) and --confidence compute the required n, prompted for on a terminal when unclaimed. Declared bars default to their feasibility minimum (a silently derived n above 100 is refused); --samples N sizes it yourself — explained, and confirmed when weak (--accept-weak-design for automation).

HTML reports are rendered by the mavai tool — mavai verdict|measure|explore|optimize <dir> [-o report.html] over the persisted artefacts. It is developed in the mavai-report project; obtain it from there, and see that project's README.md for installation instructions.

Frequently reached-for flags: --html-report <path> on any run-producing verb hands that run's artefacts to mavai, the family's report renderer, which the platform wheels carry (so pip install baseltest puts mavai on your path too, and basel --version names the one it carries); --baseline-dir, --verdict-dir, --explorations-dir, and --optimizations-dir relocate the artefact directories. Everything a run generates lands under _baseltest/.

Exit codes are contractual, made for CI: 0 success · 1 judgement failure (a declared bar or latency bound was breached) · 2 refusal (the service was never invoked: malformed file, unsupportable configuration) · 3 unsupportable (the evidence cannot carry the assertion in either direction). The getting-started guide walks through all of it, and the user guide is the complete reference — every verb, every file, every option.

The declarative core

baseltest is declarative-first. The primary way to author a test is a small, language-agnostic contract file — inputs, expectations, a service binding, a threshold — which baseltest turns into a full service contract evaluated by the statistical machinery. No statistical vocabulary is required to get a first honest result:

format: mavai-contract/1
contract: greeting-service-is-polite
service: greeting-service
inputs:
  - "Alice"
  - "Bob"
criteria:
  - threshold: 0.95
    contains: "hello"

What that core gives you today:

  • Honest output: a declared threshold yields a statistical verdict with its uncertainty stated; no threshold yields a measurement explicitly labelled as an observation, never dressed up as a pass.
  • Multiple criteria per contract: a service examined through several Bernoulli streams in one run — relevance at one bar, well-formedness at another.
  • Structured-response checks: JSON, XML, and YAML transforms with standards-pinned path expressions (RFC 9535 JSONPath, XPath 1.0) — and the same path: expressions address the structured value of any transform you register in code.
  • Measurement runs that persist a baseline artefact — the empirical record future regression tests verify against.

On the roadmap:

  • An lm-eval bridge (separate package): mavai-grade statistics over lm-evaluation-harness runs, with baseltest as the statistical engine underneath.

Status

Pre-release (0.3.0.dev0). APIs and the contract-file surface are settling; nothing here is stable yet. If the approach interests you, mavai.org explains the methodology, and punit's user guide shows the mature end of the same ideas.

Licence

See LICENSE.

Project links

Data verified by PyPI on Aug 17, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for michaelfranz from gravatar.com michaelfranz

Credits

Author: Mike Mannion

GitHub Statistics

Data verified by PyPI on Aug 17, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

Apache-2.0
View SPDX License List

Requires

Python >=3.11

Provides Extra

declarative dev

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

baseltest-0.24.0.tar.gz (540.6 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

baseltest-0.24.0-py3-none-win_amd64.whl (1.1 MB view details)

Uploaded Python 3Windows x86-64

baseltest-0.24.0-py3-none-manylinux_2_17_x86_64.whl (1.3 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ x86-64

baseltest-0.24.0-py3-none-manylinux_2_17_aarch64.whl (1.3 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ ARM64

baseltest-0.24.0-py3-none-macosx_11_0_arm64.whl (1.2 MB view details)

Uploaded Python 3macOS 11.0+ ARM64

baseltest-0.24.0-py3-none-macosx_10_12_x86_64.whl (1.2 MB view details)

Uploaded Python 3macOS 10.12+ x86-64

baseltest-0.24.0-py3-none-any.whl (294.4 kB view details)

Uploaded Python 3

File details

Details for the file baseltest-0.24.0.tar.gz.

File metadata

  • Download URL: baseltest-0.24.0.tar.gz
  • Upload date:
  • Size: 540.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for baseltest-0.24.0.tar.gz
Algorithm Hash digest
SHA256 af7c2a5ccaa50ed2d4b572dbfd1fe00b3d18a60eec689c5d2269f0f843106560
MD5 7baa3f9121f94330f6fab01b49d8fdb7
BLAKE2b-256 685ab3079e8372ebab5bc7c7ae1a8254183150f7fd326ba308ba5595e0a89202

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.24.0.tar.gz:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.24.0-py3-none-win_amd64.whl.

File metadata

  • Download URL: baseltest-0.24.0-py3-none-win_amd64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: Python 3, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for baseltest-0.24.0-py3-none-win_amd64.whl
Algorithm Hash digest
SHA256 d6a5590bfcbb53b4aca6bcef078cb9cb8d47d1baf42c9b87cdec0b7f048ee18f
MD5 b65076e9140570d52f6a5231df9aa006
BLAKE2b-256 e161f76dc7e0b6e5347f300263d07673ccfc8df4622240e1821e86c284345170

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.24.0-py3-none-win_amd64.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.24.0-py3-none-manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for baseltest-0.24.0-py3-none-manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 bacd3503e98807c7af504a6a34eaae4e05646593b47d9923183e88bbe89f9be9
MD5 940cb0d94c511918d82408faf089c6ba
BLAKE2b-256 063997f0676bb7115b72ae670b16c9aae980ca10f39ee775a350e6ae2304c854

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.24.0-py3-none-manylinux_2_17_x86_64.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.24.0-py3-none-manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for baseltest-0.24.0-py3-none-manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 3895224e683afc4f708291d261a4a62bb62b12815a44babf014f0a291bf741eb
MD5 773668a954522501ecf697de27b06b8d
BLAKE2b-256 563bb09833642e2d3ba0275b8c34e56f720af487e48b401521de89f53e0d0608

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.24.0-py3-none-manylinux_2_17_aarch64.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.24.0-py3-none-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for baseltest-0.24.0-py3-none-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 b7fe69ef2327def698c9f3ced95a05d25ddabf9628a056abad88b93c28bb11ad
MD5 05b72f5e3cfa0c133c91bb9f3a9ef47b
BLAKE2b-256 256ac9c3aabc9a41467c4cfa2b80f7e32783d2a3f788f158b809c885f4f73bcb

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.24.0-py3-none-macosx_11_0_arm64.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.24.0-py3-none-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for baseltest-0.24.0-py3-none-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 521625a14619d311a57885f4c61de69c3a7c6a0e99053b9b7243fe50d37b91be
MD5 46b1d56aa516c91631bdef5c02637c13
BLAKE2b-256 a901506fac061bf39afba04139d6ab96522a5a7c7719c2ee523ca0d8f51b21c7

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.24.0-py3-none-macosx_10_12_x86_64.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.24.0-py3-none-any.whl.

File metadata

  • Download URL: baseltest-0.24.0-py3-none-any.whl
  • Upload date:
  • Size: 294.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for baseltest-0.24.0-py3-none-any.whl
Algorithm Hash digest
SHA256 52a90343c3fa0fdaa7b0d12c77e91e9ae3e23a8f6dec4be7e4622d46f4c9e039
MD5 b58fa8ec266b06016f7109912d1da39c
BLAKE2b-256 1e5c11df0ba7d538fd806e9e8f4c0e2e14e413076050e157e409c289037cfde3

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.24.0-py3-none-any.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.24.0 This release

7 files

0.23.0

7 files

0.22.2

7 files

0.22.1

7 files

0.22.0

7 files

0.21.0

2 files

0.20.0

2 files

0.19.0

2 files

0.18.0

2 files

0.17.0

2 files

0.16.1

2 files

0.16.0

2 files

0.15.0

2 files

0.14.0

2 files

0.13.0

2 files

0.12.0

2 files

0.11.0

2 files

0.10.0

2 files

0.9.0

2 files

0.8.0

2 files

0.7.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page