baseltest

Statistically honest testing for stochastic services, in Python.

Services built on LLMs, ML models, or randomised algorithms do not pass or fail a single invocation — they succeed at a rate. baseltest treats that rate as the thing under test: run the service repeatedly, judge each response against declared criteria, and render a verdict backed by real statistics (Wilson confidence bounds, feasibility-checked sample sizes) rather than a green tick over a lucky sample.

baseltest is the Python member of the mavai framework family, alongside punit (Java) and feotest (Rust). It shares their statistical methodology — every formula is validated against the family's statistical oracle — and expresses it in Python idioms rather than porting either framework.

Where the project stands

The statistics core (baseltest.statistics) is implemented and conformance-validated against the oracle's published reference cases: Wilson score construction, threshold derivation, feasibility checking, and sampling-power arithmetic, built on scipy/statsmodels.

The framework around it is in active development.

Try it in two minutes

The repository ships a ready-to-run example that needs no API key, no network, and no Python of your own — the service under test is simulated:

baseltest requires Python 3.11 or newer. Check what you have first:

python3 --version

If that prints 3.11+ you can use python3 below. Otherwise install a newer interpreter — it will live alongside your system Python, nothing is replaced:

  • macOS (Homebrew): brew install python@3.12 → the interpreter is python3.12. If the command isn't found afterwards, run brew link python@3.12 or use the full path $(brew --prefix python@3.12)/bin/python3.12.
  • Linux (Debian/Ubuntu): sudo apt install python3.12 python3.12-venv (on older releases, via the deadsnakes PPA).
  • Any platform, pyenv: pyenv install 3.12 && pyenv local 3.12.
  • Any platform, uv: uv venv --python 3.12 creates the venv below in one step.

Then — creating the venv with the new interpreter is the step that matters; installing 3.12 alone changes nothing until a venv is built from it:

git clone https://github.com/mavai-org/baseltest.git
cd baseltest

python3.12 -m venv venv          # or python3, if yours is already 3.11+
source venv/bin/activate
python --version                 # verify: must print 3.11+ — if not, stop and re-check the venv line
pip install -e ".[declarative]"   # the baseltest package ships the `basel` command

cd examples/simulated-service
basel test fortune-teller.yaml      # judge it against its declared bar
basel measure fortune-teller.yaml --samples 200   # or: record everything, persist a baseline

(If pip ever answers with Package 'baseltest' requires a different Python, that is this issue: the active pip still belongs to an older interpreter. deactivate, delete venv/, and recreate it with the 3.11+ interpreter as above.)

You'll see a verdict with its uncertainty stated — run the test a few times and watch the observed rate move while the conclusion stays statistically honest. The measure verb is the other posture over the same file: every criterion recorded, a baseline artefact persisted. A third verb, explore, sweeps a grid of service configurations declared in the services file and writes one descriptive artefact per configuration — triage before you measure. A fourth, optimize, searches the configuration space iteratively — a declared stepper proposes each next configuration, a scorer judges each iteration, and the full history is persisted as one artefact. Everything a run generates lands under _baseltest/ in the working directory. When you have a model credential to hand, examples/language-model/ runs a real language-model service from two small files — its grid pits two models against each other (GPT-4o-mini and Claude Haiku 4.5, with an optional entry for the fully open Swiss Apertus), so one basel explore run and one diff compare them on the same job; the examples README has the step-by-step, and the getting-started guide walks through all the verbs.

The command line

The baseltest package ships one command, basel, with five verbs. The contract file carries the claim; the verb carries the posture; the invocation carries the budget.

Verb What it does Sizing
basel check <contract.yaml> Validates the contract against its services file, bindings, and path: expressions — every load-time join, zero samples: the authoring loop's compile step. No sampling.
basel explore <contract.yaml> Runs every configuration in the service's grid and writes one descriptive artefact per configuration — triage, no verdicts. --samples-per-config (default 5; no count is ever refused as too small).
basel optimize <contract.yaml> [id] Runs one declared optimization: an iterative configuration search driven by its stepper, scored per iteration, the full history persisted as one artefact — descriptive, no verdicts. With several entries declared the id is required (or --all); never guessed. --samples-per-iteration (default 20).
basel measure <contract.yaml> --samples N Records every criterion and persists the baseline artefact — the durable record future empirical bars derive from, latency profile included. --samples is required: a measurement's budget is an experimental-design decision.
basel test <contract.yaml> Judges the thresholded criteria (and any declared latency bounds): a statistical verdict with its uncertainty stated, persisted as a verdict record. Empirical criteria are sized from your stated risk: --tolerate (or the criterion's tolerate: key) and --confidence compute the required n, prompted for on a terminal when unclaimed. Declared bars default to their feasibility minimum (a silently derived n above 100 is refused); --samples N sizes it yourself — explained, and confirmed when weak (--accept-weak-design for automation).

HTML reports are rendered by the mavai tool — mavai verdict|measure|explore|optimize <dir> [-o report.html] over the persisted artefacts. It is developed in the mavai-report project; obtain it from there, and see that project's README.md for installation instructions.

Frequently reached-for flags: --html-report <path> on any run-producing verb hands that run's artefacts to mavai, the family's report renderer, which the platform wheels carry (so pip install baseltest puts mavai on your path too, and basel --version names the one it carries); --baseline-dir, --verdict-dir, --explorations-dir, and --optimizations-dir relocate the artefact directories. Everything a run generates lands under _baseltest/.

Exit codes are contractual, made for CI: 0 success · 1 judgement failure (a declared bar or latency bound was breached) · 2 refusal (the service was never invoked: malformed file, unsupportable configuration) · 3 unsupportable (the evidence cannot carry the assertion in either direction). The getting-started guide walks through all of it, and the user guide is the complete reference — every verb, every file, every option.

The declarative core

baseltest is declarative-first. The primary way to author a test is a small, language-agnostic contract file — inputs, expectations, a service binding, a threshold — which baseltest turns into a full service contract evaluated by the statistical machinery. No statistical vocabulary is required to get a first honest result:

format: mavai-contract/1
contract: greeting-service-is-polite
service: greeting-service
inputs:
  - "Alice"
  - "Bob"
criteria:
  - threshold: 0.95
    contains: "hello"

What that core gives you today:

  • Honest output: a declared threshold yields a statistical verdict with its uncertainty stated; no threshold yields a measurement explicitly labelled as an observation, never dressed up as a pass.
  • Multiple criteria per contract: a service examined through several Bernoulli streams in one run — relevance at one bar, well-formedness at another.
  • Structured-response checks: JSON, XML, and YAML transforms with standards-pinned path expressions (RFC 9535 JSONPath, XPath 1.0) — and the same path: expressions address the structured value of any transform you register in code.
  • Measurement runs that persist a baseline artefact — the empirical record future regression tests verify against.

On the roadmap:

  • An lm-eval bridge (separate package): mavai-grade statistics over lm-evaluation-harness runs, with baseltest as the statistical engine underneath.

Status

Pre-release (0.3.0.dev0). APIs and the contract-file surface are settling; nothing here is stable yet. If the approach interests you, mavai.org explains the methodology, and punit's user guide shows the mature end of the same ideas.

Licence

See LICENSE.

Project links

Data verified by PyPI on Aug 12, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.24.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for michaelfranz from gravatar.com michaelfranz

Credits

Author: Mike Mannion

GitHub Statistics

Data verified by PyPI on Aug 12, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

Apache-2.0
View SPDX License List

Requires

Python >=3.11

Provides Extra

declarative dev

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

baseltest-0.23.0.tar.gz (536.3 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

baseltest-0.23.0-py3-none-win_amd64.whl (1.1 MB view details)

Uploaded Python 3Windows x86-64

baseltest-0.23.0-py3-none-manylinux_2_17_x86_64.whl (1.3 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ x86-64

baseltest-0.23.0-py3-none-manylinux_2_17_aarch64.whl (1.3 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ ARM64

baseltest-0.23.0-py3-none-macosx_11_0_arm64.whl (1.2 MB view details)

Uploaded Python 3macOS 11.0+ ARM64

baseltest-0.23.0-py3-none-macosx_10_12_x86_64.whl (1.2 MB view details)

Uploaded Python 3macOS 10.12+ x86-64

baseltest-0.23.0-py3-none-any.whl (293.0 kB view details)

Uploaded Python 3

File details

Details for the file baseltest-0.23.0.tar.gz.

File metadata

  • Download URL: baseltest-0.23.0.tar.gz
  • Upload date:
  • Size: 536.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for baseltest-0.23.0.tar.gz
Algorithm Hash digest
SHA256 b699a6180c31056144f45d16a779c60f64c4c6ee24162246854d1e0357960518
MD5 d58ba9df716755c76bc2ca1c425cc23e
BLAKE2b-256 657b8a272241fb201eba0539120245a3ddb2d827c0c8ab167f79274bea46a6e0

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.23.0.tar.gz:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.23.0-py3-none-win_amd64.whl.

File metadata

  • Download URL: baseltest-0.23.0-py3-none-win_amd64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: Python 3, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for baseltest-0.23.0-py3-none-win_amd64.whl
Algorithm Hash digest
SHA256 98d8fe4190dcac8420f94cba7c1e88e6243e97dea79cd5eb01f0df1189d8b31c
MD5 f981727ed24ef4bfec98aacf6104b73d
BLAKE2b-256 87334419bfb1c2a5dbe7819998f05b7a454f28fc95181732676e1f7e10b1d2b7

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.23.0-py3-none-win_amd64.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.23.0-py3-none-manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for baseltest-0.23.0-py3-none-manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 a6c1520e8708e4cc990a3fc28757611d4fe0bb080bbcca4c64387680e3244167
MD5 145ac3e9e479b25eb5b20b042d37b269
BLAKE2b-256 44d48a225188d1ba902510c9cbca8c4a2b6cb24b4c5a8a296502e6f00ffefd99

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.23.0-py3-none-manylinux_2_17_x86_64.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.23.0-py3-none-manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for baseltest-0.23.0-py3-none-manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 9f228d71549d7f174cbbafd353709f4312257ec9dbd4803fd857b91e48c393b0
MD5 7a2062759098fd709dd0d736e2ff9809
BLAKE2b-256 8c73eda74809d22b10b2976105f4ccd50f2ae25399b53b8c8ba61843d86b2fc5

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.23.0-py3-none-manylinux_2_17_aarch64.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.23.0-py3-none-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for baseltest-0.23.0-py3-none-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 e7d025a269268ca41da08022bc9e7547ba44de449d60b4234a30075815b7b65e
MD5 d90e3127df09c3a07cfe89d29f04ae95
BLAKE2b-256 f76ef9f11f95b63ed2c1e1e48cfba349c61939d687e8c0a0819cee38256906df

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.23.0-py3-none-macosx_11_0_arm64.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.23.0-py3-none-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for baseltest-0.23.0-py3-none-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 7321e2d3507155f3ad0b440d105dc906dc723528d1b21733383df348c4c8305b
MD5 203a34c331a7bea52e603cc4c2e09fe3
BLAKE2b-256 69d0f642bb705f24623406eed9499da6cfa7b6c5efbfe06a22107c45b0c337ad

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.23.0-py3-none-macosx_10_12_x86_64.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.23.0-py3-none-any.whl.

File metadata

  • Download URL: baseltest-0.23.0-py3-none-any.whl
  • Upload date:
  • Size: 293.0 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for baseltest-0.23.0-py3-none-any.whl
Algorithm Hash digest
SHA256 77e48f939075c04302faa5337d8836dc80c43c0ab0781618761c239c634125f3
MD5 4023b84d87ef4b81d7cb90fc80508a82
BLAKE2b-256 1edd901c7b233e0ef3912bb5d058dd5cafbd5187f13477fb0107d66e4778e2d8

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.23.0-py3-none-any.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.24.0

7 files

This release

0.23.0 This release

7 files

0.22.2

7 files

0.22.1

7 files

0.22.0

7 files

0.21.0

2 files

0.20.0

2 files

0.19.0

2 files

0.18.0

2 files

0.17.0

2 files

0.16.1

2 files

0.16.0

2 files

0.15.0

2 files

0.14.0

2 files

0.13.0

2 files

0.12.0

2 files

0.11.0

2 files

0.10.0

2 files

0.9.0

2 files

0.8.0

2 files

0.7.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page