baseltest

Statistically honest testing for stochastic services, in Python.

Services built on LLMs, ML models, or randomised algorithms do not pass or fail a single invocation — they succeed at a rate. baseltest treats that rate as the thing under test: run the service repeatedly, judge each response against declared criteria, and render a verdict backed by real statistics (Wilson confidence bounds, feasibility-checked sample sizes) rather than a green tick over a lucky sample.

baseltest is the Python member of the mavai framework family, alongside punit (Java) and feotest (Rust). It shares their statistical methodology — every formula is validated against the family's statistical oracle — and expresses it in Python idioms rather than porting either framework.

Where the project stands

The statistics core (baseltest.statistics) is implemented and conformance-validated against the oracle's published reference cases: Wilson score construction, threshold derivation, feasibility checking, and sampling-power arithmetic, built on scipy/statsmodels.

The framework around it is in active development.

Try it in two minutes

The repository ships a ready-to-run example that needs no API key, no network, and no Python of your own — the service under test is simulated:

baseltest requires Python 3.11 or newer. Check what you have first:

python3 --version

If that prints 3.11+ you can use python3 below. Otherwise install a newer interpreter — it will live alongside your system Python, nothing is replaced:

  • macOS (Homebrew): brew install python@3.12 → the interpreter is python3.12. If the command isn't found afterwards, run brew link python@3.12 or use the full path $(brew --prefix python@3.12)/bin/python3.12.
  • Linux (Debian/Ubuntu): sudo apt install python3.12 python3.12-venv (on older releases, via the deadsnakes PPA).
  • Any platform, pyenv: pyenv install 3.12 && pyenv local 3.12.
  • Any platform, uv: uv venv --python 3.12 creates the venv below in one step.

Then — creating the venv with the new interpreter is the step that matters; installing 3.12 alone changes nothing until a venv is built from it:

git clone https://github.com/mavai-org/baseltest.git
cd baseltest

python3.12 -m venv venv          # or python3, if yours is already 3.11+
source venv/bin/activate
python --version                 # verify: must print 3.11+ — if not, stop and re-check the venv line
pip install -e ".[declarative]"   # the baseltest package ships the `basel` command

cd examples/simulated-service
basel test fortune-teller.yaml      # judge it against its declared bar
basel measure fortune-teller.yaml --samples 200   # or: record everything, persist a baseline

(If pip ever answers with Package 'baseltest' requires a different Python, that is this issue: the active pip still belongs to an older interpreter. deactivate, delete venv/, and recreate it with the 3.11+ interpreter as above.)

You'll see a verdict with its uncertainty stated — run the test a few times and watch the observed rate move while the conclusion stays statistically honest. The measure verb is the other posture over the same file: every criterion recorded, a baseline artefact persisted. A third verb, explore, sweeps a grid of service configurations declared in the services file and writes one descriptive artefact per configuration — triage before you measure. A fourth, optimize, searches the configuration space iteratively — a declared stepper proposes each next configuration, a scorer judges each iteration, and the full history is persisted as one artefact. Everything a run generates lands under _baseltest/ in the working directory. When you have a model credential to hand, examples/language-model/ runs a real language-model service from two small files — its grid pits two models against each other (GPT-4o-mini and Claude Haiku 4.5, with an optional entry for the fully open Swiss Apertus), so one basel explore run and one diff compare them on the same job; the examples README has the step-by-step, and the getting-started guide walks through all the verbs.

The command line

The baseltest package ships one command, basel, with five verbs. The contract file carries the claim; the verb carries the posture; the invocation carries the budget.

Verb What it does Sizing
basel check <contract.yaml> Validates the contract against its services file, bindings, and path: expressions — every load-time join, zero samples: the authoring loop's compile step. No sampling.
basel explore <contract.yaml> Runs every configuration in the service's grid and writes one descriptive artefact per configuration — triage, no verdicts. --samples-per-config (default 5; no count is ever refused as too small).
basel optimize <contract.yaml> [id] Runs one declared optimization: an iterative configuration search driven by its stepper, scored per iteration, the full history persisted as one artefact — descriptive, no verdicts. With several entries declared the id is required (or --all); never guessed. --samples-per-iteration (default 20).
basel measure <contract.yaml> --samples N Records every criterion and persists the baseline artefact — the durable record future empirical bars derive from, latency profile included. --samples is required: a measurement's budget is an experimental-design decision.
basel test <contract.yaml> Judges the thresholded criteria (and any declared latency bounds): a statistical verdict with its uncertainty stated, persisted as a verdict record. Empirical criteria are sized from your stated risk: --tolerate (or the criterion's tolerate: key) and --confidence compute the required n, prompted for on a terminal when unclaimed. Declared bars default to their feasibility minimum (a silently derived n above 100 is refused); --samples N sizes it yourself — explained, and confirmed when weak (--accept-weak-design for automation).

HTML reports are rendered by the mavai tool — mavai verdict|measure|explore|optimize <dir> [-o report.html] over the persisted artefacts. It is developed in the mavai-report project; obtain it from there, and see that project's README.md for installation instructions.

Frequently reached-for flags: --html-report <path> on any run-producing verb hands that run's artefacts to mavai, the family's report renderer, which the platform wheels carry (so pip install baseltest puts mavai on your path too, and basel --version names the one it carries); --baseline-dir, --verdict-dir, --explorations-dir, and --optimizations-dir relocate the artefact directories. Everything a run generates lands under _baseltest/.

Exit codes are contractual, made for CI: 0 success · 1 judgement failure (a declared bar or latency bound was breached) · 2 refusal (the service was never invoked: malformed file, unsupportable configuration) · 3 unsupportable (the evidence cannot carry the assertion in either direction). The getting-started guide walks through all of it, and the user guide is the complete reference — every verb, every file, every option.

The declarative core

baseltest is declarative-first. The primary way to author a test is a small, language-agnostic contract file — inputs, expectations, a service binding, a threshold — which baseltest turns into a full service contract evaluated by the statistical machinery. No statistical vocabulary is required to get a first honest result:

format: mavai-contract/1
contract: greeting-service-is-polite
service: greeting-service
inputs:
  - "Alice"
  - "Bob"
criteria:
  - threshold: 0.95
    contains: "hello"

What that core gives you today:

  • Honest output: a declared threshold yields a statistical verdict with its uncertainty stated; no threshold yields a measurement explicitly labelled as an observation, never dressed up as a pass.
  • Multiple criteria per contract: a service examined through several Bernoulli streams in one run — relevance at one bar, well-formedness at another.
  • Structured-response checks: JSON, XML, and YAML transforms with standards-pinned path expressions (RFC 9535 JSONPath, XPath 1.0) — and the same path: expressions address the structured value of any transform you register in code.
  • Measurement runs that persist a baseline artefact — the empirical record future regression tests verify against.

On the roadmap:

  • An lm-eval bridge (separate package): mavai-grade statistics over lm-evaluation-harness runs, with baseltest as the statistical engine underneath.

Status

Pre-release (0.3.0.dev0). APIs and the contract-file surface are settling; nothing here is stable yet. If the approach interests you, mavai.org explains the methodology, and punit's user guide shows the mature end of the same ideas.

Licence

See LICENSE.

Project links

Data verified by PyPI on Aug 11, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.24.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for michaelfranz from gravatar.com michaelfranz

Credits

Author: Mike Mannion

GitHub Statistics

Data verified by PyPI on Aug 11, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

Apache-2.0
View SPDX License List

Requires

Python >=3.11

Provides Extra

declarative dev

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

baseltest-0.22.1.tar.gz (515.2 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

baseltest-0.22.1-py3-none-win_amd64.whl (1.1 MB view details)

Uploaded Python 3Windows x86-64

baseltest-0.22.1-py3-none-manylinux_2_17_x86_64.whl (1.3 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ x86-64

baseltest-0.22.1-py3-none-manylinux_2_17_aarch64.whl (1.3 MB view details)

Uploaded Python 3manylinux: glibc 2.17+ ARM64

baseltest-0.22.1-py3-none-macosx_11_0_arm64.whl (1.1 MB view details)

Uploaded Python 3macOS 11.0+ ARM64

baseltest-0.22.1-py3-none-macosx_10_12_x86_64.whl (1.2 MB view details)

Uploaded Python 3macOS 10.12+ x86-64

baseltest-0.22.1-py3-none-any.whl (280.3 kB view details)

Uploaded Python 3

File details

Details for the file baseltest-0.22.1.tar.gz.

File metadata

  • Download URL: baseltest-0.22.1.tar.gz
  • Upload date:
  • Size: 515.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for baseltest-0.22.1.tar.gz
Algorithm Hash digest
SHA256 c47db5bce908dc7757510369ba43dcf54a9717c6dc128cc2196713ce1b414bad
MD5 622bab8e2fce663d969b904895a3206f
BLAKE2b-256 c961f4b95a8744e6b660933c41cc1c21c1e13831ff4d397807ea8146e7b52fd2

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.22.1.tar.gz:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.22.1-py3-none-win_amd64.whl.

File metadata

  • Download URL: baseltest-0.22.1-py3-none-win_amd64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: Python 3, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for baseltest-0.22.1-py3-none-win_amd64.whl
Algorithm Hash digest
SHA256 bb12cb78a8413bd4f222e080cb20ae6221d096607cf4990bcee0405cc3d9641e
MD5 0f6f629ad41efb9b971f9f19b7c0c2ad
BLAKE2b-256 a92f2b0c86198d2898a5d32158e5135f06319e94172036a5789c230b36d69f5f

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.22.1-py3-none-win_amd64.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.22.1-py3-none-manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for baseltest-0.22.1-py3-none-manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 dbc3a7e2156f7dbd58c3378c0e224ef83ecb5b9bae7c296ef6248254c34d0106
MD5 9183cc1c75b03161077a81acf830ff34
BLAKE2b-256 a8c25604dc42b8515eec946308263f163490e1d5ffeec3b599aaa4da5132ff40

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.22.1-py3-none-manylinux_2_17_x86_64.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.22.1-py3-none-manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for baseltest-0.22.1-py3-none-manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 f23d7f8b794b0d4050682e858f3f9b3b5e1e3a49cda518568e5ea4aa1385d5e2
MD5 af19435cd51c012d8285823d26d52ac7
BLAKE2b-256 65b510578de74b9398f6670d0f3cf40e784fcc6cfcbdc250366e8bc1e84ae810

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.22.1-py3-none-manylinux_2_17_aarch64.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.22.1-py3-none-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for baseltest-0.22.1-py3-none-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 facb1b1c0f02c3f67d223c9e2ac4d3f77a540457ed4f360ed0896e2afa0fe074
MD5 5db961723319fe5655d12bc04c3b95a1
BLAKE2b-256 9d0721816ed79499489a1a2e094f970580345453bff8750e09ccda6fe43f6033

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.22.1-py3-none-macosx_11_0_arm64.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.22.1-py3-none-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for baseltest-0.22.1-py3-none-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 45d8adab1c9a2e1c2e97f46ef6d13d2eeb5e12599fd0da7a25b2f17ef450e1c3
MD5 9ee919d7afdc2bbd15eae3ebe113538a
BLAKE2b-256 e03b1c9a8d8ab14081d41e0b39d68e44b48faf9a4468d35a055de4d9a2b5398b

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.22.1-py3-none-macosx_10_12_x86_64.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file baseltest-0.22.1-py3-none-any.whl.

File metadata

  • Download URL: baseltest-0.22.1-py3-none-any.whl
  • Upload date:
  • Size: 280.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for baseltest-0.22.1-py3-none-any.whl
Algorithm Hash digest
SHA256 e660cacb271534e53003ef82121297ea429e62ddfc6ce72de6bdb36c2ee2f9a1
MD5 b4f0a7aa7baef85a3f964604852853e6
BLAKE2b-256 bb9906511920e86d293f6706a353d0252797a0db8a9b75292c73be21d69c41fb

See more details on using hashes here.

Provenance

The following attestation bundles were made for baseltest-0.22.1-py3-none-any.whl:

Publisher: release.yml on mavai-org/baseltest

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.24.0

7 files

0.23.0

7 files

0.22.2

7 files

This release

0.22.1 This release

7 files

0.22.0

7 files

0.21.0

2 files

0.20.0

2 files

0.19.0

2 files

0.18.0

2 files

0.17.0

2 files

0.16.1

2 files

0.16.0

2 files

0.15.0

2 files

0.14.0

2 files

0.13.0

2 files

0.12.0

2 files

0.11.0

2 files

0.10.0

2 files

0.9.0

2 files

0.8.0

2 files

0.7.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page