Skip to main content

AnytimeMonitor — adaptive controller for TTS-MOS listening tests

AnytimeMonitor watches a MOS listening test while it runs and tells the test admin, for every pair of systems, whether the data already supports a verdict — and if not, which lever (more items, or more listeners) closes the pair fastest.

The statistical core is an anytime-valid confidence sequence (Howard, Ramdas, McAuliffe & Sekhon, 2021) on per-item paired score differences. Anytime validity means the coverage guarantee holds at any data-dependent stopping time: the admin can look at the verdicts as often as they like, stop whenever a pair resolves, and the reported confidence interval is still valid. No peeking penalty bookkeeping, no pre-registered sample size required.

This is a research prototype (v0.9). It accompanies a position paper on budget allocation in TTS evaluation (under submission, Computer Speech & Language Special Issue on the evaluation of speech and speech synthesis).

What the admin sees

For each pair of systems (A, B), one of six verdicts:

Verdict Meaning
STOP The anytime-valid CI excludes 0 (0 ∉ CI): the pair is separated.
STOP_EQUIV Strict TOST equivalence: CI ⊂ ±δ_equiv and the half-width Ψ_eff ≤ δ_equiv/2. The pair is statistically equivalent within tolerance.
CONTINUE +N Neither verdict yet. The tool predicts how many more items (+N) or listeners (+L) would close the pair, and which terminal (STOP or STOP_EQUIV) it is heading for. The headline is the cheaper option.
WAIT The listener×system variance component cannot be estimated yet (fewer than 3 listeners with ≥ 2 paired items). Structural: keep collecting before trusting any verdict.
CORPUS_LIMIT The item budget is exhausted and the listener lever can no longer close the pair. Terminal.
LISTENER_LIMIT The listener budget is exhausted and the item lever can no longer close the pair. Terminal.

Verdict naming. STOP is the separation verdict the companion paper calls STOP_SEP. STOP is the current public label and will be renamed to STOP_SEP in a future release (to pair symmetrically with STOP_EQUIV); code matching on "STOP" should expect that change.

Verdicts are recomputed from scratch at every call — the controller is queried on demand (a handful of times over a test lifecycle), not in a real-time loop. A STOP is not sticky: if you keep collecting past it, a borderline pair can revert to CONTINUE. This is honest anytime-CI behaviour (the guarantee is on coverage, not on the monotonicity of the observed difference); in practice the admin acts on the first STOP.

Install

pip install -e .          # from the repository root (pyproject.toml)
pip install -e ".[ui]"    # + Streamlit demo UI
pip install -e ".[test]"  # + pytest; then run `pytest`

Requires Python ≥ 3.10. Pinned versions used for development are in requirements.txt.

Quick start

Programmatic

import pandas as pd
from anytimemonitor import AnytimeMonitor

ctrl = AnytimeMonitor(mde_threshold=0.30)   # δ_sep, in MOS units

# Feed ratings as they arrive — long format, one row per judgment
ctrl.add_ratings(pd.DataFrame({
    "listener_id": ...,   # who rated
    "item":        ...,   # which sentence / stimulus
    "system":      ...,   # which TTS system
    "rating":      ...,   # the MOS score
}))

print(ctrl.report())                  # all pairs
dec = ctrl.decision("sysA", "sysB")   # one pair, structured
print(dec.verdict, dec.ci_lo, dec.ci_hi, dec.n_more_items)

CLI demo

Feed a ratings CSV to the controller batch by batch and watch the per-pair verdicts tighten. With no arguments it runs on a bundled synthetic sample:

python -m anytimemonitor.demo                          # bundled sample
python -m anytimemonitor.demo --input my_ratings.csv   # your own long-format CSV

Streamlit demo UI

streamlit run anytimemonitor/ui/app.py

Upload a ratings CSV (or pick a bundled synthetic example from sample_uploads/), watch the controller replay the stream batch-by-batch, and drag the time-machine slider to see the verdict and two-lever advice at any past timestep. See ui/README.md.

How it works

For a pair (A, B), the controller builds per-item paired differences d_j = mean rating of item j under A minus under B, and maintains a sub-Gaussian normal-mixture confidence sequence on their mean (Howard et al. 2021). Two ingredients adapt this to listening-test data, where ratings are clustered by listener as well as by item:

  • Bivariate cluster bootstrap SE. Resampling items gives the i.i.d.-part SE that the anytime boundary multiplies; resampling listeners gives the listener-pool floor ("plancher") — the precision ceiling that no amount of extra items can break. The two enter the half-width as a sum of squares, the item side under the anytime multiplier at level α₁ and the floor under its own critical value z_{1−α₂/2}, with α₁ + α₂ = α. When the data is too sparse to bootstrap, the controller falls back to the confidence sequence's internal running variance proxy.
  • Variance decomposition layer. Method-of-moments estimates of the system×item and residual components (per system, with a Half-Cauchy Bayesian regulariser that prevents the truncation-at-zero bias of raw MoM), plus a per-pair estimate of the listener×system component. These feed the two-lever prediction: +N items shrinks the interaction term, +L listeners lowers the floor.

The price of anytime validity is a wider interval than a fixed-n test: the item side carries the sequence's √(log t) multiplier and the floor carries a quantile rather than a single standard deviation. The gap is modest once the floor dominates and several-fold at small item counts, where the multiplier is at its largest. On top of that comes deliberate conservatism early on: below avci_t_min_ebernstein items the variance proxy falls back to the worst-case Hoeffding bound, so the first batches produce wide intervals. The controller is therefore most useful from a few dozen rated items per system onward.

The internal architecture is documented in Architecture.md (and ui/Architecture.md for the demo UI); the deeper design notes for the confidence sequence are in AVCI_PLAN.md (development repository only, not shipped in the public snapshot).

Key parameters

All knobs are constructor arguments of AnytimeMonitor; the defaults are deliberately conservative.

Parameter Default Role
mde_threshold (δ_sep) (required) The smallest effect worth resolving (MOS): seeds delta_equiv and the reported MDE. It does not gate STOP — separation fires on 0 ∉ CI regardless.
delta_equiv None (= mde_threshold) δ_equiv — equivalence margin for STOP_EQUIV. Set it below mde_threshold to create an indifference zone.
alpha 0.05 Nominal level of the confidence sequence.
power 0.80 Only sets the reported-MDE scale factor (z_{α/2} + z_power); does not affect stopping.
n_max_corpus, l_max_listeners None Item / listener budgets; enable the *_LIMIT terminal verdicts.
avci_n_planned 240 (§7 census median budget) Planned item budget; the controller derives η = 1/√(N_planned) (Howard's a-priori rule).
avci_eta None (→ 1/√avci_n_planned) Set the Howard 2021 mixing parameter η by hand instead of via avci_n_planned — η is fixed a priori, never tuned to data. Pass one or the other, not both (raises otherwise).
avci_lissys_floor_r0 0.36 (measured r_lisSys band frontier) A-priori guard on the listener floor: bounds it below by √(2·σ²₀/L) with σ²₀ = 1.5·r₀·σ̂²_eval (the ×1.5 is an MC-calibrated validity margin). Protects coverage when a small pool under-estimates σ²_lisSys. 0 disables.
avci_lissys_floor_sigma2 None Absolute σ²₀ override for the guard (expert path, no extra margin). Exclusive with avci_lissys_floor_r0 (raises otherwise). The per-pair Decision.lissys_prior_exceeded flags when data-driven floor estimates exceed the declared guard (declared margin exhausted; consider raising r₀).
avci_gamma_alpha_factor 1.0 Multiplier c on the listener floor (√c on the plancher); 1.0 assumes σ²_lisSys ≤ σ²_int. Bypassed when the bootstrap plancher is available (the normal path).
avci_proxy "ebernstein" Variance proxy: running empirical variance (ebernstein) or worst-case hoeffding.
avci_d_bounds (-4, 4) Range of the per-item paired difference (not the raw rating scale): ±(rating max − min), so the default spans a rating range of 4. The normal path bootstraps the SE from the data and ignores this (it feeds only the sparse-data fallback and a reported diagnostic); widen it for a larger rating range if you drive AnytimeCI directly on sparse data.
avci_t_min_ebernstein 10 Below this item count the ebernstein proxy falls back to hoeffding (fallback path only).
avci_ebernstein_floor_frac 0.01 Relative floor on the empirical-variance proxy, as a fraction of the hoeffding bound (fallback path only).
avci_bootstrap_B 200 Bootstrap resamples per decision call.
avci_bootstrap_seed 0 PRNG seed for the bivariate cluster bootstrap.
sigma2_ga_aggregator "median" Which bootstrap-CI point anchors the listener floor: median / mean / q95 (conservative) / None.
cold_start_ga_ratio 2.7 Conservative BTTF-anchored prior ratio (σ²_lisSys/σ²_int) used before the component is identifiable.
avci_wait_min_listeners_kge2 3 Structural WAIT gate: minimum listeners with ≥ 2 paired items.
avci_wait_max_ci_ratio 5.0 WAIT gate: max tolerated q95/median on the σ²_lisSys bootstrap CI before abstaining.

Legacy naming. Parameters and Decision fields carrying ga / gamma_alpha (e.g. cold_start_ga_ratio, sigma2_ga_*) name the listener×system variance the paper writes σ²_lisSys; the identifiers will be renamed to lissys in a future release (an API change, hence deferred).

Input data requirements

  • Long format, columns listener_id, item, system, rating (extra columns are ignored).
  • Ratings on any bounded scale. The controller works on per-item paired differences, whose extremes are ±(rating max − min); the default avci_d_bounds = (-4, 4) spans a rating range of 4. On the normal path the standard error is bootstrapped from the data, so the scale is handled automatically and this default is safe on any scale; avci_d_bounds only enters the sparse-data fallback (when the cluster bootstrap cannot run).
  • The design must let listeners rate multiple items per system (K ≥ 2 somewhere): a design where each listener rates a single item — or each item is rated by a single listener — cannot identify the listener×system component, and the controller will honestly report WAIT forever rather than emit verdicts blind to that term.

Repository layout

anytimemonitor/
├── __init__.py           ← public API: AnytimeMonitor, Decision,
│                           VarianceEstimate, estimate_components
├── controller.py         ← the AnytimeMonitor controller
├── avci.py               ← AnytimeCI — Howard 2021 sub-Gaussian
│                           normal-mixture confidence sequence
├── variance.py           ← MoM variance components + Half-Cauchy
│                           regulariser
├── demo.py               ← CLI example (batch-fed on a bundled sample)
├── ui/                   ← Streamlit demo (app.py; see ui/README.md + ui/Architecture.md)
├── sample_uploads/       ← synthetic example CSVs for the UI
├── Architecture.md       ← internal architecture (controller + statistics)
├── CHANGELOG.md          ← what changed between versions, and what it moves
├── LICENSE               ← MIT
└── requirements.txt      ← pinned development stack

The validation and simulation harnesses (synthetic-data generator, controller-vs-oracle simulator, false-positive-rate check) are not part of the installed package; they ship with the companion paper's reproduction code.

Status and caveats

  • Research prototype. The API may still change. A pytest smoke-test suite (anytimemonitor/tests/) checks the package runs end to end; the controller has additionally been validated by simulation (coverage under H₀, controller-vs-oracle gap decomposition) and exercised on Blizzard Challenge data; it has not yet been plugged into a live test platform.
  • Planning aid first. The anytime-valid CI is a valid inference under optional stopping, so it can be reported as such — but the two-lever predictions (+N items / +L listeners) are noisy projections from running variance estimates, not guarantees.
  • A full-scale validation campaign (multi-seed, multi-corpus false-positive and power sweeps) is planned alongside the companion paper; the numbers shipped here come from smaller smoke-scale runs.
  • Known bias in the per-system σ²_int estimate. variance.py's per-system path under-estimates σ²_int, by 25–100 % in Monte-Carlo checks: the column centring removes system i's own share of the item means (a 1−1/S shrinkage) and the subtracted σ̂²_eval/K̄ then over-corrects. A per-system EMS re-derivation is pending; the fix is v·S/(S−1) − σ̂²_eval/K̄. Where it bites: the estimate is exposed on the public API and feeds the demo presets. It does not reach the controller's stopping decisions at the default settings — those use the per-pair bootstrap, and the only in-controller consumer is the cold-start prior, unreachable at the default — nor any number in the companion paper. Treat estimate_components(..., per_system=True) output as indicative until the re-derivation lands.

License

MIT — see LICENSE.

Citation

A companion paper is under submission. Until it is out, please cite this repository directly.

Metadata

Release files for anytimemonitor 0.10.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for anytimemonitor 0.10.0
File Size Uploaded Provenance Actions
anytimemonitor-0.10.0.tar.gz 78.5 kB With provenance Details for anytimemonitor-0.10.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for anytimemonitor 0.10.0
File Interpreter ABI Platform Actions
anytimemonitor-0.10.0-py3-none-any.whl Python 3 none any Details for anytimemonitor-0.10.0-py3-none-any.whl

Total release size: 156.0 kB

Release files / anytimemonitor-0.10.0.tar.gz

Download URL anytimemonitor-0.10.0.tar.gz
Size 78.5 kB
Tags Source
SHA-256 checksum
How to use checksums
c05bdf3e0509a5562588eaf3de5eec3635b8ec66ef15671bdf510ec36d820ec9
BLAKE2b-256 checksum
How to use checksums
1e6558cd7e352d0424c415b70422d944702be6a6e275811ccf9ebdd9871eb55c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Aug 1, 2026.

Transparency log

Release files / anytimemonitor-0.10.0-py3-none-any.whl

Download URL anytimemonitor-0.10.0-py3-none-any.whl
Size 77.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
29e0bf5be475ced191840cdcbbb670c212b15943a33b25a5bb4e7e7b3af6f560
BLAKE2b-256 checksum
How to use checksums
459c14e773e96ff86ac90fb465ea2cc8ed5134436c85654278d142a7198a3033
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Aug 1, 2026.

Transparency log

Security for anytimemonitor 0.10.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for anytimemonitor-0.10.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Aug 1, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page