AnytimeMonitor — adaptive controller for TTS-MOS listening tests
AnytimeMonitor watches a MOS listening test while it runs and tells the test admin, for every pair of systems, whether the data already supports a verdict — and if not, which lever (more items, or more listeners) closes the pair fastest.
The statistical core is an anytime-valid confidence sequence (Howard, Ramdas, McAuliffe & Sekhon, 2021) on per-item paired score differences. Anytime validity means the coverage guarantee holds at any data-dependent stopping time: the admin can look at the verdicts as often as they like, stop whenever a pair resolves, and the reported confidence interval is still valid. No peeking penalty bookkeeping, no pre-registered sample size required.
This is a research prototype (v0.9). It accompanies a position paper on budget allocation in TTS evaluation (under submission, Computer Speech & Language Special Issue on the evaluation of speech and speech synthesis).
What the admin sees
For each pair of systems (A, B), one of six verdicts:
| Verdict | Meaning |
|---|---|
STOP |
The anytime-valid CI excludes 0 (0 ∉ CI): the pair is separated. |
STOP_EQUIV |
Strict TOST equivalence: CI ⊂ ±δ_equiv and the half-width Ψ_eff ≤ δ_equiv/2. The pair is statistically equivalent within tolerance. |
CONTINUE +N |
Neither verdict yet. The tool predicts how many more items (+N) or listeners (+L) would close the pair, and which terminal (STOP or STOP_EQUIV) it is heading for. The headline is the cheaper option. |
WAIT |
The listener×system variance component cannot be estimated yet (fewer than 3 listeners with ≥ 2 paired items). Structural: keep collecting before trusting any verdict. |
CORPUS_LIMIT |
The item budget is exhausted and the listener lever can no longer close the pair. Terminal. |
LISTENER_LIMIT |
The listener budget is exhausted and the item lever can no longer close the pair. Terminal. |
Verdict naming.
STOPis the separation verdict the companion paper callsSTOP_SEP.STOPis the current public label and will be renamed toSTOP_SEPin a future release (to pair symmetrically withSTOP_EQUIV); code matching on"STOP"should expect that change.
Verdicts are recomputed from scratch at every call — the controller is
queried on demand (a handful of times over a test lifecycle), not in a
real-time loop. A STOP is not sticky: if you keep collecting
past it, a borderline pair can revert to CONTINUE. This is honest
anytime-CI behaviour (the guarantee is on coverage, not on the
monotonicity of the observed difference); in practice the admin acts
on the first STOP.
Install
pip install -e . # from the repository root (pyproject.toml)
pip install -e ".[ui]" # + Streamlit demo UI
pip install -e ".[test]" # + pytest; then run `pytest`
Requires Python ≥ 3.10. Pinned versions used for development are in
requirements.txt.
Quick start
Programmatic
import pandas as pd
from anytimemonitor import AnytimeMonitor
ctrl = AnytimeMonitor(mde_threshold=0.30) # δ_sep, in MOS units
# Feed ratings as they arrive — long format, one row per judgment
ctrl.add_ratings(pd.DataFrame({
"listener_id": ..., # who rated
"item": ..., # which sentence / stimulus
"system": ..., # which TTS system
"rating": ..., # the MOS score
}))
print(ctrl.report()) # all pairs
dec = ctrl.decision("sysA", "sysB") # one pair, structured
print(dec.verdict, dec.ci_lo, dec.ci_hi, dec.n_more_items)
CLI demo
Feed a ratings CSV to the controller batch by batch and watch the per-pair verdicts tighten. With no arguments it runs on a bundled synthetic sample:
python -m anytimemonitor.demo # bundled sample
python -m anytimemonitor.demo --input my_ratings.csv # your own long-format CSV
Streamlit demo UI
streamlit run anytimemonitor/ui/app.py
Upload a ratings CSV (or pick a bundled synthetic example from
sample_uploads/), watch the controller replay the stream
batch-by-batch, and drag the time-machine slider to see the verdict
and two-lever advice at any past timestep. See ui/README.md.
How it works
For a pair (A, B), the controller builds per-item paired differences d_j = mean rating of item j under A minus under B, and maintains a sub-Gaussian normal-mixture confidence sequence on their mean (Howard et al. 2021). Two ingredients adapt this to listening-test data, where ratings are clustered by listener as well as by item:
- Bivariate cluster bootstrap SE. Resampling items gives the i.i.d.-part SE that the anytime boundary multiplies; resampling listeners gives the listener-pool floor ("plancher") — the precision ceiling that no amount of extra items can break, added in quadrature. When the data is too sparse to bootstrap, the controller falls back to the confidence sequence's internal running variance proxy.
- Variance decomposition layer. Method-of-moments estimates of the system×item and residual components (per system, with a Half-Cauchy Bayesian regulariser that prevents the truncation-at-zero bias of raw MoM), plus a per-pair estimate of the listener×system component. These feed the two-lever prediction: +N items shrinks the interaction term, +L listeners lowers the floor.
The price of anytime validity is a modestly wider interval than a
fixed-n test (a √log log factor), and deliberate conservatism early
on: below avci_t_min_ebernstein items the variance proxy falls back
to the worst-case Hoeffding bound, so the first batches produce wide
intervals. The controller is therefore most useful from a few dozen
rated items per system onward.
The internal architecture is documented in
Architecture.md (and
ui/Architecture.md for the demo UI); the deeper design
notes for the confidence sequence are in AVCI_PLAN.md.
Key parameters
All knobs are constructor arguments of AnytimeMonitor; the
defaults are deliberately conservative.
| Parameter | Default | Role |
|---|---|---|
mde_threshold (δ_sep) |
(required) | The smallest effect worth resolving (MOS): seeds delta_equiv and the reported MDE. It does not gate STOP — separation fires on 0 ∉ CI regardless. |
delta_equiv |
None (= mde_threshold) |
δ_equiv — equivalence margin for STOP_EQUIV. Set it below mde_threshold to create an indifference zone. |
alpha |
0.05 | Nominal level of the confidence sequence. |
power |
0.80 | Only sets the reported-MDE scale factor (z_{α/2} + z_power); does not affect stopping. |
n_max_corpus, l_max_listeners |
None |
Item / listener budgets; enable the *_LIMIT terminal verdicts. |
avci_n_planned |
240 (§7 census median budget) |
Planned item budget; the controller derives η = 1/√(N_planned) (Howard's a-priori rule). |
avci_eta |
None (→ 1/√avci_n_planned) |
Set the Howard 2021 mixing parameter η by hand instead of via avci_n_planned — η is fixed a priori, never tuned to data. Pass one or the other, not both (raises otherwise). |
avci_lissys_floor_r0 |
0.36 (measured r_lisSys band frontier) |
A-priori guard on the listener floor: bounds it below by √(2·σ²₀/L) with σ²₀ = 1.5·r₀·σ̂²_eval (the ×1.5 is an MC-calibrated validity margin). Protects coverage when a small pool under-estimates σ²_lisSys. 0 disables. |
avci_lissys_floor_sigma2 |
None |
Absolute σ²₀ override for the guard (expert path, no extra margin). Exclusive with avci_lissys_floor_r0 (raises otherwise). The per-pair Decision.lissys_prior_exceeded flags when data-driven floor estimates exceed the declared guard (declared margin exhausted; consider raising r₀). |
avci_gamma_alpha_factor |
1.0 | Multiplier c on the listener floor (√c on the plancher); 1.0 assumes σ²_lisSys ≤ σ²_int. Bypassed when the bootstrap plancher is available (the normal path). |
avci_proxy |
"ebernstein" |
Variance proxy: running empirical variance (ebernstein) or worst-case hoeffding. |
avci_d_bounds |
(-4, 4) |
Range of the per-item paired difference (not the raw rating scale): ±(rating max − min), so the default spans a rating range of 4. The normal path bootstraps the SE from the data and ignores this (it feeds only the sparse-data fallback and a reported diagnostic); widen it for a larger rating range if you drive AnytimeCI directly on sparse data. |
avci_t_min_ebernstein |
10 | Below this item count the ebernstein proxy falls back to hoeffding (fallback path only). |
avci_ebernstein_floor_frac |
0.01 | Relative floor on the empirical-variance proxy, as a fraction of the hoeffding bound (fallback path only). |
avci_bootstrap_B |
200 | Bootstrap resamples per decision call. |
avci_bootstrap_seed |
0 | PRNG seed for the bivariate cluster bootstrap. |
sigma2_ga_aggregator |
"median" |
Which bootstrap-CI point anchors the listener floor: median / mean / q95 (conservative) / None. |
cold_start_ga_ratio |
2.7 | Conservative BTTF-anchored prior ratio (σ²_lisSys/σ²_int) used before the component is identifiable. |
avci_wait_min_listeners_kge2 |
3 | Structural WAIT gate: minimum listeners with ≥ 2 paired items. |
avci_wait_max_ci_ratio |
5.0 | WAIT gate: max tolerated q95/median on the σ²_lisSys bootstrap CI before abstaining. |
Legacy naming. Parameters and
Decisionfields carryingga/gamma_alpha(e.g.cold_start_ga_ratio,sigma2_ga_*) name the listener×system variance the paper writesσ²_lisSys; the identifiers will be renamed tolissysin a future release (an API change, hence deferred).
Input data requirements
- Long format, columns
listener_id,item,system,rating(extra columns are ignored). - Ratings on any bounded scale. The controller works on per-item paired
differences, whose extremes are ±(rating max − min); the default
avci_d_bounds = (-4, 4)spans a rating range of 4. On the normal path the standard error is bootstrapped from the data, so the scale is handled automatically and this default is safe on any scale;avci_d_boundsonly enters the sparse-data fallback (when the cluster bootstrap cannot run). - The design must let listeners rate multiple items per system
(K ≥ 2 somewhere): a design where each listener rates a single item
— or each item is rated by a single listener — cannot identify the
listener×system component, and the controller will honestly report
WAITforever rather than emit verdicts blind to that term.
Repository layout
anytimemonitor/
├── __init__.py ← public API: AnytimeMonitor, Decision,
│ VarianceEstimate, estimate_components
├── controller.py ← the AnytimeMonitor controller
├── avci.py ← AnytimeCI — Howard 2021 sub-Gaussian
│ normal-mixture confidence sequence
├── variance.py ← MoM variance components + Half-Cauchy
│ regulariser
├── demo.py ← CLI example (batch-fed on a bundled sample)
├── ui/ ← Streamlit demo (app.py; see ui/README.md + ui/Architecture.md)
├── sample_uploads/ ← synthetic example CSVs for the UI
├── Architecture.md ← internal architecture (controller + statistics)
├── AVCI_PLAN.md ← deeper design notes for the confidence sequence
├── TODO.md ← development roadmap
├── archive/ ← retired code, notes and frozen runs
├── LICENSE ← MIT
└── requirements.txt ← pinned development stack
The validation and simulation harnesses (synthetic-data generator, controller-vs-oracle simulator, false-positive-rate check) are not part of the installed package; they ship with the companion paper's reproduction code.
Status and caveats
- Research prototype. The API may still change. A
pytestsmoke-test suite (anytimemonitor/tests/) checks the package runs end to end; the controller has additionally been validated by simulation (coverage under H₀, controller-vs-oracle gap decomposition) and exercised on Blizzard Challenge data; it has not yet been plugged into a live test platform. - Planning aid first. The anytime-valid CI is a valid inference under optional stopping, so it can be reported as such — but the two-lever predictions (+N items / +L listeners) are noisy projections from running variance estimates, not guarantees.
- A full-scale validation campaign (multi-seed, multi-corpus false-positive and power sweeps) is planned alongside the companion paper; the numbers shipped here come from smaller smoke-scale runs.
License
MIT — see LICENSE.
Citation
A companion paper is under submission. Until it is out, please cite this repository directly.
Metadata
Release files for anytimemonitor 0.9.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| anytimemonitor-0.9.1.tar.gz | 75.7 kB | With provenance | Details for anytimemonitor-0.9.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| anytimemonitor-0.9.1-py3-none-any.whl | Python 3 | none | any | Details for anytimemonitor-0.9.1-py3-none-any.whl |
Total release size: 150.7 kB
Release files / anytimemonitor-0.9.1.tar.gz
| Download URL | anytimemonitor-0.9.1.tar.gz |
|---|---|
| Size | 75.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c81c609179bb162f69baca79fb583a2dd6530773d7b3ccc85b9e36190735d2c1
|
|
BLAKE2b-256 checksum How to use checksums |
4495793386a332e3ec7f9d0054b786abb943176a3edf4819db47ad9eacebfffa
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 28, 2026.
Transparency logRelease files / anytimemonitor-0.9.1-py3-none-any.whl
| Download URL | anytimemonitor-0.9.1-py3-none-any.whl |
|---|---|
| Size | 75.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
c78063bd3105cb8d04207af7a62461c7a96f91565c9aada325d756c3332f5f4f
|
|
BLAKE2b-256 checksum How to use checksums |
1424a71fda48dbf8cb9a2879ac485a6b01139372c625c1b03ee9c55af6bb350a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 28, 2026.
Transparency logSecurity for anytimemonitor 0.9.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for anytimemonitor-0.9.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 28, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.