mini-omega-lock

Measure repeated judge variability and missing measurements before interpreting evaluation results. An A/B significance decision needs a separate comparison design.

pip install mini-omega-lock

v0.8.2 — Release changes

Adds mini_omega_lock.probes.compute_minimum_detectable_effect(): mean(abs(delta)) + z * population_sigma(abs(delta)) / sqrt(n) with default z=1.96. A singleton returns its absolute delta; empty/nonfinite samples and invalid z are rejected. This helper uses supplied paired observations and does not establish a sampling design, statistical power, or live-judge quality by itself.

Includes an MCP 1.x/2.x import-layout compatibility shim (FastMCP / MCPServer). Local verification used SDK 1.30.0; SDK 2.x runtime behavior is not verified, and the declared supported dependency remains mcp>=1.0.0,<2.0.0.

Start here · Standalone use · Integration/Docking

Omega Aile — Quiet precision. AI research guided by evidence.

Probe judge consistency, schema behavior, context margin and run projections before a calibration. This is a separately distributed measurement tool.

Requires Python 3.11+. Installation needs internet.

python -m pip install mini-omega-lock==0.8.2
preflight --help
python -c "from mini_omega_lock import compute_context_margin; print(compute_context_margin(system_prompt_chars=0, rubric_chars=0, longest_input_chars=380, longest_reference_chars=0, longest_response_chars=0, context_window_tokens=1000))"

The offline API example prints 0.9: a character-based context projection, not an exact tokenizer measurement or model-quality result. Full preflight CLI probes require the selected provider infrastructure. Unmeasured warnings give exit 2; threshold breaches give 3; usage/runtime errors give 1. Exit 0 means requested measurements ran, not that their values are acceptable.

omegaprompt>=1.1.0 is required and installed automatically; no calibration pipeline is needed to use the probes. This release was tested offline with omegaprompt 2.2.3. Pack judge_quality, endpoint, performance and warnings into PreflightReport, then derive_adaptation_plan. The package is not the omega-lock search engine.

Docking contracts and runnable data handoff · Full guide.

MCP: install the distribution with [mcp] and use its existing server executable. FastMCP support is bounded to MCP SDK >=1.0.0,<2.0.0; the tool names and schemas are unchanged.

What is the "noise floor"?

An LLM judge does not give the same response the same score every time. Ask it to grade one fixed (response, rubric) pair five times and you'll often get five slightly different scores. That spread is the judge's noise floor.

It matters because of one rule:

A small delta relative to observed judge variability needs more evidence before acceptance.

Relative repeated-score variability is not comparable directly to an absolute A/B delta. Use a declared comparison design with paired samples and uncertainty before accepting a candidate improvement.

# One number, no Python, CI-friendly exit codes:
preflight --provider anthropic --rubric rubric.json \
          --probe-item item.json --probe-response "4" --summary
# -> {"judge_noise_floor": 0.07, "schema_reliability": 0.0, ...}

It also measures three more pre-flight surfaces in the same pass: endpoint schema reliability, context-budget margin, and a wall-time projection for the full run.

Quick start (Python)

from omegaprompt import make_provider
from omegaprompt.domain.dataset import DatasetItem
from omegaprompt.domain.judge import Dimension, JudgeRubric
from omegaprompt.judges.llm_judge import LLMJudge
from mini_omega_lock import empirical_preflight, judge_noise_floor

judge  = LLMJudge(provider=make_provider("anthropic"))
rubric = JudgeRubric(dimensions=[Dimension(name="accuracy", description="is it correct", weight=1.0)])
probe  = DatasetItem(id="probe", input="2+2", reference="4")

judge_quality, endpoint, performance, warnings = empirical_preflight(
    judge=judge, rubric=rubric, probe_item=probe,
    probe_response="4", consistency_repeats=5,
)

print("judge noise floor:", judge_noise_floor(judge_quality))
print("measurement_status:", warnings.measurement_status)
for w in warnings:                 # retain human-readable diagnostics
    print("[mini-omega-lock]", w)

judge_noise_floor is clipped relative CV (1 - consistency). It describes repeated scores of one fixed case; it does not define the absolute A/B delta needed for significance. The default judge/gate batch uses 5 actual observations; additional configured probes add calls. Cost depends on provider pricing and usage.

Works with omegaprompt — and standalone

• Standalone: the noise-floor probe is useful on its own. Run preflight against any LLM-judge / prompt-calibration setup, get the floor + schema-reliability numbers, gate your CI on them. No omegaprompt pipeline required to read the value.
• In the ecosystem: empirical_preflight emits omegaprompt.preflight.PreflightReport records (JudgeQualityMeasurement / EndpointMeasurement / PerformanceMeasurement). Assemble them into a PreflightReport and call derive_adaptation_plan. The plan distinguishes recommendations from runtime consumption; some policy fields, including schema fallback, remain advisory. Read the artifact consumption record before claiming a policy was applied. mini-omega-lock is the empirical probe layer; omegaprompt is the engine it feeds.

It depends on omegaprompt (>=1.1.0) to build those records, so pip install mini-omega-lock pulls omegaprompt in.

vs. "just trust the eval delta"

What it measures

One empirical_preflight() call returns (judge_quality, endpoint, performance, warnings). Fail-closed numeric defaults remain compatible. Use warnings.measurement_status as the primary measurement-state contract and persist probe_accounting too. Human warnings remain diagnostics; substring inference is only a legacy fallback when structured states are absent, and does not verify historical measurement coverage.

CLI: machine summary, scorecard, threshold gates

# Flat, CI-consumable JSON (headline number + schema_version, byte-stable):
preflight ... --summary

# Single-file scorecard (stdlib only) for a PR artifact:
preflight ... --scorecard html --scorecard-out preflight.html

# Fail the build when the judge is too noisy or the endpoint too unreliable:
preflight ... --fail-over-noise-floor 0.10 --fail-under-schema-reliability 0.90

Exit codes: 0 all measured & in-bound · 2 a field fell back to a fail-closed default (unmeasured) · 3 a measured value breached a --fail-* threshold (takes precedence over 2) · 1 usage/runtime error. A measured-but-bad value alone (noisy judge, gate flip) is still 0 — it was measured.

Read more

Sibling projects: omegaprompt (calibration engine) · omega-lock (broader audit framework) · mini-antemortem-cli (analytical, no-API preflight) · antemortem-cli (pre-implementation recon).

What's new in 0.7.0

• Judge noise-floor metrics, front and centre. New judge_noise_floor() helper + a build_summary() that produces a flat, schema_version-tagged, byte-stable CI dict, and a stdlib-only render_scorecard() (Markdown / self-contained HTML).
• CLI --summary (machine summary), --scorecard md|html (+ --scorecard-out), and --fail-over-noise-floor / --fail-under-schema-reliability / --fail-under-context-margin threshold gates (new exit code 3).
• Version-agnostic publish workflow + dynamic PyPI shields that track releases automatically.
• Frozen surface unchanged: empirical_preflight, the three contract records, console scripts, and the omegaprompt>=1.1.0 pin are all the same — additive only.

See CHANGELOG.md for the full history.

Trust loop (no network)

These run entirely offline (no API keys) and are exactly what scripts/release_audit.py enforces, so local CI and the release gate stay in lockstep:

python -m pip install -e ".[dev,mcp]"
python -m pytest -q
python scripts/generate_readme_claims.py --check
python scripts/check_repo_consistency.py
python examples/demo_replay.py
python scripts/run_golden_cases.py --check
python scripts/verify_fixture_integrity.py
python scripts/release_audit.py --no-network

python examples/demo_replay.py replays empirical_preflight against a scripted fake judge; its output is byte-for-byte equal to examples/_demo_output.txt (verified by tests/test_demo_replay.py) — the "did I break the warning surface?" smoke test.

MCP server

This package also exposes ten agent-callable MCP tools (empirical_preflight, measure_judge_consistency, measure_gate_flip_rate, measure_scale_monotonicity, probe_strict_schema, compute_context_margin, compute_context_margin_from_texts, noise_floor_estimate, project_performance, derive_adaptation_plan) — regenerated list in docs/generated/claims.md.

pip install "mini-omega-lock[mcp]"
python -m mini_omega_lock.mcp           # stdio (Claude Code default)
python -m mini_omega_lock.mcp --http    # streamable-http

Want the analytical (no-API, deterministic) preflight instead? See sibling tool mini-antemortem-cli — same plugin interface, a deterministic rule-based classifier instead of LLM probes.

What this does not prove

Not a benchmark of model quality, judge quality, or provider reliability under load. Not a production-readiness proof. It measures a narrow pre-flight surface (judge noise / endpoint / context / latency) without claiming A/B significance from a repeated-item variability statistic. See docs/trust_model.md and docs/claim_ledger.md for the per-claim boundary.

License

Apache 2.0. See LICENSE.

License history. PyPI distributions of 0.1.0 shipped with an MIT LICENSE. The repository was relicensed to Apache 2.0 on 2026-04-22 (commit ff489a9); 0.2.0 and all later versions ship under Apache 2.0. Anyone who installed 0.1.0 holds an MIT license to that copy — license changes do not apply retroactively.

0.8.0: 실제 관측 공유와 측정 상태

• 같은 (rubric, item, response)의 consistency와 gate flip은 실제 judge 관측 배치를 공유합니다. 기본 3+5회가 5회로 줄며 각 지표는 요청한 3개·5개 prefix만 사용합니다. 독립 표본 수를 합산하지 않습니다. 호출 순서가 바뀌므로 stateful judge의 수치가 달라질 수 있습니다.
• 기존 4-tuple과 warnings list 동작을 유지합니다. 반환 warnings의 measurement_status와 probe_accounting을 함께 저장하세요. CLI/MCP JSON도 두 필드를 제공합니다. 요약은 상태를 우선 읽고 구형 일반 list에는 경고 문자열 호환 읽기를 사용합니다.
• 상태는 measured, unmeasured, insufficient_samples, failed, estimated, not_applicable입니다. 1회 채점은 변동 측정에 부족합니다. 실패한 호출 시간도 accounting에 포함하고, 성공 호출 latency projection과 구분합니다. provider 내부 재시도·실패 토큰 비용은 관측할 수 없으면 null/unknown으로 남깁니다.
• judge_noise_floor는 호환 이름이며 값은 clipped CV입니다. 새 별칭 judge_relative_variability도 같은 값입니다. fitness_noise_floor는 aggregate fitness의 표준편차입니다. 둘 다 임의 A/B 개선의 유의성을 판정하지 않습니다.
• CLI: 정상 실행 0은 지정된 측정 완료를 뜻합니다. 미측정·실패·표본 부족·추정은 2, 측정한 값의 사용자 threshold 위반은 3, 잘못된 입력은 1입니다. threshold 없는 성공을 judge 품질 통과로 해석하지 마세요.

전후 근거·호환성·재현 명령. 유료 provider 검증은 수행하지 않았습니다.