mini-omega-lock

English · 한국어 · English easy · 한국어 easy

Empirical preflight for repeated judge variability and missing measurements before interpreting evaluation results. It does not replace independent hold-out evaluation or an A/B significance test.

Repository: hibou04-ops/mini-omega-lock · PyPI: mini-omega-lock · import: mini_omega_lock · CLI: preflight · MCP: mini-omega-lock-mcp with mini-omega-lock[mcp]

0.8.3 — Standalone installation and statistical naming

The sole required dependency is pydantic>=2.6.0. Necessary contracts, provider boundaries and judge code are included under internal _vendor, allowing core imports, CLI and fake-provider demos without omegaprompt, omega-lock, NumPy or provider SDKs. SDKs load lazily when a live provider is selected. omegaprompt>=1.1.0 is confined to optional extra [bridge].

Requires Python 3.11+. This checkout has source version 0.8.3. The commands below install the checkout. Git tags and PyPI publication are separate; availability of the new source version on PyPI is not claimed.

python -m pip install .
preflight --help
python examples/demo_replay.py
python examples/cli_demo_replay.py

The fixture demos above run without API keys, live SDKs or network access. Installation may download dependencies. Live probes separately require the selected SDK, provider configuration and credentials.

compute_empirical_noise_ceiling

from mini_omega_lock import (
    compute_empirical_noise_ceiling,
    compute_minimum_detectable_effect,
)

assert compute_minimum_detectable_effect is compute_empirical_noise_ceiling
assert compute_empirical_noise_ceiling([-0.1, 0.1]) == 0.1

The canonical API is compute_empirical_noise_ceiling(paired_deltas, confidence_z=1.96), exposed from both the package root and mini_omega_lock.probes. The legacy compute_minimum_detectable_effect name is a backward-compatible alias to the same callable in both modules.

magnitudes = abs(paired_deltas)
ceiling = mean(magnitudes) + 1.96 * pstdev(magnitudes) / sqrt(n)

Default confidence_z is 1.96 and may be changed to a finite nonnegative value. pstdev is population standard deviation with denominator n. Empty samples, NaN/Inf observations and negative/nonfinite z are rejected; a singleton returns its absolute delta. This statistic is an empirical absolute-noise ceiling for the supplied repeated observations. It does not establish a sampling design, power, a distribution-free confidence interval or an A/B minimum detectable effect.

Measurement states and accounting

empirical_preflight() returns (judge_quality, endpoint, performance, warnings). Use local contracts from mini_omega_lock.contracts, providers from mini_omega_lock.providers and the judge from mini_omega_lock.judges.

Read warnings.measurement_status first and persist probe_accounting. States are measured, unmeasured, insufficient_samples, failed, estimated, not_applicable. The existing 4-tuple and warnings-list behavior remain. Human warnings are diagnostics; legacy substring fallback without structured states does not verify historical measurement coverage.

judge_noise_floor is clipped CV (1 - consistency); judge_relative_variability aliases the same value. fitness_noise_floor is the standard deviation of aggregate fitness. Relative CV cannot be compared directly with an absolute A/B score delta. Consistency and gate-flip share observations for the same (rubric,item,response), with each metric using its requested prefix. The default batch uses 5 actual observations; additional probes add calls.

Accounting includes failed-call time separately from successful-call latency projection. Unobservable provider-internal retries and failed-token costs remain unknown/null. Suggested schema fallback is advisory and does not prove runtime consumption.

CLI and optional integration

CLI exit codes: 0 means requested measurements completed; 2 indicates unmeasured, failed, insufficient or estimated states; 3 indicates a user threshold breach on a measured value; 1 indicates input/runtime errors. Exit 0 without thresholds is not judge-quality acceptance or deployment authorization.

# Live example: requires the selected provider SDK and credentials.
preflight --provider anthropic --rubric rubric.json \
  --probe-item item.json --probe-response "4" --summary

python -m pip install ".[mcp]"
mini-omega-lock-mcp

Supported MCP SDK range is >=1.0.0,<2.0.0; 2.x runtime behavior is unverified. Install [bridge] for parent integration. Local PreflightReport and adaptation work without the parent. Convert reports with mini_omega_lock.bridge.to_omegaprompt_report(report); wrap an existing parent provider with adapt_omegaprompt_provider(provider) before passing it to the local LLMJudge. Model classes are not assumed to share identity.

Local verification and limits

python -m pytest
python scripts/generate_readme_claims.py --check
python scripts/check_repo_consistency.py

Local contracts and fake-provider reproducibility do not establish external OOS utility, live-judge quality, load stability or production readiness. Paid-provider validation is not claimed. An independent OOS comparison must separately declare sampling units, splits, repeated-measurement correlation, uncertainty and decision rules.

Evidence and documentation

• Generated claims: English / 한국어
• Trust model · Claim ledger
• Vendored source and maintenance · Changelog
• 0.8.0 measurement-contract migration

README.ko.md is the canonical Korean document; README_KR.md is an identical compatibility copy.

Apache 2.0. LICENSE · NOTICE.