omega-lock

English · 한국어 · 한국어 별칭

Check whether a tuned candidate survives your declared audit gates. omega-lock is the gate that runs after your tuner, takes its "winning" candidate, and checks the declared constraints and train/holdout score correlation before it ships.

Source version 0.3.11. This update publishes Git source/tag identity only; it does not claim a GitHub Release object or a PyPI upload. The pinned install below uses the source tag.

python -m pip install "omega-lock @ git+https://github.com/hibou04-ops/omega-lock.git@v0.3.11"
omega-lock demo   # 60s, offline: watch a "winning" score collapse -74% on held-out data

Keywords: hyperparameter overfitting · eval / prompt regression testing · walk-forward validation · validate an Optuna study · holdout transfer check in CI.

Reproduce deployment traps in 30 seconds

From this checkout, install the optional Optuna bridge and run the keyless, offline assertions:

python -m pip install -e ".[p2]"
python examples/reproduce_optuna_traps.py

Actual stdout from the command above (local offline execution, 2026-11-02):

[
  {
    "case": "no_holdout",
    "passed": true,
    "status": "SKIP",
    "holdout_evaluated": false,
    "deployment_authorized": false,
    "pearson": null,
    "degradation_gap": null
  },
  {
    "case": "location_shift",
    "passed": false,
    "status": "FAIL",
    "holdout_evaluated": true,
    "deployment_authorized": false,
    "pearson": 1.0,
    "degradation_gap": 101.0
  },
  {
    "case": "normal",
    "passed": true,
    "status": "PASS",
    "holdout_evaluated": true,
    "deployment_authorized": true,
    "pearson": 1.0,
    "degradation_gap": 0.0
  }
]
All three deployment-gate cases reproduced without API keys or network calls.

The script checks three cases: no holdout keeps legacy passed=True / status="SKIP" while holdout_evaluated=False and deployment_authorized=False; [1,2,3] versus [-100,-99,-98] has Pearson 1 but degradation gap 101 and is blocked; unchanged held-out scores authorize under the declared gates. No API keys, paid calls or external service configuration are used.

CI consumers must read deployment_authorized, not audit-completion passed alone. gate_scores() and audit_optuna_study() now combine correlation with max_degradation_gap=1.0 by default. The gap is mean(train)-mean(holdout) in maximize orientation (sign-reversed for minimize): holdout mean must be at least train mean minus the configured limit. The Optuna bridge additionally checks the selected candidate itself, including a feasible candidate outside the top-N. The finite nonnegative score-unit limit cannot be disabled with None; predeclare a domain-appropriate value. CLI uses --max-degradation-gap and --direction maximize|minimize.

The decision compares exact finite-input arithmetic against the bound; displayed gaps are floats. Nonfinite, degenerate, overflow or underflow evidence cannot authorize deployment. These local policy checks do not establish data independence, statistical significance or external generalization. The lower-level check_kc4() and existing P1 orchestration retain their separate contracts.

The Optuna report exposes both degradation_gap (top-N mean) and selected_degradation_gap (the selected candidate). A favorable mean cannot waive the selected candidate check.

v0.3.11 — Release changes

Separates audit completion from deployment authorization and adds declared mean/selected-candidate degradation bounds. The pure helpers below are retained features.

Retains the pure omega_lock.holdout_defense helpers: compute_deflated_threshold() applies base + MDE * sqrt(1 + ln(max(1, trials))); classify_calibration_verdict() returns M0_HOLD for a failed holdout, ESTABLISHED only for a passed holdout with delta > MDE, and NOT_ESTABLISHED otherwise. Existing orchestration APIs are unchanged. These input-based rules do not measure holdout independence, statistical power, or generalization.

Start here · Standalone use · Integration/Docking

Omega Aile — Quiet precision. AI research guided by evidence.

Audit numeric candidate scores after tuning. The CLI correlation gate checks index-aligned train/holdout arrays; the library also provides search, constraints and audit trails.

Requires Python 3.11+. Installation needs internet.

python -m pip install "omega-lock @ git+https://github.com/hibou04-ops/omega-lock.git@v0.3.11"
python -c "import json; from pathlib import Path; Path('train.json').write_text(json.dumps([1,2,3,4,5])); Path('holdout.json').write_text(json.dumps([1,2,3,4,5]))"
omega-lock gate --train train.json --holdout holdout.json --report gate.html

Synthetic input: PASS and exit 0, Pearson 1.000; gate.html is the scorecard. Reverse the holdout array to exercise FAIL/exit 1. Invalid input exits 2. This checks wiring, not real-world generalization.

No sibling package is needed. omegaprompt requires omega-lock>=0.3.0,<0.4.0 at runtime. antemortem only cites a pinned source revision in CI. Optuna is a native optional bridge: install omega-lock[p2] (Optuna >=3,<5); other optimizers can supply score arrays through an explicit export.

Docking contracts and runnable data handoff · Full guide.

0.3.9 adds stable Pearson computation at extreme scales, explicit NaN/Inf rejection, a 64-Future window for optional parallel evaluation, and optional candidate-ID validation in gate_scores. Pearson alone is score linear association, not rank correlation or absolute quality. The old correlation-only path could pass holdout = train - 100; current gate_scores/Optuna deployment checks additionally enforce the mean degradation bound. See measured changes and limits.

The 30-second version

You ran a hyperparameter sweep, a prompt search, or a threshold tuner. It came back proud and pointed at the winner — the highest score on the data you tuned against.

That is exactly the number you can't trust. When you try hundreds of candidates and keep only the single best one, you don't just keep the most skillful one — you keep the luckiest one. And luck doesn't repeat. The moment you test that winner on data it has never seen, the lucky streak is gone:

on the data it was picked from   →   5.967   (real skill  +  a lucky streak)
on brand-new, held-out data      →   1.527   (only the real skill that was left)   ▼ -74.4%

This illustrates overfitting from selection. Optimizers and evaluation frameworks can include their own validation; check the actual configuration rather than assuming they lack it. omega-lock is your second opinion. It re-tests the winner on a slice the search never touched and returns a flat verdict: PASS (declared gates passed) or FAIL (a gate failed).

See it fail a lucky winner — 60 seconds, nothing to set up

omega-lock demo

A fully offline case study: a search picks a candidate that looks brilliant in training, then omega-lock re-scores it on a held-out slice.

candidate: best-by-score (selected from 125 trials)
  train score    5.967
  holdout score  1.527     ▼ -74.4%
  walk-forward transfer gate ............ FAIL   (train↔holdout correlation 0.179 < 0.3)
  hard-constraint feasibility ........... FAIL   (best_feasible ≠ best_any)

VERDICT: BLOCK — the winning score did not transfer. Selection concentrated luck.

The optimizer was thrilled with 5.967. The reality was 1.527. omega-lock stamps FAIL and your pipeline stops the deploy. That collapse is the whole product in one screen.

Drop it into CI

Point omega-lock at two score files — the scores your optimizer reported on the data it tuned against, and the scores of the same candidates re-evaluated on a held-out slice. It exits 0 (ship) or 1 (block):

omega-lock gate --train train_scores.json --holdout holdout_scores.json

# .github/workflows/overfit-gate.yml
name: overfit-gate
on: [pull_request]

jobs:
  guard:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.11" }
      - run: python -m pip install "omega-lock @ git+https://github.com/hibou04-ops/omega-lock.git@v0.3.11"

      # your tuner runs here and writes train_scores.json + holdout_scores.json
      - run: python tune.py

      # the gate: a non-zero exit fails the check and blocks the merge
      - run: omega-lock gate --train train_scores.json --holdout holdout_scores.json

When score correlation is below threshold or uncomputable, the step fails. Blocking a PR merge depends on the repository's required-check settings. The CLI prints the verdict and can write an HTML scorecard. The library audit API separately supports append-only trails; the score-array CLI does not create that trail.

Prefer Python? The same decision is one call:

from omega_lock.simple import gate_scores

result = gate_scores([1, 2, 3, 4, 5], [1, 2, 3, 4, 5])
assert result.passed, result.reasons   # fail your test suite on a bad candidate

Already have an Optuna study? Use the optional native bridge

# First install: python -m pip install "omega-lock[p2]"
import optuna
from omega_lock import audit_optuna_study

study  = optuna.load_study(study_name="my-sweep", storage="sqlite:///sweep.db")
report = audit_optuna_study(study, holdout_evaluate=score_on_holdout)  # walk-forward + feasibility on study.best_trial
print(report.deployment_authorized, report.gated_best)   # check explicit deployment authority

No new study, no rewrite of your objective, no config DSL. It also works on bare lists (Ax, Ray Tune, Hyperopt, GridSearchCV, or a hand-rolled sweep) — any leaderboard is enough.

What the gate actually checks

Three independent checks on the candidate your search already chose. Any one can block it.

Core insight: the highest score is the most suspicious number you own. A real edge survives a slice it was never shown. Luck does not.

CLI gate and library search are different surfaces

The score-array CLI audits existing scores. The library also includes GridSearch, ZoomingGridSearch and run_p1; these search paths feed the audit gates. It is the gate you bolt onto the search you already have — keep Optuna, keep your sweep, keep your eval loop, and let omega-lock judge the output.

Where it sits next to the tools you know

The empty seat omega-lock fills: an output-side overfit gate. The rule of thumb — if a number was chosen by trying many options and keeping the best, it belongs behind this gate. For where omega-lock does and does not fit the wider toolbox, see docs/TOOLKIT_POSITIONING.md.

Install

python -m pip install "omega-lock @ git+https://github.com/hibou04-ops/omega-lock.git@v0.3.11"

omega-lock demo                 # 60s offline walkthrough — watch a lucky winner collapse
omega-lock gate --help          # the CI gate (exit 0 = ship, 1 = block)

Generate a shareable dark-themed scorecard from any gate run with render_html — attach it to a PR or archive it.

READMEs: Easy / plain-English README · 한국어 README · 쉬운 한국어 README — Docs: How the transfer gate works · Power API for integrators · Trust & audit model · Toolkit positioning · CHANGELOG

Same idea, other surfaces (the Omega Aile toolkit — a held-out gate where AI workflows skip one): omegaprompt gates an overfit prompt · antemortem gates an AI agent's code plan with verified file:line citations.

_{Badge and download analytics boundaries. The badges above are static or registry-served links; they do not prove release readiness, correctness, trustworthiness, adoption, or package quality. Downloads or stars may indicate visibility, not skill — stars/downloads must not be used as audit evidence or release approval. No PyPI or GitHub download analytics are asserted here. Only the gate's PASS/FAIL on held-out data is evidence.}

_{Note on terms. This page uses plain language; the public Python API keeps its established symbols for backward compatibility (other repos import them). In code you may see: run_p1 / P1Config (run the gate + its config), check_kc4 / KCThresholds (the walk-forward transfer check + its pass thresholds, e.g. minimum transfer correlation), measure_stress (rank parameters by perturbation sensitivity), ParamSpec (a tunable parameter's range), EvalResult (one scored candidate). You never need these to use omega-lock demo, omega-lock gate, or omega_lock.simple.gate_scores(). Full reference in docs/API.md.}

The library provides an append-only audit trail (Append-only audit trail) of evaluated candidates. An optional SHA-256 hash chain makes saved audit JSON tamper-evident on verification. The score-array CLI only prints its verdict and optionally writes HTML.