cozy-eval

Quality benchmarking for generative image and video models, with the validity rules enforced by the API.

Two questions, one library, one set of validity rules over both:

• How far did the pixels move, and is that damage or just a different take? Lanes, protocol stamping, null controls, population gates.
• Did the model make what was asked for? Checklist adherence, preference, standalone quality — MIT reimplementations of methods the ecosystem otherwise only ships under non-commercial licences.

Everything is organized by what it measures, not by who wrote it. Every metric lives in cozy_eval.metrics, every metric is declared in one cozy_eval.registry, and the same protocol/null-control/population rules govern all of them.

There are good libraries for computing quality metrics. ffmpeg-quality-metrics and cvvdp do full-reference video properly; torchmetrics owns FID/KID/IS and the permissively-packaged learned scorers. This library composes those and reimplements only what is otherwise locked behind a non-commercial licence.

What no library does is stop you from computing the wrong metric. That is what this one is for.

---

The problem

You quantized a video model and you want to know if it got worse. The obvious move — render the same prompt at the same seed on both arms and measure LPIPS or PSNR between them — is the move essentially every production quantization harness makes. torchao's Flux benchmark scores same-seed LPIPS against a high-precision baseline. DeepCompressor/SVDQuant's eval config lists ["psnr", "lpips", "ssim"] against the BF16 render directory, and Nunchaku's CI hard-gates merges on assert lpips < expected_lpips * 1.15. NVIDIA's TensorRT-ModelOpt diffusers examples ship no quality eval at all, and where they acknowledge divergence the advice is "we suggest to run a few more times and choose the best one."

That distance measures fidelity to the matched render; it does not by itself establish perceptual degradation. Evidence from banked LTX-2.3 renders at 1920×1088 shows why these questions need separate reports:

1. The no-op consumes the entire budget. A compile-only control — zero quantization change — scores LPIPS 0.196–0.249 against a fleet fp8 budget of 0.25.
2. It is divergence, not drift. Distance is already 0.29–0.41 at frame 0 and flat-to-falling across the clip. Numerical error accumulates; a different take starts far apart and stays there.
3. The ranking inverts. An fp8-storage-cast arm and an unscaled-w8a8 arm carry identical weight bytes. The cast arm computes its GEMMs in bf16 and is therefore strictly the more accurate path — and it scores worse (0.3875 vs 0.3052).

Closeness to the reference and perceptual quality can rank these arms differently. An explicit paired-fidelity comparison reports the distance; quality instruments measure prompt adherence, plausibility, aesthetics and corruption separately. Anything that perturbs the sampling trajectory — quantization, torch.compile, an attention-backend swap, a scheduler change, a LoRA attach — produces a different take of the same prompt, and distance to the reference render measures how different the take is, not how damaged it is.

The academic side has largely avoided the trap (Q-Diffusion, ViDiT-Q and friends report FID/FVD/VBench, i.e. population metrics) but states its reason as sample size rather than validity, so the production side never got the message.

What this library adds

Purpose and trajectory	classify_change() keeps post-latent and trajectory-changing declarations distinct. Quality comparisons retain their guard; explicit paired-fidelity measures matched output distance without relabeling quantization.
Protocol stamping	Every result carries resolution, frame count, steps, seeds, prompt set, execution lane, hardware and same-pod status. Cross-pod or mismatched-lane arms return INDETERMINATE, never PASS.
Population semantics	Trajectory-perturbing comparisons need n ≥ 8 paired prompts. At n = 1 a shipped-clean w8a8 recipe scores an imaging index of 0.934 on one prompt and 1.005 over eight. One prompt measures the take.
Three orthogonal benchmarks	Imaging, temporal, distributional — with a banked degradation each one catches that the other two miss.
Thresholds with provenance	Every budget names the known-good and known-bad populations that fixed it and its separation margin. Provisional ones say so.
Null-control arms	A budget is only a budget where a zero-change arm sits inside it. measure_null_control() measures that per family; budgets its control trips are disregarded, and a run with disregarded budgets cannot rise above INDETERMINATE. Measured: null controls trip population_frechet at 2.3× and 6.6× the budget with zero model change.
NO-SIGNAL, not a confident FAIL	An arm of constant frames returns DEGENERATE, not FAIL with an imaging index of 0.0. There is nothing to compare; ranking it would be a category error.
Statistical honesty	Paired t with Holm correction, effect sizes, and a practical-effect floor so significance without magnitude cannot fail an artifact. No package surveyed reports a confidence interval or effect size on a quality delta.

| One implementation per name | Every MetricSpec carries a per-metric impl version beside the registry-wide METRIC_SET_VERSION, every score row carries it, and comparing rows across differing impls is a typed refusal. clean-fid measured same-named FID disagreeing by whole points on resize filter alone; a name does not identify a number, (name, impl) does. | | A cost class per metric, as data | cheap-deterministic / small-model / model-backed, declared in the registry, because that is the fact a consumer routes on. A model-backed metric with no injection seam is a registration error. | | Functional AND accumulator | Every metric is callable one-shot, and MetricAccumulator streams update()/compute() over plain-array state for corpus-sized or per-frame scoring. Both paths call the same pooling function, so they cannot disagree. |

Composed where a maintained permissive implementation exists: ffmpeg-quality-metrics and cvvdp for full-reference video, lpips for LPIPS, torchmetrics for ARNIQA/CLIP-IQA. PSNR, SSIM and MS-SSIM are ours, in numpy, one implementation each — they used to be two or three each depending on what you had installed, which is the same defect the impl version exists to prevent. The base install is numpy + msgspec only; everything else is an extra, and a fence keeps it that way.

Install

pip install cozy-eval                      # numpy + msgspec
pip install "cozy-eval[reference]"         # VMAF / ColorVideoVDP / LPIPS
pip install "cozy-eval[video]"             # optical flow + track stability (opencv)
pip install "cozy-eval[quality]"           # NIQE / MUSIQ / ARNIQA / CLIP-IQA

Licence note. No dependency here is non-commercial, and none ever will be. pyiqa was dropped outright when it relicensed Apache-2.0 → PolyForm-Noncommercial-1.0.0 at 0.1.16 (2026-07-08): NIQE and MUSIQ are ours, ARNIQA and CLIP-IQA come from torchmetrics, and BRISQUE/MANIQA/TOPIQ are gone rather than ported. DOVER and FAST-VQA are deliberately not wrapped — both are S-Lab-1.0 (non-commercial) while their setup.py files still declare MIT/Apache. parity/ keeps the replacements honest against the banked oracle.

Use

from cozy_eval import (
    ChangeKind, Protocol, score_pairs, run_population_gate,
)

protocol = Protocol(
    family="ltx-2.3-distilled",
    reference_arm="bf16 compiled",
    candidate_arm="w8a8-pcs compiled",
    changes=(ChangeKind.WEIGHT_QUANTIZATION, ChangeKind.ACTIVATION_QUANTIZATION),
    width=1280, height=704, frames=121, steps=8,
    seeds=(8,),
    prompts=("forge", "chef", "market", "portrait",
             "fabric", "street", "water", "forest"),
    execution_lane="compiled",              # both arms, or it is INDETERMINATE
    hardware="NVIDIA H100 80GB HBM3",
)

pairs = score_pairs(reference_clips, candidate_clips)   # paths, dirs, or arrays
report = run_population_gate(pairs, protocol)
print(report.summary())

PASS  [population]
ltx-2.3-distilled | w8a8-pcs compiled vs bf16 compiled | 1280x704 x121f | 8 steps |
n=8 prompts | compiled both arms | NVIDIA H100 80GB HBM3 | same pod | lane=population
  PASS  imaging
        ok  0.92 <= imaging_index <= 1.25   measured 1.0050
        ok  imaging_worst_prompt >= 0.85    measured 0.9410
  PASS  temporal
        ok  jerk_excess <= 0.04             measured 0.0132
        ok  flicker_ratio <= 1.25           measured 1.0435
  PASS  distributional
        ok  significant_features <= 0.0     measured 0.0000
        ok  population_frechet <= 0.2       measured 0.0680

And the refusal:

>>> run_reference_gate(refs, cands, protocol)   # protocol says WEIGHT_QUANTIZATION
TrajectoryPerturbingError: reference metrics (PSNR/SSIM/LPIPS/VMAF) are invalid for
['weight-quantization']: these change the sampling trajectory, so the reference
render is a different take of the prompt and distance to it measures divergence,
not damage. …

Post-latent changes take the reference lane and are valid at n = 1:

run_reference_gate(before, after, protocol_with(ChangeKind.VAE_DECODE_DTYPE))

Images are the single-frame case — same lanes, same rules, run_image_population_gate drops the temporal benchmark.

One lane, one report: assess_lane

A lane is admitted by ONE report, produced by ONE function. The model package supplies inference through its entrypoints, the tools package supplies the candidate bytes, and the family's committed assessment/ assets supply the ratified conditions and workloads:

from cozy_eval import Conditions, Workload, assess_lane

report = await assess_lane(
    reference=BF16,                      # "org/model@release/lane"
    candidate=ATTN8,
    workloads=(FL2VA_8, REF2VA_CELL),    # entrypoint + payload + prompts + seeds
    conditions=CONDITIONS,               # cozy-eval/gate-config@3, pinned by digest
    tests=("weights", "activations", "video", "audio", "quality"),
    render=my_managed_call,              # the CALLER's; cozy-eval never renders
)
if report.verdict is Verdict.PASS:
    await attach_assessment(report, destination=MODEL)

Three arms render per workload — candidate, reference, and a same-seed reference REPEAT whose distance IS the noise floor — every one carrying capture=. render is the caller's callable because cozy-eval holds no credentials and picks no placement: the renders are ordinary managed calls to the model package's own entrypoints, and the arm it is handed is what stops a memoized call from returning the reference twice and measuring a floor of zero. The candidate arm renders FIRST, so a workload its package cannot bind refuses before a reference render is paid for.

What comes back is AssessmentReport (cozy-eval/checkpoint-validation@5, canonical bytes, digest = identity):

field	content
verdict	_overall: PASS iff every REQUIRED category is PASS; any FAIL is FAIL; else INDETERMINATE. A test not run is unmeasured — never a pass
categories	weights, activations, video, audio, quality, each with its measured rows, budget rows and reasons. Audio is its own category, never folded under video
subject	every input the report read, by digest: both checkpoints, workload refs + payload digests, the request id and media digest of every render, capture identities, the weights-comparison digest, the conditions digest
environment	per arm: runtime version, worker image, accelerator, driver/CUDA, execution contract and the OBSERVED attention kernel symbol, worker_boot_id → same_pod

Conditions (cozy-eval/gate-config@3) is the family's declared policy: the ladder's k_promote/k_reject/caps and pinned inputs, plus a typed Budget per category, each carrying the calibration that fixed it. A document that requires a category it ratifies no budget for refuses at construction; a category with no ratified budget is INDETERMINATE, never a pass. Thresholds are calibration data keyed by family, never constants in code — scripts/verify_assess.py re-ratifies population_frechet for its own corpus precisely because the shipped band, calibrated on 1280x704 H100 renders, is one a zero-change arm there trips.

An explicitly nonempty activation_absolute selects separate provenance-bearing activation ceilings. Every selected statistic must be measured at every declared binding-step tap across all prompts. Missing measurements refuse, and candidate NaN/Inf fails. Exact zero repeat floors remain reported observations and do not prevent this declared absolute policy from deciding. With no absolute budgets the original ratio/cap policy, including no_floor, is unchanged. Mixed absolute budgets and ratio-mode caps refuse; bounds must have a ceiling and no positive lower bound. Creating a policy document does not establish calibration or ratification: the caller must preserve the proposed status until its control evidence has been reviewed.

Audio is the population form and not k x floor for a measured reason: the banked H3 audio floor is exactly 0, so a k x floor audio bar fails every trajectory-perturbing change. Its permanent red arm is the EasyCache-corrupted render (-50 % amplitude), which reads -6.02 dB on both audio_rms_dbfs_delta and audio_lufs_delta and FAILS audio while the very same renders PASS video.

The three measurements underneath are usable on their own — compare_weights (two checkpoints' decoded leaves through a caller-supplied WeightsView; a tensor carried by reference is identical and reads zero bytes), compare_activations (three capture trees), score_outputs (three arms of media). Install cozy-eval[jobs] to use the managed measurement leaves; the base library remains numpy + msgspec and importing cozy_eval never imports Runtime. Learned instrument dependencies are an explicit [instruments] extra.

The memoized leaves are measure_media, compare_media, measure_audio, compare_audio, and measure_activation_pair. Each returns a canonical facts FileAsset and uses Runtime's ordinary durable operation journal plus TensorFS custody. Media and ordered pair content identities, options and captured implementation/dependencies identify the work. The calling script, run ID and acceptance budgets do not identify a measurement. Changing one candidate keeps the reference facts reusable; changing only Conditions folds a new decision over the same facts. Ordinary inference still runs on every invocation.

Pass video_measurements=, audio_measurements=, or quality_measurements= to the pure score_outputs fold; compare_activations accepts measurements=. assess_lane also accepts async measure_outputs= and measure_activations= callbacks, so an ordinary client script can issue leaf calls through its existing owner and return freshly folded results. Fact readers verify content and ordered-pair identities before reuse. Quality facts exclude timing and timestamps; managed learned-model execution remains a separate serving integration and is not claimed by the CPU measurement checks.

The diagnostic ladder (quantization gates)

These individual tier decisions are diagnostic. Admitting a lane requires the whole five-category assess_lane report; a tier-only PROMOTE supplies no weight, output or audio evidence.

A derive job's fidelity gate is a three-tier ladder (weight diff → activation probe → output comparison) with a cost cliff between the tiers, so it is not one compare() call: measurement and policy are separate surfaces, and the policy is one decision per stage so the cheap tier can spare you the expensive one. run_output_gate is tier 3's measurement in one call — it dispatches on the protocol's lane and frame count to the right gate above. The policy fold lives in cozy_eval.ladder: a GateConfig (K_promote / K_reject as multiples of the probe's measured ref-vs-ref noise floor, absolute caps, pinned prompts/seeds — a contract document, pinnable by digest per family) and two functions returning a typed GateDecision (PROMOTE / REJECT / ESCALATE):

from cozy_eval import LadderVerdict, output_decision, probe_decision, run_output_gate

d = probe_decision(probe, config)          # tier 2: strict-K short-circuit
if d.verdict is LadderVerdict.ESCALATE:    # only now pay for generations
    report = run_output_gate(reference_renders, candidate_renders, protocol)
    d = output_decision(report, config, probe=probe)   # FAIL names the layer

Every tap under k_promote x floor (and under its cap) promotes without generations; NaN/Inf or any tap over k_reject x floor rejects outright, naming the tap; anything between escalates to tier 3, whose failures read the tier-2 map to say which layer degraded. probe evidence is folded by compare_activations out of the three cozy.capture/1 trees the renders themselves wrote — the capture rides the render, so a probe never needs a second construction of the model.

Null controls: is this family allowed to be judged on these budgets?

The thresholds were calibrated on one family. Render a third arm on the same pod — the same checkpoint at different seeds — and let it decide which budgets you may believe:

from cozy_eval import ChangeKind, measure_null_control

control = measure_null_control(control_pairs, control_protocol)   # ChangeKind.SEED only
report = run_population_gate(pairs, protocol, null_control=control)

INDETERMINATE  [population]
  null control n=8: DOES NOT TRANSFER — population_frechet measured 0.4688
  PASS  imaging
        ok  0.92 <= imaging_index <= 1.25   measured 1.0739
  PASS  distributional
        ok  significant_features <= 0.0     measured 0.0000
        --- population_frechet <= 0.2       measured 0.1825
        note: … DISREGARDED: the null control — identical weights, seeds only —
              measured 0.4688 …, so this budget does not transfer to this family.
  ! budgets disregarded on the null control's evidence … the best available
    verdict is INDETERMINATE.

Measured on two image families: null controls read population_frechet 0.4688 and 1.3275 against a 0.20 budget, and one read imaging_worst_prompt 0.6102 against 0.85 — at zero model change. Without the control, one of those families reads a confident FAIL on budgets its own null arm fails harder. imaging_index and significant_features transferred on both. Details: GATE.md §4b.

Reports are digest-stable bytes

A report is the product: downstream jobs take it as a digest-verified typed input, so its BYTES are part of the contract.

from cozy_eval import contract

digest = contract.save("run.json", report, protocol=proto)   # sha256:...
sealed = contract.load_verified("run.json", digest)
sealed.body            # the typed SuiteReport
sealed.impls           # {metric: impl} — what makes two runs subtractable
sealed.require_protocol()   # or a typed refusal, if the run never declared one

contract.dump writes RFC 8785 canonical bytes: sorted keys, one number spelling, no whitespace, NaN/Infinity refused. Equal reports are equal bytes, so sha256 identifies a result. contract.load refuses non-canonical bytes and an unknown schema version, and ignores fields a newer writer added. Assessment checkpoint-validation@5 can retain an unobserved execution mode as an empty value. It reports a common mode only when every arm observed the same known value; missing observations do not prove mode equality. Optional mode metadata does not erase measured metrics, while null/repeat calibration still requires its known matching mode. Other document contracts retain their existing known-mode requirement. Schema ids live in contract.SCHEMAS and bump when a field is removed, retyped or changes meaning; additive fields keep the id. Policy documents (gate-config) refuse unknown fields.

ClipScore.to_dict() / from_dict() round-trip the gate's per-clip INPUTS (including the (frames, 6) feature matrix) so a lane can re-gate without re-rendering. That is a cache, not a report: no envelope, no digest.

CLI

cozy-eval score clip.mp4 other.mp4
cozy-eval gate --reference a.mp4 --candidate b.mp4 … \
    --change weight-quantization --execution-lane compiled …
cozy-eval compare --reference a.mp4 --candidate b.mp4 --vmaf \
    --change vae-decode-dtype …

Compute budget

An eval is a batch job on somebody else's machine, so it never takes more than min(4, cpu_count) compute threads — anywhere, including a 96-core pod. One knob raises or lowers it, and everything derives from that number: BLAS/OpenMP pools, torch intra- and inter-op, cv2.setNumThreads, every ffmpeg -threads and libvmaf n_threads, and the worker pool, which splits the budget (workers = threads / 2, each worker taking an equal share) rather than multiplying it. Every run prints what it took:

cozy-eval --threads 32 score clip.mp4      # configure(threads=32) as a library
# cozy-eval: 32 compute threads, 16 workers (explicit)

Set it explicitly on a dedicated pod — the default is deliberately modest and will leave a big machine idle. Pool variables (OMP_NUM_THREADS and friends) you set yourself are never overwritten. Thread count is invisible to results: one thread and thirty-two produce bit-identical scores, and a test pins that.

Rough cost, CPU-only at the default budget: the no-reference and integrity statistics are milliseconds per clip; optical-flow temporal fidelity is a few seconds per 720p pair (it samples FLOW_PAIRS transitions, so a 3-second and a 3-minute clip cost the same); per-frame LPIPS/SSIM/MS-SSIM and VMAF scale with frame count and dominate everything else; the VLM judge is a model call per sample and belongs on a GPU.

The three benchmarks

1. Imaging — per-frame, no-reference. Detail (Laplacian variance, spectral HF ratio, local contrast) and tone (contrast, saturation, histogram entropy), as ratios to the reference arm, median-aggregated across prompts.
2. Temporal — frame-to-frame. Jerk ratio (second temporal difference over the first), exposure flicker, shimmer, motion energy. Quantization noise often shows up here and nowhere else.
3. Distributional — a paired test across the whole prompt set, with Holm-corrected significance and a practical-effect floor. Not FVD: at n = 8 a deep-feature Fréchet distance is dominated by estimator bias. Because the prompt set is identical between arms, content is differenced out by construction and a paired test has real power.

Each catches a degradation the other two miss — see the validation table in GATE.md.

Did it make what was asked for?

The gate above tells you whether pixels moved for a valid reason. The suite answers the other question: is the render actually what the prompt requested? It exists because the good evaluation code in this space is locked up — GenEval2 is CC-BY-NC, pyiqa relicensed to PolyForm-Noncommercial, DOVER and FAST-VQA are S-Lab non-commercial. This library reimplements the published methods from the papers under MIT, with the audit trail in PROVENANCE.md.

Four mostly-independent dimensions, each with exactly one gated headline number; everything else is report-only:

dimension	what it asks	headline	needs a reference?	module
similarity	how far did the pixels move	lpips	yes	metrics/similarity.py, metrics/reference.py
adherence	did it contain what was asked for	element_recall	no	metrics/adherence.py, geneval.py, ocr.py, vqascore.py
preference	would a human prefer it	pref_delta	no	metrics/preference.py, hpsv3.py
quality	does it look good on its own terms	arniqa	no	metrics/quality.py, musiq.py, signal.py

The Δ-frame temporal channel and the flow family (metrics/temporal.py), the track-stability family (metrics/tracks.py) report into the dimensions above; the gate's own two-sided budgets over a whole prompt population live in cozy_eval.benchmarks, which is a threshold table, not a metric table.

from cozy_eval import promptset, suite

report = suite.run(samples, candidates,
                   checklists=promptset.checklists_for("hard-eval-v1"))
print(report.summary("element_recall"))

The load-bearing design decisions:

• Authored, versioned checklists. element_recall is the weighted fraction of a prompt's authored checklist verified present (ocr items read literally; vqa items answered by a VLM judge, one structured call per image). Checklists are versioned with their prompt set — never generated per run, so a score is reproducible. Shipped sets: hard-eval-v1 (t2i + edit), hard-video-v1 (16 frozen t2v prompts with motion/hold dualities).
• Editing is a duality: edit_compliance (the instructed change happened) vs edit_preservation (everything else stayed put) — under- and over-editing fail on opposite halves.
• Video (cozy_eval.video.run_video): per-frame aggregation with worst-frame tails, a Δ-frame temporal channel per-frame metrics cannot see, and motion/hold checklists judged on an ordered frame strip split into three temporal windows, gated on the worst — a clip that holds its content for 10 s and drops it for the last 5 is not two-thirds compliant, it is broken, and the whole-strip mean passed exactly that arm. element_recall_drop (first window minus last) reports the direction.
• Tri-state parity verdict (free_win / conditional_parity / reject): a candidate that fulfilled the request differently but equally well is not a failure — the case a pixel-distance metric cannot express.
• Registry as data. import cozy_eval sees the complete metric table without importing a single scoring backend; external metrics join through register().

Callers supply instruments through the Judge, SoftJudge, Transcriber, and AudioJudge protocols in cozy_eval.judge. Pass the selected instrument to the scoring function, such as video.run_video(..., judge=judge). The library does not provide a transport adapter to a separately deployed judge package.

Scoring backends are extras, so the base install stays torch-free:

pip install "cozy-eval[similarity]"      # LPIPS / SSIM / MS-SSIM / PSNR
pip install "cozy-eval[judge]"           # VLM judge, CLIP fallback, Grounding DINO + SigLIP2
pip install "cozy-eval[ocr]"             # OCR items (rapidocr, Apache-2.0)
pip install "cozy-eval[preference]"      # PickScore and alternates
pip install "cozy-eval[quality]"         # ARNIQA / CLIP-IQA / MUSIQ port / NIQE
pip install "cozy-eval[hpsv3]"           # HPSv3 preference scorer (16 GB weights)
pip install "cozy-eval[video]"           # frame handling, Δ-frame channel, flow + track stability
pip install "cozy-eval[all]"

Model licences — read before you ship. The library is MIT; the models it can load are not all MIT. It never defaults to weights that cannot be used commercially, and every model it touches has a verified row in PROVENANCE.md — including "none stated", where that is the truth. parity/ holds the harness that keeps the replacements honest against the non-commercial oracles — NIQE within 2.4%, MUSIQ within 4.5% with identical rankings, CLIP-IQA bit-identical under the oracle's own prompt set, ARNIQA deliberately diverged (antialiased half-scale) with the divergence isolated and recorded. The oracle NUMBERS, not code, are banked in parity/oracles/.

Stability: everything re-exported from the cozy_eval package root (metric names, the registry, the report schema, checklist/prompt-set formats, the verdicts, the Judge protocols, the protocol/lane rules) is locked for 0.x; everything under cozy_eval.metrics.* is experimental.

Is it a render at all?

Before any of the questions below is worth asking, one has to be settled: is this output anything? A production video model of ours served pure VAE-decoded noise on billed, settled requests and passed every check in place, because the evidence banked was container metadata and a billing row. Nobody looked at pixels. Metadata is not pixels.

from cozy_eval import output_integrity

checked = output_integrity(frames)          # (T, H, W, 3), uint8 or float
if not checked.ok:
    raise RuntimeError(checked.summary())   # names NOISE, BLANK or GRID

Real video is strongly self-similar frame to frame; noise is correlated with nothing. The median adjacent-frame grey correlation separates the two with an empty middle — noise 0.29, real renders 0.92-0.99, floor 0.60 — and taking the median over pairs spread across the clip is what keeps a hard cut from reading as noise. A per-frame contrast floor catches blank output, which correlates with nothing and so cannot be caught by correlation at all. numpy only, no reference, no model: 8.3 ms on a 121-frame 1344x768 clip, which is cheap enough to run on every render including the serve path.

A third axis: is it divided into little boxes? The screen once passed four renders the owner rejected on sight for a visible lattice — at adjacent_frame_corr 0.963-0.988, a comfortable pass. That is not a tuning miss. A periodic grid is perfectly self-similar frame to frame, so a correlation floor is structurally blind to it and the defect raises the number the screen reads. grid_peak_ratio measures what a lattice is and content is not: energy in a single spatial frequency, at a fixed phase, in the same place on every frame. Owner-accepted renders measure 1.975-3.804, owner-rejected ones 7.320-14.179, and the limit is 6.0 in the empty band — 1.54× above the worst known-good, the same relative headroom the noise floor carries. It takes the whole screen to a median of 30 ms. Derivation and the per-file table: GATE.md §5b; evidence calibration/grid-blockiness.json.

It is not a quality gate, and must never be quoted as one. It catches noise, blank and gridded output. A melted or over-smoothed render scores higher than a clean one — smearing removes high-frequency temporal variation, so damage looks like stability. Fine detail is detail_verdict and the VLM rubric; motion is the temporal-fidelity family. Four axes, none sufficient alone, and the library ships a test that pins this blind spot rather than a sentence claiming it does not exist.

Do the OBJECTS hold together?

Every number above is a statistic of frames. Per-frame detail, whole-frame optical flow over sampled pairs, frame-mean luma, an ordered strip of stills shown to a VLM. An object can warble — drift, jitter and reshape itself as the camera moves through the scene — while every one of those stays clean, and that is not a hypothetical:

"Objects lose their coherence across frames. Suppose object-A is at position X,Y and we move the camera slightly; the object should move on the frame correctly as you would expect when moving through 3-dimensional space. Instead it warbles and reshapes itself."

Three metric families passed the clips that produced that sentence: the fine-detail detectors (per frame), the temporal-fidelity family (whole-frame flow statistics over decimated pairs), and the VLM strip read. The clips were rejected by the owner's eye. Nothing in the stack followed a point on an object through time, so nothing in the stack could see it.

from cozy_eval import track_verdict

checked = track_verdict(candidate_frames, reference_frames)   # same-seed control
if not checked.ok:
    print(checked.summary())
    # track stability REJECT — OBJECT WARBLE: track_stability_ratio 0.176 <
    # floor 0.9 — the candidate retains 18% of the control's coherent tracks
    # (trajectories jitter: 0.322 vs 0.120; neighbours disagree: rigidity error
    # 0.609 vs 0.366) (track_stability 0.058, ratio 0.176)

Corner features are seeded and chased frame to frame with forward-backward validated pyramidal Lucas-Kanade, and each trajectory is asked three questions: is it smooth (a point on a rigid object traces a smooth image-plane curve whatever the camera does, so warble shows up as second-derivative energy — normalized by the track's own speed, so a fast pan is not penalized for being fast), does it survive (a surface that reshapes stops matching itself), and do its neighbours agree (points on one surface keep their relative geometry through parallax and perspective, which are smooth).

metric	dimension	gates?	what it says
track_stability_ratio	similarity	yes, 0.90 – 1.40	fraction of the control's coherent tracks the candidate retains
track_stability	quality	no	fraction of seeded points that survive and move like a real 3D point
track_survival	quality	no	fraction still tracked at the end of the window
track_jitter	quality	no	median per-track acceleration energy, camera motion removed
track_rigidity_error	quality	no	median jerk of neighbour distances — the "reshapes itself" half

Both bounds are measured on labeled ground truth (calibration/track-stability.json):

set	what it is	n	ratio	verdict
rejected	sparse-attention k16/k32 arms the owner rejected, vs their own same-cell same-seed dense control	29	0.029 – 0.846, median 0.366	29 reject, 0 pass
identical	SageAttention-2 fp8 arms the owner reviewed as identical to FA3-exact, plus same-arm re-renders across a pod and a torch-line change	12	0.930 – 1.251, median 1.050	12 pass, 0 reject
bit-exact	a clip against itself, two independent decodes and two independent scorings	2	exactly 1.0	pass
over-smoothed	the 15 s arm that re-rolled into a simpler, slower take and dropped the cargo bike and the parcel it was asked for	1	2.169	reject (passed before the ceiling)

0.90 sits in an empty middle 8 points wide, and 1.40 in a second one — 1.251 to 2.169 at the shipped budget, and 1.251 to 1.512 taken across every window budget, which is the middle the bound is actually placed in. The independent negative controls agree: an untrained-selector arm and a grouped-selector arm that a separate detector already called broken score 0.000 here, and the oracle top-k arm — itself a k16 sparse render — 0.606.

The gate is two-sided, and that is not decoration. A floor-only gate is walked through by anything that does LESS: the over-smoothed row above scored 2.17× its control and PASSED, because a simpler take is easier to track. The pathology is general, not ours — CD-FVD measured that sampling motion-free video lowers FVD by 31.6–54.8%, VBench's motion_smoothness measures interpolation predictability (which blur maximizes), and EvalCrafter's fitted composite has all three motion coefficients negative. The obvious companion instrument does not rescue a one-sided gate either: motion_mag_ratio reads 0.920 whole-frame and 1.006 under VBench Dynamic-Degree top-5% pooling on that same arm, inside the 0.80–1.05 band of pairs the owner called identical, at every pooling and window tried (calibration/motion-magnitude.json). It ships declared and report-only, with the negative recorded so it is not re-proposed. What an over-smoothed arm actually loses is content, and the instrument for that is the windowed element_recall below.

The paired ratio is valid across a re-rolled take, which is the whole point: both arms are scored on their own trajectories and never compared pixel to pixel, so a trajectory-perturbing lane is measured on whether its objects hold together, not on how far its take drifted. Same property as warp_error_delta, and the reason lpips cannot be used here at all.

Untrackable content is UNMEASURED, never a pass and never a fail. Steam, water, molten glass and dense repetitive weave defeat any sparse tracker: on those cells the clean control itself holds 13-18% of its tracks and its numbers swing further between two renders of the same arm than the whole rejected-vs-clean separation. When the control's survival is below 0.25 the family refuses — measured consequence: at a 0.15 floor, a loom pair the owner judged identical would read 0.07 and be called a catastrophic reject. 7 of 36 rejected pairs and 3 of 15 clean pairs land here and are reported as unmeasured, with the reason.

Scope. This catches object warble and 3D-inconsistency under motion. It is blind to per-frame damage (melted faces, pseudo-glyphs, halos — that is detail_verdict), to content adherence (the checklist and the VLM), and to whole-clip shimmer (warp_error). A clean ratio is one axis, not a quality verdict.

What one clip of the CPU tier costs (calibration/perclip-cost.json, measured on the labeled 362-frame 1344×768 pair, four threads): decode 2.2 s per clip, integrity 0.01 s, warp_error 2.3 s per arm, the paired temporal-fidelity block 5.0 s, the paired track block 1.6 s — and, until @9, 48.5 s of signal statistics to return two luma scalars. That one family was three quarters of the bill: it computes six per-frame feature families at full resolution (an FFT, a Laplacian, a 64-bin histogram, a box filter, saturation, luma σ) whose only consumer is benchmarks.imaging(), the population lane. The per-clip path now runs a luma-only pass that reproduces luma_flicker and jerk_ratio exactly (pinned by a test, not approximately): 48.5 s → 3.0 s, and the paired per-clip gate 61.8 s → 16.3 s. No metric was lost.

Cost and the decimation pin. No new dependency — the tracker is the same BSD-licensed OpenCV the flow family already uses, because the obvious learned tracker, CoTracker, is CC-BY-NC-4.0 and this library does not ship non-commercial weights (TAP-Net/TAPIR is Apache-2.0 but adds a checkpoint download and GPU inference to a CPU tier that costs a second). Four 24-frame windows at 384-line working height: 1.2 s median per clip on four idle threads, 3.1 s median in the banked run, which was taken on a shared box under a 1-minute load of 21 — the number moves with the machine, so both are quoted. The window count is the decimation knob, and it is pinned: verdicts on the whole labeled set are identical to tracking every frame of every clip (1.9× the cost), while halving it to two windows flips one clean pair to reject. Four is the floor of the ladder, not a default that happens to be cheap.

Does it SOUND right?

Every video model we serve now emits audio — LTX-2.3 denoises audio latents in the same loop and muxes AAC, MiniMax-H3 generates 32 kHz stereo jointly with the picture. A quant lane, a cache sweep or a step-distill LoRA can destroy the soundtrack while every pixel number stays green: on one banked arm a cache sweep drove audio SNR 20.67 → 13.72 dB while the same clip's video SSIM still read 0.85. There is, as far as we can find, no published audio-degradation measurement for any caching or quantization technique on a joint audio-video model — so this is a first-contact instrument, not a reimplementation.

One call, one verdict:

from cozy_eval import audio_verdict, read_audio

result = audio_verdict(
    read_audio("candidate.mp4"),
    read_audio("bf16_anchor.mp4"),   # optional: enables the faithfulness tier
    frames=candidate_frames, fps=24.0,   # optional: enables AV-sync
)
print(result.summary())
# audio REJECT — audio_stereo_separation_db 98.6 breaches audio_stereo_separation_db <= 60 (11 measured, 6 unmeasured)

Three tiers, and a fourth outcome that is not a tier:

tier	metrics	needs
signal, reference-free	audio_rms_dbfs audio_peak_dbfs audio_lufs audio_clip_fraction audio_silence_fraction audio_dc_offset audio_spectral_flatness audio_side_dbfs audio_stereo_separation_db audio_channel_correlation	nothing — always runs
paired fidelity	audio_si_sdr (gated) audio_snr_db audio_lsd_db audio_mel_l1 audio_lufs_delta audio_align_lag_ms	a reference arm
AV-sync	av_sync_offset_ms av_sync_confidence av_sync_drift_ms (gated)	frames + fps
semantic	audio_event_recall (gated) audio_speech_exact audio_speech_fuzzy	an authored checklist + a Transcriber and/or AudioJudge
UNMEASURED	—	everything the run could not score lands in result.unmeasured with the reason, and a verdict that measured nothing is unmeasured, never pass

Audio is not a seventh dimension. The six dimensions are questions, not media: audio SNR is similarity, a sound-of-X checklist is adherence, clipping and dual-mono are quality, AV-sync is temporal. So a quant arm that wrecks the audio reaches reject through the tri-state verdict machinery that already existed.

The one shipped budget. AUDIO_DEFECTS is this library's only built-in threshold table, and the exception is deliberate: silence, clipping, dual-mono and DC are content-independent engineering faults, unlike an LPIPS budget that only means something against the population it was calibrated on. It is calibrated on 41 real fal MiniMax-H3 generations — 30 ambience plus the 11-clip transient showcase — with a further 9 of our own H100 renders HELD OUT to validate rather than fix the limits. Every limit carries its own margin over the worst known-good clip, and nothing in it is report-only at @11.

Getting there meant replacing two statistics rather than re-fitting them. v1 calibrated on the ambience half alone, and audio_peak_dbfs ≤ -0.1 and audio_silence_fraction ≤ 0.50 are exactly the two numbers that separate ambience from transient content, in opposite directions — they failed four correct renders (both arms' drummers, a domino chain, a sparse blacksmith) for containing the impulsive events an AV-sync corpus must contain. So:

• sample peak → true peak (BS.1770-4, 4× oversampled), paired with clip fraction. A loudness-normalized transient sits at full scale by construction (a known-good domino chain peaks at +0.18 dBFS), and a master clipped upstream of a lossy codec decodes under full scale — the sample peak is wrong in both directions. Known-good tops out at +0.193 dBTP; the limit is +1.0.
• silent fraction → longest consecutive silent RUN. A known-good blacksmith clip is 61 % silent and correct: three hammer strikes with gaps. A stopped generator is one long stretch. Known-good tops out at a 0.136 run; the limit is 0.25, and the old fraction limit missed a generator that stopped 65 % of the way through entirely.

calibration/run_audio_defects.py re-derives all six over the real populations: 50 known-good clips pass, 8 red arms — each made by degrading a real clip at a known severity — all fail.

What it does NOT do, stated rather than hidden. AV-sync here is event sync — an audio onset envelope cross-correlated against a visual onset envelope, closed form, no weights (Bello et al. 2005; Hershey & Movellan, NIPS 1999). It does not score lip-sync: SyncNet's weights are research-worded CC-BY and Synchformer's checkpoints carry no weights-specific licence at all, so the gap is recorded and travels in every report that carries a sync number. And when the content has no shared audio/visual onsets, sync returns UNMEASURED with the reason — measured on the fal corpus, all 18 clips are unmeasurable for event sync, because generated ambience carries no transients. Gating sync needs a prompt subset authored to contain door slams, claps and dialogue — and authoring one is harder than it sounds, which is what cozy_eval.corpus exists for.

Corpora

A corpus is prompts and their authored checklists as one versioned unit: a row IS its checklist, so there is no representation of a prompt without its assertions and no way to edit them apart. It loads from a dataset tree (corpus.json + rows.jsonl + blobs), and an edit that never re-sealed is refused at load — corpus_id@version has to name exactly one set of rows or a score taken under it is not reproducible.

$ cozy-eval corpus lint calibration/corpus/av-sync-v1 --snapshot-root ~/cozy/samples
av-sync-v1@1  9 rows / 22 items  av=9  rows_digest sha256:138d7ee181dd53cf...
  WARN  sync-gate-texture-source   anime-courier-10s   names both an impulsive
        event ['landing', 'thud'] and a sustained source ['hum', 'wind']
  2 findings, 0 error(s)

The AV-sync authoring rule is an executable check, not advice. A sync-gated prompt must name an impulsive event, and the measured trap is that steam hiss and hail read percussive on paper and generate as untrackable texture. Two of the four clips originally commissioned as sync gates were exactly that. So the lint screens the rendered exemplar with the same onset instrument that will later measure the gate: peak-to-median envelope crest ≥ 17 dB and ≥ 2 prominent onsets. On the shipped corpus the four surviving gates read 17.6–32.7 dB and the rejected texture reads 7.0–15.4 dB.

The frozen v1 sets (hard-eval-v1, hard-video-v1) stay in-package, unchanged, and load through the same validator via corpus.packaged(name).

Documentation

• GATE.md — the protocol a producer lane follows verbatim: lanes, render conditions, thresholds with calibration provenance, the full validation table with separation margins, and an explicit list of what this gate does not measure.
• PROVENANCE.md — per module: which paper it implements, whether the implementation is original, and the real licence of every dependency and model weight involved.
• calibration/ — the evidence. run_banked.py regenerates every threshold from real renders plus synthetic single-axis controls.
• parity/ — banked oracle NUMBERS (never NC code) for every method reimplemented here, with the deterministic scenes they were taken on.
• scripts/verify.py — runs the real metrics on real generated inputs and prints the values, wall time and RSS it actually got. This is how the library is verified: there is no test suite (see Status).
• scripts/verify_assess.py — the same for the one evaluation system: assess_lane end to end over real lossless renders, real capture trees and real quantized tensors, with every red arm observed (the EasyCache audio render, the 25 % blur blend, the planted x2 weight scale, a NaN capture, a cross-pod pair, a zero floor). scripts/bank_captures.py writes the cozy.capture/1 trees it folds.
• scripts/fence.py — the architecture fences: no serving/storage/catalog dependency, base install stays numpy + msgspec, the environment is read once (into cozy_eval.settings.EvalSettings), registry identity holds, and no test suite creeps back.

Status

v2 line (0.5.0, metric set cozy-eval/metrics@14; CPython 3.12 only). Six dimensions — similarity, adherence, preference, quality, temporal, distributional — with one headline per dimension and modality. Cut at @10: video-mode ssim/ms_ssim (declared image-only; pooled over frames a windowed structural similarity rewards over-smoothing), the whole Δ-frame PSNR family (a reference distance taken of a reference distance), and edge_overshoot off the report surface (scene-confounded, and it ranks arms backwards). The runner now REFUSES reference metrics on a trajectory-perturbing lane instead of printing a quotable confounded number.

There is no test suite, deliberately. Verification is running the real metrics on real inputs and recording what happened: python scripts/verify.py is 92 invariant checks and 31 metrics benchmarked live, and CI is build plus fences only.

Alpha. The thresholds are calibrated on LTX-2.3, Wan-2.2 and H3 class video at 720p–1080p, 4–30 steps; a family far outside that should carry a null-control arm (and, better, re-derive the budgets from its own clean population). population_frechet gained its second clean population at @11 — H3, n=9 pairs the owner labelled identical, 0.0665 against the banked LTX population's 0.0680 on a different family — so it gates on video, with an owner-rejected sparse-attention arm 51× away at 3.4191. It is deliberately absent from IMAGE_BUDGETS: on images the only evidence is two zero-change null controls at 0.4688 and 1.3275, and a budget that does not transfer is not a budget there.

Licence

MIT. See LICENSE.

Explicit paired output fidelity

Protocol(purpose=Purpose.PAIRED_FIDELITY, changes=(ChangeKind.WEIGHT_QUANTIZATION,), ...) asks how far matched outputs moved. Quantization still changes the trajectory; this purpose does not turn a distance into a perceptual-quality or population verdict. Render the reference, candidate and a same-seed BF16 repeat through the actual inference package with pinned prompts, seeds, settings and geometry. The scorer does not generate media and never silently resizes or trims it.

measure_repeat_control measures the same reference metrics on a protocol declaring only ChangeKind.REPEAT. Pass it as repeat_control= to run_reference_gate or run_output_gate, alongside explicit calibrated distance budgets. A missing repeat, unmeasured required metric, or a budget exceeded by the reference repeat prevents PASS. The existing different-seed population null control answers a separate question.

The one report a lane is admitted on is assess_lane's AssessmentReport (cozy-eval/checkpoint-validation@5). Its subject retains the exact candidate/reference checkpoint identities, the workload refs and payload digests, the request id and media digest of every render, the capture identities, and the conditions digest — every input the report read. The attaching owner verifies those against its retained render requests; the library does not claim that digest-shaped labels prove custody. Missing evidence stays INDETERMINATE, and an individual diagnostic tier decision cannot admit a lane on its own.

For sampled video quality, pass an explicit quality_judge= to run_video. The existing Judge interface supplies the resident instrument; the library loads no video judge. The report retains the actual APS scores, raw reply, exact model reference and sampled frame indices (up to sixteen, including the last frame). This instrument coverage is separate from full-clip integrity. use_clip=False explicitly skips the separate CLIP scorer without disabling the supplied video judge.

All thresholds still require model-family calibration. Procedural controls and instrument smoke proofs establish behavior; they do not establish H3 checkpoint quality or replace an actual run of the pinned inference package. Checkpoint publication/promotion is a separate owner decision, never an automatic act of this library.

Inspect a stored assessment

cozy-eval report inspect reads at most 8 MiB from stdin, validates the exact canonical checkpoint-validation@5 bytes through the existing contract reader, RE-DERIVES _overall from the categories, and writes one cozy-eval/report-inspection@4 projection: the byte reference, the subject's digests, the required set, the per-category verdicts (audio among them), the pod facts, and each arm's observed kernel symbol. It loads no models and accepts no path or URL. A @2 document refuses — one schema id per contract, never a partial read of a newer one. Measured values and reasons are report-author text and stay out of a host log. This is format/policy validation, not proof that the publisher ran the claimed renders.

Read-only managed weight measurements (Runtime 0.18 / Eval 0.7.1)

The jobs extra uses TensorFS 0.3.42 native source identities and scoped reads. The core assessment and policy modules keep their NumPy/msgspec-only dependency boundary.

cozy_eval.jobs.measure_weights consumes two already-held model artifacts through Runtime's injected WeightsReader. The normal ModelArtifact-to-Model binding establishes the grants; the reader opens no output transaction and exposes no Store, path or forged receipt. Install the weights extra for Runtime's qualified Torch CPU decoders; the base cozy_eval import remains NumPy/msgspec only.

from cozy_eval.jobs import measure_weights
from cozy_eval.weights import read_weights

measured = await measure_weights(reference=reference, candidate=candidate)
facts = read_weights(measured.facts.read_bytes())
report = await assess_lane(reference.manifest.digest, candidate.manifest.digest,
    workloads, conditions, render=render, weight_measurements=facts,
    measure_outputs=measure_outputs, measure_activations=measure_activations)

The leaf memoizes complete cozy-eval/weight-pair-facts@1 data, without Conditions, thresholds, verdicts or timestamps. Exact encoded identity includes all roles, so inherited tensors skip reads. Changed tensors are reconstructed through the qualified CPU decoder into float32 chunks and compared with the existing chunked float64 Frobenius calculation. A changed all-zero reference has no finite relative error and refuses rather than reporting zero.

Quantizer-time saturation is not inferable from stored decoded weights. Native views return it as unobserved, including inherited rows unless actual producer counter metadata was supplied. The optional SaturationObservation protocol serves that already-observed metadata without forcing tensor reads. Unknown counters remain null in raw rows/component summaries and are omitted from budget inputs. A policy requiring saturated_elements is INDETERMINATE; a policy using only measured Frobenius facts can still be evaluated. Editing a policy re-folds the same facts without rereading weights.

Assessment bodies now use cozy-eval/checkpoint-validation@5 because these nested counter fields are nullable. Older assessment schemas refuse on this build. The cozy-eval/report-inspection@4 output structure is unchanged; its nested report schema is now@5. Known measurement values and unrelated document contracts do not change.

Model-free controls are in scripts/verify_weight_measurements.py; they establish numeric/fact/reader behavior, not a native-reader, model-inference or quality proof.