Post-training LoRA evaluation and optimisation: generate samples from a trained LoRA and score them against the curated dataset it was trained from.
Part of the Argus suite — the stage that closes the loop after argus-forge emits a training config and you train a LoRA:
argus-quarry -> argus-curator -> argus-lens -> argus-forge -> your trainer -> argus-proof
acquire curate/export caption configs LoRA evaluate + optimise
Status: functional end-to-end. The executing verbs (
run,score,report,inspect) are live, the server triggers generation+scoring runs and serves the resulting images to the argus-studio/proofreview view. Roadmap: argus-proof epic.
Demo: proof.dragonhound.dev replays one recorded evaluation: a synthetic-identity
SDXL LoRA trained for 1600 steps, every 320-step checkpoint and a no-LoRA baseline scored by proof (144 images), and
where it peaks. It is a static page with no backend; the recording is web/public/tape/, built by
scripts/demo_tape.py <recording-dir>, and make pages exports the site to web/out/.
Install
uv pip install "argus-proof[cli]" # CLI
uv pip install "argus-proof[cli,server]" # + HTTP server for argus-studio
uv pip install "argus-proof[cli,score]" # + the full scorer stack (torch, insightface, …)
[preference] (ImageReward) can't share an environment with [score]: it pins timm==0.6.13, which pyiqa 0.1.16+
can't use. ImageReward isn't in the default scorer set, so install it on its own when you want the preference metric.
Images are on GHCR. ghcr.io/smk762/argus-proof:<version> (and latest) is the thin server/CLI and scores phash
only: identity, likeness, CLIP score and aesthetic come back empty. To score for real, use :<version>-score (latest-score),
which adds the [score] stack, with CUDA torch and onnxruntime-gpu (they fall back to CPU without a GPU). To build
from a context without .git (a git archive export), pass --build-arg VERSION=<x.y.z>.
CLI
# Generate a sample grid from a trained LoRA (prompts come from the curator
# export's captions; backend/engine/model dirs from the environment, see
# .env.example — PROOF_BACKEND, COMFYUI_BASE_URL, PROOF_MODELS_DIR):
argus-proof run subject.safetensors ./curated_export \
--checkpoint sdxl_base.safetensors --seed 1 --seed 2 --seed 3 --out runs
# A run that fails (ComfyUI retries a timed-out seed once first) leaves failed.json in its dir
# instead of a manifest, and the rest of the grid still runs; the command then exits 1 listing
# the failed run_ids. --fail-fast stops at the first failure.
# Score a generated run into a stored EvalReport (identity and likeness need --references,
# a held-out image dir that must NOT overlap the training set). A default scorer
# this install can't run is warned about on stderr; --require-scorers makes it exit 2:
argus-proof score runs/proof-l00-w00-p000 --references ./holdout \
[--require-scorers identity,likeness,clip_score,aesthetic,safety]
# Browse stored reports / print one run's digest:
argus-proof report
argus-proof report proof-l00-w00-p000 [--json]
# Summarise an export dir (prompt sources) or a run dir (manifest + images):
argus-proof inspect ./curated_export
argus-proof inspect runs/proof-l00-w00-p000
# Emit the wire-contract JSON Schema consumers codegen against (default
# schema/proof-wire.schema.json; --check exits non-zero if the committed copy is stale):
argus-proof schema [--output PATH] [--check]
gate, recommend, experiment and explore are covered in their own sections below.
Serve
argus-proof serve --port 8104 --cors # peer to lens :8100, curator :8101, quarry :8102, forge :8103
argus-proof serve --read-only # replay/demo mode: serve stored reports, 403 live eval + writes
curl -s localhost:8104/health # {"status":"ok","service":"argus-proof","version":"...","read_only":false,"capabilities":["report_manifest"]}
Routes (the backend of the argus-studio /proof view):
GET /health liveness + version + read_only flag + capabilities
GET /exports curator export dirs available to evaluate against
GET /models checkpoints + LoRAs under $PROOF_MODELS_DIR
GET /scorers scorers a run applies + whether each is installed
POST /run/stream generate + score + store one run (NDJSON RunStreamFrame lines)
POST /score score uploaded images vs an export's references (ScoreResult; stores nothing)
GET /reports stored report digests (run browser)
GET /report/{run_id} full EvalReport PUT to store one
GET /report/{run_id}/manifest the RunManifest behind a report (410 if its run dir is gone)
GET /report/{run_id}/refined passing subset, refined ranks first
GET /report/{run_id}/image/{image_id} a generated sample (ids only — no path input)
GET /report/{run_id}/image_at/{index} sample by position — seed-free URL for blind review
POST /report/{run_id}/hitl apply a review; recomputes pass-rate + verdict
POST /report/{run_id}/refine second-pass re-rank (rank: null retracts)
POST /purge/{run_id} devmode only: delete one run's report + run dir
POST /purge devmode only: delete every report + run dir
/report/{run_id}/manifest serves the SHA256-pinned setup (checkpoint, VAE,
LoRAs, sampling, prompt, seeds) so a client can carry a passed run forward, e.g.
studio's proof → darkroom handoff. It is 404 when there is no report and 410 Gone when the report was kept but the run dir was pruned. /health lists
report_manifest in capabilities, so a client can tell an older proof (route
absent) from a run with no manifest.
POST /score scores images the caller already has against an export's held-out
references/, e.g. argus-mint scoring a finished teaser against an identity (mint#51).
It stores nothing. The body is multipart/form-data:
images: 1 to 16 files (PNG/JPEG/WebP, at most 2048 px a side and 32 MiB each), scored in upload order;request: a JSONScoreRequest:
{
"export": "android-k7",
"scorers": ["likeness"],
"target_category": "identity",
"crops": [{"x": 312, "y": 40, "width": 400, "height": 900}, null]
}
scorers is likeness (DINOv2, any subject) and/or identity (ArcFace, faces; it
reads null unless target_category is identity). crops is optional, one entry
per image (null = the whole image), in the image's own pixels after its EXIF
orientation; scoring the character's box rather than the whole teaser keeps the
background out of the embedding. The reply is a ScoreResult:
{
"proof_version": "1.9",
"n_images": 2,
"images": [
{"index": 0, "filename": "t1.png", "crop": {"x": 312, "y": 40, "width": 400, "height": 900},
"metrics": {"identity": null, "likeness": 0.83, "clip_score": null, "aesthetic": null, "preference": null, "safety": null}}
],
"means": {"identity": null, "likeness": 0.79, "...": null},
"scorers": [
{"name": "likeness", "metric": "likeness", "version": null, "model": "dinov2:facebook/dinov2-base",
"model_file": {"name": "dinov2:facebook/dinov2-base", "sha256": "d73036b5…"}},
{"name": "held-out-references", "metric": "identity", "version": null, "model": "sha256+phash<=6", "model_file": null}
]
}
There is no verdict: the caller holds the scores against its own threshold. Errors:
400 (references that are training images, an export with no references/, a crop
outside its image, an unreadable image, a crops/images count mismatch, or more than
16 images), 404 (no such export), 413 (an image over 32 MiB), 422 (a malformed
request, an unknown scorer), 503 (a requested scorer isn't installed, the GPU
lease was refused, or no exports dir is configured), 403 in read-only mode. It runs
under one interactive score GPU lease and frees its models before the lease ends.
/health lists score in capabilities when the host isn't read-only.
The /purge routes are a dev tool, since reports are the evaluation record. They exist
only with ARGUS_PROOF_DEVMODE=1 (or serve --devmode), need a {"confirm": "purge"}
body, return 409 while the run (or, for POST /purge, any run) is still generating,
and read-only mode refuses them. A run's HITL review and refinement are stored in its
report, so they go with it. /health echoes devmode and lists purge in
capabilities only when the routes will work.
Concurrent reviews of the same run are serialised with a per-run file lock, so a streaming run's report write and a reviewer's HITL save can't drop updates.
Generation backend
Generation is a pluggable backend (argus_proof.backends) so swapping the engine
is a config change, not a code change — the ComfyUI adapter ships first:
from pathlib import Path
from argus_proof.backends import get_backend
from argus_proof.backends.base import make_dir_resolver
from argus_proof.backends.workflow import example_template
from argus_proof.models import RunSpec, LoRASpec, SamplingParams
backend = get_backend(
"comfyui",
workflow_template=example_template(), # or workflow.load_template(path)
resolve_model=make_dir_resolver(Path("~/ComfyUI/models")),
base_url="http://127.0.0.1:8188",
)
spec = RunSpec(
run_id="run-1",
base_checkpoint="sdxl_base.safetensors",
loras=[LoRASpec(name="subject.safetensors", weight=0.8)],
sampling=SamplingParams(sampler="dpmpp_2m", scheduler="karras", steps=30,
cfg=7.0, clip_skip=2, width=1024, height=1024),
prompt="a photo of sks person",
seeds=[1, 2, 3], # seed-set: one image per seed
)
result = backend.generate(spec, Path("out/run-1")) # writes images + manifest.json
The ComfyUI adapter drives a parametric workflow template (an API-format graph
with $placeholder values — $base_checkpoint, $positive, $seed, $steps,
$lora / $lora_weight, $clip_skip, …), polls for completion, reads back each
image's embedded PNGInfo, and emits a RunManifest that pins every
checkpoint/LoRA by SHA256 so the run reconstructs exactly. See
templates/comfyui_sdxl_lora.json
for the shipped example.
SDXL and Flux. The template is chosen by the base checkpoint's family, its
top-level folder under checkpoints/: flux/… renders
templates/comfyui_flux_lora.json
and every other family renders the SDXL template, so one server evaluates both.
PROOF_WORKFLOW_TEMPLATE replaces both for every checkpoint. The Flux checkpoints
are bare transformers, so the Flux template also loads two text encoders
(DualCLIPLoader, type flux: the $text_encoder and $text_encoder_2 slots) and
a VAE (VAELoader: $vae). They come from the run (RunSpec.text_encoders, vae)
and the manifest pins each by SHA256, like the checkpoint and LoRA, so swapping the
T5 for another build changes the manifest. A run that names none gets the Flux
defaults, t5xxl_fp8_e4m3fn.safetensors + clip_l.safetensors and ae.safetensors
(/run/stream's text_encoders/vae, argus-proof run --text-encoder/--vae); they
must be in ComfyUI's clip/ and vae/ folders and under PROOF_MODELS_DIR. The
template's encoder slots and the run's list must match one for one. A template
whose loader names a model file literally ("vae_name": "ae.safetensors") is
refused: that file would load unrecorded. Flux takes no negative prompt (the template zeroes the positive) and runs
at cfg 1 with sampling.guidance (the $guidance slot, a FluxGuidance node).
A request that leaves cfg/sampler/scheduler/guidance out gets its family's
defaults: SDXL cfg 7, dpmpp_2m/karras; Flux cfg 1, euler/simple, guidance 3.5.
Nothing a run sets is silently dropped. A run is refused when it sets a value
the template has no slot for: a VAE, a non-empty negative prompt, guidance,
or a clip_skip above 2. The diffusers and a1111 backends refuse guidance
and text_encoders outright. clip_skip is sent to ComfyUI as -max(clip_skip, 2): SDXL uses the
penultimate layer, and -1 blanks some checkpoints.
More backends. get_backend(name, ...) selects the engine by config
(PROOF_BACKEND in .env.example); scoring/report code is
unchanged regardless of which produced the run, and the RunManifest records the
engine + version:
diffusers— in-process diffusers SDXL pipeline: deterministic, no external service, weights hashed from disk (pip install "argus-proof[diffusers]"). Limitations:clip_skipis not applied, and only a handful of samplers map to a diffusers scheduler (euler,euler_a,ddim,dpmpp_2m,unipc; unknown names fall back to the pipeline default). The manifest still records the requested values, so compare across engines with care.a1111— an AUTOMATIC1111 / SD.Next/sdapiserver (checkpoint viaoverride_settings, LoRAs via<lora:…>prompt syntax; models hashed from disk).remote— a hosted/cloud endpoint that speaks the proof wire (POST /generate→RunManifest+ base64 images). The weights live remotely, so the service supplies the manifest and it's validated at the boundary; a bearerapi_keyauthenticates. Point it at a self-hosted proof-gen service or a thin wrapper in front of Replicate / fal.
The a1111 / remote adapters need no extra (stdlib HTTP); all three reuse the
shared manifest + transport helpers and are unit-tested with fakes.
GPU leases (argus-iris). The comfyui and diffusers backends hold a GPU lease
for each run from the suite's coordinator, set host-wide by ARGUS_GPU_* (unset:
no gating, as before). With ARGUS_GPU_COORDINATOR=broker, a run asks
argus-iris for a batch lease, on
comfyui (kind generate) or in-process (kind generate-local), declared in
gpu-tenant.yaml. The work goes to the ComfyUI the grant
names (else COMFYUI_BASE_URL). Between seeds proof checks should_yield():
when the broker wants the GPU back, it releases the lease and queues again,
pinned to the node the run started on, because the manifest hashed one set of
model files. A refused or timed-out lease fails the run with a message that says
so. When iris can't be reached, the lease comes from ARGUS_GPU_BROKER_FALLBACK
(lease, the host-wide flock, by default). The a1111 and remote backends
take no lease.
Scoring leases too. score_images (so argus-proof score and /run/stream) holds
one score lease (in-process, not preemptible, 3500 MiB) for the scoring pass
whenever an available scorer runs on the GPU (InsightFace, CLIP-IQA, CLIPScore; not
safety, which is a NudeNet service, and not phash). It is interactive, or batch
when it is part of a /run/stream run. The scorers load their models inside the
lease and free them before it ends, so proof keeps no model resident between leases
(residents: []): each pass reloads its models, a few seconds' cost. A refused or
timed-out lease fails scoring with a message that says so. The 3500 MiB is the torch
scorers' measured peak on an RTX 3090 (1399 MiB with the CUDA context) plus an
estimated 1500 MiB for InsightFace on CUDA, rounded up; gpu-tenant.yaml has the
breakdown.
| Variable | Value |
|---|---|
ARGUS_GPU_COORDINATOR |
broker (or lease, none) |
ARGUS_GPU_BROKER_URL |
http://<GPU_HOST>:8120 |
ARGUS_GPU_BROKER_TOKEN |
the shared caller token (secret: the host's .env, never a repo) |
ARGUS_GPU_TENANT_MANIFEST |
gpu-tenant.yaml (the image ships it at /app/gpu-tenant.yaml) |
ARGUS_GPU_BROKER_FALLBACK |
lease (default) or none |
Scoring
Generated images are scored into an EvalReport by a pluggable framework
(argus_proof.scoring). Per-image ImageScorers each fill one normalised
[0,1] metric (identity / likeness / clip_score / aesthetic / preference / safety); a
Deduper collapses Monte-Carlo near-duplicates so a cluster counts once
toward the pass rate; a DiversityScorer rewards variety. A GateConfig routes
each image to auto-pass / auto-fail / needs-HITL on a weighted composite,
so humans only rate the borderline band:
from argus_proof.scoring import score_run, ScoreContext
report = score_run(manifest, images, scorers=[...], deduper=..., diversity=...)
report.aggregate.pass_rate # computed over near-dup groups, not raw frames
report.verdict.passed # run pass/fail vs GateConfig.run_pass_rate
Concrete scorers live in argus_proof.scoring.scorers, behind the [score]
extra, and are lazy-imported (each reports is_available() so the orchestrator
skips it when the extra is absent). Shipped:
- dedup + diversity —
PhashDeduper,PhashDiversityScorer(perceptual hash, CPU-only) - identity —
IdentityScorer(InsightFace ArcFace cosine vs a held-out reference set). References must be held out from training, and a run refuses ones that aren't: before generating (/run/stream) and before scoring, each reference is compared with the export's training images (its files outsidereferences/, plus the manifest rows' sources). An exact copy is caught by SHA256; with the[score]extra, a resized or re-encoded copy is caught by a whole-image pHash within 6 bits. A crop is not caught. The report'sscorersgains aheld-out-referencesrow saying which check ran (sha256+phash<=6,sha256, or why none could). - likeness —
LikenessScorer(DINOv2 whole-image embedding cosine vs the same held-out reference set, #103). The identity score for subjects ArcFace can't see: helmeted, masked, chrome-skinned or non-human characters, and scenes or sets, whereidentityreads0.0(no face). It runs on every target category whenever there are references (Nonewithout), and the held-out refusal above covers it, since both scorers read the one reference set. DINOv2 rather than CLIP image embeddings: CLIP is trained to match captions, so two different chrome robots embed close together because both are "a chrome robot"; DINOv2 is self-supervised on images and keeps instance-level appearance. It loads throughtransformersin the[score]extra, which CLIPScore and pyiqa already bring.PROOF_LIKENESS_MODELpicks the checkpoint (defaultfacebook/dinov2-base; a Hub id or a localsave_pretraineddir for an offline host). Once loaded, itsmodel.safetensorsis pinned by SHA256 in the report's scorer provenance (ScorerProvenance.model_file), as aRunManifestpins the generation models. The raw cosine is clamped, not calibrated: unrelated images still sit around 0.1–0.4 (and simple synthetic shapes higher), so take--min-likenessfrom a real run.likenesshas weight 0 in the default composite, because it is uncalibrated and its threshold comes from the mint pilot run: it acts only through--min-likeness, alikenesshard gate (reject codeidentity_mismatch) or an explicitGateConfig.weights["likeness"]. - quality / adherence —
clip_score_scorer()(CLIPScore),pyiqa_scorer()(CLIP-IQA),image_reward_scorer()(ImageReward), each normalising its raw score to[0,1] - safety —
SafetyScorer(NudeNet ensemble,1 - unsafe); set asafetyhard gate to auto-fail unsafe images.safety_tail_aggregate()surfaces the any-hit/min/percentile tail. NudeNet runs as a server over HTTP (PROOF_NUDENET_URL, via argus-cortex'sNudeNetClient), never in-process (it is AGPL-3.0); without a URL the scorer is skipped. Point it at the suite's shared 640m server, argus-retina'sghcr.io/smk762/argus-retina-nudenet(http://<GPU_HOST>:8130on the LAN); the upstreamnotai-tech/nudenetimage is 320n. The two score the same image differently, so the report records which one ran (the safety scorer'smodel:nudenet-640m,nudenet-320n, ornudenetwhen the server doesn't say, as the upstream image doesn't), and the cross-run store keeps it as a sliceablesafety_detectorcolumn. The cutoff is tuned for 640m: an exposed score of at least 0.2 is unsafe, sounsafe_safety_floordefaults to 0.8 and an image at or below it counts (the acceptance gate'smax_unsafe_rate,recommend, andsafety_tail_aggregate()). That is NudeNet's reporting floor, so any exposed detection counts, as in curator and cull. It is deliberately conservative and may flag the odd false positive; raise the floor's exposed score if that becomes a problem. "Unsafe" is the suite's exposed-nudity set (exposed buttocks, breasts, genitalia, anus), the same as curator's
The default scorer set used by argus-proof run / score and the server
(evaluate.default_scorers(), plus phash dedup/diversity) is identity, likeness, CLIPScore,
CLIP-IQA and safety — it does not include ImageReward, so the preference
metric stays empty unless you pass image_reward_scorer() via the library API (in a [preference] environment).
GET /scorers reports which of the default set this install can actually run.
from argus_proof.scoring import score_run
from argus_proof.scoring.scorers import (
PhashDeduper, PhashDiversityScorer, IdentityScorer, clip_score_scorer, pyiqa_scorer,
)
report = score_run(
manifest, images,
scorers=[IdentityScorer(), clip_score_scorer(), pyiqa_scorer()],
deduper=PhashDeduper(), diversity=PhashDiversityScorer(),
)
The quality scorers' default
[0,1]normalization ranges are placeholders — calibratelo/hi(e.g.clip_score_scorer(lo=…, hi=…)) against real generations. Heavy backends needpip install "argus-proof[score]"; remote/hosted variants build onargus_cortex.backends.RemoteBackend(point at a service by IP/port). The spine itself is dependency-free and fully tested with fakes.
HITL review & refinement
Reports are stored per-run (argus_proof.reports.ReportStore, a directory of
<run_id>.json) and reviewed over the server (peer to the argus-studio /proof
view):
POST /report/{run_id}/hitl # 5-star ratings + reject reasons; recomputes pass-rate + verdict
POST /report/{run_id}/refine # optional second pass: re-rank the passing subset 1-5 + notes
The refinement stage (argus_proof.refinement) is a finer re-rank of just
the images that already passed — a separate layer (ImageScores.refinement)
that never overwrites the first-pass hitl_rating/reject_reasons or the run's
verdict, so both the original decision and the refined ordering are kept.
refined_ranking(report) surfaces the passing subset best-first; refining an
image that isn't in the passing subset is refused.
from argus_proof.refinement import RefinementRequest, RefinementImageUpdate, apply_refinement, refined_ranking
refined = apply_refinement(report, RefinementRequest(
rater="alice", updates=[RefinementImageUpdate(image_id="img-3", rank=5, notes="cleanest hands")],
))
best_first = refined_ranking(refined) # passing subset, refined re-ranks on top
Policy moderation (optional)
The Phase-2 safety scorer catches nudity; argus_proof.moderation extends
it to a Guard-class policy taxonomy (sexual / violence / hate / self-harm /
weapons / illegal) — over both the generated images and the input prompts /
captions, so a toxic prompt is flagged even when its output is clean.
The default detectors are Llama Guard 3: LlamaGuardImageDetector
(meta-llama/Llama-Guard-3-11B-Vision, about 22 GB in bf16) for outputs and
LlamaGuardTextDetector (meta-llama/Llama-Guard-3-8B, about 16 GB) for prompts and
captions. Install argus-proof[moderation]. Both checkpoints are gated on Hugging
Face: accept the license on each model page and set HF_TOKEN. The weights download
at first use; nothing is bundled. Device placement goes through argus-cortex's
resolve_device (device="auto" picks cuda, then mps, then cpu).
Each verdict (safe, or unsafe plus hazard codes S1–S14) maps onto the taxonomy.
A flagged category scores P(unsafe) at the verdict token rather than a hard 1.0.
Moderation fails loud rather than reading as safe. A model that can't load (no
token, license not accepted) or an unreadable verdict raises ModerationError, and an
unsafe verdict naming no code flags every category. An image is sent with the
neutral prompt "Describe this image.", so a flag comes from the image.
Licenses. Llama-Guard-3-8B is under the Llama 3.1 Community License, and Llama-Guard-3-11B-Vision under the Llama 3.2 Community License. The 3.2 license does not grant its rights for multimodal models to individuals domiciled in, or companies with their principal place of business in, the European Union (end users of a product built on them are exempt). Both carry Meta's acceptable use policy. argus-proof only downloads the weights at runtime and doesn't redistribute them. Check the terms against your use, and pass your own detectors if they don't fit.
Any object with name, version, is_available() and moderate_image(path) /
moderate_text(text) returning per-category [0,1] scores can replace them:
from argus_proof.moderation import PolicyModerator, moderate_images, moderate_texts
mod = PolicyModerator() # Llama Guard 3 Vision + text
# mod = PolicyModerator(image_detectors=[MyImageGuard()], text_detectors=[MyTextGuard()])
out = moderate_images(image_paths, mod) # per-category tails over the outputs
inp = moderate_texts(prompt_grid_variants, mod) # ...and over the inputs
out.flagged() # e.g. ["violence", "hate"], worst first
report.scorers.append(mod.provenance("output")) # version-stamp the Guard model + taxonomy
Each category gets a tail view (any-hit / max / 95th percentile — the extremes
that matter, not the mean, same rule as safety), combined conservatively across
an ensemble (most-unsafe detector wins). Detectors are pluggable and injectable, so
the taxonomy/ensemble/tail logic is dependency-free and unit-tested. Of the Llama
Guard adapters, only the model call needs the [moderation] extra; verdict parsing
and mapping are unit-tested. A reviewer's HITL flag attributes to a
category via RejectReason.category. CSAM matching stays a separate policy gate
(Thorn Safer / PhotoDNA), not an ML metric here.
CI acceptance gate
Turn "was this LoRA/dataset good enough?" into an automatable yes/no. argus-proof gate evaluates a scored EvalReport against declared thresholds and exits
non-zero when rejected, so it drops straight into CI:
# --min-pass-rate-ci-lower: Wilson lower bound — a lucky 3/3 won't pass
# --confidence: confidence level for that bound (default 0.95)
# exit 0 = accepted, 1 = rejected, 2 = unreadable report / invalid threshold
argus-proof gate eval_report.json \
--min-pass-rate 0.75 \
--min-pass-rate-ci-lower 0.7 \
--confidence 0.95 \
--min-identity 0.6 \
--min-likeness 0.7 \
--max-unsafe-rate 0.0
--min-identity floors the mean ArcFace identity (human faces); --min-likeness
floors the mean DINOv2 likeness (any subject, AcceptanceThresholds.min_likeness_mean).
Use whichever measures the subject, or both.
The pass-rate lower bound uses a Wilson score interval (argus_proof.stats, no
scipy), so acceptance is statistically defensible at small N. A configured metric
that wasn't measured fails its check rather than passing silently.
Cross-run stats
Per-run reports accumulate into a queryable store so "which checkpoint / LoRA
weight / token wins?" is answered with evidence, not vibes (argus_proof.crossrun,
pip install "argus-proof[stats]"):
from argus_proof.crossrun import CrossRunStore, run_stats, krippendorff_alpha
store = CrossRunStore("proof_stats.parquet")
store.append(run_stats(manifest, report)) # one tidy row per run (re-append updates)
for cell in store.slice_pass_rate("base_checkpoint"):
print(cell.value, cell.pass_rate, (cell.ci_low, cell.ci_high)) # pooled pass-rate + Wilson CI
# Comparing A/B experiment arms: attribute each run to its cell, then slice by the arm.
for arm in plan.cells: # an ExperimentCell (see the matrix section)
store.append(run_stats(manifest, report, step_config=arm.step_config, labels=arm.labels))
store.slice_pass_rate("step_config") # fast vs quality
store.slice_pass_rate("label:caption_strategy") # florence vs wd14 (an upstream factor)
alpha = krippendorff_alpha([{"alice": 5, "bob": 4}, ...]) # inter-rater reliability
Pass-rate slices carry a Wilson confidence interval, so a lucky 3/3 cell reads
as far less certain than 300/400; the store is parquet, keyed by run_id + versions.
Recommendations
The gate says did it pass?; argus_proof.recommend says what to change, and
where — mapping weak metrics to the suite stage that owns the fix:
argus-proof recommend eval_report.json --store proof_stats.parquet
# [lens] unsafe outputs [unsafe_rate 0.04 vs 0.00]: filter/re-caption training data… (safety first)
# [forge] identity didn't transfer [identity 0.41 vs 0.60]: add/curate more identity images…
# [lens] prompt adherence low [clip_score 0.32 vs 0.50]: revisit the captioning strategy…
# [grid] prompt adherence low [clip_score 0.32 vs 0.50]: try different prompt/token combinations…
# [checkpoint] base_checkpoint outcome varies across runs: prefer base_checkpoint='sdxl_v2'…
from argus_proof.recommend import RecommendConfig, recommend
# keep the floors in lock-step with the CI gate so the two can't disagree:
cfg = RecommendConfig.from_acceptance(thresholds)
for rec in recommend(report, config=cfg, store=cross_run_store): # store optional
print(rec.stage, rec.metric, rec.value, rec.action)
Safety first, then: low identity/likeness/aesthetic → forge (training), low adherence → lens + grid, low diversity → grid, borderline → refine (HITL). With a cross-run store it also surfaces the best checkpoint / LoRA weight — but only when the evidence separates a clear winner (non-overlapping CIs), never on a tie.
A/B experiment matrix
Compare LoRAs across more than one axis at once. An ExperimentMatrix
(argus_proof.experiment) declares factors × levels and expands to a cell per
base_checkpoint × step_config, each a full grid (LoRA checkpoint × weight ×
prompt × seed). Cost is aggregated across cells and estimated before launch,
with a --max-gpu-hours guardrail that refuses an intractable matrix:
argus-proof experiment matrix.json --export ./curated_export --max-gpu-hours 40
# experiment exp: 4 cells, 32 runs, 96 images
# [exp-c00-sdxl-a-fast] 24 images (8 runs)
# ...
# est. 0.2 GPU-hours @ 6s/image
from argus_proof.experiment import ExperimentMatrix, StepConfig, expand_experiment
from argus_proof.grid import read_export_prompts
from argus_proof.models import SamplingParams
matrix = ExperimentMatrix(
base_checkpoints=["sdxl_a.safetensors", "sdxl_b.safetensors"],
step_configs=[StepConfig(name="quality", sampling=SamplingParams(...))],
lora_checkpoints=["e10.safetensors", "e20.safetensors"], # epoch sweep
lora_weights=[0.8, 1.0],
seeds=[1, 2, 3],
labels={"caption_strategy": "florence"}, # upstream factor, carried for cross-run slicing
)
plan = expand_experiment(matrix, read_export_prompts(export_dir), max_gpu_hours=40)
for cell in plan.cells:
... # generate + score each cell.plan; cell.labels feed the cross-run store
Upstream factors (caption strategy, source-image variation) are trained into
a LoRA, so proof can't vary them — it lists them in labels, which ride on every
cell. Feed a cell's step_config/labels into run_stats(...) and the cross-run
store compares the arms directly (slice_pass_rate("step_config"),
slice_pass_rate("label:caption_strategy")). For a matrix too large to
brute-force, optuna_search() (optional [opt] extra) does sample-efficient
search over the same factor levels.
FiftyOne exploration (optional)
A power-user surface over a scored run, complementing the /proof HITL view.
argus_proof.explore turns an EvalReport into a FiftyOne
dataset — every computed field attached to its image — so you can visualise
embeddings (UMAP/t-SNE) to spot mode collapse / clusters / outliers, run the
uniqueness/near-dup brain, and triage by tag (pip install "argus-proof[fiftyone]"):
# open the App, then fold the tags you added back into a new report on close
argus-proof explore eval_report.json --images ./run-1/images --umap \
--ingest reviewed.json --rater alice
from argus_proof.explore import to_fiftyone_dataset, compute_visualization, ingest_from_dataset
ds = to_fiftyone_dataset(report, {"img-1": "run-1/images/img-1.png", ...})
compute_visualization(ds) # UMAP embedding viz (needs umap-learn)
report = ingest_from_dataset(ds, report) # round-trip: fold tags back as ratings/reasons
The round-trip is tag-driven: in the App you add rating:<1-5> / reject:<code>
tags (these are the input channel — exported samples carry the scores as fields
and only a verdict/refined display tag, so a round-trip never re-ingests the
run's own auto-computed rejects). Ingest is authoritative — an image's tags are
its full decision, so a rating:5 with no reject: tag un-rejects it — and folds
through the same apply_hitl path a review uses (the verdict recomputes identically;
pass the original gate to keep non-default thresholds). The mapping
(sample_fields/sample_tags/ingest_tags) is dependency-free and unit-tested;
only the dataset/brain/App adapters need the extra, and explore.is_available()
guards them.
Develop
make install # venv + editable install with the "dev,server,cli" extras
make test
make lint
make pages # the hosted demo's static export (web/, needs Node 22)
CI / Release
- CI runs via the shared
argus-cireusable workflow. - Release publishes to PyPI (OIDC trusted publishing) and GHCR on
v*tags: the thin image and the-scorevariant. The-scorejob runsscripts/check_scorers.pyinside the image before pushing it. CI has no GPU, so that proves the scorers import, not that they run on CUDA. - CI's
score-importsjob installs the[score]extra (CPU torch) and runs the same check, so a pin that resolves but won't import fails CI instead of quietly dropping a metric. - Once PyPI has the version, a GitHub Release is created with that version's
CHANGELOG.mdsection as its notes. Add the section before you tag; the job fails without it. - Versioning is derived from git tags via
hatch-vcs— tagvX.Y.Zto cut a release.
This repo was scaffolded from argus-pkg-template.
Run copier update to pull template changes (CI, release, tooling).
Metadata
Release files for argus-proof 0.5.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| argus_proof-0.5.0.tar.gz | 271.4 kB | With provenance | Details for argus_proof-0.5.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| argus_proof-0.5.0-py3-none-any.whl | Python 3 | none | any | Details for argus_proof-0.5.0-py3-none-any.whl |
Total release size: 440.5 kB
Release files / argus_proof-0.5.0.tar.gz
| Download URL | argus_proof-0.5.0.tar.gz |
|---|---|
| Size | 271.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
22f9422e7fa1618a1b570b8bf3c853d372dbafc4d398f0ed66441d07ca7b69c0
|
|
BLAKE2b-256 checksum How to use checksums |
bc94d6569f90bad42281b54c18b6c9a4cc6351a66231e44f12198db365011cb2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.
Transparency logRelease files / argus_proof-0.5.0-py3-none-any.whl
| Download URL | argus_proof-0.5.0-py3-none-any.whl |
|---|---|
| Size | 169.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bd9f8bfeaa0a6ec0b66230507407f73d0be37a3dddcbe39d4976b3f0526f5057
|
|
BLAKE2b-256 checksum How to use checksums |
b77e4cdea4ab86f426232da523969f925ee769fbd9c316c2de2fe419ece8b2de
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.
Transparency logSecurity for argus-proof 0.5.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for argus_proof-0.5.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.