Beyond Hit@1: Metric Choice, Implementation Channels, and a Standardized Evaluation Protocol for Zero-Shot Multi-Label Recognition

The problem. Naive evaluation of zero-shot multi-label recognition defaults to Hit@1 (top-1 accuracy), which is lenient for multi-label tasks and systematically flatters zero-shot models — even making zero-shot appear to beat supervised models. This project formalizes the metric-asymmetry finding (first surfaced in scb5-zeroshot) into a standardized evaluation protocol + open-source benchmark harness.

Status: 🚧 M0–M3 draft complete (2026-08-14). Evidence chain independently cross-validated (109/109 checks); manuscript drafted (paper/EvalAnchor.tex, 8pp, compiles clean). Pre-submission items: author lists, one bib verification, series-status re-grep.

What this research is about

For a multi-label image, a "correct" prediction means hitting any of the true labels — so top-1 accuracy (Hit@1) systematically under-punishes misses and overstates zero-shot performance. The seed finding: on TeacherBehavior (8-class multi-label), zero-shot Hit@1 = 85.56% (appears to beat the supervised linear probe's 77.10%), yet proper multi-label Sample-F1 = 59.94% vs the supervised 88–90%.

This project delivers three things:

1. The ranking-flip matrix — a systematic cross-model × cross-dataset × cross-metric study showing that metric choice changes which zero-shot model appears best.
2. The protocol — primary metrics (Sample-F1 + Macro-F1 + mAP), Hit@1 demoted to coarse screening with an explicit caveat, and a threshold-selection procedure.
3. The harness — open-source evaluation framework (splits + metric suite + one-command reproduction) on existing public datasets.

What this is NOT: a new method, theory, or dataset. "Hit@1 is lenient for multi-label" is known (Zhang & Zhou 2014, TKDE review; Madjarov et al. 2012). The contribution is the zero-shot × multi-label intersection and the systematic protocol — see AGENTS.md for the full novelty boundary.

Series relationship

This is the capstone of a 6-paper series on visual anchoring in the SCB5 classroom-behavior setting. It audits the series' own metric choices and provides the shared evaluation standard.

LA (llm-annotation)       IEEE Access ✅    concept: visual anchoring / AnchorProxy
  └─> VA (VisualAnchor)    EAAI (review)     class-level AnchorScore diagnostic
        └─> IA (InstanceAnchor) preprint      instance-level cross-backbone agreement
  └─> SV (SharedVisual)    preprint          mechanism: shared class-neighborhood geometry
scb5-zeroshot             Symmetry ✅         CAPE prompts, 5-backbone registry, metric-asymmetry seed
evalanchor (THIS)         🚧                capstone: evaluation protocol + harness

Planned repository structure

analysis/     — statistical analyses (CPU-only; ranking-flip matrix, protocol validation)
experiments/  — feature/metric extraction (GPU; reuses sibling caches)
results/      — committed evidence JSONs (single source of truth)
paper/        — manuscript (LaTeX)
todo.md       — executable task breakdown
AGENTS.md     — absorbed discipline + project-specific rules

Key results

The ranking flip (M1 v2, results/ranking_flip_report.txt; siglip2 on the corrected open_clip channel, see results/cache/README.md): the model that Hit@1 crowns "best" is overturned by every strict multi-label metric on 2 of 4 datasets, with coupled-bootstrap flip probability 1.000 and BH-FDR q ≤ 4e-4:

The flip is conditional, and the conditions are diagnostic: it occurs on the high-label-cardinality classroom data (card. 3.61, 94.6% multi-label) and the dense-annotation action data (24% empty frames), and not on the moderate-cardinality detection sets. On TeacherBehavior even Hit@1→Hit@3 flips the winner (openai vs siglip2). A loading-channel audit additionally shows the same architecture name ("siglip2") yields hit@1 0.046 vs 0.965 depending on whether weights come from HF transformers or open_clip (logit corr ≈ 0) — implementation choices act as hidden evaluation variables.

Honesty note: an earlier run of this matrix (before the channel fix) showed voc2007/coco flips that were artifacts of a collapsed siglip2 cache; they disappeared after re-extraction and a full-chain re-run (results/README.md).

Seed finding (from scb5-zeroshot, accepted at Symmetry): TeacherBehavior zero-shot Hit@1 85.56% vs Sample-F1 59.94% vs supervised 88–90%.

The protocol (M2, paper/EvalAnchor.tex, evidence results/protocol_validation.json)

• Primary suite: Sample-F1 + Macro-F1 + mAP, reported together; split verdicts reported as split.
• Hit@1 demoted to screening with a mandatory caveat: its leniency is a per-sample function of cardinality (on TeacherBehavior P(Hit@1) climbs 0.20→0.83 from 1→4 true labels while Sample-F1 plateaus at ~0.78). No predictive flip rule is claimed (n=4 datasets; the dataset-level Hit@1−Sample-F1 gap does NOT separate flipping from stable sets — documented).
• Threshold procedure: 5-fold out-of-fold per-class τ + argmax fallback. Validation: within 0.002–0.005 Sample-F1 of the in-sample oracle, vs 0.07–0.52 lost by fixed global τ.
• Retroactive series audit: scb5's CAPE prompt-set ranking (Set-C best under Hit@1, unanimous) inverts under Sample-F1 (Set-B on 4/5 backbones); VisualAnchor-style argmax-recall class difficulty is uncorrelated (point estimates) with protocol F1 on TeacherBehavior (ρ∈[−0.36,0.24], underpowered n=8) but preserved on VOC2007 (ρ=0.858) — diagnostics inherit the same conditionality.

The harness (M3, evalanchor/)

python -m evalanchor.run --dataset voc2007 --output report.json   # full suite + verdict

Audited metric suite (30 known-value + sklearn cross-checks), channel-pinned cache loading with override dir, coupled bootstrap + BH-FDR. P3.4 self-check: outputs bit-identical to the committed evidence on teacher/voc/coco.

Reproduce

python analysis/audit_metrics.py          # metric suite audit (30 checks)
python analysis/scope_inventory.py        # cache inventory -> results/scope_inventory.json
python analysis/ranking_flip.py           # headline matrix -> results/ranking_flip_report.json
python analysis/protocol_validation.py    # protocol validation + series audits
python -m evalanchor.run --dataset coco    # one-command evaluation

Citation

🚧 TBD (preprint; manuscript under preparation).

Data

Multi-label datasets (TeacherBehavior, VOC2007, COCO, MultiTHUMOS) are third-party public benchmarks; feature caches already exist in the sibling repos (InstanceAnchor/results/per_image/, scb5-zeroshot/data/feature_cache/). See AGENTS.md §"Existing assets to reuse".