VisionTrack

A from-scratch multi-object tracker, used as a controlled study of when the field's standard tricks actually help.

The tracker — an 8-state Kalman filter, an O(n³) Hungarian solver, and ByteTrack two-stage association — is implemented from first principles on NumPy, with no ML framework in the core. On top of it sits a reproducible experiment harness that measures, on real MOT17 with seed variance and paired significance tests, whether appearance and uncertainty-aware association actually improve tracking. Several of the answers are honest negatives — which is the point.

The tracker on a synthetic scene — boxes in, stable per-object IDs out. A real-MOT17 version can be rendered locally on your own copy of the dataset with python scripts/render_mot17_demo.py (the frames aren't redistributed here — MOT17 is under a non-commercial share-alike licence).

Abstract

Most tracking repositories are a thin wrapper over a detector plus a vendored tracker; every reported number is a single run on one configuration. VisionTrack inverts that: the estimation and association math is the deliverable (independently tested against SciPy and trackeval), and it is used to run a falsifiable study. We reproduce a from-scratch ByteTrack baseline on MOT17 public detections (HOTA/IDF1 verified against trackeval to within 1.4e-3), then ablate two common enhancements under seed variance and Wilcoxon significance:

• Appearance (RQ1) — a per-track re-ID cost helps association on MOT17 (ID switches 188 → 170) but the effect is small and not significant with a cheap descriptor.
• Uncertainty (RQ3) — folding calibrated Kalman uncertainty into the cost is null, and calibrating the filter to real motion is actively harmful under detector noise (ID switches 40 → 400+). The filter's apparent under-confidence turns out to be a robustness feature, not a bug.

Synthetic sanity check: seven objects over 120 frames — tracks #3 and #7 cross in the centre and keep their identities.

Research questions

The value is the same whichever way each result falls: a controlled study that shows a trick doesn't help is a contribution, not a failure.

Method

The from-scratch core (core/, NumPy only):

• Kalman filter — 8-state constant-velocity model in DeepSORT's xyah parametrization, height-scaled noise for scale-invariance, Joseph-form covariance update, Mahalanobis gating. Convergence-tested.
• Hungarian assignment — rectangular O(n³) Kuhn–Munkres with dual potentials; validated against scipy.optimize.linear_sum_assignment on 150 random matrices.
• ByteTrack two-stage association + a Tentative → Confirmed → Deleted lifecycle FSM.

The ablation surface (tracking/cost.py): the association cost is factored so each hypothesis is one weighted term with a hard gate deciding feasibility and the terms only ranking feasible pairs (DeepSORT-style fusion):

cost = w_iou·motion  ⊕  w_app·appearance  ⊕  w_unc·uncertainty      (gate: IoU + class + Mahalanobis)

At w_app = w_unc = 0 this is bit-identical to the plain 1 − IoU baseline, so every ablation toggles exactly one variable.

Rigor (eval/, experiments/):

• Metrics: from-scratch CLEAR-MOT + IDF1 + HOTA, cross-checked against trackeval end-to-end on real MOT17 (MOTA/IDF1 exact, HOTA within 1.4e-3).
• Seed variance + paired significance: every configuration is run over seeds; variants are compared paired (same sequences/seeds) with a paired bootstrap + Wilcoxon test and Cohen's d.
• Compute once: detections and appearance embeddings are cached to disk (5.2 MB detections; 6.3 MB colour-histogram or 67 MB deep-re-ID features for all of MOT17-train), so the raw ~5 GB of frames can be deleted and every experiment runs CPU-only in seconds.

Results

Baseline — from-scratch ByteTrack on real MOT17 (val-half, public detections)

Squarely in the public-detection neighbourhood (the famous ~76 MOTA ByteTrack uses a private YOLOX detector), with the expected detector ordering. scripts/xcheck_mot17_trackeval.py confirms our whole pipeline (preprocessing + metrics) matches trackeval on a real sequence.

RQ1 — appearance (MOT17 FRCNN), from-scratch histogram vs deep re-ID

Δ vs the w_app=0 motion-only baseline (HOTA 0.497 / IDF1 0.570 / AssA 0.587 / IDSW 188), same 7-sequence pairing:

Appearance's clearest effect is on ID switches. A deep re-ID embedder — a pretrained OSNet run through appearance/reid_onnx.py behind the same interface — roughly doubles the association gain over the hand-crafted histogram and cuts ID switches to 163 (−13%), earning that weight early (−14 IDSW already at w_app=0.15).

Pooling all three public detectors (DPM/FRCNN/SDP → 21 seq×detector units) for statistical power, the ID-switch and IDF1 reductions become significant (paired Wilcoxon p<0.05); association-quality (AssA/HOTA) stays marginal (p≈0.06). The instructive twist: appearance is completely inert on the weak DPM detector — its poorly-localized boxes yield mis-framed crops, so the re-ID embeddings carry no identity signal. Detection/crop quality gates whether appearance helps at all, the opposite of the "weak detector needs it most" intuition, and it doesn't stratify by crowd density or occlusion. Details →

RQ2 — learned motion residual (helps prediction, hurts tracking)

A from-scratch NumPy MLP (hand-written back-prop + Adam — no framework) predicts a correction to the constant-velocity Kalman mean, trained on GT trajectories.

Open-loop, the residual helps exactly where motion is non-linear — but wired into the tracker it hurts identity on both (DanceTrack HOTA −0.043, +29 IDSW; MOT17 HOTA −0.013, +12 IDSW). Trained on clean GT but fed the tracker's noisy estimates, its correction mis-fires and perturbs the association gate. A better predictor that makes a worse tracker. Details →

RQ3 — calibrated uncertainty (the interesting negative)

Stepping the Kalman filter along real MOT17 GT, the innovation χ² is 0.15 where a calibrated filter would give 4.0 — it is ~25× under-confident, so its 95% gate captures 100% of innovations and never rejects.

Consequences, measured:

• Folding uncertainty into the cost (w_unc) is null (the gate is inert, the signal near-constant).
• Calibrating the filter (kf_noise_scale=0.19) is catastrophic under detector noise: HOTA −0.205, ID switches 40 → 400+ (both p<0.05). A tight gate tuned to clean motion rejects noisy-but-correct detections and tracks fragment.

The loose gate is a robustness feature, not a bug. This unifies with the ablation finding that disabling the gate barely changes clean-data results. Details →

Throughput

Real-time on CPU (single-threaded): 2841 FPS at 4 objects down to 294 FPS at 64 — comfortably ≥30 FPS across the range (benchmarks/bench_tracker.py). The per-frame Kalman predict and Mahalanobis gating are batched across the whole track set (~1.4–1.5× over the per-track loop, numerically identical).

Install & use

pip install visiontrack-mot            # core: NumPy only
pip install 'visiontrack-mot[video]'   # + run on your own video files

from visiontrack import ByteTracker, TrackerConfig
tracker = ByteTracker(TrackerConfig())
for frame_detections in stream:            # list[Detection]
    observations = tracker.update(frame_detections)

Track a real video — or a live camera — from the command line (needs a YOLOX ONNX model, see docs/VIDEO.md):

visiontrack track input.mp4 out.mp4 --model models/yolox_nano.onnx
visiontrack webcam --model models/yolox_nano.onnx        # live camera, real time
visiontrack lab-evidence BUNDLE VARIANT EVENT_ID --frame 42=frame.png --crop 120,80,420,520
visiontrack lab-decision BUNDLE --status accepted --variant baseline \
  --rationale "Meets the registered criteria." --author local-reviewer \
  --decided-at 2026-09-21T08:30:00Z
visiontrack lab-decision-import BUNDLE ./Downloads/decision.json  # preview only
visiontrack lab-decision-import BUNDLE ./Downloads/decision.json --write
visiontrack lab-demo ./lab-onboarding                  # preview only
visiontrack lab-demo ./lab-onboarding --write          # create verified bundle
visiontrack lab-playback BUNDLE EVENT_ID --variant appearance  # metadata only
visiontrack lab-playback BUNDLE EVENT_ID --variant appearance --json > playback.json

lab-evidence previews only by default; add --write after checking its crop, privacy class, and limits. lab-decision likewise previews the exact choice and verified lineage before --write creates decision.json. CLI subcommands: demo, eval, ablate, track, webcam, lab-evidence, lab-decision, lab-decision-import, lab-demo, lab-playback. Full public API: docs/API.md · packaging/release: docs/RELEASE.md.

The private, client-side Reliability Lab at /lab can inspect an explicitly selected decision.json. It verifies the decision fingerprint and full report lineage in browser memory, then displays the audit record without editing it. After importing a real report, the Lab can also draft a decision from blank human inputs, preview its complete fingerprinted record, and explicitly download it. The browser never writes into the experiment bundle.

lab-decision-import closes that deliberate boundary locally. It verifies the canonical downloaded bytes, decision fingerprint, and exact experiment/source/comparison/report/variant lineage before showing a read-only preview. Only --write enters the record through immutable bundle storage; identical imports are safe and conflicting decisions are refused.

Python and browser decision records are locked to shared conformance vectors. Run npm run test:decision-contract to verify exact fingerprints, canonical JSON, Unicode handling, UTC forms, and downloaded file bytes.

Python and browser playback records are likewise locked to shared known-timing and unknown-timing vectors. npm run test:playback-contract verifies exact fingerprints and serialized bytes while rejecting remote paths, incomplete lanes, timing drift, and ambiguous missing-frame metadata.

lab-demo provides a dataset-free Reliability Lab walkthrough. It previews a fixed synthetic comparison without writing, then --write creates a complete verified bundle and prints the report.json path to import at /lab. The output root must be new and is never reused or overwritten.

lab-playback is metadata-only. It revalidates the sealed report and selected failure window, then prints synchronized lane availability, privacy counts, and complete fingerprints without displaying pixels or writing to the bundle. Add --json to emit only the canonical playback record to standard output; shell redirection is the deliberate way to save it for later browser import. The Reliability Lab accepts that local playback.json only after a verified report.json is active, rechecks its complete lineage and fingerprint, and renders synchronized timing, privacy, availability, and missing slots without resolving media paths or reading pixels. A researcher may then explicitly select each lane's declared PNG set; the browser verifies exact filenames, SHA-256 hashes, and PNG bounds before offering per-frame reveal controls. Both lanes share bounded manual and timed transport controls; automatic playback respects reduced-motion preferences and never reveals concealed pixels. CI also executes one checked-in synthetic browser workflow through the real report parser, playback validator, both lane verifiers, privacy boundary, and transport helpers.

The first data-driven results slice is available at /benchmark/explorer. It strictly validates a versioned checked-in report before rendering the dataset, protocol, provenance, published means/spread, paired deltas, and Wilcoxon significance markers. It remains read-only, selects no winner, and links to the full generated benchmark rather than replacing the existing report pages.

No install at all — run the tracker live in your browser at visiontrack.hulage.in/live: the same from-scratch association (Kalman + Hungarian + ByteTrack), ported to JavaScript and running on-device over your webcam or a sample clip.

Benchmark a set of trackers into one report (leaderboard + paired significance + ID-switch error taxonomy) — make benchmark, browse the results hub, or open the full generated synthetic report.

Reproduce

No install? Run the synthetic study in your browser: — clones, installs the harness extra, and reproduces the seed-varied, significance-tested synthetic tables (a couple of minutes, CPU-only). Notebook: notebooks/reproduce.ipynb.

make install                 # editable install with all extras
make test                    # full test suite
make reproduce-synth         # synthetic harness + RQ3 probe — NO dataset needed

Full real-data reproduction (after a one-time cache build — see docs/PHASE0.md):

python data/cache/precompute.py            --data-root ~/MOT17 --detector FRCNN --out data/cache/mot17
python data/cache/precompute_embeddings.py --data-root ~/MOT17 --detector FRCNN --cache-dir data/cache/mot17
make reproduce               # regenerates every table and figure above

Every run is pinned by a config hash; bootstrap resampling and synthetic scenes are seeded. Each phase has a runbook in docs/ (PHASE0.md … PHASE5.md).

Engineering

• Zero ML-framework dependency in the core — just NumPy. Heavy/optional deps (scipy, pandas, matplotlib, pillow, onnxruntime, torch) are isolated in extras ([experiments], [appearance], [onnx]) and lazily imported; nothing in core/ imports them.
• 560 tests: unit, property (Hungarian vs SciPy), convergence (Kalman), metric cross-checks (HOTA/IDF1 vs trackeval), video/webcam pipelines, the benchmarking tool, product-surface contracts, and end-to-end integration with a MOTA floor.
• CI on Python 3.10/3.11/3.12 + ruff.

src/visiontrack/
  core/        geometry · kalman · assignment                       ← from-scratch math
  detection/   base · synthetic · noise · mot_loader · dancetrack_loader · yolox_onnx · cached
  appearance/  embedder (colour-hist) · reid_onnx · gallery         (RQ1)
  tracking/    tracker · track (FSM) · cost (ablation surface) · config · presets (the zoo)
    motion/    oc (OC-SORT) · gmc (camera-motion, RQ4) · residual (learned motion, RQ2)
  eval/        mot (CLEAR-MOT) · hota (HOTA/IDF1) · mot17 · stats · calibration (RQ3)
  datasets/    splits (frozen) · cache (detections + embeddings)
  cli · video · viz/draw                                            ← CLI, video pipeline, drawing
experiments/   run_matrix · benchmark · error_taxonomy · tracker_zoo · profile_fps · *_study · configs/
data/cache/    precompute · precompute_embeddings · precompute_dancetrack · precompute_gmc
scripts/       xcheck_mot17_trackeval.py · render_video_demo.py · make_og_image.py

Limitations & honest negatives

• Public-detection, train/val split — reproducible and self-contained, not test-server leaderboard numbers (deliberate).
• RQ1's effect is real but small on MOT17 — a deep re-ID embedder (OSNet-x0.25, MSMT17) roughly doubles the association gain over the from-scratch histogram; pooling all three detectors (21 units) makes the ID-switch/IDF1 reduction significant, but association quality (AssA/HOTA) stays marginal and the benefit is capped by public-detection crop quality. A controlled synthetic probe (dialing inter-object appearance similarity) confirms the benefit grows with object distinctness and is significant on AssA/IDF1 — but never flips to harmful. And the decisive test — real DanceTrack (near-identical dancers, non-linear motion, the predicted "appearance hurts" case) — refutes the hypothesis: deep re-ID still significantly cuts ID switches there (217→202, p<0.05). Across all three probes a gated appearance cost is robustly beneficial-or-neutral. The "appearance hurts on DanceTrack" failure is a property of appearance-vetoing designs, not of uniform appearance itself — our cost only lets appearance rank within the motion gate, never veto a feasible match. Re-ID weights carry a non-commercial dataset licence and are not committed (regenerate from your own download).
• RQ2 (learned motion residual) is answered — an honest negative. A from-scratch NumPy MLP (hand-written back-prop, no framework) lowers open-loop next-centre error on DanceTrack (−12.5%) but not MOT17 (near-linear, ~1px CV error → +21.5% worse); crucially, inside the tracker it hurts on both (train-on-GT / infer-on-noisy-estimates distribution shift + it perturbs the association gate). Trained on GT trajectories; weights gitignored. Details → docs/PHASE4.md
• The cross-dataset RQ1 test is done on DanceTrack, with a scope caveat — detections are oracle-perturbed GT (DanceTrack ships none), which gives clean crops that favour appearance; a poor real detector would weaken it (cf. DPM). The cost is weighted-and-gated, not pure-appearance. So the refutation is scoped: a gated ranking appearance cost does not hurt, even on DanceTrack. SportsMOT (RQ2 maneuver data) is still deferred.

Interactive demo

A self-contained, pre-baked demo that makes the RQ1 result tangible: the same synthetic scene tracked two ways — motion-only vs motion + appearance — with the running ID-switch count side by side. Boxes are coloured by track ID, so a colour flip on a moving object is a switch.

• Try it live: visiontrack.hulage.in/demo — no install, runs in the browser (deployed from viz/webdemo/ via Vercel; project home at visiontrack.hulage.in).
• Build it locally: make demo → open viz/webdemo/index.html in any browser (no server, no dataset, no toolchain — inference is pre-baked from the NumPy tracker on a synthetic scene).
• On the selected scene, appearance cuts ID switches 37 → 29 (−22%) and lifts IDF1 — the study result, watchable frame by frame.

Two more live surfaces, no install: /live runs the tracker on your own webcam in the browser (the association ported to JavaScript, on-device), and /video shows the identical Python pipeline on real, crowded street footage.

New to multi-object tracking, or want to point someone non-technical at this? /teaching explains the project at two depths on one page: a builder track (install, the public API, what runs inside update()) and a plain-English track (what tracking is, why it's hard, a glossary) with no code.

Write-up

A narrative walk-through of the study — why appearance refuses to hurt (even on DanceTrack), how a better motion predictor made a worse tracker, and why the filter's loose gate is a feature: visiontrack.hulage.in/writeup. The honest negatives, explained in plain prose rather than tables.

Study guide

New to the concepts? Two self-paced, interactive learning files (single self-contained HTML, progress checkboxes saved in your browser — open in any browser):

• learning/LEARNING_PATH.html — this project, topic by topic: Kalman → Hungarian → ByteTrack → metrics → the three research questions → where to take it next, with every concept linked to the file it lives in.
• learning/CV_ROADMAP.html — the whole field, in order: a junior → mid → senior → research computer-vision roadmap (foundations → deep learning → detection/segmentation/tracking → transformers/generative/3D → production → doing research), with what-to-build and canonical resources at each stage.

License

MIT