yavih 1.0.0

A growing library of video, audio and audio-visual benchmarks.
One harness, one protocol — and your coding agent does the wiring.

yavih (Yet Another Video Harness) evaluates frozen video, audio and audio-visual encoders. You bring a model; yavih brings the benchmarks, the data recipes, the probes and the guarantees that your numbers mean the same thing as everyone else's.

Get started in one sentence

pip install yavih          # into the Python environment that runs your model
yavih skill install        # teaches your coding agent (Claude Code, Codex, Cursor, …) how to use yavih

Then tell your agent:

Benchmark my model with yavih.

That's it. The agent picks the default suite, prepares each benchmark's data, writes the small adapter that feeds your model, checks it, extracts features, trains the standard probes and hands you a report. What it cannot do on its own — a licence you have to accept, a download only a person can make — it lists for you at the end, while the rest runs.

Why yavih

A growing library

You rarely choose benchmarks one by one. A suite is a named set: default is what "benchmark my model" runs, minimal is a quick smoke pass, and yavih suite new makes your own.

yavih suite list               # default, minimal, and any suite you define
yavih bench list               # every benchmark: task, size, how to get the data, what's ready on your machine

All 11 benchmarks → the index

Each benchmark is one self-contained folder with its own page: the paper, how to get the data, and exactly how yavih runs it.

Numbers you can trust

A benchmark result is only useful if someone else can get the same number the same way. yavih freezes everything that decides what a number means and leaves everything else to you:

• Frozen by yavih: which clips, which labels, which raw files your model receives, which probe is trained on top, how it is trained, and which metric is reported — with a floor (what a model that ignores its input would score) next to every result.
• Yours to choose: the model, how it reads the media (frames, crops, audio front end), its precision and batch size. All of it is recorded with every number, none of it changes what the number is compared against.

That frozen protocol is summarised in one short hash, the eval_hash. Same hash ⇔ same data, same protocol, same probe. The report only ever puts numbers with the same hash in one column; change a clip or a metric and you get a new hash, never a quietly different number.

Add your benchmark

Tell your agent:

Add Diving48 to yavih.

It follows contributing a benchmark step by step — scaffold the folder, freeze the manifest from the public release, write the data recipe and the tests, render the header — then runs a synthetic trial and a fresh-agent trial run before opening the pull request.

How it works

prepare ──► extract ──► validate ──► probe ──► report
 data         your model   yavih checks   standard     comparable
 recipe       (a bridge)   the features   probe        tables

Your model connects through a bridge: a small Python file, written by your agent from a template, that turns a benchmark's raw files into features. It runs in your model's own environment — yavih installs without touching your torch. yavih checks the features, trains the same probe every model gets, and reports with floors and confidence intervals. Everything runs on a laptop, a workstation or as job arrays on any cluster.

License

yavih is released under the MIT License. Benchmark datasets are not part of yavih and keep their own licences and terms of use (each benchmark's page says where they come from).