LiveKit WakeWord Bench

Generate a benchmark or training dataset for your wake word, with labeled speech, near-miss phrases and realistic audio conditions. Use the benchmark to measure any detector against the same test set.

Quick start

Requires Python 3.12+ and the LiveKit CLI. Generation uses LiveKit Inference credits.

pip install livekit-wakeword-bench
lk cloud auth
lk app env -w

# Generate train, validation and test splits for your phrase.
lkwb generate "hey jarvis" --size small -o out/hey_jarvis
lkwb verify out/hey_jarvis

For a benchmark only, add --split test; for training data only, add --split train. Use --size tiny for a smoke test. Choose --plan to match your account (build by default). TTS capacity is shared with production agents: leave room for them with a lower --set tts.concurrency=N.

For unusual names, run lkwb init "your wake word" to create a pronunciation config. See Generate for sizes, pronunciation, custom settings and outputs.

What gets generated

The generator combines phoneme-guided TTS with recorded noise, room responses and speech. Each clip has 16 kHz audio, a label, phrase timing and metadata describing its voice and audio conditions.

Only positives have label 1; all negatives have label 0. Category counts depend on the size and split: training presets contain positives, adversarial negatives and synthesized speech. See the exact sample counts.

Positive and adversarial phrases are isolated or placed before/after surrounding speech (p_carrier=0.35 by default). Near misses are generated from phonetic similarity; add known confusions with hard_negatives.

Which conditions are covered

Validation and test clips sample combinations of these conditions. They are not an exhaustive test of every combination.

For example, the same wake word can be detected alone and missed with a competing speaker:

Training clips are stored clean by default so you can add fresh effects while training. Test voices, noise, rooms and recorded speakers are held apart from training. Never train on the test split or test sources. Some source corpora restrict commercial use; see sources and licenses.

Tune the recipe with YAML or --set key=value. For example, generate training data with more competing speech:

lkwb generate "hey jarvis" --size small --split train -o out/train_talkers \
    --set splits.train.augment=true \
    --set augment.p_interferer=0.6 \
    --set augment.clean_weight=0.2

Run lkwb knobs for all settings. TTS is cached, so changing only augmentation reuses synthesized speech.

How models are evaluated

The headline score is recall at the lowest tested threshold with no more than 0.5 false positives per hour (FPPH). Higher is better; change the target with --score-fpph.

Reports break results down by noise, competing speech, room, device effects, voice, accent and phrasing, with 95% confidence intervals. Breakdowns use --threshold (default 0.5), which may differ from the headline score's threshold.

Synthesized positives do not establish recall on real users. Accent labels come from TTS metadata. Treat confidence intervals wider than about 0.2 as weak evidence; rare false alarms need many hours of negative audio.

Evaluate on your dataset

For a livekit-wakeword v2 streaming model:

pip install livekit-wakeword
lkwb bench path/to/your_model.onnx --data out/hey_jarvis --limit 1000

Open results/report.html for charts and playable failures. Remove --limit for a full run before reporting results. Compare models using the same dataset, split, filters and limit.

Other models use the custom detector interface. Existing datasets need no LiveKit credentials. Omitting --data uses the public Hey LiveKit dataset, which is an example benchmark for that phrase.

Documentation

For coding agents, see AGENTS.md.