omnivad 0.1.0

OmniVAD — Cross-platform Voice Activity Detection and Audio Event Detection (based on FireRedVAD)

OmniVAD

Cross-platform toolkit for FireRedVAD — SOTA voice activity detection and audio event detection.

Three models, one toolkit, runs everywhere:

Model	What it does	Output
VAD	Speech detection (non-stream)	Speech timestamps
Stream-VAD	Real-time speech detection (frame-by-frame)	Per-frame speech probability
AED	Audio event detection (non-stream)	Speech / Singing / Music timestamps

All models are based on DFSMN architecture, ~2.2MB each (~588K params), support 100+ languages.

Packages

C/C++ Native Library (native/)

Unified C API with ncnn backend. Single header, single library.

#include "omnivad.h"

// Stream VAD — real-time, 10ms per frame
OmniVadHandle vad = omni_vad_stream_create(param, bin, means, istd, 0.5f);
omni_vad_stream_process(vad, pcm_160_samples, 160, &result);
// result.confidence = 0.95, result.is_speech = true

// Non-stream VAD — whole audio to segments
OmniVadNonStreamHandle vad = omni_vad_nonstream_create(param, bin, means, istd);
omni_vad_nonstream_process(vad, audio, num_samples, &config, &segments, &count);
// segments[0] = { start: 0.44, end: 1.82 }

// AED — speech + singing + music detection
OmniAedNonStreamHandle aed = omni_aed_nonstream_create(param, bin, means, istd);
omni_aed_nonstream_process(aed, audio, num_samples, &config, &segments, &count);
// segments[0] = { start: 0.09, end: 12.32, cls: OMNI_AED_MUSIC }

Build:

# Prerequisites: cmake, ncnn (brew install ncnn)
cd native
cmake -B build && cmake --build build -j$(nproc)

# Test
./build/test_nonstream_vad model.param model.bin cmvn_means.bin cmvn_istd.bin audio.wav

Platforms: macOS (arm64/x86_64), Linux (x86_64/aarch64), Windows (x86_64), Android (armeabi-v7a/arm64-v8a)

TypeScript/JavaScript (packages/omnivad/)

Works in both browser and Node.js via ONNX Runtime Web.

import { OmniVAD, OmniStreamVAD, OmniAED } from 'omnivad';

// Non-stream VAD
const vad = await OmniVAD.create({ modelPath: 'fireredvad_vad.onnx' });
const result = await vad.detect(audioFloat32Array);
// { duration: 2.32, timestamps: [[0.44, 1.82]] }

// Stream VAD — real-time, feed 160 samples (10ms) at a time
const svad = await OmniStreamVAD.create({ modelPath: 'fireredvad_stream_vad_with_cache.onnx' });
const frame = await svad.processFrame(pcm160);
// { confidence: 0.95, isSpeech: true, frameOffset: 42 }
svad.setMode(2); // 0=very permissive, 1=permissive, 2=aggressive, 3=very aggressive

// AED — speech + singing + music
const aed = await OmniAED.create({ modelPath: 'fireredvad_aed.onnx' });
const events = await aed.detect(audioFloat32Array);
// { duration: 22.0, events: { speech: [...], singing: [...], music: [...] }, ratios: { ... } }

Build:

cd packages/omnivad
pnpm install && pnpm build
# Output: dist/index.js (ESM, 49KB) + dist/index.cjs (CJS) + dist/index.d.ts

Key implementation details:

• Pure TypeScript fbank (80-dim mel filterbank, Povey window, pre-emphasis) — no native dependencies
• CMVN data baked in (640 bytes) — no external config files needed
• Post-processing matches Python reference exactly (4-state machine, segment merging, splitting)

Audio Pipeline

16kHz PCM → Fbank (80-dim, 25ms window, 10ms shift) → CMVN → DFSMN → Sigmoid → Post-processing → Segments
                     Povey window                        μ/σ    ~2.2MB   [0,1]    4-state machine
                     pre-emphasis 0.97                                            merge/split/extend

Model Files

Models must be downloaded separately from ModelScope or HuggingFace.

# Download PyTorch models + export to ONNX
pip install fireredvad
python -m fireredvad.bin.export_onnx --all

# Or download pre-exported ONNX models
# fireredvad_vad.onnx              — Non-stream VAD (2.3MB)
# fireredvad_aed.onnx              — Non-stream AED (2.3MB)
# fireredvad_stream_vad_with_cache.onnx — Stream VAD (2.2MB)

# For C/ncnn: convert ONNX → ncnn with pnnx
pip install pnnx
pnnx fireredvad_vad.onnx "inputshape=[1,100,80]"

Testing

# 1. Generate Python reference data (requires fireredvad Python package)
python tests/generate_reference.py

# 2. Run C vs Python accuracy test (requires ncnn models)
python tests/test_timestamp_accuracy.py

# 3. Run on any audio → TextGrid + RTF benchmark
python tests/vad_to_textgrid.py audio.wav

Accuracy (C/ncnn vs Python, 5 audio files × 3 models):

Model	Timestamp Δ	Probability Δ	Status
VAD	≤ 0.020s	≤ 0.001	Exact match
AED (singing/music)	≤ 0.010s	≤ 0.013	Exact match
AED (speech)	≤ 0.030s	≤ 0.015	Match (ncnn fp16 edge cases on event.wav)
Stream-VAD (detect_full)	≤ 0.010s	≤ 0.001	Exact match

Project Structure

omnivad/
├── native/                          # C/C++ library (ncnn backend)
│   ├── include/omnivad.h         #   Unified C API header
│   ├── src/omnivad.cpp           #   Implementation (~1100 lines)
│   ├── frontend/                    #   Fbank/FFT/WAV (from FireRedVAD)
│   ├── test/                        #   4 test programs
│   └── CMakeLists.txt
├── packages/omnivad/             # TypeScript npm package
│   ├── src/
│   │   ├── vad.ts                   #   OmniVAD (non-stream)
│   │   ├── stream-vad.ts            #   OmniStreamVAD (real-time)
│   │   ├── aed.ts                   #   OmniAED (3-class)
│   │   ├── fbank.ts                 #   80-dim mel fbank (pure TS)
│   │   ├── fft.ts                   #   Radix-2 FFT
│   │   ├── cmvn.ts                  #   CMVN with baked-in data
│   │   └── post-process.ts          #   4-state machine post-processing
│   ├── package.json
│   └── tsconfig.json
└── tests/                           # Test suite
    ├── generate_reference.py        #   Generate Python reference data
    ├── test_timestamp_accuracy.py   #   Strict C vs Python comparison
    ├── vad_to_textgrid.py           #   Audio → TextGrid + RTF benchmark
    └── data/                        #   5 test audio files + reference JSON

Performance

RTF (Real-Time Factor) on Apple M-series, lower = faster:

Model	RTF	Speed
VAD	~0.003	~330x real-time
Stream-VAD	~0.002	~500x real-time
AED	~0.002	~500x real-time

Origin & Attribution

OmniVAD is a cross-platform deployment toolkit built on top of FireRedVAD, developed by Xiaohongshu (小红书). FireRedVAD provides high-quality Voice Activity Detection models and a lightweight Audio Event Detection model that can distinguish speech, singing, and music.

Original paper: FireRedVAD (arXiv:2603.10420)

What FireRedVAD provides: DFSMN-based models (~2.2MB each), Python inference code, PyTorch training, strong VAD benchmark results (FLEURS-VAD-102 F1: 97.57%).

What OmniVAD adds: Unified C API (ncnn backend) for native deployment, TypeScript/JavaScript npm package (ONNX Runtime Web) for browser and Node.js, cross-platform build system, comprehensive test suite with accuracy validation.

License

Apache-2.0 — same as the upstream FireRedVAD.

Credits

• FireRedVAD — Kaituo Xu, Wenpeng Li, Kai Huang, Kun Liu (Xiaohongshu)
• ncnn — Tencent
• ONNX Runtime — Microsoft