inaVAMOS: Voice And Music Open Segmenter

Speech (voice activity) and music detection for audio and video files, based on self-supervised (SSL) models from INA.

inaVAMOS splits an audio stream into homogeneous segments labelled speech, music, speech+music or other. It wraps two models published on the HuggingFace Hub:

Both models use the CNN and the first transformer layer of the music2vec SSL encoder, followed by an MLP classifier and Viterbi smoothing. They share the same encoder, which is computed only once when both detectors are used.

Output labels

The two detectors make independent decisions every 20 ms, which are combined into a single timeline:

other covers everything that is neither speech nor music: silence, background noise, applause, sound effects... When a single detector is run, other means "not detected" by that detector: with -d speech, it is non-speech and can include music; with -d music, it is non-music and can include speech.

The segments of each detector are also available separately (see Python usage, and the json and textgrid output formats).

Installation

pip install inaVAMOS

The development version can be installed with pip install git+https://github.com/ina-foss/inaVAMOS.git.

ffmpeg is recommended to read any audio or video format (apt install ffmpeg, brew install ffmpeg or conda install ffmpeg). Without it, files are read with librosa (wav, flac, ogg, mp3…).

To use a GPU, install the PyTorch build matching your CUDA version before installing inaVAMOS.

The models (about 80 MB) are downloaded from the HuggingFace Hub on first use and cached. Afterwards, HF_HUB_OFFLINE=1 can be set to work offline.

Command line usage

# Print the segmentation of a file as CSV
ina-vamos -i media.mp3

# Process several files and write one result per file in a directory
ina-vamos -i *.mp4 -o results/ -f textgrid

# Voice activity detection only, on a GPU, on the first 10 minutes
ina-vamos -i media.wav -d speech --device cuda --stop 600

Output example (CSV):

label,start,stop
music,0.000,23.900
other,23.900,32.060
...
speech,63.380,65.300
other,65.300,65.980
speech,65.980,68.200

Main options (see ina-vamos --help):

python -m inavamos is equivalent to ina-vamos.

Python usage

from inavamos import Segmenter

seg = Segmenter()  # or Segmenter(detectors=["speech"]), Segmenter(device="cuda")...

# Combined timeline: a list of (label, start, stop) tuples covering the whole file
for label, start, stop in seg("media.mp3"):
    print(label, start, stop)

Segmenter.process returns a Result object with more detailed outputs:

result = seg.process("media.mp3", start=60, stop=120)

result.segments("speech")      # [Segment(label='speech', start=63.38, stop=65.3), ...]
result.segments("music")       # segments where music is detected
result.timeline()              # combined timeline, as returned by seg("media.mp3")
result.probabilities["music"]  # frame-level probabilities (numpy array, one frame every 20 ms)
result.decisions["music"]      # frame-level boolean decisions after Viterbi smoothing
result.frame_duration          # 0.02

Signals can also be given directly, as 1-D numpy arrays or torch tensors:

import librosa

audio, sr = librosa.load("media.wav", sr=16000)
result = seg.process(audio, sampling_rate=sr)

Results can be exported with inavamos.export.export(result, "textgrid").

Options

Performance

Voice activity detection

InaGVAD test set (3h37 of French TV and radio), with the InaGVAD evaluation code (0.3 s collar):

Music detection

Frame-level F1 of the music class (no collar) on the test sets of Mirex2015, OpenBMAT and Seyerlehner. The global F1 is the average over the three datasets:

The music model of inaVAMOS was trained on the training subsets of these datasets (the evaluated test subsets were held out). inaSpeechSegmenter labels speech over music as speech only, which lowers its music recall on datasets with background music. pyannote does not detect music.

These results can be reproduced with the scripts of the benchmarks directory.

Speed

On a laptop GPU (RTX 3080), inaVAMOS processes about 200 times faster than real time (the 3h37 of InaGVAD in one minute). On a laptop CPU, it is about 30 times faster than real time (an hour of audio in about 2 minutes).

Development

pip install -e ".[test]"
pytest                    # all tests (downloads the models on first run)
pytest -m "not models"    # unit tests only, without the models

The models tests check, among others, that the output is identical to the reference code published on the model cards.

To release a new version: update __version__ in src/inavamos/__init__.py, then publish a GitHub release tagged v<version> (e.g. v0.2.0). The publish workflow runs the tests, builds the package and uploads it to PyPI.

License and citation

inaVAMOS and the models it uses are distributed under the Pantagruel Research-only License (French version, which prevails, and unofficial English translation). It restricts their use to non-commercial research and development activities, by research organisations and heritage institutions (libraries, museums, archives, audiovisual heritage). For any question, contact the Pantagruel Consortium at pantagruel-licence@univ-grenoble-alpes.fr.

If you use this tool or the models, please cite:

@inproceedings{pelloin2026lrec,
  author    = "Pelloin, Valentin and Bekkali, Lina and Dehak, Reda and Doukhan, David",
  year      = "2026",
  title     = "Data Selection Effects on Self-Supervised Learning of Audio Representations for French Audiovisual Broadcasts",
  booktitle = "Fifteenth International Conference on Language Resources and Evaluation (LREC 2026)",
  address   = "Palma, Mallorca, Spain",
  publisher = "European Language Resources Association",
  url       = {https://lrec.elra.info/lrec2026-main-802},
  doi       = {10.63317/4kdn23nttrh4},
}