seprq 0.4.8

Université de Toulon · LIS · CNRS · ILLS · pyannoteAI

📄 Paper  ·  🌐 Website  ·  🤗 Hugging Face  ·  💻 GitHub

🗣️ This framework is intended to provide an easy, ready-to-use and strong SepRQ feature extractor for 🍸 cocktail-party 🍸 and multi-talker conversational scenarios 🔀 👥, while also facilitating the use of other feature extractors from the literature. We also release a BEST-RQ model trained at 50 Hz as a valuable contribution to the community.

SepRQ features are pulled from 🤗 SevKod/SepRQ; a call returns the 12 Conformer layer features usable for training or cocktail-party representation analysis. Requires PyTorch and upstream speechbrain>=1.0.3; runs on CPU or GPU.

👉 Please visit https://sevkod.github.io/SepRQ/ for the full usage and demo.

Install

pip install seprq

Usage (frozen ❄️ or fine-tuned 🔥)

from seprq import SepRQEncoder

wav = "hf.co/SepRQ/SepRQ/blob/main/ex/mix.wav"
# wav = torchaudio.load("path/to/audio.wav")

# "SepRQ" — see "Available models" for every name
encoder = SepRQEncoder("SepRQ", streams=2)

# ❄️ frozen feature extraction
encoder.requires_grad_(False).eval()  # or True for 🔥 fine-tuning

features = encoder(wav)

It accepts a file path (any format/rate — decoded, mono, resampled to 16 kHz) or a waveform tensor [num_samples], [batch, num_samples] or [batch, channel, num_samples].

The released SepRQ encoder is the 2-speaker model (streams=2).

Available models

The SepRQ / BEST-RQ weights come from the Hub (private — authenticate once with huggingface-cli login or HF_TOKEN); the HuBERT / WavLM baselines are fetched from the torchaudio pipelines, nothing extra to host.

Run a task

SUPERB separation, diarization, and enhancement, plus target-speaker extraction, personalized speech extraction, personalized VAD, target-speaker ASR, DIHARD 3 diarization (local segmentation), and WSJ0-2mix / WSJ0-3mix separation. The head is chosen from the upstream. Published for HuBERT_BASE, WavLM_BASE, and WavLM_BASE_PLUS. SepRQ downstream heads are not on the public Hub yet. Target-speaker tasks take an enrollment utterance.

from seprq import SepRQPipeline

# Separation
pipe = SepRQPipeline("separation", upstream="SepRQ", streams=2)

# Input mixture - [num_samples]
mixture = "hf.co/SepRQ/SepRQ/blob/main/ex/SS/mix1.wav"

save_to = "./output"

sources = pipe(mixture, save_to=save_to)
# [num_speakers, num_samples]

# ./output/source1.wav
# ./output/source2.wav

# Diarization
pipe = SepRQPipeline("diarization", upstream="SepRQ", streams=2)

# Input mixture - [num_samples]
mixture = "hf.co/SepRQ/SepRQ/blob/main/ex/SD/mix1.wav"

save_to = "./output"

activations = pipe(mixture, save_to=save_to)
# [time, num_speakers]

# ./output/activations.rttm

# Enhancement
pipe = SepRQPipeline("enhancement", upstream="SepRQ", streams=2)

# Input mixture - [num_samples]
mixture = "hf.co/SepRQ/SepRQ/blob/main/ex/SE/mix1.wav"

save_to = "./output"

clean = pipe(mixture, save_to=save_to)
# [num_samples]

# ./output/cleaned.wav

# Target speaker extraction
pipe = SepRQPipeline("target-speaker-extraction",
                     upstream="SepRQ",
                     streams=2)

# Input mixture - [num_samples]
mixture = "hf.co/SepRQ/SepRQ/blob/main/ex/TSE/mix1.wav"
enrollment = "hf.co/SepRQ/SepRQ/blob/main/ex/TSE/enroll1.wav"

save_to = "./output"

target = pipe(mixture, enrollment=enrollment, save_to=save_to)
# [num_samples]

# ./output/extracted_audio.wav

# Personalized speech extraction
pipe = SepRQPipeline("personalized-extraction",
                     upstream="SepRQ",
                     streams=2)

# Input mixture - [num_samples]
mixture = "hf.co/SepRQ/SepRQ/blob/main/ex/PSE/mix1.wav"
enrollment = "hf.co/SepRQ/SepRQ/blob/main/ex/PSE/enroll1.wav"

save_to = "./output"

target = pipe(mixture, enrollment=enrollment, save_to=save_to)
# [num_samples]

# ./output/extracted_audio.wav

# Personalized VAD
pipe = SepRQPipeline("personalized-vad", upstream="SepRQ", streams=2)

# Input mixture - [num_samples]
mixture = "hf.co/SepRQ/SepRQ/blob/main/ex/PVAD/mix1.wav"
enrollment = "hf.co/SepRQ/SepRQ/blob/main/ex/PVAD/enroll1.wav"

save_to = "./output"

activations = pipe(mixture, enrollment=enrollment, save_to=save_to)
# [time]

# ./output/activations.rttm

# Target-speaker ASR
pipe = SepRQPipeline("target-speaker-asr",
                     upstream="SepRQ",
                     streams=2)

# Input mixture - [num_samples]
mixture = "hf.co/SepRQ/SepRQ/blob/main/ex/TS-ASR/mix1.wav"
enrollment = "hf.co/SepRQ/SepRQ/blob/main/ex/TS-ASR/enroll1.wav"

save_to = "./output"

text = pipe(mixture, enrollment=enrollment, save_to=save_to)

# ./output/extracted_transcript.txt

# DIHARD 3 diarization
pipe = SepRQPipeline("diarization-dihard3",
                     upstream="SepRQ",
                     streams=2)

# Input mixture - [num_samples]
mixture = "hf.co/SepRQ/SepRQ/blob/main/ex/DH3/mix1.wav"

save_to = "./output"

activations = pipe(mixture, save_to=save_to)
# [time, num_speakers]

# ./output/activations.rttm
# Local segmentation only (EEND).
# Not intended for long file recordings without clustering.

# WSJ0-2mix separation
pipe = SepRQPipeline("separation-wsj0-2mix",
                     upstream="SepRQ",
                     streams=2)

# Input mixture - [num_samples]
mixture = "hf.co/SepRQ/SepRQ/blob/main/ex/WSJ2/mix1.wav"

save_to = "./output"

sources = pipe(mixture, save_to=save_to)
# [num_speakers, num_samples]

# ./output/source1.wav
# ./output/source2.wav

# WSJ0-3mix separation
pipe = SepRQPipeline("separation-wsj0-3mix",
                     upstream="SepRQ",
                     streams=3)

# Input mixture - [num_samples]
mixture = "hf.co/SepRQ/SepRQ/blob/main/ex/WSJ3/mix1.wav"

save_to = "./output"

sources = pipe(mixture, save_to=save_to)
# [num_speakers, num_samples]

# ./output/source1.wav
# ./output/source2.wav
# ./output/source3.wav

🔀 SepRQ — separation as the pretext task

SepRQ swaps masked prediction for pseudo source separation. It predicts one stream of discrete units per speaker in the mixture, at several temporal resolutions.

Fig. 1. SepRQ performs pseudo source separation in the discrete space. Random vector quantizers (RVQs) give speaker-specific discrete labels at several resolutions.

• Pseudo source separation — each separation head predicts the frozen BEST-RQ codewords of one clean speaker from the mixture. Utterance-level PIT resolves speaker order.
• Mask-free — masking can hide exactly the content needed to disentangle a mixture. SepRQ predicts units over the full utterance, which uses every frame for training.
• Multi-resolution — a separation objective follows every two Conformer layers, with frame folding going from 20 ms to 320 ms and one codebook per scale.

👥 Results

Frozen upstreams with SUPERB / TS-SUPERB downstream heads.

Multi-speaker SUPERB

Speaker diarization (SD), speech separation (SS) and speech enhancement (SE). LS = LibriSpeech, LL = Libri-Light, Mix = LL-60k + GigaSpeech-10k + VoxPopuli-24k.

TS-SUPERB

Target speaker extraction (TSE), personalized speech extraction (PSE), personalized VAD (PVAD), target-speaker ASR (TS-ASR).

Out-of-domain

EEND diarization on DIHARD 3 and ConvTasNet separation on WSJ0-2/3mix.

Citation

If SepRQ helps your research, please cite the paper: arXiv:2610.04690.

@misc{baroudi2026seprqselfsupervisedspeech,
  title         = {SepRQ : Self-Supervised Speech Mixture Representation Learning
                   via Mask-Free, Multi-Scale Source Separation},
  author        = {Séverin Baroudi and Hervé Bredin and Ricard Marxer},
  year          = {2026},
  eprint        = {2610.04690},
  archivePrefix = {arXiv},
  primaryClass  = {eess.AS},
  url           = {https://arxiv.org/abs/2610.04690}
}