MLX-RMVPE

MLX implementation of RMVPE (Robust Model for Vocal Pitch Estimation) for Apple Silicon.

This is the F0 extraction component for RVC-MLX, a native Apple Silicon implementation of Retrieval-based Voice Conversion.

What is RMVPE?

RMVPE extracts fundamental frequency (F0) from audio, essential for preserving pitch/melody in voice conversion:

Input Audio (16kHz) → RMVPE → F0 Contour (Hz) → RVC Decoder → Converted Voice

Unlike simpler methods (CREPE, pYIN), RMVPE is specifically designed for polyphonic music, making it ideal for singing voice conversion where background music may be present.

Installation

uv pip install mlx-rmvpe

For development:

git clone https://github.com/lexandstuff/mlx-rmvpe.git
cd mlx-rmvpe
uv pip install -e .

Quick Start

import librosa
from mlx_rmvpe import RMVPE

# Load model (auto-downloads weights from HuggingFace)
model = RMVPE.from_pretrained()

# Load audio at 16kHz
audio, sr = librosa.load("singing.wav", sr=16000, mono=True)

# Extract F0
f0 = model.infer_from_audio(audio)

print(f"Audio: {len(audio)/16000:.2f}s -> F0: {f0.shape[0]} frames at 100fps")
print(f"Pitch range: {f0[f0 > 0].min():.1f} - {f0[f0 > 0].max():.1f} Hz")

Manual Weight Loading

If you prefer to manage weights yourself:

from huggingface_hub import hf_hub_download
from mlx_rmvpe import RMVPE

# Download weights
weights_path = hf_hub_download(
    repo_id="lexandstuff/mlx-rmvpe",
    filename="rmvpe.safetensors"
)

# Load model manually
model = RMVPE()
model.load_weights(weights_path)
model.eval()

# Download original PyTorch weights
wget -O weights/rmvpe.pt \
  "https://huggingface.co/lj1995/VoiceConversionWebUI/resolve/main/rmvpe.pt"

# Convert to MLX format
python scripts/convert_weights.py \
  --pytorch_ckpt weights/rmvpe.pt \
  --mlx_ckpt weights/rmvpe.safetensors

API Reference

RMVPE

RMVPE(hop_length: int = 160)

Class Methods:

Parameters:

Input:

• Audio waveform at 16kHz, shape (samples,) or (batch, samples)

Output:

• F0 in Hz, shape (frames,) where frames = samples / hop_length
• Unvoiced frames have F0 = 0

RVC Integration

In the RVC voice conversion pipeline:

# 1. Extract content features with ContentVec
features = contentvec_model(audio)["x"]  # (1, T, 768) at 50fps

# 2. Extract pitch with RMVPE
f0 = rmvpe_model.infer_from_audio(audio)  # (T*2,) at 100fps

# 3. Interpolate F0 to match ContentVec frame rate
f0_interp = interpolate_f0(f0, target_len=features.shape[1])

# 4. Generate converted audio with RVC synthesizer
output = rvc_synthesizer(features, f0_interp, speaker_id)

Technical Details

Architecture

RMVPE uses a Deep U-Net with BiGRU layers:

Mel Spectrogram (128 mels)
    ↓
Encoder (5 layers, 16→32→64→128→256 channels)
    ↓
Intermediate (4 layers, 256→512 channels)
    ↓
Decoder (5 layers with skip connections)
    ↓
CNN (16→3 channels)
    ↓
BiGRU (384→512)
    ↓
Linear (512→360 pitch bins)
    ↓
Sigmoid → Pitch Probabilities

Pitch Decoding

The model outputs 360 pitch bins (covering ~32 Hz to ~1975 Hz). Decoding:

1. Find peak bin via argmax
2. Local averaging over 9-bin window
3. Convert cents to Hz: f0 = 10 * (2 ** (cents / 1200))
4. Apply voicing threshold

Frame Rate

• Sample rate: 16,000 Hz
• Hop length: 160 samples
• Frame rate: 100 fps

This is 2x faster than ContentVec (50 fps), providing higher temporal resolution for pitch.

Validation

This implementation produces numerically similar outputs to the PyTorch reference:

See IMPLEMENTATION_NOTES.md for validation methodology.

Development

Project Structure

mlx-rmvpe/
├── mlx_rmvpe/
│   ├── __init__.py
│   ├── rmvpe.py          # Main RMVPE class with from_pretrained()
│   └── model.py          # DeepUnet, BiGRU, E2E model architecture
├── scripts/
│   └── convert_weights.py # PyTorch → SafeTensors conversion
├── tests/
│   └── test_rmvpe.py
├── IMPLEMENTATION_NOTES.md
└── README.md

Running Tests

pytest tests/

Publishing to PyPI

# Build
uv build

# Upload
uv publish

License

MIT

Acknowledgments

• RMVPE - Original implementation
• RVC - Voice conversion pipeline
• MLX - Apple's machine learning framework