lfeats

lfeats provides a unified interface to extract hidden representations from various speech foundation models such as HuBERT and Whisper. While these extracted features are task-independent, the package is primarily designed for speech generation tasks including text-to-speech and voice conversion.

Requirements

• Python 3.10+
• PyTorch 2.6.0+

Documentation

• Reference Manual

Installation

The latest stable release can be installed via PyPI:

pip install lfeats

Alternatively, the development version can be installed directly from the GitHub repository:

pip install git+https://github.com/takenori-y/lfeats.git@master

Supported Models

Frame-Level Features

Token-Level Features

Utterance-Level Features

[!IMPORTANT] Users must comply with the respective licenses of the models. Please refer to the original repositories for detailed licensing information.

Supported Resamplers

Examples

Simple Usage

lfeats simplifies the process of extracting hidden states from various speech foundation models. You don't need to worry about differences between model types or input/output data types.

import lfeats
import numpy as np

# Prepare an audio waveform without zero-mean and unit-variance normalization.
# Either a NumPy array or a Torch tensor are accepted as the input of the extractor.
sample_rate = 16000
waveform = np.random.uniform(-1, 1, sample_rate)

# Initialize the extractor.
extractor = lfeats.Extractor(
    model_name="hubert",
    model_variant="base",
    resampler_type="torchaudio",
    resampler_preset="kaiser-best",
    device="cpu",
)

# Note: The model weights are automatically loaded during the first call to extractor(),
# so calling extractor.load() explicitly is optional.
extractor.load()

# Extract features.
features = extractor(waveform, sample_rate)
print(f"Shape: {features.shape}")  # (1, 50, 768)

# You can access the features as a Numpy array.
print(type(features.array))  # <class 'numpy.ndarray'>

# You can also access the features as a Torch tensor.
print(type(features.tensor))  # <class 'torch.Tensor'>

Layer Selection

lfeats allows you to extract features from specific layer(s). By default, the last layer is used.

import lfeats
import numpy as np

sample_rate = 16000
waveform = np.random.uniform(-1, 1, sample_rate)

extractor = lfeats.Extractor(model_name="hubert")

# Get the second-to-last layer output.
features = extractor(waveform, sample_rate, layers=-2)
print(f"Shape: {features.shape}")  # (1, 50, 768)

# Get the multiple layer outputs.
features = extractor(waveform, sample_rate, layers=(11, 12))
print(f"Shape: {features.shape}")  # (1, 50, 1536)

# Get all layer outputs as a concatenated vector.
features = extractor(waveform, sample_rate, layers="all")
print(f"Shape: {features.shape}")  # (1, 50, 9984)

Audio Chuking

To be computationally efficient and prevent mismatches between training and inference, long audio files can be processed by splitting them into chunks.

import lfeats
import numpy as np

sample_rate = 16000
waveform = np.random.uniform(-1, 1, 10 * sample_rate)

extractor = lfeats.Extractor(model_name="hubert")

# Processing a 10-second waveform with a 5-second chunk and 1-second overlap.
features = extractor(waveform, sample_rate, chunk_length_sec=5, overlap_length_sec=1)
print(f"Shape: {features.shape}")  # (1, 500, 768)

Sliding-Window Upsampling

Since the frame rate of speech foundation models is typically 20ms, it often doesn't match the 5ms requirement of speech generation tasks. lfeats bridges this gap by sliding the input waveform and interleaving the resulting features, providing a high-resolution output.

import lfeats
import numpy as np

sample_rate = 16000
waveform = np.random.uniform(-1, 1, sample_rate)

extractor = lfeats.Extractor(model_name="hubert")

# Extract features at a 5ms frame rate.
features = extractor(waveform, sample_rate, upsample_factor=4)
print(f"Shape: {features.shape}")  # (1, 200, 768)

Utterance-Level Feature Extraction

lfeats can extract utterance-level features, e.g., speaker embeddings, as well as frame-level features.

import lfeats
import numpy as np

sample_rate = 16000
waveform = np.random.uniform(-1, 1, sample_rate)

extractor = lfeats.Extractor(model_name="ecapa-tdnn")

# The default aggregation method for utterance-level features is averaging.
features = extractor(waveform, sample_rate, overlap_length_sec=0, reduction="mean")
print(f"Shape: {features.shape}")  # (1, 1, 192)

Command-line Interface

Once installed via pip, you can use the lfeats command directly from your terminal.

# Basic usage: extract features from a wav file
$ lfeats input.wav --output_format npz

# Process all audio files in a directory
$ lfeats input/dir --output_dir feats

# Process files listed in a file
$ lfeats input.scp --output_dir feats

# Specify model and layer
$ lfeats input.wav --model_name hubert --model_variant base --layer 12

[!TIP] For more details on all available flags and default values, simply run:

$ lfeats --help

License

This project is released under the MIT License.

Third-Party Licenses

lfeats partially incorporates the following repositories: