WhisperS2T-Reborn ⚡

An Optimized Speech-to-Text Pipeline for the Whisper Model Using CTranslate2

WhisperS2T-Reborn is a modernized fork of WhisperS2T, an optimized lightning-fast Speech-to-Text (ASR) pipeline. It is tailored for the Whisper model using the CTranslate2 backend to provide faster transcription. It includes several heuristics to enhance transcription accuracy.

Whisper is a general-purpose speech recognition model developed by OpenAI. It is trained on a large dataset of diverse audio and is also a multitasking model that can perform multilingual speech recognition, speech translation, and language identification.

Installation

pip install -U whisper-s2t-reborn

Quick Start

Transcribe a single file

import whisper_s2t

model = whisper_s2t.load_model(model_identifier="large-v3")

files = ['audio1.wav']
lang_codes = ['en']
tasks = ['transcribe']
initial_prompts = [None]

out = model.transcribe_with_vad(files,
                                lang_codes=lang_codes,
                                tasks=tasks,
                                initial_prompts=initial_prompts,
                                batch_size=32)

print(out[0][0]) # Print first utterance for first file
"""
[Console Output]

{'text': "Let's bring in Phil Mackie who is there at the palace...",
 'avg_logprob': -0.25426941679184695,
 'no_speech_prob': 8.147954940795898e-05,
 'start_time': 0.0,
 'end_time': 24.8}
"""

Batch across multiple files

Passing multiple files allows segments from different files to be batched together, making better use of the GPU:

import whisper_s2t

model = whisper_s2t.load_model(model_identifier="large-v3")

files = ['audio1.wav', 'audio2.wav', 'audio3.wav']
lang_codes = ['en', 'en', 'en']
tasks = ['transcribe', 'transcribe', 'transcribe']
initial_prompts = [None, None, None]

out = model.transcribe_with_vad(files,
                                lang_codes=lang_codes,
                                tasks=tasks,
                                initial_prompts=initial_prompts,
                                batch_size=32)

# out[0] = results for audio1.wav, out[1] = results for audio2.wav, etc.
for file_idx, transcript in enumerate(out):
    print(f"File {files[file_idx]}: {len(transcript)} segments")

Word-level alignment

To enable word-level timestamps, load the model with:

model = whisper_s2t.load_model("large-v3", asr_options={'word_timestamps': True})

Word timestamps use a small aligner model (tiny, or tiny.en for English-only models), which is downloaded when you load a model with word timestamps on. The large-v3 family (large-v3, large-v3-turbo, distil-large-v3, distil-large-v3.5) aligns with itself instead. whisper_s2t.default_word_aligner(name) tells you which one a model uses (None when it aligns with itself).

Running on CPU

model = whisper_s2t.load_model("large-v3", device="cpu", compute_type="float32")

Pass compute_type="float32" — CTranslate2 does not run float16 on CPU. Smaller models (base, small) are much more practical for CPU inference.

Supported Models

All models are available in float16, float32, and bfloat16 compute types via CTranslate2-4you on Hugging Face.

Using models offline

Models are downloaded from Hugging Face the first time you load them. To fill a folder in advance (for example, for a machine without internet access), download the model, plus its word aligner if you use word timestamps:

import whisper_s2t

whisper_s2t.download_model("small.en", compute_type="float32", cache_dir="/models")

aligner = whisper_s2t.default_word_aligner("small.en")  # "tiny.en"; None for the large-v3 family
if aligner is not None:
    whisper_s2t.download_model(aligner, compute_type="float32", cache_dir="/models")

Then load from that folder without touching the network:

model = whisper_s2t.load_model("small.en", device="cpu", compute_type="float32",
                               cache_dir="/models", local_files_only=True,
                               asr_options={'word_timestamps': True})

• Download with the same compute_type you will load with. Each precision is a separate download; other types such as int8 use the float32 files.
• With local_files_only=True, a model missing from the folder raises huggingface_hub's LocalEntryNotFoundError (a FileNotFoundError) right away, without trying the network.
• cache_dir uses the Hugging Face cache layout (models--<org>--<repo>/...). Without it, models go to the library's own cache.
• download_model() also takes force_download=True to fetch the files again, and output_dir= to save a plain copy of the files in a folder, which load_model() accepts as the model identifier.

Benchmarks

Model: distil-large-v3 · FP16 · CUDA · RTX 4090 Engine: whisper-s2t-reborn 1.7.1 · CTranslate2 4.8.1 · torch 2.11.0+cu128 Audio: sam_altman_lex_podcast_367.flac (~2 h 9 min)

Comparing openai-whisper (no batch support) against whisper-s2t-reborn.

The increased VRAM usage even at batch size 1 is largely due to the VAD model. Openai's implementation doesn't use voice activity detection. The benchmarks folder has the actual scripts used.

Acknowledgements

• Original WhisperS2T: Thanks to shashig for the original WhisperS2T project that this fork is based on.
• OpenAI Whisper Team: Thanks to the OpenAI Whisper Team for open-sourcing the Whisper model.
• CTranslate2 Team: Thanks to the CTranslate2 Team for providing a faster inference engine for Transformers architecture.
• NVIDIA NeMo Team: Thanks to the NVIDIA NeMo Team for their contribution of the open-source VAD model used in this pipeline.

License

This project is licensed under MIT License - see the LICENSE file for details.