rapido-whisper

rapido-whisper is a Whisper-only inference engine for NVIDIA GPUs, derived from CTranslate2 4.8.3 (MIT). It keeps the CTranslate2 Python API for Whisper (import ctranslate2), so faster-whisper and CrisperWhisper run on it unchanged.

Its rule is bit-identical outputs. Every change keeps tokens, scores, no-speech probabilities and word timestamps equal to the original CTranslate2 on the same GPU and cuBLAS version. Speedups that change floating point rounding are opt-in or left out.

What it contains

• Whisper only: encoder, decoder, generate, align, detect_language, and the Transformers Whisper converter (ct2-transformers-converter).
• CUDA only, native code for Ada (sm_89: RTX 40xx, L4, L40) and Blackwell (sm_120: RTX 50xx).
• Compute types float16, bfloat16 and float32. auto resolves to float16.
• CUDA graphs for decoder steps (CT2_CUDA_GRAPHS=1). Each step is captured once and replayed in later calls, for example in every window of streaming ASR. The graphs run the same kernels as the regular path. Measured on an RTX 5080 with large-v3: generate is about 20% faster (5.9 → 4.6 ms/token greedy, 719 → 611 ms beam 5 on a 15 s window).
• The CrisperWhisper decoding APIs from nyrahealth/CTranslate2: prefill, forward_step, forward_batch, *_with_attention, generate_greedy_with_attention, generate_dual_greedy and generate_speculative.

Removed from CTranslate2: every model except Whisper (translators, generators, encoders, Wav2Vec2), the CLI, CPU backends (MKL, oneDNN, Ruy, OpenBLAS, Accelerate, OpenMP), AMD HIP, tensor parallelism, FlashAttention, AWQ, int8/int16 quantization, and LLM-only layer features (rotary, ALiBi, RMSNorm, GQA/MQA, sliding windows, T5 relative positions, Gemma norms). A model that needs any of them is rejected at load time with an explicit error.

Build

Requirements: CUDA 12.8+ (sm_120 needs it), CMake ≥ 3.18 and GCC. The release flags below match the wheels; -DWITH_CUDNN=ON runs the encoder's conv1d through cuDNN instead, which changes its rounding.

git submodule update --init --recursive
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DWITH_CUDA=ON -DWITH_CUDNN=OFF -DCUDA_DYNAMIC_LOADING=ON \
  -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-12.8 \
  "-DCUDA_ARCH_FLAGS_OVERRIDE=-gencode;arch=compute_89,code=sm_89;-gencode;arch=compute_120,code=sm_120;-gencode;arch=compute_120,code=compute_120" \
  -DCMAKE_INSTALL_PREFIX=$PWD/../install
make -j$(nproc) install

cd ../python
CTRANSLATE2_ROOT=$PWD/../install pip install --no-build-isolation .

CUDA_ARCH_FLAGS_OVERRIDE passes the -gencode flags directly, because older CMake FindCUDA cannot parse two-digit architectures such as 12.0. Set LD_LIBRARY_PATH to install/lib at runtime.

Install

pip install "CTranslate2-rapid-whisper[cuda]"

The wheels are built like the upstream ctranslate2 wheels: without cuDNN and with CUDA loaded at runtime, so they produce the same outputs as the upstream wheel with the same cuBLAS. The cuBLAS version decides the exact floating point results; import ctranslate2 loads cuBLAS 12 from, in this order: a libcublas.so.12 in LD_LIBRARY_PATH, the nvidia-cublas-cu12 wheel (the cuda extra, also pulled by torch's CUDA 12 builds), the system CUDA. Pin nvidia-cublas-cu12 (e.g. ==12.8.4.1) to get the same results everywhere.

The distribution installs the ctranslate2 module, so it replaces the upstream ctranslate2 package and cannot be installed next to it. faster-whisper declares a ctranslate2 dependency; install it with --no-deps, otherwise pip installs the upstream package over this one:

pip uninstall -y ctranslate2
pip install "CTranslate2-rapid-whisper[cuda]"
pip install --no-deps faster-whisper
pip install "av>=11,<16" tokenizers onnxruntime huggingface_hub tqdm

Versions: ctranslate2.__version__ stays at the API-compatible 4.8.3, and ctranslate2.rapido_version gives the engine version.

Runtime options

Memory: every captured step graph holds ~8 MB of device memory (the launch parameters of its ~1000 kernels), and each decoding shape keeps persistent buffers: the self-attention caches at full capacity (twice, ping-pong) and the cross-attention keys/values. Random sampling (the temperature fallback) and batches that shrink during a call are decoded without graphs. Measured on an RTX 4090 with large-v3 float16, 82 interview windows of 30 s, beam 3 with temperature fallback:

Verification

The equivalence tooling lives outside the library: compare_ct2.py (1340 outputs covering greedy, beam, n-best, batches, prefixes, timestamps and faster-whisper transcribe with word timestamps), fork_check.py (long-form interviews and streaming with LocalAgreement) and stream_la.py. Against CTranslate2 4.8.3, with and without CUDA graphs, they report 0 differing tokens and 0 differing scores, no-speech probabilities and word times, and identical streaming rounds.

License

MIT, as CTranslate2. The CrisperWhisper APIs come from nyrahealth/CTranslate2 (MIT). The CrisperWhisper model weights have their own non-commercial license.