Fast GPU ASR

Zipformer and Parakeet. Built for speed.

Batched offline speech recognition for NVIDIA GPUs. Raw audio in, text and word timestamps out, through one Python API. TensorRT engines, native CUDA plugins, and GPU beam search handle inference; no NeMo or k2 installation is needed.

Batched speech recognition at up to 25,000 RTFx on B300

Zipformer CR-CTC Transducer beam 6	Parakeet V3 TDT beam 6

RTFx = total audio duration / total inference time: throughput, not request latency. Each configuration processes 157.8 hours of audio across seven English datasets.

FP16, decoder beam 6:

We reproduced the Open ASR Leaderboard English evaluation with fast-gpu-asr, using its datasets and WER scorer and reporting both RTFx and WER. Methodology.

Key Observations

• Batching matters. On B300 with FP16, batch 256 versus batch 1 delivers 28.5x for Zipformer CR-CTC Transducer and 21.6x for Parakeet V3 TDT.
• Gains taper. On that same GPU with FP16, doubling capacity from 128 to 256 changes throughput by +25.1% for Zipformer CR-CTC Transducer and +23.0% for Parakeet V3 TDT.
• BF16 is supported too. At B300/batch 256, BF16 throughput is slightly higher for Zipformer CR-CTC Transducer and Parakeet V3 TDT than FP16. WER is recorded for every configuration; precision can marginally change outputs.
• FP16 vs. FP32. At B300/batch 128, FP16 changes throughput by +22.2% for Zipformer CR-CTC Transducer and +58.5% for Parakeet V3 TDT relative to FP32.
• Mean-WER is consistent across precisions and batches. On B300, the recorded mean-WER span across all measured precisions and batches is 0.014 percentage points for Zipformer CR-CTC Transducer and 0.034 percentage points for Parakeet V3 TDT.

All results | Download CSV

Quick Start

Requirements: Linux x86-64, Python 3.12-3.14, a Turing (SM75) or newer NVIDIA GPU, and NVIDIA driver 580 or newer. The package uses CUDA 13 and TensorRT. The wheel includes all nine TensorRT plugin libraries; no local compilation or TensorRT development headers are needed.

pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install fast-gpu-asr

Use a fresh environment and install CPU-only PyTorch first. TensorRT and CuPy provide GPU inference; PyTorch does not need CUDA support. Package metadata cannot select PyTorch's CPU index for pip automatically.

Supported families: Parakeet TDT/CTC and Zipformer Transducer/CTC. You can export compatible checkpoints and fine-tuned models, not just the models shown below. Checkpoints must match the exporters' supported architectures and checkpoint/configuration formats.

The examples export FP16, batch size 8, with a 0.1 / 8 / 40-second minimum/typical/maximum audio duration profile. Transducer examples use beam 6; CTC uses greedy decoding. Checkpoints are downloaded from each model's main branch and are not bundled with the package.

Build engines on the GPU you will use, with the same TensorRT/plugin stack as inference. Export deletes and recreates its output directory. Keep checkpoints and unrelated files outside it. Building engines takes time and requires additional host/GPU memory for tactic selection.

Zipformer CR-CTC Transducer

Zipformer CR-CTC Transducer XL 290M checkpoint.

mkdir -p checkpoints/zipformer
base=https://huggingface.co/soundsgoodai/Zipformer-cr-ctc-transducer-XL-290M/resolve/main
for file in model.pt config.yaml bpe.model; do
  curl --fail --location "$base/$file" --output "checkpoints/zipformer/$file"
done

fast-gpu-asr-export-zipformer \
  --model-path checkpoints/zipformer/model.pt --output-dir exported/zipformer \
  --batch-size 8 --decoder-type transducer_modified_beam_search --beam 6 \
  --encoder-precision fp16 --decoder-precision fp16 \
  --min-audio-seconds 0.1 --opt-audio-seconds 8 --max-audio-seconds 40 \
  --optimization-level 5

Zipformer CTC

Export the same CR-CTC checkpoint's CTC head instead of its transducer head:

fast-gpu-asr-export-zipformer \
  --model-path checkpoints/zipformer/model.pt --output-dir exported/zipformer-ctc \
  --batch-size 8 --decoder-type ctc_greedy_search --beam 1 \
  --encoder-precision fp16 \
  --min-audio-seconds 0.1 --opt-audio-seconds 8 --max-audio-seconds 40 \
  --optimization-level 5

CTC requires a checkpoint with a CTC head. It needs no transducer decoder engine or predictor table, so --decoder-precision is omitted. The benchmark results above use transducer decoding, not CTC.

Parakeet V3

Parakeet TDT 0.6B V3 checkpoint.

mkdir -p checkpoints/parakeet
base=https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3/resolve/main
curl --fail --location "$base/parakeet-tdt-0.6b-v3.nemo" \
  --output checkpoints/parakeet/parakeet-tdt-0.6b-v3.nemo

fast-gpu-asr-export-parakeet \
  --model-path checkpoints/parakeet/parakeet-tdt-0.6b-v3.nemo \
  --output-dir exported/parakeet --batch-size 8 \
  --decoder-type transducer_modified_beam_search --beam 6 \
  --encoder-precision fp16 --decoder-precision fp16 \
  --min-audio-seconds 0.1 --opt-audio-seconds 8 --max-audio-seconds 40 \
  --optimization-level 5

Parakeet CTC

Both 0.6B and 1.1B are supported. Choose either checkpoint by setting model below:

model=parakeet-ctc-0.6b  # Or parakeet-ctc-1.1b.
mkdir -p checkpoints/parakeet
curl --fail --location \
  "https://huggingface.co/nvidia/$model/resolve/main/$model.nemo" \
  --output "checkpoints/parakeet/$model.nemo"

fast-gpu-asr-export-parakeet \
  --model-path "checkpoints/parakeet/$model.nemo" \
  --output-dir "exported/$model" --batch-size 8 \
  --decoder-type ctc_greedy_search --beam 1 --encoder-precision fp16 \
  --min-audio-seconds 0.1 --opt-audio-seconds 8 --max-audio-seconds 40 \
  --optimization-level 5

CTC uses one encoder engine with an integrated CTC head and GPU greedy decoding. These checkpoints are separate from the TDT models used in the benchmarks above.

Transcribe

Use any exported bundle with the same API. This example reads two mono PCM16 WAV files at 16 kHz, each no longer than 40 seconds:

import wave

import numpy as np

from fast_gpu_asr import ASR

paths = ["sample-1.wav", "sample-2.wav"]
audios = []
for path in paths:
    with wave.open(path) as wav:
        channels = wav.getnchannels()
        sample_width = wav.getsampwidth()
        sample_rate = wav.getframerate()
        if (channels, sample_width, sample_rate) != (1, 2, 16000):
            raise ValueError(f"Expected a mono PCM16 WAV at 16 kHz: {path}")
        samples = np.frombuffer(wav.readframes(wav.getnframes()), dtype=np.int16)
    audios.append(samples.astype(np.float32) / 32768.0)

# Or use "exported/zipformer-ctc" or "exported/parakeet".
asr = ASR("exported/zipformer")
texts, timestamps = asr(audios)
for path, text, words in zip(paths, texts, timestamps, strict=True):
    print(f"{path}: {text}")
    for word, start, end in words:
        print(f"  {start:.2f}-{end:.2f}: {word}")

Pass one or more nonempty 1D NumPy float32 waveforms as a list, normalized to [-1, 1] at the bundle's sample rate. Partial batches work; short clips are padded internally. Stay within the exported batch and duration limits; resample or split audio beforehand.

ASR returns a transcript and a list of (word, start, end) tuples per clip, with times in seconds. Timestamps come from decoder tokens; the final word ends at the clip's duration. Select a GPU with ASR(..., device_id=0).

Models and Inference Precision

FP32, FP16, and BF16 are supported. BF16 requires Ampere or newer. Published measurements cover A100, H200, and B300 with Zipformer CR-CTC Transducer and Parakeet V3 TDT, using beam 6 and English audio.

This is an offline batch inference library, not a streaming engine. Engines have fixed batch capacity and dynamic audio length within their exported profiles. Choose a smaller batch size or separate duration profiles when memory or latency matters more than aggregate throughput.

Implementation Details

Zipformer Encoder and Parakeet FastConformer

A single TensorRT engine turns raw waveforms into decoder-ready outputs. Features and activations stay on the GPU, and the decoder consumes the outputs directly, without a round trip through CPU memory.

• GPU-native audio frontend. Batched cuFFT transforms, cuBLAS mel projection, and custom framing, windowing, and normalization kernels keep feature extraction inside the encoder. There is no CPU feature-extraction stage or feature upload.
• Fused, layout-aware kernels. Nine native TensorRT plugin libraries across both model families combine operations such as relative-position alignment, masking, and softmax, or depthwise convolution and activation. Attention reads projection layouts directly where possible, avoiding explicit transpose buffers; dedicated Zipformer kernels handle resampling and output assembly.
• Optimizations at export time. Parakeet combines query, key, and value into one projection, folds evaluation-mode BatchNorm into convolution weights and biases, and absorbs constant feed-forward scaling into its output projections. This reduces projection launches and removes separate BatchNorm and scaling operations at inference.
• Tuned for the target GPU. TensorRT selects implementations for the exported batch size, duration profile, and precision, including eligible CUDA and cuBLAS tactics exposed by the plugins. Tactic selection happens during engine building.
• Reuse memory, replay execution. Pinned host staging, asynchronous transfers, and reusable device buffers and workspaces reduce setup work. Consecutive batches with the same shape can use CUDA graph replay to reduce kernel-launch overhead.

Beam Search Decoder

Both transducer decoders keep scores, candidate selection, and token histories on the GPU. Backpointers avoid copying histories; reusable buffers and CUDA graphs reduce allocation and launch overhead. Final selection uses length-normalized log probability.

Zipformer RNN-T

Like k2/Icefall's Zipformer reference modified_beam_search, our search emits at most one nonblank token per encoder frame. Key differences:

• Merge first, then prune. k2/Icefall selects the top beam candidates before merging duplicate histories. We combine their scores with log-sum-exp first, then retain the top beam unique hypotheses. Duplicates cannot waste beam slots, and combined scores can rescue histories that early pruning would lose.
• Precompute the predictor. GPU table lookups replace per-frame stateless predictor evaluation; only the joiner runs at each step. Export precomputes projected outputs for contexts of up to two tokens, trading memory for less work.
• Specialize the search. CUDA kernels tailored to beam, vocabulary, and context size replace Python lists and ragged bookkeeping. Prefix checks find possible duplicates; exact history comparisons guard against hash collisions.

Parakeet TDT

Like NeMo's batched ModifiedALSDBatchedTDTComputer implementation, our decoder supports batched hypotheses, GPU-resident LSTM states, reusable buffers, and CUDA graphs. Key differences:

• Merge before pruning. NeMo selects the top beam candidates, then recombines survivors. We merge candidates first, checking exact token histories, encoder positions, and zero-duration emission counts, then retain the top beam unique states.
• Search duration alternatives. NeMo pairs each parent's top token candidates with its most probable duration, forcing zero-duration blanks to advance. We expand each parent's top beam nonblank tokens over all configured durations, plus blanks over every positive duration, before merging and pruning.
• Advance differently at the symbol limit. NeMo forces a blank when its emission cap is reached. Our max_symbols_per_timestep advances one frame after the capped zero-duration token without adding a blank transition.
• TensorRT and custom CUDA kernels. Our predictor/joiner runs in TensorRT; CuPy-launched kernels handle search and state routing. NeMo uses PyTorch and supports a full conditional CUDA-graph loop; ours replays fixed search chunks with periodic host completion checks.

For both models, these search changes can affect transcripts; they do not guarantee better WER or identical upstream results. These are implementation comparisons, not matched speedup measurements against upstream decoders.

Decoder Settings

transducer_modified_beam_search uses the exported beam width. transducer_greedy_search performs greedy, single-hypothesis decoding by forcing beam=1, keeping only the best hypothesis at each step. It reuses the modified beam-search implementation rather than a separate decoder. ctc_greedy_search requires a CTC head and uses neither a transducer decoder engine nor a predictor table.

Both exporters accept --blank-penalty (default 0.0), subtracted from blank log probabilities after normalization. Positive values discourage blanks in every mode, including beam one. k2/Icefall's reference applies its penalty before softmax, so equal nonzero settings are not equivalent. Mode, beam, and penalty are saved in model_config.yaml.

Export and Runtime Notes

--encoder-precision and --decoder-precision default to fp32. Waveform frontends and CTC heads remain FP32 in both models. Changing precision can change transcripts: check WER alongside throughput.

• Zipformer precision. The final encoder projection and transducer log probabilities remain FP32. BF16 exports use FP16 for the first subsampling convolution.
• Parakeet precision. When input scaling is enabled, FP16 exports rescale the first block's output weights, biases, and LayerNorm epsilons to keep residuals in range while retaining FP16 subsampling and Conformer layers. FP32/BF16 exports instead fold the scale into the subsampling projection's weights and bias.
• Export. Build engines for the target GPU and matching TensorRT/plugin versions. Use --debug to keep intermediate ONNX files; successful exports otherwise remove them.
• Runtime. Each ASR instance reuses buffers and serializes calls with a lock. Model-bundle validation is enabled by default (validate=True).

Build From Source

Provide a CUDA-compatible C++20 host compiler and TensorRT development headers (NvInfer.h) matching the runtime in uv.lock. If the headers are outside the compiler's search paths, set CPLUS_INCLUDE_PATH to their directory. From the repository root:

uv sync --frozen --extra dev
uv run --frozen python -m fast_gpu_asr.tensorrt_plugins.build

uv installs nvcc, CUDA headers/libraries, TensorRT, and CPU-only PyTorch. PyTorch handles ONNX export. Prefix exporter commands with uv run --frozen when using the checkout. For TensorRT headers, CI uses GitHub releases, with a checksum-verified NVIDIA fallback when the tag is unavailable.

Native targets: sm_75, sm_80, sm_86, sm_87, sm_88, sm_89, sm_90, sm_100, sm_103, sm_110, sm_120, and sm_121, plus a compute_80 PTX fallback. Verify execution and memory requirements on your GPU.

Tests and Packaging

After the source-build setup above, run from the repository root:

uv run --frozen pytest
uv run --frozen ruff check .
uv run --frozen ruff format --check .
uv run --frozen python src/fast_gpu_asr/decoder/lint_gpu_kernels.py --check

GPU tests skip when CUDA is unavailable; BF16 tests require SM80 (Ampere) or newer. Tests check decoder behavior and numerical tolerances, not identical transcripts across precisions. Formatting limits are 88 columns for Python and 100 for CUDA/C++.

Build a wheel: scripts/build_wheel.sh rebuilds plugins and produces a repaired manylinux_2_27_x86_64 wheel in dist/. Pass an output directory to override it. CUDA and TensorRT remain external dependencies. Source distributions are unsupported.

CI: The workflow runs Python 3.12-3.14 CPU checks on pushes to main and pull requests. Manual dispatch with run_gpu_tests=true adds native tests and wheel smoke tests on the separately billed gpu-t4 runner; SM80-only cases skip there.

Reproduce the Measurements

Complete results and CSV | Hardware, protocol, and plot reproduction | Fresh collection workflow

The CSV regenerates the published tables and plots on a CPU-only host. Collecting new measurements requires the datasets, checkpoints, and a target GPU. The published campaign pins its source and scorer revisions.

Rebuild Published Plots

From the repository root:

uv sync --frozen --extra benchmark
uv run --frozen --extra benchmark plotly_get_chrome -y
uv run --frozen --extra benchmark python docs/benchmarks/render.py

Skip browser installation when Chrome/Chromium is already available, or set BROWSER_PATH to its executable. This regenerates two SVGs, results.md, and only the marked performance block in this README from measurements.csv.

License and Acknowledgments

Code is Apache-2.0. Model weights and datasets retain their own licenses; the code license does not grant commercial rights to noncommercial checkpoints. Built on the work of k2, NeMo, and their contributors, with NVIDIA TensorRT and CUDA libraries. See NOTICE for component attribution.