vaaksetu

A local speech-to-text server for Indian languages that speaks the OpenAI transcription API. Point an existing app's base_url at it and nothing else changes; the audio never leaves the machine.

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="anything")
with open("clip.wav", "rb") as audio:
    print(client.audio.transcriptions.create(model="whisper-1", file=audio, language="hi").text)
# कुछ अणुओं में अस्थिर केंद्रक होता है जिसका मतलब यह कि उनमें थोड़े या बिना किसी झटके से टूटने की प्रगति होती है

That output is a real response from this server for a Hindi FLEURS test clip (reference: "...जिसका मतलब यह है कि... प्रवृत्ति होती है", two words differ).

The design, in three lines

Data path. Upload (multipart, size-capped) to a temp file, ffmpeg decodes it to 16 kHz mono PCM and the temp file is deleted, the PCM is cut into chunks of at most 30 s at the quietest nearby point, each chunk goes through log-mel features, the ONNX Conformer encoder and a CTC head restricted to the requested language's tokens, and the per-chunk texts are stitched back in order. Nothing is persisted; the server has no state beyond the loaded model.

Failure mode 1: confident garbage. This model decodes whatever it is given into whichever language you name. Hindi audio sent as Tamil comes back as fluent-looking Tamil script. So language is required and validated against the 22 codes the model has heads for (an unknown code returns the list); audio is always resampled by ffmpeg to the model's 16 kHz, never trusted to already be; the optional language=auto refuses to answer when its two best candidates are close.

Failure mode 2: long audio exhausting memory. Attention cost grows with the square of the input. Measured here: a single 120 s pass peaked at 1254 MB and ran at RTF 0.25; the same recording cut into 30 s chunks (all 318 s of it) peaked at 888 MB at RTF 0.14 (onnxruntime 1.27 run, output not saved). Chunk size, upload size and duration are all capped, and each refusal names the setting that caused it.

The opinion. Drop-in compatibility beats a nicer bespoke API. The route, the form fields, the json | text | verbose_json shapes and the error envelope match OpenAI's, so the switch is one base_url line.

Why this exists

• ollama#1168 (open, 64 thumbs-up) asks for local speech-to-text; comments ask for an OpenWebUI/OpenAI-compatible transcription endpoint. ollama#11798 asks for audio input. Neither is shipped; neither targets Indian languages. (whisper.cpp#64 is about speaker diarization, not this.)
• The model's own tracker has open requests that map onto this repo: a microphone example with the ONNX files (discussion #5), deployment (#4), streaming (#14), and language detection (#13). mic, serve and --lang auto answer three of them; #14 is only partly answered (see Limits).

Install and run

pip install vaaksetu            # needs ffmpeg on PATH
pip install "vaaksetu[mic]"     # adds the microphone command
vaaksetu fetch                  # downloads ~0.8 GB, verifies SHA-256, into ~/.cache/vaaksetu/model
vaaksetu serve                  # http://127.0.0.1:8000
vaaksetu transcribe clip.wav --lang hi
vaaksetu mic --lang hi          # speak; text appears after each pause

vaaksetu mic --replay clip.wav --lang hi feeds a file through the same path at real-time speed, for demos and latency checks without a microphone.

Not yet on PyPI; from a checkout use pip install -e ..

API

Errors use OpenAI's {"error": {"message", "type", "param", "code"}} envelope. Language codes: as bn brx doi gu hi kn kok ks mai ml mni mr ne or pa sa sat sd ta te ur.

Limits and settings (environment)

Auth posture: binds to loopback by default and, with no key, trusts any local caller (the OpenAI client always sends some key; it is ignored). serve refuses a non-loopback host unless VAAKSETU_API_KEY is set. There is no TLS; put a proxy in front for anything beyond one machine. Uploads are copied to a temp file that is deleted in a finally; ffmpeg runs with an argument list, never a shell. One inference runs at a time (a lock) so concurrent requests queue instead of multiplying memory. Note that the multipart parser spools a request without a Content-Length header before the size check runs.

Measured

Test set: the first 20-25 clips of the public FLEURS test split per language (CC-BY 4.0, ungated), 150 clips and 1918 s of audio, normalised references (lower-case, punctuation removed, NFC). Model: the int8 build described below. Machine: Ryzen 7 6800H (8 cores), CPU only, onnxruntime 1.31.1, Python 3.13, Windows 11. Output of the run behind these numbers: measure/results/evaluate.txt and measure/results/lid.txt; reproduce with measure/evaluate.py and measure/lid.py (clips and weights are not in the repo). An earlier run on onnxruntime 1.27 gave WER within 0.5 points of these.

Overall RTF 0.151 (about 6.6x faster than real time). Peak resident memory 881 MB over the whole run, 762 MB of it the loaded model. A 318 s file through the HTTP server took 45 s (RTF 0.142) in the earlier onnxruntime 1.27 run, whose output was not saved. RTF excludes ffmpeg decode and was measured while another heavy job shared this machine, so treat it as an upper-ish bound. Tamil and Telugu WER is inflated by agglutination: CER is the fairer comparison there. 150 clips is a small sample; no confidence intervals were computed. The fp32 model was not measured (the disk could not hold it), so the cost of quantisation is not isolated by this repo.

Microphone latency (mic --replay, real-time feed, 0.7 s of silence ends an utterance): text appeared 1.6 s after speech stopped for a 4-5 s utterance and 2.8 s for a 10 s one (0.7 s hold + decode). Latency grows with utterance length because the model decodes whole utterances. Opening a real input stream at 16 kHz was verified on this machine; live speech through it was not.

Language detection (language=auto)

Opt-in, never the default; a request that omits language is refused. For each of the 22 heads the score is the mean over frames of the log of the probability mass the joint softmax puts on that head's tokens. On the 150 clips above, picking the top head was right 144 times (96.0%). A first scorer (per-head confidence) managed 92.0% and sent 11 of 25 Hindi clips to Urdu, which is why it was dropped. The six errors were Gujarati-as-Hindi (4) and Hindi-as-Urdu (2). With the server's rule "refuse if best minus runner-up is under 0.2 nats per frame", 128 of 150 clips are answered, all correctly, and 22 get a 422 naming the two close candidates. Caveats: only 7 of the 22 languages were tested; the 0.2 threshold was picked on the same clips it is reported on, so the 100% is optimistic; detection looks at the first 30 s only; closely related languages (Hindi/Urdu, Marathi/Hindi/Nepali, Konkani) are the weak spot. If it matters, pass language.

Model and provenance

IndicConformer 600M multilingual (AI4Bharat, MIT). The official repo is gated on Hugging Face (login required), so vaaksetu fetch pulls a public int8 repackaging (yashwork-byte/indic-conformer-600m-int8-onnx, MIT) pinned to one commit and verified against SHA-256 digests kept in src/vaaksetu/fetch.py. Its encoder is dynamically quantised from 2.4 GB fp32 to 0.65 GB. To use the official fp32 files, download them yourself and set VAAKSETU_MODEL_DIR; the layout (assets/encoder.onnx, ctc_decoder.onnx, language_masks.json, vocab.json, preprocessor.ts) is the same, and that path is not digest-checked. The card of the original says AI4Bharat now recommends Indic-Transcribe for quality; this repo targets IndicConformer because it ships ONNX and runs on a laptop CPU.

Decoding is greedy CTC. The model's TorchScript preprocessor is not executed (no torch dependency): frontend.py reads its window and mel filterbank out of the archive and reproduces the forward pass in numpy. The pickled graph is never loaded.

Limits

• Not a streaming model. mic cuts the stream into utterances at pauses and decodes each once; it is not token-by-token.
• CTC only; the RNNT decoder in the model is not wired. No translation, no diarization, no word timestamps (segments are per chunk), no srt/vtt.
• Only 7 of the 22 languages were measured; the other 15 run through the same graph with unmeasured quality (Santali, Kashmiri, Manipuri and Bodo have far less training data than Hindi).
• Needs about 1 GB of RAM free beyond the OS; a 30 s chunk adds roughly 100 MB on top of the model.
• Audio quality in the field (phone calls, noise) is not covered by FLEURS read speech; expect higher error.

Tests

pytest runs 24 tests with a fake recognizer (real server on an ephemeral port, real openai client): response shapes, oversize and over-long refusals, unknown and missing language, non-audio upload and temp-file cleanup, chunk order and cut placement, auth, auto-language refusal, utterance segmentation. pytest -m model additionally needs VAAKSETU_MODEL_DIR (and VAAKSETU_FLEURS_DIR for the Hindi clip) and is skipped otherwise.

MIT licensed.