voicedecide

Training-free decisions on audio: call routing and language identification with EmbeddingGemma 2. Audio in, a probability for every option out, plus an act / confirm / escalate action your pipeline can branch on.

No fine-tuning and no text generation. A clip is embedded once, then compared with each option:

audio → EmbeddingGemma 2 (frozen) → cosine vs. option vectors → ÷ τ → softmax → decision

• Zero-shot (default): describe the options in words.
• Few-shot prototypes (recommended): give ~10 labeled clips per option, build once, reuse.

Install

pip install voicedecide
voicedecide download        # optional: fetch the model ahead of time (~2.5 GB, cached)

The model weights are not part of the package. They are downloaded from Hugging Face on first use and cached in ~/.cache/huggingface (HF_HOME moves it; HF_HUB_OFFLINE=1 runs without network once cached). Python 3.11+. A GPU is recommended for live use; CPU works for batch jobs.

Quick start: zero-shot routing (default)

from voicedecide import Decider, Choice

d = Decider()

route = Choice.from_descriptions({
    "card_issues": ["A caller reporting a lost or stolen card",
                    "Someone whose bank card is blocked or damaged"],
    "balance":     ["A caller asking how much money is in their account"],
    "pay_bill":    ["A caller wanting to pay a bill"],
})

res = d.decide("call.wav", {"route": route})
a = res["route"]
print(a.choice, a.probability, a.action)

Write descriptions as full phrases ("A caller reporting a lost card"), not bare labels ("lost card"). Several phrasings per option are averaged.

Recommended: build prototypes once (few-shot)

from glob import glob

route = Choice.from_examples({          # ~10 of YOUR calls per option, from different speakers
    "card_issues": glob("labeled/card_issues/*.wav"),
    "balance":     glob("labeled/balance/*.wav"),
    "pay_bill":    glob("labeled/pay_bill/*.wav"),
}, encoder=d)

report = route.fit(val_files, val_labels, encoder=d)   # different clips: picks τ and thresholds
print(report)                                          # accuracy, macro-F1, ECE, action shares
route.save("packs/my-routing")                         # vectors + settings only, no audio

Nothing is trained: each option's clip embeddings are averaged into one prototype. Use clips from your own audio (same phone line, noise, accents). Prototypes from studio audio tested on phone calls is the most common way accuracy drops.

In a pipeline

d = Decider()
route = Choice.from_pack("packs/my-routing")
lang = Choice.from_pack("language-id")

def handle_call(audio_bytes):
    res = d.decide(audio_bytes, {"route": route, "lang": lang})   # one embedding, two answers
    r = res["route"]
    if r.action == "act":
        return transfer(r.choice, language=res["lang"].choice)
    if r.action == "confirm":
        return ask(f"Did you mean {r.choice.replace('_', ' ')}?")
    return to_human_agent()

audio can be a file path, encoded bytes, a file object, (array, sample_rate), or a Hugging Face audio dict. Audio is converted to mono 16 kHz. max_seconds=5 decides on the start of the clip only.

Result

a = res["route"]
a.choice          # "card_issues" (None if the audio matches no option, see out_of_set)
a.probabilities   # {"card_issues": ..., "balance": ..., "pay_bill": ...}  sums to 1
a.confidence      # 0-1: 1 - normalized entropy (how concentrated the probabilities are)
a.margin          # top-1 minus top-2 probability
a.action          # "act" | "confirm" | "escalate"
a.out_of_set      # True when the best match is below the question's calibrated floor
res.latency_ms    # {"decode", "embed", "decide", "total"}
res.to_json()

act and confirm thresholds default to 0.80 / 0.50 and are replaced by fit(), which picks the lowest probability whose decisions reach 95 % (act) and 80 % (confirm) precision on your held-out clips. Decider(strict=True) raises OutOfSetError instead of returning out-of-set answers.

Language identification

lang = Choice.from_pack("language-id")          # tested languages only
lang = Choice.from_pack("language-id", languages=["hindi", "ta", "gu_in"])

Tested languages: English, Hindi, Bengali, Tamil, Telugu, Marathi. These load by default.

The pack also contains these languages, which have not been tested: Urdu, Gujarati, Kannada, Malayalam, Odia, Punjabi, Assamese, Nepali, Sindhi, Mandarin Chinese, Cantonese, Spanish, Portuguese, French, Italian, German, Dutch, Russian, Ukrainian, Polish, Turkish, Arabic, Persian, Japanese, Korean, Vietnamese, Thai, Burmese, Indonesian, Malay, Filipino, Swahili, Hausa, Amharic.

Selecting an untested language works but warns once:

UntestedLanguageWarning: 'gu_in' (Gujarati) has not been tested.
  Tested languages: Bengali, English, Hindi, Marathi, Tamil, Telugu.
  Results for untested languages may be unreliable. Check them on your own labeled clips:
  voicedecide eval --pack language-id --languages gu_in your_clips.csv

The built-in pack has no out-of-set floor: one calibrated on FLEURS read speech also flags correctly identified phone calls. To detect speech in languages outside the pack, run lang.fit(clips, labels) on labeled clips from your own pipeline; that sets a floor matched to your audio.

A language that is not in the pack raises UnknownLanguageError with a suggestion. You can add your own with lang.add_examples({"xx": clips}) (marked untested until you run fit on your clips). Closely related languages (for example Hindi and Urdu) are expected to be confused more often.

voicedecide languages prints every language and its status.

Command line

# folder layout: labeled/<option>/*.wav
voicedecide build-pack --examples labeled/ --val val/ --out packs/my-routing
voicedecide run --pack route=packs/my-routing --pack language-id calls/*.wav > decisions.jsonl
voicedecide eval --pack packs/my-routing test.csv          # CSV columns: path,label
voicedecide languages

Options worth knowing

Limits

• Audio is cut at 320 s (model context).
• Benchmarks use public datasets (FLEURS read speech, MINDS-14 banking calls). Real calls with 8 kHz audio, noise or code-mixed speech can score lower: run voicedecide eval on your own labeled clips before going live.
• Out-of-set detection is approximate; a closely related language or intent can still be accepted. The floor set by fit() only holds for audio like the calibration clips (same channel and style).
• A small or very easy calibration set pushes τ to the edge of its grid and makes probabilities over-confident; fit() notes this in its report. Use 20+ held-out clips per option when you can.

Reproducing the numbers

• benchmarks/build_language_pack.py builds the language-id pack from FLEURS (validation split for prototypes and calibration, test split for the benchmark) and records the benchmark in pack.json.
• examples/banking-routing/build_and_eval.py builds a routing pack from MINDS-14 en-GB/en-AU and evaluates it on en-US, next to zero-shot text anchors.

License

Apache 2.0. EmbeddingGemma 2 is Google DeepMind's model (Apache 2.0), downloaded at runtime. Built-in pack vectors are derived from FLEURS (CC BY 4.0). See NOTICE.