nirnay

Fast, cheap decisions with honest confidence on the Sarvam API.

nirnay (Hindi: decision) is a small Python library for the questions inside your product that have a fixed set of answers: which team should handle this ticket, what does this user want, is this review positive. It returns the answer, a confidence score you can act on, and the exact ₹ cost, and it hands genuinely hard cases to Sarvam's thinking mode automatically.

Benchmarked on 1,616 test decisions across 5 tasks in English, Hindi, Hinglish, Bengali and Tamil, against sarvam-105b with Sarvam's default settings (nirnay v0.2, logprobs confidence):

	Sarvam default settings	nirnay
Accuracy (with label descriptions)	89.7%	91.5%
Cost per 1,000 decisions	₹43.51	₹16.14 (37%, 2.7× cheaper)
Typical response time	2.2–3.5 s	0.22–0.40 s (9–14× faster)
Decisions with no usable answer	1.2–5.4%	0%
Confidence score	—	yes

Fine print: most of the accuracy gap comes from the default sometimes giving no answer (it thinks until it hits the token limit). On the decisions it did answer, the default was slightly more accurate on 4 of 5 tasks. Read it as similar accuracy, much lower cost and latency, and no failed decisions. Full results and caveats are below.

---

Install

pip install nirnay

Requires Python 3.10+ and a Sarvam API key (dashboard.sarvam.ai), read from the SARVAM_API_KEY environment variable.

Quickstart

from nirnay import Decider

decider = Decider("sarvam-105b")

result = decider.decide(
    question="Which team should handle this customer support message?",
    choices={
        "billing": None,
        "refund": "refund policy, getting a refund, or refund status",
        "technical": None,
        "other": None,
    },
    context="Mera paisa do baar kat gaya, refund kab milega?",
)

result.choice          # "refund"
result.confidence      # 0.999 (the model's probability; calibrated if you pass a calibrator)
result.escalated       # False (True when the model was unsure and thinking mode decided)
result.probabilities   # {"refund": 0.999, "billing": 0.0003, ...}
result.usage.cost_inr  # ≈ 0.003
result.latency_ms      # ≈ 260 typical

How it works

1. One call, the model's own probabilities. The model answers with a single letter on Sarvam's V2 endpoint, which also returns how likely the model found each option (token log-probabilities). nirnay keeps the probabilities of the allowed letters, rescales them to add up to 1, and picks the most likely option. Its probability is the confidence. This is the default for models that support it (sarvam-105b, gemma4). For models without it (e.g. sarvam-105b-conversations), or with confidence="votes", nirnay instead asks for 10 answers in one call (Sarvam's n; the prompt is billed once) and uses the share that agree. Votes are cheaper but a weaker confidence signal (see v0.1 vs v0.2).
2. Escalate when unsure. If the confidence is below 0.8, nirnay asks again with Sarvam's default settings (thinking on) and uses that answer. In the benchmark this happened for 2–27% of decisions, depending on the task. If thinking runs out of tokens without answering, nirnay keeps the fast answer instead of failing.
3. Honest confidence. Optionally fit a calibrator on a few hundred labelled examples so that "90% confident" means right about 90% of the time. Set threshold= to get choice=None (route to a human) when confidence is too low.
4. Label descriptions. Pass {label: description} instead of a list to say what a category means. This was the biggest single accuracy lever in our tests (+17 points on support routing, measured with v0.1).
5. Robust parsing. Replies such as "\nB", "b.", "refund", or a letter spelt in Hindi, Bengali or Tamil script ("बी", "বি", "சி") are all understood.

Details

• The fast path runs with thinking off and a 4-token answer cap; thinking is used only for escalated decisions. Sarvam's two endpoints switch thinking off differently ("none" on V2, null on V1); nirnay sends the right one.
• Choices are shown to the model as letters (A, B, C…) and mapped back, so answers are one token.
• The fixed part of the prompt (question, options, descriptions) comes first so Sarvam's prompt cache can bill it at the cached rate (reported on V1; V2 reported no cache hits in our tests).
• Rate limits (429) and transient errors are retried with backoff; min_interval_s paces calls.

Calibration

from nirnay import Decider, IsotonicCalibrator

# raw confidences (result.raw_confidence) and whether the answer was right, from a labelled
# dev set run with the same confidence method
calibrator = IsotonicCalibrator().fit(dev_raw_confidences, dev_was_correct)
calibrator.save("calibrator.json")

decider = Decider(
    "sarvam-105b",
    calibrator=IsotonicCalibrator.load("calibrator.json"),
    threshold=0.7,  # below this, choice is None: send it to a human
)

Calibrate when you have a few hundred labelled examples and the raw confidence looks overconfident. With small or noisy dev sets the raw confidence is often already reasonable: v0.2's raw logprobs confidence had a calibration error of 0.032 across the benchmark's test sets without any calibrator (see the benchmark).

API

Decider(model="sarvam-105b", *, confidence="auto", samples=None, temperature=None, escalate_below=AUTO, escalation_max_tokens=None, threshold=None, calibrator=None, fallback=None, reasoning_effort=None, max_tokens=4, client=None, endpoint=None)

• confidence: "logprobs" (one V2 call, the model's probabilities), "votes" (several answers in one V1 call), or "auto": logprobs for sarvam-105b and gemma4, votes otherwise or when you pass samples or a V1 endpoint.
• samples: votes per decision in votes mode (1–128, default 10). 1 turns voting off (no confidence).
• escalate_below: escalate when the winner's raw confidence is below this. Defaults to 0.8 on models with thinking (both methods); None turns escalation off.
• threshold: non-escalated decisions with confidence below this return choice=None.
• calibrator: any float -> float callable, e.g. a fitted IsotonicCalibrator.
• fallback: returned as choice if nothing usable came back at all.

decider.decide(question, choices, context) -> Decision: choices is a list of labels, or a mapping {label: description or None}.

Decision fields: choice, best_guess, confidence, raw_confidence, confidence_method, probabilities, votes (votes mode), escalated, escalation_answer, abstained, valid, invalid_samples, retried, usage (input_tokens, cached_input_tokens, output_tokens, cost_inr, calls), latency_ms, model.

Calibration helpers: IsotonicCalibrator (fit, save, load), expected_calibration_error, and in nirnay.calibration: reliability_bins, risk_coverage.

SarvamClient(api_key=None, *, timeout=30, max_retries=3, min_interval_s=0) if you need to share a client or pace requests.

Benchmark

Two systems, same model (sarvam-105b), same prompt, same categories and descriptions, same answer parser:

• Sarvam default settings: only the model and messages are sent (thinking on, one answer).
• nirnay: the library as shipped (v0.2: logprobs confidence, escalation below 0.8).

Task	Data	Languages	Categories	Test n
Support routing	352 realistic support messages (AI-drafted, see limitations)	Hindi, Hinglish	11	242
Voice-assistant domain	Google Hinglish-TOP (same request in two forms)	English, Hinglish	8	240
Voice-assistant scenario	Amazon MASSIVE (same request in four languages)	English, Hindi, Bengali, Tamil	18	504
Review sentiment	AI4Bharat IndicSentiment (parallel reviews)	English, Hindi, Bengali, Tamil	2	480
Tweet sentiment	SemEval-2020 SentiMix (real code-mixed tweets)	Hinglish	3	150

Results with label descriptions

Task	Default	nirnay	Diff (95% CI)	Default failed	₹/1k default → nirnay	p50	Escalated
Support routing	90.5%	90.5%	0.0 (−3.3, +3.3)	3.3%	42.90 → 18.03	2.19 → 0.22 s	17.8%
Voice domain	95.8%	95.8%	0.0 (−1.2, +1.2)	1.2%	35.20 → 8.19	2.50 → 0.23 s	5.8%
Voice scenario	86.1%	89.9%	+3.8 (+1.6, +6.2)	5.4%	47.84 → 29.33	3.21 → 0.23 s	22.6%
Review sentiment	97.5%	99.4%	+1.9 (+0.8, +3.1)	1.5%	39.45 → 4.58	2.86 → 0.26 s	2.3%
Tweet sentiment	65.3%	66.7%	+1.3 (−4.7, +6.7)	5.3%	56.22 → 18.45	3.55 → 0.40 s	26.7%
All (weighted)	89.7%	91.5%	+1.9 (+0.8, +2.9)	3.3%	43.51 → 16.14	2.84 → 0.25 s	13.7%

Bold = statistically significant (paired; exact McNemar p < 0.05, 95% bootstrap CI). Typical = median (p50). Slow-case (p95) times improve less, 1.4–5× (8.5 s → 5.7 s overall), because escalated decisions wait for thinking mode.

Caveats

• Failed answers. The default's unanswered decisions count as wrong. On the decisions it did answer it scored 93.6 / 97.0 / 91.0 / 98.9 / 69.0% (above nirnay on four of five tasks).
• Support routing data is synthetic: drafted by an AI and judged realistic by a native speaker, with labels from the drafter. The other four tasks use public human-labelled data.
• One run per configuration; Sarvam sampling isn't deterministic (the logprobs themselves were identical in our few repeat checks). The 0.8 escalation threshold was chosen on the dev splits; the test sets are out-of-sample for it.
• The parser fixes favoured the default: teaching the parser native-script letters raised the default's scores by up to 11 points, and all of its replies were re-scored that way.
• Only the with-descriptions condition was run for v0.2. Without label descriptions, v0.1 was within noise of the default and saved less; expect the same for v0.2.
• Prices are Sarvam's published rates as of 30 Sep 2026 (₹29.28 input / ₹10.98 cached / ₹73.20 output per million tokens). Sarvam's V2 endpoint reported no prompt-cache hits, so v0.2's input is billed at the full rate.

Votes (v0.1) vs logprobs (v0.2)

v0.1 measured confidence by asking for 10 answers and counting; v0.2 reads the model's probabilities from one answer. Same 1,616 test decisions, with descriptions:

Test, n=1,616	Votes (v0.1)	Logprobs (v0.2)
Final accuracy	91.2%	91.5%
Separates right from wrong (AUROC of the fast-path confidence, higher is better)	0.794	0.887
Calibration error of the fast-path confidence (ECE, lower is better)	0.052	0.032
Confident (≥ 0.9) but wrong on the fast path	76	45
Escalated to thinking mode	9.9%	13.7%
Cost per 1,000 decisions	₹11.97	₹16.14
Typical / slow-case (p50 / p95) time	0.18 / 5.2 s	0.25 / 5.7 s

• v0.2's confidence is clearly better: higher AUROC on all five tasks, and about 40% fewer "very sure but wrong" answers. That is why it is the default: the confidence is what you act on (escalate, abstain, send to a human).
• v0.1 is cheaper and a little faster. v0.2 escalates more often, and its V2 calls get no prompt caching, so it costs ~35% more. Final accuracy is the same (difference not significant, p = 0.54). If cost matters more than confidence quality, use Decider(confidence="votes").
• On dev data, the 0.8 threshold sent decisions to thinking mode that the fast path got right only 47% of the time; the rest were right 92% of the time.

The v0.1 charts are kept in assets/benchmarks/.

Reproducing

pip install -e ".[dev]"
python experiments/prepare_bench.py                       # builds data/bench/ from the public datasets in data/raw/
python experiments/bench_run.py --system default --splits test   # Sarvam default; needs SARVAM_API_KEY
python experiments/bench_v2.py                            # nirnay v0.2 on the test sets (~₹20, ~1 h)
python experiments/analyze_bench_v2.py                    # v0.2 vs default vs v0.1
python experiments/make_v2_charts.py                      # v0.2 charts
python experiments/logprobs_dev.py                        # logprobs on the dev splits (threshold choice)
python experiments/analyze_logprobs_dev.py
python experiments/bench_run.py --system nirnay           # v0.1 (votes)
python experiments/bench_analyze.py                       # v0.1 tables
python experiments/audit_bench.py                         # v0.1 independent recomputation + significance

Dataset downloads (place under data/raw/): Hinglish-TOP (Apache-2.0), MASSIVE 1.1 (CC BY 4.0), IndicSentiment (licence not stated on its dataset card), SemEval-2020 SentiMix (CC BY 4.0). The support-routing set is private.

Roadmap

• Lower v0.2's cost: a cheaper escalation threshold chosen on fresh dev data, and prompt caching on V2 if Sarvam enables it.
• "Auto mode": detect decision-shaped calls inside an agent and route them through nirnay.
• Async client and a response cache.

Related work

The escalate-when-unsure idea builds on LLM-cascade research, notably Mixture-of-Thought cascades (ICLR 2024). nirnay applies it to Sarvam using Sarvam's own thinking switch as the two tiers, so everything stays on one India-hosted provider.

Development

Copy .env.example to .env and add your Sarvam API key.

pip install -e ".[dev]"
pytest
ruff check .

License

Released under the MIT License: see the LICENSE file.