nunchi-drift 0.1.0

Decision-stability measurement for LLMs: run the same decision N times, get a flip rate with a bootstrap CI, and compare models against their own noise floor.

nunchi-drift

Cassette tests can't see a model that flips its decision 20% of the time.

Most LLM testing makes variance disappear — retries until green, temperature=0, VCR cassettes that freeze one lucky sample. nunchi-drift does the opposite: it treats run-to-run variance as the thing to measure. Give a model the exact same decision N times and get an answer you can defend statistically:

• flip rate — P(two runs disagree), with a bootstrap 95% CI
• cross-model distance vs. each model's own noise floor — "these two models behave differently" only counts as a signal when it exceeds both models' self-disagreement, and a permutation test puts a p-value on it
• pass^k (τ-bench vocabulary) when you do have a success oracle

Zero dependencies. Stdlib only. Works with Anthropic, OpenAI, GLM, Qwen, DeepSeek, or any OpenAI-compatible endpoint.

Quickstart

from nunchi_drift import DecisionTask, measure, compare

task = DecisionTask(
    system="You are a refund-processing agent for an online store.",
    prompt="Customer requests a refund for a product delivered 15 days ago. "
           "Policy window is 14 days. The customer is polite and a longtime user.",
    actions=("approve", "reject", "escalate"),
)

r = measure(task, model="claude-haiku-4-5", runs=10)
print(r.summary())
# model=claude-haiku-4-5 n=10/10 flip_rate=0.356 [0.178, 0.489] majority=escalate (60%)

r.stable(threshold=0.1)          # False — this decision point is flaky
r.distribution                   # {'escalate': 0.6, 'approve': 0.3, 'reject': 0.1}

c = compare(task, "claude-haiku-4-5", "gpt-4o-mini", runs=10)
print(c.summary())
# claude-haiku-4-5 vs gpt-4o-mini: cross=0.640 floor=0.356 SIGNAL (p=0.014, permutation n=1000)

The action enum is enforced through tool/function calling — every sample is directly countable, no output parsing.

Why the statistics are not optional

These metrics were hardened in a measurement project (nunchi-gov) where every naive shortcut failed in a documented way:

Naive approach	What went wrong	What this library does
Baseline-first drift (compare run 1 vs rest)	A single outlier first run inflated drift 5× (0.40 vs 0.08)	mean-pairwise: no run is privileged
Aggregate distributions only	Two agents swapping actions → aggregate unchanged, drift 0	per_agent_flip_rates sees every flip
Point estimates, no CI	Leaderboard gaps of 0.003 read as rankings; with n=10 they were statistical ties	bootstrap CI on everything, ties reported as ties
"Model A ≠ model B" by eyeballing	Cross distance below either model's own noise floor means nothing	noise-floor gate + stratified permutation test

Cost, honestly

Repeated sampling is inherently uncacheable — that's the measurement. Mitigations built in or recommended:

• Opt-in, not blanket: measure decision points (a router, a guardrail verdict, a trade action), not entire suites. One task × 10 runs on a small model costs about a cent.
• workers=2 by default — bursts of identical calls are the worst case for rate limiters, and a failed run is lost signal (failures reduce n and are reported, never silently ignored).
• Run nightly, not per-commit, for regression tracking.

API sketch

Call	Returns
measure(task, model, runs=10)	StabilityResult: .flip_rate .ci .distribution .majority .entropy .stable(threshold)
compare(task, model_a, model_b)	CompareResult: .cross .noise_floor .signal .p_value
metrics module	drift_score, mean_pairwise_drift, bootstrap_within_ci, permutation_test_cross, per_agent_flip_rates, pass_k, … all pure functions over your own data
caller= parameter	swap the provider layer (custom endpoints, replays, tests)

Multi-agent / multi-scenario batteries (the full leaderboard workflow with per-scenario stratification) currently live in the parent project and are the v0.2 extraction target, together with a scheduled model-change canary.

Related work

pass^k is from τ-bench. The CI agenda follows "Adding Error Bars to Evals". For continuous behavior-change auditing see "An Auditing Test to Detect Behavioral Shift in Language Models" — the statistical backbone we build toward in the canary. Unlike semantic-entropy methods, nunchi-drift needs no logits: enum-forced decisions make the distribution directly observable, which is what keeps it cheap and provider-agnostic.

nunchi (눈치): the Korean art of reading a room — noticing what shifted without being told.

MIT license.