nunchi-drift 0.1.1

Decision-stability measurement for LLMs: run the same decision N times, get a flip rate with a bootstrap CI, and compare models against their own noise floor.

nunchi-drift

Cassette tests can't see a model that flips its decision 20% of the time.

Most LLM testing makes variance disappear — retries until green, temperature=0, VCR cassettes that freeze one lucky sample. nunchi-drift does the opposite: it treats run-to-run variance as the thing to measure. Give a model the exact same decision N times and get an answer you can defend statistically:

• flip rate — P(two runs disagree), with a bootstrap 95% CI
• cross-model distance vs. each model's own noise floor — "these two models behave differently" only counts as a signal when it exceeds both models' self-disagreement, and a permutation test puts a p-value on it
• pass^k (τ-bench vocabulary) when you do have a success oracle

Zero dependencies. Stdlib only. Works with Anthropic, OpenAI, GLM, Qwen, DeepSeek, or any OpenAI-compatible endpoint.

Live demo: a 7-model leaderboard (Claude / GPT / GLM families) measured with exactly these metrics — flip rates, CIs, statistical ties, and cross-model signals over byte-identical inputs.

Quickstart

from nunchi_drift import DecisionTask, measure, compare

task = DecisionTask(
    system="You are a refund-processing agent for an online store.",
    prompt="Customer requests a refund for a product delivered 15 days ago. "
           "Policy window is 14 days. The customer is polite and a longtime user.",
    actions=("approve", "reject", "escalate"),
)

r = measure(task, model="claude-haiku-4-5", runs=10)
print(r.summary())
# model=claude-haiku-4-5 n=10/10 flip_rate=0.356 [0.178, 0.489] majority=escalate (60%)

r.stable(threshold=0.1)          # False — this decision point is flaky
r.distribution                   # {'escalate': 0.6, 'approve': 0.3, 'reject': 0.1}

c = compare(task, "claude-haiku-4-5", "gpt-4o-mini", runs=10)
print(c.summary())
# claude-haiku-4-5 vs gpt-4o-mini: cross=0.640 floor=0.356 SIGNAL (p=0.014, permutation n=1000)

The action enum is enforced through tool/function calling — every sample is directly countable, no output parsing.

Why the statistics are not optional

These metrics were hardened in the measurement experiments behind the leaderboard, where every naive shortcut failed in a documented way:

Naive approach	What went wrong	What this library does
Baseline-first drift (compare run 1 vs rest)	A single outlier first run inflated drift 5× (0.40 vs 0.08)	mean-pairwise: no run is privileged
Aggregate distributions only	Two agents swapping actions → aggregate unchanged, drift 0	per_agent_flip_rates sees every flip
Point estimates, no CI	Leaderboard gaps of 0.003 read as rankings; with n=10 they were statistical ties	bootstrap CI on everything, ties reported as ties
"Model A ≠ model B" by eyeballing	Cross distance below either model's own noise floor means nothing	noise-floor gate + stratified permutation test

Cost, honestly

Repeated sampling is inherently uncacheable — that's the measurement. Mitigations built in or recommended:

• Opt-in, not blanket: measure decision points (a router, a guardrail verdict, a trade action), not entire suites. One task × 10 runs on a small model costs about a cent.
• workers=2 by default — bursts of identical calls are the worst case for rate limiters, and a failed run is lost signal (failures reduce n and are reported, never silently ignored).
• Run nightly, not per-commit, for regression tracking.

API sketch

Call	Returns
measure(task, model, runs=10)	StabilityResult: .flip_rate .ci .distribution .majority .entropy .stable(threshold)
compare(task, model_a, model_b)	CompareResult: .cross .noise_floor .signal .p_value
metrics module	drift_score, mean_pairwise_drift, bootstrap_within_ci, permutation_test_cross, per_agent_flip_rates, pass_k, … all pure functions over your own data
caller= parameter	swap the provider layer (custom endpoints, replays, tests)

Multi-agent / multi-scenario batteries (the full leaderboard workflow with per-scenario stratification) currently live in the parent project and are the v0.2 extraction target, together with a scheduled model-change canary.

Related work

pass^k is from τ-bench. The CI agenda follows "Adding Error Bars to Evals". For continuous behavior-change auditing see "An Auditing Test to Detect Behavioral Shift in Language Models" — the statistical backbone we build toward in the canary. Unlike semantic-entropy methods, nunchi-drift needs no logits: enum-forced decisions make the distribution directly observable, which is what keeps it cheap and provider-agnostic.

nunchi (눈치): the Korean art of reading a room — noticing what shifted without being told.

MIT license.