evallint

Audits LLM eval datasets for the flaws that make evaluations silently lie to you.

Everyone tests their model. Almost nobody tests whether their test set is any good.

The problem

A bad eval set doesn't fail loudly — it returns a number, and the number looks fine. Suppose a 20-case support-triage eval reports 85%: if 14 of those cases are billing questions, three of them are the same "I was charged twice" question reworded, and most are easy enough that any competent model passes, then that 85% is mostly measuring one scenario, counted several times, on cases that can't tell a strong model from a weak one.

The model looks fine and ships. Production disagrees. Nothing in the eval was wrong enough to notice — which is exactly why it needs auditing separately from the model.

What it checks (v1)

Discrimination failure. Runs each case against two or more models of known differing capability and flags cases where every model produces the same verdict. Those cases carry no evidence about which model is better, so an eval with 200 cases where 140 don't discriminate is really a 60-case eval wearing a 200-case costume. Results split into ceiling (every model passes — too easy), floor (every model fails — check the reference answer and grader before blaming difficulty), and inverted (a weaker model passed where a stronger one failed — usually a broken reference answer, and worth more than either).

Near-duplicate detection. Embeds each case's input with a local sentence-transformers model, computes pairwise cosine similarity, and groups linked cases into clusters. Matters because duplicates make an eval look bigger and broader than it is while silently multiplying one scenario's weight in the aggregate score — a set of 20 cases with three phrasings of the same question is really 18 scenarios, and the repeated one counts triple.

Class imbalance and basic stats. Reports class distribution, imbalance ratio, and per-class counts. Matters because aggregate accuracy hides failure on rare classes: a model that handles the 70% majority class well and everything else badly still scores ~70% overall. A class becomes unmeasurable two independent ways — its share is too small to move the aggregate, or its count is too small for its own accuracy to mean anything — so both are checked. In a 10,000-case set a 1% class still has 100 cases and is perfectly measurable; in a 20-case set a 5% class has one.

Install

Requires Python 3.12+.

pip install evallint                # core: imbalance + stats, ~33 MB installed
pip install 'evallint[embeddings]'  # adds the duplicate check (pulls torch)

Why the split. The duplicate check needs sentence-transformers, which pulls in torch — roughly 2 GB. Making that mandatory would mean downloading torch to get class-imbalance statistics, so it's an optional extra instead. The core install is numpy, click and rich.

Without the extra, the duplicate check fails with instructions rather than a traceback, and the other checks are unaffected:

The duplicate check needs sentence-transformers, which is an optional extra
because it pulls in torch (~2GB).

    pip install 'evallint[embeddings]'

Alternatively, pass your own embedder — DuplicateCheck(embedder=my_fn) takes any
callable mapping a sequence of strings to a 2-D array, so no extra is needed. Or
skip this check entirely: `evallint --skip-duplicates PATH`.

With the extra, the first run downloads all-MiniLM-L6-v2 (about 90 MB); every run after that is offline.

From source

git clone https://github.com/BhagyasriUddandam/evallint && cd evallint
uv sync    # exact versions from uv.lock, installs the `evallint` command

uv sync reproduces the environment from uv.lock and installs evallint in editable mode, so there is no separate install step. It includes the embeddings extra, so the full test suite runs.

Usage

evallint examples/sample_evalset.jsonl

evallint --json evalset.jsonl > report.json      # machine-readable, for CI
evallint --skip-duplicates evalset.jsonl         # no model download
evallint --duplicate-threshold 0.92 set.jsonl    # stricter: only very close pairs
evallint --min-class-share 0.05 set.jsonl        # tolerate smaller classes
evallint --min-class-count 20 set.jsonl          # demand more cases per class
evallint --map input=question set.jsonl          # your column names, not ours

Your column names

evallint's fields are id, input, expected, label — and almost no real eval set uses those names. Five widely-used public datasets were tried cold and all five failed on line 1, because they call the input question, prompt, ctx or turns. Common aliases are now inferred, so most files load unaided:

$ evallint gsm8k.jsonl --skip-duplicates
  · field map: input <- 'question' (inferred)
  · field map: expected <- 'answer' (inferred)

The mapping is always printed. That is the point: a loader that quietly picks the wrong column produces a confident report about the wrong data, which is the exact failure this tool exists to catch. So the rules are deliberately timid:

• A column already named input / expected / label is used as-is.

• Otherwise aliases are considered, and exactly one candidate is accepted.

• Two candidates is an error naming both, not a coin flip. TruthfulQA has category and type; evallint refuses and tells you to pick:

  Error: cannot tell which column is 'label': category, type are all plausible.
  Choose one explicitly, e.g. --map label=category
  

• Alternatives that were passed over are reported too. HellaSwag has a column literally named label, but there it means the index of the correct ending, not a class. evallint cannot know that, so it uses the canonical name and says what else was available:

  · field map: label <- 'label'  [WARNING: 'label' used as-is, but
    activity_label also present — check the meaning]
  

  That warning is load-bearing. Taking label gives "4 classes, ratio 1.2:1", which is meaningless — it's counting answer positions. --map label=activity_label gives 39 classes, ratio 14.0:1, which is the real finding. Same file, opposite conclusion, and the only thing standing between them is that evallint said which column it used.

Explicit --map always wins, and a mapped field is never overwritten by a leftover column of the same name (the displaced column is kept in metadata and reported, not dropped).

Multi-turn sets are a genuine limitation rather than a naming one: MT-Bench's prompt is a list of turns, and evallint models one case as one input string. No --map fixes that; it exits 3 and says so.

Configuration file

Thresholds can live in version control instead of a CI step, so they get reviewed like any other decision. evallint.toml, or a [tool.evallint] section in pyproject.toml:

# evallint.toml
skip_duplicates  = true
min_class_count  = 10
min_class_share  = 0.15
duplicate_threshold = 0.90
fail_on          = "warning"

Discovery searches upward from the eval-set file, so a config at the repository root applies to a set in a subdirectory. evallint.toml wins over pyproject.toml in the same directory. Precedence is CLI flag > config file > default, and that is decided by whether a flag was typed, not by its value — --min-class-count 5 overrides a config even though 5 is also the default.

--config PATH points at a specific file; --no-config ignores any file.

An unknown key or a wrong type is a hard error, not a warning:

Error: evallint.toml: unknown setting(s) min_class_shrae.
Valid settings: duplicate_threshold, fail_on, min_class_count, min_class_share, skip_duplicates

A silently ignored typo means a team believes a threshold is in force when it is not — the same quiet wrongness this tool exists to report.

Logging

-v logs progress to stderr, -vv adds detail. Quiet by default, and always stderr so --json > report.json stays clean.

INFO evallint.cli: using config /repo/evallint.toml
INFO evallint.io: loaded 20 cases from evalset.jsonl (jsonl) in 0.002s
INFO evallint.cli: running imbalance on 20 cases

As a library, evallint attaches a NullHandler and emits nothing until your application configures logging — it never configures logging on your behalf.

Using it as a CI gate

- run: evallint evalset.jsonl --fail-on warning

3 is deliberately separate from 1. "Your eval set has warnings" and "I could not finish auditing your eval set" need different reactions in a pipeline, and collapsing them would let an incomplete audit pass as a clean one. So a run where the embedding model fails to download exits 3 even under --fail-on never — you still get the report for the checks that did succeed, but the build does not go green on a one-third audit. An explicit --skip-duplicates is not treated as incomplete, because that is you choosing the scope.

The default is never, which always exits 0. That is deliberate rather than lazy: every check reports that a finding is a prompt to look, not a verdict, so failing a build by default would assert something this tool explicitly declines to claim. CI opts in.

With --json, the decision is in the payload too, so a consumer never has to re-derive it:

"gate": {"fail_on": "warning", "exit_code": 1, "tripped": true, "incomplete": []}

The explanation goes to stderr, so evallint --json ... > report.json still produces a clean file.

Accepts .jsonl, .ndjson, .json, and .csv. Only input is required: id is generated as case_1, case_2... when absent, expected and label are optional, and unrecognised fields are preserved rather than dropped.

{"id": "billing_001", "input": "I was charged twice.", "expected": "Refund the extra.", "label": "billing"}

Discrimination needs model verdicts, so it runs from the library rather than the CLI:

from evallint.io import load
from evallint.checks import DiscriminationCheck

def score(case, model) -> bool:
    """Your code, your provider. True if `model` got `case` right."""
    return grade(my_client.complete(model=model, prompt=case.input), case.expected)

# models ordered WEAKEST FIRST — that ordering is what makes inversion detectable
result = DiscriminationCheck(score, ["small-model", "large-model"]).run(load("evalset.jsonl"))
print(result.summary, result.stats["non_discriminating_share"])

The scorer is injected so evallint never talks to a provider: it works with any stack, you define what "correct" means, and the test suite needs no API key.

Repeat your runs. Model sampling makes per-case verdicts unstable, and temperature=0 is not universally available — current frontier models including Claude Opus 5 and Sonnet 5 reject the parameter outright. Pass repeats=N to score each case N times; any case whose verdict is not unanimous is reported as unstable and excluded from the counts:

result = DiscriminationCheck(score, ["small-model", "large-model"], repeats=3).run(eval_set)
result.stats["n_unstable"]          # cases whose verdict changed between runs
result.stats["unstable_case_ids"]   # ...and which ones
result.stats["n_measured"]          # the denominator the other figures actually use

On the bundled 20-case example, three repeats found 8 of 20 verdicts (40%) non-reproducible, and only one of the inversions a single run reported survived all three. The default repeats=1 keeps cost down but cannot detect this at all — which is stated in the check's own limitations.

Run it concurrently. Scoring is I/O-bound — one API call per case per model per repeat — so serial execution is the difference between minutes and hours. Measured on a 40 ms stand-in call:

DiscriminationCheck(score, models, repeats=3, max_workers=16).run(eval_set)

⚠️ Your scorer must be thread-safe to raise this above 1. evallint cannot check that for you — the scorer is your code, and a shared client, session, file handle or non-locking cache inside it will corrupt results quietly rather than crash. The default is 1, and it takes a code path with no threads in it at all, so you are unaffected until you opt in. If a concurrent run disagrees with a serial run on the same data, suspect the scorer before the eval set.

Bound the cost with sample. This is the only expensive check, so on a large set the real choice is often between auditing a sample and not auditing at all:

DiscriminationCheck(score, models, sample=100, sample_seed=0).run(eval_set)

sample_seed makes the draw reproducible, so a re-run compares against the same cases instead of a fresh subset. A sampled run says so everywhere it could be mistaken for a full one — the summary leads with it, and it is the first limitation reported:

SAMPLE of 100 of 1319 cases x 2 models: 4 discriminate, 96 do not (96%)

  · Only 100 of 1319 cases were scored (random sample, seed 0). Every figure
    above describes THAT SAMPLE, not your eval set. A clean sample does not mean
    a clean set, and a broken case that was not drawn is not reported here —
    absence of a finding is not evidence.

The remaining two knobs. pass_threshold (default 0.5) is the score at or above which a float scorer counts as a pass; it is ignored for bool scorers. max_non_discriminating_share (default 0.5) is the share of measured cases above which the check warns.

Does this check actually find anything?

Yes, on a benchmark this project did not write. Run against 100 GSM8K cases with Haiku 4.5 and Opus 5, three repeats, and no LLM judge — GSM8K answers end in #### n, so grading is a numeric comparison nobody has to trust:

100 cases x 2 models x 3 repeats: 2 discriminate, 95 do not (98%), 3 not reproducible

98% of a widely-cited reasoning benchmark carries no evidence about which of those two models is better. Both sit at the ceiling: 98.0% and 98.7%. Repeats also killed one of the two inversions a single run reported, which is exactly why the check refuses to classify unstable cases.

Full write-up, including what it does not show: docs/findings-gsm8k.md. All 800 API responses are committed, so the numbers recompute for $0.00.

Example output

Real output from evallint examples/sample_evalset.jsonl against the bundled deliberately flawed example set, at an 88-column terminal:

evallint  examples/sample_evalset.jsonl

imbalance
  20 cases across 4 classes, imbalance ratio 14.0:1
  WARN  class imbalance ratio is 14.0:1 — 'billing' has 14 cases, 'bug_report' has 1.
        Aggregate accuracy is dominated by 'billing'
  WARN  class 'password_reset' is under-represented: only 4 cases, so its accuracy can
        only land on 5 distinct values
        password_001, password_002, password_003, password_004
  WARN  class 'refund' is under-represented: 5.0% of labelled cases (below 10%); only 1
        case, so its accuracy can only land on 2 distinct values
        refund_001
  WARN  class 'bug_report' is under-represented: 5.0% of labelled cases (below 10%);
        only 1 case, so its accuracy can only land on 2 distinct values
        bug_001

duplicates
  20 cases, 3 near-duplicate clusters covering 7 cases — about 16 distinct scenarios
  (20% redundant)
  WARN  3 near-identical cases (pairwise cosine 0.81-0.91)
        billing_001, billing_002, billing_003
  WARN  2 cases have identical input text
        billing_004, billing_014
  WARN  2 near-identical cases (pairwise cosine 0.98)
        password_001, password_002

7 warnings across 2 checks

Not run
  · discrimination — needs a scoring function that runs your models, so it cannot run
    from the CLI. Use the library: DiscriminationCheck(scorer, ['weak-model',
    'strong-model']).run(load(path))

What this audit cannot tell you
  imbalance
    · This check measures whether the class distribution is UNEVEN, not whether it is
      WRONG. If production traffic really is 70% billing, a 70% billing eval may be
      exactly right. Only you know the target distribution.
    · Classes are read from the 'label' field only. If your categories live in metadata
      or are implied by the text, this check sees nothing.
    · The thresholds are conventions, not statistics. They are a prompt to look, not a
      verdict.
    · Input length is measured in characters, not tokens, so it is a rough proxy for
      cost and complexity.
  duplicates
    · Similarity is computed on the 'input' field alone. Two cases with the same input
      but different 'expected' values may be deliberate (testing consistency, or a known
      ambiguity) — read a cluster before deleting from it.
    · The threshold is a heuristic, not a decision boundary. It was checked against one
      small example set where duplicates and non-duplicates separate cleanly; a domain
      with formulaic phrasing (SQL, legal boilerplate, templated prompts) will push
      unrelated cases above it.
    · Clusters are connected components: if A is similar to B and B to C, all three are
      grouped even when A and C are not themselves similar. The pairwise range reported
      for each cluster shows when this has happened.
    · Inputs longer than the model's token limit (256 word pieces for all-MiniLM-L6-v2)
      are truncated before embedding, so two long cases that differ only in their
      endings can look identical to this check.
    · Every pair is compared, so time and memory grow with the square of the case count.
      Fine for a few thousand cases, not for a corpus.

Two things in that output are deliberate. Discrimination is listed under "Not run" rather than omitted — the CLI covers two of three checks and says so, because a report that quietly covered two thirds of the tool would be the same kind of silent lie this project exists to catch. And "What this audit cannot tell you" is not suppressible; there is no --brief flag that hides it. A clean report is exactly when a reader is most likely to over-trust it.

What this does NOT do

• It is not an eval runner. It does not execute your eval, call your model, or score responses. It audits the dataset those tools consume.
• It does not replace promptfoo, DeepEval, LangSmith, or OpenAI Evals, and is not trying to. Those run evals; they mostly assume the eval set is sound. evallint checks that assumption. Use it alongside them, upstream.
• It audits the set, not the model. Nothing here tells you whether your model is good. A clean evallint report means your eval is capable of measuring something, not that the thing it measured is passing.
• It does not tell you what to delete. Every finding is a prompt to look, not a verdict. Easy cases may be deliberate regression guards; duplicate inputs with different expected values may be testing consistency on purpose.
• It cannot tell you whether your distribution is wrong, only whether it is uneven. If production traffic really is 70% billing, a 70% billing eval may be exactly right.
• The thresholds are conventions, not statistics. 0.85 cosine, 10% share, 5 cases — all defaults, all configurable, none derived from theory. The duplicate threshold was checked against one 20-case example where duplicates and non-duplicates separate at 0.70 vs 0.81.
• Duplicate detection is O(n²) in time and memory. Fine for a few thousand cases, not for a corpus.
• Discrimination depends entirely on the model pair you choose. Two models closer in capability than you assumed will make a good eval look non-discriminating, and the check cannot tell the difference. It also cannot verify that your "weak" model is actually weaker — it can only flag when the numbers contradict the ordering you declared.
• Only three checks. Label leakage and ambiguous ground truth are real problems and are not implemented.

This section is a feature. Every check also reports its own limitations at runtime, and CheckResult raises if constructed without them — "state what you cannot tell the user" is enforced by the type, not left to discipline. A tool that states its uncertainty is more trustworthy than one that doesn't.

Roadmap (v2+)

• Label-leakage check — detect cases where the expected answer is recoverable from the input itself, which inflates scores without measuring capability.
• Ambiguous-ground-truth check — flag cases where the reference answer is one of several defensible responses, so a correct model gets marked wrong.
• Multi-turn eval sets. One case is currently one input string, so a conversational set like MT-Bench (whose prompt is a list of turns) cannot load at all. Supporting it is a schema change, not a mapping one.

Deliberately not planned: a web UI, a database, a hosted service, or becoming an eval runner.

Development

uv run pytest    # 295 tests

Every check is tested both ways: it must fire on known-bad input and stay quiet on known-good input. A check that warns about everything is worth as little as one that warns about nothing. Tests avoid re-testing other people's libraries — the duplicate check's clustering logic runs against a deterministic fake embedder, with two separate tests exercising the real model end to end.

src/evallint/
  schema.py       EvalCase / EvalSet + validation
  io.py           load JSONL / JSON / CSV
  mapping.py      your column names -> evallint's fields, always reported
  config.py       evallint.toml / [tool.evallint] discovery
  checks/
    base.py       Check interface; CheckResult refuses to exist without limitations
    discrimination.py · duplicates.py · imbalance.py
  report.py       CheckResults -> text (rich) or JSON
  cli.py          `evallint PATH`