tna-lab

Freeze a dataset, score it through trust-no-agent, and diff two runs record by record — or two versions of your agent case by case — locally, with no server and no database.

trust-no-agent answers "does this one record pass?". tna-lab answers "across a frozen dataset, under a given judge, what changed?" — it ingests records content-addressed and idempotently, freezes immutable named snapshots, scores them by calling trust-no-agent's evaluate() once per (record, evaluator) pair, and compares two runs into a per-record diff that names every improvement and regression individually instead of reporting an aggregate that can hide one. It can also compare two agent versions: the outputs each gave for the same questions, paired case by case across two datasets, under one fixed judge. It computes no score of its own: trust-no-agent is the only scorer, and every EvalResult is persisted exactly as returned, status, fingerprint, token counts and all. Storage is a directory of JSON files under .tna-lab/.

v0.2.0 is the v1 slice plus the one item its constitution deferred and this release took up: comparing two sets of outputs for the same inputs. The reasoning behind every rule it follows — one scorer, library-first, no provider abstraction, append-only versioning — is in CONSTITUTION.md. The spec, design-decision log and task breakdown behind each feature are in specs/001-dataset-run-compare/ and specs/002-compare-outputs/.

Install

uv add tna-lab      # or: pip install tna-lab

Python 3.12 exactly. The only runtime dependency is trust-no-agent.

Quickstart

Scoring needs the same two model variables trust-no-agent itself reads, plus NVIDIA_API_KEY for --live:

export JUDGE_MODEL=nvidia/nemotron-3-super-120b-a12b
export GENERATOR_MODEL=nvidia/nemotron-3-super-120b-a12b
export NVIDIA_API_KEY=...

1. Ingest records. Identity is a hash of a record's content, so re-ingesting the same file is a no-op:

cat > records.jsonl <<'EOF'
{"input": "What is the refund window?", "output": "30 days from delivery.", "expected": "30 days.", "contexts": ["Refunds: 30 days from delivery, unopened."]}
{"input": "Do you ship internationally?", "output": "No, US only.", "expected": "US only.", "contexts": ["Shipping: currently US only."]}
EOF

tna-lab ingest records.jsonl --dataset smoke     # added=2, already_present=0
tna-lab ingest records.jsonl --dataset smoke     # added=0, already_present=2

2. Freeze a snapshot. Immutable and independently addressable as smoke@v1; later ingestion cannot change it, and re-freezing an existing tag is an error, never an overwrite:

tna-lab freeze smoke v1
# smoke@v1 snapshot_id=9a6d995b0cc5... records=2

3. Score it. One trust-no-agent evaluate() call per (record, evaluator) pair. Judge calls are cached under .tna-lab/cache/, so a repeat is served from disk instead of billed again:

tna-lab run smoke@v1 --evaluator tna.ragas.faithfulness --live
# 20260928T193652Z-c79e0f03

Every result is persisted whatever its status — an error, skipped or invalid_output result is kept as returned, never dropped or coerced to a number.

4. Compare two runs. Both must have scored the same snapshot with the same evaluator; a mismatch is a named error, never a silent diff across different data:

tna-lab compare <run_id_1> <run_id_2> --evaluator tna.ragas.faithfulness

record        status  a       b       delta    class      fingerprint a → b            tokens a → b
569cddb2252c  ok      1.0000  1.0000  +0.0000  unchanged  0659fb5515c9 → 0659fb5515c9  1919/582 → 1919/582
5c8fa5fd7f4f  ok      1.0000  1.0000  +0.0000  unchanged  e13108284261 → e13108284261  1915/710 → 1915/710
summary: improved=0 regressed=0 unchanged=2 mean_score_delta=+0.0000 pass_rate_delta=-

Both runs' judge fingerprints and token counts travel with the diff, so a score that moved because the judge moved is visible as such, and so is what each score cost.

The walkthrough in full, including the failure paths, is specs/001-dataset-run-compare/quickstart.md.

Compare two agent versions by case

compare holds the records fixed and lets the judging differ: did my judge change move my scores?. compare-cases holds the judging fixed and lets the outputs differ: did my agent get better?

Ingest each version's outputs into its own dataset. A dataset's record set only ever grows, so two versions in one dataset would put every case in the run twice. Then freeze each, and score both with the same evaluator and judge:

tna-lab ingest agent-v1.jsonl --dataset agent-v1 && tna-lab freeze agent-v1 base
tna-lab ingest agent-v2.jsonl --dataset agent-v2 && tna-lab freeze agent-v2 base
tna-lab run agent-v1@base --evaluator tna.ragas.faithfulness --live    # prints <run_v1>
tna-lab run agent-v2@base --evaluator tna.ragas.faithfulness --live    # prints <run_v2>

tna-lab compare-cases <run_v1> <run_v2> --evaluator tna.ragas.faithfulness

A case is a record's input plus its expected. The output is free to differ, since that is what is being compared, and so are the retrieved contexts, which in trust-no-agent's own model are agent output too. The report opens by saying how it matched, so it can never be mistaken for compare's. This is real output from a live run, scoring tna.ragas.faithfulness for two versions of a three-question agent:

matched by case (input + expected) across snapshots — no same-snapshot guarantee
run a: 20260930T190306Z-5bd26ead (agent-v1@base)  run b: 20260930T190342Z-913b9a2f (agent-v2@base)  judge: nvidia/nemotron-3-super-120b-a12b
case          status  a       b       delta    class      ctx  out  fingerprint a → b            tokens a → b
960fcf6372d0  ok      0.0000  1.0000  +1.0000  improved   -    yes  e1a5a34e8af4 → e13108284261  1904/542 → 1914/560
eba9b124300b  ok      1.0000  1.0000  +0.0000  unchanged  yes  -    0659fb5515c9 → 89a0850639cd  1919/842 → 1927/544
unmatched in a (1):
  f277416ee994  no_counterpart  ok  0.0000  fingerprint 86e8cf884297  tokens 1903/544
unmatched in b (1):
  e5ba707427ec  no_counterpart  ok  1.0000  fingerprint 587f59c30be0  tokens 1929/858
ambiguous (0):
summary: paired=2 improved=1 regressed=0 unchanged=1 contexts_changed=1 output_changed=1 unmatched_a=1 unmatched_b=1 ambiguous=0 mean_score_delta=+0.5000 pass_rate_delta=-

Two cases paired. The first was a wrong answer in v1 (it claimed to ship worldwide against a US-only context) that v2 fixed, so out is set and it is classified improved. The second has the same answer in both versions but a different retrieved context (ctx), which is still paired, still counted, and flagged. The question only v1 has and the one only v2 has are each listed under their own side, with their own score. The report then accounts for everything it could not pair: a case present on only one side is unmatched, and a case carried by more than one record in a run is ambiguous, and listed, never guessed at. The summary line counts all of it, so "1 improved" can never quietly mean "1 improved, and 40 were not looked at".

Two runs scored under different judge or generator models are refused, with an error naming both. There is no override: a delta across two judges would credit the agent with the judge's variance.

The full walkthrough, including the ambiguity and judge-mismatch paths, is specs/002-compare-outputs/quickstart.md.

Commands

Every command takes --workspace <path> (default ./.tna-lab, never an environment variable) and --json to print the full result as JSON.

As a library

The CLI is argument parsing over these functions; nothing is reachable from one surface only, and no server or UI is involved on any path.

from pathlib import Path

from tna_lab import compare, freeze, ingest, load_run, run
from tna_lab.datasets import load_jsonl

workspace = Path(".tna-lab")

ingest(workspace, "smoke", load_jsonl(Path("records.jsonl")))
snapshot = freeze(workspace, "smoke", "v1")

record = run(workspace, snapshot, ["tna.ragas.faithfulness"])
diff = compare(record, load_run(workspace, "<earlier_run_id>"), "tna.ragas.faithfulness")

print(diff.summary.improved, diff.summary.regressed, diff.summary.mean_score_delta)

Comparing two agent versions by case takes the workspace too, because it reads the records each run scored:

from tna_lab import compare_cases

by_case = compare_cases(workspace, load_run(workspace, "<run_v1>"), load_run(workspace, "<run_v2>"),
                        "tna.ragas.faithfulness")

print(by_case.summary.paired, by_case.summary.improved, by_case.summary.regressed)
for unmatched in by_case.unmatched_a:  # in v1 only: record id, reason, and its own result
    print(unmatched.record_id, unmatched.reason, unmatched.status, unmatched.score)

run() also takes judge (a trust-no-agent JudgeConfig — pass JudgeConfig.from_env(mode="live") for what --live does) and cache_dir, which defaults to <workspace>/cache.

What v0.2.0 deliberately does not do

• Compare across different judges. compare-cases refuses two runs scored by different judge or generator models, and has no override. The raw run records in .tna-lab/runs/ stay readable for anyone who wants to look at such a pair by hand.
• Match cases loosely. "What is X?" and "What is X? " are different cases. So are two questions with different reference answers: a corrected expected leaves that case unmatched on both sides instead of comparing scores against two different yardsticks.
• Trace/OTel ingestion, or any hosted or shared view. Both stay deferred under Article VIII of CONSTITUTION.md. tna-lab ingests already-structured records and runs no server.

Scoring tna.ragas.response_relevancy live additionally needs sentence-transformers, which trust-no-agent keeps in a dependency group of its own; the other three tna.ragas.* metrics need no extra.

License

MIT — see LICENSE.