digline

_{CI is the merge queue's latest run — the gate a change passes on its way to main; a red can mean the queue turned a pull request away while main did not move.}

pip install digline

Did my change make the agent, or the prompt, worse?

An agent calls a model, chooses tools, calls again; an application calls a model with three prompts. You change one thing and run it twice: two different outputs, so reading a few of them cannot tell a regression from the model's own noise.

digline records what the agent or the prompt sends and receives, without changing your code, proposes a first suite from the recordings, and freezes a run you approved as the reference. Every later run is compared with it: answers, steps, the order of the tools, the cost. One suite per prompt, one verdict each.

from digline.core import Contains
from digline.run import Case, Response, Suite

suite = Suite(
    tenant="acme",
    environment="dev",
    name="hello",
    assertions=[Contains(needle="hello")],
    cases=[Case(id="greet")],
)


def target(case: Case) -> Response:
    return Response(output="hello world")

$ digline run --suite suite.py
digline: tenant 'acme' · 1 case × 1 sample = 1 call to the target
digline: target: 1 call · counts: none · cost: not known
2026-10-31T13-26-55-593090-00-00-f849e1e676ebb535
digline: recorded .digline/acme/runs/hello/2026-10-31T13-26-55-593090-00-00-f849e1e676ebb535.json
digline: next: digline promote --suite suite.py --run latest --replacing none  # keeps this run as the reference

$ digline promote --suite suite.py --run latest --replacing none
tenant 'acme' · hello baseline set to 2026-10-31T13-26-55-593090-00-00-f849e1e676ebb535

$ digline run --suite suite.py
digline: tenant 'acme' · 1 case × 1 sample = 1 call to the target
digline: target: 1 call · counts: none · cost: not known
2026-10-31T13-26-55-896575-00-00-f849e1e676ebb535
digline: recorded .digline/acme/runs/hello/2026-10-31T13-26-55-896575-00-00-f849e1e676ebb535.json
digline: next: digline compare --suite suite.py --run latest  # says whether anything got worse

$ digline compare --suite suite.py --run latest
Nothing got worse compared with the reference. Every case could be judged. No case is suspended. The suite is unchanged from the reference.
tenant 'acme', against baseline 2026-10-31T13-26-55-593090-00-00-f849e1e676ebb535

The reference is a file in your repo, reviewed like any other change. The answer is an exit code, so the same commands run in CI.

Coming from an agent or an application that already exists? Five walkthroughs take one from the first recorded call to a gate, every step run: an agent, from the IDE over MCP, Java on Quarkus with Bedrock, Node, Python with three prompts.

Why digline

Most evaluation tools tell you whether an output is below a threshold. digline also tells you whether it is worse than it was — the drift from 0.91 to 0.78 that trips no threshold and is the first thing a user feels.

The suite is Python, not YAML: a judge is an object, a target is a function, and what may leave a perimeter is declared in code — none of which a configuration file expresses without reinventing a language. Where a suite is plain data it can be TOML instead, and the two forms build the same objects — what TOML cannot express, it refuses by name rather than half-supporting. Built for teams shipping LLM features for someone else, who have to show a customer what was tested, when, under which commit, and who approved it.

Wondering how digline differs from promptfoo, DeepEval, or observability platforms? See How digline compares.

Quickstart

With uv (recommended):

uv init && uv add digline

or with pip in an existing environment: pip install digline.

Requires Python 3.12+, which uv fetches for you if you do not have it. On the pip path an older interpreter says ERROR: No matching distribution found with from versions: none, which does not say why — that is what it means.

If you use Claude Code, digline also installs as a plugin with an MCP server — see Claude Code and MCP below.

suite.py — complete and runnable, no API key:

"""suite.py — complete and runnable: no API key, nothing else to install."""

from digline.core import Contains, CostBudget, JudgeReply, LlmRubric
from digline.run import Case, Response, Suite

ANSWERS = {
    "where-is-my-order": "Order 4821 ships Thursday. — Northwind Support",
    "how-do-i-return": "Any item, within 30 days, unused. — Northwind Support",
}


def judge(prompt: str) -> JudgeReply:
    """Your judge. digline composes `prompt` from the rubric, the question and
    the answer; it wants a score in [0, 1] and a reason back."""
    signed = "Northwind Support" in prompt
    concise = len(prompt.split()) <= 60
    return JudgeReply(
        score=0.4 + 0.3 * signed + 0.3 * concise,
        reason=f"signed={signed}, concise={concise}",
    )


def target(case: Case) -> Response:
    """Your application, called once per case. Canned here so this runs as is."""
    text = ANSWERS[case.id]
    return Response(output=text, cost_usd=0.004 + 0.001 * len(text) / 100)


suite = Suite(
    tenant="northwind",
    environment="staging",
    name="support",
    assertions=[
        Contains(needle="Northwind Support"),
        LlmRubric(
            rubric="Does the reply answer the question in at most three sentences?",
            judge=judge,
            threshold=0.7,
            tolerance=0.05,
        ),
        CostBudget(max_usd=0.02, tolerance=0.05),
    ],
    cases=[Case(id="where-is-my-order"), Case(id="how-do-i-return")],
)

tenant="northwind" is ours, like every tenant in this repository's examples, and it has to become yours before the first run. A tenant is one customer: one perimeter, and the directory your store is keyed on, .digline/<tenant>/. Name it after the customer the suite is for. A suite copied to the next customer with this line unchanged files that customer's runs in the first one's directory, compares them against the first one's reference and promotes over it, each with exit 0.

When your judge is a real model, add a provider plugin: uv add digline-anthropic (or pip install digline-anthropic), likewise digline-openai and digline-bedrock.

$ digline run --suite suite.py
2026-08-26T15-44-09-282929-00-00-e7421ec503ccefe8

$ digline promote --suite suite.py --run latest --replacing none
tenant 'northwind' · support baseline set to 2026-08-26T15-44-09-282929-00-00-e7421ec503ccefe8

Now make it worse — delete — Northwind Support from the second answer — and ask again:

$ digline run --suite suite.py
2026-08-26T15-44-09-492722-00-00-e7421ec503ccefe8

$ digline compare --suite suite.py --run latest
2 checks got worse compared with the reference. Every case could be judged. No case is suspended. The suite is unchanged from the reference.

how-do-i-return · llm_rubric · Score fell from 1.000000 to 0.700000.
how-do-i-return · contains · Went from passing to failing (1.000000 → 0.000000).

$ echo $?
1

The exit code is the answer: 0 fine, 1 got worse, 2 the numbers are not what they look like. A case that could not be judged is one cause of 2 among several, and docs/api.md, The exit code, lists them all. None of the three is a failure of digline: a wrapper that treats a non-zero exit as a crash is reading a verdict as an error. Two more are not verdicts. 64 is the command refusing what it was asked: a missing baseline, a document it cannot read, an --out it cannot write, a suite that exits while it loads. 70 is a failure nobody anticipated. Everything lands in .digline/<tenant>/ — baselines/ committed, runs/ git-ignored through a .gitignore digline writes for you, and says so when one you already had leaves runs/ out.

What it checks

Per case — pure functions (inputs) -> Verdict, no I/O, callable on their own:

A suite pointed at an OpenAI-compatible endpoint, a gateway or a self-hosted model declares what it actually charges — [target.pricing] in TOML, override() in Python — so a CostBudget reads your prices rather than the plugin's list.

Per run — one verdict on the whole suite, the kind that goes in a contract:

Every assertion carries a threshold that can fail — there is no default that passes vacuously, and Contains("") is a ValueError when the suite loads rather than a green run — and a tolerance below which a difference from the baseline is noise. Where a number is really "k out of n", write it as one: min_agreement="2/3", and a float no k/n can produce is refused at construction.

One card each — parameters, typical values, what to watch out for — in docs/metrics.md. Custom assertion? Subclass AssertionBase, or RunAssertionBase for an aggregate: docs/api.md.

How it thinks

• The judge is yours. The core never calls a model API: you inject a function or a provider plugin's judge, and in your tests a deterministic one.
• Three states, not two — pass, fail, error. An error is neither green nor a regression: it means could not judge, and a run containing one cannot become the baseline.
• Two kinds of noise, two answers. Suite.samples asks the target more than once — the same input answered differently. Repeated grades the same output more than once — the judge changing its mind. min_agreement becomes mandatory as soon as you sample.
• A tolerance is declared; a noise floor is declared too. Where a check declares floor_standard_errors, the baseline's judged samples set a floor either side of its score — that width times √2 times their standard deviation over the square root of their number, the √2 because what moves is a difference of two means — and a drop inside it is reported as unchanged rather than as a regression: a tool that cries wolf on its own measurement error teaches people to promote past it. A run aggregate keeps the range its samples spanned instead. It never rescues a flip and never invents an interval it does not have, so you see one once you have promoted a sampled baseline and declared a width.
• Measuring is mechanical; approving is not. An agent can run the suite and tell you what moved, but the MCP server exposes no tool that promotes, and a probe among the examples proves each cycle that promote is still absent. promote is a signature, and a signature is a person's.
• An alias is a pointer, and pointers roll. compare names the model the provider said answered when it changes, digline log reads that down every stored run, and a Case(canary=True) watches behaviour where the provider says nothing: if it moves at all, the headline says the model under the alias likely changed and the run exits 1.
• Set the threshold where the system measurably is, not where you want it: the gate protects against getting worse, and raising the bar is a visible change in a pull request.
• Promote the median of several runs, not the first green one — digline view is the table you pick it from. Cases diagnose, aggregates gate.

Worked through with real numbers in docs/guide.md; the reasoning behind every fixed decision is in docs/adr/.

Claude Code and MCP

digline also installs as a Claude Code plugin. What it ships first is the judgement layer. The operating-digline skill carries the same rules as AGENTS.md, and loads when the agent is about to run a suite or read a comparison. Beside it is the MCP server, which measures, reads and explains, and cannot promote a baseline. Two lines plus one dependency: the server imports your suite, so it runs from your project's .venv.

uv add --dev digline-mcp
claude plugin marketplace add digline/digline
claude plugin install digline@digline --scope project

The agent's own shell commands are not the server. A bare digline typed there resolves through the PATH of the shell Claude Code started, which comes from your login profile and from whatever environment was active when you launched it, and it can be another project's install at another version. uv run digline resolves through your project's environment.

What it installs, and what its hook does and does not guarantee: plugins/digline/README.md.

In another agent, the skill alone installs through skills.sh: npx skills add digline/digline --skill operating-digline. That is the skill without the MCP server or the hook, taken from main rather than a release.

Commands

The same comparison reaches an agent through digline-mcp — thirteen tools that read and measure, and no promote to call — and Claude Code through the plugin, which ships that server with the judgement layer beside it. It reaches a test run through pytest-digline, one row per check, a pull request through digline/digline-action, and CI without a Python toolchain through ghcr.io/digline/digline.

From here: the examples below each run with no API key, the reasoning behind every rule is in docs/adr/, and CONTRIBUTING.md says how to set up the repository, run the tests, and what a change has to show before it lands.

Examples

The projects in examples/ each answer a question somebody actually arrives with. Every one runs with no API key, carries its committed report.html, and is a standalone project: copy the directory anywhere and uv sync works.

• I have a classifier: how do I keep it under control? — labelled cases, an agreement check, Precision and Accuracy as the gate
• I'm writing a prompt and have no application yet — a prompt in a file, and the report showing its diff next to what it moved
• I have a RAG: how do I check it doesn't make things up? — frozen retrieval, Faithfulness, PiiAbsent
• My application is Java: can I use this? — HttpTarget against a service digline cannot import
• My app is LangChain4j: what do I put in my repo? — the walkthrough: one endpoint, three files, the CI gate
• My pipeline is LangChain: what changed when I upgraded it? — the chain called in process, FakeListChatModel in CI, one line to a real model
• My agent calls the right tools, but with the right arguments? — a LangGraph agent judged on its trajectory: Before and Subsequence for the order, Never and MaxSteps on the steps, ToolCalledWith for the arguments, the tools real and the model scripted
• My RAG is LlamaIndex: is it still answering from the right page? — a live query engine, retrieval measured by Faithfulness against the page each case declares
• My team does not write Python: can we still gate a prompt? — a suite.toml and a cases.json, no code in the suite
• My suite is green today: who watches it on Thursday? — the operator loop: a scheduled re-run, draw told from drift, a declared policy that decides who is woken, and a probe proving each cycle that promote is still absent
• My agent uses an MCP server I don't control: how do I know its tool descriptions haven't changed? — a canonical tools/list dump declared as an artifact, and one added clause that stops a tool being called

What digline is not

• Not an observability platform. Dashboards over production traces are a served market. digline asks for none of that: what it needs from the system under test is a function or an HTTP endpoint. What it keeps inside your perimeter is narrower: the recorder keeps your application's calls, and an agent's turn as one episode with its steps, and digline record import writes the ones a person names, with their yes or no, as cases a suite loads (docs/record.md). Judging production responses as they arrive is designed and not yet built.
• Not a red-teaming tool. digline generates no attacks. Once one is found, it becomes a Case, and the suite makes sure it never works again.
• Not YAML. The suite is Python — or, within declared limits, TOML: cases were always data, and now the rules can be too. A judge with rules of its own, a computed request body and a custom assertion stay Python, and the loader names the wall you hit rather than half-supporting it.
• Not a funnel. Two commitments, by design and for good: no hosted service that receives your payloads, and no data collection. The baseline lives in your repo; the runs happen on your machines. If digline ever grows paid features, they will run inside your perimeter too. How digline treats its own attack surface — the published advisories and the delta-pass every minor release gets — is in SECURITY.md.

Status

1.2.0. The offline cycle — write the suite, run, compare, promote, report, and commit what a person decided — is complete, covered by tests, and used daily on a real project, and since 0.5.0 the suite may be written as data as well as in Python. From 1.0.0 a minor release does not break a public Python signature; the --json output and the stored baseline keep versions of their own, OUTPUT_VERSION and SCHEMA_VERSION, which may still move in a minor, and the baseline migrates. The production store, the bridge from production failures back to committed cases, and the reactive side are designed in ADR 0002 and not written yet.

Python 3.12+. One runtime dependency: jsonschema.

Docs

• docs/guide.md — how to reason with digline, in eight chapters and the order the problems arrive: baseline, judge noise, sampling, tolerance, threshold, which run to promote, what to gate on, what to maintain
• docs/metrics.md — a card per assertion and aggregate: when to reach for it, what it produces, what it will do to you if you are not looking
• docs/api.md — what is imported from where, every assertion and its parameters, custom assertions, and the complete example in examples/quickstart/, which a test runs on every build
• docs/declarative.md — the suite as data: the TOML format, key by key, what it deliberately cannot say, and how to move a suite between the two forms without losing its baseline
• docs/diff.md · docs/explain.md · docs/rejudge.md · docs/view.md · docs/migrate.md — each command with a surface of its own
• docs/mcp.md · docs/pytest.md — the two front ends that are not the CLI
• AGENTS.md — how a coding agent should operate digline in your repo
• plugins/digline/ — the same rules as a Claude Code plugin, with the MCP server and a hook that asks a person before promote, register, view --allow-promote, an import that writes a yes or no, and record on
• docs/adr/ — the architectural decisions, numbered, with the reasoning

License

Apache-2.0.