sauron-verifier 1.0.1

Evidence-first agentic verifier for rubric-based evaluation.

Installation • Quick start • How it works • Configuration • Reference

What is Sauron?

Sauron evaluates agent runs against rubric criteria using the execution trace, final environment state, and produced artifacts. It returns a verdict and an evidence receipt for each criterion.

Sauron adaptively gathers information through bounded, read-only queries. An investigator proposes claims, a separate verifier checks them against their sources, and a finalizer assesses whether the verified claims support a verdict. This allows Sauron to investigate long runs without loading the full trace and state into context.

Criteria that remain unresolved receive not_awarded with evidence_missing=true and an explanation.

Rubric criteria
      │
      ▼
Cluster planner
      │
      ▼
Parallel investigation sessions ─── one per cluster
      │
      ├── Query trace, state, and artifacts
      ├── Propose claims → retrieve sources → verify
      └── Finalize each criterion using verified claims
      │
      ▼
Task-wide consistency check
      │
      ▼
Per-criterion verdicts and evidence receipts
      +
Run summary

Installation

Requires Python 3.11 or newer.

pip install sauron-verifier

From source, with uv (the supported development workflow):

git clone https://github.com/rsanayei/Sauron.git && cd Sauron
uv sync                      # creates .venv with the package and dev tools
uv run sauron --help

Sauron talks to any OpenAI-compatible chat-completions endpoint (a LiteLLM proxy, or a provider that speaks that protocol). Put the endpoint in .env (copy .env.example):

LITELLM_BASE_URL=https://your-endpoint/v1
LITELLM_API_KEY=...
SAURON_MODEL=provider/your-model      # the investigator; or pass --model on each run

Model names are whatever that endpoint serves. The optional --vision surface additionally needs pdftoppm (poppler) and, for PPTX/XLSX rendering, soffice (LibreOffice) on the PATH.

Quick start

# ten toy tasks with known expected verdicts; validates the whole loop end to end
sauron smoke --model provider/your-model --out results --name smoke

# a corpus slice through the built-in adapter (a task JSONL + per-task run.zip archives)
sauron run --model provider/your-model --data-root /path/to/corpus \
    --task-ids <task_id>,<task_id> --out results --name first_run

# a corpus through your own adapter (see docs/adapters.md)
sauron run --model provider/your-model --adapter path/to/my_adapter.py \
    --out results --name mine

# metrics for a finished run, against the gold labels its adapter attached
sauron score results/first_run

(From a source checkout, prefix each command with uv run.) Runs are resume-safe: a task whose criterion records are already on disk is not re-judged. Add --fresh to refuse resuming into an existing run directory, or --dry-run to resolve the configuration and the slice without any model call.

Configuration

A run is the shipped defaults plus explicit overrides. Every knob is listed by sauron defaults; the fully resolved configuration is written to <out>/<name>/config.json. You choose the investigator model; every other default is the configuration of the reference evaluations:

See docs/configuration.md for every knob and the reasoning behind each default.

What ships here

• sauron.judges.SauronClusterJudge: the verifier. Plans criteria into evidence clusters, runs one session per cluster, and finalizes each criterion under deterministic rules plus a conditional coverage check.
• sauron.judges.SauronJudge: a single-criterion verifier, usable standalone.
• sauron.env: bounded read-only views over a trace, a state snapshot or workspace directory, and produced artifacts (text, PDF, PPTX, DOCX, XLSX), plus a scratch sandbox for run_python. Optional surfaces, off by default: live-environment MCP servers and rendered-page vision.
• sauron.verifier.evidence_store: the evidence verifier (locator schema, source resolution, deterministic checks for mechanical claims, LLM faithfulness check).
• sauron.adapters: the trajectories adapter (a task JSONL plus per-task workspace archives), a synthetic suite, and a loader for external adapters.

Development

uv sync --group dev
uv run ruff check .          # lint
uv run pytest                # offline contract tests; no network, no model calls

See CONTRIBUTING.md for the ground rules.

Documentation

• docs/architecture.md: the verification loop, its invariants, and where each part lives in the code
• docs/configuration.md: every knob, its default, and where the default comes from
• docs/adapters.md: the TaskBundle contract and how to write an adapter (examples/minimal_adapter.py is a complete one)
• docs/output.md: the files a run writes and how to read them

Reference

If you use Sauron in your research, please cite:

@misc{sanayei2026evidencefirst,
  title  = {Evidence-First Agentic Verification},
  author = {Sanayei, Reza and
            Dumitru, Razvan-Gabriel and
            Hou, Sunjie and
            Rezaei, MohammadHossein and
            Riaz, Haris and
            Vacareanu, Robert and
            Yang, Minglai and
            Liu, Ying and
            Surdeanu, Mihai and
            Zhang, Daniel Yue},
  year   = {2026},
  eprint = {2610.17724},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2610.17724}
}