retobs

PyPI

retobs is a local-first reliability layer for retrieval pipelines. It helps you integrate observable retrieval stages, evaluate a callable, compare Runs under a local release policy, and inspect recorded query and candidate-lineage evidence. It is not an answer evaluator or a leaderboard: when identity, topology, candidates, telemetry, or ground truth are unavailable, retobs reports that limit instead of inferring a conclusion.

Install

pip install "retrieval-observatory[dashboard,mcp]"

See it work first

One command, no arguments, no API keys. It builds a regression story end to end — a Test Set, a comparison, a per-query cause, and a validated fix — then hands you a dashboard to explore it.

retobs demo
retobs serve --db .retobs/demo/results.db

Everything below is the same workflow pointed at your own code.

Integrate an existing project

Create and review a plan before any mutation. Apply consumes that reviewed plan; verify reports readiness only after it observes the declared topology, candidate evidence, and telemetry health.

retobs integrate . --phase plan --output retobs/integration-plan.json
retobs integrate . --phase apply --plan retobs/integration-plan.json
retobs integrate . --phase verify --plan retobs/integration-plan.json

Unresolved required mappings or stale file hashes block apply. The apply result lists every changed file and retains reversal information in its apply record.

Evaluate a callable

retobs evaluate mypackage.search:retrieve --queries data/queries.jsonl --qrels data/qrels.jsonl --corpus data/corpus.jsonl

Use the returned Run ID with retobs report, retobs compare, and retobs inspect-query. A comparison is valid only when its required identities align; a query diagnosis is limited to evidence actually recorded.

Gate a retrieval release

Use a local, versioned policy to make the promotion boundary explicit:

retobs integrate . --phase verify --policy retobs/release-policy.yaml
retobs compare BASELINE CANDIDATE --db .retobs/results.db --policy retobs/release-policy.yaml --format html --output artifacts/retobs-release.html --fail-on hold-or-block-or-fail

PASS proves bounded non-inferiority under the declared policy; HOLD is valid but inconclusive evidence; BLOCK is missing or invalid required evidence; and FAIL is a proven policy-critical regression. Promotion readiness and lineage-diagnosis readiness remain separate. See retrieval release decisions and the Candidate Lineage Explorer.

Investigate locally

retobs serve --db .retobs/results.db

The dashboard binds to 127.0.0.1 by default. It is unauthenticated and local-first; put it behind trusted controls before exposing it beyond loopback.

What retobs records

• Evaluation Runs, manifests, query evidence, and complete or partial operator traces.
• Production traces scoped to a service and pipeline, including candidate transitions when instrumentation provides them.
• Instrumentation health: sampling, drops, serialization failures, retries, and permanent export failures.

These are evidence contracts, not guarantees that every integration can supply every field.

Integration support

First-class integration paths are plain Python, HTTP, FastAPI, LangChain, and LlamaIndex. DSPy, Haystack, and OpenAI Agents are supported examples with narrower guarantees. See integration support and the agent runbook.

Privacy and production safety

Queries, candidates, metadata, labels, and traces may be sensitive. Redaction runs before enqueue and persistence according to the integration manifest; queue capacity, overflow policy, and sampling are explicit telemetry configuration. Read privacy and security before production use.

Documentation

• Start
• Workflow
• Concepts
• CLI, SDK, and MCP reference
• Retrieval release decisions
• Candidate Lineage Explorer
• Architecture
• Releases

License: MIT.