retrieval-observatory 0.5.1

Local-first retrieval reliability for integrating, evaluating, and investigating multi-stage retrieval pipelines

retobs

retobs is a local-first reliability layer for retrieval pipelines. It helps you integrate observable retrieval stages, evaluate a callable, compare explicit Runs, and inspect recorded query evidence. It is not an answer evaluator or a leaderboard: when identity, topology, candidates, telemetry, or ground truth are unavailable, retobs reports that limit instead of inferring a conclusion.

Install

pip install "retrieval-observatory[dashboard,mcp]"

Integrate an existing project

Create and review a plan before any mutation. Apply consumes that reviewed plan; verify reports readiness only after it observes the declared topology, candidate evidence, and telemetry health.

retobs integrate . --phase plan --output retobs/integration-plan.json
retobs integrate . --phase apply --plan retobs/integration-plan.json
retobs integrate . --phase verify --plan retobs/integration-plan.json

Unresolved required mappings or stale file hashes block apply. The apply result lists every changed file and retains reversal information in its apply record.

Evaluate a callable

retobs evaluate mypackage.search:retrieve --queries data/queries.jsonl --qrels data/qrels.jsonl --corpus data/corpus.jsonl

Use the returned Run ID with retobs report, retobs compare, and retobs inspect-query. A comparison is valid only when its required identities align; a query diagnosis is limited to evidence actually recorded.

Investigate locally

retobs serve --db .retobs/results.db

The dashboard binds to 127.0.0.1 by default. It is unauthenticated and local-first; put it behind trusted controls before exposing it beyond loopback.

What retobs records

• Evaluation Runs, manifests, query evidence, and complete or partial operator traces.
• Production traces scoped to a service and pipeline, including candidate transitions when instrumentation provides them.
• Instrumentation health: sampling, drops, serialization failures, retries, and permanent export failures.

These are evidence contracts, not guarantees that every integration can supply every field.

Integration support

First-class integration paths are plain Python, HTTP, FastAPI, LangChain, and LlamaIndex. DSPy, Haystack, and OpenAI Agents are supported examples with narrower guarantees. See integration support and the agent runbook.

Privacy and production safety

Queries, candidates, metadata, labels, and traces may be sensitive. Redaction runs before enqueue and persistence according to the integration manifest; queue capacity, overflow policy, and sampling are explicit telemetry configuration. Read privacy and security before production use.

Documentation

• Start
• Workflow
• Concepts
• CLI, SDK, and MCP reference
• Architecture
• Releases

License: MIT.