Datum

Retrieval as a compiled query, not a hand-wired pipeline call.

A retrieval substrate for AI agents. Datum compiles every query into an explainable, replayable plan over one versioned, content-addressed store, so who is asking, what it may cost, which sources are trusted, and whether the evidence is sufficient are all part of the request rather than smuggled through a metadata filter.

Quickstart  ·  Python API  ·  Architecture  ·  Agent / MCP  ·  Research  ·  Roadmap  ·  Contributing

[!IMPORTANT] Project status. Datum is pre-1.0 and under active development. Every planned v1 milestone is complete and verified against a real PostgreSQL: the walking skeleton, the full hybrid retrieval pipeline (dense + BM25 + ANN, fused and reranked), the evaluation gate with abstention, MCP acceptance over the real transport, multi-format ingestion with 20-script image OCR, and the feedback loop. 258 tests, no mocked components. Benchmarks are measured and stored in-repo (benchmarks/, docs/bench/). Install with pip install datumrag (import stays import datum). The public API below is what the code does today.

Contents

• Why Datum
• Highlights
• Architecture
• Requirements
• Quickstart
• The Python API
• How retrieval works
• Use it from an agent (MCP)
• Use it from any language (HTTP)
• Write your own operator
• Security and governance
• How Datum is different
• The research behind Datum
• Roadmap
• Project layout
• Contributing
• Community and support
• Citation
• License
• Acknowledgements

Why Datum

Most retrieval code hangs off one function shaped like retrieve(query, k) -> docs. That signature is the whole problem. It is at once the logical question ("what is relevant to this query") and the physical plan that answers it, and it has exactly one structured input for everything else the system needs to know. So the identity of the caller, the budget, the trust tier of a source, and the question of whether there is enough evidence to answer at all get forced through the metadata filter, because that is the only argument available.

When four different concerns ride on one argument, a slip in any of them looks identical to a slip in the others. A relevance bug and a tenant-isolation breach become the same line of code. That is not a hypothetical. Independent teams have shipped that exact class of defect, and the same shape of write race that silently drops a memory update shows up across memory layers and vector stores. The research that motivates Datum documents these as a verified taxonomy of common failures, grounded in real issue trackers and postmortems (see The research behind Datum).

Datum takes the split that databases made decades ago. System R separated SQL, the logical request, from access paths, the physical plan that satisfies it. Datum does the same for retrieval.

# The overloaded call. Identity, budget, trust, and "is this enough?"
# all have to hide inside `filters`.
docs = retrieve(query, k=10, filters={"tenant": "acme", "min_score": 0.7})

# Datum. The request states who is asking; the plan is compiled, explainable,
# and replayable; the answer is typed evidence that is allowed to say "not enough".
evidence = corpus.search("how do I roll back a deploy", principal=alice)
if evidence.status == "insufficient_evidence":
    ...  # abstain instead of returning a confident wrong answer

The caller states the question and who is asking. Datum resolves the tenant partition before any operator runs, compiles a physical plan you can read back with explain(...), records the plan so you can replay(...) the exact evidence later, and returns a typed result that can abstain when the corpus does not support an answer.

Highlights

• 🧭 Compiled plans. Every search becomes an explicit physical plan. explain(plan_id) reads it back, and replay(plan_id) reproduces the exact evidence a plan produced even after the corpus has changed underneath it.
• 🚦 Conformance-gated operators. A physical operator cannot be registered until it passes a conformance suite (filter algebra, score contract, tenancy fail-closed, entitlement staleness). A backend that mistranslates a predicate is refused at startup. This rule has no exception for Datum's own operators.
• 🔎 Hybrid retrieval, done properly. Full-text (BM25 over Postgres), dense vectors (pgvector HNSW), and literal grep run together, fused with weighted Reciprocal Rank Fusion, then reordered by a cross-encoder reranker. Missing an embedder degrades loudly with a warning, never silently.
• 🔒 Tenant isolation by construction. The namespace partition is resolved before an operator sees the query and fails closed. A relevance change cannot become a data-leak change.
• 🧾 Typed evidence with abstention. Results carry a sufficiency estimate and a status. When the corpus does not support an answer, Datum returns insufficient_evidence rather than a confident guess.
• 🗃️ One canonical, bitemporal store. Records are content-addressed, updates are atomic supersede operations, and the write race that drops concurrent updates is closed by construction (proven by a 40-round two-writer concurrency test that ends with exactly one live record every time).
• 🧬 Provenance to the span. Section path, page, and structural location travel with a hit to the surface, so a citation points at where an answer actually lives.
• 🧩 Postgres is the only moving part. pgvector and Postgres full-text carry the whole substrate. No separate vector database or search cluster to run.
• 🤖 Agent-native. Ships as an MCP server with six verbs, five for reading and one for feedback, so a tool-calling model talks to it directly.

Architecture

Datum is nine layers with strictly one-directional imports. Storage sits at the bottom, the agent tool surface at the top, and a single composition root (Corpus) wires them together.

flowchart TB
    agent["🤖 Agent or application"]

    subgraph read["Read path"]
        direction TB
        L8["<b>L8</b> Agent tool surface (MCP)<br/>search · fetch · navigate · explain · since"]
        L6["<b>L6</b> Plan compiler<br/>resolves ACL first · EXPLAIN · replay"]
        L5["<b>L5</b> Physical operators<br/>grep · BM25 · ANN — conformance-gated"]
        L7["<b>L7</b> Evidence state<br/>typed · sufficiency · can abstain"]
    end

    subgraph write["Write path and storage"]
        direction TB
        L3["<b>L3</b> Write orchestrator<br/>assert · supersede · forget"]
        L2["<b>L2</b> Ground store<br/>bitemporal · content-addressed · atomic CAS"]
        L1["<b>L1</b> Write-ahead log"]
        L0["<b>L0</b> Object storage (content-addressed blobs)"]
    end

    L4["<b>L4</b> Derivation and views<br/>lexical + dense, rebuilt incrementally off the WAL"]

    agent -->|"request + principal"| L8
    L8 --> L6
    L6 -->|"compile + dispatch"| L5
    L5 -->|"reads"| L4
    L5 --> L7
    L7 -->|"hits + opaque hit_ids"| agent

    agent -->|"ingest"| L3
    L3 --> L2 --> L1 --> L0
    L2 -->|"WAL tail feeds derivation"| L4

What a single search() does, end to end:

sequenceDiagram
    autonumber
    actor Agent
    participant MCP as L8 Tool surface
    participant Planner as L6 Compiler
    participant Ops as L5 Operators
    participant Views as L4 Views / L2 Store
    participant Ev as L7 Evidence

    Agent->>MCP: search(query)  %% principal comes from the session, never an argument
    MCP->>Planner: compile(query, principal, budget)
    Planner->>Planner: resolve namespace ACL (fail closed)
    Planner->>Ops: run grep + BM25 + ANN, scoped to the namespace
    Ops->>Views: read lexical + dense views and live records
    Views-->>Ops: candidates
    Ops-->>Planner: candidate sets per operator
    Planner->>Planner: weighted RRF fuse, then cross-encoder rerank
    Planner->>Ev: build typed evidence + sufficiency
    Ev-->>MCP: Evidence (hits, or insufficient_evidence)
    MCP-->>Agent: hits + opaque hit_ids (no trust metadata crosses the boundary)

Requirements

• Python 3.11 or newer (developed and tested on 3.12).
• PostgreSQL 17 with the pgvector extension (CREATE EXTENSION vector). pgvector backs the dense/ANN operator; Postgres full-text backs BM25.
• Optional for hybrid retrieval: the embed extra, which pulls sentence-transformers for the dense embedder and the cross-encoder reranker. Without it, Datum still runs on grep plus BM25 and warns that the dense operator is absent.
• Optional for rich document parsing: the parse extra (docling).

Quickstart

# 1. Install. Directly from GitHub (with the dense-retrieval extra):
pip install 'datumrag[embed]'
# ...or straight from GitHub:
# pip install 'datumrag[embed] @ git+https://github.com/COLONAYUSH/Datum.git'
# ...or clone for development:
git clone https://github.com/COLONAYUSH/Datum.git
cd Datum
python -m venv .venv && source .venv/bin/activate
pip install -e '.[embed]'

# 2. Point Datum at Postgres and create a scratch database
export DATUM_PG_DSN="postgresql://localhost/datum_dev"
createdb datum_dev
psql -d datum_dev -c "CREATE EXTENSION IF NOT EXISTS vector;"

# 3. Ingest a document and search it
datum ingest ./docs/examples/runbook.md --source-id runbook --namespace tenant:acme
datum search "how do I roll back a deploy" --namespace tenant:acme

Typical output. Note that the query shares no words with the source sentence ("roll back" against a document that says "revert the release"), and dense retrieval still finds it:

status=ok  sufficiency=0.742  plan=pl_9f3c...
[1] Deploy Runbook > Rollback
    To revert the release, pin the previous image tag and redeploy the production cluster.

[!TIP] The test suite and datum eval truncate whatever database DATUM_PG_DSN points at. Always point it at a throwaway database, never at one holding content you care about.

For a step-by-step setup with troubleshooting (installing Postgres and pgvector, first-run model downloads, and common errors), see docs/SETUP.md.

datum ingest <path> --namespace NS [--source-id ID] [--dsn DSN]
    Ingest a document through the write path, then bring the views current.

datum search "<query>" --namespace NS [--dsn DSN]
    Compile and run a retrieval; print ranked hits with their section paths.

datum serve --namespace NS [--dsn DSN]
    Run the MCP server over stdio (six verbs). Point an MCP client at this.

datum eval [--corpus-dir DIR] [--regression-set FILE] [--dsn DSN]
    Ingest a fixture corpus and run the curated regression set through the live
    hybrid pipeline. Exits non-zero if any case regresses.

The Python API

Corpus is the one object you hold. It wires every layer and registers each operator through the conformance gate when it opens.

from datum import Corpus
from datum.kernel.principal import Principal

alice = Principal(id="alice", namespace="tenant:acme")

with Corpus.open("postgresql://localhost/datum_dev") as corpus:
    # Ingest. Returns the number of write ops applied; unchanged sections are no-ops.
    corpus.ingest(
        "runbook",
        "# Deploy Runbook\n\n## Rollback\nTo revert the release, pin the previous image tag.\n",
        principal=alice,
    )

    # Search. Hybrid retrieval, fused and reranked, returns typed evidence.
    evidence = corpus.search("how do I undo a bad deploy", principal=alice)
    print(evidence.status, round(evidence.sufficiency, 3))
    for hit in evidence.hits:
        print(" > ".join(hit.section_path), "::", hit.content[:80])

    # Read a hit's full content by its opaque id. Fails closed across namespaces.
    top = corpus.fetch(evidence.hits[0].hit_id, principal=alice)

    # Read the plan that produced the search, reconstructed from its trace.
    print(corpus.explain(evidence.plan_id, principal=alice))

    # Reproduce the exact evidence later, even after the corpus changes.
    same = corpus.replay(evidence.plan_id)

    # Re-run the same question against today's corpus and policy instead.
    fresh = corpus.replay(evidence.plan_id, against="current_champion")

How retrieval works

A compiled search runs three operators inside the caller's namespace and fuses their rankings with weighted Reciprocal Rank Fusion:

$$\text{score}(d) = \sum_{o \in {\text{grep},\ \text{bm25},\ \text{ann}}} \frac{w_o}{k + \text{rank}_o(d)}$$

where $\text{rank}_o(d)$ is a document's position in operator $o$'s result list, $k$ is a smoothing constant that keeps a single top rank from dominating, and $w_o$ is the per-operator weight from the plan-selection policy. Fusion by rank rather than by raw score is deliberate, because the three operators produce scores on scales that do not compare (a BM25 score and a cosine similarity are not the same unit). The fused shortlist then goes through a cross-encoder reranker, which reads the query and each candidate together and reorders them.

The views the operators read (a BM25 index and a dense-vector index) are derived from the canonical records and kept current incrementally. Because ingestion no-ops unchanged sections, only the chunks a write actually touched get re-derived.

[!NOTE] Datum's default embedder is BAAI/bge-m3 (multilingual, 100+ languages) and its default reranker is BAAI/bge-reranker-v2-m3, a matched multilingual pair, both small enough to run on a CPU. Both sit behind Protocols, so you can pass a stronger local model or a hosted API to Corpus.open(embedder=..., reranker=...) without touching anything else.

Use it from an agent (MCP)

Datum speaks the Model Context Protocol. Run the server over stdio:

datum serve --namespace tenant:acme

It exposes six verbs: search, fetch, navigate, explain, since, and feedback. The principal comes from the session, so it is never a tool argument a model could set, and what crosses the boundary is an opaque hit_id plus content, never a trust tier or authority.

{
  "mcpServers": {
    "datum": {
      "command": "datum",
      "args": ["serve", "--namespace", "tenant:acme"],
      "env": {
        "DATUM_PG_DSN": "postgresql://localhost/datum_dev",
        "DATUM_HIT_SIGNING_KEY": "set-a-stable-secret-here"
      }
    }
  }
}

Use it from any language (HTTP)

MCP is the agent-native way in. For everything else, the same six verbs are available over plain HTTP, so Node, Go, curl, or any stack that can POST JSON can use Datum without Python:

DATUM_HTTP_TOKEN=$(openssl rand -hex 24) datum serve-http --namespace tenant:acme --port 8787

curl -s http://127.0.0.1:8787/v1/search \
  -H "Authorization: Bearer $DATUM_HTTP_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query": "how do I roll back a deploy"}'

Endpoints: POST /v1/{search,fetch,navigate,explain,since,feedback} and GET /v1/health. A bearer token is required, and there is no anonymous mode; the server binds localhost by default and serves one namespace per process, so a compromised client cannot reach another tenant's partition by editing JSON. Tenancy is still enforced inside the corpus on every call, exactly as in the Python API and MCP paths, and the HTTP surface is covered by the same test suite (tests/mcp_server/test_http_api.py).

Write your own operator

An operator is anything that satisfies the Operator Protocol. Datum will not register one until it passes the conformance suite, so a backend that quietly mistranslates a filter or leaks across a tenant boundary is refused before it can ever serve a query.

from datum import ConformanceSuite, CandidateSet, CostEstimate, OperatorPlan

class MyOperator:
    kind = "my-backend"

    def plan(self, fragment, budget) -> OperatorPlan:
        return OperatorPlan(operator_kind=self.kind, params={"fragment": fragment})

    def execute(self, op_plan) -> CandidateSet:
        ...  # run the backend, return records + scores

    def cost_model(self, fragment) -> CostEstimate:
        return CostEstimate(tokens=0, dollars=0.0, latency_ms=10.0)

# Run the same gate the registry runs. If your operator fails closed correctly,
# this passes; if it can be tricked into leaking across tenants, it does not.
report = ConformanceSuite.run(MyOperator())
assert report.passed, report.failures

Security and governance

Datum treats isolation, auditability, and honest uncertainty as properties of the architecture, not features bolted on top.

How Datum is different

This compares design properties, not benchmark numbers. It is about what the architecture guarantees.

[!NOTE] On benchmarks. Every number here is measured by a harness stored in this repository, and the per-question results are committed alongside. Adversarial corpora (86 questions across two hostile documents): Datum 40/42 and 44/44 native; under the same shared models, LangChain, LlamaIndex, and Haystack score 64 to 67 of 86 (benchmarks/adversarial/). BEIR SciFact through the full governed pipeline: nDCG@10 0.694 default, 0.714 with bge-large (docs/bench/, harness in scripts/beir_scifact.py). What is also verifiable today: the suite runs green against a real PostgreSQL (not mocks), and the concurrent-write race that drops updates in other systems is closed by a two-writer concurrency test.

The research behind Datum

Datum is the design output of a large, adversarially-verified study of retrieval in agentic systems. The study surveyed the field end to end, then autopsied roughly sixty frameworks and platforms for their real, evidenced failures (issue trackers, CVEs, postmortems), and distilled a taxonomy of the failures that recur across them. An issue only counted as "common" if it appeared in at least three independent systems and survived two rounds of adversarial refutation.

The headline finding is the one this framework is built to answer: four confirmed defects collapse to one missing primitive, the split between the logical request and the physical plan. The design work also caught and corrected its own overclaiming, including finding closer prior art (LOTUS, Palimpzest) for its central idea than the first literature pass did. That honesty is part of the argument, not a footnote to it.

• The taxonomy of common failures (CI-01 through CI-27), each with evidence and a severity.
• The framework specification, post red-team revision.
• The paper that ties the failures to the design, with its figures and style rules.

[!NOTE] The full study lives in research/, the rival designs and the judgment behind the final one in design/, and the paper in paper/. For continuation context, see HANDOFF.md (status and next steps) and LEARNING.md (every lesson from building this).

Roadmap

• Foundation. Version-frozen kernel, storage, ground store, write path, security.
• Milestone A. Walking skeleton: ingest, search, fetch, navigate, explain, since, replay, end to end.
• Milestone B. Hybrid retrieval: dense + BM25 + ANN, fused with weighted RRF, cross-encoder rerank, all through the conformance gate.
• Milestone C. Evaluation gate wired to the live corpus, dense-similarity abstention, concurrency hardening.
• Milestone D. Acceptance over the real MCP transport: a real client drives all six verbs against a served corpus, tenancy fail-closed, covered in tests/mcp_server/test_serve_e2e.py.
• Multi-format ingestion. Docling-backed parser for PDF, Office, HTML, CSV, and images, with an all-format benchmark test.
• Multilingual image-text recovery. 20 writing scripts across three OCR engines, readback-verified, with the sparse gate and plurality arbitration that keep a broad engine roster hallucination-free, plus a labeled NLLB translation gloss on non-Latin image text.
• Contextual retrieval. Every chunk indexed with its own section path prefixed (no model calls needed), plus table row-groups that repeat their header.
• Adversarial benchmark. Two hostile documents, 86 questions, mechanical scoring: Datum 40/42 and 44/44 native (43/44 under the shared head-to-head config); LangChain / LlamaIndex / Haystack score 64 to 67 of 86 with identical models (benchmarks/adversarial/).
• Public benchmark. BEIR SciFact end to end through the full pipeline: nDCG@10 0.694 with the default embedder, 0.714 with bge-large, both stored in docs/bench/ with the harness at scripts/beir_scifact.py.
• Relevance feedback loop. A sixth MCP verb records judgments tied to their exact retrieval; datum calibrate grid-tunes per-tenant weights and thresholds, promotion-gated on held-out judgments. (The Phase-2 learned policy replaces the search, not the discipline.)
• Vision-describer slot. Any VLM behind a three-member Protocol; descriptions land labeled with the producing model. Ships proven and empty: no locally runnable small VLM was usable.
• As-of (time-travel) queries over the bitemporal store.
• Fine-grained, predicate-level access control.
• Cryptographic-shred forgetting.

Project layout

datum/
├── src/datum/
│   ├── kernel/        version-frozen typed contracts (Protocols + frozen dataclasses, zero I/O)
│   ├── storage/       L0 object storage + L1 write-ahead log + SQL migrations
│   ├── groundstore/   L2 bitemporal canonical store, atomic supersede, uniqueness CAS
│   ├── writepath/     L3 write orchestrator + document policy
│   ├── derivation/    L4 chunking + lexical/dense views + the derivation engine
│   ├── operators/     L5 grep / BM25 / ANN + the conformance suite that gates them
│   ├── planner/       L6 plan compiler, fusion, reranker, trace store
│   ├── evidence/      L7 typed evidence + sufficiency
│   ├── policy/        plan-selection rule table
│   ├── security/      principal context + namespace ACL (fail closed)
│   ├── mcp_server/    L8 MCP server + signed hit registry
│   ├── eval/          the regression gate
│   └── corpus.py      the composition root that wires it all together
├── tests/             the suite (runs against a real Postgres, not mocks)
├── docs/decisions.md  every deviation from the spec, numbered, with reasoning
└── pyproject.toml

Contributing

Contributions are welcome. The short version:

pip install -e '.[dev,embed]'
export DATUM_PG_DSN="postgresql://localhost/datum_dev"   # a scratch database
createdb datum_dev && psql -d datum_dev -c "CREATE EXTENSION IF NOT EXISTS vector;"
pytest -q

[!TIP] If you are adding a physical operator, the bar to clear is the conformance suite. Run ConformanceSuite.run(YourOperator()) and make report.passed true before you wire it in. That is the same gate the registry enforces at runtime.

A few house rules that keep the design honest:

• The kernel is version-frozen. Adding a top-level symbol is a deliberate, reviewed change, recorded in docs/decisions.md.
• Anything touching transactions, isolation, or ordering is tested against a real PostgreSQL, never a mock.
• Every deviation from the specification gets a numbered entry in docs/decisions.md with its reasoning.

Community and support

• Questions and ideas → open a Discussion.
• Bugs and feature requests → open an Issue.
• Security reports → please follow SECURITY.md (private disclosure), not a public issue.

Citation

If Datum is useful in your work, please cite it. The paper is in preparation; this is the current reference:

@software{datum2026,
  title        = {Datum: Retrieval as a Compiled Query for Agentic Systems},
  author       = {Kumar, Ayush},
  year         = {2026},
  url          = {https://github.com/COLONAYUSH/Datum},
  note         = {Manuscript in preparation}
}

Star history

Contributors

License

Apache License 2.0. See LICENSE.

Acknowledgements

Datum stands on PostgreSQL, pgvector, sentence-transformers, Docling, and the Model Context Protocol. The design owes a specific debt to the System R lineage in databases, and to the declarative-optimizer-over-unstructured-data work (LOTUS, Palimpzest) that reached parts of this idea first.