doxtr-rag

A multi-tenant, fail-closed Sphinx RAG (retrieval-augmented generation) extension for the doxtr ecosystem. It builds a dual-store knowledge base from your Sphinx documentation, mounted multi-format files, and security-classified Jira/Confluence cross-links (xlinks), then serves it to an Pi Agent TypeScript extension at agent time.

1. What the project is

doxtr-rag is a Sphinx extension plus a companion Pi Agent TypeScript extension that together implement a retrieval knowledge base with a hard security boundary between two physically-isolated ChromaDB stores:

• SHARED_ORG — organization-wide, safe-to-share knowledge (the Sphinx source/ tree and mounts explicitly declared shared).
• LOCAL_PRIVATE — everything else, by default. Any document, node, mount, or xlink with an ambiguous, unverifiable, or failed permission check routes here. There is no code path where an unresolved scope becomes SHARED_ORG.

During sphinx-build the extension:

1. Harvests the resolved Sphinx AST (doctree-resolved) into structural, section-boundary chunks with breadcrumbs and resolved :ref:/:doc:/:term: cross-references.

2. Extracts mounted multi-format files. Binary office/PDF formats — PDF (page-level), spreadsheets (.xlsx/.ods → Markdown tables), presentations (.pptx slide title + body + speaker notes; .keynote/.key best-effort), and Word/OpenOffice (.docx/.odt heading-aware). For .docx, extraction first uses the standard python-docx paragraph API; if that surfaces no body content — e.g. the text lives in non-standard runs, content-controls, table cells, or custom paragraph styles (like InfoLine) that python-docx does not expose as paragraphs — it falls back to reading the raw <w:t> text nodes directly from word/document.xml, preserving heading/section structure so no content is silently dropped.

   Text & markup formats (heading/section-aware chunking with breadcrumbs): Markdown (.md/.markdown, ATX # and setext ===/--- headings), reST (.rst, section-underline levels), Org (.org, *-prefixed headings), HTML (.html/.htm/.xhtml, <script>/<style> dropped, h1–h6 as section headings), XML (.xml, element text flattened to paragraphs), plain text (.txt/.text, blank-line-delimited paragraphs) and delimited data (.csv/.tsv → a single Markdown table; .tsv defaults to a tab delimiter, others are sniffed).

   Diagram sources (label-only — geometry/styling discarded, one deduplicated order-preserving chunk of clean labels per file): draw.io (.drawio, mxCell@value / object/UserObject@label text), PlantUML (.puml/.plantuml/.pu/.iuml: participants, arrow/message labels, note bodies, title, WBS/mindmap nodes and gantt task names; directives dropped) and D2 (.d2: node display labels and connection labels; reserved keywords and structural braces stripped). The full registered set is available at runtime via doxtr_rag.extractors.supported_extensions().

   Overriding extraction: the per-extension dispatch is a simple last-writer-wins registry. To override or add an extractor, call doxtr_rag.extractors.register(ext)(fn) (where fn(path, scope) yields DocumentChunks) from a module imported after doxtr_rag.extractors; the last registration for a given extension wins, so a child theme can fully replace the built-in .docx/.pdf/etc. handling with its own. A child-registered extension is honoured everywhere the pipeline discovers files, because mount scanning and doxtr-rag ingest --url selection both read the registry through the public supported_extensions() rather than a hard-coded list. Ad-hoc/independent stores are opened through the single public factory doxtr_rag.storage.open_named_store(...), so store construction can be overridden in one place too.

3. Classifies Jira/Confluence xlink targets by API-level access restriction (allow-list: only affirmatively-unrestricted → SHARED_ORG).

4. Reconciles every chunk's final scope through a single AccessControl.most_restrictive(...) resolver and ingests it into the correct ChromaDB store, addressed by a canonical (tenant, database, collection) triple, with cache-aware incremental upserts and deletion pruning.

At agent time, the Pi Agent TypeScript extension (shipped in the package at doxtr_rag/pi_extension/, installed to ~/.pi/agent/extensions/chroma-rag) registers a search_knowledge_base tool that queries both stores concurrently, merges results with Reciprocal Rank Fusion (RRF), scrubs credentials from returned context, and degrades gracefully to local-only results when the shared store is unreachable.

All embeddings use a pinned bge-m3 model standardized on 1024-dim dense vectors; parity (same model + dimension) is enforced across ingestion (Python) and query (TypeScript).

The extension is safe under parallel builds (sphinx-build -j N): it declares parallel_read_safe = True and parallel_write_safe = True. Every event it hooks runs in the main process — doctree-resolved buffers chunks and build-finished is the sole writer to the SQLite cache and Chroma store — so parallel builds are not downgraded to slow serial writes.

2. Installation

Requirements:

• Python ≥ 3.10

• The doxtr/reactor devcontainer (docker.io/doxtr/reactor:0.1.5) — run all Python/pytest/mypy and Node/tsc/vitest commands inside it.

• Node ≥ 20 (bundled in the devcontainer) for the TypeScript extension.

• The pinned bge-m3 (1024-dim) ONNX weights (~2.3 GB). In the doxtr/reactor image these are pre-baked at /opt/models/bge-m3 (DOXTR_RAG_BGE_M3_ONNX_DIR is set for you). Elsewhere, download them once with the built-in, Python-only command (no shell/curl needed):

  doxtr-rag fetch-weights          # -> $DOXTR_RAG_BGE_M3_ONNX_DIR or /opt/models/bge-m3
  

  If the weights are absent at build/query time, doxtr-rag fails with a clear error naming doxtr-rag fetch-weights rather than silently substituting a different model (that would break embedding parity).

Install the Python package (editable, with dev extras):

pip install -e .[dev]

The recommended way to install the query extension is the CLI (it copies the packaged extension into your Pi agent dir and wires the reactor defaults):

doxtr-rag --install-agent pi

To work on the TypeScript extension source directly (in a checkout):

cd doxtr_rag/pi_extension && npm install

Enable the extension in your conf.py:

extensions = [
    # ...
    "doxtr_rag",
]

3. Configuration reference

All keys are registered by doxtr_rag.setup(app) and parsed by doxtr_rag.config.SphinxRAGConfig. Set them in your Sphinx conf.py.

The six original keys

ChromaDB deployment keys

Chunk-cap keys

A format-agnostic size cap re-splits oversized chunks on paragraph/sentence/char boundaries with overlap, assigning stable <id>#part=N ids and per-part hashes.

Stable loopback ports

Embedded stores (and the embed server) are served over loopback HTTP for the JS query client. These ports are stable and deterministic so .doxtr/endpoints.json never points at a drifting port; a server already listening on its port is reused rather than duplicated.

Per-mount sidecar servers (see §8) use their own deterministic ports based at 8790, overridable via DOXTR_RAG_MOUNT_PORT_BASE.

Derived (exposed for parity enforcement)

Per-mount SHARED_ORG declaration & fail-closed defaults

A mount is shared only when it declares it explicitly:

rag_external_mounts = [
    {"path": "/data/handbook"},                          # -> LOCAL_PRIVATE (default)
    {"path": "/data/public-specs", "scope": "SHARED_ORG"},  # -> SHARED_ORG (affirmative)
]

Path routing is deny-list-before-allow-list: any resolved path containing noter/ or NDA/ always routes LOCAL_PRIVATE regardless of AST hints, only source/ (and explicitly-shared mounts) map to SHARED_ORG, and every path is resolved with realpath + confinement so source/../NDA/x traversal and symlink laundering fail closed.

HTTP store auth (env-sourced, never in conf.py)

Connection secrets for http stores come from the environment as k=v;k=v header pairs, never from the URI or conf.py:

export DOXTR_RAG_SHARED_STORE_HEADERS="Authorization=Bearer $SHARED_TOKEN"
export DOXTR_RAG_LOCAL_STORE_HEADERS="Authorization=Bearer $LOCAL_TOKEN"

The resolved URI/headers are scrubbed before any log line.

4. ChromaDB deployment guide

Each store can run embedded (on-disk) or against an external HTTP server, independently.

Embedded (on-disk)

rag_local_store_mode = "embedded"
rag_local_store_path = ".doxtr/chroma"
rag_shared_store_mode = "embedded"
rag_shared_store_uri = "/srv/doxtr/chroma_shared"   # embedded path in embedded mode

Because the Pi Agent (JS) client cannot read an embedded on-disk store directly, the build starts a loopback-bound Chroma HTTP server over each embedded store's path; both ingestion and query then go through that HTTP endpoint (one process — the server — owns the on-disk store). The resolved per-scope endpoints are written to .doxtr/endpoints.json for the query extension to read.

External HTTP server (worked example)

Run a persistent chromadb/chroma server in Docker:

docker run -d --name doxtr-chroma \
  -p 8000:8000 \
  -v /srv/doxtr/chroma-data:/data \
  -e CHROMA_SERVER_AUTHN_CREDENTIALS="$CHROMA_TOKEN" \
  -e CHROMA_SERVER_AUTHN_PROVIDER="chromadb.auth.token_authn.TokenAuthenticationServerProvider" \
  chromadb/chroma:latest

Or via docker-compose.yml:

services:
  chroma:
    image: chromadb/chroma:latest
    ports:
      - "8000:8000"
    volumes:
      - /srv/doxtr/chroma-data:/data      # persistent volume
    environment:
      CHROMA_SERVER_AUTHN_CREDENTIALS: ${CHROMA_TOKEN}
      CHROMA_SERVER_AUTHN_PROVIDER: chromadb.auth.token_authn.TokenAuthenticationServerProvider
    restart: unless-stopped

TLS: for any non-loopback remote, terminate TLS (a reverse proxy or --ssl) and use an https:// URI so StoreTarget.ssl is true.

Point the stores at it in conf.py:

rag_shared_store_mode = "http"
rag_shared_store_uri = "https://chroma.internal.example.com:8000"
rag_local_store_mode = "http"
rag_local_server_uri = "https://chroma.internal.example.com:8000"

with the auth token supplied via env (see §3).

Single server hosting both scopes (separate databases, one tenant)

A single external server MAY host both scopes as separate databases under one tenant — physical isolation is enforced at the resolved database/collection boundary, not merely by process:

rag_store_tenant = "default_tenant"
rag_shared_store_mode = "http"
rag_shared_store_uri = "https://chroma.internal.example.com:8000"
rag_shared_store_database = "doxtr_shared"
rag_local_store_mode = "http"
rag_local_server_uri = "https://chroma.internal.example.com:8000"
rag_local_store_database = "doxtr_private"

The shared-store adapter raises on any non-SHARED_ORG chunk addressed to the shared database, so LOCAL_PRIVATE vectors can never reach the shared scope even when both live on one host.

5. Shared-store writability

Whether the shared store is ingested into is decided by a runtime writability probe of the actual mount/endpoint — not by the store mode:

• rw mount / write-accepting server → locally writable: the build ingests source/ (and shared mounts) into the shared store in place.
• ro mount / read-only server → query-only: shared ingestion is skipped gracefully with an [INFO] message and the store stays fully queryable. Never a hard error.

The LOCAL_PRIVATE store (.doxtr/chroma) is always read-write.

.devcontainer mount options (requirement #22)

Mount the shared Chroma data directory rw for local shared ingestion or ro for query-only nodes:

// .devcontainer/devcontainer.json
"mounts": [
  // read-only (query-only node): shared ingestion is skipped, still queryable
  "source=/srv/doxtr/chroma_shared,target=/workspaces/docs/.doxtr/chroma_shared,type=bind,readonly"
  // ...or omit ",readonly" for a read-write (ingesting) node.
]

For an external chromadb/chroma server, apply the same rw/ro policy to that container's own persistent volume instead of a bind mount.

6. Pi Agent extension setup

The extension source ships inside the package at doxtr_rag/pi_extension/ (single source of truth). Install it into your Pi Agent config with the CLI — it copies the packaged extension and wires reactor defaults:

doxtr-rag --install-agent pi

This populates ~/.pi/agent/extensions/chroma-rag/ (a subdirectory whose package.json pi.extensions manifest makes Pi load only chroma-rag.ts and ignore the helper modules addressing.ts, rrf.ts, scrub.ts, xlang-query.ts). Works from a plain pip install — no source checkout needed.

The extension registers the search_knowledge_base tool:

search_knowledge_base({ query: string, top_k?: number }) -> Markdown context blocks

Behavior:

• Reads the resolved per-scope endpoints from .doxtr/endpoints.json (or the path in DOXTR_RAG_ENDPOINTS) — no addressing is re-derived, no loopback port is guessed.
• Queries both stores concurrently (Promise.allSettled), embeds the query with the same pinned bge-m3/1024 model as ingestion (dimension and model parity asserted before any query), and merges with RRF (1/(k+rank), rank-based, distances ignored, identical chunks deduped).
• Per-mount sidecars. When .doxtr/endpoints.json carries a top-level "mounts": [...] array (one entry per served mount sidecar — see §8), the tool queries each mount concurrently alongside the shared/private stores and folds the results into the same RRF merge, degrading gracefully per mount (an unreachable mount is logged and skipped, never fatal). Each entry is {path, scope, mode, host, port, ssl, tenant, database, collection, header_keys}. The field is optional and backward compatible: an endpoints.json with no mounts behaves exactly as before. A newly-configured vault becomes searchable automatically once its sidecar is built + served — no extension code change.
• Returns credential-scrubbed Markdown context blocks (the TS scrub() mirrors the Python pattern set verbatim).
• Offline fallback: if the shared store is unreachable or times out, returns local-only results — no crash, no unhandled rejection, no stack traces in LLM context.

KB-first default. The tool ships with promptSnippet + promptGuidelines that instruct the agent to consult search_knowledge_base before answering factual questions on any subject — because your knowledge base may hold authoritative, domain-specific, or more recent information than the model's training data (your own docs, an indexed website, product specs, or whatever you ingested). The guidance is domain-agnostic and also tells the agent to cite the returned sources, to fall back to general knowledge when results are empty or unrelated (never fabricating a citation), and to treat the KB as only as fresh as its last build (reconcile rather than blindly trust). This is a strong default, not a hard gate — tool use is agent-invoked, so a model can still skip it; for a hard guarantee you would add a harness-level turn hook.

Query-time embedding. The query side must embed with the same pinned bge-m3/1024 model as ingestion. By default this is automatic: when a build runs with the extension enabled, the pipeline auto-starts a loopback bge-m3 embed server (serving the same weights as ingestion) and publishes its URL into .doxtr/endpoints.json (embedding.embed_url). The query extension reads that URL — no manual endpoint or DOXTR_RAG_EMBED_URL needed for the common embedded deployment.

For deployments where the embed endpoint runs elsewhere, start it yourself and point the extension at it (overrides the auto-started one):

# standalone bge-m3 embed server (uses DOXTR_RAG_BGE_M3_ONNX_DIR weights)
doxtr-rag-embed-server --host 127.0.0.1 --port 8519 &
export DOXTR_RAG_EMBED_URL="http://127.0.0.1:8519/embed"

No @chroma-core/default-embed dependency. Because every query embedding is pre-computed by the bge-m3 embed server and passed to ChromaDB as queryEmbeddings, the extension never uses a collection's built-in embedder. When opening a collection it passes a no-op embeddingFunction to getCollection, which stops chromadb 3.x from instantiating its DefaultEmbeddingFunction — so the optional @chroma-core/default-embed package is not required and does not need to be installed.

One-command installer (reactor-tailored)

Inside the doxtr/reactor container, install the extension into a Pi agent with defaults tailored to the container:

doxtr-rag --install-agent pi              # copy the packaged extension (default)
doxtr-rag --install-agent pi --symlink    # or symlink to a dev checkout (live edits)
# if the console script isn't on PATH:
#   python -m doxtr_rag.cli --install-agent pi

The installer (doxtr_rag.agent_install, exercised by the test suite):

• copies (default) the packaged doxtr_rag/pi_extension/ into $PI_AGENT_DIR (default ~/.pi/agent) under extensions/chroma-rag — or --symlink to a writable dev checkout,
• runs npm install for the extension's Node deps at the destination,
• adds reactor defaults to settings.json without overwriting an existing provider/model/auth choice,
• writes a sourceable ~/.pi/agent/doxtr-rag.env with reactor defaults (DOXTR_RAG_BGE_M3_ONNX_DIR=/opt/models/bge-m3, DOXTR_RAG_EMBED_URL=http://127.0.0.1:8519/embed, DOXTR_RAG_ENDPOINTS=.doxtr/endpoints.json), each overridable from the shell.

Then source ~/.pi/agent/doxtr-rag.env and start pi.

Other agents. opencode, claude, cursor, and codex do not consume Pi's TypeScript ExtensionAPI, so the installer reports them as not-yet-supported rather than faking a config copy. Exposing search_knowledge_base to them is a planned follow-up via an MCP server wrapping the tool (opencode and Claude Code speak MCP) or each agent's native plugin format.

7. Security model

• Fail-closed scope routing: ambiguous/unverifiable → LOCAL_PRIVATE; SHARED_ORG only from an affirmative signal. AccessControl.most_restrictive is the single final-scope decision.
• Path confinement: every path resolved with realpath and asserted within an allowed root; deny-list (noter/, NDA/) before allow-list (source/); escaping symlinks are skipped fail-closed.
• Credential scrubbing: a single value-level scrubber (URLs-with-creds, JWTs, Bearer, ghp_/github_pat_, xox[baprs]-, AKIA/ASIA, PEM keys) runs over all chunk text, metadata values, audit records, and returned LLM context — mirrored verbatim in TypeScript. A metadata-key deny-list adds defense-in-depth.
• Physical store isolation: the shared-store adapter raises on any non-SHARED_ORG chunk at the resolved database boundary (holds under retries, errors, and the one-server-two-databases topology).
• Resilience: 5s connect / 10s read timeouts, ≤2 retries only on 5xx/429 (never 401/403), a per-build network budget, and fail-closed HTTP mapping.
• Audit log: restricted-xlink routing is recorded to .doxtr/security_audit.jsonl (0600, scrubbed, one [INFO] per redirect).

8. Maintenance CLI

doxtr-rag fetch-weights                             # download the pinned bge-m3 ONNX weights
doxtr-rag fetch-weights --force                     # re-download even if present
doxtr-rag --conf conf.py --rebuild --scope all      # drop & reindex collections
doxtr-rag --conf conf.py --rebuild --scope shared   # or just one scope
doxtr-rag --conf conf.py cache-info                 # inspect the SQLite cache
doxtr-rag --conf conf.py audit-info                 # summarize the audit log
doxtr-rag --install-agent pi                        # install the query extension into Pi Agent

Collections are tagged with the model revision + dimension; a model/dimension change drops and rebuilds the affected collection.

Ad-hoc ingestion (ingest)

The Sphinx build is the primary ingestion path; ingest adds a one-off path for a single file or URL (an xlink is a URL, not a file):

doxtr-rag --conf conf.py ingest (--url URL | --file PATH) \
  [--ingest-scope shared|private] [--store NAME] \
  [--source-url URL] [--tags "t1, t2"] [--doc-title "Title"]

• Exactly one of --url / --file is required.
• --url fetches the URL (httpx; requires network egress), picks an extractor from the URL extension or the response content-type, and records the URL as provenance automatically.
• --file ingests a local file; --source-url still records the originating link (useful when a page needs cookie auth the standalone fetch cannot do: fetch elsewhere, save, ingest with --file --source-url).
• --ingest-scope shared|private routes to the canonical stores via the IngestionEngine (scope reconciliation + the SHARED_ORG isolation guard both apply). private is the fail-closed default.
• --store NAME targets an independent named database, co-located with the private embedded store and created on demand. It takes precedence over --ingest-scope and is intentionally outside the shared/private isolation model (a custom store is neither the org-shared nor the local-private KB).
• --source-url/--tags/--doc-title prepend a small provenance chunk (Title / Source URL / Tags) so a semantic hit surfaces the originating link and tags.
• TLS & SSRF: URL fetch trust resolves SSL_CERT_FILE / REQUESTS_CA_BUNDLE then the system CA bundle (fail-closed). Redirects are followed but every hop is validated to reject loopback/link-local/private targets (SSRF-via- redirect guard). The discouraged DOXTR_RAG_INGEST_INSECURE=1 disables both TLS verification and the SSRF guard as a last resort.

Portable per-mount sidecar index (mount-index / mount-watch)

External mounts (e.g. a vault/) rarely change but can hold hundreds of files. A portable sidecar index co-locates a prebuilt ChromaDB next to the files at <mount>/.doxtr/chroma (with a per-mount SHA-256 cache at <mount>/.doxtr/mount-cache.db) so the folder is queryable without a make html, and a Sphinx build only re-embeds files that actually changed.

# build/refresh a mount's sidecar (incremental, SHA-gated, prunes deletions)
doxtr-rag --conf conf.py mount-index --mount-path "$(pwd)/vault"
doxtr-rag --conf conf.py mount-index --all-mounts   # every configured mount

# keep a mount's sidecar fresh in the background
doxtr-rag --conf conf.py mount-watch --mount-path "$(pwd)/vault"
doxtr-rag --conf conf.py mount-watch --mount-path "$(pwd)/vault" --once
doxtr-rag --conf conf.py mount-watch --mount-path "$(pwd)/vault" --poll-interval 10

• mount-watch is single-consumer, guarded by a best-effort <mount>/.doxtr/mount.lock (one writer per sidecar). It uses the optional watchdog package for event-driven watching when installed, else polls (default 5.0s); --once does a single incremental pass and exits.
• Sidecar servers run on deterministic loopback ports based at 8790 (override DOXTR_RAG_MOUNT_PORT_BASE) and must be running for the query side to reach a mount — a build (or .tools/doxtr-rag-serve.py) starts them and publishes them into .doxtr/endpoints.json (the "mounts" array, see §6).
• Fail-closed scope: a mount is SHARED_ORG only if it affirmatively declares it in rag_external_mounts; otherwise LOCAL_PRIVATE.

9. Development & QA

Run the full quality gate (what CI enforces):

make qa

make qa runs, and fails on any of:

• pytest with ≥ 90 % coverage (--cov-fail-under=90), including the Python↔TypeScript cross-language integration round-trip and the dox reference full-build regression (a real sphinx-build of github.com/doxtr/dox with the extension enabled, asserting the store is populated through the real event lifecycle against a committed baseline);
• mypy --strict over doxtr_rag/;
• tsc --noEmit --strict over the Pi extension;
• vitest for the extension;
• the extension-verifier regression smoke check (the host dox HTML + light/dark PDF build must still succeed within its warning baseline).

Individual targets: make test, make mypy, make tsc, make tstest, make xlang, make dox-build, make dox-pdf, make verifier.

The dox-build regression (make dox-build / tests/test_dox_reference_build.py) is the guard that tells you if a new documentation construct or extension breaks doxtr-rag: it builds the whole reference project and fails if ingestion drops below the baseline floors in tests/baselines/dox_reference.json. Update that baseline only when the dox project legitimately changes. The dox-pdf regression (make dox-pdf / tests/test_dox_pdf_ingest.py) additionally compiles the reference project to a real PDF (LaTeX → lualatex) and ingests it through the PDF extractor, exercising the binary-document path against a real-world file. Both run in the doxtr/reactor container (full extension stack + LaTeX + pre-baked bge-m3 weights) and skip gracefully elsewhere.

Test layout

• tests/ — fast unit tests (schema, scrub, cache, harvester, extractors, security, audit, storage, pipeline, config, embedding) plus the cross-language round-trip, the ad-hoc ingest path (test_ingest.py, including the SSRF-redirect guard), the CLI subcommands (test_cli_commands.py) and the portable per-mount sidecar index (test_mount_index.py).
• test_harness/ — a real Sphinx project (conf.py, source/ with source//noter//NDA/ mounts, conf_overrides/, assertions.py, test_runner.py) for integration/regression, mirroring the reference doxtr-pdf-theme-core harness.
• doxtr_rag/pi_extension/tests/ — the TypeScript vitest suite (scrub parity, RRF, metadata inflation, dimension/model parity, offline fallback, and the per-mount sidecar query merge).