pipeline/ — the zero-Claude layer

Convert · group · embed · rank · index · graph. This layer never calls an LLM.

Course-grade docs live in docs/ — install, vault adoption, the stage model, the CLI, search, converter failures, and the measured numbers. They are a Phase-1 deliverable (spec §16.1), written as the code was written. Start at docs/02-adopt-your-vault.md.

Not Claude, not a local model. The ONNX embedder is a ~130MB encoder — it has no prompt, no completion, no API key, and no network at runtime. That distinction is load-bearing: it is what makes "search works the minute you upload" true at zero Claude cost, forever (spec §2.3, §5.1).

Extraction and verification are middleware's job, because only middleware holds a human trigger. Nothing here is allowed to become a reason to call a model.

---

Install

Python 3.12 is pinned. Many wheels this depends on — onnxruntime above all — have no 3.14 build, and 3.14 is the system default on the build machine.

cd pipeline
uv venv --python 3.12
uv pip install -e ".[dev]"

One-time model download (~130MB into models/, gitignored). After this the layer is fully offline:

uv run cortex-pipeline model fetch

Optional converters

Document, audio and video conversion shell out to external binaries. They are not pipeline dependencies, deliberately — docling alone pulls in torch and costs 1.2GB, which has no business sitting in the same image as a 130MB encoder.

Tool	Handles	Install
docling	pdf/docx/pptx/xlsx, tables + OCR	uv pip install docling in a separate venv
markitdown	same formats, lighter, no OCR	uv pip install "markitdown[all]"
whisper-cli + a ggml model	audio	brew install whisper-cpp
ffmpeg	audio extraction	brew install ffmpeg
yt-dlp	video (audio-only)	brew install yt-dlp

Resolution order is PATH, then ~/.cortex-convert/bin, then the usual Homebrew locations. Override any of them with CORTEX_DOCLING, CORTEX_MARKITDOWN, CORTEX_WHISPER_CLI, CORTEX_WHISPER_MODEL, CORTEX_FFMPEG, CORTEX_YT_DLP.

A missing converter degrades, it does not crash: the item is marked failed with a reason, everything else keeps flowing, and health reports it.

---

The read-only vault rule (Phase 1, spec §16.2)

CORTEX Phase 1 adopts a second brain that already exists. So:

This layer never writes to, moves, or deletes anything inside the vault.

Enforced in vaultguard.py, not just documented — deny by default, see infer_read_only. Two things that used to write into a vault now cannot:

Artifact	Was	Now
graph.tsv	vault/00 Maps/graph.tsv	.cortex/graph.tsv
converted markdown	vault/_inbox/	.cortex/converted/

Media deletion after transcription — the one thing convert may remove — never fires on a file inside a read-only vault.

Only the repo's own vault/ is writable by default. Everything else — the buyer's real second brain, testvault/, anything — is read-only unless --writable-vault or CORTEX_VAULT_READ_ONLY=0 is passed.

During the build we target a duplicate at testvault/, rebuilt by scripts/sync-testvault.sh, so adoption can be iterated on aggressively — the reset button is one rsync. That is a build-time practice, not a product feature: in the shipped DIY course CORTEX points at the buyer's real vault in place (spec §16.2), and no code path assumes a duplicate exists.

Vault adoption

An existing vault has its own folder names, edge taxonomy and frontmatter. The §3 template assumes kebab-case PARA. Adoption maps; it does not migrate.

cortex-pipeline adopt --vault ~/vault --report adoption-report.md --json   # writes NOTHING
cortex-pipeline adopt --vault ~/vault --write-profile --json               # pin it, after review

The format is middleware's, not ours. vault.adoption.yaml at the repo root, defined by packages/agent/config/vault-adoption.example.yaml, is read by both layers. vaultprofile.py consumes it verbatim — same keys, same semantics. A second vault config would be a silent divergence, which is the failure mode BOUNDARIES.md exists for.

Two behaviours match middleware exactly:

• An unknown folder is left unclassified, never forced into the nearest role. It stays fully indexed and searchable; it just carries no role.
• An unlisted relation passes through unchanged. Edges are never renamed.

adopt drafts that same file rather than inventing a different artifact, and --write-profile is the only write it can make.

Run

# from the repo root
uv run cortex-pipeline <command> --json [args]

--json is always passed by middleware and is effectively always on. There is no human-formatted output mode, because a second output mode is a second thing that can silently diverge from the contract.

# first run against an existing vault, end to end
cortex-pipeline model fetch --json                                  # once, ~130MB
cortex-pipeline adopt --vault ~/vault --report adoption.md --json   # dry run, writes nothing
cortex-pipeline --vault ~/vault index rebuild --json
cortex-pipeline --vault ~/vault search --query "when do we refuse to quote" --json

Commands: adopt · search · ingest · status · queue · graph · audit · note · index rebuild · health · sources · measure · eval · model fetch.

Global flags are accepted before or after the subcommand — a caller that gets the order wrong gets JSON, not an argparse usage message on stderr with empty stdout. scripts/check-cli-contract.sh asserts the whole boundary, failure paths included.

The output contract (docs/pipeline-cli.md)

• stdout carries exactly one JSON document. Nothing else, ever.
• Progress, warnings and timings go to stderr as NDJSON, one object per line. Long commands stream {"stage":"embed","done":41,"total":900} for progress bars.
• Exit 0 on success. Non-zero writes {"error":{"code":...,"message":...}} to stdout, using CortexError.code values from cortexos-types.

Every payload deserialises into cortexos-types: SearchResult, GraphNeighborhood, GraphAudit, IngestProgress, PromotionQueue, SystemHealth, Note, Page<NoteRef>.

---

The stage model (spec §5.2)

Stages 1–4 are ZERO_CLAUDE_STAGES. 5–6 belong to middleware and do not exist here.

1 · Convert — stages/convert.py. docling primary, markitdown fallback for documents; whisper for audio; yt-dlp for video, audio-only, concurrency 1, media deleted after the transcript (spec §2.5 calls video the landmine, and a 127MB recording once blocked a git push for four days). Output is markdown carrying source provenance in frontmatter — into .cortex/converted/ for a read-only vault. Streaming and chunked, never whole-file-in-RAM.

2 · Group + normalise — stages/group.py. Content-hash exact duplicates and simhash near-duplicates collapse into clusters. Nothing is ever deleted. Every file stays retrievable; exactly one member per cluster carries clusterRepresentative, and that is the only one Claude is later paid to read. Boilerplate and signatures are stripped for processing only — the original on disk is untouched. Dates, emails, amounts and URLs are regexed out here.

3 · Embed + cluster — embedder.py + stages/embed.py. Local ONNX (bge-small-en-v1.5), chunked with overlap, then k-means into ~30 topics so Claude later sees clusters rather than files.

4 · Rank — stages/rank.py. The promotion queue, scored on retrieval count × task reference × cluster centrality × recency × entity density. Every entry carries a human reason string, because the Sources screen shows it verbatim. This is the end of the free pipeline, and search is fully live at this point.

Chunk offsets are provenance

Provenance.chunkOffset / chunkLength are byte offsets into the original file on disk, not into a normalised copy. The adversarial verify pass re-reads exactly that span, so chunking.py works in bytes end to end rather than encoding and guessing. If those numbers drift, the entire quality stack in spec §6 is built on sand — hence tests/test_chunking.py.

Index

SQLite, and nothing else: FTS5 (keyword) + sqlite-vec (semantic) + an edge table. Fully derived from markdown. Delete .cortex/index.db and index rebuild reconstitutes every row. The repo is truth; this file is a cache with legs.

• index rebuild — full rebuild, also on demand or corruption.
• index rebuild --incremental --paths a,b — exactly those paths, called by middleware post-commit from the write path, never from polling.
• index rebuild --incremental with no paths — resumes a killed sweep from the checkpoint.

Search fuses both arms with Reciprocal Rank Fusion — bm25 and cosine are not on comparable scales and RRF needs no per-corpus tuning. Titles get 4x bm25 weight and are prepended to the embedded representation (never to the stored chunk text, which stays byte-exact for provenance). Every hit carries a Citation. When the source date is unknown the citation says "date unknown" rather than omitting it — spec §6.4: if it can't cite, it says so.

If the semantic arm is unavailable the search returns keyword-only results with degraded: true. It never fails shut.

Graph

graph/build.py ports the vault's build-graph.py. ## Edges sections parse into graph.tsv (source \t relation \t target), which stays in the vault's human-greppable form — that file is meant to be read with awk. SQLite additionally stores each endpoint resolved to a node id.

Edge line grammar. The list marker is optional — docs/pipeline-cli.md pins the bullet-less form and the vault writes the bulleted one, so both parse. Requiring the bullet made middleware's documented form yield zero edges, silently:

causes:: [[Some Note]]                                    # the pinned contract form
- causes:: [[Some Note]]                                  # the vault's form
- proposed-causes:: [[Some Note]]                         # below the confidence gate
causes:: [[Some Note]] <!-- prov: {"sourceFile": "..."} -->

The ## Edges heading itself is configurable per vault via the profile.

Wikilinks resolve by filename stem, note title, and the slug of either — the vault writes [[Exact Filename]] while the CORTEX template uses kebab-case paths, and both must resolve or the whole graph reads as dangling edges.

Two classes of edge. Typed edges come from ## Edges sections; untyped [[wikilinks]] are emitted as references, deduplicated against typed edges so a typed edge is never re-emitted as an untyped one. On a real vault this took orphans from 330 (59%) to 11 (2%) with the typed count unchanged at 1,087. Off via index rebuild --no-wikilinks or graph.include_wikilinks: false.

graph queries take depth, relation filter and a hard node limit so the canvas never receives a 40k-node payload; clipped results report truncated: true.

audit reports orphans, dangling edges, low-confidence density, contradictions, never-retrieved nodes and expired as-of stamps. Contradictions are surfaced, never resolved — detection is conservative on purpose, because a false contradiction is worse than a missed one.

Backpressure, not OOM

• Checkpoint after every file (checkpoint.py), written atomically. An OOM kill resumes from the checkpoint instead of restarting.
• A memory-aware gate pauses the queue as RSS approaches the limit (CORTEX_MEMORY_LIMIT_BYTES, default 1.5GB).
• Never co-resident: whisper, docling and the embedder each run in their own process, which exits afterwards. measure enforces this by spawning one child per stage — which is also the only way to get a per-stage peak RSS that means anything.

---

Measured numbers

Full numbers, the machine, and the Fly sizing recommendation: eval/m1-measurements.md, regenerated by:

cortex-pipeline measure --corpus ./corpus --report eval/m1-measurements.md --json

A run where any stage crashed raises rather than reporting a plausible-looking number, because spec §17 forbids quoting a hosting cost before a real measurement exists and a fake one is worse than none.

---

Eval — the release gate

eval/questions.yaml holds the question → expected-source set of spec §6.6. Regression = don't ship.

cortex-pipeline eval --set eval/questions.yaml --json

Zero Claude, fully deterministic: a question passes when hybrid search returns one of its expected sources inside the top k. No model judges anything, which is exactly what makes it trustworthy as a gate.

version: 1
default_k: 5
questions:
  - id: money-numbers-guardrail
    question: "what am I not allowed to say publicly about deal size?"
    expect_any: ["30 Resources/Beliefs/Content Constraints.md"]
    k: 5
    tags: [guardrail]

The corpus is frozen. The gate runs against eval/snapshot/ (gitignored), pinned by eval/snapshot.manifest.json (committed), and refuses to run if a single byte moved. Live uploads once moved the score 46/50 → 45/50 with no code change; a gate whose input drifts cannot tell a regression from a corpus change. --against live still runs the set against the real vault as a labelled drift check that can never block a release.

Do not tune retrieval against this file. The 5 recorded failures are documented with their reasons in eval/known-failures.md — headroom for the detector, not bugs to paper over.

---

Tests

cd pipeline && uv run pytest

Fixtures are small committed text; no large binaries and no model download. The suite never loads ONNX — the keyword index and the fusion logic are exercised directly — so CI stays fast and needs no network.

---

Config

Reads cortex.yaml from the repo root and tolerates unknown keys by design. Paths are overridable with CORTEX_VAULT, CORTEX_STATE_DIR, CORTEX_MODELS_DIR, or the --vault / --state-dir / --repo-root flags.

What this layer will not do

• Call an LLM. Any kind. Ever.
• Commit to git. Middleware owns the single commit queue with the writer lock. This layer exposes index rebuild for middleware to call post-commit.
• Delete anything from the vault. Dedupe groups; it does not remove. The one documented exception is a recording deleted after a successful transcript.
• Write outside pipeline/, scripts/ and eval/.