PyIntake

Knowledge-base synthesis for Python — document intake, chunking, embedding generation, vector + full-text storage, hybrid retrieval, suggestion-queue governance, resource-binding lifecycle, KB versioning, and a Next.js UI on top of it all. Sibling to pycharter: pyintake owns the synthesis layer; pycharter owns the governance layer (contracts, concept registry, field bindings).

• Repository: github.com/optophi/pyintake
• Status: a working knowledge-base product on PyPI (0.0.x): durable local tier, hybrid retrieval with a measured quality gate, grounded facts with tenant-scoped reads and verifiable erasure, cited answers over HTTP, MCP and the CLI, portable attested bundles, and a Next.js explorer that reads the live KB. What is decided and what comes next is in docs/proposals/kb-one-product.md and the curated backlog beside it.
• Roadmap: see docs/proposals/ — the decision record, the backlog, and the design records behind each shipped capability.

Install

pip install pyintake                # core (pydantic + PyYAML only)
pip install 'pyintake[api]'         # + FastAPI / uvicorn HTTP surface
pip install 'pyintake[ui]'          # + UI server proxy (FastAPI + httpx)
pip install 'pyintake[local]'       # the offline tier: db + persistent indexes + local embeddings
pip install 'pyintake[db]'          # + SQLAlchemy 2 + Alembic (persistent stores)
pip install 'pyintake[postgres]'    # + psycopg[binary] for Postgres
pip install 'pyintake[pdf]'         # + pypdf for PDF document sources
pip install 'pyintake[docx]'        # + python-docx for Word documents
pip install 'pyintake[ocr]'         # + OCR for PDFs with no text layer
pip install 'pyintake[ai]'          # + generated answers with verified citations
pip install 'pyintake[all]'         # everything (api, ui, db, postgres, pdf, ai, vector)

[local] is the recommended entry point for a self-contained knowledge base: one install, no server, no API key, and no network at query time. It brings a local embedding model, a durable sqlite-vec vector index and a durable FTS5 full-text index. The ai extra still reserves a namespace for the LLM adapters that land in a later round.

Start here: a knowledge base of your own

Three commands. No server, no API key, nothing leaves the machine.

pip install 'pyintake[local]'
pyintake create                    # durable KB here: config + schema + seed, idempotent
pyintake watch ~/notes           # keep a folder ingested as it changes
pyintake ask "what did I decide about the local tier?"

init selects the durable recipe, writes a pyintake.cfg beside the database, and reports every step — so the other commands need no flags and a re-run is a no-op rather than an error. watch polls each folder and ingests only what changed: an unchanged corpus costs one digest per file and no embedding work, and an edited file replaces its previous revision instead of accumulating beside it. A file that disappears from the folder is removed on the next scan. ask answers from the passages it retrieved, citing each one.

Reads .txt, .md, .html, .eml and .py with no extra; .pdf with [pdf], .docx with [docx], and scanned PDFs with [ocr]. Full walkthrough: A knowledge base of your own.

Indexes that survive a restart

vector.backend and fulltext.backend default to inmemory, which is right for tests, bundles and small in-process KBs — and means both indexes are rebuilt from scratch every time the process starts. A deployment wants the sqlite backends instead:

vector:   {backend: sqlite}   # vectors in `embeddings`, vec0 index derived from it
fulltext: {backend: sqlite}   # FTS5, derived from `chunks`

That is what data/seed/recipes/local-sqlite.yaml selects. Both indexes are projections of durable tables, so a damaged one is a rebuild rather than a re-ingest:

pyintake db check   --recipe local-sqlite.yaml   # does each index agree with its source?
pyintake db reindex --recipe local-sqlite.yaml   # rebuild both; needs no embedding model
pyintake doctor     --recipe local-sqlite.yaml   # every health check at once; --repair rebuilds

The vector index is a brute-force scan, so query cost is linear in corpus size — measured at ~46ms per top-10 query over 100k x 384 vectors. Comfortable to ~100k chunks and usable to ~500k; beyond that the answer is the Postgres tier:

vector:   {backend: postgres}   # pgvector, HNSW index
fulltext: {backend: postgres}   # tsvector + GIN, ranked by ts_rank_cd

That tier needs pip install 'pyintake[postgres]' and the pgvector extension on the server (CREATE EXTENSION vector;). Its search is approximate where the other two are exact — an HNSW graph can miss a true neighbour, which is the trade that makes ten million vectors searchable. See Retrieval backend tiers for what each backend is for and what is stood behind.

CLI

The pyintake CLI follows the fleet multi-subcommand pattern (same as pycharter):

Ten verbs, one name each on CLI, HTTP and MCP — see The verbs for the map from every old spelling:

pyintake create                                         # create a durable KB here
pyintake connect ~/notes --retention 30d                # declare a folder as a source
pyintake sync                                           # scan the connected sources once
pyintake watch ~/notes                                  # keep a folder ingested (sync, repeated)
pyintake search "arctic tern" -k 3                      # ranked passages
pyintake ask "what did I decide about X?"                # answer, with citations
pyintake review --accept ID                             # what agents proposed; decide
pyintake export ./bundle --scope public                 # portable, policy-filtered bundle
pyintake ingest notes.md paper.txt                      # one-shot files that are not a source
pyintake api                                            # start FastAPI (port 8100)
pyintake db init                                        # alembic upgrade head
pyintake db seed                                        # bundled recipes + default KBVersion
pyintake db current                                     # show active revision
pyintake db check --recipe RECIPE                       # verify derived indexes against their sources
pyintake db reindex --recipe RECIPE                     # rebuild derived indexes from stored data
pyintake ui dev                                         # Next.js dev server (port 3100)
pyintake ui build                                       # static export → ui/static/
pyintake ui serve                                       # serve built UI + proxy /api/*
pyintake worker run                                     # poll the connected sources (sync, repeated)
pyintake worker retention --dry-run                     # fact-vector TTL and document TTL
pyintake status                                         # last scan, empty/searchable docs, never-expires warning
pyintake doctor                                         # schema, indexes, providers, sources, tenancy, graph — exit 1 if any fail
pyintake mcp --scope internal                           # KB as MCP tools (stdio)

The MCP server's scope is an operator decision

pyintake mcp binds its policy scope when it starts, and every tool closes over it:

pyintake mcp                        # internal (default) — PII values withheld
pyintake mcp --scope public         # only public/embeddable facts
pyintake mcp --scope pii            # releases PII values; opt in deliberately

No tool takes a scope, an include_pii flag, or any other policy argument. A tool argument is filled in by the language model, so a policy control expressed as one is a control the model grants itself — a prompt-injected agent would simply ask for the wider scope. Widening means restarting the server with different flags.

--requester NAME labels the agent sessions the server records, so the Trust Ledger can attribute them to the consumer that asked.

Structured sources ground more precisely

A Python module is a document like any other — discovered by the same documents globs (add **/*.py to include), loaded by a source, chunked and indexed — read a second time for its structure. Symbols become chunks cut at definition boundaries with exact source spans, and relationships (defines, imports, calls, inherits, references) become graph edges marked with how they were established: extracted, inferred, or ambiguous. Uncertain targets are omitted rather than guessed.

pyintake ingest src/shop/billing.py --query "how is an order billed" -k 3

The same tools answer: kb_search hits carry symbol and span, kb_entity symbol:shop.billing.bill lists where it is defined and who calls it, kb_graph_neighborhood walks callers (direction: incoming) or callees (outgoing) with edge provenance, and kb_agent_query answers "what does bill depend on" from the graph. No new tool, and a public-scoped server sees neither symbols, spans nor paths. See Structured grounding.

Every list tool is bounded

A tool answers into the model's context window, so no tool returns the whole knowledge base. List tools page — kb_concepts, kb_search, and kb_ontology take limit / offset and return total plus a next_offset — and every bound is a ceiling the caller cannot raise, because a limit the model can set is a limit it can set to ten thousand.

The governed vocabulary is also exposed as MCP resources (kb://concepts, kb://ontology), so a client can attach it to context once instead of spending a tool call on it.

Agents can contribute, under review

kb_propose_binding lets an agent propose that a record's field means a governed concept. The proposal is inert: it lands in the existing review queue as pending, attributed to the server's --requester, and changes nothing until a human decides. kb_suggestions reads the queue back. Neither needs a flag, because neither changes the KB.

Deciding is a separate question:

pyintake mcp --allow-review     # adds kb_decide_suggestion, kb_resolve_fuzzy

Off by default. An agent that can propose and accept is an agent writing directly, with the review queue as decoration — so turning that on is an operator's call, exactly like the policy scope.

Provenance is never the model's to state: the proposer recorded on a suggestion is the server's requester, not a tool argument.

Environment variables:

Use it from Python

from pyintake import IntakeClient

client = IntakeClient.from_recipe("data/seed/recipes/default.yaml")
client.ingest("notes.md")
client.ingest("paper.txt")

for hit in client.retrieve("arctic tern migration", k=5):
    print(f"{hit.score:.3f}  {hit.document_id}  {hit.text[:80]}…")

Use it from HTTP

The v1 surface mirrors the IntakeClient:

GET    /healthz
POST   /api/v1/ingest                  POST   /api/v1/suggestions/{id}/{accept,reject,defer}
GET    /api/v1/retrieve                GET    /api/v1/bindings
GET    /api/v1/documents               POST   /api/v1/bindings/find
GET    /api/v1/documents/{id}          POST   /api/v1/bindings/deprecate
DELETE /api/v1/documents/{id}          GET    /api/v1/kb-versions
GET    /api/v1/suggestions             POST   /api/v1/kb-versions/{id}/activate
GET    /api/v1/suggestions/{id}        POST   /api/v1/kb-versions/{id}/migrate
                                       GET    /api/v1/recipes{,/current,/{name}}

curl -X POST http://127.0.0.1:8100/api/v1/ingest \
     -H 'Content-Type: application/json' \
     -d '{"text": "the arctic tern migrates between the poles every year"}'

curl 'http://127.0.0.1:8100/api/v1/retrieve?q=arctic+tern&k=3'

Every POST /api/v1/ask is filed under a request id, echoed as X-Request-ID; send your own and the evidence record carries it. See The verbs.

Bearer tokens are an operator decision too

By default the API authenticates nobody: every caller is anonymous and reads at the internal tier, which is the same deployment you get today. Set PYINTAKE_API_TOKENS and three things become true at once:

# subject:token:scope,scope — entries separated by ';'
export PYINTAKE_API_TOKENS='svc-indexer:s3cr3t-a:retrieve,ingest;web-widget:s3cr3t-b:public'
pyintake api

The third row is the point. A scope that literally names a policy tier clamps the caller to it, so an unauthenticated request now stops at public: /retrieve answers 403, /search withholds chunk snippets, suggestions withhold the quoted record. Give a token no scopes at all (svc:tok:) and it reads at the internal ceiling, like today's anonymous caller. Comparison is constant-time, and a token is never written to a log or echoed in an error — a rejected one is recorded by a truncated digest.

The token table is a secret. Keep it in the environment or a secret store; never in a recipe, a bundle, or anything committed. Deployments that authenticate some other way (JWT, mTLS, a sidecar) still override pyintake.api.dependencies.auth.get_auth_context directly and are unaffected.

Layered architecture

┌───────────────┐  ┌───────────────┐  ┌───────────────┐
│      UI       │  │  HTTP wrapper │  │      CLI      │
│  (Next.js 16) │  │   (FastAPI)   │  │   (argparse)  │
└───────┬───────┘  └───────┬───────┘  └───────┬───────┘
        └──────────┬───────┴──────────────────┘
                   ▼
        ┌──────────────────────────────────┐
        │      INTAKE CLIENT (façade)      │
        │  wires backends from a recipe    │
        └─────────────┬────────────────────┘
                      ▼
        ┌──────────────────────────────────┐
        │         STATELESS CORE           │
        │  pyintake.core.{pipeline,        │
        │  retrieval} — pure functions     │
        └─────────────┬────────────────────┘
                      ▼
   ┌──────────────────┼───────────────────┐
   ▼                  ▼                   ▼
 BACKENDS         GOVERNANCE +         KB-VERSIONS +
 (chunkers,        BINDINGS +          ORCHESTRATION
  embedders,       MATERIALIZE         + CACHING
  chunk/vector/    + WORKER
  fulltext stores) (Phase 2)           (Phase 4 + 5)
   │
   ▼
 SQLAlchemy 2 + Alembic  ──►  SQLite / Postgres
 (pyintake.db)

Two laws (inherited from the Optophi fleet):

1. Core is stateless — pyintake.core.pipeline.ingest_document and pyintake.core.retrieval.retrieve are pure functions that take Protocol-typed backends as keyword arguments.
2. Domain artifacts are configuration — chunkers, embedders, stores, and recipes are selected by YAML validated through pyintake.config.KBRecipe.

Architectural relationship with pycharter

            pygubernator (orchestration, planned)
                 ▲       ▲
                 │       │ consumes both
                 │       │
  pyintake ──binds── pycharter
  (synthesis)        (governance)

pyintake produces:

• Chunks (text + metadata) stamped with (model, model_version) for re-embedding idempotency.
• Embeddings and a hybrid retrieval surface.
• Suggestions + Resource-binding candidates — pyintake proposes; pycharter accepts and stores the canonical record.

pyintake calls into pycharter via the PycharterClient Protocol. The concrete HTTP adapter is the next item after the pycharter Phase 0 boundary-spec conversation completes (see pycharter/docs/proposals/decoupling-generalization-backlog.md CR-1..CR-5).

Is retrieval any good?

Nine query families over a labelled corpus (tests/eval), with regression floors in the normal test gate. Five measure ranking (lexical, paraphrase, distractor, multi-hop, passage); four measure the trust claim: temporal (the newer of a superseding pair is right), unanswerable (nothing answers — the agent should say so; today it does not, and the 0.000 abstain rate is asserted on the record), permission (a public-tier answer cites nothing internal — an invariant, 0.000 leak), code (a Python module found by symbol). scripts/eval.sh prints the scorecard for any recipe.

The unit suite pins retrieval behaviour; tests/eval/ measures its quality against a labelled corpus, so changing an embedding provider or a vector backend is a measurable change rather than an act of faith.

scripts/eval.sh                       # bundled default recipe
scripts/eval.sh my-recipe.yaml        # compare another recipe

The measured baselines — the hashing embedder and the [local] model on the same corpus and machine, with the test that says whether they differ — and the targets set from them are in Retrieval quality.

The default recipe uses InMemoryHashEmbedder, a hashing vectorizer: it matches terms, so it answers lexical queries perfectly and paraphrases barely at all. Swapping in a real model is what the local-embeddings recipe does, and the harness is how you can tell it worked:

                         default (hash)    local-embeddings (BGE-small)
lexical      top-1            1.000                1.000
paraphrase   top-1            0.250                0.750
distractor   top-1            0.750                0.875
overall      nDCG             0.740                0.922

pip install 'pyintake[embeddings-local]'
scripts/eval.sh src/pyintake/data/seed/recipes/local-embeddings.yaml

Embedding providers

The API key is read from the environment, never from a recipe — recipes are version-controlled.

Answer providers

Retrieval ends with a ranked list of passages; everything a person actually asked for happens after that. Composition is a pluggable backend like every other layer, selected in the recipe:

answer:
  provider: anthropic
  model: claude-opus-5

The safety property survives the upgrade, and it is structural rather than a matter of prompting:

• A provider only ever sees policy-cleared passages. The scope decision runs before composition, so a withheld fact is not in the prompt to be paraphrased out of. A provider is handed text and nothing else — no store, no policy tags, no scope — so it cannot widen a scope it cannot see.
• Every citation is verified afterwards. A marker naming a passage that was never supplied is stripped and counted. Surviving markers are never renumbered: closing the gap left by a dropped [1] by promoting [3] would produce an answer whose every citation resolves and whose citations are wrong.
• The policy caveats are pyintake's, not the model's. A generated answer still reports how many results were withheld, though the model was never told any existed.

A transport failure degrades to quoting the passages rather than returning nothing, and the answer records that it happened. A missing API key is deliberately not absorbed — that is a misconfiguration to fix, not to answer around forever.

Documentation

Build the MkDocs site locally:

pip install 'pyintake[docs]'
mkdocs serve

Runnable examples under examples/:

Site guides: PDF ingest end-to-end, KB ingestion (Phase A).

Contributing

See CONTRIBUTING.md, AGENTS.md, and ARCHITECTURE.md. AI assistants follow the specifications in .claude/CLAUDE.md and the skills under .claude/skills/.