context-engine

Retrieval that enforces access control and PII redaction inside the query,
as a library, on your own Postgres.

Built by Promptev — the governed AI agent platform. This library is the retrieval engine behind Promptev's Context Packs, open-sourced (Apache-2.0). Use it standalone, or see what the platform adds.

Most retrieval stacks make permissions your problem: you filter results after the fact, in application code, and hope the ordering is right. This one pushes the ACL into the SQL predicate every search leg shares, so a document a caller may not see is never fetched, never ranked, and never reaches a reranker. PII redaction sits at the same depth — masked before hits are POSTed to a hosted reranker and before the tool-call audit row is written.

It is a library, not a server. No UI, no orchestration, no workflow engine, no opinion about your framework — pip install, point it at a Postgres you already run, call engine.search(). BYO everything: your database, your embedding provider, your LLM. It never calls a hosted service you didn't configure, and never implements auth, billing, or credential storage — it exposes hooks (on_usage, on_error, on_progress) and injection points (auth, principals) for your app to wire up.

On top of that it does the ordinary work well: structure-aware ingestion for PDF/DOCX/PPTX/XLSX/HTML/images, hybrid retrieval (full-text + trigram + vector, RRF-fused) with an optional graph leg, and a governed tool-execution path with approval gates and audit.

Not overclaiming: it does not prevent hallucination or guarantee grounding, and its compute() action (LLM-generated Python) is off by default — only enable it behind real OS-level isolation.

📖 Full documentation: https://promptev.ai/documentation/context-engine/

In this repo: access control (the trichotomy, enforcement, and the vector-index interaction) · the ACL-recall benchmark · cookbook · TypeScript client (@promptev/context-engine)

Why you'd choose this

1. Access control that survives the vector index. Filtering an approximate index with a WHERE clause is a post-filter — the index picks candidates first, your ACL throws them away second, and the query returns short. We measured it on 200k public documents at production embedding width, and found something the literature misses: because real permissions are topically clustered rather than random, the loss is about twice what a conventional benchmark reports — recall 0.473 against 0.980 at identical visibility, a quarter of queries empty, no error raised. Tuning does not fully close it, so this library decides between an exact and an approximate scan per query instead. How access control works · benchmark and method.

Suspect your current stack? Prove it on your own data — one read-only command, no embedding provider called:

context-engine check-acl-exposure --database-url postgresql://...

2. Redaction that runs in the right place. Not more PII detectors than everyone else — better placement. Hits are masked before the reranker POSTs them to a hosted provider and before the audit row is written, so the masked value is the only one that ever leaves the process. Wiring a redaction library into a pipeline by hand is exactly where that ordering goes wrong. (Bring Presidio for breadth — its entities become ordinary rules.)

3. It's a library, so the data never moves. No server to run, no corpus to hand over, no orchestration framework to adopt. If you're subject to rules where you stay accountable for what your AI did — the EU AI Act's Article 12 logging duties fall on the deployer, not the vendor — then "runs in my process, on my Postgres" is the answer you need, and a hosted service can't give it to you.

4. Governed tool execution in the same place as retrieval. ACL-scoped tools, encrypted credentials, a fail-closed approval gate with single-use atomic resolution, and an audit row per call. Most stacks buy this separately from a gateway.

When you outgrow the library

The library is the engine. Promptev is the platform it was extracted from. It is the same engine either way; what differs is how much of the surrounding plumbing you run yourself.

Your database, your model keys and your connectors stay your choice either way — run them yourself, or have Promptev run them. That is a deployment decision, not a difference between the two:

What the platform adds on top of the engine:

Self-hosting the engine is a first-class path and stays one — it is Apache-2.0 and never calls home. The platform is for when the plumbing around the engine becomes the job.

Install

pip install "promptev-context-engine[postgres]"
# or with uv:
uv add "promptev-context-engine[postgres]"

# before it's on PyPI, install from git:
pip install "promptev-context-engine @ git+https://github.com/promptev/context-engine"
uv add "promptev-context-engine @ git+https://github.com/promptev/context-engine"

TypeScript lives in js/ as @promptev/context-engine — same Postgres schema, so a corpus ingested by either client is searchable by the other:

npm install @promptev/context-engine pg

Needs Node 22+.

Needs Python 3.12+ and Postgres with the vector, pg_trgm, and unaccent extensions (auto-created by context-engine migrate if the role has CREATE EXTENSION).

The core engine is web-framework-agnostic — it depends on no web framework. Pick the router adapter that matches YOUR project (FastAPI, Flask, or Django) via an extra, or use none and call engine.* directly.

Optional extras — install only what you use:

pip install "promptev-context-engine[fastapi,graph]"   # or [flask] / [django]
uv add "promptev-context-engine[fastapi,graph]"

Teach your coding agent the access-control rules

Most integration code is now written with an agent's help, and the highest-cost mistake in this library is one an agent makes readily: principals=None means trusted caller, no filtering, so treating it as "nobody is logged in" returns the entire corpus.

That spelling is being retired. None is falsy, so it is what every accessor degrades to — user.groups or None, a missing key, if user else None — which made the unauthenticated path the maximum-privilege path. Say it explicitly instead:

from context_engine import TRUSTED

await engine.search(q, principals=TRUSTED)      # skip ACL, deliberately
await engine.search(q, principals=[])           # anonymous
await engine.search(q, principals=user.groups)  # scoped

None still works and warns; it raises in 1.0. The package also ships a skill stating this and the other invariants worth not guessing.

context-engine install-skill            # -> ./.claude/skills/context-engine/
context-engine install-skill --global   # -> ~/.claude/skills/
context-engine install-skill --print    # stdout, for agents that read other formats

It is an explicit command, never an install hook — the file becomes instructions inside your agent, so it should be your decision. An edited skill is never overwritten without --force.

Configure

ContextEngineConfig is the one settings object (or build it from CE_-prefixed env vars, nested with __, e.g. CE_EMBEDDING__PROVIDER).

from context_engine import ContextEngineConfig, EmbeddingConfig, LLMConfig

config = ContextEngineConfig(
    database_url="postgresql://user:pass@localhost:5432/mydb",
    embedding=EmbeddingConfig(provider="openai", model="text-embedding-3-small", api_key="sk-..."),
    # llm is optional — only for structured extraction, compute, and the graph:
    llm=LLMConfig(provider="openai", model="gpt-4o-mini", api_key="sk-..."),
)

Embeddings: openai · azure_openai · gemini · vertex_ai · voyage · cohere · custom (any OpenAI-compatible /embeddings endpoint via base_url). LLMs add anthropic · bedrock. (Anthropic has no embeddings API — pick another provider for EmbeddingConfig.)

anthropic now goes through the official anthropic SDK rather than a hand-rolled HTTP call, which is what lets it retry a 429 or a 529 like every other SDK-backed provider. It is a core dependency, pinned >=0.77,<1: the 1.x line is built on httpx2 and cannot accept the guarded httpx client the engine injects, and keeping a host-supplied or engine-guarded client on every outbound call matters more than the major version. The request body is unchanged and the SDK's telemetry headers are suppressed.

Three host-visible consequences. base_url on an anthropic LLMConfig is now honoured — it used to be ignored and the endpoint hardcoded, so a deployment that set it was talking to the public API without being told. An empty or unset api_key now fails at the call — the SDK raises TypeError("Could not resolve authentication method. Expected one of api_key, auth_token, or credentials to be set…") — instead of sending an empty x-api-key and collecting a 401. Separately, the errors a live call raises are the SDK's typed ones: an unrelated 400 is anthropic.BadRequestError rather than httpx.HTTPStatusError, and a connection failure or timeout is APIConnectionError with the httpx error as __cause__. Neither ANTHROPIC_API_KEY nor ANTHROPIC_BASE_URL is consulted — both are always passed explicitly, so the config decides the credential and the endpoint; ANTHROPIC_CUSTOM_HEADERS still injects the headers it names.

vertex_ai is the same Gemini models served from Vertex AI, and the only difference is how you authenticate: Application Default Credentials against a GCP project instead of an API key.

llm=LLMConfig(provider="vertex_ai", model="gemini-2.5-pro",
              project="my-project", location="us-central1"),

project and location are optional — leave them out and the Google SDK resolves them from GOOGLE_CLOUD_PROJECT / GOOGLE_CLOUD_LOCATION, so an existing GCP deployment needs no new configuration. The Python SDK will also discover the project from ADC when neither is set; the TypeScript client's SDK will not — measured, not assumed — so for @promptev/context-engine name the project in config or in the environment.

Credentials themselves come from ADC in every form: GOOGLE_APPLICATION_CREDENTIALS, workload identity, or gcloud auth application-default login. There is no credentials field to set. api_key is ignored when a project is present (the SDK rejects the pair outright); on its own it selects Vertex express mode.

Retries. max_retries on LLMConfig and EmbeddingConfig is the number of retries on top of the first attempt — default 2, 0 disables retrying entirely, and it reads from the environment like everything else (CE_LLM__MAX_RETRIES, CE_EMBEDDING__MAX_RETRIES, CE_VISION_LLM__MAX_RETRIES, CE_GRAPH__EXTRACTION_LLM__MAX_RETRIES). The retrying itself is the provider SDK's, because each one already classifies its own retryable errors and already honours Retry-After: openai / azure_openai / custom, gemini / vertex_ai, bedrock and anthropic all honour the field. The reranker and the voyage / cohere embedding paths do not — they are raw HTTP with no SDK to delegate to — and batch submission never retries, because batches.create is not idempotent and a retry after a 5xx that actually landed is a second paid job.

The call timeout is per ATTEMPT, so a call's ceiling is (max_retries + 1) x timeout + backoff — about 12 minutes at the defaults (240s x 3) against an endpoint that accepts connections and never answers. It layers: extract_structured_data(max_reasks=1) re-asks the model after an unusable reply, so one document is about 24 minutes, and vision's three batch attempts multiply again. Set max_retries=0 to get back the single-attempt behaviour, which is what a latency-sensitive interactive path wants. extract_structured_data's re-ask parameter used to be called max_retries and is now max_reasks, so the two names cannot be confused; passing the old one by keyword is an error.

Every outbound call the engine makes — LLM, embeddings, reranker, tool HTTP, MCP OAuth — can run on a client the host supplies instead of one the engine builds: ContextEngine(config, http_client_factory=lambda purpose: ..., mcp_resolver=...). http_client_factory is asked for an httpx.AsyncClient per purpose (llm · embeddings · reranker · tool_http · mcp_oauth, context_engine.HttpClientFactory / HttpClientPurpose); returning None keeps the engine-built client. It must be a long-lived, host-owned client — the engine never closes it, and it is called far more than once (per LLM call, per extraction attempt, per tool call). mcp_resolver is the resolver MCP transports resolve through. tool_http/mcp_oauth host clients still get the per-hop egress check and redirect refusal on every call; google-genai (gemini/vertex_ai) and boto3 (bedrock) manage their own transports and are out of scope. TypeScript: new ContextEngine(config, { fetchFactory, mcpLookup }), handing back a fetch for the same five purposes.

StorageConfig holds the deployment knobs: ann_exact_threshold (the vector leg's exact/approximate crossover — hardware- and dimension-specific, see docs/access-control.md) and the connection pool, passed through to SQLAlchemy — pool_size (5), max_overflow (10), pool_timeout (30s), pool_recycle (1800s, keep it under the idle timeout of any pooler in front of the database) and pool_pre_ping (on: without it every connection left stale by a database restart or failover fails one query first). All reachable as env vars, e.g. CE_STORAGE__POOL_SIZE.

Provision the database

context-engine migrate --database-url postgresql://user:pass@localhost:5432/mydb --dim 1536
# add --graph to also create entity/relationship/community tables

--dim must match your embedding model's width (1536 for text-embedding-3-small) and is locked in on first migrate.

migrate does not write-lock your tables. Every index it adds after the initial schema is built with CREATE INDEX CONCURRENTLY, so ingest keeps writing while the build runs. The build is slower than a plain one and it waits for transactions that were already open when it started, which is the trade for not stopping writes. migrate prints one line to stderr per index it builds, naming it, so a long wait is visibly a wait and not a hang; SELECT * FROM pg_stat_progress_create_index shows how far along it is. An index that is already present and valid is neither rebuilt nor announced.

A failed migrate is safe to re-run, and re-running is the repair. It is not one transaction: a failure part-way leaves the revisions before it applied, every statement it runs is idempotent, and running it again picks up where it stopped. An interrupted build (a cancelled statement, a killed process) leaves Postgres holding an unusable index; the failing run drops that partial copy when it can, and the next migrate drops and rebuilds whatever remains. The error says which of the two happened. If migrate stops and names an index, the cause is in the error reported with it: for ux_context_engine_tool_approvals_pending_scope that means duplicate pending approval rows to clear first.

A lock_timeout or statement_timeout on the connection you run migrate with applies to these builds, and to the cleanup that follows a failed one. migrate will not override it, so on a database with long-lived write transactions either raise it for the migration or run when those transactions are short.

Graph mode runs on Postgres alone. graph.enabled needs only extraction_llm; neo4j_uri is optional, and leaving it unset is a supported deployment shape rather than a degraded one. Entity and relationship extraction, the whole Postgres mirror, Louvain community detection and its summaries, and all four navigation actions (get_neighbors, traverse, find_related, community_summary) are Postgres-side and unchanged. What a Neo4j-less deployment gives up is multi-hop expansion: the graph leg's candidate set stops at the ACL-scoped vector seeds instead of widening to chunks two hops away through shared entities, so "what else is connected to this" narrows toward what the vector leg would have found anyway. Nothing is re-weighted — the connectivity signal falls back to one constant for every candidate, which cannot reorder a sort, and the legs are fused on rank. Neo4j stays all-or-nothing inside graph mode: a URI without a password is an error, and a password without a URI is now an error too, because it is otherwise indistinguishable from a dropped URI silently downgrading a Neo4j deployment. A Postgres-only deployment still needs the [graph] extra for networkx (community detection is not optional); the neo4j driver is imported only when a URI is set.

Deleting a document repairs the graph it contributed to. With graph.enabled, delete_document now also, after the document and its chunks are gone: recomputes frequency for the entities that document mentioned, deletes the ones nothing mentions any more (their relationships follow through the endpoint cascade), and deletes the communities that touched any of those entities. frequency therefore changes meaning — it is now the number of distinct chunks that mention the entity, which is read by the community-summary prompt (retrieval counts its own and never read the column). A deleted community is rebuilt by the next ingest(mode="graph"), which re-detects corpus-wide; until then a community that merely shared an entity with the deleted document is gone corpus-wide too, so the community leg has fewer summaries to draw on. When Neo4j is configured its chunk nodes are removed after the Postgres side commits, and a Neo4j failure never fails the delete — it is logged and reported to on_error with stage="delete_document_graph". Without graph.enabled a delete does no graph work at all.

Ingest

from context_engine import ContextEngine

engine = ContextEngine(config)

# Plain text
report = await engine.ingest(
    text="Annual leave accrues at two days per month for every band-three employee.",
    name="leave-policy",
    source_id="hr-handbook",       # a namespace to scope search/stats
    acl=["group:hr"],              # omit for an unrestricted document (visible to everyone)
)

# A file — PDF/DOCX/PPTX/XLSX/HTML/EML/images/plain text, auto-detected
with open("policy.pdf", "rb") as f:
    report = await engine.ingest(file=f, name="policy.pdf", source_id="hr-handbook")

print(report.totals)               # {"files": 1, "failed": 0, "units": 3, ...}
print(report.documents[0].status)  # "completed" | "failed" | "batch_pending"

Construction is cheap and lazy — no connection is opened until the first call that needs one. Release the pool (and the embedding client, and the Neo4j driver) with await engine.aclose(), or use the engine as an async context manager; a process-lifetime singleton can skip it, anything building engines per job or per test should not.

• mode="hybrid" (default) or mode="graph" (needs the [graph] extra).
• Scanned pages / images are transcribed to structured Markdown (tables and headings preserved) when vision_llm is configured — otherwise embedded text, with Tesseract as a fallback if [ocr] is installed. Vision is provider-agnostic: it goes through the same call_llm seam as everything else, so Claude, GPT-4o, Gemini and Bedrock all work — whatever your LLMConfig names. The [vision] extra only adds page rasterization.
• Born-digital PDFs keep their structure too, with the [pdf] extra: the text layer is converted to Markdown locally — no OCR, no LLM, no network — so a table survives as a table into the chunker instead of being flattened into a run of words. Without the extra you get today's plain-text extraction.
• A re-ingest only pays for what changed. Identical content is still deduplicated by hash and skipped outright; when the content did change, every chunk whose text did not keeps the vector the previous version already paid for — copied inside the database, never read into the process — and only the changed chunks go to the provider. A chunk matches on a hash of its whitespace-normalised text, and only when the same embedding provider/model/dimension produced the stored vector. Synchronous ingest only: batch=True submits the whole chunk set and re-embeds all of it, because a batch result is paired with the document's rows by position. Two callers racing the same document are settled by the database, not by hope: the loser of the identity race reports skipped with skip_reason="in_progress" and writes nothing while the winner is still running — including the ACL and attribute updates that request carried, so re-offer the document once the winner is done.
• Chunks are budgeted to min(2000, embedding.max_input_tokens). A chunk is one embedding input, so it is held to the provider's per-input cap — 512 tokens on cohere, where a constant 2,000 would fail the document before the first request. Changing provider or embedding.max_input_tokens therefore re-chunks each document on its next ingest, and the vectors that would have been reused are paid for once more.
• Two intake caps, both refusing rather than trimming. ingest.max_file_bytes (256 MiB) is answered without reading the document at all — os.stat for a path, Content-Length for an HTTP upload. ingest.max_rows (unset) needs the bytes, and it is EXTENSION-SCOPED: only .csv, .tsv and .xlsx are counted, because the filename extension is also what routes a file to a tabular reader at all — a spreadsheet saved as .txt, or with no extension, is ingested as text and never counted. The count at the door is a pre-check; the streaming reader re-counts as it reads and THAT count is the authoritative one — it is what settles a newline inside a quoted field and an XLSX sheet's declared row count, and it refuses at the row that passes the limit instead of after parsing the whole file. Over either raises IngestTooLarge, which the routers map to 413 with a message naming the knob. Both take None to lift the limit, and neither ever samples or truncates: a silently shortened document is a corpus that answers wrongly and looks healthy doing it. routing_core.upload_limit_bytes(engine) hands the byte cap to your own upload middleware.
• ingest() never raises on a bad document — check report.documents[i].status.

How a large document is embedded

Chunks are stored before they are embedded, and the embedding requests are packed by tokens as well as by count. What that buys you:

• A batch is closed by whichever cap it reaches first — items or tokens — against a per-provider table, written items / tokens per request / tokens per input: openai-family 256 / 250k / 8,192; gemini and vertex_ai 100 / — / 2,048; voyage 128 / 120k / 32k; cohere 96 / — / 512. Override any of them with embedding.max_batch_items, embedding.max_batch_tokens and embedding.max_input_tokens (all unset = the table). A spreadsheet is what makes this necessary: 256 rows of a wide sheet is a legal batch size and 450,000 tokens in one request is not.
• Tokens are counted, not guessed, when they can be. embedding.tokenizer="auto" (the default) uses cl100k through tiktoken only if its encoding file is already in the cache and estimates otherwise — it never downloads, because the ingest path may not fetch anything unasked. "exact" is the explicit opt-in that may fetch; "estimate" never imports it. The estimator is UTF-8 bytes ÷ embedding.token_bytes_ratio (3.0), divided again by embedding.safety_margin (0.85) for headroom; the margin applies to the batch budget only, never to max_input_tokens.
• A single chunk over max_input_tokens raises InputTooLarge instead of being split — half a chunk's vector on a row that stores the whole chunk is a silent wrong-answer bug. Cohere requests carry truncate: "NONE" for the same reason.
• A provider that rejects a batch for size halves the cap and retries, for that run only; if it still refuses a one-item batch you get EmbeddingBatchRejected naming both suspects. 429s honour Retry-After within embedding.max_retries (2); other failures back off with jitter.
• embedding.max_concurrency (4) requests are in flight per document — enough to overlap network latency, not enough to earn the deployment a 429.
• A failure keeps every batch you already paid for. The document sits at the new non-terminal status embedding with meta_data.embedding_progress = {done, total, tokens}, and await engine.resume_embeddings(document_id) embeds only the rows still missing a vector — no re-extract, no re-chunk, no second bill for the finished batches. Called with no id it sweeps every resumable document, fenced by ingest.resume_stale_seconds (600) so a cron job cannot take over a run that is still moving; naming an id is never fenced. The trade is stated plainly: a mid-embed failure no longer leaves the PREVIOUS version indexed, because the new body was written when the run claimed the row. A partly embedded document is still findable — the text legs see every chunk, the vector leg only the embedded ones.
• Progress is visible. on_progress gains a third state "progress" on the embed stage, one per completed batch, carrying {done, total, tokens}; started/done are unchanged and ingest.progress_batches (on) is the escape hatch for a consumer that cannot ignore an unknown state. The embed: log line reports chunks reused embedded requests tokens counter concurrency halvings retries for one document. A custom StorageBackend joins in by implementing set_chunk_embeddings(document_id, ids, vectors, *, progress=None, embedding_key=None) — the engine always passes the key, an opaque provider\tmodel\tdim string to store verbatim beside the vectors, because a later re-ingest reuses a vector only when it matches; drop it and every row of that backend reads as "unknown" and is never reused. A Python backend that implements neither method still works, it just writes the vectors once at the end. EmbedBatchFailed carries how far the run got.
• A file that could not be read raises ExtractionFailed, with whatever the extraction library threw chained as __cause__. Two things are NOT wrapped, because both are caught by type upstream: ImportError (a missing optional extra, which carries an install hint) and IngestTooLarge — the streaming tabular readers apply ingest.max_rows as they go, theirs is the authoritative count, and every router maps that type to 413 with the knob, the limit and the actual row count in the body.

Why a document failed

A failed document carries two answers, on purpose, because two different readers need different things.

• error is the raw str(exc) of whatever stopped the run. It is an OPERATOR's copy: an embedding provider's error is a body the provider wrote, and those have carried an API key, a model name, a request id and a quoted fragment of the document itself. engine.get_document, engine.list_documents and DocumentReport.error all keep it.
• failure_reason is one code from a fixed vocabulary, classified from the exception rather than from its text: input_too_large, provider_rejected, provider_unavailable, extraction_failed, chunk_set_changed, unknown. failure_message is the one fixed sentence for that code — never interpolated with anything the provider said.

The model-facing doors return the code and the sentence and never error: the knowledge tool's get_doc, get_docs and list (and therefore the MCP tool, which is the same function), plus GET /documents/{id} on every router. A router mounted with principals=lambda: TRUSTED is an operator's own surface and still gets error; an anonymous or principal-scoped caller does not. list carries the code only — a census of fifty documents does not need fifty copies of one of six sentences.

A document that later succeeds clears both fields.

What an ingest usage event says

hooks.on_usage is the metering seam — this package never touches billing — and every UsageEvent(kind="ingest") carries the same two mode fields in its detail, on every path that emits one (a completed ingest, a resumed one, a completed batch, a mode upgrade, a structured-extraction backfill):

• mode — the mode the document was actually PROCESSED at, never the one that was asked for. On the resume and batch paths that is the mode stored on the row, because those runs decide nothing: they finish what an earlier run started.
• mode_reason — why that is not the requested mode, as one short fixed string (currently only "tabular documents stay hybrid"), or None when the two agree. The same value DocumentReport.mode_reason carries, and it is present on every event, None included: a key that only appears on the downgrade path is a key a consumer cannot switch on.

A spreadsheet ingested with mode="graph" therefore bills as hybrid with that reason, on the first run and on every later one.

A DOCX is billed by the page count the file reports, not by a guess: pages comes from the word processor's own <Pages> (docProps/app.xml), else from the page breaks the last render left behind, else from hard and section breaks, and only falls back to the ~250-words-a-page estimate when the file says nothing. pages_source — "app_xml", "rendered_breaks", "page_breaks" or "estimate" — says which, and rides beside pages everywhere it already goes: Extracted, DocumentReport, the document's meta_data, the extract progress event and this event's detail. It is None for every other format. The declared count is believed only as far as the body could hold it — blocks (paragraphs and table rows) plus breaks, plus one — so a one-kilobyte upload declaring a hundred thousand pages is not a hundred thousand units; the cost of that rule is that a document laid out over many pages from ONE enormous paragraph falls back to the estimate. It stops a bare declared number and nothing more: padding a document with empty paragraphs satisfies the bound as cheaply, so page-based metering stays uploader-controlled — bound the cost with ingest.max_file_bytes and your own quotas on the usage events.

Spreadsheets and CSVs

CSV/XLSX documents are indexed in hybrid mode with one schema chunk per sheet; aggregate questions go to compute. The detail:

• Rows are emitted as real RFC 4180 CSV (quoted commas, doubled quotes, embedded newlines survive), streamed rather than materialised — a 22 MB CSV chunks in well under a quarter of a gigabyte of peak memory instead of failing. Empty unnamed columns are dropped; a named-but-empty column is kept, because the name is a fact.
• Each sheet gets a leading chunk with meta.chunk_kind == "schema" naming the columns, their inferred types and a few sample values, and pointing the model at compute for arithmetic. It is an ordinary chunk — hashed, embedded, retrieved and redacted like any other.
• A tabular document stays hybrid even in a graph-mode corpus and says so: report.documents[i].mode_used == "hybrid" with a fixed mode_reason. Its rows are a frame to compute over, not a graph to walk, and the per-chunk extraction LLM would be spent on thousands of near-identical lines.
• ingest.max_document_text_chars (unset) caps the stored documents.text. Leaving it unset is load-bearing: compute rebuilds its dataframes from that exact column, so a cut body is a spreadsheet that quietly answers with fewer rows than it has. Setting it is RECORDED — meta_data.text_truncated is written on every ingest, true or false, so raising the knob heals it — and both compute and discover refuse a truncated document by name rather than computing a wrong answer.

Search

result = await engine.search(
    "how much annual leave do I get",
    source_ids=["hr-handbook"],
    document_ids=None,         # optionally pin to specific documents WITHIN the sources
    principals=["group:hr"],   # TRUSTED = trusted/internal (skips ACL); [] = anonymous
    top_k=10,
)

for hit in result.hits:
    print(hit.document_name, hit.score, hit.chunk_text[:80])

Fuses full-text + trigram + vector search with Reciprocal Rank Fusion; mode="graph" adds a graph leg. compress_to_tokens=N trims weak hits to fit a prompt budget.

Thresholds follow the query's shape

A fixed threshold is wrong for some query whatever value it takes, so pg_trgm's similarity threshold moves with the query (context_engine.query_shape, the same rules in both ports). A two-character query needs a strict threshold or % degenerates into noise; a twelve-word one needs a loose one or it matches nothing: 0.45 down to 0.15, measured in characters for spaceless scripts (Chinese, Japanese, Thai, Khmer, Burmese, Lao, Tibetan) and in words for everything else. This is what the trigram leg has always done.

There is a matching floor for the vector leg, and it is OFF by default. An embedding model returns the nearest chunks, never "nothing is close enough", so a minimum cosine similarity chosen from the query (0.45 for one or two words, 0.40 for a code or an acronym, down to 0.25 for a sentence) drops the rest from the vector leg before fusion — those rows can still arrive through full-text or trigram, and a floor that empties the vector leg leaves the search running on the other legs.

The two exact-lookup shapes are checked FIRST and end it there. A query that asks for an exact string (query_type reads it as exact_match) or that contains an acronym floors at 0.40 before the word-count ladder is consulted at all, so a two-word acronym query such as VAT rules is 0.40, not the 0.45 its length alone would give it. Both are looking for a token, and the vector leg's answer to a rare token is the chunks least far away from it rather than the ones that hold it.

config = ContextEngineConfig(
    database_url=...,
    embedding=...,
    search=SearchConfig(vector_floor="adaptive"),   # opt in
)
# or, on a config you already have:
config.search.vector_floor = "adaptive"            # a typo here RAISES

With the floor on, the vector leg can return fewer candidates than limit, and nothing backfills them. That is the point — a dropped row was weak — but it means fusion sees a shorter list from that leg, and on a corpus where every chunk is far from the query the leg can come back empty. The search still runs on full-text and trigram.

Measure before you turn it on. Those numbers came from one embedding model. Cosine similarity bands differ between models — some score every pair of texts above 0.7, some cluster everything near 0.2 — so an absolute floor can empty the vector leg for short queries on a model it was not measured against, and a search that returns nothing is a worse failure than one that returns a weak row.

The graph leg is chosen by the QUERY

Leaving mode out (the default) no longer means "this corpus has a graph document somewhere, so use graph mode for everything". The leg runs when the query itself resolves to an entity the graph knows and this caller is allowed to see — otherwise one graph document would re-rank every search in the corpus by its own vocabulary, which is exactly how six chunks of an unrelated document came to fill an answer. Naming mode="graph" bypasses that gate but not the entity set: a query that names nothing has no graph question to answer and gets the hybrid answer.

The result says which of those happened, additively — every existing usage key kept its place:

result.usage["mode"]       # "graph"  — what you asked for
result.usage["mode_used"]  # "hybrid" — what actually ran
result.usage["graph"]      # {"applied": False,
                           #  "reason": "no_entities_in_query",
                           #  "entities": []}

# ... and for a query that does name one — entity names come back in the
# NORMALISED form they are stored under, not as the query spelled them:
#   {"applied": True, "reason": "requested", "entities": ["apple inc."]}

reason is one short fixed string from context_engine.GRAPH_REASONS — not_requested, graph_disabled, graph_unreachable, no_graph_documents, no_entities_in_query, graph_detection_failed, no_visible_graph_chunks, auto, requested — so a host can switch on it instead of parsing prose. graph_detection_failed is the mode-less path saying it could not work the mode out at all and fell back to the hybrid legs; an explicit mode="graph" never reports it, because that caller gets the error instead. One failure is never degraded on either path: a missing [graph] extra raises its install hint, because that is a configuration error and a deployment that believes it has a graph must be told it has not — install the extra, or set graph.enabled=False, which is answered before any import is attempted and reports graph_disabled. entities lists only the entities the query itself named, never the neighbours the leg reached through them, in their normalised (lower-cased, whitespace-collapsed) stored form. The usage event counts graph units only when the leg actually contributed.

Four knobs shape the leg, all on GraphConfig:

The leg is bounded by the same per-leg candidate limit as the other three, and only chunks with an actual graph signal reach fusion — a bare vector seed is already the vector leg's, at its proper rank, and counting it twice is what crowded better matches out. Entity resolution is index-backed (migrate --graph creates the trigram and expression indexes it needs); the fuzzy pass covers the first ~400 query tokens, so an entity named only in the tail of a very long paste may be missed.

Compute over tables

engine.compute() turns the in-scope CSV/XLSX documents into dataframes and has the LLM write code against them. compute_over_frames() is the same path for frames you already hold — an uploaded workbook, a connector's sheet, a query result — with no document to point at:

from context_engine import compute_over_frames

out = await compute_over_frames(
    {"expenses": df},              # {sheet name: DataFrame}
    "total the amount column",
    config=engine.config,
)
out["result"]

Both are off by default (enable_code_execution=True turns them on — only behind real OS-level isolation) and both apply config.redaction before the LLM sees anything: every string cell and every column header is masked, and the returned dict — generated code included — is swept on the way out.

The prompt states what the sandbox refuses, and code refused by the static check buys ONE rewrite: the model gets its own code and the refusal back and answers again, and the second attempt goes through the same check. A frame key that does not exist buys the same one rewrite (see below); no other runtime error is retried. out["attempts"] is 1 or 2 either way, and both calls' tokens are summed into out["provider_tokens"], so a retried run costs two code-generation calls. on_error fires once per FAILED attempt, with the attempt number in its context ({"stage": ..., "attempt": 1}): a retry that succeeds fires one event with attempt == 1 while attempts == 2, and a run that ends in failure fires one whose attempt equals attempts. attempts counts calls ATTEMPTED, so counting "how often the model hits the sandbox limits" from it alone over-counts by the provider-failure rate — the attempt plus stage: "compute_codegen" pair on the error event is how to tell a provider failure from a refusal.

What the frames are called. Over ONE document in scope the key is the SHEET name (dfs['sheet1']), unchanged. Over several, a document with exactly one sheet is keyed by its document NAME (dfs['revenue.xlsx']) and a document with several keeps <name>:<sheet> (dfs['book.xlsx:Q1']); two documents that share a name both fall back to <name>:<sheet>, and a key that still collides after that gets #2, #3 appended, because collapsing two frames into one would drop a document from the answer silently. The keys depend on the document SET and never on the order it arrives in — the suffixes are assigned in document-id order, so discover (which lists caller-first) and compute (which loads newest-first) give the same document the same key. Every sheet is a frame, a header-only one included: it has columns and no rows. The prompt lists the keys and says to use them exactly as listed, and a run that asks for a key that is not there fails with no frame named 'revenue.xlsx'; available: ['revenue.xlsx:Q1', 'revenue.xlsx:Q2', 'costs.csv']; did you mean 'revenue.xlsx:Q1'? — handed to the model for its one rewrite, and reported as out["error"] if the rewrite is wrong too.

The in-process sandbox behind enable_code_execution is defence in depth, not a security boundary. To run the generated code somewhere actually isolated — a subprocess, a container, a gVisor/WASM sandbox — pass code_runner= to ContextEngine(config, code_runner=...) or directly to compute()/ compute_over_frames(): a callable that runs the same (code, context, timeout) shape out-of-process (context_engine.CodeRunner names the shape). That hook is the real isolation boundary; its result is trusted only as far as JSON. Code bound for a host runner is refused only for the base reasons (unsafe imports, dunder names/attributes, open/eval/exec and the other dangerous builtins, .system/.popen/.exec/.eval); the in-process sandbox's extra refusals — file-I/O names such as to_string/read_csv, the same names string-dispatched through agg/apply/query, attribute assignment, yield, functools.update_wrapper/wraps, and any .format()/.format_map() that is not directly on a string literal — apply only when the code runs in-process. Two limits are not static refusals at all: in-process the sandbox has no class machinery, so defining a class or calling type() with three arguments fails when the code RUNS, not in the checker (and isinstance(x, type) raises with it — type inside the sandbox is a one-argument function). The code-generation prompt states whichever set will actually apply.

When the code does run in-process, out["result"] is plain data: the sandbox converts it to None/bool/int/float/str/list/dict before returning, so a DataFrame arrives as a list of row dicts (with a non-default index moved into leading columns, so groupby/describe keep their labels), a Series as a dict, a numpy scalar as a plain int/float, a date or timestamp as an ISO-8601 string, and anything else with no plain-data form as its repr. That conversion happens inside the sandbox's runtime guard and uses type-level calls only, so nothing the generated code left attached to the result can run afterwards — in the redaction sweep, in your own code, or in a JSON encoder. Ask for a number, a string, a list or a dict (which is what the prompt already asks the model for) and nothing is lost.

Two shapes are worth knowing before you parse the result. A DataFrame whose column labels are not unique comes back as {"columns": [...], "data": [[...], ...]} and a Series with repeated index labels as {"index": [...], "values": [...]}, because row dicts would silently drop every duplicate but the last — and masked headers make duplicates real (two e-mail columns both become [EMAIL]). A result past the call's output cap (max_output_length) or nested deeper than 50 levels is cut with a marker string rather than raising; the marker is an ordinary string, indistinguishable from identical text the data itself contained. The generated code is handed COPIES of the frames and a deep copy of the documents, so it cannot reach the objects you went on using, and out["documents_used"] is built from ids captured BEFORE the run — as str(...) of each id, so UUID ids come back as strings.

Schema-enforced replies

Every place the engine asks a model for JSON — community summaries, entity extraction, vision page transcription, structured extraction — sends a JSON Schema the provider enforces instead of hoping the reply parses. Nothing is configured; it applies to whatever LLMConfig names, with a fallback for a model or endpoint that predates native support:

A downgrade fires only on an error that names the schema parameter — an "invalid schema" error is a bug to fix, not something to work around — and every genuine one is logged at WARNING naming the provider. Once a fallback has actually worked the endpoint is remembered for the process, keyed by provider, base URL and model, so the refusal is paid once rather than per call. That memo has no expiry: a long-lived host that upgrades an SDK or a model calls reset_native_schema_support() (context_engine.providers.llm) to let the engine rediscover native support.

Bedrock's native path needs botocore 1.43.0 or newer — the release that added Converse's outputConfig, and strict on a tool spec with it. boto3 is not a declared extra here (a bedrock deployment brings its own), so that floor is prose rather than a resolvable constraint. An older SDK is detected on its own parameter validation, and because toolSpec.strict landed in the same release the fallback tool cannot carry strict either: the model is asked to fill the tool's schema rather than held to it, so that call honestly reports mode="prompt" and its reply is validated locally. A service-side refusal on a capable SDK is the case that does get a strict tool, and mode="tool" with it.

mode is the guarantee, and it is on every reply. native and tool mean the provider enforced the schema, so a reply that still breaks it is refused rather than re-asked — paying twice for the same bug fixes nothing. prompt means the shape was only asked for in words: the reply is validated locally before anything trusts it, and one failure buys exactly one retry that carries the validation error and the model's own answer back. A second failure is loud.

from context_engine.providers.structured import structured_call

result = await structured_call(
    config.llm,
    system="Extract the invoice fields.",
    user=text,
    schema={"type": "object",
            "properties": {"total": {"type": ["number", "null"]}}},
    schema_name="invoice",
)
result.value       # validated against the same bytes the provider was sent
result.mode        # "native" | "tool" | "prompt"
result.attempts    # 2 means the prompt path needed its one retry
result.tokens      # summed over every attempt

structured_call returns a StructuredResult or raises StructuredCallError (StructuredRefusal when the provider refused in the field it reserves for saying so). Both errors carry mode, attempts, tokens — a failed attempt was billed like a successful one — and raw, the full final reply, for a call site that wants to run its own tolerant reader over it. raw is deliberately kept out of logs: read the attribute, don't dump the exception. call_llm(..., response_schema=, schema_name=) is the layer below, if you want the mechanism without the validation.

map_reduce is the one JSON call that sends no schema — its shape is the caller's, not the engine's — so it stays on JSON mode and reports mode: "prompt" in its map_reduce_parse error context. structured_repair_count() is the number that says whether any of this is being honoured: how many enforced-mode replies needed the prose helper this process. It should stay at zero; a non-zero value means some provider accepted a response schema and then answered around it.

Schemas travel through the portable subset — the intersection of every provider's strict mode. Objects, arrays, strings, numbers, integers, booleans, nulls and $ref into a root $defs; nullable is a type array (["string", "null"]); every object gets additionalProperties: false and all its properties required; enums hold scalars only. No minItems/maxItems, no pattern/format, no anyOf/oneOf/allOf, no recursion. A schema outside it raises SchemaError before any request goes out — strict_portable() from context_engine.llm_schemas is the linter, and it is what normalises the schema that is both sent and validated against. For a shape the subset cannot describe at all — an object whose keys come from the document rather than from you — pass enforce=False: no schema goes on the wire, JSON mode does, and the reply is validated here against the schema as written (so that schema must state its own required and additionalProperties — nothing fills them in). It relaxes what may be described, never what is checked: the schema still has to stay inside the keyword vocabulary both ports' validators implement — format is not in it — and one that strays raises SchemaError before any paid call.

Host-visible effects, per call site:

• Structured extraction with field_hints returns EXACTLY the hinted keys. The schema is built from the hints and enforced, so each value is the hint's scalar type or null and extra keys are no longer possible. Without hints the payload stays free-form (enforce=False, envelope validated).
• A document with no readable quality signal is now quality="failed". The envelope's readability is an enum and a missing or invalid one is a validation failure; it no longer falls back to "medium".
• New on_error stages: structured_call_attempt and structured_call from the wrapper (per failed attempt and terminal, with mode and attempt), structured_call_repair when a provider that said it would enforce the schema returned something that is not JSON, plus entity_extraction and structured_extraction at the call sites — a failed extraction fires exactly one structured_extraction event, carrying document_id, mode and attempts. A host sees both the wrapper's per-attempt events and the site's outcome event — attempt telemetry and outcome are different questions — so a batch rescued on its retry still produced wrapper error events.
• hooks= is now accepted by extract_text_from_images and extract_pdf_pages, so vision failures reach on_error like everything else. A vision batch keeps its own three attempts around the wrapper's one retry, so the prompt path can cost up to six paid calls per batch.

Serve

The same document/search/stats surface ships as a ready adapter for FastAPI, Flask, or Django — pick the one matching your project. All three expose identical routes (POST/GET/DELETE/PATCH /documents, POST /search, GET /stats) with identical behavior (they share one framework-neutral core). auth and principals are required arguments — the package implements no auth, and resolving who the caller is has no safe default. To run a mount open on purpose, say so: principals=lambda: [] (anonymous — sees only documents with no ACL). Returning TRUSTED (import it from context_engine) means a fully trusted caller and skips ACL filtering entirely; returning None is rejected with a 500 so a missing resolver can never open a mount by accident.

A caller may only file a document under an ACL it already holds; POST /documents and PATCH /documents/{id} answer 403 otherwise, so an untrusted request can't push content into another group's retrieval scope.

FastAPI ([fastapi] extra):

from context_engine import create_router

router = create_router(engine, auth=my_auth, principals=my_principals)
app.include_router(router, prefix="/context")

Flask ([flask] extra) — auth(request) / principals(request), async views:

from context_engine import create_flask_blueprint

bp = create_flask_blueprint(engine, auth=my_auth, principals=my_principals)
app.register_blueprint(bp, url_prefix="/context")

Django ([django] extra) — include the urlpatterns in your urls.py:

from django.urls import include, path
from context_engine import create_django_urlpatterns

urlpatterns = [
    path("context/", include(create_django_urlpatterns(engine, auth=my_auth, principals=my_principals))),
]

Not on any of them? Just call engine.ingest() / engine.search() directly from your own views — the engine needs no web framework at all.

Or serve the same engine as an MCP tool surface ([mcp] extra):

context-engine mcp --database-url postgresql://... --port 8080

from context_engine import create_mcp_app, UNSCOPED
app.mount("/mcp", create_mcp_app(engine, principals=my_principals, scope=UNSCOPED))

scope is the ceiling of source ids this mounted tool may ever reach, and is REQUIRED. It is resolved per call like principals and is never a tool argument: a model can ask for any source id it likes, and a tool that believed it would let one caller read another's documents. Map your own word onto it — a project, a matter, a customer — and pass scope=lambda: ["src-a", "src-b"], or Scope(source_ids=(...), document_ids=(...)) to hand an agent specific documents. A host with one shared corpus writes scope=UNSCOPED on purpose. The caller can narrow within that ceiling and never widen past it.

Registers ONE knowledge tool, search_knowledge_base, whose action picks what it does:

Every graph answer carries the entity id beside the name — id on the start entity and on each neighbour, target_id on a traversal path, source_id/target_id on a find_related relationship — and entity accepts one. That is the handle that survives an output redaction policy: where a policy masks the names, the name in an answer is a mask token and matches nothing, so navigation would otherwise stop after one hop. The id carries no document text, so it is exempt from that masking, and it is a handle rather than a capability: it resolves behind the same visibility predicate a name does, so an id the caller cannot see gives the same entity not found an unknown name gives. It is stable while the entity exists, not a permanent handle — rebuilding the graph mints new ids. A name still works wherever it was not masked — the question, a search hit, the documents. With no unmasked name to start from, find_related is the way in: it takes a category rather than an entity, and its answer hands back the source_id/target_id to navigate from.

Combining the tools into sub-actions of one is better for the model that calls them: one description carries the decision rules once, and there is one name to route through.

Every search hit carries chunk_idx, the position of the chunk it matched inside its document, beside document_id, document_name, chunk_text, score and source_id. It is the argument get_chunks takes as start, so reading around a passage is one call rather than paging a long document from the beginning. A hit found by file name carries it too, pointing at the first chunk it showed, and a document with no chunks at all reports null.

discover answers with the SCHEMA, not an inventory, so one call is enough to write one correct compute or query_meta call instead of three exploratory ones. Every listed document carries document_type, mode, and a structure whose shape follows the type — sheets (name, column headers, row count, and frame_key: what compute will call that sheet's dataframe, over the documents THIS call listed) for a workbook, sections and last_page for a document with headings, keys for JSON, and chunks as the floor for anything else. Beside them, fields_by_type lists the extracted structured field names with each field's data type and how many in-scope documents carry it, and document_types is a census of the corpus. Both are keyed by (kind, type): kind comes from the mime and is always known, while document_type is free text an LLM wrote and exists only where extract_structured was enabled — so a PDF invoice and a spreadsheet of invoice rows stay separate rather than merging under one label.

Lists inside a structure are capped, with the remainder reported as more_columns / more_sections / more_keys / more_sheets, because discover is the first call of a conversation and an unbounded list lands in the model's context before it has asked anything. The cap is for the call that did not name its documents: discover and list take document_ids, and a discover scoped to specific documents answers their structure whole and narrows the field grouping and the census to them too. While the remainder is small enough to be worth fetching, the truncated structure carries that call already filled in; past that it says what to do instead, because a sheet with hundreds of columns is one to compute over rather than read back.

Omit mode on a search and it is worked out from the documents in scope — the graph leg is added when the deployment has a graph and something in scope was ingested that way, and it runs beside the other legs rather than ahead of them. Naming "hybrid" or "graph" forces one.

An AGGREGATE question — a total, an average, a count, a ranking, a group-by, a numeric comparison — is REFUSED when every document in scope is a spreadsheet. search and query_meta answer {"success": false, "error": …, "next_action": {"action": "compute", "query": …, "suggestion": {"frame_key": …, "measure": …, "group_by": …}}} instead of returning rows. query is the question handed straight back, so the redirect is a call to make rather than advice to translate; suggestion beside it is the vocabulary, not an argument, and it is plain DATA rather than code: frame_key is the name compute will really give that sheet's frame (the same rule discover reports, computed over the documents in this call's scope), and measure / group_by are the newest sheet's first numeric and first text column. Any of the three can be null. Search returns EXCERPTS, so totalling what it returned is totalling a sample and reporting it as the figure. A MIXED scope still searches and only carries an additive hint; a scope with no spreadsheets in it, or with nothing visible to the caller, is untouched. map_reduce is refused the same way for ANY instruction, aggregate-shaped or not — it reads a SAMPLE of each document, so the answers it merges over sheets are partial totals — and the refusal lands before any model call and before a host-supplied runner; there the question is asked of the documents that CALL would read (the same ordering and the same limit or 25 cap map_reduce itself selects by, resolved first as ids and then READ as ids, so the decision and the fan-out cannot be about different documents), so a scope of thirty whose newest twenty-five are sheets is refused rather than sampled, and a mixed target set carries a hint with its own reason. The decision is one grouped query over the documents the call would read, under the caller's ACL and the host's ceiling. Set redirect_aggregates_to_compute=False on the config to turn it off.

A question that NAMES A FILE — "get detail about INV-EG.pdf", "what is in report_2024.xlsx" — searches document NAMES instead of chunk text, and answers one row per document with matched_by: "filename" beside the ordinary hits (document_name is the name, chunk_text the document's first chunk and chunk_idx that chunk's position). Chunk text is the wrong haystack for it: the file name appears inside the document only by accident, so the one document the caller asked for is the one thing the text legs cannot reliably find. The name match is a trigram similarity on the lowercased name, under the same scope predicate and ACL as every other document read, with an exact name always first and the document id as the final tiebreak. When nothing is called that — including when the only document with that name is one the caller may not see — the ordinary search runs on the original query, so the branch can only add an answer, never remove one, and the call is metered once rather than twice. The aggregate refusal above is asked FIRST, so "what is the total in sales.csv" still goes to compute. With redirect_aggregates_to_compute=False and this branch left on, that same question is answered from the named file's FIRST chunk, which for a spreadsheet is the schema chunk: a sample of the columns, never a total. Turning the redirect off is the operator's decision, and so is that answer. Three things it does not do: mode is ignored when it answers (there is no text leg to rank), top_k is the row cap with the same floor of 1 and no ceiling search applies, and detection is ASCII-only — a document named rapport-café.pdf never triggers the branch and is answered by the ordinary search. Its usage carries legs: [] and leg_hits: {} — present and empty, because no leg ran — and omits candidates, reranked, compressed and provider_tokens, which describe work this path does not do. The rule is query_shape.filename_in_query, a pure function on the module (not exported from the package root). Set filename_search=False on the config to turn it off.

query_meta and map_reduce need an LLM, compute needs code execution explicitly enabled, and the graph actions need a configured graph — discover marks them off otherwise, and an off action called anyway answers a plain {"success": false, "error": …} result instead of hiding. discover and list page (has_more + next_cursor, sent back as cursor), and a truncated answer carries the next_page call that reads the rest. The connector-tool gateway (search_tools + execute_tool) is registered beside it — search_tools takes the same kind / name_contains / requires_approval filters and cursor as the method, and answers {"tools": [...], "next_cursor": ...}.

knowledge_tool_definition() exports the tool as data — name, description and input schema as JSON Schema — so a host wiring it into its own agent loop never hand-writes the schema. create_mcp_app also takes redaction (a policy applied per call), and compute / map_reduce callables that REPLACE the built-in ones, which is where a host applies its own rules for permission, billing and approval.

The same tool is a plain method, so an application driving its own agent loop needs no MCP at all — and cannot get a different answer, because both doors call one implementation:

answer = await engine.search_knowledge_base(
    action="search", query="annual leave", principals=caller, scope=["hr"]
)

Tools

Let an LLM/agent call http / db / mcp (external server) and function (a Python callable) tools through one governed path — ACL-checked, config AES-256-GCM-encrypted at rest, every call audited.

from context_engine import ToolConfig

# HTTP tool — the LLM fills `city` at call time
await engine.register_tool(ToolConfig(
    name="get_weather", kind="http",
    description="Fetch the current weather for a city",
    config={
        "method": "GET",
        "url": "https://api.example.com/weather",
        "headers": {"Authorization": "Bearer super-secret-token"},
        "llmQueryParameters": {"properties": {"city": {"type": "string"}}, "required": ["city"]},
    },
))  # -> call_name "http_get_weather"

# Function tool — no config, no persistence
def add(a: int, b: int) -> int:
    """Add two integers."""
    return a + b
engine.register_function_tool(add)  # -> call_name "fn_add"

out = await engine.execute_tool("http_get_weather", {"city": "Lahore"})
# -> {"result": {"status_code": 200, "data": {...}}, "usage": {"kind": "tool", "units": 1}}

Two tools with one name. Nothing makes a tool's name unique, so two registered tools can share a call name. Every tool that list_tools and search_tools return carries its tool_id; hand it back to say which one you mean. Without it a name that matches more than one tool the caller can see raises AmbiguousToolError (409 on POST /tools/execute) instead of running whichever row loaded first. The name is still required, and a tool_id the caller cannot see reads as "tool not found":

hit = (await engine.search_tools("weather", principals=["user:a"]))[0]
out = await engine.execute_tool(hit["name"], {"city": "Lahore"},
                                principals=["user:a"], tool_id=hit["tool_id"])

Discovery — search_tools ranks on the query and filters on everything else. kind (one of http/db/mcp/function, or a list), name_contains (case-insensitive substring of the call name) and requires_approval combine with AND, so an empty query with filters lists the filtered set, and an unknown kind is an empty result rather than an error:

page = await engine.search_tools(
    "", principals=["user:a"], kind="db", requires_approval=False, limit=20
)
if page.next_cursor:  # only present when more matches remain
    rest = await engine.search_tools(
        "", principals=["user:a"], kind="db", requires_approval=False, limit=20,
        cursor=page.next_cursor,   # same query + filters, or it is refused
    )

The page is the list it has always been, so a caller that ignores paging is unaffected. The walk is a live offset rather than a snapshot: a tool registered or deleted mid-walk shifts the tail.

A non-empty query also EXCLUDES: a tool none of its terms match is not in the answer, so search_tools("nonsense", kind="db") returns nothing rather than every db tool. The query ranks within what the filters selected — it does not replace them. To list without hiding anything, pass an empty query with the filters you want, or call list_tools, which never hides a tool.

Approval gate — a tool with requires_approval=True returns an approval_required payload instead of running, until resolved:

from context_engine import resolve_approval

out = await engine.execute_tool(
    "http_get_weather", {"city": "Lahore"},
    principals=["user:a"],
    approval_scope="run-42",   # opaque, chosen by you — a run id is the usual shape
)
# -> {"approval_required": {"approval_id": "...", "expires_at": "...", ...}}   (not executed yet)
await resolve_approval(engine, approval_id, "approved", approver="boss")
# the same call, same approval_scope, runs for real — exactly once

approval_scope is the claim scope: an approved record is consumed only by the same call in the same scope (and, when the record carries principals, by a caller who overlaps them). Within a scope, identical pending requests share one record, so a retrying agent gets the same approval_id back. Resolve the scope server-side — on the routers and the MCP gateway it is a dependency beside principals, never a request field. Calling without it still works this release, unscoped and with a DeprecationWarning; the next release refuses a gated call with no scope.

An approval is for one tool, not for a name. The record carries the tool_id it was opened for (None for a function tool), the approval_required payload and ApprovalRecord both show it, and an identical call to another tool that shares the name opens its own record instead of claiming this one. migrate adds the column and swaps the pending unique index for one that includes it; a record opened before that carries no tool_id, so it is no longer claimable by a registered tool and the call simply asks again.

Getting the request in front of a human (Slack, in-app) is the consumer's job — the engine owns only the single-use, atomic approval state machine. Park until expires_at; after that the record can no longer be approved.

Result budget — the returned result is cut to 8,000 characters and 100 rows unless you say otherwise. Size it to your model's window, or pass None to lift a limit. response_mode="tsv" works for any tool: every array of objects in the result — a db tool's rows, an HTTP API's data.items, a function's returned list — comes back as a TSV string under the same key, and the budget cuts at whole rows:

out = await engine.execute_tool(
    "db_sales", {"query": "SELECT * FROM orders"},
    principals=["user:a"],
    result_max_chars=200_000,  # None = no character limit
    result_max_rows=None,      # per table; a db tool's own max_rows (default 1000) still applies
    response_mode="tsv",       # -> {"success": True, "columns": [...], "rows": "id\tname\n1\t..."}
)
# a cut reports where: {"_result_shaping": {"rows": {"rows_returned": 812, "rows_omitted": 188}}}

In TSV, \N is NULL, nested values are compact JSON cells, and tab, newline, CR and backslash inside a value are backslash-escaped. Parts of a result that aren't arrays of objects stay JSON; a result that is one becomes the string (wrapped as {"result": ..., "_result_shaping": ...} when cut). The audit row always keeps the original under its own 50KB cap, whatever you pass.

An http tool can set its format once, in its config — for every method (GET, POST, PUT, PATCH, DELETE, QUERY). Omitted, it is json; a caller's explicit response_mode= still wins:

await engine.register_tool(ToolConfig(
    name="orders", kind="http",
    config={"method": "QUERY", "url": "https://api.example.com/orders",
            "response_mode": "tsv"},   # "json" (default) | "tsv"
    acl=["group:ops"],
))

The same conversion is a plain function for anything else: format_result(result, "tsv") (and rows_to_tsv(rows)) from context_engine — "json", the default, returns the input unchanged.

A failed call's error text is stripped, then masked, and the caller, the audit row (context_engine_tool_calls.error_message) and the on_tool_call hook all get the same reduced text. A db driver's message loses what carries row data first: where the driver offers structured fields the message is built from those, otherwise the first signal line is kept and the DETAIL/HINT/CONTEXT/LINE n/caret lines, the [SQL: ...]/[parameters: ...] echo and a statement echoed back verbatim are removed, and every single-quoted literal is blanked to '?' (double-quoted identifiers — the constraint and column names that say what broke — are kept). What survives then goes through the call's redaction policy, principals and key, the same ones the success path uses, so a per-tenant policy applies to failures too. If the masking itself fails — a hash rule with no key, discovered only once the tool has already run — all three surfaces get error text withheld: the redaction policy could not be applied instead of the unmasked text, and the real reason is logged for the operator. The exception type and the failure shape are unchanged; only the text is.

Connecting an mcp tool that needs OAuth ends in a popup on your origin that postMessages the result to window.opener. On failure that message is now {type: "mcp-oauth-error", code, message, correlation_id}. code is the stable name to branch on, one of idp_denied, malformed_callback, state_invalid, token_endpoint_unreachable, token_exchange_failed and internal_error; message keeps its key but is one of six fixed sentences written for the person looking at the popup, not the underlying error text — so a host that string-matched on it must read code instead. correlation_id is mcpoauth_<32hex>, shown on the page as Reference: … and logged server-side beside the detail that is deliberately not rendered or posted; the success payload carries it too. The provider's error_description, a token endpoint's response body and any exception text reach the log and nothing else.

The six sentences, so a host building its own fallback UI can match or replace them — identical in both clients:

They are the wording, not the contract — branch on code, and treat a sentence as something that can be reworded in any release.

Redaction

Span-level masking of PII (or anything you can describe with a regex or a function) as content is ingested, retrieved, or returned. It composes with ACL (ACL = who may see a document; redaction = what's masked inside it) and is a pure transform — no model, no network. The default policy is a guaranteed no-op.

from context_engine import ContextEngineConfig, RedactionPolicy, RedactionRule

config = ContextEngineConfig(
    database_url="postgresql://...",
    embedding=...,
    redaction=RedactionPolicy(rules=[
        RedactionRule(name="emails", detector="email"),                  # -> [EMAILS]
        RedactionRule(name="order_id", pattern=r"\bORD-\d{6}\b"),         # -> [ORDER_ID]
        RedactionRule(name="acct", pattern=r"Account:\s*(?P<value>\d+)"), # -> "Account: [ACCT]"
    ]),
)

A pattern rule whose regex names a group value (either (?P<value>...) or (?<value>...) — both are accepted, normalised on load) masks only that group's span instead of the whole match; a match where value didn't participate contributes no masking at all.

Built-in detectors: email · phone · ssn · credit_card · iban · api_key. The two phone forms have different rules. Without a country code it needs a - or a . between the groups — the same one in both gaps — or the (415) 555-0123 form, optionally behind a 1/+1: (415) 555-0123, 415-555-0123, 415.555.0123, 1-415-555-0123 and +1 (415) 555-0123 all match, while 415 555 0123, 415-555.0123 and a bare 4155550123 do not. Behind a leading + the rule is looser — a + then a digit run that may be split by spaces or dashes — so +14155550123 and +1 415 555 0123 both match, though a dotted +1.415.555.0123 does not. Actions: mask · hash (keyed, joinable pseudonym — needs redaction_secret_key, which falls back to secret_key) · remove. apply_at: output (default, masks on every read, can be principal-conditional via unless) · ingest (scrub before storage) · both.

hash normalises the matched text per detector before HMACing it, so one real-world value written two ways gives one token: phone keeps the decimal digits only, folded to ASCII (a non-ASCII decimal digit — Arabic-Indic, say — counts as its ASCII value, since the detectors match those too), and a 10-digit match is then hashed as if it carried a +1 — that is a HASHING rule, not a detection one, and a bare 4155550123 in text is still not a phone number at all; email trims and lower-cases; ssn and credit_card fold digits the same way; iban strips whitespace and upper-cases. api_key and every pattern rule hash the raw match — case is significant for an opaque token and a custom pattern has no detector to normalise by. A hash token stored by an earlier version for a phone, email, ssn, credit_card or iban rule no longer matches the token this version produces for the same value and key.

One engine serving many tenants can give each its own policy and hash key without a second connection pool:

tenant_engine = engine.with_redaction(tenant_policy, secret_key=tenant_hash_key)

The view shares the pool, providers and tools, applies tenant_policy on every method (ingest rules included), and hashes with tenant_hash_key, so tokens do not join across tenants. Stored tool configs still decrypt with the engine's secret_key. It is a Python call only — no HTTP or MCP argument reaches it — and closing the view closes nothing. Called with no secret_key while redaction_secret_key is unset and the policy has a hash rule, the view falls back to secret_key and warns (UserWarning), because tokens then join across every tenant sharing that fallback — pass a key, set redaction_secret_key, or filter the warning if that is what you meant. With no key at all it raises instead: a hash rule with no effective key is refused when the view is built.

A mounted MCP app takes the same pair per call instead of a view: create_mcp_app(engine, principals=..., scope=..., redaction=policy, secret_key=tenant_hash_key). Each is a value or a callable resolved fresh on every call — so one mounted app can serve many tenants — and both are host-set, never a tool argument. The connector-tool gateway takes the same pair (register_tool_gateway(..., redaction=..., secret_key=...)). It is also a per-call argument on every read path that already took redaction= — search, get_document, get_documents, get_chunks, list_documents, spreadsheet_schema, document_structure, document_types, field_summary, query_structured, map_reduce, compute, search_knowledge_base and execute_tool. Omitted, the engine's effective redaction key applies.

The detectors are deliberately a small set — the placement is the point. Redaction runs inside retrieval: hits are masked before _maybe_rerank POSTs their text to Cohere/Voyage/Jina, and tool results are masked before the audit row is written, so the masked value is the only one that leaves the process or lands in retention. That ordering is what usually goes wrong when a redaction library is bolted onto a pipeline by hand.

For broader detection, bring Microsoft Presidio ([presidio] extra) — its entities become ordinary rules and inherit the same placement guarantees:

from context_engine import presidio_detectors

RedactionPolicy(
    rules=[RedactionRule(name="person", detector="PERSON")],
    custom_detectors=presidio_detectors(["PERSON", "LOCATION"]),
)

Any Callable[[str], list[tuple[int, int]]] works as a detector, so an in-house classifier drops in the same way.

Benchmarks

Does ACL filtering cost you retrieval quality? Yes — badly — unless the vector leg is tuned for it, and the failure is silent.

Any system that applies an access-control predicate as a WHERE clause on top of an approximate vector index is doing a post-filter: the index walk is ordered by distance to the query, rows the caller may not see are discarded as they are met, and the walk stops when its candidate budget runs out. If the caller can see only a slice of the corpus, that budget can be spent entirely on rows they aren't allowed to have.

And the size of the visible slice is not the variable that matters most. Every ACL benchmark we know of assigns permissions at random. Real permissions are clustered — group:hr covers HR documents, and those are semantically adjacent by construction, so a group's slice is a region of the embedding space rather than a uniform sample.

200k documents from BEIR nq, real 1536-dim embeddings, exact brute-force oracle over eligible rows. Same corpus, same index, same queries, same number of visible rows — only the arrangement differs:

Clustering roughly doubles the recall loss. HNSW descends greedily from an entry point toward the query: scattered eligible rows are met along the walk, concentrated ones are not, so a query pointing away from the visible region spends its whole budget on rows the caller may not see.

Iterative scan does not fully close it — 0.883 clustered against 0.966 random, same setting. That residual is why this library decides between an exact and an approximate scan per query rather than tuning a parameter and hoping.

You may not be able to reproduce the failure, and that is also a result. With a GIN index on acl, all twelve default-planner rows chose an exact bitmap+sort instead of the vector index — recall 1.000, no failure, at 44–107 ms p50. Which plan Postgres picks is a cost estimate that moves with dimension, corpus size and statistics freshness, and && selectivity estimation on GIN arrays is unreliable enough to flip it without warning. The table above therefore forces the index. Whether you are exposed is decided per query, by a cost model — which is the actual argument for deciding it yourself.

Measure it on your own corpus rather than trusting ours — offline, read-only, no embedding provider called:

context-engine check-acl-exposure --database-url postgresql://...

It reports, for the ACL values that actually occur in your data, what a bare scoped query returns versus what this library returns, both scored against an exact oracle computed inside Postgres.

Written up in full: Your ACL Benchmark Is Measuring the Easy Case. Method, how clustered is constructed, the limitations — including a metric that behaved as a control rather than a result, and a retracted earlier table — are in benchmarks/acl_recall/, along with the harness and raw CSVs. Requires pgvector ≥ 0.8.

Checking your database

Managed Postgres varies in ways that matter here. To check a provider before committing to it:

CE_COMPAT_DATABASE_URL='postgresql+psycopg2://user:pass@host/db' \
    uv run pytest tests/test_managed_postgres_compat.py -v -s

Point it at a scratch database — it runs the real migrations. It verifies pgvector is new enough for iterative scan, that pg_trgm/unaccent are installable, and — the one that catches people — that their functions are actually reachable on the role's search_path. Supabase installs extensions into an extensions schema, so CREATE EXTENSION can succeed while similarity() and % stay unresolvable and the trigram leg breaks at query time rather than at migrate time.

Deep dives

This README covers the basics. Full documentation is at https://promptev.ai/documentation/context-engine/. The exhaustive design notes — redaction apply points, hash joinability, overlapping-rule merge semantics, ACL edge cases, graph internals, cost model, and known limitations — also live in the module and config docstrings (config.py, redaction.py, engine.py).