Corpus-KB
A local NLP engine for files, transcripts, and code. Ingest, search, ask, extract, and code. No cloud.
Corpus-KB serves three kinds of users. Non-coders get a browser UI. Novice coders get a CLI with a guided quickstart. Expert researchers get a qualitative-coding pipeline, an OpenAI-compatible API, and MCP tools for coding agents. Everything runs on your machine. Postgres stores the data by default; DuckDB is the single-file local engine (Storage backends). Ollama or PostgresML generates embeddings. One server exposes the catalogue over MCP stdio, HTTP, and a JSON-RPC socket. An OpenAI-compatible adapter serves /v1 chat and embeddings. See Editor integration.
Two ways to start
Pick the path that fits you. Both end at the same working system.
The novice path: three commands
Install pipx first if you do not have it. Then:
pipx install corpus-kb
corpus-kb quickstart
corpus-kb demo-kb
No repo clone is needed. quickstart checks your machine and names any
missing piece. demo-kb loads a small sample library, so search works
right away.
Two downloads come first:
- Ollama serves the local models.
- Docker runs the Postgres database. No Docker? Use the DuckDB path (Storage backends).
The default models are qwen3-embedding:4b for vectors and qwen3:4b
for answers. Each one downloads about 2.5 GB through Ollama. The setup
pulls both for you.
The browser path: from a clone
The browser UI needs the repo. First install the package:
pip install -e .
Then build the browser app once. You need Node 24 and pnpm for this step. Node is a build tool only. The running server stays pure Python:
pnpm -C ui_src install
pnpm -C ui_src build
Then start the stack. corpus-kb setup starts Postgres and loads the
schema. corpus-kb start starts the server and opens your browser:
corpus-kb setup
corpus-kb start
The browser opens the lifecycle app at http://localhost:8010/app/.
The Setup stage shows the system's health in plain language. The other
stages follow the research flow: Ingest, Review, Scale, Ask, Analyze,
Codebook, Release.
Details: docs/UI.md.
The expert path: terminals and APIs
Every operation is reachable programmatically. One line per surface:
- MCP stdio for editor agents:
corpus-kb --transport stdio— docs/API.md - HTTP API with curl examples: docs/API.md
- OpenAI-compatible adapter at
/v1(chat completions, embeddings): docs/API.md - JSON-RPC socket for local automation: docs/API.md
- CLI:
corpus-kb ingest,corpus-kb search,corpus-kb ask— docs/getting-started.md - Tracing with OpenTelemetry spans: docs/OBSERVABILITY.md
- Offline evaluation of retrieval and answers: docs/EVALUATION.md
- marimo notebooks served at
/notebooks: docs/NOTEBOOKS.md - Topics (offline BERTopic over stored embeddings): docs/ANALYSIS.md
First time on a terminal? corpus-kb quickstart walks you from zero to
a first search. The step-by-step quickstart is in
docs/getting-started.md.
Prefer Docker? A NAS or any Docker host can run the whole app with one compose command. See "Run the whole app in Docker" in docs/INSTALL.md.
What you get
- Hybrid search that blends vector similarity, full-text search, and rank fusion, with honest handling of degraded (zero-vector) chunks
- Reranking — a local judgment scorer by default, or a true cross-encoder through llama.cpp
- Answer generation with per-sentence citations, plus assertion-level verification with honest abstention
- Query routing — each question goes to vector search, SQL aggregates, or graph traversal
- Self-query filters — natural-language filters parsed into parameterized SQL predicates
- Knowledge graph with entities, relations, and BFS traversal
- Ontology-aware extraction with configurable entity/relation types and pluggable backends (regex, LangExtract, PostgresML, plus opt-in GLiNER)
- Pluggable ingestion strategies — eight built-ins (default, transcript, slides, web, survey, markdown, code, textbook), extensible three ways (docs/INGESTION.md)
- Website ingestion — a robots-respecting fetcher with same-domain crawl and heading-section chunks
- Late chunking — one model pass per document; chunk vectors pool from their own token spans
- Qualitative coding — transcript ingestion, codebooks, deductive and inductive runs, human review, governance reports, and a gated release lifecycle
- Topics — offline BERTopic clustering over stored embeddings (docs/ANALYSIS.md)
- LlamaIndex RAG backend with PGVectorStore and Ollama for local vector search
- Event sourcing for audit trails and rebuildable projections
- Multi-tenant Postgres with row-level security on every table
- Full-stack installer with hardware detection, profile-based recommendations, and guided setup
- MCP, HTTP, socket, and CLI surfaces so any editor or script can talk to it
- Browser UI with drag-and-drop ingest, cited answers, and plain-language health checks (docs/UI.md)
- Notebooks console — marimo notebooks served by the running server (docs/NOTEBOOKS.md)
Architecture
graph TB
Editor[AI Code Editor] -->|"MCP stdio"| MCP[MCP Server]
Editor -->|HTTP :8010| HTTP[Starlette API]
Editor -->|JSON-RPC socket| Socket[Unix socket / named pipe]
MCP --> Handlers[Command / Query Handlers]
HTTP --> Handlers
Socket --> Handlers
Handlers --> Domain[Domain Layer<br/>Aggregates + Events]
Domain --> ES[Event Store<br/>append-only]
ES --> Projections[Async Projections]
Projections --> PG[Postgres 17]
Projections --> DUCK[DuckDB<br/>local single-user engine]
PG --> VEC[pgvector<br/>vector search]
PG --> FTS[Postgres FTS<br/>to_tsvector]
PG --> AGE[Apache AGE<br/>Cypher graphs]
PG --> RLS[RLS on 12 tables]
Projections --> Ollama[Ollama<br/>embedding service]
- Ingest a file, directory, or raw text.
- The pipeline partitions it into chunks and embeds each chunk through Ollama or PostgresML. It extracts entities and relations, then stores the result.
- Commands append events to the event store; async projections write the read models into Postgres or DuckDB.
- Your editor queries the read models through search, SQL, or graph traversal.
Event sourcing flow
sequenceDiagram
participant C as Client
participant H as CommandHandler
participant A as Document Aggregate
participant ES as Event Store
participant P as Projection
participant DB as Postgres Tables
C->>H: ingest_file(file_path)
H->>A: create Document
A->>A: apply Ingested event
A->>A: apply ChunksAdded event
H->>ES: app.save(aggregate)
ES-->>H: event_id, version
H-->>C: {status: "success"}
ES->>P: subscribe(ChunksAdded)
P->>P: embed chunks via Ollama
P->>DB: INSERT chunks, chunks_vectors
P->>DB: UPDATE projection_checkpoints
Events are the source of truth. Projections are derived and can be rebuilt by replaying the event log. Vectors live in the chunks_vectors table and are treated as derived data, not event payload.
Documentation
| Page | What it covers |
|---|---|
| Getting started | Both paths: the browser for novices, the terminal quickstart for everyone else. Then qualitative coding from zero: the demo, your first transcript, first codebook |
| Install | Full setup from scratch: Postgres, Python, Ollama, schema, first query |
| Browser UI | The lifecycle app: stage-by-stage user guide, config, build, gates, troubleshooting |
| Features | Ingest, search, quality controls, graph, tags, research subsystem, embedding models, MCP tool reference |
| Admin | Configuration, schema, multi-tenancy, backups, monitoring, CI/CD |
| Storage backends | DuckDB vs Postgres: quickstart, engine selection, migrate-data, runtime tuning |
| Operations | The operator's contract: exit codes, ports, vocabularies, tenancy, the MCP surface |
| API | HTTP routes, the /v1 OpenAI adapter, curl examples, MCP tool reference |
| Observability | OpenTelemetry spans, exporters, zero-cost when off |
| Evaluation | Golden sets, recall/MRR/nDCG, answer metrics |
| Notebooks | The marimo console and the three example notebooks |
| Development | Architecture deep dive, testing, PR workflow, conventions |
| Research | Research command reference, config keys, embedding requirements |
| Understanding your report | Plain-language guide to every governance-report section |
| FAQ | Common questions |
| Ingestion | Full pipeline: partition, chunk, embed, extract, store, plus the strategy reference and extension ladders |
| Analysis | The analysis layer: analyze-run report, labeling, classification, topics, trends, method register |
| Analysis methods | GENERATED reference: the 30-method register with required validation and permitted claims |
| Rollback drill | Practiced procedure for backing out a bad deployment |
| Internal program docs | Archive: playbooks, research math, benchmarks, harness distribution, program plans |
Editor integration
Editors connect over MCP stdio (native, no bridge), the HTTP API, or the JSON-RPC socket:
- MCP stdio:
corpus-kb --transport stdio— themcp-configs/files (Claude Code, Cursor, Codex) and the rootopencode.json(OpenCode) launch exactly this command. Generic MCP clients such as Hermes or Cline need no config file: they point at the same stdio server. See docs/research/harness-distribution.md. The server registers every catalogue operation as an MCP tool; see docs/FEATURES.md for the tool reference. - HTTP:
corpus-kb start --transport http --port 8010— every catalogue operation has a route, plus the/v1OpenAI adapter (see docs/API.md). - JSON-RPC socket: same command, for local automation.
- The research and coding surfaces are first-class CLI commands (
corpus-kb research ...andcorpus-kb coding ...). Ingestion has a top-level front door:corpus-kb ingestwith--watchauto-heal, pluscorpus-kb ingest-statusandcorpus-kb ingest-reembed.
Every catalogue operation is reachable on all four surfaces (MCP, CLI,
HTTP, socket) and returns the same envelope on each. The parity gate in
tests/test_surface_parity_deep.py enforces this.
License
MIT License. See pyproject.toml for the full text.
Metadata
Release files for corpus-kb 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| corpus_kb-0.1.0.tar.gz | 3.4 MB | Without provenance | Details for corpus_kb-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| corpus_kb-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for corpus_kb-0.1.0-py3-none-any.whl |
Total release size: 6.1 MB
Release files / corpus_kb-0.1.0.tar.gz
| Download URL | corpus_kb-0.1.0.tar.gz |
|---|---|
| Size | 3.4 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4b15aa6d529b604ef11e71720cf2ecd94e8bf0ee1719a4bc488b21d635932521
|
|
BLAKE2b-256 checksum How to use checksums |
ed77720a266c8d75edd1ce5945ff1128f340855835bd1fa04dd0bbb58ac51a8b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.13
|
Release files / corpus_kb-0.1.0-py3-none-any.whl
| Download URL | corpus_kb-0.1.0-py3-none-any.whl |
|---|---|
| Size | 2.8 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d8cd652302b281d5680854586b676608df3d2b694eeecb5c1031e2a1915ab061
|
|
BLAKE2b-256 checksum How to use checksums |
1a815e94471496af5b34f48f89ea3d0aaa51377052b8ed79a6fafd5b7fab1809
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.13
|
Security for corpus-kb 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.