Skip to main content

Corpus-KB

A local NLP engine for files, transcripts, and code. Ingest, search, ask, extract, and code. No cloud.

Corpus-KB serves three kinds of users. Non-coders get a browser UI. Novice coders get a CLI with a guided quickstart. Expert researchers get a qualitative-coding pipeline, an OpenAI-compatible API, and MCP tools for coding agents. Everything runs on your machine. Postgres stores the data by default; DuckDB is the single-file local engine (Storage backends). Ollama or PostgresML generates embeddings. One server exposes the catalogue over MCP stdio, HTTP, and a JSON-RPC socket. An OpenAI-compatible adapter serves /v1 chat and embeddings. See Editor integration.


Two ways to start

Pick the path that fits you. Both end at the same working system.

The novice path: three commands

Install pipx first if you do not have it. Then:

pipx install corpus-kb
corpus-kb quickstart
corpus-kb demo-kb

No repo clone is needed. quickstart checks your machine and names any missing piece. demo-kb loads a small sample library, so search works right away.

Two downloads come first:

  • Ollama serves the local models.
  • Docker runs the Postgres database. No Docker? Use the DuckDB path (Storage backends).

The default models are qwen3-embedding:4b for vectors and qwen3:4b for answers. Each one downloads about 2.5 GB through Ollama. The setup pulls both for you.

The browser path: from a clone

The browser UI needs the repo. First install the package:

pip install -e .

Then build the browser app once. You need Node 24 and pnpm for this step. Node is a build tool only. The running server stays pure Python:

pnpm -C ui_src install
pnpm -C ui_src build

Then start the stack. corpus-kb setup starts Postgres and loads the schema. corpus-kb start starts the server and opens your browser:

corpus-kb setup
corpus-kb start

The browser opens the lifecycle app at http://localhost:8010/app/. The Setup stage shows the system's health in plain language. The other stages follow the research flow: Ingest, Review, Scale, Ask, Analyze, Codebook, Release.

Details: docs/UI.md.

The expert path: terminals and APIs

Every operation is reachable programmatically. One line per surface:

First time on a terminal? corpus-kb quickstart walks you from zero to a first search. The step-by-step quickstart is in docs/getting-started.md.

Prefer Docker? A NAS or any Docker host can run the whole app with one compose command. See "Run the whole app in Docker" in docs/INSTALL.md.


What you get

  • Hybrid search that blends vector similarity, full-text search, and rank fusion, with honest handling of degraded (zero-vector) chunks
  • Reranking — a local judgment scorer by default, or a true cross-encoder through llama.cpp
  • Answer generation with per-sentence citations, plus assertion-level verification with honest abstention
  • Query routing — each question goes to vector search, SQL aggregates, or graph traversal
  • Self-query filters — natural-language filters parsed into parameterized SQL predicates
  • Knowledge graph with entities, relations, and BFS traversal
  • Ontology-aware extraction with configurable entity/relation types and pluggable backends (regex, LangExtract, PostgresML, plus opt-in GLiNER)
  • Pluggable ingestion strategies — eight built-ins (default, transcript, slides, web, survey, markdown, code, textbook), extensible three ways (docs/INGESTION.md)
  • Website ingestion — a robots-respecting fetcher with same-domain crawl and heading-section chunks
  • Late chunking — one model pass per document; chunk vectors pool from their own token spans
  • Qualitative coding — transcript ingestion, codebooks, deductive and inductive runs, human review, governance reports, and a gated release lifecycle
  • Topics — offline BERTopic clustering over stored embeddings (docs/ANALYSIS.md)
  • LlamaIndex RAG backend with PGVectorStore and Ollama for local vector search
  • Event sourcing for audit trails and rebuildable projections
  • Multi-tenant Postgres with row-level security on every table
  • Full-stack installer with hardware detection, profile-based recommendations, and guided setup
  • MCP, HTTP, socket, and CLI surfaces so any editor or script can talk to it
  • Browser UI with drag-and-drop ingest, cited answers, and plain-language health checks (docs/UI.md)
  • Notebooks console — marimo notebooks served by the running server (docs/NOTEBOOKS.md)

Architecture

graph TB
    Editor[AI Code Editor] -->|"MCP stdio"| MCP[MCP Server]
    Editor -->|HTTP :8010| HTTP[Starlette API]
    Editor -->|JSON-RPC socket| Socket[Unix socket / named pipe]
    MCP --> Handlers[Command / Query Handlers]
    HTTP --> Handlers
    Socket --> Handlers
    Handlers --> Domain[Domain Layer<br/>Aggregates + Events]
    Domain --> ES[Event Store<br/>append-only]
    ES --> Projections[Async Projections]
    Projections --> PG[Postgres 17]
    Projections --> DUCK[DuckDB<br/>local single-user engine]
    PG --> VEC[pgvector<br/>vector search]
    PG --> FTS[Postgres FTS<br/>to_tsvector]
    PG --> AGE[Apache AGE<br/>Cypher graphs]
    PG --> RLS[RLS on 12 tables]
    Projections --> Ollama[Ollama<br/>embedding service]
  1. Ingest a file, directory, or raw text.
  2. The pipeline partitions it into chunks and embeds each chunk through Ollama or PostgresML. It extracts entities and relations, then stores the result.
  3. Commands append events to the event store; async projections write the read models into Postgres or DuckDB.
  4. Your editor queries the read models through search, SQL, or graph traversal.

Event sourcing flow

sequenceDiagram
    participant C as Client
    participant H as CommandHandler
    participant A as Document Aggregate
    participant ES as Event Store
    participant P as Projection
    participant DB as Postgres Tables

    C->>H: ingest_file(file_path)
    H->>A: create Document
    A->>A: apply Ingested event
    A->>A: apply ChunksAdded event
    H->>ES: app.save(aggregate)
    ES-->>H: event_id, version
    H-->>C: {status: "success"}
    ES->>P: subscribe(ChunksAdded)
    P->>P: embed chunks via Ollama
    P->>DB: INSERT chunks, chunks_vectors
    P->>DB: UPDATE projection_checkpoints

Events are the source of truth. Projections are derived and can be rebuilt by replaying the event log. Vectors live in the chunks_vectors table and are treated as derived data, not event payload.


Documentation

Page What it covers
Getting started Both paths: the browser for novices, the terminal quickstart for everyone else. Then qualitative coding from zero: the demo, your first transcript, first codebook
Install Full setup from scratch: Postgres, Python, Ollama, schema, first query
Browser UI The lifecycle app: stage-by-stage user guide, config, build, gates, troubleshooting
Features Ingest, search, quality controls, graph, tags, research subsystem, embedding models, MCP tool reference
Admin Configuration, schema, multi-tenancy, backups, monitoring, CI/CD
Storage backends DuckDB vs Postgres: quickstart, engine selection, migrate-data, runtime tuning
Operations The operator's contract: exit codes, ports, vocabularies, tenancy, the MCP surface
API HTTP routes, the /v1 OpenAI adapter, curl examples, MCP tool reference
Observability OpenTelemetry spans, exporters, zero-cost when off
Evaluation Golden sets, recall/MRR/nDCG, answer metrics
Notebooks The marimo console and the three example notebooks
Development Architecture deep dive, testing, PR workflow, conventions
Research Research command reference, config keys, embedding requirements
Understanding your report Plain-language guide to every governance-report section
FAQ Common questions
Ingestion Full pipeline: partition, chunk, embed, extract, store, plus the strategy reference and extension ladders
Analysis The analysis layer: analyze-run report, labeling, classification, topics, trends, method register
Analysis methods GENERATED reference: the 30-method register with required validation and permitted claims
Rollback drill Practiced procedure for backing out a bad deployment
Internal program docs Archive: playbooks, research math, benchmarks, harness distribution, program plans

Editor integration

Editors connect over MCP stdio (native, no bridge), the HTTP API, or the JSON-RPC socket:

  • MCP stdio: corpus-kb --transport stdio — the mcp-configs/ files (Claude Code, Cursor, Codex) and the root opencode.json (OpenCode) launch exactly this command. Generic MCP clients such as Hermes or Cline need no config file: they point at the same stdio server. See docs/research/harness-distribution.md. The server registers every catalogue operation as an MCP tool; see docs/FEATURES.md for the tool reference.
  • HTTP: corpus-kb start --transport http --port 8010 — every catalogue operation has a route, plus the /v1 OpenAI adapter (see docs/API.md).
  • JSON-RPC socket: same command, for local automation.
  • The research and coding surfaces are first-class CLI commands (corpus-kb research ... and corpus-kb coding ...). Ingestion has a top-level front door: corpus-kb ingest with --watch auto-heal, plus corpus-kb ingest-status and corpus-kb ingest-reembed.

Every catalogue operation is reachable on all four surfaces (MCP, CLI, HTTP, socket) and returns the same envelope on each. The parity gate in tests/test_surface_parity_deep.py enforces this.


License

MIT License. See pyproject.toml for the full text.

Metadata

Release files for corpus-kb 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for corpus-kb 0.1.0
File Size Uploaded Provenance Actions
corpus_kb-0.1.0.tar.gz 3.4 MB Without provenance Details for corpus_kb-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for corpus-kb 0.1.0
File Interpreter ABI Platform Actions
corpus_kb-0.1.0-py3-none-any.whl Python 3 none any Details for corpus_kb-0.1.0-py3-none-any.whl

Total release size: 6.1 MB

Release files / corpus_kb-0.1.0.tar.gz

Download URL corpus_kb-0.1.0.tar.gz
Size 3.4 MB
Tags Source
SHA-256 checksum
How to use checksums
4b15aa6d529b604ef11e71720cf2ecd94e8bf0ee1719a4bc488b21d635932521
BLAKE2b-256 checksum
How to use checksums
ed77720a266c8d75edd1ce5945ff1128f340855835bd1fa04dd0bbb58ac51a8b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.13

Release files / corpus_kb-0.1.0-py3-none-any.whl

Download URL corpus_kb-0.1.0-py3-none-any.whl
Size 2.8 MB
Tags Python 3
SHA-256 checksum
How to use checksums
d8cd652302b281d5680854586b676608df3d2b694eeecb5c1031e2a1915ab061
BLAKE2b-256 checksum
How to use checksums
1a815e94471496af5b34f48f89ea3d0aaa51377052b8ed79a6fafd5b7fab1809
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.13

Security for corpus-kb 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page