corpora-py

📚 Turn any book into a queryable text graph — then let an AI read it with you.

Install

pip install corpora-py          # or: uv add corpora-py

AUTH_REQUIRED=false corpora-api    # http://127.0.0.1:8000 — leave it running

The examples below run in a second shell and pipe through jq (brew install jq).

Auth is on by default and fails closed (401 without a Supabase JWT). AUTH_REQUIRED=false is for local use — see Settings.

C-USX conversion direction

The development API adds output_format=cusx to POST /convert: individual EPUB/PDF/HTML/XML/TEI/plain documents become compressed .cusx packages containing extended USX XML, the parsed tree and a pinned reference-linking text snapshot. The example app requests that format and uses the existing manual Hugging Face publication action. Bucket/repo settings and upload authentication are unchanged.

This option is not yet in released v6.0.0. Existing API callers retain corpus output when the option is omitted; dataset ZIPs keep their corpus import path. No corpus workflow is removed or assigned a deprecation deadline. See the C-USX package contract and consumer limits.

What you get

flowchart LR
    S["📄 EPUB · PDF · HTML<br/>XML · TEI · text"] -->|POST /convert| C["⚙️ conversion job"]
    C --> A["📦 .corpus archive<br/>every word a graph node"]
    A --> M["🤖 /mcp<br/>AI reads and searches it"]
    A --> R["🔗 /refs<br/>citable passages"]
    A --> H["☁️ /storage<br/>publish to the Hub"]

Use it

1. Convert a document

curl -sF file=@book.epub -F source_format=epub -F name='My Book' \
  localhost:8000/convert | tee job.json
# → {"job_id": "1ec2121b-…", "status_url": "/convert/1ec2121b-…", "ws_url": "…/ws"}

JOB=$(jq -r .job_id job.json)      # every command below uses it

source_format: epub · pdf · html · xml · tei · tei_zip · plain · tf_zip

2. Watch it, then take the archive

curl -s localhost:8000/convert/$JOB          # {"status": "succeeded", …}
curl -sOJ localhost:8000/convert/$JOB/download

Long jobs push the same status over a WebSocket: ws://…/convert/$JOB/ws.

3. Read it back before publishing

curl -s localhost:8000/convert/$JOB/sections            # the table of contents
curl -s "localhost:8000/convert/$JOB/content?limit=1"   # passages + tokens
curl -s localhost:8000/convert/$JOB/manifest            # title, authors, version

4. Let Claude read it

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "corpora": {
      "command": "cf-mcp",
      "args": ["--corpus", "/Users/you/.exegia/datasets/BHSA", "--name", "BHSA"]
    }
  }
}

Then ask it things. A good tool order for an agent:

describe_corpus() → list_features() → search(…, "count") → search(…, "results") → get_passages(…)

5. Cite a passage

curl "localhost:8000/refs/resolve?ref=bhsa@2021/Deut:4:2!clause1"

One grammar for every corpus — corpus@version/Section:Section!typeN:

POST /refs turns a node into one. Full grammar: skills/tf-reference-id.

6. Publish and browse a library

curl -sX POST localhost:8000/storage \
  -H 'content-type: application/json' -d "{\"job_id\": \"$JOB\"}"   # publish
curl -s localhost:8000/storage                          # what's published
curl -s localhost:8000/storage/my-book.corpus/content   # read it

Needs HF_STORAGE_REPO + HF_TOKEN.

Endpoints

Interactive docs while the server runs: http://127.0.0.1:8000/docs

Settings

Public demo = AUTH_REQUIRED=false and HF_READ_ONLY=true. Set both, or anonymous visitors can write to your Hub.

Python instead of HTTP

from admin.converters import CONVERTERS, convert_to_corpus
from admin.parsers import SourceFormat

tf_dir = CONVERTERS[SourceFormat.EPUB]("book.epub", "out/book.tf")
convert_to_corpus(tf_dir, "book.corpus", name="My Book", language_code="en")

from corpora_mcp.corpus import corpus_manager

name = corpus_manager.load("~/.exegia/datasets/BHSA", name="BHSA")
api = corpus_manager.get_api(name)          # Text-Fabric api: api.F, api.T, api.S

pip install corpora-py ships all of it — corpora_mcp, admin and common are bundled in that one wheel; there is no separate corpora-mcp on PyPI.

Docker

make docker-build-corpora        # or: docker build -f dockerfiles/Dockerfile -t corpora-py .
docker run -p 8000:8000 -v ~/.exegia/datasets:/data/datasets:ro corpora-py

Images are also published to ghcr.io/exegia/corpora-py (login required). MCP-only image: dockerfiles/Dockerfile.client · Compose: docker compose -f dockerfiles/docker-compose.yml up corpora

More

License

MIT

Reference linking library

Building a reader app? Start with the small-step client guide for copyable selection, save, retrieval and review examples.

The independently released corpora-linking core supports revision-pinned manual selections and automatic citation intake, preserving unknown works, ambiguous passages and stale anchors. Its only runtime dependency is Pydantic. Optional umbrella adapters cover PDF, EPUB CFI, HTML/browser ranges, C-USX publication and authenticated working-reference management. See application composition and limits. Linking is disabled by default; its database schema remains an unapplied proposal. The core is installed from PyPI; its independent repository owns core source, builds and publication.