corpus-core

Shared infrastructure for corpus indexing + MCP search. Powers both arxiv-radar-mcp (the public-data arXiv topical radar) and lab-corpus-mcp (the private multi-source PDF / video / lab-notes server).

What's inside:

Install

pip install corpus-core            # once published to PyPI
pip install corpus-core[pdf]       # + MinerU PDF parsing (~2 GB, mineru[core]>=2.5)
# or, during dev:
pip install -e ../corpus-core
pip install -e "../corpus-core[pdf]"   # dev with PDF extra

Quick start

from corpus_core import (
    Encoder, EmbeddingIndex,
    Chunk, chunk_markdown,
    search_text, search_semantic, similar_to,
    search_paper_text, search_paper_semantic, similar_to_paper,
    load_chunk_texts, reindex, is_junk_section,
    JobRegistry, JobHandle, JobError, Job,
    make_method_dispatcher, build_mcp_app,
    serve_stdio, serve_streamable_http,
    Dispatcher, BackgroundTaskFactory,
    fetch_url, fetch_arxiv_pdf,
    Throttle, get_arxiv_throttle,
    request_with_retry, FetchResult,
    ARXIV_RATE_LIMIT_S, DEFAULT_USER_AGENT,
)

# Submodule access also fine:
from corpus_core.embeddings import Encoder
from corpus_core.proxy import run_proxy
from corpus_core.reranker import Reranker, RerankerConfig
from corpus_core.http_fetch import fetch_url, get_arxiv_throttle

MCP concurrency and cancellation

corpus-core >= 0.3.0 requires mcp >= 2.1,<3. The MCP SDK v1.26/v1.27 receive loop could terminate after a notifications/cancelled race while the server process itself remained alive. The visible symptom was an unlimited hang of every subsequent request. The v2 low-level dispatcher keeps request cancellation local to the affected call.

Synchronous tool handlers are executed with asyncio.to_thread, so one slow search does not block transport I/O or unrelated tools. Every call also has a server-side deadline (300 seconds by default) so a client which supplies no timeout still receives a terminal error. Override it for legitimately long foreground tools with CORPUS_MCP_TOOL_TIMEOUT_S; background jobs are not limited by this value.

The worker thread behind a cancelled synchronous tool cannot be force-killed safely by Python. It may finish in the background, but cancellation is not swallowed and the MCP session continues serving other requests.

Process-level singleton contract

One cache_dir = one writing process. corpus_core.jobs.JobRegistry serialises reindex attempts within a process via acquire_reindex_lock(). Running two separate processes against the same cache_dir at the same time is not supported and will produce a corrupted index. The lockfile (<cache_dir>/fulltext/.reindex.lock) records pid + hostname + start_time so a crashed owner's lock can be recovered on the next start (same-host pid dead = stale; foreign host = operator must remove manually).

corpus_core.embeddings.Encoder and corpus_core.http_fetch.get_arxiv_throttle() are process-level singletons -- construct one instance and inject it into both arxiv-radar-mcp and lab-corpus-mcp via the encoder= / shared-throttle parameters. Never instantiate two Encoders in the same process against the same GPU (two Qwen3-4B bf16 models = ~16 GB, exhausts a 12 GB card).

Invariants downstream packages must honour

• Embedding cache layout:
  • <cache_dir>/embeddings.npy — float32, L2-normalized, shape (N, D).
  • <cache_dir>/index.json — {model, dims, n, row_for, ...metadata}.
  • Both written atomically (*.tmp → os.replace).
• Job persistence schema: <cache_dir>/jobs/<job_id>.json with fields {job_id, kind, state, progress, n_total, n_done, started_at, finished_at, result, error, args}. State ∈ {pending, running, done, failed, orphaned}.
• Chunk metadata: each chunk in EmbeddingIndex.metadata["chunks"] has {arxiv_id, section, chunk_idx, n_chars, n_tokens_est}. The arxiv_id field is the corpus-wide paper id — DOI / PMID / sha256 / any string the host project chooses.
• Encoder config duck-type: Encoder.__init__(config) reads config.embeddings.{model, batch_size, target_dim, cache_dir}. Pass any object with that shape. See corpus_core.embeddings.Config Protocol for the formal type.
• HTTP fetch invariants (http_fetch.py):
  • fetch_url writes atomically (<dest>.tmp → os.replace); on any failure (transport error, non-2xx, empty body) dest_path is not created or overwritten.
  • Throttle is one instance per source domain; all callers that share an instance share the budget. Use get_arxiv_throttle() for every arxiv.org GET so the combined image enforces 1 req / 3 sec across both downstream servers.
  • request_with_retry retries only on 429/503; other status codes fall through after the first attempt. Honours Retry-After if present, else exponential backoff 3→6→12 sec.

Used by

• arxiv-radar-mcp -- arxiv-only topical radar over the daily-arxiv-* fork family.
• lab-corpus-mcp -- private PDF / DOCX / PPTX / image corpus parsed via MinerU; can also run combined with arxiv-radar-mcp on one Qwen instance to fit a 12 GB GPU.

Note: shared GPU hosting requires the combined-supervisor (DECISIONS-136). arxiv-radar-mcp and lab-corpus-mcp each load Qwen3-Embedding-4B (~8 GB bf16). Running them as two independent standalone backends on the same GPU is not supported -- two copies total ~16 GB, causing OOM on 12 GB cards. The only supported topology for shared GPU hosting is the combined-supervisor in lab-corpus-mcp, which constructs one Encoder and injects it into both servers via the encoder= parameter on RadarServer.__init__ and the equivalent in lab-corpus-mcp.

Status

v0.2.0, in production as of 2026-05-24. Both downstream projects (arxiv-radar-mcp and lab-corpus-mcp) install corpus-core editable from the sibling repo. The combined exopoiesis/lab-corpus-gpu image on gomer holds:

• 34,627 abstract embeddings (Qwen3-Embedding-4B native 2560 dims, L2-normalized) over 5 arxiv-radar fork sources.
• 466 fulltext chunks across 51 fetched arxiv papers + 54 chunks across 2 MinerU-parsed lab PDFs.
• All in one corpus_core.embeddings.EmbeddingIndex cache layout, encoded by a single shared Encoder instance.

Build-time audit_image.py in both downstream projects checks the no-duplicate-distribution invariant — pip never ends up with two copies of any package, including torch.

HTTP fetch primitives added (2026-05-13) to close arxiv-radar-mcp's U14: http_fetch.py extracts the throttled GET + 429/503-retry pattern that previously lived inside arxiv-radar's fulltext.py. Both servers now share the singleton arxiv throttle — arxiv-radar uses it for HTML/LaTeX, lab-corpus uses it for ingest_url / ingest_arxiv_pdf PDF downloads. Same module-global lock across the whole combined image.

VRAM unload added (2026-05-24, v0.2.0). Encoder.unload() and Reranker.unload() drop the in-process model and free CUDA VRAM via torch.cuda.empty_cache() + gc.collect(). Both are idempotent and guarded by the existing model-load lock so concurrent encodes either complete first or re-load on next call. Downstream projects call Encoder.unload() after heavy one-shot work (reindex, refresh, bulk ingest) so a shared GPU host can use the freed VRAM for unrelated compute. 119 corpus-core tests green.

Phase 3 extraction is complete; PyPI publication of corpus-core deferred until the API stabilises through real-world ingest of more than the current 53 papers.

Tests

pytest -q from the repo root. The standalone test suite covers chunker, jobs, mcp_scaffold, proxy invocation, reranker config + lazy load, and basic embedding-index roundtrip with a deterministic stub encoder. Heavier integration testing (host-project Configs, real Qwen weights, real MCP sessions) lives in the arxiv-radar-mcp and lab-corpus-mcp test suites.

License

MIT.