Skip to main content

Paper Librarian

An MCP server that gives Claude a local, searchable library of research papers (arXiv + PDFs). Built on the MCP Python SDK 2.x (MCPServer, the 2.x successor to FastMCP).

Add papers by arXiv id or PDF path, then ask questions about them. Answers cite the paper, section and page, and references between papers in the library are linked in both directions.

Quick install (needs uv):

claude mcp add paper-librarian --scope user -- uvx paper-librarian

or in Claude Desktop's claude_desktop_config.json:

{ "mcpServers": { "paper-librarian": { "command": "uvx", "args": ["paper-librarian"] } } }

Demo

A session in Claude Code with the server connected.

1. Add a paper by arXiv id. add_paper downloads Attention Is All You Need, splits it into 44 searchable chunks and parses its 40 references. None are linked yet because the library was empty.

Adding the first paper

2. Claude calls the MCP tool. Asked to add BERT (1810.04805), Claude Code routes the request to the paper-librarian server.

Claude Code calling paper-librarian

3. Add a second paper; citations link automatically. BERT is added with 71 chunks and 56 references. One of them matches the Transformer paper already in the library, so the two papers are now linked.

Second paper added with a citation link

4. Ask a question; answers cite section and page. Hybrid search finds the relevant passages in both papers, and Claude cites each claim, e.g. (Attention Is All You Need, Positional Encoding, p.6).

Search answer with section and page citations

5. Save a note to a paper. add_note attaches a summary to the paper; it is returned whenever the paper is looked up with get_paper.

Adding a note to a paper


Problem statement

Researchers and students keep dozens of papers as PDFs, but an AI assistant can't use them well:

  • No memory of your library. Each chat starts from nothing; pasting whole PDFs runs into context limits and has to be repeated every time.
  • Answers from memory, not sources. Without access to the actual text, a model summarizes papers from training data, which may be outdated, vague or wrong, and can't point to where a claim comes from.
  • PDFs are hard to read programmatically. Two-column layouts, section headings, hyphenation, ligatures and reference lists in several citation styles all break naive text extraction.
  • Papers are connected, but the connections are invisible. Which of my papers cite each other? That information is buried in reference lists.

Paper Librarian solves this with an MCP server that runs locally and:

  1. ingests papers from arXiv or local PDFs into a structured library (sections, pages, references);
  2. indexes them for hybrid search (meaning-based + keyword), returning passages with section and page so every answer can be cited;
  3. parses reference lists and links citations between papers in the library;
  4. exposes all of this to Claude (or any MCP client) as tools, a resource and a prompt.

Everything — database, PDFs, embedding model — stays on your machine.


Features

  • Add papers by arXiv id (1706.03762, arXiv:1706.03762v5), arXiv URL, or local PDF path, with duplicate detection by arXiv id and file hash.
  • Layout-aware PDF extraction: two-column reading order, section detection using font weight and size, page tracking, ligature and hyphenation repair.
  • Hybrid search: vector similarity (sqlite-vec) + BM25 keyword search (SQLite FTS5), merged with reciprocal rank fusion; filters by year and paper.
  • Citation graph: reference lists split in [n], n. and author-year styles; matched to library papers by arXiv id, exact title or fuzzy title; linked forwards and backwards as papers are added.
  • Section reader: fetch the full text of any section ("Method", "Positional Encoding", …).
  • Comparison helper: gathers the most relevant passages per paper per aspect for comparison tables.
  • Notes attached to papers.
  • Local and private: one SQLite file plus a PDF folder; no external services except arXiv downloads and the one-time model download.

Tech stack

Layer Technology Why
Language Python 3.11+ (developed on 3.13) Ecosystem for PDFs, ML and MCP
MCP server MCP Python SDK 2.3 (MCPServer) Tools, resources, prompts, progress reporting, stdio transport
PDF parsing PyMuPDF 2.0 Fast text extraction with font, size and position data per span
arXiv arxiv 4.0 + requests Metadata lookup (title, authors, abstract) and PDF download
Embeddings sentence-transformers 6.2, model all-MiniLM-L6-v2 (384-dim) Small (~90 MB), CPU-friendly, good quality for retrieval
Vector index sqlite-vec 0.1.9 (vec0 table) KNN search inside the same SQLite file
Keyword index SQLite FTS5 (BM25 ranking) Built into Python's sqlite3, no extra service
Storage SQLite Single file, transactional, zero setup
Testing pytest 9, anyio, MCP in-process Client Unit tests plus end-to-end tests through a real MCP client
Build hatchling (pyproject.toml) Standard packaging; paper-librarian console script

Architecture

flowchart LR
    Claude["Claude / MCP client"] <-- "stdio (JSON-RPC)" --> Server["server.py<br/>MCPServer: 8 tools,<br/>1 resource, 1 prompt"]

    Server --> Ingest["ingest.py<br/>fetch · extract · sections · chunks"]
    Server --> Search["search.py<br/>vector + BM25 + RRF"]
    Server --> Store

    Ingest --> ArXiv[("arXiv API")]
    Ingest --> Embed["embed.py<br/>MiniLM embeddings"]
    Ingest --> Cit["citations.py<br/>split · match · link"]
    Search --> Embed

    Ingest --> Store["store.py<br/>SQLite + FTS5 + sqlite-vec"]
    Cit --> Store
    Search --> Store
    Store --> DB[("~/.librarian/library.db")]
    Ingest --> PDFs[("~/.librarian/pdfs/")]

How a paper is added (add_paper)

flowchart TD
    A["source: arXiv id / URL / PDF path"] --> B{"arXiv id?"}
    B -- yes --> C["duplicate check by arXiv id"] --> D["fetch metadata + download PDF"]
    B -- no --> E["duplicate check by sha256"]
    D --> F["extract_pdf: column-aware page text"]
    E --> F
    F --> G["split_sections: headings → segments (section, page)"]
    G --> H["chunk_segments: 180-word windows, 30-word overlap"]
    H --> I["embed chunks (MiniLM, normalized)"]
    G --> J["References section → split_references"]
    I --> K["one transaction: paper + chunks + vectors + FTS + citations + backlinks"]
    J --> K
flowchart LR
    Q["query"] --> V["embed → KNN on chunk_vectors"]
    Q --> K["sanitize → FTS5 MATCH, BM25"]
    V --> R["reciprocal rank fusion<br/>score = Σ 1/(60 + rank)"]
    K --> R
    R --> F["filters: year, paper ids<br/>cap: N hits per paper"]
    F --> S["hits: title · section · page · snippet · matched_by"]

Project structure

paper-librarian/
├── pyproject.toml            # dependencies, console script, pytest config
├── README.md
├── docs/screenshots/         # demo screenshots used in this README
├── src/librarian/
│   ├── __init__.py           # package version
│   ├── server.py             # MCP server: tools, resource, prompt, lifespan        (333 lines)
│   ├── ingest.py             # arXiv fetch, PDF extraction, sections, chunking      (457 lines)
│   ├── search.py             # hybrid search, RRF, snippets                         (198 lines)
│   ├── citations.py          # reference splitting, matching, linking               (161 lines)
│   ├── store.py              # SQLite schema and data access                        (325 lines)
│   └── embed.py              # embedding model wrapper                              ( 53 lines)
└── tests/                    # 90 tests
    ├── test_server.py        # end-to-end through an in-process MCP client          (9)
    ├── test_ingest.py        # arXiv ids, headings, sections, chunking, ingestion   (39)
    ├── test_citations.py     # splitting styles, matching, linking                  (20)
    ├── test_search.py        # keyword / semantic / hybrid, filters, snippets       (10)
    ├── test_store.py         # schema, constraints, cascades                        (9)
    └── test_smoke.py         # dependencies load                                    (3)

Code details

store.py — data model and storage

All data lives in one SQLite file. The schema:

Table Columns Notes
papers id, title, authors, year, abstract, source, arxiv_id, sha256, path, added_at id is arxiv:<id> or pdf:<sha256[:16]>; arxiv_id and sha256 are UNIQUE (duplicate protection); source is CHECKed to arxiv/pdf
chunks id, paper_id, seq, section, page, text Searchable passages; UNIQUE(paper_id, seq)
chunks_fts FTS5 over text, section External-content index kept in sync by AFTER INSERT/DELETE triggers
chunk_vectors vec0(chunk_id, embedding float[384]) sqlite-vec KNN index; same id as chunks.id so results can be fused
citations id, paper_id, seq, raw_reference, matched_paper_id matched_paper_id → papers.id, ON DELETE SET NULL
notes id, paper_id, text, created_at
meta key, value Stores embedding_dim; reopening with a different model raises instead of corrupting the index

Deleting a paper cascades to its chunks, FTS entries, vectors (via trigger), citations and notes.

Store provides typed access (Paper, Chunk, Citation, Note dataclasses): add_paper, get_paper, find_duplicate, list_papers, delete_paper, add_chunks (with vectors), get_chunks(section=…), add_citations, get_citations, get_cited_by, unmatched_citations, set_citation_match, delete_citations, add_note, get_notes.

store.transaction(commit=True) wraps writes; passing commit=False to individual methods lets a caller group several writes into one atomic transaction (used by ingestion).

The connection is opened with check_same_thread=False because MCP tools run in worker threads; the server serializes access with a lock.

ingest.py — getting papers in

Function What it does
parse_arxiv_id(source) Recognizes new-style (2310.06825v2), old-style (hep-th/9901001), arXiv: prefixes and abs//pdf/ URLs; returns the version-less id
fetch_arxiv(id, pdf_dir) Metadata via the arxiv API, PDF via requests (arxiv 4.x dropped download_pdf); validates the %PDF header
extract_pdf(path) Title (PDF metadata, else largest horizontal text on page 1), authors, year, section segments, raw References text
page_text(page) Column-aware reading order: blocks are classified left / right / full-width around the page centre; within bands separated by full-width blocks, the left column is read before the right. Marks indented lines (for reference splitting) and emphasized lines (bold or ≥1.1× body font size, for heading detection). Merges spans on the same baseline and applies NFKC normalization (fi → fi)
heading_name(line, appendix) Decides whether a line is a section heading. Known names (Introduction, Methods, …) map to canonical names; other numbered headings (3 Model Architecture) must be emphasized. Rejects years and decimals as section numbers, lines with column gaps, and sentence fragments. Letter numbering (A, B.2) only counts after References/Appendix
split_sections(pages) Walks lines, producing Segment(section, page, text, raw); a new segment starts on each page so page numbers stay accurate
chunk_segments(...) 180-word windows with 30-word overlap, never crossing a section boundary. 180 words fits MiniLM's 256-token input limit, so nothing is silently truncated
ingest(source, store, embedder, progress) The whole pipeline (diagram above); writes paper, chunks, vectors and citations in one transaction, so a failure never leaves half a paper
rebuild_citations(store) Re-parses every stored PDF's references, e.g. after improving the parser

embed.py — embeddings

Embedder is a Protocol (dim, embed(texts)), so tests use small fake embedders. SentenceTransformerEmbedder loads the model lazily (thread-safe), normalizes vectors, and offers warm_up() to load in the background at server start so the first search doesn't wait.

search.py — hybrid retrieval

Function What it does
fts_query(text) Turns free text into a safe FTS5 query (each word quoted, OR-ed), so input like "x"? AND NEAR* can't cause syntax errors
keyword_search(...) FTS5 MATCH ranked by BM25, joined to papers for filtering
vector_search(...) sqlite-vec KNN; with filters it over-fetches (10×, min 200) and filters after, since vec0 can't filter on paper columns during the scan
rrf(rankings) Reciprocal rank fusion, score = Σ 1/(60 + rank); records which methods found each chunk
search(...) Runs one or both retrievers, fuses, applies a per-paper cap, and builds SearchHits
make_snippet(text, terms) Picks the window containing the most distinct meaningful query words (short words like "of" ignored)

On a 4-paper, 10-query check (Transformer, BERT, ResNet, word2vec), hybrid search ranked the correct paper first 10/10 times.

citations.py — the citation graph

Function / class What it does
split_references(raw) Tries [n] markers (≥2), then sequential n. markers (≥3, mostly counting up), then hanging indents for author-year styles (first line at the column edge, continuation lines indented). Rejoins words hyphenated across lines
reference_arxiv_id(ref) Finds arXiv:1607.06450, abs/1409.0473, arxiv.org URLs
Matcher Matches a reference to a library paper: arXiv id → exact title (compared as lowercase alphanumerics, so Pre-\ntraining = Pretraining) → fuzzy title (≥92% of title characters found in order, after a cheap word-overlap pre-filter). Titles under 12 characters ("BERT") never match on title alone; the longest matching title wins
link_paper(store, paper, refs) Stores the paper's references matched against the library, and re-checks older papers' unmatched references against the new paper (backlinks). Self-citations are excluded. Re-running replaces the paper's previous citations

server.py — the MCP interface

create_server(db_path=None, embedder=None) builds an MCPServer whose lifespan opens the Store, creates the embedder (warming it up), and yields an AppState(store, embedder, lock). Tools get it from ctx.request_context.lifespan_context. The factory lets tests inject a temporary database and a fake embedder; main() runs the default server over stdio and quiets model-hub HTTP logging (httpx, httpx2, huggingface_hub).

Tools

Tool Arguments Returns Annotations
add_paper source paper summary, already_in_library, chunk and reference counts; reports progress open-world, idempotent
search query, limit=8 (max 30), mode (hybrid/semantic/keyword), year_from, year_to, paper_ids results[]: paper id, title, year, section, page, snippet, matched_by read-only
get_paper paper_id, section? metadata, abstract, sections (with start page), notes — or one section's full text (chunk overlaps removed, partial names accepted) read-only
get_citations paper_id references[] with in_library link, cited_by[] read-only
compare_papers paper_ids (2–8), aspects (1–8), passages_per_aspect=2 passages per paper per aspect read-only
add_note paper_id, text note id, timestamp
list_papers — count, paper summaries read-only
remove_paper paper_id removed id and title (PDF file kept) destructive

Resource: library://papers — the library as a Markdown list. Prompt: literature_review(topic) — guides Claude through search → compare → citations → a cited review.

Paper ids are accepted in full (arxiv:1706.03762) or bare (1706.03762). Expected failures (unknown paper, missing file, download error, unknown section) are raised as ToolError, so Claude gets an actionable message such as "No paper '…' in the library. Use list_papers to see paper ids."


Setup

cd paper-librarian
python -m venv .venv
.venv\Scripts\activate      # Windows  (source .venv/bin/activate on macOS/Linux)
pip install -e ".[dev]"
pytest -q                   # 90 tests

The first run downloads the all-MiniLM-L6-v2 embedding model (~90 MB).

Connect to Claude

Claude Code

claude mcp add paper-librarian --scope user -- <path-to-repo>/.venv/Scripts/python.exe -m librarian.server
claude mcp get paper-librarian   # should show: Status: ✔ Connected

Start a new Claude Code session afterwards; servers are loaded when a session starts. Remove with claude mcp remove paper-librarian -s user.

Claude Desktop — add to claude_desktop_config.json (Settings → Developer → Edit Config), then restart:

{
  "mcpServers": {
    "paper-librarian": {
      "command": "C:/path/to/paper-librarian/.venv/Scripts/python.exe",
      "args": ["-m", "librarian.server"]
    }
  }
}

Try it

  • "Add arXiv 1706.03762 and 1810.04805 to my paper library"
  • "What do my papers say about positional encoding? Cite sections."
  • "Compare BERT and the Transformer on pre-training corpus, model size and results"
  • "Which papers in my library cite Attention Is All You Need?"
  • Use the literature_review prompt with a topic

Configuration

Variable Default Meaning
LIBRARIAN_HOME ~/.librarian Library folder (library.db, downloaded pdfs/)
LIBRARIAN_EMBED_MODEL sentence-transformers/all-MiniLM-L6-v2 Embedding model (must produce 384-dim vectors)
HF_HUB_OFFLINE unset Set to 1 once the model is downloaded to skip Hugging Face checks at startup

Testing and debugging

pytest -q                                   # unit + end-to-end MCP tests
npx @modelcontextprotocol/inspector .venv/Scripts/python.exe -m librarian.server

tests/test_server.py connects the SDK's Client to the server in-process and exercises every tool, the resource and the prompt, including progress notifications and error cases.


Known limitations

  • Scanned PDFs (no text layer) are rejected; there is no OCR.
  • Unusual layouts: three-column pages, or reference lists that fit none of the three styles, may extract or split imperfectly.
  • Subsections replace parent names: text under 3.2.1 is labelled Scaled Dot-Product Attention, not Model Architecture.
  • Filtered semantic search over-fetches and filters afterwards; with many thousands of papers, a very narrow filter could miss results.
  • Ingestion holds the database lock, so other tool calls wait while a paper is being added (typically 5–30 s).
  • Hyphen rejoining also merges genuine compounds broken at a line end (sequence-aligned → sequencealigned).
  • References to papers outside the library are stored as text only (no Semantic Scholar lookup yet).

Roadmap

  • Semantic Scholar lookup for references not in the library
  • OCR fallback for scanned PDFs
  • Parent/child section hierarchy
  • Packaging for uvx paper-librarian

Metadata

Release files for paper-librarian 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for paper-librarian 0.1.0
File Size Uploaded Provenance Actions
paper_librarian-0.1.0.tar.gz 325.5 kB Without provenance Details for paper_librarian-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for paper-librarian 0.1.0
File Interpreter ABI Platform Actions
paper_librarian-0.1.0-py3-none-any.whl Python 3 none any Details for paper_librarian-0.1.0-py3-none-any.whl

Total release size: 356.6 kB

Release files / paper_librarian-0.1.0.tar.gz

Download URL paper_librarian-0.1.0.tar.gz
Size 325.5 kB
Tags Source
SHA-256 checksum
How to use checksums
c991bdf14a767293292a9affb97ed55dd0982160276cc13c27a7aa8c60a56cb2
BLAKE2b-256 checksum
How to use checksums
4771106987a9223017ede4579552bb126b674660884687965eebbd845a29919e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.2

Release files / paper_librarian-0.1.0-py3-none-any.whl

Download URL paper_librarian-0.1.0-py3-none-any.whl
Size 31.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
4d9f87804e906a1691746a47524ab338346e69e507e93529a9e0600209781695
BLAKE2b-256 checksum
How to use checksums
f8f40a8041080aee187354bea3d3f9a5936bfa90422a1444b21d6f752a80bb3b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.2

Security for paper-librarian 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page