rfclib — Local RFC Library + MCP Server

A local, self-maintaining library of every published RFC (plus BCP/STD/FYI), sourced exclusively from the official RFC Editor infrastructure (rfc-editor.org — the authoritative publisher of the RFC Series for all five streams, operating under the IETF Administration LLC; see RFC 9280, RFC 4714, RFC 7841).

Everything lives in one SQLite file and is exposed to AI agents over MCP, and to scripts over a CLI. Fully offline after the initial sync: no API keys, no external services; embeddings run locally on CPU.

Install

uvx rfclib init      # catalog: RFC index, errata, subjects (~1 min)
claude mcp add rfclib -- uvx --from "rfclib[mcp,semantic]" rfclib mcp

The core package is small (CLI, library, keyword search, ~9 MB with its dependencies); heavier features are extras:

rfclib status shows which are installed; a missing one says how to add it, and search without semantic uses keywords, identifiers and titles.

uvx fetches rfclib into an isolated, cached environment (and a suitable Python if needed); pip install rfclib works too. RFCs are then fetched from rfc-editor.org the first time they are read; uvx rfclib init --content current prefetches every current RFC so full-text section search covers them all (~10 min, ~1.5 GB).

Requirements: Python 3.12+ on Windows, macOS or Linux. Semantic search (rfclib[semantic]) uses a ~30 MB local embedding model (minishlab/potion-base-8M, model2vec) and sqlite-vec; on a Python whose sqlite3 cannot load extensions, search is keyword-only.

Sources (first-party only)

Every local file is recorded with source URL + SHA-256 for provenance.

From source

git clone https://github.com/b1tninja/rfclib && cd rfclib
uv sync --group dev              # .venv with rfclib (editable) + pytest
uv run pytest -q                 # offline tests, sample library

rfclib init                     # catalog: RFC index, errata, subject tree
rfclib status                   # where the library lives, what it holds

That is enough to use the MCP server: status, obsolete chains, document search, browsing and errata cover every RFC, and each RFC is fetched from rfc-editor.org, segmented and indexed the first time it is read (about a second). To make full-text section search cover everything up front:

rfclib init --content current   # + every current RFC, indexes, vectors (~10 min)
rfclib init --content all       # + obsolete RFCs too

Later updates are incremental (rfclib sync && rfclib index build && rfclib index embed). The official rsync service still works for a bulk mirror: scripts\mirror.ps1, then rfclib scan.

# Explore
.venv\Scripts\rfclib.exe hsearch "proving address ownership"  # hybrid — recommended
.venv\Scripts\rfclib.exe search "congestion control"          # titles/abstracts
.venv\Scripts\rfclib.exe search "KeyUpdate" --body            # BM25, sections
.venv\Scripts\rfclib.exe vsearch "proving address ownership"  # semantic only
.venv\Scripts\rfclib.exe toc rfc9000                          # outline + line ranges
.venv\Scripts\rfclib.exe get rfc9000 --start-line 6858 --lines 40
.venv\Scripts\rfclib.exe rels rfc2616
.venv\Scripts\rfclib.exe resolve bcp14
.venv\Scripts\rfclib.exe subjects get quic               # curated topic listing
.venv\Scripts\rfclib.exe status

CLI reference

All searches accept --include-obsolete (obsolete docs are hidden by design — an agent reading RFC 2616 in 2026 is a failure mode).

Retrieval indexes (full-text + semantic)

Documents are chunked by section (heuristic parser for RFC txt, exact source line ranges kept; long sections split into ~2.4k-char parts with per-part line ranges). All three search modes live in the same SQLite file: BM25 via FTS5 (search --body), KNN via sqlite-vec (vsearch), and hybrid (hsearch). Chunk IDs are content-addressed, so re-chunking preserves embeddings for every section whose text didn't change (a chunker tweak re-embeds only the diff, e.g. 15k of 298k chunks in ~11 s).

hsearch fuses up to four lists with diversified, weighted RRF (the fourth, an exact-identifier index, joins when the query contains tokens such as preferred_address, SETTINGS_INITIAL_WINDOW_SIZE or X25519, weighted by how rare they are; see docs/retrieval-design.md): section-level BM25 (terms OR'd + proximity-boosted), the embedding KNN list, and a document-level list ranked by how much of the query the doc's own title carries (IDF-weighted term coverage — the spec titled "TCP Congestion Control" beats one that merely mentions "NewReno"). Each doc appears once per list before fusion, chunks of the same doc corroborate each other, and standards-track/BCP docs get a status boost while References/boilerplate sections are penalized. Without index embed it still works (BM25 + title).

Full design rationale, measured trade-offs, and FTS5/chunker gotchas: docs/retrieval-design.md.

Accuracy loop (regression-test retrieval changes)

python scripts\eval_retrieval.py     # golden queries -> expected doc, top-5

A set of hand-picked (query → canonical-RFC) pairs run through hsearch. Any change to chunking, FTS query shape, or fusion should be built, re-embedded, and re-evaluated with this script before shipping (currently 10/12 in top-5; slowest cold query ~1.8 s — see docs/retrieval-design.md §5 for the two known-hard misses).

MCP server

Any MCP client launches the server as a subprocess over stdio. With uvx there is nothing to install first (--from "rfclib[mcp,semantic]" brings the MCP and semantic-search extras); set RFCLIB_HOME in env only to use a library somewhere other than the default per-user data directory.

Claude Code

claude mcp add rfclib -- uvx --from "rfclib[mcp,semantic]" rfclib mcp

Claude Desktop (claude_desktop_config.json) and Cursor (~/.cursor/mcp.json):

{
  "mcpServers": {
    "rfclib": {
      "command": "uvx",
      "args": ["--from", "rfclib[mcp,semantic]", "rfclib", "mcp"]
    }
  }
}

VS Code (.vscode/mcp.json):

{
  "servers": {
    "rfclib": {
      "type": "stdio",
      "command": "uvx",
      "args": ["--from", "rfclib[mcp,semantic]", "rfclib", "mcp"]
    }
  }
}

GUI clients don't always see your shell's PATH; if uvx isn't found, use its absolute path (which uvx / where uvx). From a source checkout, point command at the venv's rfclib executable instead (this repository's .mcp.json does that for development).

Tools (9)

One job per tool, all answering in compact plain text; every tool's main argument is ref (only search takes query), all addressed by refs, which are also citations. Every page of read ends with next: <ref> to continue from: RFC9000 (abstract + outline), RFC9000§9.6 (section incl. subsections), RFC9000§9.6¶2 (paragraph), RFC9000#table-2 (RFCXML block id), RFC9000:2964-2979 (txt lines); BCP14/STD7 resolve to their RFCs. Errors come back as isError results with the fix in the message; all tools are annotated read-only; search and read carry _meta["anthropic/alwaysLoad"] so they stay visible when a client defers tools behind tool search.

Prompts (user-invoked slash commands in Claude Code): /rfclib:explain <ref>, /rfclib:checklist <ref> (normative statements → implementation checklist), /rfclib:current <topic> (which RFCs are current). Resource template rfc://{ref}: @rfclib:rfc://RFC9112§6.3 attaches a section to a prompt. Completions for document ids and section numbers.

Agent guidance shipped in the repo (.claude/): the rfc-research skill (workflow, ref grammar, recipes) and a PreToolUse hook (scripts/hooks/guard_mirror.py) that denies Read/Grep/Glob on data/mirror/** and points at the MCP call to use instead.

Missing documents are fetched on demand from rfc-editor.org.

Subject views (RFC-Editor organization)

The RFC Editor curates a ~640-subject tree (web > http > http2, transport > quic, …) and assigns every document to one or more subjects; it is published only as HTML at rfc-editor.org/subjects/. rfclib subjects sync mirrors it (one polite crawl, a few minutes; HTML cached under meta/subjects/ so re-parses are offline) into subjects + doc_subjects tables. On top of that: the browse / links(direction='related') MCP tools, search(scope='documents', subject=...), subject tags in info, and the same via CLI (rfclib subjects list|get). This answers "what RFCs exist about X" and "what else is like this RFC" without a keyword query — the citation-graph view (links) stays available for normative context.

Configuration

Environment variables (the same in a shell or an MCP client's env block):

• RFCLIB_HOME: the library root. Default: the per-user data directory (%LOCALAPPDATA% fclib on Windows, ~/.local/share/rfclib or $XDG_DATA_HOME/rfclib on Linux, ~/Library/Application Support/rfclib on macOS). Never the working directory, because MCP clients start servers from arbitrary directories. Older versions defaulted to ./data; rfclib status points out such a library and how to keep using it.
• RFCLIB_NETWORK: auto (default; fetch documents on demand), on-demand, or offline (never touch the network; answer from what is local, and say so).
• RFCLIB_AUTO_INIT=1: let the MCP server initialize an empty library's catalog in the background on first use (opt-in; off by default).

The library is portable: file paths are stored relative to mirror/, so a library can be moved, copied to another OS, or mounted into a container. Several processes can share it (one MCP server per client session plus rfclib sync): SQLite runs in WAL mode with a busy timeout, downloads are written atomically, and bulk writers take a lock file.

<RFCLIB_HOME>/
  rfclib.db     one SQLite file: docs, doc_relations, files (sha256 provenance),
                chunks, chunks_fts + docs_fts (FTS5), chunks_vec (sqlite-vec),
                embed_state (content-hashed embed coverage),
                blocks + xrefs (paragraphs and section-level citations),
                seg_quality (how each document was segmented), errata
  meta/         rfc-index.xml (official manifest)
  mirror/       document files in rfc-editor.org layout (rfc/, bcp/, ...)
  models/       the embedding model (~30 MB), loaded offline

Query tips

• hsearch/search treat words as OR + proximity — natural phrasing works ("does this spec allow sending packets early?"); no operators needed.
• search --body and search(mode='exact') pass your query raw to FTS5: quote phrases ("KeyUpdate message"), combine with AND/OR/NOT, prefix with *. NEAR takes the comma form: NEAR(tls handshake, 40) (NEAR(tls handshake / 40) is a syntax error).
• Keep version tokens in queries (TLS 1.3, 802.11): they are strong discriminators and hybrid search preserves them unstemmed.
• For a long RFC, outline first, then read('RFC9000§9.6') and follow the next: ref — don't page through 8,000-line documents, and never point an agent at the mirror files (its file reader truncates them).
• RFCs since 8650 are indexed from their RFCXML source (rfclib sync --formats xml && rfclib index build); older RFCs are segmented from their own Table of Contents (rfclib/index/txtseg.py, quality per doc in the seg_quality table). Either way you get exact sections, paragraph refs, view=normative/code and section-level links.
• rfclib errata --sync-all loads every erratum from the RFC Editor bulk API once, so errata status is known offline for every document.

Troubleshooting

Content packs & Android reader

RFCs are published rolling (~5–25/month, no official batches), so updates are distributed as content packs: a .rfcpack zip holds one curated collection — document files + a pack.json manifest (per-file sha256) + an offline SQLite FTS5 index (search.sqlite) so the phone searches without a backend. Collections are tiny JSON manifests (explicit doc ids and/or selectors over series/stream/area/WG/year); built-ins:

Put your own <id>.json in data/collections/ (or --collections-dir) to define new ones. history: true pulls in the obsoleted lineage too.

.venv\Scripts\rfclib.exe packs list
.venv\Scripts\rfclib.exe packs build essentials standards-current   # -> export-packs/
.venv\Scripts\rfclib.exe packs serve --dir export-packs --port 8100

android-reader/ is a dependency-free Android app (Java, mirrors the caionline pack-install design): install a .rfcpack via the file picker, adb push, or Sync against packs serve on your LAN (downloads are sha256-verified against catalog.json, installs are verified file-by-file against pack.json before promotion). Browse packs → documents, full-text search across all installed packs, and read in the WebView via a loopback server — fully offline after sync.

Third-party export formats (packs export)

For readers outside this repo (e-book apps, offline browsing), the same collections export to standard formats — one RFC .txt becomes one clean XHTML chapter: page furniture stripped, ASCII diagrams preserved verbatim in <pre>, section anchors generated with the same heuristics as the search index, and RFC 9000/[RFC9000] references linkified (internal to the set by default; --link-refs external adds rfc-editor.org links).

.venv\Scripts\rfclib.exe packs export essentials          # -> export-formats/essentials.epub + essentials/
.venv\Scripts\rfclib.exe packs export standards-current --formats epub
.venv\Scripts\rfclib.exe packs export standards-current --formats zim   # needs: uv sync --extra zim
.venv\Scripts\rfclib.exe packs export standards-current --formats docset

• EPUB (EPUB3, pure stdlib zip): works in ReadEra/Apple Books/Play Books; the nav document gives per-document chapters nested with their top sections. Reader-side full-text search indexes the chapters.
• Dash docset (--formats docset, pure stdlib): <id>.docset bundle + .tgz — drag into Dash (or Zeal/Xcode). The docSet.dsidx lookup index carries one entry per document plus one per top-level section/appendix (RFC9110: 4.2. HTTP Messaging → deep-link straight to that anchor).
• HTML mirror: plain folder of pages + index.html — greppable, Everything/Windows-Search friendly, opens anywhere (the CHM successor, without CHM's dead tooling; CHM itself was rejected: no maintained Python writer, no Android support).
• ZIM (Kiwix) (--formats zim; optional native bindings, uv sync --extra zim — libzim wheels carry Xapian): one .zim per collection with a full-text index + title suggestions built at write time, so Kiwix apps / kiwix-serve --port 8080 export-formats\standards-current.zim give offline full-text search over the corpus. Entries are flat and lowercase (rfc9000), the main page is the collection index, and a generated 48x48 PNG icon shows in the Kiwix catalogue.

Validation: python scripts\check_export.py verifies EPUB/HTML structure (mimetype placement, OPF/nav well-formedness, anchors, cross-links, furniture stripping), python scripts\check_zim.py reopens a built ZIM with the libzim reader (entry counts, main-page redirect, metadata, illustration, anchors/links, and live Xapian suggestion + full-text queries), and python scripts\check_docset.py validates the docset bundle (plist, dsidx rows + every url/anchor resolving, tgz layout) — all end-to-end against the essentials collection.

Search engine compatibility: the packs carry an FTS5 index, but AOSP's system SQLite does not enable FTS5 — external/sqlite/dist/Android.bp on Android 11 and 16 defines FTS3/FTS4 only, and an API 36 emulator image has no fts5 module (MATCH fails with no such module: fts5; OEM ROMs sometimes add it). RfcSearch therefore probes FTS5 once per process and falls back to portable LIKE-based AND matching with density ranking when the module is absent — correct, though slower and unstemmed. Queries pass a stop-word filter: SQLite FTS5 has no native stop-word support (no built-in tokenizer option exists, verified through SQLite 3.54), so filtering happens at query time and needs no pack rebuild; RFC normative keywords (MUST/SHOULD/MAY/CAN) are deliberately kept out of the stop list because they carry meaning in this corpus. A future option for indexed search on every device is an FTS4 mirror table (FTS4 is guaranteed on AOSP) or bundling a native SQLite with FTS5.

cd android-reader
.\gradlew.bat assembleDebug          # APK: app\build\outputs\apk\debug
.\gradlew.bat test                   # installer validation unit tests
adb install app\build\outputs\apk\debug\app-debug.apk

Reader UI and Docker images

The React reader UI (rfclib ui, with its stdlib JSON API) and the Docker images that serve it live on the reader-ui branch, kept out of the package so it needs no npm toolchain. Check that branch out to build or run them.

Roadmap

1. ✅ Bulk download + incremental sync (rsync bootstrap + HTTP syncer)
2. ✅ MCP presentation (search / get / metadata / aliases / recents)
3. ✅ Full-text + semantic + hybrid retrieval, golden-set eval harness
4. ✅ React + Vite reader UI (rfclib ui): JSON API + SPA — now on the reader-ui branch
5. ✅ Cross-references: citation graph mined from txt (normative/ informative/mention + section-level anchors, rfclib refs), exposed in CLI and MCP with section deep links
6. ✅ Verified-errata overlay from the RFC Editor errata service (bulk API + per-doc fallback, rfclib errata, MCP errata)
7. ✅ Content packs (.rfcpack + collections + LAN serve) and offline Android reader (android-reader/) with FTS5 search
8. ✅ Third-party export (packs export): EPUB3, static HTML mirror, Kiwix ZIM (libzim + Xapian full-text index, optional zim extra) and Dash docset (stdlib dsidx with section deep-links) — all four over the shared txt→XHTML render layer
9. ✅ Multi-stage Docker workflow serving the reader UI — on the reader-ui branch

License

rfclib is MIT-licensed (see LICENSE). The RFCs it downloads are not: they are published by the IETF Trust under the Trust Legal Provisions. They may be copied and distributed in full but not modified; see NOTICE.