fastref 📚⚡

Fast offline documentation reference desk and skill generator for AI coding agents.

fastref gives AI coding agents an instant, local offline reference desk. It indexes documentation from any tool, framework, or site into a high-performance SQLite FTS5 database with BM25 ranking, and automatically generates self-contained Agent Skills (under ~/.agents/skills/ or ./.agents/skills/).

Coding agents can query authoritative documentation and code examples locally in sub-milliseconds without hallucinating APIs, wasting web search latency, or blowing context windows.

---

The Ingestion Pattern

Instead of fragile ad-hoc scrapers, fastref unifies documentation ingestion around two core patterns:

1. Harvest & Assemble (fastref harvest): For tools and websites that publish an llms.txt file (or website root). fastref spiders and crawls linked markdown documentation, resolves relative references, checks external domain boundaries, and bundles pages into clean, structured documentation chunks.
2. Fetch & Index Structured JSON (fastref add): For tools and documentation systems that generate machine-readable search indexes (such as Fuse.js, Algolia, or MkDocs-style search JSON endpoints). fastref ingests both flat arrays and nested section trees directly.

---

Features

• Dual Ingestion Engine:
  • llms.txt Spider & Harvester: Discovers and indexes documentation hierarchies from llms.txt files or site roots, featuring domain boundary controls, crawl depth limits, and link preview (--dry-run).
  • Structured JSON Ingestor: Automatically normalizes both flat lists and nested hierarchical search indexes into a unified schema.
• Hybrid Search (BM25 + Dense Vectors):
  • Lexical BM25: Native SQLite FTS5 for exact keywords, function names, and prefix matching.
  • On-Device Semantic Vectors: Google's LiteRT-LM with EmbeddingGemma 2 Text-270M (quantized to 157 MB, 256d normalized Matryoshka embeddings) for conceptual retrieval without external APIs or background daemons.
  • Reciprocal Rank Fusion (RRF): Merges lexical and semantic rankings for optimal recall and precision.
• Zero-Daemon Architecture: In-process inference starts in ~180ms, eliminating background sockets, daemons, and orphaned processes.
• Prompt Injection & SSRF Hardened: Defends against indirect prompt injection by stripping invisible unicode/HTML comments, sanitizing skill frontmatter, blocking private/metadata network IPs, and framing retrieved docs in untrusted reference envelopes.
• Automatic Agent Skill Generation: Creates clean SKILL.md definitions with security boundary notices and automatically bridges them into active agent harnesses (~/.agents/skills/, ~/.gemini/config/skills/, ~/.claude/skills/, ~/.config/goose/skills/).
• Harness-Agnostic: Compatible with Antigravity, Claude Code, Cursor, Codex, Goose, Aider, and any agent that supports shell commands or standard skill directories.
• XDG-Compliant Caching: Keeps database artifacts neatly organized in $XDG_CACHE_HOME/fastref (~/.cache/fastref).

---

Installation

# Using pip
pip install fastref

# Or with uv
uv tool install fastref

---

Quick Start

1. Ingest Documentation & Generate a Skill

Pattern A: Harvest from llms.txt or Site Root

# Harvest from an llms.txt index
fastref harvest mytool https://example.com/llms.txt

# Or point directly to the website root (auto-detects /llms.txt)
fastref harvest mytool https://example.com

# Preview discovered links and external domains without fetching
fastref harvest mytool https://example.com/llms.txt --dry-run

Pattern B: Ingest Pre-Built Structured Search JSON

# Index structured search index JSON directly from a URL or local file
fastref add mytool https://example.com/search_index.json
fastref add mytool ./path/to/search_index.json

By default, both commands:

1. Compile an SQLite FTS5 database to ~/.cache/fastref/<name>.db.
2. Archive the raw manifest/index to ~/.agents/skills/<name>/resources/raw_index.json.
3. Write ~/.agents/skills/<name>/SKILL.md with trigger rules for AI agents.
4. Auto-bridge the skill into installed agent environments (~/.gemini/config/skills/, ~/.claude/skills/, ~/.config/goose/skills/).

2. Search Documentation

fastref search provides three adaptive profiles:

• find (default when vectors are enabled): Returns the top focused code & API snippets (default limit: 3) without catalog clutter.
• directory (default when vectors are disabled): Returns a catalog of matching topic titles (capped at 20) plus top snippets.
• lucky: Prints the top matching document in full, followed by 5 related topic links.

# Focused search (find profile: top snippets, hybrid RRF when vectors enabled)
fastref search mytool "authentication"
fastref search mytool "middleware" --limit 5

# Directory profile: view matching topic catalog + top snippets
fastref search mytool "jwt_validation" --directory

# Lucky profile: print full content of top match + related topic links
fastref search mytool "installation" --lucky

# Show full content of all matches directly
fastref search mytool "config" --full

# Retrieve full documentation by exact URL or topic ID
fastref get mytool "/docs/auth#jwt"
fastref get mytool "/docs/users" "/docs/roles"

3. Optional: Enable On-Device Vector & Hybrid Search

Vector features are disabled by default to keep the CLI lightweight and zero-noise. When enabled, fastref uses Google's LiteRT-LM and downloads the EmbeddingGemma 2 Text-270M model (157 MB).

# Enable vector features (downloads the model artifact on first run)
fastref vector enable

# Backfill vector embeddings for an already indexed library
fastref vector index mytool

# Check vector status and indexed libraries
fastref vector status

# When vectors are enabled, fastref search automatically defaults to hybrid RRF:
fastref search mytool "how to handle webhooks asynchronously"

# Or query explicitly via semantic or lexical mode:
fastref search mytool "reactive state management" --semantic
fastref search mytool "on_mount_hook" --lexical

# Disable vector features at any time
fastref vector disable

4. Explore Documentation Structure

# View table of contents chapters and topic counts
fastref toc mytool

# Filter chapters matching a keyword
fastref toc mytool auth

5. Manage Indexed Documentation

# List all indexed tools, chunk counts, vector status, and database sizes
fastref list

# Remove an indexed tool (both cache database and skill folder)
fastref remove mytool

---

How Agents Use fastref

When fastref generates a skill, it writes a clean SKILL.md into your agent's skill directory. When you ask your agent to implement a feature or debug code using an indexed tool:

1. Discovery: The agent runs fastref search <name> "<query>" to locate exact components, function signatures, or concepts.
2. Context-Efficient Retrieval: The agent inspects targeted snippets or pulls complete topic documentation via --lucky or fastref get <name> <url>, without loading megabytes of irrelevant HTML into context.
3. Accurate Implementation: The agent codes against authoritative local documentation and tested code examples.

---

CLI Reference

fastref harvest <name> <source>

Harvest documentation from an llms.txt file or website root:

• <name>: Identifier for the tool or library.
• <source>: URL or local path to llms.txt (or website root URL).
• --max-docs <N>: Maximum documents to spider and index (default: 60).
• --max-depth <D>: Maximum spider crawl depth (default: 2).
• --dry-run: Preview discovered links and domain categorization without fetching or indexing.
• --allow-urls <prefixes>: Comma-separated URL or domain prefixes to allow (e.g. https://raw.githubusercontent.com/...).
• --allow-all-urls: Allow fetching documentation from any external URL or domain.
• --allow-remote: Allow network fetching when source is a local file.
• --allow-private-ips: Allow fetching from private, loopback, or local network IPs (e.g. localhost during dev).
• -p, --project: Install the skill into the local project directory (./.agents/skills/) instead of global user skills.
• --skills-dir <dir>: Custom destination directory for the generated skill.
• --cache-dir <dir>: Custom destination directory for the SQLite database.
• --description <desc>: Custom description for the YAML frontmatter in SKILL.md.
• --no-skill: Only build the SQLite index without generating an agent skill.

fastref add <name> <source>

Index documentation from a pre-built machine-readable search index JSON:

• <name>: Identifier for the tool or library.
• <source>: HTTP/HTTPS URL or local path to search_index.json.
• --allow-private-ips: Allow fetching from private, loopback, or local network IPs.
• -p, --project: Install the skill into the local project directory (./.agents/skills/) instead of global user skills.
• --skills-dir <dir>: Custom destination directory for the generated skill.
• --cache-dir <dir>: Custom destination directory for the SQLite database.
• --description <desc>: Custom description for the YAML frontmatter in SKILL.md.
• --no-skill: Only build the SQLite index without generating an agent skill.

fastref search <name> <query>

Search documentation with BM25 relevance ranking:

• <name>: Identifier for the indexed tool.
• <query>: Search query (supports SQLite FTS5 match expressions).
• --lucky: Print the entire top match immediately and list remaining candidate snippets.
• --full: Include full topic content for all matches directly.
• --limit <N>: Maximum results to return (default: 8).

fastref get <name> <url1> [url2 ...]

Fetch and display the full documentation text for one or more documents by URL or topic ID.

fastref toc <name> [filter]

Display Table of Contents chapters, topic counts, and sample titles, optionally filtered by keyword.

fastref list

List all indexed tools, their chunk counts, and database sizes.

fastref remove <name>

Remove an indexed tool's SQLite database cache and associated skill directories.

---

License

MIT License.