gossamer — High-Performance LLM Web Researcher

A hybrid LLM web researcher combining a Rust parsing core (PyO3) with Oxide extractors for documents and a Python orchestration layer for caching, rate limiting, budgets, tool routing, and multi-provider search.

Docs: Quick reference · Architecture · Changelog

Architecture

LLM Agent / User
       │  MCP (mcp_server.py) · CLI (cli.py) · skills (bundled)
       ▼
WebResearcherToolbox (agent_tools.py) — facade, no logic
       │  TOOL_REGISTRY (config.py): one source of truth
       ▼
Collaborators (Python: HTTP, keys, rate limits, orchestration)
fetch · search · crawl · document · discovery · 38 domain adapters
       │  JSON strings down, JSON strings up
       ▼
_core (Rust): all response parsers, HTML metadata (in-core meta_oxide
crate), SSRF/robots, budgets, guard, citations, tokens, scoring

Python decides, Rust parses. Details: Architecture.

Features

• Zero API Keys: DuckDuckGo plus 20+ keyless domain adapters (OpenAlex, Eurostat, Bundesbank, HUDOC, …)
• Multi-Provider Search: Google, Bing, Exa alongside DuckDuckGo with failover or merged results
• Domain Providers: research_by_category classifies queries (incl. German/EU terms like Leitzins, BVerfG, HICP) into scholarly / legal / patent / financial / geo — keyless-first
• Patent Providers: EPO OPS, KIPRIS, PatentsView, Lens (all key-gated, fail fast with the exact variable name; Lens aggregates WO/EP/DE/CN/US, trial is non-commercial/academic) + keyless Google Patents number lookup
• Documents: PDF/DOCX/XLSX/PPTX plus TXT/MD/CSV/JSON/XML/feeds; tables render as markdown by default; store=True, include_images=True saves PDF figures; scholarly extraction prefers publisher JATS XML (EuropePMC / PMC OA / PLOS via bobine, fused math + \tag{} displays) when the source is DOI-resolvable (prefer_jats, default on)
• Crawl: bounded relevance-ranked BFS (BM25 idfs + thesaurus + anchor context); documents collected, never fetched
• Citations: BibTeX / CSL-JSON / APA / MLA from search results, no extra network calls
• HTML Metadata: 13 formats via the in-core meta_oxide crate — no separate install, no PyPI blocker
• Guard (optional, off): JailGuard ONNX detector — annotate / redact / block
• Production-Ready: TTL + size-cap caching, per-domain rate limits, robots/SSRF compliance, retries, observability

Quick Start

Prerequisites

• Rust 1.82+ (rustup), Python 3.10+, maturin

Build & Install

git clone https://github.com/opticsWolf/gossamer && cd gossamer
pip install -r requirements.txt
maturin develop --release
# Optional: headless-browser rendering for JS-heavy pages
# (`use_smart="browser"`). Windows/macOS only — no Linux wheels exist,
# so this stays an extra (never a hard dependency) and static fetch
# remains the default.
pip install -e ".[browser]"

Basic Usage

from gossamer import ToolboxConfig, WebResearcherToolbox

tools = WebResearcherToolbox(ToolboxConfig(
    cache_dir="./cache",
    max_tokens=4000,
    model_name="gpt-4o",
))

results = tools.web_search("latest AI research papers", max_results=5, search_only=True)
content = tools.inspect_html_page("https://arxiv.org/abs/1234.5678")
pdf = tools.extract_document("https://example.com/paper.pdf")
with_figs = tools.extract_document("https://example.com/paper.pdf",
                                    store=True, include_images=True)
report = tools.research_by_category("EZB Leitzins", max_results=5)

Async Usage

results = await tools.search_web_async("rust programming")

What "async" means here (thread pool). The *_async wrappers offload the shared blocking implementation to Python's default thread-pool executor (loop.run_in_executor(None, …)), keeping the event loop responsive — but the underlying network I/O is still synchronous. Use them inside asyncio apps to avoid blocking the loop; call the sync methods otherwise. Full model: Architecture.

Tools (twelve MCP tools, everywhere)

MCP tools, CLI commands (gossamer …), and execute_tool(name, args) are the same surface, param-for-param: web_search, inspect_html_page, batch_inspect_pages, download_file, locate_pdf, extract_document, discover_resources, crawl, manage_cache, research_by_category, export_citations, check_sources. download_file saves an opaque remote file without requiring extraction; caller-supplied fallback_urls are tried sequentially with normal robots/SSRF checks, and resume=true continues a partial file with Range/If-Range (restart when the server ignores Range). Use extract_document when you also want parsed text. locate_pdf resolves a DOI to OpenAlex OA PDF/landing-page candidates (plus Unpaywall v2 when GOSSAMER_UNPAYWALL_EMAIL is configured) without downloading them. The CLI adds gossamer categories (routing table) and gossamer skills list|show|install (bundled skill management) — neither is an MCP tool. Parameters: Quick reference.

tools.get_llm_definitions()  # OpenAI-compatible function definitions
tools.execute_tool("inspect_html_page", {"url": "https://example.com"})

Domain Providers (research_by_category)

Euro terms route automatically (EZB, Leitzins, HICP, EGMR, BVerfG, DSGVO, …). For precise OpenAlex queries, gossamer research QUERY --provider openalex --filter 'type:article' --select 'id,title,doi' --title 'gradient index' --author 'Smith' passes provider-native controls; those options are rejected for other providers. --title/--author map to the verified title.search/raw_author_name.search filters and combine with --filter. Explicit scholarly multi-search is opt-in via gossamer research QUERY --providers openalex arxiv; it merges by DOI/arXiv ID, preserves each source record, and never fans out by default.

Paper collection: research → check --mode status → optional locate-pdf DOI → download URL -o FILE --expect-format pdf → extract DOI (JATS-first: publisher full text when resolvable) or extract FILE --no-prefer-jats → cite DOI (or cite FILE.pdf [--from-pdf] for a DOI detected from the file). Supplied mirror URLs are checked independently; gossamer respects robots/SSRF policy and does not bypass bot walls. The skill has the full recipe — it ships inside the wheel, and gossamer skills install copies it into your agent's discovery directory.

Configuration

Env wins over file, always. Keys: GOSSAMER_* env > legacy STITCH_* > keystore ($GOSSAMER_KEYSTORE > gossamer.json:keystore > ~/.gossamer/keys.json) > gossamer.json "keys". Config file: explicit > $GOSSAMER_CONFIG > ./gossamer.json > ~/.gossamer/config.json.

python -m gossamer.keystore --init          # 0600 template, fill it in
python -m gossamer.keystore --init-config   # gossamer.json template
python -m gossamer.keystore --check         # validate, never prints secrets

{ "max_tokens": 4000, "model_name": "gpt-4o", "fetch_mode": "auto" }

OpenAlex works without a key for casual use. Set GOSSAMER_OPENALEX_KEY (in the keystore or environment) for the higher daily budget. If you want to identify your client by email, set GOSSAMER_OPENALEX_EMAIL; it is sent as mailto and in the request headers. No placeholder email is sent when unset. Semantic Scholar is also callable without a key, but keyless requests share a pool and may receive 429s; set GOSSAMER_SEMANTICSCHOLAR_API_KEY for its individual one-request-per-second allowance.

Harness Integration (pi, Codex, Claude Code)

Same stdio server everywhere (python -m gossamer.mcp_server); keys stay in the keystore, never in client configs. Shallowest first: direct CLI (gossamer search|research|inspect|download|locate-pdf|extract|…, 1:1 with MCP) → MCP (directTools) → gossamer/skills/gossamer/SKILL.md (bundled in the wheel and served as the skills://gossamer MCP resource; gossamer skills list|show|install manages it).

pi (mcp.json, then reload):

{ "mcpServers": { "gossamer": {
  "command": "D:/User/Documents/Python/gossamer/.venv/Scripts/python.exe",
  "args": ["-m", "gossamer.mcp_server"],
  "env": { "GOSSAMER_CACHE_DIR": "D:/User/Documents/Python/gossamer/.gossamer_cache",
            "GOSSAMER_LOG_LEVEL": "WARNING" },
  "directTools": true } } }

Codex (~/.codex/config.toml):

[mcp_servers.gossamer]
command = "D:/User/Documents/Python/gossamer/.venv/Scripts/python.exe"
args = ["-m", "gossamer.mcp_server"]
startup_timeout_sec = 30

Claude Code:

claude mcp add gossamer -- D:/User/Documents/Python/gossamer/.venv/Scripts/python.exe -m gossamer.mcp_server

Keep crawls modest (max_pages ≤ 15) — long runs outlast harness timeouts.

Running Tests

Hermetic by default (no network, SSRF on):

.venv/Scripts/python.exe -m pytest -q -n auto --ignore=tests/test_live_smoke.py
GOSSAMER_LIVE=1 pytest tests/test_live_smoke.py   # opt-in endpoint-drift check

Subset markers (-m area_search|area_fetch|area_crawl|…) are assigned by filename in tests/conftest.py. Details: Architecture.

License

Dual-licensed under MIT OR Apache-2.0 — zero copyleft, zero JVM.