Gnosis — Web Scraping → Markdown for LLMs (RAG-ready, byte-level provenance)

The auditable web scraper that turns the internet into LLM-ready knowledge.
Open-source · Self-hostable · Byte-level SHA-256 · WARC archival · MIT

Gnosis is an open-source web scraper and crawler that downloads websites (and documents) and converts them into clean, LLM-friendly Markdown for RAG pipelines and knowledge bases — with a YAML provenance block on every file recording where the bytes came from, when they were fetched, and how to verify them. It is the auditable web-to-Markdown converter.

Markdown is a projection; the raw bytes + WARC are the source of truth. Unlike Firecrawl, Crawl4AI, or Jina Reader, gnosis ships byte-level SHA-256 and WARC-grade evidence out of the box.

Why Gnosis

✅ yes · ⚠️ partial · ❌ no — as of September 2026; verify each project's current capabilities and license against its own repository.

Verify it yourself

The provenance claim is the product. After a fetch:

gnosis https://docs.python.org/3/tutorial/ -o out/ --warc

# The frontmatter's bytes_sha256 is the SHA-256 of the raw response:
shasum -a 256 out/.gnosis-store/<bytes_sha256>   # matches the hash in the .md

Every markdown file is re-fetchable and re-verifiable — no sidecar bookkeeping.

Reproducible benchmark evidence: see BENCHMARKS.md.

Quick start

Requires Python 3.12+.

pip install gnosis-markdown

# One page → one markdown file (written to ./ by default; use -o to change it)
gnosis https://docs.python.org/3/tutorial/

# Crawl an entire section
gnosis https://docs.python.org/3/tutorial/ --all -o ./python-docs/

# Archive the raw bytes to WARC + a content-addressed store
gnosis https://docs.python.org/3/tutorial/ --warc

# Emit per-chunk citation manifests for RAG
gnosis https://docs.example.com --chunk

# Evaluate yourself against a corpus (one URL per line)
printf 'https://docs.python.org/3/\nhttps://example.com/\n' > urls.txt
gnosis-bench --urls urls.txt

# Convert a PDF/Office doc to Markdown (pip install gnosis-markdown[docs])
gnosis-doc report.pdf -o report.md

JS rendering is opt-in via a sidecar binary (default obscura); install the Obscura binary, then gnosis https://my-spa.example --render (or set render.engine in config).

Features

Provenance & audit (the moat)

• YAML frontmatter on every file: url, fetched_at (UTC), status_code, etag, last_modified, generator, and requested_url (when redirected).
• content_hash (SHA-256 of the markdown) and bytes_sha256 (SHA-256 of the raw response-body bytes) — hash the bytes, not the derived text.
• WARC export (--warc) + a content-addressed store keyed on bytes_sha256.
• llms.txt + llms-full.txt emitted on every crawl.
• ai.txt consent discovery — the host's ai.txt directives and llms.txt presence are recorded in the frontmatter (ai_txt / llms_txt).

ai.txt is advisory, not enforced. gnosis probes only root-level /ai.txt and /llms.txt (path-scoped files are a known limitation), records the directives it finds, and does not, by default, refuse to scrape. To stop scraping at opt-outs, use --profile strict-optout or an explicit deny_if rule.

ai.txt is advisory, not enforced. gnosis records a site's ai.txt opt-out (Training: Deny) in the frontmatter but does not, by default, refuse to scrape it. If you want scraping to stop at opt-outs, enable a policy: --profile strict-optout or write an explicit deny_if rule. "We record the opt-out and scrape anyway" is exactly the behavior a regulator will ask about — decide it deliberately, don't let it be a default.

Fetching & access

• Static-first async fetch (httpx); optional JS rendering via a sidecar binary.
• robots.txt respected (with Crawl-delay politeness); fail-open on errors.
• SSRF / private-network guard: blocks loopback/RFC1918/link-local/multicast and every redirect hop (opt out with --allow-private-network).
• Auth: Bearer, HTTP Basic (Confluence PAT), and custom headers. Reference secrets as ${ENV_VAR} (supported in config and --header values) and keep them out of config files and shell history.

Extraction & output

• Main-content detection + class-word boilerplate stripping.
• Valid GFM tables (multi-para cells, pipe escaping, sticky-header dedup).
• Metadata extraction (title/author/language/description/Open Graph).
• Chunking (--chunk) with stable chunk ids + heading paths + offsets.
• Resumable crawls (--all) via a content-hash checkpoint.

Chunk manifest example

gnosis https://docs.example.com --chunk writes <page>.md.chunks.json:

[
  {
    "doc_id": "https://docs.example.com",
    "content_hash": "1549512c...",
    "chunk_id": "c0",
    "heading_path": ["Title"],
    "start": 0,
    "end": 812,
    "char_count": 812
  }
]

Provenance: the contract

The full machine-readable contract — every field, its exact semantics, and conformance rules — is in the Capture Record Specification.

---
title: Quickstart
url: https://docs.example.com/quickstart
fetched_at: '2026-09-02T08:41:44Z'
content_hash: 1549512c...16fd     # SHA-256 of the markdown body
bytes_sha256: 85052df6...bcb31    # SHA-256 of the raw response bytes
status_code: 200
generator: gnosis/2.0.0
etag: '"61e917f4..."'
last_modified: Fri, 31 Jul 2026 16:07:37 GMT
---

Standard YAML, parseable by python-frontmatter, Jekyll, Hugo, Obsidian, and any downstream pipeline. Opt out with --no-frontmatter.

CLI reference

gnosis URL [OPTIONS]

Also available: gnosis-bench (reproducible scorecard), gnosis-doc (document → Markdown), gnosis-keygen (generate a signing keypair), gnosis-verify (verify a signed document), and gnosis-mcp (MCP server).

Installation

pip install gnosis-markdown          # core
pip install gnosis-markdown[qmd]     # optional QMD vector-DB indexing
pip install gnosis-markdown[docs]    # optional document conversion (MarkItDown)

Requires Python 3.12+. See gnosis/config/default.yaml for the full configuration reference.

Development

git clone https://github.com/SHCV-it/gnosis.git
cd gnosis
pip install -e '.[test]'
python -m pytest tests/ -q          # offline suite (localhost fixtures)

Contributing

Contributions are welcome — open an issue first to discuss. Report security issues privately (see SECURITY). See ROADMAP.md for the project plan.

License

MIT — see LICENSE.

Authors: Steffen Hoehne & Ali Zahid Raja, SHCV.IT