Gnosis

Website → clean, provenance-stamped Markdown.
Built for LLM knowledge bases, documentation pipelines, and audit-ready content archives.

PyPI License Version

Point gnosis at a URL and get clean, LLM-friendly Markdown files with a YAML provenance frontmatter block on every file — recording exactly where the content came from, when it was fetched, and how to verify it. No external bookkeeping, no hidden state. Every file is self-describing.

Table of contents

Features

Feature Description
Single page or full site One URL, or crawl every page under a path with --all
Provenance frontmatter Every file records url, fetched_at (UTC), SHA-256 content_hash, status_code, page metadata (title/author/language), and caching headers when the server sends them
Authentication built-in Bearer tokens, HTTP Basic (Confluence Cloud API tokens), or arbitrary headers — secrets always from environment variables
Clean extraction Main-content detection with ordered selectors, class-word boilerplate stripping (sidebars, breadcrumbs, cookie banners, permalink anchors), and framework-aware fixes for Confluence, Sphinx/RTD, and GitHub-style pages
Valid GFM tables Multi-paragraph cells joined with <br>, pipes escaped, sticky-header clone tables removed — tables survive ingestion
Scalable crawling Configurable concurrent fetches, politeness delay, retries with backoff, robots-aware
Scheduler-friendly Headless CLI, meaningful exit codes, JSON crawl manifest — drops into cron, n8n, Airflow, CI
Metadata extraction Title (entity-unescaped), author, language, description, Open Graph fields
Optional QMD integration Index output into a QMD knowledge base with local-LLM context generation (pip install gnosis[qmd])

Installation

Requires Python 3.12+.

git clone https://github.com/SHCV-it/gnosis.git
cd gnosis
python -m venv venv && source venv/bin/activate
pip install -e .

Optional, only if you use --qmd-index (pulls torch + transformers):

pip install -e .[qmd]

Quick start

# One page → one markdown file with provenance
gnosis https://docs.python.org/3/tutorial/

# Crawl an entire section of a docs site
gnosis https://docs.python.org/3/tutorial/ --all -o ./python-docs/

# Preview how many pages would be crawled (no downloads)
gnosis https://docs.python.org/3/tutorial/ --all --dry-run

# Faster crawl with parallel fetches
gnosis https://docs.example.com/ --all --config myconfig.yaml

Provenance: the contract

Every file gnosis writes is self-describing. Default frontmatter:

---
title: Quickstart — Trafilatura 2.2.0 documentation
url: https://trafilatura.readthedocs.io/en/latest/quickstart.html
fetched_at: '2026-08-04T10:24:17Z'
content_hash: 1549512c...16fd
status_code: 200
generator: gnosis/1.1.0
language: en
etag: '"61e917f4cd107c3bce6182b633819fcf"'
last_modified: Fri, 31 Jul 2026 16:07:37 GMT
---
Field Required Description
title Always Page title (from og:title or <title>, HTML entities unescaped)
url Always Final URL after redirects (requested_url added if it differs)
fetched_at Always UTC fetch timestamp, ISO 8601
content_hash Always SHA-256 of the markdown body — use for dedup/change detection
status_code Always HTTP status of the final response
generator Always Gnosis version that produced this file
language If present From <html lang> or og:locale
author If present From <meta name=author>, article:author, or dc.creator
description If present From <meta name=description> or og:description
site_name If present From og:site_name
published_time If present From article:published_time
modified_time If present From article:modified_time
etag If sent Response ETag header
last_modified If sent Response Last-Modified header
requested_url If redirected Original URL before redirects

Add your own constant fields per run (--frontmatter) or per config (output.frontmatter_extra) — custom keys never override the core provenance fields above:

gnosis https://example.com/docs --frontmatter 'tags: [customs, passar]' --frontmatter 'owner: kb-team'

The frontmatter is standard YAML between --- fences: parseable by python-frontmatter, Jekyll, Hugo, Obsidian, and any downstream knowledge pipeline.

Opt out per run with --no-frontmatter or globally in config:

output:
  frontmatter: false

Authenticated fetching

Secrets are read from environment variables only. They are never passed as plain CLI arguments (which leak into shell history and process tables) and never committed in config files.

Confluence Cloud with a Personal Access Token

# Set up a PAT at https://id.atlassian.com/manage/api-tokens
export CONFLUENCE_PAT="your-api-token"

gnosis "https://your-domain.atlassian.net/wiki/spaces/SPACE/pages/PAGE_ID" \
  --basic-user you@example.com \
  --basic-token-env CONFLUENCE_PAT

Bearer token (authenticated API docs, internal tools)

export MY_API_TOKEN="..."
gnosis https://internal.example.com/docs --bearer-token-env MY_API_TOKEN

Custom headers

gnosis https://example.com --header "X-API-Key: ${MY_KEY}" --header "X-Team: docs"

Via config file (multi-run / CI)

downloader:
  auth:
    type: basic                      # bearer | basic | header
    username: "you@example.com"
    password: "${CONFLUENCE_PAT}"    # ${ENV_VAR} expanded at load time

CLI reference

gnosis URL [OPTIONS]
Flag Description
-a, --all Crawl all child pages under the URL path
-n, --dry-run Discover and count pages only (requires --all)
-o, --output DIR Output directory (default: ./)
-c, --config FILE Path to YAML configuration file
-f, --overwrite Overwrite existing output files
-q, --quiet Suppress progress output
-v, --verbose Show detailed conversion diagnostics
--no-frontmatter Write bare markdown without provenance block
--frontmatter KEY: VALUE Extra constant frontmatter field (repeatable)
--header NAME: VALUE Extra request header, ${ENV_VAR} expanded (repeatable)
--bearer-token-env VAR Bearer token from environment variable
--basic-user USER HTTP Basic username (requires --basic-token-env)
--basic-token-env VAR HTTP Basic password/token from environment variable
--qmd-index Index output into QMD (requires [qmd] extra)

Configuration reference

Copy config/default.yaml and pass it with -c. Full reference:

# ── Downloader ────────────────────────────
downloader:
  timeout: 30                # Request timeout (seconds)
  retries: 3                 # Retries on 5xx / network errors
  user_agent: "Gnosis/1.1"   # User-Agent header
  rate_limit_ms: 500          # Minimum delay between requests (0 = no limit)
  respect_robots: true        # Obey robots.txt (future)
  headers: {}                 # Extra HTTP headers (${ENV_VAR} expanded)
  auth:                       # Optional: bearer | basic | header
    type: bearer
    token: "${MY_API_TOKEN}"

# ── Crawler ───────────────────────────────
crawler:
  max_depth: 10              # Crawl depth from seed URL
  max_pages: 500             # Stop after this many pages
  concurrent_requests: 5     # Parallel fetch batch size (1 = sequential)

# ── Converter ─────────────────────────────
converter:
  excluded_tags: [...]        # HTML tags stripped before conversion
  content_selectors: [...]    # Tried in order; first match ≥ 200 chars wins
  strip_classes: [...]        # Exact class-token matches to remove
  strip_class_words: [...]    # Word-level matches inside class names
  include_images: true        # Emit <img> as ![alt](src)
  absolute_urls: true         # Resolve relative links to absolute URLs

# ── Output ─────────────────────────────────
output:
  directory: "./"             # Where .md files go
  overwrite: false            # Skip existing files unless true
  extension: ".md"            # Output file extension
  frontmatter: true           # Write YAML provenance block
  frontmatter_extra: {}       # Constant fields added to every file

# ── QMD (optional) ──────────────────────────
qmd:
  enabled: false              # Enable QMD knowledge base indexing
  llm_model: "Qwen/Qwen3-0.6B"  # HuggingFace model for context generation
  llm_device: "cpu"           # cpu | cuda | auto

Exit codes

Code Meaning
0 Success (crawl mode: at least one page saved)
1 Failure — download error, file exists without -f, nothing saved, bad flags
130 Interrupted (Ctrl-C)

In --all mode a _manifest.json is written to the output directory listing every page with url, file, content_hash, fetched_at, status_code, and title — ready for schedulers and downstream audit.

How clean is the output?

Gnosis is opinionated about boilerplate. By default it:

  • strips script/style/nav/footer/aside/form/template tags and HTML comments (including Confluence's <!-- data-loadable-begin=... --> SSR markers)
  • removes elements matching exact strip_classes tokens AND boilerplate words (sidebar, toc, breadcrumb, cookie, headerlink, sourcelink, …) inside namespaced class names — bd-sidebar-primary is gone, but research-content stays
  • cleans permalink anchors from headings (# Quickstart# → # Quickstart)
  • picks the main content by precedence-ordered selectors (.markdown-body, .ak-renderer-document, .wiki-content, … before main/#content)
  • converts tables to valid GFM: multi-line cells joined with <br>, | escaped, duplicate/sticky-header rows removed
  • resolves relative links to absolute URLs; skips data:-URI images (spacers/tracking pixels)
  • unescapes HTML entities in titles (&#8212; → —)

Everything is configurable — see config/default.yaml.

Development & testing

git clone https://github.com/SHCV-it/gnosis.git
cd gnosis
python -m venv venv && source venv/bin/activate
pip install -e . pytest python-frontmatter

# Run the test suite (offline — only localhost fixtures)
python -m pytest tests/ -q -v

Project structure

gnosis/
  cli/           Click CLI (single page, crawl, dry-run, manifest)
  config/        YAML loading + typed settings dataclass
  core/
    downloader   Async HTTP client, auth, retries, FetchResult
    converter    HTML → Markdown, boilerplate stripping, metadata extraction
    crawler      BFS crawler with concurrent batch fetching
    provenance   Frontmatter generation, content_hash, render_document
  integrations/  QMD pipeline (optional, heavy deps)

The test suite covers converter quality (comment/anchor/boilerplate/table handling, shadow-table dedup, metadata), provenance generation (fields, round-trip parsing, extras merging), auth header injection (3 schemes), crawler link resolution, and CLI end-to-end behavior. Runs entirely offline.

Contributing

Contributions are welcome. Please open an issue first to discuss what you'd like to change.

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing)
  3. Run the tests (python -m pytest tests/ -q)
  4. Commit your changes with clear messages
  5. Push and open a pull request against main

Related projects

Gnosis is designed to feed documentation pipelines and LLM knowledge bases. Pair it with:

  • n8n / cron / Airflow — schedule gnosis runs and pipe results into your downstream pipeline
  • QMD — local vector search via the --qmd-index flag
  • Any Markdown-to-anything pipeline — the YAML frontmatter is parseable by python-frontmatter, Jekyll, Hugo, Obsidian, and standard static-site generators

License

MIT — see LICENSE.

Author: Steffen Hoehne, SHCV.IT

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.1.2)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for alizahidraja from gravatar.com alizahidraja

Credits

Author: Steffen Hoehne

License expression

MIT
View SPDX License List

Requires

Python >=3.12

Provides Extra

qmd

Tags

markdown web-scraping crawler html-to-markdown knowledge-base llm

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

gnosis_markdown-1.1.0.tar.gz (49.5 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

gnosis_markdown-1.1.0-py3-none-any.whl (39.8 kB view details)

Uploaded Python 3

File details

Details for the file gnosis_markdown-1.1.0.tar.gz.

File metadata

  • Download URL: gnosis_markdown-1.1.0.tar.gz
  • Upload date:
  • Size: 49.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/7.0.0 CPython/3.12.13

File hashes

Hashes for gnosis_markdown-1.1.0.tar.gz
Algorithm Hash digest
SHA256 ef4946659c0e4a8eb496417c0540d7763f31839f44e4cc3deb519fa005ebcbc3
MD5 a09ca5ac0952a7c433c80ab310ad91ce
BLAKE2b-256 32fd07fb09ed1ef177017b09867a426fe8bceb356f7dab06f91b33793cb2dc19

See more details on using hashes here.

File details

Details for the file gnosis_markdown-1.1.0-py3-none-any.whl.

File metadata

File hashes

Hashes for gnosis_markdown-1.1.0-py3-none-any.whl
Algorithm Hash digest
SHA256 d614fc695450f907c2bbb2cf46b9e0d1b7eeec5eeac5f649d427322a892c03e4
MD5 75034cc741b240e5e750cf40b68de3f6
BLAKE2b-256 5bbb8d8c027cf93880f1f1b5373460f708807122f07902136780a61d18e8aa34

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page