snapshot — offline website snapshots

Take a fast offline copy of any public website, then serve it locally.

Install

Requires Python 3.10+.

pip install web-snapshot-cli

Then run:

snapshot https://example.com ./mirror

Command generator: snapshot.sushii.dev

Usage

Snapshot a single page

snapshot https://example.com ./mirror

Crawl an entire site

snapshot --crawl https://docs.example.com ./docs --max-pages 200 --depth 4

Save pages as Markdown

snapshot https://example.com ./mirror --lang md

Crawl with filters and politeness

snapshot --crawl --include '/docs/*' --exclude '/docs/drafts/*' \
  --crawl-delay 1 --robots https://docs.example.com ./docs

Authenticated pages

snapshot --cookie session=abc123 --header "Authorization: Bearer TOKEN" \
  https://app.example.com ./mirror

Sitemap-based crawl

snapshot --crawl --sitemap https://example.com ./mirror --max-pages 500

Gobuster-style path discovery

Brute-force common paths (including robots.txt, sitemap.xml, admin panels, etc.) using bundled wordlists (common + ~18k large entries). Add your own SecLists/gobuster wordlists with -w:

snapshot --crawl --gobuster https://example.com ./mirror
snapshot --crawl -w common -w large -w /path/to/SecLists/Discovery/Web-Content/common.txt URL ./out
snapshot --crawl --gobuster --wordlist-ext html,php,asp URL ./out

Resume an interrupted snapshot

snapshot --resume --crawl https://example.com ./mirror

Dry run (no writes)

snapshot --dry-run --verbose --crawl https://example.com ./mirror

Extra args (positional)

Any key=value pairs after the output directory are merged into options:

snapshot https://example.com ./mirror crawl=true max-pages=100 lang=html concurrency=32

Restore locally

snapshot -restore ./mirror

This starts a local HTTP server (default http://127.0.0.1:8080) and opens your browser.

snapshot -restore ./mirror --port 3000 --no-open

Options

How it works

1. snapshot fetches pages with async HTTP (httpx), rewrites links to local paths, and downloads linked assets in parallel.
2. A .snapshot.json manifest is written to the output folder with metadata for restore.
3. snapshot -restore serves the saved files and maps / back to the original root page.

Output layout

mirror/
  .snapshot.json
  example.com/
    index.html
    about/
      index.html
    _assets/
      ...

License

MIT