sitewalker

Crawl a website and create a structured map of its pages.

Requirements

Python 3.11 or later

Installation

pipx install sitewalker

Usage

# Map all pages on a site (single-level crawl)
sitewalker example.com

# Recursive crawl of all internal pages
sitewalker example.com -r

# Collect external links
sitewalker example.com -e

# Collect external links and check their HTTP status
sitewalker example.com -e --check-external

# Recursive crawl with external link collection
sitewalker example.com -r -e

# Only crawl web pages (skip images, PDFs, etc.)
sitewalker example.com -r -p

# Crawl an HTTP-only site (e.g., LAN staging server)
sitewalker http://staging.lan --allow-private

# Verbose output for debugging
sitewalker example.com -r -v

The target accepts a bare domain (example.com) or a full URL (http://example.com). Bare domains default to HTTPS — if the connection fails, sitewalker exits with a message to provide the full URL.

Options

Flag	Description	Default
`-r`, `--recursive`	Recursively crawl internal links	Off
`-e`, `--external-links`	Collect external links	Off
`--check-external`	Check HTTP status of external links (requires `-e`)	Off
`-p`, `--pages`	Only crawl web pages (HTML, PHP, etc.)	Off
`-v`, `--verbose`	Enable verbose/debug output	Off
`-t`, `--timeout`	Request timeout in seconds	30
`--max-pages`	Maximum number of pages to crawl	1000
`--max-depth`	Maximum link distance from start URL (BFS)	10
`--max-external-links`	Maximum external links to check with `--check-external`	500
`--domain-delay`	Minimum seconds between requests to the same external domain	5.0
`--delay`	Delay between requests in seconds (use 0 for local)	1.0
`--allow-private`	Allow crawling domains that resolve to private IPs	Off
`--ignore-robots`	Ignore robots.txt rules	Off

Output

Results are saved to a CSV file named {domain}_{timestamp}.csv with columns:

URL — the page URL
Title — the page's <title> tag content
Status Code — HTTP response status

When using -e, external links are additionally saved to {domain}_{timestamp}_external_links.csv. The internal pages CSV is always generated. With --check-external, the external links CSV includes a Status Code column.

Security

SSRF protection: Domains that resolve to private/reserved IP addresses are blocked by default. Use --allow-private to override for legitimate internal use.
robots.txt: Respected by default. Use --ignore-robots to override.
CSV injection: Output values are sanitized to prevent spreadsheet formula injection.
Crawl limits: Recursive crawls are bounded by --max-pages and --max-depth to prevent resource exhaustion.

Roadmap

--format json — JSON output format
--images --check-alt — image inventory with alt text auditing

License

MIT

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sitewalker-0.3.2.tar.gz (8.7 kB view details)

Uploaded Aug 30, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

sitewalker-0.3.2-py3-none-any.whl (10.6 kB view details)

Uploaded Aug 30, 2026 Python 3

File details

Details for the file sitewalker-0.3.2.tar.gz.

File metadata

Download URL: sitewalker-0.3.2.tar.gz
Upload date: Aug 30, 2026
Size: 8.7 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for sitewalker-0.3.2.tar.gz
Algorithm	Hash digest
SHA256	`b80fe9e3835b0a2f92ed2dfcd016e7f436b8f58ec58c7759915f01dbd296b36f`
MD5	`940cdbb6293010f792fbb2cd0b1c3b2e`
BLAKE2b-256	`1c36dbf2fa87d1d8763d3b09e6a318fcbead49d1686138b04287add4f5d99685`

See more details on using hashes here.

Provenance

The following attestation bundles were made for sitewalker-0.3.2.tar.gz:

Publisher: publish.yml on cadentdev/sitewalker

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: sitewalker-0.3.2.tar.gz
- Subject digest: b80fe9e3835b0a2f92ed2dfcd016e7f436b8f58ec58c7759915f01dbd296b36f
- Sigstore transparency entry: 2647383846
- Sigstore integration time: Aug 30, 2026
Source repository:
- Permalink: cadentdev/sitewalker@5bb272632d1ee7b86b930c45621b15c40c2cbf4f
- Branch / Tag: refs/heads/main
- Owner: https://github.com/cadentdev
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: publish.yml@5bb272632d1ee7b86b930c45621b15c40c2cbf4f
- Trigger Event: workflow_dispatch