sitewalker

Crawl a website and create a structured map of its pages.

Requirements

• Python 3.11 or later

Installation

pipx install sitewalker

Usage

# Map all pages on a site (single-level crawl)
sitewalker example.com

# Recursive crawl of all internal pages
sitewalker example.com -r

# Collect external links
sitewalker example.com -e

# Collect external links and check their HTTP status
sitewalker example.com -e --check-external

# Recursive crawl with external link collection
sitewalker example.com -r -e

# Include images, stylesheets, and scripts in the inventory
sitewalker example.com -r -a

# Hunt broken links: non-200s summarized on stdout, with the page to fix
sitewalker example.com -r --broken-only

# Only crawl web pages (skip images, PDFs, etc.)
sitewalker example.com -r -p

# Crawl an HTTP-only site (e.g., LAN staging server)
sitewalker http://staging.lan --allow-private

# Verbose output for debugging
sitewalker example.com -r -v

The target accepts a bare domain (example.com) or a full URL (http://example.com). Bare domains default to HTTPS — if the connection fails, sitewalker exits with a message to provide the full URL.

Options

Configuration

Settings that rarely change can live in a TOML config file at ~/.config/sitewalker/config.toml (or $XDG_CONFIG_HOME/sitewalker/config.toml; override the location with --config, skip it entirely with --no-config). CLI flags always win over config values.

# Where CSVs are written when --output-dir isn't given
output_dir = "~/crawls"

# Replaces the built-in extension set used by -p/--pages.
# Include "" to keep treating extension-less URLs and directories as pages.
page_extensions = ["", "html", "htm", "php", "story"]

Recognized keys: output_dir, page_extensions. Unknown keys are an error, so typos fail loudly instead of being ignored.

Output

Results are saved to a CSV file named {domain}_{timestamp}.csv in the current directory. Use --output-dir to write elsewhere (the directory is created if needed) and --output-filename to replace the generated name:

sitewalker example.com -r -e --output-dir ~/crawls --output-filename acme-audit
# → ~/crawls/acme-audit.csv and ~/crawls/acme-audit_external_links.csv

--output-filename must be a bare name — the directory always comes from --output-dir. Columns:

• URL — the page URL
• Title — the page's <title> tag content
• Status Code — HTTP response status
• Found On — the page where this URL was first discovered (empty for the start URL). For a broken link, this is the page to fix.
• Kind — page (fetched and parsed for links) or asset (recorded only, when -a is used).

URLs discovered beyond --max-depth are included as rows with title skipped: max_depth and an empty status code, so nothing the crawler saw is invisible in the output. Use --broken-only to print a summary of every non-200 URL (with its Found On page) to stdout at the end of the run.

When using -e, external links are additionally saved to {domain}_{timestamp}_external_links.csv. The internal pages CSV is always generated. With --check-external, the external links CSV includes a Status Code column.

Security

• SSRF protection: Domains that resolve to private/reserved IP addresses are blocked by default. Use --allow-private to override for legitimate internal use.
• robots.txt: Respected by default. Use --ignore-robots to override.
• CSV injection: Output values are sanitized to prevent spreadsheet formula injection.
• Crawl limits: Recursive crawls are bounded by --max-pages and --max-depth to prevent resource exhaustion.

Roadmap

• --format json — JSON output format
• --check-alt — alt text auditing for the image inventory (-a covers the inventory itself)

License

MIT