JumpingSpider

JumpingSpider (v0.2.1) is a concurrent web crawler and broken link/asset validator with multi-format reporting (.json, .csv, .md, .html) and an interactive Terminal User Interface (TUI).

Features

• Strict Path Boundary Crawling: Recursively crawls HTML pages under the starting URL path while preventing upward or parallel directory escapes.
• Referrer & Anchor Text Tracking: Records every source page (found_on), HTML tag (<a>, <img>, <script>, <link>, <video>, <iframe>), and link/alt text for every discovered URL so you know exactly which file to update.
• External & Sibling Link Validation: Validates links outside the crawl boundary without crawling into external sites.
• Smart HEAD & Streaming Requests: Uses HEAD requests (with automatic streaming GET fallback on 405/403) for documents (.pdf, .pptx, .xlsx, .mp4, etc.), static assets, and external links to avoid downloading large binary files into memory.
• Redirect & Error Diagnostics: Captures full redirect chains (301/302 -> final URL) and categorizes failures (Timeout, SSLError, ConnectionError, HTTPError).
• Multi-Format Reports: Export results to JSON, CSV, Markdown, or styled HTML.
• Interactive TUI (--tui): Built with textual for live crawl monitoring, filtering (All, Broken, Redirects, External, Assets/Docs), referrer inspection, and one-key report exports.

Installation

Requires Python 3.13+.

From PyPI (Recommended)

# Install globally as an isolated CLI tool with uv
uv tool install JumpingSpider

# Or run directly without installing using uvx
uvx JumpingSpider https://example.com/docs/

# Or install with pip / pipx
pip install JumpingSpider

From Source (Development)

Requires uv:

git clone https://github.com/Hailstone-Labs/JumpingSpider.git
cd JumpingSpider
uv sync --all-groups

Usage

CLI Mode

# Basic crawl (outputs crawl_results.json and prints a terminal summary scorecard)
uv run JumpingSpider https://weather.ndc.nasa.gov/tempo/

# Export multiple report formats at once
uv run JumpingSpider https://weather.ndc.nasa.gov/tempo/ \
  -o tempo_results.json \
  -o tempo_broken.csv \
  -o tempo_report.md \
  -o tempo_report.html

# Export only broken links to CSV
uv run JumpingSpider https://weather.ndc.nasa.gov/tempo/ -o broken_links.csv --broken-only

# Customize concurrency, rate limit delay, and depth
uv run JumpingSpider https://example.com/docs/ -w 8 -d 0.1 --max-depth 3

# Only check internal links and skip page assets
uv run JumpingSpider https://example.com/docs/ --no-external --no-assets

CLI Options

Interactive TUI Mode

uv run JumpingSpider --tui https://weather.ndc.nasa.gov/tempo/

TUI Keyboard Shortcuts:

• Ctrl+S: Start crawl
• Ctrl+X: Stop crawl
• 1–5: Filter table (1 All, 2 Broken, 3 Redirects, 4 External, 5 Assets/Docs)
• e: Export all report formats (crawl_results.{json,csv,md,html})
• q: Quit

Development & Tests

# Run unit tests
uv run pytest -v

# Run Ruff linter
uv run ruff check .

Contributing

Contributions are welcome! Please read our Contributing Guide and Code of Conduct before submitting a pull request. For security issues, please refer to our Security Policy.

License

Distributed under the MIT License. See LICENSE for full details.