wayparam

wayparam is a modern, cross-platform CLI tool to fetch historical URLs from public web archive indexes, filter out “boring” URLs (static assets), and normalize query parameters so you can focus on endpoints that actually matter. Wayback remains the default source, with Common Crawl available as an opt-in provider.

This project is inspired by ParamSpider (same overall goal, completely rewritten with a more robust architecture, modern async I/O, better filtering, and production-friendly output behavior).

OSINT tool: wayparam does not crawl targets. It queries public archive indexes (Wayback and, optionally, Common Crawl).

Convert this example.com into something like this:

...
http://www.example.com/_next/image?q=FUZZ&url=FUZZ&w=FUZZ
https://www.example.com/_Incapsula_Resource?SWJIYLWA=FUZZ
http://www.example.com/?format=FUZZ&retailerId=FUZZ
...

Key features

• Multi-source archive collection: Wayback CDX (default), Common Crawl CDXJ, or both
• Opt-in source provenance: preserve the same normalized endpoint once per archive source in JSONL
• Async + concurrency for speed on multiple domains
• Rate limiting (--rps) to be polite with Wayback/CDX
• Retry + backoff and clearer error messages
• Lossless CDX pagination with automatic block-mode fallback
• Historical intelligence: first/last seen, capture counts, status/MIME distributions, parameter prevalence
• Filters “boring” URLs by:
  • extension blacklist/whitelist
  • optional path regex exclusion
• Canonicalization & normalization
  • drop fragments
  • normalize host/ports
  • sort parameters
  • mask parameter values (default placeholder: FUZZ)
  • optional tracking parameter removal (utm_*, gclid, fbclid, …)
• Output:
  • per-domain files (default)
  • stdout streaming for pipelines (--stdout)
  • txt or jsonl output (--format)

Installation

BlackArch Linux

wayparam is officially available in the BlackArch Linux repository.

sudo pacman -S wayparam

On an existing Arch Linux installation with the BlackArch repository enabled, the same package can be installed directly with pacman.

From PyPI (recommended)

pipx install wayparam     # isolated, keeps the CLI on your PATH
# or
pip install wayparam

Snap (Linux)

sudo snap install wayparam

The snap is strictly confined: it can write only inside your home directory, so run it from a directory under $HOME (or point -o/--outdir there). To also write to mounted media: sudo snap connect wayparam:removable-media.

apt (Debian/Ubuntu/Kali)

A .deb is attached to each GitHub release and installs the wayparam(1) manpage alongside the CLI:

sudo apt install ./wayparam_<version>_all.deb

An apt repository (Launchpad PPA) is not published yet. The Debian packaging lives in debian/; see Packaging & distribution for how it is built and what it would take to publish one.

Web interface (optional)

The interface is a separate, opt-in component: the CLI never depends on it.

sudo apt install wayparam-gui      # separate package; pulls in wayparam
wayparam-gui                       # or: snap run wayparam.gui

It prints a URL containing a one-time token and opens it in your browser:

wayparam UI: http://127.0.0.1:8765/?t=<token>

It binds to loopback only (127.0.0.1 by default), requires that token on every request, and rejects unexpected Host headers — it performs outbound requests on behalf of whoever can reach it, so it is deliberately not reachable from the network. To use it on a remote box, forward the port instead:

ssh -L 8765:127.0.0.1:8765 user@host   # then run wayparam-gui there

Inside the snap, open the printed URL yourself: a confined snap cannot launch the host browser.

From source

python -m venv .venv
# Windows: .venv\Scripts\activate
# macOS/Linux: source .venv/bin/activate
python -m pip install -U pip
pip install -e .

Development install (tests + lint)

pip install -e ".[dev]"

Quick start

1) Single domain (writes to results/)

wayparam -d example.com

2) List of domains

wayparam -l domains.txt

3) Stream to stdout (for piping), no files

wayparam -d example.com --stdout --no-files

4) JSONL output (great for tooling)

wayparam -d example.com --stdout --no-files --format jsonl

5) Include subdomains + be polite to Wayback

wayparam -d example.com --include-subdomains --rps 1 --concurrency 2

6) Customize filtering (extensions + path regex)

wayparam -d example.com --ext-blacklist ".png,.jpg,.css,.js" --exclude-path-regex "^/static/"

7) Query Common Crawl instead of Wayback

wayparam -d example.com --source commoncrawl

8) Combine Wayback + Common Crawl

wayparam -d example.com --source wayback,commoncrawl

The source order is the priority order. Results are deduplicated across providers; in JSONL, source records the provider that found a normalized URL first.

9) Preserve complete source provenance

wayparam -d example.com --source wayback,commoncrawl --provenance --format jsonl --stdout --no-files

By default, the first configured provider wins when the same normalized URL appears in several archives. With --provenance, Wayparam keeps one JSONL record per source while still deduplicating repeated variants inside each source. This mode is intentionally JSONL-only because TXT output cannot carry provenance.

10) Compare archive coverage

wayparam -d example.com --source wayback,commoncrawl --source-summary --format jsonl --stdout --no-files

--source-summary reports the normalized-URL union, per-source counts, source-exclusive counts, URLs shared by at least two sources, and the intersection present in every selected source. It requires at least two archive sources. When --max-results is used, the same cap is applied independently to each selected source, preventing an earlier provider from exhausting the whole comparison budget. A capped summary is still marked incomplete and should be treated as a bounded sample, not complete archive coverage.

11) Historical endpoint intelligence

wayparam -d example.com --history --stdout --no-files --format jsonl

12) Parameter history or a domain summary

wayparam -d example.com --params
wayparam -d example.com --summary --format jsonl

Historical modes retrieve capture metadata and automatically disable CDX collapse so that first/last seen and capture counts are meaningful. They can therefore transfer substantially more data than a normal URL run; use --from, --to, filters or --max-results to bound the analysis.

How it works (under the hood)

1. Input parsing

   • -d/--domain for a single host
   • -l/--list for multiple hosts (one per line, supports comments and basic normalization)

2. Query configured archive providers

   • Wayback remains the default and keeps its lossless auto pagination behavior
   • Common Crawl resolves latest from collinfo.json or uses explicitly pinned crawl IDs
   • Common Crawl walks CDXJ ZipNum pages and serializes/rate-limits requests by default
   • matchType=host is used by default, or matchType=domain with --include-subdomains
   • When several providers are selected, normalized URLs are deduplicated across them in source order

3. Filter “boring” URLs

   • Drops URLs that look like static assets (by extension), with optional whitelist mode
   • Optional regex filters can exclude paths (e.g., /static/, /assets/, …)

4. Canonicalize + normalize

   • Removes fragments (#...)

   • Normalizes default ports (:80, :443)

   • Parses query string and:

     • replaces values with a placeholder (default FUZZ)
     • optionally drops tracking parameters
     • sorts parameters for stable output

   • Deduplicates results

5. Output

   • By default writes per-domain results into results/
   • --stdout streams machine-readable output
   • Diagnostics (hints, logs, stats) go to stderr (safe for pipelines)

Output behavior (important for pipelines)

• stdout: only results (URLs or JSONL) when --stdout is enabled
• stderr: logs, errors, hints (VPN/proxy), optional stats, and a live progress line — the progress line is drawn only when stderr is a terminal, so redirecting or piping stderr stays clean

This means you can safely do:

wayparam -d example.com --stdout --no-files | sort -u > urls.txt

Common options

Archive sources

• --source wayback (default)
• --source commoncrawl
• --source wayback,commoncrawl (ordered priority)
• --provenance --format jsonl to keep one normalized record per selected archive source
• --source-summary to compare union, overlap and source-exclusive normalized URLs
• --cc-index latest or a crawl such as CC-MAIN-2026-39 (repeatable)
• --cc-page-size 5 (compressed Common Crawl index blocks per page)
• --cc-rps 1 (Common Crawl request rate; default intentionally conservative)
• --cc-filter status:200 (repeatable Common Crawl-native CDXJ filter)

See Archive sources for provider behavior and caveats.

Wayback/CDX

• --include-subdomains
• --from 2019 / --to 2021 (or full timestamps like 20190101000000)
• --filter statuscode:200 (repeatable)
• --no-collapse (more duplicates, more data)
• --pagination auto|blocks|resume — how to walk a multi-page result. The default auto is lossless: the CDX resumeKey walk drops one URL at each page boundary while collapse is on, so wayparam probes with one request and switches to the block API only when the result actually spans pages.
• --block-size 100 (CDX index blocks per request in block mode)

Historical analysis

• --history — aggregate first/last seen, capture counts, status codes and MIME types per normalized URL
• --params — aggregate endpoint/capture prevalence per query-parameter name
• --summary — one historical summary record per domain

These views are mutually exclusive. In historical modes, --max-results caps accepted capture rows before aggregation.

Normalization

• --placeholder X
• --keep-values (not recommended if you share logs)
• --drop-tracking / --no-drop-tracking
• --all-urls (include URLs without query parameters)

Filtering

• --ext-blacklist ".png,.jpg,.css,.js"
• --ext-whitelist ".php,.asp,.aspx"
• --exclude-path-regex "regex" (repeatable)

Performance / network

• --max-results 500 (global cap on emitted URLs; --limit / --page-size only control the CDX page size)
• --concurrency 8
• --rps 1 (recommended when using VPNs / noisy networks)
• --timeout 30
• --retries 4
• --proxy http://127.0.0.1:8080

Troubleshooting: VPN / Proxy issues (Wayback CDX)

If you see errors like “failed after retries” against the CDX endpoint, it often means:

• the VPN/proxy exit node is blocked or rate-limited by Wayback
• your VPN does TLS filtering or networking policies that break automated requests

Try:

• disconnecting VPN/proxy and rerunning
• switching to a different VPN server
• lowering --concurrency and setting --rps 1

wayparam will print a human-readable hint in English to stderr when it detects this pattern.

Man page

A manual page is included:

man ./man/wayparam.1

Testing

Install dev dependencies and run:

pip install -e ".[dev]"
pytest -q

The test suite includes httpx-level integration tests using httpx.MockTransport (no network).

License

wayparam is free software released under the GNU General Public License v3 (GPLv3). See the LICENSE file for details.

Acknowledgements

• Inspired by ParamSpider (same objective: fetch Wayback URLs, filter noise, focus on parameterized endpoints).
• Thanks to the OSINT / security community for patterns and workflows around URL collection and parameter discovery.

Disclaimer

Use responsibly and lawfully. This tool queries public web archive indexes and does not actively scan targets, but your downstream usage of collected URLs may have legal and ethical implications depending on context.