Robofinder

 _____       _            __ _           _           
|  __ \     | |          / _(_)         | |          
| |__) |___ | |__   ___ | |_ _ _ __   __| | ___ _ __ 
|  _  // _ \| '_ \ / _ \|  _| | '_ \ / _` |/ _ \ '__|
| | \ \ (_) | |_) | (_) | | | | | | | (_| |  __/ |   
|_|  \_\___/|_.__/ \___/|_| |_|_| |_|\__,_|\___|_|   
                                                     

Uncover hidden endpoints by mining every historical robots.txt snapshot from the Wayback Machine.

Why Robofinder?

Sites regularly scrub sensitive paths from robots.txt — but the Wayback Machine keeps every version ever crawled.

Robofinder queries Archive.org's CDX API to pull all historical robots.txt snapshots for a target, deduplicates every Allow, Disallow, and Sitemap directive, and prints the full list. Paths that were quietly removed from production may still be alive and reachable.

Built for: bug bounty recon · OSINT · attack-surface mapping · forgotten endpoint discovery

Install

pip install robofinder

git clone https://github.com/Spix0r/robofinder
cd robofinder
pip install .

Quick start

# Single target
robofinder -u https://example.com

# Full URLs ready to probe
robofinder -u https://example.com -c

# Pipe straight into httpx or nuclei
robofinder -u domains.txt -c -s | httpx -silent -mc 200
robofinder -u https://example.com -c -s | nuclei -t exposures/

Usage

robofinder -u URL [options]

Scan a list of domains

Create domains.txt:

https://example.com
https://target.org
api.example.com

robofinder -u domains.txt -o all_paths.txt

JSON output with jq

# Save to file
robofinder -u https://example.com -f json -o results

# Pipe to jq
robofinder -u https://example.com -f json | jq '.["example.com"].paths[]'

# Count paths per domain
robofinder -u domains.txt -f json | jq 'to_entries[] | "\(.key): \(.value.count) paths"'

Extract forgotten URL parameters

robofinder -u https://example.com -p
# returns parameter names found in historical paths, e.g.:
# id
# token
# redirect_url

Rate limiting

Robofinder ships with a token-bucket rate limiter so it never hammers Archive.org.

• Default: 2 req/s — well within Archive.org's tolerance.
• On HTTP 429: exponential back-off with automatic retry.
• On timeout/connection errors: retries with backoff (up to 7 attempts).
• Override with -r: use -r 0.5 to be conservative or -r 5 on a fast connection.

# Conservative mode
robofinder -u https://example.com -r 1 --cooldown 30

Changelog

See CHANGELOG.md.

License

MIT © Spix0r