robofinder 0.2.2

Extract historical robots.txt paths via Wayback Machine – a recon tool for bug hunters

Robofinder

 _____       _            __ _           _           
|  __ \     | |          / _(_)         | |          
| |__) |___ | |__   ___ | |_ _ _ __   __| | ___ _ __ 
|  _  // _ \| '_ \ / _ \|  _| | '_ \ / _` |/ _ \ '__|
| | \ \ (_) | |_) | (_) | | | | | | | (_| |  __/ |   
|_|  \_\___/|_.__/ \___/|_| |_|_| |_|\__,_|\___|_|   
                                                     

Uncover hidden endpoints by mining every historical robots.txt snapshot from the Wayback Machine.

---

Why Robofinder?

Sites regularly scrub sensitive paths from robots.txt — but the Wayback Machine keeps every version ever crawled.

Robofinder queries Archive.org's CDX API to pull all historical robots.txt snapshots for a target, deduplicates every Allow, Disallow, and Sitemap directive, and prints the full list. Paths that were quietly removed from production may still be alive and reachable.

Built for: bug bounty recon · OSINT · attack-surface mapping · forgotten endpoint discovery

---

Install

pip install robofinder

Install from source

git clone https://github.com/Spix0r/robofinder
cd robofinder
pip install .

---

Quick start

# Single target
robofinder -u https://example.com

# Full URLs ready to probe
robofinder -u https://example.com -c

# Pipe straight into httpx or nuclei
robofinder -u domains.txt -c -s | httpx -silent -mc 200
robofinder -u https://example.com -c -s | nuclei -t exposures/

---

Usage

robofinder -u URL [options]

Flag	Long form	Default	Description
-u	--url	—	Target URL or file with one URL per line
-o	--output	—	Output to terminal (no value) or save to file (with value)
-f	--format	txt	Output format: txt, json, or both
-c		—	Prefix each path with the target URL (full URLs)
-p		—	Extract URL parameters from historical paths
-t	--threads	1	Number of fetch threads
-r	--rate-limit	2.0	Max requests/sec sent to Archive.org
	--cooldown	10	Seconds to wait between domains
-s	--silent	—	Suppress the banner
	--debug	—	Verbose debug output (goes to stderr)

Scan a list of domains

Create domains.txt:

https://example.com
https://target.org
api.example.com

robofinder -u domains.txt -o all_paths.txt

JSON output with jq

# Save to file
robofinder -u https://example.com -f json -o results

# Pipe to jq
robofinder -u https://example.com -f json | jq '.["example.com"].paths[]'

# Count paths per domain
robofinder -u domains.txt -f json | jq 'to_entries[] | "\(.key): \(.value.count) paths"'

Extract forgotten URL parameters

robofinder -u https://example.com -p
# returns parameter names found in historical paths, e.g.:
# id
# token
# redirect_url

---

Rate limiting

Robofinder ships with a token-bucket rate limiter so it never hammers Archive.org.

• Default: 2 req/s — well within Archive.org's tolerance.
• On HTTP 429: exponential back-off with automatic retry.
• On timeout/connection errors: retries with backoff (up to 7 attempts).
• Override with -r: use -r 0.5 to be conservative or -r 5 on a fast connection.

# Conservative mode
robofinder -u https://example.com -r 1 --cooldown 30

---

Changelog

See CHANGELOG.md.

---

License

MIT © Spix0r