robofinder 0.1.0

Extract historical robots.txt paths via Wayback Machine – a recon tool for bug hunters

RoboFinder

Uncover hidden endpoints by mining every historical robots.txt snapshot from the Wayback Machine.

---

Why RoboFinder?

Sites regularly scrub sensitive paths from robots.txt — but the Wayback Machine keeps every version ever crawled.

RoboFinder queries Archive.org's CDX API to pull all historical robots.txt snapshots for a target, deduplicates every Allow, Disallow, and Sitemap directive, and prints the full list. Paths that were quietly removed from production may still be alive and reachable.

Built for: bug bounty recon · OSINT · attack-surface mapping · forgotten endpoint discovery

---

Install

pip install robofinder

Install from source

git clone https://github.com/Spix0r/robofinder
cd robofinder
pip install .

---

Quick start

# Scan a single target
robofinder -u https://example.com

# Full URLs ready to probe
robofinder -u https://example.com -c

# Extract forgotten URL parameters
robofinder -u https://example.com -p

# Save results to a file
robofinder -u https://example.com -o results.txt

# Pipe into other tools
robofinder -u https://example.com -c | httpx -silent -mc 200
robofinder -u https://example.com -c | nuclei -t exposures/

---

Usage

robofinder -u <URL> [options]

Flag	Long form	Default	Description
-u	--url	—	Target URL (required)
-o	--output	—	Save results to a file
-t	--threads	10	Number of fetch threads
-c		—	Prefix each path with the target URL
-p		—	Extract URL parameters from historical paths
-s	--silent	—	Suppress the banner
	--debug	—	Verbose debug output

Concatenate paths with the target URL (-c)

Useful for piping directly into scanners:

robofinder -u https://example.com -c
# https://example.com/admin
# https://example.com/api/v1
# https://example.com/internal

Extract forgotten URL parameters (-p)

Finds parameter names that appeared in historical paths:

robofinder -u https://example.com -p
# id
# token
# redirect_url

---

Changelog

See CHANGELOG.md.

---

License

MIT © Spix0r