CrawlerX – Advanced Web Reconnaissance Crawler

CrawlerX is an advanced, multi-threaded web reconnaissance crawler built for security researchers, bug bounty hunters, and penetration testers. It focuses on deep endpoint discovery, GET/POST parameter extraction, API detection, resource categorization, and intelligent URL validation, with optional fuzzing and common path probing.

✨ Developed by @IMApurbo 🛡️ Use only on systems you own or have explicit permission to test.

---

🚀 Core Capabilities

🔍 Intelligent Crawling

• Discovers HTML, JavaScript, CSS, JSON, and XML endpoints

• Extracts URLs from:

  • HTML attributes
  • Inline JavaScript
  • Event handlers
  • CSS files
  • JSON / API responses

• Strong filtering to eliminate JavaScript noise & false positives

---

🔗 Endpoint Discovery

• All endpoints
• Parameterized URLs
• Non-parameterized URLs
• Automatic domain + subdomain validation

Saved under:

endpoints/
├── all_endpoints.txt
├── all_endpoints.json
├── parameterized.txt
├── parameterized.json
├── non_parameterized.txt

---

🧪 GET & POST Parameter Extraction

• Extracts parameters from HTML forms

• Generates realistic default values

• Saves:

  • Parsed parameters (JSON)
  • Raw .req files (Burp-ready)

get/
├── get_urls.txt
├── get_params.json
├── *.req

post/
├── post_urls.txt
├── post_params.json
├── *.req

---

⚙️ API Endpoint Detection

Detects common API patterns:

• /api/
• /v1/, /v2/
• /rest/
• /graphql
• .json, .xml

api/
├── api_endpoints.txt
├── api_endpoints.json

---

📁 Resource Categorization

Automatically classifies discovered resources:

• Images
• JavaScript
• Stylesheets
• Fonts
• Media
• Documents
• Other

resources/
├── images.txt / images.json
├── scripts.txt / scripts.json
├── stylesheets.txt / stylesheets.json
├── fonts.txt / fonts.json
├── media.txt / media.json
├── documents.txt / documents.json
├── other.txt / other.json

---

🌳 Site Structure Mapping

Optional ASCII tree view of the entire site:

example.com
├── login
├── dashboard
│   ├── profile
│   └── settings
└── api
    └── v1

Saved to:

structure/structure.txt

---

🧠 Smart Features

• Robots.txt support (optional)
• Resume interrupted crawls using pickle state
• Incremental auto-save
• Graceful Ctrl+C handling
• Parameter fuzzing (numeric params)
• Common path probing (admin, api, backup, .env, etc.)
• Retry logic with backoff
• Verbose mode with interesting parameter highlighting

---

📦 Installation

pip install crawlerx

---

🧑‍💻 Usage

crawlerx -u <url> [options]

---

🧾 Command-Line Options

Flag	Description	Default
-u, --url	Target URL (required)	—
-o, --output	Output directory	None
--threads	Concurrent threads (1–20)	5
--depth	Crawl depth	2
--delay	Delay between requests	0.1s
--timeout	Request timeout	10s
--ua	Custom User-Agent	Browser UA
-H, --headers	Custom headers (Key:Value;)	None
--proxy	HTTP/HTTPS proxy	None
--exclude	Excluded extensions	None
--sub	Include subdomains	False
--structure	Generate site structure	False
--respect-robots	Respect robots.txt	False
--fuzz-params	Fuzz numeric parameters	False
--common-paths	Probe common paths	False
--cont	Resume from crawl state	None
--verbose	Verbose logging	False

---

🧪 Examples

Basic Crawl

crawlerx -u https://example.com

Save Output

crawlerx -u https://example.com -o results

Deep Crawl with Threads

crawlerx -u https://example.com --depth 4 --threads 10

Enable Fuzzing & Common Paths

crawlerx -u https://example.com --fuzz-params --common-paths

Resume Interrupted Crawl

crawlerx -u https://example.com --cont results/crawlerx_example.com/crawl_state.pkl

Generate Site Structure

crawlerx -u https://example.com --structure

---

📂 Output Directory Layout

crawlerx_<domain>/
├── endpoints/
├── get/
├── post/
├── api/
├── resources/
├── structure/
└── crawl_state.pkl

---

⚠️ Legal Notice

🚨 Authorized use only This tool is intended for legal security testing. Unauthorized scanning may violate laws and ethical guidelines.

---

👨‍💻 Author

• IMApurbo

---

📜 License

Licensed under the MIT License. See the LICENSE file for details.