🕷️ EvilSpider

EvilSpider is a fast, asynchronous attack-surface crawler and reconnaissance engine designed for bug bounty hunters, penetration testers, and security researchers.

Unlike traditional generic crawlers (e.g., Scrapy or Crawlee), EvilSpider focuses strictly on offensive security use cases: uncovering hidden endpoints, hunting leaked secrets and API keys, extracting parameterized URLs, fingerprinting technologies, mapping forms, and generating actionable reconnaissance dashboards.

📥 Installation

📦 Via pip

pip install evilspider

⚡ Via pipx (Recommended for isolated CLI)

pipx install evilspider

🛠️ From Source (Development)

git clone https://github.com/Baba01hacker666/evilspider.git
cd evilspider
pip install -e .

🚀 Quick Start & Smart Defaults

EvilSpider uses smart defaults — you don't even need to type crawl! Simply run EvilSpider directly with your target URL or flags:

1. Instant Recon Crawl (Smart Default)

evilspider https://example.com
# or with flags:
evilspider -u https://example.com -d 3

2. Multi-Target Crawl from File (Save findings to JSON)

evilspider -l targets.txt -t 20 -o results.json

3. Pipeline URLs to Nuclei / FFUF / HTTPX

evilspider https://example.com --urls-only | nuclei -t http/cves/

4. Generate Interactive Dark-Mode HTML Dashboard

evilspider https://example.com --report-html report.html --report-md report.md

💡 Note on Output Saving: EvilSpider keeps your filesystem clean. It only saves files to disk when you explicitly tell it to (using -o, --output, --csv, --report-html, --report-md, etc.). Otherwise, findings are streamed cleanly in the terminal.

⚡ Key Features

• 🧠 Smart Defaults: No boilerplate required. Run evilspider https://example.com directly without typing crawl.
• 🛡️ WAF & Anti-Bot Hunter (25+ Protections): Automatic signature detection for Cloudflare, AWS WAF, Akamai Kona / Bot Manager, Imperva Incapsula, Fastly Signal Sciences, F5 BIG-IP ASM, ModSecurity / OWASP CRS, Sucuri CloudProxy, Barracuda, Fortinet FortiWeb, Datadome, PerimeterX, and more. Features adaptive evasion & backoff to prevent IP bans.
• ⚡ Smart SPA & JS-Heavy App Engine: Automatically detects Single Page Applications (React, Vue, Angular, Next.js, Nuxt, Svelte, Vite) and performs multi-layer JS decoding & deobfuscation:
  • Decodes Base64-encoded hidden API endpoints and secret tokens
  • Unescapes Hex (\x..), Unicode (\u...., \u{...}), and URL-encoded strings
  • Deobfuscates JS Obfuscator string array tables (e.g. _0x1234 = ['/api/...', ...])
  • Mines Client-side Routes (React Router <Route path="...">, Vue Router, Angular Router, Next.js dynamic routes)
  • Mines Webpack & Vite dynamic chunk bundles and fetches/analyzes them recursively
  • Extracts embedded GraphQL queries & mutations
• 💾 Clean Output Saving: Only writes to disk when explicitly asked via -o / --output or report flags.
• 🚀 Async Concurrency Engine: Ultra-fast asynchronous crawler built on asyncio and aiohttp with connection pooling, rate limiting, and configurable worker threads.
• 🚨 Secret & Sensitive Data Hunter: Built-in regex rule engine scanning for 30+ types of credentials across responses (AWS keys, Google Cloud API keys, JWT tokens, Stripe keys, GitHub PATs, Slack webhooks, Private keys, Database URIs, and leaked authorization headers).
• 📝 HTML Form & Input Extractor: Detects and categorizes HTML forms (Login, Registration, File Uploads, Password Reset, Search) along with hidden CSRF tokens and field names.
• 🛠️ Tech Stack Fingerprinter: Detects CMS (WordPress, Drupal, Joomla, Shopify), Backend Frameworks (Django, Laravel, Spring Boot, Express, Rails, FastAPI), Web Servers (Nginx, Apache, Caddy, Cloudflare, IIS), and Frontend Frameworks.
• 🛡️ Security Headers & Reflection Auditor: Identifies missing security headers (CSP, HSTS, X-Frame-Options, X-Content-Type-Options), dangerous CORS misconfigurations (* with credentials), and reflected URL parameters (potential XSS sinks).
• 📖 API Documentation & Schema Parsing: Automatically parses Swagger 2.0 / OpenAPI 3.x specifications and GraphQL schemas to extract all declared endpoints, parameters, and routes.
• 🎯 Active Sensitive File Prober (probe): Fast active probing module hunting for VCS leaks (.git/HEAD, .svn), exposed configs (.env, web.config), backup archives (dump.sql, backup.zip), and debug endpoints (actuator/health, pprof) with smart soft-404 detection.
• 🌐 Scope & Filter Controls: Scope by domain, subdomains, regex whitelist/blacklist, extension filters, path exclusions, status codes, and loop prevention algorithms.
• 🎭 Browser Impersonation Profiles: Modern User-Agent and Client Hints / header profiles for Chrome, Firefox, Edge, Safari, Chrome Android, and Safari iOS.
• 📊 Multi-Format Reporting: Export findings to JSON, JSON Lines (NDJSON), CSV, Plain text URLs, Parameter wordlists, Markdown reports, and a standalone responsive Interactive Dark-Mode HTML Dashboard.

🔥 Recon Workflows

🕵️ 1) Full Hunter Mode (Secrets + Uploads + Sensitive Probing)

evilspider crawl \
  -u https://target.tld \
  --robots --sitemaps \
  --detect-uploads \
  --probe-sensitive \
  -d 4 -t 15 \
  --report-html report.html

🔑 2) Sensitive Endpoint & Backup Prober (probe)

evilspider probe \
  -u https://target.tld \
  --categories vcs,env_config,backups,api_docs,actuators_debug \
  -o probe_findings.json

📁 3) Offline JavaScript & File Analysis (extract)

evilspider extract -f app.bundle.js -o extracted_assets.json

🔐 4) Authenticated Crawl with Session Cookies & Bearer Token

evilspider crawl \
  -u https://target.tld \
  -C 'session=abc12345; role=admin' \
  --bearer-token 'eyJhbGciOiJIUzI1Ni...' \
  -H 'X-Forwarded-For: 127.0.0.1'

🌐 5) Proxied Crawl through Burp Suite / OWASP ZAP

evilspider crawl \
  -u https://target.tld \
  -x http://127.0.0.1:8080 \
  --insecure \
  --report-redirects

🎯 6) Parameter Wordlist Harvesting for Fuzzing

evilspider crawl \
  -u https://target.tld \
  --params-file params_wordlist.txt \
  --urls-only > discovered_urls.txt

# Pipe discovered parameters into ffuf
ffuf -u "https://target.tld/endpoint?FUZZ=1" -w params_wordlist.txt

🛠️ CLI Options Reference

evilspider crawl (Main Crawler)

📊 Interactive HTML Dashboard

EvilSpider generates a standalone, self-contained HTML report with:

• Summary metric cards (Endpoints, Secrets, Forms, Subdomains, Technologies, Security Issues)
• Dynamic table filtering and instant keyword search
• Dedicated tabs for Secrets, Forms, Technologies, Security Headers, API Docs, and Subdomains
• Zero external CDN dependencies (works completely offline)

🏗️ Architecture

evilspider/
├── analyzer/
│   ├── waf.py            # 25+ WAF & Anti-Bot signature detector & adaptive evader
│   ├── js_analyzer.py    # SPA detector, Base64/Hex/Unicode decoder & client route miner
│   ├── secrets.py        # 30+ regex rules for API keys, tokens & credentials
│   ├── tech.py           # Web server, framework & CMS fingerprinter
│   ├── security.py       # Security headers, CORS & reflected param auditor
│   └── api_parser.py     # Swagger 2.0 / OpenAPI 3.x & GraphQL parser
├── extractors/
│   ├── html_extractor.py # Fast HTML parser for links, scripts, frames & comments
│   ├── js_extractor.py   # JavaScript REST endpoint & webpack chunk extractor
│   ├── form_extractor.py # HTML form, input & CSRF token parser
│   └── sourcemap.py      # Source map (.js.map) unminified route extractor
├── prober/
│   ├── probe.py          # Active sensitive file prober with soft-404 detection
│   └── wordlists.py      # Curated recon path lists (VCS, configs, backups, APIs)
├── output/
│   ├── formatter.py      # ANSI terminal formatting, logo & summary tables
│   ├── reporter.py       # Multi-format reporter (JSON, CSV, MD, HTML, TXT)
│   └── templates/
│       └── report_template.html # Responsive dark-mode dashboard
├── tests/                # Comprehensive unit and integration test suite
├── _version.py           # Single source of truth for version and author metadata
├── config.py             # Config parser and CLI argument validator
├── crawler.py            # Async crawl orchestrator and queue worker pool
└── main.py               # CLI entrypoint (crawl, probe, extract, report)

⚖️ Legal Disclaimer

EvilSpider is developed for authorized security assessments, penetration testing, bug bounty reconnaissance, and educational purposes only. Do not use this tool against targets without explicit authorization. The authors assume no liability for misuse.