🕷️ EvilSpider

EvilSpider is a fast, asynchronous attack-surface crawler and reconnaissance engine designed for bug bounty hunters, penetration testers, and security researchers.

Unlike traditional generic crawlers (e.g., Scrapy or Crawlee), EvilSpider focuses strictly on offensive security use cases: uncovering hidden endpoints, hunting leaked secrets and API keys, extracting parameterized URLs, fingerprinting technologies, mapping forms, and generating actionable reconnaissance dashboards.

📥 Installation

📦 Via pip

pip install evilspider

⚡ Via pipx (Recommended for isolated CLI)

pipx install evilspider

🛠️ From Source (Development)

git clone https://github.com/Baba01hacker666/evilspider.git
cd evilspider
pip install -e .

🚀 Quick Start

1. Standard Recon Crawl

evilspider crawl -u https://example.com

2. Multi-Target Crawl from File

evilspider crawl -l targets.txt -t 20 -o results.json

3. Pipeline URLs to Nuclei / FFUF / HTTPX

evilspider crawl -u https://example.com --urls-only | nuclei -t http/cves/

4. Generate Interactive Dark-Mode HTML Dashboard

evilspider crawl -u https://example.com --report-html report.html --report-md report.md

⚡ Key Features

• 🚀 Async Concurrency Engine: Ultra-fast asynchronous crawler built on asyncio and aiohttp with connection pooling, rate limiting, and configurable worker threads.
• 🚨 Secret & Sensitive Data Hunter: Built-in regex rule engine scanning for 30+ types of credentials across responses (AWS keys, Google Cloud API keys, JWT tokens, Stripe keys, GitHub PATs, Slack webhooks, Private keys, Database URIs, and leaked authorization headers).
• 📝 HTML Form & Input Extractor: Detects and categorizes HTML forms (Login, Registration, File Uploads, Password Reset, Search) along with hidden CSRF tokens and field names.
• 📜 Deep JavaScript Analysis: Extracts REST API routes, dynamic endpoints, AJAX (fetch/axios/$.ajax) targets, Webpack chunks, and Source Map (.js.map) references from inline scripts and external JS files.
• 🛠️ Tech Stack Fingerprinter: Detects CMS (WordPress, Drupal, Joomla, Shopify), Backend Frameworks (Django, Laravel, Spring Boot, Express, Rails, FastAPI), Web Servers (Nginx, Apache, Caddy, Cloudflare, IIS), and Frontend Frameworks (React, Vue, Next.js, Nuxt, Angular).
• 🛡️ Security Headers & Reflection Auditor: Identifies missing security headers (CSP, HSTS, X-Frame-Options, X-Content-Type-Options), dangerous CORS misconfigurations (* with credentials), and reflected URL parameters (potential XSS sinks).
• 📖 API Documentation & Schema Parsing: Automatically parses Swagger 2.0 / OpenAPI 3.x specifications and GraphQL schemas to extract all declared endpoints, parameters, and routes.
• 🎯 Active Sensitive File Prober (probe): Fast active probing module hunting for VCS leaks (.git/HEAD, .svn), exposed configs (.env, web.config), backup archives (dump.sql, backup.zip), and debug endpoints (actuator/health, pprof) with smart soft-404 detection.
• 🌐 Scope & Filter Controls: Scope by domain, subdomains, regex whitelist/blacklist, extension filters, path exclusions, status codes, and loop prevention algorithms.
• 🎭 Browser Impersonation Profiles: Modern User-Agent and Client Hints / header profiles for Chrome, Firefox, Edge, Safari, Chrome Android, and Safari iOS.
• 📊 Multi-Format Reporting: Export findings to JSON, JSON Lines (NDJSON), CSV, Plain text URLs, Parameter wordlists, Markdown reports, and a standalone responsive Interactive Dark-Mode HTML Dashboard.
• 🔌 UNIX Pipeline Ready: --urls-only and --silent modes designed for piping directly into tools like nuclei, ffuf, httpx, sqlmap, or dalfox.

🔥 Recon Workflows

🕵️ 1) Full Hunter Mode (Secrets + Uploads + Sensitive Probing)

evilspider crawl \
  -u https://target.tld \
  --robots --sitemaps \
  --detect-uploads \
  --probe-sensitive \
  -d 4 -t 15 \
  --report-html report.html

🔑 2) Sensitive Endpoint & Backup Prober (probe)

evilspider probe \
  -u https://target.tld \
  --categories vcs,env_config,backups,api_docs,actuators_debug \
  -o probe_findings.json

📁 3) Offline JavaScript & File Analysis (extract)

evilspider extract -f app.bundle.js -o extracted_assets.json

🔐 4) Authenticated Crawl with Session Cookies & Bearer Token

evilspider crawl \
  -u https://target.tld \
  -C 'session=abc12345; role=admin' \
  --bearer-token 'eyJhbGciOiJIUzI1Ni...' \
  -H 'X-Forwarded-For: 127.0.0.1'

🌐 5) Proxied Crawl through Burp Suite / OWASP ZAP

evilspider crawl \
  -u https://target.tld \
  -x http://127.0.0.1:8080 \
  --insecure \
  --report-redirects

🎯 6) Parameter Wordlist Harvesting for Fuzzing

evilspider crawl \
  -u https://target.tld \
  --params-file params_wordlist.txt \
  --urls-only > discovered_urls.txt

# Pipe discovered parameters into ffuf
ffuf -u "https://target.tld/endpoint?FUZZ=1" -w params_wordlist.txt

🛠️ CLI Options Reference

evilspider crawl (Main Crawler)

📊 Interactive HTML Dashboard

EvilSpider generates a standalone, self-contained HTML report with:

• Summary metric cards (Endpoints, Secrets, Forms, Subdomains, Technologies, Security Issues)
• Dynamic table filtering and instant keyword search
• Dedicated tabs for Secrets, Forms, Technologies, Security Headers, API Docs, and Subdomains
• Zero external CDN dependencies (works completely offline)

🏗️ Architecture

evilspider/
├── analyzer/
│   ├── secrets.py        # 30+ regex rules for API keys, tokens & credentials
│   ├── tech.py           # Web server, framework & CMS fingerprinter
│   ├── security.py       # Security headers, CORS & reflected param auditor
│   └── api_parser.py     # Swagger 2.0 / OpenAPI 3.x & GraphQL parser
├── extractors/
│   ├── html_extractor.py # Fast HTML parser for links, scripts, frames & comments
│   ├── js_extractor.py   # JavaScript REST endpoint & webpack chunk extractor
│   ├── form_extractor.py # HTML form, input & CSRF token parser
│   └── sourcemap.py      # Source map (.js.map) unminified route extractor
├── prober/
│   ├── probe.py          # Active sensitive file prober with soft-404 detection
│   └── wordlists.py      # Curated recon path lists (VCS, configs, backups, APIs)
├── output/
│   ├── formatter.py      # ANSI terminal formatting, logo & summary tables
│   ├── reporter.py       # Multi-format reporter (JSON, CSV, MD, HTML, TXT)
│   └── templates/
│       └── report_template.html # Responsive dark-mode dashboard
├── tests/                # Comprehensive unit and integration test suite
├── _version.py           # Single source of truth for version and author metadata
├── config.py             # Config parser and CLI argument validator
├── crawler.py            # Async crawl orchestrator and queue worker pool
└── main.py               # CLI entrypoint (crawl, probe, extract, report)

⚖️ Legal Disclaimer

EvilSpider is developed for authorized security assessments, penetration testing, bug bounty reconnaissance, and educational purposes only. Do not use this tool against targets without explicit authorization. The authors assume no liability for misuse.