Piano

A dark-web search aggregator for defensive OSINT and threat-intelligence research.

Piano queries existing .onion search indexes (Torch, TorDex, Ahmia, and others) through Tor, extracts identifiers (cryptocurrency wallets, PGP keys, emails, CVE references, onion addresses), and presents deduplicated, BM25-ranked results through a CLI, a REST API, and an MCP server. An optional web UI provides browser-based access over HTTPS for LAN use.

The project is inspired by patterns proven in the IntelShed WorldBase darkweb_* modules and reimplements them as a standalone codebase without a hard dependency on WorldBase.

Status

Early-stage research tool. The architecture is designed to fail gracefully: a down Tor proxy, an unreachable engine, or a missing LLM API key degrades to reduced functionality rather than crashing. However, the tool has not been audited for production use, and the engine addresses may rotate without notice.

Scope and limitations

• Search-first: The primary surface queries existing search indexes. Broad spidering of arbitrary .onion domains is opt-in and depth-limited.
• Text-only: Binary MIME types are dropped at the HTTP layer.
• No PII resale: Extracted identifiers are used for local correlation only.
• Single operator: No multi-tenancy, no authentication beyond an optional API key. The HTTPS UI is intended for LAN use by a single operator.
• Engine volatility: .onion addresses change frequently. Engine addresses were verified against curated sources (edoardottt/awesome-hacker-search-engines, ahmia.fi) at the time of writing; unverified or dead engines are marked deprecated and skipped by the circuit breaker.

Architecture

Surface:  CLI (dnc_cli.py)  ·  REST API (dnc/api.py)  ·  MCP server (mcp_dnc/)  ·  Web UI (frontend/)
Intelligence:  entity_extractor  ·  ranker (BM25)  ·  correlator  ·  llm_enricher  ·  actor_profiler  ·  forensics
Acquisition:  engine_registry  ·  search_orchestrator  ·  tor_proxy  ·  tor_rotator  ·  html_parsers  ·  fetcher  ·  cache

The MCP server uses a Tool Bus pattern: 25 internal functions are compressed into 6 semantic meta-tools, reducing the tools/list payload by approximately 58% and avoiding client-side timeouts on cold start.

Quick start

Option A — Docker (recommended, full standalone)

git clone https://github.com/sookoothaii/DarkNetCrawler.git
cd DarkNetCrawler

cp .env.example .env
# Edit .env: set DNC_ENABLED=1
# For LLM enrichment: DNC_LLM_ENRICH=1, DNC_NVIDIA_API_KEY=nvapi-...
# (free key at https://build.nvidia.com/)

# Start Tor + API + Web UI
docker compose up -d --build

# Open http://localhost:8010
# CLI:   docker compose exec api python -m dnc_cli search "leaked credentials" --engines tordex --mode tor
# Logs:  docker compose logs -f api
# Stop:  docker compose down

Optional services (profiles):

# MCP server (for Devin, Claude Desktop, Windsurf, Cursor)
docker compose --profile mcp up -d

# Redis (future cache backend, v2)
docker compose --profile redis up -d

Option B — Local Python (no Docker)

git clone https://github.com/sookoothaii/DarkNetCrawler.git
cd DarkNetCrawler

cp .env.example .env
# Edit .env: set DNC_ENABLED=1, DNC_TOR_PROXY=socks5://127.0.0.1:9050
# Requires a local Tor service or Tor Browser running on port 9050

pip install -e ".[dev,tor,mcp]"

# CLI
python -m dnc_cli check
python -m dnc_cli search "leaked credentials" --engines tordex --mode tor

# REST API + Web UI
python server.py --host 0.0.0.0 --port 8010
# Open http://localhost:8010

# HTTPS (for LAN access from mobile devices)
pip install -e ".[ssl]"
python scripts/generate_cert.py
python server.py --ssl --host 0.0.0.0 --port 8010
# Open https://<lan-ip>:8010

# MCP server (for Devin, Claude Desktop, Windsurf, Cursor)
python -m mcp_dnc.server

Tor

Piano requires a Tor SOCKS5 proxy. The included docker-compose.yml provides a dedicated Tor container built from tor/Dockerfile (Debian + apt install tor, no external image dependency). The container exposes SOCKS on 127.0.0.1:9150 and the Tor control port on 127.0.0.1:9051.

Alternatively, any local Tor service (e.g. Tor Browser on port 9050) or an existing Tor container on a shared Docker network may be used — set DNC_TOR_PROXY accordingly in .env.

The fetcher uses httpx-socks with remote DNS (rdns=True) to resolve .onion addresses through the Tor proxy. Standard httpx SOCKS5 transport performs local DNS, which cannot resolve .onion hostnames.

LLM enrichment

LLM-assisted query refinement, IOC classification, and page summarization are optional and gated behind DNC_LLM_ENRICH=1. The implementation uses a standalone NVIDIA NIM cascade (nemotron-3-super-120b → nemotron-3-ultra-550b → mistral-nemotron) with Ollama qwen3:8b as an offline fallback. When the LLM is unavailable, all features degrade to regex-only extraction.

Testing

ruff check dnc mcp_dnc tests dnc_cli.py server.py
python -m pytest -x
python scripts/smoke_test.py

74 unit tests cover the engine registry, entity extraction, HTML parsers, circuit breaker state machine, TTL cache, envelope contract, BM25 ranker, Tor rotator, forensics, and configuration parsing.

License

MIT. See LICENSE.

Disclaimer

This software is a defensive research tool. The operator assumes all legal responsibility for its use. Read DISCLAIMER.md before first run.