voidcrawl 0.3.8.2

Rust-native CDP browser automation for Python via PyO3

VoidCrawl

CDP browser automation for Python — a Rust-native Chrome DevTools Protocol client exposed to Python via PyO3.

[!WARNING] VoidCrawl is research tooling for API design and web reverse engineering. You assume all legal risk for how you use it. Respect robots.txt, rate limits, and IP bans; and please don't bypass them with Tor or a VPN. Read DISCLAIMER.md before pointing it at anything.

void_crawl replaces Playwright/Selenium with a permissively-licensed (Apache-2.0) stack for rendering JavaScript-heavy pages. Built on chromiumoxide with a shared Tokio runtime.

Used by Yosoi — an AI-powered selector discovery tool for resilient web scraping.

Requirements

• Rust ≥ 1.86 (edition 2024)
• Python ≥ 3.10
• Chrome/Chromium installed on the system
• maturin ≥ 1.7 (cargo install maturin)

Installation

# Build and install into your venv
./build.sh

# Or manually:
maturin develop --release --manifest-path crates/pyo3_bindings/Cargo.toml

Quick Start

BrowserPool (recommended)

Tabs are recycled, not closed — near-instant reuse across requests.

import asyncio
from void_crawl import BrowserPool

async def main():
    async with BrowserPool.from_env() as pool:
        async with pool.acquire() as tab:
            await tab.navigate("https://example.com")
            print(await tab.title())
            print(len(await tab.content()))

asyncio.run(main())

BrowserSession (low-level)

import asyncio
from void_crawl import BrowserSession

async def main():
    async with BrowserSession(headless=True) as browser:
        page = await browser.new_page("https://example.com")
        print(await page.title())
        await page.close()

asyncio.run(main())

MCP server for Claude Code

voidcrawl-mcp is a stdio MCP server that exposes the full pool + session API as tools any MCP-speaking agent can call. Point Claude Code at it and the agent can fetch, screenshot, click, type, eval JS, detect captchas, and drive multi-step sessions — all via the same stealth-patched Chrome pool.

Install (pick one):

# Claude Code user — binary only, isolated venv:
uv tool install voidcrawl-mcp
# or
pipx install voidcrawl-mcp

# Python lib + MCP server together:
pip install 'voidcrawl[mcp]'

# From crates.io (builds from source):
cargo install voidcrawl-mcp

# From git HEAD (builds from source):
cargo install --git https://github.com/CascadingLabs/VoidCrawl voidcrawl-mcp

Wire it into Claude Code via .mcp.json:

{
  "mcpServers": {
    "voidcrawl": {
      "command": "voidcrawl-mcp"
    }
  }
}

Optional: pin the whole server to a warm Chrome profile (so every session inherits its cookies/logins):

voidcrawl-mcp --profile "Default"
# or: VOIDCRAWL_PROFILE="Default" voidcrawl-mcp

A ready-made Claude Code skill at .claude/skills/voidcrawl/SKILL.md teaches the agent when to pick voidcrawl over manual browsing, the click_visual_coords recipe for React forms, and how to react to typed CaptchaDetected errors. Claude Code picks it up automatically.

See docs/mcp-server.md for the full tool list and docs/profiles.md for profile leasing.

Docker

Pre-built multi-arch images (linux/amd64, linux/arm64) are published to GHCR on every push to main and every tagged release:

docker run -d --network host --shm-size=2g \
  ghcr.io/cascadinglabs/voidcrawl:headless-latest

export CHROME_WS_URLS="http://localhost:9222,http://localhost:9223"
python examples/basic_navigation.py

Or via compose:

docker compose -f docker/docker-compose.yml up -d

Available tags: headless-latest, headless-<version>, headless-<sha>, and the same set prefixed headful- for GPU + VNC (linux/amd64 only). See the Docker & VNC guide and the Docker Config reference for every runtime knob.

Testing

# Rust integration tests (serial — Chrome singleton lock)
cargo test -p void_crawl_core -- --test-threads=1

# Python integration tests (requires built extension + Chrome)
uv run pytest tests/ -v

Documentation

• Full API reference
• Anti-bot / CDN detection
• Challenge escalation with VNC/noVNC
• Cross-origin & closed-shadow frames
• Examples

Community

• Responsible use: see DISCLAIMER.md

Contact

[e-mail]

License

Apache-2.0