Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

crawlutron

A policy-driven, self-hosted web crawler and content-extraction service built around FastAPI. It exposes a single, machine-readable crawl contract that supports:

Positioning. crawlutron is a self-hosted, policy-driven crawler for public pages. Its caller security boundary is built-in per-caller authentication (API keys issued and revoked through /v1/keys, operator-token gated; keyed jobs and results are readable only under their key, anonymous crawls share a bounded identity). SSRF is architecture, not an add-on: every egress target is classified, fresh-resolved, IP-pinned, and re-validated per hop and per browser subresource — when the private-target guard is enabled, warned once per enabled value at first use (packages/safe-url/src/safe_url/host_policy.py:88-100). A transport-auth reverse proxy is recommended when exposing beyond localhost (TLS termination, Server header stripping, request-id logging), but built-in per-caller auth is the caller security boundary; the proxy is transport hardening, not a substitute for it. The rule framework (tests/invariants/) arms and bites every rule, and the pre-commit hook chain gates every commit. See docs/architecture-ssrf.md, docs/architecture-governance.md, and SECURITY.md.

  • HTTP fetch strategies — transparent declared-crawler HTTP and TLS-fingerprint impersonation (curl_cffi).
  • Browser rendering — headless/headed Firefox via Camoufox for JavaScript-dependent pages.
  • Document extraction — PDF/DOCX/PPTX/XLSX/TXT via markitdown, plus main-text extraction via trafilatura.
  • Content classification — 36 crawl classifications (rendered, blocked, network_error, robots_disallowed, waf_challenge_detected, etc.) so callers can branch on outcome rather than guessing.
  • Discovery endpoints — capabilities, agent hints, and a .well-known service descriptor so agents/bots can discover the contract without prior knowledge.
  • Web search — POST /v1/search discovers public URLs from a query through a keyed provider (Tavily / Brave / Serper) with a keyless DuckDuckGo fallback.
  • MCP — POST /mcp exposes search/crawl/map/render as tools over the Model Context Protocol (2026-07-28, stateless streamable HTTP).
  • SSRF protection — target URLs are validated to be public HTTP(S) only.
  • Webhooks, batching, recursion, rate limiting, robots.txt respect — best-effort, caller-owned frontier semantics.

Platforms

Linux and macOS are supported; Windows is not. The service is built on POSIX primitives the Windows runtime does not provide: fcntl advisory file locking (packages/core/src/crawlutron/safety/atomic_write.py, packages/core/src/crawlutron/orchestration/engine.py, packages/core/src/crawlutron/transport/content_cache.py), resource/sysconf-based memory sizing (packages/core/src/crawlutron/runtime/pressure_memory.py), and cgroup filesystem readings for the resource gauges (packages/core/src/crawlutron/runtime/cgroup_sense.py). Deploy it as a Linux container or run it directly on Linux/macOS.

Security residuals and environment escape hatches

Three residual weaknesses are known and accepted, with the assumption that guards them stated:

  • S1 — HTTPS DNS rebind window. HTTP targets are IP-pinned end to end. HTTPS browser navigation does two fresh DNS resolves but lets the browser connect by hostname, so a hostile DNS server can rebind between check and connect.
  • S4 — credential origins ride the egress guard. Credential origins pass the same SSRF egress check as crawl targets: a private/loopback credential origin requires CRAWLUTRON_ALLOW_PRIVATE_TARGETS=1.
  • S5 — caller key secrets are verified with salted PBKDF2-SHA256. Issued secrets are ck_<key_id>.<body> where the body is secrets.token_urlsafe(32) (256-bit entropy). At rest the key row stores a salted PBKDF2-SHA256 digest (210,000 iterations, per-key 16-byte salt, packages/core/src/crawlutron/storage/caller_key_store.py); verification is one primary-key lookup plus one PBKDF2 run, with a miss burning the same derivation against a fixed dummy digest so timing is uniform. Residual: entropy of the secret, not the KDF, is the real guard; do not reuse this pattern for human-chosen secrets.

Dangerous environment escape hatches

Every CRAWLUTRON_* variable that weakens a safety default (derived from the live settings/env code — packages/safe-url/src/safe_url/host_policy.py, packages/core/src/crawlutron/service/webhooks.py, packages/core/src/crawlutron/config/settings.py, packages/core/src/crawlutron/transport/driver_config.py):

Variable Default Weakened when Effect
CRAWLUTRON_ALLOW_PRIVATE_TARGETS unset 1 Disables the SSRF private-target guard: caller URLs may reach private, loopback, and link-local addresses including cloud metadata endpoints. Logs a SSRF private-target guard DISABLED WARNING.

Ordinary operational knobs

Two variables are sometimes mistaken for safety hatches; they are not. They change operational behavior only and are listed in the Configuration table below:

  • CRAWLUTRON_RESPECT_ROBOTS (default true) — turns robots.txt enforcement off. Robots compliance is a crawler courtesy/policy choice, not a security boundary of this service.
  • CRAWLUTRON_HEADLESS (default true) — runs the browser headed instead of headless. Changes the automation surface, weakens no guard.

See SECURITY.md for the threat model and reporting process, and docs/reviews/ADVERSARIAL_REVIEW_BROWSER_CRAWLER.md for the dated adversarial review of the crawler.

The service is intended to be deployed as a Docker container. A standalone docker-compose.yml lives at the repository root for local development; production deployments typically front it with a transport-auth reverse proxy (TLS termination, Server header stripping, request-id logging); built-in per-caller auth (/v1/keys, X-API-Key/Authorization: Bearer) is the caller security boundary, and the proxy is transport hardening, not a substitute for it.

Security Posture

crawlutron is a localhost-only service. It binds to 127.0.0.1 by default. Callers authenticate with an API key presented as X-API-Key or Authorization: Bearer — jobs and results created under a key are readable only under it, and key issuance/revocation is operator-gated (CRAWLUTRON_KEY_ADMIN_TOKEN via /v1/keys). Keyless requests crawl as a shared anonymous identity with bounded default quotas. Key verification is rate-limited (#653): five consecutive failed verifications from the same presented credential trip a 60-second cooldown that refuses further attempts with 429 key_verify_throttled before the PBKDF2 derivation runs again, so the expensive pre-auth hash cannot be invoked for free in a loop. Buckets key on a hash of the full presented secret (#675), so garbage under another caller's key id cannot lock the rightful owner out. Anonymous surface metering keys on the REMOTE ADDRESS the ASGI transport reports (client.host): behind a reverse proxy every request shares the PROXY's address and its one bucket — when fronting the service with a proxy, run the metering at the proxy or deploy as a single-tenant front. Expose it beyond localhost only behind a transport-auth reverse proxy; built-in per-caller auth (/v1/keys) is the caller security boundary, and the proxy is transport hardening, not a substitute for it.

What the SSRF defenses cover:

  • DNS pinning on every fetch — every URL is resolved fresh at fetch time and the resulting IP is pinned for the connection, so a DNS rebind cannot redirect the request mid-flight.
  • Manual redirect handling — redirects are followed one hop at a time; each hop re-runs the IP-pinning and private-IP check.
  • Browser-side SSRF — the Camoufox/Playwright navigation is pinned to a freshly-validated IP via Host: header rewriting; post-redirect URLs are re-validated before being used as document fetch targets.
  • Body-size guard — request bodies are capped at 16 MiB before pydantic validation runs.
  • HTML/DOM size guards — classification and metadata extraction truncate inputs at 4 MiB before regex / lxml processing.
  • Per-tenant cookie namespace — multi-tenant callers should set request_profile.cookie_namespace so cookies from different callers are isolated in the shared cookie jar.

Adversarial Test Coverage

The test suite was grown alongside the adversarial reviews; the following suites cover the specific attack classes the security posture is designed to defeat:

  • Truncation and drain (tests/test_body_cap_state_machine.py) — chunked handling, Connection: close on a truncated request, per-chunk trailers and headers, body-size drain after truncation, the rule that no further bytes are admitted once a chunk trips the cap, and the 413 the shipped app returns on a body over the cap.
  • File-descriptor ownership (tests/invariants/test_ownership.py) — scratch files are owned by the work writing them; the SQLite spill store replaced the file spill (tests/storage/test_crawl_spill_store.py).
  • Cache locking (tests/transport/test_dns_cache.py, tests/transport/test_robots_policy_consolidation.py) — concurrent access to the DNS and robots caches, including that robots reads go through the one chokepoint.
  • Cookie namespacing (tests/transport/test_cookie_namespace.py) — per-tenant cookies cannot cross a tenant boundary.
  • Env parsing (tests/test_env_parsing.py) — parse_bool_env edge cases, including that a typo cannot silently open a policy.
  • Webhook TOCTOU (tests/api/test_webhook_delivery.py) — one resolved target per delivery, consolidated at the chokepoint.
  • Decompression bomb tests — a real gzip bomb (~1 KB compressed → 1 MB expanded), plus deflate and brotli bomb cases, verify the decompression guard rejects over-cap output before OOM.
  • SSRF tests (tests/test_ssrf_route_subresources.py, tests/transport/test_dns_cache.py, tests/transport/test_redirect_target.py, tests/transport/test_browser_target_stability.py) — DNS pinning, route-level interception, WebSocket guard, hostname pinning even after a redirect chain, IPv6 private range blocking.
  • Classification corroboration tests — soft phrases (e.g. "blocked") require ≥2 independent signals; the test asserts a single soft signal is NOT enough to trigger a defensive classification.
  • Resource admission control — fake cgroup gates verify the two-tier weighted admission policy kills stop-the-world pauses under saturation.

This is unusually thorough for an open-source project. New attack classes should land alongside a dedicated adversarial test that fails without the fix.

Coverage is measured, not a badge. No coverage badge or coverage ratchet is claimed. The pre-push coverage hook is measured-descriptive only — no threshold, no fail-under, no ratchet until the owner rules one (.pre-commit-config.yaml:229-237). The safety boundary is the invariant suite and the fail-closed SSRF/auth guards, not a coverage percentage. See docs/architecture-governance.md.


Build & Run

The service ships with a standalone docker-compose.yml and a multi-stage Dockerfile so it can be built and run from a fresh clone without any parent monorepo or shared-helper tree.

# Build the image (multi-stage; the Camoufox browser binary is fetched once in an
# isolated stage so editing requirements.txt does not re-download ~1.3 GB).
docker compose build crawlutron

# Run it (recreates only this service).
docker compose up -d crawlutron

The container listens on 127.0.0.1:8765. The served API docs (Swagger UI) are available at /docs and the machine-readable schema at /openapi.json. For local-hostname routing (e.g. http://crawlutron.local), front the container with Caddy or another reverse proxy that owns the .local zone.

Note on the build: The Camoufox browser binary is downloaded in a dedicated camoufox-binary stage that depends only on the camoufox requirement line, then copied into the runtime image. This isolation means an unrelated change to requirements.txt (for example adding h2) no longer invalidates the multi-minute binary download.


SQLite schema migrations

Migrations live in packages/core/src/crawlutron/storage/migrations/. Every store schema — jobs, results, metrics, coordination — lives in the single cache/crawler.db; startup applies that database's pending migrations under yoyo's lock, and an exact pre-ledger table is stamped instead of recreated. The CLI names the schema explicitly because they share one file.

# Apply pending migrations (normally automatic when the store opens).
PYTHONPATH=packages/core/src .venv/bin/python -m crawlutron.storage.migrations apply cache/crawler.db crawl_jobs
PYTHONPATH=packages/core/src .venv/bin/python -m crawlutron.storage.migrations apply cache/crawler.db results

# Add a migration: add a versioned schema to manifest.py, point CURRENT_SCHEMAS at it,
# then add a yoyo step with apply and rollback SQL under the migration directory.
$EDITOR packages/core/src/crawlutron/storage/migrations/manifest.py
$EDITOR packages/core/src/crawlutron/storage/migrations/0001_crawl_jobs_initial.py

# Stamp an existing database after checking it against the declared schema.
PYTHONPATH=packages/core/src .venv/bin/python -m crawlutron.storage.migrations stamp cache/crawler.db crawl_jobs
PYTHONPATH=packages/core/src .venv/bin/python -m crawlutron.storage.migrations stamp cache/crawler.db results

# Roll back one schema's applied migrations in reverse order; do not open the
# store afterward. Rolling one schema back leaves the others applied.
PYTHONPATH=packages/core/src .venv/bin/python -m crawlutron.storage.migrations rollback cache/crawler.db crawl_jobs
PYTHONPATH=packages/core/src .venv/bin/python -m crawlutron.storage.migrations rollback cache/crawler.db results

Endpoints

Method Path Purpose
GET /health Liveness probe (always 200 if the process is up).
GET /ready Readiness probe; returns 503 when the self-managed resource guard is saturated (memory or CPU at the gate target).
POST /v1/crawl Submit one URL (or a batch of URLs) for crawling.
POST /v1/render Render a URL with the browser engine and return structured output.
POST /v1/search Discover public URLs from a query via the configured search provider.
POST /v1/map Crawl a site's link graph and return the discovered URL set (no content extraction).
GET /v1/crawl/{request_id} Fetch the result of an async crawl submission.
GET /v1/crawl/{request_id}/trace Per-URL strategy attempt trace (statuses, classifications, timings; no page bodies).
POST/GET/DELETE /v1/watches Revalidation watches: register, list, get, delete; sweeps recheck due watches and POST webhooks on change.
GET/POST /v1/jobs Whole-site resumable jobs: create (with frontier), list, inspect, pause/resume/cancel.
POST /mcp MCP 2026-07-28 endpoint exposing search/crawl/map/render as tools.
GET /v1/capabilities Declares supported strategies, classifications, extraction methods, profiles.
GET /v1/agent-hints Machine-readable guidance for agents/bots (request shapes, caveats).
GET /.well-known/crawler-service Service discovery descriptor (links to OpenAPI, agent hints, capabilities).
POST/GET/DELETE /v1/keys Caller API keys: issue, list, revoke (operator token required). Callers present the key as X-API-Key.
GET /metrics Prometheus-format metrics. Per-caller detail families require the operator token (X-API-Key: CRAWLUTRON_KEY_ADMIN_TOKEN); a presented but wrong token records an admin-bucket failure (paced 5/300s), so scrape-guessing griefs the operator doors' cooldown too.
GET /v1/policy-suggestions Candidate host policy rules rolled up from blocked-attempt evidence (proposal only; never applied).

POST /v1/search discovers public URLs from a query and returns SERP-style results ({success, query, provider, limit, data: [{url, title, description}]}). It does not crawl result URLs — pipe them into POST /v1/crawl.

The backend is operator-configured, not caller-selected:

  • CRAWLUTRON_SEARCH_PROVIDER (auto | tavily | brave | serper | duckduckgo).
  • auto (default) activates the first configured keyed provider in priority order (Tavily to Brave to Serper) and falls back to the keyless DuckDuckGo HTML endpoint when no API key is set. The keyless endpoint may be captcha-blocked from datacenter IPs, so configure a key for production use.
  • A selected provider without its key returns 503 search_provider_unavailable naming the missing CRAWLUTRON_*_API_KEY variable.

Result URLs pass the same public-URL validation as crawl targets (loopback and private targets are dropped). GET /v1/capabilities reports the active provider and the configured provider list.

MCP

POST /mcp exposes search, crawl, map, and render as tools over the Model Context Protocol (2026-07-28) — the stateless revision: no initialize handshake, no sessions. Every request carries its protocol version and client capabilities in _meta, and the transport validates the MCP-Protocol-Version, Mcp-Method, and Mcp-Name headers against the body.

  • server/discover — supported protocol versions, capabilities, server identity.
  • tools/list — the fetch, read, and job tools with JSON-Schema inputs.
  • tools/call — invokes a tool; returns text content plus structuredContent, with isError set for execution failures.
  • The tools mirror the REST endpoints and delegate to the same handlers, so all four strategies and every extraction method are available through MCP.
  • Webhook/async delivery is not available over MCP.

The endpoint is loopback-only and rejects non-loopback Origin headers (DNS rebinding defense).

Crawl request shape

POST /v1/crawl accepts either a single url or a list of urls, plus optional top-level fields: policy, validators (cached conditional-GET validators), async_mode, webhook_url, and output_format. The request/browsing profile knobs — request_profile, browser_profile, discovery, scope_regex, css_selector, xpath — live inside policy.

{
  "url": "https://example.com/article",
  "policy": {
    "strategies": ["HTTP_DECLARED_CRAWLER", "BROWSER_HEADLESS"],
    "extraction": "full",
    "limits": {
      "timeout_seconds": 30,
      "max_retries": 2,
      "challenge_wait_seconds": 4,
      "max_discovered_links": 10,
      "respect_robots": true
    },
    "request_profile": { "user_agent_family": "generic_crawler" },
    "browser_profile": { "resource_mode": "full" },
    "discovery": { "robots": true, "sitemaps": true, "feeds": true }
  }
}

Strategies (fallback chain)

strategies is an ordered fallback chain. The engine tries each in turn until one succeeds, returns a definitive blocked/error classification, or the chain is exhausted.

Strategy Description
HTTP_DECLARED_CRAWLER Transparent direct HTTP fetch with a declared-crawler identity (lightweight, fast).
HTTP_CFFI_IMPERSONATE TLS-fingerprint-impersonated HTTP via curl_cffi with browser-shaped headers, for sites that reject non-browser clients.
BROWSER_HEADLESS Firefox rendering in headless mode using Camoufox for JavaScript-dependent pages.
BROWSER_HEADED_WARMED Headed Firefox with configured warmup URLs for challenge-prone sites.

Extraction methods (policy.extraction)

  • full (default) — best-effort main-text extraction, returns both markdown and text.
  • trafilatura — main-text only via trafilatura.
  • document — markitdown-based extraction for PDF/DOCX/PPTX/XLSX/TXT (binary documents).
  • dom — raw DOM / selected-element extraction (css_selector / xpath).

Classifications

Every crawl result carries a classification from a fixed enum of 36 values (defined in packages/core/src/crawlutron/vocabulary/taxonomy.py, returned by GET /v1/capabilities under supported_classifications). Examples: rendered, not_modified, captcha_detected, waf_challenge_detected, robots_disallowed, navigation_timeout, private_network_target, network_error, http_error, url_budget_exceeded, payload_too_large. Callers should branch on classification (whose status: success | blocked | disallowed | retry_exhausted | failed | skipped the taxonomy maps onto) rather than on transport status.


Behavior notes

  • SSRF protection is on by default: only public HTTP(S) targets are allowed. Set CRAWLUTRON_ALLOW_PRIVATE_TARGETS=1 to disable it (local testing only). Enabling it logs a WARNING naming the disabled guard, and lets caller-supplied URLs reach private, loopback, and link-local addresses including cloud metadata endpoints.
  • robots.txt is respected by default (respect_robots). Disallowed targets return robots_disallowed.
  • Recursion / frontier: POST /v1/crawl recursively expands one in-memory frontier for the duration of a request and reports discovered links, sitemaps, and feeds; cross-run queue ownership remains with the caller. POST /v1/jobs instead owns a durable whole-site frontier. Each persisted frontier and in_flight entry is {"url": "...", "depth": N}; url is the canonical page identity and depth 0 is the stored canonical root. Legacy string entries load at depth 0 when they are the root and at the effective max_depth otherwise, preventing untracked additional recursion during upgrade.
  • Whole-site admission: the engine dispatcher and durable job runner share canonical-identity, scope, analytics, document, robots, crawl-depth, and page-cap admission. When discovery.sitemaps is enabled, root sitemap pages enter the durable frontier at depth 0 through that same pipeline. max_depth=0 is single-page. A positive job max_pages counts unique canonical page identities across the root, pending frontier, in-flight recovery, visited pages, and retries. Missing or zero max_pages is unbounded for jobs; /v1/crawl retains its bounded service default. A positive job cap sets cap_truncated=true in status and stats.cap_truncated=1 only when it refuses or truncates eligible queued or discovered work; normal BATCH_SIZE chunking does not set it. Already-admitted in-flight work is never dropped: recovery preserves at-least-once delivery and records stats.cap_overcommitted=1.
  • Crawl duration & async: a synchronous POST /v1/crawl has no server-side time ceiling — it returns only when the whole bounded recursion finishes, so duration scales with max_depth, max_pages, per-domain politeness delays, and the per-URL budget (limits.total_url_budget_seconds, default 180s, which fails a single tarpit URL, not the crawl). For large or recursive crawls send async_mode=true (or webhook_url) to get an immediate 202 + status_url, then poll GET /v1/crawl/{request_id}; or use POST /v1/jobs for durable, resumable, pause/cancel-able whole-site jobs. Read timeouts on the sync path must cover the chosen policy and batch size.
  • Rate limiting is enforced via global_concurrency, per_domain_concurrency, and per_domain_delay_ms, scoped to a single request: each request runs its own scheduler, so concurrent requests each enforce the limits independently and a service-wide per-domain bound is not provided (callers pacing many concurrent requests own the aggregate). Exceeding max_batch_size is rejected.
  • Webhooks / async: provide webhook_url to receive results asynchronously, or async_mode=true to get a 202 Accepted with a status_url to poll GET /v1/crawl/{request_id}. Both return immediately instead of blocking on the crawl.
  • Conditional GETs: HTTP_DECLARED_CRAWLER sends If-None-Match / If-Modified-Since when prior ETags are known, returning not_modified on a 304.
  • Resource self-tuning (#self-tuning): the engine senses its own cgroup memory/CPU ceiling and throttles recursive-batch dispatch to ride at the targets instead of OOMing or tripping the kernel CPU throttle. No operator tuning is required.
  • HTTP-aware admission (item 7): browser crawls are the memory hog (one ≈700 MiB (≈600–900 MiB RSS) browser process each), and when browser processes hold memory the whole batch used to be throttled — even cheap HTTP work. Today, when the upcoming batch is predominantly domains pinned to an HTTP strategy (they've already proven they fetch without a browser) and CPU has spare headroom, the dispatcher uses the HTTP-aware scale (ResourceGate.http_scale_factor), which rides the CPU-implied rate and is not dragged down by memory held by browser processes of other domains. Browser-bound and cold (unpinned) batches keep the conservative combined memory+CPU gate, so the extra concurrency is always ramp-bounded (≤ RECURSIVE_RAMP_MAX × multiplier) and cannot overshoot memory.

Configuration

All settings are read from CRAWLUTRON_* environment variables (sensible defaults shown):

Variable Default Meaning
CRAWLUTRON_GLOBAL_CONCURRENCY 4 (env-overridable floor) Concurrency floor for the recursive dispatcher. Real concurrency is self-managed by the resource guard; /ready no longer keys off this value.
CRAWLUTRON_PER_DOMAIN_CONCURRENCY 2 Max concurrent tasks per host.
CRAWLUTRON_PER_DOMAIN_DELAY_MS 500 Delay between requests to the same host.
CRAWLUTRON_HTTP_MAX_CONNECTIONS (RAM-derived; floor 100, cap 512) Max concurrent HTTP connections. The pool default is the floor of 100; the weighted admission derives the real HTTP concurrency from the cgroup limit as (limit − browser-pool reserve − doc-pool reserve) / 48 MiB per HTTP task — ~34 at 3 GiB with the 2-instance browser pool (~700 MiB each), ~22 with the default 2 doc workers (300 MiB each). Bound to the weighted admission's HTTP concurrency so the pool can't starve or over-provision.
CRAWLUTRON_HTTP_MAX_KEEPALIVE (RAM-derived; floor 20) Max idle keep-alive connections (≈ half of the connection ceiling, bounded to [20, 128]).
CRAWLUTRON_BROWSER_POOL_SIZE (RAM-derived; 2 at 3 GiB) Explicit browser-pool size override. Default is derived from the cgroup limit (reserves ≤50% of RAM for browsers); raising it multiplies RAM use and risks OOM.
CRAWLUTRON_SEARCH_PROVIDER auto Search backend: auto (first configured keyed provider) or explicit tavily / brave / serper / duckduckgo.
CRAWLUTRON_TAVILY_API_KEY "" Tavily Search API key (activates the tavily provider).
CRAWLUTRON_BRAVE_API_KEY "" Brave Search API key (activates the brave provider).
CRAWLUTRON_SERPER_API_KEY "" Serper.dev API key (activates the serper provider).
CRAWLUTRON_SEARCH_LIMIT_DEFAULT 5 Default result count per search.
CRAWLUTRON_SEARCH_LIMIT_MAX 20 Hard cap on results per search.
CRAWLUTRON_SEARCH_TIMEOUT_SECONDS 15 Provider call timeout.
CRAWLUTRON_MAX_BATCH_SIZE 50 Max URLs per batch request.
CRAWLUTRON_WARMUP_URLS "" Comma-separated URLs fetched once at browser startup to warm engine caches (browser.py). Warmup URLs are normalized through the same public-URL policy as crawl targets. Related: browser_profile.storage_state paths are honored only under CRAWLUTRON_CACHE_DIR (the allowlist root); without a cache dir no storage_state is accepted.

Two DNS posture values are not env-tunable — they are module constants in packages/core/src/crawlutron/transport/pinned_dns.py: DNS_WORKERS = 32 (the dedicated DNS resolver thread pool) and DNS_WAIT_SECONDS = 5.0 (the ceiling on a resolver wait; a request's own timeout tightens it further). They implement the pinned-DNS saturation law: when every worker is inside a live resolve, a new resolve refuses fast (ResolverBudgetExceeded, classifiable as dns_error) instead of queueing behind an uncancellable getaddrinfo. | CRAWLUTRON_DEFAULT_TIMEOUT_SECONDS | 30 | Default per-request timeout. | | CRAWLUTRON_DEFAULT_MAX_RETRIES | 5 | Default retry count. | | CRAWLUTRON_RESPECT_ROBOTS | true | Respect robots.txt by default. | | CRAWLUTRON_ALLOWED_HOSTS | "" | Optional host allowlist (comma-separated suffix match; case, IDNA, and port normalized). | | CRAWLUTRON_DENIED_HOSTS | "" | Optional host denylist (comma-separated suffix match; deny wins over allow). | | CRAWLUTRON_LOG_LEVEL | info | Log verbosity. | | CRAWLUTRON_LOG_FILE | ./logs/crawlutron.jsonl (repo-relative; the container image overrides it to /app/logs/crawlutron.jsonl) | Structured JSONL log path. | | CRAWLUTRON_LOG_MAX_BYTES | 10485760 | Log rotation size. | | CRAWLUTRON_LOG_BACKUP_COUNT | 3 | Number of rotated log files to keep. |

Host policy is applied at the shared public-URL chokepoint before the SSRF/private-target check. Entries are normalized as case-insensitive IDNA hostnames (_policy_host_entry in packages/safe-url/src/safe_url/host_policy.py parses each value via urlparse(...).hostname); a port in a config value is stripped during normalization and never matched — an entry is a hostname, not a host:port pair. A denied host wins over an allowed host, while an empty allowlist permits every otherwise-valid public host.

The container's production command also reads CRAWLUTRON_WORKERS (uvicorn worker count). Keep this at 1. The browser pool is a single global pool sized to the container RAM budget; running more than one uvicorn worker multiplies the browser-pool count across workers (e.g. 4 workers × pool_size 2 = 8 Camoufox processes ≈ 5+ GiB) and blows the 3 GiB memory limit. Do not raise it.

Secrets & logs: Logs and prompt JSONL can contain full page content and request metadata. Do not commit generated logs under logs/.


API documentation (served)

  • Swagger UI: /docs
  • OpenAPI JSON: /openapi.json
  • Agent hints: /v1/agent-hints
  • Capabilities: /v1/capabilities
  • Service descriptor: /.well-known/crawler-service

The /v1/render request body is a typed RenderRequest (url/urls plus an options object with timeout_seconds, max_retries, respect_robots, include_html, include_text). Malformed bodies are rejected with 422.


Project layout

.  # repository root (clone of crawlutron)
├── Dockerfile            # Multi-stage build; isolated camoufox-binary stage (context: .)
├── docker-compose.yml    # Standalone local stack (loopback-only)
├── pyproject.toml        # Package metadata + deps (single source; uv.lock pins every pin)
├── uv.lock               # Frozen transitive dependency graph + hashes
├── Makefile              # Gates + convenience targets mirroring scripts/run-tests.sh
├── packages/             # uv workspace member packages, src layout under packages/*/src/
│   ├── core/
│   │   └── src/crawlutron/   # core engine: admission kernel, models, transport, storage, service, …
│   │       ├── api/          # door-plane handler bodies (crawl_endpoint, render_endpoint, map_endpoint)
│   │       ├── cli/          # crawler CLI entry points
│   │       ├── config/       # env.py + settings.py (CRAWLUTRON_* environment surface)
│   │       ├── discovery/    # /v1/capabilities, /v1/agent-hints, /.well-known
│   │       ├── extraction/   # documents.py (trafilatura + selector extraction)
│   │       ├── health/       # /health, /ready (READINESS_STATE singleton)
│   │       ├── mcp/          # MCP protocol support shared with crawlutron_mcp
│   │       ├── models/       # Pydantic/dataclass wire + domain models
│   │       ├── orchestration/# engine.py — orchestration + fallback-chain scheduling
│   │       ├── runtime/      # metrics, memory pressure, worker dispatch, resource lifecycle
│   │       ├── safety/       # validation, bounded I/O, safe regex, SSRF helpers
│   │       ├── search/       # /v1/search provider abstraction (Tavily/Brave/Serper/DuckDuckGo)
│   │       ├── service/      # store_bootstrap, watch_cores, job lifecycle, …
│   │       ├── storage/      # job_store, result_store, watch_store, sqlite_store, migrations/
│   │       ├── transport/    # http.py, browser.py, document.py, content_cache.py, scheduler.py, robots.py
│   │       └── vocabulary/   # taxonomy.py (36-value classification enum), strategy names
│   ├── rest/
│   │   └── src/crawlutron_rest/   # REST facade: FastAPI app + routers + CLI
│   │       ├── main.py        # FastAPI factory (lifespan, app.state, router wiring)
│   │       ├── api/           # one router per concern (crawl, jobs, watches, keys, metrics, policy-suggestions, …)
│   │       └── cli/           # CLI entry (__main__, output, docker, client)
│   ├── mcp/
│   │   └── src/crawlutron_mcp/  # MCP 2026-07-28 endpoint (protocol, tools, server)
│   └── sdk/
│       └── src/crawlutron_sdk/  # client SDK mirroring the REST wire surface
├── tests/                # Behavior tests (no mocks; real HTTP + real Camoufox)
├── docs/                 # Reviews, testing guide, agent contract (docs/references/)
├── scripts/run-tests.sh  # Full battery entrypoint (mirrored by make targets)
└── .github/workflows/ci.yml  # ruff + pytest (fast)

Testing

Behavior tests run against the real stack (a local http.server, a real Camoufox browser, and real fixtures — no mocks). Use the dedicated venv:

pip install -e ".[dev]"
make test          # fast hermetic suite (default)
# or
python3 -m pytest -q

The manual test plan lives at docs/references/crawlutron-manual-test-plan.md and the agent integration contract at docs/references/crawlutron-agent-contract.md.

License

MIT — pyproject.toml (license = "MIT") and the LICENSE file agree; the NOTICE file carries the third-party attributions.

Metadata

Release files for crawlutron 0.4.0rc1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for crawlutron 0.4.0rc1
File Interpreter ABI Platform Actions
crawlutron-0.4.0rc1-py3-none-any.whl Python 3 none any Details for crawlutron-0.4.0rc1-py3-none-any.whl

Release files / crawlutron-0.4.0rc1-py3-none-any.whl

Download URL crawlutron-0.4.0rc1-py3-none-any.whl
Size 16.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
6b44deaee767be0842fe6d64dde8a4e52793a7ddbc0f91d45c9e873aad03fb01
BLAKE2b-256 checksum
How to use checksums
6eb7a5a5e86f3124156505cb8c18d40ab6f13e095ff4530c73a5326efbc95584
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Security for crawlutron 0.4.0rc1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release
Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page