escalator

Give it a URL, get clean Markdown. It climbs the cheapest rung that works — a plain HTTP fetch, the same fetch through a residential proxy, then a stealth browser — and stops at the first one that comes back with real content.

$ escalator scrape https://en.wikipedia.org/wiki/Web_scraping | head -3
# Web scraping

**Web scraping**, **web harvesting**, or **web data extraction** is [data scraping](...)

Quickstart

# 1. install uv (https://docs.astral.sh/uv/getting-started/installation/)
curl -LsSf https://astral.sh/uv/install.sh | sh

# 2. configure this machine -- finds your browser, or fetches one
uvx escalator init

# 3. use it
uvx escalator scrape https://en.wikipedia.org/wiki/Web_scraping

That is the whole on-ramp. init writes one config file, and nothing has to be hand-edited before the first run. If anything looks wrong: escalator doctor.

On a minimal Linux (a bare container, a fresh VPS) Chrome needs system libraries that a desktop already has. escalator does not install them for you — it prints the exact apt-get line and escalator doctor repeats it. One command, once:

sudo apt-get update && sudo apt-get install -y \
  libnss3 libnspr4 libatk1.0-0t64 libatk-bridge2.0-0t64 libcups2t64 libdrm2 \
  libxkbcommon0 libxcomposite1 libxdamage1 libxext6 libxfixes3 libxrandr2 \
  libgbm1 libglib2.0-0t64 libpango-1.0-0 libcairo2 libasound2t64 \
  libatspi2.0-0t64 libxcb1 libdbus-1-3 libexpat1

No batteries, by design. escalator ships unconfigured — no bundled browser, no bundled proxies, no telemetry, nothing written outside its own directories. init exists so that configuring it is a two-minute conversation instead of a README scavenger hunt.

Installation

Everything needed for the core flow is in the default install, including the browser rung — it brings no browser with it, which is what escalator browser install is for. One extra exists: escalator[mcp] adds the MCP face, and the Docker image includes it.

Docker

docker run --rm \
  -e ESCALATOR_SERVER_API_KEYS=your-key \
  -p 8000:8000 -v ./data:/data \
  ghcr.io/ruslanstarikov/escalator serve

The image carries a pinned browser and is configured entirely by environment variable — see docker-compose.example.yml for the proxy wiring. It runs as uid 1000, so a bind-mounted ./data must be writable by it.

Commands

escalator init [--yes]     configure this machine; --yes for scripts
escalator doctor [--json]  check everything, one fix per failure
escalator browser list     every browser found, and which one wins
escalator browser install  download Chrome for Testing into the data dir
escalator scrape URL       one page to stdout, so it pipes
escalator serve            the HTTP API and the MCP face
escalator --version        tool, python, platform

The ladder

policy      robots.txt (cached) + rate limit  → may short-circuit (skip/deny/wait)
http        curl_cffi, impersonate=chrome     → ~100ms; clears undefended sites
http_proxy  same, routed via residential IP   → beats datacenter-IP bans
browser     nodriver, headless Chrome         → JS/SPA + Cloudflare-class defenses
                    │
                    └─ walled on the last rung? → status="challenged". Surrender.

Two things make this more than a for loop:

200 OK is not success. A rung that returns HTTP 200 carrying a Cloudflare interstitial has not succeeded. core/detect.py classifies every response after extraction — content, thin, or blocked — and only content counts. Without that the ladder would never escalate, and the cache would learn "http works" for a domain that serves junk forever.

The cache forgets. A learned start-rung that only ratcheted upward would drift every domain toward browser+proxy and quietly inflate your proxy bill. Entries carry learned_at; past ladder.tier_cache_ttl_hours a domain retries one rung cheaper.

See DESIGN.md for why it is shaped this way — and for what it deliberately refuses to do.

Configuration

One file, written by init, at the platform config dir (~/.config/escalator/config.toml on Linux, ~/Library/Application Support/escalator/config.toml on macOS). Override the location with --config.

Precedence, everywhere:

CLI flag  >  environment  >  config.toml  >  default

Every key has an environment variable, which is how the Docker image is configured without a file existing at all:

Data — the SQLite database and any downloaded browser — lives in the platform data dir, overridable with ESCALATOR_STORAGE_DATA_DIR. Nothing is ever written outside it.

Where the browser comes from

escalator browser list shows the search, in order:

1. an explicit path — --browser-path, then ESCALATOR_BROWSER_PATH, then browser.path. If it is set and wrong, that is an error naming the path, never a silent fall-through.
2. browsers installed on this machine: real Google Chrome first, then Chromium, then Edge and Brave.
3. a browser escalator browser install downloaded earlier.

If none of those find anything, you get an error naming the two commands that fix it. Resolution never downloads on its own — a server request or a cron job should not install software as a side effect.

Using the server

escalator serve   # 127.0.0.1:8000 by default

POST /scrape   {url, markdown?, min_tier?, max_tier?, timeout_ms?}  -> FetchResult
GET  /healthz                                                       -> {status, version}

Authenticate with Authorization: Bearer <key>, where the key is one of server.api_keys. That list is the truth: remove a key and it is revoked on the next start. There is no endpoint to mint one.

A wall comes back as 200 OK with {"status": "challenged"}, not as an HTTP error. That is deliberate: an agent on the other end can react to it. Retrying in a loop will not help — escalator does not solve CAPTCHAs, by design.

With the [mcp] extra the same ladder is exposed at /mcp as one tool, scrape_url(url, force_browser=False).

Troubleshooting

Start here:

escalator doctor

It checks Python, the config file, the data directory, browser resolution, an actual headless launch, and — if a proxy is configured — one real request through it, reporting the egress IP and country with the password masked. Every ❌ comes with the one line that fixes it, and the exit code is non-zero if anything failed, so scripts can use it too. escalator doctor --json for machines.

If it still will not work, paste the whole escalator doctor output into an issue — that is what the last line of it asks for, and it is the fastest route to an answer.

Development

See CONTRIBUTING.md. In short: uv sync, uv run pytest.

License

Released into the public domain — see UNLICENSE. No warranty, no attribution required, do what you like with it.