from scraper import Scraper

with Scraper(origin="https://example.com") as scraper:
    soup = scraper.get_soup("https://example.com/")
    print(soup.select_one("h1").text)

Those four lines already reproduce a real browser's TLS and HTTP/2 fingerprint, hold one address per origin, pace themselves like a person reading, and refuse to follow decoy links.

The idea

A modern mitigation engine runs many largely independent detectors and folds them into one trust score. Admission behaves as a near-conjunction, so:

    P(evade)  ≲  min( p₁ , p₂ , p₃ , … , pₙ )
                       ▲
                       └─ the binding layer. Every other layer is
                          wasted effort until this one stops being
                          the minimum.

The weakest layer bounds the outcome. If a strategy fails on address reputation, perfecting its TLS profile gains nothing — not a little, zero. So this library diagnoses which layer is binding before it changes anything.

What a detector reads decides whether it can be satisfied.

That second distinction produces the behaviour that most sets this library apart: when the binding layer reads a possessed property, it does not rotate. Rotating discards the very history the detector is measuring, so it holds the address still and slows down. And two layers raise instead of retrying, because they read a secret you either hold or do not.

Full treatment: the model. The layer model, the emit/possess distinction and the reference patterns come from the paper this library implements — A Layered Model of Modern Web Bot Protection and the Structural Limits of Its Circumvention, included as a PDF.

What it does

The ladder

  cost   tier         reaches
  ────   ──────────   ───────────────────────────────────────────────────
     0   archive      everything — when a capture exists
    10   direct       TLS, HTTP/2 frames, header order, post-quantum keyshare
   100   clearance    + the JavaScript, Turnstile and automation-channel layers
  1000   managed      + the per-zone composite, at someone else's price

The planner picks the cheapest rung that covers the binding layer, and stops with an explanation when no configured rung does. Writing your own rung: tiers.

Installation

pip install lncrawl-scraper

Impersonation is not an extra. Layers 2–5 are one barrier and an ordinary Python client fails all four in the first round trip, so a build without it would not be a degraded scraper but one that cannot reach a protected page.

Adding reach

Two settings change what this library can do. The rest adjust how it does it.

from scraper import ExitKind, ExitSpec, Scraper, ScraperConfig
from scraper.browser import NoDriverSolver

config = ScraperConfig(
    # The only thing that moves layer 1: reputation is not something a client emits.
    # Declare the kind honestly — claiming MOBILE for a datacenter range only stops
    # this library from telling you that layer 1 is why nothing works.
    exits=[ExitSpec(url="http://user:pw@residential.test:8000", kind=ExitKind.RESIDENTIAL)],
    # The only thing that reaches the challenge layers.
    browser=NoDriverSolver(),
)

with Scraper(origin="https://site.test", config=config) as scraper:
    scraper.get("https://site.test/deep/page")
    print(scraper.explain("https://site.test/deep/page"))

site.test
  binding layer : L9 Managed JavaScript challenge — reads a hybrid property, solve
  tier          : clearance
  pacing        : 4.2s mean interval
  requests      : 48 ok / 3 failed
  clearance     : 712s left
  ladder        : direct(10) clearance(100)
  exits         : residential

When it stops

Failures name the layer and what would move it, because "403 after 3 retries" is the message that sends people to rewrite the part that was already working.

from scraper import Layer
from scraper.exceptions import Exhausted, Impassable

try:
    scraper.get(url)
except Impassable as exc:
    # Layers 18 and 19 read a secret. Nothing to retry; the message names the route.
    print(exc.detail)
except Exhausted as exc:
    # A bypass may exist; this configuration does not reach it.
    if exc.layer is Layer.IP_REPUTATION:
        print(exc.detail)
        # "no configured exit clears the reputation layer — datacenter and Tor ranges
        #  are published, so rotating between them cannot help."

Documentation

Scope

This library is for retrieving publicly accessible content. It does not attempt authentication bypass, credential abuse, or circumvention of access controls protecting non-public data — layer 19 raises rather than trying, and layer 18 raises where a signature is mandated. Where a site publishes an API or an archive holds what you need, both are cheaper than anything else here and are supported first-class for that reason.

Development

uv sync                 # deps + editable install
uv run poe lint         # ruff + pyright
uv run poe test         # pytest
uv run poe cov          # with coverage

Tests are offline: the pipeline talks to a two-method Transport, so tests/conftest.py's FakeTransport covers every tier without a network. The modules that encode judgement — diagnosis, planner, layers — are pure functions over primitives and are tested as such.

uv run poe live-all

Credits

Extracted from lightnovel-crawler.