graph-crawler
Web crawler that builds a graph of a site: pages are nodes, links are edges.
You get a Graph object you can query, export, or feed into another tool.
Fetching goes through a chain of drivers with escalation: plain HTTP first, then TLS-fingerprint HTTP, then a real browser, then an "undetected" browser. The browser step accepts any engine (CloakBrowser, patchright, stock Playwright, or your own binary).
Install
pip install graph-crawler # HTTP only (aiohttp)
pip install "graph-crawler[browser]" # + browser (stock Playwright)
pip install "graph-crawler[antidetect]" # + TLS, patchright, CloakBrowser, nodriver
| extra | what it adds |
|---|---|
browser |
driver="playwright" with bundled Chromium (playwright = alias) |
tls |
curl_cffi — TLS fingerprint tier (driver="tls") |
stealth |
patchright + playwright-stealth — engine "patchright" |
cloak |
cloakbrowser — engine "cloakbrowser" (stealth in the binary) |
undetected |
nodriver — last tier (driver="nodriver") |
antidetect |
all four above |
server |
FastAPI dashboard / REST / WebSocket (graph_crawler.api.server) |
text |
tiktoken + stop-words — count_tokens, extract_keywords |
markdown |
html2md-clean + markdown-it-py — MarkdownPlugin |
mongodb, postgresql, celery, viz, articles, fuzzy, performance, system, all |
storage backends, distributed mode, extras |
After installing a browser extra, run once:
playwright install chromium # stock Playwright / patchright
python -m cloakbrowser install # CloakBrowser binary (~200 MB)
# or point to an existing binary: export CLOAKBROWSER_BINARY=/opt/cloak/chrome
Python 3.11+.
Quickstart
import graph_crawler as gc
graph = gc.crawl("https://example.com", max_depth=2, max_pages=50)
print(len(graph.nodes), "pages")
for node in graph.nodes.values():
print(node.depth, node.response_status, node.url)
hits = graph.search("pricing")
tree = graph.url_tree()
graph.export_nodes("nodes.json")
graph.export_edges("edges.json")
Async:
graph = await gc.async_crawl("https://example.com", driver="escalating")
Reuse one crawler for many runs (keeps the driver and storage open):
with gc.Crawler(driver="escalating", storage="sqlite", storage_config={"path": "site.db"}) as c:
g1 = c.crawl("https://a.example")
g2 = c.crawl("https://b.example")
Only some pages, with rules:
from graph_crawler import URLRule
graph = gc.crawl(
"https://shop.example",
url_rules=[URLRule(pattern=r"/product/", should_scan=True),
URLRule(pattern=r"/cart", should_scan=False)],
)
Sitemap first:
graph = gc.crawl_sitemap("https://example.com/sitemap.xml", max_urls=500)
CLI:
graph-crawler crawl https://example.com --max-depth 2 --driver escalating --save example
Drivers
driver= accepts a name or a driver instance.
| name | tier | needs | notes |
|---|---|---|---|
http / async |
http | — | aiohttp, default |
tls / curl_cffi |
tls | [tls] |
Chrome TLS fingerprint, no JS |
playwright |
browser | [playwright] |
real browser; engine chosen by browser_engine |
nodriver |
undetected | [undetected] |
CDP without webdriver, async |
escalating |
all of them | whatever is installed | http → tls → browser → undetected |
escalating builds tiers lazily: a browser is only started when a cheaper
tier is blocked. Tiers whose dependency is missing are skipped.
Anti-detect: escalation and browser engines
How escalation decides
Every response goes through classify_block():
| result | when |
|---|---|
none |
normal page |
soft_block |
429, plain 503, or 200 with an empty body |
challenge |
Cloudflare "Just a moment", cf-mitigated: challenge, Incapsula, DDoS-Guard |
hard_block |
403 "Access Denied", Akamai _abck, DataDome headers |
captcha |
PerimeterX, DataDome captcha, reCAPTCHA / hCaptcha / Turnstile markup |
Anything other than none (and network errors) moves to the next tier.
The tier that worked is remembered per host, and the memory is adaptive:
- it expires after
GC_TIER_TTLseconds (default 1200 — about the lifetime of acf_clearancecookie); after that the host is probed from the cheap tier again, so you do not stay on a browser once the protection is gone; - a block on the remembered tier drops the entry and escalates from the next tier;
- only one request per host escalates at a time, the others wait for its result (no herd of browsers against one challenge);
- the memory is per process. Workers do not share it.
Each response carries what happened in FetchResponse.extras:
resp.extras["bot_block_status"] # "none" | "soft_block" | "challenge" | "hard_block" | "captcha"
resp.extras["channel"] # "http" | "tls" | "browser" | "undetected"
resp.extras["driver"] # "aiohttp" | "tls" | "playwright" | "nodriver"
resp.extras["engine"] # "cloakbrowser" | "patchright" | "playwright" | None
resp.extras["escalations"] # how many tiers were tried before this one
Using it
import graph_crawler as gc
# default chain, browser tier uses CloakBrowser if its binary is present
graph = gc.crawl("https://protected.example", driver="escalating")
# pick the browser engine, stop after the browser tier (0=http 1=tls 2=browser 3=undetected)
graph = gc.crawl(
"https://protected.example",
driver="escalating",
driver_config={"browser_engine": "patchright", "max_tier": 2},
)
Directly as a driver:
from graph_crawler.infrastructure.transport.escalation import EscalatingDriver, EscalationPolicy
async with EscalatingDriver(policy=EscalationPolicy.default(browser_engine="cloakbrowser")) as d:
resp = await d.fetch("https://protected.example")
print(resp.status_code, resp.extras["channel"], resp.extras["engine"])
Your own chain:
from graph_crawler.infrastructure.transport import DriverType
from graph_crawler.infrastructure.transport.escalation import EscalationPolicy, EscalationTier
policy = EscalationPolicy(
tiers=[
EscalationTier(channel="tls", driver=DriverType.TLS, config={"impersonate": "chrome"}),
EscalationTier(channel="browser", driver=DriverType.PLAYWRIGHT, engine="cloakbrowser"),
],
remember_per_host=False,
)
Browser engines
The browser tier is stock PlaywrightDriver; the engine decides which
Playwright module and which Chromium binary it launches.
| engine | what it is | falls back to |
|---|---|---|
cloakbrowser |
Chromium with fingerprint patches compiled into the binary | patchright |
patchright |
Playwright fork without automation leaks (Runtime.enable …) |
playwright |
playwright |
stock | — |
Fallback is automatic (one INFO log line with the reason). A silent downgrade
to a weaker stealth is dangerous for anti-detect profiles, so there is a strict
mode: GC_ENGINE_STRICT=1 (or {"browser_engine": {"strict": True}},
driver_config={"engine_strict": True} for escalating) raises ImportError
instead of falling back. Choose the engine:
gc.crawl(url, driver="playwright", driver_config={"browser_engine": {"type": "cloakbrowser"}})
or via env GC_BROWSER_ENGINE=cloakbrowser (applies when no engine is set in config).
Plug in your own binary or Playwright fork without touching the library:
from graph_crawler.infrastructure.transport.playwright.engine import register_engine
register_engine(
"my-chrome",
executable_resolver=lambda: "/opt/my-chrome/chrome", # None → fall back
launch_hook=lambda args: ([*args, "--my-flag"], {"ignore_default_args": ["--enable-automation"]}),
fallback="playwright",
)
gc.crawl(url, driver="escalating", driver_config={"browser_engine": "my-chrome"})
playwright_module="patchright.async_api" can be passed to register_engine
if your engine needs a different Playwright implementation.
Stealth lives in the engine (CloakBrowser / patchright). The library only decides when to escalate and which engine to launch.
Storage
storage="memory" (default), "json", "sqlite", "postgresql", "mongodb".
Large crawls: use sqlite or a server backend — nodes are evicted from RAM
and loaded back on access.
Tests
pip install -e ".[browser,tls,dev]"
pytest # offline, no browser needed
pytest -m manual tests/manual # real sites, network required
1 maintainer
Metadata
1 maintainer
Release files for graph-crawler 5.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| graph_crawler-5.2.0.tar.gz | 1.0 MB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| graph_crawler-5.2.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 2.3 MB
Release files / graph_crawler-5.2.0.tar.gz
| Download URL | graph_crawler-5.2.0.tar.gz |
|---|---|
| Size | 1.0 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
983815c1eea9835cdc29624a814272018c62fd4c761cb699f2def5f3809c8322
|
|
BLAKE2b-256 checksum How to use checksums |
85dc812bd3c2590a30c82ea7af3ccb044d651a6cc3195f529b14204482d3e2e8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.7
|
Release files / graph_crawler-5.2.0-py3-none-any.whl
| Download URL | graph_crawler-5.2.0-py3-none-any.whl |
|---|---|
| Size | 1.3 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
50fb8123492ec894d6ecd45314436af1af5c480d73ebdcb49be1e6b6b241476f
|
|
BLAKE2b-256 checksum How to use checksums |
dcc35a720cbaaeca0a389937a164e468cad867925f5e28a5e97b800aefb7128c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.7
|
Security for graph-crawler 5.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
