contextractor 0.4.14

Drive the Contextractor Node crawler/extractor from Python; clean main content in txt/markdown/json/html, plus raw original HTML.

contextractor

Also available as: Online playground playground, help | Apify Actor actor, help | NPM package CLI & lib package, CLI help, lib help | Source code on GitHub Social: X (Twitter)

Drive the Contextractor Node crawler/extractor from Python; clean main content in txt/markdown/json/html, plus raw original HTML.

⚠️ Alpha / experimental. This Python wrapper for Contextractor is currently an alpha, experimental release — not fully tested or officially supported, though still maintained.

Crawl web pages and extract clean main-content text — txt, markdown, json, html, or raw original HTML — from Python. Built on rs-trafilatura (extraction) and Crawlee with Playwright or a pure-HTTP Cheerio crawler (crawling).

This package is a thin, typed wrapper that drives the bundled Node CLI. A self-contained Node runtime is installed automatically as a dependency (nodejs-wheel-binaries), so no separate Node.js install is required — but no Python code touches the extraction engine itself.

Install

pip install contextractor
python -m contextractor install   # one-time: download the browser

Platform wheels are published for macOS (arm64, x86_64), Linux (x86_64, aarch64; glibc ≥ 2.28), and Windows (x64). Requires Python 3.12+.

Quick start

extract_one() fetches exactly one URL (no link-following) and returns the extracted content directly; the cheerio crawler works before any browser download:

>>> import contextractor
>>> md = contextractor.extract_one(
...     "https://www.iana.org/help/example-domains",
...     crawler_type="cheerio",
... )
>>> md.splitlines()[0]
'# Example Domains'

With one requested format the return value is a str; with several it is a dict[str, str]:

>>> contents = contextractor.extract_one(
...     "https://www.iana.org/help/example-domains",
...     formats=["markdown", "json", "original"],
...     crawler_type="cheerio",
... )
>>> sorted(contents)
['json', 'markdown', 'original']

aextract_one() is the async variant.

Crawling multiple pages

extract() crawls one or more URLs (with link-following) and writes extracted files plus a manifest.json index to output_dir:

import contextractor

summary = contextractor.extract(
    ["https://en.wikipedia.org/wiki/Web_scraping"],
    save=["markdown-kvs"],
    output_dir="./out",
)
print(summary.succeeded, "of", summary.total)

Each save token is <format>-<kvs|dataset> — the format name plus the destination — so it reads markdown-kvs, not bare markdown (the bare names like markdown and json are what extract_one(formats=[...]) takes instead).

An async variant, aextract(), takes the same arguments. Partial failures do not raise — they are reflected in summary.failed; validation and real crawl failures raise ContextractorError, and a missing browser raises MissingBrowserError pointing you at python -m contextractor install.

Options

Every crawl option is a typed keyword argument; the ExtractOptions / ExtractOneOptions TypedDicts are the authoritative list of keywords and accepted values, surfaced to editors and type checkers. Each snake_case keyword maps one-to-one onto a flag of the npm CLI:

• max_crawl_depth=3 → --max-crawl-depth 3
• headless=False → --no-headless
• save=["markdown-kvs", "json-dataset"] → --save markdown-kvs --save json-dataset

See the Python documentation for the full keyword list. Only http, https, socks4, and socks5 proxy URLs are accepted; credentials are never echoed in errors or logs. Set CONTEXTRACTOR_NODE_PATH to use a host Node binary instead of the bundled runtime.

Why Contextractor

Contextractor ships the Rust port of Trafilatura as a native (napi-rs) binding — no Python extraction runtime. On the Scrapinghub article set it scores an F1 of 0.966 (precision 0.942, recall 0.991) — ahead of go-trafilatura (0.960) and the original Python Trafilatura (0.958); see the benchmark write-up for the methodology.

It is free and open source (Apache-2.0), runs locally with no API key and no per-page credits, and its Markdown output is typically 80–90% fewer tokens than the raw HTML — cheap to feed to an LLM.

	Contextractor	Firecrawl	Jina Reader	Crawl4AI
Extraction engine	rs-trafilatura (heuristic + ML routing)	LLM / heuristic	ReaderLM neural model	LLM / heuristic
Runtime	Rust + Node (no Python engine)	hosted API / self-host	hosted API	Python
Surfaces	Apify Actor · npm CLI · npm library · PyPI	API · SDKs · self-hosted · MCP	API	Python library · crwl CLI · Docker REST API · MCP
Output formats	txt · markdown · json · html · original	markdown · html · etc.	markdown · html · text · screenshot · etc.	markdown · etc.
Crawling	Crawlee + Playwright (adaptive / browser / HTTP)	built-in	none (single URL)	built-in

Contributing

Issues and pull requests are welcome at the issue tracker. The extraction engine, npm CLI, and Apify Actor all live in the same source repository.

License

Apache-2.0