contextractor

Also available as: Online playground playground, help | Apify Actor actor, help | NPM package CLI & lib package, CLI help, lib help | Source code on GitHub Social: X (Twitter)

Drive the Contextractor Node crawler/extractor from Python; clean main content in txt/markdown/json/html, plus raw original HTML.

⚠️ Alpha / experimental. This Python wrapper for Contextractor is currently an alpha, experimental release — not fully tested or officially supported, though still maintained.

Crawl web pages and extract clean main-content text — txt, markdown, json, html, or raw original HTML — from Python. Built on rs-trafilatura (extraction) and Crawlee with Playwright or a pure-HTTP Cheerio crawler (crawling).

This package is a thin, typed wrapper that drives the bundled Node CLI. A self-contained Node runtime is installed automatically as a dependency (nodejs-wheel-binaries), so no separate Node.js install is required — but no Python code touches the extraction engine itself.

Install

pip install contextractor
python -m contextractor install   # one-time: download the browser

Platform wheels are published for macOS (arm64, x86_64), Linux (x86_64, aarch64; glibc ≥ 2.28), and Windows (x64). Requires Python 3.12+.

Quick start

extract_one() fetches exactly one URL (no link-following) and returns the extracted content directly; the cheerio crawler works before any browser download:

>>> import contextractor
>>> md = contextractor.extract_one(
...     "https://www.iana.org/help/example-domains",
...     crawler_type="cheerio",
... )
>>> md.splitlines()[0]
'# Example Domains'

With one requested format the return value is a str; with several it is a dict[str, str]:

>>> contents = contextractor.extract_one(
...     "https://www.iana.org/help/example-domains",
...     formats=["markdown", "json", "original"],
...     crawler_type="cheerio",
... )
>>> sorted(contents)
['json', 'markdown', 'original']

aextract_one() is the async variant.

Crawling multiple pages

extract() crawls one or more URLs (with link-following) and writes extracted files plus a manifest.json index to output_dir:

import contextractor

summary = contextractor.extract(
    ["https://en.wikipedia.org/wiki/Web_scraping"],
    save=["markdown-kvs"],
    output_dir="./out",
)
print(summary.succeeded, "of", summary.total)

Each save token is <format>-<kvs|dataset> — the format name plus the destination — so it reads markdown-kvs, not bare markdown (the bare names like markdown and json are what extract_one(formats=[...]) takes instead).

An async variant, aextract(), takes the same arguments. Partial failures do not raise — they are reflected in summary.failed; validation and real crawl failures raise ContextractorError, and a missing browser raises MissingBrowserError pointing you at python -m contextractor install.

Options

Every crawl option is a typed keyword argument; the ExtractOptions / ExtractOneOptions TypedDicts are the authoritative list of keywords and accepted values, surfaced to editors and type checkers. Each snake_case keyword maps one-to-one onto a flag of the npm CLI:

• max_crawl_depth=3 → --max-crawl-depth 3
• headless=False → --no-headless
• save=["markdown-kvs", "json-dataset"] → --save markdown-kvs --save json-dataset

See the Python documentation for the full keyword list. Only http, https, socks4, and socks5 proxy URLs are accepted; credentials are never echoed in errors or logs. Set CONTEXTRACTOR_NODE_PATH to use a host Node binary instead of the bundled runtime.

Why Contextractor

Contextractor ships the Rust port of Trafilatura as a native (napi-rs) binding — no Python extraction runtime. On the Scrapinghub article set it scores an F1 of 0.966 (precision 0.942, recall 0.991) — ahead of go-trafilatura (0.960) and the original Python Trafilatura (0.958); see the benchmark write-up for the methodology.

It is free and open source (Apache-2.0), runs locally with no API key and no per-page credits, and its Markdown output is typically 80–90% fewer tokens than the raw HTML — cheap to feed to an LLM.

Contributing

Issues and pull requests are welcome at the issue tracker. The extraction engine, npm CLI, and Apify Actor all live in the same source repository.

License

Apache-2.0