contextractor 0.4.4

Drive the Contextractor Node crawler/extractor from Python — clean main-content text in txt, markdown, json, or html.

contextractor

Crawl web pages and extract clean main-content text — txt, markdown, json, or html — from Python. Built on rs-trafilatura (extraction) and Crawlee + Playwright (crawling).

This package is a thin, typed wrapper that drives the bundled Node engine — a self-contained Node runtime ships with the wheel, so no Node.js install is required.

Install

pip install contextractor
python -m contextractor install   # one-time: download the Chromium browser

Platform wheels are published for macOS (arm64, x86_64), Linux (x86_64, aarch64; glibc ≥ 2.28), and Windows (x64). Requires Python 3.12+.

Quick start

Crawl one or more URLs and write extracted files plus a manifest.json index to output_dir:

import contextractor

summary = contextractor.extract(
    ["https://example.com"],
    save=["markdown-kvs"],
    output_dir="./out",
)
print(summary.succeeded, "of", summary.total)

An async variant, aextract(), takes the same arguments. Partial failures do not raise — they are reflected in summary.failed; validation and real crawl failures raise ContextractorError, and a missing browser raises MissingBrowserError pointing you at python -m contextractor install.

Single-page extraction

extract_one() crawls exactly one URL (no link-following) and returns the extracted content directly — nothing is written to disk:

import contextractor

markdown = contextractor.extract_one("https://example.com")          # -> str

contents = contextractor.extract_one(                                # -> dict[str, str]
    "https://example.com",
    formats=["markdown", "json", "original"],
)

With one requested format the return value is a str; with several it is a dict[str, str]. aextract_one() is the async variant.

Options

Every crawl option is a typed keyword argument, surfaced to editors and type checkers via the ExtractOptions / ExtractOneOptions TypedDicts (e.g. save, mode, proxy, cookies, headers, max_crawl_depth, globs). For the full, authoritative option reference see the project documentation and CLI help (contextractor extract --help); the Python keywords map one-to-one onto the CLI flags.

Only http, https, socks4, and socks5 proxy URLs are accepted; credentials are never echoed in errors or logs. Set CONTEXTRACTOR_NODE_PATH to use a host Node binary instead of the bundled runtime.

License

Apache-2.0