contextractor 0.4.13

Drive the Contextractor Node crawler/extractor from Python; clean main content in txt/markdown/json/html, plus raw original HTML.

contextractor

Also available as: Online playground playground, help | Apify Actor actor, help | NPM package CLI & lib package, CLI help, lib help | Source code on GitHub Social: X (Twitter)

Drive the Contextractor Node crawler/extractor from Python; clean main content in txt/markdown/json/html, plus raw original HTML.

⚠️ Alpha / experimental. This Python wrapper for Contextractor is currently an alpha, experimental release — not fully tested or officially supported, though still maintained.

Crawl web pages and extract clean main-content text — txt, markdown, json, html, or raw original HTML — from Python. Built on rs-trafilatura (extraction) and Crawlee with Playwright or a pure-HTTP Cheerio crawler (crawling).

This package is a thin, typed wrapper that drives the bundled Node engine. A self-contained Node runtime is installed automatically as a dependency (nodejs-wheel-binaries), so no separate Node.js install is required.

Contents

• Why Contextractor
• Install
• Quick start
• Single-page extraction
• Options
• Contributing
• License

Why Contextractor

Contextractor ships the Rust port of Trafilatura as a native (napi-rs) binding — no Python extraction runtime. On the Scrapinghub article set it scores an F1 of 0.966 (precision 0.942, recall 0.991) — ahead of go-trafilatura (0.960) and the original Python Trafilatura (0.958); see the benchmark write-up for the methodology.

It is free and open source (Apache-2.0), runs locally with no API key and no per-page credits, and its Markdown output is typically 80–90% fewer tokens than the raw HTML — cheap to feed to an LLM.

	Contextractor	Firecrawl	Jina Reader	Crawl4AI
Extraction engine	rs-trafilatura (heuristic + ML routing)	LLM / heuristic	ReaderLM neural model	LLM / heuristic
Runtime	Rust + Node (no Python engine)	hosted API / self-host	hosted API	Python
Surfaces	Apify Actor · npm CLI · npm library · PyPI	API · SDKs · self-hosted · MCP	API	Python library · crwl CLI · Docker REST API · MCP
Output formats	txt · markdown · json · html · original	markdown · html · etc.	markdown · html · text · screenshot · etc.	markdown · etc.
Crawling	Crawlee + Playwright (adaptive / browser / HTTP)	built-in	none (single URL)	built-in

Install

pip install contextractor
python -m contextractor install   # one-time: download the Chromium browser

Platform wheels are published for macOS (arm64, x86_64), Linux (x86_64, aarch64; glibc ≥ 2.28), and Windows (x64). Requires Python 3.12+.

Quick start

Crawl one or more URLs and write extracted files plus a manifest.json index to output_dir:

import contextractor

summary = contextractor.extract(
    ["https://en.wikipedia.org/wiki/Web_scraping"],
    save=["markdown-kvs"],
    output_dir="./out",
)
print(summary.succeeded, "of", summary.total)

Each save token is <format>-<kvs|dataset> — the format name plus the destination — so it reads markdown-kvs, not bare markdown (the bare names like markdown and json are what extract_one(formats=[...]) takes instead).

An async variant, aextract(), takes the same arguments. Partial failures do not raise — they are reflected in summary.failed; validation and real crawl failures raise ContextractorError, and a missing browser raises MissingBrowserError pointing you at python -m contextractor install.

Single-page extraction

extract_one() crawls exactly one URL (no link-following) and returns the extracted content directly — nothing is written to disk:

import contextractor

markdown = contextractor.extract_one("https://en.wikipedia.org/wiki/Web_scraping")  # -> str

contents = contextractor.extract_one(                                # -> dict[str, str]
    "https://en.wikipedia.org/wiki/Web_scraping",
    formats=["markdown", "json", "original"],
)

With one requested format the return value is a str; with several it is a dict[str, str]. aextract_one() is the async variant.

Options

Every crawl option is a typed keyword argument; the ExtractOptions / ExtractOneOptions TypedDicts are the authoritative list of keywords and accepted values, surfaced to editors and type checkers (e.g. save, mode, proxy, cookies, headers, max_crawl_depth, globs). See also the Python documentation. Each Python keyword maps one-to-one onto a flag of the npm CLI, documented at contextractor.com/help/npm.

Only http, https, socks4, and socks5 proxy URLs are accepted; credentials are never echoed in errors or logs. Set CONTEXTRACTOR_NODE_PATH to use a host Node binary instead of the bundled runtime.

Contributing

Issues and pull requests are welcome at the issue tracker. The extraction engine, npm CLI, and Apify Actor all live in the same source repository.

License

Apache-2.0