Markdownee

Also available as: Online playground | npm package CLI & lib | Source code on GitHub | Apify Actor Docs Getting started | Python library help Social Star us on GitHub | Follow us on GitHub

Markdownee is a web scraper that crawls websites and saves their content as Markdown, HTML or plain text for LLMs, retrieval pipelines, and research datasets.

Choose which links to follow, set page and depth limits, and select how much page content to keep. Control tables, links, images, and user comments separately.

• Boilerplate removal uses the native Python implementation of Trafilatura Core, our open-source extraction engine based on Trafilatura.

  The Core in its name means it is reduced to one task: extracting main content by removing boilerplate. Other packages handle output conversion, including Markdown.

  Core's TypeScript engine ports the original Python Trafilatura, with go-trafilatura as a DOM translation aid. Its native Python engine translates that port.

• Crawlee handles crawling and uses Playwright for browser rendering.

• Two language versions — TypeScript and Python: self-host with the npm CLI, npm library, or Python library, or use the hosted Apify Actor. Source code is on GitHub.

• Save image files with the optional image downloading mode.

This package provides the native Python library, using Crawlee Python and Python Trafilatura Core. Python exposes library APIs only; see the language differences for its supported crawl controls.

Install

Use Python 3.12 or newer:

python -m pip install markdownee

Tested PyPI platforms are macOS and Linux on x64/ARM64, and Windows x64. Native Windows ARM64 installation and HTTP fetching are verified on Python 3.12 and 3.13. impit builds from source there and needs a Rust/C++ build toolchain. 32-bit Windows (x86) is unsupported.

HTTP crawling needs no browser download. For Chromium crawling, install the Playwright browser:

python -m playwright install chromium

Firefox has its own setup in Python Help. The package installs its dependencies, including pyvips[binary]; platforms without its binary support need a compatible libvips installation.

Extract a page

Save as extract.py, then run python extract.py:

from markdownee import fetch

contents = fetch(
    "https://docs.python.org/3/tutorial/introduction.html",
    crawler_type="http",
    formats=["markdown"],
)
print(contents["markdown"])

The program prints the extracted article. fetch() follows no links; unavailable formats are omitted and request failure raises MarkdowneeError. Format selectors are txt, markdown, html, minified-html, and original; compact HTML uses the result key minified_html. Original is captured input, not cleaned output.

Use afetch() in async applications. crawl() and acrawl() write a fresh run directory and return a CrawlSummary with output and manifest paths. Image saving belongs to these file-producing calls; fetch() rejects it.

See Python Help for complete examples, options, browser controls, limits, cancellation, and language differences.

Why Markdownee

Optional image downloading keeps local assets with extracted content.

Trafilatura Core removes boilerplate. Markdownee's TypeScript and Python versions each use the matching Core implementation. Core's TypeScript engine ports original Python Trafilatura, with go-trafilatura as a DOM translation aid. Its native Python engine translates that port. Crawlee drives Playwright for browser rendering.

Use the npm CLI, npm library, native Python library, or hosted Apify Actor.

Support

Report problems through the issue tracker.

License

Licensed under Apache-2.0.