Markdownee

Also available as: Online playground | Apify Actor | npm package CLI & lib | Source code on GitHub Docs Getting started | Python library help Social Star us on GitHub | Follow us on GitHub

Markdownee is a web scraper that crawls websites and saves their content as Markdown, HTML or plain text for LLMs, retrieval pipelines, and research datasets.

Choose which links to follow, set page and depth limits, and select how much page content to keep. Control tables, links, images, and user comments separately.

• Boilerplate removal is powered by Trafilatura Core, our open-source fork of Trafilatura.

  The Core in its name means it is reduced to one task: extracting main content by removing boilerplate. Other packages handle output conversion, including Markdown.

  Trafilatura Core is ported from the original Python Trafilatura, with go-trafilatura as a DOM translation aid.

• Crawlee handles crawling and uses Playwright for browser rendering.

• Two language versions — TypeScript and Python: self-host with the npm CLI, npm library, or Python library, or use the hosted Apify Actor. Source code is on GitHub.

• Save image files with the optional image downloading mode.

This package provides the native Python library, using Crawlee Python and Python Trafilatura Core. Python exposes library APIs only; see the language differences for its supported crawl controls.

Install

Use Python 3.12 or newer:

python -m pip install markdownee

Tested PyPI platforms are macOS and Linux on x64/ARM64, and Windows x64. Native Windows ARM64 support is planned but currently blocked by impit packaging; 32-bit Windows (x86) is unsupported.

HTTP crawling needs no browser download. For Chromium crawling, install the Playwright browser:

python -m playwright install chromium

Firefox has its own setup in Python Help. The package installs its dependencies, including pyvips[binary]; platforms without its binary support need a compatible libvips installation.

Extract a page

Save as extract.py, then run python extract.py:

from markdownee import fetch

contents = fetch(
    "https://docs.python.org/3/tutorial/introduction.html",
    crawler_type="http",
    formats=["markdown"],
)
print(contents["markdown"])

The program prints the extracted article. fetch() follows no links; unavailable formats are omitted and request failure raises MarkdowneeError. Format selectors are txt, markdown, html, minified-html, and original; compact HTML uses the result key minified_html. Original is captured input, not cleaned output.

Use afetch() in async applications. crawl() and acrawl() write a fresh run directory and return a CrawlSummary with output and manifest paths. Image saving belongs to these file-producing calls; fetch() rejects it.

See Python Help for complete examples, options, browser controls, limits, cancellation, and language differences.

Why Markdownee

Choose what to fetch, which content to retain, and where to save it. Page limits, link filters, and separate controls for images, tables, links, and user comments keep those choices explicit. Optional image downloading keeps local assets with extracted content.

Trafilatura Core ports original Python Trafilatura, with go-trafilatura as a DOM translation aid. Crawlee drives Playwright for browser rendering.

Use the npm CLI, npm library, native Python library, or hosted Apify Actor.

Support

Report problems through the issue tracker.

License

Licensed under Apache-2.0.