url22md

Convert HTTP(S) URLs to clean Markdown files. Uses seven extraction tools with directed fallback chains, scores quality, and keeps the best result. Handles hundreds of URLs concurrently, tracks progress in a crash-safe JSONL report, and skips already-processed URLs on restart.

Install

pip install url22md

For full functionality (JS-rendered pages), also run:

playwright install chromium
crawl4ai-setup

Quick start

Single URL:

url22md --url "https://example.com/article"

Batch from file:

url22md --urls_path urls.txt --output_dir ./output

Pipe from stdin:

cat urls.txt | url22md --output_dir ./output

Force a specific tool:

url22md --url "https://spa-heavy-site.com" --tool 3

How it works

url22md tries extraction tools following a directed fallback chain. Each tool has a specific next-tool on failure (not a simple linear sequence). After each attempt it scores the Markdown on prose word count, sentence structure, and penalises CSS/JS boilerplate. The first result scoring above the quality threshold (0.5) is accepted. If nothing meets the threshold, the best result is kept anyway.

Tools 1-5, 8-9 run locally. Tools 6 and 7 call cloud APIs and require API keys. Tool 8 uses crawl4ai with stealth mode and anti-bot features (magic, user simulation, navigator override). Tool 9 adds a PruningContentFilter for article-only fit_markdown output. The default cascade starting from tool 1 follows: 1 → 5 → 6 → 7 → 2 (stop).

CLI reference

url22md [flags]

Input (at least one required):

Output:

Output formats:

Extraction control:

Housekeeping:

JSONL report

Each record contains:

{"url": "https://example.com", "filename": "example-com.md", "tool": "trafilatura", "success": true, "quality": 0.7, "error": null, "timestamp": "2026-03-26T22:15:00+00:00"}

With --format json or --format -, a "markdown" key with the full content is included. On the next run, URLs already in the report are skipped. Use --clean to start fresh.

Python API

run_conversion() returns a list of result records. Without format, no files are written:

from url22md import run_conversion

records = run_conversion(
    urls=["https://example.com", "https://docs.python.org/3/"],
    concurrency=10,
)
for rec in records:
    print(rec["url"], rec["tool"], rec["quality"])
    print(rec["markdown"][:200])

Each record dict has: url, filename, tool, success, quality, error, markdown, timestamp.

To also write files, pass format:

records = run_conversion(
    urls=["https://example.com"],
    output_dir="./output",
    format="md",       # writes individual .md files + JSONL report
)

Lower-level async access:

import asyncio
from url22md import convert_single_url

async def main():
    result = await convert_single_url("https://example.com", proxy_url=None)
    print(result.tool_name, result.quality_score)
    print(result.markdown)

asyncio.run(main())

Individual tool functions:

import asyncio
from url22md.tools import extract_with_trafilatura, extract_with_playwright

async def main():
    result = await extract_with_trafilatura("https://example.com")
    if not result.success:
        result = await extract_with_playwright("https://example.com")
    print(result.markdown)

asyncio.run(main())

Quality scoring:

from url22md import assess_quality

score = assess_quality("# Title\n\nA paragraph.\n\nAnother paragraph.")
print(score)  # 0.6

Proxy support

url22md supports Webshare proxies. Set these environment variables:

export WEBSHARE_PROXY_USER="your_user"
export WEBSHARE_PROXY_PASS="your_pass"
export WEBSHARE_DOMAIN_NAME="proxy.webshare.io"
export WEBSHARE_PROXY_PORT="80"

Then pass --proxy:

url22md --url "https://geo-restricted-site.com" --proxy

Cloud API keys

For tools 4 and 5, set the corresponding environment variable:

export FIRECRAWL_API_KEY="fc-..."    # tool 4
export JINA_API_KEY="jina_..."       # tool 5

These tools are only attempted when their API key is present. Without them, the cascade skips to the next tool.

Output structure

output_dir/
  example-com.md                     # Markdown content
  docs-python-org-3.md               # Markdown content
  _url2md.jsonl                      # Progress report

Filenames are generated from URLs using slugify + pathvalidate, producing filesystem-safe names like example-com-article-id-42.md.

Development

git clone https://github.com/twardoch/url22md
cd url22md
uv pip install --system -e .
playwright install chromium

# Tests (48 unit tests, no network required)
uvx hatch test

# Lint
uvx ruff check src/url22md/ tests/

# Build
uvx hatch build

# Publish
uv publish

Versioning is derived from git tags via hatch-vcs. Tag v1.2.3 produces version 1.2.3.

License

Apache 2.0