Pure-Python article extraction library and HTTP service - Drop-in replacement for readability-js-server

Navigation

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for psjinx from gravatar.com psjinx

Unverified details

These details have not been verified by PyPI
Meta
  • License: MIT License (MIT)
  • Author: Pankaj Kumar Singh
  • Tags article , extraction , html , markdown , readability
  • Requires: Python >=3.12
  • Provides-Extra: all , dev , httpx , playwright , server , ua
Classifiers
Report project as malware

Project description

Article Extractor

PyPI version Python versions License: MIT CI codecov

High-fidelity article extraction in pure Python: library, HTTP API, and CLI that turn messy web pages into clean Markdown or HTML for ingestion, archiving, and LLM pipelines.

Requires Python 3.12+

Who This Helps

  • Backend, data, and tooling teams that need reliable article text for search, RAG, or analytics.
  • Engineers who prefer a single-language stack with fast installs and reproducible results.
  • Teams that want a ready-to-ship server/CLI and a composable Python API.

Quick Start (pick one)

CLI (fastest)

pip install article-extractor
article-extractor https://en.wikipedia.org/wiki/Wikipedia --output markdown

You will see clean Markdown printed to stdout along with detected title, excerpt, and word count.

Server (Docker)

docker run -p 3000:3000 ghcr.io/pankaj28843/article-extractor:latest
curl -XPOST http://localhost:3000/ \
    -H "Content-Type: application/json" \
    -d '{"url": "https://en.wikipedia.org/wiki/Wikipedia"}' | jq '.title, .word_count'

Override cache + Playwright storage in Docker

  • docker run -e lets you pass environment variables into the container so you can raise or lower the LRU cache limit (ARTICLE_EXTRACTOR_CACHE_SIZE, ARTICLE_EXTRACTOR_THREADPOOL_SIZE, etc.) without rebuilding images Docker container run – env #techdocs.
  • Use -v/--volume to mount host directories and persist assets like the Playwright storage-state file between runs Docker container run – volume #techdocs.
  • Example: keep Playwright cookies on the host and increase the cache to 2k entries while running the published image:
docker run --rm -p 3000:3000 \
    -e ARTICLE_EXTRACTOR_CACHE_SIZE=2000 \
    -e PLAYWRIGHT_STORAGE_STATE_FILE=/data/storage-state.json \
    -v $HOME/.article-extractor:/data \
    ghcr.io/pankaj28843/article-extractor:latest

Mounting the host directory ensures /data/storage-state.json survives container restarts, so Playwright-headed sessions can defeat bot checks once and reuse the same cookies later.

Python Library

from article_extractor import extract_article

html = """
<html><body><article><h1>Title</h1><p>Content...</p></article></body></html>
"""

result = extract_article(html, url="https://example.com/demo")
print(result.title)
print(result.markdown.splitlines()[0])
print(f"Words: {result.word_count}")

Why Teams Choose Article Extractor

  • Accuracy first: Readability-style scoring tuned for long-form content and docs.
  • Clean output: GFM-ready Markdown and sanitised HTML safe for downstream pipelines.
  • Speed at scale: Caching plus early-termination heuristics keep typical pages in 50–150 ms.
  • Drop-in everywhere: Same engine across CLI, HTTP server, and Python API.
  • Test coverage that catches regressions before you do.

Installation

pip install article-extractor[server]  # HTTP server
pip install article-extractor[all]     # All optional extras

# Or with uv (fast installs)
uv add article-extractor --extra server

HTTP Server

# Local
uvicorn article_extractor.server:app --host 0.0.0.0 --port 3000

# Docker
docker run -d -p 3000:3000 --name article-extractor \
    --restart unless-stopped ghcr.io/pankaj28843/article-extractor:latest

Endpoints:

  • POST / — Extract article ({"url": "..."})
  • GET / — Service info
  • GET /health — Health check
  • GET /docs — Interactive OpenAPI UI

CLI

# Extract from URL
article-extractor https://en.wikipedia.org/wiki/Wikipedia

# From file
article-extractor --file article.html --output markdown

# One-off via Docker
docker run --rm ghcr.io/pankaj28843/article-extractor:latest \
    article-extractor https://en.wikipedia.org/wiki/Wikipedia --output text

Networking controls (CLI & Server)

  • Honor corporate proxies automatically: HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, and NO_PROXY are folded into every fetcher, or override them with --proxy=http://user:pass@proxy:8080 and optional --prefer-httpx/--prefer-playwright flags.
  • Rotate or pin User-Agents: pass --user-agent for deterministic runs or --random-user-agent to synthesize realistic desktop headers with fake-useragent (pip install article-extractor[ua] or use the all/server extras). Disable later via --no-random-user-agent.
  • Handle bot challenges: --headed --user-interaction-timeout 30 launches Chromium with a visible window, pauses for manual CAPTCHA solving, and persists cookies to ~/.article-extractor/storage_state.json (override with --storage-state).
  • Docker and server mode forward the same defaults. When running the FastAPI server, the CLI seeds these values so all requests inherit them unless the POST body supplies {"prefer_playwright": false, "network": {"proxy": "http://", "headed": true, ...}}.

Server POST example with overrides:

{
    "url": "https://example.com/paywalled",
    "prefer_playwright": true,
    "network": {
        "user_agent": "MyMonitor/1.0",
        "random_user_agent": false,
        "proxy": "http://proxy.internal:8080",
        "proxy_bypass": ["metadata.internal"],
        "headed": true,
        "user_interaction_timeout": 25,
        "storage_state": "/var/lib/article-extractor/storage_state.json"
    }
}

Python API

from article_extractor import extract_article, extract_article_from_url, ExtractionOptions

options = ExtractionOptions(min_word_count=120, include_images=True)
result = extract_article("<html>...</html>", url="https://example.com", options=options)

print(result.title)
print(result.excerpt)
print(result.success)

ArticleResult fields: title, content, markdown, excerpt, word_count, success, error, url, author, date_published, language, warnings.

Use Cases

  • LLM and RAG ingestion with clean Markdown ready for embeddings.
  • Content archiving and doc syncing without ads or cruft.
  • RSS/feed readers and knowledge tools that need readable HTML.
  • Research pipelines that batch-extract large reading lists.

How It Works

  1. Parse HTML with JustHTML.
  2. Strip noise (scripts, nav, styles) and find content candidates.
  3. Score candidates with a Readability-inspired model (density, link ratio, structure hints).
  4. Pick the winner, normalise headings/links, and emit clean HTML.
  5. Convert to GFM-compatible Markdown for downstream tools.

Configuration

HOST=0.0.0.0
PORT=3000
LOG_LEVEL=info
WEB_CONCURRENCY=2
ARTICLE_EXTRACTOR_CACHE_SIZE=1000
ARTICLE_EXTRACTOR_THREADPOOL_SIZE=0

Troubleshooting

  • JavaScript-heavy sites: pip install article-extractor[playwright] and try again.
  • Empty or short output: lower min_word_count or inspect result.warnings.
  • Ports busy: lsof -i :3000 then restart with --port 8000.

Development

git clone https://github.com/pankaj28843/article-extractor.git
cd article-extractor
uv sync --all-extras
uv run ruff format . && uv run ruff check --fix .
uv run pytest tests/ -v

Contributing

See CONTRIBUTING.md for setup, coding standards, and the full validation loop. PRs with tests and doc improvements are welcome.

License

MIT — see LICENSE

Acknowledgments

Project details

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for psjinx from gravatar.com psjinx

Unverified details

These details have not been verified by PyPI
Meta
  • License: MIT License (MIT)
  • Author: Pankaj Kumar Singh
  • Tags article , extraction , html , markdown , readability
  • Requires: Python >=3.12
  • Provides-Extra: all , dev , httpx , playwright , server , ua
Classifiers

Release history Release notifications | RSS feed

0.5.9

0.5.8

0.5.7

0.5.6

0.5.5

0.5.4

0.5.3

0.5.2

0.5.1

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

This version

0.3.0

0.2.1

0.2.0

0.1.2

0.1.1

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

article_extractor-0.3.0.tar.gz (115.4 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

article_extractor-0.3.0-py3-none-any.whl (31.1 kB view details)

Uploaded Python 3

File details

Details for the file article_extractor-0.3.0.tar.gz.

File metadata

  • Download URL: article_extractor-0.3.0.tar.gz
  • Upload date:
  • Size: 115.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: uv/0.9.21 {"installer":{"name":"uv","version":"0.9.21","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

File hashes

Hashes for article_extractor-0.3.0.tar.gz
Algorithm Hash digest
SHA256 db1a93b83e251fe40b911da8574634c7abe38b4b24b467fe36a63ec50c58f969
MD5 aea74d77d8faefb047f5c46e6c142dec
BLAKE2b-256 9e5306b066f573749641ef04a03304291e13eb72e8c36e4df03ae4b29c27fca3

See more details on using hashes here.

File details

Details for the file article_extractor-0.3.0-py3-none-any.whl.

File metadata

  • Download URL: article_extractor-0.3.0-py3-none-any.whl
  • Upload date:
  • Size: 31.1 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: uv/0.9.21 {"installer":{"name":"uv","version":"0.9.21","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

File hashes

Hashes for article_extractor-0.3.0-py3-none-any.whl
Algorithm Hash digest
SHA256 a6ff4fc39bb6a9a527468c13bc8e036dbe6445a8136475a60f34f06ff5374581
MD5 fd429a5319d5fa0538e4276cf6daeb04
BLAKE2b-256 fe42ff41d180272a9a738bd20b4270c0a7ea7d3bc9c9adb789f5b4dbc3914ba3

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page