📖 Description

PyStates is a fast, minimalist command-line utility that downloads full-length articles from major publications (The New Yorker, Wired, The Atlantic, and hundreds more) and extracts clean, readable text using the industry-leading Trafilatura library.

It automatically removes ads, navigation menus, sidebars, comments, and boilerplate, then saves the result as either beautifully formatted Markdown (.md) or clean Plain Text (.txt).

Designed for researchers, writers, journalists, students, and power users who want to archive high-quality content locally — without distractions or paywalls interfering with readability.

pystates -u https://www.newyorker.com/magazine/2025/01/20/the-new-ai-era

✨ Features

• 🚀 Blazing Fast — Downloads and processes articles in under 3 seconds
• 🛡️ 403 Bypass — Smart custom User-Agent headers to reliably avoid common blocks
• 🧼 Pristine Extraction — Powered by Trafilatura for the cleanest possible text
• 🚫 Furniture Removal — "related articles", newsletter and promo blocks are pruned before extraction, so the file contains only the article (override with --no-prune)
• 📝 Four Output Formats — Markdown (.md), Plain Text (.txt), Word (.docx) and PDF (.pdf)
• 🧹 Intelligent Filename Sanitization — Generates safe OS filenames from article titles without using re
• ✍️ Custom Filenames — Override auto-generated names with the -o flag
• 🐍 Modern Python Stack — Built for Python 3.10+ with clean architecture
• 📦 One-Command Installation — pip install -e . for instant local development
• 🌐 Broad Compatibility — Works with virtually any news, magazine, or blog site supported by Trafilatura

🚀 Installation (Local Dev Mode)

# 1. Clone the repository
git clone https://github.com/Maksum867/PyStates.git
cd pystates

# 2. Install in editable/development mode
pip install -e .

Optional: Word and PDF output

md and txt work out of the box. Word and PDF output need extra dependencies:

pip install "pystates-CLI[word]"   # .docx  (python-docx)
pip install "pystates-CLI[pdf]"    # .pdf   (reportlab)
pip install "pystates-CLI[word,pdf]"  # both

If a format is requested without its dependency installed, the CLI exits with a clear message telling you which extra to install.

Verify Installation

pystates --help

Requirements

• Python ≥ 3.9
• pip

All dependencies (requests, trafilatura) are installed automatically via pyproject.toml. argparse ships with Python.

🖥️ Usage

Command Syntax

pystates -u URL [OPTIONS]

CLI Flags

Examples

1. Basic usage (Markdown, auto-generated filename)

pystates -u "https://www.newyorker.com/magazine/2025/01/20/the-new-ai-era"

→ Creates the-new-ai-era.md

2. Save as plain text

pystates -u "https://www.wired.com/story/the-future-of-coding/" -f txt

→ Creates the-future-of-coding.txt

3. Custom output filename

pystates -u "https://www.theatlantic.com/ideas/archive/2025/07/climate-change/" \
         -o "climate_report_july_2025" \
         -f md

→ Creates climate_report_july_2025.md

4. Full-featured command

pystates \
  --url "https://www.newyorker.com/culture/cultural-comment/the-end-of-the-internet-as-we-know-it" \
  --format txt \
  --output "end_of_internet"

5. Export to Word or PDF

pystates -u "https://www.newyorker.com/books/page-turner/what-is-biography-for" -f docx
pystates -u "https://en.wikipedia.org/wiki/Ada_Lovelace" -f pdf

→ Creates What_Is_Biography_For.docx / Ada_Lovelace.pdf, with a proper title, byline (author · date · publication) and one paragraph per source paragraph.

6. View all options

pystates --help

Sample Output (Markdown)

# The New AI Era

*By Jane Smith · 2025-01-20 · The New Yorker*

<https://www.newyorker.com/culture/cultural-comment/the-end-of-the-internet-as-we-know-it>

---

Artificial intelligence is no longer a distant promise...

## A New Paradigm

The implications are profound...

📁 Project Structure

PyStates/
├── pystates/
│   ├── __init__.py          # Package metadata & version
│   ├── __main__.py          # Entry point for `python -m pystates`
│   ├── cli.py               # Argument parsing (argparse)
│   ├── exporters.py         # Filename sanitization + md/txt/docx/pdf export
│   └── parser.py            # HTTP fetching (custom UA) + Trafilatura parsing
│   └── tests/               # 75 pytest tests (unit + real-HTTP integration)
├── .github/
│   ├── workflows/
│   │   ├── tests.yml        # pytest matrix, lint, packaging
│   │   └── publish.yml      # build + publish to PyPI (Trusted Publishing)
│   └── dependabot.yml       # automatic dependency updates
├── pyproject.toml           # Modern packaging + entry points
├── CHANGELOG.md
├── .gitignore
└── README.md

⚠️ Known limitations

• JavaScript-only pages are not supported. PyStates downloads the HTML the server sends; sites that render the article in the browser (client-side React apps) may return a listing page or a paywall stub instead of the article. When the extracted text is suspiciously short, the CLI prints a warning on stderr and still writes the file.
• Bot protection blocks some sites. With a plain HTTP client, sites behind Cloudflare/Akamai may answer 403/401. PyStates reports the HTTP error and exits with status 1.
• PDF fonts. A Unicode TTF (DejaVu/Liberation) is used when present on the system, so Cyrillic and CJK render correctly on most Linux images and macOS; elsewhere ReportLab falls back to Helvetica, which only covers Latin-1.
• The default User-Agent is a fixed desktop Chrome string. It is not rotated, and it is not a "403 bypass".

🧪 Development & Testing

git clone https://github.com/Maksum867/PyStates.git
cd PyStates
pip install -e ".[dev]"
python -m pytest            # 75 tests, no network access required
ruff check . && ruff format --check .

The suite includes a real end-to-end run against a local http.server, so it verifies the whole pipeline (download → extract → sanitize → write) without touching the internet.

Releasing

1. Bump the version in pyproject.toml and pystates/__init__.py.
2. Update CHANGELOG.md.
3. Publish a GitHub Release tagged vX.Y.Z.
4. .github/workflows/publish.yml runs the tests, builds the distributions and publishes to PyPI via Trusted Publishing (no API token needed).

📜 License

This project is released under the MIT License.

See LICENSE for the full text.

PyStates — preserving the web's best writing, one clean file at a time.

Built with care by the open-source community.