A command-line tool designed to solve content preservation challenges with Ethical Scraping.
Project description
Capcat - A command-line tool designed to solve content preservation challenges with Ethical Scraping.
Captures articles from 12 built-in sources as clean Markdown files with optional self-contained HTML output. Includes 5 specialized URL processors for platforms like Medium, YouTube, and X. Supports interactive TUI and batch automation.
Installation
pipx install capcat
Requires Python 3.9+.
Quick Start
# Interactive TUI
capcat catch
# Fetch a bundle
capcat bundle tech --count 10
# Fetch specific sources
capcat fetch hn,bbc --count 15
# Archive a single article
capcat single https://example.com/article
# List available sources
capcat list sources
# Show version
capcat --version
Capcat initializes the vault automatically on first run.
Commands
| Command | Description |
|---|---|
catch |
Launch the interactive TUI |
single <url> |
Archive a single article |
fetch <sources> |
Batch fetch from sources (comma-separated) |
bundle <name> |
Fetch a pre-configured bundle |
list sources |
List all available sources |
list bundles |
List all available bundles |
add-source --url <url> |
Add a custom RSS/news source |
remove-source |
Remove a source |
generate-config |
Generate a YAML config |
init |
Explicitly scaffold vault (runs automatically on first use) |
Options
| Flag | Description |
|---|---|
--count N |
Number of articles to fetch (default: 30) |
--output DIR |
Output directory (default: current dir) |
--media |
Download images, video, audio, and PDF files |
--pdfs |
Download PDF files only (independent of --media) |
--html |
Generate self-contained HTML output |
--update |
Re-fetch and update existing articles |
-V, --verbose |
Verbose output |
-q, --quiet |
Quiet output |
-L <file> |
Log output to file |
--version |
Show version and exit |
--help |
Show help and exit |
Bundles
Pre-configured topic collections:
| Bundle | Sources | Description |
|---|---|---|
tech |
IEEE, Mashable | Consumer technology news |
techpro |
HN, Lobsters, InfoQ | Professional developer news |
ai |
MIT News, Google Research | AI research and developments |
science |
Nature, Scientific American | Scientific publications |
news |
BBC, Guardian | General news |
sports |
BBC Sport | Sports coverage |
Available Sources
Tech Pro: Hacker News (hn), Lobsters (lb), InfoQ (iq)
Tech: IEEE Spectrum (ieee), Mashable (mashable)
AI: Google Research (google-research), MIT News (mitnews)
News: BBC (bbc), The Guardian (guardian)
Science: Nature (nature), Scientific American (scientificamerican)
Sports: BBC Sport (bbcsport)
Custom: Medium, Substack (add via capcat add-source)
Output Structure
Batch mode (fetch / bundle)
News/news_DD-MM-YYYY/
├── Hacker-News_DD-MM-YYYY/
│ ├── 01_Article_Title/
│ │ ├── article.md
│ │ ├── comments.md
│ │ ├── html/
│ │ │ ├── article.html
│ │ │ └── comments.html
│ │ └── images/
│ └── 02_Another_Article/
└── BBC_DD-MM-YYYY/
Single article mode
Capcats/cc_DD-MM-YYYY-Title/
├── article.md
├── html/
│ └── article.html
└── images/
HTML output is fully self-contained - embedded CSS, no external dependencies. Open in any browser, share via email, archive permanently.
Configuration
Optional capcat.yml in your project directory:
output_base_dir: "../MyNews"
max_workers: 8
download_media: false
Config priority: CLI flag, TUI prompt, per-source Config/sources/active/<source>/config.yaml, Config/Global-settings.yaml.
Automation
# Daily tech news
0 9 * * * cd ~/news && capcat bundle tech --count 20 --html
# Weekly science digest
0 10 * * 0 cd ~/news && capcat bundle science --count 30 --media
Privacy and Ethics
- Usernames anonymized as "Anonymous" in comment archives
- Respects
robots.txt - Rate limiting: 1 request per 10 seconds
- Prefers RSS/APIs over HTML scraping
- No paywall circumvention
- Proper source attribution
Documentation
Full documentation at capcat.org:
Contributing
Open an issue or pull request on GitHub.
License
MIT License - see LICENSE.txt
Links
- Website: capcat.org
- Repository: github.com/stayukasabov/capcat
- Issues: github.com/stayukasabov/capcat/issues
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file capcat-2.0.14.tar.gz.
File metadata
- Download URL: capcat-2.0.14.tar.gz
- Upload date:
- Size: 328.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.11.15
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
9557e9ee25deb58f5686ee9bb1975521ec9964592b3fcd10b8d7368c4bb32dcb
|
|
| MD5 |
175d982bd3b33fc94768c0a88442ad65
|
|
| BLAKE2b-256 |
e247f5f5dc4001dabe33a848f08467809a50064afef747e9fbdb7f0493d09787
|
File details
Details for the file capcat-2.0.14-py3-none-any.whl.
File metadata
- Download URL: capcat-2.0.14-py3-none-any.whl
- Upload date:
- Size: 421.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.11.15
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
20a66829366188aed42307d9729a853b98d45daa066246ad3fec5bd69efd6e39
|
|
| MD5 |
8382e59f042bc8bc3e495dcd3e0d1607
|
|
| BLAKE2b-256 |
50db796271bbafc1757832740c13a6f25494079f06b5eb2104453dcea2c78684
|
