rss-retriever

Fetch articles from RSS feeds, extract their full text and images, and store them on disk.

Built around ports and adapters, so the pieces are usable independently: take the feed adapter without the storage layer, swap in your own storage backend, or drive the whole pipeline from the bundled CLI.

Install

pip install rss-retriever

Optional tracing support (OpenTelemetry SDK, OTLP exporters, and Arize Phoenix):

pip install "rss-retriever[otel]"

The base install deliberately stays light. opentelemetry-api is included because it is a no-op without an SDK, so instrumented code paths cost nothing when tracing is not configured.

Requires Python 3.12+.

Library usage

from rss_retriever import ContentExtractor, FileSystemStorage, NewsService, RSSFeedAdapter

service = NewsService(
    RSSFeedAdapter({"Phys.org": "https://phys.org/rss-feed/"}),
    FileSystemStorage("article_storage"),
    ContentExtractor(),
)

articles = service.fetch_and_store_articles(limit_per_source=5)
for article in service.get_recent_articles(limit=10):
    print(article.source_name, article.title)

Articles already present in storage are skipped before the expensive extraction step, so re-running against the same feeds is cheap.

Bulk imports

save_article rewrites both index files on every call so a crash cannot orphan an article. That is the right default for a nightly run but makes a large backfill quadratic in corpus size. Wrap bulk ingestion in batch_writes() to pay the index cost once:

with storage.batch_writes():
    for article in many_articles:
        storage.save_article(article)

Article payloads are still written immediately; only the indexes are deferred, and they are flushed even if the block raises. On a 5,000-article import this is roughly 12x faster.

Public API

Custom backends

Implement StoragePort to store somewhere other than the filesystem:

from rss_retriever import StoragePort

class S3Storage(StoragePort):
    def save_article(self, article): ...
    def get_article(self, article_id): ...
    def get_recent_articles(self, limit=50): ...
    def get_unread_articles(self, limit=50): ...
    def article_exists(self, url) -> bool: ...

CLI

export RSS_RETRIEVER_RSS_FEEDS='{"Phys.org": "https://phys.org/rss-feed/"}'
export RSS_RETRIEVER_STORAGE_DIR=./article_storage
rss-retriever

Configuration

Read from the environment by Config.from_env(). Construct Config directly to bypass the environment entirely.

There are no default feeds: a library should not fetch anything the caller did not ask for. rss_retriever.config.EXAMPLE_FEEDS holds a starting set if you want one.

Storage layout

article_storage/
├── index.json                  # article_id -> metadata, for recency queries
├── url_index.json              # url -> article_id, for deduplication
└── <source>_<url_hash>/
    ├── content.txt
    ├── content.html
    ├── metadata.json
    └── images/

Article IDs are <source_slug>_<first 10 hex of md5(url)> and are stable across runs.

Tracing

from rss_retriever.telemetry import setup_telemetry

setup_telemetry(service_name="my-service")

Configured through the standard OTEL_* environment variables. Requires the otel extra. Note that this installs a global tracer provider and sets OTLP defaults, so call it from an application entry point rather than from library code.

Development

uv sync --extra dev
uv run pytest -m "not network"   # unit tests, no network
uv run pytest                    # includes live-network tests
uv run ruff check .
uv run mypy rss_retriever/

License

MIT