FeedRay

FeedRay is a Python/FastAPI news intelligence service. It crawls configured news sources, stores article embeddings in PostgreSQL with pgvector, extracts article topics/entities/importance, groups related articles into event clusters, builds event timelines, and recommends both fresh articles and evolving stories.

Status: alpha. The backend is ready for local development and early integration. APIs, schema, and packaging may still change before a stable release.

What It Does

• Collects Google News RSS and direct RSS sources.
• Resolves publisher pages and extracts article metadata.
• Runs provider-based embedding and chat model analysis.
• Stores article topics, entities, and public importance_score.
• Groups related articles into event clusters.
• Promotes coherent pending articles into new events.
• Compresses long-running events into snapshots and timelines.
• Recommends articles and events from user interests and behavior.
• Tracks pipeline metrics, retry state, source reliability, and stale event lifecycle.

Architecture

sources
  -> article shells
  -> embedding queue
  -> chat analysis queue
  -> event assignment queue
  -> pending promotion
  -> event snapshots / lifecycle
  -> recommendations

Core modules:

Area	Module
API	feedray.api
SQLAlchemy models	feedray.db.models
Model providers	feedray.providers
Article analysis	feedray.services.analysis
Event assignment	feedray.services.event_assignment
Pending event promotion	feedray.services.pending_events
Event compression	feedray.services.event_compression
Async pipeline	feedray.services.pipeline
Crawler	feedray.scraping.crawler

Install

Package install:

pip install feedray

Crawler extras:

pip install "feedray[crawler]"
playwright install chromium

Development install:

git clone https://github.com/johnvonneumann36/FeedRay.git
cd FeedRay
pip install -e ".[dev,crawler]"

Legacy local install:

pip install -r requirements.txt

Configuration

Copy .env.example to .env, then edit local values. Do not commit .env.

Setting	Purpose
DB_USER, DB_PASSWORD, DB_HOST, DB_PORT, DB_NAME	PostgreSQL connection
JWT_SECRET_KEY, JWT_ALGORITHM	API auth token signing
FEEDRAY_EMBEDDING_PROVIDER_TYPE	Embedding provider
FEEDRAY_EMBEDDING_MODEL_NAME	Embedding model
FEEDRAY_EMBEDDING_BASE_URL	Optional OpenAI-compatible embedding endpoint
FEEDRAY_EMBEDDING_API_KEY	Embedding provider credential
FEEDRAY_CHAT_PROVIDER_TYPE	Chat provider
FEEDRAY_CHAT_MODEL_NAME	Chat model
FEEDRAY_CHAT_BASE_URL	Optional OpenAI-compatible chat endpoint
FEEDRAY_CHAT_API_KEY	Chat provider credential

Supported provider types:

Role	Providers
Embedding	ollama, openai, gemini, local_huggingface
Chat	ollama, openai, anthropic, gemini, local_huggingface

Database

FeedRay expects PostgreSQL with the vector extension available.

Warning: feedray-init-db drops all existing FeedRay tables before recreating them.

feedray-init-db

Use it only for fresh local setup or disposable development databases.

Run API

uvicorn feedray.api.app:app --host 0.0.0.0 --port 8000

Root endpoint:

GET /

Most API routes require a bearer token from /auth/login.

Jobs

Console entry points:

feedray-crawler
feedray-analyze-backfill --missing-only
feedray-event-backfill --missing-only
feedray-promote-pending-events --window-hours 48 --min-articles 2
feedray-compress-events --window-hours 24
feedray-archive-events --quiet-after-hours 48 --archive-after-hours 168

Python module equivalents:

python -m feedray.jobs.crawler
python -m feedray.jobs.analyze_backfill --missing-only
python -m feedray.jobs.event_backfill --missing-only
python -m feedray.jobs.promote_pending_events
python -m feedray.jobs.compress_events
python -m feedray.jobs.archive_events

API Highlights

Method	Path	Purpose
POST	/auth/register	Create user
POST	/auth/login	Get bearer token
GET	/articles	List/filter/search articles
GET	/articles/{id}	Article detail
GET	/events	List/filter/search event clusters
GET	/events/{id}	Event detail
GET	/events/{id}/articles	Event articles
GET	/events/{id}/timeline	Event snapshots
GET	/recommendations	Article recommendations
GET	/recommendations/events	Event recommendations
POST	/activities	Article feedback
POST	/activities/events	Event feedback
GET	/models/health	Provider health

Release Hygiene

Before publishing:

python -m pytest
python -m compileall feedray
python -m build
python -m twine check dist/*

Recommended smoke test:

python -m venv .venv-smoke
.venv-smoke\Scripts\python -m pip install dist\feedray-0.3.0-py3-none-any.whl
.venv-smoke\Scripts\python -c "import feedray; print(feedray.__version__)"

Security notes:

• .env is ignored and must stay local.
• .env.example contains placeholders only.
• chrome_profile/, logs/, dist/, and *.egg-info/ are ignored.
• Generated database, browser, log, and cache files should not be committed.

License

Apache-2.0. See LICENSE.