📖 Table of Contents

• What Is It?
• Web Interface
• Pipeline
• Agents
• Autonomous Operation
• CLI Reference
• Quick Start
• Docker
• MCP Integration
• Development
• License

🔍 What Is It?

news48 is a self-hosted news pipeline that:

1. Ingests RSS/Atom feeds from sources you choose
2. Downloads full article content (with anti-bot bypass)
3. Parses unstructured HTML into structured data via LLM
4. Fact-checks claims against external evidence
5. Purges stale data on a 48-hour retention window

All of this runs autonomously through four AI agents that schedule themselves via Dramatiq + Periodiq. The agents also learn from mistakes — saving lessons that carry across runs so they get smarter over time.

🌐 Web Interface

news48 ships a FastAPI web interface that displays the last 48 hours of verified news. In Docker it's available at http://localhost:8765 (dev) or http://localhost:8000 (prod).

Pages

Features

• AI-rewritten summaries — clear, plain-English summaries for every parsed story
• Fact-check breakdown — per-claim verdicts (verified, disputed, mixed, unverifiable) with evidence
• Tone filter — filter stories by sentiment (positive, neutral, negative) across all pages
• Trending topics — auto-generated topic clusters from article tags
• Expiring stories — catch time-sensitive reporting before it leaves the 48-hour window
• Deduplication — same story from multiple sources is shown once per category
• Category normalization — consistent category names (e.g. artificial-intelligence and artificial intelligence are merged)
• SEO-friendly — canonical URLs, Open Graph tags, JSON-LD structured data, XML sitemap
• Rate limiting — 120 req/min general, 20 req/min for search
• Security headers — X-Content-Type-Options, X-Frame-Options, CSP, Referrer-Policy

🔁 Pipeline

 seed.txt ──► seed ──► fetch ──► download ──► parse ──► fact-check
                │          │          │           │           │
                ▼          ▼          ▼           ▼           ▼
             DB feeds   DB articles  HTML → MD  structured   verdicts
                                                    data

Most commands support --json for machine-readable output and --limit to control batch size.

🤖 Agents

Four agents run on schedules through Periodiq → Redis → Dramatiq:

🧠 Self-Learning

Agents save lessons when they discover something useful. On the next run, all accumulated lessons are injected into every agent's prompt:

Run 1:  Executor fails with wrong timeout → discovers 600s works → saves lesson
Run 2:  Executor starts with "timeout for fact-check should be 600s" already loaded

Lessons are stored in data/lessons.db (SQLite), cross-pollinated across agents, and human-auditable.

news48 lessons list                              # view all
news48 lessons list --agent executor --json      # filter by agent
news48 lessons add -a executor -c "Timing" -l "Use 600s timeout for fact-checks"

🎯 Autonomous Operation

news48 is designed to run without human intervention. Every capability is measured by a rigorous Autonomous Operation Score — a weighted assessment across six dimensions that evaluates how well the system starts, monitors, heals, scales, optimizes, and contains errors on its own.

Current Score: 4.9 / 5.0 — Autonomous

System runs unattended for extended periods. Human involvement only for strategic decisions.

How It Works

Every checkpoint is enforced at one of two layers:

• [code] — Deterministic Python code paths (e.g., StartupRecoveryMiddleware runs recovery on every worker boot)
• [instruction] — Agent skill files that define rules the LLM follows (e.g., fail-safely.md mandates breaking loops after 5 repeated errors)
• [both] — Defence in depth with code + instructions working together

The full methodology — all 40 checkpoints, verification procedures, and scoring rubric — is documented in docs/autonomous-operation-score.md. The latest assessment is loaded dynamically on the /docs page.

Score Levels

📋 CLI Reference

Pipeline Commands

news48 seed <file>              # Load feed URLs from file
news48 fetch                    # Pull RSS/Atom feeds
news48 download                 # Download article content
news48 parse                    # Parse articles with LLM
news48 fact-check               # Fact-check parsed articles
news48 briefing                 # Structured news briefing (top stories, trending, stats)
news48 doctor                   # Health check of all external services
news48 stats                    # Show system statistics

Resource Management

# Feeds
news48 feeds list                          # List all feeds
news48 feeds add <url>                     # Add a feed
news48 feeds info <url-or-id>              # Feed details
news48 feeds update <url-or-id> -t "Title" # Update metadata
news48 feeds delete <url-or-id>            # Delete feed + articles
news48 feeds rss --hours 48 --output feed.xml  # Generate RSS

# Articles
news48 articles list --status parsed       # List by status
news48 articles info <id-or-url>           # Article details
news48 articles content <id-or-url>        # Show content
news48 articles update <id> --content-file <path>  # Update fields
news48 articles delete <id-or-url>         # Delete article
news48 articles reset <id> --all           # Reset failure flags
news48 articles feature <id>               # Mark as featured
news48 articles breaking <id>              # Mark as breaking
news48 articles check <id> -s verified     # Set fact-check result
news48 articles claims <id>                # Show per-claim verdicts

# Fetches
news48 fetches list                        # View fetch history

Search

news48 search articles "climate change"                    # Full-text search
news48 search articles "election" --sentiment negative -l 5  # Filtered

Agents & Plans

news48 agents status                 # Queue depths + cron schedules
news48 agents run -a parser          # Run one agent (enqueue to Dramatiq)
news48 agents run -a parser --inline # Run inline (debug, no Redis needed)

news48 plans list                    # List all plans
news48 plans list -s pending         # Filter by status
news48 plans show <plan-id>          # Show plan details
news48 plans cancel <plan-id>        # Cancel a plan
news48 plans remediate --apply       # Repair plan corruption

Observability

news48 lessons list                  # View agent lessons

Retention & Health

news48 cleanup status                # Retention policy stats
news48 cleanup purge                 # Purge old articles (default: 48h)
news48 cleanup purge --dry-run       # Preview without deleting
news48 cleanup health                # Database connectivity check
news48 cleanup summaries             # Clean truncation markers from summaries
news48 cleanup summaries --dry-run   # Preview summary cleaning

Web & MCP

news48 mcp serve                     # Start MCP server (stdio)
news48 mcp create-key --label "Dev"  # Create API key
news48 mcp list-keys                 # List active keys
news48 mcp revoke-key <key>          # Revoke a key

💡 Tip: Append --json to any command for machine-readable output.

🚀 Quick Start

Prerequisites

• Python 3.12+
• uv package manager
• An OpenAI-compatible LLM endpoint
• A Byparr instance

Install

Option A — Docker (recommended):

One-liner install:

curl -fsSL https://raw.githubusercontent.com/malvavisc0/news48/master/scripts/install.sh | bash

Or clone and run manually:

git clone https://github.com/malvavisc0/news48.git && cd news48
./scripts/install.sh

The interactive installer clones the repository, checks prerequisites, prompts for deployment mode (GPU or external LLM), generates secure passwords, and launches all services.

Option B — Local (uv):

git clone https://github.com/malvavisc0/news48.git && cd news48
uv sync --extra all
cp .env.example .env
# Edit .env with your API keys (see table below)
uv run news48 --help

Extras:

uv sync --extra cli    # CLI + agents only
uv sync --extra web    # Web server only
uv sync --extra all    # Everything

Environment Variables

Run It

# 1. Seed feeds
uv run news48 seed seed.txt

# 2. Fetch articles
uv run news48 fetch

# 3. Download content
uv run news48 download --limit 10

# 4. Parse with LLM
uv run news48 parse --limit 10

# 5. Check stats
uv run news48 stats

🐳 Docker

news48 runs entirely in Docker with separate containers for each service.

Services

† internal only

Development

# Start with live reload
docker compose up

# Web UI      → http://localhost:8765
# RedisInsight → http://localhost:8001
# Dozzle       → http://localhost:9999

# Run CLI inside container
docker compose exec dramatiq-worker news48 stats
docker compose exec dramatiq-worker news48 feeds list

# Logs
docker compose logs -f dramatiq-worker
docker compose logs -f web

# Stop
docker compose down        # keep data
docker compose down -v     # fresh start

Production

# Start
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

# Web UI → http://localhost:8000

# Backup
docker compose exec mysql mysqldump -unews48 -pnews48 news48 > backup.sql

# Update
docker compose -f docker-compose.yml -f docker-compose.prod.yml build --no-cache
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

# Stop
docker compose -f docker-compose.yml -f docker-compose.prod.yml down

External LLM (no local GPU)

Use docker-compose.external-llm.yml to skip the built-in llama.cpp server and point at any OpenAI-compatible endpoint instead:

# Set API_BASE in .env (examples):
#   API_BASE=http://host:7070/v1          # host llama.cpp
#   API_BASE=https://api.openai.com/v1    # OpenAI
#   API_BASE=https://api.groq.com/openai/v1  # Groq
#   API_BASE=http://host:11434/v1         # Ollama

docker compose -f docker-compose.yml -f docker-compose.external-llm.yml up -d

This disables the Docker llama.cpp container and model download — ideal for hosted LLM APIs or when running llama.cpp on the host machine.

CLI Wrapper

The install script adds a news48 command to ~/.local/bin that wraps docker compose exec dramatiq-worker. After installation (and sourcing your shell rc), you can run:

news48 stats                    # System statistics
news48 feeds list               # List all feeds
news48 briefing                 # News briefing
news48 search articles "query"  # Full-text search
news48 mcp create-key --label "Dev"  # Create MCP API key

The wrapper cds to the install directory and runs commands inside the dramatiq-worker container. It uses the same compose file flags as the installer (-f docker-compose.yml -f docker-compose.prod.yml).

Seeding in Docker

The sentinel agent auto-detects an empty database and creates a seed plan — so if seed.txt is in the image, seeding happens automatically.

# Manual seed (using the wrapper)
news48 seed /app/seed.txt

# Or with full docker compose command
docker compose exec dramatiq-worker news48 seed /app/seed.txt

# Verify
news48 feeds list

Worker Observability

• RedisInsight → http://localhost:8001 — inspect queues and broker state
• Dozzle → http://localhost:9999 — container log viewer
• CLI → news48 agents status --json — queue depths and cron schedules

🔌 MCP Integration

news48 exposes tools via the Model Context Protocol so AI assistants can interact with your pipeline.

Local Server (stdio)

No auth required — ideal for Claude Desktop, Cursor, etc.

uv run news48 mcp serve

{
  "mcpServers": {
    "news48": {
      "command": "news48",
      "args": ["mcp", "serve"]
    }
  }
}

Tools: get_briefing, search_news, get_article, browse_category, list_categories, list_countries

Remote Endpoint (HTTP)

The web app exposes an authenticated endpoint at /mcp. Keys are stored in Redis.

# Create a key
uv run news48 mcp create-key --label "Claude Desktop"
# → Created MCP API key: n48-aBcDeFgHiJkLmNoPqRsTuVwXyZ...
# ⚠️  Copy it now — it can't be retrieved later

# List keys (masked)
uv run news48 mcp list-keys

# Revoke a key
uv run news48 mcp revoke-key n48-...

{
  "mcpServers": {
    "news48-remote": {
      "url": "https://your-domain.com/mcp",
      "headers": {
        "Authorization": "Bearer n48-your-api-key-here"
      }
    }
  }
}

Both /mcp and /mcp/ work — the endpoint handles both paths without redirects, which is critical for browser-based MCP clients (Inspector, Claude, Cursor). CORS preflight (OPTIONS) is handled directly by the endpoint with Access-Control-Allow-Origin: *.

Tools: get_briefing, search_news, get_article, browse_category, list_categories, list_countries

🔒 All keys are prefixed n48- for secret scanner detection. If Redis is unreachable, all MCP requests are denied (fail-closed).

🧬 Development

# Run tests
uv run pytest

# Format
uv run black .
uv run isort .

📄 License

MIT — see LICENSE for details.