argos-news 0.1.0

Automated Realtime Global Observer System - Python Client & API for Middle East News

ARGOS — Automated Realtime Global Observer System

ARGOS is a 3-layer automated news pipeline that continuously scrapes articles from 4 Middle Eastern news networks, enriches them with editorial metadata, stores them locally, and exposes them via a REST API and Python client library.

---

Architecture

[ News Websites ]               ← 4 Middle East sources
      │  HTTP scraping (Scrapy)
      ▼
[ Home Server — Scrapy Engine ]  ← Runs on your machine
      │  Direct write to local DB
      ▼
[ SQLite Database ]              ← Local, unlimited storage
      │  REST API queries
      ▼
[ FastAPI API Layer ]            ← Exposed via argos.news.ydns.eu
      │  Python function calls
      ▼
[ ArgosClient Library ]          ← Import and use

Source Networks

Source	Country	Orientation	Trust	Israeli-Influenced
Al Jazeera	Qatar	center-left	0.55	No
Al Arabiya	Saudi Arabia	right	0.40	Yes
Sky News Arabia	UAE	center-right	0.48	Yes
Middle East Eye	UK (ME Focus)	left	0.48	No

---

Quick Start

1. Prerequisites

• Python 3.11+
• pip
• Linux/macOS (home server)

2. Install

# Clone the repository
cd /path/to/project

# Create virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Copy and configure environment
cp .env.example .env
# Edit .env with your settings (defaults work for local development)

3. Launch Everything

# Start API server + all 4 spiders
python launch.py

# Or start components separately:
python launch.py --api-only       # API server only
python launch.py --spiders-only   # All spiders only
python launch.py --spiders aljazeera middleeasteye  # Specific spiders

4. Access the API

• API Docs: http://localhost:8000/docs (Swagger UI)
• Health Check: http://localhost:8000/v1/health
• Latest Articles: http://localhost:8000/v1/articles/latest
• Sources: http://localhost:8000/v1/sources

5. Use the Python Client

from client import ArgosClient

# Connects to argos.news.ydns.eu automatically
client = ArgosClient()

# For local development:
# client = ArgosClient(base_url="http://localhost:8000")

# Get latest articles
latest = client.get_latest(count=10)
for article in latest:
    print(f"[{article['source_name']}] {article['title']}")

# Search with filters
results = client.get_articles(
    source_name="Al Jazeera",
    category="war",
    date_from="2024-01-01",
    page_size=50,
)
print(f"Found {results['pagination']['total_results']} articles")

# Get all sources
sources = client.get_sources()
for s in sources:
    print(f"{s['source_name']} ({s['country']}) — trust: {s['trust_score']}")

---

Running Spiders Individually

# Activate venv
source venv/bin/activate

# Run a specific spider
scrapy crawl aljazeera
scrapy crawl alarabiya
scrapy crawl skynewsarabia
scrapy crawl middleeasteye

# Limit items for testing
scrapy crawl aljazeera -s CLOSESPIDER_ITEMCOUNT=10

# Resume a previously interrupted crawl (L-02)
scrapy crawl aljazeera -s JOBDIR=./data/crawl_jobs/aljazeera

API Endpoints

Method	Endpoint	Description
GET	/v1/health	System status
GET	/v1/sources	List all source networks
GET	/v1/articles	Search with filters + pagination
GET	/v1/articles/latest	N most recent articles
GET	/v1/articles/{id}	Single article by ID

Filter Parameters (GET /v1/articles)

Parameter	Type	Description
source_name	string	Filter by news network
country	string	Filter by source country
category	string	Topic: politics, economy, tech, war, health, science, culture, sports, environment, other
topic	string	Alias for category
orientation	string	Editorial leaning
is_israeli_influenced	bool	Israeli influence flag
date_from	string	Min date (ISO 8601)
date_to	string	Max date (ISO 8601)
title	string	Title search (partial match)
tags	string	Comma-separated tags (AND logic)
page	int	Page number (default 1)
page_size	int	Items per page (default 20, max 100)
sort_by	string	Sort column (default: published_at)
order	string	asc or desc (default: desc)

---

Scheduling (Continuous Scraping)

ARGOS is designed to run continuously via cron (FR-07):

# Edit crontab
crontab -e

# Run all spiders every hour
0 * * * * cd /path/to/project && /path/to/venv/bin/python launch.py --spiders-only

# Keep API running via systemd or screen
# Option 1: screen
screen -dmS argos-api python launch.py --api-only

# Option 2: systemd service (create /etc/systemd/system/argos-api.service)

Environment Variables

See .env.example for all variables with documentation.

Variable	Default	Description
DATABASE_PATH	./data/argos.db	SQLite database path
API_HOST	0.0.0.0	API bind address
API_PORT	8000	API port
API_BASE_URL	http://argos.news.ydns.eu:8000	Public API URL
PROXY_URL	(empty)	Optional proxy for geo-blocking
LOG_LEVEL	INFO	Logging verbosity
JOBDIR	./data/crawl_jobs	Scrapy job persistence

---

Project Structure

argos/
├── launch.py                    ← 🚀 Launch everything
├── requirements.txt             ← Dependencies
├── .env.example                 ← Environment config template
├── scrapy.cfg                   ← Scrapy project config
│
├── scraper/argos/               ← Scrapy scraping engine
│   ├── items.py                 ← ArticleItem (13 fields)
│   ├── registry.py              ← SOURCE_REGISTRY + apply_registry()
│   ├── settings.py              ← Fully documented settings
│   ├── middlewares.py           ← UA rotation + proxy
│   ├── pipelines/
│   │   ├── validation.py        ← Field validation (priority 100)
│   │   ├── dedup.py             ← URL dedup (priority 200)
│   │   └── storage.py           ← SQLite write (priority 800)
│   └── spiders/
│       ├── aljazeera.py         ← Al Jazeera spider
│       ├── alarabiya.py         ← Al Arabiya spider
│       ├── skynewsarabia.py     ← Sky News Arabia spider
│       └── middleeasteye.py     ← Middle East Eye spider
│
├── api/                         ← FastAPI REST API
│   ├── main.py                  ← App entry point
│   ├── config.py                ← Environment settings
│   ├── database.py              ← SQLite query layer
│   ├── models.py                ← Pydantic response models
│   ├── openapi.yaml             ← OpenAPI 3.1 spec
│   └── routes/
│       ├── articles.py          ← /v1/articles endpoints
│       ├── sources.py           ← /v1/sources endpoint
│       └── health.py            ← /v1/health endpoint
│
└── client/                      ← Python client library
    ├── argos_client.py          ← ArgosClient class
    └── exceptions.py            ← ArgosAPIError

---

Data Model

Every article has exactly 13 fields — no more, no less:

Field	Type	Source
title	string	Article HTML
content	string	Article HTML
summary	string	Meta tags / first paragraph
category	enum	Spider classification
tags	list[str]	Article metadata
source_name	string	Registry
source_url	string	Spider URL
published_at	ISO 8601	Article HTML
scraped_at	ISO 8601	Auto-generated
country	string	Registry
orientation	enum	Registry
trust_score	float 0-1	Registry
is_israeli_influenced	bool	Registry

---

Developer

Zayani Mouhamed Mouheb / EGen Labs

ARGOS v0.1.0 — Argus Panoptes sees everything.