ARGOS — Automated Realtime Global Observer System

A 3-layer automated news pipeline that continuously scrapes, enriches, and serves articles from 4 major Middle Eastern news networks — via a REST API and Python client library.

---

Table of Contents

• Overview
• Architecture
• News Sources
• Quick Start
• Running Spiders
• API Reference
• Scheduling
• Data Model
• Configuration
• Project Structure

---

Overview

ARGOS (Argus Panoptes sees everything) automates the full lifecycle of news collection from the Middle East:

• 🕷️ Scrapes articles from 4 sources using Scrapy spiders
• 🧠 Enriches them with editorial metadata (orientation, trust score, influence flags)
• 🗄️ Stores everything locally in SQLite — no cloud dependency
• 🌐 Exposes a FastAPI REST API with filtering, pagination, and sorting
• 📦 Ships a ready-to-use Python client library

---

Architecture

[ News Websites ]
       │  HTTP scraping (Scrapy)
       ▼
[ Home Server — Scrapy Engine ]   ← Continuous crawl with dedup & validation
       │  Direct write
       ▼
[ SQLite Database ]               ← Local, unlimited storage
       │  Query layer
       ▼
[ FastAPI REST API ]              ← Exposed via argos.news.ydns.eu
       │  Python calls
       ▼
[ ArgosClient Library ]           ← Import and use in any Python project

Pipeline stages per article:

Stage	Component	Priority
Field validation	pipelines/validation.py	100
URL deduplication	pipelines/dedup.py	200
SQLite storage	pipelines/storage.py	800

---

News Sources

Source	Country	Orientation	Trust Score	Israeli-Influenced
Al Jazeera	Qatar	Center-Left	0.55	No
Al Arabiya	Saudi Arabia	Right	0.40	Yes
Sky News Arabia	UAE	Center-Right	0.48	Yes
Middle East Eye	UK (ME Focus)	Left	0.48	No

Trust scores and influence flags are defined in scraper/argos/registry.py and are attached to every article at scrape time.

---

Quick Start

Prerequisites

• Python 3.11+
• pip
• Linux or macOS (for the home server component)

1. Install

# Clone the repository
git clone <repo-url> && cd argos

# Create and activate a virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Configure environment (defaults work for local development)
cp .env.example .env

2. Launch

# Start the API server + all 4 spiders
python launch.py

# Or start components independently:
python launch.py --api-only                                   # API server only
python launch.py --spiders-only                               # All spiders
python launch.py --spiders aljazeera middleeasteye            # Specific spiders

3. Verify

Resource	URL
Swagger UI	http://localhost:8000/docs
Health check	http://localhost:8000/v1/health
Latest articles	http://localhost:8000/v1/articles/latest
Sources	http://localhost:8000/v1/sources

4. Use the Python Client

from client import ArgosClient

# Connect to the public instance
client = ArgosClient()

# Or connect to your local instance
# client = ArgosClient(base_url="http://localhost:8000")

# Fetch the 10 most recent articles
latest = client.get_latest(count=10)
for article in latest:
    print(f"[{article['source_name']}] {article['title']}")

# Search with filters
results = client.get_articles(
    source_name="Al Jazeera",
    category="war",
    date_from="2024-01-01",
    page_size=50,
)
print(f"Found {results['pagination']['total_results']} articles")

# List all sources
for source in client.get_sources():
    print(f"{source['source_name']} ({source['country']}) — trust: {source['trust_score']}")

---

Running Spiders

source venv/bin/activate

# Run a single spider
scrapy crawl aljazeera
scrapy crawl alarabiya
scrapy crawl skynewsarabia
scrapy crawl middleeasteye

# Limit output for testing
scrapy crawl aljazeera -s CLOSESPIDER_ITEMCOUNT=10

# Resume an interrupted crawl
scrapy crawl aljazeera -s JOBDIR=./data/crawl_jobs/aljazeera

---

API Reference

Endpoints

Method	Endpoint	Description
GET	/v1/health	System status
GET	/v1/sources	List all news sources
GET	/v1/articles	Search articles with filters and pagination
GET	/v1/articles/latest	Fetch N most recent articles
GET	/v1/articles/{id}	Fetch a single article by ID

Full interactive documentation is available at /docs (Swagger UI) when the server is running.

Filter Parameters — GET /v1/articles

Parameter	Type	Description
source_name	string	Filter by network name
country	string	Filter by source country
category	string	politics, economy, tech, war, health, science, culture, sports, environment, other
topic	string	Alias for category
orientation	string	Editorial leaning
is_israeli_influenced	bool	Israeli influence flag
date_from	ISO 8601	Earliest publication date
date_to	ISO 8601	Latest publication date
title	string	Partial title match
tags	string	Comma-separated tags (AND logic)
page	int	Page number (default: 1)
page_size	int	Results per page (default: 20, max: 100)
sort_by	string	Column to sort by (default: published_at)
order	string	asc or desc (default: desc)

---

Scheduling

ARGOS is designed for continuous, unattended operation. The recommended setup uses cron for spiders and systemd or screen for the API.

# Edit crontab to scrape every hour
crontab -e
0 * * * * cd /path/to/argos && /path/to/venv/bin/python launch.py --spiders-only

# Keep the API alive with screen
screen -dmS argos-api python launch.py --api-only

# Or create a systemd service for production
# → /etc/systemd/system/argos-api.service

---

Data Model

Every article is stored with exactly 13 fields:

Field	Type	Origin
title	string	Scraped from article HTML
content	string	Scraped from article HTML
summary	string	Meta tags or first paragraph
category	enum	Spider classification
tags	list[str]	Article metadata
source_name	string	registry.py
source_url	string	Spider URL
published_at	ISO 8601	Article HTML
scraped_at	ISO 8601	Auto-generated at crawl time
country	string	registry.py
orientation	enum	registry.py
trust_score	float [0–1]	registry.py
is_israeli_influenced	bool	registry.py

---

Configuration

Copy .env.example to .env and adjust as needed. Defaults work out of the box for local development.

Variable	Default	Description
DATABASE_PATH	./data/argos.db	SQLite database path
API_HOST	0.0.0.0	API bind address
API_PORT	8000	API port
API_BASE_URL	http://argos.news.ydns.eu:8000	Public API base URL
PROXY_URL	(empty)	Optional HTTP proxy for geo-blocked sources
LOG_LEVEL	INFO	Logging verbosity (DEBUG, INFO, WARNING, ERROR)
JOBDIR	./data/crawl_jobs	Scrapy job persistence directory

---

Project Structure

argos/
├── launch.py                      ← Entry point — start API and/or spiders
├── requirements.txt
├── .env.example
├── scrapy.cfg
│
├── scraper/argos/                 ← Scrapy engine
│   ├── items.py                   ← ArticleItem (13 fields)
│   ├── registry.py                ← Source metadata + apply_registry()
│   ├── settings.py                ← Scrapy settings (documented)
│   ├── middlewares.py             ← User-agent rotation + proxy support
│   ├── pipelines/
│   │   ├── validation.py          ← Field validation        (priority 100)
│   │   ├── dedup.py               ← URL deduplication       (priority 200)
│   │   └── storage.py             ← SQLite write            (priority 800)
│   └── spiders/
│       ├── aljazeera.py
│       ├── alarabiya.py
│       ├── skynewsarabia.py
│       └── middleeasteye.py
│
├── api/                           ← FastAPI REST layer
│   ├── main.py                    ← App entry point
│   ├── config.py                  ← Environment settings
│   ├── database.py                ← SQLite query layer
│   ├── models.py                  ← Pydantic response models
│   ├── openapi.yaml               ← OpenAPI 3.1 spec
│   └── routes/
│       ├── articles.py            ← /v1/articles
│       ├── sources.py             ← /v1/sources
│       └── health.py              ← /v1/health
│
└── client/                        ← Python client library
    ├── argos_client.py            ← ArgosClient class
    └── exceptions.py              ← ArgosAPIError

---

Developer

Zayani Mouhamed Mouheb / EGen Labs — ARGOS v0.3.x

Argus Panoptes sees everything.