✨ What is Open News?

Open News is intentionally not a full news platform. It is a reusable library and set of interfaces for applications that need reliable access to news data without building the entire acquisition and processing stack themselves.

                         ┌─────────────────────┐
                         │      Your App       │
                         └──────────┬──────────┘
                                    │
                         Python API / CLI / TUI
                                    │
                         ┌──────────▼──────────┐
                         │    Open News API    │
                         └──────────┬──────────┘
                                    │
                    ┌───────────────┼───────────────┐
                    │               │               │
                 Acquire         Process         Extract
                    │               │               │
                    ▼               ▼               ▼
              live sources    filter · dedupe   article text
              RSS / search    rank · enrich     metadata/media

🎯 Design goals

• Lightweight — use ordinary Python tooling where possible.
• Resilient — acquisition can fall back across multiple sources.
• Composable — public functions can be embedded into other projects.
• Local-first processing — filtering, deduplication, ranking, and summarization do not require an LLM or hosted AI service.
• Cluster-aware — group related stories and rank by outlet coverage, not just by date.
• Scriptable — JSON output makes the CLI suitable for pipelines.
• Portable — desktop Linux, macOS, WSL/Git Bash, and Termux are supported by the installation workflow.
• Stable core — v1.0.3 freezes the architecture developed and validated during the 1.0.3 pre-release cycle.

🧭 v1.0.4 at a glance

v1.0.4 builds on the v1.0.3 acquisition baseline.

v1.0.4 is a processing/output release: clustering, topic filtering, BM25/TF-IDF relevance ranking, and clean Markdown/JSON export — plus a keyboard-driven TUI refresh.

What v1.0.4 adds on top

pipeline results
       │
       ▼
filter_articles(topic=…)     narrow by topic across title/description/text/category/keywords
       │
       ▼
rank_articles(query=…)       BM25 if bm25s is installed, else TF-IDF
       │
       ▼
cluster_articles()           group by title similarity; keep every article
       │
       ▼
export.to_markdown()/to_json()   clean, schema-versioned output

All four are also available as CLI flags (--topic, --rank-query, cluster, export) and inside the TUI.

This is complemented by stronger URL resolution, aggregator-aware source attribution, hub/listing detection, deterministic test infrastructure, Android/Termux support, and a hardened installer.

🌟 Features

📦 Installation

Package name: open-news-api
Python: 3.10+

⚡ Interactive installer

curl -fsSL https://raw.githubusercontent.com/alphap365/open-news/main/install.sh | bash

The installer detects the environment, can create an isolated environment, optionally installs JavaScript rendering, configures PATH, saves CLI preferences, and verifies the resulting installation.

🐍 pip

pip install open-news-api

Pin v1.0.3 explicitly when you want the stable release:

pip install open-news-api==1.0.4

⚡ uv

uv add open-news-api

Or try the CLI without adding it to a project:

uvx --from open-news-api open-news --help

🛠️ From source

git clone https://github.com/alphap365/open-news.git
cd open-news
python -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev]"
pytest -q

Windows activation:

.venv\Scripts\activate

🌐 Optional JavaScript rendering

pip install "open-news-api[js]"
playwright install chromium

JavaScript rendering is optional. Plain HTTP extraction remains the default.

🧠 Optional NLP ranking

pip install "open-news-api[nlp]"

Enables BM25 ranking in rank_articles() (via bm25s) and higher-quality LSA summarization (via sumy). Without it, ranking falls back to pure-Python TF-IDF.

🚀 Quick Start

1. Fetch live news

from open_news import fetch

articles = fetch(category="tech", max_results=5)

for article in articles:
    print(article["title"])
    print(article["url"])

2. Search news

from open_news import search

articles = search(
    "artificial intelligence",
    query_mode="all",
    exclude_terms=["sports"],
    country="in",
    max_results=5,
)

for article in articles:
    print(f"{article['title']} → {article['url']}")

3. Extract an article

from open_news import get_article

article = get_article("https://example.com/article")

print(article["title"])
print(article["text"])
print(article["source"])

4. Discover a website

from open_news import discover_and_get

articles = discover_and_get("https://example.com", limit=5)

Open News tries RSS discovery first and falls back to same-domain crawling when appropriate.

5. Stream new search results

from open_news import stream_search

for articles in stream_search("budget 2026", refresh_interval=30):
    for article in articles:
        print(article["title"])

6. Summarize

from open_news import search_and_summarize

results = search_and_summarize(
    "renewable energy",
    limit=5,
    sentence_count=2,
)

for result in results:
    print(result["summary"])

7. Cluster related stories

from open_news import fetch, cluster_articles

articles = fetch(category="tech", max_results=40)

clusters = cluster_articles(
    articles,
    threshold=0.72,
    topic="artificial intelligence",
    rank_query="openai gemini",
    drop_singletons=True,
)

for c in clusters:
    print(f"{c['size']} outlets: {c['label']}")

8. Rank by relevance

from open_news import search, rank_articles

articles = search("eu ai act", max_results=30)
ranked = rank_articles(articles, query="EU AI Act", method="auto")

print(ranked[0]["title"], ranked[0]["_rank_score"])

9. Export

from open_news import export

export.to_markdown(clusters, path="out/clusters.md", include_all_members=True)
export.to_json(clusters, path="out/clusters.json")

Both functions return the serialized string as well, so you can pipe or stream instead of writing to disk.

🏗️ Architecture

Open News is organized around a small number of layers rather than one monolithic engine.

open_news/
│
├── api.py                  Public API boundary
├── config.py               Validated configuration objects
│
├── feeds/                  News acquisition and feed discovery
│   ├── duckduckgo_engine.py   Five-tier fetch orchestrator
│   ├── googlenews_engine.py   Google News search/fetch
│   ├── bingnews_engine.py     Bing News fallback
│   ├── yahoonews_engine.py    Yahoo News fallback
│   ├── sources.py             RSS/search sources
│   └── rss_discovery.py       RSS detection
│
├── fetch/                  URL and website acquisition
│   ├── article.py             Single article retrieval
│   ├── crawler.py             Async + JS crawlers
│   └── url_resolver.py        Redirect/hub/aggregator classification
│
├── processing/             Shared result processing
│   ├── pipeline.py            Filter → dedupe → rank → enrich
│   ├── dedupe.py
│   ├── ranker.py
│   ├── language_guard.py
│   ├── domain_filter.py
│   ├── token_filter.py
│   ├── batch.py
│   └── summarizer.py
│
└── core/                   Article extraction
    ├── extractor.py           Strategy coordinator
    ├── strategies.py          JSON-LD / Open Graph / HTML heuristics
    └── renderer.py            Playwright support

For the full data flow, see docs/architecture.md.

🖥️ CLI & 📺 TUI

CLI

open-news fetch --category tech --limit 5
open-news search "artificial intelligence" --country in --limit 5
open-news extract https://example.com/article
open-news discover https://example.com --limit 10
open-news search-site "climate policy" reuters.com
open-news summarize --query "renewable energy" --limit 5
open-news search "eu ai act" --topic "policy,regulation" --rank-query "EU AI Act" \
    --export-md out/ai-act.md --export-json out/ai-act.json
open-news cluster --query "openai" --limit 40 --drop-singletons \
    --export-md out/clusters.md --export-all-members
open-news export out/ai-act.json --md out/ai-act.md

For scripting:

open-news search "AI" --format json | jq '.[].title'

TUI

open-news-tui

The TUI provides a numbered workflow for fetching, searching, discovery, extraction, summarization, settings, and live category refresh.

🔐 Resilience & failure behavior

Open News deliberately treats acquisition as a best-effort operation.

A source failure does not automatically mean the whole fetch() operation fails. The acquisition orchestrator can continue through its fallback chain.

Likewise, article enrichment does not intentionally destroy good existing metadata merely because a later extraction step returned an empty field.

For aggregator-hosted stories, source resolution is explicitly best effort. The library records whether the source was identified as an aggregator and whether the original publisher was resolved. Consumers should not treat an unresolved aggregator attribution as ground truth.

🧪 Testing

The v1.0.3 development cycle expanded the test environment beyond ordinary unit tests.

It includes infrastructure for:

• deterministic local article-server tests;
• Android/Termux emulation;
• network-dependent test separation;
• native-dependency test separation;
• acquisition-engine contracts;
• extraction and URL-resolution behavior;
• processing and deduplication contracts;
• CLI behavior.

Run the suite locally with:

pytest -q

📚 Documentation

🧭 Versioning philosophy

Open News follows semantic-version-style release numbering.

The 1.0.3 pre-release series is intentionally treated as one development cycle rather than a collection of unrelated public feature releases:

Issue #1
   │
   ▼
1.0.3a1 → a2 → a3 → a4 → a5 → a6
   │
   ▼
1.0.3b1 → 1.0.3b2
   │
   ▼
1.0.3 stable

The stable release freezes the resulting architecture and makes the v1.0.3 behavior the documented baseline.

🤝 Contributing

Contributions are welcome, especially around:

• new acquisition fallbacks;
• extraction strategies;
• deterministic tests;
• portability improvements;
• documentation;
• performance and resource usage.

Please read CONTRIBUTING.md before opening a pull request.

📄 License

Open News is released under the MIT License. See LICENSE.