news-fetch

Python news scraper & article extractor — extract title, text, authors, date, image, and publisher from any news URL. No API key. Confidence scores included.

Fetch news. Know why it worked.

pip install news-fetch

from newsfetch import fetch

article = fetch("https://www.thehindu.com/...")
print(article.title)
print(article.text)
print(article.authors)
print(article.published_at)
print(article.image)
print(article.confidence.overall)   # 0.0–1.0
print(article.content_source)       # e.g. json-ld.articleBody

news-fetch https://example.com/article
news-fetch https://example.com/article --json
news-fetch batch urls.txt -o articles.jsonl

Why news-fetch?

A lightweight alternative to newspaper3k / newspaper4k / trafilatura wrappers — with its own extraction engine, confidence scores, and bulk + proxy support.

Feature	news-fetch
News article extraction (title, body, authors, date, image)	✅
Confidence scores + extraction provenance	✅
Bulk scraping (`fetch_many` / `fetch_iter` / CLI JSONL)	✅
Proxy + proxy rotation for thousands of URLs	✅
RSS / sitemap article discovery	✅
Async (`pip install news-fetch[async]`)	✅
Optional browser render (`pip install news-fetch[browser]`)	✅
Disk cache + robots.txt respect	✅
No API key / no account	✅
Small deps (`lxml`, `requests`, `python-dateutil`, `cssselect`)	✅

Install

pip install news-fetch
pip install news-fetch[async]     # httpx async fetch
pip install news-fetch[browser]   # Playwright fallback (then: playwright install chromium)

Requirements: Python 3.10+

Quick start

Single URL

from newsfetch import fetch

article = fetch(url)
print(article.title, article.text, article.confidence.overall)

From HTML (no network)

from newsfetch import extract

article = extract(html_bytes, url="https://example.com/story")

Bulk scraping + proxies

from newsfetch import fetch_many, fetch_iter

results = fetch_many(
    urls,
    max_workers=20,
    proxies=["http://user:pass@p1:8080", "http://user:pass@p2:8080"],
    request_delay=0.05,
)

for url, article in fetch_iter(urls, max_workers=16):
    if article:
        print(article.title)

Strict mode (production pipelines)

from newsfetch import fetch, LowConfidenceExtractionError

try:
    article = fetch(url, strict=True)
except LowConfidenceExtractionError as e:
    print(e.failed_fields, e.confidence.overall)

Discovery (RSS / sitemaps)

from newsfetch import discover

for item in discover("https://www.bbc.com", limit=10):
    print(item["url"], item.get("title"))

Async

from newsfetch import fetch_async, fetch_many_async

article = await fetch_async(url)
articles = await fetch_many_async(urls, max_concurrency=50, proxies=PROXIES)

CLI

news-fetch https://example.com/article
news-fetch get URL --json
news-fetch batch urls.txt -o out.jsonl --workers 20
news-fetch discover https://www.theguardian.com --limit 10

Cache / robots / browser

from newsfetch import fetch, Config, NewsFetcher

fetch(url, cache=True, respect_robots=True)
fetch(url, render=True)                      # needs news-fetch[browser]
fetch(url, browser_fallback=True)            # retry with Playwright if confidence is low

Custom strategy plugin

from newsfetch import NewsFetcher, CallableStrategy
from newsfetch.strategies.base import Candidate

def my_strategy(doc):
    return {"title": [Candidate("Custom", "plugin.custom", 0.99)]}

fetcher = NewsFetcher()
fetcher.register_strategy(CallableStrategy("custom", my_strategy))

Article fields

url · canonical_url · title · description · text · authors · published_at · modified_at · publisher · language · image · keywords · section · summary · word_count · reading_time_minutes · page_type · is_article · confidence · extraction · sources

article.to_dict()
article.to_json()

Release files for news-fetch 1.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for news-fetch 1.0.0
File	Size	Uploaded
news_fetch-1.0.0.tar.gz	40.7 kB	Aug 13, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for news-fetch 1.0.0
File	Interpreter	ABI	Platform	Reset
news_fetch-1.0.0-py3-none-any.whl 48.4 kB Aug 13, 2026	Python 3	none	any	Details

Total release size: 89.1 kB

Release files / news_fetch-1.0.0.tar.gz

Download URL	news_fetch-1.0.0.tar.gz
Size	40.7 kB
Tags	Source
SHA-256 checksum How to use checksums	`ec9ec39378a82ff9fba966d191aa2d86d5a9d9483f3ab0bc0cbc0533bf490a39`
BLAKE2b-256 checksum How to use checksums	`e460049d1258dc3dd51f71432c6b125dbe2a2327e9d675a948caee960d688313`
Upload date	Aug 13, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.14.0`

Release files / news_fetch-1.0.0-py3-none-any.whl

Download URL	news_fetch-1.0.0-py3-none-any.whl
Size	48.4 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`1233683d53959114e91c820595be2db44d2ff446c292b832100e57df608be07d`
BLAKE2b-256 checksum How to use checksums	`e37a9f6c86714e1958980021bf34d9b912a346e668699524bbac06d79889d4c3`
Upload date	Aug 13, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.14.0`