news-fetch 0.4.1

news-fetch is an open-source, easy-to-use news extractor with basic NLP features (cleaning text, keywords, summary) that just works.

📰 news-fetch

news-fetch extracts structured data from a news article URL with one call: Newspaper(url=...).get_dict 🌐. Under the hood it's built on newspaper4k, but it isn't just a wrapper around it — it exists to fix the gaps single-engine extraction leaves.

If this saves you time, please ⭐ star the repo — it's the main way other people find it, and it's free.

---

📚 Table of Contents

• Why news-fetch?
• Extracted Information
• Installation
• Usage
• Project Links
• Contributing
• License

✨ Why news-fetch?

	news-fetch	Plain newspaper4k
Publication / category / modified-date backfilled from JSON-LD when Open Graph tags are missing (e.g. BBC)	✅	❌
summary / keywords always populated, no NLTK corpus download needed	✅	❌ (.nlp() requires one)
One flat dict, first non-empty value wins across engines	✅	❌ (assemble it yourself)
Site-wide article discovery via sitemap/RSS, no browser automation	✅	❌
Concurrent batch scraping with per-URL failure isolation	✅	❌
Reading time & word count computed for free	✅	❌
Install footprint	~50MB, no Scrapy/boto3/Selenium	—

In detail:

• JSON-LD backfill. Many modern news sites (e.g. BBC) don't expose Open Graph tags that newspaper4k relies on for publication/category, but do embed a schema.org/NewsArticle JSON-LD block. news-fetch parses that block directly and fills in publication, category, and date_modify whenever the primary engine comes up empty — no extra dependency required, since it reuses the beautifulsoup4/requests already in the base install.
• No NLTK download required. newspaper4k's built-in summary/keyword extraction (.nlp()) needs an NLTK corpus download, which routinely fails behind corporate proxies or strict SSL setups. news-fetch falls back to a dependency-free, pure-stdlib summarizer/keyword extractor when that's unavailable, so summary/keywords are never empty.
• Every field, one flat dict, first non-empty value wins. Instead of learning three different libraries' inconsistent APIs, you get one object where each field is resolved by trying every available engine in priority order and returning the first real value.
• Site-wide article discovery, no browser automation. NewsSiteURLExtractor finds a news site's recent article URLs (with title/date, when available) via its robots.txt sitemap directives, Google News sitemaps, and RSS/Atom feeds — the same techniques real news aggregators use, and safer/more reliable than scraping a search engine.
• Reading time and word count, computed for free from the extracted article text.
• Concurrent batch scraping via Newspaper.from_urls([...]) — one bad URL doesn't take down the whole batch; it just comes back as None.
• A minimal, honest dependency footprint. The whole install is ~50MB: newspaper4k, beautifulsoup4, requests, lxml-html-clean, Unidecode — no Scrapy, no boto3, no Selenium.

📝 Extracted Information

news-fetch extracts the following attributes from news articles. You can also check out an example JSON file.

• 📰 Headline
• ✍️ Author(s)
• 📅 Publication date
• 🗞️ Publication
• 📂 Category
• 🌍 Source domain
• 📑 Article content
• 📝 Summary
• 🔑 Keywords
• 🌐 URL
• 🌐 Language
• ⏱️ Word count & estimated reading time

🔧 Installation

Install from PyPI with pip:

pip install news-fetch

Or install from source:

git clone https://github.com/santhoshse7en/news-fetch.git
cd news-fetch
pip install .

🚀 Usage

To scrape all the news details, use the newspaper function:

from newsfetch.news import Newspaper

news = Newspaper(url='https://www.thehindu.com/news/cities/Madurai/aa-plays-a-pivotal-role-in-helping-people-escape-from-the-grip-of-alcoholism/article67716206.ece')
print(news.headline)
# Output: 'AA plays a pivotal role in helping people escape from the grip of alcoholism'
print(news.word_count, news.reading_time_minutes)
# Output: 210 1

To discover recent article URLs from a news site (via its sitemaps/RSS feeds — no browser required):

from newsfetch.discovery import NewsSiteURLExtractor

site = NewsSiteURLExtractor(news_domain='https://www.bbc.com', limit=10)
for article in site.articles:
    print(article)
# Output: {'url': 'https://www.bbc.com/news/articles/...', 'title': '...', 'date': '2026-07-10T16:56:53Z'}

To scrape many URLs at once, with per-URL failures isolated:

from newsfetch.news import Newspaper

results = Newspaper.from_urls(site.urls, max_workers=5)
for result in results:
    if result is not None:
        print(result.headline)

🔗 Project Links

Source	Link
PyPI	pypi.org/project/news-fetch
Repository	github.com/santhoshse7en/news-fetch
Issue tracker	github.com/santhoshse7en/news-fetch/issues

🤝 Contributing

Pull requests are welcome! For major changes, please open an issue first to discuss what you'd like to change.

• 🍴 Fork the repo, make your change, and open a PR.
• 🐛 Found a bug or have a feature idea? File an issue.
• ⭐ Starring the repo costs nothing and genuinely helps — it's how other people searching for a news extractor find this one.

Please also read the Code of Conduct before participating.

📄 License

This project is licensed under the MIT License.