news-watch 1.2.0

news-watch is a Python package that scrapes structured news data from Indonesia's top news websites, offering keyword and date filtering queries for targeted research.

news-watch: Indonesia's top news websites scraper

Scope: Reproducible MBG topic-modeling pass over the approved window 2025-01-05 → 2026-07-17, after nr_topics=15 reduction (14 substantive topics + 3,192 outliers). Final cleaned-corpus fingerprint 470acdf80dbad4c4c66a42716c53be78eea08e04109b926bff101af58025f666.

placeholder

Corpus identity (final cutoff, derived from the authoritative data/analysis/gate_report.json and tables/run_summary.json; the legacy data/clean/gate_report.json is a byte-identical private mirror of the same report):

Stage	Count
Raw rows ingested across all increments	25,288
Rows surviving the relevance gate	18,406
Link-unique rows after dedup	8,895
Cleaned rows that drive BERTopic	8,862
Distinct contributing sources	48
playwright install chromium

Development setup: see https://okky.dev/news-watch/getting-started/

## Performance Notes

**⚠️ Works best locally.** Cloud environments (Google Colab, servers) may experience degraded performance or blocking due to anti-bot measures.

Some scrapers may work on a local machine but fail on remote servers, Linux CI, or GitHub Actions. This usually happens because of anti-bot protection, rate limits, geolocation differences, JavaScript rendering differences, or sudden source-side changes.

### Using a proxy

When running on a server or behind anti-bot blocks, route requests through a residential/datacenter proxy. The proxy applies to every layer (aiohttp, rnet, and the Playwright fallback):

```bash
# via flag
newswatch -k ihsg -sd 2025-01-01 --proxy "http://proxy.example.com:8080"

# via env (also honors standard HTTPS_PROXY / HTTP_PROXY)
export NEWSWATCH_PROXY="socks5://proxy.example.com:1080"
newswatch -k ihsg -sd 2025-01-01

import newswatch as nw
df = nw.scrape_to_dataframe("ihsg", "2025-01-01", proxy="http://proxy.example.com:8080"

Other reliability overrides (env vars): NEWSWATCH_USER_AGENT (custom User-Agent), NEWSWATCH_MAX_RETRIES (retry count, default 3).

Usage

To run the scraper from the command line:

newswatch --method <search|latest> -k <keywords> -sd <start_date> -s [<scrapers>] -of <output_format> -v

Command-Line Arguments

Argument	Description
--method	Retrieval method: search (default) or latest
-k, --keywords	Comma-separated keywords to scrape (required for search, optional for latest)
-sd, --start_date	Start date in YYYY-MM-DD format (required for search, ignored in latest)
-s, --scrapers	Scrapers to use: specific names (e.g., "kompas,viva"), "auto" (default, platform-appropriate), or "all" (force all, may fail)
-of, --output_format	Output format: csv, xlsx, json, or jsonl (default: csv)
-o, --output_path	Custom output file path (optional)
-v, --verbose	Show detailed logging output (default: silent)
--list_scrapers	List all supported scrapers and exit
--health-report	Run source health probes and print status table. JSON/CSV via --output_path
--limit	Maximum number of articles to collect in latest mode
--max-pages	Maximum pages to fetch per scraper in latest mode
--scraper-timeout	Per-scraper timeout in seconds
--progress	Print per-scraper progress lines
--daterange	Filter articles by an inclusive date window. Format: YYYY-MM-DD/YYYY-MM-DD (e.g. 2026-07-13/2026-07-14); start = 00:00:00, end = 23:59:59.999999 of the same day
--dedup-file	Path to a previous output file (JSON/JSONL/CSV); articles with matching links are skipped
--proxy	Proxy URL for all requests (e.g. http://proxy.example.com:8080 or socks5://proxy.example.com:1080). Also via NEWSWATCH_PROXY env

Examples

# Basic usage
newswatch --keywords ihsg --start_date 2025-01-01

# Latest monitoring mode
newswatch --method latest --scrapers "antaranews,kompas,viva"

# Multiple keywords with specific scraper
newswatch -k "ihsg,bank" -s "tempo" --output_format xlsx -v

# List available scrapers
newswatch --list_scrapers

Python API Usage

import newswatch as nw

# Basic scraping - returns list of article dictionaries
articles = nw.scrape("ekonomi,politik", "2025-01-01")
print(f"Found {len(articles)} articles")

# Get results as pandas DataFrame for analysis
df = nw.scrape_to_dataframe("teknologi,startup", "2025-01-01")
print(df['source'].value_counts())

# Latest monitoring
latest = nw.latest_to_dataframe(scrapers="antaranews,kompas,viva")
print(latest[["source", "title"]].head())

# Save directly to file
nw.scrape_to_file(
    keywords="bank,ihsg", 
    start_date="2025-01-01",
    output_path="financial_news.xlsx"
)

# Quick recent news
recent_news = nw.quick_scrape("politik", days_back=3)

# Get available news sources
sources = nw.list_scrapers()
print("Available sources:", sources)

See the practical guide for end-to-end CLI and Python-API examples and common research patterns. For interactive examples, see the API reference notebook.

Run on Google Colab

You can run news-watch on Google Colab

Output

The scraped articles are saved as a CSV, XLSX, JSON, or JSONL file in the current working directory with the format news-watch-{keywords}-YYYYMMDD_HH.

Each record has 8 fields: title, publish_date, author, content, keyword, category, source, link. (scrape_timestamp exists on the internal Article model but is not emitted to output.)

The output file contains the following columns:

• title
• publish_date
• author
• content
• keyword
• category
• source
• link

Retrieval Methods

• search is the default and keeps the current keyword/date research workflow.
• latest is intended for latest-news monitoring and does not require keywords.

Supported Websites (75)

Alinea.id, Al Jazeera, Antara News, AP News, Bali Post, Banten News, BBC News, Berita Jatim, BeritaSatu, Betahita, Bisnis.com, Bloomberg Technoz, CNA Indonesia, CNBC Indonesia, CNN Indonesia, The Conversation Indonesia, DailySocial, Dandapala, DDTC News, Detik, Fajar, Galamedia, Gatra, Good News From Indonesia, Grid, Harian Jogja, Hipwee, Hukumonline, IDN Financials, IDN Times, Independen.id, iNews, Investor.id, Jakarta Globe, The Jakarta Post, Jakarta Selaras, Jawa Pos, JPNN (Jawa Pos News Network), Kaltim Post (Borneo24), Katadata, KBR, Kompas, Kontan, Kumparan, Liputan6, Media Indonesia, Merdeka, MetroTV News, Mojok, Mongabay Indonesia, Niaga.Asia, NusaBali, Okezone, Pantau.com, Pikiran Rakyat, Poskota, Project Multatuli, Republika, RM.ID (Rakyat Merdeka), RMOL, RRI (RRI.co.id), SINDOnews, Suara, Suara Merdeka, Surabaya Pagi, SWA, Tempo, Tirto, Tribunnews, TVOne, TVRI News, Viva, VOA Indonesia, VOI.id, Warta Ekonomi

Notes:

• 75 registered sources: 70 with keyword search, 75 with latest mode.
• 72 stable scrapers in the current release: 68 with keyword search, 72 with latest mode.
• 3 sources under investigation.
• AP News uses topic hub pages with keyword-in-title filtering (robots disallows /search?q=*).
• Al Jazeera is latest-only via RSS feed (search page is JS-rendered).
• Reuters skipped (WAF blocked).
• Use -s all to force-run all scrapers (may cause errors/timeouts).
• Some sources are environment-sensitive and may fail on remote servers even if they work locally.
• Limitation: Kontan scraper maximum 50 pages.

Contributing

Contributions are welcome. Open an issue or pull request to add a source or improve an existing one.

Keep tests with their owning behavior: shared scraper contracts in tests/test_basescraper.py, registry metadata and discovery in tests/test_registry.py, and source-specific parsing and extraction in tests/test_scrapers_focused.py. Do not add catch-all files such as tests/test_new_scrapers.py. Before adding coverage, search the owning module and extend or parameterize an existing test when it already exercises the same observable behavior. Keep another case only for a distinct boundary, precedence rule, transition, or real error path; avoid assertions on source text, private plumbing, or registry counts.

License

This project is licensed under the MIT License - see the LICENSE file for details. The authors assume no liability for misuse of this software.

Citation

@software{mabruri_newswatch,
  author = {Okky Mabruri},
  title = {news-watch},
  year = {2025},
  doi = {10.5281/zenodo.14908389}
}

Related Work

• indonesia-news-scraper
• news-scraper