cnbctv18_scraper_safe

A polite, robots.txt-respecting Python library for collecting public CNBC-TV18 news/financial content for research and NLP sentiment training only.

What it does NOT do

• Does not bypass logins, paywalls, captchas, Cloudflare, or anti-bot systems.
• Does not ignore robots.txt or rate limits.
• Does not scrape personal data.

If robots.txt cannot be loaded or disallows a URL, the fetcher refuses (fail-closed).

Install

pip install -r requirements.txt

Quick start

from cnbctv18_scraper import (
    collect_all_categories,
    parse_articles_dataframe,
    filter_valid_articles,
    add_content_hash, deduplicate,
    add_sentiment_column, add_impact_column,
    add_stock_symbols_column,
    save_dataframe, save_unique_to_sqlite,
)
import pandas as pd

urls     = collect_all_categories(["markets", "economy", "earnings"])
articles = parse_articles_dataframe(urls["url"].head(20).tolist())
articles = filter_valid_articles(articles)              # quality filter
articles = deduplicate(add_content_hash(articles))      # content-hash dedup
articles = add_sentiment_column(articles)               # +sentiment_*
articles = add_impact_column(articles)                  # +impact_score
# Optional ticker mapping (CSV with columns: symbol,name)
articles = add_stock_symbols_column(articles, pd.read_csv("stocks.csv"))

articles["news_id"] = articles["content_hash"]
save_dataframe(articles, "articles", fmt="csv")
save_unique_to_sqlite(articles)                         # upsert into SQLite

CLI

python main.py --category markets --limit-per-category 20 --max-articles 100
python main.py --category all --max-articles 50 --stocks stocks.csv
python main.py --category economy --no-sqlite

CLI flags: --category, --limit-per-category, --max-articles, --stocks, --db, --no-sqlite.

Pipeline

Category + RSS + Sitemap URLs
   → robots check (built into fetcher)
   → fetch allowed pages (delay + retry + timeout)
   → parse article
   → quality filter      (is_valid_article)
   → deduplicate         (content_hash)
   → stock mapping       (map_stock_symbols)
   → sentiment score     (add_sentiment_column)
   → impact score        (add_impact_column)
   → store CSV + SQLite  (save_unique_to_sqlite)

Layout

cnbctv18_scraper_safe/
├── requirements.txt
├── main.py
├── README.md
├── cnbctv18_scraper/
│   ├── __init__.py
│   ├── config.py          # base URL, category map, headers, delays
│   ├── http_client.py     # requests session w/ retries + backoff
│   ├── robots_checker.py  # cached robots.txt enforcement
│   ├── html_collector.py  # polite single-page fetcher (BeautifulSoup)
│   ├── url_collector.py   # discover article URLs from category pages
│   ├── parser.py          # parse + quality-filter articles
│   ├── dedup.py           # content_hash + deduplicate
│   ├── sentiment.py       # sentiment + impact + stock-symbol mapping
│   └── storage.py         # CSV / Parquet / JSON + SQLite upsert
└── data/
    ├── raw/               # raw HTML dumps
    └── clean/             # CSV/Parquet/JSON outputs

Categories

Configured in cnbctv18_scraper/config.py — includes latest_news, markets, stocks, economy, finance, business, earnings, technology, real_estate, auto, telecom, aviation, infrastructure, healthcare, agriculture, commodities, currency, expert_views, personal_finance, videos, and more.

Settings (in config.py)

Sentiment

sentiment.py ships a small finance lexicon (gain/loss/beat/miss/upgrade/...). It's a baseline only — swap in FinBERT or a fine-tuned model for serious work.

License / use

Research and educational use only. You are responsible for complying with CNBC-TV18's Terms of Service and applicable law in your jurisdiction.