livemint-scraper-safe 0.1.0

Ethical LiveMint news collector (RSS-first, robots-aware) that produces a clean NLP dataset for sentiment training and swing-trading research.

livemint_scraper_safe

A safe, ethical Python library for collecting public LiveMint financial and news data — designed as a clean NLP dataset builder for sentiment training and swing-trading research.

This library does not produce trading signals on its own. Combine its sentiment output with technicals, volume, sector strength, and risk/reward in your trading engine. Suggested mix: Technical 40% | News Sentiment 25% | Volume 15% | Sector 10% | R:R 10%.

Principles

• Prefers official RSS feeds.
• Respects robots.txt.
• Does not bypass login, paywall, captcha, Cloudflare, or rate limits.
• Adds delay, retry, timeout, and logging.
• Saves raw and clean data to CSV and SQLite.
• Returns pandas DataFrames.

Pipeline

LiveMint RSS
   ↓
Deduplicate (md5 of title+link)
   ↓
Fetch Article Text (robots-aware)
   ↓
Normalize Date (dateutil)
   ↓
Map Stock Symbols (NSE/BSE tickers)
   ↓
Sentiment + Impact + Confidence
   ↓
Store in CSV + SQLite

Install

pip install -r requirements.txt

Run

python main.py

Outputs:

• data/clean/livemint_news_<timestamp>.csv
• data/livemint.db (SQLite, table news_clean)
• livemint_scraper.log

Programmatic use

import pandas as pd
from livemint_scraper import run_pipeline

stocks = pd.DataFrame([
    {"symbol": "RELIANCE.NS", "name": "Reliance Industries"},
    {"symbol": "TCS.NS", "name": "Tata Consultancy Services"},
])

df = run_pipeline(stocks_df=stocks, fetch_articles=True, article_limit=20)
print(df.head())

Sentiment engine

sentiment.analyze(text) returns:

{
  "sentiment": "Bullish",
  "sentiment_score": 3,
  "impact": "High",
  "confidence": 82,
  "reason": "Detected bullish cues with high impact words."
}

Upgrade path (kept behind the same analyze() interface so callers don't change):

1. Keyword sentiment (current)
2. FinBERT
3. LLM-based sentiment
4. Market impact classifier

Output schema

news_id, source, category, title, summary, article_text, link, published, published_at, fetched_at, company_symbols, text_for_sentiment, sentiment_label, sentiment_score, impact_score, confidence_score, sentiment_reason.

Folder structure

livemint_scraper_safe/
├── requirements.txt
├── main.py
├── README.md
├── livemint_scraper/
│   ├── __init__.py
│   ├── config.py
│   ├── http_client.py
│   ├── robots_checker.py
│   ├── rss_collector.py
│   ├── html_collector.py
│   ├── parser.py
│   ├── enrich.py
│   ├── dedup.py
│   ├── dates.py
│   ├── stocks.py
│   ├── sentiment.py
│   ├── storage.py
│   ├── logger.py
│   └── pipeline.py
└── data/
    ├── raw/
    └── clean/