business-standard-scraper-safe

A polite, robots.txt-respecting Python library that collects public Business Standard news for Indian market sentiment and swing-trading research.

This package is a data collector only. Prediction logic belongs in a separate package (e.g. swing_prediction_engine).

Features

• Sitemap + RSS ingestion, deduped by URL
• robots.txt enforcement on every request
• Polite client: configurable delay, retry/backoff, timeout, structured logging
• Article parsing: title, summary, body, published date
• SHA256 content-hash deduplication
• Stock-symbol mapping against an NSE master list
• Structured sentiment output: sentiment, impact, confidence, reason
• Storage: CSV snapshots + SQLite upsert (PostgreSQL-ready schema)
• CLI (bs-scraper) and importable Python API

Install

pip install business-standard-scraper-safe

Quick start

CLI

bs-scraper --limit 100 --category markets
bs-scraper --limit 50  --category all --db data/business_standard.db

CLI flags:

Python API

from bs_scraper import run, fetch_all_urls, enrich_articles, analyze, setup_logging

setup_logging()

# Full pipeline (sitemap+RSS → enrich → dedup → sentiment → stocks → CSV+SQLite)
df = run(limit=100, category="markets")

# Or use the building blocks
urls_df = fetch_all_urls(limit=200)
articles_df = enrich_articles(urls_df.head(50))
print(analyze("Reliance posts record profit and beats estimates"))

Pipeline

Sitemap + RSS
   ↓
Filter by robots.txt
   ↓
Fetch public article pages (delay + retry)
   ↓
Parse title / summary / body / date
   ↓
Deduplicate (sha256 content hash)
   ↓
Map stock symbols (NSE)
   ↓
Sentiment + impact + confidence + reason
   ↓
SQLite upsert + CSV snapshot
   ↓
Consumed by your prediction engine

Output schema

CSV files land in data/clean/business_standard_news_<timestamp>.csv. SQLite table news_clean (primary key news_id, upserted on conflict):

Sentiment payload

{
  "sentiment": "Bullish",
  "impact": "High",
  "confidence": 82,
  "reason": "Positive cues: order, growth, wins"
}

Currently Level 1 (keyword-based). Upgrade path: FinBERT → LLM sentiment → market-impact classifier. Swap the body of bs_scraper/sentiment.py::analyze to upgrade.

Stock mapping

Ships with a small NSE sample. Drop a CSV at data/stocks_master.csv with columns symbol,name to use a full universe:

symbol,name
RELIANCE.NS,Reliance Industries
TCS.NS,Tata Consultancy Services

Rules of engagement

This library deliberately does not:

• ignore robots.txt
• scrape disallowed paths (/api/, /_next/, /static/, /assets/, /search?)
• bypass paywalls, login, captcha, Cloudflare, or rate limits
• spoof browsers or rotate identities

If a path or feed is blocked, it is logged and skipped.

Project layout

business_standard_scraper_safe/
├── pyproject.toml
├── LICENSE
├── MANIFEST.in
├── README.md
├── PUBLISHING.md
├── main.py
└── bs_scraper/
    ├── __init__.py
    ├── cli.py
    ├── pipeline.py
    ├── config.py
    ├── http_client.py
    ├── robots_checker.py
    ├── sitemap_collector.py
    ├── rss_collector.py
    ├── html_collector.py
    ├── enrichment.py
    ├── parser.py
    ├── sentiment.py
    ├── stocks.py
    ├── storage.py
    └── logging_setup.py

Publishing

See PUBLISHING.md. Never paste API tokens into chat, commits, or logs. Use environment variables (TWINE_USERNAME=__token__, TWINE_PASSWORD=pypi-…) or a chmod-600 ~/.pypirc.

Roadmap

• Full NSE master list at data/stocks_master.csv
• FinBERT / LLM sentiment
• Source reliability score
• FastAPI endpoints
• Scheduler (cron / APScheduler)
• Combine with Moneycontrol, ET, LiveMint
• PostgreSQL storage with UNIQUE(news_id) ON CONFLICT DO UPDATE

License

MIT