moneycontrol-datascraper 0.2.0

Polite Moneycontrol data collector for trading research. Respects robots.txt; no anti-bot bypass.

moneycontrol_scraper_safe

A polite Moneycontrol data collector intended for trading research.

Principles

• Respects robots.txt for every request via urllib.robotparser.
• No bypassing of Cloudflare, login walls, paywalls, captchas, or rate limits.
• Prefers official RSS feeds over HTML scraping.
• Adds delay between requests (default 3s, honors any declared Crawl-delay).
• Stores both raw and clean data so research is reproducible.
• Outputs pandas DataFrames plus JSON / CSV files.

If a page is disallowed by robots.txt, or the server returns 401 / 403 / 429, the scraper stops and does not retry with tricks.

Install

pip install -r requirements.txt

Usage

# Fetch all configured RSS feeds and write clean output
python main.py rss --sentiment

# Limit to specific categories
python main.py rss --category latest_news ipo mutual_funds

# Fetch the public homepage / markets page (only if robots.txt allows)
python main.py html

# Both
python main.py all --sentiment

Set a contact email so site operators can reach you:

export MC_CONTACT="[e-mail]"

Modules

Module	Purpose
config.py	URLs, feed list, delays, paths
robots_checker.py	is_allowed() / crawl_delay() via urllib.robotparser
rss_collector.py	Polite feed fetcher using requests + feedparser
html_collector.py	Polite public-page fetcher (no auth, no bypass)
parser.py	RSS + HTML → clean pandas DataFrames
sentiment.py	Lightweight finance-flavoured sentiment scoring
storage.py	Save raw payloads (data/raw/) and clean output (data/clean/)
cli.py	argparse CLI: rss, html, all

Data layout

data/
├── raw/      # JSON payloads of feeds, raw HTML pages
└── clean/    # CSV + JSON of parsed/clean DataFrames

Categories collected

Latest market news, stock news, IPO news, economy news, business news, commodity news, personal finance news, mutual fund news, results / company headlines, and a best-effort market overview from the public homepage / markets page (only if robots.txt permits).

Disclaimer

This project is for personal research. Use the data in compliance with Moneycontrol's terms of service. Do not redistribute scraped content.