moneycontrol_scraper_safe

A polite Moneycontrol data collector intended for trading research.

Principles

• Respects robots.txt for every request via urllib.robotparser.
• No bypassing of Cloudflare, login walls, paywalls, captchas, or rate limits.
• Prefers official RSS feeds over HTML scraping.
• Adds delay between requests (default 3s, honors any declared Crawl-delay).
• Stores both raw and clean data so research is reproducible.
• Outputs pandas DataFrames plus JSON / CSV files.

If a page is disallowed by robots.txt, or the server returns 401 / 403 / 429, the scraper stops and does not retry with tricks.

Install

pip install -r requirements.txt

Usage

# Fetch all configured RSS feeds and write clean output
python main.py rss --sentiment

# Limit to specific categories
python main.py rss --category latest_news ipo mutual_funds

# Fetch the public homepage / markets page (only if robots.txt allows)
python main.py html

# Both
python main.py all --sentiment

Set a contact email so site operators can reach you:

export MC_CONTACT="you@example.com"

Modules

Data layout

data/
├── raw/      # JSON payloads of feeds, raw HTML pages
└── clean/    # CSV + JSON of parsed/clean DataFrames

Categories collected

Latest market news, stock news, IPO news, economy news, business news, commodity news, personal finance news, mutual fund news, results / company headlines, and a best-effort market overview from the public homepage / markets page (only if robots.txt permits).

Disclaimer

This project is for personal research. Use the data in compliance with Moneycontrol's terms of service. Do not redistribute scraped content.