gem-scrapscrap 1.3.1

CLI tool to scrape and download contract documents from GeM (Government e-Marketplace)

GeM Document Scraper

A Python web app that scrapes and downloads contract documents from GeM (Government e-Marketplace). Enter a category, set a date range, and download all matching contract PDFs — with automatic CAPTCHA solving.

Features

• Web UI — Clean dark-themed interface, no terminal needed
• Category Search — Autocomplete from GeM's 13,000+ categories
• Auto CAPTCHA — Solves GeM's image CAPTCHAs via Tesseract OCR
• PDF Downloads — Downloads individual files or all as ZIP
• Real-time Progress — Live logs and progress bar while scraping
• Background Processing — Scraper runs in a thread, UI stays responsive

Demo

1. Enter a category (e.g., "Ball Point Pens (V2) as per IS 3705")
2. Set date range (DD-MM-YYYY)
3. Click "Start Scraping"
4. Watch live progress, download PDFs when done

Quick Start

Prerequisites

• Python 3.9+

Install

pip3 install -r requirements.txt

Run CLI (Interactive Terminal)

python3 gem_cli.py

Run Web UI

python3 app.py

Open http://localhost:5000 in your browser.

How It Works

1. Search — User enters category + date range in the web UI
2. CAPTCHA — Opens GeM's contract page, extracts the CAPTCHA image, reads it via Tesseract OCR, submits automatically (retries up to 15 times)
3. Scrape — Collects all matching contract documents from the results
4. Download — For each document, calls GeM's AJAX endpoint to get the download URL, then fetches the PDF via HTTP
5. Serve — Downloaded PDFs are served back through the web UI for browsing/downloading

Project Structure

GeM-Document-Scraper/
├── app.py              # Flask web server (routes, job management, threading)
├── scraper.py          # Background scraping engine (Selenium + AJAX downloads)
├── actions.py          # Selenium helper functions (category, dates, CAPTCHA)
├── main.py             # Legacy CLI entry point (still works standalone)
├── templates/
│   └── index.html      # Web UI (HTML + CSS + JS, no frameworks)
├── requirements.txt    # Python dependencies
├── downloads/          # Downloaded PDFs (per job)
└── captcha/            # CAPTCHA images (temporary)

API Endpoints

Method	Route	Description
GET	/	Web UI
POST	/scrape	Start a new scraping job
GET	/status/<job_id>	Job status + logs + file list
GET	/download/<job_id>/<file>	Download a single PDF
GET	/download_all/<job_id>	Download all PDFs as ZIP

Tested Categories

These categories are known to have contracts on GeM:

Category	Notes
Ball Point Pens (V2) as per IS 3705	Office supplies, high volume
Note Sorting Machines (V2)	Banking equipment
LED Bulb with Battery as per IS 16102	Govt LED scheme
Split Air Conditioner, Wall Mount Type (V3) ISI Marked to IS 1391 (Part 2)	HVAC
Nitrogen Tyre Inflators	Automotive

Use a wide date range (e.g., 01-01-2025 to 07-07-2025) for more results.

Date Format

The web UI accepts multiple formats and auto-converts:

• DD-MM-YYYY (e.g., 01-07-2025)
• DD/MM/YYYY (e.g., 01/07/2025)
• YYYY-MM-DD (e.g., 2025-07-01)

Dependencies

Package	Purpose
Flask	Web server
Selenium	Browser automation
Tesseract OCR (pytesseract)	CAPTCHA reading
Pillow	Image processing
Requests	HTTP file downloads
Gunicorn	Production WSGI server

Deploy to Railway

1. Go to railway.app → Sign up with GitHub
2. Click New Project → Deploy from GitHub repo
3. Select Wickcore/gem-scrapscrap
4. Railway auto-detects the Dockerfile
5. Click Deploy

Your app will be live at https://your-app.up.railway.app

Note: Railway gives $5 free credits/month — enough for this app.

Notes

• CAPTCHA accuracy depends on image quality; the scraper retries up to 15 times
• Headless Chrome is used — no browser window pops up
• Each scraping job runs in its own thread with its own download directory
• GeM may change their page structure, which could break selectors

License

MIT