kmvahini
High-Performance Agricultural Market Data Toolkit for Karnataka (KSAMB / Krishi Marata Vahini)
kmvahini provides an asynchronous and parallel interface to programmatically extract, clean, cache, and analyze daily agricultural commodity price and arrival data from the Karnataka State Agricultural Marketing Board (KSAMB).
Key Features
- ⚡ High-Throughput Parallel Scraping: Decomposes multi-year and multi-commodity queries into concurrent subtasks utilizing up to 80% of available CPU cores, slashing scrape times by up to 20x (e.g. 25 years of data extracted in ~40 seconds).
- 🛡️ Robust Error Recovery & Edge Handling: Gracefully handles "No Records Found" periods, dropdown variations, and transient network timeouts with automatic retries and headless browser recycling.
- 🧹 Clean, Analysis-Ready Data: Returns standardized
pandas.DataFrameobjects with formatted DatetimeIndex, forward-filled markets, and sanitized categorical values. - 💾 Dual-Tier Caching (Local & Hugging Face Cloud): Built-in Parquet caching with automatic month-level deduplication and cloud synchronization.
- 📊 260+ Commodities & 160+ Markets: Historical records dating from 2002 to present.
Installation
Install via pip:
pip install kmvahini
To install with cloud caching support:
pip install "kmvahini[cloud]"
For development / editable installation:
git clone https://github.com/patilmanojkumar/kmvahini.git
cd kmvahini
pip install -e .
Quick Start
1. High-Speed Parallel Scraping with scrape_website
The core scrape_website function automatically parallelizes across available CPU cores:
import calendar
from kmvahini.scraper import scrape_website
# Define parameters
years = [str(y) for y in range(2010, 2026)]
months = [m.upper() for m in calendar.month_name if m] # All 12 months
commodities = ["SAFFLOWER", "SOYABEEN", "MAIZE"]
markets = ["AllMarkets"]
# Scrape using automatic ~80% CPU core parallelization
df = scrape_website(
months=months,
years=years,
commodities=commodities,
markets=markets,
workers=None, # None = auto 80% cores; or pass an explicit integer e.g. 32
show_progress=True # Unified tqdm progress bar
)
print(df.info())
print(df.head())
2. Incremental Caching API with get_data
For pipelines requiring incremental synchronization and local Parquet caching:
import kmvahini as kmv
# Fetches from local cache if present, otherwise scrapes missing months
df = kmv.get_data(
commodity="SAFFLOWER",
market="AllMarkets",
start_year=2020,
end_year=2025
)
Performance & Architecture
Under standard sequential scraping, querying 25 years of daily market records (300 monthly form submits) takes ~18 minutes per commodity due to single-browser request roundtrips and DOM rendering waits.
kmvahini introduces (Commodity × Year) task decomposition powered by Python's ProcessPoolExecutor:
User Query (e.g., SAFFLOWER, 2002–2026)
│
├── Task 1: (SAFFLOWER, 2002) ──► Worker Process 1 (Headless Chrome)
├── Task 2: (SAFFLOWER, 2003) ──► Worker Process 2 (Headless Chrome)
├── ...
└── Task 25: (SAFFLOWER, 2026) ─► Worker Process 25 (Headless Chrome)
│
▼
Consolidated, Sorted & Cleaned DatetimeIndexed DataFrame (~40 seconds total)
Benchmark on Multi-Core Workstations
| Query | Sequential Time | kmvahini Parallel (80% Cores) |
Speedup |
|---|---|---|---|
| 1 Commodity (25 Years) | ~18 min | ~42 seconds | ~25x |
| 5 Commodities (25 Years) | ~90 min | ~3.5 minutes | ~26x |
API Reference
kmvahini.scraper.scrape_website
scrape_website(
months: List[str],
years: List[str],
commodities: List[str],
markets: List[str],
workers: Optional[int] = None,
show_progress: bool = True
) -> pd.DataFrame
months: List of month names (case-insensitive:"JANUARY","January", etc.).years: List of four-digit year strings (e.g.["2023", "2024"]).commodities: List of commodity names as displayed in KSAMB dropdowns.markets: List of market names (use["AllMarkets"]for all markets).workers: Number of parallel worker processes. DefaultNoneutilizesint(cpu_count * 0.8). Setworkers=1for sequential execution.show_progress: Set toFalseto silence progress indicators.
Legal & Compliance Disclaimer
Krishi Marata Vahini is the registered trademark and service of the Karnataka State Agricultural Marketing Board (KSAMB) and the Department of Agricultural Marketing, Government of Karnataka.
kmvahini is an independent open-source research toolkit and is not affiliated, sponsored, or endorsed by KSAMB or the Government of Karnataka. Data retrieved via this package is subject to the source terms published at KSAMB Terms of Use.
License & Contributing
Distributed under the MIT License. See LICENSE for details.
- Author: Manojkumar Patil (patil.manojkumar@hotmail.com)
- Source Code: GitHub
- Issue Tracker: GitHub Issues
Metadata
Release files for kmvahini 1.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| kmvahini-1.2.0.tar.gz | 19.0 kB | Without provenance | Details for kmvahini-1.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| kmvahini-1.2.0-py3-none-any.whl | Python 3 | none | any | Details for kmvahini-1.2.0-py3-none-any.whl |
Total release size: 37.9 kB
Release files / kmvahini-1.2.0.tar.gz
| Download URL | kmvahini-1.2.0.tar.gz |
|---|---|
| Size | 19.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8f6e84f30e3ae4e86ee90ebfad529637d28f45d37dcf9d26b44da3e21c0e223d
|
|
BLAKE2b-256 checksum How to use checksums |
d0cbf5caf6caadbaf071fb0adcbf8163693e9a4f913f18dd6d5de539eb99e996
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.19
|
Release files / kmvahini-1.2.0-py3-none-any.whl
| Download URL | kmvahini-1.2.0-py3-none-any.whl |
|---|---|
| Size | 18.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
497cf66f8251908bcaff9634d6546019b3e941ec38cf754ad7e4e0792bdf2e7e
|
|
BLAKE2b-256 checksum How to use checksums |
dd9c70ba0e1115933cb1049f3cf4226e0b4c7bb53b090403d949416aa10d1c03
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.19
|
Security for kmvahini 1.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.