Skip to main content

kmvahini

PyPI version Python Versions License: MIT Code style: black

High-Performance Agricultural Market Data Toolkit for Karnataka (KSAMB / Krishi Marata Vahini)

kmvahini provides an asynchronous and parallel interface to programmatically extract, clean, cache, and analyze daily agricultural commodity price and arrival data from the Karnataka State Agricultural Marketing Board (KSAMB).


Key Features

  • ⚡ High-Throughput Parallel Scraping: Decomposes multi-year and multi-commodity queries into concurrent subtasks utilizing up to 80% of available CPU cores, slashing scrape times by up to 20x (e.g. 25 years of data extracted in ~40 seconds).
  • 🛡️ Robust Error Recovery & Edge Handling: Gracefully handles "No Records Found" periods, dropdown variations, and transient network timeouts with automatic retries and headless browser recycling.
  • 🧹 Clean, Analysis-Ready Data: Returns standardized pandas.DataFrame objects with formatted DatetimeIndex, forward-filled markets, and sanitized categorical values.
  • 💾 Dual-Tier Caching (Local & Hugging Face Cloud): Built-in Parquet caching with automatic month-level deduplication and cloud synchronization.
  • 📊 260+ Commodities & 160+ Markets: Historical records dating from 2002 to present.

Installation

Install via pip:

pip install kmvahini

To install with cloud caching support:

pip install "kmvahini[cloud]"

For development / editable installation:

git clone https://github.com/patilmanojkumar/kmvahini.git
cd kmvahini
pip install -e .

Quick Start

1. High-Speed Parallel Scraping with scrape_website

The core scrape_website function automatically parallelizes across available CPU cores:

import calendar
from kmvahini.scraper import scrape_website

# Define parameters
years = [str(y) for y in range(2010, 2026)]
months = [m.upper() for m in calendar.month_name if m]  # All 12 months
commodities = ["SAFFLOWER", "SOYABEEN", "MAIZE"]
markets = ["AllMarkets"]

# Scrape using automatic ~80% CPU core parallelization
df = scrape_website(
    months=months,
    years=years,
    commodities=commodities,
    markets=markets,
    workers=None,        # None = auto 80% cores; or pass an explicit integer e.g. 32
    show_progress=True   # Unified tqdm progress bar
)

print(df.info())
print(df.head())

2. Incremental Caching API with get_data

For pipelines requiring incremental synchronization and local Parquet caching:

import kmvahini as kmv

# Fetches from local cache if present, otherwise scrapes missing months
df = kmv.get_data(
    commodity="SAFFLOWER",
    market="AllMarkets",
    start_year=2020,
    end_year=2025
)

Performance & Architecture

Under standard sequential scraping, querying 25 years of daily market records (300 monthly form submits) takes ~18 minutes per commodity due to single-browser request roundtrips and DOM rendering waits.

kmvahini introduces (Commodity × Year) task decomposition powered by Python's ProcessPoolExecutor:

User Query (e.g., SAFFLOWER, 2002–2026)
    │
    ├── Task 1: (SAFFLOWER, 2002) ──► Worker Process 1 (Headless Chrome)
    ├── Task 2: (SAFFLOWER, 2003) ──► Worker Process 2 (Headless Chrome)
    ├── ...
    └── Task 25: (SAFFLOWER, 2026) ─► Worker Process 25 (Headless Chrome)
    │
    ▼
Consolidated, Sorted & Cleaned DatetimeIndexed DataFrame (~40 seconds total)

Benchmark on Multi-Core Workstations

Query Sequential Time kmvahini Parallel (80% Cores) Speedup
1 Commodity (25 Years) ~18 min ~42 seconds ~25x
5 Commodities (25 Years) ~90 min ~3.5 minutes ~26x

API Reference

kmvahini.scraper.scrape_website

scrape_website(
    months: List[str],
    years: List[str],
    commodities: List[str],
    markets: List[str],
    workers: Optional[int] = None,
    show_progress: bool = True
) -> pd.DataFrame
  • months: List of month names (case-insensitive: "JANUARY", "January", etc.).
  • years: List of four-digit year strings (e.g. ["2023", "2024"]).
  • commodities: List of commodity names as displayed in KSAMB dropdowns.
  • markets: List of market names (use ["AllMarkets"] for all markets).
  • workers: Number of parallel worker processes. Default None utilizes int(cpu_count * 0.8). Set workers=1 for sequential execution.
  • show_progress: Set to False to silence progress indicators.

Krishi Marata Vahini is the registered trademark and service of the Karnataka State Agricultural Marketing Board (KSAMB) and the Department of Agricultural Marketing, Government of Karnataka.

kmvahini is an independent open-source research toolkit and is not affiliated, sponsored, or endorsed by KSAMB or the Government of Karnataka. Data retrieved via this package is subject to the source terms published at KSAMB Terms of Use.


License & Contributing

Distributed under the MIT License. See LICENSE for details.

Metadata

Release files for kmvahini 1.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for kmvahini 1.2.0
File Size Uploaded Provenance Actions
kmvahini-1.2.0.tar.gz 19.0 kB Without provenance Details for kmvahini-1.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for kmvahini 1.2.0
File Interpreter ABI Platform Actions
kmvahini-1.2.0-py3-none-any.whl Python 3 none any Details for kmvahini-1.2.0-py3-none-any.whl

Total release size: 37.9 kB

Release files / kmvahini-1.2.0.tar.gz

Download URL kmvahini-1.2.0.tar.gz
Size 19.0 kB
Tags Source
SHA-256 checksum
How to use checksums
8f6e84f30e3ae4e86ee90ebfad529637d28f45d37dcf9d26b44da3e21c0e223d
BLAKE2b-256 checksum
How to use checksums
d0cbf5caf6caadbaf071fb0adcbf8163693e9a4f913f18dd6d5de539eb99e996
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.19

Release files / kmvahini-1.2.0-py3-none-any.whl

Download URL kmvahini-1.2.0-py3-none-any.whl
Size 18.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
497cf66f8251908bcaff9634d6546019b3e941ec38cf754ad7e4e0792bdf2e7e
BLAKE2b-256 checksum
How to use checksums
dd9c70ba0e1115933cb1049f3cf4226e0b4c7bb53b090403d949416aa10d1c03
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.19

Security for kmvahini 1.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.2.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page