commoner-probe

Sousveillance infrastructure for the state's mandatory disclosure systems.

A commoner probes the state's own paperwork — parliamentary questions, committee reports, state assembly records — and turns it into evidence. commoner-probe automates the acquisition so you can focus on the analysis.

pip install "commoner-probe[all]"

import commoner_probe as probe   # alias used throughout these docs

Source status

A weekly check tests whether each data source still returns current, parseable data. For what each status means, see Source freshness.

41 ✅ fresh · 9 ❌ broken · 2 🔴 down · 4 ❓ unreachable

Why this exists

Parliamentary questions, committee reports, floor debates, bills, state assembly records, CSR exports, public mining-district disclosures, Union Budget files, and faculty-recruitment ads from public universities are mandatory or official public disclosures. The data exists. The problem is that it lives across undocumented portals with inconsistent APIs, no bulk export, and PDFs that require extraction to read programmatically.

commoner-probe handles the entire acquisition pipeline:

public disclosure portals  →  manifest.jsonl  →  files/PDFs  →  extracted records  →  your analysis
                               (metadata)        (raw source)      (structured text)

Classification, topic modelling, and dossier generation are intentionally out of scope. This library does one thing: acquire public disclosure data into provenance-rich, schema-validated JSONL and source files.

Install

Requires Python 3.11+. Released on PyPI.

pip install "commoner-probe[all]"          # everything needed for acquisition + extraction
pip install "commoner-probe[all,dev]"      # + schema validation, tests, lint

The core package has zero required dependencies; each capability is an extra:

Five-minute quickstart

Step 1 — Write a topic profile

{
  "name": "climate",
  "description": "Climate change and environmental policy",
  "search_groups": {
    "climate": ["climate change", "global warming", "net zero"],
    "air_quality": ["air pollution", "AQI", "particulate matter"]
  },
  "lok_sabha_ministries": ["ENVIRONMENT", "POWER", "PETROLEUM"],
  "rajya_sabha_ministry_likes": ["ENVIRONMENT", "POWER", "PETROLEUM"]
}

Step 2 — Probe parliamentary questions

commoner-probe sansad \
  --topic topic.json \
  --out data/climate \
  --house both \
  --from-date 2019-01-01

Writes data/climate/manifest.jsonl — one record per question from both houses.

Step 3 — Probe committee reports

commoner-probe committees \
  --topic topic.json \
  --out data/climate-committees \
  --house both

One record per standing committee report (LS and RS DRSCs).

Step 4 — Extract text from PDFs

commoner-probe extract-answers --out data/climate
commoner-probe extract-answers --out data/climate-committees

Parses downloaded PDFs into answers.jsonl: Q/A pairs, committee recommendations, and government responses.

Step 5 — Load in Python

import commoner_probe as probe

c = probe.Corpus("data/climate")

for r in c.manifest_qa():
    print(r.date, r.house, r.ministry, r.title)

for pair in c.join_qa():
    if pair.answers:
        print(pair.manifest.title)
        print(pair.answers[0].question_text[:200])

What you can study

Parliamentary questions (Lok Sabha + Rajya Sabha)

Each record carries who asked (MP name, party, state), which ministry answered, question number, type (starred / unstarred), date, session, and the full PDF. After extract-answers — extracted question and answer text.

Typical research questions: ministry responsiveness rates, which MPs ask the most questions by topic, how the same policy question evolves across sessions, party-level questioning patterns.

import commoner_probe as probe
from collections import Counter

c = probe.Corpus("data/climate")
ministry_counts = Counter(r.ministry for r in c.manifest_qa())
for ministry, n in ministry_counts.most_common(10):
    print(f"{ministry}: {n}")

Standing committee reports (LS + RS DRSCs)

Committee reports come in four shapes:

Action Taken Reports (ATRs) are the government's formal written responses to committee recommendations. The atr-linkage command connects each ATR back to the original report, enabling lifecycle analysis: recommendation → government rejection/acceptance → follow-up.

import commoner_probe as probe

c = probe.Corpus("data/climate-committees")

for chain in c.join_atr_chain():
    print(f"Report: {chain.original and chain.original.title}")
    print(f"  Recommendations: {len(chain.original_observations)}")
    print(f"  Government responses: {len(chain.atr_answers)}")

Floor debates (Lok Sabha)

debates acquires the Lok Sabha "text of debate" record: one PDF transcript per sitting day. It enumerates sitting dates per Lok Sabha / session, then fetches each day's transcript (optionally downloading the PDF with a SHA-256). It is a day-by-day document acquisition — verbatim text and per-speaker segmentation are left to a downstream extraction step. The richest longitudinal record of what is said on the floor.

commoner-probe debates \
  --out data/debates \
  --loksabhas 18 \
  --download

Bills and legislation

bills fetches the sansad.in legislation catalog — every bill with its introduction date, stage dates, and status — deduplicated by a stable key (no topic profile needed; the bill list is an exhaustive catalog). Enables tracking legislative velocity, committee-scrutiny rates, and private-member-bill outcomes.

commoner-probe bills \
  --out data/bills \
  --house both \
  --bill-type "Private Member"

--download also fetches each bill's documents: the as-introduced text, the passed versions, the gazette, the synopsis, the committee report and the errata. It is off by default, because a full catalogue carries about 10,500 files and 5 GB. A resume makes no request for a URL whose outcome the manifest already records. Use --retry-failed to ask again for the failed set.

State assembly records (NeVA portals)

From 2020, sub-national governments have been adopting NIC's NeVA (National e-Vidhan Application) infrastructure under a centrally sponsored scheme run by the Ministry of Parliamentary Affairs. Most state assemblies are onboarding, though coverage varies. The state-assembly command probes any NeVA portal:

commoner-probe state-assembly \
  --portal gujarat \
  --state GJ \
  --out data/gujarat-assembly \
  --assemblies 15

State Acts, amendments, rules, and notifications (India Code)

India Code (indiacode.nic.in) is the government's own statutory-instrument archive: every state's Acts plus their amendments, rules, regulations, notifications, orders, circulars, ordinances, and statutes, each with a downloadable PDF. indiacode enumerates a state's full Act catalog and parses every instrument found on each Act's page.

commoner-probe indiacode --out data/indiacode --states "West Bengal"

import commoner_probe as probe

c = probe.Corpus("data/indiacode")
for r in c.manifest_indiacode():
    if r.is_amendment:
        print(r.state, r.short_title, r.instrument_date, r.description)

MCA CSR company-spend exports

The Ministry of Corporate Affairs CDM CSR data page exposes downloadable CSV exports by financial year. These records compare reporting/spending companies and project-sector amounts. They do not identify CSR consultants or implementing agencies unless MCA publishes that in the source export.

commoner-probe mca-csr \
  --out data/mca-csr \
  --years 2022-23,2021-22

import commoner_probe as probe

c = probe.Corpus("data/mca-csr")
for r in c.manifest_mca_csr():
    print(r.financial_year, r.status, r.filename)

Mines DMFT / PMKKKY disclosures

mines-dmft acquires raw Ministry of Mines and Odisha DMFT public disclosure files. Ministry CSVs are current cumulative snapshots timestamped by the source; treat them as snapshots, not fiscal-year series.

commoner-probe mines-dmft \
  --out data/mines-dmft \
  --sources mines-gov-in,odisha

Pair the executive disclosure snapshots with Sansad oversight records without flattening the source families:

commoner-probe evidence dmft \
  --mines-dmft-dir data/mines-dmft \
  --sansad-dir data/sansad/mines-dmft-pmkkky \
  --out data/evidence/dmft.json

Union Budget & RBI State-Finances

budget acquires fiscal source files: Union Budget SBE (Statement of Budget Estimates) spreadsheets — a static table of per-fiscal-year URL templates expanded across the requested demand numbers — and RBI State-Finances documents discovered from the RBI publication page. Each file is downloaded with existence-skip and a SHA-256, one budget_source_file record per file. Acquisition only: the spreadsheet→rows parsing stays downstream (it needs pandas).

commoner-probe budget \
  --out data/budget \
  --sources union-budget,rbi-state-finances \
  --demands 101,1,33

Academic faculty-recruitment ads

academic-jobs crawls Indian higher-education-institution (HEI) career pages for faculty-recruitment advertisements, driven by a bundled institution registry. Each ad becomes one academic_job_posting record; fetch/parse failures and empty-result cases are recorded so coverage gaps are visible rather than silent. (Migrated from an earlier academic-jobs scraper.)

commoner-probe academic-jobs \
  --out data/academic-jobs \
  --institutions iit-kharagpur,iit-bombay

UDISE+ school-census documents

udise-docs acquires the 86 documents the UDISE+ portal serves without an account: the Data Capture Format for each year from DISE 2009-10 to UDISE+ 2026-27, the annual report booklets, the metadata dictionaries and the departmental letters.

commoner-probe udise-docs --out data/udise-docs

The endpoint answers a request for a .pdf with a JSON envelope holding base64, and a name that has left the portal's bundle answers 200 with a body that is not a PDF. The probe unwraps the first and records the second as not_pdf rather than saving it.

Mirroring one site

mirror walks one host and saves every page and document it serves. Use it when the source is one author's body of work rather than an API.

commoner-probe mirror https://example.gov.in \
  --out data/example \
  --deadline 1800

It writes three artefacts and keeps all three current as the walk runs: manifest.jsonl with a sha256 per file, MANIFEST.txt in the staging-manifest format, and INDEX.md naming every page. A killed run therefore leaves a mirror somebody can read. A resume costs no request for a file the manifest vouches for, and --verify re-hashes every file the manifest names.

Commands

43 subcommands across parliament, courts, budgets, census, state registers and archives. commoner-probe --help lists them; docs/CLI.md documents each one with a worked example.

Topic profile

Controls what the probe acquires:

{
  "name": "libraries",
  "description": "Public library infrastructure and policy",
  "search_groups": {
    "public_libraries": ["public library", "rural library"],
    "policy": ["National Mission on Libraries", "RRRLF"]
  },
  "lok_sabha_ministries": ["CULTURE", "EDUCATION"],
  "rajya_sabha_ministry_likes": ["CULTURE", "EDUCATION"]
}

• search_groups — keyword groups for LS full-text search. Each query runs independently; results are union-deduped on key.
• lok_sabha_ministries — exact ministry filter for LS (case-sensitive).
• rajya_sabha_ministry_likes — ministry LIKE filter for RS (prefix match).

See examples/topics/ for working examples.

Output files

For complete field-level documentation see docs/SCHEMAS.md.

Entity resolution (--with-entities)

Pass --with-entities to commoner-probe sansad to resolve asker names to stable entity_id values. On first run the entity store is populated from the sansad.in MP roster; subsequent runs reuse the local cache.

Resolved entity IDs join across corpora and sessions — useful for studying the same MP's questioning behaviour over time or across houses.

Python API

import commoner_probe as probe

c = probe.Corpus("data/climate")

# Typed iterators
for r in c.manifest_qa():                 # ManifestQaRecord
    ...
for r in c.manifest_committee_reports():  # ManifestCommitteeReportRecord
    ...
for r in c.answers_qa():                  # AnswerQaResponse
    ...
for r in c.answers_atr():                 # AnswerAtrResponse
    ...
for r in c.answers_dfg():                 # AnswerDfgRecommendation
    ...
for r in c.atr_linkages():                # AtrLinkageRecord
    ...
for r in c.manifest_mca_csr():            # ManifestMcaCsrRecord
    ...
for r in c.manifest_mines_dmft():         # ManifestMinesDmftRecord
    ...
for r in c.manifest_doe_pay_allowances(): # ManifestDoePayAllowancesRecord
    ...
for r in c.vacancy_rows():                # VacancyRowRecord
    ...
for r in c.runs():                        # RunRecord
    ...

# Join helpers
for pair in c.join_qa():                  # manifest + extracted answers
    ...
for chain in c.join_atr_chain():          # ATR + original report + observations
    ...

# pandas (pip install commoner-probe[pandas])
df = c.to_dataframe("manifest_committee_reports")

See examples/usage.py for a runnable walkthrough. See docs/ENDPOINTS.md for source-family endpoint notes. See docs/GOV_SITE_PLATFORMS.md for which Union ministry websites are scrapeable (and which are JS-rendered SPAs, WAF- blocked, or unreachable) — read before adding a new ministry-ddg portal.

Source freshness

Government portals change without notice, and a source can keep answering while its data stops updating. A weekly workflow checks each source and records the result in SOURCES.md. When any status changes, the workflow opens or updates a pull request on the bot/source-freshness branch. The Source status section at the top of this README shows every source's current status. The workflow rewrites it each run, along with SOURCES.md, so don't edit it by hand.

To mark a host as geo-fenced after you verify it by hand, add it to geo_fenced in staleness/sources.json with a note and the date you verified it, for example "cag.gov.in": {"note": "Blocks non-India IPs", "verified": "2026-10-05"}. To run the check yourself, run python scripts/check_sources.py. To check one source, add --only <id>. As each source finishes, the check prints one line to stderr with its position, ID, outcome, and time taken, for example [29/55] mines-dmft-odisha: no response (6m 4s), so the workflow's live log shows how far a run has got.

Contributing

Bug reports, portal breakage reports, and pull requests are welcome at github.com/CommonerLLP/commoner-probe. See CONTRIBUTING.md for development setup and conventions, and CODE_OF_CONDUCT.md for community expectations. Release history lives in CHANGELOG.md.

Government portals change without notice — if a probe stops working, an issue with the failing command and its probe.log output is the most useful report.

License

MIT License — see LICENSE.

commoner-probe is sousveillance infrastructure, built for the commons. It is released under the permissive MIT license so it can serve as a shared acquisition floor that any downstream project — including the other repos in the CommonerLLP federation, whatever their own licenses — can build on without copyleft friction.

Upcoming

MP profiles and career timelines

Structured biographical data for each member: constituency, state, party, terms served, educational background, declared profession. Pairs with the Q/A corpus for studies of how MP background predicts parliamentary participation.