gov-budget-data 0.3.0

Load and analyze reconciled public-budget data with Polars and DuckDB

gov-budget-data

gov-budget-data is the user-facing Python package for discovering, loading, and analyzing reconciled public-budget data. It uses Polars for DataFrames and DuckDB for projection, filtering, and SQL over the release's Parquet files.

The package is currently an alpha. Its default API fails closed around unverified data and ambiguous time-series observations rather than returning a plausible but underspecified number.

The package code is licensed under the MIT License. Data is distributed separately: releases that name CC0-1.0 dedicate only this project's transformed structured data under CC0 and do not relicense source PDFs, source-page images, or third-party records.

Install

gov-budget-data requires Python 3.12 or newer.

python -m pip install gov-budget-data

The wheel contains the reader and analysis code, not the data corpus. Code and data are versioned separately so users can pin a data snapshot without redownloading it for every package update.

Get a data release

Downloading and opening data are separate operations:

import gov_budget_data as gbd

cached = gbd.fetch("latest")
data = gbd.open_release(cached)
print(data.info().data_identity)

fetch() is the package's only network API. It downloads an immutable release into the user cache and validates the release-index contract, archive digest, safe archive layout, size limits, and the release's inner file manifest. Pass index_sha256=... when an independently published index digest is available and the exact index bytes should also be pinned. open_release() performs no network access; it opens and revalidates an already cached or unpacked release:

data = gbd.open_release("/path/to/unpacked/release")
print(data.info())

By default, fetch() uses the project's production release index at https://dpfqddwqzniu6.cloudfront.net/releases/index.json. Pass index_url=... or set GBD_RELEASE_INDEX_URL to use a mirror or another publisher. An explicit argument takes precedence over the environment variable. Set GBD_CACHE_DIR to override the platform-standard cache directory:

export GBD_RELEASE_INDEX_URL="https://mirror.example/releases/index.json"
export GBD_CACHE_DIR="/data/gov-budget-data-cache"

Then:

cached = gbd.fetch("latest")
data = gbd.open_release(cached)

latest is convenient for exploration. Pin an explicit data-release version in reproducible analyses.

Installing the Python package never downloads data, and opening a release never updates it silently.

Discover what is available

Start with the catalog instead of guessing file names:

for dataset in data.datasets():
    print(dataset.name, dataset.fiscal_year_min, dataset.fiscal_year_max)

profile = data.describe("collections_monthly_city")
print(profile.title)
print(profile.jurisdictions)
print(profile.time_grains)
print(profile.measures)
print(profile.columns)

describe() reports coverage, trusted-row counts, dimensions, observed measures and units, temporal grains, and the physical schema without materializing the full dataset as a DataFrame. Annual observations use fiscal_year with period=None; period remains an optional within-year locator rather than carrying a synthetic annual label.

Page-typed publications expose both an umbrella dataset and virtual family__table_type datasets. Call datasets() for the exact catalog shipped by a release.

Load facts

load() returns a Polars DataFrame. Common filters and column projection are pushed into DuckDB before data reaches Python:

frame = data.load(
    "qcmr_positions",
    fiscal_year=(2022, 2026),
    columns=[
        "document_id",
        "fiscal_year",
        "period",
        "entity_id",
        "label",
        "value",
        "value_unit",
        "source_page",
    ],
)

Use filters={...} for another physical column listed by describe():

frame = data.load(
    "phila_budget_detail_adopted",
    filters={"ctx_fund": "General Fund"},
    columns=["fiscal_year", "org_entity_id", "label", "value", "value_unit"],
)

For advanced work, query registered views directly:

result = data.sql(
    """
    SELECT fiscal_year, count(*) AS rows
    FROM qcmr_positions
    GROUP BY fiscal_year
    ORDER BY fiscal_year
    """
)

connection = data.connect()
try:
    print(connection.sql("SHOW TABLES").fetchall())
finally:
    connection.close()

Each dataset has a trusted view named for the dataset and a <dataset>_raw diagnostic view. The raw view is not a second source of truth; it exists to inspect rows withheld by the default trust boundary.

Trace facts to their source

Every current v5 data release includes a checksummed source-document catalog. Inspect coverage and acquisition provenance without opening fact tables:

sources = data.documents(
    jurisdiction="city_of_philadelphia",
    fiscal_year=(2025, 2026),
    eligible=True,
)

source = data.document("phila_budget_detail_adopted_fy26_book2")
print(source.source_url)
print(source.source_sha256)

Map a loaded row back to its one-based PDF page:

row = data.load(
    "phila_budget_detail_adopted",
    columns=["document_id", "source_page", "label", "value"],
).row(0, named=True)

receipt = data.citation(row["document_id"], page=row["source_page"])
print(receipt.label)
print(receipt.page_url)
print(receipt.source_sha256)

Some historical documents do not yet have a recoverable publisher or archive URL. Their stable document id, exact source SHA-256, and page locator remain available; source_url and page_url are then None.

Find stable entities

Labels in government publications change and can be reused in different contexts. Search the shipped entity catalog and query by stable IDs:

matches = data.search_entities(
    "police",
    dimension="department",
    jurisdiction="city_of_philadelphia",
)

for match in matches[:5]:
    print(match.entity.id, match.entity.name, match.score)

Other entity helpers include:

data.entities(dimension="fund")
data.entity("phl_city_wage_tax")
data.lineage(entity_id="phl_dept_parks_and_recreation")

Use the stable entity_id and org_entity_id fields where a dataset exposes them, and declared lineage instead of joining years on presentation labels. Fund and program context is currently exposed through its verified ctx_fund* and ctx_program* fields; not every contextual label has a stable entity mapping yet.

Build an ambiguity-safe series

series() selects one semantic observation per (value_year, period) and retains units, grain, and source coordinates:

wage = data.series(
    "collections_monthly_city",
    entity_id="phl_city_wage_tax",
    column_role="fytd_actual",
)

The function does not aggregate by default. It raises AmbiguousObservationError when a query spans incompatible tags, units, table types, row kinds, organizational contexts, or repeated observations across any publication edition. Narrow the query with semantic selectors; use aggregate=... only when the remaining observations truly share one meaning. Rolling publication families must declare their canonical edition policy in the normalized dataset; series() never guesses which revision should win.

The default entity_id selects the row's subject. For schedules where a department is contextual metadata around program, class, or position rows, set entity_field="org_entity_id" and use filters={...} to select the intended fund, program, or other grain.

Declared organizational succession can be followed explicitly when the selected dataset contains the predecessor observations:

human_services = data.series(
    "pa_general_fund_enacted",
    entity_id="pa_human_services",
    entity_field="org_entity_id",
    column_role="budget_estimate",
    row_kind="total",
    follow_lineage="backward",
)

Lineage seams remain visible in the result. Undeclared splits or apportionments fail rather than being guessed.

Schema and provenance

Dataset families vary, and Parquet files are unioned by name. Common columns fall into four groups:

Group	Representative columns
Release and document identity	document_id, document_type, jurisdiction, fiscal_year, period
Observation	entity_id, column_role, column_tag, value_year, value, value_unit
Organizational context	org_entity_id, ctx_department, ctx_fund, ctx_program
Evidence and trust	source_page, table_index, row_index, parser_name, table_reconciled, table_accounted, headers_verified

Call describe(name).columns for the authoritative physical schema in the release you opened. During the alpha period, prefer named projections over persisting assumptions about SELECT *.

Every released fact retains the source document id, one-based PDF/source page, table, and row coordinates needed to identify its evidence. Source PDFs are not bundled in the Python wheel or data archive. The release ships a checksummed source catalog with acquisition URLs and exact source hashes, plus its entity catalog, organizational lineage, authoritative extraction selection, source-integrity decision, and optional annotation sidecars under the same validated manifest.

Correctness boundary

By default:

• facts from source-quarantined documents are absent, while their provenance and quarantine decision remain visible in the source catalog;
• exactly one authoritative extraction backend is selected per document;
• rows without accounted totals checks or verified headers are excluded;
• evidence-pinned source discrepancies can be accounted without being misrepresented as arithmetically reconciled;
• units and semantic context are retained in series output; and
• release files are checked against their recorded size and SHA-256 digest.

include_unverified=True and other diagnostic opt-ins are for investigation. They do not promote those rows to trusted facts. Never aggregate mixed or unknown units, and do not treat a raw display label as a stable identity.

Local pipeline data and compatibility API

Repository contributors can still pass a pipeline artifacts root or packaged release directly to the module-level functions:

frame = gbd.load("qcmr_positions", source="/path/to/release")
datasets = gbd.datasets(source="/path/to/release")

Alternatively, set GBD_DATA_ROOT. An explicit DataStore from open_release() is recommended for applications because every call remains bound to one visible data snapshot.

The installed package version is exposed directly:

import gov_budget_data as gbd

print(gbd.__version__)

Documentation

The public documentation covers installation, available datasets, release management, the complete API, worked examples, and the package's trust model.