gov-budget-data

gov-budget-data is the user-facing Python package for discovering, loading, and analyzing reconciled public-budget data. It uses Polars for DataFrames and DuckDB for projection, filtering, and SQL over the release's Parquet files.

The package is currently an alpha. Its default API fails closed around unverified data and ambiguous time-series observations rather than returning a plausible but underspecified number.

The package code is licensed under the MIT License. Data is distributed separately: releases that name CC0-1.0 dedicate only this project's transformed structured data under CC0 and do not relicense source PDFs, source-page images, or third-party records.

Install

gov-budget-data requires Python 3.12 or newer.

python -m pip install gov-budget-data

For the shared notebook presentation layer—Matplotlib and Altair themes plus Great Tables—install the optional extra:

python -m pip install "gov-budget-data[notebook]"

The civic default uses darker U.S. Web Design System color tokens; the quieter metropolitan palette is also included. Both work in Matplotlib and Altair and have light and soft canvas variants:

from gov_budget_data import presentation as pub

palette = pub.apply_style("civic")
pub.enable_altair_theme("civic")
print(palette.series)

The helpers also provide explicit dollar, percentage, and count axes; a horizontal unit label above the y scale; consistent panel proportions; thin horizontal-bar geometry; direct labels; source lines; and publication tables. See the repository's notebook style guide for complete examples.

The wheel contains the reader and analysis code, not the data corpus. Code and data are versioned separately so users can pin a data snapshot without redownloading it for every package update.

Get a data release

Downloading and opening data are separate operations:

import gov_budget_data as gbd
import polars as pl

cached = gbd.fetch("latest")
data = gbd.open_release(cached)
print(data.info().data_identity)

fetch() is the package's only network API. It downloads an immutable release into the user cache and validates the release-index contract, archive digest, safe archive layout, size limits, and the release's inner file manifest. Pass index_sha256=... when an independently published index digest is available and the exact index bytes should also be pinned. open_release() performs no network access; it opens and revalidates an already cached or unpacked release:

data = gbd.open_release("/path/to/unpacked/release")
print(data.info())

By default, fetch() uses the project's production release index at https://dpfqddwqzniu6.cloudfront.net/releases/index.json. Pass index_url=... or set GBD_RELEASE_INDEX_URL to use a mirror or another publisher. An explicit argument takes precedence over the environment variable. Set GBD_CACHE_DIR to override the platform-standard cache directory:

export GBD_RELEASE_INDEX_URL="https://mirror.example/releases/index.json"
export GBD_CACHE_DIR="/data/gov-budget-data-cache"

Then:

cached = gbd.fetch("latest")
data = gbd.open_release(cached)

latest is convenient for exploration. Pin an explicit data-release version in reproducible analyses.

Installing the Python package never downloads data, and opening a release never updates it silently.

Discover what is available

Start with the catalog instead of guessing file names:

for dataset in data.datasets():
    print(dataset.name, dataset.fiscal_year_min, dataset.fiscal_year_max)

profile = data.describe("collections_monthly_city")
print(profile.title)
print(profile.jurisdictions)
print(profile.time_grains)
print(profile.measures)
print(profile.columns)

coverage = data.coverage("collections_monthly_city")
print(coverage.documents)
print(
    coverage.editions.filter(
        pl.col("status").is_in(["internal_absent_candidate", "applicability_unknown"])
    )
)
print(coverage.unresolved)

describe() reports coverage, trusted-row counts, dimensions, observed measures and units, temporal grains, and the physical schema without materializing the full dataset as a DataFrame. Annual observations use fiscal_year with period=None; period remains an optional within-year locator rather than carrying a synthetic annual label.

coverage() performs the deeper release audit. Its documents, entities, and series frames report observation counts at progressively finer grains. Raw rows define the entity and exact-series inventory, so an all-unverified series remains visible with zero trusted observations instead of disappearing. The n_raw_* and n_trusted_* fields preserve that distinction; include_unverified=True changes which rows count as observed but does not erase it. editions is a review ledger of inferred exact-series/report candidates, and unresolved shows raw labels that still lack stable IDs while separating trusted from unverified rows. observed_underived keeps raw source editions visible when their temporal contract or catalog period cannot be derived. internal_absent_candidate means an edition falls between raw observations; applicability_unknown means it falls after the last raw observation. Neither is an expected or missing report, and neither is zero. Until producer-side applicability is declared, coverage_status remains coverage_unknown. For annual panels, each exact-series row reports the full year range between its first and last raw observation. absent_value_years have no raw row, while unverified_value_years have raw evidence but no trusted row; missing_value_years reflects the selected trust view.

Page-typed publications expose both an umbrella dataset and virtual family__table_type datasets. Call datasets() for the exact catalog shipped by a release.

Load facts

load() returns a Polars DataFrame. Common filters and column projection are pushed into DuckDB before data reaches Python:

frame = data.load(
    "qcmr_positions",
    fiscal_year=(2022, 2026),
    columns=[
        "document_id",
        "fiscal_year",
        "period",
        "entity_id",
        "label",
        "value",
        "value_unit",
        "source_page",
    ],
)

Use filters={...} for another physical column listed by describe():

frame = data.load(
    "phila_budget_detail_adopted",
    filters={"ctx_fund": "General Fund"},
    columns=["fiscal_year", "org_entity_id", "label", "value", "value_unit"],
)

For advanced work, query registered views directly:

result = data.sql(
    """
    SELECT fiscal_year, count(*) AS rows
    FROM qcmr_positions
    GROUP BY fiscal_year
    ORDER BY fiscal_year
    """
)

connection = data.connect()
try:
    print(connection.sql("SHOW TABLES").fetchall())
finally:
    connection.close()

Each dataset has a trusted view named for the dataset and a <dataset>_raw diagnostic view. The raw view is not a second source of truth; it exists to inspect rows withheld by the default trust boundary.

Trace facts to their source

Every current v5 data release includes a checksummed source-document catalog. Inspect coverage and acquisition provenance without opening fact tables:

sources = data.documents(
    jurisdiction="city_of_philadelphia",
    fiscal_year=(2025, 2026),
    eligible=True,
)

source = data.document("phila_budget_detail_adopted_fy26_book2")
print(source.source_url)
print(source.source_sha256)
print(source.publisher_reported_date)
print(source.retrieved_at)

Publisher-reported dates, checksummed catalog-observation times, archive captures, and project retrieval times remain distinct. Missing publication claims stay null; repository history is never substituted. Split artifacts also expose their parent publisher-object hash and one-based source page.

Map a loaded row back to its one-based PDF page:

row = data.load(
    "phila_budget_detail_adopted",
    columns=["document_id", "source_page", "label", "value"],
).row(0, named=True)

receipt = data.citation(row["document_id"], page=row["source_page"])
print(receipt.label)
print(receipt.page_url)
print(receipt.source_sha256)

Some historical documents do not yet have a recoverable publisher or archive URL. Their stable document id, exact source SHA-256, and page locator remain available; source_url and page_url are then None.

Find stable entities

Labels in government publications change and can be reused in different contexts. Search the shipped entity catalog and query by stable IDs:

matches = data.search_entities(
    "police",
    dimension="department",
    jurisdiction="city_of_philadelphia",
)

for match in matches[:5]:
    print(match.entity.id, match.entity.name, match.score)

Other entity helpers include:

data.entities(dimension="fund")
data.entity("phl_city_wage_tax")
data.lineage(entity_id="phl_dept_parks_and_recreation")

Use the stable entity_id and org_entity_id fields where a dataset exposes them, and declared lineage instead of joining years on presentation labels. Fund and program context is currently exposed through its verified ctx_fund* and ctx_program* fields; not every contextual label has a stable entity mapping yet.

Build an ambiguity-safe series

series() selects one semantic observation per (value_year, period) and retains units, grain, and source coordinates:

wage = data.series(
    "collections_monthly_city",
    entity_id="phl_city_wage_tax",
    column_role="fytd_actual",
    row_kind="total",
)

Current/prior tax-year cohorts use the same tax entity and an explicit qualifier:

current_wage = data.series(
    "collections_monthly_city",
    entity_id="phl_city_wage_tax",
    entity_qualifier="current",
    column_role="fytd_actual",
    row_kind="line_item",
)

The function does not aggregate by default. It raises AmbiguousObservationError when a query spans incompatible tags, units, table types, row kinds, organizational contexts, or repeated observations across any publication edition. Narrow the query with semantic selectors; use aggregate=... only when the remaining observations truly share one meaning. Default sums and means also refuse when an intended contributor is unverified, so a filtered-out row cannot silently understate a derived total. Passing include_unverified=True makes the result diagnostic; a sum with any null operand remains null rather than treating missing as zero. Rolling publication families must declare their canonical edition policy in the normalized dataset; series() never guesses which revision should win.

The default value_type="canonical" preserves the opened release's numeric dtype. New fact-schema v2 money series can be exact Decimal, while immutable v1 releases retain their original Float64 values. Statistical and charting libraries can request an explicit conversion of only the normalized value:

wage_for_model = data.series(
    "collections_monthly_city",
    entity_id="phl_city_wage_tax",
    column_role="fytd_actual",
    series_key="current_tax_year",
    value_type="float64",
)

For source-faithful chart labels, gbd.format_values(frame, style="source") adds display_value from the preserved source lexeme and declared unit basis without changing the canonical numeric value. It rejects mixed or unknown units rather than guessing.

V2 series also retain selected and raw source text, source-scale values, multipliers, displayed increments, and review evidence in sources. Common value metadata is available as top-level columns. Reviewed values are included by default only when their value, table, and header evidence all pass. For diagnostic aggregation, each source struct retains those operand trust gates, and the derived value is verified only when every operand passes them.

Every non-direct v2 value carries a content-bound evidence ID. Resolve it from the same verified release to inspect the immutable typed decision record:

receipt = data.explain_value(value_evidence_id)
print(receipt.value_disposition, receipt.evidence_detail)
print(receipt.source_value, receipt.value_multiplier, receipt.canonical_value)

retained_numeric describes the final disposition, not its trust status; use value_verified to distinguish trusted from diagnostic retained claims. Ordinary direct parses intentionally have a null evidence ID.

Some sources publish useful numeric breakdowns beside an already-counted parent row. Reviewed instances are retained as row_kind="memo": they are queryable series observations but are excluded from ETL table arithmetic. Select row kinds explicitly before aggregating raw load() output so a memo is not summed with its parent.

The default entity_id selects the row's subject. For schedules where a department is contextual metadata around program, class, or position rows, set entity_field="org_entity_id" and use filters={...} to select the intended fund, program, or other grain.

Declared organizational succession can be followed explicitly when the selected dataset contains the predecessor observations:

human_services = data.series(
    "pa_general_fund_enacted",
    entity_id="pa_human_services",
    entity_field="org_entity_id",
    column_role="budget_estimate",
    row_kind="total",
    follow_lineage="backward",
)

Lineage seams remain visible in the result. Undeclared splits or apportionments fail rather than being guessed.

Schema and provenance

Dataset families vary, and Parquet files are unioned by name. Common columns fall into four groups:

Call describe(name).columns for the authoritative physical schema in the release you opened. During the alpha period, prefer named projections over persisting assumptions about SELECT *.

Every released fact retains the source document id, one-based PDF/source page, table, and row coordinates needed to identify its evidence. Source PDFs are not bundled in the Python wheel or data archive. The release ships a checksummed source catalog with acquisition URLs and exact source hashes, plus its entity catalog, organizational lineage, authoritative extraction selection, source-integrity decision, and optional annotation sidecars under the same validated manifest.

Correctness boundary

By default:

• facts from source-quarantined documents are absent, while their provenance and quarantine decision remain visible in the source catalog;
• exactly one authoritative extraction backend is selected per document;
• rows without accounted totals checks or verified headers are excluded;
• evidence-pinned source discrepancies can be accounted without being misrepresented as arithmetically reconciled;
• units and semantic context are retained in series output; and
• release files are checked against their recorded size and SHA-256 digest.

include_unverified=True and other diagnostic opt-ins are for investigation. They do not promote those rows to trusted facts. Never aggregate mixed or unknown units, and do not treat a raw display label as a stable identity.

Local pipeline data and compatibility API

Repository contributors can still pass a pipeline artifacts root or packaged release directly to the module-level functions:

frame = gbd.load("qcmr_positions", source="/path/to/release")
datasets = gbd.datasets(source="/path/to/release")

Alternatively, set GBD_DATA_ROOT. An explicit DataStore from open_release() is recommended for applications because every call remains bound to one visible data snapshot.

The installed package version is exposed directly:

import gov_budget_data as gbd

print(gbd.__version__)

Documentation

The public documentation covers installation, available datasets, release management, the complete API, worked examples, and the package's trust model.