mloda-plugin-govdata

Connectors for German open government data, built on mloda. Request the columns you want as mloda features; the plugin handles CKAN discovery, download with caching and retries, and parsing (German CSV or publisher JSON) into a typed Arrow table.

Three example datasets cover population (GovData CSV), elections (Bundeswahlleiterin kerg.csv), and environment (UBA Air Data JSON); the Destatis connector adds GENESIS tables, with harmonization features and recipe files on top.

Status

Young but working. The three example readers and the Destatis connector run end to end, with paginated dataset search, cached downloads with retries, and unit plus property-based tests behind them. Every reader is a thin subclass of BaseGovDataReader that overrides the parse step (and the fetch step for a source with its own locator type); new datasets follow the same path (see docs/adding-a-reader.md). How the packages fit together, and where to start for a change: docs/architecture.md. Development happens in a 6-month Prototype Fund stage (June to November 2026), so the API may still shift between releases.

Usage

GovData readers

Read the Stuttgart population dataset (via GovData) as a typed PyArrow table:

from mloda.user import Feature, mloda
from mloda_plugin_govdata.feature_groups.govdata import StuttgartPopulationReader

slug = "einwohner-nach-altersgruppen-und-stadtbezirken"
result = mloda.run_all(
    [
        Feature("Einwohner", options={StuttgartPopulationReader: slug}),
        Feature("Stadtbezirk", options={StuttgartPopulationReader: slug}),
    ],
    compute_frameworks=["PyArrowTable"],
)
table = result[0]  # pyarrow.Table with the requested columns
result.plan  # resolved execution steps: which FeatureGroup ran on which framework

The options key is the reader class or its class-name string; both select the same reader. The option value is a GovData dataset slug or a direct distribution URL. The license is read from the CKAN distribution metadata. Set BaseGovDataReader.cache_dir to control where downloads are cached (harmonized features below use a separate HarmonizationFeature.cache_dir). For any other GovData CSV dataset, GovDataReader works out of the box and reads every column as a string; subclass it and set schema for typed columns.

Don't know the slug yet? Search GovData with the paginated CKAN package_search API:

from mloda_plugin_govdata.feature_groups.govdata import build_client, search_datasets

with build_client() as client:
    for dataset in search_datasets(client, "einwohner stuttgart", max_results=10):
        print(dataset.name, "|", dataset.title)

search_datasets walks the result pages lazily (page_size per request) and stops at max_results or the end of the result set.

Got a slug but not the column names? peek lists what you can request as features:

StuttgartPopulationReader.peek(slug)  # {"Stichtag": "date32[day]", "Stadtbezirk": "string", ...}

It works on every reader (BundeswahlleiterinReader.peek(kerg), UbaAirReader.peek(uba_measures_url(...))) and downloads through the cache, so the actual feature request reuses the file. peek takes a raw URL, bypassing the READER_OPTIONS validation below. A typo in a feature name fails with the available columns and a close-match suggestion instead of a raw KeyError.

The elections reader handles a direct CSV URL whose file has a multi-row merged header (Bundeswahlleiterin kerg.csv):

from mloda_plugin_govdata.feature_groups.govdata import BundeswahlleiterinReader

kerg = "https://www.bundeswahlleiterin.de/bundestagswahlen/2025/ergebnisse/opendata/btw25/csv/kerg.csv"
result = mloda.run_all(
    [Feature("Gebiet", options={BundeswahlleiterinReader: kerg})],
    compute_frameworks=["PyArrowTable"],
)

The environment reader fetches the Umweltbundesamt (UBA) Air Data v4 measures endpoint (REST JSON) and flattens it to one typed row per station and timestamp. Query parameters are per-feature options, not a pre-built URL (here: hourly ozone at station 143); a bad value is rejected during feature resolution, before any network call:

from mloda_plugin_govdata.feature_groups.govdata import (
    OPTION_UBA_COMPONENT,
    OPTION_UBA_DATE_FROM,
    OPTION_UBA_DATE_TO,
    OPTION_UBA_SCOPE,
    OPTION_UBA_STATION,
    UbaAirReader,
)

uba_options = {
    UbaAirReader: True,
    OPTION_UBA_STATION: 143,
    OPTION_UBA_COMPONENT: 3,
    OPTION_UBA_SCOPE: 2,
    OPTION_UBA_DATE_FROM: "2025-01-01",
    OPTION_UBA_DATE_TO: "2025-01-01",
}
result = mloda.run_all(
    [
        Feature("date_start", options=uba_options),
        Feature("value", options=uba_options),
    ],
    compute_frameworks=["PyArrowTable"],
)

Columns are station_id, date_start, component_id, scope_id, value, date_end, and index (the air-quality index). Component and scope ids come from the UBA components and scopes endpoints.

Destatis

The Destatis connector reads a GENESIS table (Statistisches Bundesamt or a Regionalstatistik installation) by table code, needs a free registration (GENESIS_TOKEN or GENESIS_USER / GENESIS_PASSWORD; see docs/credentials.md), and parses the ffcsv reply:

from mloda_plugin_govdata.feature_groups.destatis import DestatisReader

result = mloda.run_all(
    [Feature("value", options={DestatisReader: "12411-0010"})],  # population by Land
    compute_frameworks=["PyArrowTable"],
)

The option value is a bare table code (the server's default years), a DestatisLocator, or its dict form; the latter two narrow the selection by region, years, and classifying variables, and the dict form round-trips through a recipe file. See docs/destatis-options.md for the full parameter table. peek lists the ffcsv columns the same way as the other readers.

Harmonization

Harmonized features chain an operation onto a reader column (<column>__<operation>): value__rebased re-bases a Kreis series onto a later Gebietsstand with the BBSR keys, 1_variable_attribute_code__nuts2024 adds NUTS codes to Kreis and Gemeinde keys, and time__year_period types the period. A feature with several output columns returns them as <name>~<part>, mloda's multi-output convention: value__rebased~key, value__rebased~flag, and so on. value__rebased and __nuts2024 read their reference tables offline from HarmonizationFeature.cache_dir, so fetch them once first; see docs/harmonization.md for the example, the options, and the one-time fetch.

Recipes

A recipe file bundles the features, joins, and provenance of one run as JSON; load_recipe returns what mloda.run_all needs plus the compliance block (license, attribution, payload sha256, credential env names). Recipes ship under recipes/ in the repository, not in the published package: the Land table, the re-based Kreis series, a rate with its denominator, the Land-level join, and the three example datasets above. See docs/recipes.md.

from mloda_plugin_govdata.recipes import load_recipe

recipe = load_recipe("recipes/land_population.json")
result = mloda.run_all(recipe.features, compute_frameworks=["PyArrowTable"], links=set(recipe.links))
recipe.compliance.sources[0].attribution  # what to print next to the result

Demo

An interactive marimo notebook walks through dataset discovery, all three example datasets, a Destatis table, a recipe joining two sources (population per eligible voter by Land), and a Kreis series re-based across a merger. The notebook lives in the repository (not in the published package), so run it from a source checkout:

git clone https://github.com/mloda-ai/mloda-plugin-govdata.git
cd mloda-plugin-govdata
uv sync --all-extras
uv run marimo edit demos/govdata_demo.py

The notebook hits the live GovData, Bundeswahlleiterin, UBA, and GENESIS-Online endpoints and fetches the BBSR key file; downloads are cached locally after the first run. The Destatis chapters need GENESIS-Online credentials in the environment (GENESIS_TOKEN, or GENESIS_USER and GENESIS_PASSWORD; see docs/credentials.md) and skip themselves without them.

Related Repositories

• mloda: the core library this plugin builds on. You declare which features you need; mloda resolves how to compute them.

• mloda-registry: plugin registry and development guides for the mloda ecosystem.

Funding

Developed as part of the Prototype Fund (Round 2 / Jahrgang 02), funded by the German Federal Ministry of Research, Technology and Space (BMFTR) and supported by the Open Knowledge Foundation Deutschland. Funding code (Förderkennzeichen): 16IS26S11. Stage 1 funding period: 6 months from June 2026.