civic-data

Cleaned, normalized, and documented public open data — with a generated, code-coupled data dictionary and transformation-lineage layer, consumable as a GeoParquet lake with a self-describing CLI.

The city gives away raw open data. civic-data's value is not the data — it's the honest, reproducible record of what we did to it, why, and what's lost: schema, grain, NULL rates, per-year coverage, quarantine loss, bias caveats, and a step-by-step transformation lineage that doubles as the reproduction recipe. Because that record is generated from the live database and the ingest code, it cannot drift from reality.

v1 covers Louisville, KY: 29 domain datasets across Safety, Civic Responsiveness & Livability, Economic Distress, Investment & Growth, Housing & Affordability, and Culture & Amenities (crime, 311 service requests, landbank sales, lien orders, foreclosures, food-service establishments, ABC licenses, restaurant inspections, construction permits, short-term-rental licenses, and more — run civic-data datasets for the full current list) plus the geography spine (ZIP boundaries, council districts, neighborhoods, HUD zones, crosswalks, Census population) — millions of rows, all ingested from source, all lineage-documented (the catalog carries the exact counts).

Quick start (60 seconds, no database required)

pip install civic-data             # consumer install — DuckDB only, no Postgres
civic-data cities                  # list published city lakes
civic-data get louisville          # download the lake (sha256-verified) into ./export
civic-data query -d crime --zip 40202 --limit 5

Consuming the data

The consumer product is a GeoParquet lake described by a machine-readable catalog.json, read with DuckDB — no server, no Docker, no Postgres. Every verb takes --json (for agents) and --lake <dir> (default: ./export):

civic-data catalog --json          # the discoverability contract — start here
civic-data datasets                # list datasets by catalog key
civic-data schema crime            # one dataset's columns + shape
civic-data query -d crime --zip 40202 --year 2024 --limit 5
civic-data query -d crime --district 6          # point-in-polygon; geocoded rows only (it says so)
civic-data place 40202             # one place across every dataset: identity + counts
civic-data sql "SELECT zip_code, count(*) FROM crime GROUP BY 1"   # raw DuckDB; keys are views
civic-data query -d crime --sources   # provenance: source pulls, license, how-to-cite

Hand someone a place as a folder: civic-data place 6 --export writes a self-contained mini-lake (filtered Parquet + the district's boundary/crosswalks/population + a scoped catalog + generated llms.txt/AGENTS.md). Every verb above works against that folder via --lake.

The lake root also carries generated llms.txt and AGENTS.md so an AI agent can bootstrap itself from the artifact alone. Both are regenerated from the catalog on every export — never hand-edited (a test enforces it).

Errors

Every consumer verb fails with a non-zero exit code and a message on stderr. With --json, that message is a stable JSON envelope instead of plain text:

{"error": {"code": "LAKE_ERROR", "message": "unknown city 'gotham'. Published cities: louisville"}}

Producing the data (the reproduction path)

docker compose up -d              # PostgreSQL 16 + PostGIS on host port 5434
python -m venv .venv && . .venv/Scripts/activate   # Windows cmd: .venv\Scripts\activate.bat
                                                    # Windows PowerShell: .venv\Scripts\Activate.ps1
                                                    #   (blocked by default execution policy on some
                                                    #   machines — if so, call .venv\Scripts\python.exe
                                                    #   / .venv\Scripts\civic-data.exe directly instead)
pip install -e .[producer]        # the pipeline extras (Postgres driver, HTTP client)

civic-data ingest-zips                          # canonical ZIP boundary set (MDM) — run first
civic-data ingest-refs                          # LOJIC geocoding references (address points + intersections)
civic-data ingest-geo && civic-data ingest-population   # boundaries, crosswalks, Census ACS
civic-data ingest crime --all --promote         # source -> bronze -> silver (any of the 29 domains)
civic-data geocode-crime --write --apply        # place ~1.2M crime records (geom + derived ZIP)
civic-data datadict --with-source-audit         # generate the data dictionary + lineage
civic-data export                               # silver -> GeoParquet lake + catalog + agent docs

The generated dictionary lands in docs/data-dictionary/ — a human-readable README.md and a machine-readable data-reality.json (diffing successive runs is drift detection).

Ingest order matters: ingest-zips first. The ZIP set is the canonical geography (MDM) every record validates against; crime/311 ingest refuse to run without it.

Honesty, by construction

• Trust tiers on every dataset (trendable / viewable_caveated / snapshot_only) — a current roster is not a time series, and the catalog says which is which.
• Documented loss, not silent loss: quarantine counts, geocode coverage (spatial filters match geocoded rows only, and the CLI says so), source-shape divergences recorded as findings.
• Generated, never hand-maintained: the dictionary, the catalog, and the agent docs are all derived from the live database + code; drift guards raise when authored prose falls out of sync.

Licensing

• Code: MIT — see LICENSE.
• Data & documentation: CC BY 4.0 — see LICENSE-data.

The underlying raw data is public record, provided as-is, and its terms were verified per origin (2026-07-11): Louisville Metro publishes under the PDDL (public domain; when republishing modified data, identify the source, version, and modifications — which the generated lineage layer does per dataset); LOJIC layers are public domain (Mapping Data Source: LOJIC); Census ACS is a U.S. government work — this product uses the Census Bureau Data API but is not endorsed or certified by the Census Bureau. Every dataset's exact license text ships in its catalog provenance: civic-data query -d <dataset> --sources.

Provenance

Born out of civic-graph (the private flagship built on top of this layer, and the origin of this spine). See wiki.md for the source of truth and ai-working/plans/ for the phase plans.