portalkit

One Python API for Socrata, CKAN, ArcGIS and OpenDataSoft open data portals.

Nearly every government open data portal runs on one of four platforms. Each has its own client, its own pagination bugs, and its own way of losing your data quietly. portalkit detects which platform a URL is running and gives you the same API either way, returning pandas or geopandas.

Installation

pip install portalkit          # add [geo] for geometry support
pip install 'portalkit[geo]'

Requires Python 3.10+.

Quick start

import portalkit as pk

df = pk.load("chicago/311", limit=5000)          # a known dataset, by name

portal = pk.open("https://data.sfgov.org")       # or any portal URL
portal.search("evictions")
gdf = portal.dataset("6z8x-wfk4").to_geopandas(where="file_date > '2026-01-01'")

pk.open works out which software the portal runs by asking it. You do not have to know, and you do not need a different library per city.

Why this exists

sodapy, the client most people reach for, has been unmaintained since 2022 and only speaks Socrata. ckanapi is a faithful wrapper around CKAN's Action API with no opinion about DataFrames. ArcGIS means either Esri's SDK or writing the pagination yourself. Most projects end up with three half-written clients carrying the same three bugs.

Silently truncated downloads

Point geopandas.read_file() at an ArcGIS FeatureServer layer and you get 1000 rows back with no error and no warning, because that is the server's maxRecordCount. Socrata's default is the same. Analyses get published on top of that number.

Paging past it is not just a loop. Without an explicit sort, resultOffset and $offset are free to return the same row twice and never show you another, so both need a stable order: :id on Socrata, the object id field on ArcGIS. Some ArcGIS layers cannot page at all and have to be fetched through returnIdsOnly and batched where clauses. OpenDataSoft caps offset at 10,000 and errors past it, so anything larger goes through its export endpoint.

portalkit does all of that and checks the result against the row count the portal itself reports.

Adaptive ArcGIS page sizes

A layer's advertised maxRecordCount describes what it will return for cheap rows. Ask for the same number of full-resolution polygons and the server returns a 500 wrapped inside a 200 body, which a naive client reads as "no more rows" and stops. There is no way to know the real ceiling except to find it, so portalkit halves the page size and retries until the server copes, then keeps the smaller size. It also requests six decimal places of coordinate precision, which is about ten centimetres and roughly halves a polygon response.

Column types from the schema, not from guessing

Socrata returns every value as a string, dates included. Letting pandas infer afterwards turns ZIP code 02134 into 2134.0. The portal already published a schema saying which column is an identifier, so portalkit uses it. When the schema turns out to be wrong, which happens, the column comes back untouched rather than as a column of nulls, so you can see the mess and decide.

Geometry

Socrata sends GeoJSON, ArcGIS sends rings and paths, OpenDataSoft wraps a GeoJSON Feature in one column and its centroid in another. All of it converts to shapely with the CRS set.

Esri packs every ring of a shape into one flat list and distinguishes exteriors from holes by winding direction. Treating everything after the first ring as a hole silently deletes islands: on the Census state boundaries layer, 11 of 56 states are multi-part. portalkit classifies rings by orientation and assigns holes to the shell that contains them.

When a dataset carries both a shape and its centroid, the shape wins.

Portals

API

where, select and order go to the portal in its own query language. Filtering server-side is usually the difference between two seconds and five minutes.

Caching

Every tabular fetch is stored as Parquet, keyed by portal, dataset and the exact query, so re-running a notebook does not re-run the download.

to_geopandas() does not cache. Shapely geometry does not survive a Parquet round trip, so it would be a cache that never hits.

Command line

$ portalkit catalog chicago
reference                 dataset                  portal
------------------------  -----------------------  ------------------------------
chicago/311               311 Service Requests     https://data.cityofchicago.org
chicago/building-permits  Building permits         https://data.cityofchicago.org
chicago/crimes            Crimes, 2001 to present  https://data.cityofchicago.org

$ portalkit info chicago/311
311 Service Requests
id       v6vf-nfxy
portal   https://data.cityofchicago.org (socrata)
rows     14,517,544

$ portalkit get chicago/311 --where "sr_type = 'Pothole in Street Complaint'" -o potholes.parquet
wrote 422,462 rows to potholes.parquet

The catalog

pk.load("nyc/311") works because a small list of well-known datasets ships with the package. It is not meant to be exhaustive, it is meant to save the five minute detour of discovering that Chicago's 311 dataset is v6vf-nfxy. portalkit catalog lists what is there.

Adding your city is a pull request against src/portalkit/catalog.toml with a portal URL, a dataset id and a title. For entries you would rather not publish, point PORTALKIT_CATALOG at your own TOML file and it is merged over the built-in one.

Notes

CKAN is a catalogue first. Only resources pushed into its DataStore can be queried; everything else is a file on disk. portalkit resolves a package to its first DataStore-backed resource, or says plainly that there is not one and where the files are. Pass package:resource-id to pick a specific one.

ArcGIS addressing follows the REST URLs. Give it a service and name a layer with portal.dataset("0"), or give it the layer URL directly.

Large geometry downloads from municipal servers are genuinely slow. All 3,235 US counties at full resolution is a few hundred megabytes. The CLI shows a running row count and to_pandas takes a progress= callback. Filter server-side first where you can.

Rate limits belong to the portal, not to portalkit. One connection at a time, retries with jitter, and Retry-After respected. Most of these portals are one box run by a team of two.

Contributing

Bug reports and pull requests are welcome, especially catalog entries for new cities. uv sync then uv run pytest to get started. uv run pytest -m live runs the tests that hit real portals.

License

MIT.