portalkit

One Python API for the four pieces of software that nearly every government open data portal runs on: Socrata, CKAN, ArcGIS and OpenDataSoft. Point it at a URL, get a DataFrame back.

pip install portalkit          # add [geo] if you want geometry

import portalkit as pk

df = pk.load("chicago/311", limit=5000)  # a known dataset, by name

portal = pk.open("https://data.sfgov.org")  # or any portal URL
portal.search("evictions")
gdf = portal.dataset("6z8x-wfk4").to_geopandas(where="file_date > '2026-01-01'")

pk.open works out which software the portal runs by asking it. You do not have to know, and you do not have to import a different library per city.

Why this exists

The Python client most people reach for is sodapy, which has been unmaintained since 2022 and only speaks Socrata. ckanapi is a faithful wrapper around CKAN's Action API and has no opinion about DataFrames. ArcGIS either means Esri's SDK or writing the pagination yourself. So you end up with three half-written clients per project, and each of them has the same three bugs.

Your download is silently truncated. Point geopandas.read_file() at an ArcGIS FeatureServer layer and you get 1000 rows with no warning, because that is the server's maxRecordCount. Socrata's default is 1000 too. People publish analyses on top of that. Paging past it is not just a loop: without an explicit sort, resultOffset and $offset are free to return the same row twice and never show you another one, so both need an explicit stable order (:id on Socrata, the object id field on ArcGIS). OpenDataSoft caps offset at 10000 and errors past it, so anything bigger has to go through its export endpoint instead. portalkit does all of that and returns the number of rows the portal itself says the dataset has.

Your ZIP codes turn into floats. Socrata returns every value as a string, dates included. Let pandas infer types afterwards and 02134 becomes 2134.0. The portal already published a schema saying which column is a number and which is an identifier, so portalkit uses it. When the schema turns out to be wrong, which happens, the column comes back untouched rather than as a column of nulls, so you can see the mess and decide what to do about it.

Your geometry arrives as JSON. Socrata sends GeoJSON, ArcGIS sends rings and paths, OpenDataSoft wraps a GeoJSON Feature in one column and its centroid in another. All of it comes out as shapely, with the CRS set. geopandas stays an optional install for people who only want the table.

The rest of it

Caching. Every tabular fetch is stored as Parquet, keyed by portal, dataset and the exact query, so re-running a notebook does not re-run the download. PORTALKIT_CACHE sets the directory, cache=False skips it, portalkit clear-cache empties it. to_geopandas() does not cache: shapely geometry does not survive a Parquet round trip, so it would be a cache that silently never hits.

Filtering happens on the server. where, select and order are passed through in the portal's own query language, which is SoQL on Socrata, SQL-ish on ArcGIS, ODSQL on OpenDataSoft, and CKAN's SQL endpoint when a where is given. This is usually the difference between two seconds and five minutes.

Manners. One connection at a time, a real user agent, retries with jitter on the failures worth retrying, and Retry-After respected. Most of these portals are one box run by a team of two. Pass token= for a Socrata app token if you are going to be hammering one.

Adaptive page sizes on ArcGIS. Layers advertise a maxRecordCount that describes what they will return for cheap rows. Ask for the same number of full-resolution county polygons and the server returns a 500 wrapped inside a 200. There is no way to know the real ceiling except to find it, so portalkit halves the page size and retries until the server copes, then keeps the smaller size. It also asks for six decimal places of coordinate precision, which is about ten centimetres and roughly halves the size of a polygon response.

Command line

$ portalkit catalog chicago
reference                 dataset                  portal
------------------------  -----------------------  ------------------------------
chicago/311               311 Service Requests     https://data.cityofchicago.org
chicago/building-permits  Building permits         https://data.cityofchicago.org
chicago/crimes            Crimes, 2001 to present  https://data.cityofchicago.org

$ portalkit info chicago/311
311 Service Requests
id       v6vf-nfxy
portal   https://data.cityofchicago.org (socrata)
rows     14,517,544

column              type           label
------------------  -------------  ----------------
sr_number           text           SR_NUMBER
created_date        calendar_date  CREATED_DATE
...

$ portalkit get chicago/311 --where "sr_type = 'Pothole in Street Complaint'" -o potholes.parquet
wrote 422,462 rows to potholes.parquet

-o takes .parquet, .csv, .json, .geojson or .gpkg. The last two fetch geometry. Without -o you get the first twenty rows on stdout.

The catalog

pk.load("nyc/311") works because there is a small list of well-known datasets shipped with the package. It is not meant to be exhaustive, it is meant to save you the five minute detour of finding out that Chicago's 311 dataset is v6vf-nfxy. portalkit catalog lists what is there.

Adding your city is a pull request against src/portalkit/catalog.toml with a portal URL, a dataset id and a title. For entries you do not want to publish, point PORTALKIT_CATALOG at your own TOML file and it gets merged in.

Notes

CKAN is a catalogue first. Only resources that have been pushed into its DataStore can be queried at all; everything else is a file on disk. portalkit resolves a package to its first DataStore-backed resource, or tells you plainly that there is not one and where the files are. Pass package:resource-id to pick a specific one.

ArcGIS addressing follows the REST URLs. Give it a service and name a layer (portal.dataset("0")), or give it the layer URL directly.

Big geometry downloads from municipal servers are genuinely slow: all 3235 US counties at full resolution is a few hundred megabytes and takes a while. The CLI shows a running row count, and to_pandas(progress=...) takes a callback if you want one in a script. Filter server-side first if you can.

Rate limits are the portal's, not portalkit's. If you are pulling millions of rows from a city of 40,000 people, consider whether their server would rather you did that once and cached it.

Python 3.10+. MIT.