tabular-py

Python reader for dclimate-tabular/0 — content-addressed station/telemetry data on IPFS.

The Python counterpart to tabular-js, reading the same format from the same CIDs. Built on py-hamt, which supplies the HAMT and the content-addressed store the same way @dclimate/ipld-index does for JS.

tabular-js  ->  @dclimate/ipld-index      (HAMT, CAS, range reads)
tabular-py  ->  py-hamt                   (same, already existed)

Status

Reader only. Publishing, compaction, and rollup stay in tabular-js — the ETL that writes these datasets is JS, and a second writer would be a second thing to keep byte-identical for no current gain. Everything needed to read a dataset published by tabular-js is here.

Install

uv pip install dclimate-tabular-py

Usage

import asyncio
from tabular_py import GatewayRangeSource, StationDataset

async def main():
    source = GatewayRangeSource("https://ipfs-gateway.dclimate.net")
    ds = await StationDataset.open(source, "bafyr4if2wbttslbxpzmro427j4l4nvcrxqo4tufuffqqmqz7afj2pxyu4a")

    # nearest station to a point, then a year of readings
    near = await ds.nearest(34.05, -118.24, max_km=100)
    rows = await near.time_range("2024-01-01", "2024-12-31").elements("PRCP").rows()
    for row in rows[:5]:
        print(row.station_id, row.ts, row.values)

asyncio.run(main())

The chainable selection API mirrors tabular-js:

ds.select("USW00023174")           # explicit station ids
ds.circle(34.05, -118.24, 50)      # within 50 km
ds.rectangle(33.0, -119.0, 35.0, -117.0)
ds.polygon([[(lon, lat), ...]])
await ds.nearest(lat, lon)         # async: reads the geo index
ds.time_range(start, end)
ds.elements("PRCP", "TMAX")
ds.where(gt("TMAX", 300))          # pushed down to fragment statistics

Selections are immutable — each call returns a new StationDataset, so a base dataset can be reused across queries.

Terminal operations:

How reads stay small

A query never scans the dataset. Three things prune before any Parquet byte is fetched:

1. The station index (a HAMT keyed by station id) resolves named stations directly.
2. The geo projection answers region queries by reading one or two shard blocks instead of walking all 132k stations.
3. Fragment statistics in the manifest — per-column min/max and null counts — let a predicate skip whole fragments unread.

What survives is fetched with HTTP range requests against the exact column-chunk byte ranges the manifest records, so a query for one column of one year moves kilobytes.

One deviation from tabular-js, and why

tabular-js synthesizes Parquet FileMetaData client-side from the manifest and reads a fragment with zero footer fetches. PyArrow exposes no public FileMetaData constructor, so that trick does not transfer.

Instead this reader fetches the footer by its manifest-recorded footer_offset/footer_length in a single ranged GET, verifies it against the manifest's footer_digest, and hands the parsed metadata to PyArrow. Cost is one extra range request per fragment — and it buys a corruption check tabular-js does not perform. Footers are cached per fragment CID, so a repeated query pays it once.

Development

uv sync
uv run pytest                     # unit tests, no network
uv run pytest -m network          # conformance against the live gateway
uv run ruff check . && uv run mypy tabular_py

tests/test_conformance.py reads a real published GHCNd dataset (bafyr4if2wbttslbxpzmro427j4l4nvcrxqo4tufuffqqmqz7afj2pxyu4a, 132,437 stations, 1.15 B rows) and asserts this implementation agrees with tabular-js on decoded values.