tabular-py

Python reader for dclimate-tabular/1 and /2 — content-addressed entity/telemetry data on IPFS.

The Python counterpart to tabular-js, reading the same format from the same CIDs. Built on py-hamt, which supplies the HAMT and the content-addressed store the same way @dclimate/ipld-index does for JS.

tabular-js  ->  @dclimate/ipld-index      (HAMT, CAS, range reads)
tabular-py  ->  py-hamt                   (same, already existed)

Status

Reader only. Publishing, compaction, and rollup stay in tabular-js — the ETL that writes these datasets is JS, and a second writer would be a second thing to keep byte-identical for no current gain. Everything needed to read a dataset published by tabular-js is here.

Reads both dclimate-tabular/1 and /2. /2 stores parent and ancestry CIDs as canonical base32 strings so generic IPFS traversal does not follow the dataset's entire history. /1 stores them as IPLD links. Both decode to CID objects, and re-encoding a root preserves its original version and bytes. Data indexes, schemas, geographic projections and Parquet reads use the same paths for both versions.

SPEC_VERSION is /2; LEGACY_SPEC_VERSION is /1. Roots written under /0 remain unsupported: /1 replaced the station model with the entity model, which can also represent entities without a location. See the changelog.

Historical roots under /2 require their own retention pins; the current root's recursive pin retains its data, not previous snapshots. This reader does not manage pins or migrate published datasets.

Install

uv pip install dclimate-tabular-py

Usage

import asyncio
from tabular_py import GatewayRangeSource, EntityDataset

async def main():
    source = GatewayRangeSource("https://ipfs-gateway.dclimate.net")
    ds = await EntityDataset.open(source, root_cid)

    # nearest station to a point, then a year of readings
    near = await ds.nearest(34.05, -118.24, max_km=100)
    rows = await near.time_range("2024-01-01", "2024-12-31").elements("PRCP").rows()
    for row in rows[:5]:
        print(row.entity_id, row.ts, row.values)

asyncio.run(main())

The chainable selection API mirrors tabular-js:

ds.select("USW00023174")           # explicit entity ids
ds.circle(34.05, -118.24, 50)      # within 50 km
ds.rectangle(33.0, -119.0, 35.0, -117.0)
ds.polygon([[(lon, lat), ...]])
await ds.nearest(lat, lon)         # async: reads the geo index
ds.time_range(start, end)
ds.elements("PRCP", "TMAX")
ds.where(gt("TMAX", 300))          # pushed down to fragment statistics

Selections are immutable — each call returns a new EntityDataset, so a base dataset can be reused across queries.

Terminal operations:

How reads stay small

A query never scans the dataset. Three things prune before any Parquet byte is fetched:

1. The entity index (a HAMT keyed by entity id) resolves named entities directly.
2. The geo projection answers region queries by reading one or two shard blocks instead of walking every entity.
3. Fragment statistics in the manifest — per-column min/max and null counts — let a predicate skip whole fragments unread.

What survives is fetched with HTTP range requests against the exact column-chunk byte ranges the manifest records, so a query for one column of one year moves kilobytes.

One deviation from tabular-js, and why

tabular-js synthesizes Parquet FileMetaData client-side from the manifest and reads a fragment with zero footer fetches. PyArrow exposes no public FileMetaData constructor, so that trick does not transfer.

Instead this reader fetches the footer by its manifest-recorded footer_offset/footer_length in a single ranged GET, verifies it against the manifest's footer_digest, and hands the parsed metadata to PyArrow. Cost is one extra range request per fragment — and it buys a corruption check tabular-js does not perform. Footers are cached per fragment CID, so a repeated query pays it once.

Development

uv sync
uv run pytest -m 'not network'    # unit tests, no network
uv run pytest -m network          # conformance against the live gateway
uv run ruff check . && uv run mypy tabular_py

How this is tested against tabular-js

tests/test_golden_cids.py is the cross-language contract. It builds the same small structures tabular-js builds in its own test/wire.test.ts, and asserts they encode to the same CIDs. Because a CID is a hash of the encoded bytes, agreement means the two libraries produce byte-identical blocks for identical inputs — checkable offline, against no published dataset at all.

If a vector mismatches, fix the encoder rather than the vector. When an encoding changes deliberately, change it in tabular-js first and copy the new CID across, so the two are never quietly updated to match each other.

The rest of the suite runs against synthetic /1 and /2 datasets assembled in tests/helpers/build.py — both index variants, an entity with no position, a pre-epoch timestamp, declared gaps, and real ZSTD Parquet fragments read through byte ranges. It is a test helper, not a writer: it composes the public *_to_wire encoders and does no fragment planning, bucketing, compaction, or rollup.

tests/test_conformance.py runs against the four published /1 datasets — GHCNd (132,437 entities, 1.15 B rows), SCAN, SNOTEL, and NDBC — and a published /2 GHCNd snapshot. Known observations, station metadata and geographic lookups are checked on both GHCNd versions; /2 queries also verify that history roots are not fetched. It covers the one thing the offline suite cannot: that both libraries agree end to end on data neither of them wrote. Each root is fetched, decoded, and re-encoded back to its own CID, which is the strongest available statement that nothing was dropped, reordered, or silently defaulted.

When an ETL republishes, the roots move — update the CIDs and re-derive the asserted figures rather than loosening the assertions.