atproto-sensorthings

Flatten a dev.sensorthings ATProto repo export into tidy, geo-referenced tables for analysis.

Sensor networks on sensorthings.dev publish their observations to an ATProto repository. You can download that repository as a single CAR file (from sensorthings.dev/yourdata, or via com.atproto.sync.getRepo). This package turns that CAR into three tables:

Everything is derived from the CAR: no network access, no repo checkout, and the lexicons are not needed at runtime.

Install

The base install (parsing, the summary, and CSV output) depends only on atproto and click. Install as a command-line tool:

uv tool install atproto-sensorthings         # or: pipx install atproto-sensorthings

or add it to a project as a library:

uv add atproto-sensorthings                  # or: pip install atproto-sensorthings

GeoDataFrame and GeoJSON output need the geo extra (geopandas / shapely / pandas); GeoParquet also needs parquet:

uv add "atproto-sensorthings[geo]"           # or: pip install "atproto-sensorthings[geo]"
uv add "atproto-sensorthings[geo,parquet]"   # or: pip install "atproto-sensorthings[geo,parquet]"

Command line

# record counts and reading time span
atproto-sensorthings summary export.car

# write stations.csv, datastreams.csv, observations.csv to ./sensorthings_export
atproto-sensorthings flatten export.car

# GeoJSON (needs the geo extra); one layer at a time is also supported
atproto-sensorthings flatten export.car --format geojson --out export
atproto-sensorthings flatten export.car --format geoparquet --layer observations

CSV rows carry longitude / latitude columns, so they load as points in any GIS without the geo extra.

Library

import atproto_sensorthings as ast

repo = ast.read_car("export.car")

# pure core: plain dicts, streamed; no geopandas required
for row in ast.observation_rows(repo):
    ...

# with the geo extra: GeoDataFrames (imported lazily on first use)
stations = ast.stations_gdf(repo)        # 1 row / station
observations = ast.observations_gdf(repo)  # 1 row / reading, sorted, typed

Data notes

• Coordinates are decoded from the lexicon's integer latE7 / lonE7 (degrees times 1e7) to WGS84 degrees; altitude from altMm to metres.
• Numeric results are scaled integers: the real value is value * 10**-resultScaleFactor, applied automatically. In the observations table, value holds the decoded native result (float, string, boolean or list) and value_num is a nullable float for the numeric and boolean cases.
• Result types covered: numeric, category, boolean, array, geoPoint.
• Record types covered: observationBatch, observationFrame, observation and multiObservation, so every network's export flattens the same way.
• Frames hold each collector tick's new or changed readings until the day is sealed into observationBatch records, about two days later. A reading can be in both; it appears once, with the value from the latest frame.
• The whole export is read into memory; for the largest agency repositories prefer GeoParquet output over GeoJSON.

Licence

MIT. See LICENSE.