caeli-data

The Python client of the Caeli data repository API (https://data.caeli.space). One door for every producer and reader of datasets: the inventory platform, pipelines and colleagues all use this library; nothing writes to the tables or the buckets directly.

pip install caeli-data            # caeli-data[aws] also brings boto3, for Repository.from_secret

from caeli_data import Repository, TableRejected, IngestFailed

repo = Repository()               # CAELI_API_URL (default https://data.caeli.space) and CAELI_API_KEY
repo.whoami()                     # {"name": "tim", "role": "admin", ...}

# a pandas DataFrame with utc_time, latitude, longitude and the value columns: the platform derives the rest
# (pt_region, pt_tile, row_hash; the coordinates rounded to 5 decimals), splits per region and month, loads, waits
jobs = repo.load_dataframe(df, "cdse-sentinel-5p-l2-no2", software={"my-pipeline": "1.0"}, source_ref="download 4711")
print(jobs[0]["message"])         # "200000 rows appended in cdse_sentinel_5p_l2_no2, ... item N_EU__2026-09__v1 now ..."

# a source with its own row identity (a station id, an orbit with scanline and pixel) names it
jobs = repo.load_dataframe(df, "openaq-stations-no2", row_key="location_id")

# a prepared Parquet file with the six standard columns already in it, any regions and months
jobs = repo.load_file("prepared.parquet", "cdse-sentinel-5p-l2-no2", software={"my-pipeline": "1.0"})

A complete month delivered again is a new version of that month: mode="replace", reason="re-import of 2026-09". A service takes its key from Secrets Manager: Repository.from_secret("caeli/data-platform/api-key-inventory-platform").

The rules a table must meet are in Caeli_data_repository_conventions (section 6); the library checks them before uploading and says which rule is broken (TableRejected; for an extra column the message carries the schema change to make). IngestFailed means the platform failed; the same data sent again is the same job, never a double load. Python 3.9 and later.

A raster (a GeoTIFF on a registered grid) is one item of a raster dataset, area and period:

from caeli_data import PublishRejected

job = repo.publish_raster("nh3_2024-05.tif", "caeli-iasi-nh3-monthly", "NL", "2024-05", software={"caeli-maps": "0.1.0"})
print(job["message"])             # "NL__2024-05__v1 published as s3://caeli-file-store/datasets/.../v1/... (0.1 MB, 60x38 px ...)"
job = repo.publish_raster("nh3_2024-05.tif", "caeli-iasi-nh3-monthly", "NL", "2024-05", mode="revise", reason="recipe v2")

The worker refuses a raster that is not on the area's grid, has no CRS or nodata, or is not a Cloud-Optimized GeoTIFF; with rasterio and rio-cogeo installed the library checks the same rules before uploading (PublishRejected) and convert=True makes the COG for you. publish_file does the same for any file or model; register_grid (admin) and grids() manage the grids. The read side (catalogue, versions, downloads) and the command line follow.