Skip to main content

caeli-data

The Python client of the Caeli data repository API (https://data.caeli.space). One door for every producer and reader of datasets: the inventory platform, pipelines and colleagues all use this library; nothing writes to the tables or the buckets directly.

pip install caeli-data            # caeli-data[aws] also brings boto3, for Repository.from_secret

from caeli_data import Repository, TableRejected, IngestFailed

repo = Repository()               # CAELI_API_URL (default https://data.caeli.space) and CAELI_API_KEY
repo.whoami()                     # {"name": "tim", "role": "admin", ...}

# a pandas DataFrame with utc_time, latitude, longitude and the value columns: the platform derives the rest
# (pt_region, pt_tile, row_hash; the coordinates rounded to 5 decimals), splits per region and month, loads, waits
jobs = repo.load_dataframe(df, "cdse-sentinel-5p-l2-no2", software={"my-pipeline": "1.0"}, source_ref="download 4711")
print(jobs[0]["message"])         # "200000 rows appended in cdse_sentinel_5p_l2_no2, ... item N_EU__2026-09__v1 now ..."

# a source with its own row identity (a station id, an orbit with scanline and pixel) names it
jobs = repo.load_dataframe(df, "openaq-stations-no2", row_key="location_id")

# a prepared Parquet file with the six standard columns already in it, any regions and months
jobs = repo.load_file("prepared.parquet", "cdse-sentinel-5p-l2-no2", software={"my-pipeline": "1.0"})

A complete month delivered again is a new version of that month: mode="replace", reason="re-import of 2026-09". A service takes its key from Secrets Manager: Repository.from_secret("caeli/data-platform/api-key-inventory-platform").

The rules a table must meet are in Caeli_data_repository_conventions (section 6); the library checks them before uploading and says which rule is broken (TableRejected; for an extra column the message carries the schema change to make). IngestFailed means the platform failed; the same data sent again is the same job, never a double load. Python 3.9 and later.

A raster (a GeoTIFF on a registered grid) is one item of a raster dataset, area and period:

from caeli_data import PublishRejected

job = repo.publish_raster("nh3_2024-05.tif", "caeli-iasi-nh3-monthly", "NL", "2024-05", software={"caeli-maps": "0.1.0"})
print(job["message"])             # "NL__2024-05__v1 published as s3://caeli-file-store/datasets/.../v1/... (0.1 MB, 60x38 px ...)"
job = repo.publish_raster("nh3_2024-05.tif", "caeli-iasi-nh3-monthly", "NL", "2024-05", mode="revise", reason="recipe v2")

The worker refuses a raster that is not on the area's grid, has no CRS or nodata, or is not a Cloud-Optimized GeoTIFF; with rasterio and rio-cogeo installed the library checks the same rules before uploading (PublishRejected) and convert=True makes the COG for you. publish_file does the same for any file or model; register_grid (admin) and grids() manage the grids.

A collection is one recipe: register a raster dataset with the configuration that makes it (register_dataset(..., recipe=recipe_dict), which records it as caeli:recipe with its fingerprint()), send the fingerprint with every publication (publish_raster(..., recipe_fingerprint=fingerprint(recipe_dict))), and the platform refuses a map made with another recipe — that is a different dataset.

Reading goes through the same door (0.6.0, API 0.9.x). One call gives the rows of a table dataset as one file or one DataFrame, at any size:

repo.download("cdse-sentinel-5p-l2-no2", "no2_nl_2026-06.csv", area="NL", period="2026-06",
              columns=["product__nitrogendioxide_tropospheric_column", "product__qa_value"],
              filters=[("product__qa_value", ">=", 0.75)])
df = repo.read_table("ads-cams-forecast", regions=["N_EU"], start="2026-06-01", end="2026-06-08", columns=["pm2p5", "lead_hours"])
repo.plan("cdse-sentinel-5p-l2-no2", regions=["N_EU"], period="2026-06", columns=["product__nitrogendioxide_tropospheric_column"])

The API decides beforehand, from the tables' metadata, whether a request is small enough; a request that is too large comes back as consecutive windows in time, and download and read_table fetch them one after another and join them into one file (split=False refuses instead, with ReadTooLarge carrying the plan). Every download is pinned to one table snapshot (version= takes an item key or snapshot:<id>), filters are typed against the schema (("qa_value", ">=", 0.75), ("orbit", "in", [1, 2]), ("no2", "notnull")), and the three index columns idx_utc, idx_lat, idx_lon are always part of a download. Without Python: caeli-data plan ..., caeli-data download ... --to file.csv, caeli-data reads.

Maps and other published files are read by their item (0.6.1, API 0.10.0). An item is one area and period of a dataset (N_EU__2026-06 is the current version of that month, N_EU__2026-06__v2 one version), and its file is fetched through a link the API hands out for a quarter of an hour, straight from the file store, checked against the record's size and sha256; what is already there is not fetched again:

repo.items("caeli-iasi-nh3-tc-monthly-wgs001", start="2025-01", end="2025-12")            # the months, by area and period
repo.download_item("caeli-iasi-nh3-tc-monthly-wgs001", "N_EU__2026-06", "maps")          # one map into the folder maps/
repo.download_items("caeli-iasi-nh3-tc-monthly-wgs001", "maps", start="2025-01", end="2025-12")   # a year of maps

Without Python: caeli-data items <dataset> --from 2025-01 --to 2025-12, caeli-data fetch <dataset> --from 2025-01 --to 2025-12 --into maps, caeli-data fetch <dataset> --item N_EU__2026-06 --into maps. Admins make keys with repo.create_key(name, role, owner_email) or caeli-data key create --name piet --role reader --owner piet@caeli.nl; the key is shown once and goes to its owner through the password manager, never through a chat or a mail.

Metadata

Release files for caeli-data 0.6.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for caeli-data 0.6.1
File Size Uploaded Provenance Actions
caeli_data-0.6.1.tar.gz 64.4 kB Without provenance Details for caeli_data-0.6.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for caeli-data 0.6.1
File Interpreter ABI Platform Actions
caeli_data-0.6.1-py3-none-any.whl Python 3 none any Details for caeli_data-0.6.1-py3-none-any.whl

Total release size: 108.8 kB

Release files / caeli_data-0.6.1.tar.gz

Download URL caeli_data-0.6.1.tar.gz
Size 64.4 kB
Tags Source
SHA-256 checksum
How to use checksums
733ba87a4cf72a850f9247d61420fab63b155f546a216908006a5d8bb9783e8b
BLAKE2b-256 checksum
How to use checksums
d9132311e2e83b995f1e61209188168b6467e27f456cb3897e12391f05a8b25e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.10

Release files / caeli_data-0.6.1-py3-none-any.whl

Download URL caeli_data-0.6.1-py3-none-any.whl
Size 44.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8e2a4e3ca9dbbb81c585f98c8663fd5992e2250aeb42279fbd1d700f3e921f1c
BLAKE2b-256 checksum
How to use checksums
ad5d970445e39775873a80001919ad0d0cb5f3aecd154f478e6e9fadd81078cf
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.10

Security for caeli-data 0.6.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page