caeli-data
The Python client of the Caeli data repository API (https://data.caeli.space). One door for every producer and reader of datasets: the inventory platform, pipelines and colleagues all use this library; nothing writes to the tables or the buckets directly.
pip install caeli-data # caeli-data[aws] also brings boto3, for Repository.from_secret
# caeli-data[raster] brings rasterio and rio-cogeo for GeoTIFFs (Python 3.11 or later)
from caeli_data import Repository, TableRejected, IngestFailed
repo = Repository() # CAELI_API_URL (default https://data.caeli.space) and CAELI_API_KEY
repo.whoami() # {"name": "tim", "role": "admin", ...}
# a pandas DataFrame with utc_time, latitude, longitude and the value columns: the platform derives the rest
# (pt_region, pt_tile, row_hash; the coordinates rounded to 5 decimals), splits per region and month, loads, waits
jobs = repo.load_dataframe(df, "cdse-sentinel-5p-l2-no2", software={"my-pipeline": "1.0"}, source_ref="download 4711")
print(jobs[0]["message"]) # "200000 rows appended in cdse_sentinel_5p_l2_no2, ... item N_EU__2026-09__v1 now ..."
# a source with its own row identity (a station id, an orbit with scanline and pixel) names it
jobs = repo.load_dataframe(df, "openaq-stations-no2", row_key="location_id")
# a prepared Parquet file with the six standard columns already in it, any regions and months
jobs = repo.load_file("prepared.parquet", "cdse-sentinel-5p-l2-no2", software={"my-pipeline": "1.0"})
A complete month delivered again is a new version of that month: mode="replace", reason="re-import of 2026-09".
A service takes its key from Secrets Manager: Repository.from_secret("caeli/data-platform/api-key-inventory-platform").
The rules a table must meet are in Caeli_data_repository_conventions (section 6); the library checks them before
uploading and says which rule is broken (TableRejected; for an extra column the message carries the schema change to
make). IngestFailed means the platform failed; the same data sent again is the same job, never a double load.
Python 3.9 and later.
A raster (a GeoTIFF on a registered grid) is one item of a raster dataset, area and period:
from caeli_data import PublishRejected
job = repo.publish_raster("nh3_2024-05.tif", "caeli-iasi-nh3-monthly", "NL", "2024-05", software={"caeli-maps": "0.1.0"})
print(job["message"]) # "NL__2024-05__v1 published as s3://caeli-file-store/datasets/.../v1/... (0.1 MB, 60x38 px ...)"
job = repo.publish_raster("nh3_2024-05.tif", "caeli-iasi-nh3-monthly", "NL", "2024-05", mode="revise", reason="recipe v2")
The worker refuses a raster that is not on the area's grid, has no CRS or nodata, has other bands, another data type or
another nodata than the dataset names, or is not a Cloud-Optimized GeoTIFF; the library checks the same rules before
uploading (PublishRejected, every problem in the worker's own sentences) and convert=True makes the COG for you.
publish_file does the same for any file or model; register_grid (admin) and grids() manage the grids. The check
needs pip install "caeli-data[raster]" (rasterio and rio-cogeo in the versions of the platform's worker); without
them the library says so, in one sentence, and checks nothing silently (0.18.0).
A GeoTIFF is read and checked without a key too (0.18.0), for a service that holds the grid but no key:
from caeli_data import raster_facts, grid_of, check_raster_file, ensure_cog, period_from_name, publish_manifest
facts = raster_facts("no2_2026-06.tif") # CRS, origin, pixel size, size, bands, dtype, nodata, COG or why not, box, sha256
grid = grid_of(facts, repo.grids()) # the grid the file is on, or None
check_raster_file(facts, grid, dataset="caeli_aqm2_no2-monthly", area="NL", bands=1, dtype="float32", nodata="nan") # [] passes
period_from_name("no2_2026-06.tif", "monthly") # "2026-06"
m = publish_manifest("caeli_aqm2_no2-monthly", "NL", "2026-06", filename="no2_2026-06.tif", sha256=facts["sha256"], size_bytes=facts["size_bytes"])
repo.announce_publish(m, dry_run=True) # the platform's own check of the announcement, nothing written
The command line: caeli-data raster facts <file> (no key), caeli-data raster check <file> --dataset <key> --area NL,
caeli-data grids [--file <file>].
A person's file, CSV or Parquet, is prepared by the library without a key (0.13.0): read by the library's own rules (every column as text, cast by the dataset's schema), the six standard columns derived, one part per region and month written and checked as the worker checks it, and a report of what was made, what was left out and why:
from caeli_data import prepare_file, identity_from_label, load_manifest, upload_to_link
schema = {c["name"]: c["type"] for c in repo.columns("openaq-stations-pm1")}
row_key, key_columns = identity_from_label(repo.dataset("openaq-stations-pm1")["extra"]["caeli:identity"])
report = prepare_file("stations.csv", schema, "parts/", row_key=row_key, key_columns=key_columns)
report["refused"] # [] when the file was taken, else the sentences that say what to change
report["dropped"] # [{"reason": "no latitude", "rows": 3, "first": [17, 18, 2044]}]
for part in report["parts"]: # area, period, rows, sha256, size_bytes, problems, ...
m = load_manifest("openaq-stations-pm1", part["area"], part["period"], sha256=part["sha256"],
size_bytes=part["size_bytes"], row_count=part["rows"], schema_version=repo.schema("openaq-stations-pm1")["version"])
job = repo.announce_load(m) # the three steps of a load apart: announce, upload, wait
with open(f"parts/{part['file']}", "rb") as f:
upload_to_link(job["upload_link"]["url"], f, part["size_bytes"])
repo.wait(job["job_id"])
A table's schema is proposed from a sample file by the same reading (0.14.0, no key): each value column's type and the types its values allow, the names the platform does not take, what makes a row unique, the regions and months:
from caeli_data import propose_schema
proposal = propose_schema("stations.csv")
proposal["columns"] # [{"name": "pm1", "type": "double", "could_be": ["double", "float", "string"], ...}]
proposal["identity"] # {"coordinates": {"unique": True}, "row_keys": [{"column": "station_id", "values": 10, ...}]}
A dataset is registered with its tags (0.10.0, API 0.17.0): one to 12 short labels of what it holds, such as NO2,
satellite data, air quality, monthly. They are the keywords of its collection in the catalogue and come back
with repo.dataset(key)["tags"] and every row of repo.datasets(). The library registers a dataset only with tags
and says so before calling the API; there is no fixed list, repo.tags() is the list in use to choose from, and the
platform keeps one spelling of each tag (no2 becomes NO2 once NO2 is in use):
repo.register_dataset("zz-test-x", "table", "A test table", "hourly", columns=[{"name": "no2", "type": "double"}],
areas=["N_EU"], tags=["NO2", "hourly", "test"])
repo.tags() # [{"tag": "air quality", "datasets": 26}, ...]
repo.set_tags("caeli-tropomi-no2-trop-monthly-wgs001", ["NO2", "satellite data", "air quality", "map", "monthly"]) # registrar
A tag is 1 to 40 characters of letters, digits, spaces and the marks . - + / &; at most 12 on a dataset, none
twice (capitals and spaces do not make another tag). Without Python: caeli-data dataset register <key> --kind table --title "..." --cadence hourly --area N_EU --column no2:double:ug/m3 --tag NO2 --tag test, caeli-data dataset tags <key>, caeli-data dataset tags <key> --set NO2 --set "air quality", caeli-data tags.
A registration and a load's announcement can be tried first (0.17.0, API 0.21.0): the platform runs every check and
read of the call and writes nothing, and its answer comes back whole. A refusal is the ApiError the real call would
raise. The library sends a dry run only to an API whose /healthz says 0.21.0 or later, since an older one would make
the real call. Without Python: caeli-data dataset register <key> ... --dry-run.
plan = repo.register_dataset("zz-test-x", "table", "A test table", "hourly", columns=[...], areas=["N_EU"], tags=["NO2"], dry_run=True)
plan["dataset"]["table_name"], plan["dataset"]["tags"], plan["warnings"]
repo.announce_load(m, dry_run=True) # {"dry_run": True, "job": {...}, "behind": [the open jobs it would wait for]}
The jobs, newest first, by who announced them, dataset, area, period, kind and status (0.17.0, API 0.21.0); every key
sees every job, and mine=True narrows the list to the calling key's. job_pages reads page after page:
repo.jobs(mine=True, status="done", limit=100)
for page in repo.job_pages(dataset="zz-test-x", limit=200):
...
A table's value columns say what they are (0.11.0, API 0.19.0): a description of at most 200 characters on one line
and a unit of at most 40, set by a registrar without a new schema version (no name and no type changes, so nothing is
loaded again). Left out is kept, a text sets it, None or "" takes it away; all or nothing, so a name the table
does not have writes nothing:
repo.columns("cds-era5-single-levels") # [{"name": "u100", "type": "float", "unit": ..., "description": ...}, ...]
repo.describe_columns("cds-era5-single-levels", {"d2m": {"description": "2 metre dewpoint temperature", "unit": "K"}}) # registrar
Without Python: caeli-data dataset columns <key> (--missing: those without a description) and caeli-data dataset describe <key> --file columns.json (--dry-run shows what would change); the file is {"d2m": {"description": "...", "unit": "K"}} or the API's {"columns": [...]}.
A dataset has a short name a person would say (0.12.0, API 0.20.0), such as "OpenAQ NO2" or "CAMS forecast": 2 to 40
characters on one line, and no two live datasets share one (capitals and blanks not counting). It comes back as
repo.dataset(key)["short_name"] and on every row of repo.datasets(); an area's name and description can be set by
an admin:
repo.set_short_name("ads-cams-forecast", "CAMS forecast") # registrar; None or "" takes it away
repo.register_dataset(..., tags=[...], short_name="OpenAQ NO2") # optional at registration
repo.set_area("NL_WIDE", name="Netherlands, wide") # admin; description="..." too
Without Python: caeli-data dataset short-name <key> shows it, caeli-data dataset short-name <key> "CAMS forecast"
sets it, --clear takes it away; caeli-data area set NL_WIDE --name "Netherlands, wide".
A collection is one recipe: register a raster dataset with the configuration that makes it
(register_dataset(..., recipe=recipe_dict, tags=[...]), which records it as caeli:recipe with its fingerprint()), send the
fingerprint with every publication (publish_raster(..., recipe_fingerprint=fingerprint(recipe_dict))), and the
platform refuses a map made with another recipe — that is a different dataset.
Reading goes through the same door (0.6.0, API 0.9.x). One call gives the rows of a table dataset as one file or one DataFrame, at any size:
repo.download("cdse-sentinel-5p-l2-no2", "no2_nl_2026-06.csv", area="NL", period="2026-06",
columns=["product__nitrogendioxide_tropospheric_column", "product__qa_value"],
filters=[("product__qa_value", ">=", 0.75)])
df = repo.read_table("ads-cams-forecast", regions=["N_EU"], start="2026-06-01", end="2026-06-08", columns=["pm2p5", "lead_hours"])
repo.plan("cdse-sentinel-5p-l2-no2", regions=["N_EU"], period="2026-06", columns=["product__nitrogendioxide_tropospheric_column"])
The API decides beforehand, from the tables' metadata, whether a request is small enough; a request that is too large
comes back as consecutive windows in time, and download and read_table fetch them one after another and join
them into one file (split=False refuses instead, with ReadTooLarge carrying the plan). Every download is pinned to
one table snapshot (version= takes an item key or snapshot:<id>), filters are typed against the schema
(("qa_value", ">=", 0.75), ("orbit", "in", [1, 2]), ("no2", "notnull")), and the three index columns
idx_utc, idx_lat, idx_lon are always part of a download. Without Python: caeli-data plan ...,
caeli-data download ... --to file.csv, caeli-data reads.
Maps and other published files are read by their item (0.6.1, API 0.10.0). An item is one area and period of a
dataset (N_EU__2026-06 is the current version of that month, N_EU__2026-06__v2 one version), and its file is
fetched through a link the API hands out for a quarter of an hour, straight from the file store, checked against
the record's size and sha256; what is already there is not fetched again:
repo.items("caeli-iasi-nh3-tc-monthly-wgs001", start="2025-01", end="2025-12") # the months, by area and period
repo.download_item("caeli-iasi-nh3-tc-monthly-wgs001", "N_EU__2026-06", "maps") # one map into the folder maps/
repo.download_items("caeli-iasi-nh3-tc-monthly-wgs001", "maps", start="2025-01", end="2025-12") # a year of maps
Without Python: caeli-data items <dataset> --from 2025-01 --to 2025-12, caeli-data fetch <dataset> --from 2025-01 --to 2025-12 --into maps, caeli-data fetch <dataset> --item N_EU__2026-06 --into maps. Admins make keys with
repo.create_key(name, role, owner_email) or caeli-data key create --name piet --role reader --owner piet@caeli.nl;
the key is shown once and goes to its owner through Secrets Manager (the platform's vault), never through a chat or a mail.
A map series is bound to the recipe that makes it, and its months are made by the maps service on request (0.6.2,
API 0.11.0). The versions of a recipe are numbered by the API and kept in the record and in the library of the file
store (recipes/<name>/v<N>.json); a changed recipe becomes the next version (registrar), the same map-making fields
again change nothing. A map job is one period of a series: the API records it, queues it for the maps service and,
from maps 0.3.9 on, follows it from queued to done, exists, empty, failed or rejected:
repo.recipe("caeli-tropomi-no2-trop-monthly-wgs001") # the current version, recipe included
repo.bind_recipe_file("caeli-tropomi-no2-trop-monthly-wgs001", "recipes/tropomi-no2-trop-monthly-wgs001.json", notes="why")
repo.create_map_jobs("caeli-tropomi-no2-trop-monthly-wgs001", start="2019-01", end="2026-10") # one job per month (publisher)
repo.map_jobs(dataset="caeli-tropomi-no2-trop-monthly-wgs001", status="queued")
repo.cancel_map_job(job_id)
Without Python: caeli-data recipes, caeli-data recipe show <dataset>, caeli-data recipe bind <dataset> --file <recipe.json> --notes "why", caeli-data map-jobs create <dataset> --from 2019-01 --to 2026-10, caeli-data map-jobs list --dataset <dataset>, caeli-data map-job show <id>, caeli-data map-job cancel <id>.
The lab (0.7.0, API 0.12.0, maps 0.4.0) tries a recipe on a small box and one period without touching any dataset:
the lab tasks of the maps service run it on the box, snapped to whole cells of the recipe's grid, and write map.tif,
preview.png, bounds.json and result.json under _lab/<user>/<job id>/ in the file store, kept seven days (a
reader may ask for 20 a day, a publisher 100). A draft is the recipe a person is editing, kept in the library next to
the bound versions (recipes/<name>/draft.json) and never read by the platform itself; the style of a series is the
colour map and value range every month is shown with:
repo.lab_job(recipe=recipe_dict, period="2026-06", box=[1, 48.5, 8, 53.5]) # or dataset="...", version="current"
repo.wait_lab_job(job_id); repo.lab_files(job_id); repo.download_lab(job_id, "lab/2026-06")
repo.draft("tropomi-no2-trop-monthly-wgs001"); repo.save_draft(name, recipe_dict); repo.delete_draft(name)
repo.set_style("caeli-tropomi-no2-trop-monthly-wgs001", colormap="reds", rescale=[0, 1000]) # registrar
Without Python: caeli-data lab run --file <recipe.json> --period 2026-06 --box 1,48.5,8,53.5 --wait --into lab,
caeli-data lab list, caeli-data lab show <id>, caeli-data lab fetch <id> --into lab, caeli-data draft show | save | delete, caeli-data style set <dataset> --colormap reds --rescale 0,1000. The laptop mirror of the bound recipes
(caeli-maps-recipes/recipes) is python scripts/mirror_recipes.py --into <folder> --apply.
Deleting a dataset (0.8.0, API 0.13.0, schema v1.12) can be taken back for thirty days. A delete marks the dataset in
the record (when, by whom, why, and the day of its purge); it leaves the API, the catalogue and the tiles at once, its
queued map jobs are cancelled, but its rows, its table and its files stay until the API's sweep purges it after that
day. A real dataset is deleted by an admin who repeats its key and gives a reason; a zz-test-* dataset needs the
reason only, or goes at once with purge=True (the tests' clean start). The impact is read first:
repo.impact("caeli-tropomi-no2-trop-monthly-wgs001") # what a delete would touch, and whether it is blocked
repo.delete_dataset("caeli-tropomi-no2-trop-monthly-wgs001", reason="why, ten characters or more", confirm=True) # admin
repo.deleted_datasets() # what can still be restored, with the marks
repo.restore_dataset("caeli-tropomi-no2-trop-monthly-wgs001") # back whole, catalogue included
repo.delete_dataset("zz-test-load", purge=True) # a test dataset, gone at once
Without Python: caeli-data dataset impact <dataset>, caeli-data dataset delete <dataset> --confirm <dataset> --reason "why", caeli-data dataset list --deleted, caeli-data dataset restore <dataset>, caeli-data dataset delete zz-test-load --purge.
Deleting one item (0.9.0, API 0.14.0, schema v1.13): a month of a raster, file or model dataset, or one version of it, deleted by an admin with a reason, can be taken back for thirty days. It leaves the API, the catalogue and the tiles at once; its file stays until the sweep removes it, and its row stays for ever, so its version number is never used again (after v1 was deleted the month's next publication is v2). A month whose current item was deleted is missing until it is published again; no older version becomes current. A restore brings the item back as current when its month has no current item, as deprecated otherwise. The item is named in full:
repo.item_impact("caeli-tropomi-no2-trop-monthly-wgs001", "N_EU__2026-05__v1") # what a delete would touch, and whether it is blocked
repo.delete_item("caeli-tropomi-no2-trop-monthly-wgs001", "N_EU__2026-05__v1", reason="why, ten characters or more", confirm=True) # admin
repo.items("caeli-tropomi-no2-trop-monthly-wgs001", status="deleted") # what can still be restored, with the marks
repo.restore_item("caeli-tropomi-no2-trop-monthly-wgs001", "N_EU__2026-05__v1")
repo.delete_item("zz-test-maps", "N_EU__2026-05__v1", purge=True) # a test item's file, gone at once
Without Python: caeli-data item impact <dataset> <item>, caeli-data item delete <dataset> <item> --confirm <item> --reason "why", caeli-data items <dataset> --deleted, caeli-data item restore <dataset> <item>. The old versions
of a series go in bulk with caeli-data item clear-deprecated <dataset> --before 2026-10-01 --reason "why": it lists
the versions replaced before that day with their count and size (--dry-run stops there), asks for the dataset key,
and deletes them one by one, each with its own restore; the day must be thirty days ago or more. Publishing the same
file for a month again after its item was deleted is a new publication: publish_file and publish_raster see that
the job the file names made an item that is not the month's current one any more and announce it again.
Metadata
Release files for caeli-data 0.18.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| caeli_data-0.18.0.tar.gz | 155.5 kB | Without provenance | Details for caeli_data-0.18.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| caeli_data-0.18.0-py3-none-any.whl | Python 3 | none | any | Details for caeli_data-0.18.0-py3-none-any.whl |
Total release size: 314.2 kB
Release files / caeli_data-0.18.0.tar.gz
| Download URL | caeli_data-0.18.0.tar.gz |
|---|---|
| Size | 155.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
5c03a65e2f18ee8b03690a704d929928196ff9be8ebfd387a7301dcf555184ee
|
|
BLAKE2b-256 checksum How to use checksums |
2fd8f1e60002fbd6ef0888ec689c6d56cfe19ed6ad0177682e8b4869d62fcfed
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.10
|
Release files / caeli_data-0.18.0-py3-none-any.whl
| Download URL | caeli_data-0.18.0-py3-none-any.whl |
|---|---|
| Size | 158.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
c6747bfb9cb3143cf9e76f83fe9e4c181c6f75a12cb7ca759e4073bc4224562a
|
|
BLAKE2b-256 checksum How to use checksums |
dd241e646d4344be2e27634e91b256bcc5f871259fb32306f4b1b8475f04a546
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.10
|
Security for caeli-data 0.18.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.