collekt
A library to facilitate spatio-temporal data collection across arbitrary datasources.
A dataset config names the products to fetch, and a request supplies the region, time window, and metadata. The selected products are fanned out to pluggable source adapters. Each adapter fetches source-native files (NetCDF, GRIB2, Parquet, product archives) and records what it did in a machine-readable manifest. Gridded results can then be assembled onto a common grid and time axis.
Installation
Requires Python 3.12+ and uv.
git clone https://github.com/ai4copsec/collekt.git
cd collekt
uv sync --all-groups # or: just install
All provider clients and the gridded Assembler are installed by default, so
every source works out of the box. Clients are imported lazily, so importing
collekt stays fast.
To build the documentation locally, you will also need the quarto binary (quarto.org):
just docs
# or :
uv run quartodoc build --config docs/_quarto.yml
quarto render docs
We recommend using just (https://github.com/casey/just) with the make-like commands available in justfile.
Usage
Python
import collekt
config = collekt.DatasetConfig(
collekt.CMEMS("cmems_glorys_my", variables=["uo", "vo"], depth=[1.0, 1.1]),
collekt.CMEMS("cmems_duacs_my", variables=["ugos", "vgos"]),
collekt.CMEMS("cmems_med_currents_my", variables=["uo", "vo"], depth=[1.0, 1.1]),
)
request = collekt.Request(
region=collekt.Region.from_bbox((-6.0, 20.0, 35.0, 45.0)),
start="2023-06-15",
)
fetcher = collekt.Fetcher(request=request, config=config, output_dir="data/collections")
result = fetcher.download()
print(result.manifest_path)
# Assemble gridded sources onto a common grid and time axis.
assembler = collekt.Assembler(result)
dataset = assembler.to_xarray(grid="lowest_resolution", time="lowest_resolution")
Region can also be built from a point and radius
(Region.from_point_radius(lat, lon, radius_km)) or a GeoJSON file
(Region.from_geojson(path)).
For longer requests, pass batch_days=7 to Fetcher (or --batch-days 7 to
collekt fetch). This groups compatible retrievals while retaining daily files
for daily sources. The default behavior is unchanged; provider capabilities
determine whether a group reduces network requests. See
batch downloads.
Command line
collekt config show # inspect the bundled catalog
collekt doctor # check the environment and configuration
cat > datasets.yaml <<'YAML'
datasets:
- provider: cmems
key: cmems_glorys_my
variables: [uo, vo]
depth: [1.0, 1.1]
- provider: cmems
key: cmems_duacs_my
variables: [ugos, vgos]
- provider: cmems
key: cmems_med_currents_my
variables: [uo, vo]
depth: [1.0, 1.1]
YAML
collekt fetch --bbox -6 20 35 45 --start 2023-06-15 \
--dataset-config datasets.yaml --output-dir data/collections
Use --dry-run to plan provider requests without downloading, and --strict to
fail if any requested source is skipped.
Sources and configuration
collekt ships a curated catalog of datasets (CMEMS global and regional seas,
ECMWF Open Data, ERA5, NOAA GFS, Skytruth, HOZINT, Global Fishing Watch,
Copernicus Data Space, Geonorge, eOdyn), and a local adapter reads an existing
tabular archive on disk. Select concrete
dataset keys with DatasetConfig or its YAML form, and put provider parameters
such as variable names and depth ranges directly on each selected dataset. See
Products and
Credentials.
Development
just lint # format + lint with ruff
just test # run the test suite
just docs # build the Quarto documentation
The definition of done for any change: just lint and just test pass, with a
test for any behavior changed.
Documentation
Built with Quarto and
quartodoc from docs/; the rendered site
is published to GitHub Pages.
License
collekt does not grant you any rights to the wrapped third party providers - it only simplifies making multiple-queries to the sources you already have the permission to access.
Each wrapped service comes with its own separate "Terms of Use", e.g., Global Fishing Watch is for non-commercial use only - others may carry similar restrictions.
Copyright
Copyright (c) 2023-2026 Simula Research Laboratory, Oslo, Norway.
Acknowledgments
Part of the EU project AI4COPSEC, funded by the Horizon Europe framework programme under Grant Agreement N. 101190021.
Metadata
Release files for collekt 0.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| collekt-0.3.0.tar.gz | 110.9 kB | Without provenance | Details for collekt-0.3.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| collekt-0.3.0-py3-none-any.whl | Python 3 | none | any | Details for collekt-0.3.0-py3-none-any.whl |
Total release size: 252.5 kB
Release files / collekt-0.3.0.tar.gz
| Download URL | collekt-0.3.0.tar.gz |
|---|---|
| Size | 110.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f2892f92af0315aacf66a567e6d65ce69c46a8227a7cb512a1d75f0e4ad04b6e
|
|
BLAKE2b-256 checksum How to use checksums |
78d981d4c6e743d96cc811a6daa26805181931330effe407422791a7c6aeb4d7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / collekt-0.3.0-py3-none-any.whl
| Download URL | collekt-0.3.0-py3-none-any.whl |
|---|---|
| Size | 141.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
a5df0622db1118bc70aa53c95433ac52d3501ba6636e3158dd7425a657d1a58b
|
|
BLAKE2b-256 checksum How to use checksums |
86d769f2cd99d1df63e7d5366351c16628051a51a04e5c7c2f3ec36142eb18d7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for collekt 0.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.