Skip to main content

collekt

A library to facilitate spatio-temporal data collection across arbitrary datasources.

A dataset config names the products to fetch, and a request supplies the region, time window, and metadata. The selected products are fanned out to pluggable source adapters. Each adapter fetches source-native files (NetCDF, GRIB2, Parquet, product archives) and records what it did in a machine-readable manifest. Gridded results can then be assembled onto a common grid and time axis.

Installation

Requires Python 3.12+ and uv.

git clone https://github.com/ai4copsec/collekt.git
cd collekt
uv sync --all-groups  # or: just install

All provider clients and the gridded Assembler are installed by default, so every source works out of the box. Clients are imported lazily, so importing collekt stays fast.

To build the documentation locally, you will also need the quarto binary (quarto.org):

just docs
# or :
uv run quartodoc build --config docs/_quarto.yml
quarto render docs

We recommend using just (https://github.com/casey/just) with the make-like commands available in justfile.

Usage

Python

import collekt

config = collekt.DatasetConfig(
    collekt.CMEMS("cmems_glorys_my", variables=["uo", "vo"], depth=[1.0, 1.1]),
    collekt.CMEMS("cmems_duacs_my", variables=["ugos", "vgos"]),
    collekt.CMEMS("cmems_med_currents_my", variables=["uo", "vo"], depth=[1.0, 1.1]),
)
request = collekt.Request(
    region=collekt.Region.from_bbox((-6.0, 20.0, 35.0, 45.0)),
    start="2023-06-15",
)

fetcher = collekt.Fetcher(request=request, config=config, output_dir="data/collections")
result = fetcher.download()
print(result.manifest_path)

# Assemble gridded sources onto a common grid and time axis.
assembler = collekt.Assembler(result)
dataset = assembler.to_xarray(grid="lowest_resolution", time="lowest_resolution")

Region can also be built from a point and radius (Region.from_point_radius(lat, lon, radius_km)) or a GeoJSON file (Region.from_geojson(path)).

For longer requests, pass batch_days=7 to Fetcher (or --batch-days 7 to collekt fetch). This groups compatible retrievals while retaining daily files for daily sources. The default behavior is unchanged; provider capabilities determine whether a group reduces network requests. See batch downloads.

Command line

collekt config show       # inspect the bundled catalog
collekt doctor            # check the environment and configuration

cat > datasets.yaml <<'YAML'
datasets:
  - provider: cmems
    key: cmems_glorys_my
    variables: [uo, vo]
    depth: [1.0, 1.1]
  - provider: cmems
    key: cmems_duacs_my
    variables: [ugos, vgos]
  - provider: cmems
    key: cmems_med_currents_my
    variables: [uo, vo]
    depth: [1.0, 1.1]
YAML

collekt fetch --bbox -6 20 35 45 --start 2023-06-15 \
  --dataset-config datasets.yaml --output-dir data/collections

Use --dry-run to plan provider requests without downloading, and --strict to fail if any requested source is skipped.

Sources and configuration

collekt ships a curated catalog of datasets (CMEMS global and regional seas, ECMWF Open Data, ERA5, NOAA GFS, Skytruth, HOZINT, Global Fishing Watch, Copernicus Data Space, Geonorge, eOdyn), and a local adapter reads an existing tabular archive on disk. Select concrete dataset keys with DatasetConfig or its YAML form, and put provider parameters such as variable names and depth ranges directly on each selected dataset. See Products and Credentials.

Development

just lint    # format + lint with ruff
just test    # run the test suite
just docs    # build the Quarto documentation

The definition of done for any change: just lint and just test pass, with a test for any behavior changed.

Documentation

Built with Quarto and quartodoc from docs/; the rendered site is published to GitHub Pages.

License

BSD-3-Clause.

collekt does not grant you any rights to the wrapped third party providers - it only simplifies making multiple-queries to the sources you already have the permission to access.

Each wrapped service comes with its own separate "Terms of Use", e.g., Global Fishing Watch is for non-commercial use only - others may carry similar restrictions.

Copyright (c) 2023-2026 Simula Research Laboratory, Oslo, Norway.

Acknowledgments

Part of the EU project AI4COPSEC, funded by the Horizon Europe framework programme under Grant Agreement N. 101190021.

Metadata

Release files for collekt 0.3.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for collekt 0.3.0
File Size Uploaded Provenance Actions
collekt-0.3.0.tar.gz 110.9 kB Without provenance Details for collekt-0.3.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for collekt 0.3.0
File Interpreter ABI Platform Actions
collekt-0.3.0-py3-none-any.whl Python 3 none any Details for collekt-0.3.0-py3-none-any.whl

Total release size: 252.5 kB

Release files / collekt-0.3.0.tar.gz

Download URL collekt-0.3.0.tar.gz
Size 110.9 kB
Tags Source
SHA-256 checksum
How to use checksums
f2892f92af0315aacf66a567e6d65ce69c46a8227a7cb512a1d75f0e4ad04b6e
BLAKE2b-256 checksum
How to use checksums
78d981d4c6e743d96cc811a6daa26805181931330effe407422791a7c6aeb4d7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / collekt-0.3.0-py3-none-any.whl

Download URL collekt-0.3.0-py3-none-any.whl
Size 141.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a5df0622db1118bc70aa53c95433ac52d3501ba6636e3158dd7425a657d1a58b
BLAKE2b-256 checksum
How to use checksums
86d769f2cd99d1df63e7d5366351c16628051a51a04e5c7c2f3ec36142eb18d7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for collekt 0.3.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page