Installation • Quick Start • Documentation
[!IMPORTANT] This software is Emerging and subject to ECMWF's guidelines on Software Maturity.
earthkit-workflows is a Python library for declaring earthkit task as DAGs.
It contains an internal cascade engine for scheduling and executing task graphs almost optimally across heterogeneous platforms with complex network technologies and topologies.
It effectively performs task-based parallelism across CPUs, GPUs, distributed systems (HPC), and any combination thereof.
It is designed for a no-IO approach, where expensive storage of intermediate data is minimised whilst maximising all available transport technologies between different hardware.
Cascade is designed to work on well-profiled task graphs, where:
- the task graph is a static DAG,
- the DAG nodes are defined by tasks with well-known execution times,
- the DAG edges are defined by data dependencies with well-known data sizes,
- the characteristics of the hardware (processors, network connections) are known.
earthkit-workflows allows for declaring such task graphs using a neat fluent API, and interoperates pleasantly with the rest of the earthkit ecosystem.
Installation
Install via pip with:
$ pip install 'earthkit-workflows[all]'
For development, you can use pip install -e . though there is currently an issue with earthkit masking. Additionally you may want to install pre-commit hooks via
$ pip install pre-commit
$ pre-commit install
Quick Start
Note: this section is moderately outdated.
We support two regimes for cascade executions -- local mode (ideal for developing and debugging small graphs) and distributed mode (assumed for slurm & HPC).
To launch in local mode, in your python repl / jupyno:
import cascade.benchmarks.job1 as j1
import cascade.benchmarks.distributed as di
import cloudpickle
spec = di.ZmqClusterSpec.local(j1.get_prob())
print(spec.controller.outputs)
# prints out:
# {DatasetId(task='mean:dc9d90 ...
# defaults to all "sinks", but can be overridden
rv = di.launch_from_specs(spec, None)
for key, value in rv.outputs.items():
deser = cloudpickle.loads(value)
print(f"output {key} is of type {type(deser)}")
For distributed mode, launch
./scripts/launch_slurm.sh ./localConfigs/<your_config.sh>
Inside the <your_config.sh>, you define size of the cluster, logging directory output, which job to run... Pay special attention to definitions of your venv and LD_LIBRARY_PATH etc -- this is not autotamed.
Both of these examples hardcode particular job, "job1", which is a benchmarking thing.
Most likely, you want to define your own -- for the local mode, just pass cascade.Graph instance to the call; in the dist mode, you need to provide that instance in the cascade.benchmarks.__main__ modules instead (ideally by extending the get_job function).
There is also python -m cascade.benchmarks local <..> -- you may use that as an alternative path to local mode, for your own e2e tests.
Documentation
Not yet available.
Contributions and Support
Due to the maturity and status of the project, there is no support provided -- unless the usage of this project happens within some higher-status initiative that ECMWF participates at. External contributions and created issues will be looked at, but are not guaranteed to be accepted or responded to. In general, follow ECMWF's guidelines for external contributions.
License
See license.
1 maintainer
Release files for earthkit-workflows 0.17.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| earthkit_workflows-0.17.0.tar.gz | 10.9 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| earthkit_workflows-0.17.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 11.1 MB
Release files / earthkit_workflows-0.17.0.tar.gz
| Download URL | earthkit_workflows-0.17.0.tar.gz |
|---|---|
| Size | 10.9 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
bd29f96c0dcbb11b2a6d60c648de9a17b885efdedce21ca6bb18cad3d1eb2db9
|
|
BLAKE2b-256 checksum How to use checksums |
f03ac4ba5ba4e59a5d1d710b57508380bcdd135ad2f642906511a43f40ac6152
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.7
|
Release files / earthkit_workflows-0.17.0-py3-none-any.whl
| Download URL | earthkit_workflows-0.17.0-py3-none-any.whl |
|---|---|
| Size | 216.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
7b2ebc95628ede8a8e9136d315e64f3150d97cb21e82913224b61507e167e783
|
|
BLAKE2b-256 checksum How to use checksums |
ef34c5354243b13768b841aa9af342845d64f958ff0755c2973e87ff29ac374e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.7
|
