climate-ref-celery
This package provides Celery task generation and worker management for the Climate REF (Rapid Evaluation Framework).
The climate_ref_celery package provides a CLI tool(ref-celery) to start a worker instance for a diagnostics provider.
This worker instance will listen for tasks related to a provider and execute them.
The compute engine worker will then collect the results of these tasks and store them in the database.
This allows for the REF to be run in a distributed manner,
with multiple workers running on different machines with a centrally managed database.
Installation
pip install climate-ref-celery
Features
- Distributed task execution for metrics providers
- Worker management and monitoring
- Automatic task routing and scheduling
- Results collection and storage
- Support for multiple worker instances
Usage
Starting a Worker
To start a worker instance for a specific metrics provider:
# Using the standalone CLI
ref-celery start-worker --package climate_ref_pmp
# Or using the main REF CLI
ref celery start-worker --package climate_ref_pmp
Configuration
Each worker instance may not share the same configuration as the orchestrator. This is because the worker may be running on a different machine with different resources available or directories.
Each worker instance requires access to a shared input data directory and the output directory. If the worker is deployed as a docker container these directories can be mounted as volumes.
Each worker instance can be configured independently. Required configurations include:
- Shared input data directory
- Output directory for results
- Database connection settings
Documentation
For detailed documentation, please visit https://climate-ref.readthedocs.io/
Contributing
Contributions are welcome! Please see the main project's Contributing Guide for more information.
License
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
2 maintainers
Metadata
2 maintainers
Release files for climate-ref-celery 0.19.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| climate_ref_celery-0.19.0.tar.gz | 25.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| climate_ref_celery-0.19.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 50.2 kB
Release files / climate_ref_celery-0.19.0.tar.gz
| Download URL | climate_ref_celery-0.19.0.tar.gz |
|---|---|
| Size | 25.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7550f31855889f85465c513145ebeff4f3e92c9a4e6f9b8fb6f9f25a84fa28fb
|
|
BLAKE2b-256 checksum How to use checksums |
f7afb3506bf3360c27525669e191a11019c9081fe0c17a82f20aafab924a23f5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.12.7 {"installer":{"name":"uv","version":"0.12.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / climate_ref_celery-0.19.0-py3-none-any.whl
| Download URL | climate_ref_celery-0.19.0-py3-none-any.whl |
|---|---|
| Size | 24.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
c2b780066718fd0878dd88d46b86101359a21ca38175abe02b6737f4f2313dbc
|
|
BLAKE2b-256 checksum How to use checksums |
9a803224bc05213d1630e6dffd31b1902afa82179cffb7c6ef50f66f04793ba6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.12.7 {"installer":{"name":"uv","version":"0.12.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
