Kedro SLURM
kedro-slurm is a library that integrates Kedro pipelines with SLURM to enable distributed execution of tasks on high-performance computing (HPC) clusters. This library provides seamless integration for defining, submitting, and monitoring jobs on SLURM, leveraging SLURM's job scheduling capabilities while adhering to Kedro's pipeline structure.
INSTALLATION: pip install kedro-slurm
How do I use Kedro SLURM?
To define a SLURM-enabled node, use the kedro_slurm.pipeline.node function.
This allows you to specify SLURM resource requirements and job configurations for each node in your pipeline.
from kedro_slurm.pipeline import node
def function(input_data):
# Your node logic here
return processed_data
node = node(
func=function,
inputs="input_data",
outputs="processed_data",
name="my_slurm_node",
resources=slurm.Resources(cpus=4, memory=16, gpus=1),
configuration=slurm.Configuration(time_limit="2:00:00", partition_name="gpu"),
)
Define your pipeline by combining SLURM nodes with standard Kedro nodes. Kedro nodes will run using the library's default resource settings.
from kedro.pipeline import Pipeline, node
from kedro_slurm.pipeline import node as slurm_node
pipeline = Pipeline([
slurm_node(
func=function_1,
inputs="input_data",
outputs="processed_data",
name="slurm_node_1",
resources=slurm.Resources(cpus=8, memory=32),
configuration=slurm.Configuration(time_limit="4:00:00"),
),
node(
func=function_2,
inputs="input_data",
outputs="processed_data",
name="node_1",
),
# Add more nodes here
])
To run your pipeline on SLURM, use the custom SLURMRunner by executing the following shell command:
kedro run --async --runner=kedro_slurm.runner.SLURMRunner
Monitoring SLURM Jobs
The library offers abstractions for submitting and monitoring jobs on SLURM.
You can submit a SLURM job using the kedro_slurm.slurm.Job class with its submit method, and monitor the job using the kedro_slurm.slurm.Future class.
from kedro_slurm import slurm
resources = slurm.Resources(cpus=8, memory=32, gpus=2)
configuration = slurm.Configuration(time_limit="4:00:00", partition_name="gpu")
job = slurm.Job(
resources=resources,
configuration=configuration,
name="example_job",
command="python train_model.py",
path="./logs/%j",
)
future = job.submit()
while not future.done:
future.update()
print(f"Job status: {future._state}")
time.sleep(5)
A Future can transition through the following job states:
- RUNNING
- COMPLETED
- PENDING
- FAILED
- CANCELLED
- PREEMPTED
- SUSPENDED
- STOPPED
Default SLURM Resource Configuration
_DEFAULT_RESOURCES = slurm.Resources(cpus=4, memory=10)
_DEFAULT_CONFIGURATION = slurm.Configuration(time_limit="1:00:00")
Metadata
Release files for kedro-slurm 0.1.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| kedro_slurm-0.1.5.tar.gz | 5.7 kB | Without provenance | Details for kedro_slurm-0.1.5.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| kedro_slurm-0.1.5-py3-none-any.whl | Python 3 | none | any | Details for kedro_slurm-0.1.5-py3-none-any.whl |
Total release size: 13.3 kB
Release files / kedro_slurm-0.1.5.tar.gz
| Download URL | kedro_slurm-0.1.5.tar.gz |
|---|---|
| Size | 5.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e103894bb027f14808358eac1eb134d9247be5531148ab24983ba71825759a6b
|
|
BLAKE2b-256 checksum How to use checksums |
a0784ce051fc13de9db20236ba1635a12da4a7687fc0683f766059cb8bdd403d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
pdm/2.22.3 CPython/3.12.9 Darwin/23.6.0
|
Release files / kedro_slurm-0.1.5-py3-none-any.whl
| Download URL | kedro_slurm-0.1.5-py3-none-any.whl |
|---|---|
| Size | 7.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e29887f20c54ff52a196ad89cb73adb3344742aaa4e6c990e2a7731d522cbfb3
|
|
BLAKE2b-256 checksum How to use checksums |
1f6f67f463dcec36cd039b3998010cd0fa8828a995fc962ef08919ac756782e3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
pdm/2.22.3 CPython/3.12.9 Darwin/23.6.0
|
Security for kedro-slurm 0.1.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.