icflow is a Python package with some prototype workflow tools for use at the Irish Centre for High End Computing (ICHEC).

It is intended to help standardize some of our workflows in areas like Machine Learning by using common utilities, data-formats and data-structures.

See the project documentation on ReadTheDocs.

Features

Parameter Sweep

You can define a parameter sweep in a yaml file as follows:

title: "my_parameter_sweep"
program: "launch_program"  

parameters:
    param_0 : 1
    param_1 : "my_value"
    param_2 : [1, 2]
    param_3 : ["a", "b"]

Running:

icflow sweep --config my_config.yaml

with this fill will launch the program or script defined by program, which should be in the system PATH.

The listed parameters are passed as command line arguments to the program in the form --key value. Parameter value lists are expanded such that there is a program launch for each combination of values in the list. In the above example this will result in the following program launches:

launch_program --param_0 1 --param_1 my_value --param_2 1 --param_3 a
launch_program --param_0 1 --param_1 my_value --param_2 2 --param_3 a
launch_program --param_0 1 --param_1 my_value --param_2 1 --param_3 b
launch_program --param_0 1 --param_1 my_value --param_2 2 --param_3 b

Program launching is handled internally by ICHEC's ictasks library, with each of these program launches handled as a 'task'.

Running sweeps in parallel

You can also specify details about the number of gpu/cpu s available for running tasks by including the config and inner task_distribution sections in the sweep config This can allow for tasks to be run in parallel, see a description in ictasks covering this task_distribution and using gpus in your tasks.

title: ...
program: ... 

parameters:
    ...

config:
    task_distribution:
        ...

Installation

It is available on PyPI:

pip install icflow

Copyright

This software is Copyright of the Irish Centre for High End Computing 2024. You can use it under the terms of the GPLv3+. See the included LICENSE file for details.

Release files for icflow 0.2.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for icflow 0.2.4
File	Size	Uploaded
icflow-0.2.4.tar.gz	33.2 kB	Oct 24, 2025	Details

Built distribution (wheel)

Table of built distributions (wheels) for icflow 0.2.4
File	Interpreter	ABI	Platform	Reset
icflow-0.2.4-py3-none-any.whl 37.0 kB Oct 24, 2025	Python 3	none	any	Details

Total release size: 70.2 kB

Release files / icflow-0.2.4.tar.gz

Download URL	icflow-0.2.4.tar.gz
Size	33.2 kB
Tags	Source
SHA-256 checksum How to use checksums	`29383ffbf9f2414680f23b4bc62a28d7e588ba34ec0487f8b18ba7501daf3c53`
BLAKE2b-256 checksum How to use checksums	`5e1c5e14a12aa5f1272b14b540efdd7e316d4c92a543f5c22e81aa73e3209d3d`
Upload date	Oct 24, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.2.0 CPython/3.13.9`

Release files / icflow-0.2.4-py3-none-any.whl

Download URL	icflow-0.2.4-py3-none-any.whl
Size	37.0 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`193128ea11982ff087d2cff1ae9161f2c43c56e6c25e478a3bc2475d3fd8d629`
BLAKE2b-256 checksum How to use checksums	`740b453e1fb71c3ff8450fa63b29a0f089654cd4e2a4b5f0e06e434a24378988`
Upload date	Oct 24, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.2.0 CPython/3.13.9`