A python package that enables user to build their custom singularity image on HPC cluster
Project description
Building a singular container for HPC using globus-compute
Context
-
One of the executions configurations of globus-compute requires a registered container which is spun up to execute the user function on the HPC.
-
HPCs do not run docker containers(due to security reasons as discussed here) and support only an apptainer/singularity image.
-
Installing the apptainer setup to build the singularity image locally is not a straightforward process especially on windows and mac systems as discussed in the documentation.
Using this python library the user can specify their custom image specification to build an apptainer/singularity image which would be used to in-turn to run their functions on globus-compute. The library registers the container and returns the container id which would be used by the globus-compute executor to execute the user function.
Prerequisite.
A globus-compute-endpoint setup on HPC cluster.
Example
Consider the following use-case where the user wants to execute a pandas operation on HPC using globus-compute. They need a singularity image which would be used by the globus-compute executor. The library can be leveraged as follows:
from custom_image_builder import build_and_register_container
tutorial_endpoint = "01e21ddf-6eb4-41db-8e1d-2bcfe0c8314f"
container_id = build_and_register_container(endpoint_id=tutorial_endpoint,
image_file_name="my-test-image",
base_image_type="docker",
base_image="python:3.8",
pip_packages=["pandas"])
print("Container id ", container_id)
from globus_compute_sdk import Executor
# User function runs on the HPC node
def transform():
import pandas as pd
data = {
'City': ['New York', 'San Francisco', 'Los Angeles']
}
return pd.DataFrame(data)
with Executor(endpoint_id=tutorial_endpoint,
container_id=container_id) as ex:
fut = ex.submit(transform)
print(fut.result())
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for custom_image_builder-0.1.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | dbf1dae4322f30a0fffe6db6b39e3d8779f641dabde7c1626c0dfcf7c365217b |
|
MD5 | d3cbf0d5c36570c3614b0dbc425ea519 |
|
BLAKE2b-256 | 783eed09516144c4fc285e8f5ae440539f5ef68238db9f06d5b45ef58b0f5159 |
Hashes for custom_image_builder-0.1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e0ab9e2a6ac32fdf3c3e01445187a2a9a943a3d1698eab08c876041f2c9b309f |
|
MD5 | 5c553e924dba021474302bfb89b2ee32 |
|
BLAKE2b-256 | d0e32c5d3963bf58a7b3f2e1564a68cc14180b123fce85bf1851a08bb36b3322 |