Help to generate subsets of CVMFS
Project description
SubCVMFS-builder
Purpose
SubCVMFS-builder
aims to help scientific communities to build a subset of CVMFS containing the minimal requirements to execute applications of interests and deploy it on a remote computing infrastructure.
The tool is mainly used on supercomputers having no outbound connectivity.
Installation
- Before starting, you need to have
singularity
,CVMFS
andCVMFS-shrinkwrap
installed on the machine.
git clone https://gitlab.cern.ch/alboyer/subcvmfs-builder.git
cd subcvmfs-builder
pip install . --user
Usage
- Get the help:
subcvmfs --help
-
Trace: run applications in a singularity container and return the list of dependencies related to CVMFS.
subcvmfs trace <config> \ --apps-dir <required_input_directory> --path-list <output_file> --container <optional_input_string> --number-core <optional_input_int>
Trace is generally used when one has no idea about the dependencies of the applications to include in the subset of CVMFS.
apps-dir
should be a directory of applications structured as follows:<apps_dir>/<command>/command.sh
.container
provides a specific environment to run the applications (generally the one present in the remote computing infrastructure).path-list
should contain the result of the command, namely a list of CVMFS dependencies related to the applications to trace.number-core
can be used to run multiple applications in parallel.
-
Build: with a list of dependencies, build a subset of CVMFS using the
cvmfs-shrinkwrap
tool.subcvmfs build <config> \ --path-list <Trace_output> <another_input_list> --subset-path <output_subset_path>
Build is a required step to build a subset of CVMFS.
path-list
should be a file - or a list of files - containing the CVMFS dependencies to include in the subset. It can be obtain from Trace or built from scratch.subset-path
should the path of the generated subset.
-
Test: execute application using a subset of dependencies instead of CVMFS.
subcvmfs test <config> \ --apps-dir <required_input_directory> --subset-path <required_input_path> --container <optional_input_str> --number-core <optional_input_int>
Test is generally used before deploying a subset of CVMFS. One may want to make sure the subset is valid against a certain set of applications.
apps-dir
should be a directory of applications structured as follows:<apps_dir>/<command>/command.sh
.subset-path
should the path of a subset of CVMFS.container
provides a specific environment to run the applications (generally the one present in the remote computing infrastructure).number-core
can be used to run multiple applications in parallel.
-
Deploy: Deploy the subset of CVMFS - within a container if needed - on the remote computing infrastructure.
subcvmfs deploy <config> \ --subset-path <required_input_subset_path> --remote-location <required_input_str> --container <optional_input_path> --bootstrap <optional_input_path> --post-command <optional_input_path> --bundle-path <optional_output_path>
Deploy requires one to have access to the remote computing infrastructure.
subset-path
should the path of a subset of CVMFS.remote-location
the remote computing infrastructure that will host the subset of CVMFS.bundle-path
should be used if one wants to merge a subset of CVMFS with a container (not recommended if the subset of CVMFS is updated periodically).container
provides a specific environment to run the applications. If provided, it will be merged with the subset of CVMFS. Mandatory ifbundle-path
is present.bootstrap
the type of container. Mandatory ifbundle-path
is provided.post-command
should represent a command to run before launching the container. Mandatory ifbundle-path
is present.
Configuration
- Minimal configuration possible:
{
"cvmfs_extensions":
{
"<repository_name>":
{
"url": "<repository_url>",
"public_key": "<repository_pubkey>",
},
}
}
In this case, arguments should be passed through the CLI.
- A concrete example:
{
"cvmfs_extensions":
{
"lhcb.cern.ch":
{
"url": "http://cvmfs-stratum-one.cern.ch/cvmfs/lhcb.cern.ch",
"public_key": "/cvmfs/cvmfs-config.cern.ch/etc/cvmfs/keys/<key.pub>",
},
"lhcb-condb.cern.ch":
{
"url": "http://cvmfs-stratum-one.cern.ch/cvmfs/lhcb-condb.cern.ch",
"public_key": "/cvmfs/cvmfs-config.cern.ch/etc/cvmfs/keys/<key.pub>",
}
},
"steps":
{
"trace":
{
"apps_dir": "inputs_trace",
"path_list": "namelist1.txt"
},
"build":
{
"path_list": ["namelist1.txt", "namelist2.txt"],
},
"test":
{
"apps_dir": "inputs_test",
},
"commons":
{
"subset_path": "/path/to/subcvmfs"
}
},
"tools":
{
"parrot":
{
"http_proxy": "DIRECT"
},
"singularity":
{
"name": "/cvmfs/cernvm-prod/cvm4"
}
}
}
Arguments are entirely provided by the configuration, there is no need to provide them individually to the CLI unless overloading them is necessary. A few details here:
/tools/singularity/name
is used as thecontainer
option in the command./steps/commons
gathers parameters that are common to all the steps.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for subcvmfs_builder-1.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3dab2c64eef4b6583507869e3adb728f7e3ab8f13d6cf5b605c2e1cfdd3501d3 |
|
MD5 | 42a0b15fe938e58449f353cf108335be |
|
BLAKE2b-256 | 26f43fa553927cd5f5924362e75cb1de9de2522698f1c405e098eaed6a45a62b |