Skip to main content

Help to generate subsets of CVMFS

Project description

SubCVMFS-builder

schema

Purpose

SubCVMFS-builder aims to help scientific communities to build a subset of CVMFS containing the minimal requirements to execute applications of interests and deploy it on a remote computing infrastructure. The tool is mainly used on supercomputers having no outbound connectivity.

Installation

  • Before starting, you need to have singularity, CVMFS and CVMFS-shrinkwrap installed on the machine.
git clone https://gitlab.cern.ch/alboyer/subcvmfs-builder.git
cd subcvmfs-builder
pip install . --user

Usage

  • Get the help:
subcvmfs --help
  • Trace: run applications in a singularity container and return the list of dependencies related to CVMFS.

    subcvmfs trace <config> \
      	--apps-dir <required_input_directory>
      	--path-list <output_file>
      	--container <optional_input_string>
      	--number-core <optional_input_int>
    

    Trace is generally used when one has no idea about the dependencies of the applications to include in the subset of CVMFS.

    • apps-dir should be a directory of applications structured as follows: <apps_dir>/<command>/command.sh.
    • container provides a specific environment to run the applications (generally the one present in the remote computing infrastructure).
    • path-list should contain the result of the command, namely a list of CVMFS dependencies related to the applications to trace.
    • number-core can be used to run multiple applications in parallel.
  • Build: with a list of dependencies, build a subset of CVMFS using the cvmfs-shrinkwrap tool.

    subcvmfs build <config> \
      	--path-list <Trace_output> <another_input_list>
      	--subset-path <output_subset_path>
    

    Build is a required step to build a subset of CVMFS.

    • path-list should be a file - or a list of files - containing the CVMFS dependencies to include in the subset. It can be obtain from Trace or built from scratch.
    • subset-path should the path of the generated subset.
  • Test: execute application using a subset of dependencies instead of CVMFS.

    subcvmfs test <config> \
      	--apps-dir <required_input_directory>
      	--subset-path <required_input_path>
      	--container <optional_input_str>
      	--number-core <optional_input_int>
    

    Test is generally used before deploying a subset of CVMFS. One may want to make sure the subset is valid against a certain set of applications.

    • apps-dir should be a directory of applications structured as follows: <apps_dir>/<command>/command.sh.
    • subset-path should the path of a subset of CVMFS.
    • container provides a specific environment to run the applications (generally the one present in the remote computing infrastructure).
    • number-core can be used to run multiple applications in parallel.
  • Deploy: Deploy the subset of CVMFS - within a container if needed - on the remote computing infrastructure.

    subcvmfs deploy <config> \
      	--subset-path <required_input_subset_path>
      	--remote-location <required_input_str>
      	--container <optional_input_path>
      	--bootstrap <optional_input_path>
      	--post-command <optional_input_path>
      	--bundle-path <optional_output_path>
    

    Deploy requires one to have access to the remote computing infrastructure.

    • subset-path should the path of a subset of CVMFS.
    • remote-location the remote computing infrastructure that will host the subset of CVMFS.
    • bundle-path should be used if one wants to merge a subset of CVMFS with a container (not recommended if the subset of CVMFS is updated periodically).
    • container provides a specific environment to run the applications. If provided, it will be merged with the subset of CVMFS. Mandatory if bundle-path is present.
    • bootstrap the type of container. Mandatory if bundle-path is provided.
    • post-command should represent a command to run before launching the container. Mandatory if bundle-path is present.

Configuration

  • Minimal configuration possible:
{
	"cvmfs_extensions":
	{
		"<repository_name>":
		{
			"url": "<repository_url>",
			"public_key": "<repository_pubkey>",
		},
	}
}

In this case, arguments should be passed through the CLI.

  • A concrete example:
{
	"cvmfs_extensions":
	{
		"lhcb.cern.ch":
		{
			"url": "http://cvmfs-stratum-one.cern.ch/cvmfs/lhcb.cern.ch",
			"public_key": "/cvmfs/cvmfs-config.cern.ch/etc/cvmfs/keys/<key.pub>",
		},
		"lhcb-condb.cern.ch":
		{
			"url": "http://cvmfs-stratum-one.cern.ch/cvmfs/lhcb-condb.cern.ch",
			"public_key": "/cvmfs/cvmfs-config.cern.ch/etc/cvmfs/keys/<key.pub>",
		}
	},
	"steps":
	{
		"trace":
		{
			"apps_dir": "inputs_trace",
			"path_list": "namelist1.txt"
		},
		"build":
		{
			"path_list": ["namelist1.txt", "namelist2.txt"],
		},
		"test":
		{
			"apps_dir": "inputs_test",
		},
		"commons":
		{
			"subset_path": "/path/to/subcvmfs"
		}
	},
	"tools":
	{
		"parrot":
		{
			"http_proxy": "DIRECT"
		},
		"singularity":
		{
			"name": "/cvmfs/cernvm-prod/cvm4"
		}
	}
}

Arguments are entirely provided by the configuration, there is no need to provide them individually to the CLI unless overloading them is necessary. A few details here:

  • /tools/singularity/name is used as the container option in the command.
  • /steps/commons gathers parameters that are common to all the steps.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

subcvmfs-builder-1.0.1.tar.gz (99.5 kB view hashes)

Uploaded Source

Built Distribution

subcvmfs_builder-1.0.1-py3-none-any.whl (16.8 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page