servicex_client

Python SDK and CLI Client for ServiceX

Configuration

The client relies on a YAML file to obtain the URLs of different servicex deployments, as well as tokens to authenticate with the service. The file should be named .servicex and the format of this file is as follows:

api_endpoints:
  - endpoint: http://localhost:5000
    name: localhost

  - endpoint: https://servicex-release-testing-4.servicex.ssl-hep.org
    name: testing4
    token: ...

default_endpoint: testing4

cache_path: /tmp/ServiceX_Client/cache-dir
shortened_downloaded_filename: true

The default_endpoint will be used if otherwise not specified. The cache database and downloaded files will be stored in the directory specified by cache_path.

The shortened_downloaded_filename property controls whether downloaded files will have their names shortened for convenience. Setting to false preserves the full filename from the dataset. `

The library will search for this file in the current working directory and then start looking in parent directories until a file is found.

Command Line Interface

When installed, the client provides a new command in your shell, servicex. This command uses a series of subcommands to work with various functions of serviceX.

Common command line arguments:

If neither url nor backend are specified then the client will attempt to use the default_endpoint value to determine who to talk to.

codegens

This command will list the code generators deployed.

transforms

These commands interact with transforms that have been run

list

List transforms associated with the current user. Add the --complete flag to only show transforms that have completed.

files

List the files along with their size generated by a transform. Specify the transform request id with the -t or --transform-id flag

download

Download the files from a transform to a local directory. Specify the transform request id with -t and the directory to download to with -d. Defaults to downloading files to the current working directory.

cache

These commands allow you to work with the query cache maintained by the serviceX client.

list

Show all of the cached transforms along with the run time, code generator, and number of resulting files

delete

Delete a specific transform from the cache. Provide the transform request ID with the -t or --transform-id arg.

clear

Clear all of the transforms from the cache. Add -y to force the operation without confirming with the console.

Python SDK

Entry to the SDK starts with constructing an instance of ServiceXClient. The constructor accepts backend argument to specify a named backend from the .servicex file, or url for the direct URL to a serviceX server. With the URL option you can't provide a token from .servicex so it must either be an unsecured endpoint, or the token must be provided via the WLCG standard of a file pointed to by BEARER_TOKEN_FILE environment variable.

With an instance of ServiceXClient you can

• List the code generators deployed with the ServiceX instance
• List the transformers that have been run
• Get the current status of a specific transform

Create a Dataset Instance to Run Transforms

The ServiceX client also can create a Dataset instance that allows you to specify a query, provide a dataset identifier, and retrieve the results of the resulting transform request.

There are two types of datasets

• func_adl_dataset
• Python Function dataset

Dataset Identifiers

Before we get too deeply into the dataset classes, we should look at how to specify a dataset.

• RucioDatasetIdentifier - for retrieving data files registered with Rucio
• FileListDataset - A list of URIs for accessing files using xRootd

FuncADL Dataset

This dataset is controlled by the func_adl language. The dataset supports the Select, SelectMany, Where, MetaData, and QMetaData operators from func_adl.

Datasets

This is the abstract class for requesting data from ServiceX. You have to specify the dataset identifier you want data from and provide some sort of selection query. You can set the result format with the set_result_format operator (it's also a factory method arg for the dataset).

Operators that cause the client to interact with the server: These terminal operators will call out to the serviceX server and process results. They are all implemented as asynchronous coroutines, but they also come with synchronous versions to make it easy to do easy things.