caw 0.6.1a1

ChRIS Automatic Workflows

ChRIS Automated Workflows

A command-line client for ChRIS for pipeline execution and data mangement.

Installation

The easiest option is via pip.

Pip

pip install -U caw

# optional, for tab completion of subcommands
caw --install-completion
# optional, for secure password storage
pip install keyring

Alternatively, container images are also provided. See below.

Usage

caw [OPTIONS] COMMAND [ARGS]...

Container Usage

A Docker image is also provided. Podman and Docker work equally well.

docker run --rm --net=host -v $PWD/data:/data:ro -t fnndsc/caw:latest caw upload /data
podman run --rm --net=host -v $PWD/data:/data:ro -t fnndsc/caw:latest caw upload /data

Container isolation can make usage finicky. Volumes must be mounted for the container to read data which exists on the host filesystem. If the ChRIS backend is on a private network, the --net=host option might be necessary to resolve the server's hostname.

Alternatively, Singularity is much easier to use because of its weaker container isolation and $HOME is a bind path by default.

singularity exec docker://fnndsc/caw:latest caw upload ./data

Documentation

Details are provided by the --help commaand.

caw --help
caw search --help
caw pipeline --help
caw download --help
caw upload --help

Logging In

Multiple ways of providing your credentials are supported. The most secure way is to install the optional dependency keyring and run caw login.

# install optional dependency
pip install keyring
caw --address https://cube.chrisproject.org/api/v1/ --username 'a_crow' login

Alternatively, ChRIS user account credentials can be passed via command-line arguments or environment variables. It's safer to use environment variables (so that your password isn't saved to history) and also easier (no need to retype it out everytime).

# using cli arguments
caw --address https://cube.chrisproject.org/api/v1/ \
    --username 'a_crow'      \
    --password notchris1234  \
    search

# using environment variables
export CHRIS_URL=https://cube.chrisproject.org/api/v1/
export CHRIS_USERNAME=a_crow
export CHRIS_PASSWORD=notchris1234

caw search

Commands

• search: Search for pipelines that are saved in ChRIS.
• pipeline: Run a pipeline on an existing feed.
• upload: Upload files into ChRIS storage and run pl-dircopy.
• download: Download everything from a ChRIS url.

caw search

Search for pipelines that are saved in ChRIS.

Examples

# list all pipellines
$ caw search
https://cube.chrisproject.org/api/v1/pipelines/1/           Automatic Fetal Brain Reconstruction Pipeline
https://cube.chrisproject.org/api/v1/pipelines/2/           Infant FreeSurfer with Cerebellum Step
https://cube.chrisproject.org/api/v1/pipelines/2/           COVID-Net Chest CT Analysis and Report

# search for pipelines by name
$ caw search 'Fetal Brain'
https://cube.chrisproject.org/api/v1/pipelines/1/           Automatic Fetal Brain Reconstruction Pipeline

caw pipeline

Run a pipeline on an existing feed.

Examples

# specify source as a plugin instance ID
$ caw pipeline --target 3 'Automatic Fetal Brain Reconstruction Pipeline'

# specify source by URL
$ caw pipeline --target https://cube.chrisproject.org/api/v1/plugins/instances/3/ 'Automatic Fetal Brain Reconstruction Pipeline'

caw upload

Upload files into ChRIS storage and then run pl-dircopy, printing the URL for the newly created plugin instance.

Examples

# upload files and create a new feed by running pl-dircopy
$ caw upload something.txt picture.jpg

# upload a folder and create a new feed by running pl-dircopy
$ caw upload data/

# create a feed with a title and description
$ caw upload --name 'Caw caw, ima crow' --description 'A murder of crows' \
    something.txt picture.jpg

# create a feed and run a pipeline after the pl-dircopy instance
$ caw upload --name 'In-utero study' \
    --pipeline 'Automatic Fetal Brain Reconstruction Pipeline' \
    data/T2_*.nii

Piping

The commands caw upload and caw pipeline print out the URLs of the resources that they create. Advanced users might pipe the output of caw to other commands such as xh and jq.

caw pipeline prints out the plugin instances it creates.

caw upload prints out the feed it creates. Alternatively, the option caw upload --output plugininstances tells caw upload to print out the plugin instances it creates instead, similar to be behavior of caw pipeline.

caw download

Download files from ChRIS.

# download everything from a feed
$ caw download 'https://cube.chrisproject.org/api/v1/3/files/' results/

# download the output directory of a specific plugin instance
$ caw download 'https://cube.chrisproject.org/api/v1/plugins/instances/5/files/' results/

# download everything from a path 'chris/uploads/test'
$ caw download 'https://cube.chrisproject.org/api/v1/uploadedfiles/search/?fname=chris%2Fuploads%2Ftest' results/

# example results
$ tree results/
wow
└── uploads
    └── test
        ├── a.txt
        ├── b.txt
        ├── c.txt
        ├── d.txt
        └── e.txt

caw export

Export a registered pipeline to JSON.

caw export 'Automatic Fetal Brain Reconstruction Pipeline v1.0.0' > pipeline.json

curl -u "chris:chris1234" "https://example.com/api/v1/pipelines/" \
  -H 'Content-Type:application/vnd.collection+json' \
  -H 'Accept:application/vnd.collection+json' \
  --data "$(< pipeline.json)"

caw export Limitations

• All plugin parameters will be exported as part of plugin_parameter_defaults
• Order of lists, such as plugin_tree and plugin_parameter_defaults, may not be the same as the original

Development

python -m venv venv
source venv/bin/activate
pip install -e .

Testing

First, set up the ChRIS backend on http://localhost:8000/api/v1/ (say, using miniChRIS).

Next, install the example pipeline.

./examples/dummy_pipeline.sh
./examples/upload_reconstruction_pipeline.sh

Install testing dependencies:

pip install pytest pytest_mock

Run all tests using the command

pytest

The end-to-end test is disabled by default because it will create a ChRIS account and affect caw user settings. It is recommended to run it in a container instead.

docker build -t caw -f Dockerfile.dev .
docker run --rm --net=host --userns=host -v $PWD:/usr/local/src/caw:ro \
       -t -e CAW_TEST_FULL=y caw

Roadmap

For the next-generation ChRIS client, see chrs, and how it compares to caw: https://github.com/FNNDSC/chrs/wiki/Feature-Table