python-pfconclient 4.0.0

Install latest Docker

Currently tested platforms:

• Ubuntu 20.04+ and MAC OS X 12+

Note: On a Linux machine make sure to add your computer user to the docker group. Consult this page https://docs.docker.com/engine/install/linux-postinstall/

Fire up the full set of pfcon services

Open a terminal and run the following commands in any working directory:

$> git clone https://github.com/FNNDSC/pfcon.git
$> cd pfcon
$> ./make.sh -N -F fslink

You can later remove all the backend containers with:

$> cd pfcon
$> ./unmake.sh -N -F fslink

Python programmatic interface

Instantiate the client:

from pfconclient.client import Client, JobType

token = Client.get_auth_token('http://localhost:30006/api/v1/auth-token/', 'pfcon', 'pfcon1234')
cl = Client('http://localhost:30006/api/v1/', token)

job_id = 'chris-jid-2'

# Step 1: Submit copy job
copy_descriptors = {
    'input_dirs': ['home/user/cube'],
    'output_dir': 'home/user/cube_out',
}
cl.submit_job(JobType.COPY, job_id, copy_descriptors)

# Step 2: Poll copy status until finished
cl.poll_job_status(JobType.COPY, job_id)

# Step 3: Submit plugin job
plugin_descriptors = {
    'entrypoint': ['python3', '/usr/local/bin/simpledsapp'],
    'args': ['--prefix', 'le'],
    'auid': 'cube',
    'number_of_workers': 1,
    'cpu_limit': 1000,
    'memory_limit': 200,
    'gpu_limit': 0,
    'image': 'fnndsc/pl-simpledsapp',
    'type': 'ds',
    'input_dirs': ['home/user/cube'],
    'output_dir': 'home/user/cube_out',
}
cl.submit_job(JobType.PLUGIN, job_id, plugin_descriptors)

# Step 4: Poll plugin status until finished
cl.poll_job_status(JobType.PLUGIN, job_id)

# Step 5: Get output file metadata
resp = cl.get_plugin_job_json_data(job_id, 'home/user/cube_out')
print(resp['rel_file_paths'])

# Step 6: Submit upload job (no-op for fslink)
upload_descriptors = {
    'job_output_path': 'home/user/cube_out',
}
cl.submit_job(JobType.UPLOAD, job_id, upload_descriptors)

# Step 7: Submit delete job and poll until finished
cl.submit_job(JobType.DELETE, job_id, {})
cl.poll_job_status(JobType.DELETE, job_id)

# Step 8: Remove all containers
cl.delete_job(JobType.COPY, job_id)
cl.delete_job(JobType.PLUGIN, job_id)
cl.delete_job(JobType.DELETE, job_id)

Standalone CLI client tool

Get and print auth token with the auth subcommand:

$> pfconclient http://localhost:30006/api/v1/ auth --pfcon_user pfcon --pfcon_password pfcon1234

Submit a copy job:

$> pfconclient http://localhost:30006/api/v1/ -a <token> submit --job_type copy --jid chris-jid-2 --input_dirs home/user/cube --output_dir home/user/cube_out

Poll copy job status:

$> pfconclient http://localhost:30006/api/v1/ -a <token> poll --job_type copy --jid chris-jid-2

Submit a ds plugin job:

$> pfconclient http://localhost:30006/api/v1/ -a <token> submit --job_type plugin --jid chris-jid-2 --entrypoint python3 /usr/local/bin/simpledsapp --args '--prefix le' --auid cube --number_of_workers 1 --cpu_limit 1000 --memory_limit 200 --gpu_limit 0 --image fnndsc/pl-simpledsapp --type ds --input_dirs home/user/cube --output_dir home/user/cube_out

Poll plugin job status:

$> pfconclient http://localhost:30006/api/v1/ -a <token> poll --job_type plugin --jid chris-jid-2

Get plugin job status:

$> pfconclient http://localhost:30006/api/v1/ -a <token> status --job_type plugin --jid chris-jid-2

Delete a copy job’s container:

$> pfconclient http://localhost:30006/api/v1/ -a <token> delete --job_type copy --jid chris-jid-2

Delete a plugin job’s container:

$> pfconclient http://localhost:30006/api/v1/ -a <token> delete --job_type plugin --jid chris-jid-2

Submit a delete job:

$> pfconclient http://localhost:30006/api/v1/ -a <token> submit --job_type delete --jid chris-jid-2

Poll delete job status:

$> pfconclient http://localhost:30006/api/v1/ -a <token> poll --job_type delete --jid chris-jid-2

Delete a delete job’s container:

$> pfconclient http://localhost:30006/api/v1/ -a <token> delete --job_type delete --jid chris-jid-2

Visit the standalone CLI client wiki page to learn more about the CLI client.

Optionally setup a virtual environment

Install virtualenv and virtualenvwrapper using your OS package manager.

Create a directory for your virtual environments e.g.:

$> mkdir ~/Python_Envs

You might want to add the following lines to your .bashrc or .zshrc file:

VIRTUALENVWRAPPER_PYTHON=/usr/local/bin/python3
export WORKON_HOME=~/Python_Envs
source /usr/local/bin/virtualenvwrapper.sh

Then source the file and create a new Python3 virtual environment:

$> mkvirtualenv pfcon_client_env

To activate pfcon_client_env:

$> workon pfcon_client_env

To deactivate pfcon_client_env:

$> deactivate

Clone the repo

$> git clone https://github.com/FNNDSC/python-pfconclient.git

Run automated tests

$> cd python-pfconclient
$> workon pfcon_client_env
$> pip install -e ".[dev]"
$> pytest