pfstorage 2.0.0.2

object storage interface

Table of Contents

• Overview

  • pfstorage

• Installation

  • Python Virtual Environment

  • Using the fnndsc/pfstorage docker container

• Usage

  • Command line arguments

• EXAMPLES

  • script mode

  • server mode

Overview

This repository provides pfstorage – a library / module that speaks to an object storage backend (such as swift) and also provides logic for handling input/output data locations for the ChRIS system.

pfstorage

Most simply, pfstorage is a module that offers a regularized interface to some other backend object storage. While currently supporting swift, the long term idea is to support a multitude of backends. By providing its own interface to several storage backends, this module removes the need for client code to change when a different object storage backend is used.

While at its core a module/library, pfstorage also provides two modes of stand-alone access: (1) a command line script interface mode to the library, and (2) a persistent http server mode. In the command line mode, the main module functions are exposed to appropriate CLI. In the http server mode, a client can use curl-type http calls to call the underlying library functions.

Installation

Installation is relatively straightforward, and we recommend using either python virtual environments or docker.

Python Virtual Environment

On Ubuntu, install the Python virtual environment creator

sudo apt install virtualenv

Then, create a directory for your virtual environments e.g.:

mkdir ~/python-envs

You might want to add to your .bashrc file these two lines:

export WORKON_HOME=~/python-envs
source /usr/local/bin/virtualenvwrapper.sh

Note that depending on distro, the virtualenvwrapper.sh path might be

/usr/share/virtualenvwrapper/virtualenvwrapper.sh

Subsequently, you can source your .bashrc and create a new Python3 virtual environment:

source .bashrc
mkvirtualenv --python=python3 python_env

To activate or “enter” the virtual env:

workon python_env

To deactivate virtual env:

deactivate

Using the fnndsc/pfstorage docker container

The easiest option however, is to just use the fnndsc/pfstorage dock.

docker pull fnndsc/pfstorage

and then run (for example in http server mode access to the library):

docker run --name pfstorage -v /home:/Users --rm -ti \
       fnndsc/pfstorage \
       --ipSwift localhost \
       --portSwift 8080 \
       --forever \
       --httpResponse \
       --server

or in CLI mode:

docker run --name pfstorage -v /home:/Users --rm -ti \
       fnndsc/pfstorage \
       --ipSwift localhost \
       --portSwift 8080 \
       --msg '
        { "action": "ls",
          "meta": {
                        "path":         "",
                        "retSpec":      ["name", "bytes"]
                  }
        }'

Usage

For usage of pfstorage, consult the relevant wiki pages <https://github.com/FNNDSC/pfstorage/wiki/pfcon-overview>`.

Command line arguments

--msg '<JSON_formatted>'
The action to perform. This can be one of:

    * objPull -- pull data from storage to file system
    * objPush -- push data from file system to storage
    * ls      -- listing of data within storage

with a JSON formatted string similar to:

    * ls:
    { "action": "ls",
      "meta": {
                    "path":         "",
                    "retSpec":      ["name", "bytes"]
              }
    }

    * objPut:
    {  "action": "objPut",
        "meta": {
                    "putSpec":              "./data",
                    "inLocation":           "storage",
                    "mapLocationOver":      "./data"
                }
    }

    * objPull:
    {  "action": "objPull",
        "meta": {
                    "path":                 "chris",
                    "substr":               "/018",
                    "fromLocation":         "chris/uploads/DICOM",
                    "mapLocationOver":      "./data"
                }
    }

[--type <storageBackendType>]
The type of object storage. Currently this is 'swift'.

[--ipSwift <swiftIP>]
The IP interface of the object storage service. Default %s.

[--portSwift <swiftPort>]
The port of the object storage service. Defaults to '8080'.

[--ipSelf <selfIP>]
The IP interface of the pfstorage service for server mode. Default %s.

[--portSelf <selfPort>]
The port of the pfstorage service for server mode. Defaults to '4055'.

[--httpResponse]
In servier mode, send return strings as HTTP formatted replies
with content-type html.

[--configFileLoad <file>]
Load configuration information from the JSON formatted <file>.

[--configFileSave <file>]
Save configuration information to the JSON formatted <file>.

[-x|--desc]
Provide an overview help page.

[-y|--synopsis]
Provide a synopsis help summary.

[--version]
Print internal version number and exit.

[--debugToDir <dir>]
A directory to contain various debugging output -- these are typically
JSON object strings capturing internal state. If empty string (default)
then no debugging outputs are captured/generated. If specified, then
``pfcon`` will check for dir existence and attempt to create if
needed.

[-v|--verbosity <level>]
Set the verbosity level. "0" typically means no/minimal output. Allows for
more fine tuned output control as opposed to '--quiet' that effectively
silences everything.

EXAMPLES

script mode

pfstorage                                               \
    --ipSwift localhost                                 \
    --portSwift 8080                                    \
    --verbosity 1                                       \
    --debugToDir /tmp                                   \
    --type swift                                        \
    --msg '
    {
        "action":   "ls",
        "meta": {
            "path":         "",
            "retSpec":      ["name", "bytes"]
        }
    }
    '

server mode

start server:

pfstorage                                               \
    --ipSwift localhost                                 \
    --portSwift 8080                                    \
    --ipSelf localhost                                  \
    --portSelf 4055                                     \
    --httpResponse                                      \
    --verbosity 1                                       \
    --debugToDir /tmp                                   \
    --type swift                                        \
    --server                                            \
    --forever

query server:

pfurl --verb POST --raw                                 \
      --http localhost:4055/api/v1/cmd                  \
      --httpResponseBodyParse                           \
      --jsonwrapper 'payload'                           \
      --msg '
            {
                "action":   "ls",
                "meta": {
                    "path":         "",
                    "retSpec":      ["name", "bytes"]
                }
            }
    '