(Python) Process and File Controller
Table of Contents
This repository provides pfcon – a controlling service that speaks to remote pman and pfioh services.
Most simply, pfcon pushes local data to a remote location (by talking to a remote pfioh service), runs some process on this data in the remote space using pman, and then copies the resultant data back to a local target space.
It can be used to query and control the following (for example):
- state: Is job <XYZ> still running?
- result: What is the stdout (or stderr) from job <XYZ>?
- control: Kill job <XYZ>
Installation is relatively straightforward, and we recommend using either python virtual environments or docker.
On Ubuntu, install the Python virtual environment creator
sudo apt install virtualenv
Then, create a directory for your virtual environments e.g.:
You might want to add to your .bashrc file these two lines:
export WORKON_HOME=~/python-envs source /usr/local/bin/virtualenvwrapper.sh
Then you can source your .bashrc and create a new Python3 virtual environment:
source .bashrc mkvirtualenv --python=python3 python_env
To activate or “enter” the virtual env:
To deactivate virtual env:
We provide a slim docker image with python3 based off Ubuntu. If you want to play inside this dock and install pman manually, do
docker pull fnndsc/ubuntu-python3
This docker has an entry point python3. To enter the dock at a different entry and install your own stuff:
docker run -ti --entrypoint /bin/bash fnndsc/ubuntu-python3
Now, install pman and friends using pip
apt update && \ apt install -y libssl-dev libcurl4-openssl-dev librtmp-dev && \ pip install pfcon
If you do the above, remember to commit your changes to the docker image otherwise they’ll be lost when you remove the dock instance!
docker commit <container-ID> local/ubuntu-python3-pfcon where ``<container-ID>`` is the ID of the above container.
The easiest option however, is to just use the fnndsc/pfcon dock.
docker pull fnndsc/pfcon
and then run
docker run --name pfcon -v /home:/Users --rm -ti fnndsc/pfcon --forever --httpResponse