jupyter-openbis-server 0.4.2

Server Extension for Jupyter notebooks to connect to openBIS and download/upload datasets, inluding the notebook itself

Jupyter openBIS Server

This server is an extension to the Jupyter notebook server and is part of the jupyter-openbis-extension and jupyterlab-openbis notebook extensions. It uses the pyBIS module internally to communicate with openBIS and ommunicates with the notebook extensions via the built-in tornado webserver.

This extension has been successfully tested with Safari 12.0.3, Chrome 72.0 and Firefox 66.0. There is a known incompatibility before Firefox 61.0b13 with Tornado > 6.x (the webserver used by Jupyter). If you encounter such incompatibilities, try to downgrade to Tornado 5.1.1. However, Tornado 5.1.1 will not work with Jupyter Lab 3.

Install the server extension

The server extension will be automatically installed when you install the Jupyter Notebook Extension (the «classic» Jupyter Notebook):

$ pip install --upgrade jupyter-openbis-extension

If you need to install or upgrade the server extension alone, you can do so by:

$ pip install --upgrade jupyter-openbis-server

Make sure your extension is recognised by Jupyter and enabled (your output may vary, but ensure you see the line: jupyter-openbis-server.server OK)

$ jupyter serverextension list
config dir: /Users/your_username/.pyenv/versions/3.6.9/etc/jupyter
    jupyter-openbis-server.server  enabled
    - Validating...
      jupyter-openbis-server.server OK

Create a connection configuration file

When the module is installed, you can create a connection configuration file.

$ jupyter-openbis-conn --help
Usage: jupyter-openbis [OPTIONS]

  Generate an openBIS connection file for use in Jupyter notebooks.

Options:
  -n, --name TEXT
  -h, --hostname TEXT
  --verfiy / --no-verify
  --https / --no-https
  -u, --username TEXT
  -p, --password TEXT
  -d, --destination [/Users/your_username/.jupyter|/Users/your_username/.pyenv/versions/3.6.9/etc/jupyter|/usr/local/etc/jupyter|/etc/your_username]
                                  [default: /Users/your_username/.jupyter]
  --help                          Show this message and exit.

You can start the utility as-is to get prompted for every parameter. Username and password are optional.

$ jupyter-openbis-conn

install Jupyter extension manually

In most cases, a simple pip install --upgrade jupyter-openbis-server will install the server extension. However, in some cases (e.g. when installing via pip install -e .) you need to issue the following command to register the extension:

In the library path, e.g. /etc/jupyter/

$ jupyter serverextension enable --py jupyter-openbis-server --sys-prefix

This will create a file ~/.jupyter/jupyter_notebook_config.json with the following content:

{
  "NotebookApp": {
    "nbserver_extensions": {
      "jupyter-openbis-server.server": true
    }
  }
}

Uninstall Jupyter openBIS Server

Unfortunately, pip doesn't automatically clean up the Jupyter configuration when uninstalling. You have to do it yourself:

$ jupyter serverextension disable --py jupyter-openbis-server
$ pip uninstall jupyter-openbis-server

Server extension API documentation

XSRF Token in POST, PUT and DELETE requests

XSRF (or CSRF) stands for Cross-Site-Request-Forgery.

For all POST, PUT and DELETE requests, the following http headers must be submitted as http headers:

"X-XSRFToken": xsrf_token,
"credentials": "same-origin"

The value of the xsrf_token is the value of the _xsrf cookie which is stored in the users' browser. Without this http header information, the request will fail. All GET requests can be established without a special header.

The underlying Tornado-Webserver which handles all requests to the Jupyter serverextension will throw an error if the X-XSRF Token is not present.

Errors

Errors caused by a POST, PUT and DELETE request will result in a HTTP Status > 300 and an error message:

{
	"reason": "Incorrect username or password for openBIS instance"
}

get openBIS connections

GET /openbis/conns

Returns an array of JSON objects:

{
  "status": 200,
  "connections": [
    {
      "name": "openBIS instance",
      "url": "https://openbis.instance.ch",
      "status": "connected",
      "username": "user_name",
      "password": "******",
      "isMounted": false,
      "mountpoint": ""
    }
  ],
  "notebook_dir": "/home/user_name/project_dir"
}

• the name is the name of the connection being used when downloading or uploading dataSets (see below)
• the url of the openBIS instance
• the values of status can be either connected or not connected
• the username being used in openBIS
• the password really only consists of a number of asteriks *. If they are passed as such to re-connect to openBIS, the server tries to use the internally saved password instead. The password only lives in memory of the singleuser notebook-server and is not saved persistently.
• isMounted is either true or false, depending whether there is a current FUSE/SSHFS mountpoint available which connects to the openBIS dataStore
• mountpoint is the path to the mounted openBIS dataStore. It defaults to $HOME/<openbis hostname>

login to an openBIS connection

An openBIS connection that has to be established or has timed out: a new login has to take place.

PUT /openbis/conn

Body:

{
    "username": username,
    "password": password,
    "action": "login",
}

The action attribute defaults to login. Returns:

{
    "status": 200,
    "connection": {
        "name": "openBIS instance",
        "url": "https://openbis.instance.ch",
        "status": "connected",
        "username": "some_username",
        "password": "******",
        "isMounted": false,
        "mountpoint": ""
    }
}

logout

Logs out from an openBIS instance, i.e. the token is invalidated. The mount might still persist, as it is a separate connection. The status changes from connected to not connected

PUT /openbis/conn

Body:

{
    "action": "logout",
}

Returns:

{
    "status": 200,
    "connection": {
        "name": "openBIS instance",
        "url": "https://openbis.instance.ch",
        "status": "not connected",
        "username": "some_username",
        "password": "******",
        "isMounted": true,
        "mountpoint": "/Users/some_username/openbis.instance.ch"
    }
}

Mount to an openBIS dataStore

Prerequisites

On the Jupyter Server, FUSE/SSHFS must be installed beforehand (requires root privileges). For the actual mount to the openBIS dataStore, no special privileges are required.

For Mac OS X, follow the installation instructions on https://osxfuse.github.io

For Unix Cent OS 7, do the following:

$ sudo yum install epel-release
$ sudo yum --enablerepo=epel -y install fuse-sshfs
$ user="$(whoami)"
$ usermod -a -G fuse "$user"

Windows is currently not supported, sorry!

By default, the mountpoint is the same as the hostname of the instance and it is located inside the home of the user. FUSE/SSHFS needs an empty directory to do this, so it will automatically be created.

PUT /openbis/conn

Body:

{
    "username": username,
    "password": password,
    "action"  : "mount"
}

Returns:

{
    "status": 200,
    "connection": {
        "name": "openBIS instance",
        "url": "https://openbis.instance.ch",
        "status": "connected",
        "username": "some_username",
        "password": "******",
        "isMounted": true,
        "mountpoint": "/Users/some_username/openbis.instance.ch"
    }
}

Unmount from openBIS dataStore

PUT /openbis/conn

Body:

{
    "action"  : "mount"
}

Returns:

{
    "status": 200,
    "connection": {
        "name": "openBIS instance",
        "url": "https://openbis.instance.ch",
        "status": "connected",
        "username": "some_username",
        "password": "******",
        "isMounted": false,
        "mountpoint": ""
    }
}

Register a new openBIS connection

For the lifetime (runtime) of the Jupyter server, this will create a connection to openBIS.

POST /openbis/conns

Body:

{
    "name": connection_name,
    "url": connection_url,
    "username": username,
    "password": password
}

Unregister/delete a new openBIS connection

For the lifetime (runtime) of the Jupyter server, this will drop an existing openBIS connection:

DELETE /openbis/conn/<connection name>

Upload a dataSet

POST /openbis/dataset/<connection_name>/<permId>/<downloadPath>

Download a dataSet

GET /openbis/dataset/<connection_name>/<permId>/<downloadPath>

• the connection_name is the name of the connection given in the connections dialog.
• the permId is the identifer of the dataSet that needs to be downloaded.
• the downloadPath is the absolute path on the host system where the dataSet files should be downloaded to. The downloadPath must be URL-encoded to not to be confused with the URL itself.

In case of a successful download, the API returns a JSON like this

{
    'url'       : conn.url,
    'permId'    : dataset.permId,
    'path'      : path,
    'dataStore' : dataset.dataStore,
    'location'  : dataset.physicalData.location,
    'size'      : dataset.physicalData.size,
    'files'     : dataset.file_list,
    'statusText': 'Data for DataSet {} was successfully downloaded to: {}'.format(dataset.permId, path)
}

In case of an error, the API returns one of these errors (HTTP Status > 200):

general connection error

HTTP-Status: 500
{
	"reason": 'connection to {} could not be established: {}'.format(conn.name, exc)
}

dataSet not found error

HTTP-Status: 404
{
	"reason": 'No such dataSet found: {}'.format(permId)
}

dataSet download error

HTTP-Status: 500
{
	"reason": 'Data for DataSet {} could not be downloaded: {}'.format(permId, exc)
}

Save requirements.txt and runtime.txt file

Note: The requirements list and the runtime must be evaluated by executing actual Python or R code from wtihin a notebook cell. The Python used by the Jupyter server might differ from the Python used by the kernel. The usual pip freeze doesn't work, as we cannot access the pip CLI from within Python.

For the Python requirements.txt we use this script:

import pkg_resources
print(
	"\n".join(
		["{}=={}".format(i.key, i.version) for i in pkg_resources.working_set]
	)
)

For the Python runtime.txt:

import sys
print('python-' + str(sys.version_info[0]) + '.' + str(sys.version_info[1]))

Once submitted to the server, the server will join the relative notebook_path (from the UI) with the server-side notebook_dir. These files will be stored in the same location on the filesystem as the notebook itself.

POST /openbis/requirements

Body:

{
    "notebook_path": notebook_path,
    "requirements_list": state.requirements_list,
    "requirements_filename": state.requirements_filename,
    "runtime": state.runtime,
    "runtime_filename": state.runtime_filename
}