stackyter 0.11

Local display of a jupyter notebook running at CC-IN2P3

Introduction

This script will allow you to run a jupyter notebook (or lab) on a distant server (initialy at CC-IN2P3) while displaying it localy in your local brower. It was initialy intended to help LSST members to interact with the datasets already available at CC-IN2P3 using Python (in a stack or a DESC environment), but can be use for other purposes that need a Jupyter environment. It can be used in the following mode:

• For LSST/DESC members:

  • LSST stack environment using the --vstack option.

  • DESC catalogs environment using the --desc option

  • Your personal setup using the --mysetup option

• For all users:

  • With your personal setup using the --mysetup option

  • On any host (not only CC-IN2P3) using the --host AND the --mysetup options together.

Caveat: Jupyter must be available on the distant host for this

script to work.

Installation

Latest stable version can be installed with pip:

pip install stackyter

To upgrade to a newer version:

pip install --upgrade stackyter

To install in a local directory:

pip install --user stackyter            #  in your home directory
pip install --prefix mypath stackyter   #  in 'mypath'

Usage

stackyter.py [options]

Options

The configuration file can contain any (or all) options available through command line. An example of such a file can be found here.

Optional arguments are:

-h, --help           show this help message and exit
--config CONFIG      Configuration file containing a set of option values.
                     The content of this file will be overwritten by any
                     given command line options. (default: None)
--username USERNAME  Your CC-IN2P3 user name. If not given, ssh will try to
                     figure it out from you ~/.ssh/config or will use your
                     local user name. (default: None)
--workdir WORKDIR    Your working directory at CC-IN2P3 (default:
                     /pbs/throng/lsst/users/<username>/notebooks)
--vstack VSTACK      Version of the stack you want to set up. (E.g. v14.0,
                     w_2017_43 or w_2017_43_py2) (default: v14.0)
--desc               Setup a DESC environment giving you access to DESC
                     catalogs ('proto-dc2_v2.0' is for now the only
                     available catalog). This option overwrites the '--
                     vstack' and '--mysetup' options. (default: False)
--packages PACKAGES  A list of packages you want to setup. Coma separated
                     from command line, or a list in the config file. You
                     can use the `lsst_distrib` package to set up all
                     available packages from a given distrib. (default:
                     lsst_distrib)
--jupyter JUPYTER    Either launch a jupiter notebook or a jupyter lab.
                     (default: notebook)
--host HOST          Name of the target host. This option may allow you to
                     avoid potential conflit with the definition of the same
                     host in your $HOME/.ssh/config, or to connect to an
                     other host than the CC-IN2P3 ones (Jupyter must also be
                     available on these hosts). Default if to connect to CC-
                     IN2P3. (default: cca7.in2p3.fr)
--libs LIBS          Path(s) to local Python librairies. Will be added to
                     your PYTHONPATH. Coma separated to add more than one
                     paths, or a list in the config file. A default path for
                     jupyter will be choose if not given. (default: None)
--bins BINS          Path(s) to local binaries. Will be added to your PATH.
                     Coma separated to add more than one paths, or a list in
                     the config file. A default path for jupyter will be
                     choose if not given. (default: None)
--labpath LABPATH    You must provide the path in which jupyterlab has been
                     installed in case it differs from the (first) path you
                     gave to the --libs option. A default path for
                     jupyterlab will be choose if not given. (default: None)
--mysetup MYSETUP    Path to a setup file (at CC-IN2P3) that will be used to
                     set up the working environment. Be sure that a Python
                     installation with Jupyter (and jupyterlab) is available
                     to make this work. The LSST stack won't be set up in
                     this mode. 'vstack', 'libs', 'bins' and 'labpath'
                     options will be ignored. (default: None)

Distant host configuration

The --host option allows you to connect to any distant host. The default option used to create the ssh tunnel are -X -Y -tt -L. If you want to configure your ssh connection, edit your ~/.ssh/config file using, for instance, the following template:

Host ccjupyter
Hostname cca7.in2p3.fr
User lsstuser
GSSAPIClientIdentity [e-mail]
GSSAPIAuthentication yes
GSSAPIDelegateCredentials yes
GSSAPITrustDns yes

You can then use the stackyter script as follows:

stackyter.py --host ccjupyter

Or put the value for that option (along with others) in your config.yaml file.

LSST environment

Version of the LSST stack

All available versions of the LSST stack at CC-IN2P3 can be found under:

/sps/lsst/software/lsst_distrib/

These versions (and all the others) have been built under CentOS7, and must be used under a compatible system (CentOS7 or Ubuntu). To connect to a CentOS7 machine on CC-IN2P3, use --host cca7.in2p3.fr instead of --host ccage.in2p3.fr (cca7 is the default value of this script).

Python 2 (2.7) and 3 (>3.4) are available for almost all weeklies, with the following nomencalture:

Version	< w_2017_27	w_2017_27
Python 2	w_2017_XX	w_2017_XX_py2
Python 3	w_2017_XX_py3	w_2017_XX

Latest releases of the LSST stack, as of 11-07-2017, are:

Version	Comment
v14.0	Current stable version of the stack (Python 3 only)
w_2017_43_py2	Latest weekly release for Python 2
w_2017_44	Latest weekly release for Python 3

Keep in mind that using Python 2 in an LSST context is not encouraged by the community, and will not be supported anymore. The latest weekly for which Python 2 has been installed at CC-IN2P3 is w_2017_4 (see online documentation).

Note: Since version w_2017_40, the ipython module is included in the stack installation at CC-IN2P3 as an add-on. This module is not part of the officiel LSST distribution and will not be set up with the lsst_distrib package.

Use the LSST stack

Many examples on how to use the LSST stack and how to work with its outputs are presented there.

A few data sets have already been re-processed using the LSST stack, and their outputs are available for analysis at different places on CC-IN2P3:

• SXDS data from HSC: /sps/lsst/dev/lsstprod/hsc/SXDS/output

• CFHT data (containing clusters): /sps/lsst/data/clusters

• CFHT D3 fieald: /sps/lsst/data/CFHT/D3

Additional features

• ds9 is automatically available since version 0.9, and can be called in a Jupyter terminal.

DESC environment

You can automatically set up a DESC environment that will give you access to DESC catalogs such as the lattest proto-dc2_v2.0. A test notebook is available on this github page. Download and test it to make sure that everything is working properly. In this environment, the following ressources are available:

• A miniconda3 install with Jupyter (notebook and lab) and Ipython;

• The GRC (Generic Catalog Reader) and grc-catalogs packages, allowing you to easily load and read the DESC catalogs;

• The following DESC catalogs (more info can be found on the grc-catalogs web page):

  • proto-dc2_v2.0

• You can also use the --libs or --bins options to complete this set up with your personnal libraries (Python 3 only for now).

Personal environment

As stated in the introduction, and instead of seting up the LSST working environment, you can set up your personal working environment by using the --mysetup option. Given a setup file located at CC-IN2P3, you can simply do:

stackyter.py --username myusername --mysetup /sps/.../mysetup.sh

Your local setup file will be sourced at connection as followed:

source /sps/.../mysetup.sh

Your setup file must at least contains what is needed to make Jupyter available. In this mode, the LSST stack will not be setup.

Questions?

• If you have any comments or suggestions, or if you find a bug, please use the dedicated github issue tracker for this page.

• Why stakyter? For historical reason: stackyter = LSST stack + Jupyter. It was initially intended for LSST members to easily use the LSST software stack and interact with the data sets.