OverBoard

OverBoard is a lightweight yet powerful dashboard to monitor your experiments.

Load remote experiments in real-time through SSH/FTP and others.

Custom visualisations (tensors and any custom plot with familiar MatPlotLib syntax).

A sortable table of hyper-parameters with Python-syntax filtering.

Plot percentile intervals (shaded plots), custom X/Y axes, and hyper-parameters (bubble plots).

Installation

You can install the dependencies with:

• With Conda: conda install pyqt=5.12 pyqtgraph=0.11 -c conda-forge

• With pip: pip install pyqt5==5.12 pyqtgraph==0.11

Finally, OverBoard itself can be installed with: pip install fs==2.4 overboard (Conda seems to be too strict when installing PyFileSystem/fs, so pip should be preferred.)

Optional:

• pip install fs.sshfs to support remote files through SSH.

• PyOpenGL 3.1 (either through conda or pip) if you intend to use custom 3D plots with PyQtGraph.

• MatPlotLib if you intend to use it for custom plots.

Installation - logger only

Your scripts can log data without installing the full GUI and its dependencies (so your remote GPU cluster does not need PyQt at all).

Just use: pip install overboard_logger

And remember to import overboard_logger instead of overboard in your scripts.

Usage

• Main interface: python3 -m overboard <logs-directory>

• Logging experiments is simple:

from overboard import Logger

with Logger('./logs') as logger:
  for iteration in range(100):
    logger.append({'loss': 0, 'error': 0})

You can also pass in a meta keyword argument, which can be a dict with hyper-parameters names and values (or other meta-data), to help organize your experiments. These will be displayed in a handy table, which supports sorting and filtering. The meta data can also be an argparse.Namespace, which is useful if your hyper-parameters are command-line arguments parsed with argparse.

By default a unique folder (using the current timestamp) is created for the logs. For full documentation on initialization arguments and other methods, type pydoc overboard on the command-line (Python built-in doc viewer).

You can also check the examples directory:

• examples/basic.py: A minimal example. Generates some test logs.
• examples/mnist.py: The mandatory MNIST example. Also shows a custom MatPlotLib plot (a confusion matrix).
• examples/example_2d.py: Example custom 2D plot, with PyQtGraph (faster than MatPlotLib).
• examples/example_3d.py: Example custom 3D plot, with PyQtGraph. Requires PyOpenGL.

A note about importing: You can either import the Logger class from overboard or from overboard_logger. If you installed the "logger only" version as described above (no dependencies), then you can only import from overboard_logger.

Remote experiments

It's as simple as:

python -m overboard ssh://username:password@hostname/path-to-experiments

...replacing your SSH user name, password, host name/server, and the directory where OverBoard should look for experiments. The default port is 22, which you can override with :23 or another port number after the host name.

There are a number of other settings, which you can append after the path. For example, append ?keepalive=60&config_path=~/.ssh/config to set the keepalive packets interval to 60 seconds, and specify an OpenSSH configuration file path.

Other remote file systems are available by using appropriate prefixes (e.g. ftp://).

Interface tips & tricks

• Click a plot title to hide it (moving it to the bottom) or to show it again.

• Click a plot line to select that experiment in the table, and vice-versa.

• Use the mouse wheel to zoom and drag to pan in a plot or custom visualization.

• Click the "A" button in the bottom-left of a plot to reset the zoom/pan.

• Choose an hyper-parameter as the "X axis" in the sidebar to compare different hyper-parameters graphically (bubble plot). Click a bubble to highlight that experiment. If the hyper-parameter is numerical but has wildly different orders of magnitude, check "Treat X as categorical". By default, each bubble's Y coordinate (e.g. accuracy) is taken from the last iteration; in "Scalar display" you can choose the maximum or minimum across iterations.

• If you have multiple runs with different random seeds (stored as hyper-parameter "seed" for example), select "Merge: seed" in the sidebar to merge them into shaded plots (you can then select whether to show the mean, median, range or standard deviations).

• You can split all experiments into different plots by selected "Panels: One per run" in the sidebar, and selecting a metric to plot in "Y axis". Similarly, they can be split by a hyper-parameter value (e.g. a different algorithm in each panel, but each has multiple plots/runs).

Author

João Henriques, Visual Geometry Group (VGG), University of Oxford