👁️ panoptes is a real-time dashboard for Snakemake workflows — see every running pipeline, which jobs are done, running, or failed, and drill into any one of them, all from a browser instead of tail -f on a log file.

panoptes is a service that can be used by:

• 🧬 Data scientists, bioinformaticians, etc. that want to have a general overview of the progress of their pipelines and the status of their jobs
• 🖥️ Administrations that want to monitor their servers
• 🌐 Web developers that want to integrate the service in bigger web applications

Note: panoptes currently supports workflows written in snakemake.

⚠️ Snakemake 9 users: the legacy --wms-monitor flag was removed upstream. Monitoring is now delivered via a logger plugin — see Snakemake 9 support below.

Table of contents

• ⚡ Quickstart
• 📦 Installation
  • 💻 Local
  • 🐳 Containers
• 🧪 Run an example workflow
• 🐍 Snakemake 9 support
• 🚦 Workflow statuses
• 🔌 panoptes API
• 🤝 Contribute
• 💬 Contact

Quickstart

The fastest way to see panoptes running is the published 🐳 container image — no Python environment to set up:

docker run -p 5000:5000 ghcr.io/panoptes-organization/panoptes:latest

Then open http://127.0.0.1:5000. From here, point a Snakemake 9 workflow at it with --logger panoptes (see Snakemake 9 support) to watch a real run come in.

Prefer conda or pip instead? See Installation below for a persistent, non-throwaway setup.

Installation

Requirements:

• Python>=3.11
• virtualenv
• sqlite3

Local

pypi

Create virtual environment

python3 -m venv venv

Activate virtual environment

source venv/bin/activate

Install via pypi

pip install panoptes-ui

conda

Create conda environment

conda create --name panoptes -c conda-forge -c bioconda panoptes-ui

Activate conda environment

conda activate panoptes

Source code

Clone repo

git clone https://github.com/panoptes-organization/panoptes.git

Enter repo

cd panoptes

Create virtual environment

python3 -m venv venv

Activate virtual environment

source venv/bin/activate

Install all requirements

pip install .

Run the server

By default the server binds to 0.0.0.0:5000 — i.e. it listens on all network interfaces (open http://127.0.0.1:5000 in your browser) — and generates the sqlite database .panoptes.db.

Because 0.0.0.0 means every interface, the server is reachable from other machines on your network (and, on Linux, on any 127.x.x.x loopback address — the whole 127.0.0.0/8 block is loopback by standard). If you only need local access, restrict it with:

panoptes --ip 127.0.0.1

The running version is shown in the web UI under About (and is available programmatically as panoptes.__version__).

Environment variables:

Using PostgreSQL (or another SQLAlchemy-supported database)

SQLite is the zero-setup default, but panoptes talks to its database purely through SQLAlchemy, so PANOPTES_DB_URL can point at a PostgreSQL server instead:

pip install 'panoptes-ui[postgres]'   # pulls the psycopg2 driver
export PANOPTES_DB_URL='postgresql+psycopg2://user:password@dbhost:5432/panoptes'
panoptes

When to prefer PostgreSQL over the default SQLite file:

• Containers — state lives outside the container, so restarts and redeployments keep the history without volume mounts.
• HPC / network filesystems — SQLite relies on file locking that is unreliable on NFS; if the panoptes working directory is on network storage, use PostgreSQL.
• Several gunicorn workers or panoptes instances — PostgreSQL handles concurrent writers natively instead of serializing on a file lock.

The full test suite runs against a real PostgreSQL in CI, so this stays supported. Other SQLAlchemy-supported databases (e.g. MySQL/MariaDB) are expected to work the same way but are not exercised by CI.

Installing via conda or using the biocontainer instead of pip? The bioconda package (and therefore the container image) ships the psycopg2 driver from recipe version 1.6.0 onward, so PANOPTES_DB_URL can point at PostgreSQL out of the box — which pairs naturally with running the container against an external database instead of mounting volumes for SQLite.

Using the development server

panoptes

Using a WSGI server

Install all necessary packages (see above), plus a WSGI server (e.g. gunicorn or waitress), and run the server:

gunicorn --access-logfile logs/access.log --error-logfile logs/error.log --timeout 120 --bind :5000 panoptes.app:app

Containers

Docker

Requirements:

• docker

Pull the image that is automatically built from bioconda. Replace <tag> with a release from the list of available tags: https://quay.io/repository/biocontainers/panoptes-ui?tab=tags

docker pull quay.io/biocontainers/panoptes-ui:<tag>

Then run the container with:

docker run -p 5000:5000 -it <image-id> panoptes

Note: In this case the database is stored within the docker image, so every time you restart the server the database will be empty. You would need to mount the volumes to make the database persistent.

Docker compose

Requirements:

• docker
• docker-compose

Build

docker-compose build

Run

docker-compose up -d

Server should run on: http://127.0.0.1:8000

Stop

docker-compose down

Singularity

You can also deploy the server with singularity. To do so pull the image with singularity. Replace <tag> with a release from the list of available tags: https://quay.io/repository/biocontainers/panoptes-ui?tab=tags

singularity pull docker://quay.io/biocontainers/panoptes-ui:<tag>

And then we can start the server by running:

singularity exec panoptes-ui:<tag>

Run an example workflow

A small reference pipeline (samtools sort/index → htseq-count → merge across four samples) that already wires up --logger panoptes lives at snakemake_example_workflow. Follow the instructions there to exercise this server end-to-end.

Snakemake 9 support

Starting with Snakemake 9, the --wms-monitor flag that older panoptes versions relied on has been removed. Monitoring is instead delivered through logger plugins.

To stream events from a Snakemake 9 workflow to panoptes, install the companion logger plugin with either pip or conda:

pip install snakemake-logger-plugin-panoptes
# or
conda install -c conda-forge -c bioconda snakemake-logger-plugin-panoptes

Then pass --logger panoptes to Snakemake:

snakemake \
    --cores 1 \
    --logger panoptes \
    --logger-panoptes-address http://127.0.0.1:5000

Re-runs under a stable id

By default every invocation registers a new workflow in panoptes. If you want re-runs of the same pipeline to show up as the same workflow — e.g. a run fails halfway, you fix the problem and resume — give it a stable id:

snakemake \
    --cores 1 \
    --logger panoptes \
    --logger-panoptes-address http://127.0.0.1:5000 \
    --logger-panoptes-workflow-id my-pipeline

When a run starts with an id panoptes already knows, the existing workflow entry is reset and reused instead of a second entry being created, so the dashboard tracks the latest state of that pipeline under one entry.

Restart protection: the entry is only reset when the previous run is in a finished state (Done, Error, Cancelled, Stale, …). A run interrupted with Ctrl+C or failed mid-way ends up as Error, so the normal fail-fix-restart cycle just works. But if the previous run still shows Running — either it is genuinely alive, or its process was killed hard (kill -9) and never got to report — panoptes will not wipe its history: the new run is tracked under a fresh entry with a random suffix (e.g. my-pipeline-1a2b3c4d) instead. To restart under the original id in that case, first cancel the stuck entry (the ban button in the UI, or POST /api/workflow/<id>/cancel) and then re-run — mirroring the cancel-before-delete rule.

The plugin lives in its own repository: panoptes-organization/snakemake-logger-plugin-panoptes. It registers a workflow with panoptes via GET /create_workflow on the first event and then translates Snakemake's LogEvent records (JOB_INFO, JOB_STARTED, JOB_FINISHED, JOB_ERROR, SHELLCMD, PROGRESS, ERROR, RUN_INFO) into the JSON payloads that panoptes' /update_workflow_status endpoint already understands. When a run ends without error, the plugin also emits a workflow_success event so panoptes can mark the workflow Done even when Snakemake never reported done == total — e.g. with --until, where the reported total is the whole DAG but only a subset actually runs.

Workflows orchestrated by Snakemake < 9 continue to work unchanged via the legacy --wms-monitor http://<host>:<port> flag.

Using panoptes via the Snakemake Python API

If you drive Snakemake programmatically instead of via the CLI, enable the panoptes logger by passing the plugin's settings in OutputSettings.log_handler_settings — the API equivalent of --logger panoptes --logger-panoptes-address ...:

from pathlib import Path

from snakemake.api import SnakemakeApi
from snakemake.settings.types import OutputSettings, ResourceSettings
from snakemake_logger_plugin_panoptes import LogHandlerSettings

panoptes_settings = LogHandlerSettings(
    address="http://127.0.0.1:5000",  # where the panoptes server runs
    workflow_id="my-workflow",        # optional: stable id, so re-runs reuse
                                      # the same workflow entry in panoptes
)

with SnakemakeApi(
    OutputSettings(
        log_handler_settings={"panoptes": panoptes_settings},
    )
) as snakemake_api:
    workflow_api = snakemake_api.workflow(
        resource_settings=ResourceSettings(cores=2),
        snakefile=Path("Snakefile"),
    )
    dag_api = workflow_api.dag()
    dag_api.execute_workflow()

Requires snakemake>=9 and snakemake-logger-plugin-panoptes (see above) in the same environment. Everything else (workflow registration, per-job events, end-of-run success reporting) behaves exactly as with the CLI flags.

Workflow statuses

The web pages poll the JSON API every few seconds and refresh automatically when the data changes, so a dashboard left open tracks running workflows without manual reloads. A workflow page also shows a per-rule progress breakdown below the overall progress bar, so a run with hundreds of jobs but few rules stays legible at a glance.

panoptes API

Panoptes provides the following API endpoints:

To communicate with panoptes the following endpoints are used by snakemake:

Contribute

Please see the Contributing instructions.

CI server

Changes on master (and pull requests against it) trigger a GitHub Actions build that runs the test suite and a live end-to-end run of the example workflow.

Contact

In case the issues section is not enough for you, you can also contact us via discord