panoptes-ui 1.6.0

panoptes: monitor computational workflows in real time

Bioinformaticians and data scientists, rely on computational frameworks (e.g. snakemake, nextflow, CWL, WDL) to process, analyze and integrate data of various types. Such frameworks allow scientists to combine software and custom tools of different origin in a unified way, which lets them reproduce the results of others, or reuse the same pipeline on different datasets. One of the fundamental issues is that the majority of the users execute multiple pipelines at the same time, or execute a multistep pipeline for a big number of datasets, or both, making it hard to track the execution of the individual steps or monitor which of the processed datasets are complete. panoptes is a tool that monitors the execution of such workflows.

panoptes is a service that can be used by:

• Data scientists, bioinformaticians, etc. that want to have a general overview of the progress of their pipelines and the status of their jobs
• Administrations that want to monitor their servers
• Web developers that want to integrate the service in bigger web applications

Note: panoptes currently supports workflows written in snakemake.

Snakemake 9 users: the legacy --wms-monitor flag was removed upstream. Monitoring is now delivered via a logger plugin — see Snakemake 9 support below.

Installation

Requirements:

• Python>=3.11
• virtualenv
• sqlite3

Local

pypi

Create virtual environment

python3 -m venv venv

Activate virtual environment

source venv/bin/activate

Install via pypi

pip install panoptes-ui

conda

Create conda environment

conda create --name panoptes -c conda-forge -c bioconda panoptes-ui

Activate conda environment

conda activate panoptes

Source code

Clone repo

git clone https://github.com/panoptes-organization/panoptes.git

Enter repo

cd panoptes

Create virtual environment

python3 -m venv venv

Activate virtual environment

source venv/bin/activate

Install all requirements

pip install .

Run the server

By default the server binds to 0.0.0.0:5000 — i.e. it listens on all network interfaces (open http://127.0.0.1:5000 in your browser) — and generates the sqlite database .panoptes.db.

Because 0.0.0.0 means every interface, the server is reachable from other machines on your network (and, on Linux, on any 127.x.x.x loopback address — the whole 127.0.0.0/8 block is loopback by standard). If you only need local access, restrict it with:

panoptes --ip 127.0.0.1

The running version is shown in the web UI under About (and is available programmatically as panoptes.__version__).

Environment variables:

Variable	Default	Description
PANOPTES_DB_URL	sqlite:///.panoptes.db	Database URL — any SQLAlchemy-supported database; see Using PostgreSQL.
PANOPTES_STALE_HOURS	48	Hours without any event before a Running workflow is marked Stale (e.g. its snakemake process was killed and never reported back). Set to 0 to disable. A Stale workflow flips back to Running if events resume, and can be deleted directly.

Using PostgreSQL (or another SQLAlchemy-supported database)

SQLite is the zero-setup default, but panoptes talks to its database purely through SQLAlchemy, so PANOPTES_DB_URL can point at a PostgreSQL server instead:

pip install 'panoptes-ui[postgres]'   # pulls the psycopg2 driver
export PANOPTES_DB_URL='postgresql+psycopg2://user:password@dbhost:5432/panoptes'
panoptes

When to prefer PostgreSQL over the default SQLite file:

• Containers — state lives outside the container, so restarts and redeployments keep the history without volume mounts.
• HPC / network filesystems — SQLite relies on file locking that is unreliable on NFS; if the panoptes working directory is on network storage, use PostgreSQL.
• Several gunicorn workers or panoptes instances — PostgreSQL handles concurrent writers natively instead of serializing on a file lock.

The full test suite runs against a real PostgreSQL in CI, so this stays supported. Other SQLAlchemy-supported databases (e.g. MySQL/MariaDB) are expected to work the same way but are not exercised by CI.

Installing via conda or using the biocontainer instead of pip? The bioconda package (and therefore the container image) ships the psycopg2 driver from recipe version 1.6.0 onward, so PANOPTES_DB_URL can point at PostgreSQL out of the box — which pairs naturally with running the container against an external database instead of mounting volumes for SQLite.

Using the development server

panoptes

Using a WSGI server

Install all necessary packages (see above), plus a WSGI server (e.g. gunicorn or waitress), and run the server:

gunicorn --access-logfile logs/access.log --error-logfile logs/error.log --timeout 120 --bind :5000 panoptes.app:app

Containers

Docker

Requirements:

• docker

Pull the image that is automatically built from bioconda. Replace <tag> with a release from the list of available tags: https://quay.io/repository/biocontainers/panoptes-ui?tab=tags

docker pull quay.io/biocontainers/panoptes-ui:<tag>

Then run the container with:

docker run -p 5000:5000 -it <image-id> panoptes

Note: In this case the database is stored within the docker image, so every time you restart the server the database will be empty. You would need to mount the volumes to make the database persistent.

Docker compose

Requirements:

• docker
• docker-compose

Build

docker-compose build

Run

docker-compose up -d

Server should run on: http://127.0.0.1:8000

Stop

docker-compose down

Singularity

You can also deploy the server with singularity. To do so pull the image with singularity. Replace <tag> with a release from the list of available tags: https://quay.io/repository/biocontainers/panoptes-ui?tab=tags

singularity pull docker://quay.io/biocontainers/panoptes-ui:<tag>

And then we can start the server by running:

singularity exec panoptes-ui:<tag>

Run an example workflow

A small reference pipeline (samtools sort/index → htseq-count → merge across four samples) that already wires up --logger panoptes lives at snakemake_example_workflow. Follow the instructions there to exercise this server end-to-end.

Snakemake 9 support

Starting with Snakemake 9, the --wms-monitor flag that older panoptes versions relied on has been removed. Monitoring is instead delivered through logger plugins.

To stream events from a Snakemake 9 workflow to panoptes, install the companion logger plugin with either pip or conda:

pip install snakemake-logger-plugin-panoptes
# or
conda install -c conda-forge -c bioconda snakemake-logger-plugin-panoptes

Then pass --logger panoptes to Snakemake:

snakemake \
    --cores 1 \
    --logger panoptes \
    --logger-panoptes-address http://127.0.0.1:5000

Re-runs under a stable id

By default every invocation registers a new workflow in panoptes. If you want re-runs of the same pipeline to show up as the same workflow — e.g. a run fails halfway, you fix the problem and resume — give it a stable id:

snakemake \
    --cores 1 \
    --logger panoptes \
    --logger-panoptes-address http://127.0.0.1:5000 \
    --logger-panoptes-workflow-id my-pipeline

When a run starts with an id panoptes already knows, the existing workflow entry is reset and reused instead of a second entry being created, so the dashboard tracks the latest state of that pipeline under one entry.

Restart protection: the entry is only reset when the previous run is in a finished state (Done, Error, Cancelled, Stale, …). A run interrupted with Ctrl+C or failed mid-way ends up as Error, so the normal fail-fix-restart cycle just works. But if the previous run still shows Running — either it is genuinely alive, or its process was killed hard (kill -9) and never got to report — panoptes will not wipe its history: the new run is tracked under a fresh entry with a random suffix (e.g. my-pipeline-1a2b3c4d) instead. To restart under the original id in that case, first cancel the stuck entry (the ban button in the UI, or POST /api/workflow/<id>/cancel) and then re-run — mirroring the cancel-before-delete rule.

The plugin lives in its own repository: panoptes-organization/snakemake-logger-plugin-panoptes. It registers a workflow with panoptes via GET /create_workflow on the first event and then translates Snakemake's LogEvent records (JOB_INFO, JOB_STARTED, JOB_FINISHED, JOB_ERROR, SHELLCMD, PROGRESS, ERROR, RUN_INFO) into the JSON payloads that panoptes' /update_workflow_status endpoint already understands. When a run ends without error, the plugin also emits a workflow_success event so panoptes can mark the workflow Done even when Snakemake never reported done == total — e.g. with --until, where the reported total is the whole DAG but only a subset actually runs.

Workflows orchestrated by Snakemake < 9 continue to work unchanged via the legacy --wms-monitor http://<host>:<port> flag.

Using panoptes via the Snakemake Python API

If you drive Snakemake programmatically instead of via the CLI, enable the panoptes logger by passing the plugin's settings in OutputSettings.log_handler_settings — the API equivalent of --logger panoptes --logger-panoptes-address ...:

from pathlib import Path

from snakemake.api import SnakemakeApi
from snakemake.settings.types import OutputSettings, ResourceSettings
from snakemake_logger_plugin_panoptes import LogHandlerSettings

panoptes_settings = LogHandlerSettings(
    address="http://127.0.0.1:5000",  # where the panoptes server runs
    workflow_id="my-workflow",        # optional: stable id, so re-runs reuse
                                      # the same workflow entry in panoptes
)

with SnakemakeApi(
    OutputSettings(
        log_handler_settings={"panoptes": panoptes_settings},
    )
) as snakemake_api:
    workflow_api = snakemake_api.workflow(
        resource_settings=ResourceSettings(cores=2),
        snakefile=Path("Snakefile"),
    )
    dag_api = workflow_api.dag()
    dag_api.execute_workflow()

Requires snakemake>=9 and snakemake-logger-plugin-panoptes (see above) in the same environment. Everything else (workflow registration, per-job events, end-of-run success reporting) behaves exactly as with the CLI flags.

panoptes in action

Workflow statuses

Status	Meaning
Running	The workflow is registered and events are arriving.
Done	All jobs finished (done == total), or the plugin reported end-of-run success (e.g. --until runs).
Error	A job or the workflow reported an error.
Cancelled	Explicitly cancelled via POST /api/workflow/<id>/cancel.
Stale	No events for more than PANOPTES_STALE_HOURS (default 48h) — the snakemake process was probably killed. Reverts to Running if events resume.
No Execution	Snakemake reported there was nothing to be done.

The web pages poll the JSON API every few seconds and refresh automatically when the data changes, so a dashboard left open tracks running workflows without manual reloads. A workflow page also shows a per-rule progress breakdown below the overall progress bar, so a run with hundreds of jobs but few rules stays legible at a glance.

panoptes API

Panoptes provides the following API endpoints:

Endpoint	Method	Description
/api/service-info	GET	Server status
/api/workflows	GET	Get all workflows
/api/workflow/<workflow-id>	GET	Get workflow status
/api/workflow/<workflow-id>/jobs	GET	Get all jobs of a workflow
/api/workflow/<workflow-id>/job/<job-id>	GET	Get job status
/api/workflow/<workflow-id>	PUT	Rename a workflow Expects a dictionary with new name (e.g. {'name': 'my new workflow name'})
/api/workflow/<workflow-id>/cancel	POST	Cancel a workflow (sets its status to Cancelled). Use this to move a workflow that is stuck as Running (e.g. a dry run, or a process killed with kill -9) into a deletable state.
/api/workflow/<workflow-id>	DELETE	Delete a workflow. A workflow that is still Running is protected and returns 403; cancel it first via the endpoint above.
/api/workflows/all	DELETE	Clean up database (deletes all workflows, including running ones)

To communicate with panoptes the following endpoints are used by snakemake:

Endpoint	Method	Description
/api/service-info	GET	Server status (same as above)
/create_workflow	GET	Get a unique id/name str(uuid.uuid4()) for each workflow
/update_workflow_status	POST	Panoptes receives a dictionary from snakemake that contains: - A log message dictionary (JSON-encoded) - The current timestamp - The unique id/name of the workflow. (e.g. {'msg': json.dumps(message), 'timestamp': time.asctime(), 'id': id})

Contribute

Please see the Contributing instructions.

CI server

Changes on master (and pull requests against it) trigger a GitHub Actions build that runs the test suite and a live end-to-end run of the example workflow.

Contact

In case the issues section is not enough for you, you can also contact us via discord