snakemake-logger-plugin-panoptes 0.3.0

Snakemake logger plugin that forwards workflow events to a panoptes server.

snakemake-logger-plugin-panoptes

A Snakemake 9 logger plugin that forwards workflow events to a running panoptes server, replacing the legacy --wms-monitor integration that was removed in Snakemake 9.

Requirements

• Python 3.11 or newer
• Snakemake 9 or newer
• A running panoptes server

Install

# via PyPI
pip install snakemake-logger-plugin-panoptes

# or via bioconda
conda install -c conda-forge -c bioconda snakemake-logger-plugin-panoptes

After installing, Snakemake should advertise the plugin in --help:

snakemake --help | grep logger-panoptes
#   --logger-panoptes-address VALUE
#   --logger-panoptes-timeout VALUE

Use

Start a panoptes server, then run Snakemake with the plugin enabled:

snakemake \
    --cores 1 \
    --logger panoptes \
    --logger-panoptes-address http://127.0.0.1:5000

You can also point at the address via an environment variable:

export SNAKEMAKE_LOGGER_PANOPTES_ADDRESS=http://127.0.0.1:5000
snakemake --cores 1 --logger panoptes

Settings

Flag	Env var	Default	Description
--logger-panoptes-address	SNAKEMAKE_LOGGER_PANOPTES_ADDRESS	(required)	Base URL of the panoptes server.
--logger-panoptes-timeout	—	10.0	Per-request HTTP timeout in seconds.
--logger-panoptes-workflow-id	SNAKEMAKE_LOGGER_PANOPTES_WORKFLOW_ID	(auto uuid)	Stable workflow id. Re-running with the same id reuses and resets the existing panoptes workflow instead of creating a new one.

Give a run a stable id so re-runs update the same panoptes entry instead of piling up new ones:

snakemake \
    --cores 1 \
    --logger panoptes \
    --logger-panoptes-address http://127.0.0.1:5000 \
    --logger-panoptes-workflow-id my-pipeline

How it works

On workflow start the plugin calls GET /create_workflow to register a new workflow with the panoptes server and remembers the returned workflow id. When --logger-panoptes-workflow-id is set it is passed along, and the server reuses (and resets) any existing workflow with that id so re-runs land on the same id. It then translates Snakemake LogEvent records (JOB_INFO, JOB_STARTED, JOB_FINISHED, JOB_ERROR, SHELLCMD, PROGRESS, ERROR, RUN_INFO) into the JSON message format that panoptes' /update_workflow_status endpoint already understands.

Network errors are logged but never crash the workflow.

If the panoptes server is served over HTTPS with a certificate signed by a private/corporate CA, point the plugin at the CA bundle via one of REQUESTS_CA_BUNDLE, SSL_CERT_FILE, or CURL_CA_BUNDLE (checked in that order); the plugin uses it to verify the connection.

For a working example workflow that uses this plugin, see snakemake_example_workflow.

Development

pip install -e '.[dev]'   # editable install with test deps (quote .[dev] for zsh)
pytest

License

MIT