snakemake-executor-plugin-sge

A Snakemake executor plugin for Sun Grid Engine (SGE), Univa Grid Engine (UGE), and Open Grid Scheduler (OGS) clusters.

Features

• Every job is an array job – singletons become -t 1-1, batches become -t 1-N. One submission path, one tracking path, no special cases.
• Group jobs as a single array job – Snakemake group jobs (several rules bundled together) are submitted as one qsub call.
• Rich qsub flag coverage – queue, project, parallel environment, wall time, memory (per-slot and total), priority, requeue, reservation, notify, mail, hold_jid, task concurrency, env-var export, join logs, and arbitrary extra flags.
• Per-rule resource overrides – any global setting can be overridden via rule resources: (e.g. sge_queue, sge_pe, sge_extra, …).
• Two-stage status polling – qstat for in-flight jobs, qacct for completed jobs; configurable retries and optional qacct disabling.
• Clean log management – per-task stdout/stderr, optional auto-delete on success, age-based rotation.

Installation

pip install snakemake-executor-plugin-sge

Quick start

snakemake --executor sge --jobs 100

Common options

Per-rule resource overrides

rule my_rule:
    resources:
        runtime        = 120,          # minutes → h_rt=02:00:00
        mem_mb         = 8192,         # total memory (divided by threads per slot)
        mem_mb_per_cpu = 2048,         # per-slot memory (takes precedence)
        threads        = 4,
        sge_queue      = "highmem.q",
        sge_pe         = "smp",
        sge_project    = "myproject",
        sge_extra      = "-l gpu=1",   # arbitrary extra qsub flags
        sge_resources  = {"h_cpu": "24:00:00", "arch": "lx-amd64"},
        sge_join_logs  = True,
        sge_requeue    = True,
        sge_priority   = -100,
        sge_notify     = True,
        sge_mail_on    = "e",
        sge_mail_address = "me@example.com",
        sge_hold_jid   = "12345",
        sge_task_concurrency = 8,
        sge_export_env = False,
    threads: 4
    shell: "..."

Array-job design

Every qsub submission uses qsub -t start-end. Execution commands are stored in a JSON file on the shared filesystem before qsub is called. The submission script reads $SGE_TASK_ID, looks up the compressed command, and evaluates it. This avoids command-line length limits and all heredoc/quoting problems.

License

MIT