snakemake-executor-plugin-sge
A Snakemake executor plugin for Sun Grid Engine (SGE), Univa Grid Engine (UGE), and Open Grid Scheduler (OGS) clusters.
Features
- Every job is an array job – singletons become
-t 1-1, batches become-t 1-N. One submission path, one tracking path, no special cases. - Group jobs as a single array job – Snakemake group jobs (several rules bundled together) are submitted as one qsub call.
- Rich qsub flag coverage – queue, project, parallel environment, wall time, memory (per-slot and total), priority, requeue, reservation, notify, mail, hold_jid, task concurrency, env-var export, join logs, and arbitrary extra flags.
- Per-rule resource overrides – any global setting can be overridden via
rule
resources:(e.g.sge_queue,sge_pe,sge_extra, …). - Two-stage status polling –
qstatfor in-flight jobs,qacctfor completed jobs; configurable retries and optional qacct disabling. - Clean log management – per-task stdout/stderr, optional auto-delete on success, age-based rotation.
Installation
pip install snakemake-executor-plugin-sge
Quick start
snakemake --executor sge --jobs 100
Common options
| CLI flag | Default | Description |
|---|---|---|
--sge-queue |
— | Default queue (-q) |
--sge-pe |
— | Parallel environment for multi-thread jobs |
--sge-project |
— | Project (-P) |
--sge-export-env |
True |
Export environment (-V) |
--sge-join-logs |
False |
Merge stdout+stderr (-j y) |
--sge-requeue |
— | Requeue on failure (-r y/n) |
--sge-reservation |
— | Resource reservation (-R y/n) |
--sge-notify |
False |
Send notify signal |
--sge-mail-on |
— | Mail events (b, e, a, s, n) |
--sge-mail-address |
— | Mail address (-M) |
--sge-hold-jid |
— | Hold until job(s) finish |
--sge-array-limit |
75000 |
Max tasks per qsub -t call |
--sge-task-concurrency |
— | Max concurrent array tasks (-tc) |
--sge-logdir |
.snakemake/sge_logs |
Log directory |
--sge-keep-successful-logs |
False |
Keep logs for successful jobs |
--sge-use-qacct |
True |
Use qacct for finished-job detection |
--sge-extra |
— | Raw extra qsub flags (global) |
--sge-jobname-prefix |
— | Prefix for SGE job names |
Per-rule resource overrides
rule my_rule:
resources:
runtime = 120, # minutes → h_rt=02:00:00
mem_mb = 8192, # total memory (divided by threads per slot)
mem_mb_per_cpu = 2048, # per-slot memory (takes precedence)
threads = 4,
sge_queue = "highmem.q",
sge_pe = "smp",
sge_project = "myproject",
sge_extra = "-l gpu=1", # arbitrary extra qsub flags
sge_resources = {"h_cpu": "24:00:00", "arch": "lx-amd64"},
sge_join_logs = True,
sge_requeue = True,
sge_priority = -100,
sge_notify = True,
sge_mail_on = "e",
sge_mail_address = "me@example.com",
sge_hold_jid = "12345",
sge_task_concurrency = 8,
sge_export_env = False,
threads: 4
shell: "..."
Array-job design
Every qsub submission uses qsub -t start-end. Execution commands are
stored in a JSON file on the shared filesystem before qsub is called. The
submission script reads $SGE_TASK_ID, looks up the compressed command, and
evaluates it. This avoids command-line length limits and all heredoc/quoting
problems.
License
MIT
1 maintainer
Metadata
1 maintainer
Release files for snakemake-executor-plugin-sge 0.6.25
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| snakemake_executor_plugin_sge-0.6.25.tar.gz | 30.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| snakemake_executor_plugin_sge-0.6.25-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 61.8 kB
Release files / snakemake_executor_plugin_sge-0.6.25.tar.gz
| Download URL | snakemake_executor_plugin_sge-0.6.25.tar.gz |
|---|---|
| Size | 30.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
53432b4e9c5f41cfb52eb9f58acc96bebdae24cfe422ed99659872229c78d8a7
|
|
BLAKE2b-256 checksum How to use checksums |
67a877c6a5808423c64674f7e44d3899468b8d367176060bb19a4381737a9a57
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / snakemake_executor_plugin_sge-0.6.25-py3-none-any.whl
| Download URL | snakemake_executor_plugin_sge-0.6.25-py3-none-any.whl |
|---|---|
| Size | 31.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bb3642dacbc2f07081de66d185d4520d6f5500d9ccdbc4ae322df5aa0ee8198a
|
|
BLAKE2b-256 checksum How to use checksums |
c324ed7f57f1f0b6ee03e36cef598a4823f091a63c03b4f66ed0710f70797212
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
