snakemake-executor-plugin-aws-basic-batch

A Snakemake executor plugin for AWS Batch that uses pre-configured job definitions.

For full documentation, see the Snakemake Plugin Catalog.

Unlike the standard AWS Batch plugin which dynamically creates job definitions, this "basic" plugin relies on existing job definitions. This allows all resource configuration to be managed externally (e.g., via Terraform/CloudFormation). Additionally, workflow files and dependencies must bundled in the container image.

Usage

snakemake --executor aws-basic-batch \
  --aws-basic-batch-region us-east-1 \
  --aws-basic-batch-job-queue my-queue \
  --aws-basic-batch-job-definition my-job-def \
  --aws-basic-batch-tags "project=genomics,user=alice" \
  --default-storage-provider s3 \
  --default-storage-prefix s3://my-bucket/workdir

Workflow-Level Tags

Apply tags to all submitted jobs (both regular and coordinator) for cost tracking, filtering, etc.:

--aws-basic-batch-tags "project=genomics,run=exp1,costcenter=research"

Tags are comma-separated key=value pairs. Can also be set via the SNAKEMAKE_AWS_BASIC_BATCH_TAGS environment variable.

Coordinator Mode

Run the entire workflow as a fire-and-forget AWS Batch job:

snakemake --executor aws-basic-batch \
  --aws-basic-batch-coordinator true \
  ...

The coordinator job runs Snakemake itself on AWS Batch, submitting and monitoring rule jobs. Your terminal can disconnect after submission.

Optional coordinator-specific settings:

• --aws-basic-batch-coordinator-queue - Job queue for the coordinator (defaults to main queue)
• --aws-basic-batch-coordinator-job-definition - Job definition for the coordinator (defaults to main job definition)
• --aws-basic-batch-coordinator-job-name-prefix - Custom prefix for coordinator job names (defaults to snakemake-coordinator)
• --aws-basic-batch-coordinator-job-uuid - Custom UUID/identifier for coordinator job names (defaults to auto-generated UUID)

Per-Job Resource Customization

Override CPU, memory, queue, or job definition on a per-rule basis using Snakemake's resource system:

rule compute_heavy:
    output: "result.txt"
    resources:
        aws_batch_vcpu=4,
        aws_batch_mem_mb=8192,
        aws_batch_gpu=1,
        aws_batch_job_queue="high-memory-queue",
        aws_batch_job_name_prefix="myproject",
        aws_batch_scheduling_priority=100,
        aws_batch_job_uuid="my-run-id"
    shell: "python compute.py > {output}"

These values override the base job definition's resource configuration at submission time via AWS Batch's containerOverrides.resourceRequirements.

Requirements

• Workflow and dependencies must be included in the container image
• Job definitions should have appropriate IAM roles for S3 access and Batch job submission (for coordinator mode)