KSUBMIT - Kubernetes Job Submission Tool with Shorthand Commands

A Python-based CLI tool that parses shell scripts with UGE-like #$ directives and translates them into Kubernetes jobs.

Overview

KSUBMIT makes it easy to submit batch jobs to Kubernetes clusters using a familiar syntax inspired by Univa Grid Engine (UGE). It allows you to:

• Parse shell scripts with UGE-like #$ directives
• Generate Kubernetes job specifications
• Submit jobs to Kubernetes
• Monitor job status and logs
• Manage job lifecycle

Installation

# Install from PyPI
pip install ksubmit

# Or install from source
git clone https://github.com/yourusername/ksubmit.git
cd ksubmit
pip install -e .

Quick Start

1. Initialize KSUBMIT configuration:

kconfig init

2. Create a job script (example.sh):

#!/bin/bash
#$ -N my-job
#$ -I ubuntu:latest
#$ -l h_vmem=4G
#$ -l h_rt=01:00:00
#$ -pe smp 2
#$ -mount data=./data
#$ -retry 2
#$ -v ENV_VAR=value

echo "Starting job..."
# Your commands here
echo "Job completed!"

3. Submit the job:

krun example.sh

4. Check job status:

kstat <job-id>

5. View job logs:

klogs <job-id>

Command Reference

Job Script Directives

ksubmit supports the following UGE-like directives in job scripts:

Linting Rules

ksubmit validates job scripts against the following rules to ensure they are correctly formatted and semantically valid:

Configuration

KSUBMIT stores configuration in ~/.ksubmit/config.yaml. You can view or update this configuration using:

# Initialize configuration
kconfig init
kconfig init --namespace my-namespace --email me@example.com

# View configuration
kconfig list

# Update configuration
kconfig set namespace my-namespace
kconfig set email me@example.com

# Reset configuration
kconfig reset

Examples

Simple Example Script

Here's a simple example of a job script that runs a Python container and prints a message:

#!/bin/bash
#$ -N hello-world
#$ -I docker.io/library/python:3.10
#$ -l h_vmem=2G
#$ -l h_rt=00:10:00
#$ -pe smp 1
#$ -v MSG="Hello from ksubmit"

echo "$MSG"

This script:

• Names the job "hello-world"
• Uses the Python 3.10 container image
• Requests 2GB of memory
• Sets a runtime limit of 10 minutes
• Requests 1 CPU
• Sets an environment variable MSG
• Runs a simple echo command to print the environment variable

Multi-Job Workflow Example

Here's a more complex example that demonstrates a workflow with multiple job blocks and dependencies:

#!/bin/bash
#$ -N global-setup
#$ -v REFGENOME=hg38
#$ -mount data=./data
#$ -mount code=./pipeline

#$ -I docker.io/continuumio/miniconda3
#$ -N preprocess
#$ -l h_vmem=16G
#$ -l h_rt=04:00:00
#$ -pe smp 4
#$ -file ./pipeline/env.yaml
#$ -entrypoint bash
conda env create -f /mnt/code/env.yaml || true
conda activate pepatac
python /mnt/code/run.py --genome $REFGENOME --input /mnt/data/sample.tsv

#$ -I docker.io/biocontainers/samtools:v1.9-4-deb_cv1
#$ -N analyze
#$ -after preprocess
#$ -l h_vmem=8G
samtools view -b /mnt/data/sample.bam > /mnt/data/sample.filtered.bam

This script:

• Sets up global directives that apply to all job blocks
• Creates two job blocks with different container images
• Establishes a dependency where the second job waits for the first to complete
• Mounts local directories into the container
• Copies a file into the container
• Uses environment variables across job blocks

Basic Job Submission

krun my-script.sh

Following Job Logs

klogs <job-id> --follow
# or
klogs <job-id> --output-file logs.txt

Deleting a Job

kdel <job-id>

Testing

The project uses pytest for testing. Tests are organized in a structure that mirrors the ksubmit package structure.

Running Tests

To install test dependencies:

pip install -e ".[dev]"

To run all tests:

python -m pytest

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.