molcrafts-molq 0.4.0

Unified job queue for local execution and HPC schedulers

molq

Unified job queue for Python workloads that need the same submission API on a laptop, workstation, or HPC cluster.

Quick Start

pip install molcrafts-molq

from molq import Duration, JobResources, Memory, Submitor

local = Submitor("devbox", "local")

job = local.submit(
    argv=["python", "train.py"],
    resources=JobResources(
        cpu_count=4,
        memory=Memory.gb(8),
        time_limit=Duration.hours(2),
    ),
)

record = job.wait()
assert record.state.value == "succeeded"

Features

• One Submitor API for local, slurm, pbs, and lsf
• Typed submission inputs with Memory, Duration, Script, JobResources, JobScheduling, and JobExecution
• SQLite-backed persistence with WAL mode and UUID job identities
• Reconciliation and blocking waits through JobReconciler and JobMonitor
• First-class retry lineage with persisted attempt history
• Molq job-id dependencies and inspectable dependency metadata
• Profiles from ~/.molq/config.toml plus reusable defaults
• Cleanup and lightweight daemon workflows for retention and reconciliation
• Event hooks through EventBus
• Rich CLI for submit, list, status, watch, logs, history, inspect, cleanup, daemon, monitor, and cancel
• Default stdout/stderr capture for every submitted job, stored under the submission working directory unless overridden

Retry and Dependency Example

from molq import RetryBackoff, RetryPolicy, Submitor

slurm = Submitor("hpc", "slurm")

train = slurm.submit(
    argv=["python", "train.py"],
    retry=RetryPolicy(
        max_attempts=3,
        backoff=RetryBackoff(initial_seconds=10, maximum_seconds=60),
    ),
)

eval_job = slurm.submit(
    argv=["python", "eval.py"],
    after_success=[train.job_id],
)

Profile Example

~/.molq/config.toml

[profiles.gpu]
scheduler = "slurm"
cluster_name = "hpc"

[profiles.gpu.defaults.resources]
cpu_count = 8
memory = "34359738368"

[profiles.gpu.defaults.scheduling]
queue = "gpu"

[profiles.gpu.retry]
max_attempts = 3

CLI usage:

molq submit slurm --profile gpu python train.py
molq daemon slurm --profile gpu --once
molq cleanup slurm --profile gpu --dry-run

By default, ~/.molq stores global state such as config.toml and jobs.db. Per-job artifacts such as generated scripts, manifests, and default logs are written under the submission working directory at .molq/jobs/<job-id>/ unless you override jobs_dir or explicit output paths.

Documentation

• Getting Started for installation and first-job examples
• Scheduler Guide for backend capabilities and scheduler options
• Monitoring Guide for lifecycle, polling, and dashboards
• API Reference for the exported classes and functions
• CLI Reference for command-line usage

---

Built by MolCrafts with love.