Awaitless

Durable, bounded, event-driven jobs for AI coding agents.

Awaitless turns a long local, SSH, or Slurm command into a persistent job with a stable job_id. An agent calls MCP once to submit, once to wait, and receives the exit code, bounded logs, and declared JSON Artifacts—without writing shell commands or repeatedly spending context on polling.

简体中文

Why Awaitless

• Survives the client: closing the terminal or interrupting wait does not cancel the managed job. Reuse the same ID from a new client.
• Agent-native MCP tools: submit_job, wait_for_job, get_job_status, get_job_logs, cancel_job, and list_jobs use the standard stdio protocol.
• Schedules cluster work: the Slurm backend persists scheduler IDs and maps queue/accounting state, exit codes, cancellation, logs, and Artifacts.
• Returns bounded context: stdout and stderr tails share a configurable byte budget; complete logs stay on disk.
• Returns machine-readable results: declared JSON Artifacts are parsed into parsed_results.
• Handles real cluster edges: SSH liveness uses a wrapper-owned heartbeat and does not assume separate login sessions can see the same PID namespace.

Install

The distribution name is awaitless-runner. It installs the awaitless CLI and awaitless-mcp stdio server. Awaitless requires Linux, Python 3.10+, and Bash. SSH and Slurm hosts also require OpenSSH (ssh and sftp) locally.

python -m pip install awaitless-runner
awaitless doctor --json

From a source checkout:

python -m pip install -e .

Agent-native MCP quick start

Point your MCP client at the installed stdio command (adapt the outer key to your client's configuration format):

{
  "mcpServers": {
    "awaitless": {
      "command": "awaitless-mcp",
      "args": ["--config", "/home/me/.config/awaitless/config.toml"]
    }
  }
}

The server uses the official modelcontextprotocol/python-sdk. The Agent can now call submit_job with an argv array and later call wait_for_job with the returned ID. Each MCP invocation opens the existing Service and SQLite store; there is no Awaitless daemon, HTTP endpoint, or Web service. Stopping the stdio server does not stop a submitted job.

Acceptance criterion: install the PyPI package and configure one MCP server; the Agent can submit to Slurm, survive a client disconnect, and receive a structured result without writing Awaitless CLI commands.

Quick start

Submit returns before the job finishes:

awaitless submit --json --name build -- ninja -C build

{"job_id":"job_019F...","state":"running","backend":"local"}

Then make one blocking call:

awaitless wait job_019F... --json

If that client is closed or interrupted, start a new one and run the same wait command with the saved ID. The managed job keeps running.

Useful one-shot operations:

awaitless status <job-id> --json
awaitless logs <job-id> --tail 200 --json
awaitless cancel <job-id> --grace-period 5s --json
awaitless list --state running --json
awaitless inspect <job-id> --json

SSH and structured Artifacts

Declare a host in ~/.config/awaitless/config.toml:

[defaults]
backend = "local"
log_tail_lines = 200
max_return_bytes = 65536
poll_interval = 2

[hosts.gpu]
hostname = "gpu.example.com"
port = 22
user = "developer"
identity_file = "~/.ssh/id_ed25519"
remote_job_dir = "~/.awaitless/jobs"
# gssapi_authentication = false
# connect_timeout = 8
# operation_timeout = 20

operation_timeout is the minimum timeout for one SSH control operation, not a job runtime limit. Use submit --timeout to limit the job itself.

Submit a remote command and declare its result:

awaitless submit --json \
  --host gpu \
  --cwd /workspace/project \
  --timeout 2h \
  --artifact results/benchmark.json \
  -- ./run_benchmark.sh

On completion, wait --json reports Artifact existence, size, and modification time. A declared JSON file within the return budget is also exposed directly:

{
  "state": "succeeded",
  "exit_code": 0,
  "truncated": false,
  "parsed_results": {
    "correctness": true,
    "latency_us": 24.7
  }
}

Relative local Artifacts are resolved from the submission working directory, even if a later client runs elsewhere. --log-dir /path/to/logs creates an isolated /path/to/logs/<job-id>/ directory per job.

Slurm backend

Configure a scheduler host and its default resource request:

[defaults]
backend = "slurm"
host = "cluster"
poll_interval = 10
log_tail_lines = 200
max_return_bytes = 65536

[hosts.cluster]
hostname = "login.cluster.example"
user = "developer"
backend = "slurm"
gssapi_authentication = false
operation_timeout = 30
slurm_accounting_grace = 120
slurm_job_dir = ".awaitless/slurm/jobs"

[hosts.cluster.slurm]
partition = "compute"
account = "research"
nodes = 1
ntasks = 1
cpus_per_task = 1
time = "00:30:00"

With defaults.host configured, MCP calls may omit both backend and host. submit_job may override the allowlisted options account, constraint, cpus_per_task, gres, mem, nodes, ntasks, partition, qos, and time through slurm_options. The backend sends the batch script to sbatch over stdin, persists the returned Slurm ID, checks active state with squeue, recovers terminal state/exit code/runtime with sacct, and cancels with scancel. User computation is therefore scheduled on an allocated compute node—never launched as a process on the SSH login node. A separate SFTP data channel creates the private job directory and reads only the bounded log tails and declared Artifacts.

Slurm PENDING maps to Awaitless pending; active scheduler states map to running; COMPLETED maps to succeeded; CANCELLED maps to cancelled; TIMEOUT/DEADLINE map to timed_out; scheduler, node, launch, OOM, and preemption failures map to failed. ExitCode values such as 7:0 and signal terminations are preserved as process-style exit codes.

Real MCP → Slurm disconnect demo

On 2026-08-10, two separate MCP stdio clients ran the checked-in demo against a real Slurm 25.11.2 cluster:

node099 is the allocated compute node. The reproducible driver is scripts/mcp_slurm_demo.py, and the raw structured evidence is assets/mcp-slurm-demo.json.

Real experiment: 12 polls to 2 calls

On 2026-08-10, the reproducible experiment ran the same sleep-only workload on a real SSH login node: twelve 1 KiB log records, 4.5 seconds apart. It used no CPU- or GPU-intensive work. The traditional side repeatedly fetched its entire log snapshot twelve times; Awaitless used one submit and one wait.

That is 72,704 fewer returned log bytes (85.5%) and 13 → 2 agent-visible CLI calls (84.6%). The twelve traditional log snapshots were [1024, 2048, 3072, 4096, 5120, 6144, 8192, 9216, 10240, 11264, 12288, 12288] bytes. "Calls" here means agent-visible CLI invocations; Awaitless's internal SSH control operations do not trigger additional agent turns. The byte figures are decoded log content, not estimated tokens or network wire bytes.

The runnable method and raw result are in benchmarks/.

Awaitless vs. alternatives

Source notes: GNU nohup, tmux, Pueue, the Slurm overview, and the Codex Goal mode guide. Awaitless uses Slurm for allocation instead of replacing the cluster scheduler.

Reliability model

• The local runner and user command have independent sessions and process groups; cancellation targets the whole validated group.
• SQLite uses WAL, and active-to-terminal transitions are transactional so completion, cancellation, and stall detection cannot overwrite each other.
• SSH wrappers atomically persist exit_code and finished_at. A lightweight heartbeat handles hosts where separate SSH sessions cannot inspect the same PID namespace; PID, process group, and /proc start time remain a fallback.
• SSH cancellation persists intent before signaling the validated process group. OpenSSH host-key verification keeps its secure defaults.
• Slurm control-plane SSH calls are restricted to sbatch/squeue/sacct/scancel; file access uses SFTP, and arbitrary computation exists only inside the submitted batch script.
• Suspected credential values are redacted from metadata, and the executable run specification is stored with mode 0600.

States are starting, running, stalled, succeeded, failed, cancelled, timed_out, and lost. --stall-timeout 20m reports a stalled job but does not cancel it automatically.

CLI exit codes: 0 success, 1 internal error, 2 invalid usage, 3 job failure, 4 job/client wait timeout, 5 cancelled, 6 lost, and 7 SSH connection failure.

Development

PYTHONPATH=src python3 -m unittest discover -s tests -v
ruff check src tests benchmarks

GitHub Actions runs the test suite on every supported CPython release from 3.10 through 3.14, then builds the distributions, checks the PyPI README, and runs an installed-wheel CLI/Artifact smoke test. Version tags use PyPI Trusted Publishing without a stored API token.

The Codex Skill lives in skills/awaitless. The v0.1 product requirements are in docs/PRD.zh-CN.md, and the v0.2 Agent/Slurm acceptance contract is in docs/v0.2.zh-CN.md.

License

MIT