neurarch-slurm

Know what a GPU job needs before it takes a reservation. A SLURM cli_filter plugin that sizes a training job's model when it is submitted, on the submit host, and warns (or refuses) when the GPU it asks for cannot hold the training peak, and warns when it asks for far more than the job needs.

It is the SLURM sibling of neurarch-skypilot and reuses it rather than repeating it: the sizing is that package's Sizer, the verdict is its judge, and the card table is its cards.json. A job judged by SkyPilot and the same job judged here get the same numbers and the same sentence, apart from where the request is written (--gres instead of accelerators:). tests/test_parity.py holds that over 384 job/request pairs.

What a submit looks like

$ sbatch examples/finetune-qwen3-8b.sbatch
Neurarch: Qwen/Qwen3-8B needs 297.9 GB to train at batch 8 per device, 2048 tokens (peak 259.1 GB
x 1.15 headroom), and a10g:1 holds 24 GB. No batch fits a10g:1: the weights, gradients and
optimizer state alone need 180.9 GB. No single card holds it; the smallest request that does is
A100-80GB:4, sharded (FSDP or ZeRO-3) at the same global batch of 8. Sized offline by the neurarch
CLI from the model's structure; nothing was trained. Change --gres and resubmit.

That message is real output of the helper (neurarch-slurm) with the neurarch CLI from this repo sizing Qwen/Qwen3-8B on 2026-10-17, as the plugin prints it to the user's terminal before returning slurm.ERROR. It was not captured from a real sbatch, so whatever line sbatch adds of its own after a refusing cli_filter is not shown (see What was and was not run). In the default warn mode the same message is printed with "Submitted anyway" appended and the job is submitted.

How it works

sbatch / srun / salloc (user's process, submit host)
  └─ cli_filter.lua  slurm_cli_pre_submit(options)
       ├─ no GPU in --gres/--gpus/--gpus-per-node/--tres-per-node  -> allow, nothing spawned
       ├─ no #NEURARCH in the script and no NEURARCH_MODEL in env  -> allow, nothing spawned
       ├─ SLURM_JOB_ID set (a step inside an allocation)           -> allow, nothing spawned
       └─ neurarch-slurm --script ... --gres ... --nodes ...
            ├─ facts: #NEURARCH directives (validated by neurarch_skypilot.facts)
            ├─ sizing: neurarch_skypilot.Sizer (neurarch CLI offline, then hosted plan)
            └─ verdict: neurarch_skypilot.judge, SLURM wording
          prints decision=allow|warn|reject and message= lines

Why cli_filter and not job_submit.lua. job_submit.lua runs inside slurmctld, under its locks: a plugin there that waits seconds on a sizing call stalls every submit on the cluster. cli_filter runs in the user's own sbatch/srun/salloc process on the submit host, so it can take the time a sizing call takes and the controller never waits. The cost of that choice, said plainly: cli_filter is not an enforcement boundary. It runs in the client, so a user with their own build of sbatch, or anything that submits through slurmrestd, skips it. It is the place to tell people, before they wait in a queue for a card that cannot hold their job; it is not a control.

Fail open, always. No helper on the host, no output, a crash, a sizing source that is down, a GPU type with no capacity on file, a directive that cannot be read, an untyped gpu:2 with no site default: every one of these lets the submit through, with a warning where there is something to say. reject mode refuses only the two verdicts that mean the job should not take the reservation as asked: the request cannot hold the training peak, or the run is over the cost cap. An over-request is a warning in every mode.

Telling the check about the job

The job's facts go in the batch script as #NEURARCH directives, read the way sbatch reads #SBATCH: from the top of the file, stopping at the first line that is neither blank nor a comment.

#!/bin/bash
#SBATCH --gres=gpu:a10g:1
#NEURARCH model=Qwen/Qwen3-8B batch=8 seq-len=2048
#NEURARCH lora-rank=16 attention=sdpa hours=6

key	meaning	default
model	Hugging Face id, or a path to a .neurarch.json from neurarch-trace	required
batch	per device	32
seq-len	tokens per sample	2048, or the model's context if shorter
lora-rank	omit for a full fine-tune	none
attention	sdpa, flash or eager	sdpa
hours	expected wall hours, for the cost cap	none

Why directives, rather than --comment or the environment:

• The GPU request and the facts it is judged against sit in one file and change in one diff, so whoever reviews the job script sees both.
• --comment is free text that sites already use for accounting tags and workflow ids; taking it over collides with whatever is there.
• An exported NEURARCH_MODEL outlives the job it was meant for, and would quietly size every later submit from that shell.

The environment (NEURARCH_MODEL, NEURARCH_BATCH, ... the same names the SkyPilot policy reads) is used only when the script names no model, for srun/salloc and for sbatch reading its script from stdin. The two sources are never mixed key by key. A value that cannot be read (batch=eight) is reported and the job is submitted unchecked: it is never guessed.

The training command has to use the same values; nothing passes the directives on to it. examples/finetune-qwen3-8b.sbatch shows the pattern.

The GPU request

--gres=gpu:<type>:<n>, --gpus-per-node and --tres-per-node=gres/gpu:... are per node and are multiplied by --nodes (its minimum, for a range: the smallest allocation the job accepts is the one that has to hold it). --gpus is the job's total. A multi-card request is sized as one job sharded across all its cards, the same rule as npx neurarch rightsize and the SkyPilot policy: the peak at the global batch against the summed memory, with 1.15x headroom.

GPU type names are site-specific. Without a map, a type is matched case-insensitively to a Neurarch card id or SkyPilot name (a10g, l40s, h100, h200, a100-80gb; note that a bare a100 means the 40 GB card there). With a map, only the site's own types are judged and every suggestion names a card the cluster has:

{ "a10": "A10G", "a100": "A100-80", "h100": "H100", "l40s": "L40S" }

Card ids: T4, L4, A10G, A100-40, L40S, A100-80, H100, H200. A type outside the map (or the table) is not judged.

Install

On every host users submit from (login nodes), and on compute nodes if jobs submit jobs:

# 1. The helper and the package it builds on (neurarch-skypilot comes with it).
pip install neurarch-slurm
which neurarch-slurm          # put this path in cli_filter.lua: `helper`

# 2. The sizing CLI, offline (Node 18+). npx --no never fetches at submit time,
#    so install it, or point `cli` in cli_filter.lua at a checkout:
#      cli = "node /opt/neurarch/packages/neurarch-cli/bin/neurarch.mjs"

# 3. The plugin, next to slurm.conf, and turn it on.
neurarch-slurm --print-lua > /etc/slurm/cli_filter.lua
echo 'CliFilterPlugins=cli_filter/lua' >> /etc/slurm/slurm.conf

CliFilterPlugins is loaded by the client commands (sbatch, srun, salloc) from slurm.conf; keep slurm.conf identical across hosts as usual. A site that already has a cli_filter.lua keeps it: dofile this file at its top (it defines a global neurarch table and the three slurm_cli_* functions) and then redefine slurm_cli_pre_submit to call neurarch.pre_submit(options, offset) alongside its own logic.

Settings

At the top of cli_filter.lua, in the neurarch table. They live in the admin-owned file on purpose: nothing a user exports can switch the check off or change the mode.

field	meaning	default
helper	command that runs the helper; absolute, since sbatch has the user's PATH	/usr/local/bin/neurarch-slurm
mode	warn or reject	warn
gpu_map	path to the JSON map above	none
default_gpu_type	what an untyped --gres=gpu:N means here	none (untyped requests are not judged)
max_cost_usd	refuse (in reject mode) a run whose hours x cards x list price is over this	none
prices	path to a JSON object, card id to USD per GPU-hour	Modal on-demand list prices
annotate	append neurarch:<verdict> need=<GB> on <request> to --comment, so sacct -o comment keeps what admission said	false
sizing	auto (CLI, then hosted), cli or hosted	auto
cli	the sizing CLI command	neurarch on PATH, else npx --no neurarch
timeout_s	seconds per sizing source	30
debug	show the helper's stderr	false

Start on warn, read what it prints for a week, set gpu_map, then switch to reject.

The repository's .neurarch.yml

The helper looks for a .neurarch.yml next to the batch script, then in the submit directory, up to the repository root (or NEURARCH_POLICY_FILE), and applies it through the same CLI that sizes the job (neurarch policy apply). A waiver scoped to this cluster (cluster: "gpu-dev", matched against SLURM_CLUSTER_NAME or NEURARCH_CLUSTER) turns a reject into a warning that names its owner and expiry; budget.cards_allowed and the other budget lines reject in reject mode and warn in warn mode. Everything that goes wrong on our side still fails open: a policy file with no CLI to read it is reported and the submit goes through as asked.

Where the number comes from

Exactly where neurarch-skypilot gets it, through the same code: weights, gradients and AdamW state plus activations, computed from the model's structure (a Hugging Face config.json or a traced graph), calibrated on measured runs and published next to accelerate estimate-memory at https://neurarch.com/memory-calibration.html. Arithmetic: nothing is trained, downloaded beyond config.json, or run on a GPU. The offline CLI reads config.json from huggingface.co (set HF_TOKEN for a gated repo); the hosted plan takes a .neurarch.json graph and does not size LoRA. See the SkyPilot README for the details of each source.

What was and was not run

Run, on 2026-10-17:

• python3 -m unittest discover -s tests (47 tests): directive and --gres parsing, every decision and fail-open path, the helper as a subprocess with a stand-in sizing CLI, and the SkyPilot parity grid.
• The helper against the real neurarch CLI from packages/neurarch-cli, sizing Qwen/Qwen3-8B from Hugging Face. The refusal at the top of this file and the borderline LoRA case in the example script are its output.
• tests/test_lua.py runs cli_filter.lua under a real Lua interpreter with a stub slurm table and the real helper. It is skipped where no lua is installed (it was skipped on the machine this was written on) and runs in CI, which installs lua5.4.

Not run:

• No real slurmctld, sbatch, srun or salloc. The plugin follows the documented cli_filter/lua interface (slurm_cli_setup_defaults, slurm_cli_pre_submit, slurm_cli_post_submit, option names as the long flags), but the option table it reads has only been a stub. Two things in particular need checking on a live cluster: that options["argv"] carries the batch script path in your Slurm version (if it does not, the check falls back to the NEURARCH_* environment and a script-only job is silently not judged), and that assigning options["comment"] takes effect.
• No timing at scale: each judged submit spawns Python, and a cold sizing call fetches one config.json. Jobs without a GPU request or without a model named spawn nothing.

Not done yet

• The time limit (--time) is not used for the cost cap; hours= is.
• Verdicts are not recorded anywhere but the user's terminal and, with annotate, the job's comment.
• No cache across submits: a user submitting 50 jobs sizes the model 50 times.
• --gpus-per-task, MIG and sharded GPUs (shard, mps) are not judged.
• Not published to PyPI.

Tests

cd packages/neurarch-slurm && python3 -m unittest discover -s tests

Standard library only; neurarch-skypilot is imported from the sibling directory, so nothing needs installing.

MIT licensed.