HPC-DS1

A small, confidence-aware decision model that tunes HPC jobs, used the way Jev is: you call decide(state) and get back a choice, a confidence, and every option's probability. Confident decisions run automatically; uncertain ones go to an engineer.

Everything is trained and evaluated on real measurements. Every runtime in the dataset comes from actually running eight workloads on this machine (six HPC kernels, a neural-network training step, and backend request handling): an Intel i5 (10 cores / 16 threads, 24 GB) with an RTX 4050 Laptop GPU. Nothing is simulated.

import hpc_ds1

model = hpc_ds1.load()
state = model.observe("stencil", 160, {"ranks": 1, "threads": 1})   # runs the job once, for real
d = model.decide(state)                     # d.choice, d.confidence, d.probabilities, d.should_escalate
result = model.tune("stencil", 160, {"ranks": 1, "threads": 1})   # closed loop, every step runs for real

Results (held-out problem sizes the model never saw)

¹ Tries every neighbouring config at every step (what a careful engineer does by hand). ² Can still reach within 2% of the best runtime reachable from that state.

• Confidence: calibration error is 0.106 after temperature scaling, within the 0.12 target. At a 0.85 threshold, 59% of decisions run automatically and 97% of those are best moves; 38% ask to confirm and 2% escalate. A 5-seed ensemble was also trained; it was not better calibrated on validation, so the single network is used.
• Live tuning sessions: eight sessions, one per workload, every step executed on the hardware: 85–114% of the best known config re-run at the same time (the spread is run-to-run timing noise). Earlier six-kernel run, for reference: six sessions, one per kernel, with every step executed on the hardware during evaluation. All six ended on the same device and precision as the best known configuration. Measured against that config re-run back-to-back under the same conditions, they reached 73–111%. The spread is run-to-run timing noise of about ±25% on millisecond GPU runs; one session ended on the identical config and still read 93%.
• Never-seen workload (train on 7, test on the 8th): accuracy ranges from 86% (FFT, training step) down to 27% (server), and calibration degrades. The familiarity guard escalates 100% of those decisions: 240 confidently wrong automatic moves become 0, while flagging 0 of 286 familiar test decisions (results/guard.json).

Full numbers: results/metrics.json and results/generalization.json; interactive: the layer explorer (below).

System 1: the whole configuration before anything runs

hpc_ds1/system1.py answers from the job description alone: workload, size and the machine. A performance model learned from every real measured run predicts each configuration's runtime and failure risk. An ensemble of eight, each trained on a different bootstrap resample of problems and with a loss that ranks the configs of each problem, scores all 44 configurations in one pass (about 9 ms on CPU); its disagreement becomes the confidence.

hpc-ds1 predict --workload mltrain --size 4096   # instant: GPU, bf16 — nothing is run
hpc-ds1 plan    --workload stencil --size 160    # dual process: System 1 picks, System 2 only if needed

Dual process. If System 1 is confident, the job is familiar, and the first real run is no slower than 1.5× the prediction, the configuration is kept. Otherwise System 2 (the step-by-step tuning loop) starts from System 1's pick, and unfamiliar jobs are escalated. Sessions ("start": "system1") and fleet tasks use this by default.

• Cross-validated over all 72 problems (9 folds by size): 83.3% of instant picks are within 5% of the best configuration, reaching 96.9% of best speed, with 0 crashing picks. The dual process gives up about 0.4 points of speed against System 2 alone for 5x fewer real runs.
• Confidence: calibrated (error about 0.05), but with 72 problems it barely separates good picks from bad: AUROC 0.41–0.55 across every signal set tried (results/experiments/system1_confidence_*.json), i.e. noise-dominated. More measured problems (level 5, in use) is the fix, not more tuning. About 40% of jobs are confident enough to skip System 2. The recipe was chosen by cross-validation without touching the test sizes (scripts/tune_system1.py, results/system1_tuning.json); it traded about one problem's worth of accuracy for this.
• Limitations: System 1's confidence barely separates good picks from bad, so System 2's check matters. Results on the 8 test sizes are a small sample.
• On a workload System 1 never saw, picks can be badly wrong (the server workload, held out, picks the GPU, which has no server version). The size and workload checks escalate those.

Level 5: System 2 teaches System 1

Every real run made through HPC-DS1 on a machine (tuning sessions, plan, fleet tasks, the playground) is logged to ~/.hpc_ds1/data/live_runs.jsonl in the same format as the collected measurements (turn off with HPC_DS1_LOG_LIVE=0). hpc-ds1 learn retrains System 1 on the collected measurements plus those live runs and saves ~/.hpc_ds1/model/system1.pt, which hpc_ds1.load() then prefers. Problems the facility actually runs become familiar over time; rerun scripts/train_system1.py to re-measure accuracy.

Reproduce: python scripts/train_system1.py → model/system1.pt, results/system1.json.

Ask it anything: a Laya-style question encoder

hpc_ds1/encoder.py answers free-text questions about a job, Jev-style: you give a question, a type (noul = yes/no/unclear, or choice with options), and the job; a fine-tuned ModernBERT-base reads question + options + the job described as text in one pass and scores the options (about 29 ms on the GPU, temperature-calibrated).

The job text includes System 1's runtime estimate for every one-step change ("moving the job to the GPU x0.12; switching to bf16 x0.75; ..."), so the encoder reads performance evidence instead of guessing it. During training these estimates come from System 1 models that never saw that problem (scripts/system1_oof.py), so they are as imperfect as the live ones.

curl -X POST http://127.0.0.1:8765/v1/ask -H "Content-Type: application/json" -d '{"workload": "mltrain", "size": 4096,
  "config": {"ranks": 2, "threads": 4}, "questions": {"gpu": {"type": "noul", "question": "Would the graphics card speed this up?"}}}'

In the console playground, any question key that isn't one of the built-in five goes to the encoder.

Every training question is answered from the real measurements (e.g. "will bf16 be faster here?" is labelled by the two measured runtimes). Seven question families, several phrasings each, and synonyms (GPU / CUDA device / accelerator, bf16 / bfloat16 / half precision, ...); the last phrasing of every family and one synonym per term ("graphics card", "MPI ranks", ...) are never trained on.

Yes/no questions about the effect of one change ("would the graphics card make this faster?") are answered from System 1's estimate for that change, found with a transparent lexicon: 83–85% on held-out questions versus 68–78% for the encoder (scripts/eval_router.py); every answer reports its source. Strong on "CPU or GPU?" and "fp32 or bf16?" (100%), "which of these two setups is faster?" and "is this near the best?" (85–92%); "will this one change make it faster?" improved from about 60% to 68–78% with the hints. Weak: "will this setup crash?" (50–78%). Adding System 1's failure predictions to the text was tried and was worse on validation, so it is off (results/experiments/). The familiarity guard marks answers about unfamiliar jobs. Weights are not in git: HF_HOME=models/hf python scripts/train_encoder.py (downloads ModernBERT-base, about 600 MB, then about 8 minutes on an RTX 4050).

Quick start

Python 3.10+. For GPU moves, first install the CUDA build of PyTorch for your driver from pytorch.org; CPU-only works too (GPU moves are then unavailable).

pip install hpc-ds1                 # or: pip install "hpc-ds1[encoder]" for free-text questions
hpc-ds1 serve                       # console at http://127.0.0.1:8765

From source (to retrain or run the tests):

git clone https://github.com/Tarunkumar060106/hpc-ds1.git
cd hpc-ds1
pip install -e ".[test]"
python -m pytest tests -q

hpc-ds1 predict --workload mltrain --size 4096                # System 1: instant, nothing is run
hpc-ds1 plan    --workload stencil --size 160                 # System 1, then System 2 only if needed
hpc-ds1 decide  --workload gemm --size 2048 --ranks 2 --threads 4
hpc-ds1 tune    --workload stencil --size 160                 # asks before running uncertain moves
hpc-ds1 tasks                                                 # what the model knows

Your own machine. The shipped models were trained on one laptop (i5 + RTX 4050); on other hardware the familiarity guard escalates every move until you run hpc-ds1 setup (about 1 hour, or --quick for about 30 minutes), which measures your machine and trains on it.

Free-text questions. Install with the encoder extra, then download hpc-ds1-encoder-v1.0.0.zip from the v1.0.0 release (about 600 MB, too big for PyPI) and unzip it into ~/.hpc_ds1 so that ~/.hpc_ds1/models/encoder/scorer.pt exists. In a source checkout, models/encoder/ also works, or set HPC_DS1_ENCODER.

HTTP: POST /v1/decide, /v1/predict, /v1/ask, /v1/trace, /v1/tune, /v1/sessions, /v1/fleet/...; GET /v1/health, /v1/schema, /v1/queue, /v1/audit. Without telemetry in a request, the server runs the job once to measure it.

Security: hpc-ds1 serve has no HTTPS and no console login in v1. Run it on a trusted network or inside a private overlay network such as Tailscale or ZeroTier; do not expose it to the public internet.

How it works

Workloads (hpc_ds1/kernels.py). Eight real workloads, each splittable across ranks:

Your own task: write a Kernel subclass with @register in a file in ~/.hpc_ds1/plugins/ (example: examples/plugins/etl_batch.py), then hpc-ds1 setup --workloads <name>. Until it is measured, decisions about it are escalated.

Execution (hpc_ds1/runner.py). Ranks are separate processes with their own memory, connected to rank 0 by pipes. Every rank computes its shard with T threads, and rank 0 gathers and combines the results. This is the distributed-memory pattern MPI uses, run on one machine without an MPI install. Each run is checked against an fp32 reference; a config that crashes or gives a wrong answer is recorded as failed. Telemetry is measured: runtime (median of repeats), CPU time across all ranks, GPU busy time, and communication/gather overhead.

Configurations and actions (hpc_ds1/space.py). A config is {ranks 1–8, threads 1–16, GPU on/off, fp32/bf16}, which gives 44 per problem. There are nine actions: ADD/REMOVE_RANKS, ADD/REMOVE_THREADS, USE_GPU/CPU, USE_BF16/FP32 and STOP_TUNING.

Data (scripts/collect.py, hpc_ds1/dataset.py).

• 3,168 real runs over 72 problems (8 workloads x 9 sizes): 2,589 ran correctly and 579 failed or have no such version (server has no GPU or bf16 path).
• The failures are bf16 FFT and SpMV on this CPU, and GPU bf16 FFT at non-power-of-two sizes.
• Each correct config becomes a state. Its label is the first move of the best chain of measured, clearly faster (>5% and above noise) moves.
• Several moves are often equally good, so training uses soft targets that share credit among moves reaching within about 2% of the best.

Model (hpc_ds1/net.py, 0.59M parameters: the "small" size, chosen on validation loss among 0.1M, 0.59M and 2.85M).

• Encoders: four modular encoders (hardware, workload, config, telemetry), each with pre-norm residual blocks.
• Fusion: attention over the four tokens.
• Heads: action, stop, and expected-improvement heads; invalid actions are masked.
• Calibration: temperature scaling fitted on validation.

Tuning loop (hpc_ds1/tuner.py).

1. The model decides; below the threshold, the decision is escalated.
2. The chosen config is run for real.
3. A faster result is kept. A slower or failing one is reverted and excluded, and the model is asked again.
4. The session ends on STOP or after two misses in a row.

Reproduce everything on your machine

python scripts/collect.py          # ~45 min: measures every config of every kernel (resumable)
python scripts/train.py            # trains, calibrates, evaluates, runs live sessions, exports the UI data
python scripts/generalization.py   # leave-one-kernel-out + input ablations

collect.py measures whatever machine it runs on, so a different machine produces a model for that machine. Keep the machine plugged in and otherwise idle while collecting.

Limitations

• One machine. The model has only seen this laptop. The hardware features are constant in the training data, so it can't transfer to other hardware without collecting there (hpc-ds1 setup, which has so far only been tested on copies of this laptop's data).
• Fleet = many independent tasks. The fleet places whole tasks on machines (a high-throughput cluster); it does not yet split one job across machines.
• No HTTPS or console login in v1 (see Quick start).
• Question encoder: about 87% on unseen wording and problems, but it can be confidently wrong on "will this one change make it faster?"; use System 1 and System 2 for actual decisions.
• Ranks are processes, not MPI. The communication pattern (shard, compute, gather to rank 0) is real, but it runs over pipes on one machine, not over a network.
• Generalizing to new kinds of work is partial (see results), and calibration degrades on them. That's exactly where escalating to an engineer matters most.
• Evaluation replays real measurements for the hundreds of tuning sessions, because running them all live would take hours. The live sessions show the same behaviour.
• Telemetry: memory use and run-to-run noise are recorded but deliberately not model inputs. Both depend on what the measuring process did earlier, not on the job, and they made identical jobs look different live versus during collection.

Layout

hpc_ds1/      workloads, runner, model, System 1, encoder, guard, tuner, API, server, fleet, CLI
scripts/      collect, train, generalization, fit_guard, train_system1, tune_system1, train_encoder, ...
hpc_ds1/ui/          console.html (facility console), index.html (layer explorer)
hpc_ds1/models/      hpc_ds1.pt (step model + guard), system1.pt
hpc_ds1/data/real/   measurements.jsonl (3,168 real runs), hardware.json
hpc_ds1/results/     metrics, generalization, guard, system1, encoder (shown on the Evaluation page)
results/             training logs, out-of-fold estimates, experiments/ (every variant tried)
examples/     plugins/etl_batch.py (bring your own task)
tests/        test_project.py
docs/         DEMO.md, architecture/ (diagram)
archive/      the earlier simulator-based prototype (code only), kept for history