sixtytwo-cli 0.3.32

Sixtytwo CLI: `sixtytwo rent` reserves reliability-backed GPUs; `sixtytwo` qualifies, monitors, and NCCL-benchmarks your own GPU clusters, with Slurm/SkyPilot integration.

sixtytwo

sixtytwo-cli is the command-line client for sixtytwo, built for the terminal and for coding agents. It covers two jobs:

• Rent reliability-backed GPUs. sixtytwo rent reserves GPUs that sixtytwo has already qualified, watches them while you run, and refunds the unused time if hardware goes bad.
• Qualify and monitor your own clusters. sixtytwo runs hardware checks, NCCL benchmarks, straggler detection, and live fault monitoring on GPUs you already have, on both NVIDIA and AMD hardware, and keeps a per-node trust score from every test, fault, and recovery.

sixtytwo rent stays light (no torch), so it runs anywhere. The qualification and monitoring commands pull in the GPU stack only when you ask for it.

Quick start

pip install sixtytwo-cli

# Rent a qualified GPU
sixtytwo rent login                    # paste a token from sixtytwo.ai/account
sixtytwo rent catalog
sixtytwo rent A100_SXM --exec "nvidia-smi -L"  # bounded first test; auto-settles
sixtytwo rent H100_SXM -n 1 -H 2              # 1x H100 for 2 hours

# Or check the GPUs you already have
sixtytwo init
sixtytwo doctor

Install

Requirements: Python 3.11 or newer; macOS (Apple silicon), Linux x86_64/aarch64, or Windows.

Optional extras:

pip install 'sixtytwo-cli[gpu]'        # nvidia-ml-py + torch for local GPU checks
pip install 'sixtytwo-cli[skypilot]'   # SkyPilot cloud adapter (see below)
pip install 'sixtytwo-cli[server]'     # local dashboard + collector daemon

Rent GPUs

sixtytwo rent catalog                  # list SKUs and prices
sixtytwo rent H100_SXM -n 8 -H 4       # reserve 8x H100 for 4 hours, wait for READY
sixtytwo rent ls                       # list my reservations
sixtytwo rent status <reservation-id>
sixtytwo rent ssh <reservation-id>     # open a shell on a READY reservation

Every node is validated before you get it and monitored while you use it. Add --region us|eu|asia to pin a geography; billing follows actual usage. The storefront at sixtytwo.ai covers the same flow in the browser.

Qualify and monitor your own GPUs

These commands read a sixtytwo.yaml, so run init once first. It detects this machine (and your scheduler, if any) and writes the config into the current directory; everything below loads it.

sixtytwo init                          # writes ./sixtytwo.yaml (run this first)
sixtytwo doctor                        # validate this box: drivers, GPUs, topology
sixtytwo doctor --all-nodes --json     # validate every node in the cluster
sixtytwo test --quick --all            # fast per-node qualification
sixtytwo test --full gpu-01,gpu-02     # full suite, including NCCL
sixtytwo launch --pre-check python train.py   # qualify, then run with live monitoring
sixtytwo nodes                         # trust scores and fault history

doctor --all-nodes reports each node's GPU inventory, not just the local host's. Results land in a local trust registry, so a node's history (tests, faults, recoveries) follows it over time.

Straggler detection runs at both stages. test --full flags a GPU that is slower than its node peers (an identical kernel timed per GPU) or below its model's reference baseline, before you commit a run to it. During a run, launch watches per-rank step times and flags a rank that falls behind the others, so a GPU that degrades into a straggler mid-job is caught too.

Slurm clusters

Run init on the login node (or inside an allocation). It detects Slurm, fills the node list from sinfo/scontrol, and sets remote.mode: slurm, so test and doctor --all-nodes dispatch each node's checks through srun. Confirm the GPU partition and per-node GPU count in sixtytwo.yaml (the per-node count can't be detected from a GPU-less login node):

cluster:
  slurm:
    partition: gpu        # your GPU partition
    gpus_per_node: 8

On a large or multi-partition cluster init lists every node it can see, so trim cluster.nodes to the GPU nodes you actually want to qualify before running doctor --all-nodes or test --all.

SSH clusters (no scheduler)

Any set of boxes you can ssh into can be treated as a cluster: set remote.mode: ssh and list the nodes, where entries accept user@host:port so cloud pods with one public port per node (RunPod, Vast) work directly:

cluster:
  nodes:
    - root@203.0.113.7:45663
    - root@203.0.113.9:11542
remote:
  mode: ssh
  ssh_key_path: ~/.ssh/id_ed25519

doctor --all-nodes and test --all --parallel 8 then drive every node over ssh.

Kubernetes clusters (distributed PyTorch)

On a GPU Kubernetes cluster you can run a real multi-node torch.distributed test, a collective-bandwidth sweep plus a few DDP training steps, across nodes:

sixtytwo k8s-dist --nodes 2 --gpus-per-node 8 --gpu-type H100_SXM

It launches one pod per node as a Kubernetes Indexed Job, with ranks coordinated by torch's c10d rendezvous over a headless Service, so there is no Kubeflow or MPI operator to install and no SSH between pods. The result is scored against the same NCCL thresholds as the rest of the suite. Add --dry-run to print the manifests without touching the cluster, --json for the full report, or --no-cleanup to leave the Job up for inspection. The machine you run it from needs only kubectl access; torch runs in the pods.

Prerequisites: Kubernetes 1.24+, the NVIDIA device plugin and the nvidia RuntimeClass (the usual GPU-pod setup), at least two GPU nodes, and a sixtytwo namespace with a sixtytwo-agent service account:

kubectl create namespace sixtytwo
kubectl create serviceaccount sixtytwo-agent -n sixtytwo

SkyPilot

There are two ways to combine sixtytwo with SkyPilot:

# 1. Qualify a cluster you launched on any cloud SkyPilot supports
sixtytwo skypilot qualify my-cluster --mode quick

# 2. Provision sixtytwo's own GPUs through SkyPilot with `cloud: sixtytwo`
pip install 'sixtytwo-cli[skypilot]'
sixtytwo skypilot install              # register sixtytwo as a SkyPilot cloud
sky show-gpus --cloud sixtytwo

sixtytwo skypilot install is opt-in: it wires the adapter into the active virtualenv so sky launch can reserve reliability-backed GPUs with cloud: sixtytwo. Run sixtytwo skypilot --help for the full set of subcommands.

Topology and fabric health

The full test suite maps how a node is actually wired and checks the fabric it sits on:

• GPU topology: the NVLink and XGMI link matrix on every node, with PCIe-only or mixed-link wiring flagged before it slows a collective.
• InfiniBand fabric: the fabric is mapped to show which leaf switch each node hangs off (one sweep per cluster, reused across nodes), and port counters are classified so congestion reads as congestion rather than a cable fault.
• GPU to NIC affinity: an NCCL probe across all GPUs verifies rail-optimized placement.
• Diagnosis with opt-in repairs: failures map to a remediation plan (restart fabric manager, reset wedged GPUs, fix clock sync). Everything is off by default and dry-run first; fixes only execute when recovery.in_place_fixes.enabled is set, stop at the first verified repair, and never escalate to a disruptive action without a fresh diagnosis.

Hosted dashboards

Stream telemetry to sixtytwo.ai and watch the fleet from a browser:

sixtytwo connect --source ssh --name my-cluster   # register (token from the account page)
sixtytwo monitor                                  # stream per-GPU telemetry
sixtytwo install-agent                            # or install as a systemd service

The cluster page shows live per-node tiles, a node x GPU utilization heat map, and a 24h history; on Slurm clusters the agent also attributes GPU hours to users and accounts, feeding a chargeback report that exports to CSV.

Metrics for Grafana

sixtytwo metrics serve exposes a Prometheus /metrics endpoint backed by the local trust registry: trust scores, fault counters, recovery downtime, and per-check status. sixtytwo metrics export-grafana prints a curated dashboard you can import as is.

sixtytwo metrics serve --host 0.0.0.0 --port 9620
sixtytwo metrics export-grafana --output sixtytwo-overview.json

License

Commercial. The bundled LICENSE governs use. Learn more at sixtytwo.ai.