sixtytwo-cli 0.3.10

Sixtytwo CLI: `sixtytwo rent` reserves reliability-backed GPUs; `sixtytwo` qualifies, monitors, and NCCL-benchmarks your own GPU clusters, with Slurm/SkyPilot integration.

sixtytwo

sixtytwo-cli is the command-line client for sixtytwo, built for the terminal and for coding agents. It covers two jobs:

• Rent reliability-backed GPUs. sixtytwo rent reserves GPUs that sixtytwo has already qualified, watches them while you run, and refunds the unused time if hardware goes bad.
• Qualify and monitor your own clusters. sixtytwo runs hardware checks, NCCL benchmarks, straggler detection, and live fault monitoring on GPUs you already have, and keeps a per-node trust score from every test, fault, and recovery. NVIDIA and AMD Instinct (MI300X family) are both supported: AMD nodes are checked through rocm-smi and amd-smi, validated against real MI300X hardware.

sixtytwo rent stays light (no torch), so it runs anywhere. The qualification and monitoring commands pull in the GPU stack only when you ask for it.

Install

pip install sixtytwo-cli

Requirements: Python 3.11 or newer; macOS (Apple silicon), Linux x86_64/aarch64, or Windows.

Optional extras:

pip install 'sixtytwo-cli[gpu]'        # nvidia-ml-py + torch for local GPU checks
pip install 'sixtytwo-cli[skypilot]'   # SkyPilot cloud adapter (see below)
pip install 'sixtytwo-cli[server]'     # local dashboard + collector daemon

Rent GPUs

sixtytwo rent catalog                  # list SKUs and prices
sixtytwo rent H100_SXM -n 8 -H 4       # reserve 8x H100 for 4 hours, wait for READY
sixtytwo rent ls                       # list my reservations
sixtytwo rent status <reservation-id>
sixtytwo rent ssh <reservation-id>     # open a shell on a READY reservation

Every node is validated before you get it and monitored while you use it. Add --region us|eu|asia to pin a geography; billing follows actual usage. Prefer the browser? The live storefront is at sixtytwo.ai.

Qualify and monitor your own GPUs

These commands read a sixtytwo.yaml, so run init once first. It detects this machine (and your scheduler, if any) and writes the config into the current directory; everything below loads it.

sixtytwo init                          # writes ./sixtytwo.yaml (run this first)
sixtytwo doctor                        # validate this box: nvidia-smi or rocm-smi, dcgmi, topology
sixtytwo doctor --all-nodes --json     # validate every node in the cluster
sixtytwo test --quick --all            # fast per-node qualification
sixtytwo test --full gpu-01,gpu-02     # full suite, including NCCL
sixtytwo launch --pre-check python train.py   # qualify, then run with live monitoring
sixtytwo nodes                         # trust scores and fault history

doctor --all-nodes reports each node's GPU inventory, not just the local host's. Results land in a local trust registry, so a node's history (tests, faults, recoveries) follows it over time.

Straggler detection runs at both stages. test --full flags a GPU that is slower than its node peers (an identical kernel timed per GPU) or below its model's reference baseline, before you commit a run to it. During a run, launch watches per-rank step times and flags a rank that falls behind the others, so a GPU that degrades into a straggler mid-job is caught too.

Slurm clusters

Run init on the login node (or inside an allocation). It detects Slurm, fills the node list from sinfo/scontrol, and sets remote.mode: slurm, so test and doctor --all-nodes dispatch each node's checks through srun. Confirm the GPU partition and per-node GPU count in sixtytwo.yaml (the per-node count can't be detected from a GPU-less login node):

cluster:
  slurm:
    partition: gpu        # your GPU partition
    gpus_per_node: 8

On a large or multi-partition cluster init lists every node it can see, so trim cluster.nodes to the GPU nodes you actually want to qualify before running doctor --all-nodes or test --all.

SSH clusters (no scheduler)

Any set of boxes you can ssh into is a cluster. Set remote.mode: ssh and list the nodes; entries accept user@host:port, so cloud pods with one public port per node (RunPod, Vast) work directly:

cluster:
  nodes:
    - root@203.0.113.7:45663
    - root@203.0.113.9:11542
remote:
  mode: ssh
  ssh_key_path: ~/.ssh/id_ed25519

doctor --all-nodes and test --all --parallel 8 then drive every node over ssh. Validated end to end against live multi-node rigs.

SkyPilot

There are two ways to combine sixtytwo with SkyPilot:

# 1. Qualify a cluster you launched on any cloud SkyPilot supports
sixtytwo skypilot qualify my-cluster --mode quick

# 2. Provision sixtytwo's own GPUs through SkyPilot with `cloud: sixtytwo`
pip install 'sixtytwo-cli[skypilot]'
sixtytwo skypilot install              # register sixtytwo as a SkyPilot cloud
sky show-gpus --cloud sixtytwo

sixtytwo skypilot install is opt-in. It wires the adapter into the active virtualenv so sky launch can reserve reliability-backed GPUs with cloud: sixtytwo. Run sixtytwo skypilot --help for the full set of subcommands.

Topology and fabric health

The full test suite maps how a node is actually wired and checks the fabric it sits on:

• GPU topology: NVLink/XGMI link matrices (nvidia-smi topo, rocm-smi showtopo). PCIe-only or mixed-link nodes are flagged before they slow a collective.
• InfiniBand fabric: ibnetdiscover maps which leaf switch each node hangs off (one fabric-wide sweep per cluster, reused across nodes), and port counters are classified so congestion reads as congestion instead of a cable fault.
• GPU to NIC affinity: an NCCL probe across all GPUs verifies rail-optimized placement.
• Diagnosis with opt-in repairs: failures map to a remediation plan (restart fabric manager, reset wedged GPUs, fix clock sync). Everything is off by default and dry-run first; fixes only execute when recovery.in_place_fixes.enabled is set, stop at the first verified repair, and never escalate to a disruptive action without a fresh diagnosis.

Hosted dashboards

Stream telemetry to sixtytwo.ai and watch the fleet from a browser:

sixtytwo connect --source ssh --name my-cluster   # register (token from the account page)
sixtytwo monitor                                  # stream per-GPU telemetry
sixtytwo install-agent                            # or install as a systemd service

The cluster page shows live per-node tiles, a node x GPU utilization heat map, and a 24h history. On Slurm clusters the agent also attributes GPU hours to users and accounts, so the chargeback report (with CSV export) fills itself in.

Metrics for Grafana

sixtytwo metrics serve exposes a Prometheus /metrics endpoint backed by the local trust registry: trust scores, fault counters, recovery downtime, and per-check status. sixtytwo metrics export-grafana prints a curated dashboard you can import as is.

sixtytwo metrics serve --host 0.0.0.0 --port 9620
sixtytwo metrics export-grafana --output sixtytwo-overview.json

License

Commercial. The bundled LICENSE governs use. Learn more at sixtytwo.ai.