remote-cluster-controller 0.3.2

Remote cluster controller: push, pull, run, shell over SSH + rsync

rcc

Remote cluster controller. Generalizes the classic ssh host "cd dir && cmd" + rsync project/ host:dir/ workflow into a small CLI with per-project config.

Install

uv tool install remote-cluster-controller
# or
pipx install remote-cluster-controller

Both install the rcc command.

Quickstart

cd my-project
rcc init
# edit .rcc/config.toml to set host and remote_dir
rcc push --dry-run
rcc push
rcc run -- nvidia-smi
rcc run -t -- htop
rcc run -s 'squeue -u $USER | head'   # shell snippet: pipelines, $vars survive
rcc run --env EPOCHS=10 -- python train.py
rcc run --docker -- python train.py   # run through the profile's remote Docker target
rcc shell
rcc pull jobs/results/                 # fetch just that subtree (bypasses rccignore)
rcc status
rcc close

Configuration

.rcc/config.toml (per project, gitignored):

default = "myhost"

[profiles.myhost]
host = "myhost"
remote_dir = "/abs/path/on/remote"
# Bastion hop / explicit key (encapsulate instead of an SSH-config dance):
# proxy_jump = "bastion.example.com"
# identity_file = "~/.ssh/id_ed25519"

# Per-profile env defaults, honored by `rcc run` and `rcc job submit`:
[profiles.myhost.env]
TRITON_CACHE_DIR = "/scratch/cache"

# Remote-owned paths to protect from --delete/--mirror (rccignore-with-teeth):
# keep_remote = ["logs/", "cache/", "*.safetensors", "last_service.env"]

# Default port-forward for `rcc tunnel`:
# [profiles.myhost.tunnel]
# remote_port = 8080
# local_port = 18080        # optional; defaults to remote_port
# remote_host = "localhost" # optional; set to a compute node to reach a service there

# Optional Docker target for `rcc run --docker`.
# Docker runs on the SSH remote host, so images and mount sources are resolved
# there, not on your laptop.
[profiles.myhost.docker]
image = "pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime"
workdir = "/workspace"
gpus = "all"
shm_size = "16g"
network = "host"

[[profiles.myhost.docker.mounts]]
source = "{remote_dir}"
target = "/workspace"
mode = "rw"

[[profiles.myhost.docker.mounts]]
source = "/scratch/datasets"
target = "/data"
mode = "ro"

[profiles.myhost.docker.env]
HF_HOME = "/workspace/.cache/huggingface"

Per-command overrides: --profile, --host, --remote-dir. Excludes: edit .rcc/rccignore (gitignore syntax).

Machine-readable config

rcc config prints the resolved profile. For automation:

rcc config --json                    # stable JSON object of the resolved profile
rcc config --get host                # single value, unquoted
rcc config --get remote_dir
rcc config --get env.TRITON_CACHE_DIR
rcc config --get tunnel.remote_port
rcc config --get docker.image
rcc config --get docker.workdir
rcc config --get docker.env.HF_HOME

This lets wrappers drop fragile sed/regex parsing of free text.

Sync: push / pull

rcc push                             # whole project → remote
rcc push jobs/sweep                  # just a subpath (bypasses rccignore)
rcc pull                             # whole remote → project
rcc pull jobs/sweep/                 # fetch a subtree even if it's in rccignore
rcc pull jobs/sweep/ ./out/          # ...reconstructed under ./out/
rcc push --no-ignore                 # bypass rccignore for a whole-project transfer
rcc push --include '*.bin'           # extra include globs

Output: quiet by default (issue #6)

push/pull no longer dump rsync's progress bar and stats block at you. A real transfer prints a single summary line:

$ rcc push
Pushed 42 files (12.34MB sent) → alpha:/srv/app/
$ rcc pull
Pulled 1 file (35B received) → alpha:/srv/app/

Pass -v/--verbose (a root flag, before the subcommand) to see everything rsync prints — the live progress bar, per-file list, and full stats:

$ rcc -v push
sending incremental file list
...

--dry-run output is unaffected (it was already concise). On failure the rsync stderr tail is surfaced so you don't need --verbose just to see why.

Deletion safety

The default is non-destructive. There are two distinct deletion modes:

• --delete — bounded sync: deletes remote/local files inside the non-ignored transfer scope that have vanished from the source. Excluded files are spared.
• --mirror — dangerous full mirror: also removes rccignore-excluded files (--delete-excluded). Use this only when you truly want local→remote mirroring.

--keep-remote GLOB (repeatable) and the profile-level keep_remote = [...] list protect job-owned paths — logs/, cache/, *.safetensors, last_service.env — from both modes. The guard rail lives in the tool, not your memory.

rcc push --delete                     # safe bounded sync
rcc push --mirror --keep-remote 'logs/'   # full mirror, but never touch logs/

Clear dry-runs

--dry-run no longer buries deletions in a wall of f.f..... lines. Deletions get their own section so a destructive transfer can't hide:

$ rcc pull --delete --dry-run
Dry run for /srv/app/
Would DELETE:
  - jobs/old.txt
  - cache/v3/
Would RECEIVE (download):
  + jobs/new.txt

Running commands: run

rcc run -- nvidia-smi                 # exec-style: tokens passed verbatim
rcc run -s 'a && b | c'              # shell snippet: pipelines, $vars, quotes survive
rcc run --env EPOCHS=10 --env GPU=0 -- python train.py   # remote env (repeatable)
rcc run --env-file .env -- python serve.py               # load KEY=VAL lines
rcc run --cwd /scratch/run -- bash job.sh                # override working dir
rcc run -t -- htop                    # allocate a PTY

--env layers on top of the profile's [env] defaults, removing the injection-fragile KEY=VAL cmd hand-quoting wrappers used to need.

Docker execution on the remote host

rcc run --docker wraps the same remote command in docker run or docker exec on the SSH remote host. This keeps the daily command terse while the stable image, mounts, container workdir, GPU flags, and container env live in .rcc/config.toml.

rcc run --docker -- python train.py --epochs 10
rcc run --docker -s 'python train.py 2>&1 | tee train.log'
rcc run --docker-image my-debug-image -- bash
rcc run --docker-container devbox --docker-workdir /workspace -- python -V

With the profile above, rcc run --docker -- python train.py executes roughly:

ssh myhost 'cd /abs/path/on/remote && docker run --rm \
  --gpus all \
  --shm-size 16g \
  --network host \
  --mount type=bind,src=/abs/path/on/remote,dst=/workspace \
  --mount type=bind,src=/scratch/datasets,dst=/data,readonly \
  --workdir /workspace \
  --env TRITON_CACHE_DIR=/scratch/cache \
  --env HF_HOME=/workspace/.cache/huggingface \
  pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime \
  python train.py'

Important semantics:

• The Docker image is resolved by Docker on the remote machine. If the image only exists locally, push it to a registry or load it on the remote first.
• Docker mount sources are remote-host paths. {remote_dir} expands to the resolved profile remote_dir; {cwd} expands to the effective remote working directory.
• Profile [env] values are still exported in the remote shell and are also passed into the container. [docker.env] is container-only and wins over profile env. CLI --env wins over profile env, and --docker-env wins last inside the container.
• --docker-image and --docker-container are mutually exclusive. Image mode uses docker run --rm; container mode uses docker exec.
• --docker is currently foreground-only; --detach --docker is rejected until the tmux background runner grows container support.

For one-off overrides:

rcc run --docker-image img \
  --docker-workdir /app \
  --docker-mount '{remote_dir}:/app:ro' \
  --docker-gpus all \
  -- pytest

Streaming + captured output together (tee)

A long-standing snag for wrappers: plain streaming shows output live but discards the text; plain capture buffers everything so the call looks hung. --result-json PATH tees — output streams to your terminal and rcc writes a structured result to PATH on exit:

rcc run --result-json /tmp/r.json -- srun python train.py   # live output now
# then: {"returncode": 0, "stdout": "...", "stderr": "...", "command": [...]}

A wrapper using subprocess.run(argv) (streaming) gets the live output, then reads /tmp/r.json for the authoritative exit code + full captured text.

Detached runs for non-SLURM hosts (--detach + rcc bg)

rcc job is Slurm-only; on a plain SSH host there used to be no built-in way to launch a long command detached and reattach. Now there is — backed by tmux, so it survives disconnects:

rcc run --detach --name sweep -- python train.py   # launch in a tmux session
rcc bg ps                                          # list rcc-launched sessions
rcc bg logs sweep -f                               # tail the captured log
rcc bg attach sweep                                # attach to the live session
rcc bg wait sweep                                  # block until it exits (exit code)
rcc bg stop sweep                                  # kill it

State lives under remote_dir/.rcc-runs/<name>.{log,status}, so it survives disconnects and can be pulled back. Requires tmux on the remote (auto-detected; exit 127 with a hint if missing). The --name is optional (auto-generated); logs/attach/wait/stop default to the sole running session if you omit it. This removes the hand-rolled tmux orchestration every non-SLURM consumer needed.

Port-forwarding: tunnel

rcc tunnel                            # uses the profile [tunnel] defaults
rcc tunnel --remote-port 8080         # ...or specify explicitly
rcc tunnel --remote-port 8080 --remote-host head01 --local-port 18080

rcc tunnel opens a local port-forward reusing rcc's SSH ControlMaster (the same connection status/close manage), collapsing the manual ssh -L 8080:head:8080 host tail every workflow used to end with. Ctrl-C closes it.

Slurm jobs (rcc job)

For HPC login nodes running Slurm, rcc job wraps the common verbs so you never have to shell-quote a --format= value, a $USER, or a pipeline (the friction that motivated this command — see issue #1).

rcc job submit train.sh --extra-env EPOCHS=10   # sbatch, prints the JOBID
rcc job submit train.sh -W --dependency afterok:524614   # block until done + chain after a prior job
rcc job list                                     # squeue for your user
rcc job list --json                              # ...as structured records for wrappers
rcc job status 524614                            # sacct -j <id> (fixed format)
rcc job status 524614 --json                     # ...as structured records (main + steps)
rcc job tail 524614 -f                           # tail -f slurm-524614.out
rcc job wait 524614                              # poll until done, exit with job's code
rcc job wait 524614 --on RUNNING                 # ...or until it reaches a state
rcc job cancel 524614                            # scancel

Notes:

• The submit/list/status/cancel verbs sniff for sbatch on the remote and exit 127 with a hint on non-Slurm hosts; tail skips the check (tail is universal).
• list and status use a fixed, readable --format=; you never type one. Their --json variants return pipe-delimited output parsed into structured records (issue #2 P2). list --json emits one record per active job; status --json emits one record per row sacct returns (the main job plus each step — distinguished by the JobID field, e.g. 524614.batch). Each row with a State gains an ok flag, and each row with an ExitCode RETURN:SIGNAL gains parsed integer exit_code/signal fields, so an OOM-killed step shows up as "exit_code": 137, "signal": 9.
• job submit -W/--wait blocks until the job finishes and rcc exits with the job's exit code — closing the submit→monitor loop in one command. --dependency passes --dependency=<TYPE:JOBID> straight to sbatch for chaining (e.g. afterok:524614).
• job tail reads slurm-<JOBID>.out inside remote_dir by default. Pass --file NAME for jobs that set a custom --output.
• job wait polls squeue until the job leaves the queue, then classifies the final state via sacct: exits 0 on COMPLETED, the job's exit code on failure, 124 on timeout. --on STATE returns early (e.g. once RUNNING). Unlike sbatch --wait, it surfaces the job's outcome without minutes-long silence.

For one-off Slurm commands that aren't wrapped, use the shell-string mode of rcc run, which also sidesteps the quoting problem:

rcc run -s 'sacct -j 524614 --format=JobID,State,Elapsed,ExitCode,Reason'

Roadmap / not yet implemented

Tracked in the issue tracker; the four originally-deferred asks from issues #2/#3 are now implemented (run tee, detached tmux runs, job wait, status --json). What remains is the Slurm-specific parsing work, which needs a live scheduler to validate:

• Rank-aware Slurm logs (job tail --rank N, distributing across the job's node list) — issue #2 P2. Needs a multi-node job to validate the node/rank mapping; deferred until testable against a live scheduler.

rcc config --json, rcc status --json, and rcc job list/status --json are available.

Releasing (maintainers)

Releases are published to PyPI via trusted publishing (OIDC): no API tokens are stored anywhere. A GitHub Actions workflow (.github/workflows/release.yml) builds the sdist + wheel and publishes under the pypi environment, where PyPI trusts it by repository + workflow filename + environment.

One-time setup (PyPI web UI)

Only the PyPI project owner can do this. Go to https://pypi.org/manage/project/remote-cluster-controller/settings/publishing/, under Add a publisher → GitHub, and register:

Field	Value
Owner	ResearchComputer
Repository	remote-cluster-controller
Workflow name	release.yml
Environment name	pypi

(The Environment name must match the environment: pypi line in the workflow.)

Until this is done, the first publish run will fail at the Publish to PyPI step with a clear PyPI error about an unknown publisher.

Optional hardening: under the repo's Settings → Environments → pypi, add a required reviewer so every publish needs a manual approval.

Cutting a release

The git tag drives the release and must match the version in pyproject.toml (a workflow step enforces this, so a mismatch fails fast instead of publishing the wrong version):

# 1. bump version in pyproject.toml and src/rcc/__init__.py
# 2. commit, then:
git tag v0.3.2
git push origin v0.3.2

Pushing the tag triggers the workflow. Watch it:

gh run watch

There is also a workflow_dispatch trigger (Actions tab → Run workflow) for re-runs; it publishes whatever version is in pyproject.toml on the selected branch.

Design

See the design docs for the full picture:

• docs/superpowers/specs/2026-04-23-rcc-design.md — original v1 design.
• docs/superpowers/specs/2026-06-28-rcc-issues-2-3.md — issues #2 & #3: automation ergonomics, sync safety, lifecycle verbs, and a condensed usage tour.