svtop

svtop monitors NVIDIA GPU usage across multiple SSH servers. It provides an interactive TUI for humans and stable, plain-text or JSON commands for scripts and agents.

The project keeps its state in ~/.svtop, runs no daemon, and does not require a central server.

Features

• Concurrent multi-server GPU polling over SSH
• Real-time TUI with aggregate server rows and expandable per-GPU details
• Agent-readable inventory via svtop list --format json
• Portable, versioned configuration export and import
• Verified local-machine-to-server SSH key setup and status checks
• OpenSSH config fragment generation without modifying ~/.ssh/config
• Dual-pane local/server transfer UI using rsync or an OSS cache
• Per-server proxy jump chains
• Password and private-key authentication
• Local configuration and connectivity diagnostics

svtop intentionally does not configure server-to-server keys or a full-mesh trust network. SSH setup is limited to the machine running svtop connecting to configured servers.

Install

pip install svtop

For local development:

pip install -e .

Check the installed version and available commands:

svtop --version
svtop --help

Monitor and configure

Launch the monitor:

svtop
svtop run --interval 2.5 --timeout 15

Manage server profiles interactively:

svtop config

Configuration is stored in ~/.svtop/servers.json and ~/.svtop/settings.json. Server selectors accepted by non-interactive commands are an exact server name or ID. Use an ID when names are ambiguous.

List servers and GPUs

Probe all configured servers and print a fixed-column table:

svtop list
svtop -l

Print the stable JSON contract intended for agents and scripts:

svtop list --format json

The document uses kind: "svtop.inventory" and schema_version: 1. It contains server IDs, endpoints, authentication metadata, probe status, GPU metrics, and a summary. Passwords, private-key contents, and private-key paths are never included. Error messages are sanitized before output. An unavailable GPU metric is represented as JSON null and as - in the table instead of dropping the GPU row.

Useful inventory options:

# Do not connect; return configured endpoints only.
svtop list --offline --format json

# Select one or more profiles in the requested order.
svtop list --server train-a --server 0f3a... --format json

# Make partial probe failure visible to automation with exit code 3.
svtop list --strict --timeout 20 --jobs 4 --format json

Per-server status values are online, offline, fingerprint_mismatch, unknown_host_key, auth_error, unreachable, or probe_error. A valid probe from a host with no NVIDIA GPUs is online with gpus: []. Without --strict, reachable and failed servers are reported together and the command exits successfully.

Export and import configuration

Create a private portable export:

svtop export svtop-config.json
svtop -e svtop-config.json

Exports use kind: "svtop.config" and schema_version: 1, are written atomically with mode 0600, and do not overwrite an existing file unless --force is supplied. Symlink destinations are rejected.

Credential migration rules:

• A profile originally added with a password is exported with that retained password even if svtop later switched its active login to a key. This lets the destination machine bootstrap its own key.
• If such a profile no longer has its saved password, export fails instead of silently creating a broken migration file. --allow-incomplete records the missing requirement explicitly.
• A profile originally added with a private key exports only a portable key hint and an external_private_key requirement. Private-key contents are never copied into the export.
• --redact-secrets removes saved passwords for a shareable inventory-style export. A redacted export cannot restore those credentials by itself.
• Secret-bearing stdout export is blocked unless --unsafe-stdout is given.

Optionally verify retained password credentials before writing:

svtop export svtop-config.json --verify-auth

Inspect an import without changing local state:

svtop import svtop-config.json --dry-run --format json

Apply it with an explicit conflict policy:

svtop import svtop-config.json --conflict fail
svtop import svtop-config.json --conflict skip
svtop import svtop-config.json --conflict replace
svtop import svtop-config.json --conflict rename

To replace the complete local inventory, use both confirmation flags:

svtop import svtop-config.json --replace --yes

An applied import writes the server and portable settings together and creates private timestamped backups of existing files. Invalid schemas, duplicate IDs, duplicate endpoints, and malformed profiles are rejected before any write.

Treat a non-redacted export like a password file. Transfer it over a secure channel and remove it when migration is complete.

Local-to-server SSH access

Open a configured server by name or ID:

svtop ssh train-a
svtop ssh open train-a

Install this machine's public key on selected servers, verify a fresh key-only connection, and update only profiles that passed verification:

svtop ssh setup train-a train-b
svtop ssh setup --all --jobs 8

By default, svtop reuses a suitable local key. If none exists it creates a dedicated Ed25519 key under ~/.svtop/keys. Use --key PATH to select another private key. A generated key is explicitly reported as passwordless together with its path and SHA-256 fingerprint. Password prompts and unknown-host confirmations are collected before concurrent work begins; use --non-interactive for unattended runs with already saved credentials.

Verify key-only access without installing anything:

svtop ssh status
svtop ssh status train-a --format json

Generate an OpenSSH fragment for review or manual inclusion:

svtop ssh config

This command only prints text. It never edits ~/.ssh/config.

Interactive ssh open and ssh setup show the unknown host fingerprint and ask before recording it. Non-interactive checks reject unknown keys by default; --accept-new trusts only previously unseen keys. Paramiko and OpenSSH both read ~/.svtop/known_hosts and the normal user known-hosts file, while newly accepted svtop keys are written to the managed file. Changed keys still fail. Structured commands report a changed key as fingerprint_mismatch, and interactive ssh open prints the same classification before returning the SSH process exit code. Avoid --accept-new when you cannot independently verify the endpoint.

Diagnostics

Check local tools, profiles, and key files:

svtop doctor
svtop doctor --format json

Also test key-only connectivity:

svtop doctor --network --server train-a

Diagnostic JSON uses kind: "svtop.ssh-doctor" and schema_version: 1.

Exit codes

Non-interactive commands use these codes:

Code	Meaning
0	Completed successfully
2	Command-line syntax or option error reported by the argument parser
3	Partial failure or warning, including list --strict probe failures
4	Invalid input, configuration, selection, or unsafe operation refused
5	Complete SSH setup/status failure or diagnostic failure

An SSH subprocess may return its own non-zero exit code from svtop ssh open. JSON result documents for SSH setup, status, and doctor also include an exit_code field.

Transfer mode

Open the dual-pane transfer UI:

svtop transfer
svtop transfer --parallel 4
svtop transfer --relay oss
svtop transfer --relay rsync

Quick keys:

• Top bar Left / Right: choose Local or a configured server
• Pane path bar: paste a path and press Enter
• Tab: switch active pane
• 1 / 2: cycle left/right endpoint
• d: toggle transfer direction
• Enter or double-click: open a directory
• Space: mark or unmark an item
• F5: transfer the current or selected items
• Backspace: go to the parent directory
• r: refresh the active pane

The oss relay uses ossutil cp through the configured OSS cache. The rsync relay uses direct transfer for local/server operations and local staging for server/server operations. Transfer workspace state is saved in ~/.svtop/settings.json.

Remote requirements

• SSH access to each server
• nvidia-smi in the remote PATH for GPU monitoring
• rsync locally and on relevant servers for rsync transfer mode
• ossutil wherever OSS transfer mode is used
• A local OpenSSH client when proxy jump chains or shell access are used

Proxy jump chains are configured per profile as comma-separated user@host[:port] entries, for example ops@jump-a,core@jump-b:2222.

Security

Password authentication stores the password in plaintext in ~/.svtop/servers.json with mode 0600. Prefer key authentication for normal operation, restrict access to the account running svtop, and protect any non-redacted export as a secret.

svtop never exports private-key contents, never automatically modifies the user's OpenSSH config, and never installs keys between managed servers.