multiSSH3

Introduction

multiSSH3 is a fast, flexible way to run commands and move files across many hosts in parallel, while watching everything live.

Use it from the CLI for quick fleet actions or import it as a Python module for automation and orchestration.

multiSSH3 understands host groups from env files, expands ranges, reuses SSH sessions, and presents clean outputs for human or machine (json / table).

Demo Video

https://github.com/user-attachments/assets/5121bb20-4b32-4000-8081-2ad39571781e

Screenshots

Running date on host group all

Running free on host group all

Running ping on host group all

Curses Help Window within curses display

Curses single window mode with key '|'

Running free -h on host group all with json output

Running free -h on host group all with greppable table output

Broadcasting ./test.txt to host group us at /tmp/test.txt

Syncing /tmp/test.txt from local machine to host group all (at /tmp/test.txt)

Gathering /tmp/test.txt from host group all to local machine at /tmp/test/<hostname>_test.txt

Running power status using ipmitool on host group all with IPMI interface prefixing

Note: DB6 in image does not have IPMI over ethernet connection. It had failed back to running ipmitool over ssh.

Running date; df -h on host group all showing the N-Host Diff (threshold 0.6, the default then; it is 0.9 now)

Running echo hi on host range <hostname>[10-20]*

Running echo hi on host range 127.0.0.1-100

Summary shown with some hosts reporting error

Command syntax / output / runtime comparism with Ansible ad-hoc

Note: if you like to use ansible, you will likely be running playbooks. This comparison is just for showing ansible is not for running ad-hoc commands.

Command syntax / Runtime comparism with pdsh

Output comparism with pdsh

Highlights

• Run commands on many hosts simultaneously and asynchronously ( configurable max connections )

• Live interactive curses UI with per-host status. ( send input to all hosts asynchronously )

• Broadcast / gather files (rsync -> scp), with --file_sync syntaxic sugar

• Host discovery via env variables, files, ranges, and DNS; smart cached skip of unreachable hosts

• IPMI support with interface IP prefixing and SSH fallback

• Concurrency tuned for large fleets with resource-aware throttling

• Easily persist defaults via config files; ControlMaster config helper for speed

• Importable as a Python module for automation frameworks

• No client side code / dependencies! ( calling system ssh / rsync / scp )

• Support windows with openssh (server and client) !

Why use it?

• If you think ansible is too slow

• If you think ansible is too clunky / cluttered

• If you think pdsh is too complicated / simple

• If you think pdsh output is too messy

• See progress in real time, not after-the-fact logs

• Operate at scale without drowning your terminal

• Keep your host definitions in simple env files and ranges ( customizable groupable DNS )

• Drop-in for scripts: stable exit codes, compact summaries, and capable to produce machine-friendly output

Install

Install via

pip install multiSSH3

multiSSH3 will be in cli available as

mssh
mssh3
multissh
multissh3
multiSSH3

It also installs mssh-complete, the shell autocomplete engine behind mssh --completion (see Autocomplete).

Need Python 3.6+

Configuration

Config File Chain

multissh treat config files as definable default values.

Configed values can be inspected by simply running

mssh -h

to print the help message with current default values. Passwords are the exception: for -p and -ip it only says whether one is set.

Defaults are read from the following chain map stored as json files, top ones overwrite the bottom ones.:

The current directory is not searched: a cloned repo or an unpacked archive could otherwise change any setting, including which ssh runs. Use --config_file to load a config file from there; mssh prints a note when ./multiSSH3.config.json exists (read automatically up to 6.26).

Config and env files found on these default paths are only loaded when they are owned by you or root and nobody else can write to them (group write is fine when the group is your own private group). Others are skipped with a warning that shows the chmod fix. Files you name with --config_file, --env_file or --env_files are always loaded.

--store_config_file writes the file readable by you only (0600), as it can hold passwords. The /etc locations above are written 0644 so every user can read them. An existing file's permissions are only ever tightened, and a file whose permissions cannot be restricted is left untouched (the config is printed instead).

A config file that is a symlink (e.g. into a dotfiles repository) is written through the link; its backup (.bak, the default answer when the file exists) is a copy next to the link.

Each value is checked against the type of the setting's default. Values whose meaning is clear are converted ("true" / "false", "yes" / "no" for true / false settings, numbers written as strings, a list of host names for DEFAULT_HOSTS); anything else, such as "DEFAULT_TIMEOUT": "fifty" or an encoding Python does not know, is ignored with a warning naming the file and the key, and the setting keeps its value from the rest of the chain (no warning when a more specific file sets that key anyway). null for a string setting means its default (DEFAULT_HOSTS: null: no default hosts). A file that is not a JSON object is skipped the same way. --config_file applies a file exactly as the chain does, _sshPath, DEFAULT_ENV_FILE and ERROR_MESSAGES_TO_IGNORE included.

Generating / Storing Config File

To store / generate a config file with the current command line options, you can use

mssh --store_config_file [STORE_CONFIG_FILE_PATH]

--generate_config_file prints the same config (the built-in defaults, the config files mssh loaded and the command line options) as JSON to stdout and writes no file. --generate_config_file > PATH saves it, but with your shell's file mode (often readable by everyone), while --store_config_file writes it readable by you only.

--config_file PATH does not choose where --generate_config_file writes: it only names a config file to load. A PATH that does not exist gets Warning: Config file 'PATH' not found, ignoring it. and the config is still printed to stdout.

When the file exists, --store_config_file asks whether to overwrite it, move it to PATH.bak first (the answer after 10 seconds) or stop. Given --generate_config_file as well, it overwrites the file without asking and without a backup.

Without a path, --store_config_file writes to the config file mssh loaded, or to ~/.multiSSH3.config.json if none was loaded.

You can modify the json file or use command line arguments to update values and store them as defaults.

mssh --timeout 180 --store_config_file

will store

...
"DEFAULT_CLI_TIMEOUT": "180"
...

into ~/.multiSSH3.config.json (or the config file mssh loaded)

If you want to store password, it will be a plain text password in this config file. This will be better to supply it everytime as a CLI argument but you should really consider setting up public-key authentication.

On some systems, scp / rsync will require you use public-key authentication to work.

Host Key Checking

By default mssh does not check host keys. Every ssh, scp and rsync (through --rsh) it runs gets -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null: any host key is accepted and none is remembered, so a key that changes goes unnoticed (ssh's Warning: Permanently added ... line is left out of the output). This keeps large or often reinstalled fleets from stopping at host key questions, but mssh cannot tell a host from a machine pretending to be it. Someone who can redirect the connection (a forged DNS or ARP answer, a taken-over IP address) receives the commands you run and answers them, and with -p (sshpass) receives the password itself.

To have host keys checked, add this to your config file:

"SSH_STRICT_HOST_KEY_CHECKING": true

mssh then passes neither option, and ssh checks host keys against ~/.ssh/known_hosts as your ssh config says (StrictHostKeyChecking, ask by default in OpenSSH). Since mssh talks to many hosts at once, add their keys to ~/.ssh/known_hosts beforehand (after checking the fingerprints), or set StrictHostKeyChecking accept-new in ~/.ssh/config to trust a host's key on the first connection and refuse a different one later. There is no command line option for it; --store_config_file keeps the setting once it is in a config file. A script that imports multiSSH3 can set multiSSH3.SSH_STRICT_HOST_KEY_CHECKING = True.

--copy_id gives ssh-copy-id the same two options (see -ci, --copy_id), so new nodes need no host key answer; with SSH_STRICT_HOST_KEY_CHECKING true it checks host keys as your ssh config says.

SSH ControlMaster Configuration

Note: you probably want to set presistent ssh connections to speed up each connection events.

You can add the following to your ~/.ssh/config file to enable ControlMaster with a 1 hour persistence by running

mssh --add_control_master_config

# Added by multiSSH3.py to speed up subsequent connections (mssh --add_control_master_config).
# ssh keeps one connection per host open for an hour after its last use and shares it. Its socket
# goes in /run/user/<uid> while you have a login session, else in /dev/shm/multiSSH3-<uid> when that
# is a directory only you can write to (su, sudo, cron, containers); with neither, ssh does not share.
Match exec "/bin/sh -c 'u=$(id -u) || exit 1; test -O /run/user/$u -a -w /run/user/$u'"
  ControlPath /run/user/%i/ssh_sockets_%C
Match exec "/bin/sh -c 'u=$(id -u) || exit 1; test -O /run/user/$u -a -w /run/user/$u && exit 1; d=/dev/shm/multiSSH3-$u; test -d $d || mkdir -m 700 $d 2>/dev/null; test -O $d || exit 1; case $(ls -ld $d) in drwx?-??-*) ;; *) exit 1;; esac'"
  ControlPath /dev/shm/multiSSH3-%i/ssh_sockets_%C
Host *
  ControlMaster auto
  ControlPersist 3600
# End of the block added by multiSSH3.py

The block applies to every ssh you run, not only mssh's. ssh fails outright (exit 255, nothing run) when the directory of its ControlPath is missing, so each Match exec line checks its directory first:

1. /run/user/<uid>, the per-user directory systemd keeps while you have a login session.
2. Else /dev/shm/multiSSH3-<uid> (under su, sudo, cron and in containers): mssh's runtime directory there, created with mode 0700 when missing, and used only if it is yours and nobody else can write to it. ssh trusts whatever listens where it looks for its master and hands it your terminal.
3. Else ssh connects without sharing, as without the block (also on systems that have neither).

The sockets never go in your home directory: on clusters it is often NFS, GPFS or Lustre, where unix sockets may not work. The checks run in /bin/sh whatever your login shell (csh and fish too) and work with OpenSSH 7.4 and later. ssh runs each Match exec with your $SHELL -c, twice per ssh: that adds about 2 ms with bash, more with shells that read startup files for -c (tcsh reads .cshrc and /etc/profile.d/*.csh, fish its config.fish), and with a $SHELL that is not an executable every ssh fails. The checks also run when the autocomplete's background checker calls ssh -G; they only look at local directories (and may create /dev/shm/multiSSH3-<uid>).

Up to multiSSH3 6.31 the block was Host * with ControlPath /run/user/%i/ssh_sockets_%C, which makes every ssh fail where /run/user/<uid> is missing. Run mssh --add_control_master_config again: it replaces that block, and only it, keeping the rest of the file and its permissions as they were. Running it once more changes nothing.

IPMI Logins

With -M, --ipmi, mssh logs in to each host's BMC with one IPMI definition after another until one works. A definition is a dict with any of username, password, args, ipmi_method, interface_ip_prefix and retry_lockout_time (what -iu, -ip, -ia, -im, -mpre and -irlt set), and hostname_keyword: a definition with one is only tried on hosts whose name contains it.

A wrong login can count towards the BMC locking the account, so mssh tries each login once, and only logins you gave it, in this order:

1. The command line's, when it has any of -iu, -ip, -ia, -im, -mpre or -irlt: the DEFAULT_IPMI_* values with those options applied.
2. DEFAULT_IPMI_DEFINITIONS (config file), in order. A key a definition leaves out comes from the DEFAULT_IPMI_* values, or from the command line when it gives that option: -ia='-I lanplus' reaches every definition without its own args.
3. DEFAULT_IPMI_USERNAME / DEFAULT_IPMI_PASSWORD (config file), when step 1 did not try them.

When a definition of step 2 applies to the host, the login of step 1 or 3 is only tried if -iu or -ip is given, or if a config file sets DEFAULT_IPMI_PASSWORD and the login is not admin / admin. A configured password admin counts (root / admin is the login of some boards); a DEFAULT_IPMI_USERNAME without a password only fills in the definitions that leave their username out. admin / admin itself, the built-in login (which --generate_config_file and --store_config_file write out too), is only tried on a host nothing else is configured for, or when typed as -iu admin -ip admin. To try it after your logins, list it: {"username": "admin", "password": "admin"}.

A host that fails every login then gets ipmitool run on it over ssh (see -M, --ipmi), once: mssh does not run a command line it already ran on that host.

--store_config_file stores the command line's IPMI options as the DEFAULT_IPMI_* values and keeps your DEFAULT_IPMI_DEFINITIONS. A definition in the list with all seven keys equal to the DEFAULT_IPMI_* values or to the built-in ones is not a login of its own: it marks where the DEFAULT_IPMI_* login goes. mssh writes one there (the list starts with one, and --store_config_file puts it first when the command line had IPMI options, as they were tried first). Config files written by mssh 6.31 and earlier can hold such a copy of the built-in admin / admin definition: it is read the same way, so it no longer makes mssh try admin / admin before your login.

Environment Variables for Hostname Aliases

multissh3 is able to read hostname grouping / aliases from environment variables.

By default mssh reads env variables and recursive resolves them if specified with a hostname that is not be able to be resolved from /etc/hosts. This functions as a pseudo dns service for hostname grouping.

Use [0-9], [0-f], [a-Z] ranges in the hostname strings for range expansion.

Use , to separate multiple hostnames / hostname groups.

Hostname Resolution Logic

First, mssh will expand [a-b] ranges in the hostname strings.

• Check if an IPv4 range is given
  • Expand using ipv4 range expansion logic.
  • Return
• Check if range given is all numerical
  • Expand using numerical expansion logic.
  • Resolve hostnames
  • Return
• Check if range given is all hex characters
  • Expand using hex expansion logic.
  • Resolve hostnames
  • Return
• Else
  • Expand using alphanumeric expansion logic.
  • Resolve hostnames
  • Return

When hostname need to be resolved, mssh will check in the following order:

• Return if it is an ipv4 address
• Return if hostname is in the hosts file (/etc/hosts, or --host_file): its first IPv4 address there, else its first IPv6 address
• If -C, --no_env is not specified and hostname is in current terminal environment variables
  • Redo the whole range -> hostname resolution logic with the resolved env variable value.
• hostname is in map generated from env_file(s) specified by --env_files ENV_FILES
  • Redo the whole range -> hostname resolution logic with the resolved env variable value.
• Lookup using socket.getaddrinfo() to query dns server ( IPv4 preferred, IPv6 accepted ). Names are looked up in parallel.
• If the ssh config (ssh -G) maps the name to another HostName or through a ProxyJump / ProxyCommand, it is valid ( #IP# uses the HostName's address when it resolves ). A catch-all block such as Host * only counts when it gives the name a HostName that resolves (e.g. HostName %h.corp.example), so typos are still rejected.

TLDR:

ipv4 address expansion -> range expansion -> identify ipv4 -> resolve using environment variables ( if not no_env ) -> env map from env_files -> remote hostname resolution

An alias's value is split into hosts as the hosts argument is (,, spaces, ;, ...). An alias with an empty value (eu='') stands for no hosts.

Inside its own expansion, directly or through other aliases, an alias's name is not expanded again but taken as a host name, as bash does with aliases: web="web,web-b" runs on the host web and on web-b. When no host has that name, mssh reports the loop and runs on the rest: Error: alias loop through 'web' (web -> app -> web): 'web' is not a valid hostname or IP address!

Default Env Files

Because command environment variables take precedence over env files, you can specify -C, --no_env or set "DEFAULT_NO_ENV": true in config file to disable environment variable lookup.

Use -ef ENV_FILE, --env_file ENV_FILE to specify a single env file to replace the default env file lookup chain. ( Only this file will be used. )

Use -efs ENV_FILES, --env_files ENV_FILES to append files to the end of the default env file lookup chain. ( Files will be loaded first to last. )

Later files take precedence over earlier files.

The current directory is not searched, as files there could redirect host aliases. Use --env_files hosts.env to add one. mssh points this out when a host name it cannot find is defined in ./.env, ./host.env or ./hosts.env (read automatically up to 6.26).

Example Env Hostname File

An example hostname alias file will look like:

us_east='100.100.0.1-3,us_east_prod_[1-5]'
us_central=""
us_west="100.101.0.1-2,us_west_prod_[a-c]_[1-3]"
us="$us_east,$us_central,$us_west"
asia="100.90.0-1.1-9"
eu=''
rhel8="$asia,$us_east"
all="$us,$asia,$eu"

( You can use bash replacements for grouping. )

Hostname Range Expansion

mssh is also able to recognize ip blocks / number blocks / hex blocks / character blocks directly.

For example:

mssh testrig[1-10] lsblk
mssh ww[a-c],10.100.0.* 'cat /etc/fstab' 'sed -i "/lustre/d" /etc/fstab' 'cat /etc/fstab'

A range needs both ends: node[1-] or node[-3] is not expanded, and mssh rejects it as an invalid host name.

A host you give that names no host at all, such as a range counting down ( node[10-1], 10.0.0.20-10 ) or an empty alias, is an error like a name that does not resolve: mssh says why, counts it as a failed host and lists it in failed_hosts. When none of the hosts you gave names a host, mssh stops with Exiting: no hosts to run on without running anything. Empty groups inside an alias ( like eu='' in the example above ) are fine. If -X, --skip_hosts excludes every host, mssh warns.

Misc Features

Interactive Inputs

It also supports interactive inputs. ( and able to async boardcast to all supplied hosts )

mssh www bash

mssh cache all inputs and send them to all hosts when they are ready to receive inputs. What you type goes to the commands' input, not to ssh's own questions (see below).

Password, Passphrase and Host Key Questions

ssh asks for a password, a key passphrase or to confirm a new host key on your terminal. mssh lets it only when one command runs at a time without the curses display: -nw with one host, or -nw -m 1 to answer each host in turn. Otherwise ssh runs without a terminal and fails at once with its reason, such as Permission denied (publickey,password). or Host key verification failed.: behind the curses display its questions would be hidden and the run would never end, and several ssh asking at once would split what you type between them and could show a typed password on screen.

After the output, a Note: line on stderr names the hosts whose ssh wanted a password (or a key passphrase) it could not ask for. For many hosts, use ssh keys (ssh-agent for a key with a passphrase) or -p (needs sshpass), and with SSH_STRICT_HOST_KEY_CHECKING add new hosts' keys to known_hosts first. Ctrl-C still stops the commands: mssh stops them itself, as it does for -t.

Without a terminal, ssh would send each password it cannot ask for empty, three times per host, and every one is a failed login in the host's auth log (fail2ban). So mssh then gives ssh, scp and rsync -o NumberOfPasswordPrompts=0, except with -p and when SSH_ASKPASS_REQUIRE tells ssh to ask through an askpass program. A ProxyJump host's ssh does not get it: if that host may want a password, set NumberOfPasswordPrompts 0 for it in your ssh config.

Input is typed into the curses display, so it needs a terminal on stdin as well as on stdout. When stdin is redirected (< file, a pipe, a while read loop, xargs), mssh shows no curses display, leaves stdin unread, and commands get /dev/null as their input, as with --no_watch.

Piping the Output

The output is printed when the commands finish, so it can be piped like any command's:

mssh all 'dmesg' | less

If the reader exits before reading all of it (| head, or q in less), mssh stops quietly as other command line tools do: no further commands or --repeat runs start, and the exit code still counts the hosts that failed in what ran. If that happens before anything has run (the output of -h, -V, a usage error or --generate_config_file), the exit code is 141, as for other tools SIGPIPE ends.

Curses Window Size Control

By default, it will try to fit everything inside your window.

DEFAULT_WINDOW_WIDTH = 40
DEFAULT_WINDOW_HEIGHT = 1

While leaving minimum 40 characters / 1 line for each host display by default. You can modify this by using -ww WINDOW_WIDTH, --window_width WINDOW_WIDTH and -wh WINDOW_HEIGHT, --window_height WINDOW_HEIGHT.

It is also possible to modify the window size within curses display by pressing keys:

? or F1 : Toggle Help Menu   
_ or +  : Change window hight
{ or }  : Change window width
< or >  : Change host index  
|(pipe) : Toggle single host
Ctrl+D  : Exit               
Ctrl+R  : Force refresh      
↑ or ↓  : Navigate history   
← or →  : Move cursor        
PgUp/Dn : Scroll history by 5
Home/End: Jump cursor        
Esc     : Clear / hide view  

Esc clears the input line. On an empty input line it hides the live view; the commands keep running and their output is printed when they finish. The live view leaves out ssh's own notices, as the output at the end does ( ERROR_MESSAGES_TO_IGNORE ): mssh does not keep the hosts' keys, so ssh says Warning: Permanently added ... on every connection, and a quiet command's window showed only that. You can also toggle this help in curses by pressing ? or F1. The one-character keys (?, _, +, {, }, <, >, |) act only on an empty input line; after you have typed something they are typed like any other character, so a ? can still reach the hosts. F1 toggles the help at any time.

Command String Replacement

mssh will replace some magic strings in the command string with host specific values.

Note: #HOST# and #HOSTNAME# are the supplied hostname / ip before any resolution.

Note: Resolved name is the IP / Hostname with user appended and ip prefix applied. This is what got used to connect to the host.

When a Host Does Not Answer

mssh has ssh, scp, rsync's ssh and ssh-copy-id give up on a host that takes the connection but never answers (an overloaded or hung sshd), or that stops answering while its command runs (it went off the network):

• -o ConnectTimeout=30: when connecting and sshd's first words take longer than 30 s, the host fails with Connection timed out during banner exchange (older OpenSSH: ssh: connect to host ... Connection timed out), return code 255, and counts as unreachable. A file copy takes twice as long, mssh tries scp after rsync failed.
• -o ServerAliveInterval=15 -o ServerAliveCountMax=4: after about 60 s without an answer from a connected host, it fails with Timeout, server ... not responding., return code 255. A command that prints nothing for a long time is not affected: sshd answers the keepalives.

Change the values with DEFAULT_SSH_CONNECT_TIMEOUT, DEFAULT_SSH_SERVER_ALIVE_INTERVAL and DEFAULT_SSH_SERVER_ALIVE_COUNT_MAX in the config file; 0 leaves the option to ssh (without a keepalive interval, mssh does not pass its count either).

Your own values win. ssh uses the first value it gets and a command-line -o beats the ssh config, so mssh leaves an option out when you give it:

• with -ea for ssh or scp, e.g. -ea '-o ConnectTimeout=120' (with -ea '-F FILE', mssh reads that ssh config instead of the two below);
• in your ssh config, ~/.ssh/config or /etc/ssh/ssh_config and the files they Include, also for single hosts (Host slow* / ConnectTimeout 120). Only when these files mention one of the three options does mssh ask ssh -G about each host whether it sets it, which runs the config's Match exec lines once more. ConnectTimeout 0 there means no timeout for those hosts; ServerAliveInterval 0 cannot be told from not setting it.

A shared connection (ControlMaster, see SSH ControlMaster Configuration) does the keepalive of the ssh that opened it. One mssh opened gives up on a host that went away, and the next ssh connects anew (where ConnectTimeout applies). One that a plain ssh without ServerAliveInterval in your ssh config opened does not notice: commands through it wait until the kernel gives up on the connection, which takes about 15 minutes on Linux. Setting ServerAliveInterval in your ssh config covers those too.

Autocomplete

mssh completes hosts, options, option values and remote commands when you press TAB, in bash, zsh and fish.

mssh --completion install     # set it up for the shell you run this from
mssh --completion uninstall   # remove it again
mssh --completion bash        # or print the script (bash, zsh or fish)

Install writes the script to ~/.local/share/multiSSH3/ and adds a marked block that loads it to ~/.bashrc (bash) or $ZDOTDIR/.zshrc (zsh, which install asks, so a ZDOTDIR set in ~/.zshenv counts; ~/.zshrc when ZDOTDIR is not set); for fish it writes files under ~/.config/fish/ ($XDG_CONFIG_HOME/fish/ when that is set). It prints every file it changes, and leaves alone a script file in its way that mssh did not write. Uninstall removes exactly what install added. bash login shells (ssh logins, macOS terminals) read ~/.bash_profile, ~/.bash_login or ~/.profile instead of ~/.bashrc; those usually load ~/.bashrc, and install prints the line to add when yours does not.

A new block is appended to your rc file; changing or removing it replaces the file in one step, so a full disk or quota leaves it as it was. A link stays a link (the file it points to is the one changed), and the file keeps its permissions and owner (not ACLs or other extended attributes). If the block's # >>> mssh completion >>> or # <<< mssh completion <<< line is missing or repeated, install and uninstall change nothing and print the line numbers to fix by hand. When mssh may not change the file (read-only, or linked into a read-only store as home-manager does, or not a regular file, such as a link to /dev/null), or cannot replace it safely to change a block (bind-mounted, hard-linked, or another user's), it leaves the file as it is and prints the lines to add, replace or delete by hand.

• Hosts: host strings you typed before, env file aliases, /etc/hosts, ssh config Host entries, known_hosts and the ARP table. Large clusters are offered as ranges (node[0001-2000]) until what you typed narrows them down. Hosts mssh recently found unreachable come last (with the user this line gives them: root@node05 and node05 are told apart).
• Options and values: option names, and values you used before (-u, -t, ...). Paths complete as files. Passwords are never completed or shown, the ones in your config file included.
• Commands: commands from your mssh history, the rest of a command from this machine's own completion (mssh grp systemctl rest<TAB>), and the magic strings (#HOST#, ...). A command typed as separate words gets the next words of commands you ran, with the quotes they ran with ('ssh*' comes as \'ssh\*\', so the remote shell gets the same command), never a word holding a space: mssh runs each word as its own command when one of them holds a space, so type such a command in quotes (mssh grp "grep 'connection refused' /var/log/syslog").

Ranking uses the rest of the line on both sides of the cursor: after mssh gpu[01-16] " the commands you run on those hosts come first, and in the hosts of mssh ▮ "systemctl restart slurmd" the hosts you restarted slurmd on come first.

Completion never removes what you typed, prints nothing, and falls back to the shell's normal completion when anything is slow or missing.

In zsh, TAB needs zsh's completion system: compinit has to run when zsh starts, before or after the mssh block in ~/.zshrc (oh-my-zsh, prezto and the like run it, and so does Ubuntu's /etc/zsh/zshrc). mssh --completion install says so when ~/.zshrc does not seem to start it. Grey suggestions, warnings and Ctrl-R work without it.

It learns from an index of your history and hosts, kept in mssh's private runtime directory ($XDG_RUNTIME_DIR/multiSSH3, else /run/user/<uid>/multiSSH3, /dev/shm/multiSSH3-<uid> or multiSSH3-<uid> in the temp dir) and rebuilt in the background when its sources change. mssh --completion install builds it right away. A reboot clears it: the first TAB after one starts the build and, until the index is ready (about a second for a long history), offers options and what your shell history holds. mssh-complete rebuild rebuilds it now; mssh-complete eval replays your history and shows how much typing completion saves.

Grey suggestions

While you type an mssh line, the most likely rest of it shows in grey after the cursor: the rest of the line when one past invocation clearly fits, only the rest of the current word when only that is clear, or the start every known value shares when none is (what TAB would insert), nothing when mssh is not sure. → (or End, Ctrl-F) takes all of it, Alt-→ / Ctrl-→ / Alt-F one word. Enter runs only what you typed.

• zsh: built in; with zsh-autosuggestions loaded, mssh becomes its first strategy on mssh lines. With setopt sh_glob (or emulate sh) in ~/.zshrc, grey text comes only through zsh-autosuggestions, and zsh shows no warnings (below): zsh's add-zle-hook-widget, which both need, finds no hooks under that option.

• bash with ble.sh: built in, as ble.sh's first suggestion source.

• plain bash (4.4 or later): turn on the mssh line editor, which takes over after you type an mssh name and a space, and hands the line back to readline on any key it does not handle (↑, Ctrl-K, ...) or when you delete into mssh :

  mssh --completion install bash editor
  

  Installing again keeps it on; mssh --completion uninstall bash, then mssh --completion install bash, turns it off.

• fish: fish's own suggestions already use mssh's completion.

Ctrl-R

Ctrl-R on an mssh line opens a menu below the prompt with the candidates for the word under the cursor (hosts, commands for those hosts, option values), how often and how long ago you used them; Tab switches to whole past invocations. Type or paste to filter, ↑/↓ to move, Enter to insert, Esc to leave the line as it was. On any other line Ctrl-R does what it did before (readline's search, fzf, atuin, ...).

A paste only filters, nothing in it runs, whether or not the terminal marks it as pasted (bracketed paste). Unmarked pastes are told by timing: a line break with another one less than 50 ms behind it was pasted. So Enter pressed twice within 50 ms, or Enter and the next line's keys arriving together over a laggy link, filter instead of choosing (press Enter again). Other keys typed right after Enter follow the choice, as typed.

In plain bash, mssh takes over Ctrl-R only when it can call the old binding on other lines. It leaves Ctrl-R alone when Ctrl-R is unbound, and on bash 4.2 when Ctrl-R runs a shell command bound with bind -x (fzf's key bindings do this), as bash 4.2 cannot list such bindings. Ctrl-R then does what it did before on every line, mssh lines included: there is no menu, and without the line editor (bash 4.4 or later) the warnings are not shown either. TAB completion works as usual. On bash 4.2, Ctrl-R bound to readline's search or to a macro is taken over as on newer versions.

Warnings before Enter

While you type an mssh line, words already typed that mssh would trip over are marked, and one line under the prompt says why. Nothing is ever blocked: Enter runs what you typed.

• Unknown host: a name mssh would reject (no IP, alias or hosts-file entry, not in DNS, not mapped by your ssh config), with the host you probably meant: gpu017: unknown host — did you mean gpu17?
• Range surprises: a range or alias with members mssh would reject: node[1-16]: 9 of 16 hosts unknown (node1, node2, node3, …) — did you mean node[01-16]?. Braces the shell expands make several words of the hosts, and mssh runs the second as the command: node{1,2}: the shell makes 2 words of this; mssh would run node2 as the command — did you mean node1,node2?
• Unreachable host: one that failed in your recent mssh runs (will be skipped with -su), or that answered ping recently and stopped. Like mssh, the warning tells root@node05 (typed so, or with -u root) apart from node05.
• Conflicting options: two options mssh refuses together, an option given twice, or one that does nothing on this line (-iu does nothing: only used with --ipmi).
• Options inside an unquoted remote command: mssh reads its options anywhere on the line, so a word of an unquoted remote command that names an mssh option is that option, and any other word starting with - stops mssh with an error: -u is mssh's --username here — quote the command: 'journalctl -u slurmd'. Options after a quoted command (mssh web1 'uptime' -q) are mssh's, as meant, and are not marked. Unlike other warnings, this one also marks the word you are typing when it is the line's last word and already an mssh option (mssh all df -h), as Enter would run the line as it stands.

Hosts the shell rewrites before mssh runs ($NODES, $(cat hosts.txt), `...`, ~, !$) are not checked: mssh gets what the shell makes of them. In single quotes or after a backslash they reach mssh as typed and are checked as mssh reads them.

zsh, bash with ble.sh and the plain-bash line editor mark the words and show the reason; fish 4 prints the reason under the line after a space; fish 3 and plain bash without the editor list the warnings at the top of the Ctrl-R menu.

With zsh-syntax-highlighting before zsh 5.9 (or before its 0.8), source it last in .zshrc, as it asks: loaded later (a plugin manager's deferred loading), it removes the marks and the grey text's colour at every key.

Hosts are checked in the background by mssh-complete probe: DNS, ssh -G (which prints what your ssh config says about the name) and ping. It never connects or logs in to a host itself, so it cannot trip fail2ban. One exception comes from your ssh config: ssh -G runs the commands of Match exec lines, so a Match exec command that contacts the host (such as nc -z %h 22 to choose a jump host) runs for every host name you type, before Enter. Check ~/.ssh/config and /etc/ssh/ssh_config (and the files they Include) if fail2ban matters. The prober runs at low priority, as many checks at once as mssh's own max_connections allows. What it learns lives in mssh's runtime directory ($XDG_RUNTIME_DIR/multiSSH3, else /run/user/<uid>/multiSSH3, /dev/shm/multiSSH3-<uid> or the temp dir) and is gone after a reboot. It remembers when up to 10 000 hosts last answered ping, for a day (longer when MSSH_COMPLETE_PING_TTL or mssh's unreachable host expiry is longer); -uhe above that on the line finds no older answers. mssh-complete eval --diagnostics replays your history and lists the hosts mssh would reject.

Tested with bash 4.2 to 5.3, zsh 5.5 to 5.9 (with compinit loaded; zsh-autosuggestions 0.7.1; zsh-syntax-highlighting 0.7.1 and 0.8.0), fish 3.3 to 4.0 and the 2026-09-08 ble.sh nightly. Since 6.28 mssh records hosts in its history as typed; older history lines hold them expanded.

After upgrading mssh, run mssh --completion install again to get the new shell glue. Glue from an older mssh keeps working, without the newer features.

The plain-bash editor binds the space key, so every space typed in bash runs a tiny shell function; Ctrl-R becomes a key macro (the binding it replaced is called for other lines). Uninstall restores both in new shells. Such a function also fires bash's DEBUG trap, which some tools take for the start of a command: a command timer kept with the DEBUG trap (as some prompts keep theirs) counts from the first space typed instead of from Enter, and the preexec hooks of bash-preexec (used by atuin and iTerm2's shell integration) may run at that space, before the line is complete.

In plain bash (without ble.sh), Ctrl-R and the line editor take their keys at the first prompt, through PROMPT_COMMAND. Where PROMPT_COMMAND is read-only (some audit setups on shared clusters make it so), they are left out without a message; TAB completion still works.

Options

Here details the options for multiSSH3 6.31 @ 2026-10-13

Options can go before, between or after the hosts and the command words. -- ends the options: the word after it is the hosts, unless they came before it, and every later word is a command word, even one that starts with -:

mssh web1 -- ls -la              # runs "ls -la" on web1 (without "--", -la is an unknown option)
mssh -q -- web1 grep -r TODO /srv

This is how Python 3.12.8+ reads --, and mssh reads it the same way on every Python. Up to mssh 6.31, older Pythons (before 3.12.8, and 3.13.0) read the options after a -- again when it came before the hosts: mssh -- web1 -u root set the username there.

-u, --username USERNAME | DEFAULT_USERNAME

• You can specify the username for all hosts using this option and / or specifying username@hostname per host in the host list.

-p, --password PASSWORD | DEFAULT_PASSWORD

• You can specify the password for all hosts using this option. Although it is recommended to use SSH keys / store password in config file for authentication.
• -p needs sshpass, which types the password when ssh asks for it and answers nothing else. When ssh asks again or asks about a host key, sshpass gives up without a word, and mssh says why next to the return code: 5 the password was not accepted, 6 ssh asked whether to trust a host key that is not in known_hosts (the host's own with SSH_STRICT_HOST_KEY_CHECKING true, else a ProxyJump host's), 7 known_hosts has a different key for the host's IP address. A command of yours that prints nothing can exit with the same codes, so the line says unless the command itself exited with N; with SSH_STRICT_HOST_KEY_CHECKING false and no ProxyJump (or ProxyCommand) in the ssh config, 6 and 7 can only be the command's and mssh adds nothing.
• Warning: by default mssh does not check host keys, so the password goes to whichever machine answers at the host's address. See Host Key Checking.
• Note: a password given on the command line is plain text. It can be seen by other local users (e.g. in ps and your shell history), is echoed in the > mssh ... line mssh prints at start, and is recorded in the history file (-hf, default ~/.mssh_history) so the command can be replayed (an IPMI password, -ip, is not). mssh creates a new history file readable by its owner only (mode 600); an existing history file keeps its permissions. Use -I, --no_history to keep a command out of the history, or set DEFAULT_PASSWORD in a config file (also plain text, so keep that file private) to avoid typing it on the command line.
• mssh does not show the passwords from your config file: -h and the autocomplete only say whether one is set, the > mssh ... line and the history leave them out, and the commands the TUI, --debug and --copy_id show have *** in their place. Only --generate_config_file (and --store_config_file when it cannot write the file) print the config itself, passwords included. Like a password typed on the command line, they reach sshpass and ipmitool as arguments, where other local users can briefly see them in ps.

-k, --key, --identity [KEY] | DEFAULT_IDENTITY_FILE

• You can specify the identity file or folder to use for public key authentication. If a folder is specified, it will search for a key file inside the folder.
• A ~ at the start of the path means $HOME, as for the rest of mssh, also where $HOME is not your home directory in the passwd file ( sudo -E, containers ): ssh is given the key file mssh found there. The history file keeps the ~.
• This option implies --use_key.
• If this option is not specified but --use_key is specified, it will search for identity files in DEFAULT_IDENTITY_FILE.
• If no value is specified, it will search DEFAULT_SSH_KEY_SEARCH_PATH.
• ssh then logs in with that key only (-o IdentitiesOnly=yes), not first with every key in your ssh-agent: with six or more there, sshd ends the login ( Too many authentication failures ) before the key is tried. A key held by the agent is still used when the file given is it or its .pub. A key file that does not exist is passed to ssh as before, which then uses the agent's keys.

-uk, --use_key | DEFAULT_USE_KEY

• Attempt to use public key authentication to connect to the hosts.
• Will search for identity file in DEFAULT_IDENTITY_FILE if --identity is not specified.

-ea, --extraargs EXTRAARGS | DEFAULT_EXTRA_ARGS

• Extra arguments to pass to the ssh / rsync / scp command. Put in one string for multiple arguments.
• The string is split into arguments as a shell would: quote an argument that contains spaces.
• For a file copy with rsync, ssh's options that take a value go to the ssh that rsync runs, and the other arguments to rsync: the option followed by its value as the next word (-o ConnectTimeout=5, -p 2222, -i FILE, -J HOST, -F FILE, -l USER, ...), or -oName=value and -pPORT. scp takes ssh's options itself; ssh's -p PORT is given to it as -P PORT.
• Example:

mssh -ea="--delete" -f ./data/ allServers /tmp/data/
mssh -ea="-o ProxyCommand='ssh -W %h:%p bastion'" allServers uptime

-11, --oneonone | DEFAULT_ONE_ON_ONE

• Run commands in one-on-one mode, where each command corresponds to each host, front to back.
• Hosts are taken in the order they are given. A range or an environment variable expands in ascending order in its place, e.g. b,a[1-2] pairs the commands with b, a1, a2.
• Hosts skipped with -X, --skip_hosts are left out before pairing, so give one command per remaining host. An unreachable host keeps its command, which is skipped.
• If command list length is not equal to expanded host list length, an error will be raised.

-f, --file FILE | None

• The file to be copied to the hosts. Use -f multiple times to copy multiple files.
• When file is specified, the command(s) will be treated as the destination path(s) on the remote hosts.
• A path that exists is copied as written, glob characters in its name included (report[1].txt); any other is expanded as a glob, so a quoted glob (-f '*.log') is expanded by mssh.
• Files are copied with rsync when it is installed on this machine, else with scp as with -W, --scp.
• rsync writes each file under a temporary name and renames it into place once it is complete, so a running program can be replaced and nobody reads a half-written file.
• A copy that is stopped (-t, Ctrl-C, a lost connection) leaves the old file in place and keeps the part already copied in a .rsync-partial directory beside it; the next copy of that file continues from there and removes the directory.
• rsync connects with the same ssh program and options as mssh's other commands, -k identity file included. A host given as an IPv6 address is written in brackets for rsync and scp (root@[fd00::5]:/dest/).
• As with rsync, a directory given with a trailing slash (./data/) copies its contents into the destination, one without (./data) copies the directory itself into it.
• A source that is a symbolic link to a file is copied as that file, under the link's name, as scp does (rsync's --copy-links). Links inside a directory stay links, so a link is copied as a link, with a warning before the copy, when it points to a directory or to nothing, or when a directory is copied in the same run (--copy-links would follow the links inside it too).
• A host where rsync could not run (rsync is not installed there or could not start, an rsync turned down mssh's options as macOS's rsync 2.6.9 does, or ssh failed to connect) is retried with scp, laying the files out as rsync would have: scp copies the entries of a ./data/ directory, and a lone directory goes inside the destination directory (scp fails if that directory does not exist, instead of creating it). Not retried: an empty ./data/ (scp has nothing to copy), a ./data/ with more entries than scp's command line holds (about 1 MB of names) and, in gather mode, a remote dir/ source going into a local directory that exists or with other sources (scp would copy the directory itself into it).
• rsync keeps extended attributes (-X). Where the destination cannot keep them (NFSv3, vfat, ...), rsync copies the files and fails with 23; mssh then runs rsync once more without -X, a quick pass that ends with the copy's own result, and warns that the attributes were not copied.
• Any other rsync failure, such as 23 (some files were not transferred) or 24 (some files vanished), is the host's result, with rsync's return code.
• rsync copies files as they are: permissions, times, owner and group (-a). Owner and group go by name: on a host that knows the same user name, the file belongs to that user whatever its uid there; only for a name the host does not know is the number kept, so a file of uid 1000 here belongs to whoever is uid 1000 there. Owners are only set when the receiving side is root: when mssh logs in to the hosts as root (with -G, when mssh runs as root here). Logged in as another user, the files belong to that user, keeping only groups that user is in. scp (-W, and the retry above) keeps no owners: new files belong to the user it logs in as.

-s, -fs, --file_sync [FILE_SYNC] | DEFAULT_FILE_SYNC

• Operate in file sync mode, sync path in <COMMANDS> from this machine to <HOSTS>.
• Treat --file <FILE> and <COMMANDS> both as source and source
• Destination path will be inferred from source path ( Absolute Path ).
• Each path is copied on its own, so a path that is a symbolic link to a file is copied as that file (see -f): a link such as /etc/localtime becomes a plain copy of the file it points to.
• -fs can also be followed by a path, a syntaxic sugar for specifying the path after the option.
• Example:

mssh -fs -- allServers ./data/ # is equivalent to
mssh -fs ./data/ allServers # is equivalent to
mssh -fs -f ./data/ allServers # is equivalent to
# if the cwd is at /tmp,
mssh -f ./data/ allServers /tmp/data/

-W, --scp | DEFAULT_SCP

• Use scp for copying files by default instead of trying rsync.
• Can speed up operation if we know rsync will not be available on remote hosts.
• scp's own rules apply: a trailing slash makes no difference, ./data/ copies the directory itself (to /dest/data when /dest exists).
• Remote paths are taken as written, as rsync takes them, spaces and characters such as ' or $ included; a remote source keeps its glob characters (/var/log/*.log). scp's legacy protocol (the default before OpenSSH 9.0, or -O) hands remote paths to the host's shell, so mssh puts a backslash before each character the shell reads specially. To know which protocol the local scp uses, mssh runs it once with -v and a stand-in for ssh that connects nowhere.

-G, -gm, --gather_mode | False

• Gather files from the hosts instead of sending files to the hosts.
• Will send remote files specified in <FILE> to local path specified in <COMMANDS>.
• Likely you will need to combine with the Command String Replacement feature to let each host transfer to a different local path.
• A source that rsync brings back as a symbolic link (it is one on the host, as /etc/os-release is on Debian) would point into this machine, or nowhere: mssh copies it again as the file it points to (--copy-links), as on push. A link to a directory, or one that points to nothing on the host, stays the link, with a warning (and, for a link to nothing, rsync's exit code 23). Not for a dir/ or glob source, nor when the local destination is itself a link.

-t, --timeout TIMEOUT | DEFAULT_CLI_TIMEOUT

• Idle timeout in seconds: a host's command is stopped when it has printed nothing for this long. A command that keeps printing is not stopped, however long it runs.
• When using 0, timeout is disabled.
• mssh stops only its own ssh ( or scp, rsync, ... ) on this machine, as Ctrl-C does. Without a terminal, the command on the host gets no signal: it runs on until it ends, or until it next writes output to the closed connection. The host shows Timeout! and what was stopped, with return code 124; a host stopped by Ctrl-C shows Ctrl C detected, Emergency Stop! and return code 130. To have the command end with its connection, give it a terminal with -ea=-tt ( for commands, not file copies ): its stdout and stderr then come as one stream.
• For CLI interface, will use DEFAULT_CLI_TIMEOUT as default.
• For module interface, will use DEFAULT_TIMEOUT as default.
• Without -t a host that does not answer still cannot hold up the run, ssh gives up on it (When a Host Does Not Answer).

-T, --use_script_timeout | False

• In CLI, use DEFAULT_TIMEOUT as timeout value instead of DEFAULT_CLI_TIMEOUT.
• This is to emulate the module interface behavior as if using in a script.

-r, --repeat REPEAT | DEFAULT_REPEAT

• Repeat the commands for a number of times.
• Commands will be repeated in sequence for the specified number of times.
• Between repeats, it will wait for --interval INTERVAL seconds.

-i, --interval INTERVAL | DEFAULT_INTERVAL

• Interval between command repeats in seconds.
• Only effective when REPEAT is greater than 1.
• mssh waits only between repeats: the first run starts at once, and nothing waits after the last one.

-M, --ipmi | DEFAULT_IPMI

• Use ipmitool to run the command instead of ssh.
• Will strip ipmitool from the start of the command if it is present. Without a command, runs power status.
• Will replace the host's resolved IP address header with DEFAULT_IPMI_INTERFACE_IP_PREFIX in this mode
• Ex: 10.0.0.1 + DEFAULT_IPMI_INTERFACE_IP_PREFIX='192.168' -> 192.168.0.1
• If a login fails, mssh tries the next IPMI definition that applies to the host (see DEFAULT_IPMI_DEFINITIONS in All Changable Config Options), then falls back to ssh: it logs in to the host itself (with -u / -p when given, at the -pre address rather than the -mpre one) and runs ipmitool <command> there. Without ipmitool on this machine it goes to ssh at once.
• Which IPMI logins it tries, and in which order: see IPMI Logins.

IPMI logins default to admin / admin. Unless -iu / -ip or a config file say otherwise, mssh runs ipmitool -U admin -P admin against every host's BMC. -u / -p (DEFAULT_USERNAME / DEFAULT_PASSWORD) are not used for IPMI logins, only for the ssh fallback. A wrong password can count towards a BMC's lockout, so give the right ones:

• on the command line: mssh -M -iu USER -ip PASSWORD ... (tried before any other definition), or
• stored: mssh -iu USER -ip PASSWORD --store_config_file, which writes DEFAULT_IPMI_USERNAME, DEFAULT_IPMI_PASSWORD and a matching DEFAULT_IPMI_DEFINITIONS.

-mpre, --ipmi_interface_ip_prefix IPMI_INTERFACE_IP_PREFIX | DEFAULT_IPMI_INTERFACE_IP_PREFIX

• The prefix of the IPMI interfaces. Will replace the resolved IP address with the given prefix when using ipmi mode.
• This will take precedence over INTERFACE_IP_PREFIX when in ipmi mode.
• Ex: 10.0.0.1 + -mpre '192.168' (or '192.168.') -> 192.168.0.1

-pre, --interface_ip_prefix INTERFACE_IP_PREFIX | DEFAULT_INTERFACE_IP_PREFIX

• The prefix of the for the interfaces. Will replace the resolved IP address with the given prefix when connecting to the host.
• Will prioritize IPMI_INTERFACE_IP_PREFIX if it exists when in ipmi mode.
• Ex: 10.0.0.1 + -pre '172.30' (or '172.30.') -> 172.30.0.1
• Only IPv4 addresses are rewritten. mssh connects to hosts without one by name, and says which before the run.

-iu, --ipmi_username IPMI_USERNAME | DEFAULT_IPMI_USERNAME

• The username ipmitool (or redfishtool) logs in to the hosts' BMCs with, in --ipmi mode.
• Defaults to admin. It does not fall back to -u / DEFAULT_USERNAME: see the note under -M, --ipmi.
• With -im ssh_ipmitool it is the ssh login name instead.

-ip, --ipmi_password IPMI_PASSWORD | DEFAULT_IPMI_PASSWORD

• The password ipmitool (or redfishtool) logs in to the hosts' BMCs with, in --ipmi mode.
• Defaults to admin. It does not fall back to -p / DEFAULT_PASSWORD: see the note under -M, --ipmi.
• With -im ssh_ipmitool it is the ssh password instead (sent with sshpass).
• Unlike -p, it is neither echoed in the > mssh ... line nor recorded in the history file, and when it is given, neither are -iu, -ia, -im, -mpre and -irlt: with the configured password they would make a login you did not give, tried on every BMC (each failed login counts towards the BMC locking the account). A replayed history line tries the configured logins, as --ipmi alone does; give the IPMI options again to use yours.

-ia, --ipmi_args IPMI_ARGS | DEFAULT_IPMI_ARGS

• Extra arguments for ipmitool (or redfishtool) in --ipmi mode, in one string, placed as they are before the command. -ea arguments only go to ssh / scp / rsync.
• Ex: mssh -M -ia='-I lanplus' bmc[1-4] 'power status' runs ipmitool -H <address> -U <user> -P <password> -I lanplus power status. Use =, as the value starts with -.

-im, --ipmi_method IPMI_METHOD | DEFAULT_IPMI_METHOD

• How --ipmi reaches a host:
  • ipmitool (default): runs ipmitool -H <address> -U <user> -P <password> [IPMI_ARGS] <command> on this machine.
  • redfish: runs redfishtool -S Always -a -r <address> -u <user> -p <password> [IPMI_ARGS] <command> on this machine (Systems without a command).
  • ssh_ipmitool: logs in to the host over ssh with the IPMI username and password (-iu / -ip, so admin / admin unless set) and runs ipmitool <command> there.
• Any other value fails with IPMI method ... not supported!.

-irlt, --ipmi_retry_lockout_time IPMI_RETRY_LOCKOUT_TIME | DEFAULT_IPMI_RETRY_LOCKOUT_TIME

• Seconds to wait after a failed login that reached the BMC before the next attempt on that host (the next IPMI definition or the ssh fallback), so a BMC that locks an account after failed logins can unlock it first.
• Failures that never reached a BMC (timeouts, no route, connection refused, ...) do not wait.
• -1 (default) or 0: never wait.

-S, -q, -nw, --no_watch | DEFAULT_NO_WATCH

• Disable the curses terminal display and only print the output.
• Note this will not be 'quiet mode' traditionally, please use -Q, -no, --quiet, --no_output to disable output.
• Useful in scripting to reduce runtime and terminal flashing.
• With one host, or with -m 1, ssh can ask for passwords, passphrases and host keys on your terminal; otherwise it cannot. See Password, Passphrase and Host Key Questions.

-ww, --window_width WINDOW_WIDTH | DEFAULT_WINDOW_WIDTH

• The minimum character length of the curses window.
• Default is 40 characters.
• Will try to fit as many hosts as possible in the terminal window while leaving at least this many characters for each host display.
• You can modify this value in curses display by pressing { or }.

-wh, --window_height WINDOW_HEIGHT | DEFAULT_WINDOW_HEIGHT

• The minimum line height of the curses window.
• Default is 1 line.
• Will try to fit as many hosts as possible in the terminal window while leaving at least this many lines for each host display.
• You can modify this value in curses display by pressing _ or +.
• When the terminal is smaller than ww * wh allows for every host, fewer hosts are shown at a time (at least one), and < / > rotate through the rest.

-B, -sw, --single_window | DEFAULT_SINGLE_WINDOW

• Use a single window mode for curses display.
• This shows a single large window for a host for detailed monitoring.
• You can rotate between hosts by pressing < or > in curses display. ( also works in non single window mode )
• You can toggle single window mode in curses display by pressing | ( pipe ).

-R, -eo, --error_only | DEFAULT_ERROR_ONLY

• Print Success if all hosts returns zero.
• Only print output for the hosts that returns non-zero.
• Useful in scripting to reduce output.

-Q, -no, --no_output, --quiet | DEFAULT_NO_OUTPUT

• Do not print any output.
• Note: if using without --no_watch, the curses display will still be shown.
• Useful in scripting when failure is expected.
• Note: return code will still be returned correctly unless -Z, -rz, --return_zero is specified.

-Z, -rz, --return_zero | DEFAULT_RETURN_ZERO

• Return 0 even if there are errors.
• Useful in scripting when failure is expected and bash is set to exit when error occurs.
• Not when mssh itself is stopped: SIGTERM (kill, a batch scheduler, a service manager), SIGHUP (the terminal closing, unless started with nohup) or SIGQUIT (Ctrl-\) kills the running commands, starts no more and exits with 143, 129 or 131.

-C, --no_env | DEFAULT_NO_ENV

• Do not load the command line environment variables for hostname resolution.
• Only use /etc/hosts -> env files specified in --env_files ENV_FILES -> DNS for hostname resolution.
• Useful when environment variables can interfere with hostname resolution ( for example not reloading environment variables after refreshing them ).

-ef, --env_file ENV_FILE | None

• Replace the env file look up chain with this env_file. ( Still work with --no_env )
• Only this file will be used for env file hostname resolution.
• Useful when you want to use a specific env file for hostname resolution.

-efs, --env_files ENV_FILES | DEFAULT_ENV_FILES

• The files to load the environment variables for hostname resolution.
• Can specify multiple. Load first to last. ( Still work with --no_env )
• Useful when you want to add additional env files for hostname resolution.

-m, --max_connections MAX_CONNECTIONS | DEFAULT_MAX_CONNECTIONS

• The maximum number of concurrent connections to establish.
• Default is 4 * cpu_count connections.
• Useful for limiting the number of simultaneous SSH connections to avoid overwhelming the compute resources / security limits
• Note: mssh will open at least 3 files per connection. By default some linux systems will only set the ulimit -n to 1024 files. This means about 300 connections can be opened simultaneously. You can increase the ulimit -n value to allow more connections if needed.
  • You will observe Warning: The number of maximum connections {max_connections} is larger than estimated limit {estimated_limit} ..... if the max connections is larger than estimated limit.
  • mssh will also throttle thread generation if the estimated limit is lower than 2 * max_connections to avoid hitting the file descriptor limit as python will use some file descriptors when setting up threads.
• Note: mssh runs the hosts on at most this many threads, however many hosts there are (with --ipmi, one thread per host: a host waiting out an IPMI lockout runs nothing). Each connection also needs its ssh process and two threads reading its output, about 4 processes / threads in all. Under a process / thread limit (ulimit -u, a container's pids limit) too low for that, mssh starts fewer threads, and a host that cannot start its ssh or threads waits for the running commands to end (Warning: Too many processes or threads. retrying...): the hosts run fewer at a time.
  • You will observe Warning: The system allows no more threads (ulimit -u, pids limit), running the hosts at most N at a time. when it does.
• Behind a bastion (ssh -J, or ProxyJump in ~/.ssh/config), every connection goes through the bastion's sshd, and its MaxStartups (default 10:30:100) turns away connections that have not logged in yet once more than 10 are opening: 30% of new ones, rising to all of them at 100. mssh opens up to -m connections at once, so with the default some hosts fail with kex_exchange_identification: read: Connection reset by peer / Connection closed by UNKNOWN port 65535 (exit code 255). The same can happen on a host many clients connect to at once.
  • mssh tries such a host again, up to 3 times, 0.2 to 3 s later, when ssh's lines show the connection was dropped before login: they name the address ssh connected to, or UNKNOWN port 65535 through a ProxyJump / ProxyCommand, with nothing else printed. A command that may have started is never run again. Older OpenSSH, such as 8.0 on RHEL 8, names no address, so its ssh and scp runs are not tried again; rsync runs are, as they received nothing and only sync again.
  • When hosts still fail this way, a hint after the summary says how many and why. Use -m 10 (or fewer) for runs through a bastion, or raise MaxStartups on it.

-j, --json | DEFAULT_JSON_OUTPUT

• Output in json format.
• Will also respect -R, -eo, --error_only and -Q, -no, --quiet, --no_output options.
• Every host has its own entry, also when hosts return the same output. ( The default output shows such hosts once, as node[1-3]; json and greppable output merged them too in 6.05 to 6.31. )

-w, --success_hosts | DEFAULT_SUCCESS_HOSTS

• By default, a summary of failed hosts is printed.
• Use this option to also print the hosts that succeeded in summary as well.
• Useful when you want to do something with the succeeded hosts later.
• Note: you can directly use the failed / succeeded host list string as it should be fully compatible with mssh host input.

-P, -g, --greppable, --table | DEFAULT_GREPPABLE_OUTPUT

• Output in greppable table.
• Each line contains: Hostname / Resolved Name / Return Code / Output Type/ Output
• Note a host can have multiple lines if the output contains multiple lines.
• Every host has its own lines, also when hosts return the same output, so grepping for a host finds them.
• Useful in a script if we are piping the output to a log file for later grepping.

-x, -su, --skip_unreachable | DEFAULT_SKIP_UNREACHABLE

• Skip unreachable hosts.
• Note: a host whose command hits -t, --timeout before printing anything is considered unreachable: mssh cannot tell a host that never answers from a command that is quiet from the start. A host that printed something answered, so it is not.
• By default mssh set this to true to speed up operations on large host lists with some unreachable hosts.
• Unreachable hosts will be tried again when their timeout expires.
• A skipped host is not run and counts as a failed host: it shows return code 255 ( what ssh returns when it cannot connect ) with a line saying it was skipped, and the summary lists it after failed_hosts: as Skipped N hosts unreachable in a recent run, without trying them: .... So mssh does not exit 0 or print Success when it skipped a host.
• mssh stores the current run unreachable hosts in memory and if skip_unreachable is true, it will store them in unavailable_hosts.csv in mssh's private runtime directory: $XDG_RUNTIME_DIR/multiSSH3, else /run/user/<uid>/multiSSH3, /dev/shm/multiSSH3-<uid>, or a multiSSH3-<uid> folder in the system temp folder (the first that only you can write to). If none can be used, they are only kept for the current run.
• To force mssh to not use unreachable hosts from previous runs, you can use -a, -nsu, --no_skip_unreachable to set skip_unreachable to false.

-a, -nsu, --no_skip_unreachable | not DEFAULT_SKIP_UNREACHABLE

• Do not skip unreachable hosts.
• This forms an mutually exclusive pair with -x, -su, --skip_unreachable.
• This option sets skip_unreachable to false.

-uhe, --unavailable_host_expiry UNAVAILABLE_HOST_EXPIRY | DEFAULT_UNAVAILABLE_HOST_EXPIRY

• The expiry time in seconds for unreachable hosts stored in the temperary unavailable hosts file.
• Default is 600 seconds ( 10 minutes ).
• Note: because mssh stores a hostname: expire_time pair in the unavailable hosts file, opeartor is able to use different expiry time for different runs to control how long unreachable hosts are skipped and they will be expired at the correct time.
• Note: mssh stores the expire time in monotonic time, which restarts at every boot, so the file also records the boot it was written in and a file from an earlier boot is ignored (the temp folder can survive a reboot). Without a boot id to compare, a file holding an entry that expires further away than unavailable_host_expiry is taken to be from an earlier boot.

-X, -sh, --skip_hosts SKIP_HOSTS | DEFAULT_SKIP_HOSTS

• A comma separated list of hosts to skip.
• This field will be expanded in the same way as the host list.
• Useful when you want to skip some hosts temporarily without modifying the host list.

--generate_config_file | False

• Print the config (the built-in defaults, the config files mssh loaded and the current command line options) as JSON to stdout, and exit when no command is given. No file is written.
• With --store_config_file, writes that file instead, overwriting an existing one without asking and without a backup.

--config_file [CONFIG_FILE] | None

• Additional config file path to load options from.
• Will be loaded last thus overwriting other config file values.
• Use without value to use multiSSH3.config.json in the current working directory.
• It only names a file to read, also with --generate_config_file, which still prints to stdout. A CONFIG_FILE that does not exist gets Warning: Config file 'CONFIG_FILE' not found, ignoring it. and is not created.

--store_config_file [STORE_CONFIG_FILE] | None

• Store the current command line options to a config file: what --generate_config_file prints, readable by you only (0600, or 0644 for the /etc locations).
• Without a path, writes to the config file mssh loaded, or to ~/.multiSSH3.config.json if none was loaded.
• If the file exists, asks whether to overwrite it, move it to STORE_CONFIG_FILE.bak first (the answer after 10 seconds) or stop.

--debug | False

• Enable debug mode.
• Print host specific debug messages to hosts's stderr.

-ci, --copy_id | False

• copy_id mode, use ssh-copy-id to copy public key to the hosts.
• Will use the identity file if specified in -k, --key, --identity [KEY] as the key to install.
• Will respect -u, --username and -p, --password options for username and password. With -p and sshpass, the hosts get the key at once; without them, ssh-copy-id asks for the password on the terminal, one host after the other, and Ctrl-C stops before the next host.
• ssh-copy-id checks host keys as every ssh mssh runs does: unless SSH_STRICT_HOST_KEY_CHECKING is true, it gets -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null, so new nodes, not yet in known_hosts, need no answer to a host key question (with -p, sshpass cannot give one). With SSH_STRICT_HOST_KEY_CHECKING true, it checks them as your ssh config says.
• From -ea, --extraargs it gets what every ssh-copy-id takes (that of OpenSSH 7.4 on CentOS 7 and later): -o and -p, and ssh's -J as -o ProxyJump=. The other words are left out: ssh's -i (the -i of ssh-copy-id names the key to install), -F (the ssh-copy-id of OpenSSH 8.0 and older fails on it) and rsync's options.
• A host ssh-copy-id failed on is a failed host: it counts towards the exit code and is listed in failed_hosts, also when a command runs after the copy. Without a command, mssh exits with 127 if it finds no ssh-copy-id.
• Example: mssh -ci -u admin -p 'PASSWORD' newnode[01-20]

-I, -nh, --no_history | DEFAULT_NO_HISTORY

• Do not store command history.
• By default, mssh store command history in HISTORY_FILE.
• Useful in scripts when you do not want to store command history.

-hf, --history_file HISTORY_FILE | DEFAULT_HISTORY_FILE

• The file to store command history.
• By default, mssh store command history in ~/.mssh_history.
• The history file is a TSV ( tab separated values ) file with each line containing: timestamp, mssh_path, options, hosts, commands.
• A line records the command so that replaying it (the Ctrl-R picker, mssh-complete) runs it the same way: the hosts as typed and every option that changes what runs, where, or what is shown; a flag when it is on, an option with a value when the value is not the default (the config files' DEFAULT_*). -T, -uk (or -k with a folder), --script and -s PATH are recorded as the timeout, key file and options they chose, --config_file and --host_file with their absolute paths. Not recorded: -ip (and with it the other IPMI options, see -ip), and --generate_config_file, --store_config_file and --add_control_master_config, which do something besides the run. The > mssh ... line mssh prints at start lists the same options.

--script | False

• Script mode, syntatic sugar for -SCRIPT or --no_watch --skip_unreachable --no_env --no_history --greppable --error_only.
• Useful when using mssh in shell scripts.

-e, --encoding ENCODING | DEFAULT_ENCODING

• The encoding to use for decoding the output from the hosts.
• Default is utf-8.
• Characters the local terminal's encoding cannot show (for example under LANG=C with Python 3.6) are printed as escapes like \u2713.
• An encoding Python cannot decode text with (unknown, or a codec such as rot13) is refused before any host runs.

-dt, --diff_display_threshold DIFF_DISPLAY_THRESHOLD | DEFAULT_DIFF_DISPLAY_THRESHOLD

• How alike the outputs of hosts must be (0 to 1, roughly the share of lines they have in common) to be shown as one block in N-body diff display mode: the lines they share are shown once, and the lines only some of them printed are shown under the names of those hosts.
• Hosts with the same output are always shown once. Default 0.9.
• Useful when the outputs are large and mostly similar.
• Set to 0.0 to always use diff display mode.
• Set to 1.0 to never use diff display mode (only the same outputs are merged).
• Only for the normal output, not -j or -P.
• Note: This uses custom N-body diff algorithm. Uses some memory.

--force_truecolor | DEFAULT_FORCE_TRUECOLOR

• Force enable truecolor support in curses display.
• Useful when your terminal supports truecolor but is not detected correctly.

--add_control_master_config | False

• Add ControlMaster configuration to your ~/.ssh/config file to enable persistent ssh connections.
• This will help speed up connections to multiple hosts.
• The configuration added, and where ssh puts its sockets, is described in SSH ControlMaster Configuration.
• Run it again to replace the block multiSSH3 6.31 and older added, which makes ssh fail where /run/user/<uid> is missing.

--host_file HOST_FILE | DEFAULT_HOST_FILE

• The hosts file mssh looks host names up in before DNS ( see Hostname Resolution Logic ). Default: /etc/hosts.
• In the format of /etc/hosts: an IP address, then its names, per line. Text after # is a comment, and lines that do not start with an IP address are skipped.
• A name listed more than once gets its first IPv4 address, else its first IPv6 address.
• It is not a list of hosts to run on. mssh warns when the file does not exist or has no address lines.
• mssh uses these addresses for #IP#, -pre and to check host names. ssh is still given the host name ( or the -pre address ), so it must find the name on its own ( DNS, /etc/hosts, ~/.ssh/config ).

--completion ARG [ARG ...] | None

• Shell autocomplete: install [SHELL] [editor], uninstall [SHELL], or print the glue for bash, zsh or fish. See Autocomplete.

-V, --version | False

• Print the version of multiSSH3 and exit.
• Will also print the found system binary for calling when setting up connections.

Exit Codes

mssh exits with the number of failed hosts, so 0 means every host ran the command and returned 0. A failed host is one that:

• returned non-zero ( counted once, however many commands or -r, --repeat runs it failed ),
• was skipped because it was unreachable in a recent run ( see -x, --skip_unreachable ),
• or is a host name that does not resolve, or a host you gave that names no host ( like node[10-1] ).

The count stops at 255, as an exit status only has 8 bits. -Z, --return_zero makes it 0. When no host is left to run on ( every host you gave is one of the last kind ), mssh stops with Exiting: no hosts to run on and the same count. If -X, --skip_hosts excludes every host, nothing runs, mssh warns and exits 0.

Before anything runs, mssh can also stop with: 2 for a bad option, 66 when the -f, --file sources are not found, 255 when -11, --oneonone gets a different number of commands and hosts, and 1 for Ctrl-C.

The return code shown for each host is the command's own, or: 255 when ssh could not connect ( or the host was skipped as unreachable ), 124 when -t, --timeout stopped it, 130 when Ctrl-C stopped it ( the first Ctrl-C also keeps hosts still waiting for a connection slot from starting: they say Not started: Ctrl C ), and -1 for an error in mssh itself.

Usage

Use mssh --help for more info.

Remote commands with options

mssh reads its options anywhere on the line, also among the words after the hosts. Quote a remote command that has options of its own, so it reaches mssh as one word:

mssh web1 'ls -la /var/log'
mssh web1 "bash -c 'echo \$HOSTNAME'"
mssh gpu[01-16] 'tail -n 50 /var/log/slurmd.log' -q   # -q is mssh's own: no curses window

Left unquoted, a word starting with - is read by mssh:

• A word that names an mssh option is that option: mssh web1 df -h prints mssh's help, mssh web1 journalctl -u slurmd connects as user slurmd (-u, --username), mssh web1 tail -f /var/log/x copies this machine's /var/log/x to the hosts, as tail (-f, --file), and mssh web1 ps -aux reads -a -u x. This is also what lets mssh's options follow the command (mssh web1 uptime -q).
• Any other word stops mssh with an error (mssh web1 ls -la: unrecognized arguments: -la) that says to quote the command. On Python 3.6, mssh warns and adds such words to the commands instead.

So that a remote command's option is never taken silently:

• The autocomplete marks such a word before you press Enter, also the word you are still typing at the end of the line (see Warnings before Enter). It sees your quotes, so options after a quoted command are not marked.

• mssh itself prints a note on stderr before it runs:

  Note: mssh read -h after the command 'df' as its own option (--help: mssh prints its help and runs nothing). If it was meant for df, quote the command: mssh all 'df -h'
  

  By then the shell has removed the quotes, so mssh cannot tell mssh web1 'uptime' -q from mssh web1 uptime -q. It says nothing where the line shows the option was meant for mssh: after a command word holding a space or a shell character (|&;<>()$, quotes, backslash), which only quotes could give it, as in mssh gpu[01-16] 'tail -n 50 /var/log/slurmd.log' -q; and for flags that only change how mssh shows the output (-q, -j, -g, -Q, -R, -w, -B, -Z, -I, --debug, --force_truecolor) after the command's last word. Options that take a value (-t 30), -h, -V and flags that change what runs (-a, -ci, -M, -G, ...) are noted. To give mssh options without a note, put them before the hosts: mssh -q -t 30 web1 uptime.

Option names count in full only: --user is not --username and -c is not -ci, so mssh web1 bash -c 'echo hi' stops with an error instead of running ssh-copy-id. Short options can still be written together (-qj is -q -j, -SCRIPT is -S -C -R -I -P -T) and take their value attached (-uroot, -t30).

Putting the command after -- is no substitute for quotes: every word after the hosts is a command of its own, so mssh web1 -- bash -c 'echo hi' runs bash, -c and echo hi one after another (when each word is a single word, as in mssh web1 -- ls -la, mssh asks whether to join them).

Below is a sample help message output from multiSSH3 6.32 @ 2026-10-14 (no config file, 120 columns; the lines after the options list the programs found on that machine)

$ mssh -h
usage: mssh [-h] [-u USERNAME] [-p PASSWORD] [-k [IDENTITY_FILE]] [-uk] [-ea EXTRAARGS] [-11] [-f FILE]
            [-s [FILE_SYNC]] [-W] [-G] [-t TIMEOUT] [-T] [-r REPEAT] [-i INTERVAL] [-M]
            [-mpre IPMI_INTERFACE_IP_PREFIX] [-pre INTERFACE_IP_PREFIX] [-iu IPMI_USERNAME] [-ip IPMI_PASSWORD]
            [-ia IPMI_ARGS] [-im IPMI_METHOD] [-irlt IPMI_RETRY_LOCKOUT_TIME] [-S] [-ww WINDOW_WIDTH]
            [-wh WINDOW_HEIGHT] [-B] [-R] [-Q] [-Z] [-C] [-ef ENV_FILE] [-efs ENV_FILES] [-m MAX_CONNECTIONS] [-j]
            [-w] [-P] [-x | -a] [-uhe UNAVAILABLE_HOST_EXPIRY] [-X SKIP_HOSTS] [--generate_config_file]
            [--config_file [CONFIG_FILE]] [--store_config_file [STORE_CONFIG_FILE]] [--debug] [-ci] [-I]
            [-hf HISTORY_FILE] [--script] [-e ENCODING] [-dt DIFF_DISPLAY_THRESHOLD] [--force_truecolor]
            [--add_control_master_config] [--host_file HOST_FILE] [--completion ARG [ARG ...]] [-V]
            [hosts] [commands ...]

Run a command on multiple hosts, Use #HOST# or #HOSTNAME# to replace the host name in the command.

positional arguments:
  hosts                 Hosts to run the command on, use "," to seperate hosts. (default: all)
  commands              the command to run on the hosts / the destination of the files #HOST# or #HOSTNAME# will be replaced with the host name.

options:
  -h, --help            show this help message and exit
  -u, --username USERNAME
                        The general username to use to connect to the hosts via ssh. Will get overwrote by individual username@host if specified. (default: None)
  -p, --password PASSWORD
                        The password to use to connect to the hosts via ssh, (default: none)
  -k, --identity_file, --key, --identity [IDENTITY_FILE]
                        The identity file to use to connect to the hosts. Implies --use_key. Specify a folder for program to search for a key. Use option without value to use ~/.ssh/ (default: None)
  -uk, --use_key        Attempt to use public key file to connect to the hosts. (default: False)
  -ea, --extraargs EXTRAARGS
                        Extra arguments to pass to the ssh / rsync / scp command. (Use -ia for ipmi arguments) Put in one string for multiple arguments.Use "=" ! Ex. -ea="--delete" (default: None)
  -11, --oneonone       Run one corresponding command on each host. (default: False)
  -f, --file FILE       The file to be copied to the hosts. Use -f multiple times to copy multiple files
  -s, -fs, --file_sync [FILE_SYNC]
                        Operate in file sync mode, sync path in <COMMANDS> from this machine to <HOSTS>. Treat --file <FILE> and <COMMANDS> both as source and source and destination will be the same in this mode. Infer destination from source path. (default: False)
  -W, --scp             Use scp for copying files instead of rsync. Need to use this on windows. (default: False)
  -G, -gm, --gather_mode
                        Gather files from the hosts instead of sending files to the hosts. Will send remote files specified in <FILE> to local path specified in <COMMANDS>  (default: False)
  -t, --timeout TIMEOUT
                        Idle timeout in seconds: stop a host's command after this long without output. Set default value via DEFAULT_CLI_TIMEOUT in config file. Use 0 for disabling timeout. (default: 0)
  -T, --use_script_timeout
                        Use shortened timeout suitable to use in a script. Set value via DEFAULT_TIMEOUT field in config file. (current: 50)
  -r, --repeat REPEAT   Repeat the command for a number of times (default: 1)
  -i, --interval INTERVAL
                        Interval between repeats in seconds (default: 0)
  -M, --ipmi            Use ipmitool to run the command. (default: False)
  -mpre, --ipmi_interface_ip_prefix IPMI_INTERFACE_IP_PREFIX
                        The prefix of the IPMI interfaces (default: )
  -pre, --interface_ip_prefix INTERFACE_IP_PREFIX
                        The prefix of the for the interfaces (default: None)
  -iu, --ipmi_username IPMI_USERNAME
                        The username to use to connect to the hosts via ipmi. (default: admin)
  -ip, --ipmi_password IPMI_PASSWORD
                        The password to use to connect to the hosts via ipmi. (default: set, not shown)
  -ia, --ipmi_args IPMI_ARGS
                        The arguments passed to ipmitool to use to connect to the hosts via ipmi. (default: )
  -im, --ipmi_method IPMI_METHOD
                        The method to use to connect to the hosts via ipmi. (default: ipmitool)
  -irlt, --ipmi_retry_lockout_time IPMI_RETRY_LOCKOUT_TIME
                        The time in seconds to retry the lockout of the hosts via ipmi. (default: -1)
  -S, -q, -nw, --no_watch
                        Quiet mode, no curses watch, only print the output. (default: False)
  -ww, --window_width WINDOW_WIDTH
                        The minimum character length of the curses window. (default: 40)
  -wh, --window_height WINDOW_HEIGHT
                        The minimum line height of the curses window. (default: 1)
  -B, -sw, --single_window
                        Use a single window for all hosts. (default: False)
  -R, -eo, --error_only
                        Only print the error output. (default: False)
  -Q, -no, --no_output, --quiet
                        Do not print the output. (default: False)
  -Z, -rz, --return_zero
                        Return 0 even if there are errors. (default: False)
  -C, --no_env          Do not load the command line environment variables. (default: False)
  -ef, --env_file ENV_FILE
                        Replace the env file look up chain with this env_file. ( Still work with --no_env ) (default: None)
  -efs, --env_files ENV_FILES
                        The files to load the mssh file based environment variables from. Can specify multiple. Load first to last. ( Still work with --no_env ) (default: [])
  -m, --max_connections MAX_CONNECTIONS
                        Max number of connections to use (default: 4 * cpu_count)
  -j, --json            Output in json format. (default: False)
  -w, --success_hosts   Output the hosts that succeeded in summary as well. (default: False)
  -P, -g, --greppable, --table
                        Output in greppable table. (default: False)
  -x, -su, --skip_unreachable
                        Skip unreachable hosts. Note: Hosts that time out before printing anything are considered unreachable. Note: multiple command sequence will still auto skip unreachable hosts. (default: True)
  -a, -nsu, --no_skip_unreachable
                        Do not skip unreachable hosts. Note: Hosts that time out before printing anything are considered unreachable. Note: multiple command sequence will still auto skip unreachable hosts. (default: False)
  -uhe, --unavailable_host_expiry UNAVAILABLE_HOST_EXPIRY
                        Time in seconds to expire the unavailable hosts (default: 600)
  -X, -sh, --skip_hosts SKIP_HOSTS
                        Skip the hosts in the list. (default: None)
  --generate_config_file
                        Restore / generate using the build-in default configs with default config file chain and command line argument to write to --store_config_file / stdout
  --config_file [CONFIG_FILE]
                        Load additional config file to apply, Warning: will over write CLI options. Use without a path will use multiSSH3.config.json
  --store_config_file [STORE_CONFIG_FILE]
                        Refresh / store the specified config file with new params and command line args into the config file. Defaults to the first config file found in search path: /home/user/.multiSSH3.config.json
  --debug               Print debug information
  -ci, --copy_id        Copy the ssh id to the hosts
  -I, -nh, --no_history
                        Do not record the command to history. Default: False
  -hf, --history_file HISTORY_FILE
                        The file to store the history. (default: ~/.mssh_history)
  --script              Run the command in script mode, short for -SCRIPT or --no_watch --skip_unreachable --no_env --no_history --greppable --error_only
  -e, --encoding ENCODING
                        The encoding to use for the output. (default: utf-8)
  -dt, --diff_display_threshold DIFF_DISPLAY_THRESHOLD
                        The threshold of lines to display the diff when files differ. {0-1} Set to 0 to always display the diff. Set to 1 to disable diff. (Only merge same) (default: 0.9)
  --force_truecolor     Force truecolor output even when not in a truecolor terminal. (default: False)
  --add_control_master_config
                        Add ControlMaster configuration to ~/.ssh/config to speed up multiple connections to the same host.
  --host_file HOST_FILE
                        The hosts file mssh looks host names up in before DNS, in the format of /etc/hosts (an IP address, then names, per line). Not a list of hosts to run on. Default: /etc/hosts
  --completion ARG [ARG ...]
                        Shell autocomplete: print the glue for bash, zsh or fish, or "install [SHELL] [editor]" / "uninstall [SHELL]" it for the current user ("editor": the plain-bash mssh line editor)
  -V, --version         show program's version number and exit

Terminal color capability: 24bit
Found bins: 
{'/usr/bin/ipmitool',
 '/usr/bin/rsync',
 '/usr/bin/scp',
 '/usr/bin/sh',
 '/usr/bin/ssh',
 '/usr/bin/ssh-copy-id',
 '/usr/bin/sshpass'}
Missing bins: 
{'redfishtool'}
Config file chain: 
['~/multiSSH3.config.json',
 '~/.multiSSH3.config.json',
 '~/.config/multiSSH3/multiSSH3.config.json',
 '/etc/multiSSH3.d/multiSSH3.config.json',
 '/etc/multiSSH3.config.json']
Hosts Env files searched:
['/etc/profile.d/hosts.sh',
 '~/.bashrc',
 '~/.zshrc',
 '~/host.env',
 '~/hosts.env']

mssh reads words starting with '-' as its own options, also after the hosts. To pass options to the remote command, quote the whole command: mssh web1 'ls -la'

Note: The default values can be modified / updated in the Config file.

Importing as a Module

You can also import multiSSH3 as a module in your python scripts.

Host Object

The Host object represents a host and its command execution state. The main execution function run_command_on_hosts returns a list of Host objects.

class Host:
	def __init__(self, name, command, files = None,ipmi = False,interface_ip_prefix = None,scp=False,extraargs=None,gatherMode=False,identity_file=None,shell=False,i = -1,uuid=uuid.uuid4(),ip = None):
		self.name = name # the name of the host (hostname or IP address)
		self.command = command # the command to run on the host
		self.returncode = None # the return code of the command
		self.output = [] # the output of the command for curses
		self.stdout = [] # the stdout of the command
		self.stderr = [] # the stderr of the command
		self.lineNumToPrintSet = set() # line numbers to reprint
		self.lastUpdateTime = time.monotonic() # the last time the output was updated
		self.lastPrintedUpdateTime = 0 # the last time the output was printed
		self.files = files # the files to be copied to the host
		self.ipmi = ipmi # whether to use ipmi to connect to the host
		self.shell = shell # whether to use shell to run the command
		self.interface_ip_prefix = interface_ip_prefix # the prefix of the ip address of the interface to be used to connect to the host
		self.scp = scp # whether to use scp to copy files to the host
		self.gatherMode = gatherMode # whether the host is in gather mode
		self.extraargs = extraargs # extra arguments to be passed to ssh
		self.resolvedName = None # the resolved IP address of the host
		# also store a globally unique integer i from 0
		self.i = i if i != -1 else _get_i()
		self.uuid = uuid
		self.identity_file = identity_file
		self.ip = ip if ip else getIP(name)
		self.current_color_pair = [-1, -1, 1]
		self.output_buffer = io.BytesIO()
		self.stdout_buffer = io.BytesIO()
		self.stderr_buffer = io.BytesIO()
		self.thread = None
		self.interactive_stdin = False # forward the curses TUI key presses to stdin (set while the TUI runs), else /dev/null
		self.skipped = False # True when mssh did not run the command: the host was unreachable in a recent run

A host skipped because it was unreachable in a recent run ( see -x, --skip_unreachable ) is in the returned list without being run: its skipped is True, its returncode is 255 and its stderr says why. start_run_on_hosts() does not start it.

Example:

import multiSSH3
ethReachableHosts = multiSSH3.run_command_on_hosts(nodesToCheck,['echo hi'],return_unfinished = True) # return_unfinished will return the host objects immediately without waiting for all hosts to finish.
ipmiReachableHosts = multiSSH3.run_command_on_hosts(nodesToCheck,['power status'], timeout = timeout, ipmi=True, password=password, skip_unreachable=False) # ipmi command
# You can use blocks like the following block to wait on a specific host list to finish ( That does respect timeout even if the host is hanging )
# Following block also implement an exponential backoff.
sleep_interval = 0.0001
while any([host.returncode is None for host in ethReachableHosts]):
  time.sleep(sleep_interval)  # avoid busy-waiting
  if sleep_interval < 0.01:
    sleep_interval *= 1.1
# You can also use join_threads to wait on all running threads ( including timedout hanging threads )
multiSSH3.join_threads()
...

run_command_on_hosts Function

The run_command_on_hosts function includes almost all the options available in the CLI. ( some options modify the global variables and can be set before calling the function. )

def run_command_on_hosts(hosts = DEFAULT_HOSTS,commands = None,oneonone = DEFAULT_ONE_ON_ONE, timeout = DEFAULT_TIMEOUT,password = DEFAULT_PASSWORD,no_watch = DEFAULT_NO_WATCH,json = DEFAULT_JSON_MODE,called = _DEFAULT_CALLED,max_connections=DEFAULT_MAX_CONNECTIONS,file = None,ipmi = DEFAULT_IPMI,interface_ip_prefix = DEFAULT_INTERFACE_IP_PREFIX,return_unfinished = _DEFAULT_RETURN_UNFINISHED,scp=DEFAULT_SCP,gather_mode = False,username=DEFAULT_USERNAME,extraargs=DEFAULT_EXTRA_ARGS,skip_unreachable=DEFAULT_SKIP_UNREACHABLE,no_env=DEFAULT_NO_ENV,greppable=DEFAULT_GREPPABLE_MODE,will_update_unreachable_hosts=_DEFAULT_UPDATE_UNREACHABLE_HOSTS,no_start=_DEFAULT_NO_START,skip_hosts = DEFAULT_SKIP_HOSTS, window_width = DEFAULT_WINDOW_WIDTH, window_height = DEFAULT_WINDOW_HEIGHT,single_window = DEFAULT_SINGLE_WINDOW,file_sync = False,error_only = DEFAULT_ERROR_ONLY,quiet = False,identity_file = DEFAULT_IDENTITY_FILE,copy_id = False, unavailable_host_expiry = DEFAULT_UNAVAILABLE_HOST_EXPIRY,no_history = True,history_file = DEFAULT_HISTORY_FILE,
):
	"""
	Run commands on multiple hosts via SSH or IPMI.

	Parameters:
		hosts (str or iterable): Hosts to run the command on. Can be a string (comma/space-separated) or iterable. Default: DEFAULT_HOSTS.
		commands (list or None): List of commands to run on the hosts. If files are used, defines the destination. Default: None.
		oneonone (bool): If True, run each command on the corresponding host (1:1 mapping). Default: DEFAULT_ONE_ON_ONE.
		timeout (int): Idle timeout: stop a host's command after this many seconds without output. Default: DEFAULT_TIMEOUT.
		password (str): Password for SSH/IPMI authentication. Default: DEFAULT_PASSWORD.
		no_watch (bool): If True, do not use curses TUI; just print output. Default: DEFAULT_NO_WATCH.
		json (bool): If True, output results in JSON format. Default: DEFAULT_JSON_MODE.
		called (bool): If True, function is called programmatically (not CLI). Default: _DEFAULT_CALLED.
		max_connections (int): Maximum concurrent SSH sessions. Default: 4 * os.cpu_count().
		file (list or None): Files to copy to hosts. Default: None.
		ipmi (bool): Use IPMI instead of SSH. Default: DEFAULT_IPMI.
		interface_ip_prefix (str or None): Override IP prefix for host connection. Default: DEFAULT_INTERFACE_IP_PREFIX.
		return_unfinished (bool): If True, return hosts even if not finished. Default: _DEFAULT_RETURN_UNFINISHED.
		scp (bool): Use scp for file transfer (instead of rsync). Default: DEFAULT_SCP.
		gather_mode (bool): Gather files from hosts (pull mode). Default: False.
		username (str or None): Username for SSH/IPMI. Default: DEFAULT_USERNAME.
		extraargs (str or list or None): Extra args for SSH/SCP/rsync. Default: DEFAULT_EXTRA_ARGS.
		skip_unreachable (bool): Skip hosts marked as unreachable. Default: DEFAULT_SKIP_UNREACHABLE.
		no_env (bool): Do not load environment variables from shell. Default: DEFAULT_NO_ENV.
		greppable (bool): Output in greppable table format. Default: DEFAULT_GREPPABLE_MODE.
		will_update_unreachable_hosts (bool): Update global unreachable hosts file. Default: _DEFAULT_UPDATE_UNREACHABLE_HOSTS.
		no_start (bool): If True, return Host objects without running commands. Default: _DEFAULT_NO_START.
		skip_hosts (str or None): Hosts to skip. Default: DEFAULT_SKIP_HOSTS.
		window_width (int): Minimum width per curses window. Default: DEFAULT_WINDOW_WIDTH.
		window_height (int): Minimum height per curses window. Default: DEFAULT_WINDOW_HEIGHT.
		single_window (bool): Use a single curses window for all hosts. Default: DEFAULT_SINGLE_WINDOW.
		file_sync (bool): Enable file sync mode (sync directories). Default: DEFAULT_FILE_SYNC.
		error_only (bool): Only print error output. Default: DEFAULT_ERROR_ONLY.
		quiet (bool): Suppress all output (overrides other output options). Default: False.
		identity_file (str or None): SSH identity file. Default: DEFAULT_IDENTITY_FILE.
		copy_id (bool): Use ssh-copy-id to copy public key to hosts first. Without commands, the ssh-copy-id runs are returned as the hosts. Default: False.
		unavailable_host_expiry (int): Seconds to keep hosts marked as unavailable. Default: DEFAULT_UNAVAILABLE_HOST_EXPIRY.
		no_history (bool): Do not record command history. Default: True.
		history_file (str): File to store command history. Default: DEFAULT_HISTORY_FILE.

	Returns:
		list: List of Host objects representing each host/command run.
	"""

All Changable Config Options ( Global Variables )

Update these values in the config file to change the default behavior of multiSSH3.

Values with leading _ will not be included in the config file generated by mssh but are respected when set in the config file.

AUTHOR = 'Yufei Pan'
AUTHOR_EMAIL = 'pan@zopyr.us'
DEFAULT_HOSTS = 'all'
DEFAULT_USERNAME = None
DEFAULT_PASSWORD = ''
DEFAULT_IDENTITY_FILE = None
DEFAULT_SSH_KEY_SEARCH_PATH = '~/.ssh/'
DEFAULT_USE_KEY = False
DEFAULT_EXTRA_ARGS = None
DEFAULT_ONE_ON_ONE = False
DEFAULT_SCP = False
DEFAULT_FILE_SYNC = False
DEFAULT_TIMEOUT = 50
DEFAULT_CLI_TIMEOUT = 0
# seconds after which ssh gives up on a host that does not answer (ConnectTimeout) or stops answering
# (ServerAliveInterval x ServerAliveCountMax), unless -ea or the ssh config set them; 0 leaves them to ssh
DEFAULT_SSH_CONNECT_TIMEOUT = 30
DEFAULT_SSH_SERVER_ALIVE_INTERVAL = 15
DEFAULT_SSH_SERVER_ALIVE_COUNT_MAX = 4
DEFAULT_UNAVAILABLE_HOST_EXPIRY = 600
DEFAULT_REPEAT = 1
DEFAULT_INTERVAL = 0
DEFAULT_IPMI = False
DEFAULT_IPMI_INTERFACE_IP_PREFIX = ''
DEFAULT_IPMI_USERNAME = 'admin'
DEFAULT_IPMI_PASSWORD = 'admin'
DEFAULT_IPMI_ARGS=''
DEFAULT_IPMI_METHOD = 'ipmitool'
DEFAULT_IPMI_RETRY_LOCKOUT_TIME = -1
def get_default_ipmi_definition():
	return {'interface_ip_prefix': DEFAULT_IPMI_INTERFACE_IP_PREFIX,
	'username': DEFAULT_IPMI_USERNAME,
	'password': DEFAULT_IPMI_PASSWORD,
	'args': DEFAULT_IPMI_ARGS,
	'ipmi_method': DEFAULT_IPMI_METHOD,
	'hostname_keyword': '',
	'retry_lockout_time': DEFAULT_IPMI_RETRY_LOCKOUT_TIME,
}
DEFAULT_IPMI_DEFINITIONS = [get_default_ipmi_definition()]
DEFAULT_INTERFACE_IP_PREFIX = None
DEFAULT_NO_WATCH = False
DEFAULT_WINDOW_WIDTH = 40
DEFAULT_WINDOW_HEIGHT = 1
DEFAULT_SINGLE_WINDOW = False
DEFAULT_ERROR_ONLY = False
DEFAULT_NO_OUTPUT = False
DEFAULT_RETURN_ZERO = False
DEFAULT_NO_ENV = False
DEFAULT_ENV_FILE = ''
DEFAULT_ENV_FILES = ['/etc/profile.d/hosts.sh',
					 '~/.bashrc',
					 '~/.zshrc',
					 '~/host.env',
					 '~/hosts.env',
					 ]  # no current-directory files (could redirect host aliases), use --env_files
DEFAULT_NO_HISTORY = False
DEFAULT_HISTORY_FILE = '~/.mssh_history'
def _cpu_count():
	# os.cpu_count() returns None when the count cannot be determined
	return os.cpu_count() or 1
DEFAULT_MAX_CONNECTIONS = 4 * _cpu_count()
DEFAULT_JSON_MODE = False
DEFAULT_PRINT_SUCCESS_HOSTS = False
DEFAULT_GREPPABLE_MODE = False
DEFAULT_SKIP_UNREACHABLE = True
DEFAULT_SKIP_HOSTS = ''
DEFAULT_HOST_FILE = '/etc/hosts'
DEFAULT_ENCODING = 'utf-8'
DEFAULT_DIFF_DISPLAY_THRESHOLD = 0.9
SSH_STRICT_HOST_KEY_CHECKING = False
FORCE_TRUECOLOR = False
ERROR_MESSAGES_TO_IGNORE = [
	r'^Pseudo-terminal will not be allocated because stdin is not a terminal',
	r'^Connection to \S+ closed( by remote host)?\.$',
	r'^Warning: Permanently added ',
	r'^mux_client_request_session: ',
	r'^ControlSocket .* already exists, disabling multiplexing$',
	r'^Killed by signal \d+\.$',
]
POSSIBLE_SSH_KEY_FILES = ['id_ed25519','id_ed25519_sk','id_ecdsa','id_ecdsa_sk','id_rsa','id_dsa']
__DEFAULT_COLOR_PALETTE = {
	'cyan': (86, 173, 188),
	'green': (114, 180, 43),
	'magenta': (140, 107, 200),
	'red': (196, 38, 94),
	'white': (227, 227, 221),
	'yellow': (179, 180, 43),
	'blue': (106, 126, 200),
	'bright_black': (102, 102, 102),
	'bright_blue': (129, 154, 255),
	'bright_cyan': (102, 217, 239),
	'bright_green': (126, 226, 46),
	'bright_magenta': (174, 129, 255),
	'bright_red': (249, 38, 114),
	'bright_white': (248, 248, 242),
	'bright_yellow': (226, 226, 46),
}
COLOR_PALETTE = __DEFAULT_COLOR_PALETTE.copy()
_DEFAULT_CALLED = True
_DEFAULT_RETURN_UNFINISHED = False
_DEFAULT_UPDATE_UNREACHABLE_HOSTS = True
_DEFAULT_NO_START = False
_etc_hosts = {}
__ERROR_MESSAGES_TO_IGNORE_REGEX =None
__DEBUG_MODE = False

SSH_STRICT_HOST_KEY_CHECKING = False (the default) passes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null to every ssh / scp / rsync, so host keys are not checked. See Host Key Checking.

DEFAULT_IPMI_DEFINITIONS lists the IPMI logins --ipmi tries on each host, in order, until one works. Each is a dict with any of interface_ip_prefix, username, password, args, ipmi_method and retry_lockout_time (what -mpre, -iu, -ip, -ia, -im and -irlt set; missing keys come from the DEFAULT_IPMI_* values), and hostname_keyword: a definition with one is only tried on hosts whose name contains it. Which logins are tried, and in which order (the command line's, these definitions, the DEFAULT_IPMI_* login), is in IPMI Logins. For example

"DEFAULT_IPMI_USERNAME": "ADMIN",
"DEFAULT_IPMI_PASSWORD": "bmc-password",
"DEFAULT_IPMI_DEFINITIONS": [
    {"hostname_keyword": "gpu", "username": "root", "password": "gpu-bmc-password", "args": "-I lanplus"}
]

makes mssh try root (with -I lanplus), then ADMIN, on hosts whose name contains gpu, and only ADMIN on the others. After the definitions, a host still failing gets the ssh fallback described under -M, --ipmi.

DEFAULT_HOST_FILE is the hosts file (/etc/hosts format) mssh looks host names up in before DNS; --host_file sets it for one run. Autocomplete offers the names in the one a config file sets.

Note: If color palette need to be changed, please change the COLOR_PALETTE values instead of the __DEFAULT_COLOR_PALETTE as if values are missing from __DEFAULT_COLOR_PALETTE, mssh may raise errors.

When imported and calling as a module, called will be set to _DEFAULT_CALLED.
When True, it sets several flags to make mssh behave better when called as a module.

• if skip_unreachable is not set, it will skip unreachable hosts within one command call
• Will clear global keyboard input cache
• Will set global quiet mode to True

_DEFAULT_UPDATE_UNREACHABLE_HOSTS is used to control whether to update the unreachable hosts file when mssh detects unreachable hosts.

_DEFAULT_NO_START is somewhat analogous to "demo mode" where mssh will return Host objects without actually allocating and starting the execution threads. Although they still can be started using start_run_on_hosts() but you should really be looking for return_unfinished instead for asynchronicity.

_etc_hosts is a custom dict to use instead of reading /etc/hosts file for hostname resolution.

ERROR_MESSAGES_TO_IGNORE lists regexes for ssh's own notices, which mssh leaves out of each host's stderr (also in --json and --greppable output). Each regex is searched anywhere in every stderr line, so anchor yours with ^ and $ to match whole lines (in the JSON config file, write a regex backslash as \\). The defaults are anchored to the lines OpenSSH prints, so a remote program's error that uses the same words, such as curl's Recv failure: Connection reset by peer, is kept. Connection resets are not hidden: kex_exchange_identification: read: Connection reset by peer is how sshd's MaxStartups turns away connections opened at once. Config files that --store_config_file or --generate_config_file wrote before 6.20 hold a copy of the old, unanchored defaults; mssh ignores such an unchanged copy and uses the current defaults.

__ERROR_MESSAGES_TO_IGNORE_REGEX is the compiled regex pattern for error messages to ignore. If custom defined, mssh will ignore ERROR_MESSAGES_TO_IGNORE and use supplied regex pattern instead. In a config file, give the pattern as a string. An empty ERROR_MESSAGES_TO_IGNORE hides no stderr line.

When ssh cannot connect to a host (it exits with 255 after ssh: connect to host HOST port PORT: REASON), the default output shows the line as SSH not reachable!, SSH connection timed out! or Cannot find host! followed by the port and ssh's reason, e.g. SSH connection timed out! (port 22: Connection timed out). Without the host name, unreachable hosts merge into one block. --json and --greppable output keep ssh's line, and a remote program's error that ends the same way is shown as printed.

__DEBUG_MODE is used to control whether to print host thread execution debug information to its stderr.