Linux.com

Why you would need a utility like this when, using openSSH, you can create a file containing your commands and use a bash for loop to run it on a list of remote hosts, one at a time? One advantage of a parallel SSH utility is that commands can be run on several hosts at the same time. For a short-running task this might not matter much, but if a task needs an hour to complete and you need to run it on 20 hosts, parallel execution beats serial by a mile. Also, if you want to interactively edit the same file on multiple machines, it might be quicker to use a parallel SSH utility and edit the file on all nodes with vi rather than concoct a script to do the same edit.

Many of these parallel SSH tools include support for copying to many hosts at once (a parallel version of scp) or using rsync on a collection of hosts at once. Because the parallel SSH implementations know about all the hosts in a group, some of them also offer the ability to execute a command "on one host" and will work out which host to pick using load balancing. Finally, some parallel SSH projects let you use barriers so that you can execute a collection of commands and explicitly have each node in the group wait until all the nodes have completed a stage before moving on to the next stage of processing.

Parallel ssh (pssh)

The Parallel ssh project includes parallel versions of the shell (pssh), scp (pscp), rsync (prsync), and kill (pnuke).

pssh is packaged for openSUSE as a 1-Click install, is available in Ubuntu Hardy Universe and the Fedora 9 repositories. I used the 64-bit package from the Fedora 9 repositories.

All of the Parallel ssh commands have the form command -h hosts-file options, where the hosts-file contains a list of all the hosts that you want to have the command executed on. For example, the first pssh command below will execute the date command on p1 and p2 as the ben user. The optional -l argument specifies the username that should be used to log in to the remote machines.

# cat hosts-file
p1
p2

# pssh -h hosts-file -l ben date
[1] 21:12:55 [SUCCESS] p2 22
[2] 21:12:55 [SUCCESS] p1 22

# pssh -h hosts-file -l ben -P date
p2: Thu Oct 16 21:14:02 EST 2008
p2: [1] 21:13:00 [SUCCESS] p2 22
p1: Thu Sep 25 15:44:36 EST 2008
p1: [2] 21:13:00 [SUCCESS] p1 22

Normally the standard output from the remote hosts is not shown to you. The -P option in the last invocation displays the output from both remote hosts as well as the exit status. If you are running more complex commands you might like to use -i instead to see each remote host's output grouped nicely under its hostname rather than mixed up as the output comes in from the hosts. You can also use the --outdir pssh option to specify the path of a directory that should be used to save the output from each remote host. The output for each host is saved in separate file named with the remote machine's hostname.

You can use the --timeout option to specify how long a command can take. It defaults to 60 seconds. This means that if your command fails to complete within 60 seconds on a host, pssh will consider it an error and report it as such, as shown below. You can increase the timeout to something well above what might be acceptable (for example to 24 hours) to avoid this problem.

# pssh -h hosts-file -l ben -i "sleep 65; date"
[1] 21:19:26 [FAILURE] p1 22 Timeout
[2] 21:19:26 [FAILURE] p2 22 (4, 'Interrupted system call')

The pscp command takes the same -h, -l, and --timeout options and includes a --recursive option to enable deep copying from the local host. At the end of the command you supply the local and remote paths you would like to copy. The first pscp command in the example below copies a single file to two remote hosts in parallel. The following ssh command checks that the file exists on the p1 machine. The second pscp command fails in a verbose manner without really telling you the simple reason why. Knowing that I was trying to copy a directory over, I added the --recursive option to the command and it executed perfectly. The final ssh command verifies that the directory now exists on the p1 remote host.

$ mkdir  example-tree
$ date > example-tree/df1.txt
$ date > example-tree/df2.txt
$ mkdir  example-tree/subdir1
$ date > example-tree/subdir1/df3.txt

$ pscp -h hosts-file -l ben example-tree/df1.txt /tmp/df1.txt
[1] 21:28:36 [SUCCESS] p1 22
[2] 21:28:36 [SUCCESS] p2 22

$ ssh p1 "cat /tmp/df1.txt"
Thu Oct 16 21:27:25 EST 2008

$ pscp -h hosts-file -l ben example-tree /tmp/example-tree
...
python: Python/ceval.c:2918: set_exc_info: Assertion `frame != ((void *)0)' failed.
Aborted

$ pscp -h hosts-file -l ben --recursive  example-tree /tmp/example-tree
[1] 21:29:57 [SUCCESS] p1 22
[2] 21:29:57 [SUCCESS] p2 22

$ ssh p1 "ls -l /tmp/example-tree"
total 24
-rw-r--r-- 1 ben ben   29 2008-09-25 16:01 df1.txt
-rw-r--r-- 1 ben ben   29 2008-09-25 16:01 df2.txt
drwxr-xr-x 2 ben ben 4096 2008-09-25 16:01 subdir1

The prsync command uses only a handful of the command-line options from rsync. In particular, you cannot use the verbose or dry-run options to get details or see what would have been done. The command shown below will rsync the example-tree into /tmp/example-tree on the remote hosts in a manner similar to the final command in the pscp example.

$ prsync -h hosts-file -l ben -a --recursive  example-tree /tmp

The main gain of the prsync command over using the normal rsync command with pssh is that prsync gives a simpler command line and lets you sync from the local machine to the remote hosts directly. Using pssh and rsync, you are running the rsync command on each remote machine, so the remote machine will need to connect back to the local machine in order to sync.

The pslurp command is sort of the opposite to the pscp in that it grabs a file or directory off all the remote machines and copies it to the local machine. The below command grabs the example-tree directory from both p1 and p2 and stores them into /tmp/outdir. The -r option is shorthand for --recursive. As you can see, for each remote host a new directory is created with the name of the host, and inside that directory a copy of example-tree is made using the local directory name supplied as the last argument to pslurp.

# mkdir /tmp/outdir
# pslurp -h hosts-file -L /tmp/outdir -l ben -r /tmp/example-tree example-tree

# l /tmp/outdir
drwxr-xr-x 3 root root 4.0K 2008-10-16 21:47 p1/
drwxr-xr-x 3 root root 4.0K 2008-10-16 21:47 p2/
# l /tmp/outdir/p2
drwxr-xr-x 3 root root 4.0K 2008-10-16 21:47 example-tree/
# l /tmp/outdir/p2/example-tree/
-rw-r--r-- 1 root root   29 2008-10-16 21:47 df10.txt
-rw-r--r-- 1 root root   29 2008-10-16 21:47 df1.txt
...
drwxr-xr-x 2 root root 4.0K 2008-10-16 21:47 subdir1/

You can use environment variables to make things easier with Parallel ssh. You can use the PSSH_HOSTS variable to name the hosts file instead of using the -h option. Likewise, the PSSH_USER environment variable lets you set the username to log in as, like the -l pssh command line option.

Share    Print    Comments