This is a read-only archive. Find the latest Linux articles, documentation, and answers at the new Linux.com!

Linux.com

Feature: Tools & Utilities

Parallel SSH execution and a single shell to control them all

By Ben Martin on October 30, 2008 (8:00:00 AM)

Share    Print    Comments   

Many people use SSH to log in to remote machines, copy files around, and perform general system administration. If you want to increase your productivity with SSH, you can try a tool that lets you run commands on more than one remote machine at the same time. Parallel ssh, Cluster SSH, and ClusterIt let you specify commands in a single terminal window and send them to a collection of remote machines where they can be executed.

Cluster SSH

Cluster SSH works a little differently from Parallel ssh. Instead of using a single terminal to issue commands to multiple machines, it creates an xterm for each node in the cluster and a controlling window to let you give input for all the nodes in the cluster.

Cluster SSH is available in Ubuntu Hardy Universe, the Fedora 9 repositories, and as a 1-Click install for openSUSE 11. I used the 64-bit package from the Fedora 9 repositories.

You specify one or more machines to connect to on the command line, and Cluster SSH opens a new xterm window for each connection and a single Tk window to let you control Cluster SSH and send input to all the xterm windows at once. The windows generated by running cssh p1 p2, where p1 and p2 are remote machines, is shown in the screenshot below. When it starts up, Cluster SSH will position the xterm windows in a tiled formation for you as shown.

The Cluster SSH File menu lets you see the history of the input you have provided. The history display expands the Cluster SSH window to contain a text area that displays only your input. You can also quit your session from the File menu. Closing the Cluster SSH window will also end the session and close all the xterm windows associated with it.

The Hosts menu lets you Retile Hosts, which repositions your xterms and the main control window in a manner similar to the one used at startup. The Hosts menu also lets you add a new connection to another host to the current session, as well as disable and enable any of the hosts in the current session. The latter option is useful if you are administering a collection of several machines and you want to run a command on all machines except one. For example, you might want to run commands on client machines but when you are playing with iptables rules you might want to exclude the firewall machine temporarily.

Cluster SSH has two machine-wide configuration files, /etc/clusters and /etc/csshrc, which are used to define the machines in a cluster (just a group of hosts you want to connect to) and general settings respectively. You can also have ~/.csshrc as a per-user configuration file. There is no corresponding ~/.clusters file, but you can use the extra_cluster_file configuration directive in your ~/.csshrc file to tell Cluster SSH where to look for your personal version of /etc/clusters.

The clusters file has the form: groupname user1@host1 user2@host2 ... where the usernames are optional. With a clusters file you can simply specify cssh groupname to connect to all the hosts in that group. The below commands show a personal clusters file and a ~/.csshrc file that tells Cluster SSH to use that clusters file. The final line invokes Cluster SSH to connect to the specified hosts as various users.

# cat /root/.cssh-clusters rougenet p1 ben@p2 # cat /root/.csshrc extra_cluster_file = ~/.cssh-clusters # cssh rougenet

The software defines four hotkeys, which you can rebind to other keys in your csshrc file. Ctrl-q closes the current session, Ctrl-+ lets you add a new host to the current session, Alt-n pastes the remote hostname into each xterm when the main Cluster SSH window is active, and Alt-r retiles the xterm windows. As an example of Alt-n, if you typed echo Alt-n into the CSSH window in the screenshot, the left terminal would show echo vfedora864prx1 in its xterm, and the right terminal would show echo vfedora864prx2. There is no keybinding to insert the hostname that you are running Cluster SSH from; that might be handy if you were rsyncing data from your controlling machine to all the nodes in the session.

You can specify the user to log in to the remote hosts with using the -l option. A username specified this way has higher priority than any usernames specified in your clusters file.

A collection of terminal configuration parameters for your csshrc file let Cluster SSH use a different terminal than xterm for the connections. Unfortunately, getting gnome-terminal or konsole to work as the terminal is not as trivial as specifying terminal = konsole because Cluster SSH sends information to the terminal with the assumption that it will accept xterm options. I found that terminal = Eterm works as a drop-in replacement if you are fond of that terminal. However, issues like Cluster SSH always passing -font options when starting the terminal make gnome-terminal unsuitable as a drop-in replacement. To get other terminals to work you might have to create a wrapper script that forwards to the terminal only arguments that will not upset it.

Share    Print    Comments   

Comments

on Parallel SSH execution and a single shell to control them all

Note: Comments are owned by the poster. We are not responsible for their content.

Parallel SSH execution and a single shell to control them all

Posted by: Anonymous [ip: 64.89.94.194] on October 30, 2008 01:05 PM
Another product that you should check out is called Func https://fedorahosted.org/func
from the product description:
Func is a secure, scriptable remote control framework and API. It is intended to replace SSH scripted infrastructure for a variety of datacenter automation tasks (such as taking hardware inventory, running queries, or executing remote commands) that are run against a large amount of systems at the same time. Func provides both a command line tool and a simple and powerful Python API. It is also intended to be very easy to integrate with your provisioning environment and tools like Cobbler.

So say you need to update just your test machines you do:
func "*.test.example.org" call yumcmd update

#

Re: Parallel SSH execution and a single shell to control them all

Posted by: Anonymous [ip: 67.173.12.114] on October 30, 2008 02:20 PM
Func is more feature rich then some of these applications but its also far more difficult to setup. Its hard to beat "yum install pssh" and be able to do what's listed in this article.

#

Parallel SSH execution and a single shell to control them all

Posted by: Anonymous [ip: 76.98.241.86] on October 31, 2008 06:32 PM
Capistrano (http://www.capify.org) is another tool to do this. It's billed as a deployment tool but is essentially a tool for defining sequences of operations to execute on remote hosts in parallel.

#

This isn't good enough?

Posted by: Anonymous [ip: 76.103.250.180] on November 01, 2008 01:46 AM
For x in `cat hosts`; do ssh -f $x "do stuff" & done && wait;

#

Re: This isn't good enough?

Posted by: Anonymous [ip: 98.215.194.203] on November 01, 2008 09:22 PM
no, its not. Lets take a sample size of 32 hosts and run a quick command on each:


$ time for f in `cat hosts`; do ssh $f 'ls / > /dev/null'; done
real 2m45.195s

That's roughly 5.15 seconds per host. If this were a 5000 node network we're looking at about 7.1 hours to complete this command. Lets do the same test with pssh and a max parallel of 10:

$ time pssh -p 10 -h hosts "ls > /dev/null"
real 0m17.220s

That's some considerable savings. lets try each one in parallel and set the max to 32:
$ time pssh -p 32 -h hosts "ls > /dev/null"
real 0m7.436s

If one run took about 5 seconds, doing them all at the same time also took about 5 seconds, just with a bit of overhead. I don't have a 5000 node network (anymore) but you can see there are considerable savings by doing some things in parallel. You probably wouldn't ever run 5000 commands in parallel but really thats a limit of your hardware and network. if you had a beefy enough host machine you probably could run 50, 100 or even 200 in parallel if the machine could handle it.

#

Re(1): This isn't good enough?

Posted by: Anonymous [ip: 99.156.88.107] on November 02, 2008 07:54 PM
It's absolutely not good enough. 4 or so years ago a coworker and I wrote a suite of parallel ssh tools to help perform security related duties on the very large network in our global corp. With our tools on a mosix cluster using load balanced ssh-agents across multiple nodes we could run upto 1000 outbound sessions concurrently. This made tasks such as looking for users processes or cronjobs on 10,000+ hosts world wide a task that could be done in a reasonable amount of time, as opposed to taking more than a day.

#

Parallel SSH execution and a single shell to control them all

Posted by: Anonymous [ip: 24.14.35.105] on November 02, 2008 12:43 PM
I use the parallel option to xargs for this. Tried shmux, and some other tools, but xargs seems to work best for me. Just use a more recent gnu version. Some older gnu versions, some aix version, etc... have some issues. Only real gotcha that I've run into is that it will stop the whole run if a command exits non-zero. Just write a little wrapper that exits 0 and you're good to go. I've used this in 2 ~1000 server environments to push code(pipe tar over ssh for better compatibility), and remotely execute commands.

#

Parallel SSH execution and a single shell to control them all

Posted by: Anonymous [ip: 66.151.59.138] on November 04, 2008 04:09 PM
Ticketmaster wrote a really good tool to do parallel systems administration tasks like this called "onall".

It is released under the gplv3 and can be downloaded from:
http://code.google.com/p/onall/

I do something like this to execute a command on all hosts and set the timeout to 10 seconds:
host -l internal.dns.zone | awk '{print $1}' | onall -t10 "uptime"

#

Parallel SSH execution and a single shell to control them all

Posted by: Anonymous [ip: 87.230.108.21] on November 06, 2008 10:08 AM
What I'm curious about is this:
"""
if you want to interactively edit the same file on multiple machines, it might be quicker to use a parallel SSH utility and edit the file on all nodes with vi rather than concoct a script to do the same edit.
"""

I would have found a short note on which of these three is capable of doing so very helpfull. Cluster SSH's description sounds as though it would be the tool that could do it. But I just don't have the time to test it just yet.
Anyone tried that yet? Or knows to which tool this statement refers to?

#

Re: Parallel SSH execution and a single shell to control them all

Posted by: Anonymous [ip: 75.25.174.30] on November 14, 2008 05:33 AM
I don't see ssh keys mentioned anywhere...I'm not getting how this allows authentication to happen. Is this thing secure?

#

This story has been archived. Comments can no longer be posted.



 
Tableless layout Validate XHTML 1.0 Strict Validate CSS Powered by Xaraya