Running Remote Commands on Multiple Servers with SSH and Dsh

shell

Dsh is a tool that will allow you to use SSH to run commands on multiple servers at once. When properly used with ssh-agent, dsh can run unattended commands on many different hosts at once.

Installing Dsh

On Debian GNU/Linux-based systems (including Ubuntu), you can install dsh quite simply with apt-get install dsh.

If you aren't using a Debian system, or are on another Unix-based system, you can always install from the dsh sources using the instructions at the dsh home page.

Configuring Dsh

The main dsh configuration file is /etc/dsh/dsh.conf (or $HOME/.dsh/dsh.conf). Here is a working sample (note that this file, with different options, is installed for you when you install the Debian dsh package):

verbose = 0 remoteshell = ssh showmachinenames = 0 waitshell=0 # whether to wait for execution before moving onto next host #remoteshellopt=...

In the default config file supplied with Debian or Ubuntu, you will have to change the remoteshell option from rsh to ssh, and the waitshell option from 1 to 0. You can also just cut-n-paste the version listed above.

Other Dsh Configuration Files

There are some other files located in /etc/dsh. Here are their descriptions:

Dsh Command Line Switches

The following are the most useful dsh options:

SwitchDescription
-aAll machines
-g serversUse the group servers
-cUse concurrent connections
-wWait for one machine to finish before moving onto next
-vVerbose output
-MShow machine name, useful with -c

Configuring SSH Key Access and Running Dsh

Let's say that you wanted to use dsh to install package updates using apt-get update and apt-get dist-upgrade. Since you need root access on each remote host, you will have to create a public/private key pair on the machine you are launching dsh from, (using ssh-keygen -t dsa). Next, you append the public key just created (this will be in ~/.ssh/id_dsa.pub by default) to each remote host's /root/.ssh/authorized_keys file. You then add the private key to a local ssh-agent instance, so the passphrase is cached. Once this is done, you will be able to run the remote commands on all the configured remote machines without being prompted for a passphrase. Here are the steps, in detail:

Note: I recommend you tighten the security of your remote SSH servers by at least following the few steps detailed in Five Minutes to a More Secure SSH.

Now you are ready to actually run dsh. Test dsh access to the servers you specified in one of your dsh config file groups.

dsh -M -w -g servers -- 'w'.

You may be prompted to accept the remote host key if this is the first time you are logging in to the remote host(s). The -w switch ensures you process only one host at a time, so that you can answer y to the host key question at each machine in turn. You will not be prompted for a passphrase, since you have cached it in the your local SSH authentication agent.

Assuming the access works (and you should have the output of the w command on all three hosts visible in your console or xterm), try running apt-get update and apt-get dist-upgrade on each host. This time, you will not be prompted to accept each remote host key:

dsh -M -g servers -c -- 'apt-get update' dsh -M -g servers -c -- 'apt-get -y dist-upgrade'

You should see the output of the commands mixed together, as they are all running concurrently. The -M switch helps in this regard, since it prepends the machine name to each output line. When you are done, all machines in the host group servers should be updated. Because the dist-upgrade may occasionally prompt for your input (despite the -y), or even abort with an error, you may wish to run it first with -u and -s, coupled with the dsh -w switch, to show you which packages will be upgraded while at the same time simulating what will happen if the apt-get command is actually run.

dsh -M -g servers -w -- 'apt-get -u -s dist-upgrade'

If you want to only download the package updates, but not actually install them, use the -d switch to apt-get.

Sample Dsh Session

In this example, the /etc/dsh/group/servers file looks like this:

root@10.1.1.101 root@10.1.1.102 root@10.1.1.103

Here is the console session (spacing added for readability):

dmaxwell@lab0:~$ ssh-add Could not open a connection to your authentication agent. dmaxwell@lab0:~$ eval `ssh-agent` Agent pid 27831 dmaxwell@lab0:~$ ssh-add Enter passphrase for /home/dmaxwell/.ssh/id_dsa: Identity added: /home/dmaxwell/.ssh/id_dsa (/home/dmaxwell/.ssh/id_dsa) dmaxwell@lab0:~$ ssh-add -l 1024 dd:a9:8f:bd:e8:9e:7e:5e:cc:10:9f:e4:a4:c2:52:22 /home.dmaxwell/.ssh/id_dsa (DSA) dmaxwell@lab0:~$ dsh -M -w -g lab -- 'w' 10.1.1.101: 12:58:36 up 239 days, 3:28, 1 user, load average: 0.00, 0.00, 0.00 10.1.1.101: USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT 10.1.1.101: dmaxwell pts/0 lab0.unixlore.net 03Jan12 16:11m 0.00s 0.00s -bash The authenticity of host '10.1.1.102 (10.1.1.102)' can't be established. RSA key fingerprint is 62:9b:77:a2:b2:9b:bb:f7:a1:f2:de:24:ad:22:12:df. Are you sure you want to continue connecting (yes/no)? yes 10.1.1.102: Warning: Permanently added '10.1.1.102' (RSA) to the list of known hosts. 10.1.1.102: 12:58:55 up 324 days, 18:06, 1 user, load average: 0.00, 0.00, 0.00 10.1.1.102: USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT 10.1.1.102: dmaxwell pts/0 lab0.unixlore.net 15Jan12 16:11m 0.06s 0.06s -bash 10.1.1.103: 12:56:53 up 234 days, 3:10, 1 user, load average: 0.00, 0.00, 0.00 10.1.1.103: USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT 10.1.1.103: dmaxwell pts/1 lab0.unixlore.net 27Feb12 15:47m 0.01s 0.01s -bash dmaxwell@lab0:~$ dsh -M -w -g lab -- 'w' 10.1.1.101: 12:58:51 up 239 days, 3:28, 1 user, load average: 0.00, 0.00, 0.00 10.1.1.101: USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT 10.1.1.101: dmaxwell pts/0 lab0.unixlore.net 03Jan12 16:11m 0.00s 0.00s -bash 10.1.1.102: 12:59:09 up 324 days, 18:07, 1 user, load average: 0.00, 0.00, 0.00 10.1.1.102: USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT 10.1.1.102: dmaxwell pts/0 lab0.unixlore.net 15Jan12 16:12m 0.06s 0.06s -bash 10.1.1.103: 12:57:06 up 234 days, 3:10, 1 user, load average: 0.00, 0.00, 0.00 10.1.1.103: USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT 10.1.1.103: dmaxwell pts/1 lab0.unixlore.net 27Feb12 15:47m 0.01s 0.01s -bash dmaxwell@lab0:~$