Running Remote Commands on Multiple Servers with SSH and Dsh
Dsh is a tool that will allow you to use SSH to run commands on multiple servers at once. When properly used with ssh-agent, dsh can run unattended commands on many different hosts at once.
Installing Dsh
On Debian GNU/Linux-based systems (including Ubuntu), you can install dsh quite simply with apt-get install dsh.
If you aren't using a Debian system, or are on another Unix-based system, you can always install from the dsh sources using the instructions at the dsh home page.
Configuring Dsh
The main dsh configuration file is /etc/dsh/dsh.conf (or $HOME/.dsh/dsh.conf). Here is a working sample (note that this file, with different options, is installed for you when you install the Debian dsh package):
verbose = 0
remoteshell = ssh
showmachinenames = 0
waitshell=0 # whether to wait for execution before moving onto next host
#remoteshellopt=...
In the default config file supplied with Debian or Ubuntu, you will have to change the remoteshell option from rsh to ssh, and the waitshell option from 1 to 0. You can also just cut-n-paste the version listed above.
Other Dsh Configuration Files
There are some other files located in /etc/dsh. Here are their descriptions:
- /etc/dsh/machines.list: A list of all hostnames or IP addresses, one per line. These are the remote servers you wish to run commands on. You can prefix each with the username, as in user@servername.
- /etc/dsh/group/all: This is really a symlink to ../machines.list.
- /etc/dsh/group/servers: This is a file you create. It lists a chosen subset of all the server IP's or hostnames in machines.list, one per line. You will actually use the word servers as part of your dsh commands. You can choose any name for this file, and can have multiple group files. The format of the file is the same as machines.lst.
Dsh Command Line Switches
The following are the most useful dsh options:
| Switch | Description |
|---|---|
| -a | All machines |
| -g servers | Use the group servers |
| -c | Use concurrent connections |
| -w | Wait for one machine to finish before moving onto next |
| -v | Verbose output |
| -M | Show machine name, useful with -c |
Configuring SSH Key Access and Running Dsh
Let's say that you wanted to use dsh to install package updates using apt-get update and apt-get dist-upgrade. Since you need root access on each remote host, you will have to create a public/private key pair on the machine you are launching dsh from, (using ssh-keygen -t dsa). Next, you append the public key just created (this will be in ~/.ssh/id_dsa.pub by default) to each remote host's /root/.ssh/authorized_keys file. You then add the private key to a local ssh-agent instance, so the passphrase is cached. Once this is done, you will be able to run the remote commands on all the configured remote machines without being prompted for a passphrase. Here are the steps, in detail:
Note: I recommend you tighten the security of your remote SSH servers by at least following the few steps detailed in Five Minutes to a More Secure SSH.
- Login to your own workstation, as an unprivileged user
- If you don't already have a public/private SSH keypair, run ssh-keygen -t dsa. Use the default locations/filenames, and be sure to use a strong passphrase.
- Append ~/.ssh/id_dsa.pub to the /root/.ssh/authorized_keys file of the other servers you wish to run commands on.
- Make sure the remote authorized_keys files have permissions of 0600 by running chmod 0600 /root/.ssh/authorized_keys on each remote box.
- If you are running in console mode (versus X, which always runs under an ssh-agent in Debian and most other GNU/Linux distributions), type `eval ssh-agent` at the shell prompt (note the back-quotes). This will start a new ssh-agent process with a properly configured environment.
- Type ssh-add, this will prompt you to add your private key to the local authentication agent.
- Verify that the identity is added with ssh-add -l
Now you are ready to actually run dsh. Test dsh access to the servers you specified in one of your dsh config file groups.
dsh -M -w -g servers -- 'w'.
You may be prompted to accept the remote host key if this is the first time you are logging in to the remote host(s). The -w switch ensures you process only one host at a time, so that you can answer y to the host key question at each machine in turn. You will not be prompted for a passphrase, since you have cached it in the your local SSH authentication agent.
Assuming the access works (and you should have the output of the w command on all three hosts visible in your console or xterm), try running apt-get update and apt-get dist-upgrade on each host. This time, you will not be prompted to accept each remote host key:
dsh -M -g servers -c -- 'apt-get update'
dsh -M -g servers -c -- 'apt-get -y dist-upgrade'
You should see the output of the commands mixed together, as they are all running concurrently. The -M switch helps in this regard, since it prepends the machine name to each output line. When you are done, all machines in the host group servers should be updated. Because the dist-upgrade may occasionally prompt for your input (despite the -y), or even abort with an error, you may wish to run it first with -u and -s, coupled with the dsh -w switch, to show you which packages will be upgraded while at the same time simulating what will happen if the apt-get command is actually run.
dsh -M -g servers -w -- 'apt-get -u -s dist-upgrade'
If you want to only download the package updates, but not actually install them, use the -d switch to apt-get.
Sample Dsh Session
In this example, the /etc/dsh/group/servers file looks like this:
root@10.1.1.101
root@10.1.1.102
root@10.1.1.103
Here is the console session (spacing added for readability):
dmaxwell@lab0:~$ ssh-add
Could not open a connection to your authentication agent.
dmaxwell@lab0:~$ eval `ssh-agent`
Agent pid 27831
dmaxwell@lab0:~$ ssh-add
Enter passphrase for /home/dmaxwell/.ssh/id_dsa:
Identity added: /home/dmaxwell/.ssh/id_dsa (/home/dmaxwell/.ssh/id_dsa)
dmaxwell@lab0:~$ ssh-add -l
1024 dd:a9:8f:bd:e8:9e:7e:5e:cc:10:9f:e4:a4:c2:52:22 /home.dmaxwell/.ssh/id_dsa (DSA)
dmaxwell@lab0:~$ dsh -M -w -g lab -- 'w'
10.1.1.101: 12:58:36 up 239 days, 3:28, 1 user, load average: 0.00, 0.00, 0.00
10.1.1.101: USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
10.1.1.101: dmaxwell pts/0 lab0.unixlore.net 03Jan12 16:11m 0.00s 0.00s -bash
The authenticity of host '10.1.1.102 (10.1.1.102)' can't be established.
RSA key fingerprint is 62:9b:77:a2:b2:9b:bb:f7:a1:f2:de:24:ad:22:12:df.
Are you sure you want to continue connecting (yes/no)? yes
10.1.1.102: Warning: Permanently added '10.1.1.102' (RSA) to the list of known hosts.
10.1.1.102: 12:58:55 up 324 days, 18:06, 1 user, load average: 0.00, 0.00, 0.00
10.1.1.102: USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
10.1.1.102: dmaxwell pts/0 lab0.unixlore.net 15Jan12 16:11m 0.06s 0.06s -bash
10.1.1.103: 12:56:53 up 234 days, 3:10, 1 user, load average: 0.00, 0.00, 0.00
10.1.1.103: USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
10.1.1.103: dmaxwell pts/1 lab0.unixlore.net 27Feb12 15:47m 0.01s 0.01s -bash
dmaxwell@lab0:~$ dsh -M -w -g lab -- 'w'
10.1.1.101: 12:58:51 up 239 days, 3:28, 1 user, load average: 0.00, 0.00, 0.00
10.1.1.101: USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
10.1.1.101: dmaxwell pts/0 lab0.unixlore.net 03Jan12 16:11m 0.00s 0.00s -bash
10.1.1.102: 12:59:09 up 324 days, 18:07, 1 user, load average: 0.00, 0.00, 0.00
10.1.1.102: USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
10.1.1.102: dmaxwell pts/0 lab0.unixlore.net 15Jan12 16:12m 0.06s 0.06s -bash
10.1.1.103: 12:57:06 up 234 days, 3:10, 1 user, load average: 0.00, 0.00, 0.00
10.1.1.103: USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
10.1.1.103: dmaxwell pts/1 lab0.unixlore.net 27Feb12 15:47m 0.01s 0.01s -bash
dmaxwell@lab0:~$
![[FSF Associate Member #286]](http://static.fsf.org/nosvn/associate/fsf-286.png)