One of the things you quickly learn as you become a more sophisticated systems administrator is that performing the same task on several servers is both tedious and error-prone. Of course, if your infrastructure has been deployed in an ad-hoc manner, sometimes you don’t have a choice; the task is unique on each machine simply because each machine has a unique configuration. However, once you have made the transition to stock configurations on multiple machines, you gain the ability to use new tools to perform tasks in parallel on a number of systems at once. The tendency for many admins is to set up public-key authentication and hand-roll a script. This isn’t necessarily a bad approach; sometimes a quick-and-dirty script is all you really need to get the job done. On the other hand, there are tools which can make the process more elegant, consistent, and professional. The two I know of are ClusterSSH and Capistrano.
I’m fairly new to Capistrano, but I’ve been using it to perform some simple tasks for a couple of months, and I’m really pleased with the results. I first encountered Capistrano when I was looking for a simple way to remove the SNMP agent from a group of about 20 machines. I knew I could whip up a script, but I had remembered reading a blog post somewhere (sorry, I don’t remember where) about ClusterSSH; several Google searches later I had found both ClusterSSH and Capistrano. At the time, I was learning Ruby (in order to get involved in Puppet development), so Capistrano seemed the natural choice.
According to the website, Capistrano was originally written to help deploy Ruby on Rails applications. Like many tools, it has grown beyond its initial mission, and is now a powerful tool for systems administration. Capistrano makes some assumptions about your infrastructure. First, you must be using SSH to access your systems, because Capistrano does not support older methods such as telnet (and if you are still using telnet, shame on you!) Second, your systems must have a POSIX shell in the default system path. For most *nix systems these days, this is a given. Finally, to *really* utilize Capistrano correctly, you should be using public-key authentication to access your servers. You don’t necessarily need to use password-less keys (in fact, I suggest you resist that temptation in general). Just use ssh-agent to keep your passphrase in memory. All of these assumptions are true for my environment, so I got started.
Installation was trivial using RubyGems. Next I needed to create a “capfile” - the Capistrano equivalent of a “Makefile”. The “capfile” gives Capistrano information about your environment and defines “roles” and “tasks”. “Roles” describe groups of systems, “webservers” or “firewalls” for example. “Tasks” are the actions you wish to perform. After reading through the Basics on the Capistrano website, it was fairly simple to define a task to do what I wanted:
task :remove_snmp do
run "sudo aptitude -y purge snmp"
end
One complication of my environment is that most of the systems are not directly accessible from outside. We have an SSH “bastion host” which acts as a gateway to the internal network. Luckily, Capistrano is ready for this. I added the following to the beginning of my capfile:
set :gateway, "ssh-gateway.example.com"
This tells Capistrano to first establish an SSH connection to ssh-gateway.example.com, and connect from that machine to the systems we want to run commands on. “But how does Capistrano know what systems to run commands on?” you ask. Good question! That is what we need a “role” for:
role :de_snmp, "server1.example.com", "server2.example.com", "server3.example.com", "server4.example.com", "server5.example.com", "server6.example.com"
Before you run a task on all of your systems at once, you probably want to test it first. I usually try the task by hand on one or two systems. Once I am happy with the procedure and the results, I pick several less-critical (or easy to rollback) hosts to do a second pass on. Finally, after all is well with those hosts, I run the task on the full group. Here is our completed “capfile” for the second pass:
1
2
3
4
5
6
7
| set :gateway, "ssh-gateway.example.com"
role :de_snmp, "server1.example.com", "server2.example.com", "server3.example.com", "server4.example.com", "server5.example.com", "server6.example.com"
task :remove_snmp, :roles => :de_snmp do
run "sudo aptitude -y purge snmp"
end |
As you can see, we’ve modified our “task” definition to apply to the “role” we created. All we have to do now is run cap remove_snmp and Capistrano will do its job! There is plenty of output, so it is fairly easy to review what is happening (though it can be challenging to follow the events on a single server unless you know your way around grep; you do, right?)
This is just the tip of the iceberg. Capistrano is very sophisticated, allowing your tasks to be self-documenting, divided into namespaces, use variables, and more! Perhaps the most powerful feature is the ability to define “transactions” and “rollback” functions. Although you still have to manually define what a “rollback” means, Capistrano allows you to do that once, and use the capability as often as necessary. In addition, you have the full power of Ruby at your command in Capistrano scripts. I haven’t had the need to explore these features but as I do I’ll be sure to share my experiences here!