Thoughts on System Administration

The lack of posts is due to a general lull in my work lately. I’ve got a couple of posts in the queue, but they need more development. Stay tuned!

One of the things you quickly learn as you become a more sophisticated systems administrator is that performing the same task on several servers is both tedious and error-prone. Of course, if your infrastructure has been deployed in an ad-hoc manner, sometimes you don’t have a choice; the task is unique on each machine simply because each machine has a unique configuration. However, once you have made the transition to stock configurations on multiple machines, you gain the ability to use new tools to perform tasks in parallel on a number of systems at once. The tendency for many admins is to set up public-key authentication and hand-roll a script. This isn’t necessarily a bad approach; sometimes a quick-and-dirty script is all you really need to get the job done. On the other hand, there are tools which can make the process more elegant, consistent, and professional. The two I know of are ClusterSSH and Capistrano.

I’m fairly new to Capistrano, but I’ve been using it to perform some simple tasks for a couple of months, and I’m really pleased with the results. I first encountered Capistrano when I was looking for a simple way to remove the SNMP agent from a group of about 20 machines. I knew I could whip up a script, but I had remembered reading a blog post somewhere (sorry, I don’t remember where) about ClusterSSH; several Google searches later I had found both ClusterSSH and Capistrano. At the time, I was learning Ruby (in order to get involved in Puppet development), so Capistrano seemed the natural choice.

According to the website, Capistrano was originally written to help deploy Ruby on Rails applications. Like many tools, it has grown beyond its initial mission, and is now a powerful tool for systems administration. Capistrano makes some assumptions about your infrastructure. First, you must be using SSH to access your systems, because Capistrano does not support older methods such as telnet (and if you are still using telnet, shame on you!) Second, your systems must have a POSIX shell in the default system path. For most *nix systems these days, this is a given. Finally, to *really* utilize Capistrano correctly, you should be using public-key authentication to access your servers. You don’t necessarily need to use password-less keys (in fact, I suggest you resist that temptation in general). Just use ssh-agent to keep your passphrase in memory. All of these assumptions are true for my environment, so I got started.

Installation was trivial using RubyGems. Next I needed to create a “capfile” - the Capistrano equivalent of a “Makefile”. The “capfile” gives Capistrano information about your environment and defines “roles” and “tasks”. “Roles” describe groups of systems, “webservers” or “firewalls” for example. “Tasks” are the actions you wish to perform. After reading through the Basics on the Capistrano website, it was fairly simple to define a task to do what I wanted:

task :remove_snmp do
  run "sudo aptitude -y purge snmp"
end

One complication of my environment is that most of the systems are not directly accessible from outside. We have an SSH “bastion host” which acts as a gateway to the internal network. Luckily, Capistrano is ready for this. I added the following to the beginning of my capfile:

set :gateway, "ssh-gateway.example.com"

This tells Capistrano to first establish an SSH connection to ssh-gateway.example.com, and connect from that machine to the systems we want to run commands on. “But how does Capistrano know what systems to run commands on?” you ask. Good question! That is what we need a “role” for:

role :de_snmp, "server1.example.com", "server2.example.com", "server3.example.com", "server4.example.com", "server5.example.com", "server6.example.com"

Before you run a task on all of your systems at once, you probably want to test it first. I usually try the task by hand on one or two systems. Once I am happy with the procedure and the results, I pick several less-critical (or easy to rollback) hosts to do a second pass on. Finally, after all is well with those hosts, I run the task on the full group. Here is our completed “capfile” for the second pass:

1
2
3
4
5
6
7

set :gateway, "ssh-gateway.example.com"
 
role :de_snmp, "server1.example.com", "server2.example.com", "server3.example.com", "server4.example.com", "server5.example.com", "server6.example.com"
 
task :remove_snmp, :roles => :de_snmp do
  run "sudo aptitude -y purge snmp"
end

As you can see, we’ve modified our “task” definition to apply to the “role” we created. All we have to do now is run cap remove_snmp and Capistrano will do its job! There is plenty of output, so it is fairly easy to review what is happening (though it can be challenging to follow the events on a single server unless you know your way around grep; you do, right?)

This is just the tip of the iceberg. Capistrano is very sophisticated, allowing your tasks to be self-documenting, divided into namespaces, use variables, and more! Perhaps the most powerful feature is the ability to define “transactions” and “rollback” functions. Although you still have to manually define what a “rollback” means, Capistrano allows you to do that once, and use the capability as often as necessary. In addition, you have the full power of Ruby at your command in Capistrano scripts. I haven’t had the need to explore these features but as I do I’ll be sure to share my experiences here!

I was pretty unhappy with Drupal as a blog software so I’ve switched to Wordpress. Since I have a fairly limited audience and not a lot of old content, I’m pretty sure this won’t disrupt people much. The code segments in the old posts don’t look as nice as I would like, but I’m not going to focus on fixing that unless I get a lot of complaints.

Thanks for bearing with me!

One of the most exciting features of Puppet is the way it enables the sharing of systems administration tools - “recipes” in the Puppet parlance. Puppet allows us to collect manifests, definitions, and even plugins into modules which can then be shared with others.

David Schmitt is the author of the Complete Configuration found on the Puppet wiki. This is an amazing example of what can be done with Puppet, but can be a bit overwhelming at first. Happily, it is reasonably easy to integrate David’s modules into your Puppet configuration one at a time. Today I’d like to show you how to use David’s apt module to manage the dpkg & apt databases, as well as the keyrings associated with them.

In order to get started with David’s modules, we’ll need to take a few steps. The first step is upgrading to the latest release of Puppet. In an earlier post I advocated using the Debian packages; on further reflection, it might be best to install from source (or roll your own package). Puppet is a bit of a moving target right now, and you’ll definitely want to keep up. Installing from source is easy enough. On Debian Etch, I used the following steps to install from source:

1
2
3
4
5
6
7
8
9
10

aptitude remove puppet puppetmaster
aptitude install libopenssl-ruby libshadow-ruby1.8 libxmlrpc-ruby
curl -kO https://reductivelabs.com/downloads/facter/facter-1.3.7.tgz
tar xzf facter-1.3.7.tgz
curl -O http://reductivelabs.com/downloads/puppet/puppet-0.24.1.tgz
tar xzf puppet-0.24.1.tgz
cd facter-1.3.7
ruby install.rb
cd ../puppet-0.24.1
ruby install.rb

In short: remove the old version, install the prerequisite ruby libraries, download the latest source of Facter and Puppet, and install them both. The init scripts didn’t work after this, so I rolled my own.

The new version of Puppet allows module authors to distribute plugins within their modules. However, to make use of this, we need to add a couple of lines to the [main] section of our puppet.conf file:

  pluginsync = true
  pluginsource = puppet://$server/plugins

You’ll want to create a “modules” subdirectory to store your modules in. According to the Puppet Best Practices page, this directory should be in the same place as your “manifests” directory. For me, that means /etc/puppet:

total 36K
drwxr-xr-x  6 root root 4.0K 2008-01-02 13:47 .
drwxr-xr-x 71 root root 4.0K 2008-01-02 13:44 ..
drwxr-xr-x  7 root root 4.0K 2008-01-02 13:47 files
-rw-r--r--  1 root root  664 2008-01-02 13:47 fileserver.conf
drwxr-xr-x  4 root root 4.0K 2008-01-02 13:57 manifests
drwxr-xr-x  4 root root 4.0K 2008-01-02 13:47 modules
-rw-r--r--  1 root root  250 2008-01-02 13:47 puppet.conf

Next, we’ll want to download the modules we wish to use. All of David’s modules depend on the “common” module, so we will want “common” and “apt”. David makes it easy to pull down each module independently. Here’s what I did:

1
2
3
4
5
6
7
8
9
10
11
12

cd /etc/puppet/modules
mkdir common
cd common
git init
git remote add -t common -m common -f origin git://git.black.co.at/manifests/
git pull
cd /etc/puppet/modules
mkdir apt
cd apt
git init
git remote add -t apt -m apt -f origin git://git.black.co.at/manifests/
git pull

You should now have local copies of both the “common” module and the “apt” module. If you read the documentation, you will notice that these modules require the installation of the “lsb-release” package. I’m not sure why David doesn’t have the module install the package, but he doesn’t. So, let’s have puppet install it. A new “package” resource in our default node should do the trick:

25
26
27

  package { sudo: ensure => installed }
  package { lsb-release: ensure => installed }
}

Next, we need to integrate the new modules into our puppet configuration. Now that we’re getting into more complex configurations, we’ll want to start breaking our puppet configuration into separate files. In the same directory as site.pp (/etc/puppet/manifests), create a file called modules.pp containing:

1
2

import “common”
import “apt”

As you add more modules, you’ll import them here.

We won’t be explicitly using the common module at this time, so all we need to do is add two lines to site.pp. At the top of the file, we’ll want:

1

import modules.pp

to pull in the file we just created. We also want to update the default node to make use of the apt module:

7
8
9
10

node default {
  include apt
 
  file { ‘/root/.ssh/authorized_keys’:

It turns out that David’s modules also depend on a filebucket resource named “server”. This is a site-wide resource, so site.pp is the ideal place to put it:

3
4
5
6
7

filebucket { server:
  server => “server1.example.com”
}
 
node default {

After all these modifications, our site.pp looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

import modules.pp
 
filebucket { server:
  server => “server1.example.com”
}
 
node default {
  include apt
 
  file { ‘/root/.ssh/authorized_keys’:
    owner => root,
    group => root,
    mode => 644,
    source => ‘puppet:///root/.ssh/authorized_keys’
  }
 
  file { ‘/etc/sudoers’:
    owner => root,
    group => root,
    mode => 440,
    source => ‘puppet:///etc/sudoers’
    require => [ Package[“sudo”] ]
  }
 
  package { sudo: ensure => installed }
  package { lsb-release: ensure => installed }
}

I had to do one more thing to get this to really work. Reading through David’s module, you would expect /var/lib/puppet/modules/apt to be created by Puppet. Unfortunately, this doesn’t seem to happen. I had to create this directory by hand. I’ll let you know if/when I figure out why.

For basic usage of the module, this is all you need! David’s module defaults to a very sane basic apt configuration. The only tweak I felt was necessary was a modification to modules/apt/templates/sources.list.erb to use a US Debian mirror. You can modify this file to your liking for your default sources.list. You can also use the $custom_sources_list variable in a node definition to provide a customized file for a specific node:

node “oddball.example.com” {
  $custom_sources_list = “#This box is located in Canada, so use a local mirror
deb http://ftp.ca.debian.org/debian etch main contrib non-free
deb http://security.debian.org/ etch/updates main contrib non-free” include “apt”
}

David has a number of other modules I’d like to try; as I get them working I’ll be sure to share my experiences here.

As I write these entries about Puppet I want to make it clear that the configurations I post may not necessarily be “best practices”. I am attempting to follow a logical evolution of steps involved in becoming familiar with Puppet. As the configuration grows more complex I will introduce the various best practices which make complex Puppet configurations manageable. If you are looking to dive straight in to a fully-realized puppet configuration, take a look at the Complete Configuration page of the Puppet wiki.

In my last post, I described how to distribute an SSH authorized_keys file using Puppet. My reasoning for this was to help us utilize our existing set of home-grown scripts to administer machines - with Puppet we don’t have to wonder if our keys are set up on every host. Well, I’m sure a few of you spotted the flaw in this. Most of us have learned not to run scripts as root when it isn’t necessary. Instead we use sudo to grant limited root powers for specific commands. Sudo is a well-designed piece of software; it’s configuration file, the sudoers file, is setup in such a way that the same sudoers file may be used on many machines. This makes it ripe for management in Puppet.

Building on top of the configuration we created in my last post, here is the site.pp manifest after adding sudo to the mix:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

node default {
  file { ‘/root/.ssh/authorized_keys’:
    owner => root,
    group => root,
    mode => 644,
    source => ‘puppet:///root/.ssh/authorized_keys’
  }
 
  file { ‘/etc/sudoers’:
    owner => root,
    group => root,
    mode => 440,
    source => ‘puppet:///etc/sudoers’
    require => [ Package[“sudo”] ]
  }
 
  package { sudo: ensure => installed }
}

The first definition is mostly familiar; we defined a file resource like it last time. There is something new here, however:

14

    require => [ Package[“sudo”] ]

The require parameter describes a dependency. Before this resource is applied, Puppet will look for a package resource with the name “sudo” and apply that resource first. Next we define that package resource:

17

  package { sudo: ensure => installed }

All we want is to make sure that sudo is installed. Perhaps the coolest part of this is that it doesn’t matter that I am using Debian here, Puppet supports a wide range of package systems under the hood, and it will choose the one most appropriate for the system it is configuring. If I add a FreeBSD machine to my network, I should not have to make any changes to my Puppet configuration - I can still depend on sudo getting installed!

Next, we need to add an etc mount to our Puppet fileserver. Here is the resulting fileserver.conf:

1
2
3
4
5
6
7

[root]
  path /etc/puppet/files/root
  allow 10.0.0.0/8
 
[etc]
  path /etc/puppet/files/etc
  allow 10.0.0.0/8

Put your sudoers file in /etc/puppet/files/etc and wait for your clients to check in with puppetmasterd. Alternately, you can log in to a client and run puppetd –test to pull down the new configuration. If sudo was not installed, it should be, and /etc/sudoers will be downloaded as well.

In my last post I mentioned Puppet. If you are a systems administrator and you haven’t taken a look at Puppet, stop reading and go look.

No, really. Open a new tab and head over to the About Puppet page of the Puppet wiki. Because Puppet is one of those tools which will utterly change the way you work.

Puppet can be described in many ways, but the most accurate (and powerful) way to think about it is as a language for describing systems. In fact, Puppet may end up redefining the term “systems programming” completely. Puppet provides the toolkit you need to build the Holy Grail of systems administration: the all-powerful, all-important, but seldom-realized Configuration Management Database!

Now that you are rolling your eyes and muttering under your breath, thinking you’ve heard this all before, I’d like to show you what Puppet can actually do. I think you will be pleasantly surprised…

In general usage, Puppet is a client/server application. Puppet clients running puppetd periodically contact a Puppet server running puppetmasterd. Puppet can also be used in stand-alone mode with the puppet command-line utility, but we’ll have to come back to that in a later post. For now, we need to get Puppet installed before we can move on to the good stuff.

I’m using Puppet to manage a number of Debian Etch servers. Some of these servers are virtualized Xen instances (domUs), the rest are the Xen hosts (dom0s). Lucky for me, Puppet is available as a Debian package. Unfortunately, like many Debian packages, the package is lagging behind the rather rapid release cycle of Puppet. However, the Puppet team has provided excellent instructions on installing Puppet from the “Unstable” release of Debian. These instructions worked perfectly for me and I was able to install Puppet 0.23.2, which is fairly up-to-date.

If you aren’t using Debian, take a look at the Installation Guide - I can’t vouch for the instructions for other platforms, but the documentation, where it exists, is generally excellent.

With my experience in setting up Puppet, I recommend setting up a dedicated server or virtualized system and calling it puppet.example.com (substituting your domain for example.com, of course!). There are several reasons for this. First, Puppet clients default to connecting to the host puppet in the domain which matches their assigned FQDN. Second, Puppet uses SSL certificates to secure communication between the clients and the server, and the certificates puppetmasterd will generate will use a CN corresponding to the system’s FQDN. This means if you install puppetmasterd on server1.example.com with an appropriate CNAME in DNS, clients who attempt to connect will get a certificate mismatch error. There is a way around this, but I think it is easier to just set up a new machine if you have the luxury. Last, this system will have an inordinate amount of power over your network, so you want to make sure it is as secure as you can make it - this suggests a dedicated system. That being said, I went another route and installed Puppet on an existing admin machine.

So, choose a server which will act as your “Puppetmaster” and install puppet and puppetmaster (or whatever the packages are called on your platform). At this point, you won’t be able to start either Puppet daemon; puppetmasterd requires a minimal “manifest” before it can function (we’ll discuss the term “manifest” in a moment), and puppetd requires a working puppetmasterd to talk to!

“What, exactly, is a manifest?” I hear you say. I could refer you to the Glossary of Terms on the Puppet wiki, but I think a little more explanation is in order. A manifest is nothing more or less than a description, written in the Puppet language, of one or more possibly interconnected and inter-related systems. The Puppet language allows us to fully describe many of the “nouns” of systems administration in such a way as to specify the configuration required to create a functioning system.

I have a very specific reason for describing manifests from such a high-level perspective. See, as I mentioned above, Puppet has the potential to change the way systems administration is done; it requires a different perspective to really leverage that potential - you have to begin thinking in terms of high-level concepts instead of low-level details of command-line arguments and configuration file formats. Yes, these details will (probably) always be part of systems administration (especially while you are still becoming comfortable with Puppet), but the high-level approach gives you the power to step back and deal with systems at a policy level.

Before you can write a manifest, you need to consider a system. By system, I don’t mean a server, I mean a collection of concepts which encapsulate some functionality on your network. The first system I considered was the SSH authorized_keys file. The “old way” of doing systems administration usually involves a number of home-grown scripts run over SSH using public-key authentication (so we don’t have to type the root password 100 times!). Since I like to make small, incremental changes when I move to a new way of doing something, I thought “Let’s see if puppet can make it easier to do my normal style of administration.” I wrote this simple manifest to do that:

1
2
3
4
5
6
7
8

node default {
  file { ‘/root/.ssh/authorized_keys’:
    owner => root,
    group => root,
    mode => 644,
    source => ‘puppet:///root/.ssh/authorized_keys’
  }
}

I put this into /etc/puppet/manifests/site.pp (This file is the starting point for puppet, and we’ll be changing it a lot in later posts). Let’s work through this file line by line:

1

node default {

tells Puppet that this manifest applies to all nodes (the special node name “default” is applied to all client nodes which contact puppetmasterd and do not have a specific node defined for their fqdn).

2

  file { ‘/root/.ssh/authorized_keys’:

tells Puppet that we are defining a “resource” of the native type “file“. This resource is named /root/.ssh/authorized_keys. The built-in types all do logical things with the name of the resource; in this case, the path parameter of the file will be taken from the name. The colon at the end of the line is a bit of syntax.

The next several lines all define “parameters” of the file type. These parameters are the adjectives of Puppet, they help describe the object which Puppet handles. The first three parameters are fairly self-explanatory:

3
4
5

  owner => root,
  group => root,
  mode => 644,

Indicates that this file (/root/.ssh/authorized_keys) should be owned by root, group ownership should be root, and the permissions should be 644. I took these settings from the existing authorized_keys file on hosts I already manage with SSH and public-key authentication.

The last line of this manifest is where things get a little more interesting.

6

  source => ‘puppet:///root/.ssh/authorized_keys’

The source parameter tells Puppet where to find this file. This can be a fully-qualified filesystem path or a URI. The documentation is unclear about which URI types are supported - it says only puppet:// URIs are supported, but mailing list traffic suggests http:// is supported as well. In any case, the puppet:// URI type tells puppetd to request the file from the puppetmasterd server. The file is copied into the specified location, then ownership and permissions are set.

Suddenly, every Puppet client you configure will have the same authorized_keys file for root! This makes those homegrown admin scripts a bit more reliable, because you can count on your SSH key being authorized on every host. (Once you set up Puppet, that is…)

Let’s not get ahead of ourselves. How does puppetmasterd know which file to give out? On Debian, the Puppet file server is configure in /etc/puppet/fileserver.conf. Here’s one that will work for the manifest above:

1
2
3

[root]
  path /etc/puppet/files/root
  allow 10.0.0.0/8

You will need to substitute your own network address in the allow statement, of course.

This configuration exports the path /etc/puppet/files/root at the URI puppet:///root. Simply make a subdirectory .ssh and place the master authorized_keys file you want distributed into that directory, and you’re all set to configure your first Puppet client.

When you choose a system to be your first Puppet client, don’t choose the same system you are using as a “Puppet Master”. There are a couple of gotchas that confuse the issue of managing the local machine. They aren’t complex, but they’re irritating enough to skip over while you first experiment. If you are using Xen, just provision another VM and install Puppet on it!

If you followed my advice and set up a dedicated system named puppet.example.com, once you install Puppet, you can skip straight to the certificate signing. If you didn’t, you’ll need one extra step. On Debian, the file /etc/puppet/puppet.conf controls certain default options for the various Puppet binaries. Add this to your puppet.conf file:

1
2

[puppetd]
  server=server1.example.com

Substitute the name of your Puppet master server for server1.example.com

Next, on the Puppet client, issue the command puppetd –test (You’ll run this as root). You’ll get a fairly informative message which indicates that you haven’t got a certificate configured yet. This is okay. On the Puppet master, issue the command puppetca –list. This command lists the pending certificate requests for Puppet. You should see the FQDN of your puppet client there. I leave it as an excercise to the reader to do due diligence on this signing request, once you feel good about it, sign the certificate on the Puppet master by issuing the command puppetca –sign client_fqdn. Back on the client, issue puppetd –test again. You should see some messages describing in detail how puppet is changing your machine. Once this is done, take a look at /root/.ssh/authorized_keys. If you aren’t drooling over the potential of Puppet at this point, I’m shocked.

Puppet can do a lot more than I’ve discussed here. This is just a small sample, meant to get you thinking. I’ll be writing more about my experiences with Puppet, as well as discussing best practices, a maintainable configuration, and more about how Puppet can change your conceptions of systems administration, in future posts.

I thought I’d write an introductory post to tell you a little about myself. I’ve been working professionally in the world of systems administration for seven years. My focus is on Unix variants such as BSD and Linux, but I am a generalist as opposed to a specialist; I have administered Windows systems, Macs, and even a few oddballs like NeXT and BeOS. If you’d like to see more about my experience, take a look at my resume (http://plathrop.tertiusfamily.net).

I have a growing passion for automation and a people-centered view of systems administration. To me, a systems administrator’s primary goal should be to act as a mediator between people and the technological systems which exist to support them. That’s right, I feel our profession is about people, not about computers! Although this is a somewhat radical view in our “community”, I apparently share it with a few others such as Luke Kanies (developer of the extremely awesome configuration management tool Puppet).

I think that our profession is entering a period of transition which has much in common with the recent changes in the world of software development; much as “coders” became “software engineers”, “sysadmins” are becoming “systems engineers” - or at least we should be if we wish to remain relevant. That change will be accompanied by a growing body of tools, standards, and best practices - and it is time for professional administrators to start thinking hard about what we want these to look like. Part of the reason I started this blog is to put my thoughts out there and hopefully start this conversation.

A friend and former co-worker has been writing interesting and useful articles over at his tech blog. I’ve been thinking about doing something similar for quite some time; guess seeing him do it was adequate inspiration.

I’ll be writing about systems administration and my experiences as a systems administrator (I know, fairly obvious from the title).

Stay tuned…