Thoughts on System Administration

As I write these entries about Puppet I want to make it clear that the configurations I post may not necessarily be “best practices”. I am attempting to follow a logical evolution of steps involved in becoming familiar with Puppet. As the configuration grows more complex I will introduce the various best practices which make complex Puppet configurations manageable. If you are looking to dive straight in to a fully-realized puppet configuration, take a look at the Complete Configuration page of the Puppet wiki.

In my last post, I described how to distribute an SSH authorized_keys file using Puppet. My reasoning for this was to help us utilize our existing set of home-grown scripts to administer machines - with Puppet we don’t have to wonder if our keys are set up on every host. Well, I’m sure a few of you spotted the flaw in this. Most of us have learned not to run scripts as root when it isn’t necessary. Instead we use sudo to grant limited root powers for specific commands. Sudo is a well-designed piece of software; it’s configuration file, the sudoers file, is setup in such a way that the same sudoers file may be used on many machines. This makes it ripe for management in Puppet.

Building on top of the configuration we created in my last post, here is the site.pp manifest after adding sudo to the mix:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

node default {
  file { ‘/root/.ssh/authorized_keys’:
    owner => root,
    group => root,
    mode => 644,
    source => ‘puppet:///root/.ssh/authorized_keys’
  }
 
  file { ‘/etc/sudoers’:
    owner => root,
    group => root,
    mode => 440,
    source => ‘puppet:///etc/sudoers’
    require => [ Package[“sudo”] ]
  }
 
  package { sudo: ensure => installed }
}

The first definition is mostly familiar; we defined a file resource like it last time. There is something new here, however:

14

    require => [ Package[“sudo”] ]

The require parameter describes a dependency. Before this resource is applied, Puppet will look for a package resource with the name “sudo” and apply that resource first. Next we define that package resource:

17

  package { sudo: ensure => installed }

All we want is to make sure that sudo is installed. Perhaps the coolest part of this is that it doesn’t matter that I am using Debian here, Puppet supports a wide range of package systems under the hood, and it will choose the one most appropriate for the system it is configuring. If I add a FreeBSD machine to my network, I should not have to make any changes to my Puppet configuration - I can still depend on sudo getting installed!

Next, we need to add an etc mount to our Puppet fileserver. Here is the resulting fileserver.conf:

1
2
3
4
5
6
7

[root]
  path /etc/puppet/files/root
  allow 10.0.0.0/8
 
[etc]
  path /etc/puppet/files/etc
  allow 10.0.0.0/8

Put your sudoers file in /etc/puppet/files/etc and wait for your clients to check in with puppetmasterd. Alternately, you can log in to a client and run puppetd –test to pull down the new configuration. If sudo was not installed, it should be, and /etc/sudoers will be downloaded as well.

In my last post I mentioned Puppet. If you are a systems administrator and you haven’t taken a look at Puppet, stop reading and go look.

No, really. Open a new tab and head over to the About Puppet page of the Puppet wiki. Because Puppet is one of those tools which will utterly change the way you work.

Puppet can be described in many ways, but the most accurate (and powerful) way to think about it is as a language for describing systems. In fact, Puppet may end up redefining the term “systems programming” completely. Puppet provides the toolkit you need to build the Holy Grail of systems administration: the all-powerful, all-important, but seldom-realized Configuration Management Database!

Now that you are rolling your eyes and muttering under your breath, thinking you’ve heard this all before, I’d like to show you what Puppet can actually do. I think you will be pleasantly surprised…

In general usage, Puppet is a client/server application. Puppet clients running puppetd periodically contact a Puppet server running puppetmasterd. Puppet can also be used in stand-alone mode with the puppet command-line utility, but we’ll have to come back to that in a later post. For now, we need to get Puppet installed before we can move on to the good stuff.

I’m using Puppet to manage a number of Debian Etch servers. Some of these servers are virtualized Xen instances (domUs), the rest are the Xen hosts (dom0s). Lucky for me, Puppet is available as a Debian package. Unfortunately, like many Debian packages, the package is lagging behind the rather rapid release cycle of Puppet. However, the Puppet team has provided excellent instructions on installing Puppet from the “Unstable” release of Debian. These instructions worked perfectly for me and I was able to install Puppet 0.23.2, which is fairly up-to-date.

If you aren’t using Debian, take a look at the Installation Guide - I can’t vouch for the instructions for other platforms, but the documentation, where it exists, is generally excellent.

With my experience in setting up Puppet, I recommend setting up a dedicated server or virtualized system and calling it puppet.example.com (substituting your domain for example.com, of course!). There are several reasons for this. First, Puppet clients default to connecting to the host puppet in the domain which matches their assigned FQDN. Second, Puppet uses SSL certificates to secure communication between the clients and the server, and the certificates puppetmasterd will generate will use a CN corresponding to the system’s FQDN. This means if you install puppetmasterd on server1.example.com with an appropriate CNAME in DNS, clients who attempt to connect will get a certificate mismatch error. There is a way around this, but I think it is easier to just set up a new machine if you have the luxury. Last, this system will have an inordinate amount of power over your network, so you want to make sure it is as secure as you can make it - this suggests a dedicated system. That being said, I went another route and installed Puppet on an existing admin machine.

So, choose a server which will act as your “Puppetmaster” and install puppet and puppetmaster (or whatever the packages are called on your platform). At this point, you won’t be able to start either Puppet daemon; puppetmasterd requires a minimal “manifest” before it can function (we’ll discuss the term “manifest” in a moment), and puppetd requires a working puppetmasterd to talk to!

“What, exactly, is a manifest?” I hear you say. I could refer you to the Glossary of Terms on the Puppet wiki, but I think a little more explanation is in order. A manifest is nothing more or less than a description, written in the Puppet language, of one or more possibly interconnected and inter-related systems. The Puppet language allows us to fully describe many of the “nouns” of systems administration in such a way as to specify the configuration required to create a functioning system.

I have a very specific reason for describing manifests from such a high-level perspective. See, as I mentioned above, Puppet has the potential to change the way systems administration is done; it requires a different perspective to really leverage that potential - you have to begin thinking in terms of high-level concepts instead of low-level details of command-line arguments and configuration file formats. Yes, these details will (probably) always be part of systems administration (especially while you are still becoming comfortable with Puppet), but the high-level approach gives you the power to step back and deal with systems at a policy level.

Before you can write a manifest, you need to consider a system. By system, I don’t mean a server, I mean a collection of concepts which encapsulate some functionality on your network. The first system I considered was the SSH authorized_keys file. The “old way” of doing systems administration usually involves a number of home-grown scripts run over SSH using public-key authentication (so we don’t have to type the root password 100 times!). Since I like to make small, incremental changes when I move to a new way of doing something, I thought “Let’s see if puppet can make it easier to do my normal style of administration.” I wrote this simple manifest to do that:

1
2
3
4
5
6
7
8

node default {
  file { ‘/root/.ssh/authorized_keys’:
    owner => root,
    group => root,
    mode => 644,
    source => ‘puppet:///root/.ssh/authorized_keys’
  }
}

I put this into /etc/puppet/manifests/site.pp (This file is the starting point for puppet, and we’ll be changing it a lot in later posts). Let’s work through this file line by line:

1

node default {

tells Puppet that this manifest applies to all nodes (the special node name “default” is applied to all client nodes which contact puppetmasterd and do not have a specific node defined for their fqdn).

2

  file { ‘/root/.ssh/authorized_keys’:

tells Puppet that we are defining a “resource” of the native type “file“. This resource is named /root/.ssh/authorized_keys. The built-in types all do logical things with the name of the resource; in this case, the path parameter of the file will be taken from the name. The colon at the end of the line is a bit of syntax.

The next several lines all define “parameters” of the file type. These parameters are the adjectives of Puppet, they help describe the object which Puppet handles. The first three parameters are fairly self-explanatory:

3
4
5

  owner => root,
  group => root,
  mode => 644,

Indicates that this file (/root/.ssh/authorized_keys) should be owned by root, group ownership should be root, and the permissions should be 644. I took these settings from the existing authorized_keys file on hosts I already manage with SSH and public-key authentication.

The last line of this manifest is where things get a little more interesting.

6

  source => ‘puppet:///root/.ssh/authorized_keys’

The source parameter tells Puppet where to find this file. This can be a fully-qualified filesystem path or a URI. The documentation is unclear about which URI types are supported - it says only puppet:// URIs are supported, but mailing list traffic suggests http:// is supported as well. In any case, the puppet:// URI type tells puppetd to request the file from the puppetmasterd server. The file is copied into the specified location, then ownership and permissions are set.

Suddenly, every Puppet client you configure will have the same authorized_keys file for root! This makes those homegrown admin scripts a bit more reliable, because you can count on your SSH key being authorized on every host. (Once you set up Puppet, that is…)

Let’s not get ahead of ourselves. How does puppetmasterd know which file to give out? On Debian, the Puppet file server is configure in /etc/puppet/fileserver.conf. Here’s one that will work for the manifest above:

1
2
3

[root]
  path /etc/puppet/files/root
  allow 10.0.0.0/8

You will need to substitute your own network address in the allow statement, of course.

This configuration exports the path /etc/puppet/files/root at the URI puppet:///root. Simply make a subdirectory .ssh and place the master authorized_keys file you want distributed into that directory, and you’re all set to configure your first Puppet client.

When you choose a system to be your first Puppet client, don’t choose the same system you are using as a “Puppet Master”. There are a couple of gotchas that confuse the issue of managing the local machine. They aren’t complex, but they’re irritating enough to skip over while you first experiment. If you are using Xen, just provision another VM and install Puppet on it!

If you followed my advice and set up a dedicated system named puppet.example.com, once you install Puppet, you can skip straight to the certificate signing. If you didn’t, you’ll need one extra step. On Debian, the file /etc/puppet/puppet.conf controls certain default options for the various Puppet binaries. Add this to your puppet.conf file:

1
2

[puppetd]
  server=server1.example.com

Substitute the name of your Puppet master server for server1.example.com

Next, on the Puppet client, issue the command puppetd –test (You’ll run this as root). You’ll get a fairly informative message which indicates that you haven’t got a certificate configured yet. This is okay. On the Puppet master, issue the command puppetca –list. This command lists the pending certificate requests for Puppet. You should see the FQDN of your puppet client there. I leave it as an excercise to the reader to do due diligence on this signing request, once you feel good about it, sign the certificate on the Puppet master by issuing the command puppetca –sign client_fqdn. Back on the client, issue puppetd –test again. You should see some messages describing in detail how puppet is changing your machine. Once this is done, take a look at /root/.ssh/authorized_keys. If you aren’t drooling over the potential of Puppet at this point, I’m shocked.

Puppet can do a lot more than I’ve discussed here. This is just a small sample, meant to get you thinking. I’ll be writing more about my experiences with Puppet, as well as discussing best practices, a maintainable configuration, and more about how Puppet can change your conceptions of systems administration, in future posts.

I thought I’d write an introductory post to tell you a little about myself. I’ve been working professionally in the world of systems administration for seven years. My focus is on Unix variants such as BSD and Linux, but I am a generalist as opposed to a specialist; I have administered Windows systems, Macs, and even a few oddballs like NeXT and BeOS. If you’d like to see more about my experience, take a look at my resume (http://plathrop.tertiusfamily.net).

I have a growing passion for automation and a people-centered view of systems administration. To me, a systems administrator’s primary goal should be to act as a mediator between people and the technological systems which exist to support them. That’s right, I feel our profession is about people, not about computers! Although this is a somewhat radical view in our “community”, I apparently share it with a few others such as Luke Kanies (developer of the extremely awesome configuration management tool Puppet).

I think that our profession is entering a period of transition which has much in common with the recent changes in the world of software development; much as “coders” became “software engineers”, “sysadmins” are becoming “systems engineers” - or at least we should be if we wish to remain relevant. That change will be accompanied by a growing body of tools, standards, and best practices - and it is time for professional administrators to start thinking hard about what we want these to look like. Part of the reason I started this blog is to put my thoughts out there and hopefully start this conversation.

A friend and former co-worker has been writing interesting and useful articles over at his tech blog. I’ve been thinking about doing something similar for quite some time; guess seeing him do it was adequate inspiration.

I’ll be writing about systems administration and my experiences as a systems administrator (I know, fairly obvious from the title).

Stay tuned…