Thoughts on System Administration

In Puppet parlance, a module is a collection of manifests (including classes and definitions), templates, and files which, taken together, describe a recipe for configuring something via Puppet. Puppet has a number of facilities which make modules incredibly useful and labor-saving.

Modules have a standard internal organization which is described in detail on the ModuleOrganisation wiki page. A trivial module can be as simple as a manifests directory containing only a single manifest: init.pp. However, as modules grow more complex, you’ll want to break up your manifests and add templates in a templates directory, files in a files directory, and a README file which explains how to make use of your module.

I’m in the process of polishing up several modules, including LDAP, Kerberos, memcached, and ntp modules. In some ways, I’m duplicating work; several implementations of these modules are available. However, Puppet modules are still evolving, and I wanted to try my hand at module writing. Also, the existing modules did not work quite right for us. Some of them fail to properly isolate site-specific information from the recipe, for example. Others had complex interdependencies that I didn’t like.

There are several techniques that I think will evolve as “best practices” for Puppet module design. These are:

1. Keep site-specific customization out of your modules.
2. Your module should implement a minimal working configuration “out of the box”.
3. Use variables to make it easier to patch your module for use on other platforms.
4. Break a module up into sensible classes and defined types to make it easy to customize via inheritance.
5. Use init.pp for module-wide variables and/or common functionality, but break all other classes/defines into separate files.

Let’s take a look at each of these in detail:

Keep site-specific customization out of your modules.

Modules are, at heart, meant to be shared with others. Try to write your modules so they are as site-neutral as they can be. Use variables that can be set in a higher-level scope like site.pp to control how the module works, instead of baking your site’s settings into the module. For example, in my LDAP module I do this:

class ldap::common {
  case $ldap_base_dn {
    "": { $ldap_base_dn = "dc=example,dc=com"
      warning("ldap_base_dn not set, using default $ldap_base_dn")
    }
  }
 
  case $ldap_admin_dn {
    "": { $ldap_admin_dn = "cn=admin,$ldap_base_dn"
      warning("ldap_admin_dn not set, using default $ldap_admin_dn")
    }
  }
 
  case $ldap_admin_password {
    "": { fail("ldap_admin_password not set!")
    }
  }

Then, in site.pp, I set the variables appropriately:

# Site Variables
$ldap_server           = "ash001.example.com"
$ldap_admin_password   = "testing"
$ldap_base_dn          = "dc=example,dc=com"

Your module should implement a minimal working configuration “out of the box”.

This principle is also tied in with the fact that modules were meant to be shared. When someone is looking for a module, they are going to want something that works with little-to-no configuration, because this will give them confidence that the module will work once it is fully configured. Just like full-blown pieces of software, if a module doesn’t have an easy initial setup, people will get confused as to whether or not it even works. My LDAP module provides a ldap::master class which implements a very basic configuration: the slapd package is installed, a working configuration file is set up (without SSL or any of the other goodies), and the service is started. If the user sets just one variable, $ldap_admin_password in their init.pp and includes ldap::master on a node, they will be able to verify that LDAP is up and running with an example configuration. Even better, if they set the other variables, this configuration will be customized to their site with little effort on their part. It might be better to add in all the bells and whistles (SSL at a minimum), but I’m not sure yet where I stand on that.

Use variables to make it easier to patch your module for use on other platforms.

Currently, in the init.pp of my LDAP module, I set variables like this:

  $ldappackage       = "slapd"
  $ldapservice       = "slapd"
  $ldapdir           = "/etc/ldap"
  $ldaputilpackage   = "ldap-utils"
  $ldapclientpackage = "libnss-ldap"

and use them like this:

  package {
    $ldappackage:     ensure => installed;
  }
 
  file { "$ldapdir/slapd.conf":
    content => template("ldap/slapd.conf.erb"),
    require => Package[$ldappackage],
    notify  => Service["$ldapservice"],
  }
 
  service { $ldapservice:
    require   => [ Package[$ldappackage], File["$ldapdir/slapd.conf"] ],
    ensure    => running,
    enable    => true,
  }

These variables are set up for Debian now, because that is the distribution I’ve standardized on. If later I (or someone I’ve shared the module with) wants to add support for another distribution, they can set up case statements, and not have to modify the rest of the module!

  case $operatingsystem {
    debian: {
      $ldappackage       = "slapd"
      $ldapservice       = "slapd"
      $ldapdir           = "/etc/ldap"
      $ldaputilpackage   = "ldap-utils"
      $ldapclientpackage = "libnss-ldap"
    }
    centos: {
      $ldappackage       = "openldap"
      $ldapservice       = "slapd"
      $ldapdir           = "/etc/openldap"
      $ldaputilpackage   = "openldap-utils"
      $ldapclientpackage = "libnss-ldap"
    }
  }

I have not tested this, as I don’t use CentOS, so don’t use these!

Break a module up into sensible classes and defined types to make it easy to customize via inheritance.

Since you’ve pulled all the site-specific customization out of your package, and you are providing a minimal working configuration, people are going to want to do other things with your module that are appropriate to their site. If you have everything lumped into one big “ldap” class, this is going to be difficult for them. I break thinks up into ldap::common for things common to all the other classes, ldap::client for things needed to query the LDAP servers, ldap::master for the primary LDAP server, and later I’ll provide ldap::slave for replication slaves.

Use init.pp for module-wide variables and/or common functionality, but break all other classes/defines into separate files.

This is for your sanity as well as the sanity of those who might want to modify your module. I started with everything in one big init.pp file and it rapidly went out of control. Puppet does some awesome automagical lookups that you can take advantage of. For example, my ldap::master class is defined in a file called master.pp; when someone tries to load ldap::master, Puppet automatically searches for a .pp file named “master” in the “ldap” module!

So far, this is all the wisdom I have to impart on the subject. I’ll be sure to post links here when these modules are ready for prime-time.

One of the most exciting features of Puppet is the way it enables the sharing of systems administration tools - “recipes” in the Puppet parlance. Puppet allows us to collect manifests, definitions, and even plugins into modules which can then be shared with others.

David Schmitt is the author of the Complete Configuration found on the Puppet wiki. This is an amazing example of what can be done with Puppet, but can be a bit overwhelming at first. Happily, it is reasonably easy to integrate David’s modules into your Puppet configuration one at a time. Today I’d like to show you how to use David’s apt module to manage the dpkg & apt databases, as well as the keyrings associated with them.

In order to get started with David’s modules, we’ll need to take a few steps. The first step is upgrading to the latest release of Puppet. In an earlier post I advocated using the Debian packages; on further reflection, it might be best to install from source (or roll your own package). Puppet is a bit of a moving target right now, and you’ll definitely want to keep up. Installing from source is easy enough. On Debian Etch, I used the following steps to install from source:

1
2
3
4
5
6
7
8
9
10

aptitude remove puppet puppetmaster
aptitude install libopenssl-ruby libshadow-ruby1.8 libxmlrpc-ruby
curl -kO https://reductivelabs.com/downloads/facter/facter-1.3.7.tgz
tar xzf facter-1.3.7.tgz
curl -O http://reductivelabs.com/downloads/puppet/puppet-0.24.1.tgz
tar xzf puppet-0.24.1.tgz
cd facter-1.3.7
ruby install.rb
cd ../puppet-0.24.1
ruby install.rb

In short: remove the old version, install the prerequisite ruby libraries, download the latest source of Facter and Puppet, and install them both. The init scripts didn’t work after this, so I rolled my own.

The new version of Puppet allows module authors to distribute plugins within their modules. However, to make use of this, we need to add a couple of lines to the [main] section of our puppet.conf file:

  pluginsync = true
  pluginsource = puppet://$server/plugins

You’ll want to create a “modules” subdirectory to store your modules in. According to the Puppet Best Practices page, this directory should be in the same place as your “manifests” directory. For me, that means /etc/puppet:

total 36K
drwxr-xr-x  6 root root 4.0K 2008-01-02 13:47 .
drwxr-xr-x 71 root root 4.0K 2008-01-02 13:44 ..
drwxr-xr-x  7 root root 4.0K 2008-01-02 13:47 files
-rw-r--r--  1 root root  664 2008-01-02 13:47 fileserver.conf
drwxr-xr-x  4 root root 4.0K 2008-01-02 13:57 manifests
drwxr-xr-x  4 root root 4.0K 2008-01-02 13:47 modules
-rw-r--r--  1 root root  250 2008-01-02 13:47 puppet.conf

Next, we’ll want to download the modules we wish to use. All of David’s modules depend on the “common” module, so we will want “common” and “apt”. David makes it easy to pull down each module independently. Here’s what I did:

1
2
3
4
5
6
7
8
9
10
11
12

cd /etc/puppet/modules
mkdir common
cd common
git init
git remote add -t common -m common -f origin git://git.black.co.at/manifests/
git pull
cd /etc/puppet/modules
mkdir apt
cd apt
git init
git remote add -t apt -m apt -f origin git://git.black.co.at/manifests/
git pull

You should now have local copies of both the “common” module and the “apt” module. If you read the documentation, you will notice that these modules require the installation of the “lsb-release” package. I’m not sure why David doesn’t have the module install the package, but he doesn’t. So, let’s have puppet install it. A new “package” resource in our default node should do the trick:

25
26
27

  package { sudo: ensure => installed }
  package { lsb-release: ensure => installed }
}

Next, we need to integrate the new modules into our puppet configuration. Now that we’re getting into more complex configurations, we’ll want to start breaking our puppet configuration into separate files. In the same directory as site.pp (/etc/puppet/manifests), create a file called modules.pp containing:

1
2

import “common”
import “apt”

As you add more modules, you’ll import them here.

We won’t be explicitly using the common module at this time, so all we need to do is add two lines to site.pp. At the top of the file, we’ll want:

1

import modules.pp

to pull in the file we just created. We also want to update the default node to make use of the apt module:

7
8
9
10

node default {
  include apt
 
  file { ‘/root/.ssh/authorized_keys’:

It turns out that David’s modules also depend on a filebucket resource named “server”. This is a site-wide resource, so site.pp is the ideal place to put it:

3
4
5
6
7

filebucket { server:
  server => “server1.example.com”
}
 
node default {

After all these modifications, our site.pp looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

import modules.pp
 
filebucket { server:
  server => “server1.example.com”
}
 
node default {
  include apt
 
  file { ‘/root/.ssh/authorized_keys’:
    owner => root,
    group => root,
    mode => 644,
    source => ‘puppet:///root/.ssh/authorized_keys’
  }
 
  file { ‘/etc/sudoers’:
    owner => root,
    group => root,
    mode => 440,
    source => ‘puppet:///etc/sudoers’
    require => [ Package[“sudo”] ]
  }
 
  package { sudo: ensure => installed }
  package { lsb-release: ensure => installed }
}

I had to do one more thing to get this to really work. Reading through David’s module, you would expect /var/lib/puppet/modules/apt to be created by Puppet. Unfortunately, this doesn’t seem to happen. I had to create this directory by hand. I’ll let you know if/when I figure out why.

For basic usage of the module, this is all you need! David’s module defaults to a very sane basic apt configuration. The only tweak I felt was necessary was a modification to modules/apt/templates/sources.list.erb to use a US Debian mirror. You can modify this file to your liking for your default sources.list. You can also use the $custom_sources_list variable in a node definition to provide a customized file for a specific node:

node “oddball.example.com” {
  $custom_sources_list = “#This box is located in Canada, so use a local mirror
deb http://ftp.ca.debian.org/debian etch main contrib non-free
deb http://security.debian.org/ etch/updates main contrib non-free” include “apt”
}

David has a number of other modules I’d like to try; as I get them working I’ll be sure to share my experiences here.

As I write these entries about Puppet I want to make it clear that the configurations I post may not necessarily be “best practices”. I am attempting to follow a logical evolution of steps involved in becoming familiar with Puppet. As the configuration grows more complex I will introduce the various best practices which make complex Puppet configurations manageable. If you are looking to dive straight in to a fully-realized puppet configuration, take a look at the Complete Configuration page of the Puppet wiki.

In my last post, I described how to distribute an SSH authorized_keys file using Puppet. My reasoning for this was to help us utilize our existing set of home-grown scripts to administer machines - with Puppet we don’t have to wonder if our keys are set up on every host. Well, I’m sure a few of you spotted the flaw in this. Most of us have learned not to run scripts as root when it isn’t necessary. Instead we use sudo to grant limited root powers for specific commands. Sudo is a well-designed piece of software; it’s configuration file, the sudoers file, is setup in such a way that the same sudoers file may be used on many machines. This makes it ripe for management in Puppet.

Building on top of the configuration we created in my last post, here is the site.pp manifest after adding sudo to the mix:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

node default {
  file { ‘/root/.ssh/authorized_keys’:
    owner => root,
    group => root,
    mode => 644,
    source => ‘puppet:///root/.ssh/authorized_keys’
  }
 
  file { ‘/etc/sudoers’:
    owner => root,
    group => root,
    mode => 440,
    source => ‘puppet:///etc/sudoers’
    require => [ Package[“sudo”] ]
  }
 
  package { sudo: ensure => installed }
}

The first definition is mostly familiar; we defined a file resource like it last time. There is something new here, however:

14

    require => [ Package[“sudo”] ]

The require parameter describes a dependency. Before this resource is applied, Puppet will look for a package resource with the name “sudo” and apply that resource first. Next we define that package resource:

17

  package { sudo: ensure => installed }

All we want is to make sure that sudo is installed. Perhaps the coolest part of this is that it doesn’t matter that I am using Debian here, Puppet supports a wide range of package systems under the hood, and it will choose the one most appropriate for the system it is configuring. If I add a FreeBSD machine to my network, I should not have to make any changes to my Puppet configuration - I can still depend on sudo getting installed!

Next, we need to add an etc mount to our Puppet fileserver. Here is the resulting fileserver.conf:

1
2
3
4
5
6
7

[root]
  path /etc/puppet/files/root
  allow 10.0.0.0/8
 
[etc]
  path /etc/puppet/files/etc
  allow 10.0.0.0/8

Put your sudoers file in /etc/puppet/files/etc and wait for your clients to check in with puppetmasterd. Alternately, you can log in to a client and run puppetd –test to pull down the new configuration. If sudo was not installed, it should be, and /etc/sudoers will be downloaded as well.

In my last post I mentioned Puppet. If you are a systems administrator and you haven’t taken a look at Puppet, stop reading and go look.

No, really. Open a new tab and head over to the About Puppet page of the Puppet wiki. Because Puppet is one of those tools which will utterly change the way you work.

Puppet can be described in many ways, but the most accurate (and powerful) way to think about it is as a language for describing systems. In fact, Puppet may end up redefining the term “systems programming” completely. Puppet provides the toolkit you need to build the Holy Grail of systems administration: the all-powerful, all-important, but seldom-realized Configuration Management Database!

Now that you are rolling your eyes and muttering under your breath, thinking you’ve heard this all before, I’d like to show you what Puppet can actually do. I think you will be pleasantly surprised…

In general usage, Puppet is a client/server application. Puppet clients running puppetd periodically contact a Puppet server running puppetmasterd. Puppet can also be used in stand-alone mode with the puppet command-line utility, but we’ll have to come back to that in a later post. For now, we need to get Puppet installed before we can move on to the good stuff.

I’m using Puppet to manage a number of Debian Etch servers. Some of these servers are virtualized Xen instances (domUs), the rest are the Xen hosts (dom0s). Lucky for me, Puppet is available as a Debian package. Unfortunately, like many Debian packages, the package is lagging behind the rather rapid release cycle of Puppet. However, the Puppet team has provided excellent instructions on installing Puppet from the “Unstable” release of Debian. These instructions worked perfectly for me and I was able to install Puppet 0.23.2, which is fairly up-to-date.

If you aren’t using Debian, take a look at the Installation Guide - I can’t vouch for the instructions for other platforms, but the documentation, where it exists, is generally excellent.

With my experience in setting up Puppet, I recommend setting up a dedicated server or virtualized system and calling it puppet.example.com (substituting your domain for example.com, of course!). There are several reasons for this. First, Puppet clients default to connecting to the host puppet in the domain which matches their assigned FQDN. Second, Puppet uses SSL certificates to secure communication between the clients and the server, and the certificates puppetmasterd will generate will use a CN corresponding to the system’s FQDN. This means if you install puppetmasterd on server1.example.com with an appropriate CNAME in DNS, clients who attempt to connect will get a certificate mismatch error. There is a way around this, but I think it is easier to just set up a new machine if you have the luxury. Last, this system will have an inordinate amount of power over your network, so you want to make sure it is as secure as you can make it - this suggests a dedicated system. That being said, I went another route and installed Puppet on an existing admin machine.

So, choose a server which will act as your “Puppetmaster” and install puppet and puppetmaster (or whatever the packages are called on your platform). At this point, you won’t be able to start either Puppet daemon; puppetmasterd requires a minimal “manifest” before it can function (we’ll discuss the term “manifest” in a moment), and puppetd requires a working puppetmasterd to talk to!

“What, exactly, is a manifest?” I hear you say. I could refer you to the Glossary of Terms on the Puppet wiki, but I think a little more explanation is in order. A manifest is nothing more or less than a description, written in the Puppet language, of one or more possibly interconnected and inter-related systems. The Puppet language allows us to fully describe many of the “nouns” of systems administration in such a way as to specify the configuration required to create a functioning system.

I have a very specific reason for describing manifests from such a high-level perspective. See, as I mentioned above, Puppet has the potential to change the way systems administration is done; it requires a different perspective to really leverage that potential - you have to begin thinking in terms of high-level concepts instead of low-level details of command-line arguments and configuration file formats. Yes, these details will (probably) always be part of systems administration (especially while you are still becoming comfortable with Puppet), but the high-level approach gives you the power to step back and deal with systems at a policy level.

Before you can write a manifest, you need to consider a system. By system, I don’t mean a server, I mean a collection of concepts which encapsulate some functionality on your network. The first system I considered was the SSH authorized_keys file. The “old way” of doing systems administration usually involves a number of home-grown scripts run over SSH using public-key authentication (so we don’t have to type the root password 100 times!). Since I like to make small, incremental changes when I move to a new way of doing something, I thought “Let’s see if puppet can make it easier to do my normal style of administration.” I wrote this simple manifest to do that:

1
2
3
4
5
6
7
8

node default {
  file { ‘/root/.ssh/authorized_keys’:
    owner => root,
    group => root,
    mode => 644,
    source => ‘puppet:///root/.ssh/authorized_keys’
  }
}

I put this into /etc/puppet/manifests/site.pp (This file is the starting point for puppet, and we’ll be changing it a lot in later posts). Let’s work through this file line by line:

1

node default {

tells Puppet that this manifest applies to all nodes (the special node name “default” is applied to all client nodes which contact puppetmasterd and do not have a specific node defined for their fqdn).

2

  file { ‘/root/.ssh/authorized_keys’:

tells Puppet that we are defining a “resource” of the native type “file“. This resource is named /root/.ssh/authorized_keys. The built-in types all do logical things with the name of the resource; in this case, the path parameter of the file will be taken from the name. The colon at the end of the line is a bit of syntax.

The next several lines all define “parameters” of the file type. These parameters are the adjectives of Puppet, they help describe the object which Puppet handles. The first three parameters are fairly self-explanatory:

3
4
5

  owner => root,
  group => root,
  mode => 644,

Indicates that this file (/root/.ssh/authorized_keys) should be owned by root, group ownership should be root, and the permissions should be 644. I took these settings from the existing authorized_keys file on hosts I already manage with SSH and public-key authentication.

The last line of this manifest is where things get a little more interesting.

6

  source => ‘puppet:///root/.ssh/authorized_keys’

The source parameter tells Puppet where to find this file. This can be a fully-qualified filesystem path or a URI. The documentation is unclear about which URI types are supported - it says only puppet:// URIs are supported, but mailing list traffic suggests http:// is supported as well. In any case, the puppet:// URI type tells puppetd to request the file from the puppetmasterd server. The file is copied into the specified location, then ownership and permissions are set.

Suddenly, every Puppet client you configure will have the same authorized_keys file for root! This makes those homegrown admin scripts a bit more reliable, because you can count on your SSH key being authorized on every host. (Once you set up Puppet, that is…)

Let’s not get ahead of ourselves. How does puppetmasterd know which file to give out? On Debian, the Puppet file server is configure in /etc/puppet/fileserver.conf. Here’s one that will work for the manifest above:

1
2
3

[root]
  path /etc/puppet/files/root
  allow 10.0.0.0/8

You will need to substitute your own network address in the allow statement, of course.

This configuration exports the path /etc/puppet/files/root at the URI puppet:///root. Simply make a subdirectory .ssh and place the master authorized_keys file you want distributed into that directory, and you’re all set to configure your first Puppet client.

When you choose a system to be your first Puppet client, don’t choose the same system you are using as a “Puppet Master”. There are a couple of gotchas that confuse the issue of managing the local machine. They aren’t complex, but they’re irritating enough to skip over while you first experiment. If you are using Xen, just provision another VM and install Puppet on it!

If you followed my advice and set up a dedicated system named puppet.example.com, once you install Puppet, you can skip straight to the certificate signing. If you didn’t, you’ll need one extra step. On Debian, the file /etc/puppet/puppet.conf controls certain default options for the various Puppet binaries. Add this to your puppet.conf file:

1
2

[puppetd]
  server=server1.example.com

Substitute the name of your Puppet master server for server1.example.com

Next, on the Puppet client, issue the command puppetd –test (You’ll run this as root). You’ll get a fairly informative message which indicates that you haven’t got a certificate configured yet. This is okay. On the Puppet master, issue the command puppetca –list. This command lists the pending certificate requests for Puppet. You should see the FQDN of your puppet client there. I leave it as an excercise to the reader to do due diligence on this signing request, once you feel good about it, sign the certificate on the Puppet master by issuing the command puppetca –sign client_fqdn. Back on the client, issue puppetd –test again. You should see some messages describing in detail how puppet is changing your machine. Once this is done, take a look at /root/.ssh/authorized_keys. If you aren’t drooling over the potential of Puppet at this point, I’m shocked.

Puppet can do a lot more than I’ve discussed here. This is just a small sample, meant to get you thinking. I’ll be writing more about my experiences with Puppet, as well as discussing best practices, a maintainable configuration, and more about how Puppet can change your conceptions of systems administration, in future posts.