Photo of David Winter

david winter

Setting up a Puppet Master and Agents on EC2

Now you’ve mastered how to use the EC2 command line tools (Ninja, right?), you’ll want to manage all of those newly created instances in the easiest way possible. Puppet is the answer. If you’re not familiar with Puppet, back in March I wrote an introduction to it which would be good to read before continuing. If you are familiar, read on.

Previously I’ve only used Puppet ‘agentless’, or simply, standalone. This is great for developing small sites that don’t require multiple instances, however with larger projects, you’ll need a master so that you can leave that to deal with the ‘agents’ that connect to it.

Master and Agents

The example we’ll use for this post is for a web hosting project. You want to create a new instance for each customers, with a web server and database. Those packages will be installed and configured centrally from one master server so that all the instances are identical in that sense.

When first connecting an agent to a master, they have to exchange SSL certificates so that they know who is talking to who. The default way to do this is to manually check and sign the requests of agents as they attempt to connect to the master. This is time consuming an inefficient.

Instead, we’ll tell the Puppet master to automatically sign all incoming agent requests. That may sound like a security risk because anyones Puppet agent instance could try and connect. But we’ll use EC2 security groups to protect our server. For each new web server instance we create, we’ll assign it a security group called puppets. Then on our Puppet master, we’ll only allow other instances to connect to Puppet who are in the puppets group. This means we can safely allow the master to auto-sign requests, knowing that only our web server instances are allowed to do so.

The Puppet master

We’ll need to configure the master so that it’s ready to auto-sign puppet agents.

IP address and DNS

We need a dedicated Elastic IP for our Puppet master:

ec2-allocate-address

The output of that command will print out an IP address. It’s not currently associated to an instance, but it’s there ready to associate.

Each Elastic IP address is also given a public facing DNS name. It’s in the format of:

ec2-<IP-ADDRESS>.<REGION>.compute.amazonaws.com

So if the IP we were allocated was 123.456.789.012 and we are using the eu-west-1 EC2 region, the DNS name would look like:

ec2-123-456-789-012.eu-west-1.compute.amazonaws.com

Note: The periods in the IP address have been replaced with dashes.

The clever thing with these DNS names is that when you use them to communicate between EC2 instances, the DNS name resolves to the private IP address of the instance associated to it. This means you’re keeping the traffic internal to EC2 and it’s not going out onto the internet.

Why’s this important? Because when you stop an instance, the IP is unassociated from it, and next time you start the instance, the private IP address is not guaranteed to be the same. Using the DNS name, you know it’ll always use the correct private IP address by the DNS name being resolved.

Note: To make your systems more flexible in future, and not so tied to a DNS name with an IP address in it, it’s worth setting up a DNS CNAME record on your domain that forwards onto the Elastic IP DNS name, such as puppetmaster.mydomain.com. That way, in your systems, you’re using your domain name, and you only then need to change the DNS CNAME record in future if you change the Elastic IP.

Security groups and key pair

Create the security group that our Puppet agents will use.

ec2-create-group puppets -d "Puppet agents"

Don’t worry about assigning ports to this group yet, at this stage, we just need to ensure it exists.

Now create the security group for the Puppet master:

ec2-create-group puppetmaster -d "Puppet master"

For the puppetmaster group we want to allow TCP port 8140 to only the puppets security group:

ec2-authorize -P tcp -p 8140 -o puppets puppetmaster

We’ll throw in SSH too as it’s always handy to login just to see how things are:

ec2-authorize -P tcp -p 22 puppetmaster

Generate a new key pair for the master, and save it to a file, and set permissions:

ec2-create-keypair puppetmaster | sed 1d > puppetmaster.key
chmod 600 puppetmaster.key

Master instance and setup

Almost ready to create the instance, but when we initially do, it’ll be a base install of Ubuntu with packages that we need not installed, such as Puppet master.

A great feature of EC2 is that of user data scripts. These are scripts that are run when the instance is first created. We need to create a script that installs and configures Puppet master.

Create the following file and call it puppetmaster.sh:

#!/bin/bash
set -e -x
EC2_HOSTNAME=ec2-123-456-789-012.eu-west-1.compute.amazonaws.com
export DEBIAN_FRONTEND=noninteractive
hostname $EC2_HOSTNAME
echo $EC2_HOSTNAME > /etc/hostname
aptitude -y install puppetmaster
echo "*" > /etc/puppet/autosign.conf
service puppetmaster restart

With the above, the only line you need to change is the value of the EC2_HOSTNAME variable, which should be either the DNS name of the Elastic IP or the CNAME record you created at your domain.

The script will set the hostname of the instance, and then install Puppet master. The order of this is important, because when Puppet master is installed, it uses the hostname of the instance to create an SSL certificate for the master to use when auto signing agent certificates. If you have an incorrect hostname, the Puppet agents will complain and fail when trying to communicate with the master.

We then also tell Puppet master to autosign all requests, and then restart the service for the changes to take affect.

Now create the EC2 instance with the following:

ec2-run-instances --user-data-file puppetmaster.sh -t t1.micro -g puppetmaster -k puppetmaster ami-3b65664f

The instance will not be created, and started, and the user data script will run. In the output of this command, you’ll get details of the instance that has been created, along with an instance ID that starts with a prefix of i-. Use that ID to associate the Elastic IP to the instance:

ec2-associate-address -i i-INSTANCEID 123.456.789.012

Where 123.456.789.012 is the Elastic IP address you was allocated earlier.

Your master server is now setup and ready.

Setup your manifests

Although puppet master is ready to go, we don’t currently have any manifests setup. So for this example, we’ll just do a basic web server setup.

SSH into the master:

ssh -i puppetmaster.key ubuntu@ec2-123-456-789-012.eu-west-1.compute.amazonaws.com

Puppet master, by default, stores it’s manifests and modules at /etc/puppet. We’ll create a simple manifest that installs nginx and adds an index.html file to the default virtual host.

sudo nano /etc/puppet/manifests/site.pp

In the manifest file, add:

package { 'nginx':
	ensure => present,
}

service { 'nginx':
	ensure  => running,
	require => Package['nginx'],
}

file { '/usr/share/nginx/www/index.html':
	content => '<h1>Hello from Puppet master!</h1>',
	require => Package['nginx'],
}

Now when we boot up new Puppet agent instances, Puppet master will install nginx on them, and update the default index.html file with our changes.

The Puppet agents

You’ve already created a security group for these instances, so we’ll just authorise SSH and HTTP traffic for it.

ec2-authorize -P tcp -p 22 puppets
ec2-authorize -P tcp -p 80 puppets

Create a keypair for the puppet agents to use:

ec2-create-keypair puppets | sed 1d > puppets.key
chmod 600 puppets.key

We need to create another user data script for the agents in order for them to install Puppet when they are first created. Create a file called puppets.sh and put in it:

#!/bin/bash
set -e -x
EC2_HOSTNAME=ec2-123-456-789-012.eu-west-1.compute.amazonaws.com
export DEBIAN_FRONTEND=noninteractive
aptitude -y install puppet
echo "
[agent]
server=$EC2_HOSTNAME
" >> /etc/puppet/puppet.conf
sed -i /etc/default/puppet -e 's/START=no/START=yes/'
service puppet restart

Again, you just need to update the EC2_HOSTNAME variable to point to the DNS name (or custom CNAME) pointing to the Elastic IP for the master server. The rest of the script installs Puppet, and points the Puppet agent to the master server. It also sets the Puppet agent to start when the instance is started or rebooted.

Now you can create an instance:

ec2-run-instances --user-data-file puppets.sh -t t1.micro -g puppets -k puppets ami-3b65664f

Give the instance enough time to be created and booted. The following is then happening:

  1. Puppet is being installed
  2. It’s being configured to use our puppet master
  3. The puppet agent requests to the puppet master, which accepts
  4. Puppet master sends over the manifests (and modules if you specify any) and starts executing them

In our case, that means it’ll install nginx and our test index.html file.

So that we can check it has worked, run ec2-describe-instances and in the output, find the instance that has the security group puppets. That’ll be the one we just created. We want the public DNS name. Copy that, and paste it into a browser and you should see after a few minutes once it’s had time to setup:

Hello from Puppet master!

Our Puppet agent has connected and hooked up to the master successfully. Now you can continue to create other agent instances knowing that they’ll connect up the same. Those instances can then all be managed by the Puppet master.

Success.