This post is the third of a few explaining how I set up my home lab setup to run a Kubernetes cluster on a Proxmox hypervisor using Infrastructure-as-code tools like Terraform and Puppet.

In the last parts we setup the base Proxmox setup and a cloud-init template (Part 1), then defined our infrastructure using Terraform with the Proxmox Provider (Part 2). In this post we’ll be actually setting up and using Puppet Bolt to run the previous Terraform logic as well as setup the base Puppet agent on these nodes required in a future part.

Puppet Bolt

Puppet Bolt is an agent-less task orchestration tool (similar to Ansible) that we’ll be using to actually drive our provisioning process. We’ll be creating a module with a plan that runs our previous Terraform logic, using the outputted server names and IP’s to setup the base Bolt/Puppet dependencies.

Note: Puppet Bolt is an open source utility. If you are already familiar with Puppet Enterprise, the paid version of Puppet, then using Puppet orchestrator should be preferred - although it follows a very similar process to what will be described with Bolt.

Install Bolt

Installing is straight forward, but will depend on your OS. Refer to the latest Bolt instructions on how to install it on your system. If you are following on from Part 1, this should be Debian 10 (Buster) steps which is shipped with the current Proxmox release as below.

wget https://apt.puppet.com/puppet-tools-release-buster.deb
sudo dpkg -i puppet-tools-release-buster.deb
sudo apt-get update 
sudo apt-get install puppet-bolt

We can test this is installed and working by running bolt task show.

Create our Puppet Bolt module

Install Puppet Development Kit

The easiest way to create a Puppet module is to use the PDK utility. Installing this differs depending on OS again (refer here) - but for reference on Debian 10 this is as follows (at time of writing the documentation doesn’t include this OS, but they do ship the release file needed when searching apt.puppet.com).

wget https://apt.puppet.com/puppet-tools-release-buster.deb
sudo dpkg -i puppet-tools-release-trusty.deb
sudo apt-get update 
sudo apt-get install pdk

Initialize our module

Now that we have the development toolkit, we can actually create the module. This should be created within one of the module paths directories, by convention this should be /root/Boltdir/site-modules (but may differ depending on OS and you can modify the path if required in./bolt.yaml)

cd /root/Boltdir/site-modules && pdk new module

# Follow prompts to set suitable defaults, I chose name terraform_provision
#    and de-selected Windows support, leaving RHEL and Debian selected

PDK will then setup a certain file structure consistent with the Puppet style guide, making it easier for others to modify your code if you were to share it on Puppet Forge. It should look similar to the following, but may differ slightly depending on PDK version.

terraform_provision/
├── data/                                 # Hiera data directory.
│   └── common.yaml                       # Common data goes here.
├── files/                                # Any files we'll be using like .confs will go here
├── plans/
│   └── init.pp                           # The terraform_provision plan we'll be modifying
├── specs/
│   ├── default_facts.yml
│   └── spec_helper.rb  
├── inventory.yaml
├── Puppetfile                            # A list of external Puppet modules to deploy.
├── README.md
└── hiera.yaml                            # Hiera's configuration file. The Hiera hierarchy is defined here.

The module contents and folder structure will likely not make much sense to you, but you should be able to blindly follow along to setup similar, although it would be worth having a read through Puppet’s official documentation on modules here to get a better understanding if you were intending to expand this logic.

This is a good time to perform the initial commit and push to your Git repo provider. At this stage I won’t be sharing my module, but will be providing all key components in the following sections.

Expand our module

Next we will add a Puppet Bolt Plan that uses our previously defined Terraform logic. Puppet Plans are a grouping of tasks or related business logic (read more here).

./inventory.yaml

Firstly, we’ll be defining a Puppet Bolt inventory file in the top level of our newly created module. In Bolt, inventory files are a way to store information about targets and any groupings or specific transport mechanisms like SSH/WinRM (read more here)

We’ll fill this out as below, adding the private key path of the public keys added to the Terraform input file in the last part. In our case we’ll leave targets empty since we’ll actually populate this from the server outputs from Terraform.

---
groups:
- name: terraform
  config:
    transport: ssh
    ssh:
      private-key: "~/.ssh/id_rsa-bolt"
      user: centos
      host-key-check: false
  targets: []

plans/init.pp

At this stage, we’ll only be creating just one plan. We can do this by creating a .pp file in the plans/ directory (creating it if it doesn’t exist). In Puppet, the init.pp file is special and allows you to reference a class, task or plan by the name only (e.g. we can later call bolt plan run terraform_provision). If we were to add other plans they’d be referenced differently like bolt plan run terraform_provision::my_plan.

At this stage we will just use init.pp to define our plan. Since we want this logic to be re-usable for different Terraform modules we’ll be defining an input parameter tf_path that relates to the Terraform module path we setup in the last part.

The current version of Bolt I’m using (v2.5) actually ships with built-in Terraform tasks, so we will then set Bolt to run on localhost and ensure it will return the outputs we defined in the Terraform outputs.tf file from last part with the parameter 'return_output' => true storing this to a variable apply_result .

If you also remember from last part, we also output a list of key/values in format; ‘server_name’: ‘ipconfig’. So we can then iterate over these items, splitting out the key fields to create a list of Puppet targets. With these we then use the built-in function apply_prep() on these targets which will setup any required packages needed for Bolt, gather any system facts with Facter for use by us in this module later and lastly will setup a Puppet agent with the built-in task puppet_agent::install if not already installed.

Combining all the above explanations, we end up with the following logic.

plan terraform_provision(String $tf_path) {
  $localhost = get_targets('localhost')

  run_task('terraform::initialize', $localhost, 'dir' => $tf_path)
  $apply_result = run_plan('terraform::apply', 'dir' => $tf_path, 'return_output' => true)

  $targets = $apply_result["servers"]["value"].map |$name, $ipconfig| {
    $plugin_hooks = {'puppet_library' => {'plugin' => 'puppet_agent', '_run_as' => 'root'} }
    # Terraform gives us ipconfig which is cloudinit format like 'ip=192.168.0.111/24,gw=192.168.0.1'
    #   so we need to split it, for some reason there is no regex, only regsubst??
    $ip = split(split($ipconfig, '/')[0],'=')[1]
    Target.new('name' => $name, 'uri' => $ip, 'plugin_hooks' => $plugin_hooks).add_to_group('terraform')
  }

  # Setup Puppet agent
  apply_prep($targets)
}

At this stage we’ve completed our Terraform/Bolt specific logic which makes a nice time to wrap up this part, to later expand on configuration management with Puppet in next part.

Testing it works (optional)

As per the last part though, we can optionally test our logic actually works. Just be mindful that if you ran the Terraform commands manually earlier, then the Bolt plan may not provision any servers as Terraform should detect there’s nothing to change depending on the statefile (if there is no statefile it will likely destroy and re-create the nodes - which can be a bad thing for production servers!). You could force Terraform to delete these nodes in the by running terraform destroy in the Terraform directory.

export TERRAFORM_DIR="<your Terraform path>"
bolt plan run terraform_provision -i inventory.yaml tf_path=$TERRAFORM_DIR

Conclusion

In this part we defined the Puppet Bolt logic to make use of our Terraform logic from Part 2 as well as setup the required components for Puppet to configure the servers further. In the next part we’ll expand on this Bolt logic to setup our Puppet master with Bolt and point the other nodes to it to perform the Puppet configuration management required to manage our environment.

References

This was partially based on logic defined by Puppet employee Lucy Wyman in an official Puppet blog post (note: Lucy’s code, particularly the inventory version 1 YAML, is deprecated for the latest Bolt release)