Logical thoughts exposed

A journey through my life as a infrastructure developer.

Put Some Data in Your Module

Puppet 4.9 cemented the final touches to putting data in puppet modules with the release of hiera 5. Which by the way hiera is now part of Puppet project instead of a separate gem as with previous versions.

So now that that whole params.pp mess is over we can finally move forward and create modules with rich sets of data without resorting to the params.pp hack.

So lets get started. In order to put data in your module you need to have a few things.

Requirements

Ideally, you should be running puppet 4.9 to take advantage of hiera 5 and data in modules. But if you happen to be running puppet 4.7-4.8 you can still utilize hiera 4 which contains similar features.

Just note that when upgrading to puppet 4.9+ hiera 4 implementations will need to eventually be onverted to hiera 5. This article will only focus on hiera 5 but the methods below are extremely similar for hiera 4.

  • Puppet 4.9+

Data in module the manual way

  1. Create a hiera.yaml file inside your module with at a minimum the following contents. Also figure out which backend type to use.
  2. Create a data directory mkdir data. This should be in your module’s root directory.
  3. Create a common.yaml file
  4. Optionally, create a data/os directory
  5. Add hiera data to common.yaml
1
2
3
4
5
6
7
8
9
10
  ---
  version: 5
  defaults:
    datadir: data
    data_hash: yaml_data
  hierarchy:
      - name: "OS family"
        path: "os/%{facts.os.family}.yaml"
      - name: "common"
        path: "common.yaml"

Note: only keys that start with your module’s class name will be referenced, unless you are doing hiera interpolation.

Automatic way

Retrospec automates all the steps above. So to get started install retrospec.

gem install puppet-retrospec facter and perform the following.

  1. cd your_module_path
  2. retrospec puppet module_data

Retrospec will clone down it’s templates and create all the necessary files to get module data up and running. Yup! A single command. Pretty easy right? Best of all you can do this over and over on all your modules!

If you doing advanced lookup backends retrospec also allows additional options should you want to create a custom function for lookup. See retrospec puppet module_data -h for more options.

retrospec puppet module_data -b data_hash -n my_custom_hash -t native

1
2
3
4
5
Options:
  -b, --backend-type=<s>     Which hiera backend type to use (hiera, data_hash, lookup_key, data_dig) (default: hiera)
  -n, --backend-name=<s>     The name of the custom backend (default: custom)
  -t, --function-type=<s>    What type of function to create the backend type with (native or v4) (default: native)
  -h, --help                 Show this message

Most of the time you will just want to use the defaults but some will be cooking up some fancy backends and need a way to easily generate the lookup backend.

Hierarchy Setup

Unfortunately, Retrospec does not put the data in your hiera files. It does dump all the class parameters into common.yaml but you still need to sort out the values and hierarchy configuration for each parameter key. This might take some time to figure out.

If your module currently uses params.pp I would start there.

Break Into Your Puppet Code

The puppet 4 language introduced a slew of new features that gives us enormous amounts of power and flexibility. So much so that puppet code can become complex and often hard to understand at a glance.

A way to combat this complexity is to write unit tests to validate the end state. However, one of the shortcomings of unit testing with rspec-puppet is that you can only test against the end state. Because puppet is a “compiled” language there is not a good way to perform testing on intermediary states like the value of variables.

With this in mind, we need a way to get inside the puppet compiler and see how our code works from a logical standpoint instead of just a static end state. The puppet-debugger is a tool that allows us to discover and explore the inner workings of puppet code in real time as if you are standing inside the compiler with a big poking stick.

Dr. Farnsworth poking a dead space creature

Below I will show you the steps needed to combine rspec-puppet and the puppet-debugger to step into your code during compilation time. So bring a stick because there will be many opportunities to poke things.

Requirements

In order to use the puppet-debugger please follow the steps below.

Install puppet-debugger gem

Ensure you have installed the puppet-debugger gem

gem install puppet-debugger

or

Add it to your Gemfile

gem 'puppet-debugger', '>= 0.6'

Add the nwops-debug puppet module

You will also want to include this module in your .fixtures.yml file if using rspec-puppet.

1
2
debug:
   repo: https://github.com/nwops/puppet-debug

Puppet 4 also requires us to specify our dependencies in metadata before using any ruby based functions. So we will need to add the debug module to the metadata’s dependency list.

The module’s metadata.json file should look something like this. For more info on metadata click here

1
2
3
4
"dependencies": [
    {"name":"puppetlabs-stdlib","version_requirement":">= 4.11.0"},
    {"name":"nwops-debug","version_requirement":">= 0.1.1"}
  ]

Install gems and fixtures

  1. bundle install
  2. bundle exec rake spec_prep

Usage

To break into our puppet code we need to first set a breakpoint. This is done by using the debug::break() function. You can put this breakpoint anywhere in your puppet code. Once inside the debugger REPL, start poking variable values and run other debugger commands.

1
2
3
4
5
6
7
8
9
class testmod (
  $var1 = 'value'
) {
  debug::break()
  file{"/tmp/${var1}":
    ensure => present
  }

}

Now in order to break into the puppet code you need to have puppet evaulate the code. This can be done in multiple ways. The easiest is to use rspec-puppet, but you could also use puppet debugger or even puppet apply commands.

In the example below we just need to create a simple unit test and then run bundle exec rake spec

1
2
3
4
5
6
it do
    is_expected.to contain_file("/tmp/value")
        .with({
          "ensure" => "present"
        })
end

Debug using rspec animated gif

As I mentioned above you can also use the puppet debugger directly and playback puppet code. ie. puppet debugger --play examples/debug.pp

Debug using rspec animated gif

When using puppet apply, just remember you either need to include the class like include testmod from within the debugger or puppet code. Just remember when using with puppet apply puppet will actually make changes so be very careful. This is why the puppet debugger command exist since it does not make changes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
root@f6e624d48fe2:~/testmod# puppet apply manifests/init.pp
Ruby Version: 2.2.7
Puppet Version: 4.10.0
Puppet Debugger Version: 0.6.1
Created by: NWOps <corey@nwops.io>
Type "exit", "functions", "vars", "krt", "whereami", "facts", "resources", "classes",
     "play", "classification", "types", "datatypes", "benchmark",
     "reset", or "help" for more information.

From file: init.pp
          1: class testmod (
          2:   $var1 = 'value'
          3:
          4:
          5: ) {
      =>  6:   debug::break()
          7:   file{"/tmp/${var1}":
          8:     ensure => present
          9:   }
         10:
         11: }
1:>> exit
Notice: Compiled catalog for f6e624d48fe2 in environment production in 700.28 seconds
Notice: /Stage[main]/Testmod/File[/tmp/value]/ensure: created
Notice: Applied catalog in 0.03 seconds

I hope these examples have given you an overview of all the different ways to invoke the puppet debugger and utilization of the debug::break() function. You should now be equipped to start poking fun at puppet.

Stay tuned next time while I show how to debug with different facter sets using facterdb and the puppet-debugger.

Benchmarking Your Puppet Code

Did you know the puppet debugger can measure how fast your puppet code runs? Starting with release 0.6.1 of the puppet-debugger you can now perform simple benchmarks against your puppet code. So if you ever wondered how fast your puppet code is or that custom puppet function you wrote now there is a way to benchmark those things.

All you need to do is enable benchmark mode, and then use the debugger as you normally do. Every time puppet evaluates your input a benchmark will be returned with your result of running the puppet code. This can be helpful when trying to determine which code might take a long time to evaluate. A simple use case is measuring your puppet functions.

1
2
3
4
5
6
7
8
9
10
11
12
13
1:>> benchmark
Benchmark Mode On
2:BM>> md5('dsfasda')
 => [
  [0] "1b0590733e5d6ad1f122d95c47558564",
  [1] "Time elapsed 50.57 ms"
]
3:BM>> md5('dsfasda')
 => [
  [0] "1b0590733e5d6ad1f122d95c47558564",
  [1] "Time elapsed 0.64 ms"
]
4:BM>>

As an example, running these functions back to back shows how the puppet application lazy loads puppet functions. The first run took 50 ms which included loading the function, while the second only half a millisecond since the puppet function was already loaded.

The benchmark feature is now available with puppet-debugger release 0.6.1.

gem install puppet-debugger or on the web at https://www.puppet-debugger.com

If you have a unique use case I would love to hear about it.

Benchmark Demo

Testing DataTypes With the Puppet Debugger

The puppet language comes with a lot of extremely useful syntax and concepts. However, sometimes it is difficult to understand how these work or how to use them. Datatypes are found in almost every programming language so it is no surprise that puppet 4 has a similar feature for validating parameter data. If you have never used a puppet datatype before have a look here first.

The other day I was showing my client the awesome power of puppet datatypes and I was using the puppet-debugger to illustrate how datatypes work. By using the puppet debugger the client was immediately able to understand datatypes without writing a manifest, so I wanted to share how to use datatypes using the debugger to the rest of the world because datatypes can be very complex and using the debugger will help you understand how they work.

Let us first detail how to use a datatype with puppet parameters.

Using a datatype

Below is a snippet of a puppet defined type that has no datatype validations. Obviously the result of running this code would fail.

1
2
3
4
5
6
7
8
define foo(
  $bar = 'barbar'
  ) {
    file{'/tmp/test':
      ensure => $bar,
    }
  }
  foo{'test': bar => [1,2,3]}

So to prevent an error, we should be checking the value of bar in the parameter. Previously you might have used validate_string($bar) which was a function from the stdlib module that performed validations. This validation would occur only when the catalog is applied to a system. But that is way too late, we need to fail faster. So if you want an error to popup well before you deploy the code you need to inform Puppet that you expect a certain type of data. Hence the word ‘datatype’.

Adding a datatype would cause the compilation to fail immediately at compile time which you can check by writing a unit test or running puppet apply.

1
2
3
4
5
6
7
8
define foo(
  Enum['directory', 'present', 'file'] $bar,
  ) {
    file{'/tmp/test':
      ensure => $bar,
    }
  }
  foo{'test': bar => 'present' }

However, if you are not ready to jump into unit testing, you can utilize the puppet-debugger to bridge the gap. Using the puppet-debugger is the easiest, fastest way to validate that the parameter datatype accurately reflects the type of data your manifest requires.

Installing the Puppet Debugger

To get started you need to install the puppet-debugger on any system with puppet >= 3.8

gem install puppet-debugger

Testing a DataType with the Debugger

In order to test the data against the datatype we are going to use the =~ operator. This operator lets us test out the datatype. The data being tested goes on the LHS (left hand side) while the datatype goes on the RHS (right hand side).

Examples:

  1. true =~ Boolean
  2. 'true' =~ String
  3. 'https://www.google.com' =~ Stdlib::HttpsUrl

Puppet Datatypes Demo

If the value on the LHS matches the requirements on the RHS, puppet will return true, other false is returned and the value does not match the datatype.

Testing a DataType with the Debugger using the new function

Another way to test your datatype is use the [new operator](https://docs.puppet.com/puppet/latest/function.html#new_. This only works with some datatypes like structures and core types. But if you have a complex datatype like a structure you can use the new operator to test out the custom datatype. In this example puppet shows an error because we did not specify an owner attribute on line 11.

More Info about Abstract Data Types

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
1:>> type MyType = Struct[{
  2:>>         mode => Enum[read, write, update],
  3:>>         path            => Optional[String[1]],
  4:>>         NotUndef[owner] => Optional[String[1]]}]
5:>> MyType.new({
  6:>> mode => 'read',
  7:>> path => '/dev/null',
  8:>> owner => 'root'
  9:>> }
  10:>> )
 => {
   "mode" => "read",
  "owner" => "root",
   "path" => "/dev/null"
}
11:>> MyType.new({mode => 'read'})
 => Evaluation Error: Error while evaluating a Method call, Converted value from MyType = Struct[{'mode' => Enum['read', 'update', 'write'], 'path' => Optional[String[1, default]], NotUndef['owner'] => Optional[String[1, default]]}].new() has wrong type, expects a value for key 'owner' at /var/folders/v0/nzsrqr_n40d4v396b2bqjvdw0000gp/T/puppet_debugger_input20170424-91861-6grwv7.pp:1:11
12:>>

As you can see above the puppet-debugger makes it dead simple to test datatypes. For repeatable datatype testing you will want to write unit tests instead which is not covered in this article. Additionally, the stdlib module also has some nice datatypes that you can use in your modules.

I hope this has been of value to you, please share with others and star the puppet-debugger project if you enjoy using it.

Testing Hiera Data

As a puppet consultant I often run into the same problems with multiple clients. Many times I can reuse a magical script and my value instantly becomes obvious. I like to think I have things figured out, but sometimes there are just problems we as a community have not solved yet.

The problem I am talking about is hiera validation. Most of us are too busy learning puppet, ruby, markdown, and git that testing is not a priority until your puppet code blows up in your face. But for those who know hiera well, and understand what bad data means, than read on.

1
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find data item lvm::volumes in any Hiera data file and no default supplied on node puppetmaster3278.nwops.io

What do we test

Traditionally, the answer to testing hiera has been to just check for correct YAML syntax. But the problem with this is that YAML is very accepting of various inputs. We need to validate hiera data not YAML.

What we care about

  • How the hell do we test the data going into hiera?
  • How do we test that the keys we type, match up with the puppet parameters in the modules?
  • How do we test that the values in the keys are in the same format that the puppet code is expecting?
  • How do you test that you added all the necessary keys?
  • How do you test your ability to create correct YAML syntax?
  • How do you do all this is under 1 second?

How do we test

Up until now most of use used our eyeballs to test hirea data. But my client won’t keep me around forever and I can’t just say look for these errors. Using YAML.load_file('common.yaml') does not do a damn thing! After all we are validating hiera data not YAML.

At a previous client I used rspec-puppet and the hiera-puppet-helper to not only mock hiera but to also test real hiera data with my unit tests. But anytime someone changed the data the tests broke.

Below is an example of using rspec-puppet with hiera.

1
2
3
4
5
6
7
8
9
10
require 'hiera'
hiera_file = File.expand_path(File.join(__FILE__,  '..', '..', '..', '..', 'hieradata', 'spec.yaml'))
shared_context :hiera do
  let(:hiera_config) do
    hiera_file
  end
  hiera = Hiera.new(:config => hiera_file)
end

it { is_expected.to contain_file('/tmp/test').with_content("hello")}

But none of the above solutions work reliably. And the reason behind our inability to test hiera data is we don’t know what we are testing. We don’t know what kind of data needs to go in those values. There is no definition or map that magically tells us what should and should not put in our hiera data. Or is there?

This is where a module schema becomes invaluable. The module schema details the exact definition of all the parameters for that module. So all we need to do is extract the schemas from every module being used into a giant master schema.

Creating a master schema might seem impossible because every single implementation of hiera data is unique. But I assure you its not impossible, just incredibly tedious. But we are devops dammit! Lets automate that shit!

Building a master schema

I have previously explained how to build a module schema. And I have added support for auto generating schemas with retrospec puppet tool. But we need to build something slightly different for validating hiera data. We need a master schema that contains all the schemas from all the modules we are using.

There are a few ways we can build up a master schema.

1. Use existing hiera data

The hiera data you have contains all the keys and values that are currently being used. Ruby makes it easy to turn YAML files into native ruby objects. So you can read your hiera data files and map all the keys and values into a schema pretty quickly. The only downside is your schema won’t be very specific until you take some time to define the schema with complex data types that might be lurking in your puppet code.

I have written such a script for my client, and it works pretty awesome. My CI job fails when someone inserts hiera data without an associated mapping for it. It will even suggest the mapping to use in the master schema. Basically your just working backwards to create a schema when given data.

One additional trick is to ensure that all the hiera data keys are set to required: true in your schema. Because the key already exists in your hiera data it will help enforce spelling mistakes in key names.

2. Use Puppetfile to dynamically generate schemas on the fly

For now this method is more of a pipe dream. But in a perfect world where every module contains a well defined schema. We could easily read the contents of the module’s schema for each module defined in the Puppetfile and merge together the contents into a single master schema that would be unique for each permutation of the Puppetfile.

That script might look something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#!/usr/bin/env ruby
@master_schema = {}

# returns schema as a string
def get_remote_schema(url)
  uri = URI.parse(url)
  http = Net::HTTP.new(uri.host, uri.port)
  request = Net::HTTP::Get.new(uri.request_uri)
  response = http.request(request)
  response.body
end

# implement the mod method that Puppetfile uses
def mod(name, opts={})
  schema = YAML.load(get_remote_schema(opts[:url])
  @master_schema.merge!(schema['hostclass']) # host class parameters only
end

eval(File.read('Puppetfile') # read and eval the puppetfile

# write the master schema out
File.open('master_schema.yaml', 'w') {|file| file.write(@master_schema.to_yaml)}

With this method we are relying solely on the developers schema to validate our hiera. Of course you could always maintain a better more static master schema. Additionally, you could make some pull requests to update the developer’s schema which in turn benefits everyone.

And this is where it becomes tedious. Since nobody has ever thought about creating schemas for their puppet code it might take some time to build up your master schema.

Your schema can be as little or big as you want. The better it is the more errors that will be caught. So lets move on and assume you have a well defined master schema.

Building a script to validate your data

Now that you have a master schema and lots of hiera data to validate, how do you validate all the keys across all the files? Basically you need to build a script that uses the kwalify parser to validate hiera files against your master schema. Below are some snippets from a much bigger script that does this validation.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# round up all the hiera files
@hiera_files = Dir.glob(File.join('data','**', "*yaml"))
@hiera_files.each do |file|
  validate_file(file)
end

# an instance of the kwalify parser
# since we don't need to create a new instance each time
# we cache the object here
# returns a kwalify validator parser instance
def parser
  unless @parser
    ## load schema data
    schema = Kwalify::Yaml.load_file(@schema_file)
    ## create validator
    validator = Kwalify::Validator.new(schema)
    @parser = Kwalify::Yaml::Parser.new(validator)
  end
  @parser
end

# use the kwalify validator to validate each file
# returns an array of errors if any.
def validate_file(file)
  logger.debug "Validating file: #{file}"
  ## load document and parse
  document = parser.parse_file(file)
  begin
    errors = parser.errors || []
  rescue Kwalify::SyntaxError => e
     return [e]
  end
end

There is actually a lot more to this script. One example is that hiera allows us to define a key in any file. But this validation doesn’t know that because it works with one file at a time. So if a schema requires a key and that hiera data file doesn’t contain the key, validation fails. So we have to treat all the files as one big file. We can either concat all the files together or load every file into a giant hash and use the hash to remember which keys have been validated already when they are required. Below is an example of building a giant hash.

1
2
3
4
5
# load em up!  
@referenced_keys = {}
@hiera_files.each do |file|
  @referenced_keys.merge!(YAML.load_file(file))
end

So validation fails because it cannot find a required key but we can use some logic to determine it a thats really a problem by using our giant hash.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
  def validate_file(file)
    logger.debug "Validating file: #{file}"
    ## load document and parse
    document = parser.parse_file(file)
    begin
      errors = parser.errors || []
    rescue Kwalify::SyntaxError => e
       return [e]
    end
    # if a given key is already defined somewhere else we don't
    # want to mark the key as required
    errors.find_all do |error|
      if error.error_symbol == :required_nokey
        # since this is a required no key error there is no path
        # so we must find the key we are missing
        key = error.to_s.match(/key\s'(.*):'/)[1]
        logger.debug "Looking up key #{key}"
        !@referenced_keys.has_key?(key) # if key is found this is not an error
      else
        # all other errors should be returned as normal errors
        true
      end
    end
  end

And really thats all there is to it. These are just snippets from the entire script. The other part of my script deals with creating maps given the data and also some pretty output. But hopefully it should give you an idea how to build your own validator using the kwalify parser. I would share the rest of my script but its not ready for prime time and I don’t have the capacity to maintain another gem for public consumption. You could also go the other route by concatenating all the files together and using kwalify -lf master_schema.yaml giant_hiera_data_file.yaml but that might have some drawbacks.

Summary

I have shown you how to create module schemas. And this article shows you how to create a master schema in order to validate your data. So once you build up the master schema and create a validation script, the payoff is huge! Everyone wins. So go forth and give this a try.

How to Build a Module Schema

As a long time puppet module developer and puppet consultant I have noticed some trends over the years. Puppet modules are becoming increasingly sophisticated and complex. When consuming forge modules I often spend time figuring out how it works and what the developer requires for parameters. Especially when it comes to inserting data into hiera.

Schemas help validate the user is doing what the developer intended.

Creating a schema might not seem necessary but anybody using your module will greatly appreciate it. So if your a module developer here is how you can create a schema.

Install Kwalify

Kwalfy is a project that allows you to easily write a schema and parse the yaml file against the schema you created.

To install: gem install kwalify

Create a Schema Yaml file

The schema details all the inputs for your module and there are a two ways to create one. Manual or Automatic. If doing manually head over to the kwalify docs to see what options you can specify. For automatic generation use the retrospec puppet tool to auto generate a schema. But retrospec doesn’t do enough. The schema definition does not accurately reflect parameter types because puppet is a loosely typed language, so even retrospec cannot determine with the resource exactly needs. So you will need to further define your schema if you have complex data. However, retrospec will get you started and you can update the schema as needed.

The module schema should be named modulename_schema.yaml and placed in the root of your module.

1
2
3
cd /Users/user1/github/puppetlabs-apache
retrospec puppet
+ /Users/user1/github/puppetlabs-apache/puppetlabs-apache_schema.yaml

Map Your parameters

Your schema file should detail exactly what your puppet class or definition is expecting. Be extremely detailed. Your users will love you.

Simple example

1
2
3
4
5
6
7
8
9
10
11
12
---
  type: map
  mapping:
    hostclass:
      type: map
      mapping:
        "tomcat::catalina_home":
          type: str
          required: false
        "tomcat::user":
          type: str
          required: false

Complex Example

Example of a more detailed schema, this is an example using the puppet lvm module. Not everyone will have a complex schema, but some of us will.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
---
type: map
mapping:
  hostclass:
    type: map
    mapping:
      "lvm::volume_groups":
        type: map
        required: false
        mapping:
          =:
            type: map
            required: true
            mapping:
              'physical_volumes':
                type: seq
                desc: 'A list of physical volumes to use for the volume group'
                sequence:
                  - type: str
                required: true
              'logical_volumes':
                type: map
                required: true
                mapping:
                  =:
                    type: map
                    desc: 'The logical volume name'
                    required: true
                    mapping:
                      'size':
                        type: str
                        pattern: /[1-9]\d?[kKmMgGtT]{1}/
                        required: true
                        desc: 'The size of the logical volume'
                      'mountpath':
                        type: str
                        required: true
                        desc: "Where to mount the logical volume"
                      'fs_type':
                        default: ext4
                        desc: "The filesystem type to use when formating. "
                        type: str
                        required: false
                        enum: ['swap', 'ext4', 'ext3']

Use a data type

Just like in puppet 4 kwalify allows the use of data types. I suspect str, any, bool, number, map, and seq will be the most used. Below are all the types we can use in our schema:

  • str
  • int
  • float
  • number (== int or float)
  • text (== str or number)
  • bool
  • date
  • time
  • timestamp
  • seq
  • map
  • scalar (all but seq and map)
  • any (means any data)

Use a constraint

You can require and item, and require a specific type, and you can also require the value to a specific set. This is very similar to puppet 4 data types.

  • regex patterns
  • enums
  • ranges
  • length

Document the default values and description

Now this might seem redundant because you already documented these in puppet.
But if the schema contains the description and default values, the user only has to view a single file instead of many files. Furthermore, with proper tooling we should be able to either pull this data out of puppet or use the schema to populate the puppet code with default values and comments that are defined in the schema.

1
2
3
4
5
6
'fs_type':
  default: ext4
  desc: "The filesystem type to use when formating. "
  type: str
  required: false
  enum: ['swap', 'ext4', 'ext3']

See Rules and Constraints for more info.

Make items required

You can require parameters if your puppet code doesn’t supply a default. If the user doesn’t supply the fs_type, validation will fail.

1
2
3
4
'fs_type':
  type: str
  required: true
  enum: ['swap', 'ext4', 'ext3']

Validate Your schema with Kwalify

To ensure your schema is a valid schema use the following command. This simply validates your schema against the kwalify tool.

kwalify -m your_schema.yaml

Create a sample data set for testing purposes

Below is an example of data set that would be used with the puppet lvm module. This would most likely be in hiera.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
hostclass:
  lvm::volume_groups:
    vg00:
      physical_volumes:
        - /dev/sda2
        - /dev/sdd
        - /dev/sdc
      logical_volumes:
        audit:
          size: 160M
          mountpath: /var/audit
        lv04:
          size: 4G
          mountpath: /tmp
        lv_swap:
          size: 0G
          fs_type: swapp
          mountpath: swap
    vg01:
      physical_volumes:
        - /dev/sdb
      logical_volumes:
        lv01:
          size: 32M
          mountpath: /system/app
        lv02:
          size: 16M
          mountpath: /system/oracle
        lv03:
          size: 40M
          mountpath: /var/opt/oracle
        backup:
          size: 0K
          mountpath: /backup

Validate your data against your schema

Now we will validate the data against our schema. Notice that my sample data set contains a few errors. Did you even notice? This is intentional, because this shit won’t work if you accidentally had these values in hiera. Because that would never happen …

1
2
3
4
5
kwalify -lf test_schema.yaml /tmp/test_schema_data.yaml
/tmp/test_schema_data.yaml#0: INVALID
- (line 16) [/hostclass/lvm::volume_groups/vg00/logical_volumes/lv_swap/size] '0G': not matched to pattern /[1-9]\d?[kKmMgGtT]{1}/.
- (line 17) [/hostclass/lvm::volume_groups/vg00/logical_volumes/lv_swap/fs_type] 'swapp': invalid fs_type value.
- (line 33) [/hostclass/lvm::volume_groups/vg01/logical_volumes/backup/size] '0K': not matched to pattern /[1-9]\d?[kKmMgGtT]{1}/.

Now the cool about this type of validation is that it is immediate and extremely specific. If every module developer had a well defined schema, there would be no errors in our hiera data.

Summary

There is still lots to learn but the process for creating a schema is the same across every module. So hopefully this article is enough to get you started. Having a schema for your module allows the end user to easily assemble a master schema in order to validate all of their hiera data. It could also be used to create a README file that details all the parameters as well. I suspect that if schema files take off we can see some further tooling around generation and possibly more use cases for schemas.

So stay tuned as my next article talks about just how to validate hiera using schemas.

Leveraging Docker for Puppet Development

Last week I decided to format my macbook pro after several years of gem and vagrant clutter. My computer was suffering from lag, wasted space, spinning beach balls, and weird crashes. So after I backed up and formatted, I decided I was going to do things differently from now on because starting from scratch is such a pain. Since I had some free time on hand I thought I could take a chance to explore docker as a development environment.

After reading this great docker development article I figured I could do the same for puppet development as puppet development sorta falls under ruby development.

Now since I use bundler bundles for all my gems and puppet modules. I often accumulate lots of .bundle directories with the same gems installed across many repos. This means I often have to run bundle install and find /repos/ -name '.bundle' -type d -exec rm -rf {} \; which can take some time. I am sure you have seen this before? This pain becomes exponential on airplane wifi.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Fetching gem metadata from https://rubygems.org/...........
Fetching gem metadata from https://rubygems.org/..
Resolving dependencies...
Installing rake (10.4.2)
Installing CFPropertyList (2.2.8)
Installing addressable (2.3.8)
Installing backports (3.6.4)
Installing builder (3.2.2)
Installing hitimes (1.2.2)
Installing timers (4.0.1)
Installing celluloid (0.16.0)
Installing coderay (1.1.0)
Installing multi_json (1.11.0)
Installing gherkin (2.12.2)
Installing cucumber-core (1.1.3)
Installing diff-lcs (1.2.5)
Installing multi_test (0.1.2)
.
.
.
Installing typhoeus (0.7.1)
Installing travis (1.7.7)
Installing travis-lint (2.0.0)

Its this kind of repetition and wasted space that really grinds my gears. Furthermore, I can only test one puppet version at a time before running into gem conflicts.

So then I thought, Docker has the layered filesystem thingy, and reusable images which could greatly speed up the testing process and save space. I can even use this workflow in my CI pipelines too. So why wouldn’t I use containers?

Now the whole point is to ditch ruby version managers and bundler and rely purely on the system ruby and pre installed gems. We have the ability with docker images to mix and match ruby versions with puppet to give us the ultimate testing matrix.

I have mentioned in the past about the importance of unit testing your puppet code here. If you have spent any time unit testing your code you have probably only tested against a single version of puppet because it takes too much time to test against a good version matrix unless your using travis. However, by using containers we can easily swap out versions with a single command thereby reducing the feedback loop.

Now, before we start using docker for development, there are a few things you need. This should work on most platforms, but some instructions may differ slightly especially around using boot2docker.

Setup boot2docker

We need boot2docker to run the Docker container platform since Docker does not natively run on OS X.

  1. Install boot2docker brew install boot2docker
  2. boot2docker init
  3. boot2docker up
  4. eval `boot2docker shellinit`
  5. docker version # test docker works
  6. docker run --rm --hostname=puppetdev -t logicminds/centos-puppetdev:latest3.8.x puppet --version

Now if your curious if docker-machine can be used in place of boot2docker, docker-machine suffers from a volume mount issue. Furthermore, you need to ensure your puppet modules are under your home directory as detailed here.

ENSURE YOUR MODULE CODE IS CHECKED OUT UNDER YOUR HOME DIRECTORY

Using puppet development containers

The point of using containers for development is to not only conserve time but also isolate dependency issues. Additionally, some gems require development dependencies that you may not want on your system. So keeping them in a container allows you to keep your laptop in pristine condition while also having the ability to throw away your development environment with ease.

While the container images are static in nature, the container in which your test runs on will only last for the duration of the test. The lifecycle of containers is short and only meant to perform the task before exiting (unless its a long running process like a guard, webservice, or pry). So consider this an ephemeral development environment. When we start a container we are going to mount the current module directory on your machine into the docker container so that you can change your code on your computer and the container will automatically see these changes. Then we pass in any command we want the container to run. So before we start using docker, lets dissect the command first.

So given the following docker command, we can break down the parts below: docker run --rm -v $(PWD):/module --workdir /module --hostname=puppetdev -t logicminds/centos-puppetdev:latest3.8.x puppet --version

  1. docker run (simple docker command to start a container given an image)
  2. —rm (remove the container upon exit)
  3. -v $(PWD):/module (mount a volume designated by the $PWD environment variable to /module inside the container
  4. —workdir (change the working directory inside the container to /module
  5. —hostname (give the container a real hostname)
  6. -t (start s pseudo-TTY) – makes the colors show
  7. logicminds/centos-puppetdev:latest3.8.x (the docker image to use when starting the container)
  8. puppet —version (the command the container will run)

Run Multiple versions of puppet just by switching docker images

1
2
3
4
5
6
7
8
Coreys-MacBook-Pro% docker run --rm -v $(PWD):/module --workdir /module --hostname=puppetdev -t logicminds/centos-puppetdev:latest3.8.x puppet --version
3.8.1
Coreys-MacBook-Pro% docker run --rm -v $(PWD):/module --workdir /module --hostname=puppetdev -t logicminds/centos-puppetdev:latest3.7.x puppet --version
3.7.5
Coreys-MacBook-Pro% docker run --rm -v $(PWD):/module --workdir /module --hostname=puppetdev -t logicminds/centos-puppetdev:latest4.1.x puppet --version
4.1.0
Coreys-MacBook-Pro% docker run --rm -v $(PWD):/module --workdir /module --hostname=puppetdev -t logicminds/centos-puppetdev:latest3.2.x puppet --version
3.2.4

Interactive Usage

Now if you want the container to stick around so you can login and play around just pass the -i.
The -i makes the container interactive (if the command is /bin/bash you can login into the container and play around)

The interactive option only makes sense with shells like /bin/bash, or when the process your running requires some interaction.

Sample Workflows

Basic testing against three versions of puppet

  1. cd ~
  2. git clone https://github.com/logicminds/gitlab_mirrors.git
  3. cd gitlab_mirrors
  4. docker run --rm -v $(PWD):/module --workdir /module --hostname=puppetdev -t logicminds/centos-puppetdev:latest3.8.x rake spec
  5. docker run --rm -v $(PWD):/module --workdir /module --hostname=puppetdev -t logicminds/centos-puppetdev:latest3.7.x rake spec
  6. docker run --rm -v $(PWD):/module --workdir /module --hostname=puppetdev -t logicminds/centos-puppetdev:latest4.1.x rake spec
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
Coreys-MacBook-Pro% docker run --rm -v $(PWD):/module --workdir /module --hostname=puppetdev -ti logicminds/centos-puppetdev:latest3.8.x rake spec
Cloning into 'spec/fixtures/modules/stdlib'...
remote: Counting objects: 6896, done.
remote: Compressing objects: 100% (129/129), done.
remote: Total 6896 (delta 83), reused 22 (delta 22), pack-reused 6738
Receiving objects: 100% (6896/6896), 1.44 MiB | 852.00 KiB/s, done.
Resolving deltas: 100% (2987/2987), done.
HEAD is now at da11903 Merge pull request #299 from apenney/432-release
Cloning into 'spec/fixtures/modules/git'...
remote: Counting objects: 29, done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 29 (delta 0), reused 25 (delta 0), pack-reused 0
Unpacking objects: 100% (29/29), done.
/usr/bin/ruby -I/home/puppet/.gem/ruby/gems/rspec-core-3.2.3/lib:/home/puppet/.gem/ruby/gems/rspec-support-3.2.2/lib /home/puppet/.gem/ruby/gems/rspec-core-3.2.3/exe/rspec --pattern spec/\{classes,defines,unit,functions,hosts,integration\}/\*\*/\*_spec.rb --color
..................

Finished in 2.16 seconds (files took 0.64817 seconds to load)
18 examples, 0 failures

Coreys-MacBook-Pro% docker run --rm -v $(PWD):/module --workdir /module --hostname=puppetdev -ti logicminds/centos-puppetdev:latest3.7.x rake spec
Cloning into 'spec/fixtures/modules/stdlib'...
remote: Counting objects: 6896, done.
remote: Compressing objects: 100% (129/129), done.
remote: Total 6896 (delta 83), reused 22 (delta 22), pack-reused 6738
Receiving objects: 100% (6896/6896), 1.44 MiB | 853.00 KiB/s, done.
Resolving deltas: 100% (2987/2987), done.
HEAD is now at da11903 Merge pull request #299 from apenney/432-release
Cloning into 'spec/fixtures/modules/git'...
remote: Counting objects: 29, done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 29 (delta 0), reused 25 (delta 0), pack-reused 0
Unpacking objects: 100% (29/29), done.
/usr/bin/ruby -I/home/puppet/.gem/ruby/gems/rspec-core-3.2.3/lib:/home/puppet/.gem/ruby/gems/rspec-support-3.2.2/lib /home/puppet/.gem/ruby/gems/rspec-core-3.2.3/exe/rspec --pattern spec/\{classes,defines,unit,functions,hosts,integration\}/\*\*/\*_spec.rb --color
..................

Finished in 2.19 seconds (files took 0.64453 seconds to load)
18 examples, 0 failures

Coreys-MacBook-Pro% docker run --rm -v $(PWD):/module --workdir /module --hostname=puppetdev -ti logicminds/centos-puppetdev:latest4.1.x rake spec
Cloning into 'spec/fixtures/modules/stdlib'...
remote: Counting objects: 6896, done.
remote: Compressing objects: 100% (129/129), done.
remote: Total 6896 (delta 83), reused 22 (delta 22), pack-reused 6738
Receiving objects: 100% (6896/6896), 1.44 MiB | 845.00 KiB/s, done.
Resolving deltas: 100% (2987/2987), done.
HEAD is now at da11903 Merge pull request #299 from apenney/432-release
Cloning into 'spec/fixtures/modules/git'...
remote: Counting objects: 29, done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 29 (delta 0), reused 25 (delta 0), pack-reused 0
Unpacking objects: 100% (29/29), done.
/usr/bin/ruby -I/home/puppet/.gem/ruby/gems/rspec-core-3.2.3/lib:/home/puppet/.gem/ruby/gems/rspec-support-3.2.2/lib /home/puppet/.gem/ruby/gems/rspec-core-3.2.3/exe/rspec --pattern spec/\{classes,defines,unit,functions,hosts,integration\}/\*\*/\*_spec.rb --color
..................

Finished in 2.87 seconds (files took 1 second to load)
18 examples, 0 failures

Using pry with containers

  1. cd ~
  2. git clone https://github.com/logicminds/gitlab_mirrors.git
  3. cd gitlab_mirrors
  4. echo “require ‘pry’\nbinding.pry” >> spec/spec_helper.rb (example of pry interaction only)
  5. docker run --rm -v $(PWD):/module --workdir /module --hostname=puppetdev -ti logicminds/centos-puppetdev:latest3.8.x rake spec
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Coreys-MacBook-Pro% docker run --rm -v $(PWD):/module --workdir /module --hostname=puppetdev -ti logicminds/centos-puppetdev:latest3.2.x rake spec       
Cloning into 'spec/fixtures/modules/stdlib'...
remote: Counting objects: 6896, done.
remote: Compressing objects: 100% (129/129), done.
remote: Total 6896 (delta 83), reused 22 (delta 22), pack-reused 6738
Receiving objects: 100% (6896/6896), 1.44 MiB | 406.00 KiB/s, done.
Resolving deltas: 100% (2987/2987), done.
HEAD is now at da11903 Merge pull request #299 from apenney/432-release
Cloning into 'spec/fixtures/modules/git'...
remote: Counting objects: 29, done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 29 (delta 0), reused 25 (delta 0), pack-reused 0
Unpacking objects: 100% (29/29), done.
/usr/bin/ruby -I/home/puppet/.gem/ruby/gems/rspec-core-3.2.3/lib:/home/puppet/.gem/ruby/gems/rspec-support-3.2.2/lib /home/puppet/.gem/ruby/gems/rspec-core-3.2.3/exe/rspec --pattern spec/\{classes,defines,unit,functions,hosts,integration\}/\*\*/\*_spec.rb --color

From: /module/spec/spec_helper.rb @ line 3 :

    1: require 'puppetlabs_spec_helper/module_spec_helper'
    2: require 'pry'
 => 3: binding.pry
    4: puts "hello"

[1] pry(main)> exit
 hello
 ..................

 Finished in 2.14 seconds (files took 45.41 seconds to load)
 18 examples, 0 failures

Login to the container and look around

  1. docker run --rm -v $(PWD):/module --workdir /module --hostname=puppetdev -ti logicminds/centos-puppetdev:latest3.8.x /bin/bash
  2. which puppet
  3. puppet —version
  4. exit
1
2
3
4
5
6
7
8
9
10
11
Coreys-MacBook-Pro% docker run --rm -v $(PWD):/module --workdir /module --hostname=puppetdev -ti logicminds/centos-puppetdev:latest3.8.x /bin/bash
[puppet@puppetdev module]$ which puppet
~/bin/puppet
[puppet@puppetdev module]$ which ruby
/usr/bin/ruby
[puppet@puppetdev module]$ ruby --version
ruby 2.0.0p598 (2014-11-13) [x86_64-linux]
[puppet@puppetdev module]$ puppet --version
3.8.1
[puppet@puppetdev module]$ facter virtual
docker

Using bundler bundles to override the default gemset in the container

Now the containers I built are somewhat opinionated by this Gemfile. So if you need something else without building a new container image you can use bundler bundles. Although this workflow now becomes a two step process and bundle install is now being used which slows down your workflow. This uses the container to download and compile all the gems and places them inside a bundle folder for later use which is persistent on your machine. So each successive docker run will be able to use the prebuilt gems, provided you use bundle exec rake spec Additionally, bundler seems to spend a few seconds doing background tasks for each use which can be annoying.

  1. cd ~
  2. git clone https://github.com/logicminds/gitlab_mirrors.git
  3. cd gitlab_mirrors
  4. rm Gemfile.lock
  5. docker run —rm -v $(PWD):/module —workdir /module —hostname=puppetdev -ti logicminds/centos-puppetdev:latest3.8.x bundle install —standalone
  6. docker run —rm -v $(PWD):/module —workdir /module —hostname=puppetdev -ti logicminds/centos-puppetdev:latest3.8.x bundle exec rake spec

Puppetdev Container Repo

These images are available on the docker hub so its as easy as docker pull logicminds/centos-puppetdev:latest3.8.x

All images and various tags run on centos. Depending on the ruby version required by puppet this usually dictates which version of centos is used. This is due to centos7 containing ruby 2.0 by default and centos6 using ruby 1.9.3. There is no ruby version manager in the mix since we rely on the system ruby.

Currently Available tags:

  • latest4.1.x
  • latest3.2.x (not really compatible with ruby 2.0, but thats only during agent runtime which isn’t used during testing)
  • latest3.7.x
  • latest3.8.x

These are easy to make so others versions will be available upon request

Going a step further

If your tired of typing these commands you can create an alias for each puppet version.

1
2
3
4
# Puppet environment aliases
alias puppet38='docker run --rm -v $(PWD):/module --workdir /module --hostname=puppetdev -ti logicminds/centos-puppetdev:latest3.8.x'
alias puppet37='docker run --rm -v $(PWD):/module --workdir /module --hostname=puppetdev -ti logicminds/centos-puppetdev:latest3.7.x'
alias puppet41='docker run --rm -v $(PWD):/module --workdir /module --hostname=puppetdev -ti logicminds/centos-puppetdev:latest4.1.x'

Then to run a test against a container just use an alias puppet38 rake spec.

As you can see using containers is a great way to conserve time and space on your machine. Not having to wait 60s for a bundle install to run can save you time. There are probably a few more workflows out there as well. Imagine running guard in the background of a container that checks your code across multiple versions of puppet! Its like having a mini CI on your laptop. So give containers a try for your development environment.

References

http://www.hokstad.com/docker/patterns https://medium.com/@treeder/why-and-how-to-use-docker-for-development-a156c1de3b24

Testing Your Infallibleness - Part 1

Audience: System Admin, Self taught coder, Computer Scientist

Summary: How to leverage puppet-retrospec to generate your puppet unit test suite

Time Involved: 10 minutes

How many of you remember when the spell checker first came out? It had such a huge impact in everyones life because instantly everyone who used the spell checker appeared as if they had won the international spelling bee. How long did it take you to switch from looking up every single word in a paper dictionary to simply right clicking on a word? Assuming you grew up in the 90’s, we had to use books to spell check, crazy isn’t it. Spell checker turned horrible spellers into artisan letter assemblers overnight. With that said, I consider the spell checker to be the first form of unit testing because the spell checker tested every word of your summer vacation essay against a simple algorithm to ensure the word passed a basic spelling test.

So what does spell checking have to do with configuration management tools like Chef and Puppet? Basically, after you learn some basic CF (Configuration Management) programming you will need to start testing your own code, just like the spell checker. If your new to development which includes anybody using configuration management, unit testing your CF code brings a spotlight to bugs without having to pay much attention. This is especially important because a sysadmin’s workday is bombarded with distractions, especially from your trusty feline sidekick. So lets move forward and review some basic automated testing principles.

When it comes to testing there are two types one must test. You might have seen these before but these types are unit and acceptance/integration testing.

Unit testing

The unit test performs a very simple test to ensure that you do not have syntax errors and your conditional logic works against a set of supplied assumptions. Does your conditional statements work as intended? Do your variables interpolate correctly? Do the functions you use, perform the magic you expected? These are the things you should be asking yourself when building unit tests.

Writing your first unit test

With regards to configuration management code we will first test against compile time bugs because that is where the bulk of your mistakes will be caught and its also the fastest.

Part of the problem with new developers and lack of test code is the amount of time taken away from writing the “real” code. There are plenty of articles on how to write test code, but its often difficult to find something that caters to the absolute beginner and just getting the code setup to run tests is often too much. So questions like, whats a helper?, whats rspec?, mocking, mocha, shoulda, doubles, fixtures, unit test, integration test, a/b test? The problem is that you already don’t know what your doing and to make matters worse, now you have to learn new terminology to test the code that you barely know how to write. The mountain of knowledge needed to run a basic test serves as a barrier to becoming a better programmer and is often what separates a junior from a senior level programmer. Testing should be just as easy as writing the code itself. While I cannot change the fact that good testing practices is a skill in itself. I can at least automate some basic testing patterns and remove the immediate barrier to becoming a better configuration management programmer.

With this in mind I would like to introduce Retrospec. Retrospec is a tool that will generate puppet rspec test code based on the code inside the manifests directory. Its sole purpose is to get your module setup with automated testing, stat! Additionally, Retrospec will actually write some basic test code for you. Yes, you heard me right, it will write tests for you, even while you sip beer. While the generated tests are only basic checks it lays the groundwork for you to create even more advanced test cases and increase your BeerOps time. So lets get started.

The first thing you need to do is install retrospec gem install puppet-retrospec

We will use a puppet module I wrote for non-root environments as an example. Go ahead and clone this repo so you can follow along.

1
2
  git clone https://github.com/logicminds/puppet-nonrootlib
  cd puppet-nonrootlib

Now we just need to do some house cleaning to show how retrospec generates files easily. Obviously, you won’t do this in your own module, unless you really want to.

1
2
  rm -rf spec/  # helps show magic
  rm -f Gemfile Rakefile .fixtures.yml

You can run retrospec -h by itself to see what options you have.

1
2
3
4
5
6
7
8
9
10
11
12
$ puppet retrospec -h
Options:
        --module-path, -m <s>:   The path (relative or absolute) to the module
                                 directory (Defaults to current directory)
       --template-dir, -t <s>:   Path to templates directory (only for
                                 overriding Retrospec templates)
  --enable-user-templates, -e:   Use Retrospec templates from
                                 /Users/cosman/.puppet_retrospec_templates
    --enable-beaker-tests, -n:   Enable the creation of beaker tests
   --enable-future-parser, -a:   Enables the future parser only during
                                 validation
                   --help, -h:   Show this message

Now that we have a module without tests we can use Retrospec to retrofit the module with some basic tests. Many of these basic files are needed for the puppetlabs_spec_helper and rspec-puppet.

Note: If you have set your bundle path to be something other than GEM_HOME you will need to install retrospec in that path.

  `bundle exec gem install puppet-retrospec`, otherwise you will get an error.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
  $ puppet retrospec     # no need for -m if in the module directory
  + /private/tmp/puppet-nonrootlib/Gemfile
  + /private/tmp/puppet-nonrootlib/Rakefile
  + /private/tmp/puppet-nonrootlib/spec/
  + /private/tmp/puppet-nonrootlib/spec/shared_contexts.rb
  + /private/tmp/puppet-nonrootlib/spec/spec_helper.rb
  + /private/tmp/puppet-nonrootlib/.fixtures.yml
 !! /private/tmp/puppet-nonrootlib/.gitignore already exists and differs from template
 !! /private/tmp/puppet-nonrootlib/.travis.yml already exists and differs from template
  + /private/tmp/puppet-nonrootlib/spec/classes/
  + /private/tmp/puppet-nonrootlib/spec/classes/nonrootlib_spec.rb
  + /private/tmp/puppet-nonrootlib/spec/classes/rpm_spec.rb
  + /private/tmp/puppet-nonrootlib/spec/defines/
  + /private/tmp/puppet-nonrootlib/spec/defines/init_script_spec.rb
  + /private/tmp/puppet-nonrootlib/spec/defines/sysconfig_spec.rb

Bam! Wasn’t that easy?

Now Retrospec isn’t perfect but it did save us several hours of BeerOps time for us by generating all these files. You should see something similar like below when you run cat spec/classes/nonrootlib_spec.rb. This gives us a easy place to start. Keep following and I’ll show you some basic testing patterns to fix some of this generated code. Retrospec only generates files that you do not already have as noted by the !! and + symbols.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
require 'spec_helper'
require 'shared_contexts'

describe 'nonrootlib' do
  # by default the hiera integration uses hiera data from the shared_contexts.rb file
  # but basically to mock hiera you first need to add a key/value pair
  # to the specific context in the spec/shared_contexts.rb file
  # Note: you can only use a single hiera context per describe/context block
  # rspec-puppet does not allow you to swap out hiera data on a per test block
  #include_context :hiera


  # below is the facts hash that gives you the ability to mock
  # facts on a per describe/context block.  If you use a fact in your
  # manifest you should mock the facts below.
  let(:facts) do
    { }
  end
  # below is a list of the resource parameters that you can override.
  # By default all non-required parameters are commented out,
  # while all required parameters will require you to add a value
  let(:params) do
    {
      #:install_core => $home_dir,
      #:owner => $id,
      #:group => $id,
      #:etc_dir => $home_dir/etc,
      #:bin_dir => $home_dir/bin,
      #:var_dir => $home_dir/var,
      #:usr_dir => $home_dir/usr,
      #:tmp_dir => $home_dir/tmp,
      #:initd_dir => $home_dir/etc/init.d,
      #:lib_dir => $home_dir/lib,
      #:sysconfig_dir => $home_dir/etc/sysconfig,
      #:run_dir => $home_dir/var/run,
      #:lock_dir => $home_dir/var/lock,
      #:subsys_dir => $home_dir/var/lock/subsys,
      #:log_dir => $home_dir/var/log,
    }
  end
  # add these two lines in a single test block to enable puppet and hiera debug mode
  # Puppet::Util::Log.level = :debug
  # Puppet::Util::Log.newdestination(:console)
  it do
    is_expected.to contain_file('[$install_core, $etc_dir, $bin_dir, $var_dir, $usr_dir, $tmp_dir, $initd_dir, $lib_dir, $sysconfig_dir, $run_dir, $lock_dir, $subsys_dir, $log_dir]').
             with({"ensure"=>"directory"})
  end
  it do
    is_expected.to contain_file('$home_dir/bin/service').
             with({"ensure"=>"present",
                   "content"=>"template(nonrootlib/service.erb)",
                   "require"=>"File[$bin_dir]"})
  end
  it do
    is_expected.to contain_file('$home_dir/.bash_profile').
             with({"ensure"=>"present",
                   "content"=>"template(nonrootlib/.bash_profile.erb)",
                   "require"=>"File[$install_core]"})
  end
end

Test Prep

Once you retrospec your module many tests are generated but need to be prepped for testing. Note the tests will fail until you refactor the test code. Since testing puppet code relies heavily on other gems we need to use bundler to download all these dependencies.

  1. cd puppet-nonrootlib # if not already in the directory
  2. gem install bundler # unless bundler is already installed
  3. bundle install # installs all the gems necessary for puppet unit testing ( You should be in the module directory)
  4. bundle exec rake spec_prep # sets up fixtures, not necessary if using rake spec
  5. bundle exec rspec spec/classes/nonrootlib_spec.rb # run your test against a single test file

Normally you would run bundle exec rake spec but I wanted to just run a single test file so I used rspec directly.

Lets go ahead and open the spec/classes/nonrootlib_spec.rb file and refactor the test code, because out of the box, these tests will fail.

Basic mocking

Below is an example of how you mock facts in a rspec-puppet testing environment. I refactored spec/classes/nonrootlib_spec.rb to work by specifying the facts to mock like home_dir and id. Note, these are just mocks so directories and users don’t actually have to exist. I only need to mock the facts that are used in my manfiest code. Go ahead and update your spec/classes/nonrootlib_spec.rb file to match the facts block below.

1
2
3
  let(:facts) do
    { :home_dir => '/home/user1', :id => 'user1' }
  end

Note: You can only mock hiera values, facts, params, and functions with rspec-puppet. This is all you really need to influence conditional logic in your code as your will be testing against the catalog. I am only covering facts and params in this article since the other items are considered an advanced topic. Plus your attention span can’t handle much more anyways.

Parameter Mocking

Below is a real example of how you can mock parameters to incluence your conditional logic. It follows the same syntax as mocking facts. You may have noticed that retrospec comments out any parameters with default values. However, I specified each parameter value statically as I consider specifying the parameter values better for long term test maintainability. If a future developer changes the default parameter values in your manifest code some of these test will break, so its good practice to set them in stone here. But your not required to do this which is why they are commented out in the initial test generation. This is where you can mock parameter values and test against different scenarios. Its worth noting that without Retrospec you would had to specify every parameter by hand.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
let(:params) do
    {
      :install_core => '/home/user1',
      :owner => 'user1',
      :group => 'user1',
      :etc_dir => '/home/user1/etc',
      :bin_dir => '/home/user1/bin',
      :var_dir => '/home/user1/var',
      :usr_dir => '/home/user1/usr',
      :tmp_dir => '/home/user1/tmp',
      :initd_dir => '/home/user1/etc/init.d',
      :lib_dir => '/home/user1/lib',
      :sysconfig_dir => '/home/user1/etc/sysconfig',
      :run_dir => '/home/user1/var/run',
      :lock_dir => '/home/user1/var/lock',
      :subsys_dir => '/home/user1/var/lock/subsys',
      :log_dir => '/home/user1/var/log',
    }
end

Basic Tests

Since the manifest code creates a bunch of directories you can speed up your test creation by using the ruby each iterator and iterate around the resources you are creating by defining an array. This only works because all the resources have the same attributes with the exception of the name. Alternatively, you could statically define each test case as well, but call me lazy. Rspec-puppet which is the testing library required for testing puppet code will query the catalog that puppet generated during the testing process.

When you write a test it should mentally read, “The manifest named nonrootlib when compiled into a catalog is expected to contain the file resource XX with ensure set to directory.”

1
2
3
4
5
6
7
8
9
 dirs = ['/home/user1', '/home/user1/etc', '/home/user1/bin','/home/user1/usr', '/home/user1/tmp', '/home/user1/etc/init.d',
    '/home/user1/lib', '/home/user1/etc/sysconfig', '/home/user1/var/run', '/home/user1/var/lock', '/home/user1/var/lock/subsys',
    '/home/user1/var/log'
    ]
 dirs.each do | dir|
     it do
       is_expected.to contain_file(dir).with({"ensure"=>"directory"})
     end
 end

I have also gone ahead and removed the content line from the service and bash_profile resource because I will discuss verifying content in a future article. So go ahead and replace the contents below in your own spec/classes/nonrootlib_spec.rb file just like below.

1
2
3
4
5
6
7
8
9
10
  it do
    is_expected.to contain_file('/home/user1/bin/service').
             with({"ensure"=>"present",
                   "require"=>"File[/home/user1/bin]"})
  end
  it do
    is_expected.to contain_file('/home/user1/.bash_profile').
             with({"ensure"=>"present",
                   "require"=>"File[/home/user1]"})
  end

The finished test code after refactor.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
require 'spec_helper'
require 'shared_contexts'

describe 'nonrootlib' do

  # below is the facts hash that gives you the ability to mock
  # facts on a per describe/context block.  If you use a fact in your
  # manifest you should mock the facts below.
  let(:facts) do
    { :home_dir => '/home/user1', :id => 'user1' }
  end
  # below is a list of the resource parameters that you can override.
  # By default all non-required parameters are commented out,
  # while all required parameters will require you to add a value
  let(:params) do
    {
      :install_core => '/home/user1',
      :owner => 'user1',
      :group => 'user1',
      :etc_dir => '/home/user1/etc',
      :bin_dir => '/home/user1/bin',
      :var_dir => '/home/user1/var',
      :usr_dir => '/home/user1/usr',
      :tmp_dir => '/home/user1/tmp',
      :initd_dir => '/home/user1/etc/init.d',
      :lib_dir => '/home/user1/lib',
      :sysconfig_dir => '/home/user1/etc/sysconfig',
      :run_dir => '/home/user1/var/run',
      :lock_dir => '/home/user1/var/lock',
      :subsys_dir => '/home/user1/var/lock/subsys',
      :log_dir => '/home/user1/var/log',
    }
  end
  # add these two lines in a single test block to enable puppet and hiera debug mode
  # Puppet::Util::Log.level = :debug
  # Puppet::Util::Log.newdestination(:console)
  dirs = ['/home/user1', '/home/user1/etc', '/home/user1/bin','/home/user1/usr', '/home/user1/tmp', '/home/user1/etc/init.d',
   '/home/user1/lib', '/home/user1/etc/sysconfig', '/home/user1/var/run', '/home/user1/var/lock', '/home/user1/var/lock/subsys',
   '/home/user1/var/log'
   ]
  dirs.each do | dir|
    it do
      is_expected.to contain_file(dir).with({"ensure"=>"directory"})
    end
  end
  it do
    is_expected.to contain_file('/home/user1/bin/service').
             with({"ensure"=>"present",
                   "require"=>"File[/home/user1/bin]"})
  end
  it do
    is_expected.to contain_file('/home/user1/.bash_profile').
             with({"ensure"=>"present",
                   "require"=>"File[/home/user1]"})
  end
end

Working Example

At this point your ready to test and should see similar output (I have omitted some deprecation warnings)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
$ bundle exec rake spec_prep
$ bundle exec rspec spec/classes/nonrootlib_spec.rb


nonrootlib
  should contain File[/home/user1] with ensure => "directory"
  should contain File[/home/user1/etc] with ensure => "directory"
  should contain File[/home/user1/bin] with ensure => "directory"
  should contain File[/home/user1/usr] with ensure => "directory"
  should contain File[/home/user1/tmp] with ensure => "directory"
  should contain File[/home/user1/etc/init.d] with ensure => "directory"
  should contain File[/home/user1/lib] with ensure => "directory"
  should contain File[/home/user1/etc/sysconfig] with ensure => "directory"
  should contain File[/home/user1/var/run] with ensure => "directory"
  should contain File[/home/user1/var/lock] with ensure => "directory"
  should contain File[/home/user1/var/lock/subsys] with ensure => "directory"
  should contain File[/home/user1/var/log] with ensure => "directory"
  should contain File[/home/user1/bin/service] with ensure => "present" and require => "File[/home/user1/bin]"
  should contain File[/home/user1/.bash_profile] with ensure => "present" and require => "File[/home/user1]"

  Finished in 0.35775 seconds (files took 0.86506 seconds to load)
  14 examples, 0 failures

Now that you have created your first rspec-puppet test you should be able to start testing your infallibleness on your own module and find out just how perfect your code is. With the help of Retrospec this should be pretty easy. But remember unit testing is not just about testing your code, its about maintaining code integrity long after you have left. Because many times when new folks are added to a team they make a ton of mistakes until they are familiar with the code base. So its important to build a safety net for them with basic unit tests that allows them to test against a feature set outlined in the test code.

There are many things that I did not discuss in this article that are very important but Retrospec automated these things such as .fixtures.yml, spec_helper, Rakefile, Gemfile and others. So stay tuned and I’ll cover the these items in a later article.

Remotely Controlling Your Server With IPMI

In 2010 my life changed dramatically as I moved from Atlanta, GA to Portland, OR. During this time I maintained a 100% remote position where I controlled twenty unique datacenters all over the country with my computer and a VPN connection. Due to to the nature of the company, problems only seemed to present themselves during the early hours (After 12AM). On several occasions I found the need to turn off a production system due to power or cooling issues. Traditionally, this was easily accomplished by picking up the phone and instructing a person to perform a task. However, finding an able body in the middle of the night sometimes extended downtime past a few hours. And god forbid they might turn off the wrong system. This led to unnecessary downtime in my opinion that could have been prevented by using out-of-band management.

What is out-of-band management? Essentially, out-of-band management is a device that allows you to perform actions on a server as if you were standing in front of it. This type of management works independently of the OS, so if the OS crashes you still have a way in. The standard protocol behind out-of-band management is called IPMI. However, many device manufactures had similar solutions before IPMI was even available. Solutions like HP’s iLO, IBM’s OSA, Dell’s DRAC, and Sun’s iLom. When IPMI came along, server manufactures shoe-horned IPMI support into their proprietary out-of-band management devices. The end result is a buggy IPMI device that “mostly” works. However, what IPMI provides is standard mechanism to control any kind of server despite its origin. Fortunately, newer IPMI devices are more compliant with the IPMI protocol than their predecessors.

Setup

The first thing you need to know about IPMI is that it is not configured out the box. In many situations you will need to configure the IPMI device from the BIOS or upon bootup. If your lucky enough to be using a server that sets up IPMI automatically via DHCP then your in luck. For example, HP ilo devices will use the dhcp assigned ip address when plugged in. Some devices like HP ilo come with a unique username/password to initially login to the device, while others have some sort of insecure default credentials hidden in the manual. Essentially what you need to do is login (via ssh or http) to the device and configure to your requirements. In a later article I will detail how you can automate the deployment of IPMI devices so you never have to perform this step manually again.

Locating that special port

As you can see in the following picture the ilo device is the special out-of-band management device that allows us to use IPMI commands to control the server remotely. It looks just like a normal ethernet port but is usually marked specifically for the BMC controller (ilo, ipmi, drac, …). So just plug a ethernet cable in and find the IP address using nmap, dhcp logs, or other mechanism.

Server with IPMI device

Remotely controlling that server

Usually each IPMI device has a web interface for configuring and controlling the device. So initially, you will have to use this interface to set it up. In the picture below I have logged into the device and clicked on power management and then clicked momentary press to turn the server on. Disclaimer: I am not responsible for you powering on/off servers in any of your environments. Make sure you know what your doing.

ilo page

Once the IPMI device turns the server on the options change and we can now see the server is powered on.

ilo page

Some of you may be asking yourself, is there a CLI tool to perform these functions? The answer is yes, however the CLI commands are generic in nature and some functionality may not be available in the opensource CLI tools. But, the majority of what you want to do should be supported.

These tools are freeipmi and ipmitool both of which perform the same function but differ in implementation. Additionally, there is openipmi which works as a driver that freeipmi and ipmitool use for communication with the IPMI device. But openipmi is not necessary if you know the username, password and ip address of the IPMI device.

With one of these tools installed you can skip the web UI and remotely control the system from any unix based system running either ipmitool or freeipmi.

1
2
ipmitool -H 192.168.1.21 -U admin -P password -I lanplus power on
Chassis Power Control: Up/On

Futher reading

Now if your wondering just how to configure your IPMI device using configuration management have a look at the bmclib puppet module for configuring each IPMI device. Bmclib internally uses ipmitool and openipmi so that you don’t have to.

Additionally, if you are looking for a way to interact with the IPMI device pragmatically you can use the Rubyipmi library directly without having to know any knowledge of ipmitool or freeipmi.

There is a ton of imformation you can get out of the IPMI device like sensor information and event logs so its worth exploring with Rubyipmi if you plan on doing sensor monitoring of some sort.

A great example of monitoring with Rubyipmi and Sensu.

Summary

Now that you know how to remotely control your servers you can spend more time doing other things since you won’t be on the phone instructing other people to turn the server off. Furthermore, stand up and move your legs since you just removed the need to walk to the server room and reduced your daily pedometer score.