A Nice Fedora Environments Graphic

On the Wiki.

I don't see puppet01 on the diagram, but I think that's an easier(-for-me)-to-remember synonym for lockbox01, under Support.

Vagrant & Stack Haemer: More Tools that Look Cool

I saw Vagrant and asked around a little to see if anyone I know has tried it.  Not yet.

Ben Whaley said he'd poked at Stack Hammer, a project that seems to pal around with Puppet.  When I pointed out to him that they'd misspelled "Stack Haemer," he recommended I fork the project on GitHub, fix the spelling, and send them a pull request.

Both seem pointed at a problem I'm bumping up against: now that making VMs is easy, both on my box and in the cloud, how do I manage and keep track of them?  Puppet's a piece of that, but it ain't all of it.

FI Mailing List Has Lots of Traffic

When I signed up for the Fedora Infrastructure mailing list, I picked "digest."  I didn't want to have to filter individual Fedora messages in my mailbox. 

I expected to get -- I dunno -- a digest a month?  A week, maybe even?  I'm getting a couple of digests a day.  Lots of traffic.

Plus, it's a fire hose of chatter about terms and topics I know nothing about: Koji, AMPQ, SELinux, ....

It's great.  My goal is always to be the dumbest guy in my peer group.

Puppet-Lint and Other Toys

Random notes on puppet-related things I'm ignorant of but look worth exploring when I get a minute:

Lots of folks seem to recommend puppet-lint.

Jan Vansteenkiste has tips on integrating puppet with Jenkins, and puppet-module patterns that I should probably read through.

Tim Sharpe makes reference to a tool called rspec-puppet that I think is still over my head.

I feel farther behind every day.  Ah well.

Should the Init.pp Files Be Split Up?

Looks to me like the module/manifests/init.pp files should be broken up.

Today, they're monolithic, and hold all the class and subclass definitions for a module.

Puppet looks at all the *.pp files for a module under manifests/.  The suggested structure for this directory is that each subclass be defined in its own manifest, so that foo/manifests/init.pp defines class foo, foo/manifests/bar.pp defines class foo::bar and foo/manifests/mumble/frabitz.pp defines class foo::mumble::frabitz .

The refactor-init script in github:jsh/fedora-infrastructure creates this structure from an existing init.pp.  (My guess is it's still too simple-minded and incomplete, but it's a start.)

Unit tests are supposed to mirror this structure, and mk-manifest-unit-tests, also on GitHub, does this job.

GitHub Repo Will Hold My Scripts

I've made a GitHub repo to hold scripts.  Who knows?  Maybe someone else will find them useful, too.  Even if not, it'll give me a nice synchronization point, so I can get them from any machine.

The URL?
git://github.com/jsh/fedora-infrastructure.git

Making Unit Tests to Find Module Problems

The puppet repository has grown like Topsy, so some pieces that were top-of-the-line when they were first created are no longer best-practice.

One of these is the module structure. The top-level README suggests this structure:

    modulename

        /files       # For config files

        /manifests   # For puppet manifests

        /plugins     # For puppet plugins

        /templates   # For config templates

        README       # A brief description of the module

The plugins/ subdirectory has now been renamed lib/, and the new, best-practices structure includes a tests/ directory with unit tests for each class.

Running these tests with a --noop flag will trigger warnings about a variety of deprecated features and syntax errors.  For example, this one

notice: DEPRECATION NOTICE: Files found in modules without specifying 'modules' in file path will be deprecated in the next major release.  Please fix module 'bash' when no 0.24.x clients are present

is warning that the line

source => 'puppet:///bash/bashrc',

in bash/manifests/init.pp should now be updated to read

source => 'puppet:///modules/bash/bashrc',

Here's a script that checks the structure of a module, adds anything that needs to be in it, and runs the unit tests.

---

#!/bin/bash -eu

#

# mk-puppet-module

#  impose current best-practices structure on the named modules

#  Usage: mk-puppet-module [-c] [-g] modulename

#    -c: create anything that doesn't exist

#    -g: verify that we're working in a git repository and

#        remove any empty directories.

trap '{ (( errors == 0 )) || echo "$module: $errors errors"; }' EXIT

errors=0

usage="usage $0 [ -c ] [ -g ] modulename"

warn() { echo $* >&2; (( errors += 1 )); }

die() { echo $* >&2; exit 1; }

module_dirs="files manifests lib templates tests"

if [ ${PWD##*/} = "modules" ]; then

    modulepath=$PWD

else

    die "$PWD not a modules directory"

fi

# parse options

while getopts cg opt; do

    case $opt in

        c) create=1  ;;

        g) gitwork=1 ;;

    esac

done

: ${create:=''}

: ${gitwork:=''}

# now get the module names

shift $((OPTIND - 1))

[ $# = 1 ] || die $usage

module=$1

# validate gitrepo

if [ "$gitwork" ]; then

    puppetrepo=/git/puppet

    [ $(git config --get remote.origin.url) = "$puppetrepo" ]

fi

# check for existence

if [ -d $module ]; then

   cd $module

elif [ "$create" ]; then

   warn "creating $module"

   mkdir $module; cd $module

else

   warn "$module missing"

   exit 1;

fi

  

if [ -f README ]; then

    true

elif [ "$create" ]; then

    warn "creating $module/README"

    echo "Placeholder README for $1" > README

else

    warn "$module/README missing"

fi

# validate module directory structure

for dir in $module_dirs; do

    if [ -d $dir ]; then

    true

    elif [ "$create" ]; then

        mkdir -p $dir

    else

        warn "$module/$dir missing"

    fi

done

# validate manifests subdirectory

cd manifests

if [ -f init.pp ]; then

    grep -q "^class $module" init.pp ||

        warn "$1/manifests/init.pp missing class definition"

elif [ "$create" ]; then

    warn "creating $module/manifests/init.pp"

    printf "class $1 {\n}\n" > init.pp

else

   warn "$1/manifests/init.pp missing"

fi

# validate tests subdirectory

for i in *; do

    [ -d ../tests ] || continue  # don't bother if there's no tests directory

    if [ -f ../tests/$i ]; then

        true

    elif [ "$create" ]; then

    warn "creating $module/tests/$i"

        perl -lane 'print "include $1" if /^class\s+(\S+)/' init.pp > ../tests/$i

    else

        warn "$tests/$i missing"

    fi

    if [ -f ../tests/$i ]; then

    puppet apply --modulepath=$modulepath -v --noop ../tests/init.pp

    fi

done

if [ "$gitwork" ]; then

    git add .

    git clean -f -d

    git reset

fi

I Turn SELinux Back On, and Other Cleanup

When I was having trouble getting Fedora updates, I noticed that SELinux is turned on by default.  I've always treated it as a nuisance, so I turned it off,
by changing "SELINUX=enforcing" to "SELINUX=disabled".

That turned out not to be the problem, so I asked Kevin Fenzi whether I should turn it back on.  He recommends I do, so I did and rebooted.

I also removed the clone of the puppet config files from my other box, at his recommendation.

What's an Empty Module Look Like?

One module is particularly compact: ip6tables.

The init.pp file is empty.  There is, however, a README, which says this:

=====================
ip6tables
=====================
-----------
Usage
-----------
The ip6tables utility controls the network packet filtering code in the,
Linux kernel. If you need to set up ipv6 firewalls, you should install
this package.

Looks like a module that someone's just creating.  A quick look at the log says that's exactly right: it was just created a few days ago, with the comment.

    Add in minimal files for ip6tables.


This tells me the skeleton FI is using for its modules.  What are the files?

$ tree .
.
|-- manifests
|   `-- init.pp
|-- README
`-- templates
    `-- iptables.erb

The third file, iptables.erb, is also still empty.   I'll get to watch them grow.

Unfortunately, I don't know much of anything about ip6tables, or, really, any of ip6.  Perhaps that'll turn into "fortunately," if it prompts me to learn something so I can understand what I'm seeing.  It's all in how I look at it.

The pessimist says, "The glass is half empty."
The optimist says, "The glass is half full."
The engineer says, "The glass is twice as big as it needs to be."

The Law of Unintended Consequences Strikes

With the best of intentions, I grabbed a Fedora 16, live CD, and used it to install F16 on a small, VirtualBox VM.

Once installed, the next step was to pull all the updates.  Or, well, you know, try.

$ sudo yum update

Cannot retrieve repository metadata (repomd.xml) for repository:

Uh.

Getting past this took time. I tried a bunch o' stuff recommended by the experts of the interwebs.  I turned off (just in case) SELinux.  I made sacrifices to a rubber chicken.  I even read the documentation. Eventually, I found something that worked (sslverify=0 in /etc/yum.conf).  

That the distro has this problem out-of-the-box would seem like a bug.

However, I find I am now reluctant to figure out how to file a bug report, since, as a member of the Fedora team, I might then be asked to help fix it and I have no idea how to.

No good deed goes unpunished.

Q: What goes "Ha ha ha <THUNK!>" ?

A: A man laughing his head off.

Kevin has now assured me that this sequence:

$ yum clean all

$ yum update

would probably have done the trick.  Next time.

I Can Email the List

Kevin diagnosed and fixed my mailing-list problem.  I'd subscribed as a Gmail "plus" address, but submitted email as my normal, nonplussed self, so it wasn't recognizing me.

Formatting Follies

I need to figure out how to post code blocks.  I'll do some experiments here.

 $ ls
foo

Well, that works.  I have to go in and insert line breaks manually.  Harumph. I see the question a lot on the web, and no answers.  Be nice if Blogger gave geeks a way to do that easily.

You Have Much to Learn, Grasshopper

I just got mailed my third infrastructure-Digest.  This one had something in it I had a question about, so I asked it.  I got an immediate response, saying my submission was being held, awaiting approval, because I was a non-member posting to a members-only list.

I thought, "If I'm not a member, why am I getting the digests?"

I have much to learn.

On to Modules

Having taken a first look at the nodes, I think I'll also take a quick tour of the modules they include.

- Where are they?

$ cd puppet/modules

- How many modules are there?

$ ls | wc -l
131

A fair number.  I have no standard of comparison.  At this level, it's a flat directory structure, which makes the number pretty big, but I haven't yet seen another way to organize, group, and simplify this structure.

- Which modules have a substructure?

$ ls */manifests/*.pp | grep -v init.pp

Yikes!  All the modules have everything in their init.pp files.  This is a bit different from the examples I saw in class, which farmed out different pieces of the modules into different .pp files.  Okay, that's interesting.

- How big are the init.pp files?

$ wc -l */manifests/*.pp | sort -n

Some are pretty big.  httpd's is 537 lines long; however, almost a third of them fit in a single screen

$ wc -l */manifests/*.pp | awk '$1 < 24' | tee /tmp/short | wc -l
42

The answer to life, the universe, and the number of Fedora Infrastructure Puppet modules that will fit on a terminal screen.

- How many have subclasses?
The course I took showed us subclasses, but I'm not comfy with them yet, so it'd be good to look through some examples to understand them.

$ grep 'class.*::'  */manifests/*.pp | uniq | tee /tmp/subclassed | wc -l
112

- What short, init.pp files have subclasses?

Some modules that'll fit on a screen even have subclasses, since 112+42 > 131.  Which ones?

$ comm  -12 /tmp/short /tmp/subclassed | wc -l
34

Great!  Most of the short init.pp files have subclasses.  Files short enough that I can read them on the screen, but with enough complexity that I'll learn how folks really use subclasses.

Where My Nodes Leads Me

What Fedora Infrastructure machines does Puppet help configure?  These must be listed under manifests/nodes. Time to explore a little ... I'll just follow my nodes.

- Go into the manifests/nodes subdirectory. What's there?

Nodes managed by puppet.

- How many nodes are listed? 

$ ls *.pp | wc -l
176

- Which of these hostnames are unknown? 

$ for i in *.pp; do ping -c1 ${i/.pp/} > /dev/null; done |& tee ~/unknown | wc -l

6

Are these down, or just hosts that no longer exist?

- How many, known or unknown, are un-ping-able? 

$ for i in *.pp; do ping -c1 ${i/.pp/} &> /dev/null || echo $i; done |& tee ~/cannotping | wc -l
17

Are they blocking pings, or are these really not Puppet-configurable?

- Are any descriptions duplicates of one another? 

$ for i in *.pp; do perl -pe 's/^node.*//' $i | md5sum; done | sort | uniq -c | awk '$1 > 1'
      2 0543743a9b438bc5c5530cbbe4719584  -
      3 05c68ad0e20d10c5ffa8605748a64ace  -
      2 21c2d6043fb8fd87cda5092eaa76d129  -
      2 2b56c5db637e1c59f19a50818ef6fe32  -
      2 5744cbd978245e061b48e050ae15c804  -
     21 aacdff160a5397d6cb6c9809ed910eca  -
      2 d05aed91de3e7af1d29956db1cbcc65a  -

(Here, I pull off the "node" line because otherwise identically configured nodes have different node names.)

If Puppet lets node files point at more than one host, the number of node files would drop.  Does it?  I'll have to look that up.

- Which machines share the modal (most frequent) description? 

$ for i in *.pp; do printf  "$i"; perl -pe 's/^node.*//' $i | md5sum; done | grep aacdff

For example, most of these have almost the same name:

 x86-NN.ph2.fedoraproject.org.pp .

Does Puppet let nodenames be regular expressions (or globs)?  I'll have to look that up.

- How big are the descriptions?

$ for i in *.pp; do wc -l $i | sort -n | sed -n '1p;$p'

93 insight02.dev.fedoraproject.org.pp
4 app06.fedoraproject.org.pp

Some of these only include one module. Some are huge. I wonder whether any standard statistical distribution, like a Poisson, fits the sizes.

First Steps

Well, I have to start somewhere.  I'll just start poking around to see what I find, and follow my nose.

- Google for "Fedora infrastructure puppet."  Where are notes for how to check out puppet?

http://infrastructure.fedoraproject.org/infra/docs/puppet.txt

These notes have an older flavor than my course.  Ben Ford says that puppet is now moving to the git/svn model of  "command subcommand flags," and that what was once "puppetca" is now "puppet cert".  The notes look like they use the older syntax.

- How do I get to the machine for checking out the puppet git repo?

$ ssh lockbox01.phx2.fedoraproject.org

- What's the command to check it out?

$ git clone /git/puppet

  Instructions say only check it out on lockbox01.  I suspect this means just use it there, since anywhere else wouldn't make sense. Exploring the repo locally is easier, so I check it out on my own machine like this:

$ git clone lockbox01.phx2.fedoraproject.org:/git/puppet

- The top-level "main" -- the place Puppet looks first -- is manifests/site.pp.  This installation just has site.pp import other files in the directory.  What manifests are imported that aren't in the repo?

private/*.pp

There is no private directory, but this suggests that a typical practice might be to put your own private puppet manifests into a local directory by this name, for experimentation.

My Enlistment

Week before last, I took a three-day, Puppet course, taught by Ben Ford, of Puppet labs.  Nice course, but now I need to use it.  The best way would be to work on a big, Puppet installation, but my company doesn't have one.

As luck would have it, Kevin Fenzi, who ramrods Fedora Infrastructure, gave a nice, Boulder Linux Users Group (BLUG) talk in February and mentioned that they use puppet.  Also, his group has an apprentice program, where novices can get adult supervision.

Every Thursday night, I wander over to the Southern Sun, my neighborhood brewpub, for supper and beer with folks I know from BLUG.  Afterwards, we head over to Caffe Sole, a coffee shop in the same shopping center, and geek out.  Regulars include Chuck Hipschmann, Rich Johnson, and ... Kevin.


Last Thursday night, over coffee and with Kevin Fenzi's coaching, I signed on as a Fedora Apprentice.