Giving Git a run-out

Something apparently snapped while I was at OSCON, and I apparently collapsed my distributed source control management quandary down to Git. I think in the end it doesn't matter all that much, since they're so similar in basic functionality, and I think I mostly got tired of sitting on the fence looking over but not being willing to commit to a specific dSCM.

Once I decided I'd go ahead with Git, my main priority was to get to the point where I could do my development on Puppet in it, which is especially important since it's the only real way for me to figure out if it will work for me, not that I really know what "work for me" means.

There are two crucial steps to testing an SCM for me: Getting Puppet's code into it, with as much history as possible, and making it available for others to have access to.

Getting the code was moderately easy, but made harder by the fact that when I first made my Subversion repository, when SVN was just starting to get popular, so I started without the typical branches/tags/trunk directory set. Here's the command I used in the end:

git svnimport -A ~/puppet-users -i -v http://reductivelabs.com/svn/puppet/ > /tmp/git.out

I tried git-svn, but it never got past revision 567 or so (which is when I switch to the popular directory structure). In addition, I was never able to actually get a working copy of the repository up to that point.

The puppet-users file contains a mapping from svn-style user names to email addresses:

luke = Luke Kanies <luke@domain.com>
lutter = David Lutterkort <dlutter@domain.com>
mpalmer = Matthew Palmer <mpalmer@domain.org>

I redirect output to a file, because it produces a bunch of output (I've got about 2800 revisions) and I don't actually care about any of it, and in addition, because I use iTerm, it takes a whole freaking cpu to scroll a terminal.

This basically worked, except that it started at revision 600 (arbitrarily close enough to the time when I changed the directory structure in the repository).

To make the repository shareable, I first just exported it via http, which was pretty easy, but then I was told I need to use git-server for performance reasons. I built a Puppet module to set it all up, and although the server doesn't work as well as I like (I really like SVN's auth file, which allows me to control who has access to the 32 repositories I maintain).

I'm getting some gritching from the Australians, and it's not like it's perfect, but at least I know I want something like that.

At the least, this has been a great experiment, and I figure we'll spend a week or so messing around with it. I'm not sure I can afford the time to experiment with all of the competitors; Matt's really pushing on darcs, but... I dunno, it seems niche, and at this point, I'm niche enough for all of us.

Tue, 07 Aug 2007 | Tags: tools, git, scm, svn, oscon, oscon07

How I Add Code to My Presentations

Many of my presentations include source code (including yesterday's), and I always struggle a bit to find a way to get attractive, syntax highlighted code into my presentations. This is further complicated by the fact that I often include Puppet code, and there are currently only syntax highlighters for Vim and Emacs, so the apparently-normal Enscript highlighting won't work for me.

So, I hacked up a Vim script that can create marked up HTML, then I threw that into a Rake file that converts my code samples into HTML. Here's the code:

outputs = []

FileList['examples/*'].each do |input|
    output = File.join("output", File.basename(input)).sub(/\.\w+/, ".html")
    outputs << output

    file output => ["Rakefile", input] do
        vimsyn(input, output)
    end
end

def vimsyn(file, outfile = nil)
    filetype = case file
        when /\.pp/: "puppet"
        when /\.rb/: "ruby"
        when /\.mk/: "make"
        when /\.sh/: "bash"
        when /\.txt/: "text"
        else
            raise "Unknown File type for %s" %  file
        end
    name = File.basename(file).sub(/\..+/, '')
    outfile ||= "/tmp/outfile"
    synfile = outfile + ".tmp"
    codefile = file

    filetype="puppet"

    puts "Creating %s" % outfile
    expr = %Q%/usr/bin/vim -f -n -X -e -s -c  'set filetype=#{filetype}'  -c 'syntax on'    -c 'let html_use_css=1'  -c 'run syntax/2html.vim'  -c 'wq! #{synfile}' -c 'q'  #{codefile}%

    `#{expr}`
    html = File.read(synfile)
    html.sub!(/<title>.+<\/title>/, "<title>#{name}</title>")
    html.sub!(/^(pre.+) \}/) { $1 + "font-size: 22pt; }" }
    html.gsub!(/text-decoration: underline;/, '')
    File.open(outfile, "w") { |f| f.print html }
    File.unlink(synfile)
end

task :clean do
    outputs.each do |file|
        if FileTest.exists?(file)
            File.unlink(file)
        end
    end
end

task :default => outputs

I usually (but not always) use Apple's Keynote. So, for this presentation, I used Keynote's "Web Views" to include the HTML directly in the presentations. There were some significant hitches to doing so: First and most stupidly, Keynote won't include local file:/// URLs, only http:///, which means that you can't embed local files. This is apparently by design (someone mumbled something about security, which is, um, inane).

The second hitch is that there does not appear to be any way to change the text size within the web view, so you have to generate the html with the correct size. This is pretty stupid, considering that we otherwise make sure we use scalar graphics instead of PNGs or whatever so they're easy to resize. This makes it pretty annoying to use lots of code samples, because they're invairably different lengths and thus should be different sizes for best readability.

The last problem is that the web views update automatically be default, which means that if you go off the 'net, they lose all of their content. Thus, you have to go through and mark them all to be updating manually.

I also use OmniGraffle to build all of my graphics, which works easily and makes pretty attractive pictures.

This isn't an awesome solution, but it seems to be working for now, anyway, and hopefully it will make it a bit easier for others to get code into their presentations.

Thu, 26 Jul 2007 | Tags: ruby, keynote, markup, highlighting, oscon, oscon07

Day 3 at OSCON

Now that UbuntuLive has officially ended, I'm at OSCON entirely. I got my talk on Racc done at FOSCON yesterday; I feel like it didn't go all that well, but I only had 10 minutes, so you can't expect all that much. My Puppet talk is in a bit over an hour; you can get the slides if you're interested, although I generally don't put much information in my slides -- I find it a lot easier to talk when the slides are just reminders, rather than the content itself. Otherwise I end up trying to read the slides instead of talking, which is no good.

I'm stoked to be finished with the talk, since I can completely relax now, it'll be just parties and the hallway track.

Wed, 25 Jul 2007 | Tags: travel, oscon, oscon07, pdx, portland

Preparing for OSCON

Looks like my insane conference travel season is starting again. I'm heading to Portland, OR for OSCON again, and my talk is on Wednesday at 5:20pm. I'll probably also do a talk or two (hopefully) at FOSCON (as you might already know), and Ubuntu LIVE starts the day after I arrive. I was planning on having some downtime in Portland before OSCON, but now I'll likely be hanging around the Ubuntu conference instead. Considering that I just submitted a talk for LinuxConf Australia, I'm hoping to put out 0.23.1 tomorrow, and I still need to write my presentation for OSCON, I'm not getting much in the way of downtime.

If any of you are going to be at OSCON, send me a note or something; we should probably organize a Puppet BoF or Meetup or something. Any volunteers?

Thu, 19 Jul 2007 | Tags: travel, oscon, oscon07, conferences, portland, pdx

Ruby Talk Proposals for FOSCON

I got an email from Thomas Lockney about FOSCON, asking if I were interested in talking about the crazy stuff I'm doing with Ruby in Puppet. Here's what I sent him:

Generating classes rather than declaring them

...which I do throughout Puppet. For instance, here is a recent set of classes I created in Puppet:

module Puppet
    newtype(:maillist) do
        @doc = "Manage email lists.  This resource type currently can only create
            and remove lists, it cannot reconfigure them."

        newparam(:name, :namevar => true) do
            desc "The name of the email list."
        end

        newparam(:description) do
            desc "The description of the mailing list."
        end
        ...

Both newtype and newparam create new classes. I've got enough of this that I've actually created a module to do this for me.

I only do this when I'm storing references to the class or module by name, and this classgen module sticks the generated class into a hash or array as necessary, sets up constants (which I don't use but Ruby does), sets attributes, calls initializations hooks, and much more.

I could fake a lot of this using the 'inherited' class method is called on the parent class, but the class when passed to that method is uninitialized (no associated code has yet been evaluated), it does not work well for subclasses of subclasses, and it doesn't make it easy to reload classes in memory.

Using Racc

...to write a real domain specific language. Everyone asks why I do this, instead of just using Ruby, but this talk would focus more on the how and what than the why. As far as I know, I've got the only two complete and open-source Racc parsers available in the wild. Puppet's parser uses abstract syntax trees and thus is a far more complicated and thus more enlightening parser, while Naginator's parser is pleasantly simple to understand and is a good example. In both cases, I've created my own lexer, since Ruby seems to be without a good lexer generator.

Puppet's configuration system

I've created a stand-alone configuration class that handles all of my Puppet run-time configuration. You define configuration settings with a section and a default:

self.setdefaults(:main,
        :trace => [false, "Whether to print stack traces on some errors"],
        :autoflush => [false, "Whether log files should always flush to disk."],
        :syslogfacility => ["daemon", "What syslog facility to use when logging to
            syslog.  Syslog has a fixed list of valid facilities, and you must
            choose one of those; you cannot just make one up."],
        :statedir => { :default => "$vardir/state",
            :mode => 01777,
            :desc => "The directory where Puppet state is stored.  Generally,
                this directory can be removed without causing harm (although it
                might result in spurious service restarts)."
        },
            ....

See Puppet's configuration for many more examples.

I keep a global configuration object at Puppet.config, set all of my items there, and then I can access them or set them throughout my system with ease.

There are some really great things about this system:

It's autodocumenting, because every configuration item has a documentation string associated with it directly (see the configuration reference for the current docs).
It's got methods for setting options within getopt, so you can trivially make any valid config option a valid getopt option, and the integration correctly handles booleans (e.g., --no-noop disables and --noop enables).
Any configuration parameter can reference another parameter, which is especially useful for directory paths. Just use a '$' before the parameter name, and the config class will expand the item when it is asked for (not when it's set, so it's using lazy evaluation).

The best thing about the class, though, is that it relies on Puppet. Any configuration item that manages files or directories can be converted into a Puppet resource and then evaluated, thus creating the directories if necessary and setting owner, group, and/or mode as necessary. Alternatively, you can generate Puppet code that you can then run manually if you prefer. You can even have Puppet create any necessary users and groups. You can use specific configuration sections (as defined during the configuration specification process), and only the files and directories from those sections will be used, so you can have hundreds of settings but only actually manage the settings you're using.

Provider localization

Puppet's portable resource types, like packages and users, are generally backed by less portable providers, like useradd, dpkg, apt, and about 22 other package providers. These providers vary in suitability for a given platform based on lots of different criteria, and Puppet still has to choose a default provider out of all of those that are suitable.

In addition, most providers are largely just wrappers around external binaries, which they call then parse their output. Because this is done so often throughout the providers, it's important that these calls to external binaries be easy to perform, along with it being easy to manage where the binaries are.

I've created a few simple methods that handle all of this functionality for me. I've got a confine class method that can confine a provider to only systems matching the specified critiria, which can include the presence of a file (usually a binary), comparing the output from Facter so you can confine providers to only Darwin systems, for instance, or arbitrary booleans, even returned from a block.

I went a bit further in handling required binaries by providing a commands method that allows you to specify required commands, either qualified or not, and the provider will then base its suitability on the presence of those commands, store the path for the command so the provider does not need to keep using a full path, and then define a method named after each command that will call out to the command, protecting the call and failing appropriately if the command is missing.

So, I can have simple provider definitions like this:

Puppet::Type.type(:package).provide :apt, :parent => :dpkg, :source => :dpkg do
    # Provide sorting functionality
    include Puppet::Util::Package

    desc "Package management via ``apt-get``."

    has_feature :versionable

    commands :aptget => "/usr/bin/apt-get"
    commands :aptcache => "/usr/bin/apt-cache"
    commands :preseed => "/usr/bin/debconf-set-selections"

    defaultfor :operatingsystem => :debian
    ...

Here I've specified three required commands, and now, for instance, I can just call aptget :install, :ruby (since the command accepts symbols as well as strings) and Puppet will call the right binary with those arguments and return the output, or throw an exception if the command fails.

You can see that I've also got a defaultfor method, which I can use to define whether a given provider is the default for some set of systems. We usually use operating system names and releases, but it can be anything that Facter knows about.

Note that this is another example of class generation -- each provider is a generated class, and you can specify as a parent class either a normal Ruby constant or the name of another provider to use as the parent class. Note that I'm also passing other options to the provider, which will get set before the associated code is evaluated.

There are plenty of other things that I think are pretty cool about Puppet, like the new provider features (search for features), which provide automatic documentation and validation, but these four are probably the coolest and most surprising things I do with Ruby in Puppet. If you need more talks, just let me know. :)