Checking whether classes have been evaluated
Another snippet I used for testing.
Bug #1165 requests the ability to check for whether a class exists, but we've already got that:
class yay {
notify { "This is a test": }
}
class other {
if defined(Class[yay]) {
notify { "Defined works just fine": }
}
}
include yay, other
Prints:
notice: This is a test notice: Defined works just fine
Works for me. :)
Thu, 03 Apr 2008 | Tags: puppet, bug, class, snippet
John Willis on Open Source
John Willis just made a great post on the Open Source vs. Commercial software for enterprise management:
Now contrast this with the infamous iLike/Puppet story. Last year iLike.com added a new Facebook application to their service and they gained 35k new users in 24 hours and over 700k new users in just a couple of days. Within a three week period they went from 3 million users to 6.7 million users. Less than a year later, they now have 23 million users. Surely, they must have used an industrial strength multi-million dollar provisioning system - 'not'. Their provisioning system was FREE. With the help of a consulting company called HJK they used an open source product called Puppet to manage their server growth. HJK boasts that with a really short services engagement, one or two weeks, they can achieve 10 minute psychical system bare metal installs. On Xen images they can provision a system in two minutes. In other words, for about 15k in services and another 15k for a Puppet maintenance contract, HJK can provide a solution that the Big Four charges over a million plus and they can deliver the solution in two weeks instead of 3 months.
Fri, 21 Mar 2008 | Tags: opensource, puppet, opennms, hjk
ralsh is Awesome
So, I'm testing ticket #1099, and I run this snippet of code:
user { testing: ensure => present, home => "/var/tmp" }
Then this one:
user { testing: ensure => present, home => "/tmp" }
Sure enough, the home directory changes (although it doesn't actually move the directory, thankfully), so I clearly didn't do due diligence on accepting the bug information from my client. Now I need to remove the user. Sure, I can modify and reexecute the file, but why should I, when I can just do this:
luke@culain(0) $ sudo puppet/bin/ralsh user testing ensure=absent
notice: /User[testing]/ensure: removed
user { 'testing':
uid => 'absent',
home => 'absent',
password => 'absent',
gid => 'absent',
groups => 'absent',
comment => 'absent',
ensure => 'absent',
shell => 'absent'
}
luke@culain(0) $
The extra output of the user is kinda silly, and really only matters when printing rather than modifying users, but still, I use this all the time, and I'm quite fond of it and its silly name.
Tue, 26 Feb 2008 | Tags: tools, ralsh, puppet
On Ruby Interviews James Turnbull and Reviews the Puppet Book
Pat of On Ruby has posted a brief review of James Turnbull's Puppet book, Pulling the Strings with Puppet:
This book is filled with helpful code samples and pointers to external resources that look very useful. It's well written and easy to understand. As good a tool as Puppet looks to be, this looks like an equally good book to get you going. If you're doing configuration management for anything more than a box or two, run, don't walk, and pick up your copy of Pulling Strings with Puppet.
Pat also posted an interview with James:
Who gets the credit (Or is it blame?) for the title of your book, 'Pulling Strings with Puppet'
That'd be my editor and the marketing guys at Apress. Do you know how excited marketing people are when a product allows amusing alliteration and puns' :) But I like it -- it's both kitsch and catchy.
Looks like the book is really helping with the visibility of Puppet, which is great, and people seem to even like the book so far. :)
Wed, 13 Feb 2008 | Tags: puppet, book, interview, jamesturnbull, review, ruby, onruby
Example code: Testing relationships to base classes
I think I promised a while ago to start posting the example code I use for tests.
Here's the code I used to reproduce #1030:
class base {
notify { "This is the base class": }
}
class sub inherits base {
notify { "This is the sub class": }
}
include sub
notify { "This is the main class": require => Class[base] }
Sure enough, I get this exception:
luke@phage(0) $ test.pp Could not find dependency Class[base] for Notify[This is the main class] at /Users/luke/bin/test.pp:41 luke@phage(1) $
Turns out the problem is in lib/puppet/parser/compile.rb; or rather, the problem is that the code is in this file, instead of in lib/puppet/parser/ast/hostclass.rb. The Compile class creates a resource for every class it evaluates, which is how these relationships work, but it's that AST class that actually knows when base classes are evaluated by a subclass, so it needs to be creating these resources.
Thu, 07 Feb 2008 | Tags: puppet, example, code
LCA Video is Posted
I'm finally back from LCA (and San Francisco and Seattle, although I haven't posted my LCA pictures yet). It looks like they've already posted the video (search for "luke", then click the ogg link).
As always, there were a lot of great presentations and even more great attendees. I'm not yet recovered enough to give out the link love, and really, I didn't schmooze as much as I should have -- a little too much time in the cocktail bars and trying to find a spicy meal, I think.
Hopefully the video turned out well; I think I kind of stuttered for the first ten minutes or so but then found my groove. I originally intended to spend the talk on directly using Puppet, but after some client work in Seattle I realized it made more sense to really focus on the idea of the Resource Abstraction Layer (RAL), which seemed to go over well. The previous presenter went quite long, despite my best attempts to rudely let him know he should leave, so I had to skip quite a few slides.
Tue, 05 Feb 2008 | Tags: travel, lca, lca08, mel8ourne, puppet, presentation, video, download
Closing the first DTrace loop
I ended up spending what time I had for a couple of days at LCA optimizing Puppet's lexer. As I mentioned in my first DTrace post, I'm mostly just trying to use existing scripts for now and worry about understanding the darn thing later.
It turns out that the Swiss Army Knife of available scripts for Ruby is the rb_calltime.d script in the DTraceToolkit. This one gives us pretty much everything we might want to know in a first pass of debugging a Ruby program, including (most importantly for me) the inclusive and exclusive elapsed times for each method in the system (DTrace insists upon calling them functions, but it's Ruby, so we know better).
Unfortunately, I'd already hacked out most of the optimization before I discovered this script, and it doesn't seem to want to run at the moment, so all I can do is show the current data. Here are the methods that take the most time inclusively (meaning that it counts the time from entry to exit, and thus just tells us where the time is being spent in the overall program, not where the actual problems might lie):
lexer.rb func Puppet::Parser::Lexer::munge_token 12062622 lexer.rb func Puppet::Parser::Lexer::TokenList::lookup 12410291 branch.rb func Puppet::Parser::AST::Branch::initialize 19084001 methodhelper.rb func Hash::each 19144097 methodhelper.rb func Object::set_options 20712067 ast.rb func Puppet::Parser::AST::initialize 22431107 lexer.rb func Array::each 23868735 parser_support.rb func Class::new 28520157 lexer.rb func Puppet::Parser::Lexer::find_string_token 29610463 lexer.rb func Puppet::Parser::Lexer::find_regex_token 29844180 parser_support.rb func Puppet::Parser::Parser::ast 36132924 lexer.rb func Puppet::Parser::Lexer::find_token 44581509 lexer.rb func Object::catch 46187059
And here are the exclusive times (i.e., only counting the time spent in each method, not the time between entry and exit):
ast.rb func Puppet::Parser::AST::initialize 1719039 lexer.rb func Puppet::Parser::Lexer::Token::convert 1920189 branch.rb func Puppet::Parser::AST::Branch::initialize 2062436 lexer.rb func Object::catch 2262789 lexer.rb func StringScanner::match? 2471109 parser_support.rb func Class::new 3112051 lexer.rb func Puppet::Parser::Lexer::skip 3282333 lexer.rb func Puppet::Parser::Lexer::find_token 4930508 lexer.rb func Puppet::Parser::Lexer::munge_token 5232851 lexer.rb func Puppet::Parser::Lexer::find_regex_token 5572052 lexer.rb func Puppet::Parser::Lexer::TokenList::lookup 6659161 parser_support.rb func Puppet::Parser::Parser::ast 7144763 lexer.rb func Hash::[] 7179650 methodhelper.rb func Hash::each 14954253 lexer.rb func Puppet::Parser::Lexer::find_string_token 17045065 lexer.rb func Array::each 20768233 - total - 132806477
Given this data, we're spending about 1/7th of the total parsing time just in the find_string_token method, which currently looks like this:
def find_string_token
matched_token = value = nil
# We know our longest string token is three chars, so try each size in turn
# until we either match or run out of chars. This way our worst-case is three
# tries, where it is otherwise the number of string chars we have. Also,
# the lookups are optimized hash lookups, instead of regex scans.
[3, 2, 1].each do |i|
str = @scanner.peek(i)
if matched_token = TOKENS.lookup(str)
value = @scanner.scan(matched_token.regex)
break
end
end
return matched_token, value
end
The method is responsible for determining if the next token is a simple string-based token. I've optimized it by taking the fact that the longest string-based tokens are three characters (the <<| and |>> tokens), so I look for three character matches, then two, then one. If I don't get a match by then, then we don't have a match. I could probably optimize further, since these three character tokens are pretty darn rare, especially compared to the one character tokens, but I'd need to hard-code a lot more knowledge about the token list, and really, this iteration should be delimited by an automatic determination of the longest token, rather than hard-coding it.
This really isn't a very good write-up of what I did with DTrace or how it was helpful, other than showing the interesting differences between the exclusive and inclusive data, and letting you know that the rb_calltimes.d script is the one to start with, but hey, that's more than I could find when I started looking, so hopefully this will get you somewhere.
I expect to continue spending more time using DTrace for optimizations, and I'll hopefully start uploading my data so I don't have to worry about taking these snapshots. Graphs would sure be nice....
Tue, 05 Feb 2008 | Tags: tools, dtrace, puppet, performance, optimization, profiling, profiler
Podcast with Hyperic
I know it's been a long time since I posted, and there's lots to post about, but it's been a very long month with little time.
Until I get my act together (which likely won't happen until I'm in Melbourne for LCA), here's at least a snippet.
I did a podcast with John Mark Walker of Hyperic a couple of weeks ago when I was in San Francisco for the Velocity summit.
I actually haven't had a chance to listen to it yet, but apparently I do some smack talk or something. Give it a listen.
Mon, 21 Jan 2008 | Tags: industry, podcast, puppet, hyperic, interview
0.24.1 is out
As promised, 0.24.1 is out. Here's the changelog:
- Updated vim filetype detection. (#900 and #963)
- Default resources like schedules no longer conflict with managed resources. (#965)
- Removing the ability to disable http keep-alive, since it didn't really work anyway and it should no longer be necessary.
- Refactored http keep-alive so it actually works again. This should be sufficient enough that we no longer need the ability to disable keep-alive. There is now a central module responsible for managing HTTP instances, along with all certificates in those instances.
- Fixed a backward compatibility issue when running 0.23.x clients against 0.24.0 servers -- relationships would consistently not work. (#967)
- Closing existing http connections when opening a new one, and closing all connections after each run. (#961)
- Removed warning about deprecated explicit plugins mounts.
The main fixes are that http keep-alive works now (meaning you should only have one connection open to your server, not tens or hundreds), the backward compatibility problem is fixed, and default schedules work again.
Fri, 21 Dec 2007 | Tags: puppet, release
Puppet on Ohloh
Looks like someone added Puppet to Ohloh. I hadn't heard of the site before, but it looks pretty interesting. I dunno how well it will succeed in its goal, but I really like the idea of there being a site that draws in the conversations about a given project and shows how different projects connect.
Fri, 21 Dec 2007 | Tags: puppet, opensource, project, social
Tags
