Jay and I converge on testing
Those four people who have been reading this blog for a while know I've been struggling to think and program like Jay Fields. In particular, he seems to have presented a few rules in the past that don't like to be used together:
- Inline setup code: You apparently should not use before or setup blocks with your code; instead, you should have all of the setup code in every method.
- Mocks instead of Stubs: Here he claims that it's always better to use expects instead of stubs.
- One expectation per test: With only one expectation, you're only testing one feature at a time, and you're guaranteed that each behaviour is tested separately.
Now, let's do a simple combinatorial exercise, and put these three rules together:
- You should have all related code in your test
- You should only use Mocks (expects), not Stubs (stubs)
- You should only have on expectation (expects) per test
It's pretty clear that, like the old saw about programming ("All programs can be reduced to one line of code with a bug in it"), Jay is pointing us toward tests that can largely only be one line of code. Yeah, I know sometimes setup methods don't involve any mocking, but often they do.
And, since your tests can only be one line of code, they can't test very much, which means that all of your methods need to also be one line of code, else they aren't testable. (Yes, I'm being a touch extreme here, but that is where the arrow is pointed, anyway.)
You can see how this would kinda drive me bonkers. Some local dev friends have been trying to help me see the error of my ways, mostly so my code would stop looking like such crap; I've learned a helluva lot in the last 8 months or so, and most of it has actually made my code look more like Jay would recommend. I will say it's blindingly obvious Jay is doing internal development at enterprises, rather than developing as part of a consistent team producing software that is distributed to the wider world.
But one can only go so far, and the three rules above, in combination, are just way too far. I've often kind of sputtered expasperatedly at Jay's posts, especially his announcement of his new testing tool, expectations. Again, I can kind of see where he's going with that, but you've got another thing coming if you think I'm using it, especially given how happy (at least, relative to test/unit) I am with RSpec.
Also, I think it's just stupid having all setup code inline. DRY ("don't repeat yourself") is just as true in your test code as anywhere else, and having a maintainable test code base is, IMO, more important than having your normal code base be maintainable, because tests are kind of unnecessary. If you have good, readable, maintainable tests, then people who contribute will also contribute tests. If your tests are all 50 lines long and have lots of repetition, then 1) you've got 5x the amount of code you should, which is wicked expensive, and 2) you've got so much code no one will look at it. Yay, never getting patches with tests in them. My favorite example of this is Steve Yegge's rant Code's Worst Enemy; he describes his 500k line Java project with no tests, which is a lot of code but much less code than if it had tests. I've experienced in Puppet that test code seems to be much harder to maintain that normal code (although maybe it's just own crap test code, not normal test code), and having 5x test code than normal code would make me just quit writing unit tests entirely.
So, I am absolutely overjoyed to announce that Jay has changed one of his rules: He now recommends stubs over mocks. This is clearly just for setup code and such, but it's a big step. He even goes into using stub_everything, which I find is the only way to build tests that aren't fragile. For instance, say you start with this code:
class MyClass
def go
start()
finish()
end
end
describe MyClass do
before
@me = MyClass.new
@me.stubs(:start)
@me.stubs(:finish)
end
it "should start when going" do
@me.expects(:start)
@me.go
end
it "should finish when going" do
@me.expects(:finish)
@me.go
end
end
Now you find you need a validation method, so you add this test:
it "should validate when going" do
@me.expects(:validate)
@me.go
end
Update: Fixed code to actually call @me.go in the validate test.
Oops. Now your single test passes, but your two old tests break, because you were only stubbing start and finish, instead of using stub_everything. Your setup code needs to be modified to take this new call into account (or, if you're Jay, you need to modify every test in your suite; yay). This comes up constantly. If you specifically mock or stub methods during setup, then you are almost guaranteed to have cascading failures when you expand your code.
Anyway, the point is, if I tried to follow Jay's rules, then the above trivial change -- I add one line of code to a very simple method -- would result in me adding a test for that line, plus at least one line of code in every other test in that suite. Instead, if I use stub_everything, then I add my new test and I'm done. (Well, kind of; notice I'm not actually testing the order of the method calls, which is actually pretty tough.)
My recommendation is to read Jay, since he's clearly thinking and talking about aspects of testing that not many others are, but read him with a skeptical eye, and be willing to say "That's just nuts!" and write your own relatively abstracted test code. And if you're working with people who can't think to look at their setup code when a test fails, then you need to find a different job.
Tue, 06 May 2008 | Tags: ruby, testing, programming, bdd, tdd, jayfields
A little more RSpec
I've been using RSpec more thoroughly recently, so I figured it's time produce a post covering some of the things I've learned.
Particularly, thanks to much forcing by Rick, I have now come to realize that RSpec involves writing two very coupled and very important documents: The spec, which describes how your code should behave, and the tests, which are attached to the spec and theoretically prove that your code does behave this way.
This was a big step in realization for me, because I had always thought of my tests as just that: tests. That is, their job was to make sure my code was behaving as I thought it should be behaving. When working with Rick the other day, though, he kept saying my spec was bad and I kept wanting to shout that I understood my tests were bad, let's just fix them.
After enough prodding, though, and some judicous using of RSpec's formatting, I came to see that he views them as separate things, and that I should too.
With RSpec, you can obviously run your tests:
luke@phage(0) $ bin/spec unit/indirector/indirector.rb ......... Finished in 0.010605 seconds 9 examples, 0 failures luke@phage(0) $
But you can also run them and produce a document that looks suspiciously like a spec:
luke@phage(0) $ bin/spec unit/indirector/indirector.rb --format s Puppet::Indirector when available to a model - should provide a way for the model to register an indirection under a name Puppet::Indirector when registering an indirection - should require a name when registering a model - should create an indirection instance to manage each indirecting model - should not allow a model to register under multiple names - should make the indirection available via an accessor Puppet::Indirector when redirecting a model - should give model the ability to lookup a model instance by letting the indirection perform the lookup - should give model the ability to remove model instances from a terminus by letting the indirection remove the instance - should give model the ability to search for model instances by letting the indirection find the matching instances - should give model the ability to store a model instance by letting the indirection store the instance Finished in 0.014107 seconds 9 examples, 0 failures luke@phage(0) $
Note that it's still running the tests here (as shown by the lack of failures at the bottom), it's just producing output differently.
I was surprised to find Rick wanting to read this output again and again, completely ignoring any associated test code. It finally hit me, of course, that if we aren't doing a good job of describing the expected behaviour, then how can we do a good job of testing it?
When we first started working through some code recently, he kept niggling about some code that wasn't in the spec, which kinda miffed me because it was clearly being tested. When I finally realized what he meant about the difference between the spec and the tests, no matter how tightly coupled they are, I realized why it mattered so much to him: There was behaviour in my code that I essentially hadn't documented, even if I was confident I was testing that it was working.
Certainly an interesting realization.
Sun, 23 Sep 2007 | Tags: ruby, rspec, puppet, development, programming, testing, bdd
Tags
