Aslak Hellesøy

Here is a challenge for you: If I modify an arbitrary line of code in your app, will your specs detect it and fail? They should! Enter "Heckle":http://glu.ttono.us/articles/2006/12/19/tormenting-your-tests-with-heckle. It takes up that challenge. (Sorry, Windows folks - no support here yet). Consider the following class (greeter.rb):

class Greeter
  def initialize(person = nil)
    @person = person
  end

  def greet
    @person.nil? ? "Hi there!" : "Hi #{@person}!"
  end
end

And this spec (greeter_spec.rb):

require 'greeter'

context "Greeter" do
  specify "should say Hi to person" do
    greeter = Greeter.new("Kevin")
    greeter.greet.should == "Hi Kevin!"
  end
end

If run normally it will pass. If you use "RCov":http://eigenclass.org/hiki.rb?rcov you'll get 100% coverage. Now, run it with Heckle:

aslakhellesoy$ spec greeter_spec.rb --heckle Greeter

..

Finished in 0.000831 seconds

2 specifications, 0 failures

**********************************************************************
***  Greeter#greet loaded with 3 possible mutations
**********************************************************************

3 mutations remaining...
2 mutations remaining...
1 mutations remaining...

The following mutations didn't cause test failures:

def greet
  if @person.nil? then
    "E\001N\017qf9J$}Z\vn\\uY\025*>Rdb&\036\031eI\021e\vsqQ:\020\016\rS&?w\0362N\017\037Ao.\vI\001s<\v\031\"\\\027"
  else
    "Hi #{@person}!"
  end
end

That means your specs are not up to the challenge I stated initially. Let's add another spec to plug this hole (greeter_spec.rb):

require 'greeter'

context "Greeter" do
  specify "should say Hi to person" do
    greeter = Greeter.new("Kevin")
    greeter.greet.should == "Hi Kevin!"
  end

  specify "should say Hi to nobody" do
    greeter = Greeter.new
    greeter.greet.should == "Hi there!"
  end
end

And run with Heckle again:

aslakhellesoy$ spec greeter_spec.rb --heckle Greeter

..

Finished in 0.000845 seconds

2 specifications, 0 failures

**********************************************************************
***  Greeter#greet loaded with 3 possible mutations
**********************************************************************

3 mutations remaining...
2 mutations remaining...
1 mutations remaining...
No mutants survived. Cool!

The next release of "RSpec":http://rspec.rubyforge.org/ (0.7.5) will have --heckle built-in. If you can't wait for the release, get the code from subversion:

gem install heckle --include-dependencies 

svn checkout svn://rubyforge.org/var/svn/rspec/trunk rspec
cd rspec/rspec
rake gem
gem install pkg/rspec-0.7.5.gem

Huge thanks to the "Seattle.rb":http://seattlerb.rubyforge.org/ crowd "Kevin Clark":http://glu.ttono.us/ (who turned Heckle into a real project and wrote the blog entry this one is based on), "Ryan "Zenspider" Davis":http://blog.zenspider.com/ (who wrote the initial Heckle code), "Eric Hodel":http://segment7.net/ (who wrote most of the tools that Heckle is based on - ParseTree and RubyToRuby) and finally to "Ivan Moore":http://oocode.com/ (another ex-colleague) who wrote the initial mutation testing tool, "Jester":http://jester.sourceforge.net/. I expect a bright future for Heckle. One of the missing features right now is proper HTML reports that show diffs of the original and mutated code, stats about what code is the worst etc. It won't take long...