Aslak Hellesøy

Here is a challenge for you: If I modify an arbitrary line of code in your app, will your specs detect it and fail? They should! Enter "Heckle":http://glu.ttono.us/articles/2006/12/19/tormenting-your-tests-with-heckle. It takes up that challenge. (Sorry, Windows folks - no support here yet). Consider the following class (greeter.rb):

class Greeter
  def initialize(person = nil)
    @person = person
  end

  def greet
    @person.nil? ? "Hi there!" : "Hi #{@person}!"
  end
end

And this spec (greeter_spec.rb):

require 'greeter'

context "Greeter" do
  specify "should say Hi to person" do
    greeter = Greeter.new("Kevin")
    greeter.greet.should == "Hi Kevin!"
  end
end

If run normally it will pass. If you use "RCov":http://eigenclass.org/hiki.rb?rcov you'll get 100% coverage. Now, run it with Heckle:

aslakhellesoy$ spec greeter_spec.rb --heckle Greeter

..

Finished in 0.000831 seconds

2 specifications, 0 failures

**********************************************************************
***  Greeter#greet loaded with 3 possible mutations
**********************************************************************

3 mutations remaining...
2 mutations remaining...
1 mutations remaining...

The following mutations didn't cause test failures:

def greet
  if @person.nil? then
    "E\001N\017qf9J$}Z\vn\\uY\025*>Rdb&\036\031eI\021e\vsqQ:\020\016\rS&?w\0362N\017\037Ao.\vI\001s<\v\031\"\\\027"
  else
    "Hi #{@person}!"
  end
end

That means your specs are not up to the challenge I stated initially. Let's add another spec to plug this hole (greeter_spec.rb):

require 'greeter'

context "Greeter" do
  specify "should say Hi to person" do
    greeter = Greeter.new("Kevin")
    greeter.greet.should == "Hi Kevin!"
  end

  specify "should say Hi to nobody" do
    greeter = Greeter.new
    greeter.greet.should == "Hi there!"
  end
end

And run with Heckle again:

aslakhellesoy$ spec greeter_spec.rb --heckle Greeter

..

Finished in 0.000845 seconds

2 specifications, 0 failures

**********************************************************************
***  Greeter#greet loaded with 3 possible mutations
**********************************************************************

3 mutations remaining...
2 mutations remaining...
1 mutations remaining...
No mutants survived. Cool!

The next release of "RSpec":http://rspec.rubyforge.org/ (0.7.5) will have --heckle built-in. If you can't wait for the release, get the code from subversion:

gem install heckle --include-dependencies 

svn checkout svn://rubyforge.org/var/svn/rspec/trunk rspec
cd rspec/rspec
rake gem
gem install pkg/rspec-0.7.5.gem

Huge thanks to the "Seattle.rb":http://seattlerb.rubyforge.org/ crowd "Kevin Clark":http://glu.ttono.us/ (who turned Heckle into a real project and wrote the blog entry this one is based on), "Ryan "Zenspider" Davis":http://blog.zenspider.com/ (who wrote the initial Heckle code), "Eric Hodel":http://segment7.net/ (who wrote most of the tools that Heckle is based on - ParseTree and RubyToRuby) and finally to "Ivan Moore":http://oocode.com/ (another ex-colleague) who wrote the initial mutation testing tool, "Jester":http://jester.sourceforge.net/. I expect a bright future for Heckle. One of the missing features right now is proper HTML reports that show diffs of the original and mutated code, stats about what code is the worst etc. It won't take long...

The BDD buzzword (Behaviour-Driven Development) has caught on quite nicely. Unfortunately a lot of people seem to think it’s about syntax, which it is not.

Writing things like context and specify doesn’t mean you’re doing BDD, just like writing tests doesn’t mean you’re doing TDD.

There are other features of RSpec than its syntax that makes it a BDD library.

Expressed intent and Testdox-like reports

Writing specs (or examples or tests) that serve as documentation and at the same time reveal the intent of the code is a common practice among seasoned TDDers. BDD defines this practice as mandatory. RSpec makes the practice easy to follow (using context and specify). The point is – any syntax is fine. You can follow the same practice using FIT or JUnit (or one of the dozens of clones), a naming convention and a reporting tool like Testdox that can generate something like this:

A full stack
- should raise an exception on push
- should allow a pop

RSpec also makes it easier to express intent in the code itself, using constructs like

name.should_match /lak/

Again – this syntax makes it easier to write code that follows the BDD philosophy, but it isn’t – per se – BDD.

Just using context, specify and should_* without understanding that the key points are to express intent and to make it visible won’t bring you more than somewhat nicer syntax. BDD is a technique, not a syntax.

Mock objects

Webster’s definition of “behavior” says: anything that an organism does involving action and response to stimulation.

Code designed according to BDD (at least down at the code level) lets you verify what an object does in response to stimulation (calling a method on it). It’s not about looking at return values from that method. It’s about verifying what other objects the stimulated object communicates with.

This is where mock objects come in. Mock objects (connected to the object you poke at) tells you that. Without mock objects it is terribly hard to know what an object does – you don’t know much about its behaviour. So what a lot of people do instead is to use state-based testing That is fine too, but it ain’t BDD.

So here is an appeal to all of you creating “BDD” libraries: Does your library make it easy to communicate the overall responsibilities of the code? Can you use the library to express behaviour?

If not, chances are you’ve just created an RSpec clone, possibly promoting the BDD cargo cult

At one of the projects in BEKK we’re using Watir extensively to verify that the web application is working as expected. We’re using RSpec to drive Watir.

Every now and then the Watir specs fail (either on a developer machine or on the Continuous Integration machine). For each failure we want to know what HTML the browser had, and what the browser looked like at the time of failure. Knowing this makes the process of hunting down the cause much easier, because it gives us more context information.

Watir comes with built-in support for screenshots via Watir::ScreenCapture, but it is a really nasty hack so we didn’t want to use that. We also wanted a nice-looking HTML report for all the specs, with a screenshot of the browser (and a link to the HTML) for each of the failing specs.

About the same time as I released Win32::Screenshot, Luke Redpath contributed a much improved HTML report for RSpec. (The RSpec website has some more examples of passing and failing specs).

Getting screenshots into the RSpec HTML reports is quite easy. First, in the spec’s teardown we have to take a screenshot and store it in a file with a unique name. (I’ve simplified the code a little here to illustrate the technique – the full code is available in RSpec’s svn). Let’s start with the spec:

context "Google's search page" do
  setup do
    @browser.goto('http://www.google.com')
  end

  specify "should find rspec's home page when I search for rspec" do
    @browser.text_field(:name, "q").set("rspec")
    @browser.button(:name, "btnG").click
    @browser.contains_text("rspec.rubyforge.org").should_not_be nil # should_contain_text is RSpec sugar
  end

  specify "should find rspec's home page when I search for 'better than fudge' (will probably fail)" do
    @browser.text_field(:name, "q").set("better than fudge")
    @browser.button(:name, "btnG").click
    @browser.contains_text("rspec.rubyforge.org").should_not_be nil
  end

  specify "should not find Ali G when I search for respec" do
    @browser.text_field(:name, "q").set("respec")
    @browser.button(:name, "btnG").click
    @browser.contains_text("Ali G").should_be nil
  end

  teardown do
    # Take the screenshot 
    width, height, bmp = Win32::Screenshot.foreground
    img = Magick::Image.from_blob(bmp)[0]
    # dir and spec number are defined outside
    img_path = "#{dir}/#{spec_number}.png" 
    img.write(img_path)

    # Get the HTML from Watir and save it too
    File.open("#{dir}/#{spec_number}.html", "w") {|io| io.write(browser.html)}
  end
end

RSpec will output a HTML report when you run it with —format html, but it doesn’t link to images or the HTML source. We have to write a little extension to RSpec’s default HTML formatter:

class WebTestHtmlFormatter < Spec::Runner::Formatter::HtmlFormatter
  def extra_failure_content
    @output.puts "
" 
    @output.puts "
HTML source
" 
  end
end

Finally we have to tell RSpec (which is driving Watir via our specs) to use our WebTestHtmlFormatter when it runs the specs:

spec --require web_test_html_formatter.rb --format WebTestHtmlFormatter

The astute reader might have noticed that in the clickable screenshot above I have a Safari window and not an Internet Explorer one. The fact is, you can use this very same technique on a Mac, using Dave Hoover’s SafariWatir. The only difference is that you’ll use OS X’ built-in screencapture command to take the screenshots.

If you want to try this out, install the latest RSpec (0.7.4 or later). Then check out RSpec with subversion and peek inside vendor/web_spec. (The path within the repository may change soon).