Typo suddenly started dying on me over at Mediatemple, so I finally upgraded to Mephisto. Actually, I tried Mephisto here a while ago, but it wasn’t compatible with Ruby 1.8.2, which is all you get at Mediatemple’s Grid Server. Luckily for me Mephisto now runs on Ruby 1.8.2 again!

Some of my articles won’t read well until I do some more tweaking, and the Atom feed at Feedburner needs to be updated. I’ll fix that in a couple of weeks.

Context

You have a continuous integration (CI) system with a long execution time – 10 minutes or more. It may be the “full build” in Fast build/full build or the only build – it doesn’t matter. Most of the CI cycle time is spent running automated tests, and the CI system is not sending notifications before all tests have run – regardless of intermediate failures.

The team wants to be notified about a test failure as soon as it happens. The team also wants to get enough information to be able to start fixing any failing tests without having to wait for the CI system to complete the current cycle.

Solution

1. Add failure notification capabilities into the test framework.
2. Make test results easily available before the full test cycle has completed.

There are several ways to do this. I will present a couple of alternatives for each of these points:

Failure notification in the test framework

First of all – I recommend lamps and sounds as notification mechanisms.

You’ll have to understand the internals of your test framework (JUnit, Fitnesse, RSpec, what have you) and hook into the place where test failures are recorded. Once you find out how to write and enable a “failure hook” in your test framework, write one that sends out notifications. You have at least two options here.

Reuse the CI system’s notification gear

Most CI systems have several notification mechanisms built in. If the CI system is open enough you can either reuse some of its notification code, or you can talk to it over some protocol. If not, tell your CI tool vendor about this pattern so they can implement it.

Roll your own

It’s not that hard really – this is what I did. For lamps I can recommend heyu or some other X11 command line tool/api. For sounds I recommend win32/sound or the Java Sound API

Example (using RSpec)

(Usually Rspec is super fast, but on one of our current projects we’re using it to run a truckload of Watir).

require 'win32/sound'

class Spec::Runner::Reporter
  alias old_spec_finished spec_finished
  def spec_finished(name, error=nil, failure_location=nil)
    if(error && !@notified)
      # This is a little wrapper around Net::SSH
      linux_box do |box|
        box.cmd "heyu on A2" # red
        box.cmd "heyu off A1" # green
      end
      Win32::Sound.play(File.dirname(__FILE__) + '/sounds/pacman_dies.wav')
      @notified = true
    end
    old_spec_finished(name, error, failure_location)
  end
end

Make test results easily available before the full test cycle has completed

When developers see the red lamp and hear pacman die they’re dying to fix the problem – and they want to see the test results although the whole test suite is still running.

Most decent automated test tools write results to one or more text files on disk – little by little as tests progress. All you need is to serve these files with a web server. That’s easy with Ruby’s built-in WEBrick.

require 'webrick'
s = WEBrick::HTTPServer.new(
  :Port => 80,
  :DocumentRoot => "path/to/where/test/results/are/written" 
)
trap("INT") { s.shutdown }
s.start

Just make sure this server is running all the time.

Motivation

The same as the motivation in Fast build/full build