Context
You have a continuous integration (CI) system with a long execution time – 10 minutes or more. It may be the
“full build” in
Fast build/full build or the only build – it doesn’t matter.
Most of the CI cycle time is spent running automated tests, and the CI system is not sending notifications before all tests have run – regardless of intermediate failures.
The team wants to be notified about a test failure as soon as it happens. The team also wants to get enough
information to be able to start fixing any failing tests without having to wait for the CI system to complete the current cycle.
Solution
- Add failure notification capabilities into the test framework.
- Make test results easily available before the full test cycle has completed.
There are several ways to do this. I will present a couple of alternatives for each of these points:
Failure notification in the test framework
First of all – I recommend lamps and sounds as notification mechanisms.
You’ll have to understand the internals of your test framework (JUnit, Fitnesse, RSpec,
what have you) and hook into the place where test failures are recorded. Once you find out how to write and
enable a “failure hook” in your test framework, write one that sends out notifications. You have at least two options
here.
Reuse the CI system’s notification gear
Most CI systems have several notification mechanisms built in. If the CI system is open enough you can either reuse some of its notification code, or you can talk to it over some protocol. If not, tell your CI tool vendor about this pattern
so they can implement it.
Roll your own
It’s not that hard really – this is what I did.
For lamps I can recommend heyu or some other X11 command line tool/api.
For sounds I recommend win32/sound or the Java Sound API
Example (using RSpec)
(Usually Rspec is super fast, but on one of our current projects we’re using it to run a truckload of Watir).
require 'win32/sound'
class Spec::Runner::Reporter
alias old_spec_finished spec_finished
def spec_finished(name, error=nil, failure_location=nil)
if(error && !@notified)
# This is a little wrapper around Net::SSH
linux_box do |box|
box.cmd "heyu on A2" # red
box.cmd "heyu off A1" # green
end
Win32::Sound.play(File.dirname(__FILE__) + '/sounds/pacman_dies.wav')
@notified = true
end
old_spec_finished(name, error, failure_location)
end
end
Make test results easily available before the full test cycle has completed
When developers see the red lamp and hear pacman die they’re dying to fix the problem – and
they want to see the test results although the whole test suite is still running.
Most decent automated test tools write results to one or more text files on disk – little by little as tests progress.
All you need is to serve these files with a web server. That’s easy with Ruby’s built-in WEBrick.
require 'webrick'
s = WEBrick::HTTPServer.new(
:Port => 80,
:DocumentRoot => "path/to/where/test/results/are/written"
)
trap("INT") { s.shutdown }
s.start
Just make sure this server is running all the time.
Motivation
The same as the motivation in Fast build/full build
