CruiseControl.rb, by ThoughtWorks, is an elegant Continuous Integration dashboard for Ruby projects. This article shows how to call the RoR command “rake stats” once per build, capture the results, and chart them to track trends over time.

Our goal is this chart:

If you think of more trends to display, our architecture will make them easy to add. We start by counting the lines of tests & code, and calculating their ratio. These signals occupy different orders of magnitude, so a logarithmic scale reveals their common slopes. We use Gnuplot to make such charts easy; this paltry output is only the beginning of Gnuplot’s abilities.

CruiseControl.rb

This exemplary Ruby on Rails application provides these important files and folders:

app/controllers/projects_controller.rb	we will add a statistics view here
app/models/project.rb	we will add supporting methods here
builder_plugins/installed/statistician.rb	when the builder encountered significant events, such as successful builds, it calls registered methods in here. We add a new statistician.rb class to call rake stats and record its results
lib/	you might plop gnuplot.rb in here; otherwise, get it with gem install gnuplot
projects/your_application/statistics.yaml	we add a database of your application’s statistics
public/images/charts/your_application.svg	to view a chart, we render its SVG file into here

builder_plugins/installed/statistician.rb

Add this file there, and bounce your server:

require 'fileutils'
require 'code_statistics'

class Statistician

  def initialize(project)
    @project = project
  end

  def build_finished(build)
    run_in_here = @project.path + '/work'
    FileUtils.cd(run_in_here){  append_stats(@project)  }
  end
end

STATS_FOLDERS = [
  %w(Controllers        app/controllers),
  %w(Helpers            app/helpers), 
  %w(Models             app/models),
  %w(Libraries          lib/),
  %w(APIs               app/apis),
  %w(Components         components),
  %w(Integration\ tests test/integration),
  %w(Functional\ tests  test/functional),
  %w(Unit\ tests        test/unit)
].freeze

class CodeStatistics;  attr_reader :statistics;  end

def collect_stats
  #  code callously ripped from statistics.rake !
  folders = STATS_FOLDERS.select{|name, dir| File.directory?(dir) }
  cs      = CodeStatistics.new(*folders)
  statz   = cs.statistics
  tyme    = Time.now.to_i
  yaml    = { "build_#{ tyme }" => statz }.to_yaml
  return yaml.sub(/^---/, '')  #  abrogate that pesky document marker!
end

def append_stats(project)
  yaml = collect_stats
  plop = project.path + '/statistics.yaml'
  File.open(plop, 'a+'){|f| f.write(yaml) }
end

Project.plugin :statistician

That code registers a callback, to call when any project ends. It uses code_statistics.rb to collects line counts for your project, then it appends them to the end of projects/your_application/statistics.yaml. Install that module, trigger a few builds, and examine that YAML file to see your trends in their raw format.

project.rb

The callback indexed the database using a timestamp - the number of seconds since 1970. To read the database, we pull everything into an Array and sort it by that timestamp.

Add this to app/models/project.rb:

require 'fileutils'
require 'gnuplot'

class Project

  def get_stats
    statistics = File.read(path + '/statistics.yaml')
    statistics = YAML::load(statistics)
    statistics = statistics.map{|k,v| [k.sub('build_', '').to_i, v] }
    return statistics.sort
  end
...

That method took out the prefix “build_“, and converted the timestamp back into an integer.

gnu_plot_stats

Now that we have an array of statistics, we need to convert them into a chart. You can add the following code anywhere that projects_controller.rb can see; I happened to add mine to the end of app/models/project.rb:

require 'gnuplot'

def timefmt;  '%Y/%d/%m-%H:%M';  end

def fetch_codelines(stat, fields)
  return stat.values_at(*fields).map{|values| values['codelines'] }.sum 
end

def ftime(timestamp)
  Time.at(timestamp).strftime(timefmt)
end

ALL_TESTS = %w(Unit\ tests Functional\ tests)
ALL_CODE = %w(Components Libraries Helpers Controllers Models)

def gnu_plot_stats(project, signals = {})
  signals = { 'Code' => ALL_CODE, 
              'Test' => ALL_TESTS }.merge(signals)
  name = @project.name
  path = "#{RAILS_ROOT}/public/images/charts"
  Dir.mkdir(path) unless File.directory?(path)
  output_file = "#{path}/#{name}.svg"
    #  drastically prevent false positives in manual tests
  File.unlink(output_file) if File.exist?(output_file)

  Gnuplot.open do |gp|
    Gnuplot::Plot.new( gp ) do |plot|
        #  decorate the chart
      plot.xdata 'time'
      plot.key 'outside title "   Code Lines   "'
      plot.grid 'ytics'
      plot.timefmt timefmt.inspect # for quote marks
      plot.term 'svg'
      plot.output output_file
      plot.title name
      plot.logscale 'y'
      stats = project.get_stats
        #  set the time range
      timestamps = stats.map(&:first)
      statistics = stats.map(&:last)
      mini = timestamps.first - 60
      maxi = timestamps.last  + 60
      plot.xrange "['#{ ftime(mini) }':'#{ ftime(maxi) }']"
      times = timestamps.map{|v|  ftime(v.to_i)  }
      maxi = 0

        #  collect each signal and add its plot line

      plot.data = signals.keys.sort.map do |legend|
        fields = signals[legend]
        
        values = statistics.map{|stat|
                    if fields.respond_to? :call
                      fields.call(stat)
                    else
                      fetch_codelines(stat, fields)
                    end
                  }
        maxi = [maxi, *values].max
        
        Gnuplot::DataSet.new( [times, values] ) { |ds|
          ds.with = "linespoints"
          ds.title = legend
          ds.using = '1:2'
          ds.linewidth = 4
        }
      end

        #  set the chart height
      next_order_of_magnitude = 10 ** (Math.log10(maxi) + 1.1).to_i
      plot.set 'yrange', "[0.01:#{next_order_of_magnitude}]"
    end
  end  #  this 'end' writes the output SVG
  
    #  Gnuplot can't beautify the SVG enough, so we tweak these things
  svg  = File.read(output_file)
  doc  = REXML::Document.new(svg)
  node = REXML::XPath.first(doc, "//text[ '#{name}' = . ]")
  return unless node
  node.attributes['style'] = 'fill: #507ec0; font-family:Arial,sans; font-size: 0.8cm;'
  node = REXML::XPath.first(doc, '/svg')
  node.attributes['height'] = nil
  node.attributes['width'] = nil
    #  and finally write the SVG again
  File.open(output_file, 'w'){|f| doc.write(f) }
end

The top-level method, gnu_plot_stats, can call with more signal commands, as an option hash. A signal’s key is its legend, and its value is either an array of signals to sum, or a lambda to call, for more advanced processing. That’s how we will find the ratio of test to code lines.

project_controller.rb

To get this chart into our users’ faces, we need a new action in a controller. Providing links to this action is left as an exercise for the reader.

Add this statistics action to app/controllers/projects_controller.rb:

class ProjectsController < ApplicationController
  layout 'default'

  def statistics
    @project = Projects.find(params[:id])
    name = @project.name
    
    gnu_plot_stats @project,
      'Test:Code' => lambda{|stats|
        test = fetch_codelines(stats, ALL_TESTS)
        code = fetch_codelines(stats, ALL_CODE)
        return test.to_f / code.to_f
        }
    render :layout => true,
           :inline => %(
        <embed src="/images/charts/#{name}.svg" 
              type='image/svg+xml' width='100%' height='400'
              style='background: url(/images/big_top_gradient.png);' />
      )
  end
...

That action generates a new SVG file, then renders it in an embedded SVG viewer.

To collect the ratio of test to production code, we pass in a lambda that calculates this. Note that Gnuplot has a “math nanny” feature, which rejects logarithmic scales with input data at 0. A finished project would find a way to clip these points.

To Do

This simple project suggests many cleanups and extensions. We might…

• link the dashboard page to our URIs, such as http://localhost:3333/projects/statistics/lemmings
• productize the code as a plugin
• configure different projects with different signals
• annotate a chart with significant integration messages
• featurize the chart with time ranges
• stack several charts, with different signals in each one
• roll-up all the statistics into a team’s summaries.

The ultimate goal is a Big Visible Chart, or Information Radiator, for CruiseControl.rb projects.