My pair programmer and i tried to reproduce the problem using two computers, but we couldn’t. We were only running one Rails instance, but we know that in the production environment there are multiple load-balanced servers pointing to one database, so we had an inkling that we’d be able to produce it using multi-threading.

To give it a test, we wrote a Rake task which we ran in two terminal windows to mimic the simultaneous access. The Rake task looked something like this:

require File.join(File.dirname(__FILE__), '..', '..', 'config', 'environment.rb')
namespace :test do
  task :take_next_for, :login do |t, args|
    user = User.find_by_login(args[:login])
    user.take_next_item
    puts user.item.inspect
  end
end

This is easily called by running:

rake test:take_next_for['ann']

We ran it for two users simultaneously and inspected the output. Sure enough they were being assigned the same item.

Since there is only one database, we knew that we could fix it with a carefully placed transaction and lock on the database. But we wanted to add a Cucumber feature so that we could be sure it was working, and to give us confidence that the bug would not come back again in the future.

  Scenario: Two users take next item simultaneously
    Given a user with login "ann"
    And a user with login "bob"
    And an available item called "Item 1"
    And an available item called "Item 2"
    When two users attempt to take the next item at the same time
    Then they should each have taken different items

Notice we can’t actually say who gets which item – it’s a race condition. We can only check that both of them have an item and that they are not the same item. We could alternatively check that both of the items have successfully been taken.

Testing this concurrency issue in Cucumber turned out to be somewhat tricky. We tried using simple Ruby threads in Cucumber, but it wasn’t properly simultaneously. I guess the single Cucumber environment still only does one thing at a time. So it was back to the Rake task.

When /^two users attempt to take the next item at the same time$/ do
  t1 = Thread.new { `RAILS_ENV=cucumber rake test:take_next_for['ann']` }
  t2 = Thread.new { `RAILS_ENV=cucumber rake test:take_next_for['bob']` }
  t1.join
  t2.join
end

We ‘join’ the two threads to make sure they’ve both finished before carrying on.

It’s slow because it loads up a whole new Rails environment for each of the Rake tasks, but that is exactly what we want to do, to mimic the concurrency of the production system.

The next problem we encountered was that Cucumber scenarios are run inside a transaction which means that a Rake task running outside of it cannot see the users and items we just created. So we had to tag the scenario as @no-txn so that they would be available externally and @clean-up-afterwards so that we could remove them from the database.

After "@clean-up-afterwards" do
  User.destroy_all
  Item.destroy_all
end

With this in place the Cucumber scenario failed as we hoped it would! Then it was simply a matter of creating a transaction from the moment we find the next item (with a database lock) until we have successfully assigned the item. This is a simplified version of what we ended up with:

class User < ActiveRecord::Base
  has_one :item
 
  def take_next_item
    transaction do
      item = Item.available.by_priority.find(:first, :lock => true)
      self.item = item
    end
  end
 
end

The Cucumber scenario passed and the problem was solved. In the live system, if two users now try to take an item at the same time, one of them has to wait a moment until the database has finished assigning to the first user so that it can assign a different item to the second user.

How would you have tested a concurrency issue like this? Are there better ways of imitating a multi-server production environment than the solution we came up with?

See my Apprenticeship task presentation on youtube.

Last time i wrote, i was just starting to work on Pages and a PageFactory to find them or create them as necessary. Since then i’ve added a persistence strategy which saves wiki pages to the filesystem. I pass the desired directory to the strategy so that test pages can be kept separate from real pages. This was necessary as i wanted to clear out the test directory every time after use and i didn’t want to accidentally delete real pages.

After that little burst of progress i stalled for a long time as i knew i needed to start writing the web interface and i didn’t know where to begin. James gave me a good idea which was to set up a simple Webserver class that would wrap around the web server of my choice. I have chosen WEBrick for the time being, but it shouldn’t be too difficult to swap in another one later, probably with another strategy pattern. James suggested i start by visiting a page and looking for the text ‘Hello world’. I thought this was a good idea and so i started finding out about WEBrick servlets.

With that start i was suddenly able to leap ahead, testing and implementing everything i need so that my wiki is now able to create, show and edit pages through the web interface. I feel as if i am very close now. Possibly the only thing left to do is add a way to link between pages and then i can truly call it a wiki.

Test driving the development of this whole project has been extremely educational. I know i started off getting too involved in writing a test framework, so Enrique pulled me back and made me think of the simplest thing that could possibly work. I began with one method: assert_equal. That was enough in the beginning. As time went on i added two more methods: assert_true and assert_contains. When necessary i added useful debugging information to report failures. At the appropriate time i refactored the testing framework into a module that could be included into multiple test classes.

I know i also started thinking about the web server too early. I started off trying to work with ERB and templates, as i am familiar with Rails, and in the end all i needed was a few servlets. As Enrique made me realise, the most important part of a wiki is not that it runs on a web server. The most important thing is content. Pages, with content, saved to a file system. Having focussed on that and got a real solid, well tested mechanism for saving and retrieving pages, i knew that the WEBrick servlets could rely on the underlying framework, and when the time came, it did not let me down.