Eden Development Staff Notes

Recently i encountered a concurrency problem of the type where there is a queue of things to do, and users press a button to be automatically assigned the next item in the queue. The bug report was that two users could get assigned the same item.

My pair programmer and i tried to reproduce the problem using two computers, but we couldn’t. We were only running one Rails instance, but we know that in the production environment there are multiple load-balanced servers pointing to one database, so we had an inkling that we’d be able to produce it using multi-threading.

To give it a test, we wrote a Rake task which we ran in two terminal windows to mimic the simultaneous access. The Rake task looked something like this:

require File.join(File.dirname(__FILE__), '..', '..', 'config', 'environment.rb')
namespace :test do
  task :take_next_for, :login do |t, args|
    user = User.find_by_login(args[:login])
    user.take_next_item
    puts user.item.inspect
  end
end

This is easily called by running:

rake test:take_next_for['ann']

We ran it for two users simultaneously and inspected the output. Sure enough they were being assigned the same item.

Since there is only one database, we knew that we could fix it with a carefully placed transaction and lock on the database. But we wanted to add a Cucumber feature so that we could be sure it was working, and to give us confidence that the bug would not come back again in the future.

  Scenario: Two users take next item simultaneously
    Given a user with login "ann"
    And a user with login "bob"
    And an available item called "Item 1"
    And an available item called "Item 2"
    When two users attempt to take the next item at the same time
    Then they should each have taken different items

Notice we can’t actually say who gets which item – it’s a race condition. We can only check that both of them have an item and that they are not the same item. We could alternatively check that both of the items have successfully been taken.

Testing this concurrency issue in Cucumber turned out to be somewhat tricky. We tried using simple Ruby threads in Cucumber, but it wasn’t properly simultaneously. I guess the single Cucumber environment still only does one thing at a time. So it was back to the Rake task.

When /^two users attempt to take the next item at the same time$/ do
  t1 = Thread.new { `RAILS_ENV=cucumber rake test:take_next_for['ann']` }
  t2 = Thread.new { `RAILS_ENV=cucumber rake test:take_next_for['bob']` }
  t1.join
  t2.join
end

We ‘join’ the two threads to make sure they’ve both finished before carrying on.

It’s slow because it loads up a whole new Rails environment for each of the Rake tasks, but that is exactly what we want to do, to mimic the concurrency of the production system.

The next problem we encountered was that Cucumber scenarios are run inside a transaction which means that a Rake task running outside of it cannot see the users and items we just created. So we had to tag the scenario as @no-txn so that they would be available externally and @clean-up-afterwards so that we could remove them from the database.

After "@clean-up-afterwards" do
  User.destroy_all
  Item.destroy_all
end

With this in place the Cucumber scenario failed as we hoped it would! Then it was simply a matter of creating a transaction from the moment we find the next item (with a database lock) until we have successfully assigned the item. This is a simplified version of what we ended up with:

class User < ActiveRecord::Base
  has_one :item
 
  def take_next_item
    transaction do
      item = Item.available.by_priority.find(:first, :lock => true)
      self.item = item
    end
  end
 
end

The Cucumber scenario passed and the problem was solved. In the live system, if two users now try to take an item at the same time, one of them has to wait a moment until the database has finished assigning to the first user so that it can assign a different item to the second user.

How would you have tested a concurrency issue like this? Are there better ways of imitating a multi-server production environment than the solution we came up with?