Coding is like gardening...

Introducing Jabbersonic

I’ve been working on improving our build process recently and have been inspired by several articles on Extreme Feedback devices.

Specifically, this blog post at Pragmatic Automation caught my attention: I really liked the idea of being able to listen to complex systems such as a project management flow. Wouldn’t it be cool if you could control a varied soundscape with, say, Jabber messages?

So after four hours of hacking late last night, Jabbersonic was born.

It’s pretty easy to get started. If you’re on a mac with iLife installed it should work out of the box:

sudo gem install gosu xmpp4r-simple
git clone git://github.com/ChrisMDP/jabbersonic.git
cd jabbersonic
bin/server  

This will kick off a gosu app which will listen for Jabber messages sent to the provided account and play different sounds corresponding to different events. An example of usage:

Jabbersonic in Action

Jabbersonic in Action

It’s currently set up for a continuous integration system – it should be trivial to make the Hudson Jabber plugin talk to it, and write a tiny API app for Hoptoad for example. It’s not confined to project management though: there’s a simple configuration file which allows you to make it work for pretty much any complex system you might want to model.

For more information, see the README. Opinion is divided here as to whether a soundscape is actually useful for project management or just an irritation: I guess it will mostly depend on the sound design, but we’ve yet to try it out properly.

Let me know if you try it out, or use it for anything useful.

Four things (of many) we learnt from Liz Keogh

Liz KeoghWe’ve had the great pleasure of having Liz Keogh doing some agile coaching with us for the last two days.

I invited Liz after meeting her at the BDD workshop I ran last week. We were very lucky that she was free and available at short notice. I felt we had a pretty good agile system set up here, but we’d not scaled it to a larger team (6+) before and we were coming up against issues (such as the best ways of estimating) which we felt Liz could help us with.

It’s been enormously beneficial to have Liz with us. Much of our practice were validated: pair programming, co-location, BDD, outside-in, continuous integration. These are core tenets of our process which have been proven in many companies. Instead of changing our basic practice, Liz has helped us to the next level through much conversation and discussion.

Here’s a few snippets of what we’ve learned and how we’re tweaking our process to give our clients the maximum number of features with the shortest possible lead time.

Use Feature Injection to force the value question

One small change with big ramifications is the concept of Feature Injection.

In brief, this means changing the way you write stories so that instead of having:

As a 
I want 
so that 

You have:

In order to 

will need
.

Rather than just a way of specifying technical stories, we realised that by putting stakeholder value first, we are forcing the question of what’s actually necessary about this feature that causes us to do it in the first place. This ties our stories very much to stakeholder value, which ensures that everyone knows exactly what value is being provided when they’re working on a story.

Read more about Feature Injection here.

Keep completed stories visible

Once our stories are clearly about stakeholder value, it’s important for everyone to see exactly which features currently exist, preferably broken down by stakeholder. We’re working on ways of getting the features list that we’ve already completed more visible to our clients. Once everyone can see what’s been delivered so far, it’s much clearer what should be worked on next.

One way of doing this might be a small Rack app which handles /admin/features and returns all the features grouped by tag… does anyone know of a Rails plugin that might do something similar?

Ensure you can always release

If we’re truly doing BDD, then we are delivering tiny slices of functionality that can be released immediately. Often we batch these into releases, often accompanied by a rather laborious and crunch-like polish process.

However, there’s no reason that we shouldn’t be ensuring that the site is release ready at any point. If this is the case, the stakeholders have the option to say “Done – ship it!” whenever they like, and we’re able to do this straight away, rather than after a three week polish + skin period, and the dreaded IE6 compatibility check.

Lesson learned: Get a site release-able as fast as possible, and keep it there.

Put performance tests in the build

As developers we are always concerned about timing our performance improvements. How do we balance over-optimising a site with doing performance testing too late and risking a meltdown? How do we ensure that performance is kept at an acceptable level?

If the goal is to always keep a project release ready, then we need to ensure that our new system hasn’t introduced any serious performance issues, which bring it below a minimum performance level. If you are guaranteeing to your client that your site can handle a certain level of traffic, then anything that breaks that effectively breaks the build. Therefore, what better way to do this than make your performance tests part of your build?

And there’s more…

I plan to blog further about some of the other things we learned: including the major insight about the benefits of a Lean approach, which is worthy of its own post.

Do you have strong opinions about the above? Do you disagree? Let us know!

Rails Underground BDD Workshop on 23rd July

A couple of days ago I ran a BDD workshop the day before Rails Underground. Skills Matter videoed the event and will provide the podcast + slides here, but if you’re impatient download the slides here.

I really enjoyed doing my first talk about Ruby/Rails – teaching it was great fun. Even though I’m sure we can improve further, we get a lot of value from BDD, Cucumber and RSpec and it was good to pass that on and give something back to this excellent community.

The workshop was free – if you got something out of it please make a donation to the Red Cross through the Rails Underground donation page.

Update: the video is now available here. If you were there, there’s a rating and feedback page for it.

The ever-moving market

Rob Dempsey posted an interesting article a little while back on the future of software development firms in the new economy. It discusses offshoring, the recession, and a shift towards the employment of permanent staff within companies for Ruby on Rails work.

This is the sort of thing I muse about all the time: I keep a close eye on the web development market to ensure Eden is still in a position to add real value to potential customers. The results are important for deciding what we want Eden to look like in the future.

Ultimately, I want any decision to use Eden to be the best decision for a customer’s business. If it’s not, I prefer to point people elsewhere. So how do some of the indicators Rob mentions stack up against our business model? Are we still the best decision a customer can make?

Offshoring still a way off

The rise in offshoring development doesn’t particularly worry me. I’ve worked with offshore agencies before and there are some excellent companies out there, but the distance and the communication issues do present barriers to the addition of value. The price is lower, but the value added is commensurate.

If the spec is very (very) clear then I think that an offshoring project could work: but the risks are high and difficult to mitigate. I would hate to tie down any of our customers to a particular spec, which is why we always work in an agile a fashion as possible. It’s easier to manage risk with an agile process and to do agile well you need very clear communication pathways.

Bringing it in-house

I have also noticed a move to using permanent staff and in-house contractors to get websites built. The pool of people with Ruby on Rails skills is growing and there’s often no need to hire a contractor at a distance.

However, hiring a big experienced team takes time and the best people are difficult to find. My experience leads me to believe companies will always look to specialist development agencies to get larger jobs started quickly, and to ensure they’re done to a certain standard. We have a number of customers to whom we’ve provided development services in this fashion, often as a kick-start to an in-house team in the process of being hired.

Shifting our focus

We have had to make changes recently: I realised a few months ago we needed to shift our focus away from smaller projects, where the scope was tighter and contractors or offshore teams could price us out of the market. We grew the team to the point where we’re no longer a small group of coders with our own small independent projects. We’re now a well functioning cohesive development agency, taking on larger projects in bigger teams.

We’re looking to diversify too: we’re full of ideas for new web applications we’d just love to build, and we’ll be making some time over the summer to kickstart some prototypes. Watch this space.

Web developers: we always have to be thinking about the market we’re in, especially one as fast moving as ours, where things seem to change by the week. What are your thoughts on the ever-moving web development market?

MOO: A project for the BBC

MOOI’ve just finished updating the website to let you know we’ve been working with the BBC for the last three months on updating an exciting internal project called MOO.

This has been an exciting project to work on and a lot of fun. There were some pretty big technical challenges on the way, but I’m very proud of the team: they really rose to the challenge and delivered some excellent work.

Read the full case study here for more details on what it’s all been about.

Billing with Integrity (part 5)

This is the goal. The method is the by-product.

This is the goal. The method is the by-product.


(You may want to read parts one, two, three and four first).

So after much discourse on billing methods and the complicated relations that exist between a client and supplier, the question remains: how do we do it? The following post outlines how we try to “bill with integrity”.

We do weekly billing

Virtually all our billing is now done on a weekly basis. We charge a fixed price (yes I know, bear with me), for between one to four developers for a week.

We prefer to put either two or four developers on a project for a week. This is best for most projects, so that they can pair program on the tasks. One developer is ok for R&D, prototyping work or very small amounts of work, but not recommended as you lose all the advantages of pairing.

When asked to quote work, we give an expectation of budget. We might recommend that a client book 3-5 weeks of time for 2 developers for example. It’s easy for everyone to work out how much that will cost, which promotes transparency.

We don’t charge extra for management time. All our management time with this billing method is rolled into the weekly cost. Therefore no nasty ‘management fees’ on the bill – what you’ve agreed to is what you get billed.

We build slack time into the project. All our rates are based on the fact that developers are unlikely to be able to spent more than four days of development time per week on a project. There are bug fixes, side projects, internal work, research, learning, review meetings, sick days, etc to consider.

We don’t stop at “enough”. We only guarantee to complete three days of work at the beginning of a project. That doesn’t seem like much a week, but because we don’t stop after the amount of work we’ve guaranteed, we’re not wasting the client’s time. If we have time we keep on going, all the way through to the five days if we can. Therefore we often over-deliver on a week’s booked work. We strive to ‘gently exceed expectations’ (as detailed in The Pragmatic Programmer; which is a must read, by the way). Pivotal Tracker helps track the velocity for a particular project, so we know how much we can expect to deliver on a project on any particular week.

So how is that different to fixed price?

There are elements of a fixed price methodology in this billing strategy. We do fix the price for a week’s work, for example. However, we don’t fix all the features that we can might deliver within a week, and we’re conservative with what we guarantee. This ensures that there is an appropriate amount of contingency built in, so if a project starts to get bogged down it doesn’t affect the schedule. Our estimates are done in half-day chunks using the planning poker method, which also makes them pretty accurate.

If a client is insistent that they need a fixed price, we simply say that we’ll add a certain percentage to the quoted cost for the number of iterations we’ve recommended to reflect the fact that we’re taking on more risk in the project. We’re quite honest about our reasons for doing this. If a client finds that difficult, we’re probably not the right company for them.

Aren’t you still charging a variable rate?

Well, sort of. You could say we’re charging per week as opposed to ‘per hour’, so what’s the difference? There are a number of distinct advantages that this model has over ’straight’ variable billing.

No need to clock time, for managers or developers. Variable billing requires meticulous records of time spent. We hardly ever need to do this now, saving the hassle of timesheet keeping, approval and complex invoicing.

Weeks are much easier to schedule than hours. Before switching to this model I found myself heading round the office at least four times a day to make sure everyone had enough work to do. Even when we made this more automatic, it was still a problem – it was really hard to commit to definite deadlines and everything was too fluid.

There’s much less context-switching for developers. With fully variable billing, everyone still found they were switching around between projects several times a day, which can be very confusing and inefficient. Now there’s at most 2-3 projects running at once, and there’s much less switching around – less than once a day for most people.

The team is much more focused over a whole week. Before we found we were effectively acting as several independent freelancers, all talking to individual clients and getting on with their work in isolation. This worked ok with much smaller projects, but it doesn’t scale to multiple team members on one project. ‘Silos’ of knowledge began to build up, and getting a new team member up to speed was hard. Now we’ve moved mostly away from variable billing, we’ve been able to pair program over the last few weeks and that’s really helped to break down the silos and promote communication. We’ll be posting about our experiences with pairing in a future post – it’s been so significant it’s probably worth another series :)

Some closing thoughts

It’s been great to write this series, because it’s really forced me to think about our billing strategy hard again, striving to ensure we have integrity about our billing. You can’t exactly write a series such as this and not be walking the talk, right? My thoughts have evolved as I’ve written it; there are some things about our current billing method that would be quite different had I not written this series.

A few extra thoughts as I round this up:

Billing is hard. That should be painfully obvious from some of the previous posts. We’ve learnt an enormous amount from our successes, but even more from our mistakes.

Ultimately it’s about the relationship. A great relationship with the client will help to soften any billing issue you get into. Likewise, a bad relationship will be hard to save if things go south.

And lastly, probably the most important thing:

We’re not done with improving our billing model. We’re on a journey, same as everyone. There are disadvantages to our current billing model too, which we’re looking for solutions to. Have you found a better way of working? Let us know!

Thanks for reading.

redirect_to :back not reliable

Suppose you want to allow someone to make a comment and then return to the page they were on. An easy way to do this in Rails is to set the CommentsController method to redirect_to :back

def create
  comment = Comment.new(params[:comment])
  if comment.save
    redirect_to :back
  end
end

But beware! The trouble with this is that Firefox has an option to disable referrers, so you can’t rely upon the HTTP_REFERER being set.

So you could set up a session variable when you display your comment form, to later be used in the controller. The current url is found in request.path

<% session[:return_to] = request.path -%>
def create
  comment = Comment.new(params[:comment])
  if comment.save
    redirect_to session[:return_to]
  end
end

Sure, people can turn off session cookies, but if they do that then you have all sorts of other Rails problems to do with authenticity tokens and user authentication, so I think this is a fairly safe method.

Of course, an even better option is to add the comment by AJAX and avoid the need for a page refresh at all … but that can be the topic of a different blog post!

Under the hood: Not-so-basic authentication

Recently I worked on a project that required a single login to access administration options. There was no need for a full-blown RESTful authentication solution – I was advised to “Just use basic auth!”

Rails makes it easy. You probably know the standard example. You put a before_filter :authenticate in the controllers that require it, and set it up in the Application Controller.

def authenticate
  authenticate_or_request_with_http_basic do |user, password|
    user == 'admin' && password == 'pass'
  end
end

It’s all well and good … until you want to add a log out button. The browser stores the successful login credentials in a sort of cookie, and applies them to every page which requests basic authentication. Once you’ve logged in, it’s actually quite hard to make the browser forget you until you quit and restart the browser. It’s hard, but not impossible. If you can force basic authentication to fail, the browser will throw away the credentials.

So the solution is to add a session variable that says “NO SRSLY, LOG ME OUT PLS!” This is the logout action (a destroy method in a Sessions Controller)

def destroy
  session[:logout_requested] = true
  flash[:notice] = "You have logged out successfully"
  redirect_to(root_path)
end

Now for the tricky bit. The way this works is subtle and takes a moment to figure out each time I think about it. We change the authenticate method in the Application Controller so that as well as checking the username and password, it also ensures that this flag has not been set. Meaning we can cause basic authentication to fail when we want it to.

def authenticate
  authenticate_or_request_with_http_basic do |user, password|
    user == 'admin' && password == 'pass' && session[:logout_requested] != true
  end
  session[:logout_requested] = nil
end

Next time our user goes to a page which requires authentication, the browser still provides the correct username and password, but the flag causes the basic authentication to fail. Obviously we then have to clear the flag straight away, otherwise the user would not be able to get back in again even with the correct credentials. The user must type in the correct login name and password again to be able to get back in.

Perhaps we want to know whether the user is logged in or not, so that we know whether to display an edit button. We can set another session variable. Conveniently, authenticate_or_request_with_http_basic returns a boolean value.

def authenticate
  session[:logged_in] = authenticate_or_request_with_http_basic do |user, password|
    user == 'admin' && password == 'pass' && session[:logout_requested] != true
  end
  session[:logout_requested] = nil
end

Remember to set the flag to false when you log out. Also remember that this flag could be true, false or nil so a check in the Application Controller looks like this:

def logged_in?
  session[:logged_in] == true
end

Finally it’s worth noting that the username and password do not have to be hard-coded like this. It’s simple for an example, but don’t think that’s all there is to basic authentication. There’s nothing to stop you comparing against values in a settings table or even doing a user lookup à la RESTful authentication.

def authenticate
  session[:logged_in] = authenticate_or_request_with_http_basic do |email, password|
    user = User.authenticate(email, password)
    if user && session[:logout_requested] != true
      self.current_user = user
      true
    else
      self.current_user = nil
      false
    end
  end
  session[:logout_requested] = nil
end

Thanks to Richard and Tris for their help in figuring out the not-so-basic aspects of basic authentication! :)

Want to work for us?

A quick note to draw your attention to the fact that we’re hiring again… this time we’re looking for mid-level or senior web developers. Do have a look at our jobs pages for more information.

Hiring in the midst of a recession is always a bit scary, but we’re finding that we’re very well placed as a company to withstand it. I’ve never been busier writing features and following up leads, and we’re rapidly becoming well-known for our team’s excellent work (I’m so proud of them :-)

So, if you’d like to join us, now is the time.

Pivotal Tracker: Fantastic app, shame it’s free

I’ve been digging around for a decent agile PM tool for a while, and stumbled across Pivotal Tracker on Twitter a couple of days ago.

Within about thirty minutes of trying it out, I was totally hooked. We’ve rolled our own basic project management tool internally, but we’ve yet to get the time to put proper iterative development support into the app. Pivotal Tracker just works. With a clean UI and some great reporting tools, it’s all I need to run an agile project.

The one big downside is this: it’s completely free.

Wait, you say: surely that’s a good thing? Well, yes and no.

Why great software should cost money

When I pay for an app on the net, I expect a few things:

  • It’s going to be up pretty much all the time.
  • It’s not going to vanish tomorrow.
  • It’s not going to suddenly cost a fortune.

With Pivotal Tracker, I’ve no idea when they’re going to take the software down for maintenance. I’ve seen reports of it going down without notices for 15 minutes here and there, which is fine if it’s a non-critical service. It is however most definitely not fine when my client is trying to comment on features. I also have no guarantee that it’s not going to suddenly disappear into the night, taking all my stories and possibly the project itself with it.

The other issue is that if I’m paying $20/month for it, I’ve a reasonable expectation that it’s not going to cost say $200/month next month. It’s well within Pivotal’s rights to withdraw the free service and charge what they like for the paying version, without any notice.

Are we going to use it anyway? Yes we are: it’s too good to ignore. However, we’ve put in place the following cron job to mitigate the risk:

*/20 * * * * curl -H "X-TrackerToken: " -H
 "Content-type: application/xml" -X GET
 https://www.pivotaltracker.com/services/v2/projects/39382/stories
 > stories-backup.xml

This little piece of magic copies all our stories in XML form to our backup server every 20 minutes, so we’re not screwed if the site vanishes without notice. It would only take us about an hour to import this XML into our old system should we need to. That takes care of the main risk for me.

The conclusions: Some software shouldn’t be free. If you’re using Pivotal Tracker or any other free service in anger, make sure you’ve got a backup plan.