Coding is like gardening...

Author Archive

Rails Underground BDD Workshop on 23rd July

A couple of days ago I ran a BDD workshop the day before Rails Underground. Skills Matter videoed the event and will provide the podcast + slides here, but if you’re impatient download the slides here.

I really enjoyed doing my first talk about Ruby/Rails – teaching it was great fun. Even though I’m sure we can improve further, we get a lot of value from BDD, Cucumber and RSpec and it was good to pass that on and give something back to this excellent community.

The workshop was free – if you got something out of it please make a donation to the Red Cross through the Rails Underground donation page.

Update: the video is now available here. If you were there, there’s a rating and feedback page for it.

The ever-moving market

Rob Dempsey posted an interesting article a little while back on the future of software development firms in the new economy. It discusses offshoring, the recession, and a shift towards the employment of permanent staff within companies for Ruby on Rails work.

This is the sort of thing I muse about all the time: I keep a close eye on the web development market to ensure Eden is still in a position to add real value to potential customers. The results are important for deciding what we want Eden to look like in the future.

Ultimately, I want any decision to use Eden to be the best decision for a customer’s business. If it’s not, I prefer to point people elsewhere. So how do some of the indicators Rob mentions stack up against our business model? Are we still the best decision a customer can make?

Offshoring still a way off

The rise in offshoring development doesn’t particularly worry me. I’ve worked with offshore agencies before and there are some excellent companies out there, but the distance and the communication issues do present barriers to the addition of value. The price is lower, but the value added is commensurate.

If the spec is very (very) clear then I think that an offshoring project could work: but the risks are high and difficult to mitigate. I would hate to tie down any of our customers to a particular spec, which is why we always work in an agile a fashion as possible. It’s easier to manage risk with an agile process and to do agile well you need very clear communication pathways.

Bringing it in-house

I have also noticed a move to using permanent staff and in-house contractors to get websites built. The pool of people with Ruby on Rails skills is growing and there’s often no need to hire a contractor at a distance.

However, hiring a big experienced team takes time and the best people are difficult to find. My experience leads me to believe companies will always look to specialist development agencies to get larger jobs started quickly, and to ensure they’re done to a certain standard. We have a number of customers to whom we’ve provided development services in this fashion, often as a kick-start to an in-house team in the process of being hired.

Shifting our focus

We have had to make changes recently: I realised a few months ago we needed to shift our focus away from smaller projects, where the scope was tighter and contractors or offshore teams could price us out of the market. We grew the team to the point where we’re no longer a small group of coders with our own small independent projects. We’re now a well functioning cohesive development agency, taking on larger projects in bigger teams.

We’re looking to diversify too: we’re full of ideas for new web applications we’d just love to build, and we’ll be making some time over the summer to kickstart some prototypes. Watch this space.

Web developers: we always have to be thinking about the market we’re in, especially one as fast moving as ours, where things seem to change by the week. What are your thoughts on the ever-moving web development market?

MOO: A project for the BBC

MOOI’ve just finished updating the website to let you know we’ve been working with the BBC for the last three months on updating an exciting internal project called MOO.

This has been an exciting project to work on and a lot of fun. There were some pretty big technical challenges on the way, but I’m very proud of the team: they really rose to the challenge and delivered some excellent work.

Read the full case study here for more details on what it’s all been about.

Billing with Integrity (part 5)

This is the goal. The method is the by-product.

This is the goal. The method is the by-product.


(You may want to read parts one, two, three and four first).

So after much discourse on billing methods and the complicated relations that exist between a client and supplier, the question remains: how do we do it? The following post outlines how we try to “bill with integrity”.

We do weekly billing

Virtually all our billing is now done on a weekly basis. We charge a fixed price (yes I know, bear with me), for between one to four developers for a week.

We prefer to put either two or four developers on a project for a week. This is best for most projects, so that they can pair program on the tasks. One developer is ok for R&D, prototyping work or very small amounts of work, but not recommended as you lose all the advantages of pairing.

When asked to quote work, we give an expectation of budget. We might recommend that a client book 3-5 weeks of time for 2 developers for example. It’s easy for everyone to work out how much that will cost, which promotes transparency.

We don’t charge extra for management time. All our management time with this billing method is rolled into the weekly cost. Therefore no nasty ‘management fees’ on the bill – what you’ve agreed to is what you get billed.

We build slack time into the project. All our rates are based on the fact that developers are unlikely to be able to spent more than four days of development time per week on a project. There are bug fixes, side projects, internal work, research, learning, review meetings, sick days, etc to consider.

We don’t stop at “enough”. We only guarantee to complete three days of work at the beginning of a project. That doesn’t seem like much a week, but because we don’t stop after the amount of work we’ve guaranteed, we’re not wasting the client’s time. If we have time we keep on going, all the way through to the five days if we can. Therefore we often over-deliver on a week’s booked work. We strive to ‘gently exceed expectations’ (as detailed in The Pragmatic Programmer; which is a must read, by the way). Pivotal Tracker helps track the velocity for a particular project, so we know how much we can expect to deliver on a project on any particular week.

So how is that different to fixed price?

There are elements of a fixed price methodology in this billing strategy. We do fix the price for a week’s work, for example. However, we don’t fix all the features that we can might deliver within a week, and we’re conservative with what we guarantee. This ensures that there is an appropriate amount of contingency built in, so if a project starts to get bogged down it doesn’t affect the schedule. Our estimates are done in half-day chunks using the planning poker method, which also makes them pretty accurate.

If a client is insistent that they need a fixed price, we simply say that we’ll add a certain percentage to the quoted cost for the number of iterations we’ve recommended to reflect the fact that we’re taking on more risk in the project. We’re quite honest about our reasons for doing this. If a client finds that difficult, we’re probably not the right company for them.

Aren’t you still charging a variable rate?

Well, sort of. You could say we’re charging per week as opposed to ‘per hour’, so what’s the difference? There are a number of distinct advantages that this model has over ’straight’ variable billing.

No need to clock time, for managers or developers. Variable billing requires meticulous records of time spent. We hardly ever need to do this now, saving the hassle of timesheet keeping, approval and complex invoicing.

Weeks are much easier to schedule than hours. Before switching to this model I found myself heading round the office at least four times a day to make sure everyone had enough work to do. Even when we made this more automatic, it was still a problem – it was really hard to commit to definite deadlines and everything was too fluid.

There’s much less context-switching for developers. With fully variable billing, everyone still found they were switching around between projects several times a day, which can be very confusing and inefficient. Now there’s at most 2-3 projects running at once, and there’s much less switching around – less than once a day for most people.

The team is much more focused over a whole week. Before we found we were effectively acting as several independent freelancers, all talking to individual clients and getting on with their work in isolation. This worked ok with much smaller projects, but it doesn’t scale to multiple team members on one project. ‘Silos’ of knowledge began to build up, and getting a new team member up to speed was hard. Now we’ve moved mostly away from variable billing, we’ve been able to pair program over the last few weeks and that’s really helped to break down the silos and promote communication. We’ll be posting about our experiences with pairing in a future post – it’s been so significant it’s probably worth another series :)

Some closing thoughts

It’s been great to write this series, because it’s really forced me to think about our billing strategy hard again, striving to ensure we have integrity about our billing. You can’t exactly write a series such as this and not be walking the talk, right? My thoughts have evolved as I’ve written it; there are some things about our current billing method that would be quite different had I not written this series.

A few extra thoughts as I round this up:

Billing is hard. That should be painfully obvious from some of the previous posts. We’ve learnt an enormous amount from our successes, but even more from our mistakes.

Ultimately it’s about the relationship. A great relationship with the client will help to soften any billing issue you get into. Likewise, a bad relationship will be hard to save if things go south.

And lastly, probably the most important thing:

We’re not done with improving our billing model. We’re on a journey, same as everyone. There are disadvantages to our current billing model too, which we’re looking for solutions to. Have you found a better way of working? Let us know!

Thanks for reading.

redirect_to :back not reliable

Suppose you want to allow someone to make a comment and then return to the page they were on. An easy way to do this in Rails is to set the CommentsController method to redirect_to :back

def create
  comment = Comment.new(params[:comment])
  if comment.save
    redirect_to :back
  end
end

But beware! The trouble with this is that Firefox has an option to disable referrers, so you can’t rely upon the HTTP_REFERER being set.

So you could set up a session variable when you display your comment form, to later be used in the controller. The current url is found in request.path

<% session[:return_to] = request.path -%>
def create
  comment = Comment.new(params[:comment])
  if comment.save
    redirect_to session[:return_to]
  end
end

Sure, people can turn off session cookies, but if they do that then you have all sorts of other Rails problems to do with authenticity tokens and user authentication, so I think this is a fairly safe method.

Of course, an even better option is to add the comment by AJAX and avoid the need for a page refresh at all … but that can be the topic of a different blog post!

Under the hood: Not-so-basic authentication

Recently I worked on a project that required a single login to access administration options. There was no need for a full-blown RESTful authentication solution – I was advised to “Just use basic auth!”

Rails makes it easy. You probably know the standard example. You put a before_filter :authenticate in the controllers that require it, and set it up in the Application Controller.

def authenticate
  authenticate_or_request_with_http_basic do |user, password|
    user == 'admin' && password == 'pass'
  end
end

It’s all well and good … until you want to add a log out button. The browser stores the successful login credentials in a sort of cookie, and applies them to every page which requests basic authentication. Once you’ve logged in, it’s actually quite hard to make the browser forget you until you quit and restart the browser. It’s hard, but not impossible. If you can force basic authentication to fail, the browser will throw away the credentials.

So the solution is to add a session variable that says “NO SRSLY, LOG ME OUT PLS!” This is the logout action (a destroy method in a Sessions Controller)

def destroy
  session[:logout_requested] = true
  flash[:notice] = "You have logged out successfully"
  redirect_to(root_path)
end

Now for the tricky bit. The way this works is subtle and takes a moment to figure out each time I think about it. We change the authenticate method in the Application Controller so that as well as checking the username and password, it also ensures that this flag has not been set. Meaning we can cause basic authentication to fail when we want it to.

def authenticate
  authenticate_or_request_with_http_basic do |user, password|
    user == 'admin' && password == 'pass' && session[:logout_requested] != true
  end
  session[:logout_requested] = nil
end

Next time our user goes to a page which requires authentication, the browser still provides the correct username and password, but the flag causes the basic authentication to fail. Obviously we then have to clear the flag straight away, otherwise the user would not be able to get back in again even with the correct credentials. The user must type in the correct login name and password again to be able to get back in.

Perhaps we want to know whether the user is logged in or not, so that we know whether to display an edit button. We can set another session variable. Conveniently, authenticate_or_request_with_http_basic returns a boolean value.

def authenticate
  session[:logged_in] = authenticate_or_request_with_http_basic do |user, password|
    user == 'admin' && password == 'pass' && session[:logout_requested] != true
  end
  session[:logout_requested] = nil
end

Remember to set the flag to false when you log out. Also remember that this flag could be true, false or nil so a check in the Application Controller looks like this:

def logged_in?
  session[:logged_in] == true
end

Finally it’s worth noting that the username and password do not have to be hard-coded like this. It’s simple for an example, but don’t think that’s all there is to basic authentication. There’s nothing to stop you comparing against values in a settings table or even doing a user lookup à la RESTful authentication.

def authenticate
  session[:logged_in] = authenticate_or_request_with_http_basic do |email, password|
    user = User.authenticate(email, password)
    if user && session[:logout_requested] != true
      self.current_user = user
      true
    else
      self.current_user = nil
      false
    end
  end
  session[:logout_requested] = nil
end

Thanks to Richard and Tris for their help in figuring out the not-so-basic aspects of basic authentication! :)

Want to work for us?

A quick note to draw your attention to the fact that we’re hiring again… this time we’re looking for mid-level or senior web developers. Do have a look at our jobs pages for more information.

Hiring in the midst of a recession is always a bit scary, but we’re finding that we’re very well placed as a company to withstand it. I’ve never been busier writing features and following up leads, and we’re rapidly becoming well-known for our team’s excellent work (I’m so proud of them :-)

So, if you’d like to join us, now is the time.

Pivotal Tracker: Fantastic app, shame it’s free

I’ve been digging around for a decent agile PM tool for a while, and stumbled across Pivotal Tracker on Twitter a couple of days ago.

Within about thirty minutes of trying it out, I was totally hooked. We’ve rolled our own basic project management tool internally, but we’ve yet to get the time to put proper iterative development support into the app. Pivotal Tracker just works. With a clean UI and some great reporting tools, it’s all I need to run an agile project.

The one big downside is this: it’s completely free.

Wait, you say: surely that’s a good thing? Well, yes and no.

Why great software should cost money

When I pay for an app on the net, I expect a few things:

  • It’s going to be up pretty much all the time.
  • It’s not going to vanish tomorrow.
  • It’s not going to suddenly cost a fortune.

With Pivotal Tracker, I’ve no idea when they’re going to take the software down for maintenance. I’ve seen reports of it going down without notices for 15 minutes here and there, which is fine if it’s a non-critical service. It is however most definitely not fine when my client is trying to comment on features. I also have no guarantee that it’s not going to suddenly disappear into the night, taking all my stories and possibly the project itself with it.

The other issue is that if I’m paying $20/month for it, I’ve a reasonable expectation that it’s not going to cost say $200/month next month. It’s well within Pivotal’s rights to withdraw the free service and charge what they like for the paying version, without any notice.

Are we going to use it anyway? Yes we are: it’s too good to ignore. However, we’ve put in place the following cron job to mitigate the risk:

*/20 * * * * curl -H "X-TrackerToken: " -H
 "Content-type: application/xml" -X GET
 https://www.pivotaltracker.com/services/v2/projects/39382/stories
 > stories-backup.xml

This little piece of magic copies all our stories in XML form to our backup server every 20 minutes, so we’re not screwed if the site vanishes without notice. It would only take us about an hour to import this XML into our old system should we need to. That takes care of the main risk for me.

The conclusions: Some software shouldn’t be free. If you’re using Pivotal Tracker or any other free service in anger, make sure you’ve got a backup plan.

Billing with Integrity (part 4)

How do you think a client would feel if he got this bill?

How do you think a client would feel if he got this bill?

(You may want to read parts one, two and three first).

Given that “fixed price fixed scope” billing is flawed, a natural next step for many small businesses is to try variable billing. This time we’ll look at some of the pros and cons of such an approach.

By “variable billing” I am referring to the practice of billing per hour (or per day) for your services. It’s a pretty common way of charging out time in the web development industry. Going back to the four variables of software development from my previous post, when billing in this way you are effectively fixing quality (or you should be!), and tying time and cost together. Therefore you are allowing scope and time/cost to vary based on the client requirements.

The advantages

On the surface, variable billing appears to be the answer to all our problems. We don’t have to worry about estimates, because the client is paying per hour, so we’ll just take as long as we need to get things done. Changes in the project are handled easily: we just drop what we’re doing and change direction, so we can give the client the features they want. Goodbye to underbid projects completed for free!

So why is the client often unhappy with this sort of arrangement? Why won’t many clients even sign up to it?

The disadvantages

Many of the advantages are to do with transferring risk away from the developer and onto the client. If the developer is freed from the risk associated with getting a feature done, then that means the client is bearing the full brunt of the uncertainty. Clients want a fixed price, so they can compare you to other suppliers. Just doing variable pricing makes the proposal process very difficult.

There’s also the lack of trust. If you’re just setting up a relationship with the client, they don’t know you from Adam: they have no idea whether whether you’re as good as your word. You might say 2 days for a feature, but if the bill comes in for 4 days, they aren’t going to be happy. It only takes one large unexpected bill to wreck a client relationship.

Another main disadvantage is more subtle. With this sort of approach, a client can sometimes micromanage the project, as they’re rightly concerned about getting value for money. Therefore it’s possible to get inundated with feedback, minor tweaks, irrelevant bug reports and requests for status updates. This burns through yet more of the developer’s time, adding to a client’s bill and dragging down project efficiency. Often clients will want a very detailed breakdown of exactly what time was spent on their project, causing more management overheard.

Conclusion: Useful in some circumstances

Variable billing works well when there’s a large amount of goodwill between client and developer. It doesn’t work so well at the beginning of a client relationship. It’s worked well for us on the two extremes of projects we take on – the very short and the very long:

* Software consultancy, where we’ve been providing short term expert advice on an ad hoc basis.
* Tiny changes, where there’s only a day or two of little tweaks to make to a site.
* Open ended long term projects, where we provide development resource over months/years.

Sometimes projects degenerate into variable billing as the trust grows, a client starts believing our estimates, and everything becomes more informal.

Interestingly, on our most successful long term collaboration using this billing method we have a regular monthly budget which we can’t exceed, but we use that up in a variable ‘per hour’ manner. This is strikingly similar to our current chosen method of billing, which I’ll discuss in the next and final post in this series.

Tune in next time for an outline of our current method (you may want to subscribe for the latest updates). If you’re in a hurry for an “answer”, you could always peek at our pricing pages if you want to know how we bill :) Thanks to Kent Beck’s excellent essay on Optional Scope Contracts (PDF) for many of the ideas in this series, and the feedback I’ve received from various people.

Twitter integration from your Rails app

We’ve learnt a bit about Twitter from writing ykyat.com.

I particularly want to write about how we fetch Twitter user icons. The icons are stored on AWS (Amazon Web Services) and cannot be deduced from the user name. You have to go through the Twitter API to find the image location.

The first thing I tried was the Twitter4R library. This seems to be a very powerful library, and if you were writing a full Twitter client in Ruby you’d definitely want to consider it. It was simple to get the user icons, but the library just felt a little over-the-top for our needs.

Twitter4R seemed to require authentication with every request, and we soon hit the API limit. I realised this is odd because you really don’t need to authenticate to look at the XML or JSON data about a Twitter user. I decided to go back to basics and do it myself. This is the code for parsing the JSON data and picking up the user icon URL:

def icon_url_for_user(username)
  require 'open-uri'
  require 'json'
  buffer = open("http://twitter.com/users/show/#{username}.json").read
  result = JSON.parse(buffer)
  result['profile_image_url']
end

See! Easy!

Most people don’t change their user icon very often, so once we know where to find a user’s icon, we don’t need to ask Twitter for it again for an arbitrary amount of time. A week seems quite sensible. To that end, we created a lookup table in our database to match Twitter user names to their user icon URL. We added an index to the user name column because it acts as the primary key lookup.

When we want to know a user’s icon, we first look up in our table. If we don’t yet have it, or if the updated_at date is more than a week ago, we check with Twitter for the image location. Otherwise we use our cached location.

Fast and easy! :)