The Broken Build

*Bling*, your email client exclaims. You ignore it. You’re on a roll. Things are flying by. At this pace you will be out of the office an hour before you expected. 3 minutes later. *bling*. You wonder whether you should check you email now. After hesitating a few seconds you continue working, only mildly distracted. Another 3 minutes later. *Bling*. After looking swiftly at your watch you decide to check your inbox.

“Arrrgggh, it’s the build server. Three mails … *bling* four … Oh man! Well, let’s see what happened.”

Browsing through hundreds of rows of stacktraces and other hard to read output you realize that someone has checked in code that looks for a particular file on the file system. “When will they ever learn” you say to yourself. You rub your sore shoulders briefly and whip your head from side to side and get back to work.

Time flies, 45 minutes and fifteen more build server mails later your are pretty near your boiling point. The code you are working on is not turning out the way you wanted it to and things are not running very smoothly. You decide to stop monitoring the build server and do so with an angry click. Instantly you have unsubscribed from the status reports of the build server.

“Finally, some piece of mind”

This behavior is just a symptom of an underlying problem as you probably guessed. Other symptoms might be turning off a lava lamp, turning the volume down on the build server speakers, removing certain packages from the build or even down right stopping a build server. In short: stopping information from reaching its intended audience.

Examining and taking care of the underlying problem can be a tedious task. But it is nevertheless the most important thing to do. I’d say, stop working on anything new until you have corrected and I do mean corrected, not suppressed the error.

First you must find the (root)cause of the problem. Then you have to make sure it does not show up again. Here are a few examples of more or less hard to handle causes that I have come across and been forced to deal with.

1. Using resources / Slow tests
Slow tests slows development down. They usually have several other drawbacks as well. Michael Feathers writes more about unit tests.

2. Cyclic dependencies
What came first, the chicken or the egg?

3. Threaded tests
How many people can really handle threads? Do not make the tests harder than the code. Really!

4. Memory leakage
Can be really hard to find. Run your application in a profiler from time to time to prevent this.

Then there are probably a million more reasons why your build would break. Here is how I think you should proceed.

– Automate your build
– Fix all (root)problems as soon as they appear
– Make sure the build creates deployment ready software (maybe even deploy)
– Worship your build-server
– Repeat the above as long as you want to deliver high quality code


About Ola Ellnestam

Agile Coach, Systems developer, Agile Practitioner and father of three.

2 responses »

  1. Nice post! I think it’s easy to end up in this situation when you haven’t fostered a culture that respects the CI build. In my experience most developers like to fix things that are broken, but if they believe it’s someone else’s problem, that’s another issue.

  2. Ola Ellnestam says:

    Yeah. I believe so too. I think that the Not My Problem attitude is very closely related. I mean to blog on in later in the series.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s