I was talking with a colleague of mine today and he mentioned to me that he is having trouble getting his team to deliver a working build. He has large development team with over 20 developers, with the majority of them working offshore.
I asked him if his team was doing daily builds. He said no and asked me if my team does daily builds. For the most part we do (sometimes we slack :( ). It has always been a fundamental belief that many of the integration problems that rise during large scale software development can be exposed by building the entire project every day and running all of the project's unit tests.
My colleague asked me if I could document some of my thoughts on the subject of daily builds. Rather then write them up in an e-mail, I decided to post them in "Monkeys with Keyboards."
John's Thoughts on Builds
- Build Daily (If not more often) - Pulling all of your source code from the source control system and building it helps shake out stupid integration issues. Remember people make mistakes and forget to check code in or get sloppy with their code and do not check to see if it compiles. (Ohh, this small change won't break anything! ) Checking out the code daily and building the entire application finds these issues.
A few years ago I was working in Boston leading a team of 13 developers. We had daily build schedule and because of this we were able to always give our QA group a build within 2 hours or less.
A team right down the hall from us did their builds once ever two weeks and would build the entire application on the developers machine. It took them 2 days to deliver the build and often times the QA department would find major and blatantly obvious problems with the build.
All of these things could have been avoided if they would have done a daily build and then quickly run through all of the screens in their application.
- Build in a clean room environment - When your team does the build do it on a clean machine and not on a developer's desktop's. A developer's desktop is suppose to be a play area and can have often have configuration and JAR files on it that can cause integration problems to be hidden. Nothing is more irritating to discover that the application built on a developer's desktop only to find out there desktop environment was configured differently then the machine on where the build is actually going to run.
- Automate your builds - If builds are not easy to do developers will find a way to avoid them. Automate your builds to the point where it is simple as a push of the button. A well-behaved build script:
- Check out all of the source code from the source code repository.
- Compile all of the code
- Run all of the unit tests and code coverage
- Run all documentation (i.e. JAVADocs tasks)
- JAR/ZIP any deliverables up
- Push the build out to an integration server
- Notify the buildmaster that the build is complete
- Cleanup any temporary build files or tasks
- Smoke test your builds - After the build has been deployed the buildmaster should walk through each of the screens and check to make sure major pieces of functionality are working. This does not mean checking every detail of the screen, but rather seeing if after you build your application does the server it runs fall over. The idea behind "smoke" tests are not new. I believe they were first implemented as a practice at Microsoft and were documented in Steve McConnell's book Code Complete.
For smoke test's your team should have a simple checklist of functionality that is run through after each build. If all of the items on the checklist pass then the build has passed the smoke test and is ready for consumption by testers.
- Enforce check-in standards - Developers never like to check in their code until it is "done." Done is a relative code. My team's rule of thumb is this:
Check code in everyday. Only check your code in if it compiles.Code is the one of truest sources of looking at the health of a project. If a developer does not know what they are doing wouldn't you rather find the problems right away rather then wait until the very end and find out that there are serious problems in the code being delivered.
The enforcing of checking-in standards is extremely important in a distributed development environment because the developers do not have the luxury of talking to each other throughout the day. Since problems can take days to weeks to manifest themselves, have the ability to look at code everyday is critical.
- Everyone is responsible for the build (Share the pain) - Everyone on the team should know how to do the build and should take turns doing the builds. This is particularly important when instituting daily builds in a team that has never done. After a few nights of having to stay late to fix a build problem, developers will very quickly become more conscious of how they check-in code.
- If a build fails fix it or re-deploy the old build - This is a big one. If a build fails it is the buildmaster's responsibility to have the person who broke the build fix the code. If that person is not available, it is the build master's responsibility to fix the problem.
If the buildmaster can not fix the problem they have to recover the development/testing server back to the previous days build. This is critical because:
If you can setup your build to use a continous integration tool like Cruise Control or AntHill. These tools will poll your source control system every X minutes and check to see if new files have been checked in since the last build. If the tool(s) finds new files, they can automatically kick of the build. Remember:
The success of your build practices will be strongly correlated to how easy and automated the build process is.
- Developers often forget that QA and business analysts rely on a build to be stable for testing and/or demoing. Nothing frustrates them more when they go out to the application and it is just broke. Often times these folks are held accountable for getting through their test cases within an X amount of time. Having the test environment unavailable because the build is broke puts them behind schedule. (Believe or not developers are not the only ones who have to work weekends.)
- A build is a testimony to where the team is at in the development efforts. Not having a new build available everyday can end up hiding serious problems until the end of the project.
- Automated - The build runs with minimal effort from the development team.
- Repeatable - Follow a process to ensure that the same steps are used regardless of who does the build.
- Dependable - Builds must be available to a QA or business analyst with minimal effort.
- Opaque - Builds must expose problems quickly and obviously. The quicker a problem is exposed to daylight, the quicker the development team can work to resolve the issue.