main banner


Demystifying Agile, Part 5

In this post, I will continue with my last posts topic, and describe additional engineering practices and techniques commonly used to aid agile development.

Engineering Practices, Techniques and Tools to aid agile development and to lower technical debt II

Continuous Integration (CI) and Continuous Delivery (CD)

Continuous integration is a software development practice that consist of frequently integrating the work done, by each developer in the team, with the latest integrated codebase (mainline) of the software; this practice is done often (continuously) with the goal of preventing integration problems and the so called “Integration Hell”.

“Integration Hell” happens when the time it takes to integrate changes back into the mainline is longer that the time it took to develop them. By Integrating often, the risk of encountering integration problems is highly reduced; and if problems are found, it would be easier and faster to fix them as the changes have been introduced recently.

A typical CI goes like this:

When working on coding tasks, a developer first gets a copy of the currently integrated source code (mainline code) into his local development environment (working copy). Then, he proceeds to do his development work on that local copy. As time passes his local copy starts to become outdated against the mainline (as other developers integrate their work back into it); so the longer he takes to integrate his changes back to the mainline, the higher the risk of encountering code conflicts during his integration.

He will integrate his code changes back to the mainline only after he has successfully tested them locally; and, only after he has downloaded and successfully integrated, with his working copy, all code changes that took place on the mainline codebase since the time he got his working copy (local integration); then he will produce a successful local build.

To integrate work back to mainline, a developer will move all his local tested code changes to the mainline codebase, all conflicts should be resolved and then all integration test should be executed and passed, and a mainline build should be made to consider it a successful integration. When problems are found mainline should revert to its previous successful integration state immediately, and then the developer should fix his changes in his local environment and retry the integration.

There is no rule for how often to integrate, but a good rule of thumb is that each developer in the team has to integrate his work at least once a day.

There are no specific tools required to do CI, however there are many types of tools and techniques that have been developed throughout the years that will help doing CI seamless, some of them are:

Source Code Management Systems, Version Control Systems, Configuration Management tools, Self-Testing Code, Tools to produce Automated Builds, Continuous Integration Servers,  Tools to produce Automated Tests, etc.

To be effective, CI relies on the following practices:

Maintain a single source repository: Place all files needed (code, test scripts, configuration files, database loads, install scripts, etc.) to create a fresh working build of the software into one single repository that must be reachable by all team members. A mainline branch in a Source Code System is the ideal place (just keep the branching to a minimum so that everyone in the dev team bases their work on the mainline codebase most of the time).

Automate the build: The build process should be automated to the point that anyone should have the capability to build the software just by issuing a single command; use the IDE build functionality to do local builds but use a build tool like Apache Ant, MSBuild, Make, etc. to create the mainline builds (or master builds).

Make the build self testing: Produce an automated suit of tests, then each time an automated build is made the automated test suite should be executed to ensure the software is stable and usable; these automated tests should be integrated with the build process so that the build process fails when any of these tests fail. Use Self-Testing code (or use TDD) and automated test tools like Selenium, Watir, Sahi, Xunit, etc.

Everyone commits to mainline everyday: If a developer integrates often (at least once a day) his work, the risk of facing integration problems will be reduced. During integration, automated and self-testing builds are made, a successful build means the developer has green light to commit his changes back to mainline. If problems arise, they will be easier and quicker to fix, as these integrations are happening often.

Every commit to mainline should build on an integration machine: Whenever a commit is done to mainline (changes are pushed back into its source code) an automated self testing build should be created on an integration machine (this can be done manually or by using a CI server to do it); this will ensure mainline is always on a healthy state.

Fix broken builds immediately: If the mainline build fails, fixing it should be top priority as keeping a stable integrated code as a base for all development is the primary point of CI. Producing a successful build is the way to ensure the recent code changes were integrated successfully and the code base is stable. When the build fails, the integration should revert to the previous stable integration (rolling back the changes) and fixes should be done on a development environment.

Keep the build fast: This one is self explanatory, but how fast? well XP promotes the 10 minute build rule, however it really depends on the project requirements and the expertise of the development team. Most of the time the automated tests of the self testing build are the ones that consume most of the building time; each team has to find the right balance between the extensiveness of the automated test executed during the build vs how fast the build can be created; to keep the team focused and working as CI promotes often mainline commits.

Test in a clone of production environment: You should aim to create a staging environment that is used to test the builds; this environment should be as similar to production environment as possible (expenses, hardware limitations, large number of production configurations, etc might play against you) and be aware that any differences in these environments present a risk of a possible failure in production. Virtualization tools can be useful.

Make it easy to everyone to get the latest deliverables: Anyone that is involved in the project should be able to get the latest working deliverables (and be able to run it) so they can inspect it, try it out, do demonstrations, do exploratory testing, see what are the latest changes, etc. Designate a store where these deliverables will be placed and ensure everyone has access to it.

Everyone can see what is going on (the progress and results of the latest build): It should be easy for everyone to find out the outcome of the mainline build (success or failure) and also who is responsible for the integration that broke the build. People can get creative, like using virtual semaphores or lava lamps; or physical things like flags or toys; even sirens.

Automate Deployment: when doing CI, teams are moving code from one environment to another often (even many times a day); so It is a good idea to use tools that allow the team to automate the deployment process, this will save time and reduce the risk of manual errors during deployment (consider also automatic rollbacks to roll back a faulty deployment). By doing this, the team is encouraged to deploy often, getting new features to users smoothly adds more value to the process.

As you can see Continous Integration has many benefits like: the reduction in number of bugs in the different environments (by running an automated self-testing build bugs are found quickly), the ability to release often and quickly, the reduction of risk (as integration is done often and in short increments) and the visibility of the state and effort needed to integrate and build a stable software version.

If you are adopting CI a good starting point is introducing the CI practices into your process one at a time; automating the Build is a common first step; then move further to adopting the other practices like self-testing builds, and reducing build times, etc.

There are many tools that have been developed to help teams and organizations do CI, the most populars are: Jenkins, TeamCity, CircleCI, Travis, Codeship, GitLab CI, CruiseControl, Bamboo, TFS, etc.

Continuous Delivery (CD) is, as Martin Fowler explains it,

“a software development discipline where you build software in such a way that the software can be released to production at any time”.

Continuous Delivery builds on top of Continuous Integration and extends it by automating the steps needed to deploy the software to production at any moment (like automating acceptance tests); in order to do it the team must be able to:

  • Work under a collaboration culture with all personnel involved in the delivery of the software (DevOps culture).

  • Automate all automatable steps in the delivery process (Development Pipeline).

  • Keep the software in a deployable state throughout the development lifecycle.

  • Provide automated feedback, on the production readiness of the software, each time something gets changed.


By doing continuous delivery the organization will be capable of deploying to production, on demand, any version of the software with a single command.

There is a variation of CD called continuous deployment; this variation automatically deploys to production without waiting for request to do so. In order to do continuous deployment the organization needs to be doing continuous delivery already.

The benefits of CD include reducing the risks associated with deployment processes (less things can go wrong when doing often deployments in small increments), progress measurement is more accurate (development done vs deployed to production done), and the team can get user feedback sooner (finding course deviations faster).

So, that’s it, I have described the most common engineering techniques and practices that will aid you during agile development; also, I have mentioned some tools that can be used to help you adopt them.

In the next post I will give you a summary and the references used during the writing of the entire blog series.

Raúl N.

Ever met one of those guys whose code should’ve been his natural language? Well, meet Raul, a Computer Systems Engineer whose hobbies include video games, playing with his ferret pets, reading and movies. He has over 12 years of experience as a developer and has a lot of experience with Microsoft Technologies (specially .NET) as well as architecture and software development process; he has written some really good articles about this topics.