ribbon

Hosted Continuous Integration Comparative

Continuous Integration is one of the most valuable practices the Extreme Programming (XP) community have contributed. Avoiding the “in my PC works”-effect has a clear impact on the ROI of every project so managers happily invest in processes, tools and training for this. Virtualization technologies have made possible and cost-effective to outsource all the hardware needed to maintain these continuous integration processes paying only for the use (SaaS). The term coined for this concept is Hosted Continuous Integration. You can google or ask in Stackoverflow and will find a number of solutions and comparatives like this one, but products will be fortunately changing, introducing new features and modifying their prices. However the criteria to evaluate which is the best solution for you should be more stable. In this article we want to help you with your decision by giving the most relevant criteria based on our experience with customers and what we and others have shared in software development ecosystems forums like ecosistemas-software (in Spanish). If you are a developer and want that your CTO or CFO gives you the budget to move your CI systems to a SaaS solution, share this article with them because it is written also for them.

Criteria

Assuming you have already decided to go for a hosted solution and put your continuous integration systems in the cloud, now you have to choose among a number of solutions available. In order to evaluate the suitability of a service you need to consider several aspects. Some of them will try to point financial aspects out and some others will be more focused on technical details. A balanced and careful set of criteria should let you take the cheapest solution for the short and the long term according to your technical requirements and estimations of growth.

CI tool

Some years ago, CruiseControl (a Continuous Integration tool made by Thoughtworks, the company where one of the fathers of CI works) was almost the only open source CI tool available. Today Jenkins is undoubtedly the most used one, specially in those projects based in Java. It is also an open source project, based on the previous work on Hudson (now property of Oracle), with a myriad of plugins that help you integrate Jenkins with almost any existing scenario.

Some people say that “Jenkins is a cron with steroids”. It depends on how important are those steroids for you. Maybe you just need a script run by a cron process. Depending on your technological stack (i.e. language, libraries, runtimes, build tools…) you may need a different CI tool and some vendors have opted for a custom solution. If you haven’t selected your CI tool yet you should also consider the learning curve.

Build tool

The main purpose of a continuous integration system is to build artifacts. Sometimes the artifacts are code to be run for testing and QA purposes, sometimes they are code to be run in a deployment environment (e.g. in a demo environment or even for production), sometimes they are just documentation (e.g. the site of the project build with some QA reports, API docs, project management reports, etc). For the CI system, the artifacts are not relevant, but it is the build tool.

Some well-known open-source tools are Maven, Ant, Make, Rake, framework dependent script (e.g. Grails) or any custom shell script you can write. Depending on your technological stack you can have restrictions for the build tool and, as a consequence, you will also have restrictions on the hosted solution you can choose.

Technological stack independence

Written in Java and using Maven or Ant as its preferred build tool, Jenkins is associated with the Java technological stack, although this is not correct. Jenkins can also launch any executable script so that you can use it to build any artifact and integrate the build results in the console and reports offered by Jenkins as part of the building process. There are plenty of examples showing how to use Maven to build projects written in C/C++, PHP or Javascript.

But whichever your technological stack, you should look for the less restrictive CI tool or you can put an obstacle to your adaptation possibilities in the future. You never know if you will need to change your technological stack next year.

Vendor lock-in

Vendor lock-in means that you cannot easily change your provider without incurring in an excessive cost. If the service you are hiring is based on industry standards or de facto standards (like Jenkins), then you are in a better situation to move from one vendor to another. Of course this also applies to hosted CI services.

SCM (Source Control System) or VCS (Version Control System)

The input for every IC system is the SCM. Changes in the source code or configuration files are usually used to stimulate the start of the build process. Today it is one of the first components of the software development ecosystem that companies are moving to a SaaS formula. It is very common nowadays using Github, Bitbucket, etc. But this is not new, developers used to contribute to open source projects are very familiar with Sourceforge, Codehaus or similar where everybody shares a Subversion (SVN) instance.

Some of the hosted IC solutions provide their own SCM so you can start working faster. But you have to consider if they let you configure your own external SCM. This can be blocking for your whole ecosystem.

Cost

Money is important, very important, specially if we are talking about SaaS. You need to have an estimation of how much would this service cost for a start and as the business goes. In this case, the business of a continuous integration system is:

  • to retrieve the changes from a SCM,
  • to build all the required artifacts (executables, metrics, documents, reports…),
  • to keep the artifacts in an artifact repository (depeding on your release management policy),
  • sometimes also running tests,
  • and sometimes even deploying the artifacts in a number of environments (including production)

Depending on the provider you select, for the same profile, the cost can be very different. Some hosted solutions can offer flat rates, but it is not usual, so you have to worry about the differences and how they affect to you. It’s very common that these services are free for open source projects.

  • Fee: You have to pay even if you are not using the system. Your accountant would say this is a fixed cost.
  • Price: You pay depending on the use. There are different policies, like CPU-cycles, agents/workers, etc. All of them have in common that they increase the cost as the system grows. Your accountant would say this is a variable cost. Be specially conscious of the cost depending on the kind of task because slow tasks (e.g. building the documentation) that usually grow very fast (in terms of CPU time and storage) can be extremely expensive in the long term if you are paying for the time of use.

More criteria

We have presented the more relevant aspects you should consider when choosing a hosted CI service, but there are more and, depending on your peculiarities, they can be critical.

Configuration flexibility

Not all the hosted CI services are offering Jenkins and not all of them let you choose the plugins. You should also have to consider the freedom and flexibility you are having.

One of the most popular plugins ever made for Jenkins is that one showing the result of the build using a picture of Chuck Norris. Motivation and humour are important for a healthy development team. But who decides which are the plugins you can use in your hosted CI is the vendor. It has to be like that because they have to guarantee the service for all their customers. Remember that you are paying for the use of the software: SaaS. Before choosing your hosted CI, you should check that you are allowed to run all the plugins you need or introduce any other peculiarity.

Notifications

We want to know the result of the build tasks. Sometimes we are happy with an RSS channel, but sometimes we want more, like emails to everyone involved in the changes for that build (according to the SCM log), or to everyone in the department to stop introducing more changes until the build is fixed, etc. The more flexible the better for you. It shouldn’t impact in the cost but can be very expensive in terms of time and effort of your team trying to adapt your process to the tool. This is particular case of the previous criteria.

Artifacts repository

Your build process produces artifacts that you could keep to feed other build processes or simply to have them versioned. Having an artifact repository can help you to reduce your bill by reducing the time of looking for dependencies and the storage space by avoiding redundancies.

Some examples

Now we list in alphabetical order some of the more significant providers of a continuous integration service. Please, if you know more examples or do you want us to include or change anything on it, just let a comment in this post. We will be very pleased to keep this list as up-to-date as possible.

Based on Jenkins as the CI tool

  • Clinker: private and dedicated internal SVN repository, supports external SVN and Git repositories, private and dedicated internal Maven artifacts repository (Nexus), supports Ant and Maven, pricing plans are flat rates, focused on Java, PHP and Javascript projects
  • CloudBees: free for open source projects, private and internal SVN and Git repositories, private and internal Maven artifacts repository, supports Ant and Maven, pricing plans based on the number of users, build time, storage and a fee
  • FaZend: free (no plans), private and internal SVN repository, supports Ant and Maven, focused on Java and PHP projects
  • Hosted CI: supports external SVN and Git repositories, focused on iOS and Mac projects in general, pricing plans based on number of users, build time and storage
  • Shining Panda: free for open source projects, no pricing plans, the price is based on a build time depending on the platform (Linux or Windows) plus a fee, focused on Java and Python projects, excelent build environment, supports external Mercurial, Git, CVS, SVN and Bazaar repositories

Based on custom or proprietary CI tools

  • Bamboo: free for open source projects, pricing based on agents, supports Ant, Maven and Grails, does not provide a building environment, supports external CVS, Git, Mercurial, Perforce and Subversion repositories
  • Circle CI: projects described with YAML, very integrated with Github, supports Bundle, focused on Ruby projects
  • Continuous.io: free for open source projects, pricing plans based on number of executions, number of collaborators (participants) and projects, focused on Python and Ruby projects, also support Java, Clojure, Node.js and PHP, custom build tool, requires an Amazon EC2 account, supports external Git repositories
  • Semaphore: pricing plans based on the number of projects, focused on Ruby projects, based on Rake as the build tool, supports external Git repositories (focused on Github)
  • Travis CI: projects described with YAML, offers a library of build environments managed with Chef therefore it supports most of the technological stacks, free (no plans excepting a PRO option for personal support)

Conclusions

When choosing a hosted solution for your continous integration you shouldn’t forget to consider the following variables:

  • technological stack (including language, libraries, runtimes, build tools, etc)
  • integration with your own ecosystem (including other hosted services your team is already using, like Github, etc)
  • cost (fees, agents, process time…)
  • vendor lock-in avoidance

Obviously we recommend Clinker because it is our pretty son, but also because it is a full ecosystem ready-to-use for almost every software development team you can find. Based on a pluggable architecture, Clinker can be adapted very easily to your own existing infrastructure too and offers competitive price plans, beginning by downloading the Virtual Appliance for free, and a very low learning curve because it is based on the most used open source tools, like Jenkins, Maven, etc.

We hope this comparative being helpful for you. Our plans are to keep it updated, but if you disagree or have any suggestion (e.g. you know any solution that we should include), please leave us your comment here or by twitter @clinkerhq. We will be very happy to know your opinion.