Getting Started With Continuous Delivery

Posted on Posted in Continuous Delivery, development processes, quality assurance

Basically, Continuous Delivery is simply a clever idea to make software development more simple, more reliable and more efficient. Sounds like simply tweaking the development process here and there, doesn’t it? But when you’re an experienced developer starting with Continuous Delivery, you’ll be surprised to meet a whole bunch of new tools you’ve never heard of. I found this pretty confusing: CD is a development process, but when I asked how to employ CD in my company, all of sudden people started to talk about tools. Worse, you’ll even learn new vocabulary, and you’ll have to acquire a lot of knowledge that’s previously been the exclusive domain of your operations department.

On second thought, that’s not surprising: One of the key ideas of Continuous Delivery is to make developers and operations collaborate more closely (an idea commonly known as “devops”). Operations departments have always been using their own tools, and will continue to do so for good reasons, so developers have to make themselves familiar with the tools of the operations department. In particular, they have to get familiar to Unix shells, which is quite a chunk to swallow if you’re a visually oriented developer like me.

Adopting Continuous Delivery has at least three challenges:

  • The development process changes.
  • The cultural shift. Both developers and operations have to learn about each others job, and they have to do tasks they could previously delegate to “the other department”.
  • Your team has to install and use several tool they wouldn’t need without CD. Expect to invest in courses and trainings.
    • This article presents some of the basic ideas of Continuous Delivery, some of the tools used for Continuous Delivery and – more important – why they’re used. Because – to quote Ana M. Del Carmen GarcĂ­a Oterino – it’s tempting to sell CD tools and to tell you that’s all you need to do Continuous Delivery – but that’s only a start. Continuous Delivery, she continues, is a big word. Treat it with respect.

      Strict quality gates

      Let’s start with a very important message to your manager: Continuous Delivery is about being able to publish the developers branch of your project at any give time – without sacrificing quality. Quite the contrary, Continuous Delivers adds strict quality gates to your project, so most likely the quality of your software becomes better by switching to CD. There’s another reason why quality gets better: releasing software to productions is a skill that needs to be trained. Publish new versions of your software every week, and you gather so much experience that everything goes smoothly. Publish a new version of your software once a year, and you run into all kinds of trouble. Some time ago I’ve heard about a company that didn’t upgrade their software for two years. When they were forced to update their software in the end, the entire development stopped for months because every developer was needed to help with the update. Frequent updates eliminate this kind of expensive stand-stills completely, thus reducing the risk of releasing new versions considerably.

      The runtime environment is part of your program

      Companies selling application servers told us for years that it’s possible to consider web applications and application servers as two separate things. The sad truth is, that this is nonsense. Almost every web application depends on the configuration of the application server. Even worse, it also depends on the libraries installed in the container. As a consequence, most companies abandoned the old idea of deploying multiple web applications on a single application server. Instead, each instance of an application server runs only a single Java application, or even a single microservice. This policy requires many application servers. But that’s not a big deal these days: virtual machines have made this approach cheap and simple.

      Continuous Delivers follows this path to its logical end. Instead of delivering a war file developers deliver a virtual machine running the application.

      That’s where the new tool chain of CD comes into play. Most of these tools allow you to create, maintain and distribute virtual machines more efficiently and – more important – in a reproducible way. Simplifying things a bit, you put the virtual machines into your Git repository. However, Git isn’t optimized for versioning fully equipped virtual machines, so in reality Git stores small scripts describing how to build a virtual machine. This also makes it easier to update parts of the infrastructure.

      What’s the difference to Continuous Integration?

      At first glance, Continuous Delivery is simply Continuous Integration further developed. Like Continuous Delivery, Continuous Integration adds a quality gate to the development process. After every commit, the code is compiled and the unit test are run. If either compilation or tests fail, the commit is rejected (usually by sending the developer a friendly mail asking them to fix their code). Continuous Delivery adds automated acceptance tests running on the target system. Obviously, you can’t run these tests on the production servers, but what you can do is to run the code on a virtual machine and push the tested virtual machine to production later. Such a virtual machine consists of the operating system, the JVM, the database engine (and sometimes even the database data), the application server and of course the application itself.

      As I mentioned above, instead of putting the entire virtual machine under version control there’s a small file (or set of files) describing the virtual machine meticulously.

      Setting up a virtual machine to test an application

      In the CD community, there’s a popular pair of tools to define and set up a virtual machine which is used for tests:

      • Vagrant allows you to configure and run a virtual machine based on a script.
      • Ansible allows you to install software in a virtual machine run by Vagrant.

      At this point, introductory courses tend to become very technical. They start to tell you how to install and use the tools. I’d prefer to provide you with some easily readable article which gives you an impression what the tools do. I’m going to write about the gory details in follow-up articles. Nonetheless, I’ll show you what a (simplified) script to define a Tomcat environment looks like. First, we use Vagrant to set up a virtual machine:

      VAGRANTFILE_API_VERSION = "2"
       
      Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
        config.vm.box = "mango-showcase"
        config.vm.box_url = 
               "https://github.com/kraksoft/vagrant-box-ubuntu/releases...
                        /download/4.10/ubuntu-14.10-amd64.box"
        config.vm.network :forwarded_port, guest: 8080, host: 8080
       
        config.vm.provision "ansible" do |ansible|
          ansible.playbook = "playbook.yml"
        end
       end
      

      This is a simplified version of the Vagrant file I found at http://pelle.io.

      This script tells Vagrant to download a particular Ubuntu version and to expose the default port of Tomcat – which is going to run within the virtual machine – to the host (which is your desktop PC). In other words, it allows you to access the address http://localhost:8080 from the Internet Explorer of your Windows PC. The request is forwarded to the Tomcat running on Linux with the virtual machine.

      The next three lines of the script deal with “provisioning” – in other words, with installing software on the Ubuntu VM. It’s possible to do this in the Vagrant file, but for some reason, it’s better to run an Ansible script playbook.yml:

      - name: install tomcat8
            apt: name=tomcat8 state=present
      

      Now all that’s needed to build and start the virtual machine and Tomcat is to type vagrant up on the command line.

      Of course, that’s simply a default Tomcat. However, checking out your source code from a repository, compiling it via Maven or Gradle and installing it to Tomcat, is only a matter of a couple of additional lines to the playbook.yml.

      The nice thing is, that Vagrant and Ansible give you a really clean Tomcat installation. There’s no long history of changes. Every modification has to be described in one of the script files. This is particularly useful when it comes to updating parts of the infrastructure. To switch to another version of Ubuntu – or another Linux distribution, for that matter – you only have to change a single line of the Vagrant file. Similarly, installing another JDK or another version of Tomcat is simply another entry in the playbook.yml. To install this, you get rid of the old Vagrant VM by typing vagrant destroy on the command line, followed by a vagrant up. Usually that’s only a matter of minutes. Compare that to the tedious process of installing another version of the software on your desktop PC! Plus, on the desktop PC there’s always the chance that there are remainders from previous installations. There’s no such pollution in a virtual machines provided by Vagrant.

      Deploying the production environment

      By now we’ve got everything we need to pull your application from the repository, build and configure it in a reproducible way and to run it in production. It’s possible to use Vagrant in production. However, hardly anybody does so. Most web servers run on Linux, and Linux already contains a better tool to run virtual machines: Docker. The advantage of Docker is that it uses far less resources than a virtual machine run by Vagrant.

      So we’re back to square one. We need to learn about Docker, and figure out how to install software on Docker.

      That’s where the Ansible playbook comes in handy: Ansible can also install software in a Docker container. Basically, we can use the same script we’ve already used in the test environment.

      So why do we need Vagrant at all? Well, Docker runs exclusively on Linux. Developers using a Windows PC or a Mac can’t use Docker. To start a virtual machine we need to install software like VMWare or VirtualBox, and we need Vagrant to set up virtual machines using a script. If you’re developing on Linux, you could probably do without Vagrant.

      However, Vagrant has special support for Docker. For one, you can use Vagrant to run and configure a Docker image (which in turn is more or less the same as a virtual machine). Second, Vagrant recognized if it’s running on Linux. In this case, it doesn’t start a virtual machine to run a Docker image, but uses the lightweight Linux way to start it.

      What about the development process?

      By now, we’ve got everything we need to get started with Continuous Delivery. We can build both testing and production environments using automated scripts. But when we try to use our Continuous Integration tools like Jenkins to install Continuous Deployment, we’re in for another surprise. Jenkins is not up to the task. That’s because CD’s development process consists of several steps, which need to be automated. Jenkins only covers the first of these steps. Dedicated tools like Thoughtworks Go allow you to define complex build pipelines.

      Such a complex build pipeline can integrate various build artefacts into an application. Just think of microservices. Your application consists of a collection of microservices, which all have to work correctly both individually and with each other. Before pushing a new version of a microservice into production, you have to run an integration test. If you’ve got an automated integration test, you can start it by a script. Managing such a complex build pipeline – which pretty soon becomes a build mesh – is beyond the scope of Jenkins.

      Quality gates

      I’d like to conclude the article with an example of the quality gates I already mentioned at the beginning. Remember, CD is a process that allows you to push code to production at any given time. Rumors have it that companies like Flicker push up to six releases to production every day. This can only be achieved by automated testing. In most cases, Continuous Delivery does not imply that every commit is pushed automatically to production. Usually there’s a quality gateway that requires manual testing. Wikipedia has such an example:
      Continuous Delivery process diagram by Jez Humble, put under a CC BY-SA 1.0 license
      Continuous Delivery process diagram by Jez Humble, put under a CC BY-SA 1.0 license
      The green bars are successfully passed quality gates, where the red bar are quality gates that rejected a particular change. This is a simple example, so it doesn’t differ much from traditional processes, but there’s a strong focus on adding automated tests in order to build trust to the release.

      Feature toggles

      Talking of trust, there’s a surprising change to the development process: Continuous Delivers discourages branches. Instead, it prefers feature toggles. The rationale behind this is that feature toggles can be activated by the automated tests. So a feature is tested much earlier than if it’s developed in a separate branch, which usually is ignored by your Continuous Integration and Continuous Delivery tools.

      Wrapping it up

      The first contact to Continuous Delivery can be pretty confusing. The small change in the development process, the simple goal to be able to publish the current developer release at any given time, has a lot of consequences:

      • Developers have to learn a lot about the tools of the operations department. They even have to use them on a daily basis.
      • The runtime environment should be considered part of the application. In theory, CD doesn’t require this, but the build-test-release process becomes a lot more reliable.
      • This, in turn, requires tools that are able to build, configure, manage and run such an environment. Keywords are Vagrant, Ansible and Docker.
      • You need to invest a lot into test automation.
      • The build-test-deploy process needs to be managed by tools. Again, that’s not necessary in theory, but it makes things much more reliable.

2 thoughts on “Getting Started With Continuous Delivery

  1. Thanks for sharing the detailed information,great stuff please share more article like this in future

    1. Well, well, well. If you’re flooding my blog with unwanted spam, please do it in an interesting way. And follow the grammar rules of the English language. It goes without saying that I’ve deleted the URL in your comment. BeyondJava is not a commercial site, and it will never be.

Leave a Reply

Your email address will not be published.