A Deployment pipeline is an important concept in Continuous Delivery. In Deployment pipeline we break the build process into distinct stages In each stage we get feedback to move into the next stage.
It is a collaborative effort between various groups involved in delivering Agile Software Development.
Often the first state in Deployment pipeline is compiling the code and converting into binaries. After that, we run the automated tests. Depending on the scenario, there are stages like performance testing, security check, usability testing etc in a Deployment pipeline.
In DevOps our aim is to automate all the stages of Deployment pipeline with a smooth running Deployment Pipeline, we can achieve the goal of Continuous Delivery.
At an abstract level, a deployment pipeline is an automated manifestation of your process for getting software from version control into the hands of your users. Every change to your software goes through a complex process on its way to being released. That process involves building the software, followed by its process of these builds through multiple stages of testing and deployment.
Steps in the Deployment Pipeline include
- The deployment pipeline begins when a developer commits code to a joint versioning system.
- Prior to doing this commit, the developer will have performed a series of pre-commit tests on their local environment. The failure of the pre-commit tests of course means that the commit does not take place.
- A commit then triggers an integration build of the service being developed. This build is tested by the integration tests.
- If these tests are successful, the build is promoted to a quasi-production environment – the staging environment.
- Then it is promoted to production under close supervision.
- After another period of close supervision, it is promoted to normal production.
- The specific tasks may vary a bit for different organization, For example, a small company may not have a staging environment or special supervision for recently deployed version. A larger company may have several different production environments for different purposes.
- One way to define continuous integration is to have automatic triggers between one phase and the next, upto integration tests. That is, if the build is successful then integration tests are triggered. If not, the developer responsible for the failure is notified.
- Continuous delivery is defined as having automated triggers as far as the staging system.
- Once a service is deployed into production it is closely monitored for a period and then it is promoted into normal production.
- At this final stage, monitoring and testing still exist but the service is no different from other services in this regard.
- The committed code moves through the different steps described above but the code does not move on its own. Rather, it is moved by tools. The tools are controlled by their programs (scripts) or by developer/operator commands.
- Traceability is of utmost importance in this movement. Traceability means that, for any system in production, it is possible to determine exactly how it came to be in the production. This means keeping track not only of source code but also of all commands to all the tools that acted on the elements of the system. Individual commands are difficult to trace, therefore, it is better to use scripts instead of commands.
- Treating infrastructure-as-code means that the infrastructure-scripts and movement-scripts should be subject to the same quality control as application source code. It should be regulated in same fashion.
A Basic Deployment Pipeline
The above figure shows a typical deployment pipeline and captures the essence of the approach.
- The Process starts with the developers committing changes into their version control At this point, the continuous integration management systems responds to the commit by triggering a new instance of our pipeline. The first (commit) stage of the pipeline
o compiles the code,
o runs unit tests,
o performs code analysis and
o creates installers
- The Second stage is typically composed of longer-running automated acceptance tests. Again your CI server should let you split these tests into suites which can be executed in parallel to increase their speed and give you feedback faster – typically within an hour or two. This stage will be triggered automatically by the successful completion of the first stage in your pipeline.
- At this point, the pipeline branches to enable independent deployment of your build to various environments – in this case, UAT, Capacity testing and Production.
Commit Stage
A new instance of your deployment pipeline is created upon every check-in and, if the first stage passes, results in the creation of a release candidate. The aim of the first stage in the pipeline is to eliminate builds that are unfit for production and signal the team that the application is broken as quickly as possible. We want to expend a minimum of time and effort on a version of the application that is obviously broken. So, when a developer commits a change to the version control system, we want to evaluate the latest version of the application quickly. The developer who checked in then waits for the results before moving on to the next task. There are a few things we want to do as part of the commit stage
- Compile the code (if necessary)
- Run a set of commit tests
- Create binaries for use by the later stages
- Perform analysis of the code to check its health
- Prepare artifacts such as test databases for use by the later stages.
If you only implement a commit stage in your development process, it usually represents an enormous step forward in the reliability and quality of the outputs of your teams. However, there are several more stages necessary to complete what we consider to be a minimal deployment pipeline.
Automated Acceptance Test Gate
A comprehensive commit test suite is an excellent litmus test for many classes of errors, but there is much that it won’t catch. Unit tests, which comprise the vast majority of the commit tests are so coupled to the low level API that it is often hard for the developers to avoid the trap that the solution works in a particular way rather than asserting that it solves a particular problem.
Commit tests that run against every check-in provides us with timely feedback on problems with the latest builds and on bugs in our application. But, without running acceptance tests in a production-like environments, we know nothing about whether the application meets the customer specifications, nor whether it can be deployed and survive in the real world. If we want timely feedback on these topics we must extend the range of our continuous integration process to test and rehearse these aspects of our system too.
The goal of the acceptance test stage is to assert that the system delivers the value the customer is expecting and that it meets the acceptance criteria. The acceptance test stage also serves as regression test suite, verifying that no bugs are introduced into existing behavior by the new changes.
The automated acceptance test gate is the second significant milestone in the lifecycle of a release candidate.
Subsequent Test Stages
The acceptance test stage is a significant milestone in the lifecycle of a release candidate. Once this stage has been completed, a successful release candidate has moved on from something that is largely the domain of the development team to something of wider interest and use.
For the simplest deployment pipelines, a build that has passed acceptance testing is ready for release to users, at least as far as the automated testing of the system is concerned. If the candidate fails this stage, it by definition is not fit to be released.
The progression of the release candidate to this point has been automatic with successful candidates being automatically promoted to the next stage. If you are delivering software incrementally it is possible to have an automated deployment to production. But for many systems, some form of manual testing is desirable before release, even when you have a comprehensive set of automated tests. Many projects have environments for testing integration with other systems, environments for testing capacity, exploratory testing environments and staging/production-like environments. Each of these environments can be more or less production-like and have their own unique configuration.
Preparing to Release
There is a business risk associated with every release of a production system. At best, if there is a serious problem at the point of release, it may delay the introduction of valuable new capabilities. At worst, if there is no sensible back-out plan in place, it may leave the business without mission-critical resources because they had to be decommissioned as part of the release of the new system. The mitigation of these problems is very simple when we view the release step as a natural outcome of our deployment pipeline. Fundamentally we want to
- Have a release plan that is created and maintained by everybody involved in the delivering of the software including developers and testers as well as operations, infrastructure and support personnel.
- Minimize the effect of people making mistakes by automating as much of the process as possible, starting with the most error-prone stages
- Rehearse the procedure often in production-like environments so that you can debug the process as possible, starting with the most error-prone stages
- Rehearse the procedure often in production-like environments so that you can debug the process and the technology supporting it
- Have the ability to back out a release if things don’t go according to plan
- Have a strategy for migrating configuration and production data as part of the upgrade and rollback processes.
Implementing a Deployment Pipeline
Whether you are staring a new project from scratch or trying to create an automated pipeline for an existing system, you should generally take an incremental approach to implementing a deployment pipeline. In general the steps look like this
- Model your value stream and create a walking skeleton
- Automate the build and deployment process
- Automate unit tests and code analysis
- Automate acceptance tests
- Automate releases
Build and Deployment Scripting
One of the core principles of environment management is that changes to testing and production environments should only be made through an automated process. That means that you should not log into such systems remotely or perform deployments. They should always be entirely scripted.
Some of the principle of Build and Deployment Pipeline Scripting are as follows
Create a Script for Each Stage in your Deployment Pipeline
Taking this approach ensures that our scripts have a well-defined structure that helps us to keep them clean during maintenance and minimizes dependencies between components of our build and deployment system. Luckily, the deployment pipeline provides an excellent organizing principle for dividing up responsibilities between build scripts. When you first start your project, it makes sense to have a single script containing every operation that will be performed in the course of executing the deployment pipeline, with dummy targets for steps that are not yet automated. However, once your script gets sufficiently long, you can divide it up into separate scripts for each stage in your pipeline. Thus you will have a commit script containing all the targets required to compile your application, package it, run the commit test suite, and perform static analysis of the code. You then need a functional acceptance test script that calls your deployment tool to deploy the application to the appropriate environment, then prepare any data, and finally runs the acceptance tests.
Use an Appropriate Technology to Deploy Your Application
In a typical deployment pipeline, most stages that follow a successful commit stage, such as the automated acceptance test stage and user acceptance test stage, depend upon the application being deployed to a production-like environment. It is vital that this deployment is automated too. However, you should use the right tool for the job when automating deployment, not a general-purpose scripting language (unless the deployment process is extremely simple).
Use the Same Scripts to Deploy to Every Environment
It is essential to use the same process to deploy to every environment in which your application runs to ensure that the build and deployment process is tested effectively. That means using the same scripts to deploy to each environment and representing the differences between environments—such as service URIs and IP addresses—as configuration information to be managed separately. Separate out configuration information from the script and store it in version control, providing some mechanism for your deployment script to retrieve it. It is essential that both build and deployment scripts work on developers’ machines as well as on production-like environments, and that they are used to perform all build and deployment activities by developers.
Ensure the Deployment Process Is Idempotent
Your deployment process should always leave the target environment in the same (correct) state, regardless of the state it finds it in when starting a deployment. The simplest way to achieve this is to start with a known-good baseline environment, provisioned either automatically or through virtualization. This environment should include all the appropriate middleware and anything else your application requires to work. Your deployment process can then fetch the version of the application you specify and deploy it to this environment, using the appropriate deployment tools for your middleware.
Evolve Your Deployment System Incrementally
Everyone can see the appeal of a fully automated deployment process: “Release your software at the push of a button.” You don’t have to have completed all of the steps to get value from your work. The first time you write a script to deploy the application in a local development environment and share it with the team, you have saved lots of work of individual developers. Start by getting the operations team to work with developers to automate deployment of the application into a testing environment. Make sure that the operations people are comfortable with the tools being used to deploy. Ensure that developers can use the same process to deploy and run the application in their development environments. Then, move on to refining these scripts so they can be used in the acceptance test environment to deploy and run the application so that the tests can be run. Then, move further down the deployment pipeline and ensure the operations team can use the same tools to deploy the application into staging and production.