The pace of the industry today is pressuring software developers to build, test, and release software more frequently than ever. To achieve this pace, teams have built two core processes into their workflow: Continuous Integration and Continuous Deployment.

Continuous Integration and Continuous Deployment (CI/CD) is a subset of the DevOps workflow that automates application code management and its safe, predictable shipping. In this article, we’ll take a deep dive into CI/CD. We’ll look at the individual roles of Continuous Integration and Continuous Deployment as part of the software development life cycle, and we’ll examine different components and stages.

What Is Continuous Integration (CI)?

Continuous Integration (CI) is an automated process whereby software developers integrate new code into existing code. This process involves merging code changes into the application’s repository, and then running builds and tests to check the correctness and functionality of the updated application.

Why CI?

There are several reasons why CI is necessary for modern application development.

Fewer bugs

Before CI, the process of merging and testing new code was more cumbersome, so engineering teams executed this process less frequently. As teams merged code less frequently, the number of code changes that needed merging would grow, typically leading to substantially large pieces of code that needed integration. The introduction of such large pieces of code inevitably led to undetected bugs, and these bugs ultimately made their way into the production environment.

Because CI simplifies the task of integrating code, software developers can push and test code more frequently—and in smaller chunks—which reduces the risk of introducing bugs.

Faster code integration

With CI, software developers can create and merge pull requests more efficiently. When software development teams work on different features in multiple branches, the CI process can work alongside source control management tools to resolve code conflicts.

Automated testing

CI makes it easy for software developers to create automated testing plans and associated scripts. This saves a lot of time for software testers. Such testing can be updated and enhanced in the CI process as testing matures.

Compatible libraries

With the more frequent running of the CI process, the application source code is constantly tested for successful integration and the absence of bugs. This frequency of integration helps to ensure third-party dependencies, with their own frequent updating, have been checked for vulnerabilities and continued compatibility.

The CI Process

Generally, the CI process consists of several steps.

Create a pull request for the new code changes

A software developer writes code and pushes these updates from the local branch to the development branch of the central repository. The software developer then creates a pull request, alerting the team of the intention to merge the development branch into the main branch.

Code review

Typically, the next step in the process is for a senior member of the development team to review the code changes in the development branch. That team member then either approves or rejects the pull request. When rejecting the pull request, the reviewer usually specifies a reason.

Build

The approval of a pull request triggers an automated process for building the application with the new code included. The result of this build process is some sort of build artifact. That artifact might be a zip file containing the updated application, a compiled binary, or some other form altogether.

Test

Next, a series of automated tests are run, validating the correct functionality of the newly built application.

Merge

If the build succeeds and all of the tests pass, then the development branch code merges into the main branch.

What Is Continuous Deployment (CD)?

Continuous Deployment and Continuous Delivery both use the abbreviation, CD. However, when we use CD in this article, we are referring to Continuous Deployment (CD). After we cover Continuous Deployment, we’ll clarify how it differs from Continuous Delivery.

CD is an automated software release process that picks up immediately where the CI process leaves off. The output of a successful CI process—a tested and verified application with code changes merged in—is the input for Continuous Deployment.

Why CD?

CD empowers software development teams to build, test, and deploy code changes with confidence. With the bulk of the CI process using automation, developers can be confident that the new application has been thoroughly tested for bugs and vulnerabilities. The CD process, also using automation, ensures that the updated application is properly deployed to production and made available for users.

The codification and automation of CD prevent configuration steps from being forgotten or environment settings from getting mistyped. The result is high confidence that the application will be deployed seamlessly and less reliance on a specific team member to perform the deployment.

Also, as organizations must constantly innovate to stay competitive, CD largely eliminates the time lost in manual software testing or configuration. These savings allow for iterative development and a faster time to market.

The CD Process

Generally, the CD process consists of five key steps.

Step 1: Pre-production deployment

The CD pipeline takes the build artifact resulting from a successful CI pipeline run, and then deploys that artifact to a pre-production (also known as “staging”) environment.

Step 2: Pre-production tests

In this phase, the CD pipeline runs a suite of automated tests on the pre-production environment to verify that the application is working as expected. This is also the stage where checks for security vulnerabilities and compliance may be run.

Step 3: Production deployment

If all of the pre-production tests pass successfully, the CD pipeline deploys the application to the production environment. This deployment step typically involves:

• Provisioning of resources
• Installation of the updated application
• Installation of any application dependencies
• Configuration of the application environment

Before CD, these tasks were performed manually, and this was time-consuming and error-prone.

Step 4: Production tests

In this step, the CD pipeline tests the application in the production environment. Because the updated application was tested in the CI pipeline and again in the pre-production environment, some DevOps teams see this production testing step as optional. However, this step is crucial for catching errors that may not have existed in the pre-production environments, due to differences in infrastructure or configuration settings.

Step 5: Monitoring and feedback

This final step involves constantly monitoring the health and stability of the application and gathering user feedback. Such feedback forms the basis of the next iteration of the development cycle. It’s worth noting that some DevOps teams see these “Day 2” concerns as their own processes, separate from the CD pipeline.

Continuous Delivery vs Continuous Deployment

In our discussion so far, we have considered the Continuous Integration process, which results in a build artifact, and a Continuous Deployment process, which deploys that artifact to production.

Continuous Delivery takes the build artifact from CI, deploys and tests it in a pre-production environment, and then waits for manual approval before deploying the application to production. Why might there be a need for manual approval? Typically, this gives time and space to perform manual testing—whether by Quality Assurance (QA) engineers or others—before deploying an application to production.

On this distinction, Martin Fowler tweeted:

The difference between Continuous Delivery and Continuous Deployment is that Continuous Delivery requires manual testing and explicit approval for deployment, whereas Continuous Deployment automates the whole process.

The Role of CI/CD in Software Development

CI/CD has removed many of the pain points that were present in monolithic application development and big-bang deployments. 

Improved Code Management

With CI/CD, different teams can simultaneously work from the same codebase using Git branches. The code repository and the CI tool manage each team’s changes, so it’s easy for development leads to see the presence and status of tasks in progress.

For example, tech leads and reviewers can look at the Git repository to see the current branches and the last time the code was pushed into the branch. They can compare that branch’s code with the main branch. Similarly, they can look at previous runs of a CI pipeline and see any errors that occurred.

No Code Conflicts

With big-bang deployments (going from nothing to a whole lot of everything all at once), software developers could overwrite each other’s work when checking in code. A CI/CD pipeline mitigates this risk by enforcing step-wise deployments, a method that minimizes siloed development and eliminates code conflicts.

Automation

Automation tools in the CI pipeline help verify that the checked-in code adheres to specified coding guidelines, thereby improving the overall code standard. Both CI and CD largely automate functional and integration testing across different environments. This encourages teams to write more tests and create other in-depth checks. By automating builds, tests, and releases, CI/CD prevents risks associated with manual code handling. 

Increased Productivity

The CI/CD process aligns with the agile development methodology, which prioritizes iterative development and the fast delivery of code. CI/CD enables multiple teams to work in parallel on existing feature improvements, new features, or bug fixes. This improves the teams’ overall throughput and productivity. 

Automated Software Versioning

In multi-team setups without strong conventions, code tags can cause significant confusion. For example, Team A builds a software feature, ships it, tags it as v3, and then immediately begins working on v4 of the entire application. Around the same time, Team B ships their software feature. After checking the codebase, they tag their new version as v4. When Team A ships their newly developed feature also as v4, this collision will create confusion about deployed code versions.

CI/CD tools have special ways to prevent conflicts when naming deployment versions. With CI/CD, no team needs to manually tag software deployments with a version number.

Easier Rollbacks

Most CI/CD pipelines are designed so that any failure during code integration or deployment results in a graceful rollback, reverting the source code repository (for CI) or the target environment (for CD) to their original states. 

What Is a CI/CD Pipeline?

Up to this point, we have used the terms “CI/CD process” and “CI/CD pipeline” synonymously. Technically, a CI/CD pipeline is a software-based workflow that automates software source code integration and application deployment. Because the CI/CD pipeline closely resembles the CI and CD processes we have discussed so far, many practitioners use “process” and “pipeline” interchangeably.

You can trigger the CI/CD pipeline at different points. One team may configure a pipeline to start immediately after a pull request is approved and the code merges into the main branch. Another team may trigger the pipeline on the development branch immediately after a pull request is opened.

CI/CD Pipeline Components

CI/CD pipelines are usually written in YAML and contain explicit commands for each action. However, some tools have made it easy to create pipelines with graphical UIs, where developers can fill fields in a configuration form. Behind the scenes, CI/CD tools typically generate the pipeline code in YAML.

A CI/CD pipeline can comprise jobs, stages, and tasks.

• A stage is a major division in a pipeline that can consist of multiple jobs or tasks.
• A job represents an action performed in the pipeline. It can be defined on its own or as a child of a stage.
• A task is an abstracted, reusable packaged script with a set of inputs, typically provided by the developer. An example can be a task that installs scripts on a machine depending on the subnet it’s located in.

The image below shows a sample pipeline architecture.

Here’s a sample YAML file that breaks down this workflow into separate components.

stages:
- stage: First Stage
  jobs:
  - job: echo "Hello world"
  - job: echo "Hello CI/CD"

- stage: Last Stage
  jobs:
  - job: echo "Hello last stage"
  - job: echo "Goodbye CI/CD"

steps:
- task: UsePythonVersion@0
  inputs:
    versionSpec: '3.7'
    architecture: 'x64'

This CI/CD pipeline has two stages, four jobs, and one task. The stages keyword states that the file would contain many stages. The code snippet above shows two stages: “First Stage” and “Last Stage.” The First Stage consists of two jobs, which show the messages “Hello world” and “Hello CI/CD” to the user. 

Like stages, the jobs keyword states that the section will contain multiple jobs. A job is declared with the job keyword. 

- stage: First Stage
  jobs:
  - job: echo "Hello world"
  - job: echo "Hello CI/CD"

Similar to the First Stage, the Last Stage also contains two jobs that print text messages.

- stage: Last Stage
  jobs:
  - job: echo "Hello last stage"
  - job: echo "Goodbye CI/CD"

Finally, there’s also a task to install python3.7.

- task: UsePythonVersion@0
  inputs:
    versionSpec: '3.7'
    architecture: 'x64'

There are no dependencies between jobs and tasks. A developer can create a job without any tasks by providing a bash script the job will run. However, there are other complex use cases where using tasks would save you time.

Types of CI/CD Pipelines

There are different ways to create and customize CI/CD pipelines.

Basic pipeline

A basic CI/CD pipeline runs simple workflows that don’t need customization. This type of pipeline runs a series of independent jobs sequentially.

Directed acyclic graph (DAG) pipeline

DAGs allow you to define relationships between jobs in your CI/CD pipeline. Using some checks and logic, you can choose to run some related jobs in sequence and other jobs in parallel. This helps the pipeline run more efficiently and quickly. The image below shows a sample DAG workflow.

Here, Group A jobs are related to one another, while Group B jobs are also related to one another. After the CI/CD pipeline runs Other jobs, Group A and B can start running in parallel. However, the jobs in each group will run sequentially.

Merge request pipeline

The merge request CI/CD pipeline runs when you create a new merge request (pull request) into the main branch. It can also run multiple times during the lifespan of the merge request. Other scenarios that trigger this pipeline include:

• When you push a new commit, and the merge request is still open.
• When you manually trigger the pipeline for the merge request.

Merged results pipeline

A merged results pipeline is a special kind of merge request pipeline. However, the merged results pipeline runs against the results of the main branch and the merged source code. For example, immediately after a pull request is merged into the main branch, this pipeline runs on the main branch, with the newly merged code version as its input.

Merge train

A merge train is a queue of merge requests waiting to merge into the code repository’s main branch. A train can have many merge requests, with each merge request running its merge results pipeline. Each pipeline will include the merge request changes that are in front of it in the train. Merge trains prevent obsolete changes from entering the main branch and breaking the software. Ultimately, this approach ensures that only successfully merged result pipelines make it to the production environment.

How merge trains work:

Let’s assume a merge train has three merge requests (A1, A2, and A3). It will create three parallel merged results pipelines.

• The first pipeline runs on the changes of A1 combined with the main branch.
• The second pipeline runs on the changes of A1 and A2 combined with the main branch.
• The third pipeline runs on the changes of A1, A2, and A3 combined with the main branch.

If the A2 pipeline fails, it leaves the train. The A3 pipeline still continues, but only with the changes for A1 and A3. If A1 finishes successfully, it merges into the main branch, and A3 runs. If more merge requests get into the train, they will contain the A1 changes already in the main branch, and the A3 merge request changes already in the merge train.

Parent-child pipeline

As a CI/CD pipeline grows, some challenges can arise:

• Referencing similar jobs multiple times in the same CI/CD pipeline increases the possibility of bugs reaching production. When a dependent job is changed, developers must remember to update every reference to it in the pipeline. 
• Multiple subcomponents, referenced jobs, and tests can result in a bloated pipeline that is difficult to maintain.
• The pipeline often needs to be rewritten for speed optimization.

With parent-child pipelines, software developers can split a large CI/CD pipeline into reusable templates. This makes configurations more readable, and developers can easily understand and manage the workflow.

Multi-project pipelines

CI/CD pipelines can be set up across multiple projects so that when a pipeline in one project starts, it triggers another pipeline in a different project.

Stages in CI/CD

We have already discussed individual processes within CI and CD separately. A fully automated CI/CD pipeline, therefore, will have the following broad stages:

Stage 1: Source Control 

After software developers write code, they use Git commands to push the code into the development branch of the code repository. Then, they open a pull request. The pull request is reviewed by one or more tech leads. If the change is approved, the pipeline merges the development branch into the main branch.

Stage 2: Building the Code

In this phase, the pipeline packages the code for deployment. This may include building the binaries (or executables) from the source code and their dependencies.

Stage 3: Automated Testing and Validation

The CI/CD pipeline runs automated unit and integration tests against the built artifact from the previous stage. If tests fail, the main branch is reverted to the original code. This allows the developer to create a quick fix and deploy it before the tests are performed again. When all the tests are successful, the workflow moves to the next stage.

Stage 4: Deploying to Pre-production

The CI/CD pipeline connects to a pre-production environment and deploys the newly built (and tested) code. 

Stage 5: More Automated Testing and Validation

The pipeline runs a final set of end-to-end tests against the application in the pre-production environment, ensuring the application is working as expected.

Stage 6: Deploying to Production

After all the tests run successfully, this step makes the application changes live in the production environment.

Conclusion

CI/CD is the broad term for a workflow model that allows software teams to build, test, and ship code in a fast and reliable manner. 

Kong offers an industry-best API Gateway for modern, microservices-based, distributed applications. Software developers and DevOps engineers often need to deploy and configure Kong and its plugins in different environments. CI/CD processes can help them create deployment and configuration scripts for Kong. And with tooling like decK and inso, you can easily integrate them into your existing CI/CD tool chain to deploy and manage Kong easily. These can then be tested, and later applied, both safely and predictably. This prevents the need for post-deployment manual adjustments. It also eliminates the possibility of configuration drift between environments. 

To see how you can use Kong products and integrate them into your API-centric development work, book a personalized demo today.