Project
Continuous Delivery

Continuous Delivery

Continuous delivery goes beyond your software artifacts and must encompass all of your infrastructure on which you build and deploy your software.

In practice, this means having scripted, automated deployment of all IT infrastructure, all environments and all databases and other parts of the infrastructure used by deployment jobs.

There are levels of continuous delivery and by identifying which level you are on, you can determine what areas you need to pay more attention to.

Maturity level

Level 0

No continuous delivery. At this level you have no repeatability in your delivery processes.

  • Build management: at this level manual processes are used for building software. No management of artifacts and reports is done.
  • Deployment environment: manual processes are used to deploy software. Environment specific binaries are produced and environments are configured manually.
  • Release management: releases are infrequent and unreliable.
  • Testing: manual testing after deployment.
  • Data management: migrations are not versioned and are performed manually.
  • Configuration: version control not used for managing configuration of build environments.

Level 1

You have a repeatable, documented and partially automated process.

  • Build management: regular automated builds and testing. Any version can be recreated from source control using automated process.
  • Deployment environment: automated deployment to some environments. Creation of new deployment environments is cheap and easy. All configuration is under version control.
  • Release management: painful and infrequent, but reliable, releases. Limited traceability of requirements to a release.
  • Testing: automated tests written as part of story implementation.
  • Data management: migrations are handled using automated scripts that are versioned as part of the application.
  • Configuration: version control is in use for everything needed to recreate software: source, build scripts, deployment scripts, database migrations.

Level 2

You have automated processes applied across the full application lifecycle.

  • Build management: automated build and test is implemented each time a change is committed. Dependencies are versioned. You have high level of reuse of scripts and tools.
  • Deployment environment: fully automated process that can be triggered with a push of a button. Same process is used to deploy to any environment.
  • Release management: change management and approval process defined and enforced. Regulatory and compliance testing is part of the release process.
  • Testing: automated unit, integration and acceptance tests. Test written as part of daily development.
  • Data management: migrations are handled using automated scripts that are versioned as part of the application.
  • Configuration: Libraries and dependencies managed under version control. Any old version of the configuration can be checked out and software can be rebuilt using tools and dependencies that were specific to that version.

Level 3

You have a fully automated process that is measured and controlled.

  • Build management: build metrics are gathered, made visible and acted on. Builds are never left in a broken state.
  • Deployment environment: deployments fully managed. Release and rollback is fully tested.
  • Release management: target environment and application health is monitored and managed.
  • Testing: quality metrics and trends are tracked. Test and code coverage is tracked. Non-functional requirements are defined and measured.
  • Data management: database updates and rollbacks are tested with every deployment. Database performance is monitored.
  • Configuration: Developers merge new changes to main at least once a day. Long lived branches are only used for releases.

Level 4

You are focusing on process improvement.

  • Build management: teams regularly meet to discuss integration problems and resolve these with automation, faster feedback and better visibility.
  • Deployment environment: all deployment environments are managed effectively. Provisioning of environments is fully automated.
  • Release management: operations and delivery teams regularly collaborate to manage risks and reduce release cycle time.
  • Testing: production rollbacks are rare. Majority of bugs are uncovered and fixed during automated testing. Defects that are found in production are fixed immediately.
  • Data management: database updates fully automated. Release to release performance is excellent.
  • Configuration: regular validation that configuration management policy supports effective collaboration, rapid deployments and auditable change management.

Continuous delivery in larger teams

The most effective way to organize large teams (or teams of teams) for rapid delivery is to separate the larger system into separate components where each component implements all of the requirements that are necessary for continuous delivery.

A component is a piece of software that your application depends on that your team developes. In contrast, libraries are external components that your team does not develop. Libraries are updated infrequently while components are updated much more frequently. A component define functional boundaries where details can be hidden behind a well defined API.

Even though individual classes or C files can be considered to be components as well, usually a component is a collection of classes working together. It is also more practical to define a components as an internal library inside your project.

This definition mandates that your components follow the same set of requirements with respect to unit, integration and acceptance testing as your overall application and it also means that each component can be replaced at any time using another component that implements the same API. This is important because in a large team, this split of the application into components helps a lot with managing complexity of the system.

Best practices

Here is the recap of best practices of continuous delivery.

  • Test automation: A comprehensive test suite with tests that are automated and reliable - meaning that they reliably fail on buggy code and reliably pass code that is ready for production. Make sure that you have 100% coverage for unit tests and that all work is verified by system and integration tests.

  • Deployment automation: Deployments have to be automated and not require any manual automation. Anything that has been accepted automatically by tests into production must be easy to automatically deploy without risk. This of course requires your team to be very thorough at expressing all possible concerns and documenting absence of bugs through tests that run on every build.

  • Trunk based development: Characterized by one main integration branch and only a handful of other branches with very short lifetimes (less than one day) that are only merged into main through the merge request process. Trunk based development helps the team implement continuous integration at repository level.

  • Merge request workflow: All code needs to pass code review and should only merged into main once all concerns were answered. The team needs to be good at automating checks for any concerns identified by other team members so that the review process can be faster next time.

  • Shifting left on security: Integrating security scanning into the design and testing phases of the development cycle. Usage of tools like Valgrind to check for buffer overflows and developing tests specifically for testing security of the application. This will ensure that continuous deployments can be done safely.

  • Loosely coupled applications: An architecture that allows teams to test and deploy their applications on demand without required orchestration with other services or devices. A loosely coupled architecture of the whole network of devices allows your team to work independently without being held back by other parts of the organization.

  • Loosely coupled components: similar to the requirement of loosely coupled application, loosely coupled components help the team organize the complexity of a large system so that components can be refactored without affecting a lot of other code.

  • Continuous integration: applied not just to source code but to the whole system including deployment environments and production data. This process should be automated and should happen as part of every day development.

  • Configuration and database change management: Database and configuration changes do not slow the team down. This can be achieved by storing any database changes as scripts in version control and avoiding doing any changes to the database manually.

  • Code maintainability: It is easy for developers to upgrade, change and extend existing code and find necessary examples for adding new functionality to the code base. Design patterns are very important practice that improves maintainability and gives the team a set of patterns to duplicate.

Tools

  • Puppet: this is an enterprise level software configuration management tool which includes its own declarative language to describe system configuration.
  • Ansible: this is an alternative deployment automation tool written in python and is more lightweight than puppet but is not as fully featured.
Martin SchröderMartin Schröder
16 years  of experience

About the author

Martin is a full-stack expert in embedded systems, data science, firmware development, TDD, BDD, and DevOps. Martin serves as owner and co-founder of Swedish Embedded Consulting.

Expertise

Embedded Firmware
Zephyr RTOS
Scrum
Continuous Delivery

Contact Martin

By completing signup, you are agreeing to Swedish Embedded's Terms of Service and Privacy Policy and that Swedish Embedded may use the supplied information to contact you until you choose to opt out.

Confirm your contact information

Thank you!

An email has been sent to you with a link to book a short call where we can discuss your project further. If you have any further questions, please send an email to info@swedishembedded.com