Calculating..Register to attend

Minimum Requirements for Embedded Build Pipeline

Your CI pipeline determines quality of your code by checking and rejecting potential problems before they get into production firmware.

Minimum Requirements for Embedded Build Pipeline

It is a well known fact that you should always keep your main git trunk in a fully working deployable state. I have covered the reasons for this in great detail in the DevOps Infrastructure Series.

While having a "develop" branch may sound like an attractive idea - it is really a hidden shortcut that you are trying to convince yourself to take in order to avoid making sure that every single commit is fully done the moment you are done with it. Once you are fully done with a ticket, the merge request that solves the ticket must always at all times be in a fully deployable state.

In the devops training we also look at how we deal with partial features: we simply make sure that they are always deployable (in other words as stable as we can get them to be) but they do not have to be enabled for the user if they are a work in progress! But the partial work in progress must always be production quality code.

This is the only way you can fully move on from a task and avoid having a lingering backlog of things you have to "fix" later.

We can achieve this by making sure that our CI system is top notch.

In this insight I’m going to do my best to summarize the minimal set of checks that we currently do on our own repositories and also help our customers implement in their firmware.

This list is by no means exhaustive and does not contain project specific checks which must be included based on project requirements.

This list only contains the generic checks that every single embedded C project should have in place regardless of size.

The Role Of CI

CI stands for "Continuous Integration" however these two words have as many meanings as there are software projects around. So we have to define these two words.

First and foremost continuous integration means that:

Every single piece of work, no matter how small, must be fully completed and integrated into production-ready software in the best possible way without decreasing stability of the software.

When we have a full continuous integration workflow in place, we can say with high degree of certainty what is completed and what is not. The necessity of "going back to fix things later" has to be eliminated at all costs because it is the primary cause of incorrect timeline estimates and delayed releases. When we know that what was done is done (and remains done as more things get done) we can be sure that our backlog represents the full picture of the tasks that are remaining.

CI helps us achieve this by being a companion to developers which can quickly point out when something that was previously done has become broken so that developer can fix the issue and avoid the situation where something has been pushed into production and a problem is discovered afterwards requiring developer to go back and fix it (classic case of "what is done is not really done but rather we hope it’s done).

By working closely with the CI scripts and making sure that all changes to the project are verified by CI we can move away from hoping and move towards actual knowing that our software is in good shape at all times.

What Checks CI Should Do

We can divide the role of CI into several distinct areas:

  • Repository integrity: where we verify actual file system structure of the repository and make sure that all necessary files exist and are placed where they should be.
  • Patch integrity: these checks verify that one whole patch (including all changes to all files) is well formed. This often identifies issues not just in C files but in other files as well since it looks at one full merge request as a single diff.
  • Software integrity: these checks consist of unit, integration and system tests and verify that the software still continues to work after we have made changes to it.
  • Post build integrity: these checks can be used to verify build artifacts (that there is nothing missing from them) after a full build and test run has been completed. It is mainly used to ensure that we have all the reports in place and that they contain sensible data.

For every task in the ticket management system, the workflow is as follows:

  1. Create a new branch for the task off of main branch in git (branch names typically start with ticket number).
  2. Complete the task while pushing changes to this branch.
  3. Make sure that all CI checks pass and that nothing has been forgotten (this is largely handled by CI).
  4. CI checks ensure that all new changes are of production grade quality so once they complete successfully, the changes are squashed and rebased on the main trunk (squashing and rebasing ensures clean, linear history).

The process is then repeated for the next task where the new branch is based off of the new trunk.

Repository Integrity Checks

Every repository has unwritten rules where files should be placed. So long as these rules are "unwritten" you can be sure that they will be violated or simply forgotten. Repository integrity checks ensure that these rules are always adhered with (code review process ensures that these rules are never removed or changed without a discussion).

Here are some examples of repository integrity checking:

The list of most basic checks includes:

  • General directory structure: imposes specific pattern that must be adhered to for each type of directory (doc, include, tests, samples etc). This helps keep directory structure organized.
  • Relationships between sources tests: every C file should have a corresponding unit test with matching name.
  • Documentation: every documentation chapter has specific required sections.

Running all the checks is simple when we are using robot framework:

robot integrity # run all robot files in integrity folder

Patch Integrity Checks

Each time a developer completes work on a task, all the changes are captured in the diff between the ticket branch and main trunk. Since we only have one trunk that represents latest production version of the software, we only have to worry about one patch.

When we look at all changes as a single patch, we can run additional checks that are concerned with the formatting of the patch itself.

These checks include:

  • API Usage: code scans that catch incorrect or undesirable API calls.
  • Comments: verifies that comments have proper formatting (readability).
  • Commit message: if relevant, checks commit message and tags like "Signed-off-by".
  • Indentation and line breaks: making sure that there is no mixed indentation anywhere and other checks related to line spacing.
  • Spelling: running the patch against a list of common spelling errors.
  • Stray spaces and dos line endings: this check minimizes pointless diffs during development (because IDEs and text editors often remove stray spaces automatically causing lines to appear changed when no actual code was changed - which increases code review time for no good reason).
  • Source code formatting: this is doable for most source files including C, Python and CMake files. Consistent code formatting minimizes lines appearing in diff only because their formatting was changed by the IDE. clang-format is configured and integrated into the build process to handle this.
  • File permissions: source files should not have executable permissions (which interestingly can happen when developers mix windows and linux). Other checks include making sure that only scripts folder has executable scripts.

Checking the patch has the advantage of only having to check files that are part of the change set and not have to scan the whole repository. This is often a good optimization compared to the more brute force approach of scanning all files. Example of how this can be done for a range of commits can be found here.

Patch checking should not try to understand the semantic meaning of the code but rather just look at the diff as a single text file and search for patterns that we do not want to see in our project code. This is often easier and faster to maintain than having to maintain a custom version of clang-tidy (clang-tidy checks are much more thorough but are limited to software code only - they do not cover patterns one may see in documentation files outside of the source code).

Software Integrity Checks

In contrast to patch checking, software integrity checking goes deep into the source code. This includes:

  • Static analysis: which must be done with exact compile flags that we use when we are compiling the firmware. It is more complex to integrate but usually gives very good results. Static analysis can find problems that depend on values of variables. A good example of this is conditions that result in overflow of an array. To check this, the static analyzer must parse the code and understand how to solve for the conditions that result in an error. Many of these checks are hard to see by just looking at the code - only a static analyzer that can see the whole call tree has the view necessary to identify many obscure problems.
  • Unit testing with mocks: this approach is used for logical integrity checking of the source code. In order to be effective, unit testing must capture all code paths and must be guided by both code and branch coverage. Unit testing does not replace integration testing because unit testing does not integrate the module with any other modules - it simply mocks all outside code using a combination of test code and automatically generated infrastructure from included header files.
  • Integration testing: these tests must capture use case scenarios of using a module from programmer perspective. We don’t care about user perspective because we are testing an API and we also don’t care about testing all code branches because we are not testing internal logic of the module (that is done by unit testing). In integration testing, we take a module and consider it to be a black box only accessible through its public API. We must then write tests that cover all functions in the public api. Line coverage can be used as a hint that shows what functions have been tested, but in contrast to unit testing there is no hard requirement of 100% branch and line coverage as is the case with unit testing - because coverage in this case is not a precise indicator of how well the api has been tested. A better indicator is "can we quickly and easily use the API in practice and get expected results when using it both in simulation and on real hardware?".
  • System testing: these tests take the perspective of the user and treat the whole firmware image as a black box. This black box has an API: the memory of the processor. Through this API the "black box" is able to interact with the outside world. To test this API we run the production firmware in a simulator and have automated scripts (usually written in robot framework language) interact with code that is designed to simulate behavior of real hardware devices that are accessible to the firmware through memory mapping or through secondary interfaces (like SPI and I2C - which are still memory mapped on CPU level but simulation abstractions make sure we can interact with higher level devices without worrying about details of the low level protocols). Instead of code coverage, system testing uses instructions executed and user story scenario coverage as main metrics of completeness. Once all user requirements have been codified in scripted form, it becomes a matter of verifying them against real firmware running in a simulation in order tell whether they have been correctly implemented or not.

Doing software testing very often leads to substantial improvements in software architecture. The primary reason for this is that programmers are forced to think "how can I test this code?" which leads to less dependencies and cleaner design because dependencies often lead to difficulty in testing.

It is also very common to discover API bugs while testing (particularly while ding in depth unit testing) because these bugs are often located in code branches that rarely run. On example is return values: do we follow a consistent return value policy in our software? Are we returning correct error codes? Are we unlocking resources before we return?

Post-Build Integrity Checks

Post build integrity checks aim at checking the final artifacts produced by the build process. These include:

  • SPDX Software BoM: have we generated a software bom with all required data?
  • Binaries: do we have binaries and do they have correction version data?
  • Packages: have we generated installer packages?
  • Documentation: do we have documentation and does the final PDF/HTML file contain all necessary sections (for example release notes relevant to the current version?).

These checks are easiest to implement in a similar way as directory integrity checks (ie using robot framework and python) but with the only difference that they are executed after a full build and test run has completed.

What have we learned

This quick summary illustrates the basic set of checks that can help you keep bugs at bay and minimize the possibility of problems piling up in the code without knowledge of the developer team.

More importantly these checks keep your main trunk in deployable state at all times. Without checks there is simply no way to tell whether trunk is working or not - which requires expensive "testing" to be done using physical devices. With checks in place, testing is all captured in reproducible way which means that it is cheap to test for every single commit. Which in turn means that trunk is always tested and ready for deployment.

When you have unit and integrity testing in place, you instantly see when some code changes break something outside of the module you are editing.

When you have system testing in place, you always see whether user is affected by the changes you are committing.

With repository integrity checking you make sure that you never forget to add additional files (such as documentation) after you have added a new C module.

With post build integrity checking you never forget to include some important file in the release installer or archive.

You simply have a choice: either you have to implement all of this yourself, or you can let Swedish Embedded Group do it for you!

Getting help

When we take a C code base that does not have any checks in place and we want to modernize it, we need to do it by following a step by step process.

First, we put in place checks that do not require any changes to the logic of the source code. This includes:

  • CI Infrastructure: repository cleanup making sure that all build actions are scripted and executable through CI. Even if your project already has CI infrastructure, it is likely that it needs to be cleaned up and prepared for the addition of all the checks that we will be adding to the repository as a whole. There must be no action at all that is not achievable through CI scripts. The build, the testing, the deployment - it must all be scripted.
  • Infrastructure for testing: allowing unit, integration and system tests to be executed automatically for each merge request. This involves adding necessary CMake code and CI scripts for building and running all tests. It also includes setting up scripts for automatically checking code coverage so that we can later enforce the requirement that all source code must be tested before it gets into trunk (because remember - we want trunk to be deployable at all times!)
  • Code formatting, cleanup and patch checks: this will likely require almost every file to be altered in some way - but usually does not require any substantial changes that affect functionality of the source code. We want to get formatting out of the way so that we can minimize merge conflicts while developers continue working on other tickets unrelated to repository cleanup.

Once we have the build infrastructure in place, the next step is to start cleaning up the source code to eliminate obvious problems with the code itself. This is done without substantial architectural changes and focuses primarily on two things:

  • Static analysis: this often requires changes to the source code and may modify APIs - but the changes are usually localized and do not affect the functionality of the code - only its form.
  • Compiler flags: this is similar to static analysis but focuses on making the most of the functionality provided by the compiler. Compilers often have many useful warning options which are not enabled by default. We enable all of these options and also make sure that compilation always fails if there are any warnings at all. Warnings usually have reasoning behind them and we don’t want any potential bugs to be allowed only because we let warnings through without aborting compilation.

After this, the code is good shape to start refactoring and testing it. This is also split into multiple steps because we want to avoid breaking the code while we are refactoring it.

  • High level integration tests: we start by making sure that we can build the main application as part of a test that runs on posix platform. This test will run simple checks against the main application to make sure it boots and enters into correct state after startup. But more importantly, this step requires us to do the necessary refactoring to separate application code from HAL layer. It is very important to be able to build the source code for posix platform because all unit tests will run natively on the CI machine and many integration tests as well (although integration tests are not limited to posix platform and can run on hardware as well).
  • Integration testing of application modules: we then proceed "downwards" and start with integration testing of existing modules. This usually requires refactoring in the form of making sure that modules use object oriented approach, that they can be built relatively separated from other components and that they continue to work as we go deeper into the code. Integration testing is much easier to quickly setup and does not require us to go deep into the code. We do not want to start with unit testing because all of the refactoring that will likely be necessary as testing progresses would also mean that we would have to rewrite unit tests - this is why a top down approach of integration testing first is much better.
  • Unit testing of source code logic*: once integration testing is in place, we are in a good position to start refactoring and verifying application logic. This is a rather tedious process to do for the first time in a project’s lifetime. It is easy to continue doing once you have it in place - but the first time around requires revisiting and understanding every bit of the software - which can take some time. However, in the process many many bugs are also fixed because all code is standardized and reviewed as the process continues (which is guided by branch and line coverage as reported by the infrastructure we have setup in the first step).
  • Simulation testing: in parallel with unit testing, simulation testing can be setup as well. Since simulation testing does not care about source code (it works on final binary directly) we don’t need to wait to implement it. It can be done in parallel with unit testing - however it is better to do it after initial integration tests are in place because as we progress with testing the code it also becomes easier to debug. So when bugs are identified in the final binary, it is easier to catch then once we already have integration testing in place because we can then try to reproduce the bug in an existing test and once we have done so we then fix it in the source code and verify it in the integration test. The simulation testing is a higher level test that expects the software to already work correctly and then tests whether the software can actually be used in a way expected by the user.

What are you waiting for? Schedule a call and let’s discuss how we can implement this in your embedded firmware project as well!