Continuous Testing
Testing is a cross-functional activity that involves the whole team. Building quality into your build process means writing automated tests at multiple levels and running these tests as part of your deployment pipeline.
Continuous testing solves the problem of not knowing whether the software still works after a small change has been made. It gives you confidence that you can change any small detail and know for sure that you did not break anything else. This confidence gives you the ability to clean up technical debt whenever you see it instead of waiting to clean it up until 'after the release'.
Types of tests
It is convenient to group tests into business facing tests vs. technology facing tests as well as tests that support developers vs. tests that find faults in the product.
The tests that support developers and face technology are naturally lower level and are usually written in the same language as the application. While tests that are business facing and which test the whole product are usually written in a higher level language and do not directly interface with the code, but rather test the product end to end as a whole unit.
Business facing tests supporting developers
These tests are meant to verify that the acceptance criteria for a user story is met.
These tests should be written and reviewed by the users. Acceptance tests are critical from the agile perspective because they verify that the customer got what they wanted. More importantly, they also continuously verify that these requirements are fulfilled because they continuously run in CI and so regressions are less likely.
Several excellent tools exist for writing these tests including:
RobotFramework, Cucumber, behave. These frameworks help you express the
acceptance tests in plain language and to separate the test descriptions from
the operations needed to implement them.
Technology facing tests that support developers
These tests are written and maintained entirely by developers. This includes
unit, integration and system tests and also deployment tests.
The purpose of these tests is to verify the software components at different levels. Unit tests for example check that individual classes or source files implement correct behaviors. In the case of unit tests in particular, you must make heavy use of mocking to decouple test from other components.
Integration tests verify interactions between components. You should limit the modules that you compile as part of these tests to only the modules between which you want to verify interactions. Avoid the temptation to include complex hierarchies of dependencies. This makes your tests harder to debug and also it makes it hard for you to actually verify that given inputs into your components give the expected outputs. Use mocking to mock the dependencies.
The combination of unit and integration tests should cover at least 80% of all paths through your code. Use coverage information generated by your compiler tools to track your branch coverage to verify that this is true.
System tests verify that a deployed application works. These tests run against the built production executable and typically involve testing on real hardware. In these tests you verify that it is possible to interact with the software and that all expected interaction patterns work.
Note that system tests are different from user acceptance tests in the scope of the testing. User acceptance tests only verify what is important for the user, while system tests verify that the whole system works in an end to end fashion.
Business facing tests that critique the product
These are manual tests intended to verify that the product delivers to the users what the users require.
The second role of these tests is to verify that these expectations are in fact correct. It could be for instance that automated user acceptance tests make the wrong assumptions. The only way to correct this is to do manual testing and then make adjustments to the automated tests so that they are more inline with the manual tests that make the most sense.
One particular variant of these tests are in fact sprint demos. When developers demonstrate to stakeholders at the end of each agile sprint what they were able to achieve. These demos invite feedback from stakeholders and ensures that what is being built is in line with what the stakeholders want.
Technology facing tests that critique the product
These tests focus on additional requirements such as performance, security, capacity and power consumption.
These tests usually require specialized equipment such as robotic automation and measurement tools as well as scripts that interact with these tools.
Tools
Almost every language has its own set of frameworks that provide both unit testing and mocking tools. Here are some frameworks you can use.
It helps to split the tools between unit testing and mocking categories. Some unit testing frameworks include mocking support, but not all. This means you often have to combine two kinds of libraries - one for assertions and one for mocking support when you develop your unit tests.
C
-
Unit Testing Frameworks:
- Unity: A simple and flexible unit testing framework for C.
- ZTest: A framework for testing firmware built on top of Zephyr RTOS
-
Mocking Frameworks:
- CMock: Works well with Unity for generating mock functions for C. Works in Zephyr using custom extension.
- CMocka: A unit testing framework for C with support for mock objects.
C++
- Unit Testing Frameworks:
- GoogleTest: Google’s C++ test framework.
- Boost.Test: Universal and generic testing framework that is part of Boost library.
- Catch2: A modern C++ native test framework for unit testing and behavioral testing.
- Mocking Frameworks:
- GoogleMock: Offers the ability to create mock classes for C++ testing.
- FakeIt: A simple yet very expressive headers only mocking library for C++.
Python
- Unit Testing Frameworks:
- unittest: The built-in unit testing framework for Python.
- pytest: A powerful framework that makes it easy to write simple tests, yet scales to support complex functional testing.
- Mocking Frameworks:
- unittest.mock: A library for testing in Python. As of Python 3.3, it's part of the standard library.
JavaScript
- Unit Testing Frameworks:
- Jest: A delightful JavaScript Testing Framework with a focus on simplicity.
- Mocha: A feature-rich JavaScript test framework running on Node.js and in the browser, making asynchronous testing simple.
- Jasmine: A behavior-driven development framework for testing JavaScript code.
- Mocking Frameworks:
- Sinon: Standalone test spies, stubs, and mocks for JavaScript.
- Jest: Comes with built-in mocking capabilities.
- Proxyquire: Proxies
require()in order to allow mocking dependencies in Node.js.
TypeScript
- Unit Testing Frameworks:
- Jest: With TypeScript support out-of-the-box, it's a popular choice for testing TypeScript projects.
- Mocha with ts-node: Mocha can be used for TypeScript testing in combination with ts-node.
- Mocking Frameworks:
- ts-mockito: A TypeScript mocking library inspired by the Mockito framework for Java.
Bash
- Unit Testing Frameworks:
- bats-core (Bash Automated Testing System): A testing framework for Bash that provides a simple way to test scripts and command line tools.
- shunit2: A unit testing framework for shell scripts akin to xUnit frameworks.
- Mocking Frameworks:
- Mockery: Part of the bats-core ecosystem, allowing for mocking of Bash commands within tests.
Lua
- Unit Testing Frameworks:
- busted: A unit testing framework with a focus on being easy-to-use for Lua.
- LuaUnit: A unit-testing framework for Lua, inspired by several other xUnit frameworks.
- Mocking Frameworks:
- luamock: A simple mocking library for Lua.
- Mocka: A Lua library that provides mocking functionality.
Lisp
- Unit Testing Frameworks:
- FiveAM: A simple, extensible testing framework for Common Lisp.
- CLUnit: A unit testing framework for Common Lisp.
- Mocking Frameworks:
- Mockingbird: A Common Lisp library for mocking and stubbing.
Java
- Unit Testing Frameworks:
- JUnit: A simple framework to write repeatable tests. It is an instance of the xUnit architecture for unit testing frameworks.
- TestNG: Inspired by JUnit and NUnit, but introducing some new functionalities that make it more powerful and easier to use.
- Mocking Frameworks:
- Mockito: Tasty mocking framework for unit tests in Java.
- EasyMock: Allows mock creation, verification, and stubbing.
- PowerMock: A framework that extends other mocking frameworks such as Mockito and EasyMock with more powerful capabilities.