A large part of the success of agile adoptions is due to the automated testing approach used in agile projects. Because many of these techniques were pioneered in the development of web applications it can be hard to see how these techniques can be leveraged for a project where the software being built is for an embedded application. Discover ways to leverage agile testing techniques for embedded systems. Whether you are building a medical device, embedded controller, or Internet of Things device learn how to leverage these testing practices to create fully automated tests that fit into a DevOps build pipeline and help your team create higher quality, more reliable software. Test automation is the best way to maintain and execute a comprehensive suite of regression tests that allows you to take back control of your testing process while increasing test coverage. Learn how to be in control of your test process by stepping up your test automation to the next level. Embedded development and Internet of Things development is often done on platforms that lack modern software development and test automation tools. The more esoteric or the smaller the target audience, the less likely tool vendors are to create products that directly support the deployment environment. This can make getting started with test automation using older tools that are not as actively supported by vendors can be a challenge that has to be overcome by a team that wants to move toward a Continuous Deployment process. This session is aimed at people that are trying to adopt agile and continuous delivery with embedded technology, but might be worried that it can’t work in their particular environment due to their industry, technology stack, culture, or regulatory environment.

1. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 1
Agility. Security. Delivered.
Agile Testing for Embedded and
IoT Software Development

2. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 2
About Coveros
• Coveros builds security-critical applications using
agile methods.
• Coveros Services
• Agile transformations
• Agile development and testing
• DevOps and continuous integration
• Application security analysis
• Agile & Security training
• Government qualifications
• DCAA approved rates and accounting
• TS facility clearance
Areas of Expertise

3. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 3
What is embedded software
• Embedded software is software that controls a device
• A device is a combination of software, firmware and hardware that
make up a system
• The system has hardware components that are either sensors or
manipulators
• The software and firmware control the hardware components, using
the sensors to discover and the manipulators to carry out actions
• Examples include:
• Medical devices
• Self driving cars
• Fitness wearables
• HVAC systems

4. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 4
What is agile testing
• Agile testing is the the set of practices used to build quality in and
make sure that software created by the team is of high quality
• The whole team is responsible for quality
• Test automation is integrated into the Continuous Integration and
Continuous Delivery pipeline
• Testing of new functionality is conducted during the same sprint or
iteration in which the functionality is developed
• Unit testing is used as part of the quality assurance process
• Acceptance testing is conducted using acceptance criteria created
directly from the requirements

5. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 5
User Acceptance Testing (UAT)
• UAT is the process of getting stakeholders to write down what results
they want to see from the software in order to determine it works as
the requirements intended
• Acceptance criteria are written before development of the software
begins
• Acceptance criteria are a form of requirements that are used to let
the team know what key aspects of the requirements need to be met
in order for the stakeholders to feel that they can accept the software
increment
• Testing of acceptance criteria happens at the end of an increment and
gives the team feedback on how well they have created software that
the stakeholders expected.

6. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 6
Behavior Driven Development (BDD)
• BDD is a test automation process for testing software based on the
requirements
• The tests are written before the code is written or at least
independently of the code
• For some teams BDD is used to automate UAT tests
• Many BDD tools exist such as Cucumber and Behave
• Most tools use the Gherkin language, which uses the following
format:
• Given initial condition
• When some action or trigger
• Then resulting action

7. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 7
Test Driven Development (TDD)
• TDD, also called Test First Development, is the process of using unit
tests to help design the code for an application
• The developer will write a test of what they intend the code to do and
then write the code to make the test pass
• Red, Green, Refactor cycle
• This has a number of positive results:
• Makes the code more testable
• Makes the code more modular or self contained
• Makes the code more maintainable
• Gives the code a regression test suite
• Makes the intent of the code easier to understand

8. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 8
Continuous Integration (CI)
• CI is the process of integrating and testing code upon check in
• In CI, integration is the process of integrating a developer’s code with
the code already in the repository
• The goal of CI is to make sure the code can build and pass an initial
set of regression tests before other testing and quality assurance is
applied to a specific build
• CI uses a build server such as Jenkins to do a clean build of the code
and run a suite of unit tests
• If the build breaks, the people that checked in since the last
successful build are informed and they are expected to get the build
back to a working state

9. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 9
Continuous Delivery (CD)
• CD is the process of taking a build through a build pipeline that
consists of a CI process followed by several quality gates that verify
the code through various quality checks including:
• Automated functional tests
• Static code analysis
• Automated non-functional tests like performance and security
• Automated deployment into progressively higher environments
• It could include on demand deployment to a manual test environment
• On demand deployment to a UAT environment
• On demand deployment to a demo environment
• Continuous Deployment is automatically progressing to production
including automatically deploying into production

10. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 10
Value Stream Mapping
• Value stream mapping is the process of determining how a product is
created in your environment from inception to deployment and use in
production
• By creating a value stream map you can figure out how many steps
you have in your delivery process and how long each step takes
• Once you have a map of your process you can start to optimize and
automate the different steps to streamline your process
• The goal is to reduce the time it takes to move requirements through
your process and get new features into production

11. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 11
Build Pipeline
• The build pipeline is the part of your value stream that takes checked
in code and makes it into high quality, tested software that is either in
production or ready to be distributed to customers
• CI/CD are a subset of your build pipeline
• Automation, including test automation, is the only way to make a
build pipeline work in a fast and efficient manner
• Many of the quality gates in a build pipeline are testing focused and
require strong testing acumen in order to help deliver high quality
software

12. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 12
Version Control
• Version control is the process of managing the code over time by
allowing a project to keep track of the state of the code as it
progresses
• All code, including testing code and test automation code needs to
be checked into version control
• Application code and testing code should be versioned together since
the test code acts on the application and changes over time to keep in
sync with changes to the application code
• Version control tools include:
• Subversion
• Git
• Others

13. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 13
Configuration Management
• Configuration Management are the tools and process used to
determine how software is setup in any given environment and how
to account for all of the different files needed to make the software
work
• This includes binaries, configuration files and environment settings
• Most build pipelines have a component that manages configuration
elements for each deployment so that the team can know exactly
what is deployed and tested for every environment

14. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 14
Testing Software vs. Hardware vs. System
• With embedded systems you are testing three things:
• Software – The code that manages the system
• Hardware – The mechanical and/or electrical parts of the system
• System – The combination of hardware and software
• Sometimes firmware is tested like software, sometimes it is tested
like hardware (depending on how firm it is)
• Isolating what you are testing can help you create test automation
that covers a much of the product as possible
• Some amount of manual testing might be unavoidable

15. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 15
Software Testing
• During software testing you are testing only the software part of your
system
• This means that all input (sensor data) should be supplied by a known
source, if possible by recording the data and playing it back, or
simulating it
• Output can be ignored or simulated
• Record and playback, or simulation can be less expensive and easier
to setup
• You should be focused on testing the business logic of the software,
i.e. the calculations and determinations that software makes
• You can also focus on testing the user interface or interactive
elements of the system

16. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 16
Hardware Testing
• Hardware testing can be agile as well
• Hardware can have a different cadence than software
• Agile hardware testing is about automating as much of the testing
process as possible
• Software tools, both homegrown and third party, can be used to
automate hardware testing
• Using the system software the controls that hardware to test the
hardware is acceptable but keep the focus on the hardware

17. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 17
System Testing
• System testing is the combination of the software and hardware
working together
• Depending on the nature of your embedded system, this can be hard
to automated or automate completely
• Depending on the nature of your embedded system there may be a
good deal of tooling that can help you test the system or there may
be nothing at all
• Testing the system is critical because you don’t know how the
software and hardware will interact until you actually have them
interact

18. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 18
Test Automation vs. Manual Testing
• Agile testing focuses on test automation and in some types of
applications it is possible to automate nearly all of the testing
• With the focus on test automaton it is easy to think that manually
testing has no place in agile but that isn’t true
• Manual testing is often a precursor to functional test automation
• Manual testing can be cheaper or more effective for some features of
an embedded system
• Exploratory testing can exercise the system in a way that test
automation will not and find deeper issues

19. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 19
How much can you automate?
• Look at this from a number of points of view:
• What can you afford
• What do you have the skills or abilities to automate
• What is possible
• What is the most beneficial
• Finding a balance between test automation and manual testing will
be a challenge that all projects face
• The answer will change over time as you learn and as your code
improves

20. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 20
Coding Standards (for tests too)
• Using coding standards means:
• Making the coding standards
• Enforcing coding standards using code reviews
• It doesn’t matter what specific coding standards you use so much as
you have them and the team agrees to them
• If possible, use an existing coding standard for your technology
• Make coding standards compliance part of your code review
• Better yet, use a code formatter to apply the standard during code
commit
• Apply coding standards to your tests as well

21. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 21
Code Inspections
• Code inspections include a number of things:
• Code reviews
• Static code analysis
• Secure code reviews
• Format and style reviews
• Automate what you can, format and style are easy to automate
• Static code analysis can be a good input into code reviews
• Code review tools can help, things like Crucible and ReviewBoard
• A secure code review is different than a code review and is a different
skill set, you should master both

22. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 22
Unit Testing
• Unit tests are developer tests that are used to make sure that the
code that they write behaves as the developer intended
• Unit tests are written in the language and technology platform of the
application code and often use a framework like Unity (see
http://www.throwtheswitch.org/unity/)
• Unit tests might be run on the dev platform or the target platform,
this often depends on how the tooling works
• Sample code:
void test_FunctionWhichReturnsLocalVariable_ShouldReturnTheCurrentCounterValueAgain(void)
{
TEST_ASSERT_EQUAL(0, FindFunction_WhichIsBroken(78));
TEST_ASSERT_EQUAL_HEX(0x5a5a, FunctionWhichReturnsLocalVariable());
}

23. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 23
BDD Sample Code
Scenario: Subtract two numbers
Given I have a calculator on
When I enter ”25" into the calculator
And I enter ”10" into the calculator
And I press subtract
Then the result should be "15" on the screen
Scenario: Add two numbers
Given I have a calculator on
When I enter ”25" into the calculator
And I enter ”45" into the calculator
And I press add
Then the result should be ”70" on the screen

24. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 24
Mock Objects
• Mock objects are replacements for the dependencies of the code
under test
• They are used to make it easier to test the code by responding the
same way all the time
• There are a number of good mock object frameworks including
CMock (see http://www.throwtheswitch.org/cmock/)

25. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 25
Test Fixtures
• Test fixtures are the code that creates a fixed state that is used to
baseline tests so that the test environment is the same each time the
tests are run
• Test frameworks do this by exposing methods or functions that allow
for the configuration of dependent objects or for the creation of mock
objects
• For a data related test a test fixture might load a specific set of data
into the database
• Most unit testing frameworks and many test automation tools have
test fixtures build into them

26. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 26
Working with Simulators
• A simulator is software or hardware used for testing that has the
same interface as the application or system it is replacing
• Simulators provide a consistent interface for testing, much like test
fixtures or mock objects
• Simulators are more full featured and can react to input, in more
varied ways that other testing tools
• The idea is that a simulator will react more like the real component in
the real world
• Simulators can be expensive to build and maintain but there may be
no other option

27. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 27
Testing services for embedded software
• Many Internet of Things (IoT) devices use services
• Services are Internet accessible bits of functionality that do things
like:
• Collect data
• Calculate results
• Coordinate resources
• Testing these services can be easier to automate then the devices
themselves
• Scale can be an issue for services
• Network availability or reliability can be an issue for the devices
• Performance testing is often important for testing services

28. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 28
What to take away from the presentation
• Test automation is the key to agile testing
• Do test automation planning while you do test planning
• Create test suites that test your software separately from your
hardware and another suite that tests the system as a whole

29. © COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 29
Questions?
• Tom Stiehm
• tom.stiehm@coveros.com
• @thomasstiehm