The document discusses how simulation can help automate testing and integration for embedded systems development. Simulation allows testing to be fully automated by treating hardware and software systems as virtual models that can be easily and endlessly configured in software. This enables continuous integration practices like testing any changes immediately, running many tests in parallel, and testing unusual configurations and fault conditions. Simulation makes test automation scalable, flexible, and able to cover more of the problem space than is possible with physical hardware alone.

1. Jakob Engblom, Product Line Manager, System Simulation, Wind River
Graham Morphew, Sr. Director of Product Management, System Simulation, Wind River
Need To Automate Test And Integration Beyond
Current Limits? Use Simulation
Moderator:
Brandon Lewis, OpenSystems Media
Speakers:

2. Agenda
 Housekeeping
 Presentation
 Questions and Answers
 Wrap-up

3. 3 © 2015 Wind River. All Rights Reserved.
Need To Automate Test and
Integration Beyond Current
Limits? Use Simulation
3
Maximize Velocity – Webinar Series
Agile, Continuous Practices and DevOps
Graham Morphew & Jakob Engblom
Wind River

4. 4 © 2015 Wind River. All Rights Reserved.
Embedded Software Trends
 Software driving majority of
value and innovation
 IoT driving future differentiation,
value & service
 Customer expectations have
changed to demand new
features more often and faster
 Pushing software organizations
to change and be more agile
and efficient
Change
Change
Change

5. 5 © 2015 Wind River. All Rights Reserved.
Faster Delivery Of Software To Customers
Agile Continuous Practices Cross Disciplined Teams DevOps
New development practices and organization styles
getting established for embedded software development
Key Challenges:
• Access
• Collaboration
• Automation
Automation is key because manual processes don’t scale.
Covered in previous webinar
Covered in upcoming webinar

6. 6 © 2015 Wind River. All Rights Reserved.
Developer changes
or adds code
Build and test:
single unit
Build and test:
multiple units
Build and test:
entire system
Code
OK
Immediate
Coffee break or lunch
Daily
Continuous Practices
 Quick feedback to developers
 Find different classes of errors at each level
 Avoid big-bang integration & associated problems
 Avoid regressions by rerunning old tests
 = higher quality and shorter lead times
(Typical time frames)
Needs to be Automated!

7. 7 © 2015 Wind River. All Rights Reserved.
The Hardware Test Lab
bus, network, …
Data generator or
world model, running
on hardware
Target provisioning
and control
JTAG, serial,
Ethernet, flash
programmer, …
System under
test
Test manager
Build server
Lab test rig – critical resource
Automation based
on coordinating
several separate
pieces of hardware

8. 8 © 2015 Wind River. All Rights Reserved.
The Challenge of Automation in Hardware
 Hardware test lab is difficult to build, scale and maintain
 How many lab setups can you afford?
 Reconfiguration is manual, even when tests are automatic
 Automation is often brittle
 Impact
• Large equipment expenses
• Unable to scale
• Automation of failure

9. 9 © 2015 Wind River. All Rights Reserved.
Simulation system – all just software
The Simulation Test Lab
Simulated bus,
network, …
Data generator or
world model, running
as software only
Target provisioning
and control
System under
test - simulated
Test manager
Build server
Simulator
tools
Model of
the world
 Use the existing test
management system
 Automation based on
software control, not
hardware control!

10. 10 © 2015 Wind River. All Rights Reserved.
Simulation Makes Automation a Reality
 Automate any operation on the target system
 Test in the context of any system or subsystem
 Parallelize, scale & reconfigure test suites in any way
 Use any server to run any test – flexibility*
 Send the context of test failure to developers to solve*
Value using simulation for automation
• Make embedded test automation possible & cost effective
• Do more, effective testing and get higher quality releases
• Deploy faster and more frequently

11. 11 © 2015 Wind River. All Rights Reserved.
Parallelization and Performance
 Simulation provides automation and access, which enables
more tests to be run in parallel
Test 1 Test 2 Test 4Test 3 Test 5
simulation
Test 1
Test 2
Test 4
Test 3
Test 5
 Overnight tests take a few minutes
 Weekly tests become daily tests
 Test more often = less chance for bugs to creep
back into the software – higher quality
1/5 the time

12. 12 © 2015 Wind River. All Rights Reserved.
Test management server
Test run system
(such as Jenkins)
Test generation
and management
Test server 1
Simics
board
Networkboard
board
Testing
agent/
slave
node
Parallel Testing and
Automation Setup
Physics/Electronics/World
Test server n
Simics
board
Network
board
board
Testing
agent/
slave
node
Physics/Electronics/World
board
 Test system launches
simulated test labs
 Each test can be of a different
configuration
 Sends out test cases &
collects results

13. 13 © 2015 Wind River. All Rights Reserved.
Simulation
Expanding Test Coverage: Variation
Simulation
Board A
Simulation
Board A
Network
Board B
Network
Board A
Board A
Board B
Board C
Simulation
Board B
Program 1
Program 2
Program 3
Program 4
generates
generates
generates
 System setups are just
software
 Change setup program
to get a new
configuration
 Infinite supply of each
type of system
 Randomly generate
setups for “didn’t think
of that” tests
 System creation &
setup is automatable
Input file
World / Physics
Simulation

14. 14 © 2015 Wind River. All Rights Reserved.
Expanding Test Coverage: Faults
Fault injection is a natural match for simulation
 Test fault detection, handling, recovery, reporting, …
 POST, diagnostic software, firmware, applications, system, …
 For example:
– Corrupt and drop network packets
– Feed bad data from the environment
– Crash processors
– Change memory contents
– Modify system configuration on the fly
 Simulation makes faults:
– Repeatable
– Non-destructive
– Scriptable and automatable!

15. 15 © 2015 Wind River. All Rights Reserved.
 Automatically capture test failures
 Analyze using record and replay of tests
 Run and debug in reverse
When Things go Wrong
More on Collaboration with simulation in our next webinar.

16. 16 © 2015 Wind River. All Rights Reserved.
Key Take-Aways
 Continuous integration and testing requires automation
 Automation is made easier by using simulation
 Simulation provides the ability to automate any test
 Simulation allows testing and automation of what
cannot be tested with hardware

18. Audience Q & A
Jakob Engblom,
Product Line Manager,
System Simulation,
Wind River
Graham Morphew,
Sr. Director of Product Management,
System Simulation,
Wind River

19. Thanks for joining us
Event archive available at:
http://ecast.opensystemsmedia.com/
E-mail us at: jgilmore@opensystemsmedia.com