This is the presentation we gave in 2009 during Agile Testing Days in Berlin. Even though it is already more than 2 years old, many things we said during the talk are very valid today. Some things did not change at all.
2. • Test Consultancy at VIIQ.
• Agile Test Team leader at Philips CareServant.
• 10 years of experience in the Testing Field (7 years
V-Model, 3 years Agile).
• Some experience with coding.
• Bachelor degree in Information Technics.
• Married to Jeanne, has a dog (Sasha) and 2 cats.
Martijn Gelens
3. • Currently, Philips Software Engineering Services,
MiPlaza, Software Designer.
• Ph.D. in Computer Security from University of
Twente, Enschede, The Netherlands.
• M.Sc. in Software Engineering at Warsaw University
of Technology.
• 7 years of experience as an embedded software
developer (SterKom (Poland) and freelancer).
• Modelling Languages: my master thesis + one year
internship at Institute National des
Télécomunications (INT), Evry, Cedex, France.
• Agile, test-invected since one year.
• Married to Beata and also has two cats (Mufasa and
Ursus).
Marcin Czenko, Ph.D.
4. Wim van de Goor
• Agile Software Team Leader at Philips Software
Engineering Services, Philips MiPlaza.
• 8 years experience with eXtreme Programming.
• Agile mentor and Coach.
• Advised us on Agile Principles.
5. Contents
• Agile Testing
What Do we want to do?
• Constraints
What can we do?
• Agile Embedded Testing
How do we test?
• Conclusions
6. What do we want ?
• Prerequisites
• Test Strategy
• Acceptance Criteria
• Continuous Integration
• Testing
7. Prerequisites
• Testing is integrated in the team's Way of
Working.
• Acceptance Criteria are defined, discussed and
explored beforehand.
• Test Driven Development.
• Continuous Build and Integration.
• Testing is structured (specify, verify, report).
8. Test Strategy
• Product risks and mitigation.
• Tester-role.
• Definition of Done.
• Quality gates.
• Testing is structured (specify,
verify, report).
9. Test Strategy (cont.)
• Deliverables from outside (also HW).
• Issue procedure and attendees.
• Reporting (test reports, PMI, ...).
• Emergency scenarios.
15. Test Driven Development
• Means: “Write unit tests
before code”.
• Integrate with your
Continuous Integration
environment.
• Automate the Acceptance
Tests.
19. Light
• No cross-compilation required.
• Usually mainstream OS (Windows/Linux).
• Wide range of testing/mocking
frameworks available.
• Standard hardware - no or very limited
hardware level programming required.
• No dependence on a particular vendor
(supplier).
20. Heavy
• Small memory (no OS), limited performance,
limited debugging possibilities.
• Limited cross–compilers: often only C, and
Embedded (Extended) C++ available.
• Vendor specific.
• Difficult to find a testing/mocking framework.
• Custom hardware (ramp-up).
30. Development Board
• Selecting the right board can be
challenging (expensive ?).
• Chip selection driven by the availability
of the right board.
• The board selection driven by the
availability of the BSP and OS.
36. The effect on Agile
Principles
• Longer ramp-up time.
• Resistance to modify hardware
(introduces up-front design).
• Limited response to changes.
• Higher risk.
37. How to proceed ?
• Often we cannot remove queues &
batches in HW development (are we
going to be able doing so in any
predictable future ?).
• Reducing the queue size is also often not
an option.
• Is there a lean solution ?
45. Testing Framework (C)
• Many frameworks are simple ports of
the frameworks for PC-based
development.
• Increased stack consumption.
• Dynamic memory allocation.
46. CMock
• Easy to understand. Easy to customise.
Lightweight.
• Comes with Supporting Ruby-based
Mocking Framework.
• Ready For “Heavy Embedded” - tests
executed in batches.
47. Testing Frameworks (C++)
• Run-Time Type
Information
(RTTI).
• Exceptions.
• ISO C++ compiler
needed.
• Gnu or Microsoft
are preferred.
48. Testing Frameworks (C++)
• We could not find a framework that
compiles on GreenHills and WindRiver
C++ compilers (forget IAR Extended
Embedded C++).
• It was cheaper and more effective to
come with your own simple testing
frameworks.
49. yaffut
• Our choice for unit testing in C++.
• Just one header file.
• Not meant for embedded: needs RTTI,
and C++ exceptions.
• Easy to understand and customise. We
made an RTTI and Exception-free
version.
50. Mocking
• We did not succeed in using existing
frameworks. Our best candidate
GoogleMock does not even compile and
it is quite complex.
• Does it mean that no one is doing TDD
on embedded ? Probably not.
51. Mocking - way to go
• Think of your own framework.
• We needed three “evenings” to create
our own mocking framework.
54. Conclusions
• Agile in Heavy Embedded is a challenge: we cannot change it, but
we can understand it and try to reduce its impact on agile software
development.
• The tester should be experienced in working with hardware,
perhaps even more than a developer.
• There is no one way: what you can do depends on the constraints
you have (e.g. light to heavy).
• Hardware development is far from being lean - and there is not
that much we can change.
• Development and support tools are far behind the needs of the
agile teams.
• Let your agile testing framework grow with your code.
55. Conclusions
• Agile in Heavy Embedded is a challenge: we cannot change it, but
we can understand it and try to reduce its impact on agile software
development.
• The tester should be experienced in working with hardware,
perhaps even more than a developer.
• There is no one way: what you can do depends on the constraints
you have (e.g. light to heavy).
• Hardware development is far from being lean - and there is not
that much we can change.
• Development and support tools are far behind the needs of the
agile teams.
• Let your agile testing framework grow with your code.
Be agile.