How to Break Software: Embedded Edition

MQ
PM Tutorial
9/30/2013 1:00:00 PM

"How to Break Software:
Embedded Edition"
Presented by:
Jon Hagar
Consultant

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Jon Hagar
Grand Software Testing
Jon Hagar is a systems-software engineer and tester consultant supporting software product
integrity and verification and validation, with a specialization in embedded and mobile software
systems. For more than thirty years Jon has worked in software engineering, particularly testing,
supporting projects including control system (avionics and auto), spacecraft, mobile-smart
devices, IT, and attack testing of smart phones.

8/20/2013

How to Break Software:
Embedded Edition
Attacks to find common bugs
quickly in embedded software
systems

Jon D. Hagar
embedded@ecentral.com
Jon.d.hagar@gmail.com
http://www.breakingembeddedsoftware.com/
Jon Hagar Copy right 2013

1

Some Thoughts


What are our objectives?
◦ Definitions and introductions
◦ Understand Applicable Mobile and Embedded
Test Concepts
◦ Practice some testing


2
Software Test Attacks To Break Mobile and Embedded Devices

1

8/20/2013

This is a Workshop Tutorial


A bit of a talking head (with charts)
◦ Based on my book



Attendees should be prepared to:
◦
◦
◦
◦



Do some reading & thinking
Use the reference material
Talk & ask questions
Share (lessons learned and retrospectives)

LET’S PLAY TEST… and try out some new things…..
 It might get LOUD


3

Agenda


Definitions and introductions



Risk-based concepts



Exploratory approaches



Attacking scenario(s)



Attacking the hardware-software interface



Wrap up and references


4

2

8/20/2013

Definitions
Embedded Software Systems . . .








Interact with unique hardware/systems to solve specialized
problems in the “real world”
◦ IT software runs with largely “generic” hardware
◦ Users are barely aware the device uses or has software
Usually have significant hardware interface issues and concerns
◦ Initialization, noise, power up/down, timers, sensors, etc.
Often are resource constrained
◦ RAM, ROM, stack, power, speed, time, etc.
Typically has a restricted or no Human User/Computer Interface
(HCI) but is evolving rapidly
Often no way (or only a risky way) to update and/or change the
software
Involves risks, hazards, safety, and/or some specialized domain
knowledge and logic/algorithms usually controlling hardware


5

Close Cousins: Mobile, Smart, &
Handheld
As the names imply, these are devices—small, held in the hand, often
connected to communication networks, including:
◦ Cell and smart phones – apps (not covered today)
◦ Tablets
◦ Medical devices
 Typically have:
◦ Many of the problems of classic “embedded” systems
◦ The power of PCs/IT
◦ More user interface (UI) than classic embedded systems
◦ Fast updates
 Are getting more powerful, memory and features (software, e.g., apps)
 The “hot” area of computers/software
◦ Testing rules are “evolving”



6

3

8/20/2013

What do these look like?
Examples
– Avionics systems: planes, cars, rockets, military. . .
– Telecom: switch, routers, phones, cell devices
– Transportation: traffic control, railroad, trucking
– Industrial control: lighting, machines, HVAC, nuclear/power
– Medical: pacemaker, AEDs, defibrulators, dispensers, etc.
– Home and office systems: control, entertainment (TV box)
– And the list goes on
• Now starting to include PDA’s and other items that “blur”
the lines

7

Fundamental Software Capabilities Defined
Dr. James Whittaker lists four capabilities:
1. Software accepts inputs from its environment
2. Software produces output and transmits it to its
environment
3. Software stores data internally in one or more
data structures
4. Software performs computations using input or
stored data

Embedded devices can be refined with
◦
◦
◦

Function in/with Time
Use/control of unique hardware, OR
Real world specialization of items 1 and 2


8

4

8/20/2013

Knowing the Bug (error) - Defined
•

Handheld/Embedded software has
similar defects to other software
• Requirements & Design
• Logic & Math
• Control Flow
• Data
• Initialization & Mode changes
• Interfaces
• Security
• Gaming
• etc. . .

But adds context defects/issues
• Software and hardware
development cycles done in
parallel, where aspects of the
hardware may be unknown to the
software development effort
• Hardware problems which are
often fixed with software late in
the project
• Small amounts of dense complex
functions often in the control
theory or safety/hazard domains
• (a big one) Very tight real-time
performance issues (often in millior micro-second ranges)


9

“World” of Mobile-Smart/Embedded Software
Response-Outputs

Stimulus-Inputs
Expected
Unexpected




Wanted
Hardware

Unwanted

Software

Inputs and outputs involve hardware, software, and
humans
Time dependent
◦ NOTE: most software has “time” (performance) issues but here
things are often “hard real time”
◦ Embedded and real-time “time” may be a requirement


10

5

8/20/2013

Exercise: Why do we test?


Handheld Mobile/Embedded Software (or any
software)?


11

YAM* Lifecycle Embedded (*yet another model)
 Software - Many builds, iterations and increments
 Test “circles” around schedule milestones
start

Lab drop

end

Build 1
start

Eng drop lab drop

end

Build 2

…………
…Prototype………
………Prototype n…..
Test Efforts

But what about the hardware lifecycle?

12

6

8/20/2013

Example High Level Embedded Lifecycle
System Creation
Hardware Build
Hardware Build
Hardware Build

Hw
Issue

Software Build
Software Build
Software Build
Test/V&V

Software Build
Software Build

Results: Software is “late”


13


My Assumptions…





This is not a “general” class on systems,
software, and/or testing and
I assume the following knowledge:
◦
◦
◦
◦
◦
◦
◦

Test plans and planning
Requirements testing
Test labs and building labs
Standards you operate under (yes, there are many)
Tools you use
Testing experience (a software system)
Embedded design for testability is an accepted
practice

That you want something more . . .


14

7

8/20/2013

If What I Assume is False…
(when you get home)









Reference list is available to do some reading
Other full classes are available
You are reading books
You will ask questions
You are looking to have an epiphany
You are ready to learn

Keep in mind that I do not have all the answers

15

Section 1: Testing
Preliminary


16

8

8/20/2013

Exercise: Test the Embedded Game






Break into teams
Load the 20Q app on smart phone (if you want)
Define a test
Define some rules: No destructive testing please
List of requirements
◦ This is a handheld game
◦ You think of something (say spinach) and it figures out what you are thinking
by proposing 20 questions to you
◦ Questions begin with animal, vegetable, mineral and go from there
◦ Game has non-standard input keys, display screen, and embedded software
◦ Game knows things and will figure out what you are thinking of



Now . . . build a test for this device


17

What do you mean you do not have
enough?


What is wrong?



What do you need to do testing?



Is this not the world many testers live in?



We should start simple in testing, but maybe
this is not simple enough?


18

9

8/20/2013

So, Let’s Back up a Little


Let me give you some attack support concepts &
techniques (in case you don’t know these)



You can apply these if you are a staff tester or a
“crowd source” contractor



This is a simulation, but in the real world, often you
will just be given the software or a device to test --- You CAN test…….


19

Risk and Exploratory-Attack Testing


You cannot test everything



Risk(s) based testing helps bound the test scope
problem



Testing is about providing information and understanding



Exploration gets you started with whatever you have (or
don’t have)


20

10

8/20/2013

Basic to Attacks: Risk-Based Testing


Address, mitigate, attack and retire product
risks



Do you remember what a risk is?
◦ Potential problem - consequence and effect
◦ Occurrence – likelihood or chance of happening
◦ Impact – what happens



Do this from the beginning (proposal) to the
end (retirement) of the product (Hw-Sw)
lifecycle



Risks should feed the Attacks (more on that
later)


21

Sample Product Risks Testers Should
Consider


Safety



Security



Hazard



Business impacts



Control (loss of. . . )



Computation



Functional elements



Non-functional



Data



Regulation (s) and legal factors



Output noise



Environment and input factors



System factors – complexity, interfaces, human/non-human


22

11

8/20/2013

How to Use Risk Analysis in Testing


Goal oriented testing (where to focus)



Priority of attack and scenario
◦ Never enough time to test everything
◦ Can define the “un attacked” (risks)



Minimization of risks by focusing on the scary or
critical first



Provide information back to the team sooner


23

Risk-Based Testing Process (simple)


Identify the product



Find product supporting information



Identify risks associated with the product



Risk priority (what you will test first?)



The resulting risks by priority define the
attacks


24

12

8/20/2013

Risk Analysis Throughout the Test Process





Many testers just think “requirements” in embedded,
but…
Always be “thinking” risks, since it can drive and control
your testing
Do this by team brainstorming (make lists)
Tests and analysis provide learning/data
points/information
◦ Errors in an area of code?
◦ Hardware that doesn’t work?
◦ Piece of code from a vendor is more complex?
◦ Operations the system will/can do?
 Particularly off nominal and unusual (where bugs hide)


25

Exercise: Redeaux
Back into teams
 Conduct a risk exercise for your device


Risk Statement ( If x, then y happens)


Priority

26

13

8/20/2013

Risks Should Define
Exploration
For mobile-embedded, exploratory testing
can be important


27

Exploratory – Attack Testing


What is it?
◦
◦
◦
◦



Scientific “methods”
Engineering understanding
May call it something else, but most of us do it
Attacks “target” specific bugs using test techniques

How and when to apply?
◦ As early in a lifecycle as possible (with prototypes, models,
etc.)
◦ When you want to “learn” and test at the same time
◦ When being a little “informal” is OK
◦ All the time?


28

14

8/20/2013

Exploratory–Attack Testing Definition
Bach/Kaner:
“Exploratory testing is simultaneous learning,
test design and test execution.”
 Exploratory testing has rules and concepts
 Underlying it is a “model” of human understanding of
software and knowing how that fails
 NOT AD HOC: Ad hoc has all too often been associated
with sloppiness, carelessness, no documentation, nonrepeatable, and so forth—but may have its place at times


29

In Embedded


Exploratory testing is situational - Use it when…











Need rapid feedback
Learning
Upfront rapid learning
Attacking
There are risks
Need independent assessment
Targeting a defect
Prototyping
Need info
Testing beyond the requirements


30

15

8/20/2013

General Concept of Exploratory
Many authors define it as:
 Time/Schedule (limited)
 The Tester (your team)
 A Testing Mission (also called “Charter”)
 Results
 Usually in the form of opened Defects
 Sometimes an annotated Mission statement and
opened Defects list
 Maybe a “report”
 Retrospective (more on that in a minute)


31

Exploratory Critical Components
 Test Design
 Critical Thinking
 Diverse Ideas
 Rich Resources
 Careful Observation

32

16

8/20/2013

Pattern for This Class (one of many)









Have an outline (top level plan and/or risk list)
Create a flip chart, notecard, state model, or some
representation of each test task
◦ No “heavy” weight documentation of the “test case”
◦ See Exploratory Charter (test objective)
Have a Target concept or charter (Risk, Attack, Bug, Learning,
etc.)
Have a schedule/time box (hours — not more than 1-2 days)
Do the test
◦ Design test
◦ Execute test
◦ Learn about the product: change the risk list, modify/add
tests, and so on
Repeat the process as needed


33

Exploratory Test Card (Charter)
Name of Test:

Who is testing (test team)

What to Test:
◦ Risk (s):

Success Criteria:
1.

◦ Attack

2.

◦



3.

Other (requirements, …..)



Support items needed:



Role (User you play during the test):
Actions:
◦ 1.
◦ 2.
◦ 3.



Others Steps

Results (bugs, observations, lessons learned, positives, issues, concerns, more risks….)
Document all of these.


34

17

8/20/2013

Exercise: So Let’s Go Back to
the Embedded Game Device to Do Testing


Test of the Game App



Use the risks (chart 25) for the device to define test
objectives



Apply Exploratory-Attack
◦ Do a Charter



Learn – do one cycle of exploration


35

Group Flip Chart
Feedback - Retrospective


What did you accomplish?
◦ Did you find any bugs? If so, how many?



What did you think of?



What would you do differently?


36

18

8/20/2013

Section 2:
So we have Risk Analysis &
have done a first Exploratory Test
Basic and addressed in many books
What’s next?
Lets get to different levels of testing, embedded
devices, and more fun

37

What is an Attack


Attacking your software–In part, the process of attempting
to demonstrate that a system (hardware, software, and
operations) does not meet requirements, functional and
non-functional objectives
◦ Embedded/handheld software testing must include “the system”
(hardware, software, operations, users, etc.)



Attacks go after common modes of failure and bugs to
demonstrate that “does not meet” exists



We go after our enemy with many approaches


◦
◦
◦
◦
◦

Tools
Levels
Attacks
Techniques
Etc.

38

19

8/20/2013

An Attack Is…


Based on a common mode of failure seen over and
over

◦ Maybe seen as a negative, when it really is a positive
◦ Goes after the “bugs” that may be in the software
◦ Based on or using classic test techniques and test concepts
 Lee Copeland’s book on test design
 Many other good books



A Pattern (more than a process) which must be
modified for the context at hand to do the testing



Testers learn these in a domain after years and form a
mental model (most good testers attack)


39

Kinds of Attacks


Whittaker offers a good starting point for software
attacks in general that can be applied to embedded:
◦ User Interface Attacks
◦ Data and Computation
◦ File System Interface
◦ Software/OS Interface



Whittaker’s “How to Break Software” lists 23 attacks



“Software Test Attacks to Break Mobile and
Embedded Devices” (my book) adds 32 attacks and 8
sub attacks


40

20

8/20/2013

Embedded Attack Classification










Developer Attacks (unit/code testing)
Control System Attacks
Hardware-Software Attacks
Mobile and Embedded Software Domain Attacks
Time Attacks (Performance)
Human User Interface Attacks
Smart and/or Mobile Phone Functional App Attacks
Mobile/Embedded Security Attacks
Generic Attacks
◦ Functional, mind mapping, and combinatorial tests


41

The Software You Test


Do you know how it fails?



Do you test for success or failure or both?



Will this workshop give you all the answers and all possible
attacks?
◦ No, but you can start asking questions and thinking


42

21

8/20/2013

Introducing the Robots
Requirements – in the class hand out for each robot grouping
Rules (this exercise takes some thinking and reading)
◦ NO destructive testing (Please BE CAREFUL with the robots)
◦ There are bugs to be found (record and report them)
◦ Each group defines an attack and gets “time” on the devices (but time in our
environment is limited—just as it is in the real world)
 Environment
◦ This room, but we have some “test tools”
◦ This is software testing, but within the hardware it is embedded in
 This will be a simple testing process, but use the attack concepts and report
experiences back in the debrief
1. Define risks based on hardware, software, requirements, and bugs (risk list)
2. Conduct each attack session using a charter
3. Define a test attack using provided attack pattern (handout)
4. Will do a debrief after each test session
5. Group will rotate different robot configurations




43

Additional Considerations








We will try to run at least 2 different attacks
Use the concepts we used on games
Ask questions (of each other & me)
No destructive testing and I am the “tester” (you
must tell me what to do)
Use the tools at hand
But first we need to think about embedded users
– next exercise


44

22

8/20/2013

More Test Time








We are going to do another attack on a different
embedded device
Each group will be tasked with one of the two attacks
Each group should complete a charter and see the “test
master” for access to the device
You’ll have a robot with software loaded
Practice identifying: Risks, Users, Exploration, and Attacks
Follow the “suggested patterns” of the attacks
We’ll go until no time left


45

Exercise: Understanding Users for
Embedded Attacks


Let’s list some users of the games and
robots because . . .
◦ Users play into risks, attacks, bugs, what to look for
◦ You should be able to do the same for the robots (or
any software you test)


46

23

8/20/2013

Exercise Answer: List Users


47

Now you have a few basics:
•
•
•
•


Risk thinking
Exploration
Software’s users
Attack patterns are provided next

48

24

8/20/2013

Attack Group 15


Scenarios and actions over time


49

Attack Stories, Tours, and Scenarios


Call them what you will



There are subtle differences depending on whose
material you have read
They are how the system gets used end-to-end
They combine use, users, information, techniques,
tools, and (maybe) attacks





50

25

8/20/2013

Apply This Attack When…



Time interacts with the software, events, inputs,
and outputs
Checklist of things to look for and consider
(possible bugs)
◦ Order problems
◦ Too Long

◦ Too Fast
◦ Not at Right Time mark or point
◦ Late
◦ Late or early
◦ Early
◦ Deadlocked caused by a race condition(hard to find)
◦ Extra input or output events
◦ Missing events
◦ Wrong input/output within events

51

Attack Factors


What - Look for things not in the right order



Who – Test team



Where – Lab and/or field testing where hardware
and software interact
◦ Tools may be important here


52

26

8/20/2013

How (a generic pattern)?


Understand what the system does or is supposed to do
◦ a sequence of events or functions
◦ Look in: concepts of operations, user guides, use cases, models, & any other
information that will detail functions and usage over time
◦ From these, organize a sequence or set of sequences



First attack case: Focus on a typical situation based on requirements and/or use
cases



Second attack case: Consider the off–normal, non–failure modes

◦ Look for the failure modes and effects—does the software recover well?
◦ Review and understand system errors and failure history from the field


Build up histories of attacks based on outputs and log files
◦ Warning: log files can contain large amounts of detailed data and this can
also adversely affect the performance (especially timing) of the software



Conduct risk analysis as the effort progresses



Final Attack cases: Build Extreme cases such as “Soap Opera” Tests



Warning: Watch becoming “script” bound


53

Exercise – Tell Me Stories



Work in groups
Key points

◦ Define a “story/scenario” with a name and outline
◦ Use the check list on chart 51 (note what is used)
◦ Follow pattern of chart 52 (note steps on charter)
 Can you build a Tour (combination of story patterns)



Products
◦
◦
◦
◦





Risk list
User list
Test charter
Note bugs (if any)

Complete the feedback retrospective
Do several attacks (note what each one is)
Expand to “extreme cases”


54

27

8/20/2013

Attack 7


Digitals v Analog Integration


55

Attack Hw-Sw Interface Group
Here we are attacking the hardware-tosoftware interface
Many bugs
- Developers

hide in the interfaces
often miss these


56

28

8/20/2013

Attack: Analog-Digital Hw-Sw Interfaces


When - The software is “controlling” the unique
hardware



What – Look at the interface, hardware (as a user), and
what the software is controlling



Who – Test team (independent)



Where – Lab where the hardware and software are both
present



Bugs to look for (next page)


57

Taxonomy: A2D and D2A Bug
Possibilities
Type
A2D

A2D

A2D

D2A

D2A

D2A

Situation
Impact
A2D representation information is lost Software
because measurement is not precise computation is
based on incorrect
data
A2D information is contaminated with Software
noise
computation use
noise when it
should not
A2D information is calculated
Computation has
correctly
unknown error

D2A conversion losses “least
significant bits” (LSB) in conversions,
but bits are, in fact, important
because computer word sizes are too
small
D2A information does not account for
noise of the real world

D2A information is calculated
correctly because of internal factors

Output to analog
device is wrong

Software
computation does
not include a factor
for noise
Computation has
unknown error

Notes
Number of bits used to store the analog
converted data is not large enough or
sampling rate to get bits is not correct.
The noise term may not be known,
accounted for, or misrepresented.

Sources of error can come from:
calibrations used on variables, variables
lacking initialization, or calculations are not
done with enough accuracy (single versus
double floating point
Number of bits stored from the digital
world to the analog world do not have
enough precision, so analog data is
incorrect.
The analog values are not correct given the
noise of the real world (output data may be
lost in the noise).
Sources of error can come from:
calibrations used on variables, variables
lacking initialization, or calculations are not
done with enough accuracy (single versus
double floating point


58

29

8/20/2013

How?















Up front data gathering and analysis are important beginnings – what do we
know (or can ask about)
Identify input devices
Identify output devices
Define the input disturbances (unexpected system inputs)
Define possible output disturbances (unexpected system outputs)
Determine what is or is not possible in the test environment
Conduct a risk analysis (see likely bugs table)
Identify the users of the device and software
- Testers should be aware that embedded systems have resource
constraints in memory, CPU usage, and time
Use the above information to define an exploratory chart attack
Go run that attack
Learn
Design
Repeat (until time runs out)
Think!


59

Questions to Ask with this Attack


If the hardware is a prototype (not like what will be in the field), will
that impact testing or test results?



If a simulation is used, what bugs might be missed because actual
hardware or software is not used?



If the test inputs and environment are not representative of the real
world both in terms of expected and unexpected values, what risks
will be acceptable?



If the hardware is not understood, will testing be weak?



If the major sources of “noise” is not defined, will the system be
susceptible to impacts from unexpected inputs or outputs?



All of these questions will involve test tradeoffs, acceptable risk, and
compromise
◦ 40% of this kind of attack should be “normal” situations
◦ Start with normal and move to off-normal and stress cases


60

30

8/20/2013

Exercise – A2D/D2A



Work in groups
Key points

◦ Think how to look for bugs of chart 58
◦ Use the questions on chart 60 (note what is used)
◦ Follow pattern of chart 59 (note steps on charter)
 Can you expand with stories/tours?



Products
◦
◦
◦
◦





Risk list
User list
Test charter
Note bugs (if any)

Complete the feedback retrospective
Do several attacks (note what each one is)
Expand to “extreme cases”


61

Feedback – Retrospective Session


What did you accomplish?
◦ Bugs?
◦ Tests?



What things did you think of?
◦ Wish I had a _____???? I need more time???

What favors or opposes an attack?
 What would you do differently next cycle?



62

31

8/20/2013

Final Thoughts . . .


63

Wrap Up


This tutorial covers some basic introduction
(key attacks) and sampling
◦ There are many more

Understanding your local context and error
patterns is important (one size does NOT fit all)
 Attacks are patterns…you still must THINK!
 These attacks target Embedded and Mobile



64

32

8/20/2013

More Attacks (from my book and others)


Attack 1: Static Code Analysis



Attack 18: Bugs in Timing Interrupts and Priority Inversion



Attack 2: Finding White–Box Data Computation Bugs



Attack 19: Finding Time Related Bugs



Attack 3: White–Box Structural Logic Flow Coverage



Attack 20: Time Related Scenarios, Stories and Tours



Attack 4: Finding Hardware–System Unhandled Uses in Software



Attack 21: Performance Testing Introduction



Attack 5: Hw-Sw and Sw-Hw signal Interface Bugs

Attack 22: Finding Supporting (User) Documentation Problems





Sub–Attack 22.1: Confirming Install–ability



Attack 6: Long Duration Control Attack Runs



Attack 23: Finding Missing or Wrong Alarms



Attack 7: Breaking Software Logic and/or Control Laws



Attack 24: Finding Bugs in Help Files



Attack 8: Forcing the Unusual Bug Cases



Attack 25: Finding Bugs in Apps



Attack 9 Breaking Software with Hardware and System
Operations



Attack 26: Testing Mobile and Embedded Games



Attack 27: Attacking App–Cloud Dependencies

9.1 Sub–Attack: Breaking Battery Power



Attack 28 Penetration Attack Test



Attack 10: Finding Bugs in Hardware–Software Communications



Attack 28.1 Penetration Sub–Attacks: Authentication — Password Attack

Attack 11: Breaking Software Error Recovery





Attack 28.2 Sub–Attack Fuzz Test



Attack 29: Information Theft—Stealing Device Data



Attack 12: Interface and Integration Testing



Attack 29.1 Sub Attack –Identity Social Engineering



12.1 Sub–Attack: Configuration Integration Evaluation



Attack 30: Spoofing Attacks



Attack 13: Finding Problems in Software–System Fault Tolerance



Attack 30.1 Location and/or User Profile Spoof Sub–Attack



Attack 14: Breaking Digital Software Communications



Attack 30.2 GPS Spoof Sub–Attack



Attack 15: Finding Bugs in the Data



Attack 31: Attacking Viruses on the Run in Factories or PLCs



Attack 16: Bugs in System–Software Computation



Attack 32: Using Combinatorial Tests



Attack 17: Using Simulation and Stimulation to Drive Software
Attacks

Attack 33: Attacking Functional Bugs






65

Summary: Thank You (ideas used from)
James Whittaker (attacks)
Elisabeth Hendrickson (simulations)
 Lee Copeland (techniques)
 Brian Merrick (testing)
 James Bach (exploratory & tours)
 Cem Kaner (test thinking)







Many teachers
Generations past and future
Books, references, etc.


66

33

8/20/2013

Book List (my favorites)


“Software Test Attacks to Break Mobile and Embedded Devices”



“How to Break Software” James Whittaker, 2003
◦ And his other “How To Break…” books







“Testing Embedded Software” Broeckman and Notenboom, 2003
“A Practitioner’s Guide to Software Test Design” Copeland, 2004
“A Practitioner’s Handbook for Real-Time Analysis” Klein et. al., 1993
“Computer Related Risks”, Neumann, 1995
“Safeware: System Safety and Computers”, Leveson, 1995



Honorable mentions:
◦ “Embedded System and Software Validation” Roychoudhury, 2009
◦ “Systems Testing with an Attitude” Petschenik 2005
◦ “Software System Testing and Quality Assurance” Beizer, 1987
◦ “Testing Computer Software” Kaner et. al., 1988
◦ “Systematic Software Testing” Craig & Jaskiel, 2001
◦ “Managing the Testing Process” Black, 2002

– Jon Duncan Hagar, due out late 2013
(http://www.crcpress.com/product/isbn/9781466575301)


67

More Resources
•

www.stickyminds.com – Collection of test info
My Web site:
www.breakingembeddedsoftware.com

•

Association of Software Testing

•

– BBST Classes http://www.testingeducation.org/BBST/
•

Your favorite search engine


68

34

8/20/2013

Definitions (for this class)













Taxonomy – the practice and science of classification.
Test – the act of conducting experiments on something to determine the
quality and provide information
Test case – one set of inputs, environmental set up, and results (expected
and unexpected)
Attack – to set up, forcefully and attempt to “damage” the system or
software, using tools, methods, and techniques
Bug (error) – results that depart from the expected (from requirements,
design, standards, user, etc.)
Lifecycle – from beginning-to-end, the steps, stages, and activities to create
(birth-to-death)
Procedure – a particular way of accomplishing tests, usually written (one or
more test cases)
Tour – a journey to find information (tests) with a focus/direction (story)
Scenario – a sequence of events with a test plot or story
Script – see procedure, but normally uses automation
Users – someone/something that interacts with the system/software (can be
human or machine, or?)
Quality – value to someone and that they will pay for


69

Name of Test:


What to Test:
– Risk (s):

Success Criteria:
1.

– Attack

2.

– Other (requirements, …..)

•

3.

•


•

Actions:
– 1.
– 2.
– 3.

•

Others Steps


70

35

8/20/2013

Name of Test:


What to Test:
– Risk (s):

Success Criteria:
1.

– Attack

2.


•

3.

•


•

Actions:
– 1.
– 2.
– 3.

•

Others Steps



71

Name of Test:


What to Test:
– Risk (s):

Success Criteria:
1.

– Attack

2.


•

3.

•


•

Actions:
– 1.
– 2.
– 3.

•

Others Steps



72

36

8/20/2013

Name of Test:


What to Test:
– Risk (s):

Success Criteria:
1.

– Attack

2.


•

3.

•


•

Actions:
– 1.
– 2.
– 3.

•

Others Steps



73

Name of Test:


What to Test:
– Risk (s):

Success Criteria:
1.

– Attack

2.


•

3.

•


•

Actions:
– 1.
– 2.
– 3.

•

Others Steps



74

37

How to Break Software: Embedded Edition

Recomendados

Recomendados

Más contenido relacionado

Similar a How to Break Software: Embedded Edition

Similar a How to Break Software: Embedded Edition (20)

Más de TechWell

Más de TechWell (20)

Último

Último (20)

How to Break Software: Embedded Edition