Test Design and FreeNest 1.4 Draft Material

2. Test Design and Management
in context of IFDK reference product

3. About this course
material
- This material if for general training for Test
Design and management
- Material is more supportive in class room
- Material will be updated during courses
- FreeNest Portable Project Platform is used to
demonstrate things only in practice. This is
not limiting usage for material for other
training environments (I hope )
About

4. Example of IFDK
product concept
IFDK = Internal Flame Drum Kit
Requirement

5. Different aspects to
product
QA Tester
Customer
Seller
Developer
Yläotsikko

6. Different requirement levels
Customer/Business/Stake
Holder Requirements
System Requirements Scalability
Stability
Performance
Design Requirements
Security
Performance
Stress
Component Requirements Usabilty
Implementation

7. V-Model in testing
Verification and Validation
Customer Requirements Acceptance Testing
System Requirements System Testing
Sub System Requirements Integration Testing
Component Requirements Component/Unit Testing
Verification = Are we building the product right?
Validation = Are we building the right product?
Yläotsikko

8. Why we need requirements from testing point of view?
”Traditional Testing Levels”
„Test Engineers Area“
Customer/Business/Stake
Acceptance Testing
Holder Requirements
System Requirements System Testing
Design Requirements Integration Testing
„Developer's Area“
Component
Unit Testing
Requirements
Implementation

9. IFDK System
Verification and Validation
IFDK Product Ideas
VALIDATION Product
Features
Customer/Business Acceptance
Requirements Testing
Validation = Are we building the right product?
Use Cases
Verification = Are we building the product right?
System
Requirements System
Testing
User Storys Sub System Integration
Requirements Testing
Architecture&
Component Design& Component / Unit
VERIFICATION
Requirements Requirements Implementation Testing
VALIDATION vs VERIFICATION?
Yläotsikko

10. You have vision of product
What means testing in brief?
Product Design & Testing & Quality Assurance
Implementation
Ready
Features to test Test Case
Use Cases Test Case
Can we deliver Ready to
Product Deliver
User Storys Test Case
Not
ready to
deliver
? Customer
Requirements Test Case
Yläotsikko

11. Verification vs Validation?
Verification = Are we building the product right? Validation = Are we building the right product?
Verification Validation
Product Design & Testing & Quality Assurance
Implementation
Features Test Case
Ready
to test
Use Cases Test Case
Can we deliver Ready to
Product Deliver
User Storys Test Case
Not
ready
? Customer
Requirements Test Case
to
deliver

12. SW Development Process (Waterfall)
Requirement
Gathering/Evaluatio
n
Design
Implementation
Verification &
Validation
Error Managment
Process Maintenance
Mile Stone 1 Mile Stone 2 Mile Stone 3
Task1 Task1 Task1 Task1 Task1 Task1

13. Testing Orientation
“SYSTEM TESTING”
Black Box Testing
Perspective
Grey Box Testing
“CODE LEVEL TESTING”
White Box Testing
Perspective
http://en.wikipedia.org/wiki/Software_testing

14. Black Box vs White Box Testing
? ?
Black Box Testing
for selected Unit Testing is White Box testing
component
Yläotsikko

15. TESTING LEVELS

16. Unit/Module/Compone
nt Testing
VALIDATION
Customer/Business Product Acceptance
Requirements Testing
System
Requirements System
Architecture& Testing
Sub System Design& Integration
Requirements Implementatio Testing
n
Component Component / Unit
Requirements Testing
VERIFICATION
Yläotsikko

17. How to Test?
What should be tested?
How ?

18. How to test?
What should be tested?
How ?

19. How to verify component
implementation
-Unit Testing
-Code Coverage
-Branch Coverage
-Complexity
Analyse
Yläotsikko

20. Component /Unit Testing
Developer
Implemented Class Unit Test Frame Work
TestClass
Class
Test Method Call Attributes
Attributes
TestMethodCall
Methods Method Result
A=1
B=2
MethodCountValues( int x, int y) C=Class.TestMethodCo
z=x+y+1 untValues(A+B)
Return z C<>3 FAIL
C=3 PASS

21. Code Coverage
An analysis method that determines which parts of
the software have been executed (covered) by the
test suite and which parts have not been executed,
e.g. statement coverage, decision coverage or
condition coverage.
http://en.wikipedia.org/wiki/Code_coverage
http://www.atlassian.com/software/clover/
Yläotsikko

22. Branch coverage
The percentage of branches that have been
exercised by a test suite. 100% branch coverage
implies both 100% decision coverage and 100%
statement coverage.
Yläotsikko

23. Code Complexity
Example tool CCCC
https://wiki.jenkins-
ci.org/display/JENKINS/CCCC+Plugin
http://sourceforge.net/projects/codeanalyze-
gpl/?source=recommended
Yläotsikko

24. Integration Testing
VALIDATION
Customer/Business Product Acceptance
Requirements Testing
System
Requirements System
Architecture& Testing
Sub System Design& Integration
Requirements Implementatio Testing
n
Component Component / Unit
Requirements Testing
VERIFICATION
Yläotsikko

25. Integration Testing

26. How To Test ?
What should be tested?
How ?
http://prosentti.vero.fi/veropros_tietojen_syotto2011.asp

27. Why Integrate first? Avoid Big Bang!
Web Service
HW Component
Data Base
Component/Application
10% tested
Tested Component/Application
Yläotsikko

28. System Testing
VALIDATION
Customer/Business Product Acceptance
Requirements Testing
System
Requirements System
Architecture& Testing
Sub System Design& Integration
Requirements Implementatio Testing
n
Component Component / Unit
Requirements Testing
VERIFICATION
Yläotsikko

29. How to test?
What should be tested?
How ?

30. System Testing in Large
DB ?
Application Sales
& Gateways DB
Application
& Gateways
What should be tested? Application
How ?
CRM DB
Room
Reserva
tion DB

31. System Acceptance Testing
VALIDATION
Customer/Business Product Acceptance
Requirements Testing
System
Requirements System
Architecture& Testing
Sub System Design& Integration
Requirements Implementatio Testing
n
Component Component / Unit
Requirements Testing
VERIFICATION
Yläotsikko

32. What to test?
http://www.123rf.com http://www.123rf.com
What should be tested before so customer could be so happy ?
How ?

33. IFDK Verification/Validation (Testing Organization)
Product Release
Acceptance
Test Engineer
IFDK System Acceptance Testing
System
Test Engineer
Test Manager
IFDK System Testing
Project Manager
System Testing Designer/Coder
Integration
Test Automation Test Engineer
Error Manager
Feature Unit/Integration Testing
Feature Pack
Validation
Verification
Report Test
Error
Staus Management
Database
Database

34. What should be tested?
Input ?
Output?

35. What is Test Design?
REQ-X
REQ-O
What I
should
REQ-Z check ?
REQ-Y
Implementation
IDEAL
Yläotsikko

36. Why we need test design?
• Discuss about reasons for test design?
• Why we need to do design?
• Stupid work  ! I wan’t to progress!??

37. Test Case Design

38. UNDERSTAND YOUR TEST LEVEL
Product Release
Acceptance
Test Engineer
IFDK System Acceptance Testing
System
Test Engineer
Test Manager
IFDK System Testing
Project Manager
System Testing Designer/Coder
Integration
Test Automation Test Engineer
Error Manager
Feature Unit/Integration Testing
Feature Pack
Validation
Verification
Report Test
Error
Staus Management
Database
Database

39. What Information Test Case should contain?
Add Information about case • Test Case Id
• Test Case owner/writer
• Date
• comments
• Verify drum track player pause mode functionality.
• Do this with IFDK software release X and playing song
”Show must go on by Freddy Mercury”
Verify what?
• Test should be done using android emulator
Using configuration? environment and using your hands, ears and eyes”
With tools?
• Pre State:
• Android emulator is running
• Release X is installed on emulator
• Test Case Steps:
• 1. Open drum kit player application
• 2. Select song ”Show must go on”
• 3. Start to play
• 4. Press Pause and check song is paused
• 5. Check memory usage from system application
• 6. Press Play
Define pre-state • 7. jump to 4 several time (<10)
Define Steps • 8. Listen song to the end
Define end-state • 9. Exit player using ”exit button”
• End State:
• IFDK Kit in main screen mode
What is verdict? • If Pause is working result is PASS. If Pause mode failed
result is FAIL

40. Why we need test design again!
• Stupid work! This takes ages! This Test Case
documentation is old as soon I have changed
some implementation? Why you need to do
so hard documentation? Give me a one good
reason!

41. Checklist?
Check UI is working Working?
Check color change Working?
Check Counter value after 50 logins Working?
Check disable mode for counter Working?
Checklist can be working great in small
team! 
What happens if team is disbanded to
other projects? And you are new
maintainer for this project?

42. Agile Thinking?
• We have to automate all tests!! No sense to
create documentation ?
• Who does automation without a design?

43. Where I find sources for test design?
Specification based Testing design
Negative Testing
Design based test desing Test Case
Test Design Method
M
Requirement based test design i Test
Defect based test design x Engineer’s Write Step
e Daily Job? Step
Functional test design d Step
• Customer's Idea
• Brainstorm
NonFunctional test design
• Intitution
• Exploratory

44. How to create Test Case???
Acceptance Test Case
Functional Test Case
Check Non-Functional Test Cases
different
sources & WRITE
What is strategy Choose Field Test Case A
Testing level for Test Case
Case Type
Interoperability Test Case
Test
design
Case!
Conformance Test Case
Regression Test Case

45. Test Case Design in agile framework
User Story: As a user I would like to use my google account for login
Implementation
Done
Definition for
Acceptance Criteria
Definition for test cases
Check list Tested
1. Verify Test using test
2. Verify Test automation?
3. Verify Test
4. Verify Test

46. Test Driven Development
TDD in all levels!
Define Architecture &
Design draft Tests Case
Design?
Design Tests Implement Code

47. Unit Testing

48. Test Driven Development and Unit
Testing
Developer
DEFINE IMPLEMENT
TEST CODE
CASES AGAINST
Unit Test Frame Work Implemented Class TESTS
FIRST!!!
TestClass Class
Attributes Attributes
Test Method Call
TestMethodCall Methods
Method Result
A=1
B=2 MethodCountValues( int x, int y)
C=Class.TestMethodCo z=x+y+1
untValues(A+B) Return z
C<>3 FAIL
C=3 PASS

49. Ideal project team and unit testing
Software Product
Integration test engineer #1 Integration Integration test engineer #2
Test Sand Box Test Sand Box Test Sand Box Test Sand Box
Developer 1 Developer 2 Developer 3 Developer 4
TESTS TESTS TESTS TESTS
Implemented Implemented Implemented
Implemented
Software Software Component Software Component
Software Component
Component #3 #4
#2
#1

50. Integration Test with stubs/mocs
STUB/MOCK Component STUB/MOCK Component
Simulated
Interface
Log
Tested Component/Application
Control Interface
Messages/Events
Control
Configure
Scripted STUB Interface

51. Integration Testing with simulated
Interfaces
Simulated
Bluetooth
Interface
Fake Fake
Spotify Facebook
Server Server
Simulated
MIDI
Interface

52. Integration Test with stubs
STUB/MOCK Component STUB/MOCK Component
Simulated
Interface
Log
Tested Component/Application
Control Interface
Messages/Events
Control
Configure
Scripted STUB Interface
Yläotsikko

53. in practice #1 IFDK android setup
WEB SERVER simulating
STUB/MOCK Component
Service interface
Simulated
Interface
Tested Component
Application Trace/Log
Control Interface Activate/Control
Messages/Events
Control
Configure
Scripted STUB Interface
Yläotsikko

54. in practice #2 server component
testing
Mock Server/Daemon WEB SERVER
Simulated
Interface
Tested Component Trace/Log
Application Needed
Fake Application
Control Interface Activate/Control
Messages/Events
Operating System
Automated
Test Interface
Junit Scripted Interface Scripted STUB Interface
Yläotsikko

55. Integration Testing
Implementa Test
tion Framework
Error
Database
Test
Plan
Component
2
Implementa Test
tion Framework

56. Regression Test
Testing of a previously tested program following
modification to ensure that defects have not been
introduced or uncovered in unchanged areas of the
software, as a result of the changes made. It is
performed when the software or its environment is
Changed.
(ISTQB Syllabus)

57. Regression testing reason for
REGRESSION
automation?
TEST PLAN
Test Case Execute Regression ?
#1 Test when needed?
Test Case
#3
Test Automation Solution
Test Case
#4
Export Results as
Test Case Report
#7
Test Case
#8

58. How to create Test
Case???
Functional Test Case: Check also.....
●Requirement
●Use Case
●Feature Verify functionality of
● Write a Case ●Correct functionality
●User Story XXXX path
●Miss-usage of
functionality
Functional? ●Boundary Check
Your
Create a Test Sources Which
Case! For Test
Case Type? Regression Test
Case??
Non-Functional?
●Customer's Idea
●Brainstorm Non-Functional Test
●Intitution
Cases Check also.....
●Exploratory
Check Possiblity to
●
automated testing?
●Verify Stability of XXXX Write a Case
●Verify Performance of
XXX
●Verify Security of XXXX
●Verify Usability of XXXX
●Verify Scalability of
XXXX
●etc...
Yläotsikko

59. IFDK Verification/Validation (Testing Organization)
Product Release
Acceptance
Test Engineer
IFDK System Acceptance Testing
System
Test Engineer
Test Manager
IFDK System Testing
Project Manager
System Testing Designer/Coder
Integration
Test Automation Test Engineer
Error Manager
Feature Unit/Integration Testing
Feature Pack
Validation
Verification
Report Test
Error
Staus Management
Database
Database

60. System Testing in Small
Appcliation
DB
What should be tested?
How ?

61. Application Functional testing
Application
What should be tested?
How ?

62. Application code level testing

63. Test Case Management Basics
• Using Testlink
• How to write test case?
• Creating a test plan

64. About Testlink
TestLink is a web based Test Management tool. The application provides Test
specification, Test plans and execution, Reporting, Requirements specification and
collaborate with well-known bug trackers.

65. Test Management – Testlink
PASS
Feature
TestCase
Implementation
TestCase
TestCase
TestCase
TestCase
FAIL
Test Plan/Suite
Bug Report
SourceCode Test
Error
Version Management
Database
Control Database
TESTLINK

66. Test Plan Life Cycle
Test
Plan 1.0
Test
Plan 2.x
Needs
Still
Routine Work
Test Effort
REL 0.1 REL 0.2 REL 0.5 REL 1.0
Week 26 Week 52
Week 0

67. Selected Test Cases = Test Plan/Suite
Test Plan V 0.1
Test Case 1 Test Case 1
Test Case 2 Test Case 2
V2.0 V2.0
Test Case 3 Test Case 3
V2.0 V2.0

68. Test Case Pool and version control
Test Case 2 Test Case 3 Test Case 4
Test Case 1 V1.0 Draft
V1.0
Test Case 2 Test Case 2
V2.0 V2.0
Test Case 2
V3.0

69. Test Management
What we can do with Testlink
Tool

70. Who executes test cases and when?
Test Plan V 0.1
Test Case 1
Test Case 2
V2.0
Test Case 2
V2.0

71. What should be tested? Priority
setting
Test Case 2 Test Case 3
Test Case 1
V2.0 V2.0
P4 P1 P3

72. What is a target?
Product
RELEASE/BUILD
Test Plan V 0.1
VERSION 1.1
Executes
Test Case 1
Product
Executes
RELEASE/BUILD
Test Case 2
V2.0
VERSION 1.2
PASS / FAIL
Test Case 3
V2.0 Executes
Product
RELEASE/BUILD
VERSION 1.3
PASS / FAIL

73. Setting Configuration
Tested
Release/configuration
Version 0.4
Error/De
fect
Report
Test
Plan
Version Test Error/De
0.2.2
Cases fect
For Report
Features
Error/De
Version
0.2.2.2 fect
Report

74. Testlink use….

75. Needed Tools
TESTLINK:
http://sourceforge.net/projects/testlink/
BUGZILLA: http://www.bugzilla.org/

76. Execution of Test Case
Test Engineer daily job ?
Yläotsikko

78. What is generated as results from test case execution
TOOLS
SUT/DUT LOG FILE
TEST CASE Test Case ”IFDK- Indications Events
unit” NOTIFICATIONS
ENVIRONMENT/
EMULATOR
SUT = System Under Test
DUT = Device Under Test
Yläotsikko

79. Regression Test
Testing of a previously tested program following
modification to ensure that defects have not been
introduced or uncovered in unchanged areas of the
software, as a result of the changes made. It is
performed when the software or its environment is
Changed.
(ISTQB Syllabus)

80. Test Plan & Regression
TEST PLAN 1 REGRESSION TEST PLAN 1.1
TEST PLAN
Test Case Test Case Test Case Test Case
#1 #2 #9 #2
Test Case
#1 New
Test Case Test Case TCases Test Case Test Case
#3 #4 Added! #10 #11
Test Case
#3
Test Case Test Case Test Case Test Case
#5 #6 #5 #6
Test Case
#4
Test Case Test Case Test Case Test Case
#7 #8 #9 #12
Test Case
#7
Test Case Test Case Test Case Test Case
#9 #10 #13 #10
Test Case
#8

81. Regression testing reason for
REGRESSION
automation?
TEST PLAN
Test Case Execute Regression ?
#1 Test when needed?
Test Case
#3
Test Automation Solution
Test Case
#4
Export Results as
Test Case Report
#7
Test Case
#8

82. Product verification/validation and test automation A
Product Release
Acceptance
Test Engineer
System Acceptance Testing
System
Acceptance Testing Test Engineer
Test Manager
System Testing
Project Manager
Functional System Testing A Load, Stress, Designer/Coder
performance
Integration
Regression Testing A Test Engineer
Test Automation
Feature Unit/Integration Testing Engineer
Integration Testing
Component Integration Testing A
Validation
Feature
Component Verification
A
Unit Testing

83. Heading
- Sed posuere interdum sem.
- Quisque ligula eros ullamcorper quis, lacinia
quis facilisis sed sapien.
- Mauris varius diam vitae arcu. Sed arcu
lectus auctor vitae, consectetuer et venenatis
eget velit.
- Sed augue orci, lacinia eu tincidunt et
eleifend nec lacus.
Yläotsikko

84. Component Design
Class
Attributes
Methods
Yläotsikko

85. Feature example 1 (Invented on course 2009-2010)
Calory Counter:
Player can measure calories during training session. This can be
seen as exercise result in web service eg. Facebook application
Feature X * n
Energy usage
Yläotsikko

86. SW Development Process (Agile)
Product Backlog
Task4
User Story X
Task5
Task1 Task6
User Story Y Task7
Task2
Task8
User Story Z Task3
Task9
User Story Z Sprint Sprint Sprint Sprint
Design Design Design Design
Implementation Implementation Implementation Implementation
Verification Verification Verification Verification
Yläotsikko

87. Yläotsikko