1. Test ExecutionTest Execution
& Reporting& Reporting

2. AgendaAgenda
 General Test procedureGeneral Test procedure
 Executing testsExecuting tests
Set up systems and softwareSet up systems and software
Execute test casesExecute test cases
 Verifying and analyzing resultsVerifying and analyzing results
 ReportingReporting

3. General Test ProcedureGeneral Test Procedure
A Single test normally includes three steps:A Single test normally includes three steps:
 PlanningPlanning
 ExecutionExecution
 EvaluationEvaluation
Tester Plan testing
Execute
testing
Evaluate
testing

4. General Test ProcedureGeneral Test Procedure
Plan testing
Execute testing
Evaluate testing
If I click OK, without input into text
field, the application will probably
crash.
Let’s see if it does
It sure did!

5. PlanningPlanning
 It is the first step to determine what should be tested. AlsoIt is the first step to determine what should be tested. Also
referred as test case design.referred as test case design.
 Design the test cases (inputs and outputs) used to test theDesign the test cases (inputs and outputs) used to test the
system:system:
 Identify the test case (Identify the test case (What has to be testedWhat has to be tested).).
 Specify the test activities(Specify the test activities( How to proceedHow to proceed).).
 Indicate how test results should be defined (Indicate how test results should be defined (Correct outcomeCorrect outcome).).

6. ExecutionExecution
It’s Show Time!It’s Show Time!
Execution of a test is a three stepExecution of a test is a three step
Process:Process:
 Preparing the Test InfrastructurePreparing the Test Infrastructure
 Conduct Test.Conduct Test.
 Verify Results and Report Results.Verify Results and Report Results.

7. Prepare Test InfrastructurePrepare Test Infrastructure
 The prepare Test Infrastructure taskThe prepare Test Infrastructure task
ensures that all the components such asensures that all the components such as
technical resources, human resources,technical resources, human resources,
configuration, data, and test designconfiguration, data, and test design
specification are in place for successfulspecification are in place for successful
execution of the test.execution of the test.
 The key to setting up the testingThe key to setting up the testing
environment is to establish a set ofenvironment is to establish a set of
standards that everyone in the project canstandards that everyone in the project can
adhere to.adhere to.

8. Prepare Test InfrastructurePrepare Test Infrastructure
Establish a set of standards that theEstablish a set of standards that the
organization follows.organization follows.
 The key to setting up the testingThe key to setting up the testing
environment is to establish a set ofenvironment is to establish a set of
standards that everyone in thestandards that everyone in the
project can adhere to.project can adhere to.

9. PreparePrepare Test InfrastructureTest Infrastructure
 Components of the testingComponents of the testing
environment include:environment include:
Supporting software
An inventory of hardware
The code under test
A workbook or execution logs

10. PreparePrepare Test InfrastructureTest Infrastructure
Supporting softwareSupporting software
 Includes operating systems, middleware,Includes operating systems, middleware,
system utilities, application, patches and sosystem utilities, application, patches and so
on.on.
 Supporting software is stored on CD orSupporting software is stored on CD or
available on a shared file system.available on a shared file system.
 All supporting software will need to beAll supporting software will need to be
installed in the environment in the exactinstalled in the environment in the exact
sequence required by the components.sequence required by the components.

11. PreparePrepare Test InfrastructureTest Infrastructure
Supporting softwareSupporting software
 Detailed installation steps will need to be providedDetailed installation steps will need to be provided
to you as part of the implementation of the testto you as part of the implementation of the test
cases.cases.
 When installing multiple supporting softwareWhen installing multiple supporting software
elements, the order of installation is critical to theelements, the order of installation is critical to the
proper functioning of the environment.proper functioning of the environment.

12. PreparePrepare Test InfrastructureTest Infrastructure
An inventory of hardware:An inventory of hardware:
 Properly setting up the test-execution environmentProperly setting up the test-execution environment
requires to have access the inventory of hardwarerequires to have access the inventory of hardware
that is needed, along with any special instructionsthat is needed, along with any special instructions
for configuring the hardware.for configuring the hardware.
 It is important to ensure that you execute the testIt is important to ensure that you execute the test
case in the exact hardware configuration identifiedcase in the exact hardware configuration identified
by the test cases.by the test cases.
 Tester must confirm that the hardware meets allTester must confirm that the hardware meets all
requirements and specifications such as those forrequirements and specifications such as those for
memory, disk space, processor speeds and so on.memory, disk space, processor speeds and so on.

13. PreparePrepare Test InfrastructureTest Infrastructure
Code under test:Code under test:
 Code under test, also referred to as a driver or aCode under test, also referred to as a driver or a
build is the packaged new code that is ready to bebuild is the packaged new code that is ready to be
installed and tested.installed and tested.
 Properly preparing the environment for test-caseProperly preparing the environment for test-case
execution also requires that you have access to theexecution also requires that you have access to the
code under test.code under test.
 Code is stored in a well-known place with namingCode is stored in a well-known place with naming
conventions that indicate the exact driver or buildconventions that indicate the exact driver or build
levels.levels.
 Detailed procedures need to be in place to enableDetailed procedures need to be in place to enable
you to install the code accurately and efficiently.you to install the code accurately and efficiently.

14. PreparePrepare Test InfrastructureTest Infrastructure
Workbook or execution logs:Workbook or execution logs:
 Workbooks and execution logs are used to recordWorkbooks and execution logs are used to record
the test-execution activity.the test-execution activity.
 Workbooks are used when manually running tests,Workbooks are used when manually running tests,
while execution logs are used to capture executionwhile execution logs are used to capture execution
of tests run programmatically.of tests run programmatically.
 Store the log files in a well-known place withStore the log files in a well-known place with
naming conventions to indicate the date capturednaming conventions to indicate the date captured
and the build level being tested.and the build level being tested.

15. Conduct TestConduct Test
Execute Test caseExecute Test case
 The purpose of the task is toThe purpose of the task is to
implement the test executionimplement the test execution
plan, produce test results, andplan, produce test results, and
compare between the expectedcompare between the expected
test results and actual testtest results and actual test
results.results.
 After set up the environment,After set up the environment,
ready to execute the tests.ready to execute the tests.
 Execute each test exactly asExecute each test exactly as
defined in the test-casedefined in the test-case
implementation.implementation.

16. Execute Test case – The processExecute Test case – The process
Software
to be tested
Test Case
Output
Verifier
Test Case
Generation

17. Testing processTesting process
Design test
cases
Prepar e test
data
Run pr ogram
with test da ta
Compar e results
to test cases
Test
cases
Test
data
Test
results
Test
repor ts
InputInput
Actual OutputActual Output
Comparison ProcessComparison Process
Test case : Specifies
• Inputs + pre-test state of
the software
• Expected results (outputs
and state)
Test case : Specifies
• Inputs + pre-test state of
the software
• Expected results (outputs
and state)

18. Testing policesTesting polices
 Only exhaustive testing can show a programOnly exhaustive testing can show a program
is free from defects. However, exhaustiveis free from defects. However, exhaustive
testing is impossible,testing is impossible,
 Testing policies define the approach to beTesting policies define the approach to be
used in selecting system tests:used in selecting system tests:
• All functions accessed through menusAll functions accessed through menus
should be tested.should be tested.
• Where user input is required, all functionsWhere user input is required, all functions
must be tested with correct and incorrectmust be tested with correct and incorrect
input.input.

19. Testing guidelinesTesting guidelines
 Testing guidelines are hints for the testingTesting guidelines are hints for the testing
team to help them choose tests that willteam to help them choose tests that will
reveal defects in the systemreveal defects in the system
• Choose inputs that force the system toChoose inputs that force the system to
generate all error messages.generate all error messages.
• Design inputs that cause buffers to overflow.Design inputs that cause buffers to overflow.
• Repeat the same input or input series severalRepeat the same input or input series several
times.times.
• Force invalid outputs to be generated.Force invalid outputs to be generated.
• Force computation results to be too large orForce computation results to be too large or
too small.too small.

20. Verify the existence of the DefectVerify the existence of the Defect
 Verify that the element under test,Verify that the element under test,
the execution environment and thethe execution environment and the
test case-implementation aretest case-implementation are
consistent with the test case designconsistent with the test case design
 Verify and confirm the discrepancyVerify and confirm the discrepancy
between the expected test resultsbetween the expected test results
and actual test results.and actual test results.
Verify and Analyze ResultsVerify and Analyze Results

21. Analyze the DefectAnalyze the Defect
 The purpose of the task is toThe purpose of the task is to
analyze and resolve variancesanalyze and resolve variances
encountered during test execution.encountered during test execution.
 Each variance is analyzed toEach variance is analyzed to
determine the cause.determine the cause.
 Analysis may indicate that the causeAnalysis may indicate that the cause
of the variance is a defect, or that aof the variance is a defect, or that a
change request is required.change request is required.
Verify and Analyze ResultsVerify and Analyze Results

22. Analyze the DefectAnalyze the Defect
 Investigate further andInvestigate further and
gather as muchgather as much
information as possibleinformation as possible
about the symptoms.about the symptoms.
 Analyze failures by varyingAnalyze failures by varying
conditions, options andconditions, options and
settings to gauge thesettings to gauge the
severity of the defectseverity of the defect
Verify and Analyze ResultsVerify and Analyze Results

23. Analyze failures by varying conditions, options and settingsAnalyze failures by varying conditions, options and settings
 Change theChange the conditionsconditions under which an error occurred byunder which an error occurred by
changing what you do. As an example, if the problem seemschanging what you do. As an example, if the problem seems
related to entering data very quickly, try slowing your rate ofrelated to entering data very quickly, try slowing your rate of
input.input.
 Change theChange the options and settingsoptions and settings of the program underof the program under
test. As an example, if the problem seems related to desktoptest. As an example, if the problem seems related to desktop
screen-resolution settings, try varying the settings.screen-resolution settings, try varying the settings.
 Change theChange the software and hardware environmentsoftware and hardware environment. For. For
example, if the defect seems related to a low-end hardwareexample, if the defect seems related to a low-end hardware
configuration, try using the lowest configuration specified inconfiguration, try using the lowest configuration specified in
the requirements.the requirements.
 Change theChange the datadata used by the program. As an example, if theused by the program. As an example, if the
defect seems related to strings exceeding a certain numberdefect seems related to strings exceeding a certain number
of characters in length, create a second set of test data toof characters in length, create a second set of test data to
check the hypothesis.check the hypothesis.
Verify and Analyze ResultsVerify and Analyze Results

24. Documenting yourDocumenting your
execution resultsexecution results
 Test logs to recordTest logs to record
test executiontest execution
 Create a testCreate a test
incident report forincident report for
problemsproblems
ReportingReporting
Run Test
Test Logs
ERROR
REPORT
TEST CASES
Verify & Analyze Result
Bug Tracking

25. Effective defectEffective defect
reportsreports
 Convey the properConvey the proper
message, gets themessage, gets the
job done andjob done and
simplifies thesimplifies the
process forprocess for
everyone.everyone.
 Can Pig RideCan Pig Ride
technique to writetechnique to write
effective incidenteffective incident
reports.reports.
ReportingReporting
C Condense: Say it clearly but briefly
A
Accurate: Is it a user error or set-up problem?
System problem? Application problem?
Test-case problem?
N
Neutral: Just the facts; no zingers, no humor,
no emotion, no sarcasm
P Precise: Explicitly, what is the problem?
I Isolate: What was done to isolate the problem?
G
Generalize: What was done to understand
how general the problem is?
R
Re-create: What is essential to
re-create this problem?
I
Impact: What is the impact if the bug
were to surface in a customer environment
D
Debug: What will the developer
need to debug this problem?
E Evidence: What will prove the existence of the error?

26. Test-Defect reportTest-Defect report
 The IEEE Std. for Software Test Documentation forThe IEEE Std. for Software Test Documentation for
Test Defect template offers the following fields.Test Defect template offers the following fields.
ReportingReporting
• Incident Summary report Identifier
Specify the unique identifier assigned to this test-defect
report.
•Incident Summary
Summarize the incident. Identify the test items involved,
indicating their version and revision level. Supply references to the
appropriate test-procedure specification, test-case specification and
test log.

27.  Incident Description
Provide a description of the incident that
includes the following items:
 Inputs
 Expected results
 Actual results
 Date and time
 Procedure steps
 Environment
 Attempts to repeat
 Testers
 Observers
 Related activities and observations
Include any of these that may help to
isolate the correct the cause of the incident.
Test-Defect reportTest-Defect report
 The IEEE Std. for Software Test Documentation forThe IEEE Std. for Software Test Documentation for
Test Defect template offers the following fields.Test Defect template offers the following fields.
ReportingReporting

28. ResolveResolve
 True defects - referred to the DefectTrue defects - referred to the Defect
Management process.Management process.
 Rated for severity and a priorityRated for severity and a priority
 Change Requests - referred to the ChangeChange Requests - referred to the Change
Management processManagement process
Prioritizes the revision of requirements,Prioritizes the revision of requirements,
the specification, design and constructionthe specification, design and construction
of each item.of each item.
Changed items reenter the test processChanged items reenter the test process
ReportingReporting