3. Software Development Life Cycle
Requirements Tests Defects
What do we
make and how
should it
behave
Make sure it
works as
stated in the
Requirement
Definition Sets Environments
Actual Results
do not equal
Expected
Results
4. What is a Test Framework?
A Test Framework:
• Links tests to other SDLC items
• Is NOT a Test Automation Framework but often contains one
• Allows for rapid creation of tests from reusable components
• Separates data from logic (REUSABILITY)
• Provides a standardized test “language” and reporting structure for an application
under test
6. 3 D’s of Testing: Define
Define Workflows (the easy part)
• Startwith requirements
• Each requirementshouldbe at least one test case
• Identify and detailcommon workflows thru the applicationundertest
• If you use User Stories,you are already mostly there
7. 3 D’s of Testing: Define
Define Requirement Example: QAComplete.com ROI Calculator
• Role:As a QAComplete Prospect
• Perspective: Interestedin determiningwhat QAComplete can do for me
• Context: On the ROI Calculatorpage in a web browser
8. The 3 D’s of Testing: Define
Define Example Workflow—QAComplete.com ROICalculator
• The desired path with minimal variation
• User selects values for the 4 inputs or leavesthe defaults
• User reviewsthe 3 pieces of returned data
• User clicksPrint
• User clicksContact Us
9. How to Build a Test
Framework—The 3 D’s
Decompose
10. The 3 D’s of Testing: Decompose
Decompose the Workflow
• Definitions should be as reasonably atomic as possible, yet still useful
• Break down:
• Actions that provide an input (data)
• Results of those actions
• Workflow logic
• Exit points from the workflow
11. The 3 D’s of Testing: Decompose
Decompose Workflow Example—QAComplete.com ROI Calculator
1.Userselectsortypesinavalueforteammembers
2.Userselectsortypesinavalueforrequirementsto
beimplementedthisyear
3.Userselectsortypesinavaluefortestcasesper
requirements
4.UserleavesSaaSasaselection,orswitchestoOn-
Premise,orcanswitchbackandforth
5.QACompleteImplementationcostinUSD
6.SavingsavailableduetoQACompleteinUSD
7.ROIinthefirstyearasapercent
8.UsercanclickContactUslink
9.UsercanclickPrint,whichwillprovideaprintable
versionof1-7
Inputs
Outputs
Exits
14. The 3 D’s of Testing: Decide
What Environments /Contexts to Cover?
• Operating Systems
• Browsers
• Devices (Mobile & Tablets)
• Security Permissions
• User Roles
• Conflicting Software
15. The 3 D’s of Testing: Decide
What Tasks and Steps Repeat?
• After decomposing all major workflows:
• Find commonly reused steps and actions
• Find actions and checks that use the same data
• Example: Workflow 1 and Workflow 2 from the ROI Calculator
16. The 3 D’s of Testing: Decide
What to Automate?
• EnvironmentSetup/Teardown
• DataEntry
• FormFilling
• Varyingdatainputsinarepetitiveprocess
• Exposingbackenddata(APIs,DBtable,etc…)
• Repetitive/boringtasksthatarepronetoinattentionerrors
• Taskswithhighreusevalueacrossmanyworkflows
• Testswithtimingorscreenresponsivenessasacriteriaforsuccess
• Manynon-functionaltesttypes,suchasperformancetesting
• Captureresults
18. Software Development Life Cycle
Requirements Tests Defects
What do we
make and how
should it
behave
Make sure it
works as
stated in the
Requirement
Definition Sets Environments
Actual Results
do not equal
Expected
Results
19. Elements of a Test Framework
• Library
• Test Data Sources
• HelperFunctions
• Test Environments
• Modules
• Structure/ Hierarchies
20. Elements of a Test Framework
• Library:Arepositoryofallyourdecomposedscripts,separatedintotheircomponents.
• Example:InputsonlyfromWorkflow2,separatedfromInputsselectedbysliderinWorkflow1,separatefrom
thecheckpointsusedinboth
• TestDataSources:Arepositoryofalldatasources
• Example:Inputsandexpectedresultsusedbybothworkflowsisonedatasources
• HelperFunctions:Arepositoryofalldecomposedtestscripts,automatedormanual,thatarenotinputsorchecks
• Example:Setupscripts,cleanupscripts,backenddatacapture,other“tasks”
• TestEnvironments:Alistofallcoveredtestingenvironments,brokenoutbytape(OS,Browser…)
• Bestwhencombinedwithassociatedhelperfunctionssuchassetupscriptsandaccessinformation
21. Elements of a Test Framework
Modules
• Thecombinationoflibraryitemswithanyhelper
functionsandtestdatasources—plusenvironments
• Example:ROICalculator
• Workflow1inputs“ROIslider”usingData
SourceA
• ROICalculatorChecksusingDataSourceA
• Workflow2inputs“ROItypein”usingData
SourceA
• ROICalculatorChecksusingDataSourceA
• RepeatforCoveredEnvironments:IE,
Chrome,Firefox,etc.
Structure/Hierarchies
• The“folder”structureofmodules
• Oftenorganizedinasimilarformatastheapplication
undertest
• Groupedinparent/childrelationships
22. Elements of a Test Framework
Defects
• Importanttorelatetotestsfor
retesting
• Importanttorelateto
requirementsfor“brokenas
designed”
Releases
• Listofallchangingrequirements
ingivenperiod/build
• Agreatframeworkcanbuildit’s
testplanjustfromthislistasthe
planisequaltoalllinked
requirementsandtheirtests
Requirements
• Mustberelatedtotestsfor
coveragereportingtoensurethat
allchangesaretested
• Feedstestdesign
23. Join us at 8:30 AM | 2:00 PM EDT
November 7th
https://smartbear.com/resources/webinars/elements-of-a-ui-automation-framework/
Carson Underwood
QA Engineer
O’Reilly Auto Parts
Bria Grangard
Product Marketing Manager
SmartBear Software
Notas del editor
Bria
Bria
Bria leads, Jeff adds comments (feel free to interrupt but Bria will initiate conversation because she actually understands this slide)
Let’s star high level with the software development lifecycle. While you all are probably very familiar with this let’s talk about some points.
Requirements. Where it all begins. What do we want to make, what will we make, and how should whatever we make really behave.
Need to have a solid foundation for success… you need to know what your requirements are and what you want them to accomplish. No confusion. It’s kind of like taking that moment to prepare for your essay before you write it.
For Tests you can have multiple types of tests.
Test Definition
Test set
What environments you want to run the test on-operating systems, browsers, resolutions, etc.
Defects—it’s inevitable. Your tests are going to find some bugs and there will be some problem. But if you have a defect you want to be able to log it properly and tie it to the correct test and requirement.
Releases—all of these things are encapsulated in a release. What are you pushing out for the next major or minor release/build.
Jeff leads
Bria
Bria leads
Framework starts as a collection of workflows
Every requirement should have some associated tests. You shouldn’t have a requirement without a test because then how do you know whether or not your requirement is successful.
It’s important to look and see if there is a common thread between your workflows for the application under test and if so… note them. These will be important later as you continue to build out your test framework
You may be using user stories.. If this is the case you’re already almost there for defining your workflows
Bria leads
Now let us walk you through an example. Recently I was working on a project-–QAComplete.com. For this website I needed to define my requirements.
In order to do this I needed to put myself in the place of whomever will be visiting this website. This is important to get perspective of what is necessary.
Role: sets expectations, is a way to fill in unwritten requirements
Perspective: separate from role to make later reporting easier, provides insight into desires and mindset
Context: very important as it provides a starting point AND ENVIRONMENT, in more complex tests provides input data as well—you cannot forget this part of things and sometimes people do. What is the context in which you are defining your requirements.
Jeff leads
Jeff leads
Jeff leads
Little bit of an art and usually requires knowledge of the application under test
Bria leads—Jeff definitely adds comments lol
Note there is no Data, but formats are specified
Jeff leads
Bria leads
Bria leads
So now it’s time to make some decisions. You’ve defined key things to be successful, and you’ve broken them down into various parts, but now you have to make some decisions for each part. S…me of these decisions may include.
Jeff leads
Jeff leads—Bria interjects
Bria leads
Bria leads
Let’s bring it back. We know about how to craft a testing framework but what are the actual elements of the testing framework for every stage of the SDLC
Jeff leads
Jeff Leads
It may be helpful to specify INPUTS, OUTPUTS, Logic, and exits
There are implicit exits as well (close window, click to other area of website)