a step to step guide to do software test automation on large projects.

1. Testing Automation
From Start to Finish
Liang Gao

2. Testing Automation Attribute:
贵 expensive
2 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

3. Testing Automation Attribute:
冰山 iceberg
3 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

4. Agenda
∗ Realize when you need automation
∗ Always remember the goal
∗ Find the way to control your product
∗ API design
∗ Define the development process
∗ Script Quality
∗ Do I need an automation framework?
∗ Define the execution process
∗ Team management
∗ Equipment management
4 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

5. Automation is useful for….
∗ Frequently functionality regression testing
∗ Sample based performance testing
∗ Repetitive work for system and solution testing
5 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

6. Frequently functionality regression
testing
∗ Faster turnaround
∗ Reduce human errors
∗ Cost saving
6 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

7. Sample based performance testing
7 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

8. system and solution testing
∗ Semi-automation
∗ Configure 1000 VPN tunnels
∗ Create 10000 routing table entries
∗ …..
8 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

9. Agenda
∗ Realize when you need automation
∗ Always remember the goal
∗ Find the way to control your product
∗ API design
∗ Define the development process
∗ Script Quality
∗ Do I need an automation framework?
∗ Define the execution process
∗ Team management
∗ Equipment management
9 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

10. The Goal
∗ No matter manual or auto
Accurate
Testing Report
10 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

11. Agenda
∗ Realize when you need automation
∗ Always remember the goal
∗ Find the way to control your product
∗ API design
∗ Define the development process
∗ Script Quality
∗ Do I need an automation framework?
∗ Define the execution process
∗ Team management
∗ Equipment management
11 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

12. Remote Control Your Products
Personal
Computer Web Browser
Truck
Tablet Video
Printer
Camera
Telecommuter
House with
Web Server
Router Video
Refrigerator Phone
Television Woman -
End User
Scanner
Television -
Closed-circuit
Weather
Station Woman -
Set Top Box Sitting
(STB)
12 Tunnel This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

13. Remote Control Your Testing
∗ 2 ways of software manual testing (or Device Control)
∗ Mouse and Eye (expectation)
∗ Keyboard and Eye (expectation)
∗ Automation is to repeat
∗ Mouse and Eye – GUI automation
∗ Keyboard and Eye – CLI automation
13 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

14. GUI Automation
∗ Windows based GUI automation
∗ AutoIT
∗ Browser based GUI automation
∗ QTP
∗ WinRunner
∗ Silktest
∗ AutoIT
∗ Selenium
14 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

15. CLI Automation
∗ Expect
∗ TCL Expect
∗ Perl Expect
∗ Python Expect
15 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

16. Embedded devices State of the
Union
∗ Most network devices (network equipment)
∗ Management interface: Browser or console
∗ Mobile devices
∗ Keyboard simulation
∗ Serial port, USB port or Bluetooth
16 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

17. A Sigma Approach (in progress)
Keyboard simulation,
predefined key sequence
(XML), render engine and
Screen capture
Result Verification
(manual, only once)
Replay keyboard simulation,
Screen capture and Image
comparison
17 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

18. Pick the Tool is just beginning
万里长征第一步
18 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

19. Agenda
∗ Realize when you need automation
∗ Always remember the goal
∗ Find the way to control your product
∗ API design
∗ Define the development process
∗ Script Quality
∗ Do I need an automation framework?
∗ Define the execution process
∗ Team management
∗ Equipment management
19 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

20. API Design
∗ Lessons learned: API (Library) design is key for
∗ Productivity
∗ Modularity
∗ Maintainability
∗ Expansion
∗ Don’t start with programming, start with API design
∗ 磨刀不误砍柴工 – It is true…
20 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

21. API Design
∗ Design pattern or other OO technique apply
∗ Need senior software architect blessing
∗ More design brain storming sessions, more reviews,
the better.
∗ Spend more time on the API design, it will pay off in
the long run
21 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

22. API Principle
Scripts
API
Control Library
22 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

23. API Design - Principle
•Connect to Device Device_id = connect_to_device();
∗ Script only contains testing actions, no software
•Issue XXX command to send Status = Send_SMS(device_id, $string);
independent info
SMS messages
Assert($status);
•Verify messages have been
Status = Get_SMS(sent, $string)
sent successfully
Assert($status)
•Verify messages saved in
the SMS sent folder Status = delete_SMS(sent,$string);
•Delete message Assert($status)
•Verify message has been
deleted
23 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

24. API User Manual
∗ Most important document for script development
API 名称
描述
输入参数 参数 1
参数 2
参数 3
返回值
使用举例
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.
24

25. Agenda
∗ Realize when you need automation
∗ Always remember the goal
∗ Find the way to control your product
∗ API design
∗ Define the development process
∗ Script Quality
∗ Do I need an automation framework?
∗ Define the execution process
∗ Team management
∗ Equipment management
25 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

26. Script Development Process
×Ô¶¯»¯½Å±¾¿ª·¢Á÷
³Ì²Î¿¼
26 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

27. Define Script Quality
∗ All common development coding standard applies
∗ Version control
∗ Standard template
∗ Standard header
∗ 20% of the code is comments
∗ Well indented
∗ Use API and functions
∗ Enough error checking
27 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

28. Agenda
∗ Realize when you need automation
∗ Always remember the goal
∗ Find the way to control your product
∗ API design
∗ Define the development process
∗ Script Quality
∗ Do I need an automation framework?
∗ Define the execution process
∗ Team management 28 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

29. Define Script Quality
∗ Testing script has it own quality too
∗ Exactly match the manual testing steps
∗ Just by reading logs, we know exactly what scripts are
trying to do
∗ Device and hardware/software environment independent
∗ Enough debugging message when test case failed
∗ Clean out environment before exit or before execution
29 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

30. Script Quality Checklist
Sigma Script Quality Others Script
Quality Checklist
Checklist
30 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

31. Agenda
∗ Realize when you need automation
∗ Always remember the goal
∗ Find the way to control your product
∗ API design
∗ Define the development process
∗ Script Quality
∗ Do I need an automation framework?
∗ Define the execution process
∗ Team management 31 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

32. Automation Framework ？
∗ Always remember our goal
∗ You need at least an execution engine for
∗ Error resuming batch run
∗ Report generation
32 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

33. Automation Framework
∗ Want to design and implement an Automation
Framework?
∗ Checkout
<<Automation Framework: Design and
Implementation>>
33 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

34. Agenda
∗ Realize when you need automation
∗ Always remember the goal
∗ Find the way to control your product
∗ API design
∗ Define the development process
∗ Script Quality
∗ Do I need an automation framework?
∗ Define the execution process
∗ Team management 34 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

35. Execution Process
Always remember the goal
35 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

36. Execution Process
∗ Image upgrade process
∗ Parallel execution
∗ Total run time
∗ Debugging time
∗ Report collection and generation
36 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

37. Image upgrade process
∗ Where to get the image
∗ Can you automatically get the image and upgrade to
your product
∗ Call this a “clean” procedure
37 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

38. Parallel execution
∗ To save total run time – sometimes even automation
is too long
∗ This is where the script need to be environment
independent
∗ Be aware on the equipment cost issue
∗ How many parallel scripts can an engineer run and
manage (also equipments)
38 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

39. Run Time
∗ Total run time per engineer
∗ Total Debugging time per engineer
∗ Total reporting time per engineer
39 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

40. Report collection and generation
∗ If it is parallel, need a process to parallel report
collection and generation
∗ Testing Information Database?
∗ Web Interface?
40 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

41. Agenda
∗ Realize when you need automation
∗ Always remember the goal
∗ Find the way to control your product
∗ API design
∗ Define the development process
∗ Script Quality
∗ Do I need an automation framework?
∗ Define the execution process
∗ Team management 41 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

42. Team Management
∗ Who write the scripts?
∗ Who write the framework
∗ Who execute the scripts
∗ Are they happy
42 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

43. Who Write Scripts
∗ Whoever write the test plan, write the test scripts
∗ Knows test case the best
∗ Not professional coder
∗ Hard for project management
∗ Professional scripter
∗ Need to understand the test case first
∗ High productivity
∗ Good Quality
43 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

44. Who Write Framework
∗ Or any other testing related tools
∗ If a tester develop a great testing tool, who maintains
it?
∗ Who to coordinate the common share library
development
∗ Who to develop testing tools if there is a request
∗ Most of the time developer does this.
∗ If have resources, think about a Tools Team
44 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

45. Who Execute the Scripts
∗ Whoever write the scripts, execute the scripts
∗ Low productivity
∗ Bad script portability
∗ Unpredictable schedule (bad for scheduled release)
∗ Dedicated resources (professional executer)
∗ Streamline resources
∗ High productivity
∗ Fixed total run time
∗ Most modern companies do it this way
45 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

46. Interaction process
∗ Test cases writer and professional scripter
∗ Scripter and executer
46 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

47. Case Writer and Scripter
∗ Repeatable test cases
47 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

48. Scripter and Runner
∗ Script integration process
∗ Newly develop scripts submit to runner
∗ Use “State” to manage
∗ Script maintenance process
∗ New image change
∗ Use “State” to manage
48 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

49. Team Building
∗ Manual tester, scripter are doing creative things
∗ Script runner may be board
∗ Role rotation
∗ Clear role for a clear career path
49 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

50. Streamlining the team
脚本开发 脚本开发 脚本开发 脚本开发
脚本执行 脚本执行 脚本执行
50 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

51. The Goal
∗ No matter manual or auto
Accurate
Testing Report
51 This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.