Test automation brings benefits but also struggles like many failure reports. Currently, temporary unstable failures make up 73.4% of issues but investigating them wastes time. The presenter proposes an auto recover system to automatically re-run tests predicted to fail due to temporary issues. This could reduce wasted monthly operation time from 75 hours to around 25 hours by automating recovery of the most common failure type. The system would use OCR, screenshots and previous data to predict failure reasons and automatically re-run tests to improve test automation efficiency.

1. Improve test automation operation
June 30th, 2020
Sadaaki, Emura
Leisure Product Dept.
Rakuten, Inc.

2. 2
Who am I
Name : Sadaaki Emura (nickname M)
Join in Rakuten : 2007
Career : Embedded engineer (2000-2002)
Web engineer (2002-2015)
Product manager (2015-2016)
QA (2016~)
Role: Test Automation Engineer Lead
Birthplace : Kanazawa-city
Hobby: jog , climbing , horse racing

3. 3
Organization
Developer group QA group
Manual test team Test automation team
@IllustAC
ココ
の人
:5 services

4. 4
Organization

5. 5
Today story
1. Test Automation operation is struggling
2. Improve test automation operation

6. 6
Test automation brings benefits in proportion to coverage!
• Reduce total cost of regression test
• Quick feedback
• Take work easy
© FREEIMAGES

7. 7
Test automation brings struggle actually
©photoAC
focus
• Many failure reports come
• Synchronize latest specification
• a few test automation engineers

8. 8
Classify failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable
Failure report
classif
y

9. 9
Classify failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable
• Good activity !
• Happy to report engineer!
©いらすとや

10. 10
Classify failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable
• Script has bug
• Script covers old specification
• Scenario dependency
©いらすとや

11. 11
Classify failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable
• Test data invalid
• Application is not ready
©いらすとや

12. 12
Classify failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable
• Network disconnected
• 500 error (releasing operation?)
• 503 error (what’s happened?)
©いらすとや

13. 13
Classify failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable

14. 14
Classify failure report
• Find bug
• Test automation script issue
• Environment issue
• Temporary unstable
0… 5.3
21.1
73.4
Fing bug Test automation script issue
Environment issue Temporary unstable

15. 15
0… 5.3
21.1
73.4
Operation for Temporary unstable
1.Investigate failure report (common)
2.Run test automation script again
3.Check if result is success
Ex. Operation Cost
1 operation = 2 mins
Daily operation = 100 * 2 = 200 mins
Monthly operation = 200 * 22 = 75 hours
Temporary Unstable
Waste
time!

16. 16
Today story
1. Test Automation operation is struggling
2. Improve test automation operation

17. 17
0… 5.3
21.1
73.4
Operation for Temporary unstable
1.Investigate failure report (common)
2.Run test automation script again
3.Check if result is success
Ex. Operation Cost
1 operation = 2 mins
Daily operation = 100 * 2 = 200 mins
Monthly operation = 200 * 22 = 75 hours
Temporary Unstable
Waste
time!

18. 18
Auto recover system
• Classify failure report automatically
• Re run test automation script when “Temporary unstable”
Improve test automation operation

19. 19
Auto recover system
Test Result is
Failure
Predict reason
Temporary
unstable
Re run
Yes

20. 20
Auto recover system
Test Result is
Failure
Predict reason
Temporary
unstable
Re run
Yes
Test result output
• Console error message by Ranorex
• Error page screenshot

21. 21
Auto recover system
Test Result is
Failure
Predict reason
Temporary
unstable
Re run
Yes
Use data
• Console error message
• Screenshot
• Previous prediction data
Use Technology (Python)
• Tesseract-OCR
• Deep Neural Network with Keras

22. 22
Auto recover system
Test Result is
Failure
Predict reason
Temporary
unstable
Re run
Yes
Input test report
Extract message
from screenshot
Text base
prediction
Image
base
prediction
No result
Output prediction
In progress
Feedback

23. 23
Auto recover system
Test Result is
Failure
Predict reason
Temporary
unstable
Re run
Yes
• Reduce waste time operation
• Not waste test resource and time

24. 24
Auto recover system benefit
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Oct-19 Nov-19 Dec-19 Jan-20 Feb-20 Mar-20
auto recovery manual recovery
Ex.
Date : March-20
Temporary unstable Ratio : 75%
Operation time without system: 100 hours
100 * 0.65 = 65 hours
saved
65%

25. 25
Summary
• Test automation operation takes time and includes boring task
• Let’s automate some of operation
• Auto recover system is not perfect yet, keep improvement