Our team automation testing activity. This year theme is to improve automation testing speed. show issues , solution and result .

1. Improve Automation Testing
~ speed up deployment ~
Aug. 28, 2018
Sadaaki Emura
Rakuten, Inc.

2. 2
Who am I
Name : Sadaaki Emura (nickname M)
join in Rakuten: 2007
Group: Service Quality Assurance in OSPD
established in 2016
Role: Test Engineer Lead
Hobby: jog , climbing , horse racing

3. 3
OSPD organization
Product
Manager
Engineer QA

4. 4
Our team (QA)
Scope : 5 services
Members : 8 + off site (if needed)
Skill : QA specialist (almost not engineer)
Mission : Quality Assurance
by manual & automation test

5. 5
Our Automation Testing activity

6. 6
Automation testing activity
2016
Proof
of
concept
2017
Expansion
2018
Improve
speed

7. 7
Why is this year theme “improve speed”?
Make impact
= release many apps frequently
photoAC

8. 8
What was bottleneck in development speed?
Specification
Design
Development Test (E2E) Development
Test Design
Test case / script
Implement
Test Execution
scope

9. 9
What was bottleneck in development speed?
1. Setup test environment takes time
2. Test feedback was late
3. Scripting for PC and SP takes time

10. 10
1. Setup test environment takes time
photoAC
What was bottleneck in development speed?
Example
Install apps (1st time) 120 min
Setup devices 5 min
Run apps 2 min
Closing 5 min
:

11. 11
2. Test feedback was late (E2E)
■One job has
Mouse click : 3 sec
Input form : 5 sec
Page transition : 10 sec
:
2-3 mins / page
10 mins / job
What was bottleneck in development speed?
■Single process worsens speed
Test
job
Test
job
Test
job
10 mins

12. 12
For PC For SP
What was bottleneck in development speed?
3. Scripting for PC and SP takes time
Apps support PC and SP
Need to create 2 script

13. 13
How did we resolve?

14. 14
Solution to resolve these bottleneck
1. Every time, Everywhere, Everyone can use
2. Many test run at same time
3. One script covers multiple devices
Features

15. 15
Test script
management
Mobile Labo Architecture
Chrom
e
FF
IE
iPhon
e
Andro
id
Edge
Safari
CI environment
AppiumAPI
SeleniumAPI
SeleniumAPI
Client side
Test
Environment
(Mobile Labo)
Clients
Windows server MAC server MAC server
Test script
management
Test script
management
https://jenkins.io/
Java
iPhon
e
Andro
id

16. 16
Mobile Labo image
Appium
Selenium
Mac
Windows
Smart phone
https://qiita.com/emurin
Document ▼

17. 17
1. Every time, Everywhere, Everyone can use
per person every time
As Is
Install apps (1st time) 120 min
Setup devices 5 min
Run apps 2 min
Closing 5 min
:
To Be
Connect API 1 min

18. 18
2. Many test run at same time
FireFox Chrome
android iOS iOS
One JENKINS runs a couple of test script

19. 19
3. One script covers multiple devices
PC
SP
Able to run test in each device
※apps rule
- same function
- same class name (HTML)
:
example : responsive site

20. 20
What’s happened?

21. 21
Improvement effect 1
Example (keiba)
Betting Lite (投票Lite) in Horse Racing
PC & SP are same function
• Step is Choose race, Bet , Confirm ,
Complete
• 9 betting method / 4 type
• Selecting order of placing(Max 3rd)
⇒ 736 test case x 2 devices(PC , SP)
PC SP

22. 22
Improvement effect 1 (initial phase)
Manual testing
Previous automation
Mobile labo
120 hour
60 hour
Method Speed
30 hour
※1 include scripting , setup task
※2 include scripting task
※1
※2

23. 23
Improvement effect 1 (CI phase)
Manual testing
Previous automation
Mobile labo
120 hour
1.5 hour
Method Speed
0.5 hour
※1 include setup task
※1

24. 24
Improvement effect 2
Example (Wedding)
Wedding Service is responsive site
• Reserve wedding hall
• Review wedding hall
• Search wedding hall
:
⇒ 302 test case x 2 devices(PC , SP)
PC SP

25. 25
Improvement effect 2 (initial phase)
Manual testing
Previous automation
Mobile labo
50 hour
20 hour
Method Speed
11 hour
※1 include scripting , setup task
※2 include scripting task
※1
※2

26. 26
Improvement effect 2 (CI phase)
Manual testing
Previous automation
Mobile labo
50 hour
0.5 hour
Method Speed
0.2 hour
※1 include setup task
※1

27. 27
Improvement effect
Mobile Labo improves speed
FREEIMAGES/Geo Okretic

28. 28
Next challenge

29. 29
photoAC
Next challenge
encounter issues

30. 30
Issue 1 : clash with others
iPhon
e 1
Andro
id
Chro
me
iPhon
e 2
iPhon
e 1
Andro
id
Chro
me
iPhon
e 2
clash
Able to use devices
Not able to use same device
at same time

31. 31
Issue 2 : need to know device location
iPhon
e 1
Andro
id 1
Appium API
MAC server 1
iPhon
e 2
Andro
id 2
iPhon
e X
Andro
id X
Appium API
MAC server X
…
Clients set configuration
- Appium IP address & port
- Capabilities (UDID etc)
Where ?

32. 32
New problem and Next step
Improve speed more
1. Clash with others
2. Need to know device location
FREEIMAGES/Alexander