Mobile testing and its part in testing tools

Mobile testing and its part in testing tools
Kari Kakkonen, March 25, 2014, ASTQB Software Testing Conference, San Francisco
© Copyright Knowit Oy 2013 | Confidential | Version 1.0

Kari Kakkonen, Knowit
• Roles
• Knowit, Director, Quality and Competences & Lead
Consultant
• Finnish Software Testing Board FiSTB, chairman
• ISTQB international exam and expert working group member
• Finnish Software Measurement Association FiSMA ry board
member 2004-2007
• Chairman of research project STX, Lappeenranta University
of Technology
• Achievements
• Testing promoter since 1996
• Tietoviikko-magazine 100 most influential IT-persons listing
2010
• Presentations in national and international conferences
• Candidate for Tester of the Year 2010 (TestausOSY – FAST)
• TestausOSY/FAST founding member
• Education
• ISTQB Advanced certified
• SPICE provisionary assessor certified
• M.SC, Helsinki University of Technology / Aalto-university
• Marketing studies, University of Wisconsin-Madison
• Domain background
• Embedded
• Public sector
• Pension insurance
• Training provider
• Telecom
• Finance
• Manufacturing
• Services
• Training ISTQB Advanced and Foundation + Knowit Testing
Professional
• Service management
• Test management
• Test assessments (TPI Next etc.) and test development
• Agile testing services
© Knowit Oy / SECRET (Internal slide)
Twitter: @kkakkonen
LinkedIn: fi.linkedin.com/in/karikakkonen/

110
145
154
189
215 215 220
2007 2008 2009 2010 2011 2012 2013
Trusted ICT Services Partner in Nordics
Experts in
1800
Sweden, Finland, Norway,
Denmark, Russia and Estonia
Knowit AB is listed on the Nordic Exchange in Stockholm
Grouprevenue(MEUR)

Case description
© Copyright Knowit Oy 2013 | Confidential | Version 1.0 4

Client
• Worldwide company with many products
• Top of the line in their business
• Consumer products (B2C)
• Established software testing, including automation
• Now wanted their first mobile app
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 5

Three perspectives of mobile solutions
Technology
• Responsive web
• Mobile optimized web
• Native application
• Hybrid application
Target group
• Consumers (B2C, B2B2C)
• Retailers and suppliers (B2B)
• Stakeholders
• Own personnel
Benefits
• Growth in sales
• Customer loyalty
• Savings
Client choices
in bold

Mobile app scope
• To enable users to access the service also from mobile devices
• not only through the established web service
• Utilize mobile device’s location services heavily
• First platform - iOS
• For anonymity the actual functionality is here called just “activity”
• Supporting functionalities in a bit more detail, meaningful things for testing here

Expected challenges
• Connectivity to web service, integral part of the service
• Usability – will the vast user base like it?
• Can we automate?

Functionalities of the application
Location
• Activity
• Data
• Map
• Graphs
• Camera
Profile
• Latest info
• Following
• Fans
• Shouts
• Edit Profile
Personal
• Latest info
• Calendar
Other
• Following
• Shouts
• Settings

Our team
• Compact scrum-team
• 2 developers in Sweden
• Project manager, 2 testers and 1 designer in Finland
• All participating in daily scrum meetings
• Customer committed to be in close contact with the team on daily basis
• Customer geographically dispersed
• Remote connection within team and to customer (Lync, Skype)

We piloted automation

Knowit test automation process used in the
mobile project
Preparation & piloting Implementation Testing
Testing
Recording and
scripting
Test execution
New tests
Script maintenance
Reporting
Feasibility
study
Piloting
POC
Implementation
Training
1.version of
software
under test

Feasibility study
Phase Description
• Benefits of automation in this project?
• Discussions about test automation and tools with customer
Proof Of
Concept
(PoC)
• Verifying that the test automation tool is compatible with application
under test
• Identifying parts that are impossible/no benefit to automate
• Installation of test tool to machines
• Placing the hooks to software
• Automating 1-2 test cases
• Investment calculations and workload estimations
• Scripting test cases
• Running test cases in development environment
• New versions of sofware under test in regular intervals
Piloting
• Execution and maintenance of test cases
• Scripting of new test cases
• Reporting
Implementation
Testing

Tool options
Availabletools
MonkeyTalk
Sikuli
Testing with Frank
Xcode
Selenium
NativeDriver
Robot Framework
Junit
eggPlant for mobile
Silk4mobile
Piloted
Robot Framework
Silk4mobile
Xcode
Chosen
Silk4mobile
Xcode

Criteria for selection
• Target operating system iOS
• Several iOS options (iPad, iPad mini, iPod, iPhone)
• Other operating systems in the future (Android, Windows)
• Cost-effectiveness
• Ease of installation and learning
• Existing experiences of use

Silk4mobile
• Supports all mentioned mobile platforms
• Knowit Silk4Mobile know-how
• Straightforward implementation
• Requires hooks in the software or a jailbreak
• Borland is a Knowit partner

Xcode
• Default Apple development environment
• Includes test automation environment
• Automation tests run with emulator
• Operational testing in emulator
• Used in Knowit as development and unit test environment

Project implementation

Testing process
Development
Developers’ tests
Test team
• Smoke test
• UI testing
• Field testing
• Automation
Beta testing
Feedback
• Internal testing
• Beta testing

Early challenges
• Challenges of automation
• Tried out Silk4Mobile (Borland)
• Requires hooks within the app
• Part of iOS test automation requires ‘jailbreaking’
• iOS7
• Apple updated its operating system at the same time with this project -> retesting
• Other challenges
• Application needed a lot of field testing in true environment
• A lot of outdoor use
• The web service also updated during the project
• Web service interfaces were modified with changes to app behavior
• Downtimes in web service
• Testing consisted of interaction between app, location information from phone, and web service21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 20

Test planning
• Division of testing tasks between testers and beta testers
• Testers continuously validated the correct functionality of the app features
• Beta testers were required to validate real life scenarios
• Levels of use
• Basic use: activities and following the web service
• Heavy use: more activities, following the web services, shares own information to social media, likes
and comments of friends
• Benchmarking: does app + web service work with same ease and content as web service only?
• Use of exploratory testing

Test management - ChiliProject…
• User story - > test scenarios
(different variety) - > field tests
• Incidents, bugs and other issues
reported back to Chili
• Roadmap
• Sprints
• Features
• Bugs
• Status
• Priority
• Target version
• Files
• Reports

Implementation of tests
Basicuser
Indoor
Outdoor
City location
Forest
On the water Sailing
Specialcases
Network is lost
At sea
In a forest
In the cityConnection is lost
Web service is down
Data is transferred after
connection is restored

Beta testing – field testing
• After smoke tests app moved to beta testing stage
• Purpose of beta testing
• To get the actual end-user experience
• Real life use cases
• Achievements / what was good?
• Wide coverage
• Use cases : active and basic users, like-minded group of active users discussing with others online and at
different events, continuously growing number activities, usage in different countries, various programs, all
iPhone models plus iPad and iPod…
• Validation testing at the end of the project
• A group of test users, about 10-20 persons who were not involved in the project in any way, but had
previously used the web service, from heavy to basic users in different countries
• Using mobile app normally and giving feedback

Summary
• Challenging start
• Challenging test environment in mobile testing
• Mobile test automation can be tricky, settled to unit test automation only
• Demanding usage environment in this project – field testing was a success
• Despite the challenges, comprehensive and versatile testing
• A total of 140 functionalities and user stories tested
• 445+ issues were reported to Chili, fixed and verified
• Especially happy customer and successful stable application
• Based on this project the customer ordered a new mobile application development and testing from
Knowit

Test tool survey reflections
© Copyright Knowit Oy 2012 | Luottamuksellinen

Mobiles
roll…
Number of smartphones worldwide:
Q3 / 2012 1 000 000 000
2015 2 000 000 000
Source: Strategy Analytics

Test automation options
Mobile
testing
Emulator
Remote
device
testing
• Devices as
a service
Device
specific
automation

Mobile testing software
Open source
• Robot framework
• Robotium
• JUnit / OCUnit
• Appium
• Calabash
• MonkeyTalk
• Sikuli
• Testing with Frankk
Commercial
• Borland Silk4Mobile
• Squish – Frologic
• Keynote
DeviceAnywhere
• Testdroid
• Appurify
• EggPlant for mobile

The survey

Background
• Research target: Software testing tools usage in Finland
• Research is a part of Master Degree of Business Informatics thesis for Metropolia in co-
operation with Knowit Oy
• Used research method was quantitative method with web-form survey
• Survey was sent to IT- and testing professionals in Finland
• Knowit customer contacts
• TestausOSY society
• Soliditet register of Bisnode Finland
• Survey was done in February 2013
• You could choose many options in most questions

Respondent’s background
• Response rate was 39 % i.e. total 107 responses
• 64 % was men and 36 % women
• Age distribution; 84 % age between 31 to 50 years
• Education; highly educated, 86 % have university level education
• 87 % of respondents from South of Finland and 10 % from West of Finland

Profession/role in the company
38%
4%
6% 5% 5%
13%
3%
1%
3% 2% 3%
1% 1% 1%
12%
4%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
Most of the respondents were test
managers/ test coordinators and system
testers

Operating area of the company
The largest operating areas were
Information and telecommunication,
Finance and insurance business and
Industry
Other service
12 %
Science or technical
area
7 %
Finance and
insurance business
21 %
Information and
telecommunication
35 %
Retail business
2 %
Industry
16 %
International
organization and
commission
1 %
Business area
unknown
2 %
Mining
1 %
Administration and
support
1 % Medical
2 %

Size of the company
2%
7%
5%
24%
14%
45%
1%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
1 - 10 51 – 100 11 - 50 101 – 500 501 - 1000 I don't know Over 1001
employees
The respondents were mostly from
large companies

Current life cycle model in the project
5%
67%
13%
7%
21%
29%
50%
24%
11%
5% 8%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Current projects life cycle models
were Scrum, V-model and Iterative
model

Future life cycle model
18%
25%
4% 2%
5%
8% 7%
16%
40%
9%
30%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
Scrum seems to be future
trend, Kanban and Lean are
following, V-model use is
going down

Which employees of your company are using software
testing tools?
72% 74%
89%
19%
24% 28% 25%
3%
9%
64%
6%
80%
49%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Testing tools are used by testing
professionals and developers

What kind of training do you have for software
testing?
5% 1%
25%
2% 1% 0% 0% 0% 7% 12% 15%
1% 2%
3% 6%
49%
4% 1% 4%
0% 0%
27% 24% 18%
3%
14%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Men
Women
74 % of respondents have ISTQB/ISEB Foundation
course, next most common are tool specific
courses and scrum master course.

What kind of testing tools are in use?
88%
60%
48%
4%
0%
20%
40%
60%
80%
100%
Commercial
software
testing tools
Open Source
testing tools
Proprietary
tools
Other
Tools from many sources are used
side by side

Which way the testing tools are acquired?
40%
6%
3% 4%
25%
22%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
Bought Cloud
service
Other way Rented Open
Source
Coded by
company
itself
Normally testing tools are acquired
by buying. Next most common is
open source usage and self coding
According to this research cloud
services are not yet a popular
acquisition method..

Which way the testing tools are acquired by company
size? (number of persons in the company)
1
1
2
1 - 10
5
5
2
2
1
11 - 50
5
42
2
2
1 Open
Source
Bought
12
22
3
13
2
3 101 - 500
6
142
7
1
1501 - 1000
27
466
24
1 5
over 1000
51 – 100

What test management tools do you have in the
company?
1%
14%
9%
2%
54%
3% 2%
5%
2%
47%
1% 1% 1%
5%
1%
9%
0%
10%
20%
30%
40%
50%
60%
Most popular Test
management tools were HP:
QC, Atlassian: Jira and
TestLink

Test management- vs office-tools
1%
54%
9%
47%
0% 1% 1% 1% 1% 0% 0% 0% 5% 9%
1% 2% 0% 0% 5% 3% 2% 2% 1%
13%
8%
7%
2% 1%
1% 1%
1%
0%
10%
20%
30%
40%
50%
60%
70%
No
Yes
Also QC and Jira users are
utilizing often office tools (spread
sheets, word processors)

What incident management tools do you have in the
company?
41%
11%
3%
5%
3%
12%
25%
5%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
Atlassian:
JIRA
IBM
Rational
ClearQuest
Polarion Proprietary
tools
HP: Quality
Center
Bugzilla
(Open
source)
Other Mantis
Atlassian: Jira is clearly the largest
in this category and second open
source tool Bugzilla – QC is
surprisingly very little used for
incident management.

What test execution tools do you have in the
company?
33%
7%
3% 4% 2% 1% 2% 2% 2% 2% 1% 3% 4% 5%
21%
2% 4%
29%
4% 4%
45%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
Open source tool Selenium was
most popular, next HP:QTP and open
source tool Robot Framework. There
are a large number of different tools
in use in this category.

What test design tools do you have in the company?
1% 1% 1% 2% 2%
46%
2% 3%
8%
1%
4% 4% 2%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
Major test design tool was HP: QC, note also
MBT tools and test technique tools (CTE,
allpairs)

What test harness/ unit test framework tools do you
have in the company?
27%
5%
17%
2% 3%
1%
12%
7% 7%
2%
7%
2%
0%
5%
10%
15%
20%
25%
30%
These tools are mostly used by
developers. Most used tools in this
category were CppUnit, Junit and .TEST

What specialized unit test frameworks tools do you
2% 2%
4%
1%
6%
1%
2%
5%
1%
0%
1%
2%
3%
4%
5%
6%
Specialized unit test frame work tools are
less used but most popular is JBehave

What performance/ load/ stress testing tools do you
1%
3%
7%
1%
4% 4%
7%
22%
7%
12%
4%
24%
0%
5%
10%
15%
20%
25%
30%
Most popular in this category
were open source tool Jmeter
and HP Performance Center.

Overall usage of all software testing tools
88
4 4
12
4 4
98
32
21
7 3 4 6
24
15
27
5
26
5 5 2
29
45
15
3 5
9
3
31
12
0
20
40
60
80
100
120
%ofrespondents
98 % have QC in use or have used and 88 % have
JIRA in different testing phases.

Purpose of software testing tools?
79% 76%
80%
29%
48% 50%
77%
30%
4%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Reporting Test
execution
Planning test
cases
Generating
test cases
Requirement
management
Defect
tracking
Defect
control and
management
Reviews Other
Typical testing activities have the lead in
this category, i.e. planning test cases,
reporting and test execution.

Do you use ‘office tools’ in testing (Spreadsheet, word
processor etc.)?
35%
1%
17%
12%
1% 1%
10%
19%
1%
3%
0%
5%
10%
15%
20%
25%
30%
35%
40%
Reporting Requirement
management
Test data Test design Defect control
and
management
Test
management
Test execution Documentation Design Calculation
• Yes, office tools in use 80 %
• Which functions are you using them for?
Office tools are used to same
functions as the software
testing tools

Testing tool markets directions
• Open source is gaining momentum, in midsize companies
• Big (commercial) test tool suites work for big companies
• Cloud (software testing tool as a service) is just emerging, but a growing trend
• Specialized tools, like mobile test automation, emerge all the time
• Are not first considered testing tools (or testers are not the users)
• Are soon swallowed by big test suite providers
• Integrated tool sets for multiple platforms are seen as most useful
• Support e.g. all mobile platforms and all browsers

Recommendations
• Pay attention to test tool life cycle – does it have future?
• Open source – the user and developer base
• Commercial – company size, market shares
• Consider tool integrations
• Mix and match open source and commercial
• Integrate tools to work together – e.g. Jira – QC – Robot Framework
• Look for good tools
• Superb usability
• Excellent technical flexibility
• Even small vendors grow soon with superb tools
• Word of mouth is big player (Finnish Robot Framework phenomena)

”There are plenty of software testing tools.
Expertise is needed to select effective testing
tools to suitable areas of testing processes”

Questions?
@kkakkonen or kari.kakkonen@knowit.fi

Mobile testing and its part in testing tools

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (20)

Similar a Mobile testing and its part in testing tools

Similar a Mobile testing and its part in testing tools (20)

Más de Kari Kakkonen

Más de Kari Kakkonen (20)

Último

Último (20)

Mobile testing and its part in testing tools