Those slides have been presented @ AST 2012, the 7th International Workshop on Automation of Software Test (part of ICSE 2012) See also http://www.henrymuccini.com/mobiletesting

1. Università degli Studi dell’Aquila
Henry Muccini, Antonio Di Francesco, Patrizio Esposito
Dipartimento di Ingegneria, Scienze dell’Informazione e Matematica,
University of L’Aquila
www.henrymuccini.com, @muccinihenry
@AST 2012, the 7th International Workshop on Automation of Software Test, Zurich, June 02 2012

2. 2
Goal and RQs
Goal of this presentation is to convey two
messages to the audience:
 testing applications running on mobile devices is
an important task
 those applications require dedicated testing
techniques

3. 3
This work is conducted in the context of a
series of activities conducted by my
Department on mobile applications:
 Testing of (context-aware) applications
 Engineering of mobile applications
 A course we launches one year ago on
“applications for mobile devices)
 A lab we are planning to build on those topics

4. # of downloads:
29 billions of mobile apps
downloaded in 2011.
abiresearch
185 billions of downloaded
apps by 2014.
Gartner
Market:
$ 128 billions (2011).
$ 150 billions (2012).
Strategy Analytics

5. http://code.google.com/p/android/issues/list

6. 6
M-health
Navatar
M-Payments

7. 7
Mobile Computing & Context-Awareness
In mobile computing an application is considered to be
mobile if it runs on an electronic device that may move
(e.g., mp3 readers, digital camera, mobile phones).
In context-aware computing, an application is aware
of the computing environment in which it runs, and
adapts/reacts according to its computing, user,
physical, or time context [Schilit1994,Chen2000].

8. Type of Mobile Apps
Native applications
Web applications
Hibrid applications

9. Why differentiating Context-aware apps from Apps4Mobile?
From a testing perspective:
 The type and variaty of inputs are unpredictable
 Low programming support
From a technological perspective:
 Sensors-based applications are becoming pervasive
From a market perspective:
 $ 96 billions in 2015 (gartner)
 “context-aware mobile applications will change the business
world…” [Beth Schultz]
 ”creating applications for use in the mobile context will ultimately
create better applications that scale across connected devices”
[Eric Schmidt]
 Projects ongoing

10. 10
Peculiarities of mobile applications
Type of Mobile
Application Peculiarity Implications on Testing
Mobile Connectivity Reliability, performance, security,
and functional testing through
different networks
Apps4Mobile
Limited Resources Performance and functional
monitoring
Autonomy Monitoring for energy consumption
MobileApps
User Interface GUI Testing
Context Awareness Context-dependent functional and
extra functional testing
Mobile Applications
Adaptation Adaptation correctness
New programming languages New white box and black-box
testing, byte-code analysis
New O.S.s Compatibility and O.S. testing
Diversity of phones and O.S.s Diversity coverage
Touch Screens Usability, and response to screen
touch

11. 11
Peculiarities and Implications on testing
Type of Mobile
Application Peculiarity Implications on Testing
Mobile Connectivity Reliability, performance, security,
and functional testing through
different networks
Apps4Mobile
Limited Resources Performance and functional
monitoring
Autonomy Monitoring for energy consumption
MobileApps
User Interface GUI Testing
Context Awareness Context-dependent functional and
extra functional testing
Mobile Applications
Adaptation Adaptation correctness
New programming languages New white box and black-box
testing, byte-code analysis
New O.S.s Compatibility and O.S. testing
Diversity of phones and O.S.s Diversity coverage
Touch Screens Usability, and response to screen
touch

12. 12
Peculiarities and Implications on testing
Type of Mobile
Application Peculiarity Implications on Testing
Mobile Connectivity 512 MB of RAM,
Reliability, performance, security,
64 GB of disk space, testing Ghz dual core
and functional and 1 through
different networks
CPU
Apps4Mobile
Limited Resources Performance and functional
monitoring
Autonomy Monitoring for energy consumption
MobileApps
User Interface GUI Testing
Context Awareness Context-dependent functional and
extra functional testing
Mobile Applications
Adaptation Adaptation correctness
New programming languages New white box and black-box
testing, byte-code analysis
New O.S.s Compatibility and O.S. testing
Diversity of phones and O.S.s Diversity coverage
Touch Screens Usability, and response to screen
touch

13. 13
Peculiarities and Implications on testing
Type of Mobile iPhone 4S:
Application Peculiarity
200 hours autonomy in stand-by Testing to
Implications on …drops
Mobile Connectivity Reliability, performance, security,
9 hours when Wi-Fi is enabled … drops to
and functional testing through
6 hours when a 3G connection is active
different networks
Apps4Mobile
New OS releases to increase autonomy
Limited Resources Performance and functional
monitoring
Autonomy Monitoring for energy consumption
MobileApps
User Interface GUI Testing
Context Awareness Context-dependent functional and
extra functional testing
Mobile Applications
Adaptation Adaptation correctness
New programming languages New white box and black-box
testing, byte-code analysis
New O.S.s Compatibility and O.S. testing
Diversity of phones and O.S.s Diversity coverage
Touch Screens Usability, and response to screen
touch

14. 14
Peculiarities and Implications on testing
Type of Mobile
Application Peculiarity Implications on Testing
Mobile Connectivity Reliability, performance, security,
and functional testing through
different networks
Apps4Mobile
Huge effort
Limited Resources Performance and functional
monitoring
Autonomy Monitoring for energy consumption
MobileApps
User Interface GUI Testing
Context Awareness Context-dependent functional and
extra functional testing
Mobile Applications
Adaptation Adaptation correctness
New programming languages New white box and black-box
testing, byte-code analysis
New O.S.s Compatibility and O.S. testing
Diversity of phones and O.S.s Diversity coverage
Touch Screens Usability, and response to screen
touch

15. 15
Peculiarities and Implications on testing
Type of Mobile
Application Peculiarity Implications on Testing
Mobile Connectivity Reliability, performance, security,
and functional testing through
different networks
Apps4Mobile
Limited Resources Is the app going to work infunctional
Performance and any
monitoring
environment and under any unpredictable
Autonomy scenarios? Monitoring for energy consumption
MobileApps
User Interface GUI Testing
Context Awareness Context-dependent functional and
extra functional testing
Mobile Applications
Adaptation Adaptation correctness
New programming languages New white box and black-box
testing, byte-code analysis
New O.S.s Compatibility and O.S. testing
Diversity of phones and O.S.s Diversity coverage
Touch Screens Usability, and response to screen
touch

16. 16
Peculiarities and Implications on testing
Type of Mobile
Application Peculiarity Implications on Testing
Mobile Connectivity Reliability, performance, security,
and functional testing through
different networks
Apps4Mobile
Limited Resources Performance and functional
monitoring
Autonomy Monitoring for energy consumption
MobileApps
User Interface GUI Testing
Context Awareness Context-dependent functional and
extra functional testing
Mobile Applications
Adaptation Adaptation correctness
New programming languages 1,800 different hw/sw
New white box and black-box
configurations analysis
testing, byte-code
New O.S.s Compatibility and O.S. testing
Diversity of phones and O.S.s Diversity coverage
Touch Screens Usability, and response to screen
touch

17. 17
Challenges
Test Selection for context-aware mobile
applications
Challenges: unpredictabile and high variable inputs
Potentials:
 Create higher level libraries for developers
 Coverage criteria
 Scenario-based approaches

18. Context-aware programming
public class SensorActivity extends Activity, implements SensorEventListener {
private final SensorManager mSensorManager;
private final Sensor mAccelerometer;
public SensorActivity() {
mSensorManager = (SensorManager)getSystemService(SENSOR_SERVICE);
mAccelerometer = mSensorManager.getDefaultSensor(Sensor.TYPE_ACCELEROMETER);
}
protected void onResume() {
super.onResume();
mSensorManager.registerListener(mAccelerometer, SensorManager.SENSOR_DELAY_NORMAL);
}
protected void onPause() {
super.onPause();
mSensorManager.unregisterListener(this);
}
public void onAccuracyChanged(Sensor sensor, int accuracy) {
}
public void onSensorChanged(SensorEvent event) {
}
}

19. 19

20. 20
Challenges
Security testing
Challenge:
 the mobility of the device into networks with different
security levels increases security.
 a trojan might have access to personal date, private
networks, and credit card information
 the rich contextual information presents real privacy
concerns.
 the democratization of mobile application publishers also
increases the number of apps store the final user does not
know anything about.

21. 21
Challenges
Performance and Reliability testing
Challenge:
 Performance and reliability of mobile applications strongly
depends on the mobile device resources, on the device
operational mode, on the connectivity quality and
variability, and other contextual information.
Potentials:
 new techniques to explicitly consider characteristics related
to (changing) contexts and different devices.
 Run-time analysis techniques to prevent performance
degradation.

22. 22
Challenges
Product Line
Challenge:
 130 devices running Android, with 1,300 hw/sw
configurations
Potentials
 new systematic combinatorial/product line testing
techniques
 run-time analysis of instrumented beta versions of the
application
 testing as a service

23. 25
Conclusions
Test automation to decrease the cost of testing
 mobile applications must be cheap
 outsourcing, cloud- and crow-based testing
Testing automation towards all the different layers
 current bugs are due to interoperability problems that exist
today among the application, application framework,
operating system, and hardware (sensoring) layers