Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
Effectiveness Measurement Framework for
Field-Based Experiments Focused on Android
Devices
CLIHC
2013
Ivan Pretel, Ana B. ...
Outline

Introduction
Usability, Quality and Effectiveness
Context model focused on Mobile Devices

Context model Capturer...
Introduction
Usability, Quality and Effectiveness
Context model focused on Mobile Devices

Context model Capturer
System V...
CLIHC
2013
4/37
Introduction (I)
►

According to the last Ericsson Mobility Report
(November 2013)
► By

Q3 2013, total mobile subscriptio...
Introduction (II)
►

Testing tools have changed and have been focusing on
the web domain

►

But, the web domain is focuse...
Introduction (III)

►

Approach
►A

mobile-based tool to automatically evaluate the
effectiveness of interactions and capt...
Introduction
Usability, Quality and Effectiveness
Context model focused on Mobile Devices

Context model Capturer
System V...
Usability, Quality and Effectiveness (I)
►

Usability
► ISO

9241-11
► “Usability is the extent to which a product can be ...
Usability, Quality and Effectiveness (II)
►

Quality according to ISO 9126
►

”Represents a property of the software produ...
Usability, Quality and Effectiveness (III)
►

Effectiveness metrics (ISO/IEC 9126)
► Task

Effectiveness

► What

proporti...
Introduction
Usability, Quality and Effectiveness
Context model focused on Mobile Devices

Context model Capturer
System V...
CLIHC
2013
13/37
Context model focused on Mobile Devices (I)
►

It has been taken into account and has been defined
several times
► Physica...
Context model focused on Mobile Devices (II)
►

Context of Use model based on 3 main components

►

Nuevo modelo basado en...
Introduction
Usability, Quality and Effectiveness
Context model focused on Mobile Devices

Context model Capturer
System V...
CLIHC
2013
17/37
Context model Capturer - Lab vs Field (I)
►

Capturing tools
► Lab-based
► Log

information in a highly controlled
environ...
Context model Capturer - Lab vs Field (II)

► If

you do not want to influence the context, you have to
remove all the bia...
Context model Capturer - Purposed System
►

The purposed system is formed by a tiny Android library
and a server to store ...
Context model Capturer – Library (I)
►

Interaction attributes
► Using

one command

► log(event_type,

interface, object)...
Context model Capturer – Library (II)
►

Static attributes
► Captured

►

during the registration

Automatic

►

Manual

C...
Context model Capturer – Library (III)
►

Dynamic attributes
► Captured
►

►

during the interaction (all automatically)
A...
Introduction
Usability, Quality and Effectiveness
Context model focused on Mobile Devices

Context model Capturer
System V...
System Validation – Experiment (I)

4 subjects
► A tiny memory game
► 1 day registering interactions
► Contexts
►

► At

h...
System Validation – Experiment (I)
►

Steps
►

Download the Android app
– From http://qualityinuseserver.appspot.com

►
►
...
System Validation – Experiment (I)

►

Focusing on the effectiveness
measurement
► Task:

To end up with all of the cards
...
System Validation
►

Effectiveness metrics
►

►
►

Task Effectiveness (TE)
► TE = 1 – ΣAi * error weight
► The weight of a...
System Validation – Results

►
►
►

The outdoor contexts (PT and S) affect subjects and lead them to be
less effective tha...
Introduction
Usability, Quality and Effectiveness
Context model focused on Mobile Devices

Context model Capturer
System V...
Conclusion

►

The effectiveness and the new context model, which can
bias the usability and the quality of mobile applica...
Conclusion
►

Found issues
► Security
–android.permission.INTERNET
android.permission.ACCESS_FINE_LOCATION
–android.permis...
Thank you for your
attention
¡Pura vida!
CLIHC
2013
33/37
Effectiveness Measurement Framework for
Field-Based Experiments Focused on Android Devices
Iván Pretel

{ivan.pretel@deust...
All rights of images are reserved by the original
owners*, the rest of the content is licensed under a
Creative Commons by...
Próximo SlideShare
Cargando en…5
×

de

Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 1 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 2 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 3 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 4 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 5 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 6 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 7 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 8 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 9 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 10 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 11 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 12 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 13 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 14 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 15 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 16 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 17 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 18 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 19 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 20 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 21 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 22 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 23 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 24 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 25 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 26 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 27 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 28 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 29 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 30 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 31 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 32 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 33 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 34 Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Slide 35
Próximo SlideShare
Presentatie Jong & Moedig case snijdersschool
Siguiente
Descargar para leer sin conexión y ver en pantalla completa.

0 recomendaciones

Compartir

Descargar para leer sin conexión

Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices

Descargar para leer sin conexión

Most of the mobile phones have turned into full-connected devices. This provides companies with a perfect channel to interact with their potential clients and employees. The quality of the experience with these applications can directly affect the profits of the company it represents. Focusing on the mobile field and its extremely dynamic context, the quality of the experience can highly fluctuate. Inside this field, several methods and tools have been developed by defining a context of use. However, current methods can only capture it through adding external capture tools (added cameras, human observers...) that can change the experience. The main contribution in this article is a new approach to automatically measure effectiveness through a tiny but powerful mobile tool that can capture interaction metrics and the surrounding context without biasing the measured experience.

Libros relacionados

Gratis con una prueba de 30 días de Scribd

Ver todo

Audiolibros relacionados

Gratis con una prueba de 30 días de Scribd

Ver todo
  • Sé el primero en recomendar esto

Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices

  1. 1. Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices CLIHC 2013 Ivan Pretel, Ana B. Lago DeustoTech - Deusto Institute of Technology, University of Deusto http://www.morelab.deusto.es December 6, 2013 CLIHC 2013 1/37
  2. 2. Outline Introduction Usability, Quality and Effectiveness Context model focused on Mobile Devices Context model Capturer System Validation Conclusion CLIHC 2013 2/37
  3. 3. Introduction Usability, Quality and Effectiveness Context model focused on Mobile Devices Context model Capturer System Validation Conclusion CLIHC 2013 3/37
  4. 4. CLIHC 2013 4/37
  5. 5. Introduction (I) ► According to the last Ericsson Mobility Report (November 2013) ► By Q3 2013, total mobile subscriptions were around 6.6 billion. By the end of 2019, they are expected to reach around 9.3 billion. CLIHC 2013 Introduction 5/37
  6. 6. Introduction (II) ► Testing tools have changed and have been focusing on the web domain ► But, the web domain is focused not only on its own but also on the mobile one ► Mobile device -> More contexts than ever ► Issue ► The quality of mobile applications fluctuates depending on the context in which they are used ► When we try to capture it, we modify it CLIHC 2013 Introduction 6/37
  7. 7. Introduction (III) ► Approach ►A mobile-based tool to automatically evaluate the effectiveness of interactions and capture metrics of the surrounding context without adding disturbing elements CLIHC 2013 Introduction 7/37
  8. 8. Introduction Usability, Quality and Effectiveness Context model focused on Mobile Devices Context model Capturer System Validation Conclusion CLIHC 2013 8/37
  9. 9. Usability, Quality and Effectiveness (I) ► Usability ► ISO 9241-11 ► “Usability is the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” – “Effectiveness is the level of accuracy and completeness with which users achieve specified goals” ► ISO 9126-4 ► “Usability is the capability of the software product to be understood, learned, used and attractive to the user, when is used under specified conditions” – “Effectiveness is the capability of the software product to enable users to achieve specified goals with accuracy and completeness in a specified context of use” Usability, Quality and Effectiveness CLIHC 2013 9/37
  10. 10. Usability, Quality and Effectiveness (II) ► Quality according to ISO 9126 ► ”Represents a property of the software product defined in terms of a set of interdependent attributes (usability, security, reliability, performance, complexity, readability, reusability) expressed at different levels of detail and also taken into account the particular context of use” ► Can be measured by 3 quality aspects ► Internal Quality – From an internal view (e.g. spent resources) ► External Quality – Its behaviour (e.g. number of wrong expected reactions) ► Quality in Use – “The quality of software that user can perceive when the software is used in an explicit context of use” – It is measured by 4 main capabilities » Effectiveness, satisfaction, productivity and safety Usability, Quality and Effectiveness 10/37 CLIHC 2013
  11. 11. Usability, Quality and Effectiveness (III) ► Effectiveness metrics (ISO/IEC 9126) ► Task Effectiveness ► What proportion of the goals is achieved? – Measures the quantity of the goals achieved by a user –  0<=TE<=1 ► Task Completion ► What proportion of the tasks is completed? – Measures the level of success the user achieves performing tasks –  0<=TC<=1  ► Error Frequency ► What is the frequency of errors? – Measures the number of times that an error is made within a given period –  0<=EF<=1  Usability, Quality and Effectiveness 11/37 CLIHC 2013
  12. 12. Introduction Usability, Quality and Effectiveness Context model focused on Mobile Devices Context model Capturer System Validation Conclusion CLIHC 2013 12/37
  13. 13. CLIHC 2013 13/37
  14. 14. Context model focused on Mobile Devices (I) ► It has been taken into account and has been defined several times ► Physical location ► Weather attributes ► Involved communities & stakeholders ► Goals ►… ► Tries to answer ► Who are the users? ► What are they doing? ► Where? ► Which resources are nearby? Context model focused on Mobile Devices CLIHC 2013 14/37
  15. 15. Context model focused on Mobile Devices (II) ► Context of Use model based on 3 main components ► Nuevo modelo basado en tres pilares CLIHC 2013 Context model focused on Mobile Devices 15/37
  16. 16. Introduction Usability, Quality and Effectiveness Context model focused on Mobile Devices Context model Capturer System Validation Conclusion CLIHC 2013 16/37
  17. 17. CLIHC 2013 17/37
  18. 18. Context model Capturer - Lab vs Field (I) ► Capturing tools ► Lab-based ► Log information in a highly controlled environment using specific devices and users – Control: high – Context variables: limited – Objectivity: low ► Field-based ► Log using added agents and invasive evaluation methods – Control: very low – Context variables: unlimited – Objectivity: depends on the added agents Context model Capturer CLIHC 2013 18/37
  19. 19. Context model Capturer - Lab vs Field (II) ► If you do not want to influence the context, you have to remove all the biasing elements which can influence the captured data and use only the elements which is composing it Capture using only the mobile device ► This tool should capture the context model via the built-in mobile sensors and logging interaction events CLIHC 2013 Context model Capturer 19/37
  20. 20. Context model Capturer - Purposed System ► The purposed system is formed by a tiny Android library and a server to store and log the performed interactions 1. The app-to-test (ATT) should be integrated with the library ► 2. This library automatically captures context and interaction information and stores it in a local database ► 3. When the device has internet connection and its owner wants, all the information is uploaded ► CLIHC 2013 Context model Capturer 20/37
  21. 21. Context model Capturer – Library (I) ► Interaction attributes ► Using one command ► log(event_type, interface, object) CLIHC 2013 Context model Capturer 21/37
  22. 22. Context model Capturer – Library (II) ► Static attributes ► Captured ► during the registration Automatic ► Manual CLIHC 2013 Context model Capturer 22/37
  23. 23. Context model Capturer – Library (III) ► Dynamic attributes ► Captured ► ► during the interaction (all automatically) Android API Device Sensors ► External Services CLIHC 2013 Context model Capturer 23/37
  24. 24. Introduction Usability, Quality and Effectiveness Context model focused on Mobile Devices Context model Capturer System Validation Conclusion CLIHC 2013 24/37
  25. 25. System Validation – Experiment (I) 4 subjects ► A tiny memory game ► 1 day registering interactions ► Contexts ► ► At home [H] ► Public Transport [PT] ► Walking Down the street [C] ► At work (office) [T] CLIHC 2013 System Validation 25/37
  26. 26. System Validation – Experiment (I) ► Steps ► Download the Android app – From http://qualityinuseserver.appspot.com ► ► ► Sign up on the testing platform through the app Play the game Upload info to server CLIHC 2013 System Validation 26/37
  27. 27. System Validation – Experiment (I) ► Focusing on the effectiveness measurement ► Task: To end up with all of the cards flipped face up in less than 15 turns ► More than 15 turns: task is not completed ►8 pairs of cards ► It means the best round is made by only 8 interactions and 0 errors CLIHC 2013 System Validation 27/37
  28. 28. System Validation ► Effectiveness metrics ► ► ► Task Effectiveness (TE) ► TE = 1 – ΣAi * error weight ► The weight of an error is 1/7 = 0.143 – Max number of errors is 6 with 8 correct interactions – If you make 7 errors » (1- (7*1/7) = 0) Task Completion (TC) ► TC = won rounds / played rounds Error Frequency (EF) ► EF = Number of errors / Total turns CLIHC 2013 System Validation 28/37
  29. 29. System Validation – Results ► ► ► The outdoor contexts (PT and S) affect subjects and lead them to be less effective than in the indoor contexts (H and W). TE indicates that the S context is the context where subjects have the worst results The most efficient context is W, although EF exposes that is where subjects make more mistakes CLIHC 2013 System Validation 29/37
  30. 30. Introduction Usability, Quality and Effectiveness Context model focused on Mobile Devices Context model Capturer System Validation Conclusion CLIHC 2013 30/37
  31. 31. Conclusion ► The effectiveness and the new context model, which can bias the usability and the quality of mobile applications, can be automatically captured and measured by the own mobile device minimizing the interaction bias CLIHC 2013 Conclusion 31/37
  32. 32. Conclusion ► Found issues ► Security –android.permission.INTERNET android.permission.ACCESS_FINE_LOCATION –android.permission.ACCESS_NETWORK_STATE android.permission.ACCESS_WIFI_STATE –android.permission.MODIFY_AUDIO_SETTINGS android.permission.RECORD_AUDIO –android.permission.WRITE_EXTERNAL_STORAGE ► Weather variables capture depends on the Internet connection ► Developers have to specify where are the errors ► Next steps ► Solve the found issues ► Increase the number of users ► Calculate correlations CLIHC 2013 Conclusion 32/37
  33. 33. Thank you for your attention ¡Pura vida! CLIHC 2013 33/37
  34. 34. Effectiveness Measurement Framework for Field-Based Experiments Focused on Android Devices Iván Pretel {ivan.pretel@deusto.es} DeustoTech - Deusto Institute of Technology, University of Deusto http://www.morelab.deusto.es CLIHC 2013 34/37
  35. 35. All rights of images are reserved by the original owners*, the rest of the content is licensed under a Creative Commons by-sa 3.0 license. * • • • • • • http://www.fotopedia.com/items/flickr-2830319467 http://www.fotopedia.com/items/flickr-4609827016 http://en.wikipedia.org/wiki/File:Skyline_oklahoma_city.JPG http://www.flickr.com/photos/grimsanto/6190478735/ http://commons.wikimedia.org/wiki/File%3AAstronomy_Amateur_3_V2.jpg http://commons.wikimedia.org/wiki/File%3AAldrin_Apollo_11_cropped.jpg CLIHC 2013 35/37

Most of the mobile phones have turned into full-connected devices. This provides companies with a perfect channel to interact with their potential clients and employees. The quality of the experience with these applications can directly affect the profits of the company it represents. Focusing on the mobile field and its extremely dynamic context, the quality of the experience can highly fluctuate. Inside this field, several methods and tools have been developed by defining a context of use. However, current methods can only capture it through adding external capture tools (added cameras, human observers...) that can change the experience. The main contribution in this article is a new approach to automatically measure effectiveness through a tiny but powerful mobile tool that can capture interaction metrics and the surrounding context without biasing the measured experience.

Vistas

Total de vistas

465

En Slideshare

0

De embebidos

0

Número de embebidos

7

Acciones

Descargas

7

Compartidos

0

Comentarios

0

Me gusta

0

×