With physical mobile interaction techniques, digital devices can make use of real-world objects in order to interact with them. In this paper, we evaluate and compare state-of-the-art interaction methods in an extensive survey with 149 participants and in a lab study with 16 participants regarding efficiency, utility and usability. Besides radio communication and fiducial markers, we consider visual feature recognition, reflecting the latest technical expertise in object identification. We conceived MobiMed, a medication package identifier implementing four interaction paradigms: pointing, scanning, touching and text search. We identified both measured and perceived advantages and disadvantages of the individual methods and gained fruitful feedback from participants regarding possible use cases for MobiMed. Touching and scanning were evaluated as fastest in the lab study and ranked first in user satisfaction. The strength of visual search is that objects need not be augmented, opening up physical mobile interaction as demon- strated in MobiMed for further fields of application.

1. Distributed Multimodal Information Processing Group Technische Universität München
MobiMed:
Comparing Object Identification Techniques
on Smartphones
Andreas Möller1, Stefan Diewald1, Luis Roalter1, Matthias Kranz2
1Technische
Universität München, Germany
2Luleå University of Technology, Department of Computer Science,
Electrical and Space Engineering, Luleå, Sweden
October 15, 2012
NordiCHI, Copenhagen, Denmark

2. Distributed Multimodal Information Processing Group Technische Universität München
Outline
Background and Motivation
Scenario and Prototype
User Study
Discussion and Conclusion
Oct 15, 2012 A. Möller, S. Diewald, L. Roalter, M. Kranz 2

3. Distributed Multimodal Information Processing Group Technische Universität München
Background and Motivation
• Idea of bridging the gap between the physical and the virtual world
for easier interaction and additional functionality
– Connect physical objects with virtual representations by tags
(Want et al., 1999)
– Physical mobile interaction (Rukzio, 2006)
• Investigation and comparison of different interaction techniques done earlier,
BUT:
– meanwhile outdated technologies (e.g. IR)
– older comparisons based on (nowadays) limited hardware
(VGA cameras, small screens, slow mobile CPUs)
– new technologies have emerged (e.g. vision-based approaches)
– user knowledge and experience has changed
 Suggesting a new comparison of (state-of-the-art) interaction techniques
Oct 15, 2012 A. Möller, S. Diewald, L. Roalter, M. Kranz 3

4. Distributed Multimodal Information Processing Group Technische Universität München
Outline
Background and Motivation
Scenario and Prototype
User Study
Discussion and Conclusion
Oct 15, 2012 A. Möller, S. Diewald, L. Roalter, M. Kranz 4

5. Distributed Multimodal Information Processing Group Technische Universität München
Scenario for Physical Mobile Interaction
• MobiMed:
identifying medication packages
with the smartphone
• Target groups: active people
pursuing a healthy lifestyle, elderly
people
• Physical mobile interaction to get
information on drugs
– package insert
– side effects
– active ingredients
– cross-correlations
Oct 15, 2012 A. Möller, S. Diewald, L. Roalter, M. Kranz 5

6. Distributed Multimodal Information Processing Group Technische Universität München
Investigated Interaction Types
Touching Scanning
(radio tags, e.g. NFC or RFID) (visual tags, e.g. bar codes)
Pointing Text Input
(tag-less vision-based identification) (e.g. name, ID, …)
Oct 15, 2012 A. Möller, S. Diewald, L. Roalter, M. Kranz 6

7. Distributed Multimodal Information Processing Group Technische Universität München
Excursus: Pointing (Vision-based Recognition)
• Image processing is used to detect
visual features of an image
• A query in feature space returns
similar images from a reference
database
• Good choice of feature type allows
very reliable results (e.g. MSER)
– High distinctiveness (e.g. by
using text-related features)
– Scale invariance (works at
different distances)
– Rotation invariance (works at
different angles)
• Enabled by rise in mobile CPU
performance (multi-core...)
Oct 15, 2012 A. Möller, S. Diewald, L. Roalter, M. Kranz 7

8. Distributed Multimodal Information Processing Group Technische Universität München
Prototype
• Implementation as Android application
• 47,000 drugs in query database
• 100,000 reference images
Oct 15, 2012 A. Möller, S. Diewald, L. Roalter, M. Kranz 8

9. Distributed Multimodal Information Processing Group Technische Universität München
Outline
Background and Motivation
Scenario and Prototype
User Study
Discussion and Conclusion
Oct 15, 2012 A. Möller, S. Diewald, L. Roalter, M. Kranz 9

10. Distributed Multimodal Information Processing Group Technische Universität München
Research Questions
• RQ1: What advantages and disadvantages of identification techniques,
as presented in MobiMed, can be determined?
– ...in terms of effectiveness? large-scale, online
– ...in terms of efficiency? lab
• RQ2: Which method is preferred by users?
– ...a priori? large-scale, online
– ...after practical use? lab
• RQ3: What potential do people see for MobiMed as a whole?
– ...a priori? large-scale, online
– ...after practical use? lab
Oct 15, 2012 A. Möller, S. Diewald, L. Roalter, M. Kranz 10

11. Distributed Multimodal Information Processing Group Technische Universität München
Methodology
• Online study
– Human Intelligence Task at Amazon mTurk
– 149 participants
• 74 females, 75 males
• 17-79 years (average: 31, standard deviation: 11)
– Questionnaire survey
• Lab study
– 16 participants
• 6 females, 10 males
• 22-69 years (average: 31, standard deviation: 12)
– Experimental task + Questionnaire survey
• Identification of 10 packages
with each of four methods
• Within-subjects design, permuted order
Oct 15, 2012 A. Möller, S. Diewald, L. Roalter, M. Kranz 11

12. Distributed Multimodal Information Processing Group Technische Universität München
Results: RQ1 (Individual Method Comparison)
Method
Advantages
Disadvantages
Scanning
Quick, precise, high Visual code + camera
familiarity
required, need to find and
focus on code
Touching
Hassle-free, fool-proof, NFC augmentation and
quick
NFC-capable phone
required, privacy skepticism
Pointing
Intuitive to use, „most Computational demand,
human form“ of interaction, ambiguous results possible
works from any angle, works
also with catalog/website
images, no product tagging
required
Text
Highest familiarity, accurate, High amount of typing,
search term flexibility
misspelling, slow, difficult
Oct 15, 2012 A. Möller, S. Diewald, L. Roalter, M. Kranz 12

13. Distributed Multimodal Information Processing Group Technische Universität München
Results: RQ1 (Efficiency)
Oct 15, 2012 A. Möller, S. Diewald, L. Roalter, M. Kranz 13

14. Distributed Multimodal Information Processing Group Technische Universität München
Results: RQ2 (User Preferences)
-3 = strongly disagree, +3 =strongly agree
Observations/interpretations:
• Touching was only #3 in online survey, but rated best in lab study
• Possible explanation: low familiarity (as soon as people used it, they liked it)
Oct 15, 2012 A. Möller, S. Diewald, L. Roalter, M. Kranz 14

15. Distributed Multimodal Information Processing Group Technische Universität München
Results: RQ3 (Utility of Tool in Scenario)
• Information sources on drugs: • Suggestions for additional
– Doctor/pharmacist (75%) features
– Package insert (69%) – Price comparison
– Books/internet (56%) – Active ingredient analysis
– Self-diagnose
• Would you be interested in – Personalized medication
MobiMed as alternative source for management
drug information? 88%
• Would you use a system such as
MobiMed? 82%
• Average amount of money subjects
would spend: $8.40 (aged >25:
$14.01)
Oct 15, 2012 A. Möller, S. Diewald, L. Roalter, M. Kranz 15

16. Distributed Multimodal Information Processing Group Technische Universität München
Results: RQ3 (Usability of Prototype)
Oct 15, 2012 A. Möller, S. Diewald, L. Roalter, M. Kranz 16

17. Distributed Multimodal Information Processing Group Technische Universität München
Outline
Background and Motivation
Scenario and Prototype
User Study
Discussion and Conclusion
Oct 15, 2012 A. Möller, S. Diewald, L. Roalter, M. Kranz 17

18. Distributed Multimodal Information Processing Group Technische Universität München
Discussion and Conclusion
• Physical Mobile Interaction is popular and efficient
– Was preferred over conventional (text) search
– Was faster than text search
• Touching and Scanning evaluated best
– Fastest and most popular physical mobile interaction methods
– Touching faster and more popular than scanning in lab study
– Scanning more popular in online survey (familiarity)
• Vision-based Search (pointing) as future alternative?
– Natural; works for any object (no augmentation needed)
– Reliability/speed improvement needed, but almost as fast as scanning
• Best method depends on intended scenario
• General demand for medical apps
Oct 15, 2012 A. Möller, S. Diewald, L. Roalter, M. Kranz 18

19. Distributed Multimodal Information Processing Group Technische Universität München
Thank you for your attention!
Questions?
?
?
andreas.moeller@tum.de
www.vmi.ei.tum.de/team/andreas-moeller.html
Oct 15, 2012 A. Möller, S. Diewald, L. Roalter, M. Kranz 19

20. Distributed Multimodal Information Processing Group Technische Universität München
References
• Slide 3:
– Rukzio, E. Physical mobile interactions: Mobile devices as pervasive mediators for interactions
with the real world. PhD thesis, 2006
– Want, R., Fishkin, K., Gujar, A., and Harrison, B. Bridging physical and virtual worlds with
electronic tags. In Proceedings of the SIGCHI conference on Human factors in computing
systems: the CHI is the limit, ACM (1999), 370–377.
• Slide 10: https://www.mturk.com/mturk/welcome
• All other images: Microsoft ClipArt 2012
Oct 15, 2012 A. Möller, S. Diewald, L. Roalter, M. Kranz 20

21. Distributed Multimodal Information Processing Group Technische Universität München
Paper Reference
• Please find the associated paper at:
http://dx.doi.org/10.1145/2399016.2399022
• Please cite this work as follows:
• Andreas Möller, Stefan Diewald, Luis Roalter, and Matthias Kranz. 2012.
MobiMed: comparing object identification techniques on smartphones. In
Proceedings of the 7th Nordic Conference on Human-Computer Interaction:
Making Sense Through Design (NordiCHI '12). ACM, New York, NY, USA,
31-40. DOI=10.1145/2399016.2399022 http://doi.acm.org/
10.1145/2399016.2399022
Oct 15, 2012 A. Möller, S. Diewald, L. Roalter, M. Kranz 21

22. Distributed Multimodal Information Processing Group Technische Universität München
If you use BibTex, please use the following entry
to cite this work:
@inproceedings{Moller:2012:MCO:2399016.2399022,
author = {M"{o}ller, Andreas and Diewald, Stefan and Roalter, Luis and Kranz, Matthias},
title = {MobiMed: comparing object identification techniques on smartphones},
booktitle = {Proceedings of the 7th Nordic Conference on Human-Computer Interaction: Making Sense Through Design},
series = {NordiCHI '12},
year = {2012},
isbn = {978-1-4503-1482-4},
location = {Copenhagen, Denmark},
pages = {31--40},
numpages = {10},
url = {http://doi.acm.org/10.1145/2399016.2399022},
doi = {10.1145/2399016.2399022},
acmid = {2399022},
publisher = {ACM},
address = {New York, NY, USA},
keywords = {object identification, physical mobile interaction, pointing, scanning, touching},
}
Oct 15, 2012 A. Möller, S. Diewald, L. Roalter, M. Kranz 22