Our presentation on mobile interactive hologram verification at ISMAR 2013 in Adelaide, Australia ABSTRACT: Verification of paper documents is an important part of checking a person’s identity, authorization for access or simply establishing a trusted currency. Many documents such as passports or paper bills include holograms or other view-dependent elements that are difficult to forge and therefore are used to verify the genuineness of that document. View-dependent elements change their appearance based both on the viewing direction as well as dominant light sources, thus it requires special knowledge and training to accurately distinguish original elements from forgeries. We present an interactive application for mobile devices that integrates the recognition of the documents with the interactive verification of viewdependent elements. The system recognizes and tracks the paper document, provides user guidance for view alignment and presents a stored image of the element’s appearance depending on the current view of the document also recording user decisions. We describe how to model and capture the underlying spatially varying BRDF representation of view-dependent elements. Furthermore, we evaluate this approach within a user study and demonstrate that such a setup captures images that are recognizable and that can be correctly verified.

1. Graz University of Technology
1hartl@icg.tugraz.at ISMAR 2013
Mobile Interactive Hologram Verification
A. Hartl, J. Grubert, D. Schmalstieg, G. Reitmayr
hartl@icg.tugraz.at

2. Graz University of Technology
2hartl@icg.tugraz.at ISMAR 2013
Motivation
View-Dependent Elements
 Strong dependence on viewing angle/light sources
 Security elements (e.g., holograms)
 Often only a few distinct views
 Color copies, substitutes etc.
Document Verification
 ID-cards, passports, banknotes
 Checking security elements
 Several security levels
=> Level 1: check with a manual

3. Graz University of Technology
3hartl@icg.tugraz.at ISMAR 2013
Contribution
Hologram Verification with Mobiles – Feasibility Check
 Capturing hologram patches
 View alignment/user guidance
 Mobile AR prototype
 Evaluation within user study
 Revision of the prototype

4. Graz University of Technology
4hartl@icg.tugraz.at ISMAR 2013
Hologram Capture
Assumption
 Total radiance from a point on the hologram is dominated by a single light source
 Fixed distance to surface
Dominant Light Source
 LED light on mobile phone
 Fixed to the camera with offset vector o: l ~ P + R . o
Representation
 Spatially Varying BRDF
 I=I(x,y,l,d); l,d as unit vectors; 2 DOF each
 6D appearance model per channel
 High frequency representation for capturing sharp edges
required => use of texture, keeping warped patches
 5D model; indexed by R and location x,y on the document

5. Graz University of Technology
5hartl@icg.tugraz.at ISMAR 2013
View Alignment
Objective
 Mobile method for alignment with a given 6 DOF pose
 Reasonable accuracy despite mobile environment
Approach
 Visual guidance with iron sights and a virtual horizon
 1 matching direction of the viewing ray
 2 position along the ray
 3 in-plane rotation
1
2
3

6. Graz University of Technology
6hartl@icg.tugraz.at ISMAR 2013
Evaluation – User Study
Questions
1. How accurate can the user acquire viewing directions with guidance?
2. Can the user verify a hologram with the proposed approach?
3. How would the approach compare with a digital manual?
4. Would it be feasible to build an automatic system?
Study – Hologram Verification Task
 Wrapped 50 Euro banknotes; AR prototype vs. check with a manual
 Controlled environment
 Learning phase for both AR system and digital manual
 No hints on similarity/dissimilarity
 Participants could stop at any time
 Decision on validity required
Participants
 17 volunteers; 1 female
 Little experience with holograms

7. Graz University of Technology
7hartl@icg.tugraz.at ISMAR 2013
Maneuvering and Task Performance
Alignment Error
 Translation: -8mm-10mm
 Rotation: -8 degrees - 8 degrees
 Largest error with first view
 Reasonable accuracy for non-experts.
Task Completion Times and Soundness
 No difference in soundness
„I think, the hologram is real.“
 Significant effect of interface on completion
time; higher temp. effort with AR system
 Users were able to verify holograms
with both setups.

8. Graz University of Technology
8hartl@icg.tugraz.at ISMAR 2013
Subjective Assessment
NASA TLX Weighted Scores
 Workload assessment
 Higher physical/temporal demand
with AR system: users are forced
to move to the right pose
 No clear evidence of either interface
on validity of the element
AttrakDiff Questionnaire
 Measuring attractiveness, usability
 lower usability for AR system
 Higher scores for hedonic dimensions

9. Graz University of Technology
9hartl@icg.tugraz.at ISMAR 2013
Subjective Assessment and Automation
IMI (Intrinsic Motivation Inventory)
 Measuring intrinsic motivation
 No effect on value/usefulness
 Significant effect on interest/enjoyment:
higher motivation with AR system
Patch Matching
 Registration with optical flow
 Normalized cross correlation (NCC)
 4/6 views have NCC scores > 0.75
 Repeatable image capture of hologram patches

10. Graz University of Technology
10hartl@icg.tugraz.at ISMAR 2013
Revised Prototype
Comments from User Study
 Physical strain => automatic recapture during final alignment
 Cognitive load => display of captured data, change of local decisions
 Live-view; alignment ranges
Informal Study (7 Participants)
 Comparison with previous iteration
 Higher confidence (5/7, 2 equal)
 Equal strain/effort (5/7, 2 less)
 Live-view and visual cues are useful (verbal)
ISMAR Demo Session
recorded patch
live-view
reference patch

11. Graz University of Technology
11hartl@icg.tugraz.at ISMAR 2013
Conclusion
Feasibility check for hologram verification on mobiles
 Mobile SVBRDF image capture using dominant light source
 User guidance approach using iron sights and the virtual horizon
 Mobile AR prototype system
 Evaluation in user study and comparison with digital manual
Findings
 Capture of holograms and view alignment worked reasonably well.
 Higher motivation to use the AR system, but it did not provide more value.
 Hologram verification on mobiles seams feasible.
Future work
 Evaluation of the revised prototype with substitutes or real fakes.
 Crafting a less straining approach with dense matching.
 Improving tracking robustness.
This work is supported by Bundesdruckerei GmbH.

12. Graz University of Technology
12hartl@icg.tugraz.at ISMAR 2013
References
R. L. van Renesse. Optical Document Security. Artech House, third edition, 2005.
T.-H. Park and H.-J. Kwon. Vision inspection system for holograms with mixed patterns. In CASE, pages 563–567, 2010.
M. Haindl and J. Filip. Visual Texture. Advances in Computer Vision and Pattern Recognition. Springer Verlag, 2013
J. Jachnik, R. A. Newcombe, and A. J. Davison. Real-time surface light-field capture for augmentation of planar specular surfaces. In ISMAR,
pages 91–97, 2012
P. Ren, J. Wang, J. Snyder, X. Tong, and B. Guo. Pocket reflectometry. n Proc. SIGGRAPH 2011, SIGGRAPH ’11, pages 45:1–45:10, New York,
NY, USA, 2011. ACM.
Y.-C. Cheng, J.-Y. Lin, C.-W. Yi, Y.-C. Tseng, L.-C. Kuo, Y.-J. Yeh, and C.-W. Lin. Ar-based positioning for mobile devices. In ICPPW, pages 63–
70, 2011
K. Chintamani, A. Cao, R. Ellis, and A. Pandya. Improved telemanipulator navigation during display-control misalignments using augmented reality
cues. Systems, Man and Cybernetics, Part A: Systems and Humans, 40(1):29–39, 2010
S. G. Hart and L. E. Staveland. Human Mental Workload, chapter Development of NASA-TLX (Task Load Index): Results of empirical and
theoretical research. North Holland Press, Amsterdam, 1988
M. Hassenzahl, M. Burmester, and F. Koller. AttrakDiff: Ein Frage-bogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität.
In Mensch & Computer 2003: Interaktion in Bewegung, pages 187–196, Stuttgart, Germany, 2003. B. G. Teubner.
E. McAuley, T. Duncan, and V. V. Tammen. Psychometric properties of the intrinsic motivation inventory in a competitive sport setting: A
confirmatory factor analysis. Research quarterly for exercise and sport, 60(1):48–58, 1989
T. P. M., Urschler, C. Zach, R. Beichel, and H. Bischof. A duality based algorithm for tv-l1-optical-flow image registration. In MICCAI, pages 511–
518, 2007