This document provides an overview of a project to develop an augmented reality face recognition application for mobile devices. It recaps the concept, which is to use a smartphone camera and face recognition to access extra information about people from social networks. It describes scenarios, technical details, an implementation using the MVC pattern in iOS with classes like a Facebook request wrapper and user model. It outlines a storyboard, shows problems encountered, and provides statistics on the project planning and hours spent on various aspects of the work. Evaluation results from user testing of paper and digital prototypes are also summarized.

1. Augmented reality face recognition
for mobile devices
Gerry Hendrickx
Promoter: Professor Erik Duval
Coordinator: Gonzalo Parra
1

2. Overview
• Recap
• Implementation
• Storyboard/Screen transition diagram
• Evaluation
• Demo
• Problems
• Todo
• Planning
• Statistics
2

3. Recap
3

4. Recap: Concept
Augmented reality face recognition for mobile devices…
Camera of smartphone will be
Real world augmented with extra used to accomplish goals
information, by using a camera -> Extra information on screen
Extra information will be
about people, by recognizing
their faces
Extra information will be
taken from social networks
… with social network information
4

5. Recap: Scenarios
1. Get contact info
2. Create a face based contact book
3. Quick access to slides/publications by
recognizing speaker
4. Quick access to social network information
5

6. Recap: Technical info
• iOS – Face detection
• iPhone
• Face.com – Free, SDK, private namespace,…
6

7. Recap: Paper prototyping
• 3 iterations
7

8. Paper prototyping: Iteration 1
8

9. Paper prototyping: Iteration 1
9

10. Paper prototyping: Iteration 1
10

11. Paper prototyping: Iteration 2
• Elaboration of UI3
• New functionality: History, Settings, Overview
11

12. Paper prototyping: Iteration 2
• Positive: • Negative:
– Clean UI – Not enough home
– iOS style buttons
– The concept – Unclear contact icon
– Not him/her button!
12

13. Paper prototyping: Iteration 3
• Adapted to negative points
• Tested with prof Duval and 5 assistants
• Should improve:
– Incorrect button
– Still too much clicking to get somewhere =>
Tabbed Bar
– Delete person from history
13

14. Implementation
14

15. Implementation: MVC
Controller View Model
SocialRecognizerViewController SocialRecognizerView UserModel
CameraViewController CameraView
OverviewViewController OverviewView RecognizedPerson
HistoryViewController HistoryView UserModel
SettingsViewController SettingsView UserModel
FBRequestWrapper
15

16. Implementation: Class diagram
16

17. Implementation: Class diagram
17

18. Implementation: FBRequestWrapper
– Singleton
– Handles all requests from Facebook
– Has delegate-methods
– Preferred way to implement it!
18

19. Implementation:
CameraViewController
– Uses AVCaptureSession instead of
UIImagePickerController
– Linked with Face.com through API
– Uses iOS5 face detection to track faces
– Redraws facebox each frame
– Tried to be memory efficient
19

20. Implementation: UserModel
– Saves UID’s and names of friends
Reduces # requests
– Saves list of recognized persons
– Saves the settings
20

21. Implementation: RecognizedUser
– Saves UID and name of recognized person
– Saves profile picture
– Saves all other information received from
Facebook request
21

22. Storyboard/Screen transition
Diagram
22

23. Storyboard/Screen transition Diagram
23

24. Demo
24

25. Storyboard
25

26. Storyboard
26

27. Storyboard
27

28. Evaluation
28

29. Evaluation: Iteration 1
– Tested with 7 smartphone users
– 4 already tested the paper prototype
– Current state of the app
– Focus on face recognition
– Results and comments were saturated fast
29

30. Evaluation: Iteration 1
– Think aloud
– Tasks
– Extra questions about satisfaction
– USE questionnaire
30

31. Evaluation: results
– Tapping the face is cumbersome
– Statusinfo
31

32. Problems
32

33. Problems
– Knowing which face belongs to who
– Tapping the face
– Augmented part is failing…
33

34. Todo
34

35. Todo
– Adapt to results evaluation
– Implement history and settings
– Training algorithm
– Multiple social networks
– Private namespace
Requires database
Which order???
35

36. Planning
36

37. Planning
37

38. Statistics
38

39. Statistics
# Blog posts 21
# Comments on other blogs 10 (hcifetcher results: 3075)
# Tweets concerning thesis 81
Total # of hours worked 395
# of hours on literature study 20
# of hours on related work 10
# of hours on reports/blog posts 26
# of hours dedicated to other theses 18
# of hours on iOS learning 52
# of hours on design 35
# of hours on paper prototype testing 28
# of hours on paper prototype creating 35
and evaluating
# of hours on implementation 166
# of hours on digital prototype testing 5
39

40. Thank you for listening!
Questions?
40