A Usability Evaluation carried out on my second year Brunel Group project.
A.R.C. (Augmented Reality Communicator), is an augmented reality social networking application , designed and built for my second year group project at Brunel University.
Optimizing AI for immediate response in Smart CCTV
A.R.C. Usability Evaluation
1. h
CS2003 Usability Engineering Report For A.R.C.
by Jan P.C. Hanson
2014
StudentID: 1339404
Level 2
Abstract
A.R.C. (Augmented Reality Communicator) is a project, in its infancy, that tries to provide
social media functionality with a focus on professional collaboration. This puts it somewhere
between Google's suite of tools and LinkedIn. This document provides some background
to the project ostensibly from a usability engineering perspective, as well as the covering
the requirements gathering procedure as well as usability testing of the application and the
ramications thereof.
1
3. 1 Background
ARC stands for Augmented Reality Communicator. It is an attempt to produce a social networking
application that uses augmented reality (in the form of live camera preview overlays and map
overlays) to visualise a social network aimed at making professionals' lives easier.
It is widely known that Social Networking Sites (SNS's) have become widely used in the last 15
years, and based on anecdotal evidence, LinkedIn appears to be the SNS of choice for users wanting
to engage in social networking in a professional setting. (Duggan Smith 2013) recorded that
42% of adults now use multiple SNS's, of the online population measured in this study 71% use
Facebook making this by far the most popular SNS on the list, while 22% use LinkedIn(in addition
to, or as their sole SNS).
(Hampton et al.,2014) noted that when it came to the Snowdon NSA scandal, 58% of Facebook
users and 59% of twitter users were unwilling to discuss the topic of government surveillance
programs, while 35% of people would be unwilling to discuss it at work. By extension this may go
some way to explaining the divide in usage; LinkedIn is by denition an SNS used by employers
and employees for the purpose of people marketing themselves to each other, and thus only 'work
safe' communications are generally posted.
(Quan-Haase Young 2010) reported Facebook users primarily using the site for maintaining
pre-existing relationships and keeping in touch with school friends, in this same paper they also
explain that instant messaging fullls dierent needs to that of a social networking site such as
facebook; that of maintaining and supporting relationships with distant others. Facebook currently
incorporates instant messaging also. Couple this multi-mode communication with the percieved
openess of communication (anecdotal), and it is small wonder why Facebook is the more ubiquitous
SNS.
With ARC I would like to attempt to bridge the divide between LinkedIn and Facebook, to provicde
a social networking application that can be used in a professional context without the apparent
restrictive feel that comes with knowing one is in full view of any potential employer.
I believe that one way to achieve this is to make the experience enjoyable, to this end, the results
of (Saket et al., 2015) show that users enjoy information visualisation more when graphs are
embellished. This lead to my choice to incorporate AR as a central part of this project. The paper
goes on to explain the use of a modied Flow model to explain and partially quantify how and
why certain tasks are enjoyed more than others:
Challenge - Generally, enjoyment occurs when the challenge in an activity matches the
skill of the participant.
Focus - Enjoyable activities require complete attention on the task at hand.
Clarity - Enjoyment occurs when the goals of the activity are clearly understood, and the
users knows what he/she is working towards.
Feedback - Enjoyement occurs when the task undertaken provides immediate feedback.
Control - An application should provide a feeling of control over the environment specieed.
Immersion - The participants should lose themselves in an activity.
(ISO 9241-11, 1998) denes usability as:
3
4. The extent to which a product can be used by specied users to achieve specied
goals with eectiveness, eciency and satisfaction in a specied context of use.
(Speicher, M., 2015) expands on this denition by specifying what is meant by :product, specied
users, specied goals and specied context, as well as dening the level of usability metric used:
usability 2 LEVEL PRODUCT USERS GOALS CONTEXT
This provides a useful semi-formalism to dene, more consistnatly, the parameters for this usability
evaluation.
LEVEL Internal External In Use
PRODUCT application code prototype's and mobile application
and comments. screenshots.
USERS Programmers test participants test participants
GOALS implementing new completion of test (reading posts, creating posts,
functionality tasks, adding friends, viewing
friend information), using
all available visualisation
methods
CONTEXT eclipse and provided observation in real world,
documentation plus controlled interviews, questionaire
interviews environment plus
interviews
Table 1.
(Speicher, M., 2015) denes LEVEL under three distinct catagories:
Internal metrics, which measure a set of static attributes (e.g., related to software archi-
tecture and structure). In the context of this evaluation this would refer to the ability for
a programmer to maintain and modify the code.
External metrics, which relate to the behavior of a system (i.e., they rely on execution of
the software). This would be a simple measure of whether the software performs the specic
tasks set out in the requirements.
In-use metrics, which involve actual users in a given context of use. This last entry would
then refer to the application used in 'real life' context.
2 Aims and Objectives
2.1 Aims
The aim of this usability evaluation is three-fold: Firstly to determine potential user's most wanted
features; Secondly to determine potential user's ecacy with the proposed interface and lastly to
discover usability pitfalls with the system.
4
5. 2.2 Objectives
Undertake a literature review
Determine requirements gathering tools
Collect user requirements
Create basic prototype
Evaluate basic prototype
Analyse results
Design Java application for Android platform
Evaluate application with potential users.
Analyse results
Write usability report.
3 Research Methods
3.1 Requirements Gathering
(M.F. DiAngelo C.J. Petrun, 1995) work through a very detailed and well specied methdology
known as CRTS (Custormer Requirements and Task Specication), which focuses on gaining
requirements based solely on customer input, through an extensive series of expert-led sessions.
Unfortunately utilizing this in its full capacity would have taken a restrictive amount of time. I
would like to take some key points from this and apply them to my own requirements gathering
methods, namely: that user requirements should come entirely from the customer (with the caveat
that, there exist some pre-existing requirements for this project), and that requirements should
be developed into a 3-tier system with top tier attributes of the system(i.e. the system should
be: consistant etc), middle tier indicators for these attributes(i.e. no down time), and lower tier
measurables (i.e. monthly down time).
3.1.1 Pre-Existing Requirements
At the outset of this project some arbitrary requirements were prescribed for the project (David
Bell, 2014) Which include the use of augmented reality, and the requirement that the project must
be completed using Google's Java Android framework. The justicastion for the use of AR in this
particular context is given in the background section.
3.1.2 Semi-Structured Interviews
The rst method that I elected to use for the purpose of requirements gathering is a set of semi
structured interviews performed on a test population of potential users. (Doody Noonan, 2013)
building on the work of (Holloway Wheeler, 2010), describe the eective use of semi-structured
interviews to allow a free ow of ideas and context, dicult to obtain via structured interviews,
to be developed; while allowing the researcher to eliminate some of the observational bias inherent
5
6. in unstructured interviews. The question themes used to direct the course of the interview, within
the context of this project where as follows: What role do you think an SNS could play in your
day to day professional activities, how comfortable are you using GPS/Sensors, What features do
you use most/would you most like.
3.1.3 Brainstorming
In conjunction with the method specied above it was decided upon to use brainstorming. Both
(M.F. DiAngelo C.J. Petrun, 1995) and (AL-Hothali, 2012) describe it's use for the elicitation of
requirements in software developement. In the context of ARC, this was performed by a group of 9
Potential users and the system developer using the group passing technique described in (Osborne,
1963)
3.1.4 Survey
Lastly a survey was released onto Facebook and Twitter, asking a series of descriptive, true/false,
selection and Likert style questions. These questions attempt to ascertain(amongst other things)
potential users' most favoured/wanted features from an SNS and users current SNS usage (Hanson,
2014).
3.2 Prototype Design
A series of prototypes was used to aid both the design process and the usability of this project. This
process was started by using the POP application to produce an interactive paper-style prototype
as advised by (Wilson Fletcher, 2014). Following this a HTML mockup was created to more closely
simulate the look and feel of the end product (Van Buskirk Moroney, 2003).
The look and feel of these prototypes was deliberately left rough while engaged in usability testing
as (Van Buskirk Moroney, 2003) noted that by doing so it is possible for the test user to
concentrate more on the functional usability, as they are less likely to forgive errors because of a
polished appearance.
3.3 Usability Testing
As explained in the 'Background' section of this paper, and recommended by (Speicher, 2015), the
actual usability is evaluated acording to three metrics (internal, external and in-use).
3.3.1 Internal
This is the metric that is hardest to quantify, and from the consumers point of view, the least
important. However, if this project were to continue in the long run, maintainable, easy to read
code ultimately aids usability as new features are less likely to be prone to crippling bugs, and
those bugs that do crop up should (by denition) be easier to nd and x.
Because of this, I have decided to spend some time evaluating these attributes of ARC by giving
the source code to 5 developers not involved with this project and asking them to implement some
test functionality. A time limit of one week was given to them, after which their code was submitted
and the developers interviewed.
The developers were of varying skill levels and asked to implement a new text based view with for-
ward and backward buttons to view the next/previous post based on chronological order. The only
information available to the developers was the sourcecode documentation and a brief description
6
7. amounting to what is contained in the rst paragraph of this report. The code that they submitted
was not used in the nal product.
3.3.2 External
This metric dened the testing of the initial POP prototypes and the HTML Mockup. Test par-
ticipants were asked to review these prototypes in the context of completing a series of test tasks
aimed at assessing the 4 aspects of usability proposede by (Shackel, B., 1991). These test tasks
were performed under observation in laboratory conditions and the time taken to complete the
tasks was recorded. The same tests(where applicable) where performed on facebook and linkedin.
No information was given to the test subjects other than what is contained in the rst paragraph
of this report. Semi-Structured interviews were then carried out given the theme of (Shackel, B.,
1991)'s 4 criterion: eectiveness, learnability, exibility and attitude. (appendix 7.1))
3.3.3 In-Use
This context denes the usability in a real word scenario, therefore the app was given to test
participants, and they were asked to use it o site as if it were any other mobile application. A time
limit of two days was set after which they were given a questionnaire based on a modied set of
questions from (Thuseethan et al, 2014) to ascertain the usability of the product in its current state.
Lastly unstructured interviews were conducted (Doody Noonan, 2013) where participants were
asked for there overall thoughts on the product. Test participant comments were used to drive the
interview in an attempt to reduce researcher bias. (appendix 7.1))
4 Presentation of Results
4.1 Requirements Gathering
4.1.1 Survey
Figure 1. Which of these social networks do you use
7
8. Figure 2. How often do you check your network(s)
Figure 3. How often do you post messages
Figure 4. Which of these features would you nd useful in a professional context
8
9. Figure 5. Password Security
Figure 6. GPS usage
Figure 7. Connectivity
Figure 8. Movement Sensors
4.1.2 Brainstorming Interviews
As a result of Brainstorming and interviews the following points where made regarding features
and behaviour
i. A map view that has posts shown at the geographical location where they were created
when clicking on them, should expand to show post.
ii. There should be a timeline view for those times where GPS availability is limited (tube
stations, inside certain buildings etc), this should be the default page.
iii. A view that shows a live feed from the camera, overlayed with posts bassed on geographical
location, when clicked on they should expand to show the conent of the post.
iv. There should be dierent levels of relationship (colleague, friend, employer), to limit and
control who gets access to particular information.
9
10. v. User Info should contain geographical information, but you should be able to chose who
views it.
vi. Location should be monitored and recorded, so that other users can get see status inform-
ation in real time.
vii. Views should be made easily accessible, maybe placing icons prominently on all screens, at
the top of the screen.
viii. Any view should be accessible from inside any other view, to make it easy to navigate.
ix. There should be no cap on how large a post entry should be, after a certain size the post
should be folded.
4.2 Usability Testing
4.2.1 Internal
4 out of the 5 developers where able to complete the task in the alotted time. Implementing a text
based post view with forward and back buttons to view the next/previous post in chronological
order.
After interviews with the developers, the following points where concluded:
After realising that the project conformed to an MVP architecture, and that the strategy
pattern (Gamma et al, 1994) had been used to implement the Views portion of the MVP
architecture; it was conceptually straightforward to t new functionality into the framework.
Sourcecode documentation was good but a little inconsistant, with some functionality well
documented, while other functionality had comments missing or incomplete, leading to delay
in nishing the test task.
The use of a dummy model caused some confusion, as it was unclear by the specication of
the test task as to whether this was meant to be used or not.
Naming conventions were slightly mixed, similar to the documentation, leading to a small
measure of confusion in which method parameters where being passed and which were used
internally to certain classes.
4.2.2 External
Figure 9. Time taken to write a test post (seconds)
10
11. Figure 10. Time taken to view a post using the camera view (seconds)
Figure 11. Time taken to view a post using the map view (seconds)
Figure 12. Time taken to view friend information (seconds)
interview results(amalgam of HTML POP):
functionality clear , placed in standard places on screen
all expected functionality of SNS but no frills bordering on too basic
iconography clear
good use of screen real estate
11
12. No clear indication of how to access friend information.
lack of information formatting
Notes:
4.2.3 In-Use
Figure 13. Usability Questionnaire results table
(see appendix 7.1 for question numbers)
Figure 14. Usability Questionnaire results graph
12
13. Interview results:
not enough funtionality implemented to really be usable
the functionality that is there is easy to use and navigate.
map and camera easy to use
no warning of gps o
good use of screen real estate (doesnt feel cluttered)
crashes on switching between camera/map/post without returning to main screen rst
lack of ability to edit posts once created
lack of information formatting
feels very basic
5 Discussion of Results
5.1 Internal (refer to 4.2.1)
The general reaction of the developers tasked with implementing was positive, with all of the
developers except one manageing to implement the required functionality. The reason that
developer was not able to complete the task was due to a lack of familiarity with object ori-
ented techniques and design patterns.
There were however some points to take away from this excercise: it was noted that the comments
were slightly inconsistant between classes as were variable naming conventions. This led to some
delays in coding as developers spent more time than was necessary deciphering the provided
documentation before they could start.
This can be rectied by going through the source code and rewriting the comments so that doxygen
will produce more consistant docs. The naming conventions may take more work but is also
perfectly feasible.
5.2 External (refer to 4.2.2)
As can be seen from gure's 9 - 12 the times taken to complete the various tasks is of similar length
to that of facebook and linkedin with the exception of viewing friend information. With regards to
gures 9 - 11 the HTML and pop mockups take a little longer but not a signicant amount, and
not enough to produce signicant negative sentiment according to the interviews conducted.
The time taken to view friend information is signicantly longer than facebook and linkedin, and
this is echoed in the interviews conducted, with the general consensus being that it isnt clear how
to do it, and that it was only through guestimation that the task was completed. This may be
due to the fact that there is no actual 'friends' tab or section within the app and the only way to
access friend information is by tapping on the users icon or name.
This needs to be rected in the nal product. Another thing that was commented on was that
there was a lack of formatting to the information, both post info and user info. This made the
interface a little o putting and is an issue that also must be addressed.
13
14. 5.3 In Use (refer to 4.2.3)
Looking at the interview results it is clear that there are some serieous usabaility issues that need
to be addressed, First among these is the fact that the system crashes under certain conditions,
this is a fatal aw and must be rectied before anything else. The next is the lack of error
messages (interviews question 13[g 13-14, appendix 7.1]), this is something vital for the app
to communicate with the user during situations such as the gps not being switched on. without
an error message the user will likely just wonder why the app isnt working and stop using it. The
other points mentioned in theinterview results are less of a priority but still important. The lack of
information formatting would probably help the app feel more polished, adding extra functionality
would make it feel more complete, but theses I feel are probably a lower priority than changing
the structure of posts so they can be edited. It was noted in the interviews and in question 14
that there is general sentiment that it is dicult to recover from a mistake if one is made, and this
would probably go some way towards xing that.
On the plus side, the app was generally well recieved with regards to ease of use, learnability and
peoples attitude towards it. (see other questions [appendix 7.1 g 14]). Overall the app has both
attractors and detractors in roughly equal measure, hinting that the app shows promise but needs
work.
14
15. 6 References Bibliography
[1]Maeve Duggan, Aaron Smith,. (2013). Social Media Update 2013. Available: http://pewin-
ternet.org/Reports/2013/Social-Media-Update.aspx. Last accessed 10 Mar 2015.
[2]Hampton, K.N., Rainie, L., Lu, W., Dwyer, M., Shin, I., Purcell, K. (2014). Social Media
and the Spiral of Silence.' Pew Research Center, Washington, DC. Available at http://www.pew-
internet.org/2014/08/26/social-media-and-the-spiral-of-silence/
[3]Anabel Quan-Haas , Alyson L. Young. (2010). Uses and Gratications of Social Media: A
Comparison of Facebook and Instant Messaging. Bulletin of Science, Technology Society. 30
(5), p350-361.
[4]Bahador Saket, Carlos Scheidegger, Stephen Kobourov. (2015). Towards Understanding Enjoy-
ment and Flow in Information Visualization.arXiv:1503.00582. 1 (1), p1-11.
[5]ISO 9241-11:1998: Ergonomic requirements for oce work with visual display terminals (VDTs),
Part 11: Guidance on usability, 1998.
[6]Maximilian Speicher. (2015). What is Usability? A Characterization based on ISO 9241-11 and
ISO/IEC 25010. arXiv:1502.06792. 1 (1), p1-10.
[7]M. F. DiAngelo., C. J. Petrun.. (1995). Collecting Product based Usability Requirements. IBM
Systems Journal. 34 (1), p4-19.
[8]David Bell. 2014. CS2001 - Group Project. [ONLINE] Available at:
https://blackboard.brunel.ac.uk/webapps/portal/frameset.jsp?tab_tab_group_id=_2_1url=%2Fwebapps%2Fbla
board%2Fexecute%2Flauncher%3Ftype%3DCourse%26id%3D_17217_1%26url%3D. [Accessed
16 March 15].
[9]Doody 0, Noonan M (2013) Preparing and conducting interviews to collect data. Nurse
Researcher. 20, 5, 28-32.
[10]Holloway I, Wheeler S (2010) Qualitative Research in Nursing and Healthcare.Third edition.
Wiley-Blackwell. Oxford
[11] Samaher Abdullah AL-Hothali, Noor, Anusuyah Subbarao Abdulrahman AL-Zubaidi, . (2012).
Requirements Elicitation For Software Projects. International Journal of Computer Science and
Information Security. 10 (11), p64-71.
[12] Osborn, A.F. (1963) Applied imagination: Principles and procedures of creative problem
solving (Third Revised Edition). New York, NY: Charles Scribner's Sons.
[13]Wilson Fletcher, (2014), 'Prototyping - The Tools: What We Use
and When To Use Them', CS2003: Usability Engineering, Available at:
https://blackboard.brunel.ac.uk/webapps/portal/frameset.jsp?tab_tab_group_id=_2_1url=%2Fwebapps%2Fbla
board%2Fexecute%2Flauncher%3Ftype%3DCourse%26id%3D_17219_1%26url%3D, [Accessed
16 March 15]
[14]Shackel, B. (1991). Usabilitycontext, Framework, Denition, Design and Evaluation. Shackel,
B. and Richardson, S., Ed. Human Factors for Informatics Usability. pp.21-37. Cambridge, UK,
Cambridge University Press.
[15]Thuseethan, S., Achchuthan, S., Kuhanesan, S.. (2014). Usability Evaluation of Learning Man-
agement Systems in Sri Lankan Universities. Global Journal of Computer Science and Technology.
15 (1), p15-24.
[16]Gamma, E., Helm, R., Johnson, R., Vlissides, J. (1994)Strategy. Design Patterns, Elements of
Reusable Object-Oriented Software. Westford, MA: Addison Weseley.
15
16. 7 Appendix
7.1 questionnaire questions
1. I think that I would like to use ARC frequently
2. I found ARC unnecessarily comlex
3. I thought ARC was easy to use
4. I found the various functions of ARC well integrated
5. I thought there was too much inconsistancy in the system
6. I would imagine that most people would learn to use ARC quickly
7. I felt very condent using ARC
8. I needed to learn a lot of things before I could get going with ARC
9. I liked using the ARC interface
10. Overall the system was easy to use
11. It was easy to learn to use the system
12. I believe I could become productive using this system
13. The system gave error messages
14. Whenever I made a mistake using the system, I could recover easily and quickly
15. ARC enables me to communicate with colleagues asynchronously
16. I am condent in using this technology
7.2 Test text for 'create post' test task
my name is [insert name], i am [insert age] years old, i live in [insert city].
7.3 Programme of Work
On following pages.
16
18. Name Begin
date
End date Outcome
Idea generation 9/15/14 9/27/14 App Ideas
Requirements
gathering
9/20/14 10/23/14 Requirements
Survey 9/20/14 10/23/14 required functionality, Current SNS usage
Interviews 9/20/14 9/26/14 Soft features, Functionality, preffered SNS usage
Brainstorming 9/26/14 10/1/14 ideas for functionality, priority of functionality
App Design 10/24/1
4
11/13/14 prototype design
Prototype 10/30/1
4
12/4/14 Prototypes
basic mockup 10/30/1
4
11/9/14 working POP prototype
html mockup 11/10/1
4
11/16/14 working HTML mockup
Working Prototype 11/17/1
4
12/4/14 Prototype android application
Usability Evaluation 1/3/15 2/4/15 Evaluation of the usability of prototypes
Literature Review 1/3/15 1/15/15 Assesment of usability engineering methods
applicable
Developer review 1/16/15 1/25/15 developers have knowldege of framework usability
Prototype reviews 1/16/15 1/25/15 testers have knowldege of app usability
Interviews 1/26/15 1/30/15 Contextual appraisal of prototypes
Observation of tasks 1/26/15 1/30/15 semi-quantitative/qualitative appraisal of prototypes
Questionnaire 1/26/15 2/4/15 ADJUST AND ADD
writing usability report 1/16/15 3/21/15 Usability Evaluation Report
CS2003 - A.R.C. Usability
Engineering Mar 20, 2015
Tasks 2
19. Name Default role
Jan P.C. Hanson project manager
CS2003 - A.R.C. Usability
Engineering Mar 20, 2015
Resources 3
20. CS2003 - A.R.C. Usability
Engineering Mar 20, 2015
Gantt Chart 4
Name Begin... End date Outcome
Idea generati...9/15/14 9/27/14 App Ideas
Requirement... 9/20/14 10/23/14 Requirements
Survey 9/20/14 10/23/14 required functionality, Current SNS usage
Interviews 9/20/14 9/26/14 Soft features, Functionality, preffered SNS...
Brainstor... 9/26/14 10/1/14 ideas for functionality, priority of function...
App Design 10/24... 11/13/14 prototype design
Prototype 10/30... 12/4/14 Prototypes
basic moc...10/30... 11/9/14 working POP prototype
html moc... 11/10... 11/16/14 working HTML mockup
Working P...11/17... 12/4/14 Prototype android application
Usability Eval...1/3/15 2/4/15 Evaluation of the usability of prototypes
Literature... 1/3/15 1/15/15 Assesment of usability engineering metho...
Developer...1/16/15 1/25/15 developers have knowldege of framework ...
Prototype... 1/16/15 1/25/15 testers have knowldege of app usability
Interviews 1/26/15 1/30/15 Contextual appraisal of prototypes
Observati... 1/26/15 1/30/15 semi-quantitative/qualitative appraisal of ...
Questionn...1/26/15 2/4/15 ADJUST AND ADD
writing usabil...1/16/15 3/21/15 Usability Evaluation Report
2014 2015
September October November December January February March April
21. CS2003 - A.R.C. Usability
Engineering Mar 20, 2015
Resources Chart 5
Name Default role
Jan P.C. Hanson project manager 400% 200% 300% 200% 200%
2014 2015
September October November December January February March April