Presentation of Prof. Heikki Topi at edTech Symposium, September 12

1. 9/13/13
1
Improving ERP Usability
Through User-System
Collaboration
Tamara Babaian, Wendy Lucas,
Heikki Topi, and Jennifer Xu
CIS Department, Bentley University
Presentation by Heikki Topi
University of Eastern Finland
September 12, 2013
Background
¢ Enterprise Resource Planning (ERP)
systems integrate business processes
across the entire enterprise
¢ ERP tasks automate business processes
that typically:
l Have well-defined hierarchical sub-task
structures
l Involve multiple users and system modules
l Can extend over days/weeks
¢ Many other types of large-scale enterprise
systems share characteristics with ERPs

2. 9/13/13
2
Background
¢ Anecdotal evidence of poor usability of
ERP systems
¢ Poorly designed user interfaces have
negative impact on bottom line due to:
l Increases in new user training time
l Decreases in end-user productivity
l Poor user adoption rates
(from Forrester Research studies of enterprise
and business application usability - Ragsdale
2004; Herbert 2006)
Motivation
¢ Research initiative driven by lack of
attention paid to addressing significant
usability shortcomings of ERP systems
by both manufacturers and the usability
community
¢ Focus on understanding ways in which
users interact with ERP software and
degree to which interaction model
supports tasks being performed
¢ Address gap between capabilities of ERP
systems and how they meet each user’s
objectives

3. 9/13/13
3
Motivation
¢ User’s spend weeks (months) in training, yet
often characterize ERP systems as ‘brutal,’
‘unforgiving,’ and ‘unintuitive,’ because
l System’s logic (process and data flow) opaque
to users
l Relationships and interdependencies between
tasks difficult to determine
l Semantics of fields, forms, and interfaces hard
to understand
l Little if any support for error resolution
¢ Users rely on in-house made documentation,
notes, and support by IT departments
Categorization of Usability
Problems from a Field Study*
¢ Difficulty in identifying and accessing the
correct functionality
¢ Lack of transaction execution support
¢ System output limitations
¢ Inadequate support in error situations
¢ Incompatibility between the users’ and
the system’s terminology
¢ Usage-related problems arising from the
overall complexity of the system
*Topi, Lucas, Babaian. Identifying Usability Issues with an ERP Implementation" in Proc. ICEIS’2005.

4. 9/13/13
4
Broader Problematic System
Characteristics
¢ Users lack an understanding of how their
narrowly defined tasks are related to the broader
process in which they are taking part
¢ Underlying business processes and their
relationships are hidden from users
¢ Data-to-process and process-to-process
relationships too complex for a user to
understand
¢ Local requirements lead to manual tailoring
efforts that are very expensive, particularly over
time
Typical ERP
Screens
“Easy” Access Menus
Data entry form

5. 9/13/13
5
User notes – example 1
User notes – example 2 –process
description
and 11 more
pages

6. 9/13/13
6
General Approach
¢ Apply Collaboration Theory to the
design and evaluation of ERP user
interfaces
l Collaboration between user and
the system, with system acting as
a collaborative partner helping the
user achieve system-related
goals
• Note that this is not CSCW
Human-Computer Collaboration
Theory*
¢ Commitment to Joint Activity (CJA)
l Each party in the interaction commits to the joint
activity and is aware of the context surrounding
their collaboration
¢ Mutual Responsiveness (MR)
l Each party responds to the intentions and
actions of the other by adjusting his own
behavior based on the behavior of the other’s
¢ Commitment to Mutual Support (CMS)
l Each party is committed to supporting the
efforts of the other when that party needs
help and will recognize the need and provide
assistance
*Bratman, 1992; Grosz & Kraus, 1996

7. 9/13/13
7
Conceptual Model of
System-User Collaboration
Current Research Program
¢ Three-year NSF grant (2008-2011)
with a one-year no-cost extension
(2011-12)
¢ Multiple research perspectives and
methods
¢ Four faculty members (two with a CS
background; two with an IS
background) and two IS doctoral
students actively involved

8. 9/13/13
8
Three Broad Principles
¢ Greater collaborative strength of
software yields greater usability
¢ Embedding knowledge about the
users, tasks, processes, domains,
prior usage, and characteristics of
interface components into an ERP
systems will strengthen its
collaborative capabilities
¢ Usage logs are a true reflection of
reality
Four Primary Methods
¢ Field studies (both interview and survey) to
identify usability issues with existing ERP
systems from the end users’ perspective
¢ Formulation of design principles based on
user input and collaboration theory to guide
interface design
¢ Applying design principles to the development
of illustrative prototypes and evaluating them
in laboratory studies
¢ Specifying new usability evaluation
methodologies and evaluating them in the lab
and in the field

9. 9/13/13
9
Research Goal
¢ Improve the usability of ERP systems by
increasing the collaborative capabilities of
their interfaces through:
l Development of design guidelines and
interface evaluation techniques based on
collaboration theory
l Implementation of prototype interfaces to
validate design and evaluation
methodologies
Related Research
¢ Human-Computer Collaboration
l Terveen (1995)
¢ Human factors issues and usability characteristics of ERP
systems
l Bishu et al. (1999)
l Calisir and Calisir (2004)
¢ User-centered/human-centered design
l Maguire (2001)
¢ Activity modeling
l Constantine (2006): Activity theory & usage-centered design
¢ TAM, UTAUT
l Davis (1989), Venkatesh & al. (2003)

10. 9/13/13
10
Sample Publications and
Working Papers
¢ Xu, J. & Topi, H. “A Conceptual Model for Collaborative ERP
Systems,” Bentley University Working Paper.
¢ Lucas, W., Xu, J., and Babaian, T. Visualizing ERP Usage Logs
in Real Time. In ICEIS ’13.
¢ Lucas, W. and Babaian, T. “Implementing Design Principles for
Collaborative ERP Systems,” In DESRIST ‘12
¢ Babaian, Lucas, Xu, and Topi, 2010,” Usability through System-
User Collaboration: Deriving Design Principles for Greater ERP
Usability,” In DESRIST ’10
¢ Cooprider, Topi, Xu, Dias, Babaian, and Lucas, 2010, “A
Collaboration Model for ERP User-System Interaction,” In
HICSS ‘10
¢ Lucas and Babaian, 2009, “Reasoning for Intelligent System-
User Interactions with Enterprise Resource Planning Systems,”
IJCAI 2009 Workshop on Intelligence and Interaction
Connections with UEF
Projects
¢ We believe that the key principles of
the human-computer collaboration
model can be successfully applied to
all types of enterprise systems
¢ We are interested in testing the ideas
developed in the ERP context in other
environments

11. 9/13/13
11
Connections with UEF
Projects
¢ “The main aim of DAVID is to enhance
corporate leadership by combining
leadership technologies, findings of
information systems research,
visualization and language
technologies.” (Dai, 2013, p. 77)
¢ These goals are, at the general level, not
far from the goals of enterprise systems
à could we apply HCC principles to
evaluate and potentially improve DAVID?
Connections with UEF
Projects
¢ Expansion of the usability concept
beyond user interface issues – our
research encourages all designers to
consider the collaborative relationship
between the users and the system
from very early stages of the design
l Moving from ease of use to usability
and user experience

12. 9/13/13
12
Example: Illustrations of
CJA, MR, and CMS
CJA-User
¢ How do you know that this is your debit piece and this
is your credit piece? You wouldn't know that until you
walked through it all and then you understood
conceptually. (need for commitment to learn)
¢ So, that's the one drawback I would say versus some
other systems that's not very intuitive. So, you need to
know the screens. And, you need to be trained in what
the process is. (commitment to get training)
¢ No. No. I think it's just (up to) you to understand what
you're trying to do and say 'Oh, yeah. Your (journal
actions are going to have to be) balanced, of course.
(commitment to certain subtasks)

13. 9/13/13
13
CJA-User – Survey Items
¢ I try to stay aware of the progress of
my tasks
¢ In general, I know how to complete
tasks in <Enterprise System>
¢ I know how to perform my tasks with
<Enterprise System>
¢ I know the business goals I am trying
to achieve using <Enterprise System>
CJA-System
¢ And so, you can drill in here and see why you short-
paid someone, or why you paid the amount you did. It
makes sense. It's like a story when you go through
them. (+)
¢ I think that the system is actually very helpful. This little
help button right here, which will launch into either the
definitions for where you are, if you're on a particular
field, but then it will also launch into help with the
overall applications, so I actually think it's very helpful.
(+)

14. 9/13/13
14
CJA-System
¢ User: Depending on what I'm going to try to do here. I
forget if I'm in account payable or accounts receivable
here. Interviewer: How can you tell if you need to know?
User: I actually don't know. I'd probably just go back and
then I'd look at the menu path that I was in. So here I'm in
accounts receivable. (-)
¢ When you first see this layout stuff, it doesn't make
sense. But then once you know what it is it does make
sense. (-)
¢ Honestly, for me personally, every time I've gone and
tried to drill in through the folders, I can never get to
where I think I need to be. (-)
CJA-System – Survey Items
¢ <Enterprise System> always informs
me of the progress of my tasks
¢ In general, I feel that <Enterprise
System> knows how the tasks should
be completed
¢ <Enterprise System> knows how
business tasks should be completed

15. 9/13/13
15
CJA-Total – Survey Items
¢ <Enterprise System> and I have a shared
understanding about how to complete
business tasks while we work on them
together
¢ <Enterprise System> and I share goals
for tasks that need to be completed when
we work together
¢ <Enterprise System> and I share
understanding about the progress of our
tasks while we work on them together
MR-User
¢ If you're used to doing an upload of the same thing, you
might have to manually type that in, if the upload
doesn't work. You always have to adapt to the system.
¢ Interviewer: Are there any examples of situations
where the system forces you to do things in a certain
way that is different from your preferred way of doing it,
accomplishing a certain goal? User: Geez, I don’t
know. I think after all these years I think the system has
just beaten me down.

16. 9/13/13
16
MR-User – Survey Items
¢ I have learned how to use <Enterprise
System> in order to adapt to its way of
working
¢ I take into consideration how
<Enterprise System> does things
¢ I understand how <Enterprise
System> typically gets things done
and adjust my work accordingly
MR-System
¢ The system just doesn't know what the problem is. It
doesn't know how to diagnose those. So this one says,
no suitable documents found. Message [XXXXX]. (-)
¢ I would say just getting rid of some of the fields that you
don’t need to look at [for this task] instead of having to
go through them all. It’s just a waste of time. (-)

17. 9/13/13
17
MR-Systems – Survey Items
¢ <Enterprise System> is able to
learn from how I work with it
¢ <Enterprise System> adjusts
to the way I perform tasks
¢ <Enterprise System> takes
into consideration my previous
actions
CMS-System
¢ Yes, it can give you error signs, saying
something is not right, or this field is incorrect, or
this has already been inputted. (+)
¢ No, it doesn't give you like error messages that
say hey, you need to do x, y, z. (-)
¢ The thing is that in this case [of getting lost], the
system is not reaching out to you saying that
you obviously need help. It’s me having to go
find it. (-)

18. 9/13/13
18
CMS-System – Survey Items
¢ In case of an error, <Enterprise
System> provides mechanisms for
fixing the problem
¢ <Enterprise System>'s error
messages provide detailed guidance
and instructions to help me solve
problems
¢ <Enterprise System>'s error
messages use terms that I can
understand
CMS-User – Survey Items
¢ I recognize the error situations in which
<Enterprise System> needs additional
input from me
¢ I do my best to understand <Enterprise
System>'s error messages in order to
solve the problem
¢ I do my best to diagnose and solve
problems in error situations

19. 9/13/13
19
Example: Proposed Design
Principles (Babaian, Lucas,
Xu, and Topi, DESRIST
2010)
Design Principle 1
¢ DP1. The user interface should provide a
mechanism for customizing the vocabulary of
terms used by the system in its communication
to the user, the composition of business
transactions, and the content of the system's
informational output to match the practices of
the organization. There should be a
mechanism for incorporating the customizations
from an earlier version of the system to a
later one.

20. 9/13/13
20
Design Principle 2
¢ DP2. The system should provide navigational
and progress guidance to a user performing a
transaction, indicating the broader context of
each interaction in terms of the related business
process components and specifying the
completed and remaining parts. A sufficiently
competent user should be able to turn off this
guidance if it becomes a distraction.
Design Principle 3
¢ DP3. When the system detects a problem, it
should identify the possible causes and ways
of resolving it. If the fix is obvious, the system
should inform the user and perform it. If it isn't
obvious, the possible causes and resolution
scenarios should be presented to the user and
be readily executable. If the system is unable
to identify resolution strategies, it should present
the user with the relevant data and transactions.

21. 9/13/13
21
Design Principle 4
¢ DP4. In presenting selection choices, the
system should utilize what it knows about
the user, the organization, the task, and the
context, and provide faster access to the more
likely choices than the less likely ones. Where
the choice of data or action is obvious, the
system should have an option of not waiting
for the user to enact it. The user should have
an option to replace/cancel the system's
provided choice of data/action.
Example: Moving towards a
Collaborative Walkthrough /
Collaborative Critique

22. 9/13/13
22
Collaborative Critique
¢ A new usability walkthrough method
¢ Questions that the evaluators answer
were formulated from the human-
computer collaboration paradigm
¢ The first version of the method has
been developed and gone through the
first round of testing
¢ CHI’12 Extended Abstract
Interface Evaluation:
Cognitive Walkthrough*
¢ Will the user try to achieve the right
effect?
¢ Will the user notice that the correct action
is available?
¢ Will the user associate the correct action
with the effect trying to be achieved?
¢ If the correct action is performed, will the
user see that progress is being made
toward solution of the task?
*[Wharton et al, 1994]

23. 9/13/13
23
Viewing Cognitive Walkthrough
Through a Collaborative Lens
¢ Based on the user’s overall goal, will the system
recognize the next step in the process and either act to
perform that step or, if the user’s input is necessary,
present a set of alternative actions from which the user
may make a selection?
¢ Does the system help the user identify the next action
and present it in a highly visible manner?
¢ Does the system present a meaningful set of alternative
actions based on the user’s overall goal?
¢ Will the system keep the user informed about the
consequences of actions taken by either the user or the
system, as they relate to progress made toward the
achievement of the task?
Key Questions
¢ Does the allocation of responsibilities between
the system and its user within the task align with
their respective natural strengths?
¢ Is the system-user communication mechanism
effective for both novice and experienced users?
¢ Does the system make effective use of its
resources in the process of working on the task
and communicating with the user?
¢ What are the system capabilities and
information that could be added to improve
interaction?

24. 9/13/13
24
Example: Use of Process
Graphs to Improve System-
User Communication
(Babaian, Lucas, and Topi,
2007)
Typical ERP task interface

25. 9/13/13
25
49
Improving System-User
Communication Via Process Graphs
Add Material
Edit Material
Add Vendor
Process Graphs Form
Hierarchies

26. 9/13/13
26
Benefits
¢ Improves system-user communication
of system-prescribed processes and
progress through task
¢ Simplifies navigation to related tasks
¢ Enables preview of future steps and
revisiting of previously completed
steps
Example: Improving System-
User Communication based
on Usage Logs (Babaian
and Lucas, 2011)

27. 9/13/13
27
Improving System-User
Communication via Usage Logs
¢ Embed model capturing relationships
between users, actions, tasks, and
processes into the system so that it can
reason about itself and its users
l Capture usage data via input-aware
components à supporting users with basic
system usage and in error situations
l Ongoing usability evaluations
l Automated planning for diagnosing and
resolving errors
Usage Logs + Task-Interface
Model
¢ Enables
l playback of past interactions
l usage log mining
l estimation of user’s experience level
¢ Makes relevant usage histories available
to all users:
l Show how Jack posted a goods
receipt.
¢ Enables automated usage-based support
mechanisms
l Show all account numbers used by
inspectors
l Which fields in the GR are always
filled?
Business
Processes, Tasks
Interface
components
Usage log Users
ERP
database

28. 9/13/13
28
Usage Log Examples
¢ Identify required fields based on prior usage
¢ Reconstruct completed processes
l Remind user how process was completed before
via a “playable” reconstruction process
l Show user how others completed the same
process
¢ Adapt level of instructional/error support
based on user experience
Data Model
Represents and links:
o Domain data
o User data
o Task data
o UI data
o User input log

29. 9/13/13
29
Automated Usability
Evaluation
¢ Keystroke analysis
l Components that are prone to user entry
errors
¢ Time to complete a task
l Subtasks that are more difficult to complete
¢ Time between tasks
l Navigation difficulties
¢ Number of times a task is revisited
l Tasks requiring additional support
Concluding Observations
¢ Underlying theoretical framework as a strong guide for a
research program that approaches the same underlying
research question from multiple perspectives
¢ A planned multi-method approach will build a much
stronger evidentiary foundation than any single method
would
¢ The empirical evidence gathered so far provides support
for our key propositions, but getting to a point where we
can conclusively reject alternative hypotheses will still take
time
¢ Design principles and guidance for design of future
enterprise systems will be in the center of the research
program outcomes
¢ The benefits of a multidisciplinary research team are
significant

30. 9/13/13
30
Work that Continues in the
Project
¢ More in-depth analysis of the interview data
¢ Further formalization of collaboration theory
applied to the ERP usability context
¢ Finalizing the survey instrument to enable
positivist theory testing
¢ Further development of collaborative critique for
evaluating interface usability
¢ Continuation of lab studies with experimental
collaborative interfaces
¢ Continued development and testing of the
prototypes
¢ Further articulation of the design principles
Acknowledgements
¢ This material is based in part upon work
supported by the National Science
Foundation under Grant No. 0819333. Any
opinions, findings, and conclusions or
recommendations expressed in this material
are those of the author(s) and do not
necessarily reflect the views of the National
Science Foundation.