HM404

1. © 2017 American Health Information Management Association
© 2017 American Health Information Management Association
Health Informatics Research Methods:
Principles and Practice, Second Edition
Chapter 6: Evaluation Methods

2. © 2017 American Health Information Management Association
Learning Objectives
• Demonstrate appropriate use of terms related to
evaluation methods and theory.
• Explain the rationale for conducting an
evaluation or evaluation research.
• Apply evaluation or evaluation research
methods to a scenario.
• Outline the steps in an evaluation of health
information technology.

3. © 2017 American Health Information Management Association
Introduction: Evaluation and
Evaluation Research
• Evaluation
– A process to determine perceived value by
stakeholders
• Evaluation research
– A specific kind of research that includes social
science methods and a theoretic framework
or model as part of exploratory or hypothesis-
driven research

4. © 2017 American Health Information Management Association
Evaluation Research
Methods
• Use a theoretic framework
• Exploratory or hypothesis driven
• Determine the variables based on
theoretic constructs
• Develop research procedures to study the
theoretic constructs
• Collect and analyze the data

5. © 2017 American Health Information Management Association
Forms of Evaluation and Evaluation
Research
• Two categories of evaluation
– Formative evaluation occurs during the program cycle to gather
feedback from stakeholders or end-users to adjust the process
or the software so that the desired effects are achieved.
– Summative evaluation occurs at or near the end of the project
and is undertaken to assess effectiveness of the program.

6. © 2017 American Health Information Management Association
Formative Evaluation
• Needs assessment
– The process of determining, analyzing, and prioritizing
needs, and in turn, identifying and implementing solution
strategies to resolve high-priority needs
• Implementation evaluation
– Monitoring how well the planned events are actually
occurring
– Whether the implementation is meeting the expected
timeframes
• Process evaluation
– How well the project was carried out

7. © 2017 American Health Information Management Association
Examples of Formative
Evaluation
• Improvement in delivery methods with regard to
technology used
• The quality of implementation of a new process
or technology
• Information about the organizational placement
of a given process
• How well a planned process aligned with an
actual process

8. © 2017 American Health Information Management Association
Summative Evaluation
• Two types
–Impact evaluation
• Assesses the intended or unintended net
effects of the program or technology
–Outcome evaluation
• Determines whether the program or
technology has caused demonstrable
effects as defined in the project goals

9. © 2017 American Health Information Management Association
Summative Evaluation (cont.)
• Two components
–Cost-effectiveness analysis
• Compares the costs to the outcomes
of two or more courses of action
–Cost-benefit analysis
• Compares the total costs of a program or project
with its benefits

10. © 2017 American Health Information Management Association
Complex Adaptive Systems (CAS)
• An entity with many diverse and autonomous
components or parts (called agents) that are
interrelated and interdependent with many
interconnections
– The agents behave as a unified whole in learning
from experience and in adjusting to changes in
the environment.
– Each agent maintains itself in an environment
which it creates through its interactions with other
agents.
10

11. © 2017 American Health Information Management Association
Healthcare Organizations as
CAS
• Healthcare organizations have
– Diverse agents that learn such as
• Providers, patients, and other stakeholders
– Nonlinear interdependencies
• Small changes in the system may or may not result in a change at the
system level
• Reduce predictability
– Self-organization
• Patterns emerge based on the interaction of the components
– Emergence
• Patterns emerge from the system based on the individual agents as
well as the specific interactions undertaken within the system
– Coevolution
• Both the system and the agent evolve

12. © 2017 American Health Information Management Association
Sociotechnical Model for Health
Information Technology
• The eight dimensions provide a means to
evaluate aspects of health information
technology.
• Each can be assessed for what is working
well and what could be improved.
• Evaluation findings can be used to iteratively
revise software, processes, or components of
the HIT to improve the use of the technology
or the technology itself to improve safe use of
technology in healthcare.

13. © 2017 American Health Information Management Association
Design Principles for Health
Information Technology
• Five rights framework states that optimal
design and implementation will be achieved if
we provide:
– The right information
– To the right person
– In the right format
– Through the right channe.
– At the right time in workflow
• The five rights are essentially a framework for
formative evaluation for HIT.

14. © 2017 American Health Information Management Association
User Centered Design
• The product, service, technology, or process
should be designed for the user and their
task.
• There should be consistency of design
elements with other components of the
system.
• The language and terminology should be
simple and use terminology that the user is
familiar with.

15. © 2017 American Health Information Management Association
User Centered Design Model
Source: Creately 2016. Reprinted with permission.

16. © 2017 American Health Information Management Association
User Centered Design (cont.)
• The user interface (UI) should be simple and
transparent.
• The user should know that their actions are
successful and feedback to this effect should be
provided.
• Information should be provided about how to
navigate the system.
• Fundamental aspects of user centered design are
to functionality aligned with the task.
• Design the technology to reduce errors.

17. © 2017 American Health Information Management Association
The System Usability Scale
(SUS)
• A frequently-used tool that is simple yet
informative
• Consists of a 10-item questionnaire originally
created by John Brooke in 1986
• The SUS can be used to evaluate a wide variety of
products and services including:
– Hardware
– Software
– Mobile devices
– Websites
– Applications.

18. © 2017 American Health Information Management Association
Evaluation
• The evaluator develops the theory of change.
• The theory of change is a statement that
explains how activities will produce a series
of results and impact and is used to develop
a logic model.
• In the logic model the desired effect is
identified first and then the inputs, activities,
resources, and outputs required to achieve
these effects are identified.

19. © 2017 American Health Information Management Association
Logic Model Example
Source: University of Kansas Workgroup for Community Health and Development 2016. Reprinted with permission.

20. © 2017 American Health Information Management Association
Evaluation Questions
• Evaluation questions should be developed to
guide the evaluation.
• Questions should be identified and answered
based on the logic models and the critical
activities.
• By posing the questions, data will be
collected to answer these key questions and
important aspects of the project will be
assessed.

21. © 2017 American Health Information Management Association
Evaluation Data Collection and
Activities
• Once the evaluation questions are
determined, procedures will need to be
established to collect the data to answer the
questions.
• Quantitative and qualitative data can be
collected through a variety of methods.
• The process of data collection should be
feasible and the data accessible for
collection.

22. © 2017 American Health Information Management Association
Evaluation Research Examples
• Process evaluation
– O’Malley and colleagues (2005) examined
key areas related to the use of coded data. In
this study, the process of inpatient coding was
examined via:
• A review of the literature
• Flow charting the process
• Interviews
• Discussion with coders and the users of the coded data

23. © 2017 American Health Information Management Association
Evaluation Research Examples
(cont.)
– O’Malley and colleagues also provide a useful
guide to calculating accuracy measures for
coding. The most common statistics are:
• Sensitivity
• Specificity
• Positive predictive value
• Negative predictive value
• The kappa (k) coefficient

24. © 2017 American Health Information Management Association
Evaluation Research Examples
(cont.)
• Goal-based and impact evaluation
– Shah and colleagues (2005) designed a
computerized medication alert system to decrease
adverse events with medications but the alerts often
were overridden.
• The system was redesigned to improve clinician acceptance
• The first goal-based evaluation reflect frequent dismissal
• The system was redesigned with user-centered design
principals and re-evaluated for impact
• The impact evaluation reflected that high-severity medication
alerts were accepted.
24

25. © 2017 American Health Information Management Association
Other Evaluation Resources
• Office of the National Coordinator (ONC)
testing process for certification of HIT
• Agency for Healthcare Research and
Quality (AHRQ) Evaluation Toolkit