AMIA 2012 Panel on Visual Analytics for Healthcare Organizer: Adam Perer, PhD Research Scientist IBM T.J. Watson Research Center, Hawthorne, NY Panelists: Ben Shneiderman, PhD Professor, Computer Science University of Maryland, College Park, MD Yuval Shahar, PhD Professor, Head of the Medical Informatics Research Center Ben Gurion University, Beer Sheva, Israel Jeffrey Heer, PhD Assistant Professor, Computer Science Stanford University, Stanford, CA David Gotz, PhD Research Scientist IBM T.J. Watson Research Center, Hawthorne, NY Abstract With the proliferation of medical information technology, users at all levels of the healthcare system have access to more data than ever before6. This data can be of tremendous value but is often difficult to access and interpret. For example clinicians are often faced with the challenging task of analyzing large amounts of unstructured, multi-modal, and longitudinal data to effectively diagnose and monitor the progression of a patient’s disease4,5. Similarly, patients are confronted with the difficult task of understanding the trends and correlations within data related to their own health. At the institutional level, healthcare organizations are faced with the desire to use data to improve overall operational efficiency and performance, while continuing to maintain the quality of patient care and safety. Recent advances in visualization and visual analytics have the potential to help each of the user groups listed above do more with the often overwhelming amount of data available to them 1,3,7,8. However, to be successful, visualization designers and clinicians must work together closely to ensure that the right technologies are used to help address the meaningful problems. Unfortunately, despite the continuous use of scientific visualization and visual analytics in medical applications, the lack of communication between engineers and physicians has meant that only basic visualization and analytics techniques are currently employed in clinical practice2,9. The goal of this panel is to present state-of-the-art visualization applications for healthcare and engage the leading physicians and clinical researchers at AMIA to discuss the areas in healthcare where additional visualization techniques are most needed.

1. Panel on
Visual Analytics
for
Healthcare
AMIA 2012 - November 5, 2012
Moderator: Adam Perer, IBM Research

2. Visual analytics combines
automated analysis with
interactive visualizations
to understand, reason
and make decisions from
big data
Definition adapted from Daniel Keim, Jörn Kohlhammer,Geoffrey Ellis and Florian Mansmann’s
Mastering the Information Age Solving Problems with Visual Analytics

3. Panelists
• Ben
Shneiderman
University of Maryland
Pattern Finding in
Point & Interval
Event Sequences

4. Panelists
Yuval Shahar
Ben-Gurion University
Visual Analytics
for Discovery of
Time-Oriented
Clinical
Knowledge

5. Panelists
David Gotz
IBM Research
Visual Analytics
for Healthcare

6. Panelists
Diana Maclean
Stanford University
Finding What
to Look For
Exploratory Visual
Analytics for Online
Health Communities

7. Pattern Finding in
Point & Interval Event Sequences
!
Ben Shneiderman ben@cs.umd.edu @benbendc
Founding Director (1983-2000), Human-Computer Interaction Lab
Professor, Department of Computer Science
Member, Institute for Advanced Computer Studies
University of Maryland
College Park, MD 20742

8. Interdisciplinary research community
- Computer Science & Info Studies
- Psych, Socio, Poli Sci & MITH
(www.cs.umd.edu/hcil)

9. Patient Histories: Our Research
Tool Event Records Display
Types
LifeLines Points, One Individual
Intervals
LifeLines2 Points Many Individual,
Summary
Similan Points Many Individual
LifeFlow Points Many Individual,
Aggregate
EventFlow Points, Many Individual,
Intervals Aggregate
www.cs.umd.edu/hcil/toolname

10. LifeLines: Patient Histories
www.cs.umd.edu/hcil/lifelines

11. LifeLines2: Contrast+Creatine
www.cs.umd.edu/hcil/lifelines

12. LifeLines2: Align-Rank-Filter & Summarize
www.cs.umd.edu/hcil/lifelines

13. LifeLines2: Align-Rank-Filter & Summarize
www.cs.umd.edu/hcil/lifelines2

14. Similan: Search
www.cs.umd.edu/hcil/similan

15. LifeFlow: Aggregation Strategy
Temporal
Categorical Data
(4 records)
LifeLines2 format
Tree of Event
Sequences
LifeFlow Aggregation
www.cs.umd.edu/hcil/lifeflow

16. LifeFlow: Interface with User Controls

22. EventFlow: Original Dataset

23. LABA_ICSs Merged

24. SABAs Merged

25. Align by First LABA_ICS

26. Reduce Window Size

27. Overview + Details

28. Current Directions
- Motif Simplifications: Find & Replace
- Query Features
- Menus & Graphical Query
- Absence for Points & Intervals
- Flexible Temporal Patterns
- Scalability
- Event types, Events, Records, Query complexity
- LP Algorithm
- Long-term Case Studies

29. 30th Annual Symposium
May 22-23, 2013
www.cs.umd.edu/hcil
@benbendc

30. Visual Analytics
for Discovery of Time-Oriented
Clinical Knowledge
or:
You need to Know Something
if You Want to Know More
Yuval Shahar, M.D., Ph.D.
Medical Informatics Research Center
Department of Information Systems Engineering
Ben Gurion University,
Beer Sheva,
Israel
yshahar@bgu.ac.il

31. Declarative Knowledge
in the Medical Domain
• Many medical tasks, especially those involving chronic
patients, require extraction of clinically meaningful concepts
from multiple sources of raw, longitudinal, time-oriented data
• Example: Modify the standard dose of the drug, if during treatment, the
patient experiences a second episode of moderate anemia that has persisted
for at least two weeks
• Examples of clinical tasks that require temporal reasoning:
– Therapy
• Following a treatment plan based on a clinical guideline
– Monitoring and Diagnosis
• Searching for a gradual increase of fasting blood-glucose level
– Quality assessment
• Comparing observed treatments with those recommended by a guideline
– Research
• Discovery of hidden dependencies over time between clinical parameters

32. The Need for Intelligent Mediation:
The Gap Between Raw Data
and Meaningful Concepts
Clinical databases store raw, time-stamped data
BUT:
Clinicians and decision-support applications reason in
terms of abstract, clinically meaningful concepts,
typically over time periods, AKA Temporal abstractions
! Automated computation of concepts or of temporal
patterns derivable from raw data, using knowledge,
supports monitoring, interactive data mining, and
automated discovery of meaningful temporal patterns

33. Bridging the Gap:
Knowledge-Based
Monitoring, Exploration, and Discovery
• A distributed architecture that caters for three needs:
– Automated means for monitoring and recognition of
meaningful known patterns, in time-oriented data, by
applying temporal-abstraction knowledge from multiple
domain-specific knowledge sources to data from multiple
data sources
– Interactive, human-operated means for dynamic visual
exploration of a time-oriented data repository, using on-the-
fly integration with domain-specific knowledge, to identify
new meaningful patterns and add them to the knowledge base
– Automated analysis, enumeration, and discovery of new
meaningful, significant temporal-abstraction patterns
(relationships amongst temporal-abstraction intervals)

34. The Temporal-Abstraction Ontology
(Shahar, Artif. Intell. 1997)
• Used by the Knowledge-Based Temporal-Abstraction Method;
Includes:
• Events (interventions) (e.g., insulin therapy; surgery; irradiation)
- part-of, is-a relations
• Parameters (measured raw data and derived [abstract] concepts)
(e.g., hemoglobin values; anemia levels; liver toxicity grade)
- abstracted-into, is-a relations
• Patterns (e.g., crescendo angina; paradoxical hyperglycemia)
- component-of, is-a relations
• Abstraction goals (user views)(e.g., diabetes therapy)
- is-a relations
• Interpretation contexts (effect of regular insulin; pregnancy; infant)
- subcontext, is-a relations
• Interpretation contexts are induced by all other entities

35. Temporal-Abstraction Output Types
• State abstractions (LOW, HIGH)
• Gradient abstractions (INC, DEC)
• Rate Abstractions (SLOW, FAST)
• Pattern Abstractions (CRESCENDO)
- Linear [one-time] patterns
- Periodic [repeating] patterns
- Fuzzy patterns (partial match)

36. Temporal-Abstraction Knowledge Types
• Structural (e.g., part-of, is-a relations)
- mainly declarative/relational (BMI = ƒ(Wt,Ht))
• Classification (e.g., value ranges; patterns)
- mainly functional (BMI = Wt/Ht^2)
• Temporal-semantic (e.g., concatenable property)
- mainly logical (anemia is concatenable; pregnancy is not)
• Temporal-dynamic (e.g., interpolation functions)
- mainly probabilistic (anemia periods can be bridged)

37. The IDAN Temporal-Abstraction Mediator
[Boaz and Shahar, Artif. Intell. Med 2005]
Medical knowledge Knowledge- Medical
service acquisition expert
tool
Temporal- Decision
Standard medical
abstraction support
vocabularies service controller system
End user
(care
Medical data Access Temporal-
service provider)
abstraction
service

38. The GESHER Knowledge Structuring and Maintenance Tool:
Creating a Declarative Knowledge Map from Medical Concepts
[Hatsek et al., OMIJ 2010]
A knowledge
map
Constraints on
concept values
Structured text
description

39. The KNAVE-II Single-Subject Browsing and Exploration Interactive Interface
[Shahar et al., Artif. Intell. Med 2006]
Overall pattern
Medical knowledge Intermediate
browser interpretations
Concept search Raw clinical data

40. Evaluation of KNAVE-II
(Palo Alto Veterans Administration Health Care System)
(Martins , Shahar, et al., Artif. Intell. Med. 2008)
! 14 clinicians with varying medical/computer use backgrounds
! Each user was given a brief demonstration of the interface
! DB: more than 1000 bone-marrow transplantation patients (2-4yrs)
! Each user asked to answer 10 queries common in oncology protocols
! A cross-over study design compared the KNAVE-II module versus two existing
methods: paper charts and an electronic spreadsheet (ESS)
! (The 2nd phase, using more difficult queries, compared only versus the ESS)
• Direct Ranking comparison: KNAVE-II ranked first in preference by all users
• SUS Usability Scores: KNAVE-II 69, ESS 48, Paper 46 (P=0.006)
• Time: In the first evaluation: Users were significantly faster using KNAVE-II,
up to a mean of 93 seconds difference versus paper, and 27 seconds versus the
ESS, for the hardest query (p = 0.0006); In the second evaluation: The
comparison with the ESS showed a similar trend for moderately difficult queries
(P=0.007) and for hard queries (p=0.002); the two hardest queries were
answered a mean of 277 seconds faster when using KNAVE-II rather than ESS
• Correctness: for KNAVE-II 92% [110/120]; for ESS 57% [69/120], in the
second study; scores were significantly higher for all queries (p<0.0001)

41. Exploration of Subject Populations:
The VISITORS System
[Klimov and Shahar, Artif. Intell. Med. 2010; J. Intell. Info. Sys. 2010]
• VISualizatIon and exploration of Time-Oriented raw
data and abstracted concepts for multiple RecordS
– Knowledge-based time-oriented interpretations of the raw data
– Graphical construction of subject-selection query expressions
– Visual display and interactive exploration
– Use of absolute time as well as relative time (from some event)
– Multiple-record aggregation and association
• Evaluated for functionality and usability by clinicians
and knowledge engineers, with encouraging results

42. A VISITORS Select Subject Query (1)
• Demographic Constraints:
– Male subjects, who are Young (age≤20) or Old (age≥70)
OR relation
Query is automatically and
incrementally being created from
the user s graphical specification

43. A VISITORS Select Subjects Query (2)
• Knowledge based constraints Hemoglobin state was
abstracted as less than
Normal, for at least seven
days, starting at a time point
that is at least two weeks
after the allogenic BMT
WBC count was
increasing during the
same period

44. A VISITORS Select Time Intervals Query
• Find time intervals (in a monthly resolution) during which the
HGB value state was considered lower than “normal” for more
than 50% of the subjects

45. The VISITORS Multiple-Records
Main Interactive-Display Interface
Subject groups
Medical knowledge
browser Multiple-subjects
raw data
Distribution of derived
patterns over time
Concept search

46. Temporal Association Charts
Abstractions for the same [Klimov and Shahar, Meth. Info. Med. 2010]
subject group are
connected; support and
confidence are indicated
by width and hue
The data of each
subject are connected
by a line

47. A Temporal-Mediation Application Example:
The MobiGuide Project
• Coordinated by Mor Peleg, Haifa University, Israel
• Funded by the EU; an FP7 Integrated Project
• 13 partners from 5 countries
• Monitoring of chronic patients through bodily sensors and a smart phone
– Cardiac arrhythmia patients in Italy
– Diabetes and high blood pressure in high-risk pregnancy in Spain
• Provision of alerts to the patients through the mobile phone, and guideline-
based decision support to their care providers through the Web
• Abstraction of raw time-oriented monitored and historical patient data, to
support interpretation, alerting, decision support, quality assessment, and
mining performed by a temporal mediator

48. Temporal Data Mining:
Mining Temporal Interval Related Patterns
A Temporal Interval Related Pattern (TIRP) is a conjunction of
temporal relations among symbolic time intervals (i.e., abstractions)
{A1 o B, A1 o D, A1 m C1, A1 b C2, A1 b A2, B o D, B c C1, B b C2, B b
A2, C1 b C2, C1 b A2, C2 o A1 , D c C1,D o C2}

49. KarmaLego – Fast TIRP Mining
[Moskovitch & Shahar, IDAMAP 2009, AMIA 2009]!
*Ri = {Before, After, During, Overlaps…}

50. A KarmaLego Example:
Looking at a Diabetes Dataset
[Moskovitch & Shahar, AMIA 2009]!
• Contains 2038 diabetic patients data accumulating over five
years (2002-2007) , monitored by a large HMO
• Includes monthly measurements such as of HbA1c, Glucose,
and Cholesterol values, and medications purchased, including
diabetic (insulin-based) medications, statins, and beta-
blockers, normalized by the Defined Daily Dose (DDD)
• The laboratory-test values were abstracted using the KBTA
method, based on domain expert specifications
• The medication doses were abstracted, using the Equal- Width
Discretization method, into three states!

51. Exploration of Diabetes TIRPs:
An Example of discovered Patterns
[Moskovitch & Shahar, AMIA 2009]
0.26 0.18 0.22 0.28
0.25 0.23 0.33 0.42 0.29
Shown : Levels of [vertical] support; [No. cases/Horizontal support]
D.inc, D.dec, De.stab: drug dose gradient; H.dec, H.inc, H.stab: HbA1C gradient
F = Finishes; M = Meets; S = Starts (temporal relations)

52. The KarmaLego Visualization Tool (I)!

53. The KarmaLego Visualization Tool (II)!

54. Automated Classification:
Using TIRPs as Features
[Moskovitch & Shahar, IDAMAP 2009]
• The TIRPs discovered by KarmaLego can be used as features for classification
• Classification was rigorously evaluated in several medical domains
• Example: An ICU dataset of patients who underwent cardiac surgery at the
Academic Medical Center in Amsterdam during April 2002-May 2004
• Static data include details such as age, gender, surgery type
• Temporal data (HR, BP, FiO2…) measured each minute during first 12
hours
• Classification task: Determine whether the patient was mechanically
ventilated more than 24 hours during her postoperative ICU stay
• 664 patients; 196 patients were mechanically ventilated for more than
24hrs (29.5%)
• Multiple aspects were investigated: The temporal-relations fuzziness
factor value, the discretization method, the feature selection method…
• Overall accuracy: 79.6% for most combinations involving 5 discrete
states using a very simple equal-width discretization method

55. Summary:
Intelligent Abstraction, Exploration, and Discovery
of Time-Oriented Data and Their Abstractions
• It takes knowledge to obtain even more knowledge!
• Distributed integration of time-oriented clinical data and knowledge
• Faster identification of new patterns
• Goal-directed: By supporting intelligent, interactive visual exploration, by a
domain expert, of the contents of the accumulating time-oriented database
• Data-driven: by automated discovery of frequent temporal patterns
• Quick adaptation to new patterns, by enabling human experts to easily
modify the knowledge base
• Visualization provides concise, meaningful summaries of large amounts of
time-oriented data in terms familiar to the clinicians
• Temporal abstractions can also be used for generation of natural language summaries
• Suggests an iterative process in which new discovered and
validated knowledge is added to the knowledge base and is
exploited for the discovery of further medical knowledge

56. David&Gotz&
Healthcare&Analy4cs&Research&Group&
IBM&T.J.&Watson&Research&Center&
VISUAL'ANALYTICS'FOR'
HEALTHCARE'

57. Making'healthcare'smarter' n
My&Team&at&IBM&Research&
Healthcare'AnalyBcs'Research'Group'
IBM'T.J.'Watson'Research'Center'
Yorktown'Heights,'New'York'
'
hMp://www.research.ibm.com/healthcare/'
Data'
StaBsBcs'
Mining'
Visual' Clinical'
AnalyBcs' Medicine'

58. Making'healthcare'smarter' n
Personalized&EvidenceDBased&
Medicine&
Pa4ent& Clinician&
Search'and'
Analysis'
Tens'of'Thousands'to'10+'Million'Pa9ents'
• Several'Years'of'Data'Per'Pa/ent'
• Thousands'of'Features'
• Demographics'
• Diagnoses'
• Labs'
• Procedures'
• Claims'
• Unstructured'Physician'Notes'

59. Making'healthcare'smarter' n
Personalized&EvidenceDBased&
Medicine&
Pa4ent& Clinician&

60. Making'healthcare'smarter' n
Many&Opportuni4es&for&Visualiza4on&
Similarity
Analysis ? Clinically similar to
x1
? Q
x2
Q
…
3'
xN
Q
Query patient
Patient similarity
assessment in clinical
1' Visual cohort refinement
factor/feature space
x x1 x
1 2 K
1
x
1
, x2
2
,… , 1
x
K
2 2
…
…
…
x x x
?
1 2
N N N
K
Patient population
2' Visual outcome analysis

61. Making'healthcare'smarter' n
Cohort&Refinement&and&Manipula4on&
SolarMap'
FacetAtlas'

62. Making'healthcare'smarter' n
DICON&
• Iconic&mul4dimensional&visualiza4on&of&cohorts&
&
&
&
• Icons&are&interac4ve&
– Compare&
– Merge&
– Split&

63. Making'healthcare'smarter' n
DICON&Demonstra4on&

64. Making'healthcare'smarter' n
Visual&Outcome&Analysis&

65. Making'healthcare'smarter' n
OuTlow:'A&Temporal&Pathway&Visualiza4on&
Patient Outcome Time-stamped Events
Aggregate'
Alignment&Point&
[A]& [A,B]&
[A,B,C,D]&
[&]& [B]& [A,C]&
[A,B,C]&
Average outcome = 0.4 [A,B,C,E]&
Average time = 10 days
[C]& [B,C]& Number of patients = 10

66. Making'healthcare'smarter' n
OuTlow:&Visual&Encoding&
Past NOW Future
Horizontal&
posi4on&
shows& D
sequence&of&
states.& A
Height&is&
number&of&
C people&
E
B
Color&is&
outcome& Width&is&dura4on&of&
measure& transi4on&

67. Making'healthcare'smarter' n
OuRlow&Demonstra4on&

68. Making'healthcare'smarter' n
Interac4ve&Cohort&Refinement&
• So&far…&
AnalyBcs' VisualizaBon'
• However,&cohort&analysis&is&not&always&a&oneDway&
process&
Persisted Cohorts
Cohort
Cohort
Cohort Cohort
Cohort
Cohort
Views Cohort Analytics

69. Making'healthcare'smarter' n
Itera4ve&Visual&Cohort&Analysis&

70. Making'healthcare'smarter' n
Conclusion&
Similarity
Analysis ? Clinically similar to
x1
? Q
x2
Q
…
3'
xN
Q
Query patient
Patient similarity
assessment in clinical
1' Visual cohort refinement
factor/feature space
x x1 x
1 2 K
1
x
1
, x2
2
,… , 1
x
K
2 2
…
…
…
x x x
?
1 2
N N N
K
Patient population
2' Visual outcome analysis

71. Finding What to Look For
Exploratory Visual Analytics for
Online Health Communities
Diana MacLean, PhD Candidate
Advised by Jeffrey Heer
Computer Science Dept.
Stanford University
malcdi@stanford.edu

72. Exploratory Visual Analytics
• Start of research cycle (rinse, repeat)
• Goals
• Create a mental map of the data
• Drive hypotheses generation
• Useful for
• Big(ish) data
• Researchers with partial/full domain expertise

73. What do users talk about? Do forums contain novel, useful information?
Can online health forum participation help patients?
FORUM CONTENT

74. The Raw Data

75. Creating a Co-Occurrence Graph
Wgat allergy that might trigger asthma?
And she has had allergy and asthma problems since birth.
It could be asthma, or you could have a heart condition.

76. Creating a Co-Occurrence Graph
Wgat allergy that might trigger asthma?
And she has had allergy and asthma problems since birth.
It could be asthma, or you could have a heart condition.
asthma allergy

77. Creating a Co-Occurrence Graph
Wgat allergy that might trigger asthma?
And she has had allergy and asthma problems since birth.
It could be asthma, or you could have a heart condition.
asthma allergy

78. Creating a Co-Occurrence Graph
Wgat allergy that might trigger asthma?
And she has had allergy and asthma problems since birth.
It could be asthma, or you could have a heart condition.
asthma allergy

79. Creating a Co-Occurrence Graph
Wgat allergy that might trigger asthma?
And she has had allergy and asthma problems since birth.
It could be asthma, or you could have a heart condition.
asthma allergy

80. Creating a Co-Occurrence Graph
Wgat allergy that might trigger asthma?
And she has had allergy and asthma problems since birth.
It could be asthma, or you could have a heart condition.
heart
condition asthma allergy

81. Pain Forum

82. Pain Forum

83. Pain Forum

84. Pain Forum
Hypothesis: A common discussion pattern in the Pain
Management forum is, “I took [DRUG X] after having
surgery on [BODY PART Y].”

85. Pain Forum : Body Parts

86. Pain Forum : Body Parts

94. Hypothesis 1: A data-driven derivation of “pain types”
from health forum data would closely mirror an
expert-derived categorization.
Hypothesis 2: We can use data-driven categorization
to map out symptom types for conditions that are less
understood (e.g. Lyme Disease).

95. Asthma Forum

96. Asthma Forum

97. Asthma Forum
Hypothesis: Forums
related to specific
conditions have smaller
vocabularies of medically-
relevant terms.

101. Hypothesis: Drugs are
grouped by function/
application. We can mine
this data to determine
which drugs people are
using to treat certain
conditions.

103. When%you%go%outside,%try%
wearing%a%scarf%over%your%nose%
and%mouth%to%see%if%it%quells%the%
reac8on%

104. When you go outside, try
wearing a scarf over your nose
and mouth to see if it quells the
reaction
After attack I got an enroumous
amount of mucs (half of trash
bas of napkins and more)
especially after attack.

106. Hypothesis: This is spam.

107. Allergies Forum

108. Allergies Forum

109. Summary
• First: figure out what to look for
• Exploratory visual analytics can help us
marshal hypotheses
– Quickly
– Even with big (ish) data
– But without accuracy /completeness guarantees
• Visualizations can be playful
– Fun/accuracy trade-off
– Can we engage non-experts (users), too?