This document describes the development of a visual language system (VLsys) for representing medical concepts. The VLsys uses iconic representations combined in a modular way to depict complex concepts. It was evaluated against a word-based display and found to support faster understanding and identification of related concepts. The VLsys focuses on coordination and supporting collaborative discussion across medical experts through its use in desktop environments. Key elements of the VLsys include: - Representing medical concepts like diseases, drugs, and tests through iconic symbols that are combined modularly - Organizing icons in a hierarchical structure with contextual relationships - Providing text explanations on rollover of icons - Enabling identification and exploration of related concepts The VLsys

1. Exploring the Design Space for Situated Glyphs to
Support Dynamic Work Environments
Fahim Kawsar1,2, Jo Vermeulen3, Kevin Smith2, Kris Luyten3, Gerd Kortuem2
1Bell Labs, 2Lancaster University, 3Hasselt University

2. Motivation

3. #3$*MN!6*D&6*5$E$)(8$5*&6*#3$*-'"&)*
6#'"/* '"6#%,:$"#6* D$%$* ,6$5* D3')$* >(G$6* D$%$* 5$2%$&6$5* '"* $&23* -(%:C* .$#&')$5*
#$:* 6,23* &6* #3$* 8%'"#$5* -(%:6C* 13$* 5$62%'8#'("* &"5* 6,//$6#'("6* -(%* $%%(%* 3&"5)'"/* D$%$*
$%$*‣5$6'/"$5* -(%* >(#3* /&#3$%'"/*
Medical Professionals deal with large amounts of dynamic data.
6,88)'$5*-(%*#3$*,6$%6*-(%*%$5,2'"/*#3$*8(#$"#'&)*$%%(%6C*
346'2'&"6* &"5* ",%6$6* D'#3(,#* &"4* 7"*$&64*#(*,6$*IJ$)8K*:$",*D&6*8%(E'5$5*#(*#3$*,6$%6*
!"#$"%&'(%)*+,"%'-'. /0)12/31*/4#'45'6"$/+12'(4#+"7*0'
. 89:;9<=' ><' '
2#'("6* &"5* :(#'E&#'("6* (-* #3$* ,6$%6* D'#3*&*6':8)$*&22$66C*@'/,%$*U*63(D6*#3$*6$)$2#$5*MN!6*
‣ Encoding patient information visually?*)$"#*0'*,"#'54%@"$'0@122'*"1@0'45';'A'B'$"0/C#"%0'*,1*'C1*,"%"$'122'45'*,"'/#$/D/$)12'/+4#0'
"6'5$%&#'("*("*#3$*646#$:C**
for quick understandability.
(-* #3$* :$5'2&)* '"-(%:&#'("* 646#$:* #3&#* D$* 5$E$)(8$5*
*4C"*,"%'*,"#&'10'1'*"1@&'+4#+"/D"$'1#$'$"0/C#"$'*,"'/#$/D/$)12'/+4#0'/#*4'1'0E0*"@'FE'+4#+"/D/#C'
$* D&6* $E&),&#$5* &"5* &"&)4F$5* >4* &6*-'"&)*8%(#(#48$C*13$*646#$:*)&"/,&/$*D&6*(%'/'"&))4*
1#$'$"0/C#/#C'1'C2E7,'0E0*"@'*4'4%C1#/3"'1#$'+1*"C4%/3"'*,"'/#$/D/$)12'/+4#0'/#*4'1'+4,"%"#*'0E0*"@'
1,%H'63*>,#*D&6*#%&"6)&#$5*#(*O"/)'63C* $"0+%/F"$'1F4D"G'H#"'0)+,'0E0*"@&'0,4I#'F"24I'JK/C)%"'BL&'I10'$"0/C#"$'FE'0*)$"#*0'?"1#'
‣ Combining
$G8$%#6C* NO6* 8%$8&%$5* &* )'6#* (-* icons to represent complex medical concepts with quick readability
M%"0"#0&'N1D/$'O%4#"%&'P421#'?*4D"%'1#$'Q)R"'S44$0G''
$%*8$%-(%:'"/*D'#3*#3$*&88)'2&#'("*'"*
'
! K/C)%"'B'
Medical Information System with Iconic User Interfaces
! The Visualisation of Medical Concepts
Visualization System
(Salman et al., 2010) ! (Zender et al., 2007)
?*)$"#*'$"0/C#"%0T'?"1#'M%"0"#0&'N1D/$'O%4#"%&'P421#'?*4D"%'1#$'Q)R"'S44$0'
! U'
!
$%012*#3"*IQ&#'$"#*J'6#(%4K*MN!*(-*#3$*-'"&)*8%(5,2#* U'
* '
V1#C/F2"'HFW"+*0'
X0',10'F""#'#4*"$'7%"D/4)02E&'/+4#0',1D"'F""#'7%4D"#'"55"+*/D"'1*'%"7%"0"#*/#C'*1#C/F2"'4FW"+*0&'*,"'
21#C)1C"'"Y)/D12"#*'45'#4)#0'J!"#$"%&':<<ZLG'[#'*,"'$"D"247@"#*'45'*,"'%"Y)/0/*"'/424C/+12'4%'
Visual Language for Medical Concepts /4+,"@/+12'HFW"+*'/+4#0&'$"0/C#"%0'2"D"%1C"$'*,"'0E0*"@1*/+'7%47"#0/*/"0'45'0+/"#*/0*0'FE'$"0/C#/#C'
51@/2/"0'45'*1#C/F2"'4FW"+*'/+4#0G'[+4#'51@/2/"0'@/@/+'*,"'71%"#*9+,/2$'0*%)+*)%"'45'*,"']6Q?'1#$'FE'
$4/#C'04'+%"1*"$'/+4#0'I/*,'1'^7%4_/@1*"'+4#*"_*`'*,1*'"#1F2"0'4#"'/+4#`0'@"1#/#C&'*,"'71%"#*&'*4'
143

4. 4. Verbal concepts are linked with the VLsys’s visual
example of the VLsys representation. The verbal concept identifier appears
Display with text on roll-over, like a software ‘tool tip.’
explanation. An Iconic Language for the Graphical
5. The VLsys presents all concepts in an interactive
Visual Language System for representing Medical display (see Cell 3-4).
Representation of Medical Concepts (Lamy et
Concepts (Zender et al., 2010) !"#$"%&'()*$+,-./0)1'(2$),&$3%('2'.,$")4',5!"##$%!!&'( )**+&,,---./01234536*789.512,':;"<(=:;,$,'(
al., 2008)
Display example from a
preliminary VLsys
Icons (in circles) represent concepts
in neuronal plasticity / Alzheimer’s
Disease, organized by ontological
structure with the most frequent
concepts larger, the less frequent
smaller. Connecting the icons are
line glyphs, which animate on roll-
over, representing semantic
relations: regulate, modulate,
produce, etc..
Each icon pair may represent one
Visual Language System for Representing Medical Concepts 6/11/10 4 or scores of papers or data sets.
Clicking an icon pair brings up a
A disorder characterized by recurrent episodes of paroxysmal brain dysfunction due to a sudden, disorderly, and window (not shown here) with all
excessive neuronal discharge. the related papers (or other data
MeSH tree number C10.228.140.490 sources), with links to sources.
In this definition we identified the three key terms as: !"#$"%&'()*$+,-./0)1'(2$),&$3%('2'.,$")4',5!"##$%!!&'(
Using a fisheye views approach,
successive levels of interaction
repeated
neuronal reveal more concepts related to
visualized ones.
(electrical) discharges [seizures]. In a pilot study, this display was
From this breakdown we created individual icons for head, nervous system, electrical discharge and repetition (see Figure 4). understood both faster and more
We can machine assembled these icons into an epilepsy icon. Modular machine assembly is important because it enables a accurately than a word-based
limited number of icons and glyphs to be combined in software to represent a large number of concepts (see Figure 4). display (Telemakus). For
identification tasks requiring
Figure 1
reasoning or associationExamples of VCM icons created by combining central pictograms, external shapes, colors and top-right pictograms
Figure 4. Modular icon construction. Reading left to right, a human head icon is combined with a circle glyph, which is combined with a brain + brain stem the
VCM icon
Examples of VCM icons created by combining central pictograms, external shapes, colors and top-right picto-
icon to make nervous system, which is combined with an electric glyph to make seizure, which is combined with a repetition glyph (arrows) to construct aggregate mean time for this
Epilepsy (repeating seizures). grams. Simple physiological or pathological states are represented by combining a color that indicates the moment at which
display was 18% faster than the
Drug mo
the patient state occurs, a shape that distinguishes pathological (square) and physiological (circle) states, and a central picto-
word-based display, andgram. These icons can be reused for building drug and test icons.
for
identification of related concepts, ties of th
the mean time for this display was
twice as fast as the word-based
Both methods can be combined in the same icon, e.g. the The external shape is brown for antecedents, red for cur-
ning. Th
display. square with the downward arrow combined with the heart
with an ECG signal indicates bradycardia.
rent states and orange for risks. Risks play an important
role in the expression of adverse effects of drugs (see icons square a
difficult to represent: the disease epilepsy for example. Moreover, in a language as rich, complex and numerous as medicine it
(a), (b) and (d) on Figure 1).
indicate
VCM icons for drugs
is not practical to develop a custom-designed icon for every term. Our strategy to overcome this is to develop a modular icon VCM represents the therapeutic classes of drugs using the dose). T
Moreover, we foundiconaprimitives that epilepsy is often broken software to represent complex concepts. Through evaluation we found that
system of that concept such as can be combined in down into several classifications along well-defined lines. icon of the disease they are indicated for (as given by the
For example, the MeSH description for epilepsy indicates that: words that, when analyzed, can be broken into individual representational
medical definitions often consist of several key ATC Anatomical Therapeutical Chemical drug classifica- ifiers to
tion [43]). A green top-right pictogram is added to the dis-
icons. For example, systems are generally based MeSH as:
Epilepsy classification epilepsy is described in upon:
(1) clinical features of the seizure episodes (e.g., motor seizure),
ease icon to obtain the drug icon (see Figure 4). The top-
right pictogram is a cross if the route of administration is
property
(2) etiology (e.g., post-traumatic), not specified, or a tablet, a syringe or a pomade tube to means d
‣ Used effectively for visualising a patient’s situation, providing instructions,
(3) anatomic site of seizure origin (e.g., frontal lobe seizure),
(4) tendency to spread to other structures in the brain, and
(5) temporal patterns (e.g., nocturnal epilepsy).
Figure 2
Examples of VCM disease or sign icons
Examples of VCM disease or sign icons. Icon (a) simply
indicate drugs with an oral, injectable or topical adminis-
tration route, respectively.
offering task support.
indicates the anatomical and functional location of the dis-
(MeSH tree C10.228.140.490 continued) ease. Icon (b) provides additional information about the gen-
VCM icons for tests
In drug monographs, tests are mostly mentioned for the
Building s
This structure offers five categories along which icon representations can be developed. Figure 5 shows an example of a eral pathological mechanism, using a shape modifier, and icon
partial iconic representation of various types of epilepsy based on these categories.
(c) provides additional information about the function
involved, by modifying the central pictogram. Icon (d) com-
follow-up of potential adverse effects of drugs. Thus, VCM
represents a test using the icon of the risk it can follow up. One or m
Figure 5. Pilot epilepsy vocabulary. The ‘root’ epilepsy icon is modified by icon primitives to represent five forms of epilepsy, from left to right: bines both the shape modifier and the pictogram modifica-
tion.
A blue top-right pictogram is added to the risk icon to
obtain the test icon. The top-right pictogram is a stetho- ical elem
‣
generalized, temporal lobe, absence, tonic-clonic, and myoclonic, plus a related concept: infantile spasms.
All are used in Desktop Environment, primarily used as a communication scope, a test tube, a ray or a signal, for clinical tests, bio-
ing cont
or adver
medium for informed discussion across medical experts. >8?3!(!1@!'"
67)5%$,809%/$,.1$-./$('1)1'.,$78/7.2%2:
to right
ture. The
(e.g. the
Visual Language for Medical Concepts
to three
Using this approach, iconic ‘words’ form basic vocabularies from which numerous specific icons can be assembled. These
repeated icons form icon families, which forms a context on which many specific vocabularies can be constructed: a disease Figure 3
Examples of VCM disease icons using various shape modifiers in the se
vocabulary, a protein/gene vocabulary, a study or trial vocabulary, etc..
Images clustered together in a shared space or context, such as a circle or square containing shape, are interpreted Examples of VCM disease icons using various shape avoid (e
together. (Arnheim, 1969, p. 55 ff.) By controlling image hierarchy and/or reading sequence the VLsys can cause some images
to be read as primary and others as secondary. By altering the visual form of the images in the reading sequence or hierarchy, modifiers. In icon (a) a pathological agent entering in the is empty
the reader can be stimulated to understand one icon as the primary concept and another image as a supporting or modifying square from the left represents the etiology, in (b) the cell in the drug

5. ‣ Focuses primarily on co-ordination, and supporting collaborative discussion,
planning, (and supporting nomadic activities to some extent) across medical
professionals and patients.
‣ does not address information presentation modality - visualisation aspects.
‣ Mostly confined to one space. Ecological Validity and Pervasiveness 421
Downloaded By: [Favela, Jesus] At: 03:53 20 April 2010
(Bardram et al. 2006, 2009, 2010) (Favela et al., 2010)
FIGURE 3 The CHIS system (a): The user uses the handheld to learn of the
availability and location of colleagues and services (b) and to send context-aware
messages (c).
information and services according to user context. For instance, when a physi-
cian carrying a PDA approaches a patient, the system can automatically display
that patient’s clinical record.
A second version of CHIS integrates public displays that personalize the infor-
mation shown to the user and offer opportunistic access to clinical information
(Favela et al., 2004). For instance, if a physician approaches a public display, it
detects her presence and personalizes the system, highlighting in the floor map
recent additions to clinical records of her patients, messages and services most
relevant to her current work activities (Figure 3a). Information from the mobile
device can also be seamlessly moved to the public display and vice versa. For
instance, a physician can drag a window in the display over her picture to transfer
the information contained in it to her PDA.
A third and final version of CHIS incorporates mechanisms that let users
manage their privacy based on contextual information (Tentori et al., 2006). For
instance, a physician can configure the floor map to show only the area where he
is located and his role instead of his precise location and name.
There is a significant complexity in evaluating such a system in situ. On one
(Scupelli et al., 2010) (Wilcox et al., 2010)
hand, the system needs to be integrated with the Hospital Information System
and an Electronic Patient Record. In addition, the reliability of the location estima-
tion component must be assured (Castro & Favela, 2008). Thus, we opted for an
approach in which some of these components can be simulated, in order to reduce
the cost and the complexity of the evaluation.
Situated Displays in Health Care

6. Nurse can provide
saline to patient one
or three.
Saline Nurse can support patient
Machine UD two with regular morning
routine.
Patient One PO PO
Patient Two
PO
Patient
Three
UD: Use Device
PO : Perform On
Discover
Understand
Select
(Zender et al., 2006, Bardram et al., 2006, 2008. Favela et al., 2010) Act

7. Research Goal

8. Activity-aware tiny displays distributed across the environment providing task information
Discover
Understand
Select
Act

9. Situated Glyphs

10. ‣ Adaptive and multivariate graphical units that provide in­situ task
information.
‣ Visual representations of human activities that are integrated in the
environment and linked to physical entities - like objects and people -
and are adaptive and mobile.
‣ Situated Glyphs can be realised by micro display networks.
Situated Glyphs

11. What
Information Affinity How
Representation Fidelity
Situated
Glyphs
Where
When Spatial Distribution
Temporal Distribution
Design Space
Situated Glyphs

12. ‣ Followed Nurses to understand their daily routine, information needs, and
current work practices (semi structured Interview)
‣ 20-22 beds distributed over 10 rooms in 4 areas.
‣ 1 Head Nurse, 8-10 Nurses, 4-5 at duty all the time
‣ Typical Stay Time for Patients : 3 weeks
Formative Study at the Department of Geriatric Psychiatry of the District Hospital
Mainkofen, Germany supporting elderly patients suffering from dementia.
Formative Study

13. Activity Pattern
A: do x x : action
A: do x with y y : device / object
z : patient
A: do x to z
A: do x with y to z
Insight -1
Action
Activity
Objects (Optional)
Activity Pattern

14. Activity Pattern
A: do x x : action
A: do x with y y : device / object
z : patient
A: do x to z
A: do x with y to z
Action
Activity
Objects (Optional)
Activity Pattern

15. What
Information Affinity How
Representation Fidelity
Situated
Glyphs
Where
When Spatial Distribution
Temporal Distribution
Information Affinity
What to Present in Glyphs (and to Whom)

16. Identity Confirmation
Relationship
Information Instructions
Explanation Affinity
Status
Trends
‣ Identity and Relationship - Which equipment to use with Whom.
Insight - 2
‣ Status - Current operation status (ok, faulty) for equipment, last checkup time for
patients, etc.
‣ Instructions - Guideline to perform a complex operation in sensitive areas, e.g.
Emergency Room, Intensive Care Unit etc.
‣ Confirmation - Feedback of task completion, and recorded.
‣ Explanation - Why the equipment is not working
‣ Trends : Equipment status history, patient medication record, etc.
Information Types

17. Identity Confirmation
Relationship
Information Instructions
Explanation Affinity
Status
Trends
‣ Identity and Relationship - Which equipment to use with Whom.
‣ Status - Current operation status (ok, faulty) for equipment, last checkup time for
patients, etc.
‣ Instructions - Guideline to perform a complex operation in sensitive areas, e.g.
Emergency Room, Intensive Care Unit etc.
‣ Confirmation - Feedback of task completion, and recorded.
‣ Explanation - Why the equipment is not working
‣ Trends : Equipment status history, patient medication record, etc.
Information Types

18. What
Information Affinity How
Representation Fidelity
Situated
Glyphs
Where
When Spatial Distribution
Temporal Distribution
Representation Fidelity
How to Present Glyphs

19. ‣ Symbolic | Iconic | Indexical (Semiotics Signs)
Symbolic Signs Iconic Signs (Static | Animated)
Symbolic Colour Coded Signs Iconic Signs (Static | Animated)
(a) Symbolic (b) Iconic
Direct Images Direct Images with Text Colour Coded Text and Number
(c) Indexical
Learning from Semiotics

20. Representation Fidelity
Symbolic Iconic Indexical
(abstract) (metaphoric) (direct)
Identity & Yes Yes Yes
Relationship (With Colour, Text
(With Colour) (With Multiple Icons)
Information Affinity
and Number)
Yes Yes Yes
Status
(With Colour)
Instruction
Insight - 3
No
Yes
Yes Yes
Confirmation Yes Yes
(With Colour Code)
Explanation No No Yes
Trends No Yes Yes
(With Animation)
Information Vs Fidelity [with Semiotics]

21. Representation Fidelity
Symbolic Iconic Indexical
(abstract) (metaphoric) (direct)
Identity & Yes Yes Yes
Relationship (With Colour, Text
(With Colour) (With Multiple Icons)
Information Affinity
and Number)
Yes Yes Yes
Status
(With Colour)
Instruction No Yes Yes
Yes
Confirmation Yes Yes
(With Colour Code)
Explanation No No Yes
Trends No Yes Yes
(With Animation)
Information Vs Fidelity [with Semiotics]

22. Colour Representing
‣ Situated Glyphs are represented (b)
with colour, text, number and measure Entity and Relationship
blood
3
arrows. Instruction | Explanation
pressure Confirmation | Trends
‣ Identity and relationship are
managed by a simple colour code Entity A Identity
system (e.g. Nurse A is only allowed
to interact with patients and 19 3 Entity B Identity
equipments marked with a given Status
colour code).
‣ Glyphs are initially abstract, on
approaching an entity more detail is (a) (c)
revealed.
Missing / Faulty
Status
Represents “red” coded nurse’s activity of measuring blood pressure with a “red” coded patient numbered
“3”, using a “red” coded blood monitoring device numbered “19” which is working fine.
Colour + Number + Text + Symbol
Example Glyphs

23. Alternative Glyph Design

24. Text Snippet + Symbolic + Temporal Stream
Design Alternatives

25. Iconic + Abstract + Temporal Stream
Design Alternatives

26. What
Information Affinity How
Representation Fidelity
Situated
Glyphs
Where
When Spatial Distribution
Temporal Distribution
Spatial Distribution
Where to Present Glyphs

27. Eppler et al. 2004, Muller et al. 2009, Pousman et al. 2006
Demanding
Fragmentation of Attention
Context
Switch
Fine Grained
Situated
Information Capacity
Fragmentation of Attention vs Information Capacity

28. Entity Centric
Glyph is placed at each entity, e.g.,
patient, equipment
Activity Centric
Glyph is placed at the location of the
Insight - 4
activity, e.g.,patient’s bed.
Co-Location Centric
Glyph is placed at the glance-able
space, e.g., nearby wall
Design Alternatives

29. Entity Centric
Glyph is placed at each entity, e.g.,
patient, equipment
Activity Centric
Glyph is placed at the location of the
activity, e.g.,patient’s bed.
Co-Location Centric
Glyph is placed at the glance-able
space, e.g., nearby wall
Design Alternatives

30. What
Information Affinity How
Representation Fidelity
Situated
Glyphs
Where
When Spatial Distribution
Temporal Distribution
Temporal Distribution
When to Present Glyphs

31. Temporal Distribution
Before
Before During After
Relationship Yes Yes Yes
Information Affinity
Status Yes - -
During
Insight - 5
Instruction Yes Yes -
Confirmation - - Yes
Explanation - Yes -
After
Trends Yes - -
Information Vs Timing

32. Temporal Distribution
Before
Before During After
Relationship Yes Yes Yes
Information Affinity
Status Yes - -
During
Instruction Yes Yes -
Confirmation - - Yes
Explanation - Yes -
After
Trends Yes - -
Information Vs Timing

33. System

34. iPod Touch with Custom Shield
51 mm x 30 mm
Jennic JN5139 Micro Controllers
OELD-160-G1 Displays (160x128 Pixels)
Contiki Operating System
TCP/IP Suite on top of ZigBee
with TeCO
Visit us at the Demo Session
Hardware Prototype

35. Kawsar & Kortuem, "Supporting Interaction with the Internet of Things across
Objects, Time and Space" Internet of Things 2010 Conference (IoT-2010)
RESTful Software Infrastructure

36. Follow-up Studies

37. Temporal Distribution
Control Mediation
Study 1 - @ Mainkofen Hospital with Nurses

38. Spatial Distribution
Study 2 - @ Controlled Lab Settings

39. ‣ Situated Glyphs as visual representations of human activities to
provide in-situ task support information.
‣ Activity patterns expressed in action-object pairs for a hospital ward
dedicated to dementia patients.
‣ A design space analysis for situated glyphs
‣ Information Affinity - Type of Information Need
‣ Representation Fidelity - Semiotics Signs Mapping
‣ Spatial Distribution - 3 Design Alternatives
‣ Temporal Distribution
‣ A prototype wearable display for situated glyphs - as building blocks
for miniature display network.
Take Away Points

40. Fahim
Kawsar
@raswak
h-p://www.fahim-­‐kawsar.net
Jo
Vermeulen
@jozilla
h-p://jozilla.net/
Gerd
Kortuem
@kortuem
h-p://www.kortuem.com