Augmenting Online Clinical Care Discussions with Published Literature

Knowledge Linkages
Augmenting Online Clinical Care Discussions
with Published Literature
Sam Stewart, Syed Sibte Raza Abidi
NICHE Research Group
Faculty of Computer Science
Dalhousie University, Halifax, Canada
November 30, 2010
Sam Stewart (Dal) Knowledge Linkages November 30, 2010 1 / 35

Outline
Introduction
Problem Description
Knowledge Linkage Framework
Preliminary Results
Conclusion

Introduction
Introduction
Pediatric pain management is a complex subject
children lack the cognitive ability to properly express their
pain, which can lead to incorrect interventions.
Lack of specialized knowledge or training in pediatric pain
management.
Because of the temporal and physical restrictions that
clinicians face, traditional educational systems are not a
plausible solution
Web 2.0 technologies provide alternate knowledge dissemination
mediums for clinicians to converge and share their knowledge.

Introduction
Rationale for Knowledge Linkages
Pediatric Pain Mailing List (PPML)
Brings together over 700 pediatric pain practitioners from
around the world to share their clinical experiences and seek
advice
The knowledge shared on the PPML is practice-based rather
than evidence based
It is important to augment the practice-based (tacit) knowledge
on the PPML with explicit knowledge
The goal of this project is to establish knowledge linkages
between discussions on the PPML and published
literature on Pubmed.

Introduction
Project Objectives
The objective of knowledge linkage is to reaﬃrm the
practice-related recommendations on the PPML with
evidence-based literature from Pubmed
The outcome of the project will allow users to
search through PPML archives to ﬁnd topics of interest
Retrieve research articles related to the topics from
Pubmed, through a “single-click” evidence retrieval
strategy.

Introduction
Knowledge Linkage Framework
PPML
Archives
Online
Disucssion
Forum
Message
Parsing
Filtered
Messages
Threading
Algorithm
Threads
Mapping
To MeSH
Thread
MeSHThread MeSHThread
Papers
Information
Retrieval

Project Framework
Step 1: Processing the Archives
The archives are stored in simple ASCII text ﬁles, organized by
month, starting June 1993 and ending December 2008
The messages are processed to extract the sender, date, subject
line and content of the messages
The messages are ﬁltered to remove non-substantive content
PPML
Archives
Online
Message
Parsing
Filtered
Messages
Threading
Algorithm
Threads
Mapping
To MeSH
Thread

Project Framework
Step 2: Threading
A thread is a series of messages centred around a common
subject.
They are the embodiment of experiential knowledge on the
PPML.
The messages are assigned to threads using their subject lines.
PPML
Archives
Online
Message
Parsing
Filtered
Messages
Threading
Algorithm
Threads
Mapping
To MeSH
Thread

Project Framework
Step 3: Mapping to MeSH
Threads
Mapping
To MeSH
Thread
MeSH
The threads are parsed and connected to
formal MeSH terms, using the Metamap
program
Metamap creates a mapping score, which is a
measure of the strength of the connection
Each mesh-based thread is used to query
PubMed

Project Framework
Metamap: Mapping free text to MeSH
Metamap is a program developed by Dr. Alan Aronson at the
NLM that maps biomedical text to the MeSH lexicon
Each mapping is assigned a score that is a measure of the
strength of the mapping.
1000×(Centrality+Variation+2×Coverage+2×Cohesiveness)/6
The scores provide a baseline measure of how well the mapped
MeSH term represents the original term in the thread

Project Framework
Example
Sample Statement
‘‘The report stated that when music therapy is used, the
babies required less pain medication. Does anyone know of any
published reports of empirical research demonstrating the
effect?’’
Source MeSH Term Score
music therapy Music Therapy 1000
the babies Infant 966
less pain medication Pain 660
less pain medication Pharmaceutical Preparations 827
published reports Publishing 694
empirical research Empirical Research 1000

Project Framework
Step 4: Literature Search StrategyPPML
Archives
Online
Disucssion
Forum
Message
Parsing
Threading
Algorithm
Mapping
To MeSH
Thread
Papers
Information
Retrieval
Passively links the threads to published medical literature
Naive approach: Retrieve all papers that contain every MeSH
term in the thread. If no papers exist the algorithm would drop
the lowest scoring terms and reiterate

Project Framework
Naive Approach
The naive approach has several problems
It doesn’t provide any kind of ordering on the resulting
papers
It doesn’t fully utilize the MeSH scores
It doesn’t take into account the possibility of incorrect
mappings
One of the challenges of mapping free text with Metamap is its
inaccuracy.
The presence of a false MeSH term with a high MeSH score will
prevent the retrieval of useful papers

Project Framework
Improved Search Strategy
Our improved search strategy makes full use of the Metamap
scores
It also addresses the problem of incorrect mappings
It is based on the Extended Boolean Information Retrieval
(eBIR) algorithm
Customizes the algorithm to deal with pediatric pain by
adding a specialized ﬁlter
Let (Mi , mi ) be MeSH term i and the associated Metamap score.
Q = [Infant OR Child OR Adolescent] AND
[(M1, m1) ORP
(M2, m2) ORP
. . . (Mn, mn)] (1)

Project Framework
Step 5: Discussion Forum
Archives
Online
Disucssion
Forum
Mapping
To MeSH
Thread
Papers
Information
Retrieval
An online forum is being developed that allows practitioners to
interact with the PPML discussions and review the research
articles for a speciﬁc discussion thread.
The forum will be navigated by a standard search function, or by
a search function based on MeSH terms
As well the threads will be organized into a hierarchy based on
their MeSH terms

Project Framework
Discussion Forum

Project Framework
Example Thread

Project Framework
Linked Papers

Results
Example
This ﬁrst example is the ﬁrst thread ever transmitted on the
PPML
It is on the subject of Music Therapy
The following slides show the discussion, then the list of MeSH
terms mapped, then a sampling of the papers returned

Results
Music Therapy I
Sender: 1
Subject: Music Therapy
Date: Mon Jun 28 21:19:36 ADT 1993
Thread: 3, false
The last several days, the local NBC station aired a ”medical report”
about the use of music therapy. The report was from Miami and
included a short report on the use of music therapy in a NICU. The
report stated that when music therapy was used, the babies required
less pain medication. Does anyone know of any published reports of
empirical research demonstrating this eﬀect?

Results
Music Therapy II
Message
Sender:2
Date: Tue Jun 29 08:25:12 ADT 1993
Thread: 3, true
I would suggest that you might contact **** ******** in Pediatrics
at Washington University Medical School. Her research is on
neonatal pain and she might know where the local station picked up
the report. I haven’t seen any data on the topic.

Results
Music Therapy III
Message
Sender: 3
Date: Tue Jun 29 10:20:41 ADT 1993
Thread: 3, true
I’m not aware of speciﬁc studies conducted using music therapy to
reduce the need for pain medication (i.e., music therapy to manage
pain). However, several cognitive interventions have been used quite
eﬀectively to manage pain. Donald Meichenbaum developed a
technique in the early 1970s called stress inoculation training which
combines aspects of self-instruction training and relaxation training.

Results
Music Therapy MeSH Terms
MeSH Score Papers
Music Therapy -4802 1621
Pain -4215 237890
Research -2000 254813
Pharmaceutical Preparations -1688 438290
Infant, Newborn -1660 422011
Education -1654 492705
Teaching -1320 52938
Vaccination -1320 44092
Intensive Care Units, Neonatal -1000 6847
Pediatrics -1000 34612
Empirical Research -1000 8897
Relaxation Therapy -1000 5677
Vision, Ocular -966 18516
Behavior -966 879624
Infant -966 795209
Air -966 16342
Awareness -861 9711
Schools, Medical -861 17436
Biomedical Research -827 28157
Publishing -694 27222
Cognition -589 76048

Results
Music Therapy Papers Returned
Bo LK, Callaghan P. Soothing pain-elicited distress in Chinese
neonates. Pediatrics:2000,105(4). 10742370.
Cignacco E, Hamers JP, Stoﬀel L, van Lingen RA, Gessler P,
McDougall J, Nelle M. The eﬃcacy of non-pharmacological
interventions in the management of procedural pain in preterm
and term neonates. A systematic literature review. European
journal of pain (London, England):2006,11(2). 16580851.
Kemper KJ, Danhauer SC. Music as therapy. Southern medical
journal:2005,98(3). 15813154.
Tagore T. Why music matters in childbirth. Midwifery today
with international midwife:2009,(89). 19397157.

Results
Pilot Study
A pilot study was conducted on all messages from 2007 and 2008
100 threads were reviewed to determine
1 the accuracy of the message parsing
2 the accuracy of the thread assignment
3 The accuracy of the papers returned
The message parsing was successful on 74% of the messages
The threading was successful on 92% of the messages

Results
Recall, Precision, Utility
Precision and relative recall were compared between the modiﬁed
search strategy, the eBIR model, and a traditional VSM.
Relative recall is for comparing search strategies of
unannotated databases
Precision =
Number of relevant papers returned by the search
Total number of papers returned
Recall =
Number of relevant papers returned by the search
Number of relevant papers returned by all searches

Results
Precision
2 4 6 8 10 12 14
0.000.050.100.150.20
Top k papers
Precision
q
q
q
q
q
q
q
q q q
q q q
q
q
q
Custom
VSM
eBIR

Results
Recall
2 4 6 8 10 12 14
0.00.20.40.60.8
Top k papers
RelativeRecall
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q
Custom
VSM
eBIR

Results
Precision-Recall
0.2 0.4 0.6 0.8
0.080.100.120.140.160.18
Relative Recall
Precision
q
q
q
q
q
q
q
q q q
q
q q
q
qq
Custom
VSM
eBIR

Results
Search Strategy Results
The precision of the modified algorithm is significantly higher
than the other two algorithms at k = 15 (p-values of 0.013 and
0.003 respectively)
The recall, however, is only significantly different between the
modified and ebir models (p < 0.0001) and not with the VSM
algorithm (p = 0.351)
Ultimately, a search is “good” if it returns at least one pertinent
result
Utility at level k is an indicator of whether the search returns a
relevant paper in the first k results.

Results
Utility vs. k
2 4 6 8 10 12 14
0.00.10.20.30.40.50.6
Top k papers
Utility
q
q
q
q
q
q q
q
q
q
q
q
q q
q
q
Custom
VSM
eBIR

Conclusion
Conclusion
The mapping of experiential to explicit clinical knowledge is
critical, given the rapid changes in medical knowledge and its
application in specialized domains
Clinical experiences should be supported by clinical evidence, and
this has been achieved through our Knowledge Linkage
Framework
Presented a method of leveraging web 2.0 techniques by
incorporation medical information retrieval strategies to improve
the overall medical knowledge base
Automatic query generation, using clinical terms and contexts, is
a unique aspect of the research

Conclusion
Future Work
The next step is to provide open access to a wide number of
users and get their feedback
More time should be spent looking into the variables within the
eBIR algorithm and the modiﬁed algorithm
Q = [Infant ORp1
Child ORp1
Adolescent] ANDp2
[M1 ORp3
M2 ORp3
. . . ORp3
Mn]
Tweaking the Metamap scores, either within the Metamap
system or through post-processing, should also be explored

Conclusion
Acknowledgement
This work is carried out with the aid of a grant from the International
Development Research Centre (IDRC), Ottawa, Canada

Conclusion
Thank you

Appendix
Appendix

Appendix Metamap
Metamap Algorithms
There are three general types of matches:
Simple match a direct connection between the recognized noun
and the UMLS term
Complex match when a noun phrase can be mapped directly to
a combination of UMLS semantic types
Partial match when part of the noun/noun-phrase does not map
to UMLS
The general mapping strategy is, for each term SPECIALIST
recognizes: generate all variants of the noun-phrase, form the
candidate set of all the UMLS strings that contain 1 of the
variants, sort the candidate set by the strength of mapping,
combine candidates for disjoint parts of the noun-phrase, then
select the mapping with the best score.

Appendix Metamap
Variants
The variants are all composite parts of a noun-phrase, along
with all acronyms, abbreviations and synonyms of those terms,
all variants of those variants, etc . . .
For the term ocular the following ﬁgure depicts the generation of
the variants

Appendix Metamap
Metamap Scores
The scores range from [-1000, 0], with lower scores being better
The score is based on 4 metrics: centrality, variation, coverage
and cohesiveness. The ﬁnal score is calculated as:
−1000×(Centrality +Variation+2×Coverage +2×Cohesiveness)/6

Appendix Metamap
Metamap Scores I
Centrality a 1/0 indicating whether the match is to the head of the
phrase
Variation A measure of the distance the matched term is from the
root word. The distance, D, is a sum of the following
variations. The score is calculated as 4
D+4
.:
spelling: 0
inﬂectional: 1
synonym/acronym/abbreviation: 2
derivational: 3.

Appendix Metamap
Metamap Scores II
Coverage How much of both the UMLS string and the phrase are
involved in the match. The number of words in each
phrase are computed, as well as the spans of each term,
i.e., the length of the matching terms, ignoring
non-matching terms. The score is calculated as
2
3
Span
UMLS Length
+
1
3
Span
term length
Cohesiveness Like coverage, but focusing on connected terms. It
calculates the length of connected components (the
maximal sequence of connected words in both terms),
and takes a weighted mean again, this time of the sum
of squares.
2
3
SS UMLS con comps
UMLS length2 +
1
3
SS phrase con comps
phrase length2

Appendix Metamap
Sample Metamap Scores
From PPARCH.199603
Noun UMLS Cent. Var. Cov. Coh.
aired Air 1 D=1;4/5 1 1
of music therapy Music Therapy 1 0;1 1 1
a NICU ICU, Neonatal 1 0;1 1 1
the babies Infant 1 D=1; 4/5 1 1
less pain medication Pain 0 0;1 2
3
1
1
+ 1
3
1
3
2
3
1
1
+ 1
3
1
9
Pharm Prep 1 0;1 2
3
2
2
+ 1
3
1
3
2
3
2
2
+ 1
3
1
9
of any published reports Publishing 0 0;1 2
3
1
1
+ 1
3
1
2
2
3
1
1
+ 1
3
1
4
empirical research Empirical Research 1 0;1 1 1
Noun UMLS TOTAL
aired Air −1000 × (1 + 4/5 + 2(1) + 2(1))/6 = −966
of music therapy Music Therapy −1000
a NICU ICU, Neonatal −1000
the babies Infant −1000 × (1 + 4/5 + 2(1) + 2(1))/6 = −966
less pain medication Pain −1000 × (0 + 1 + 2( 7
9
) + 2( 19
27
))/6 = −660
Pharm Prep −1000 × (1 + 1 + 2( 7
9
) + 2( 19
27
)/6 = − − 827
−660−827
2
= −743.5
of any published reports Publishing −1000(0 + 1 + 2( 10
12
) + 2( 36
48
))/6 = −694
empirical research Empirical Research −1000

Appendix Metamap
Extended Boolean Information Retrieval (eBIR)
The eBIR system incorporates query weights into the traditional
BIR model
Let the set of query terms be A = {(A1, s1), . . . , (An, sn)}, where
Ai is the ith
query term, and si is the associated score
Let the OR and AND queries be
QOR(p) = {(A1, s1) ORp
. . . ORp
(An, sn)}
QAND(p) = {(A1, s1) ANDp
. . . ANDp
(An, sn)}
The selection of p eﬀects the inﬂuence of high-scoring terms on
the returned query scores.

Appendix Metamap
Modified IR Algorithm
The problem with applying the eBIR algorithm to this project is
that it doesn’t address the issue of specialized domains
MeSH keywords such as Pediatrics or Pain could be implicitly
representative of all conversations on the list
Our algorithm modified the eBIR algorithm by adding a
specialized filter
adding an AND operator to the query

Appendix Metamap
The new query would modify the search query by adding Infant,
Child or Adolescent to the set of MeSH terms, as demonstrated
in equation (2)
Let (Mi , mi ) be MeSH term i and the associated Metamap
score.
Q = [Infant ORp
Child ORP
Adolescent] ANDP
[(M1, m1) ORP
(M2, m2) ORP
. . . (Mn, mn)] (2)
Using the eBIR algorithm the next step would be to apply query
weights to the terms in the specialized ﬁlter and then ﬁnd a
suitable value for p.

Appendix Metamap
In order to accommodate the ﬁlter the eBIR algorithm was
modiﬁed, making the AND operator a strict Boolean operator,
and leaving the query weights on the OR operator
The decision was also made to set p = 1.
Q = [Infant OR Child OR Adolescent] AND
[(M1, m1) ORP
(M2, m2) ORP
. . . (Mn, mn)] (3)
The result is a search strategy customized to the pediatric pain
domain, that makes full use of the Metamap scores to return a
pertinent set of papers

Augmenting Online Clinical Care Discussions with Published Literature

Recomendados

Recomendados

Más contenido relacionado

Similar a Augmenting Online Clinical Care Discussions with Published Literature

Similar a Augmenting Online Clinical Care Discussions with Published Literature (20)

Último

Último (20)

Augmenting Online Clinical Care Discussions with Published Literature