1. Understanding the EMR Error Control Practices
among Gynecologic Physicians
Ritu Kharea, Yuan Anb, Sandra Wolfa, Paul Nyirjesy,a Longjian Liua, Edgar Choua
aDrexel University College of Medicine
bDrexel University College of Information Science and Technology
Philadelphia, PA, USA
iConference 2013,
February 13 2013
1

2. Motivation: EMR Errors
 EMR Errors
◦ incomplete, inaccurate, or inconsistent information
entered in Electronic Medical Records (Brown & Patterson, 2001;
Phillips & Gong, 2009)
 Occur because
◦ unusable EMR interfaces situated within demanding
clinical environment
◦ clinicians inadvertently make mistakes while
documenting patient visits and diagnosis information
 Are expensive …
◦ poor data quality
◦ unsafe quality of care
◦ physicians liable for medical malpractice (Classen, Pestotnik, Evans,
Lloyd, & Burke, 1997; Fichman, Kohli, & Krishnan, 2011).
2

3. Controlling EMR Errors
 Develop computational error control algorithms (Redwood, Rajakumar, Hodson, & Coleman,
2011).
◦ alert physicians in real time, and minimize further medical errors
 An “inside-out” approach
Step1: Understand the existing error control mechanisms
Step II: Design the algorithms according to the observed limitations, and
opportunities.
 Existing EMRs offer limited error control functionality
◦ Clinicians resort to MANUAL techniques to review, detect, and
resolve the errors (Phillips & Gong, 2009).
 We take the first step toward algorithm development, and
investigate the manual practices.
◦ assess abilities of physicians to detect EMR errors
◦ elucidate their strategies
◦ derive implications for algorithm design 3

4. The Error Simulation User Study
 Outpatient Clinics  Study Workflow
◦ clinicians document the patient 1. Fabricate gynecologic visit
visit information into the EMRs in scenarios and develop the
an on-the-spot “narrative”
corresponding EMR patient visit
manner.
notes.
◦ documentation occurs under
extreme time constraints 2. Purposefully introduce several
◦ conducive to a variety of data
EMR errors into the notes.
errors (George & Bernstein, 2009). 3. Present experienced gynecologic
physicians with the flawed notes,
 Gynecologic Field of Medicine and ask them to
◦ Physicians responsible for ◦ Identify any data errors
documenting information on ◦ Reveal error detection and
 yeast infections, bacterial vaginitis, resolution strategies
menstrual cycle issues, pre-natal
and post-natal complaints, regular
gynecologic examination, etc.
4

5. Providers find it far more efficient to enter impromptu
EMR Gynecologic Visit Notes narrative notes (as opposed to structured data) during
patient visits (Doğan et al., 2010).
ABC, ABC, 34 yr old F, DOB: DD-MM-YY
Type: Amb New Pt Visit A typical gynecoloigic
Owner: XYZ, XYZ Status: Final
visit note is organized
Reason for Visit
Ms ABC presents for a new patient visit
into 19 Sections
Chief Complaints (1) Reason for Visit
Yeast Infections (2) Chief Complaint
HPI
After having the PE and DVT had to go off birth control which she was on for endometriosis. Few months later, started to have yeast infections.
(3) History of Present
Initially, noticed external itching, soreness, redness. Her OB/GYN told she had yeast and put her on fluconazole to take as two dose treatment. Illness (HPI)
Sometimes it would work but not reliable. Treated herself monthly in this manner. Started to have more internal symptoms, as well as little (4) Allergies
discharge. Started nystain-triamcinolone which helped the external itching. They then tried inserts x7 days , it helped again. Now finds that her
partner is getting sore at times. Finally went on fluconazole once a week, started it a few months ago. Has not been much better on this regimen.
(5) Current Medicines,
Allergies (6) Active Problems
Benadryl: CAPS (7) Past Medical History
Lovenox: SOLN
Coumadin: TABS
(PMH)
Latex Exam Gloves MISC (8) Past Surgical History
PMH: (PSH)
Bipolar disorder
Current Meds
(9) Family History
Diflucan 150 mg oral tablet: Qty0, R0, RPT (10) Personal or Social
Terconazole 0.8% Vaginal Cream; Qty0; R0; RPT History
Probiotic CAPS; ; Qty0; R0; RPT
Assessment
(11)Gynecologic History
• Bacterial vaginosis (12)Obstetric History
Tests (13)Review of Systems
LABS ORDERED; yeast culture
Orders
(14)Vital Signs
Fluocinolone Acetonide 0.025% External Ointment; APPLY SPARINGLY TO AFFECTED AREA(S) TWICE DAILY; Qty1; R1; Rx. (15)Physical Examination
Plan (PE)
Reviewed at length w/pt. Discussed vulvar hygiene sheet. Discussed pathophysiology of lichen simplex. Start fluocinolone ointment. Stop
fluconazole as she has no evidence of VVC today. RTC – 1 month.
(16)Assessment
Signature (17)Tests
Electronically Signed by XYZ, MM-DD-YY (18)Plan
(19)Orders
5

6. The 5 Kinds of EMR Errors
 Inconsistent Information
◦ Contradictory information
across sections
 Incorrect Information
◦ Based on note scenario
◦ Based on clinical guidelines
 Incomplete Information
◦ Essential information
omitted
 Missing Section
◦ An entire section omitted
 Miscellaneous Errors
◦ Inappropriate placement
◦ Use of un-established
acronyms
6

7. User Study Design
 Participants  Data
◦ 11 females, 9 males ◦ 7 fabricated visit notes
◦ Gynecologic physicians with  different hypothetical
Drexel College of Medicine gynecologic patients
 Using Allscripts EMRs since 2008 or later
 Purposefully introduced errors
 Gold standard errors (total 97)
prepared by 2 clinical investigators
with 20 years of patient visit
Number of Participants documentation
1 year  Introduced more incomplete and
experience
missing section errors as they are
5 2 year more frequent in real world
8 experience
3 year
5 experience
2 Total Number of Introduced Errors
4 year Inconsistent
experience
6 Incorrect
48
Incomplete
25
54 Missing
Section
Miscellaneous
7

8. User Study Design
 Objectives
◦ assess participants’ ability to detect/resolve errors
◦ explicate intuitive strategies
◦ infer guidelines for algorithm design.
 Conducted one-to-one session with each participant.
◦ Analysis Stage
 present the paper prototypes of the patient notes
 participant carefully studies the note, detect any data error(s), and
document/annotate them on the same sheet of paper.
◦ De-briefing Stage
 the participant answers follow-up questions regarding the detected
errors
 what makes you conclude that certain data are erroneous?
 what in your medical training allowed you to detect this error?
 what measures would you take to resolve a certain error?
 why do you think these errors occur?
8

9. Results: Note Analysis Stage (avg. 32.6 min)
 Re-grouped Errors:  Correlation (Pearson's) between
◦ Mod-liability: missing section, missing ◦ Task performance(recall) and task
information, misc. errors duration
◦ Hi-liability: incorrect and incomplete ◦ Task performance and years of
information experience
 Error Precision: 100% ◦ Not significant at 0.05 level
◦ Each detected error could be mapped to a Time Experience
gold standard item spent
Hi-liability 0.29 0.43
 Error Recall: Recall for hi-liability (0.49)
Mod-liability 0.1 0.19
statistically higher(p<=0.05) than recall for
mod-liability (0.36)
 Best Recall Performance 70%
◦ Participant P5 had 4 years experience
and spent 53 minutes reviewing the
note
 Lowest Recall 17%
◦ Participant P10 had 1 year experience
and spent only 19 minutes reviewing
the note
9

10. Results: Debriefing Stage (avg. 13.3 min)
Participants were very confident of their performances during the
note analysis stage, and were very vocal about their experiences.
 How do you gain the  Why do the errors occur?
ability to detect errors? ◦ Because of poor
◦ Field experience of writing EMR documentation practices
notes in clinical settings  17 participants believe that
 4 participants physicians
◦ Training in medical school  Should write for others to be
 6 participants able to read
 Should ask more questions of
◦ Both (Experience + Academic the patient
training)  Should write in a list format
 5 participants
◦ Because of system design
 5 participants
 Allscripts EMR propagates all
problems through previous visits
creating obsolete information
 Clinicians tend to write free text
note because structured
interface is not friendly enough.
10

11. Results: Debriefing Stage
What are the triggers for detecting mod-liability errors?
 Detection of abnormal history events
◦ if history of abnormal pap smear in the gynecologic history section is observed, it
must always accompany more information such as the diagnosis date
◦ If history of hypertension in the family history section is specified then it should
also be specified which family member suffered hypertension.
 General Information Recall
◦ If the patient is having an annual visit, then HIV screening must be
present in at least one of the sections.
◦ If the patient is over 60 years of age, then a health monitoring plan
should be created and specified in the Plan section
11

12. Results: Debriefing Stage
What are the triggers for detecting hi-liability errors?
 Observation of discrepant information between two sections
◦ The reason for visit should be consistent with the active problem list.
◦ The information on the same drugs should match across different
sections, e.g., in one of the study notes
 strength of the drug “Fluconazole” is 200mg in the Plan section, and strength of the drug
“Diflucan” (market brand name for same drug) is 20mg in the Orders section
 Only 5 participants detected the above error in our study
 Detection of abnormal results
◦ Any abnormal body mass index should be alerted in the Plan and
Assessment sections.
 Identification of broken information links
◦ Each abnormal result from Physical Examination, should be linked to a
corresponding diagnosis in the Assessment section. Each diagnosis item
should have a corresponding item in the Plan section.
12

13. General Implications
for Error Control Algorithm Design
 Despite the experience and expertise, error recall performance <
50% (3 participants had >55%, 5 participants had <30%)
 It is imperative to replace existing manual strategies with effective computational
algorithms
 The participants delivered statistically better performance for hi-
liability errors than mod-liability errors
 Underlines the significance of learning from their expert abilities to minimize
potential physician liability
 The computed correlations between performance and
experience/time were not significant
 No clear conclusion regarding learning-based algorithms
 The results on the provenance of abilities suggest that future
algorithms should
 incorporate domain knowledge from a wide range of sources
 learn and infer from the contextual information in the EMR data
13

14. Guidelines to programmatically fire the Error Detection Triggers
Trigger Computational Technique Knowledge Sources
General Information Recall Basic if-then rules Clinical guidelines
Extraction of key information such as • Centers for Disease Control and Prevention
age, visit type, etc. • American Congress of Obstetricians and Gynecologists
Detection of Abnormal Basic if-then rules • Davis’s Laboratory and Diagnostic Tests
Results Extraction of examination results. • Agency for Healthcare Policy and Research
• Archimedes 360 Medical Calculator
Detection of abnormal Advanced if-then rules Conceptual model for drugs, disease conditions, and habits.
history event Extraction of abnormal events and their • UMLS RxNorm
attributes • DailyMed
Extraction of medications and their • MedlinePlus
attributes.
Observation of discrepant Comparison of problems, and Controlled vocabulary for describing problems and drugs,
information between two medications across sections linkages between drug ingredients, and brand names, drug-
sections Drug and Disease recognition drug interactions.
• DrugBank
• UMLS
• FDA National Drug Code Directory
• Classification of Diseases, Functioning, and Disability
• RxDrugs
• Davis’s Drug Guide
Identification of broken Extraction of results, diagnosis, plan, Drug indication, prescriptions, physical examination resources.
links across multiple order information • SIDER 2
sections Linking items from different sections, • Health Assessment Through the Life Span
and discovering the missing links. • Outlines in Clinical Medicine
• DailyMed
• NDF-RT
14

15. Study Summary and Contributions
As a pre-step to design EMR error control algorithms, we learn algorithm design lessons from
their abilities and behaviors of physicians on gynecologic visit notes.
 Participants could detect only 49% of the inaccuracy and inconsistency
errors, and only 36% of the omission errors from the notes.
◦ Need for more effective and efficient error control solution.
 An in-depth investigation of manual strategies helped develop guidelines
for algorithm design.
1. 5 data triggers that naturally prompt participants to sense a potential
error in gynecologic notes: detection of abnormal examination results, recall of generic clinical
guidelines, detection of abnormal history events, observation of discrepant information, identification of broken
information links.
2. Participants can identify the triggers using NLP abilities, and an immense
amount of intuitive domain knowledge accumulated through experience
and medical school training.
◦ In addition to sophisticated NLP techniques, the algorithms should incorporate a wide
range of federally established free resources for clinical guidelines, controlled
vocabularies, drugs, diseases, drug indications, gynecologic best practices, etc.
3. We briefly provide the linkages among triggers, NLP techniques, and the
relevant trustworthy knowledge sources.
15

16. Study Limitations
 Demanding schedules of our participants
◦ Could devote limited time to the study
◦ Set of derived implications is by no means complete
 Study had inherent bias
◦ Participants knew in advance that the note contain errors
◦ This is contrary to real-world practices
 Frequency of errors introduced in each note (avg. 13) is
not based on empirical evidence (due to lack of related work)
◦ Some participants might have assumed the notes to contain
fewer errors and terminated their analysis earlier
16

17. Conclusions and Future Work
In the USA, medical errors kill more people than highway accidents every year. (Kohn, Corrigan, & Donaldson, 1999).
“30 years from now the EMRs should make sense – otherwise defeats the purpose of EMRs”
– a study participant on documentation malpractices
 the first step to algorithm design by exploring an untapped knowledge
resource, i.e., the physicians.
 Explored their abilities to detect EMR data errors
 Derived algorithm design implications from their intuitive knowledge and
personal strategies.
 In comparison to the manual expert strategies, the existing automated
algorithms only scratch the surface of error control
 We plan to design customized algorithms for gynecologic notes by building
on the identified triggers
◦ To simulate the narrative information extraction
 existing NLP algorithms to extract drug, disease and specific clinical information from
texts (Doğan et al., 2010; Li et al., 2012; Névéol & Lu, 2010).
◦ To simulate the physicians’ knowledge in the head
 utilize, integrate, and organize several available trustworthy knowledge sources hosted by
the US Government.
17

18. Acknowledgements
- The 2 anonymous reviewers
- The 20 User Study Participants
- Drexel’s Jumpstart grant for health informatics
- Drexel’ Institutional Review Board
THANK YOU
18