2. Questions about digital health technologies
Question Information needed Type of study
1. Can it be done ? That a prototype exists Feasibility study
2. Is it safe ? Hazard analysis; accuracy of Inspection eg. HAZOP; pilot
data, advice etc. tests of accuracy, usability
3. Is it acceptable? User attitudes, usage rates Interviews, survey, log file
analysis
4. Is it clinically Size of benefit vs. harms Pragmatic randomised trial
effective ?
5. Is it cost Incremental cost NICE technology appraisal
effective? effectiveness ratio
3. “We know it works”
“… Full advanced life-support did not decrease mortality or
morbidity... during advanced life-support, mortality was greater
among patients with Glasgow Coma Scale scores < 9”
Stiell IG et al. CMAJ. 2008; 178: 1141-52
Solution: do a trial - Liu & Wyatt, JAMIA 2011
4. Plausible DHC technologies that failed
Diagnostic decision support (Wyatt, MedInfo ‘89)
Integrated medicines management for a children’s
hospital (Koppel, JAMA 2005)
MSN messenger triage (Eminovic, JTT 2006)
Telehealth for COPD (Polisena, JMIR 2010)
Smart home applications ? (Martin, CDSR 2008):
“The effects of smart technologies to support
people in their homes are not known. Better quality
research is needed.”
7. The serious games evidence base
1 randomised trial:
– 91 learners about triage randomly allocated to serious game
or card sort control
– Outcomes: skill at triage of 8 simulated cases
– Results:
• Accuracy (0 or 1 errors on 8 cases): 91% game, 80% control (p=0.02)
• Time taken: game 456, card sort control 435, p=0.155
(Knight, de Freitas, Dunwell et al. Resuscitation 2010; 81: 1175-9)
0 Systematic reviews
8. Cochrane review on phone consultation / triage
Range of study types: 5 RCTs, 1 CCT, 3 ITS up to 2007
Results: 3/5 studies on GP visits showed drop, but 2
showed rise in later visits
Of 7 studies on A&E usage, 6 showed no change & 1
showed an increase
2 studies looked at deaths – no difference
“Phone consultation appears to reduce the no. of
surgery contacts & OOH visits by GPs, but questions
remain on service use, safety, cost & pt. satisfaction”
(Burin et al, CDSR 2010)
9. e-prescribing & adverse drug events
4 studies showed significant reduction in ADEs by 30-84%
1 study showed non-sig. increase in risk by 9%
[1] Ammenwerth E, et al. JAMIA 2008;15(5):585-600.
10. TeleHealth in diabetes, bronchitis & heart failure
Diabetes (Farmer et al SR, 2005):
– Slight reduction of HbA1C by 0.1% (95% CI -0.4% to 0.04%)
– Use of services no different or increased with telehealth
Bronchitis (Polisena et al SR, 2010):
– Mortality may be greater in telephone-support group (RR = 1.2;
95% CI 0.84 to 1.75)
– Reduced hospitalization and A&E visits, but impact on hospital
bed days varied
Heart failure (Inglis et al, CDSR 2010):
– Reduced mortality by 44% (RR 0.66, CI 0.54-0.81, p < 0.001)
– Reduced CHF-related admissions by 23% (RR 0.77)
– However, recent large RCT negative (Chaudry NEJMed, Dec 2010)
11. The evidence base for digital healthcare ?
Quality Safety Prevention Productivity
Electronic records + ++ ++ +/-
Phone consultations ++ +/- - ++
Email, SMS + +/- - +/-
Decision support +++ ++ ++ ++
Telemedicine + + - +
Remote monitoring +/- - - +
Serious games + + + +/-
Virtual reality + + +/- +
Importance of context:
• Setting: community, primary, secondary care
• Users: nurses, doctors, therapists, the patient
• Care groups – age, disease, severity…
• Other benefits: improved patient access, education
12. Case study: decision support systems
“A knowledge-rich system that processes two or more
items of patient data to generate encounter-specific
advice or interpretation” Wyatt & Spiegelhalter, 1991
Used to generate:
Advice about diagnosis
Prescribing alerts & reminders
Interpretation of lab test results…
13. New errors introduced by DSS
Inappropriate drug form selected for route
– e.g. capsules for IV administration
Inappropriate product selected
Incorrect dose, frequency, formulation from
dropdown menu
Inappropriate selection of default doses
Missed drug allergies / high severity interactions
– high override rate
Duplicate orders
– system could not distinguish regular, one off & PRN orders
Failure to stop drugs that are no longer required
Increased drug monitoring errors
– eg. fail to include diluents if CV line not selected
15. Reasons for negative ACORN trial
ACORN was too slow (1986 !) so advice
given too late
Nurses not empowered to act on advice
Cardiac care unit always full
ACORN also used in 15% of controls
Patients, not nurses, randomised –
learning effect
16. When do decision support systems work ?
Success rates across trials
Target clinical practice Clinical practice Patient outcomes
Diagnosis 40% 4/10 0% 0/5
Disease management 62% 23/37 18% 5/27
Single drug prescribing, 62% 15/24 11% 2/18
dosing
Prevention 76% 16/21 0% 0/1
Multi-drug prescribing 80% 4/5 0% 0/4
Overall 64% 62/97 13% 7/52
Garg et al, JAMA 2005, 293: 1223-38
17. Home-grown vs commercial systems
Commercial: 96%
reduction to 26%
increase (NS)
Home grown: strong
effect 99% to 16%
reduction (p<0.05)
Ammenwerth E, et al. JAMIA 2008;15(5):585-600.
18. Cost effectiveness of DSS to advise
on warfarin dosage
Clinically effective & promote safety: trials show 3-
13% increase in patient time within therapeutic
range
Very cost effective: incremental cost effectiveness
ratio £2200 per quality adjusted life year (NICE
technology appraisal 2005)
19. Conclusions
1. Technology can harm as well as help
2. Evidence is often context sensitive
3. Some evidence of effectiveness, little on cost
effectiveness / savings
4. The wrong kind of evaluation studies (eg. Chaudry)
can mislead
5. Working with EU Commission to improve design of
evaluation studies in Framework 7/8 projects
6. IDH will be publishing evidence reviews on selected
technologies over next few months
Notas del editor
Effects of e-Rx on potential ADEs:*Sig. RR from 35% to 98% (6/9, RR=0.02-0.65)*Sig. RR of 29% (1/9, RR=1.29, 95%CI: 1.04-1.60): [Mitchell D, et al. J Inform Tech Healthcare 2004;2(1): 19-29] small study*Inconclusive result (2/9): RR 0.63, 95%CI: 0.38-1.05 [Gandhi 2005] & RR 0.97, 95%CI: 0.16-5.82 [Bizovi 2002] small studies [Gandhi TK, et al. J Gen Intern Med 2005;20(9):837-841, Bizovi KE, et al. AcadEmerg Med 2002;9(11):1168-1175]Effects of e-Rx on ADEs:*Sig. RR from 30% to 84% (4/6, RR=0.16-0.70)*Non-sigRR of 13% (1/6, RR=0.87, 95%CI: 0.39-1.94) [Mullett CJ, et al. Pediatrics 2001;108(4):E75]*Small & non-sig. RR of 9% (1/6, RR=1.09, 95%CI: 0.92-1.29): [Bates DW, et al. JAMA 1998:280(15):1311-1316]
Home-grown systemsvs Commercial systems:* Higher relative risk reduction (RRR) by home-grown than commercial systems.*Home-grown CPOEs (11/23): Sig. RR from 13% to 99% (RR=0.01-0.87, 95%CI: 0.00-0.94)*Commercial CPOEs (12/23): Non-sig changes, ranges from 96% RR to 26% RR (RR=0.04-1.26, 95%CI: 0.00-1.55)** Reason: Home-grown systems can be modified & adapted more easily and quickly to local needs.