Presentation_Thesis defense _Thi Ha VO. 16.12.2015
•Descargar como PPTX, PDF•
2 recomendaciones•592 vistas
The document describes the development and validation of a new multidimensional tool called CLEO for assessing the potential impact of pharmacist interventions (PIs). It involved: 1. Conducting a systematic review of existing models and tools which identified limitations and informed the development of the optimal SP(ECH)O-P model. 2. Developing the CLEO tool based on the model, with 3 dimensions (clinical, economic, organizational) and defined scoring criteria. 3. Validating the CLEO tool by testing inter-rater and intra-rater reliability on samples of PIs, which showed moderate to substantial agreement, demonstrating the tool is reliable for assessing impacts of PIs
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Presentation_Thesis defense _Thi Ha VO. 16.12.2015
Similar a Presentation_Thesis defense _Thi Ha VO. 16.12.2015 (20)
Más de HA VO THI
Más de HA VO THI (20)
Último
Último (20)
Presentation_Thesis defense _Thi Ha VO. 16.12.2015
- 1. Assessment of the potential impact of pharmacist interventions: development and validation of the CLEO multidimensional tool Thi Ha VO Pharmacist, PhD Student Director: Assoc.Prof. Pierrick Bedouch Codirecteur: Prof. Benoit Allenet Laboratory TIMC-IMAG UMR CNRS 5525 PhD Defense - 16 December 2015
- 2. 1. Introduction 2. Development of a new tool 2.1. Review of models and tools 2.2. Development of the CLEO tool 3. Validation of the CLEO tool 3.1. Test in a general practice 3.2. Test in a specific clinical service 4. Discussion & Perspectives 5. Conclusion Plan
- 3. Drug-related problems (DRP) 2-27%2 of hospital admission 50%1 DRPs are preventable Solution = team working with clinical pharmacists Introduction – What’s clinical pharmacists’ rolesIntroduction – What’s role of clinical pharmacists? 1. McKenney JM et al. Drug-related hospital admission. Am J Hosp Pharm. 1976;33(8):792-5 2. Kohn LT et al. To Error is Human. 1999
- 4. Collect patient information Identify DRPs Suggest Pharmacist interventions (PIs) Medication Review Pharmacist intervention (PI): «any action by a clinical pharmacist that directly results in a change in patient management or therapy»1. (Dooley at al. BJCP 2004) Introduction – What’s pharmacist interventions ?Introduction – What’s a pharmacist intervention?
- 5. Introduction – How to evaluate impacts of PIs ? Impacts of the PI When ? Potential or actual impacts Where ? Hospital, community pharmacy How ? Models, tools, process of evaluation… What ? Clinical, economic, humanistic… For whom ? Patient, physician, nurse, pharmacist, hospital, society Evaluation of impacts of PIs is essential for demonstrating the added value of pharmacists, continuing quality improvement, research and education… Introduction – How to evaluate PIs? Reponses to these questions are important to develop a new tool for assessing impacts of PIs
- 6. Overhage and Lakes, USA,1999 5 distinct tools found in 10 studies from a review of articles from 1966 to 1997 Overhage and Lakes. Practical, reliable, comprehensive method for characterizing pharmacists’ clinical activities. Am J Health-Syst Pharm. 1999; 56:2444-50 Introduction – Tools to evaluate PIs?
- 7. Conduct an updated systematic review of tools for assessing potential impacts of PIs Review models of evaluations and develop an optimal model for evaluation of PIs Objectives Develop and validate a multidimensional tool for assessing potential impact of PIs
- 8. 1. Introduction 2. Development of a new tool 2.1. Review of models and tools 2.2. Development of the CLEO tool 3. Validation of the CLEO tool 3.1. Test in a general practice 3.2. Test in a specific clinical service 4. Discussion & Perspectives 5. Conclusion Plan
- 10. 873 Abstracts scanned Exclusion criteria: • Specific type of DRPs/PIs only • Actual impacts only • Economic impact only • Non-accessible and review articles 82 tools in 133 studies: models & content, structure, validity A systematic review – Method Keywords: DRP* AND PI* Inclusion criteria : • Langue: English, French • Abstract available • Peer review journals • Pharmacists involved • Explicit description of tools Databases - Pubmed (1986-2013) - PsycINFO (1999-2013) - PASCAL (1997-2013) - CINAHL (1993-2013) Review of tools for assessing potential impacts of PIs Hand searchs - References of (review) articles - Thesis of Quélennec
- 11. • Model of Donabedian • Model of Kozma et al. • Pharmacoeconomic model • Risk assessment matrix SP(ECH)O-P model Models of evaluations A systematic review – Results & Discussion Indicators of content of tools • Structure • Process • Outcome - Clinical - Humanistic - Economic • Probability A. Models & content of tools
- 12. A systematic review – Results & Discussion STRUCTURE Settings Staffs: trained pharmacists Equipments Information resources Financal stability ….NOT FOUND IN TOOLS PROCESS Technical Consistency to standards Informational intervention Acceptance by HCPs… Interpersonal Satisfaction of HCPs Collaboration between HCPs Continuation of care… OUTCOME Clinical Humanistic Economic … 1. Model of Donabedian 1966 Quality of a pharmacist intervention Donabedian et al (1966). Evaluating the quality of medical care. Milbank Mem Fund Q.
- 13. A systematic review – Results & Discussion STRUCTURE Settings Staffs: trained pharmacists Equipments Information resources Financal stability …. PROCESS Technical N° DRPs/PIs Consistency to standards Agreement between HCPs Acceptance Interpersonal Satisfaction of HCPs Collaboration btw HCPs OUTCOME CLINICAL • Severity of DRPs: harm, ADR… • Significance of PIs: Health care resources avoided (emergency visit, hospitalization…) ECONOMIC • Direct costs: cost of implementation of the PI cost saving, cost avoidance • Indirect costs: missing working • Intangible costs: pain, suffering… HUMANISTIC • Patient’s satisfaction • Knowledge • Medication compliance • Quality of life… 2. Model of Kozma et al. 1993 Outcomes of a pharmacist intervention Kozma et al (1993). Economic, clinical, and humanistic outcomes: a planning model for pharmacoeconomic research. Clin Ther.
- 14. STRUCTURE Settings Staffs: trained pharmacists Equipments Information resources Financal stability …. PROCESS Technical N° DRPs/PIs Consistency to standards Agreement between HCPs Acceptance Interpersonal Satisfaction of HCPs Collaboration between HCPs CLINICAL • Severity of DRPs: harm, ADR, therapeutic failure • Significance of PIs: Health care resources avoided (eg., emergency visit, hospitalization…) ECONOMIC • Direct costs: cost of implementation of the PI cost saving, cost avoidance • Indirect costs: missing working • Intangible costs: pain, suffering… • HUMANISTIC • Patient’s satisfaction • Knowledge • Medication compliance • Quality of life • … 3. Pharmacoeconomic model Value of a PI Value = Differences between the scenarios without and with the PI A systematic review – Results & Discussion Schumock GT et al. (2000). Method to assess the economic outcomes of clinical pharmacy services. Pharmacotherapy.
- 15. 15 STRUCTURE Settings Staffs: trained pharmacists Equipments Information resources Financal stability …. PROCESS Technical N° DRPs/PIs Consistency to standards Agreement between HCPs Acceptance Interpersonal Satisfaction of HCPs Collaboration between HCPs CLINICAL • Severity of DRPs: harm, ADR, therapeutic failure • Significance of PIs: Health care resources avoided(eg., emergency visit, hospitalization…) ECONOMIC • Direct costs: cost of implementation of the PI cost saving, cost avoidance • Indirect costs: missing working • Intangible costs: pain, suffering… HUMANISTIC • Patient’s satisfaction • Knowledge • Medication compliance • Quality of life • … Severity X Probability = Risk Matrix 4. Risk assessment matrix A systematic review – Results & Discussion National patient safety agency (2008). A risk matrix for risk managers.
- 16. STRUCTURE Settings Staffs: trained pharmacists Equipments Information resources Financal stability …. PROCESS Technical N° DRPs/PIs Consistency to standards Agreement between HCPs Acceptance Interpersonal Satisfaction of HCPs Collaboration between HCPs OUTCOME Clinical Humanistic Economic … CLINICAL • Severity of DRPs: harm, ADR, therapeutic failure • Significance of PIs: Health care resources avoided (eg., emergency visit, hospitalization…) ECONOMIC • Direct costs: cost of implementation of the PI cost saving, cost avoidance • Indirect costs: missing working • Intangible costs: pain, suffering… HUMANISTIC • Patient’s satisfaction • Knowledge • Medication compliance • Quality of life • … Value = Differences between the scenarios without and with the PI Severity X Probability = Risk Matrix An optimal model of evaluation of impacts of PIs: SP(ECH)O-P
- 17. Multi- or mono-dimensional Independent/Integrated Ordinal/Nominal Numeric/Non-numeric Explicit/Implicit Opened/closed B. Properties of structure of tools A systematic review – Results & Discussion Few tools have these optimal properties of structure A. Models & content of tools
- 18. C. Properties of validation of tools A systematic review – Results & Discussion Inter-rater reliability • Agreement between raters • 48/133 studies Intra-rater reliability • Agreement of same raters between times • 2/133 studies Validity • Agreement between raters’ and gold standard ratings • 8/133 studies A. Models & content of tools B. Properties of structure of tools Need of test inter-rater, intra-rater reliability and validity of a new tool
- 19. 1. Introduction 2. Development of a new tool 2.1. Review of models and tools 2.2. Development of the CLEO tool 3. Validation of the CLEO tool 3.1. Test in a general practice 3.2. Test in a specific clinical service 4. Discussion & Perspectives 5. Conclusion Plan
- 20. 20 Development of the CLEO tool PROCESS OF DEVELOPMENT • GROUP: 7 pharmacists of the working group of SFPC (French Society of Clinical Pharmacy) • WHY ? A simple, comprehensive, reliable tool • WHERE ? Hospital • WHAT ? clinical, humanistic, economic, process-related. No probability. • WHEN? Potential impact • FOR WHOM ? For the patient, hospital, HCPs • HOW ? 3 dimensions, 4-7 categories, opened • RESULTS: 6 direct meetings, 5 versions of the CLEO tool SCORE IMPACTS CLINICAL (CL) -1C Negative 0C Null 1C Positive – Humanistic 2C Favorable – Minor 3C Favorable – Major 4C Favorable – Vital ND Non-determined ECONOMIC (E) -1E Negative 0E Null 1E Positive ND Non-determined ORGANIZATIONAL (O) -1O Negative 0O Null 1O Positive ND Non-determined Score (CL, E, O)
- 21. Score Impact The clinical impact is evaluated according to the most likely case expected, not the worst/best case -1C Nuisible The PI can lead to adverse outcomes on clinical status, knowledge, satisfaction, patient adherence and/or quality of life of the patient. 0C Null The PI can have no influence on the patient regarding the clinical status, knowledge, satisfaction, patient adherence and or quality of life of the patient. 1C Minor The PI can improve knowledge, satisfaction, medication adherence and/or quality of life of the patient OR the IP can prevent damage that does not require monitoring/treatment. 2C Moderate The PI can prevent harm that requires further monitoring/treatment, but does not lead or dose extend a hospital stay of the patient. 3C Major The PI can prevent harm which causes or lengthens a hospital stay OR causes permanent disability or handicap. 4C Lethal The PI can prevent an accident that causes a potentially intensive care or death of the patient. ND Non- determined The available information does not determine the clinical impact. 21 1. CLINICAL IMPACT6 levels of Hatoum’s tool Severity categories from the NCC MERP Index Humanistic indicators = low levels from the Williams et al.’s tool Development of the CLEO tool The « Clinical impact » focuses on clinical and humanistic impacts of the PI for the patient.
- 22. 22 2. ECONOMIC IMPACT Score Impact Definition -1E Increase of cost The PI increases the cost of the drug treatment of the patient. 0E No change The PI does not change the cost of drug treatment of the patient. 1E Decrease of cost The PI saves the cost of drug treatment of the patient. ND Non-determined The available information does not allow determining the economic impact. The cost of drug therapy contains two main elements: • The cost of drugs • The cost of monitoring of drug therapy (e.g., clinical, kinetic, biological monitoring ...). The cost of drug therapy is based on the financial cost of a hospital. Development of the CLEO tool The « Economic impact » dimension focuses on cost savings of PIs based on the financial cost for a hospital
- 23. 23 3. ORGANIZATIONAL IMPACT Score Impact Definition -1O Negative The PI reduces the quality of care process. 0O Null The PI does not change the quality of the care process. 1O Positive The PI increases the quality of the care process. ND Non-determined The available information does not identify the organizational impact. The organizational impact is coded regarding the overall impact on the quality of the care process from the perspective of health care providers (eg, time savings, improved security, knowledge, job satisfaction of nursing staff; facilitating professional tasks or teamwork, continuity of care, etc.) Development of the CLEO tool The « Organizational impact » dimension focuses on benefits of PIs on process of care for HCPs
- 24. 24 Development of the CLEO tool A CASE STUDY • Description: Woman 85 years old was treated by AUGMENTIN (amoxicillin + clavulanic acid) IV for sinusitis. • PM: the patient was known to be allergic to beta-lactams (angioedema). • IP: change to PYOSTACINE (pristinamycin) 500mg tablet. SCORE IMPACTS CLINICAL (CL) -1C Negative 0C Null 1C Positive – Humanistic 2C Favorable – Minor 3C Favorable – Major 4C Favorable – Vital ND Non-determined ECONOMIC (E) -1E Negative – Increase of cost 0E Null – No change 1E Positive – Decrease of costs ND Non-determined ORGANIZATIONAL (O) -1O Negative 0O Null 1O Positive ND Non-determined Score (3C, -1E, 1O)
- 25. 1. Introduction 2. Development of a new tool 2.1. Review of models and tools 2.2. Development of the CLEO tool 3. Validation of the CLEO tool 3.1. Test in a general practice 3.2. Test in a specific clinical service 4. Discussions & Perspectives 5. Conclusions Plan
- 26. Test in a general practice - Method 1st rating 2nd rating 3rd rating 50 PIs from the Act-IP® database 30 PIs from the 6 pharmacists’ hospital practice 10 PIs from the same 30 PIs The CLEO v1 The CLEO v2 The CLEO v2 Raters TestScenarios Tool InTER-rater reliability InTER-rater reliability InTRA-rater reliability 7 pharmacists of the group of SFPC The SP(ECH)O-P model The CLEO
- 27. Test in a specific practice - Method Kappa value – Agreement < 0 : poor 0.00-0.20: slight 0.21-0.40: fair 0.41-0.60: moderate 0.61-0.80: substantial 0.81-1.00: almost perfect Agreement: % Validity/Reliability: Weighted kappa (kw) Statistical test Gisev et al (2013). Interrater agreement and interrater reliability: key concepts, approaches, and applications. Res Social Adm Pharm. Expected weighted kappa: kw ≥ 0.41 (moderate agreement) Interpretation of kappa value by Landais & Koch:
- 28. Test in a general practice - Results 1st rating 2nd rating 3rd rating The CLEO v1 The CLEO v2 The CLEO v2 Raters TestTool InTER-rater reliability CL: 36%, kw = 0.34 X E: 65%, kw = 0.53 O: 57%, kw = 0.26 X InTER-rater reliability CL: 39%, kw = 0.41 E: 90%, kw = 0.93 O: 62%, kw = 0.39 ! InTRA-rater reliability CL: 33%, kw = 0.38 ! E: 80%, kw = 0.70 The CLEO v2 CL nearly validated E validated O nearly validated Expected weighted kappa: kw ≥ 0.41 (moderate agreement)
- 29. 1. Introduction 2. Development of a new tool 2.1. Review of models and tools 2.2. Development of the CLEO tool 3. Validation of the CLEO tool 3.1. Test in a general practice 3.2. Test in a specific clinical service 4. Discussion & Perspectives 5. Conclusions Plan
- 30. Test in a specific practice - Method Objective: To assess validity and reliabilty of the CLEO tool used in daily practice in a Centralized Preparation Unit of Anticancer Drugs (CPU) at the Grenoble University Hospital Type of study: mono-center, prospective Services included : • Complete Hospitalisation in hematology and oncology • One-day Hospital in hematology, oncology, pneumology, hepato- gastroenterology and radiotherapy Period of study : 10 weeks (July 2014 to September 2014) Sample: 214 PIs related to 167 patients.
- 31. 1st rating 2nd rating 3rd rating Ward-based pharmacist (P2) Hematology 43 PI Oncology radiotherapy 146 PI Pneumology 33 PI Hepatogastro enterology 15 PI 1 panel for each speciality 4 panels Pharmacist at the CPU (P1) Expert panel (EP) Composition of each panel : 4 members (a specialist – physician, a clinical pharmacist, a pharmacist at the CPU and a pharmacist at a pharmacovigilance center) The CLEO v2 The CLEO v2 The CLEO v2 InTER-rater reliability (P1-P2) Validity (P1-EP) Validity (P2-EP) Raters TestTool Test in a specific practice - Method Process of ratings
- 32. Test in a specific practice - Method 1st rating 2nd rating 3rd rating Ward-bassed pharmacist (P2) Pharmacist at the CPU (P1) Expert panel (EP) The CLEO v2 The CLEO v2 The CLEO v2 Inter-rater reliability (P1-P2) CL: 51%, kw = 0.48 E: 71%, kw = 0.61 O: 60%, 0.27 X Validity (P1-EP) CL: 41%, kw = 0.32 ! E: 68%, kw = 0.53 O: 57%, kw = 0.17 X Validity (P2-EP) CL: 54%, kw = 0.56 E: 81%, kw = 0.75 O: 49%, kw = 0.11 X Raters TestThe tool The CLEO v2 CL nearly validated E validated O NOT validated Expected weighted kappa: kw ≥ 0.41 (moderate agreement)
- 33. 1. Introduction 2. Development of a new tool 2.1. Review of models and tools 2.2. Development of the CLEO tool 3. Validation of the CLEO tool 3.1. Test in a general practice 3.2. Test in a specific clinical service 4. Discussion & Perspectives 5. Conclusion Plan
- 34. A updated systematic review of tools of assessing potential impacts of PIs was conducted with: • 82 distinct tools found in 133 studies: rich resources for development of new tools • 6 models of evaluations • 12 recommendations of methods of assessment of potential impacts of PIs: theoretical, psychometric and pragmatic properties Discussion & Perspectives 1. Principal findings
- 35. The SP(ECH)O-P model: • The first specific model in literature for evaluation of PIs • A comprehensive model including 6 main types of indicators • Relationships between types of indicators Discussion & Perspectives 1. Principal findings
- 36. ▷Content & Structure: humanistic indicators included, health care resouces included, structure inspired the Hatoum’s tool, content inspired from NCC MERP, 7 categories ▷Validation properties 33-54%, Kw = 0.32-0.56 VALIDATED ▷ ▷Content & Structure: cost savings (drug, monitoring), 4 categories ▷Validation properties 68-90% , Kw = 0.53-0.93 VALIDATED ▷Content & Structure: integration of many organizational indicators and persepctives of HCPs, 4 categories ▷Validation properties 49-63% , Kw = 0.11-0.39 NOT VALIDATED Economic impact Test of the CLEO in 2 studies - Results Clinical impact Organizational impact Content, Structure & Validity of the CLEO tool 1. Principal findings
- 37. Discussion & Perspectives Improvement of validation properties: training, examples Establisment relationship between potential and actual clinical impact Development of separate tools for measuring humanistic impacts Specific study for improvement of organizational impact: eg., qualitative studies on perceptions of HCPs of organizational impacts Estimation of direct costs (cost savings, cost avoidance and cost of implementation of a PI) Adding estimation of probability 2. Validity of findings – Quality of the CLEO tool Develop of assessment matrix (a global score)
- 38. Discussion – The CLEO tool 2. Validity of findings – Validity of research method Internal validity - Confounding: perceptions of raters - Maturation of raters: familiarity with the tool - Testing bias: awareness of ratings - Selection bias of PIs - Statistical test: %, kw External validity - Raters: 7 raters involved to develop another tool 10 years ago - Ratings: requirement of confirmations of ratings in some cases - Settings: a specific practice (CPU) - Tool: the French-written CLEO tool for hospital
- 39. Database: Adding only the « Clinical impact » and « Economic impact » into the Act-IP® database. French hospitals: test for daily use in other services/hospitals (Lyon, Annecy, Epernay…) Community pharmacies: modifications of the CLEO Other langues/countries: Vietnam 276,851 documented PIs 1723 pharmacists Senior 45% Resident 50% Student 6% 666 hospitals University 34% General 62% Psychiatric 4% Act-IP© http://www.actip.sfpc.eu Perspectives
- 40. 1. Introduction 2. Development of a new tool 2.1. Review of models and tools 2.2. Development of the CLEO tool 3. Validation of the CLEO tool 3.1. Test in a general practice 3.2. Test in a specific clinical service 4. Discussion & Perspectives 5. Conclusion Plan
- 41. The updated systematic review of tools for assessing of potential impacts of PIs synthesized models, content, structure and validation properties of tools. The specific and comprehensive model for evaluations of PIs, named SP(ECH)O-P was developed. The CLEO tool of assessing potential impacts of PIs (including 3 dimensions: Clinical Impact, Economic Impact, and Organizational impact) was constructed and tested in 2 studies. However, only « Clinical impact » and « Economic impact » dimensions were validated. Conclusion
- 42. 42 Discussion, conclusion Working group “Standardizing and demonstrating the value of clinical pharmacy activities” of the SFPC (the French Society of Clinical Pharmacy) Group of clinical practitioners (pharmacists, physicians) at the Grenoble University Hospital Acknowledgements
- 43. 43 It is not hard to make decisions once you know what your values are. Roy E. Disney
Notas del editor
- Hello every body My name’s Thi Ha Vo, a pharmacist – and a lecturer at Hue Unversity of Medicine and Pharmacy in Vietnam. I’m a PhD student in Grenoble since 4 years. Today, I’m pleasure to present the results of my thesis study: « Assessment of the potential impact of PIs: development and validation of the CLEO multidimensional tool ».
- The plan of my presentation consists of five parts. Firstly, I introduce the context and why I conduct this study. Then, process of development of a new tool, named CLEO are presented. Next, I present the process of validation of the CLEO in 2 studies. Finally, I finish my presentation by some discussions and perspectives
- - As you may know, DRPs are ones of major problems of health care systems. Some examples of DRPs are: adverse drug reactions, overdose, drug interactions, drug contra-indications… DRPs are causes of 2-27% of hospital admissions. About 50% of DRPs are potentially preventable. And Clinical pharmacists play a key role to work with other HCPs such as physicians, nurses to optimize drug use for patients.
- There are a variety of clinical pharmacy services that pharmacists can do. Of which, MR is one of major contribution of pharmacists. MR means that pharmacists collect all relevant patient information and review updated knowledge of treatment in literature in order to identify DRPs and then suggest changes, named PIs. . PI is defined as any action by a clinical pharmacist that directly results in a change in patient management or therapy. For example, change the drug, route of admistration, change of dosage..
- Evaluation of impacts of PIs is essential for demonstrating the added value of clinical pharmacists, continuing quality improvement, research and education…That is reasons why we want to develop a tool for evaluation of impacts of PIs. However, at the begining of the study, we raised many questions concerning evaluation of PIs: + When should we evaluate them, whether should we evaluate potential or actual impacts ? + Where should our tool target to be used ? Only in hospitals or both hospital and community pharmacies + How about models or tools of evaluations, and process of evaluation known in literature ? + Which impacts do we need to evaluate ? Clinical, economic impacts and anything else ? + For whom do the PIs provide impacts ?
- In 1999, Overhage and Lakes in USA conducted a review of tool for assessing impacts PIs from 1966 to 1997. They found 5 distinct tools in 10 studies. Only the tool of Folli et al was widely used. The tool of Folli assessed severity of error. The tool of Hatoum et al evaluated clinical impact of PIs The tool of Brandt et al. and used by Walters et al. focussed on only drug administration errors. The classification of Neville and coll focussed on potential effects and inconvenience of prescription errors to the patients. In a study of community pharmacies, Rupp and coll categorized severity of error into health resources avoided Based on results obtained,Overhage and Lakes developped a new tool including 2 dimensiosn: severity of error inspired from Folli, significance of PIs inspired from the tool of Hatoum. High reliability was demonstrated for this new tool. This validated tool has been widely used and adopted in other studies. However, results of reliability were low in many studies. And this tool has some other limitations: although the tool of Overhage et al consists of 2 dimensions, it only focused on clinical impact and doesn’t evaluate other impacst, such as economic, humanistic impacst
- Therefore, the our study aimed to conduct an updated systematic review of tool for assessing potential impacts of PIs. (2) Review models of evaluations and develop an optimal model for evaluation of PIs In order to reach the objective: (3) Develop and validate a multidimensional tool for assessing potential impacts of PIs
- Firstly, I present the review of models and tools of evaluation in literature
- The results of this systematic review was published in the journal Drug Safety a week ago.
- Concerning the method of the review, we followed the guidlines of Cochrance and PRISMA for systematic review. 2 pharmacists conducted the review independently. We used combination of different keywords of DRPs and PIs and searched in 4 databases: Pubmed, PsyINFO, PASCAL, CINAHL for the period 1986-2013. As final results, we scanned 873 abstracts and found 82 distinct tools in 133 studies.
- Few studies described how they constructed their tools, which models of evaluations inspired them to develop their tools. I will present 4 main models which were applicable to construct tools in literature and then introduced a new model we constructed for assessing impacts of PIs. These models consists of 6 types of indicators of content of tools: related to structure, process, clinical, humanistic and economic outcomes, and probability of impacts.
- According to the model of Donabedian, the author stated that quality of any heath care interventions depends on 3 types of indicators: related to structure of care, process of care, and outcomes of health care interventions. Application of this model to evaluation of quality of PIs, we need to evaluate also properties of structure in which pharmacists conducted interventions such as: staffs, equipments, drug information resources… However, these structure-related indicators were not found in tools because PIs aimed to influence on the individual patient and process of care rather than influence on structural settings. Some process-related indicators were found in different tools such as consistency of practice to standards, acceptance of PIs by physicians, satisfaction or collaboration btw HCPs
- Concerning outcomes of PIs, according to the model of Kozma et al applicable to PIs, the outcomes of pharmaceutical products or services consists of clinical, economic and humanistic outcomes. Clinical outcomes are defined as medical event that occur due to disease or medical interventions. In tools indentified, clinical impacts was estimated in two ways. The first (and most common) way was to assess the severity of harm to the patient avoided by the PI. The second way was to estimate the resources that would have been required to manage the patient as a result of the DRP such as hospitalization, emergency visit… humanistic outcomes are defined as the consequences of disease or treatment on patient functional status or quality of life. In tools, some humanistic indicators found in some tools were patient’s satisfaction, knowledge of patients, medication compliance and quality of life of patient Concerning economic outcomes, economic outcomes as defined as direct cost, indirect cost and intangible costs.
- According to the pharmacoeconomic model applicable to the PI, the value of a PI depends on difference of inputs and outputs between the scenario without and with the PI. Inputs means resources consumed related to structure and process of care in order to conduct the PI. Inputs includes 2 types of costs: costs of implementation of the PIs such as salary to pay pharmacists, telephone, fax; or costs savings such as cost of drug change, monitoring. Outputs means outcomes of the PI, including clinical, humanistic and economic outcomes. Clinical and humanistic outcomes can be converted to economic value. For example, importance of clinical outcomes of the PI were determined by health care resources avoided by the PIs which can be converted to cost avoidance of the PI. Or humanistic outcomes can be converted to intangible costs. However, value of a PIs found in all studies using tools were calculated from only 3 types of direct costs: cost savings plus cost avoidance less cost of implementation. Indirect and intangible costs were ignored because it was so difficult to estimate.
- According the risk assessment matrix, combination of severity and probability of each consequences contributes to overal priority risk. However, a few tools combined severity and probability of only clincal impact.
- - By combining the above general models to the case of evaluation of a PI and using results obtained from the systematic review, we developed an intergrated model, named SP(ECH)O-P. According to this model, the value of a PI depends on 6 types of indicators: related to structure and process and clinical, economic and humanistic outcomes, and probability. The value of the PI is calculated from the differences of these indicators between the scenarios without and with a PI.
- Concerning to structure of tools, we found some prefered properties of structure of tools, including: The structure was multidimensional, meaning the tool can answer many different questions related to impacts of PIs Different types of indicators were evaluated independently Each dimension consists of ordered and numeric levels Each levels was defined clearly Structure of tools was opened, meaning each dimension has a category like non-determined or others for cases when available information did not allow to determine the impacts. - We found in the review, few tools had these optimal properties.
- Concerning to properties of validation of tools, about a third of tools was tested for inter-rater reliability (that means agreement of ratings between different raters), intra-rater raliability (that means agreement of ratings by the same rater between different times) and validity (that means agreement between ratings of raters and gold standard ratings such as an expert panel) Of that, many tools tested had low agreement. - In short, a new tool need to be tested for 3 variables: inter and intra rater reliability and validity.
- Now, I will present how the CLEO tool was constructed
- - The group of 7 clinical pharmacists who worked in 6 French hospital, belonging to the French Society of CP was involved in developted the new tool. Based on results of review and inputs of the experts, we discussed a lot to find answers to questions raised. Aftter 6 direct meetings, 5 versions of the CLEO were suggested. Finally, we wanted to develop a simple, comprehensive and reliable tool for each pharmacist who can rated indepedently in their hospital. This is the final version of the CLEO. The CLEO tool consists of 3 dimensions: clinical, economic and organizational impacts. By assessing impacts of PI with the CLEO, each PI can have a score of 3 value.
- To develop the « Clinical impact », we inspired the structure of 6 levels of Hatoum’s tool. For integration of humanistic indicators into the « clinical impacts », we inspired the tool of Williams et al. In the tool of Williams et al., they included patient’s knowledge and patient compliance into low and mild levels of clinical impact. In our tool, we included 4 major humanistic indicators: patient’s knowledge, satisfaction, compliance and quality of life into the level 1 – minor positive impact. The definitions of levels 2, 3, 4 were inspired from the severity categories of the NCC MERP – a popular severity classification of actual harm of medication error. In short, the clinical impact focuses on clinical and humanistic impacts of the PI for the patient
- The economic impact consists of 4 categories: increase of cost, no change, decrease of cost and non-determined. The economic impact evaluates cost savings related to drug or monitoring of drug therapy. In bref, the economic impact dimension focuses on cost savings of PIs based on the financial cost of the hospital
- - Concerning the organizational impact, the organization impact is rated the impact of the PI on process of care from perspective of HCPs into 4 categories: negative, null, positive and non-determined. We included many process-related indicators found in the review, meaning benefits of PIs for HCP such as time savings, improved security, knowledge, job satisfaction of nursing staffs, facilitating professional tasks… In bref, the organization impact dimensiosn focuses in benefits of PIs in process of care for HCPs
- - For example, in one case study: … Because the PI can prevent allergic reaction (such as angioedema) for the patient which requires hospitalization, therefore the clinical impact accoding to the CLEO was the level 3C. Because we had to change to more expensive drug for the patient, so the level of the eco impact was -1E. With this PI, the patient was changed from IV to orall administration, which was more simple and less time-consuming for nurses. Therefore, the PI had a positive orga impact. At final, this PI has a score: 3C, -1E, and 1O.
- Now, I will present process of validation of the CLEO tool. The first one tested the CLEO tool in a general practice
- - Objective of this study was to test reliability of different verions of the CLEO when assessing impacts of variety of types of PIs sellected from different context. - After the we developped a model of evaluation and six versions of the CLEO tools. Two versions of the CLEO were tested, named CLEO v1, v2. For the first ratings, 7 pharmacists of the group rated impacts of 50 PIs sellected from the French database of PIs, named Act-IP in order to calculate the inter-rater reliability. We questioned also satisfactions of the pharmacists and suggestions for modifications of the CLEO. Based on these suggestion, we modified and obtained the CLEO v2. For the second ratings, we asked 7 pharmacists who worked in 6 hospitals to collect 10 PIs for each one in their hospital. Then, we choosed 30 PIs with complete informations and assured to balanced types of DRPs and PIs. And required 7 pharmacists to rate impacts using the CLEO v2 in order to calculate inter-rater reliability For the third ratings, we sellected randomly 10 PIs from the same 30 PIs and asked the pharmacists to rate the second time 2 month lahers in order to calculate the intra-rater reliability
- Regaring statistical test, we used 2 variables for evaluating reliability or validity. Those were « percentage of agreement » and « weighted kappa » Percentage of agreement: is defined as the percentage of possible cases scores/ratings are identical Kappa score relates to the extent of variability and error inherent in a measurement. The kappa provides a chance-corrected index and are based on the ratio of the proportion of times agreement is observed, to the maximum proportion of times that the raters could agree (both corrected for chance agreement). Therefore, the kappa score is more exact to demonstrate true agreement. We used the weighted kappa instead kappa. Whereas kappa considers only total (“all-or-none”) agreement or disagreement, weighted kappa allows for the assignment of weights to different categories such that similar categories can be in partial agreement. We used the interpretation of kappa score proposed by Landis and Koch. Based on results of low kappa score obtained in many studies in the review, we targeted a weighted kappa equal or more than 0.41 (moderate agreement)
- The results obtained were that Kappa Scores of the CLEO v2 were higher than ones of the CLEO v1. For the inTER-rater reliability of the CLEO v2, kw scores were equal to 0.41 for clinical impact, much higher than 0.41 for the economic impact, and slightly lower 0.41 for the organizational impact, however percentage of agreement of « organizational impact » was quite high 62%. Concerning to the inTRA-rater reliability of the CLEO v2, the results were slightly lower than 0.41 for clincial impact and much higher 0.41 for economic impact. We did not tested intra-rater reliability for organizational impact. In conclusion, the CL impact was nearly validated, the E impact was validated and the O impact was nearly validated. Therefore, the CLEO v2 was then used for test in the next study.
- The version 2 of the CLEO was then tested for validity and inter-reliability in daily practice at a centralized preparation unit of anticancer drugs (CPU) at the Grenoble Univeristy hospital This was a mono-center, prospective study which lasted for 10 weeks, including Complete Hospitalisation in hematology and oncology One-day Hospital in hematology, oncology, pneumology, hepato-gastroenterology and radiotherapy Finally, we collected 214 PIs related to 167 patients
- The process of ratings as follows: For the first ratings, a pharmacist at the CPU who suggested a PIs during the study period evaluated impacts of their PI immediately For the second ratings, a second pharmacist who worked in services with physicians and patients collected all informations related to PIs several days later and coded. For the third ratings, all PIs were evaluated in consensus by expert panels. There were 4 expert panels for 4 specific therapeutic domains And each expert panel consisted of 4 members: a specialist – physician, a clinical pharmacist, a pharmacist at the CPU and a pharmacist at a pharmacovigilance center We compared results of ratinsg btw 2 pharmacists in order to calculate inter-rater reliability We compared results of ratings btw each pharmacist and expert panels to calculate validity of the CLEO tool.
- the validity between the second pharmacist and EPs were higher levels of agreement than one bewteen the first pharmacist and Eps for the clinical and economic impact, but the same low level of agreement for the organizational impact In general, - The results of inter-rater reliability and validity obtained were: Regarding the clinical impact, kappa was higher than 0.41 except kappa of validity between the first pharmacist and EP was slightly lower than 0.41. Redarding the economic impact, kappa were much higher than 0.41. Regarding the organizational impact, kappa were much lower than 0.41 In conclusion, the « CL » impact was nearly validated, the E impact was validated, and the O impact was not validated.
- From results obtained, we had some discussions and perspectives
- Secondly, our systematic review of tools of assessing potential impacts of PIs contributed to update the knowledge on this topic. . 82 distinct tools found in 133 studies provided rich resources for other researchers or practitioner to develop new tools. A great numbers of tools found in our review showed that after the first step to implementation of clinical pharmacy services for many decades, nowadays, pharmacists focuses more and more on demonstration of benefits of their services Futhermore, currently, there are no formal guidelines concerning methods of assessing potential impacts of PIs, taking into account the results of this review, we suggested 12 recommendations about theoretical, psychometrics and pragmatic properties of methods of assessment.
- Concerning to principal findings of the study, the first major finding was we constructed a new model, SP(ECH)O-P model for evaluation of PIs The advantages of this model are: + Firstly, it is a specific model of evaluation of PIs + Secondly, the model is comprehensive, including 6 major types of indicators + Thirdly, the model described also the relationships between types of indicators
- The third principal findings were related to the CLEO tool. The CLEO tool evaluates the « clinical and humanistic impact » for the patient, the cost savings for the hospital and the organizational impact – benefits for HCPs. The CLEO tool was tested in 2 studies for inter, intra reliability and validity. And the kappa score for the « CL » impact was from 0.32 to 0.56. When comparing to other tools having 5-7 categories in literature, our results were better than many tools and just worse than the tool of Overhage and Lakes, and the tool of Struck et collegues. Therefore, we considered the CL impact as validated. The kappa score for E impact was high enough with the value from 0.53-0.93. Then, the E impact was valiated. The kappa score the O impact was lower than 0.41 in two studies, then the it is not validated.
- Regarding validity of findings of the work, we discuss firtly quality of CLEO tool and how to improve it Concerning to the clinical impact, + Improvement of validation properties by training pharmacists, providing examples and guide of ratings should be conducted. + there is a need of study who that will establishrelationship between potential and actual clinical impact Concerning to humanistic impacts, we need developement of separate tools for measuirng humanistic impacts Concerning to economic impact, we need further economic studies which will estimate 3 types of direct costs (including..) and even indirect and intangible costs. Concerning to the organizational impact, we need to conduct qualitative studies on perceptions of HCPs to defined clearly organizational impacts and we can try to modify the « organizational impact » by seperation of organizational indicators and types of HCPs. We need also to add estimation of probability and develop an assessment matrix which combined all important impacts into a global score
- - Concerning to the validity of research method: There were some main threats to internal validitity in the study + Confounding factors: results of agreement btw raters in two studies may be affected by perceptions of raters themselves rather than functionality of the tool. To minimize this bias, we choose a great mumber of raters (7 for the first one and 10 for the second one) + Maturation of raters: raters rate more consistently overtime because they becom be more familiar to the tool, espectically in the first study, the pharmaacist of the Group SFPC who was involved in development of the tool. To deal this bias, we conduct the second study in which raters were all new to the tool. + Testing bias: evaluators are aware of that their ratings will be compared to others, than, thay are likely to rate based on what researchers expect rather than based on their true perception. To minimize this threat, we informed clearly objectives of studies was to test the functionality of the tool. We minimized bias of selections of samples of PIs by applyng different method of sellection in 2 studies We used weighted kappa to test reliability/validity, then weighted kappa values tend to be higher than unweighted kappa values. Therefore, comprarison to ther studies need to consider this point. Concerning to the external validity of findings in this work, the studies may have included unique characteristics that make the results unique to the study, and cannot be transfered to others. + Fistly, the group of SFPC was involved in development another tool 10 ys ago, which can influence the ratings in these studies. However, this is minor threat because of long times ago. + Secondly, in the second rating of the first study, we required confirmations of rating in some cases when the investigators found one valuator rated so differently compared to majority of raters. The objective was to prevent cognitive or unintentional errors. However, only few PIs were needed to do so. Therefore, this bias was minor. + We tested in daily practice in one specific setting (CPU), and other largem, multi center studies are needed. + Finally, the CLEO tool was written in French and tested by French hospital pharmacists. Other studies are needed for translation and tool adoption for use in other languages/countries.
- Results of this work can open some perspectives At the present, some colleages in different French hospitals from Lyon, Annecy, Epernay…contacted us for testing the CLEO in their hospitals We are going to add the « Clinical impact » and « Economic impact » into the French database of PI, named Act-IP. This databased is constructed by the SIG from 2006. At the present, there are over 276.00 documented PIs. We need also to modify the CLEO for use in community pharmacies Finally, after finishing the study in France, I will return back to Vietnam, and I will conduct another study for tool translation and adoption in Vietnam.
- From results obtained, we had some discussions and perspectives
- I would like to thanks the working group of SFPC and the group pf physicians and pharmacists at the Grenoble University Hospital who were involved in this study. I thank also to laboratory TIMC-IMAG, Unversity Grenoble Alpes.
- Thank you for your attention!