Amee 2009 7 E5 JCH

•

1 recomendación•247 vistas

Jane Holland

Presented at AMEE 2009 - Item-Writing & Standard Setting

Educación Salud y medicina Deportes

Introduction of Standard Setting in undergraduate medical education: re-emphasizing the importance of item-writing. Jane Holland, Alice McGarvey, Richard Arnett, David Croke Royal College of Surgeons in Ireland

Standard setting – preparation for pilot Module selection Nutrition & Energy Junior Cycle Semester 1 n = 281 19 - anatomy, biochemistry pharmacology, physiology Lecturers / Judges Examination Multiple choice, n = 30 Method Modified Angoff (1971) Jane Holland, RCSI

Standard setting – training of judges 59.2 60.7 Workshop Dr Katherine Boursicot November 2007 Principles & specific methods Exercise 10 MCQs (JC1 NE 2007) Jane Holland, RCSI

JC1 NE 2008: Modified Angoff “Classic” Angoff MCQ cut score 50 58 MCQ Class mean 63.6 54.6 Overall class mean 61.1 59 Fails 12 15.4 14 judges 30 MCQs 3 ½ hours!! *All tabular values are percentages Jane Holland, RCSI

JC1 NE 2008: Hofstee 57 % Jane Holland, RCSI

RCSI Curriculum Forum, 29th March 2008 Jane Holland, RCSI

Item writing – training of writers Workshop Dave Swanson / Kathy Holtzman April 2008 Guidelines NBME / MRCS Jane Holland, RCSI

Item writing – pilot within NE module 13 unusable 64 suitable! 7 -ve 4 T/F 2 misc Call for questions 60 MCQs required 71 MCQs requested Guidelines / SBA stems 77 MCQs received 83% Jane Holland, RCSI

JC1 NE 2009: Modified Angoff 12 judges 30 MCQs Cut-score = 48 Item difficulties DIF IDEAL Consortium Item Analysis package Item probabilities Prob Modified Angoff (1971) Jane Holland, RCSI

JC1 NE 2008: DIF vs Prob Prob KR-20 = 0.735 DIF rs = 0.277 (p = 0.13) Jane Holland, RCSI

JC1 NE 2009: DIF vs Prob Prob KR-20 = 0.769 DIF rs = 0.599 (p = 0.001) Jane Holland, RCSI

Summary of results & take home message 2008 2009 MCQ cut score 58 48 MCQ Class mean 63.6 63.0 Item difficulties vs. item probabilities (rs) 0.277 (p=0.13) 0.599 (p=0.001) A well-structured item, while designed to more accurately assess the candidate’s knowledge of the subject material, also enables more accurate rating of items during the standard setting process and is an important consideration when introducing these procedures de novo. Jane Holland, RCSI

Más contenido relacionado

Similar a Amee 2009 7 E5 JCH

Aligning tests to standards

Fariba Chamani

Developing short answer questions (sa qs)

Javed Iqbal

Loughborough DHD seminar

Simon Bates

Stich Qol Mark

hospital

Robust Methods for Health-related Quality-of-life Assessment

dylanturner22

Measurement of Course learning Outcome using new NCAAA course report template

Ahmedani Shoaib

Patricia Khashayar

The distribution of research efforts done in the field of Natural Language Processing (NLP) has not been uniform across all natural languages. It has been observed that there is a significant gap between the development of NLP tools in Indic languages (indic-NLP), and in European languages. We aim to explore different directions to develop an automatic question answering system for Indic languages. We built a FAQ-retrieval based chatbot for healthcare workers and young mothers of India. It supported Hindi language in either Devanagri script or Roman script. We observed that, in our FAQ database, if there exists a question similar to the query asked by the user, then the developed chatbot is able to find a relevant Question-Answer pair (QnA) among its top-3 suggestions 70% of the time. We also observed that performance of our chatbot is dependent on the diversity in the FAQ database. Since database creation requires substantial manual efforts, we decided to explore other ways to curate knowledge from raw text irrespective of domain. We developed an Open Information Extraction (OIE) tool for Indic languages. During the preprocessing, chunking of text is performed with our fine-tuned chunker, and the phrase-level dependency tree was constructed using the predicted chunks. In order to generate triples, various rules were handcrafted using the dependency relations in Indic languages. Our method performed better than other multilingual OIE tools on manual and automatic evaluations. The contextual embeddings used in this work does not take syntactic structure of sentence into consideration. Hence, we devised an architecture that takes the dependency tree of the sentence into consideration to calculate Dependency-aware Transformer (DaT) embeddings. Since the dependency tree is also a graph, we used Graph Convolution Network (GCN) to incorporate the dependency information into the contextual embeddings, thus producing DaT embeddings. We used a hate-speech detection task to evaluate the effectiveness of DaT embeddings. Our future plan is to evaluate the applicability of DaT embeddings for the task of chunking. Moreover, the broader aim for the future is to develop an end-to-end pronoun resolution model to improve the quality of triples and DaT embeddings. We also aim to explore the applicability of all our works to solve the problem of long-context question answering.

A Framework for Automatic Question Answering in Indian Languages

IIIT Hyderabad

The distribution of research efforts done in the field of Natural Language Processing (NLP) has not been uniform across all natural languages. It has been observed that there is a significant gap between the development of NLP tools in Indic languages (indic-NLP), and in European languages. We aim to explore different directions to develop an automatic question answering system for Indic languages. We built a FAQ-retrieval based chatbot for healthcare workers and young mothers of India. It supported Hindi language in either Devanagri script or Roman script. We observed that, in our FAQ database, if there exists a question similar to the query asked by the user, then the developed chatbot is able to find a relevant Question-Answer pair (QnA) among its top-3 suggestions 70\% of the time. We also observed that performance of our chatbot is dependent on the diversity in the FAQ database. Since database creation requires substantial manual efforts, we decided to explore other ways to curate knowledge from raw text irrespective of domain. We developed an Open Information Extraction (OIE) tool for Indic languages. During the preprocessing, chunking of text is performed with our fine-tuned chunker, and the phrase-level dependency tree was constructed using the predicted chunks. In order to generate structured triples, various rules were handcrafted using the dependency relations in Indic languages. Our method performed better than other multilingual OIE tools on manual and automatic evaluations. The contextual embeddings used in this work does not take syntactic structure of sentence into consideration. Hence, we devised an architecture that takes the dependency tree of the sentence into consideration to calculate Dependency-aware Transformer (DaT) embeddings. Since the dependency tree is also a graph, we used Graph Convolution Network (GCN) to incorporate the dependency information into the contextual embeddings, thus producing DaT embeddings. We used a hate-speech detection task to evaluate the effectiveness of DaT embeddings. Our future plan is to evaluate the applicability of DaT embeddings for the task of chunking. Moreover, the broader aim for the future is to develop an end-to-end pronoun resolution model to improve the quality of triples and DaT embeddings. We also aim to explore the applicability of all our works to solve the problem of long-context question answering. The distribution of research efforts done in the field of Natural Language Processing (NLP) has not been uniform across all natural languages. It has been observed that there is a significant gap between the development of NLP tools in Indic languages (indic-NLP), and in European languages. We aim to explore different directions to develop an automatic question answering system for Indic languages. We built a FAQ-retrieval based chatbot for healthcare workers and young mothers of India. It supported Hindi language in either Devanagri script or Roman script.

A Framework For Automatic Question Answering in Indian Languages

IIIT Hyderabad

Adjusting for Differential Item Functioning in the EQ-5D-5L Using Externally-...

Office of Health Economics

f NSB010 Student CPS Guide Page 7 of 81 In tro C P S 1 C P S 2 C P S 3 C P S 4 C P S 5 C P S 6 C P S 7 C P S 8 C P S 9 Guidelines for documentation S Situation: Identify patient and what is new for that shift B Background: What are the relevant pieces of information about your client medical history? A Assessment: What is new about your client’s condition? What happened during your shift R Recommendation: What is the plan for the next shift? PATIENT PROGRESS NOTE Happy Valley Nursing Home Victoria Park Road KELVIN GROVE QLD 4154 Bed: 4 URN: QUT302872 PATIENT: Sam Shepherd DOB: 31/05/1936 (81yo) Date of Admission: 10/05/2015 Date/ Time Notes 13/07/20151420 1420 Mr Shepherd has had three loose bowels motions this shift and was incontinent for all of these. He has a history of hypertension, AF, R) CVA 2010 L) sided weakness, glaucoma, Type II diabetic, and Asthma. Vital signs assessed – BP 130/90, PR 88(irreg), R 16, temp 36.5oC. Complaining of abdominal cramping in the lower quadrants, pain score 5/10. Dr Trainer has been notified and has requested a stool sample to be sent to pathology and to give Panadol x 2 for pain. Panadol given at 13:30 and Mr Shepherd stated that this had given him some relief – pain score 3/10. Mr Shepherd is to remain on bed rest with toilet privileges. Stool sample has not yet been collected. M. Smith (SMITH)RN S B A R The patients’ medical file is a legal document. When writing in a patient’s medical file: § Writing must be legible and in black or blue pen § Any mistakes are to be crossed out with a single line then signed and dated § All entries need to have a date and time, and be signed with your designation § Don’t leave blank space. IF IT ISN’T DOCUMENTED IT HAS NOT OCCURRED Assessment Task 1 Assessment name: Critique of a Video Vignette Learning outcomes measured: 1. Critique a health assessment, recognising normal health parameters and identifying actual and potential health problem / s and accurately document findings. 2. Apply knowledge of the key NMBA Registered Nurse Standards for Nursing Practice, National Safety and Quality Health Service Standards and national health priorities for effective and appropriate decision making, planning and action. 3. Apply the underpinning knowledge of anatomy, physiology and pathophysiology to support evidence-based decisions for planning and action. 4. Apply clinical reasoning, decision making and communication skills to inform care planning of fundamental safe person- centred nursing care across the lifespan. Length: 750 words Estimated time to complete task: Approximately 3 - 5 hours Weighting: Satisfactory or Unsatisfactory Individual/Group: Individual Authentic Assessment: Yes Formative/Summative: Formative but mandatory How ...

f NSB010 Student CPS Guide Page 7 of 81 Intro .docx

ssuser454af01

Clinical Prediction Rules

Jen-Hsiang Chuang

Presented by Sarah B. McBrien, Curriculum & Assessment Manager, University of Nebraska Medical Center College of Medicine Facing curriculum reform? Need to look closer at what content is being tested where? ExamSoft's longitudinal reports can help educators make informed, evidence-based decisions during the curriculum reform and program evaluation phase. This presentation will focus on how categories can be used to help you mine valuable data about program outcomes, and will provide the more advanced user with concrete examples of how you can use that data to ensure quality in your new or existing program.

Using Categories to Direct Curriculum Reform and Evaluation

ExamSoft

ALERT Presentation - Ching - IPSSW 2014

INSPIRE_Network

F. Benvenuti - Tele-rehabilitation service for stroke survivors with residual...

Wydział ds. eZdrowia, Departament Polityki Zdrowotnej, Urząd Marszałkowski w Łodzi

05 Programme evaluation

東京大学医学系研究科医学教育国際研究センター

6_State of the Department_ OB-Gyn 2013

Julian Thach

Differential Item Functioning among Students with Disabilities and English La...

Kevin Krost

A new method for valuing quality of life: Direct elicitation of personal util...

Office of Health Economics

Accurately answering verbose queries that describe a clinical case and aim at finding articles in a collection of medical literature requires capturing many explicit and latent aspects of complex information needs underlying such queries. Proper representation of these aspects often requires query analysis to identify the most important query concepts as well as query transformation by adding new concepts to a query, which can be extracted from the top retrieved documents or medical knowledge bases. Traditionally, query analysis and expansion have been done separately. In this paper, we propose a method for representing verbose domain-specific queries based on weighted unigram, bigram, and multi-term concepts in the query itself, as well as extracted from the top retrieved documents and external knowledge bases. We also propose a graduated non-convexity optimization framework, which allows to unify query analysis and expansion by jointly determining the importance weights for the query and expansion concepts depending on their type and source. Experiments using a collection of PubMed articles and TREC Clinical Decision Support (CDS) track queries indicate that applying our proposed method results in significant improvement of retrieval accuracy over state-of-the-art methods for ad hoc and medical IR.

Optimization Method for Weighting Explicit and Latent Concepts in Clinical De...

Saeid Balaneshinkordan (Balaneshin-kordan)

Similar a Amee 2009 7 E5 JCH (20)

Aligning tests to standards

Developing short answer questions (sa qs)

Loughborough DHD seminar

Stich Qol Mark

Robust Methods for Health-related Quality-of-life Assessment

Measurement of Course learning Outcome using new NCAAA course report template

A Framework for Automatic Question Answering in Indian Languages

A Framework For Automatic Question Answering in Indian Languages

Adjusting for Differential Item Functioning in the EQ-5D-5L Using Externally-...

f NSB010 Student CPS Guide Page 7 of 81 Intro .docx

Clinical Prediction Rules

Using Categories to Direct Curriculum Reform and Evaluation

ALERT Presentation - Ching - IPSSW 2014

F. Benvenuti - Tele-rehabilitation service for stroke survivors with residual...

05 Programme evaluation

6_State of the Department_ OB-Gyn 2013

Differential Item Functioning among Students with Disabilities and English La...

A new method for valuing quality of life: Direct elicitation of personal util...

Optimization Method for Weighting Explicit and Latent Concepts in Clinical De...

Último

Mehran University Newsletter Vol-X, Issue-I, 2024

Mehran University of Engineering & Technology, Jamshoro

Klinik_ Apotek Onlin 085657271886 Solusi Menggugurkan Masalah Kehamilan Anda Jual Obat Aborsi Asli KLINIK ABORSI TERPEECAYA _ Jual Obat Aborsi Cytotec Misoprostol Asli 100% Ampuh Hanya 3 Jam Langsung Gugur || OBAT PENGGUGUR KANDUNGAN AMPUH MANJUR OBAT ABORSI OLINE" APOTIK Jual Obat Cytotec, Gastrul, Gynecoside Asli Ampuh. JUAL ” Obat Aborsi Tuntas | Obat Aborsi Manjur | Obat Aborsi Ampuh | Obat Penggugur Janin | Obat Pencegah Kehamilan | Obat Pelancar Haid | Obat terlambat Bulan | Ciri Obat Aborsi Asli | Obat Telat Bulan | Pil Aborsi Asli | Cara Menggugurkan Konten | Cara Aborsi Tuntas | Harga Obat Aborsi Asli | Pil Aborsi | Jual Obat Aborsi Cytotec | Cara Aborsi Sendiri | Cara Aborsi Usia 1 Bulan | Cara Aborsi Usia 2 Tahun | Cara Aborsi Usia 3 Bulan | Obat Aborsi Usia 4 Bulan | Cara Abrasi Usia 5 Bulan | Cara Menggugurkan Konten | Kandungan Obat Penggugur | Cara Menghitung Usia Konten | Cara Mengatasi Terlambat Bulan | Penjual Obat Aborsi Asli | Obat Aborsi Garansi | Kandungan Obat Peluntur | Obat Telat Datang Bulan | Obat Telat Haid | Obat Aborsi Paling Murah | Klinik Jual Obat Aborsi | Jual Pil Cytotec | Apotik Jual Obat Aborsi | Kandungan Dokter Abrasi | Cara Aborsi Cepat | Jual Obat Aborsi Bergaransi | Jual Obat Cytotec Asli | Obat Aborsi Aman Manjur | Obat Misoprostol Cytotec Asli. "APA ITU ABORSI" “Aborsi Adalah dengan membendung hormon yang di perlukan untuk mempertahankan kehamilan yaitu hormon progesteron, karena hormon ini dibendung, maka jalur kehamilan mulai membuka dan leher rahim menjadi melunak,sehingga mengeluarkan darah yang merupakan tanda bahwa obat telah bekerja || maksimal 1 jam obat diminum || PENJELASAN OBAT ABORSI USIA 1 _7 BULAN Pada usia kandungan ini, pasien akan merasakan sakit yang sedikit tidak berlebihan || sekitar 1 jam ||. namun hanya akan terjadi pada saatdarah keluar merupakan pertanda menstruasi. Hal ini dikarenakan pada usiakandungan 3 bulan,janin sudah terbentuk sebesar kepalan tangan orang dewasa. Cara kerja obat aborsi : JUAL OBAT ABORSI AMPUH dosis 3 bulan secara umum sama dengan cara kerja || DOSIS OBAT ABORSI 2 bulan”, hanya berbedanya selain mengisolasijanin juga menghancurkan janin dengan formula methotrexate dikandungdidalamnya. Formula methotrexate ini sangat ampuh untuk menghancurkan janinmenjadi serpihan-serpihan kecil akan sangat berguna pada saat dikeluarkan nanti. APA ALASAN WANITA MELAKUKAN ABORSI? Aborsi di lakukan wanita hamil baik yang sudah menikah maupun belum menikah dengan berbagai alasan , akan tetapi alasan yang utama adalah alasan-alasan non medis (termasuk aborsi sendiri / di sengaja/ buatan] MELAYANI PEMESANAN OBAT ABORSI SETIAP HARI, SIAP KIRIM KESELURUH KOTA BESAR DI INDONESIA DAN LUAR NEGERI. HUBUNGI PEMESANAN LEBIH NYAMAN VIA WA/: 085657271886

Jual Obat Aborsi Hongkong ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...

ZurliaSoop

FSB Advising Checklist - Orientation 2024

Elizabeth Walsh

SOC 101 Demonstration of Learning Presentation

camerronhm

How to setup Pycharm environment for Odoo 17.pptx

Celine George

Python Notes for mca i year students osmania university.docx

Ramakrishna Reddy Bijjam

Unit 3 Emotional Intelligence and Spiritual Intelligence.pdf

Dr Vijay Vishwakarma

Interdisciplinary_Insights_Data_Collection_Methods.pptx

Pooja Bhuva

80 ĐỀ THI THỬ TUYỂN SINH TIẾNG ANH VÀO 10 SỞ GD – ĐT THÀNH PHỐ HỒ CHÍ MINH NĂ...

Nguyen Thanh Tu Collection

Spatium Project Simulation student brief

Association for Project Management

SKILL OF INTRODUCING THE LESSON MICRO SKILLS.pptx

Amanpreet Kaur

The basics of sentences session 3pptx.pptx

heathfieldcps1

Making communications land - Are they received and understood as intended? webinar Thursday 2 May 2024 A joint webinar created by the APM Enabling Change and APM People Interest Networks, this is the third of our three part series on Making Communications Land. presented by Ian Cribbes, Director, IMC&T Ltd @cribbesheet The link to the write up page and resources of this webinar: https://www.apm.org.uk/news/making-communications-land-are-they-received-and-understood-as-intended-webinar/ Content description: How do we ensure that what we have communicated was received and understood as we intended and how do we course correct if it has not.

Making communications land - Are they received and understood as intended? we...

Association for Project Management

How to Manage Global Discount in Odoo 17 POS

Celine George

Fostering Friendships - Enhancing Social Bonds in the Classroom

Pooky Knightsmith

Single or Multiple melodic lines structure

dhanjurrannsibayan2

Jamworks pilot and AI at Jisc (20/03/2024)

Jisc

Basic Civil Engineering notes first year Notes Building notes Selection of site for Building Layout of a Building What is Burjis, Mutam Building Bye laws Basic Concept of sunlight ventilation in building National Building Code of India Set back or building line Types of Buildings Floor Space Index (F.S.I) Institutional Vs Educational Building Components & function Sills, Lintels, Cantilever Doors, Windows and Ventilators Types of Foundation AND THEIR USES Plinth Area Shallow and Deep Foundation Super Built-up & carpet area Floor Area Ratio (F.A.R) RCC Reinforced Cement Concrete RCC VS PCC

Basic Civil Engineering first year Notes- Chapter 4 Building.pptx

Denish Jangid

Micro-Scholarship, What it is, How can it help me.pdf

Poh-Sun Goh

Introduction to Nonprofit Accounting: The Basics

TechSoup

Amee 2009 7 E5 JCH

1. Introduction of Standard Setting in undergraduate medical education: re-emphasizing the importance of item-writing. Jane Holland, Alice McGarvey, Richard Arnett, David Croke Royal College of Surgeons in Ireland

2. Standard setting – preparation for pilot Module selection Nutrition & Energy Junior Cycle Semester 1 n = 281 19 - anatomy, biochemistry pharmacology, physiology Lecturers / Judges Examination Multiple choice, n = 30 Method Modified Angoff (1971) Jane Holland, RCSI

3. Standard setting – training of judges 59.2 60.7 Workshop Dr Katherine Boursicot November 2007 Principles & specific methods Exercise 10 MCQs (JC1 NE 2007) Jane Holland, RCSI

4. JC1 NE 2008: Modified Angoff “Classic” Angoff MCQ cut score 50 58 MCQ Class mean 63.6 54.6 Overall class mean 61.1 59 Fails 12 15.4 14 judges 30 MCQs 3 ½ hours!! *All tabular values are percentages Jane Holland, RCSI

5. JC1 NE 2008: Hofstee 57 % Jane Holland, RCSI

6. RCSI Curriculum Forum, 29th March 2008 Jane Holland, RCSI

7. Item writing – training of writers Workshop Dave Swanson / Kathy Holtzman April 2008 Guidelines NBME / MRCS Jane Holland, RCSI

8. Item writing – pilot within NE module 13 unusable 64 suitable! 7 -ve 4 T/F 2 misc Call for questions 60 MCQs required 71 MCQs requested Guidelines / SBA stems 77 MCQs received 83% Jane Holland, RCSI

9. JC1 NE 2009: Modified Angoff 12 judges 30 MCQs Cut-score = 48 Item difficulties DIF IDEAL Consortium Item Analysis package Item probabilities Prob Modified Angoff (1971) Jane Holland, RCSI

10. JC1 NE 2008: DIF vs Prob Prob KR-20 = 0.735 DIF rs = 0.277 (p = 0.13) Jane Holland, RCSI

11. JC1 NE 2009: DIF vs Prob Prob KR-20 = 0.769 DIF rs = 0.599 (p = 0.001) Jane Holland, RCSI

12. Summary of results & take home message 2008 2009 MCQ cut score 58 48 MCQ Class mean 63.6 63.0 Item difficulties vs. item probabilities (rs) 0.277 (p=0.13) 0.599 (p=0.001) A well-structured item, while designed to more accurately assess the candidate’s knowledge of the subject material, also enables more accurate rating of items during the standard setting process and is an important consideration when introducing these procedures de novo. Jane Holland, RCSI

Amee 2009 7 E5 JCH

Recomendados

Recomendados

Más contenido relacionado

Similar a Amee 2009 7 E5 JCH

Similar a Amee 2009 7 E5 JCH (20)

Último

Último (20)

Amee 2009 7 E5 JCH