SlideShare una empresa de Scribd logo
1 de 90
Descargar para leer sin conexión
TESSERACT

            Educational


                    Nursing Workshop 2/12/2013
Multiple Choice
Item Writing Workshop

               Doug Mauger EdD
               School Of Instructor Education
               dmauger@vcc.ca

                             Nursing Workshop 2/12/2013
Workshop Agenda
•   Introductions
•   Overview
•   Perspectives and Attitudes
•   Stakeholders
•   Validity and Reliability
•   Alignment
•   Classifying skills and outcomes
•   Best Practices in MC item writing
•   Table of Specification or Blueprint
•   Take Away
•   Next Workshop



                                          Nursing Workshop 2/12/2013
Introductions
• Name:
• What topics do you teach?
• How long have you been teaching?
• One thing you know about Item writing
• What do you want to get out of the workshop?



                                          Nursing Workshop 2/12/2013
Workshop Agenda
• Introductions
• Overview
• Perspectives and Attitudes
•   Stakeholders
•   Valdity and Reliability
•   Alignment
•   Classifying skills and outcomes
•   Best Practices in MC item writing
•   Table of Specification or Blueprint
•   Take Away
•   Next Workshop



                                          Nursing Workshop 2/12/2013
Put each
 rider on
  the top
     of a
    horse
            Nursing Workshop 2/12/2013
Put each
 rider on
  the top
     of a
    horse
            Nursing Workshop 2/12/2013
Put each
 rider on
  the top
     of a
    horse
            Nursing Workshop 2/12/2013
Put each
 rider on
  the top
     of a
    horse
            Nursing Workshop 2/12/2013
Put each
 rider on
  the top
     of a
    horse
            Nursing Workshop 2/12/2013
Nursing Workshop 2/12/2013
Nursing Workshop 2/12/2013
What are your
attitudes to
creating exams?




        Nursing Workshop 2/12/2013
Workshop Agenda
• Introductions
• Overview
• Perspectives and Attitudes
• Stakeholders
•   Validity and Reliability
•   Alignment
•   Classifying skills and outcomes
•   Best Practices in MC item writing
•   Table of Specification or Blueprint
•   Take Away
•   Next Workshop



                                          Nursing Workshop 2/12/2013
Stakeholders Gathering




                   Nursing Workshop 2/12/2013
Workshop Agenda
•   Introductions
•   Overview
•   Perspectives and Attitudes
•   Stakeholders
• Validity and Reliability
•   Alignment
•   Classifying skills and outcomes
•   Best Practices in MC item writing
•   Table of Specification or Blueprint
•   Take Away
•   Next Workshop



                                          Nursing Workshop 2/12/2013
Instrument……Confidence




                         Nursing Workshop 2/12/2013
Points to Ponder

   There were flaws in 46% of the total
   questions, with the number of flaws per
   test ranging from 36% to 65%. In a
   sample of examinations distributed to
   first and second year medical students
   (Downing, 2005)

 Downing, S. M., 2005. The effects of violating standard item writing principles on tests and students: the con- sequences of using flawed test items on
 achievement ex- aminations in medical education. Advances in Health Science Education 10 (2), 133–143.



                                                                                                                                          Nursing Workshop 2/12/2013
Process Validity
    Activity



               Nursing Workshop 2/12/2013
Validity and Reliability




                           Nursing Workshop 2/12/2013
Reliable but Not Valid




                         Nursing Workshop 2/12/2013
NOT Valid and NOT Reliable




                             Nursing Workshop 2/12/2013
Valid and Reliable




                     Nursing Workshop 2/12/2013
Validity
            Content
           Predictive
           Construct
       Consequences
            Process

                 Nursing Workshop 2/12/2013
Reliable
       Test – Retest
      Alternate Form
        Inter Rater




              Nursing Workshop 2/12/2013
Types of Validity
    Activity



                Nursing Workshop 2/12/2013
1. Process, Content Validity (inadequate sample)

2. Process Validity (stress)

3. Content Validity (inadequate preparation)

4. Predictive Validity (inconsistent results)

5. Consequence Validity (induced anxiety)

6. Reliability/Process Validity (possible bias)


                                                  Nursing Workshop 2/12/2013
7. Process Validity (students not able to provide best
performance)

8. Content Validity (different instructor’s test reflects
different instruction)

9. Construct Validity (improper form of assessment)

10. Content Validity/Alignment (instruction and
assessment do not match)



                                                  Nursing Workshop 2/12/2013
Why is Validity and
Reliability important
to you as a
professional?




             Nursing Workshop 2/12/2013
Workshop Agenda
•   Introductions
•   Overview
•   Perspectives and Attitudes
•   Stakeholders
•   Validity and Reliability
• Alignment
•   Classifying skills and outcomes
•   Best Practices in MC item writing
•   Table of Specification or Blueprint
•   Take Away
•   Next Workshop



                                          Nursing Workshop 2/12/2013
Alignment

 Content and Assessment
       Alignment
      Are Critical
       For Valid
      Assessment

                   Nursing Workshop 2/12/2013
Intended
Curriculum

WHAT we plan
on teaching
Implemented
Curriculum
HOW we teach
the curriculum
Achieved
Curriculum
Assessment
Evidence of
Student
Learning
Outcomes:            Intended        Implemented      Achieved


                        What             How          Assessment
Complexity of content




                                   Perfect Alignment
Complexity of content
                        What     How    Assessment




                               Scenario 1
Complexity of content
                        What     How    Assessment




                               Scenario 2
Complexity of content
                        What     How    Assessment




                               Scenario 3
Complexity of content
                        What     How    Assessment




                               Scenario 4
Complexity of content
                        What     How    Assessment




                               Scenario 5
Complexity of content
                        What     How    Assessment




                               Scenario 6
Complexity of content
                        What     How    Assessment




                               Scenario 7
Complexity of content
                        What     How    Assessment




                               Scenario 8
Complexity of content
                        What        How     Assessment




                               Perfect Alignment
How has Alignment
  Impacted You?




               Nursing Workshop 2/12/2013
Workshop Agenda
•   Introductions
•   Overview
•   Perspectives and Attitudes
•   Stakeholders
•   Valdity and Reliability
•   Alignment
• Classifying skills and outcomes
•   Best Practices in MC item writing
•   Table of Specification or Blueprint
•   Take Away
•   Next Workshop



                                          Nursing Workshop 2/12/2013
Nursing Workshop 2/12/2013
Blooms Taxonomy revised




                          Nursing Workshop 2/12/2013
Sorting
  skills/outcomes
         from
Simple to Complex

               Nursing Workshop 2/12/2013
Blooms Taxonomy revised




                          Nursing Workshop 2/12/2013
Remembering/Understanding Level


       Describes the circulation through the heart

       Defines the term systole




                                         Nursing Workshop 2/12/2013
Application Level
Application (AP)
theories in providing care to clients (e.g., a




            Apply the concepts of aging in
            developing interventions for the
            elderly



                                                 Nursing Workshop 2/12/2013
Critical Thinking Level revised
Blooms Taxonomy




    Develop a plan for delivering services to elderly persons who
    are homebound

    Evaluate nursing research based on predetermined criteria

    Analyzes the organizational structure of the community health
    agency and its impact on client services.

                                                        Nursing Workshop 2/12/2013
Sorting
skills/outcomes
       into
3 classifications

                Nursing Workshop 2/12/2013
Four Knowledge Dimensions
 Factual Knowledge
 The basic elements students must know to be acquainted with a discipline or
 solve problems in it.

 Conceptual Knowledge
 The interrelationships among the basic elements within a larger structure
 that enable them to function together.

 Procedural Knowledge
 How to do something, methods of inquiry, and criteria for using skills,
 algorithms, techniques, and methods.

 Meta-Cognitive Knowledge
 Knowledge of cognition in general as well as awareness and knowledge of
 one's own cognition


                                                                     Nursing Workshop 2/12/2013
Verbs




        Nursing Workshop 2/12/2013
Nursing Workshop 2/12/2013
Identify
  skills/outcomes
         By
Knowledge Dimension &
   Cognitive Process
                  Nursing Workshop 2/12/2013
Writing a MC Item
             What is the biggest
             challenge for you in sorting
             outcomes?




                                 Nursing Workshop 2/12/2013
Workshop Agenda
•   Introductions
•   Overview
•   Perspectives and Attitudes
•   Stakeholders
•   Valdity and Reliability
•   Alignment
•   Classifying skills and outcomes
• Best Practices in MC item writing
• Table of Specification or Blueprint
• Take Away
• Next Workshop



                                        Nursing Workshop 2/12/2013
Writing a MC Item




                    Nursing Workshop 2/12/2013
Writing a MC Item




                    Nursing Workshop 2/12/2013
MC Item Anatomy




                  Nursing Workshop 2/12/2013
Steps to writing a MC Item item

 1.   Selecting the competency/outcome
 2.   Construct the stem
 3.   Write plausible distractors
 4.   Peer edit
 5.   Item analysis




                                         Nursing Workshop 2/12/2013
ATOA and NOTA




                Nursing Workshop 2/12/2013
Points to Ponder Item
Constructing a MC
   Results showed that when “all-of-the-above” is
   used as the correct answer, the item is more
   difficult for all students, despite the literature
   assumption that it provides a cueing effect to
   students. Research findings corroborate
   literature assumptions that high ability students
   score significantly higher than other ability
   students in this type of option.

  Huang, Yi Min, Trevisan, Michae lThe Impact of the “all-of-the-above” Option and Student Ability on Multiple Choice Tests International Journal for the
  Scholarship of Teaching and Learning
  http://www.georgiasouthern.edu/ijsotl Vol. 1, No. 2 (July 2007) ISSN 1931-4744 © Georgia Southern University

                                                                                                                                     Nursing Workshop 2/12/2013
Constructing a MC Item




  Huang, Yi Min, Trevisan, Michae lThe Impact of the “all-of-the-above” Option and Student Ability on Multiple Choice Tests International Journal for the
  Scholarship of Teaching and Learning
  http://www.georgiasouthern.edu/ijsotl Vol. 1, No. 2 (July 2007) ISSN 1931-4744 © Georgia Southern University

                                                                                                                                     Nursing Workshop 2/12/2013
Points to Ponder Item
Constructing a MC
 The present study examined the effect of NOTA on multiple choice
 questions in a large classroom setting. Specifically, we
 manipulated the place- ment of NOTA across versions of the test
 such that, for each of 20 experimental questions, NOTA was either
 not present, replaced the key, or re- placed the high, medium, or
 low frequency dis- tractors respectively. We found that NOTA sig-
 nificantly increased the difficulty of multiple choice questions only
 when it was used as the key. We found no effect of NOTA replacing
 any of the dis- tractors, regardless of their effectiveness. Addi-
 tionally, we found no significant effect of NOTA on discrimination
 scores, regardless of its place- ment.


 2010 Matthew V. Pachaia, David DiBattistab, Joseph: The effect of ’None of the Above’ on multiple choice questions in a first year classroom A Preprint
 submitted to Advances in Health Science Education

                                                                                                                                     Nursing Workshop 2/12/2013
Constructing a MC Item




                         Nursing Workshop 2/12/2013
Correct or Best
  Correct Answer




  Best Answer




                   Nursing Workshop 2/12/2013
Nursing Workshop 2/12/2013
Content
Free Test
        Nursing Workshop 2/12/2013
Varieties of MC Items


   1. Single Correct Answer
   2. Multiple Response
   3. Combined Response
   4. Multiple Response


                              Nursing Workshop 2/12/2013
Writing a MC Item

 1. Design each item to measure an important learning
    outcome
 2. Present a single clearly formulated problem in the
    stem of the item
 3. State the stem of the item in simple, clear language
 4. State the stem in positive form wherever possible
 5. Word the alternatives clearly and concisely
 6. Keep the alternatives mutually exclusive



                                                Nursing Workshop 2/12/2013
Writing a MC Item

 7. Keep the alternatives homogeneous in content
 8. Keep the alternatives free from clues as to which
     response is correct
 9. Avoid the alternatives AOTA and NOTA
 10. Use as many functional distractors as are feasible.
 11. Include one and only one correct or clearly best
     answer for each item




                                                  Nursing Workshop 2/12/2013
Writing a MC Item
Writing a MC Item
 12. Present the answer in each of the alternative
     positions approximately an equal number of
     times, in a random order
 13. Layout the items in a clear and consistent manner
 14. Use proper grammar, punctuation, and spelling
 15. Avoid using unnecessarily difficult vocabulary
 16. Analyze the effectiveness of each item after each
     administration of the test.




                                                Nursing Workshop 2/12/2013
Rules Broken
     in the
Content Free Test


               Nursing Workshop 2/12/2013
Writing a MC Item
               Item writing……




                           Nursing Workshop 2/12/2013
Workshop Agenda
•   Introductions
•   Overview
•   Perspectives and Attitudes
•   Stakeholders
•   Valdity and Reliability
•   Alignment
•   Classifying skills and outcomes
•   Best Practices in MC item writing
• Table of Specification or Blueprint
• Take Away
• Next Workshop



                                        Nursing Workshop 2/12/2013
Blueprint or TOS




                   Nursing Workshop 2/12/2013
Remembering/Understanding Level




                                  Nursing Workshop 2/12/2013
Application Level
Application (AP)
theories in providing care to clients (e.g., a




                                                 Nursing Workshop 2/12/2013
Critical Thinking Level revised
Blooms Taxonomy




                                  Nursing Workshop 2/12/2013
Blueprint or TOS




                   Nursing Workshop 2/12/2013
Workshop Agenda
•   Introductions
•   Overview
•   Perspectives and Attitudes
•   Stakeholders
•   Valdity and Reliability
•   Alignment
•   Classifying skills and outcomes
•   Best Practices in MC item writing
•   Table of Specification or Blueprint
• Take Away
• Next Workshop



                                          Nursing Workshop 2/12/2013
Writing a MC Item


            What
            So what
            Now what
                    Nursing Workshop 2/12/2013
Writing a MC Item




                    Nursing Workshop 2/12/2013
Workshop Agenda
•   Introductions
•   Overview
•   Perspectives and Attitudes
•   Stakeholders
•   Valdity and Reliability
•   Alignment
•   Classifying skills and outcomes
•   Best Practices in MC item writing
•   Table of Specification or Blueprint
•   Take Away
• Next Workshop

                                          Nursing Workshop 2/12/2013
Writing a MC Item
   Thank You




                    Nursing Workshop 2/12/2013
ReferencesKrathwohl, David R. (2001). A Taxonomy for
  Anderson, Lorin W. &
  Learning, Teaching and Assessing: a Revision of Bloom’s Taxonomy.
    New York. Longman Publishing.

     http://shpliteracy.pbworks.com/w/file/fetch/54533674/NEW%20Blooms%20Taxono
     my%20Question%20Stems.pdf




                                                                      Nursing Workshop 2/12/2013

Más contenido relacionado

La actualidad más candente

HCS 451 Remember Education--hcs451.com
HCS 451 Remember Education--hcs451.comHCS 451 Remember Education--hcs451.com
HCS 451 Remember Education--hcs451.comJaseetha61
 
Progress report, Utang na Loob
Progress report, Utang na LoobProgress report, Utang na Loob
Progress report, Utang na LoobMiriam College
 
Documenting Your Teaching Efforts in a Way that Counts
Documenting Your Teaching Efforts in a Way that CountsDocumenting Your Teaching Efforts in a Way that Counts
Documenting Your Teaching Efforts in a Way that Countstatetomika
 
Advancing the Methods of Evaluation of Quality and Safety Practice and Educa...
Advancing the Methods of Evaluation of Quality and Safety Practice and Educa...Advancing the Methods of Evaluation of Quality and Safety Practice and Educa...
Advancing the Methods of Evaluation of Quality and Safety Practice and Educa...Daniel McLinden
 
Evidence to action: Why TESTA works
Evidence to action: Why TESTA worksEvidence to action: Why TESTA works
Evidence to action: Why TESTA worksTansy Jessop
 

La actualidad más candente (6)

HCS 451 Remember Education--hcs451.com
HCS 451 Remember Education--hcs451.comHCS 451 Remember Education--hcs451.com
HCS 451 Remember Education--hcs451.com
 
Progress report, Utang na Loob
Progress report, Utang na LoobProgress report, Utang na Loob
Progress report, Utang na Loob
 
Trainee in Difficulty
Trainee in DifficultyTrainee in Difficulty
Trainee in Difficulty
 
Documenting Your Teaching Efforts in a Way that Counts
Documenting Your Teaching Efforts in a Way that CountsDocumenting Your Teaching Efforts in a Way that Counts
Documenting Your Teaching Efforts in a Way that Counts
 
Advancing the Methods of Evaluation of Quality and Safety Practice and Educa...
Advancing the Methods of Evaluation of Quality and Safety Practice and Educa...Advancing the Methods of Evaluation of Quality and Safety Practice and Educa...
Advancing the Methods of Evaluation of Quality and Safety Practice and Educa...
 
Evidence to action: Why TESTA works
Evidence to action: Why TESTA worksEvidence to action: Why TESTA works
Evidence to action: Why TESTA works
 

Similar a MC Item Writing Workshop

ASSESSMENT-EDITED slide share.pptx
ASSESSMENT-EDITED slide share.pptxASSESSMENT-EDITED slide share.pptx
ASSESSMENT-EDITED slide share.pptxbeminaja
 
Rubrics - Research and Enterprise Committee presentation (UoN)
Rubrics - Research and Enterprise Committee presentation (UoN)Rubrics - Research and Enterprise Committee presentation (UoN)
Rubrics - Research and Enterprise Committee presentation (UoN)Adel Gordon
 
How to Benefit from Meaningful Learning Assessments and Evaluations
How to Benefit from Meaningful Learning Assessments and EvaluationsHow to Benefit from Meaningful Learning Assessments and Evaluations
How to Benefit from Meaningful Learning Assessments and EvaluationsLambda Solutions
 
Work Based Projects
Work Based ProjectsWork Based Projects
Work Based Projectsmdxaltc
 
Designing Your Programme (May 2015)
Designing Your Programme (May 2015)Designing Your Programme (May 2015)
Designing Your Programme (May 2015)AQD Winchester
 
Fostering a culture change in assessment and feedback through TESTA
Fostering a culture change in assessment and feedback through TESTAFostering a culture change in assessment and feedback through TESTA
Fostering a culture change in assessment and feedback through TESTATansy Jessop
 
Rubric\'s Cube--Complimenting, Critiquing, and Challenging Student Work (NELB...
Rubric\'s Cube--Complimenting, Critiquing, and Challenging Student Work (NELB...Rubric\'s Cube--Complimenting, Critiquing, and Challenging Student Work (NELB...
Rubric\'s Cube--Complimenting, Critiquing, and Challenging Student Work (NELB...Mark Eutsler
 
Classroom diagnostic tools training 9.23.14
Classroom diagnostic tools training 9.23.14Classroom diagnostic tools training 9.23.14
Classroom diagnostic tools training 9.23.14nickpaolini81
 
Evaluating Student Success Initiatives
Evaluating Student Success InitiativesEvaluating Student Success Initiatives
Evaluating Student Success InitiativesBradley Vaden
 
Evaluating Student Success Initiatives
Evaluating Student Success InitiativesEvaluating Student Success Initiatives
Evaluating Student Success Initiatives3CSN
 
Assessing students in clinical practice
Assessing students in clinical practiceAssessing students in clinical practice
Assessing students in clinical practicesgray2
 
CHAPTER 4 Assessing Student Learning Outcomes.pptx
CHAPTER 4 Assessing Student Learning Outcomes.pptxCHAPTER 4 Assessing Student Learning Outcomes.pptx
CHAPTER 4 Assessing Student Learning Outcomes.pptxRhumJezzCabElp
 
Compare addie vs dick and carey model
Compare addie vs dick and carey modelCompare addie vs dick and carey model
Compare addie vs dick and carey modelMichelleDiaz90
 

Similar a MC Item Writing Workshop (20)

Creating Meaningful Rubrics
Creating Meaningful Rubrics Creating Meaningful Rubrics
Creating Meaningful Rubrics
 
ASSESSMENT-EDITED slide share.pptx
ASSESSMENT-EDITED slide share.pptxASSESSMENT-EDITED slide share.pptx
ASSESSMENT-EDITED slide share.pptx
 
Rubrics - Research and Enterprise Committee presentation (UoN)
Rubrics - Research and Enterprise Committee presentation (UoN)Rubrics - Research and Enterprise Committee presentation (UoN)
Rubrics - Research and Enterprise Committee presentation (UoN)
 
How to Benefit from Meaningful Learning Assessments and Evaluations
How to Benefit from Meaningful Learning Assessments and EvaluationsHow to Benefit from Meaningful Learning Assessments and Evaluations
How to Benefit from Meaningful Learning Assessments and Evaluations
 
Work Based Projects
Work Based ProjectsWork Based Projects
Work Based Projects
 
How Can We Use Assessment Techniques to Enhance the Learning of Pareducators?...
How Can We Use Assessment Techniques to Enhance the Learning of Pareducators?...How Can We Use Assessment Techniques to Enhance the Learning of Pareducators?...
How Can We Use Assessment Techniques to Enhance the Learning of Pareducators?...
 
Designing Your Programme (May 2015)
Designing Your Programme (May 2015)Designing Your Programme (May 2015)
Designing Your Programme (May 2015)
 
Fostering a culture change in assessment and feedback through TESTA
Fostering a culture change in assessment and feedback through TESTAFostering a culture change in assessment and feedback through TESTA
Fostering a culture change in assessment and feedback through TESTA
 
Rubric\'s Cube--Complimenting, Critiquing, and Challenging Student Work (NELB...
Rubric\'s Cube--Complimenting, Critiquing, and Challenging Student Work (NELB...Rubric\'s Cube--Complimenting, Critiquing, and Challenging Student Work (NELB...
Rubric\'s Cube--Complimenting, Critiquing, and Challenging Student Work (NELB...
 
RUBRICS.pptx
RUBRICS.pptxRUBRICS.pptx
RUBRICS.pptx
 
Classroom diagnostic tools training 9.23.14
Classroom diagnostic tools training 9.23.14Classroom diagnostic tools training 9.23.14
Classroom diagnostic tools training 9.23.14
 
Evaluating Student Success Initiatives
Evaluating Student Success InitiativesEvaluating Student Success Initiatives
Evaluating Student Success Initiatives
 
Evaluating Student Success Initiatives
Evaluating Student Success InitiativesEvaluating Student Success Initiatives
Evaluating Student Success Initiatives
 
Assessing students in clinical practice
Assessing students in clinical practiceAssessing students in clinical practice
Assessing students in clinical practice
 
CHAPTER 4 Assessing Student Learning Outcomes.pptx
CHAPTER 4 Assessing Student Learning Outcomes.pptxCHAPTER 4 Assessing Student Learning Outcomes.pptx
CHAPTER 4 Assessing Student Learning Outcomes.pptx
 
Module 4, ed 103
Module 4, ed 103Module 4, ed 103
Module 4, ed 103
 
Assessment
AssessmentAssessment
Assessment
 
CET Unit 8 Assessment
CET Unit 8 AssessmentCET Unit 8 Assessment
CET Unit 8 Assessment
 
Unit 3- Topic 3.pptx
Unit 3- Topic 3.pptxUnit 3- Topic 3.pptx
Unit 3- Topic 3.pptx
 
Compare addie vs dick and carey model
Compare addie vs dick and carey modelCompare addie vs dick and carey model
Compare addie vs dick and carey model
 

MC Item Writing Workshop

  • 1. TESSERACT Educational Nursing Workshop 2/12/2013
  • 2. Multiple Choice Item Writing Workshop Doug Mauger EdD School Of Instructor Education dmauger@vcc.ca Nursing Workshop 2/12/2013
  • 3. Workshop Agenda • Introductions • Overview • Perspectives and Attitudes • Stakeholders • Validity and Reliability • Alignment • Classifying skills and outcomes • Best Practices in MC item writing • Table of Specification or Blueprint • Take Away • Next Workshop Nursing Workshop 2/12/2013
  • 4. Introductions • Name: • What topics do you teach? • How long have you been teaching? • One thing you know about Item writing • What do you want to get out of the workshop? Nursing Workshop 2/12/2013
  • 5. Workshop Agenda • Introductions • Overview • Perspectives and Attitudes • Stakeholders • Valdity and Reliability • Alignment • Classifying skills and outcomes • Best Practices in MC item writing • Table of Specification or Blueprint • Take Away • Next Workshop Nursing Workshop 2/12/2013
  • 6. Put each rider on the top of a horse Nursing Workshop 2/12/2013
  • 7. Put each rider on the top of a horse Nursing Workshop 2/12/2013
  • 8. Put each rider on the top of a horse Nursing Workshop 2/12/2013
  • 9. Put each rider on the top of a horse Nursing Workshop 2/12/2013
  • 10. Put each rider on the top of a horse Nursing Workshop 2/12/2013
  • 13. What are your attitudes to creating exams? Nursing Workshop 2/12/2013
  • 14. Workshop Agenda • Introductions • Overview • Perspectives and Attitudes • Stakeholders • Validity and Reliability • Alignment • Classifying skills and outcomes • Best Practices in MC item writing • Table of Specification or Blueprint • Take Away • Next Workshop Nursing Workshop 2/12/2013
  • 15. Stakeholders Gathering Nursing Workshop 2/12/2013
  • 16. Workshop Agenda • Introductions • Overview • Perspectives and Attitudes • Stakeholders • Validity and Reliability • Alignment • Classifying skills and outcomes • Best Practices in MC item writing • Table of Specification or Blueprint • Take Away • Next Workshop Nursing Workshop 2/12/2013
  • 17. Instrument……Confidence Nursing Workshop 2/12/2013
  • 18. Points to Ponder There were flaws in 46% of the total questions, with the number of flaws per test ranging from 36% to 65%. In a sample of examinations distributed to first and second year medical students (Downing, 2005) Downing, S. M., 2005. The effects of violating standard item writing principles on tests and students: the con- sequences of using flawed test items on achievement ex- aminations in medical education. Advances in Health Science Education 10 (2), 133–143. Nursing Workshop 2/12/2013
  • 19. Process Validity Activity Nursing Workshop 2/12/2013
  • 20. Validity and Reliability Nursing Workshop 2/12/2013
  • 21. Reliable but Not Valid Nursing Workshop 2/12/2013
  • 22. NOT Valid and NOT Reliable Nursing Workshop 2/12/2013
  • 23. Valid and Reliable Nursing Workshop 2/12/2013
  • 24. Validity Content Predictive Construct Consequences Process Nursing Workshop 2/12/2013
  • 25. Reliable Test – Retest Alternate Form Inter Rater Nursing Workshop 2/12/2013
  • 26. Types of Validity Activity Nursing Workshop 2/12/2013
  • 27. 1. Process, Content Validity (inadequate sample) 2. Process Validity (stress) 3. Content Validity (inadequate preparation) 4. Predictive Validity (inconsistent results) 5. Consequence Validity (induced anxiety) 6. Reliability/Process Validity (possible bias) Nursing Workshop 2/12/2013
  • 28. 7. Process Validity (students not able to provide best performance) 8. Content Validity (different instructor’s test reflects different instruction) 9. Construct Validity (improper form of assessment) 10. Content Validity/Alignment (instruction and assessment do not match) Nursing Workshop 2/12/2013
  • 29. Why is Validity and Reliability important to you as a professional? Nursing Workshop 2/12/2013
  • 30. Workshop Agenda • Introductions • Overview • Perspectives and Attitudes • Stakeholders • Validity and Reliability • Alignment • Classifying skills and outcomes • Best Practices in MC item writing • Table of Specification or Blueprint • Take Away • Next Workshop Nursing Workshop 2/12/2013
  • 31. Alignment Content and Assessment Alignment Are Critical For Valid Assessment Nursing Workshop 2/12/2013
  • 35. Outcomes: Intended Implemented Achieved What How Assessment Complexity of content Perfect Alignment
  • 36. Complexity of content What How Assessment Scenario 1
  • 37. Complexity of content What How Assessment Scenario 2
  • 38. Complexity of content What How Assessment Scenario 3
  • 39. Complexity of content What How Assessment Scenario 4
  • 40. Complexity of content What How Assessment Scenario 5
  • 41. Complexity of content What How Assessment Scenario 6
  • 42. Complexity of content What How Assessment Scenario 7
  • 43. Complexity of content What How Assessment Scenario 8
  • 44. Complexity of content What How Assessment Perfect Alignment
  • 45. How has Alignment Impacted You? Nursing Workshop 2/12/2013
  • 46. Workshop Agenda • Introductions • Overview • Perspectives and Attitudes • Stakeholders • Valdity and Reliability • Alignment • Classifying skills and outcomes • Best Practices in MC item writing • Table of Specification or Blueprint • Take Away • Next Workshop Nursing Workshop 2/12/2013
  • 48. Blooms Taxonomy revised Nursing Workshop 2/12/2013
  • 49. Sorting skills/outcomes from Simple to Complex Nursing Workshop 2/12/2013
  • 50. Blooms Taxonomy revised Nursing Workshop 2/12/2013
  • 51. Remembering/Understanding Level Describes the circulation through the heart Defines the term systole Nursing Workshop 2/12/2013
  • 52. Application Level Application (AP) theories in providing care to clients (e.g., a Apply the concepts of aging in developing interventions for the elderly Nursing Workshop 2/12/2013
  • 53. Critical Thinking Level revised Blooms Taxonomy Develop a plan for delivering services to elderly persons who are homebound Evaluate nursing research based on predetermined criteria Analyzes the organizational structure of the community health agency and its impact on client services. Nursing Workshop 2/12/2013
  • 54. Sorting skills/outcomes into 3 classifications Nursing Workshop 2/12/2013
  • 55. Four Knowledge Dimensions Factual Knowledge The basic elements students must know to be acquainted with a discipline or solve problems in it. Conceptual Knowledge The interrelationships among the basic elements within a larger structure that enable them to function together. Procedural Knowledge How to do something, methods of inquiry, and criteria for using skills, algorithms, techniques, and methods. Meta-Cognitive Knowledge Knowledge of cognition in general as well as awareness and knowledge of one's own cognition Nursing Workshop 2/12/2013
  • 56. Verbs Nursing Workshop 2/12/2013
  • 58. Identify skills/outcomes By Knowledge Dimension & Cognitive Process Nursing Workshop 2/12/2013
  • 59. Writing a MC Item What is the biggest challenge for you in sorting outcomes? Nursing Workshop 2/12/2013
  • 60. Workshop Agenda • Introductions • Overview • Perspectives and Attitudes • Stakeholders • Valdity and Reliability • Alignment • Classifying skills and outcomes • Best Practices in MC item writing • Table of Specification or Blueprint • Take Away • Next Workshop Nursing Workshop 2/12/2013
  • 61. Writing a MC Item Nursing Workshop 2/12/2013
  • 62. Writing a MC Item Nursing Workshop 2/12/2013
  • 63. MC Item Anatomy Nursing Workshop 2/12/2013
  • 64. Steps to writing a MC Item item 1. Selecting the competency/outcome 2. Construct the stem 3. Write plausible distractors 4. Peer edit 5. Item analysis Nursing Workshop 2/12/2013
  • 65. ATOA and NOTA Nursing Workshop 2/12/2013
  • 66. Points to Ponder Item Constructing a MC Results showed that when “all-of-the-above” is used as the correct answer, the item is more difficult for all students, despite the literature assumption that it provides a cueing effect to students. Research findings corroborate literature assumptions that high ability students score significantly higher than other ability students in this type of option. Huang, Yi Min, Trevisan, Michae lThe Impact of the “all-of-the-above” Option and Student Ability on Multiple Choice Tests International Journal for the Scholarship of Teaching and Learning http://www.georgiasouthern.edu/ijsotl Vol. 1, No. 2 (July 2007) ISSN 1931-4744 © Georgia Southern University Nursing Workshop 2/12/2013
  • 67. Constructing a MC Item Huang, Yi Min, Trevisan, Michae lThe Impact of the “all-of-the-above” Option and Student Ability on Multiple Choice Tests International Journal for the Scholarship of Teaching and Learning http://www.georgiasouthern.edu/ijsotl Vol. 1, No. 2 (July 2007) ISSN 1931-4744 © Georgia Southern University Nursing Workshop 2/12/2013
  • 68. Points to Ponder Item Constructing a MC The present study examined the effect of NOTA on multiple choice questions in a large classroom setting. Specifically, we manipulated the place- ment of NOTA across versions of the test such that, for each of 20 experimental questions, NOTA was either not present, replaced the key, or re- placed the high, medium, or low frequency dis- tractors respectively. We found that NOTA sig- nificantly increased the difficulty of multiple choice questions only when it was used as the key. We found no effect of NOTA replacing any of the dis- tractors, regardless of their effectiveness. Addi- tionally, we found no significant effect of NOTA on discrimination scores, regardless of its place- ment. 2010 Matthew V. Pachaia, David DiBattistab, Joseph: The effect of ’None of the Above’ on multiple choice questions in a first year classroom A Preprint submitted to Advances in Health Science Education Nursing Workshop 2/12/2013
  • 69. Constructing a MC Item Nursing Workshop 2/12/2013
  • 70. Correct or Best Correct Answer Best Answer Nursing Workshop 2/12/2013
  • 72. Content Free Test Nursing Workshop 2/12/2013
  • 73. Varieties of MC Items 1. Single Correct Answer 2. Multiple Response 3. Combined Response 4. Multiple Response Nursing Workshop 2/12/2013
  • 74. Writing a MC Item 1. Design each item to measure an important learning outcome 2. Present a single clearly formulated problem in the stem of the item 3. State the stem of the item in simple, clear language 4. State the stem in positive form wherever possible 5. Word the alternatives clearly and concisely 6. Keep the alternatives mutually exclusive Nursing Workshop 2/12/2013
  • 75. Writing a MC Item 7. Keep the alternatives homogeneous in content 8. Keep the alternatives free from clues as to which response is correct 9. Avoid the alternatives AOTA and NOTA 10. Use as many functional distractors as are feasible. 11. Include one and only one correct or clearly best answer for each item Nursing Workshop 2/12/2013
  • 76. Writing a MC Item Writing a MC Item 12. Present the answer in each of the alternative positions approximately an equal number of times, in a random order 13. Layout the items in a clear and consistent manner 14. Use proper grammar, punctuation, and spelling 15. Avoid using unnecessarily difficult vocabulary 16. Analyze the effectiveness of each item after each administration of the test. Nursing Workshop 2/12/2013
  • 77. Rules Broken in the Content Free Test Nursing Workshop 2/12/2013
  • 78. Writing a MC Item Item writing…… Nursing Workshop 2/12/2013
  • 79. Workshop Agenda • Introductions • Overview • Perspectives and Attitudes • Stakeholders • Valdity and Reliability • Alignment • Classifying skills and outcomes • Best Practices in MC item writing • Table of Specification or Blueprint • Take Away • Next Workshop Nursing Workshop 2/12/2013
  • 80. Blueprint or TOS Nursing Workshop 2/12/2013
  • 81. Remembering/Understanding Level Nursing Workshop 2/12/2013
  • 82. Application Level Application (AP) theories in providing care to clients (e.g., a Nursing Workshop 2/12/2013
  • 83. Critical Thinking Level revised Blooms Taxonomy Nursing Workshop 2/12/2013
  • 84. Blueprint or TOS Nursing Workshop 2/12/2013
  • 85. Workshop Agenda • Introductions • Overview • Perspectives and Attitudes • Stakeholders • Valdity and Reliability • Alignment • Classifying skills and outcomes • Best Practices in MC item writing • Table of Specification or Blueprint • Take Away • Next Workshop Nursing Workshop 2/12/2013
  • 86. Writing a MC Item What So what Now what Nursing Workshop 2/12/2013
  • 87. Writing a MC Item Nursing Workshop 2/12/2013
  • 88. Workshop Agenda • Introductions • Overview • Perspectives and Attitudes • Stakeholders • Valdity and Reliability • Alignment • Classifying skills and outcomes • Best Practices in MC item writing • Table of Specification or Blueprint • Take Away • Next Workshop Nursing Workshop 2/12/2013
  • 89. Writing a MC Item Thank You Nursing Workshop 2/12/2013
  • 90. ReferencesKrathwohl, David R. (2001). A Taxonomy for Anderson, Lorin W. & Learning, Teaching and Assessing: a Revision of Bloom’s Taxonomy. New York. Longman Publishing. http://shpliteracy.pbworks.com/w/file/fetch/54533674/NEW%20Blooms%20Taxono my%20Question%20Stems.pdf Nursing Workshop 2/12/2013

Notas del editor

  1. No need to fold the paperNo need to cut or rip the paper
  2. No need to fold the paperNo need to cut or rip the paper
  3. No need to fold the paperNo need to cut or rip the paper
  4. No need to fold the paperNo need to cut or rip the paper
  5. No need to fold the paperNo need to cut or rip the paper
  6. Knowledge/Comprehension (K/C)This level refers to the ability to recall previously learned material and to understand its meaning. It includes such cognitive abilities as knowing and understanding definitions, facts, principles and interpreting data (e.g., knowing the effects of certain drugs, interpreting data appearing on a client’s chart).
  7. Application (AP)This level refers to the ability to apply knowledge and learning to new or practical situations. It includes applying rules, methods, principles and nursing theories in providing care to clients (e.g., applying principles of therapeutic communication to the nursing care of clients).In this question, the examinee would be required to use the cognitive ability of Application. That is, the examinee would be required to apply, from among the general measures to counteract pyrexia, those that are appropriate at this particular stage (e.g., onset). The examinee would also be required to apply knowledge of measures that should be taken initially given that an elderly person can become debilitated very rapidly from the effects of a fever.
  8. Critical Thinking (CT)The third level of the taxonomy deals with higher-level thinking processes. It includes the ability to judge the relevance of data, to deal with abstractions and to solve nursing problems (e.g., identifying priorities of care, evaluating the effectiveness of nursing actions). The nurse should be able to identify cause-and-effect relationships, distinguish between relevant and irrelevant data, formulate valid conclusions and make judgments concerning the needs of clients.In this question, the examinee would be required to use the cognitive ability of Critical Thinking. That is, the examinee would be required to solve a nursing situation while considering the relationship between the client’s age and manifestations that have not been directly observed by the practical nurse. The examinee would have to make a judgment about the seriousness of this situation (e.g., possibility of meningitis) and determine the most appropriate intervention.
  9. Knowledge/Comprehension (K/C)This level refers to the ability to recall previously learned material and to understand its meaning. It includes such cognitive abilities as knowing and understanding definitions, facts, principles and interpreting data (e.g., knowing the effects of certain drugs, interpreting data appearing on a client’s chart).
  10. Application (AP)This level refers to the ability to apply knowledge and learning to new or practical situations. It includes applying rules, methods, principles and nursing theories in providing care to clients (e.g., applying principles of therapeutic communication to the nursing care of clients).In this question, the examinee would be required to use the cognitive ability of Application. That is, the examinee would be required to apply, from among the general measures to counteract pyrexia, those that are appropriate at this particular stage (e.g., onset). The examinee would also be required to apply knowledge of measures that should be taken initially given that an elderly person can become debilitated very rapidly from the effects of a fever.
  11. Critical Thinking (CT)The third level of the taxonomy deals with higher-level thinking processes. It includes the ability to judge the relevance of data, to deal with abstractions and to solve nursing problems (e.g., identifying priorities of care, evaluating the effectiveness of nursing actions). The nurse should be able to identify cause-and-effect relationships, distinguish between relevant and irrelevant data, formulate valid conclusions and make judgments concerning the needs of clients.In this question, the examinee would be required to use the cognitive ability of Critical Thinking. That is, the examinee would be required to solve a nursing situation while considering the relationship between the client’s age and manifestations that have not been directly observed by the practical nurse. The examinee would have to make a judgment about the seriousness of this situation (e.g., possibility of meningitis) and determine the most appropriate intervention.