1. Designing and Developing Online Course Assessments Day 2: Strategies and Resources Dr. Veronica Diaz, Maricopa Community Colleges Dr. Patricia McGee, The University of Texas at San Antonio
2. Day 2 Agenda Welcome Part 1: What are the types of effective assessments in an online course? Part 2: How can we select the appropriate assessment strategy? Q&A
9. Rubrics Specifically state the criteria for evaluating student work Are more specific, detailed, and disaggregated than a grade and can help students to succeed before a final grade Can be created from Language in assignments Comments on students’ papers, or Handouts intended to help students complete an assignment
10. Development Steps Identify what you are assessing (e.g., critical thinking, writing, process, participation) Identify the characteristics/behavior of what you are assessing (e.g., presenting, problem-solving) Decide what kind of scales you will use to score the rubric (e.g. checklists, numerical, qualitative, or numerical-qualitative) Describe the best work you could expect using these characteristics: top category Describe the worst acceptable product using these characteristics: lowest category
11. More Steps Develop descriptions of intermediate-level products and assign them to intermediate categories: 1-5: unacceptable, marginal, acceptable, good, outstanding 1-5: novice, competent, exemplary Other meaningful set Test it out with colleagues or students by applying it to some products or behaviors and revise as needed to eliminate ambiguities
12. Rubric Tips Develop the rubric with your students Use same rubric that was used to grade Use examples to share with students, so they can begin to understand what excellent, good, and poor work looks like Have students grade sample products using a rubric to help them understand how they are applied In a peer-review process, have students apply the rubric to each other’s work before submitting it for official grading
13. Benefits Allows assessment to be more objective and consistent Focuses instruction to clarify criteria in specific terms Clearly shows the student how their work will be evaluated and what is expected Promotes student awareness of about the criteria to use in assessing peer evaluation
17. Contributes to… Ownership and deeper learning Motivation Independent and creative thinking Self awareness of learning as compared to that of others
18. Peer Assessment…. Strategies Participation in peer assessment scales, processes Rating Peer Critique/Review Mentoring/tutoring/coaching Rubrics: for drafts, in conjunction with instructor assessment Tools Wiki or Google Docs CMS rating function Rubric, generic survey IM, Twitter, Facebook
23. Frequency of Feedback Pedagogical Timing After practice At completion/achievement of objective Calendar Schedule Indirect – weekly Direct - bi-monthly
24. Where and how… From http://www.french-in-aude.com/pages/skype.htm From http://www.thethinkingstick.com/?p=604
27. Involving the learner… Incorporate meta-cognitive assessments Provide a strategy for self-assessment and progress Track progress Compare work to that of others Grade own performance
29. Self-Assessment Techno CATs Before/after unit Reflection Feedback on design Feedback on technology Increases accountability Engages students Starts discussions Early alert Practice
30. Examples What was the one most useful thing you learned in this assignment, unit or module? What suggestions would you give other students on ways to get the most out of this assignment, unit or module? In what area did you learn or understand the most? Least? List three ways you think you have developed or grown as a result of this assignment, unit or module? What did you learn about writing, research, (or any other skill) from this assignment, unit or module? What problems did you encounter in this assignment, project, unit, or tool that was used? What unit/module of this course was your best work and why?
32. Case-based instruction Well-bounded cases are presented to students as a focus for discussion and analysis. One situation or case becomes the focal point for an instructional sequence. Cases can illustrate a real world situation that requires application of learned course content. Cases can be provided in segments, as learners become prepared to address different components of the case.
33. Case Assessment Teams Final outcome against pre-determined criteria Completion Performance against other teams Individual Contribution Objective assessments
34. Case-based Assessment Student Generated Pre-determined and communicated context, focus, format Peer review/critique Expert review/critique Contest Instructor Provided Case is focus of formal assessment Solution-resolution-outcome is withheld Students complete case
36. Project-based learning A long term instructional activity in which students work as a group as they focus on a question, problem, event or interest, investigate and negotiate understanding, and produce a product that represents their understanding. (Brown & Campione, 1994)
37. Considerations for Online Collaborative Projects Virtual or field? Team or individual? Cooperative or collaborative? Progressive?
38. Types of Projects Research Debate Presentation Teach/mentor/coach Design-Develop Simulation (virtual worlds)
39. Possible Tools Presentation (such as Voicethread, Bubbleshare, Yawnbuster) Videocasts (Such as Viddler, Videostream) Social networking (such as Ning, Epsilen) Virtual Worlds (such as Second Life)
40. Helpful techniques Set benchmarks Use peer review/assessments Provide regular and informative feedback if not formal assessment Set criteria for performance, completion, scope, and achievement
42. What are they? A way to organize, summarize, and share artifacts, information, and ideas about [you decide] A sampling of the breadth and depth of a person's work conveying the range of abilities, attitudes, experiences, and achievements
43. Uses Reflection Reference Progressive Evaluation Sharing Employer oriented Course, unit, program based Repository
44. Tools & Considerations Tool Ideas Internal CMS Web pages External Wikis Google Sites Blogs Considerations Interactive Support multimedia Security and privacy Allows assessment (scoring and data aggregation based on a rubric) Portable (exportable) Storage space
45. How can we select the appropriate assessment strategy?
47. POLL: Which best describes your experience with student teams? Have used them successfully (students like them and I do too) Have used them, but students don’t like them Have never used them I’d use them more often if I could use them effectively They don’t fit well with my courses 47
48. Using Teams Based on the work of Larry Michaelsen (University of Oklahoma) http://teambasedlearning.apsc.ubc.ca/ 3 Keys Promoting ongoing accountability Using linked and mutually reinforcing assignments Adopting practices that stimulate idea exchange 48
49. Promoting Ongoing Accountability Require pre-group work Require group members to express individual opinions and monitor via another member Include peer evaluation in grading Readiness Assurance Process Test over readings Group: Test, discuss, reach consensus and retest Provide information for peer feedback process 49
57. Team Teaching Tips Outline learning goals Teach team skills Clear and detailed instructions Rubric Stages of team development Forming - polite but untrusting Storming - testing others Norming - valuing other types Performing - flexibility from trust 52
58. Cooperation and Collaboration COOPERATION – Each team member takes on a specific and unique role, e.g., editor, task manager, researcher, etc. The instructor assign tasks, monitors accountability, sets a clear procedure, devices a strategy for oversight of parts as relate to the whole COLLABORATION – Team members contribute perform same role and compile contributed pieces, e.g. all students research, interview, collect, examine, etc. Assign roles, accountability, clear procedure
59. Team Contracts Purpose, goals, and missions Expectations Roles Conflict resolution strategies Meetings Communication Decision-making policy Agendas Record-keeping 54
60. Other Resources Team Based Learning (Michaelsen) http://teambasedlearning.apsc.ubc.ca/?page_id=9 Video Demonstrations http://teambasedlearning.apsc.ubc.ca/v/michaelsenvid.html 55
62. 4 Questions What do I want students to be able to DO after this unit of instruction (behavioral outcomes) What will students have to KNOW to do XYZ (learning outcomes) How can I ASSESS whether or not students have successfully mastered key course concepts? (feedback) How can I tell if students will be able to USE their knowledge of key course concepts? (application) 57
72. Individual Classroom Assessment A method used to inform you on …. Students learning Effectiveness of course content Effectiveness of teaching methods
73. Individual Assessment Benefits Learner-centered Teacher-prompted Mutually beneficial Formative Fast to administer and interpret Non threatening Ongoing Foster trust between student and instructor
74. Basic Development Steps Choose a learning goal to assess Choose an assessment technique Apply the technique Analyze the data and share the results with students Respond to the data, i.e., make modifications as necessary
75. 5 Suggestions for Online CATs Customize to your specific needs and learning environment Should be consistent with your instructional philosophy Test out a CAT and assess their effectiveness Allow extra time to carry out and respond to the assessment Let students know what you learn from their feedback and how you and they can use that information to improve learning
76. Online CAT Examples Chain notes One-sentence summary Application cards Student-generated test questions Can be easily modified or converted to an online environment
77. CATS as Formative Assessments CATs (most can be easily converted) Classroom Assessment in Web-Based Instructional Environment: http://pareonline.net/getvn.asp?v=9&n=7 Teaching Tips: http://honolulu.hawaii.edu/intranet/committees/FacDevCom/guidebk/teachtip/teachtip.htm#assessment
79. Exercise Review the CATs and pick one Identify the goal for your CAT Explain why this CAT is helpful/necessary in this particular area of the course How and when will students receive feedback on the CAT
81. Posting vs. processing But I posted six messages?? You didn’t tell me I had to respond to other posts I didn’t have anything to say! I didn’t understand the instructions!!!
82. Start with.. General discussion criteria in the syllabus Reiterate criteria and instructions as first message in discussion thread Provide examples of an appropriate post Facilitate with public feedback to the group and private feedback to the individual
83. Possible discussion criteria Number of posts (least effective) Number of words Type of responses: Answer a question Pose a question Respond to another post Provide new information, examples, or evidence Quality of response
85. Participation vs. contribution Did the learner arrive on time? Did the learner remain in chat room the entire time? Did the learner contribute? Did the learner interact with others? Did learner follow chat conventions? What was frequency of contribution? Where chat expectations communicated? How did student meet chat expectations? How was chat assessed?
86. Chat strategies Breakout rooms with specific outcomes Presentations with pere-determined criteria (Rubric) Debates (winner) Discussion (participation and contribution) Learner generated conventions
Patricia - Most of online learning is independent in the sense that the learners are never physically together. Collective learning experiences, particularly asynchronous ones, are difficult to capture. Although social presence and interaction increases engagement, caution must be taken to make sure that it results in learning.
Veronica
Veronica -Make sure you communicate this clearly to students in an online environment
Veronica
Veronica
Veronica
Veronica
Veronica
Michaelsen, L. K. (1992). Team learning: a comprehensive approach for harnessing the power of smallgroups in higher education. To Improve the Academy, 11, pp. 107-122Rafiq, Y., & Fullerton, H. (1996). Peer assessment of group projects in civil engineering. Assessment &Evaluation in Higher Education, 21, pp. 69-81.Somervell, H. (1993). Issues in assessment, enterprise and higher education: the case for self-, peer andcollaborative assessment. Assessment & Evaluation in Higher Education, 18, pp. 221-233.
Patricia
Veronica
Patricia - Assessments and feedback can be conducted in real time or in off time. For example, synchronous activities (chats, VOIP, etc.) can be assessed by using checklists or rubrics and this information conveyed to the learner. Asynchronous feedback and assessments more closely mirror traditional classroom assessments in that instructor or peer takes time to score against pre-determined criteria.
Patricia - In an online course, you may assess things you would not assess in a classroom because of the isolation of the learner, the developmental phase of the learner, and the requirement to engage the learner while documenting their successful participation in the course. Oftentimes online courses limit assessment to assignments, quizzes and tests, because these are the tools built into course management systems. However, the value of practice, activity, and interaction is just as important in an online course as in a traditional classroom. Unlike a classroom students may not know when they are achieving these successful. Assessment is one tactic to give them feedback about their performance. You can make a choice about whether this assessment ‘counts’ as a grade or not.
Patricia
Patricia
Patricia - Grading against a set of criteria is how we typically view assessment in education. However, ungraded assessment may be expedient and provide reassuring feedback to students just knowing that they have done “enough” can motivate and give students confidence.
Patricia
Patricia
Veronica
Veronica
Patricia
Patricia
Patricia
Patricia
Patricia
Patricia
Patricia – projects require instructional decisions that relate to the appropriate type of assessmentVirtual projects may be documented by a project, report or presentation for which criteria are pre-established. Field-base projects may require corroboration by a third party or collection or documentation of experience.Cooperative Learning – individuals work together where each person contributed a piece of the whole’ requires interdependence to achieve a goal.Collaborative Learning - individuals work independently to contribute to a goal.Progressive projects may be completed piecemeal throughout the course rather than a single activity or assignment.
Veronica
Veronica
Veronica
Veronica
Veronica
Veronica
Veronica
Veronica
Veronica
V
Veronica
Veronica
Veronica
Veronica
Veronica
Veronica
Veronica
Veronica - Formative assessment is generally carried out throughout a course or project. Formative assessment, also referred to as educative assessment, is used to aid learning. In an educational setting, formative assessment might be a teacher (or peer) or the learner, providing feedback on a student's work, and would not necessarily be used for grading purposes.
Veronica
Veronica
Veronica
Veronica
Veronica
Patricia
Patricia
Patricia
Patricia
Patricia
Patricia
Patricia
Patricia and Veronica – making assessment that is learner-centered, progressive, relevant, and aligned with objectives and effective practices will help you ensure a reliable and valid set of strategies that contributes to the successful design and implementation of an online course.