Instructurecon 2012 (#instcon12) presentation. In this session we’ll take a look at selected discussion posts to determine indicators of quality, depth of reflection, and evidence of student learning. Participants will be able to apply the technique to student writing in course discussions at their own institutions.

1. Depth Perception:
Identifying quality posts in
online discussions
Carrie L Saarinen
Instructional Designer, Brown University
Lecturer, Salem State University
Social Media Strategist, NERCOMP
Huge Fan, Of Awesome
@clsaarinen

2. A little background....
Comparison of blogged and written
reflections in two medicine clerkships
Fischer, M. A., Haley, H.-L., Saarinen, C. L. and Chretien,
K. C. (2011), Medical Education, 45: 166–175
DOI: 10.1111/j.1365-2923.2010.03814.x

3. Methods for Assessment of Student Writing
Levels of Reflection Mezirow J. A critical theory of adult learning and education. Adult Educ 1981;32 (1):3–24.
Non-
Reporting only
reflective
Becoming aware of a specific perception, meaning or behavior of one‟s
own or of ways one has of seeing, thinking or acting
Reflection Becoming aware of how one feels about the way one is perceiving, thinking
on or acting
experience Becoming aware of one‟s value judgments about one‟s perceptions,
thoughts, actions and habits as positive or negative, good or bad, liked or
disliked
Questioning whether one‟s own concepts of understanding are adequate
Reflection Recognizing that one is making judgments about people prematurely, based
on on limited information
awareness Developing awareness of underlying assumptions that lead to the above
Critically questioning one‟s underlying beliefs

4. Examples of Student Writing
The CT [computed tomography scan] showed a partially obstructing lesion in the
patient’s colon, with irregular borders, that looked highly suspicious for
malignancy. A colonoscopy that had been already scheduled for that morning
Non-reflective
confirmed a mass in the patient’s colon that would need to be resected. The intern
that I was working with discussed this information with me and told me that the
patient’s attending wanted to break the news to this patient.
Still, it was surprising to me because I was not expecting to like being in the
Reflection on hospital so much more. I really liked being with my patients in the acute crisis and
experience being able to help them through some huge decisions. It just seems like there isn’t
always enough time for that on the out-patient side’
This experience [of] interacting [with] and caring for this patient changed my view on the
outlook of those with significant disease. I began to view these patients not as individuals
who possess a decreased ability to appreciate the variety of life’s
experiences, but as individuals with changing competencies. Through the care of cancer
patients and others with end-of-life issues, I came to appreciate the complexities of this
Reflection on
aspect of medicine. Certainly medicine is more than simply miraculous recoveries and
awareness complete cures, as these are few and far between. To care for a patient then, requires an
understanding of the realities of modern medicine and a willingness to devote oneself to
caring for the entire person despite such realities. In this approach, we must not only
convince patients that we are navigating them through an arduous journey, but to, in fact,
embark on their journey ourselves’

5. flash forward

6. While teaching a
class called
“Technology in the
21st Century
Classroom” I found
that my students
were displaying the
various levels of
reflection in the
online discussions.
Excerpt from lead-in to an online
discussion:
“This cartoon was drawn in 1969 - the
era of Nixon, Woodstock, Vietnam. It
was also the year the first ATM
machines were available and ARPANET,
the predecessor of the internet, was
developed. It's amazing to think that 43
years ago people were already
speculating that classrooms would be
equipped with massive computer
technology and students would be
interacting with them individually.”
Image source: MAKE Magazine
http://blog.makezine.com/2009/10/22/ret
ro-futuristic-classroom-enforcer/.

7. In this fully online class (grad students, MEd
program) I realized that my comments to their
online discussions were connected to what I
learned while examining levels of reflection.
“Your reflection this week is amazing. Very deep
and insightful. There is evidence that this topic and
the discussion has resonated with you. And that you
got something out of the group conference. You
recognize this and have taken the time to really
think about it, to process it, to apply this new
perspective.
Excellent work this week, in the live chat and
discussion forum. BRAVO!”

8. Hey, cool.

9. Levels of Reflection
Applied to Online Discussions
Can the levels concept be tweaked to fit this learning experience? this learning
environment?
Non-reflective Student summarizes information, course content.
Student connects information, course content with personal
Reflection on experience and demonstrates awareness that what has been learned
experience is meaningful. Becomes aware of thought processes and reasoning in
relation to new information.
Student identifies ways in which new knowledge may impact behavior,
decisions, or performance in the future. Student is aware of his/her
Reflection on
own learning and concludes the new knowledge can be used in a
awareness
meaningful way. May question methods because of new knowledge
and begin to develop new ideas.
Note: Be sure to document and share the levels along with general discussion
expectations for the course in the syllabus or course overview materials.

10. Levels of Reflection
Applied to Online Discussions in Canvas
Standard Rubric:
Meets, exceeds,
below expected
class engagement
Add the levels of
reflection on learning
learning

11. Discussion guidelines - example
• Recognize the purpose of and expectations for discussions.
• Read the instructions, prepare for the discussion by completing the work assigned.
• Be concise, be clear.
• Gather and use references to materials. Don‟t copy and paste text from source - use
inline citations to course materials and/or links to non-course materials.
• Follow instructions (ie post once, comment twice) and contribute as instructed (on
time)
• Include respectful point/counter point arguments with your opinion stated but open to
argument
• Provide examples of experiences that relate to the discussion and support your
argument

12. Standard Rubric:
Meets, exceeds, below
expected
class engagement
Add the levels of
reflection on
learning
learning

13. Levels...with sample posts
These selections are from a discussion about Digital Citizenship in relation to building 21st century skills. The
discussion was part of a master‟s program in education.
Digital citizenship needs to be taught in conjunction with exposing children to
Non-reflective technology. It is important that our students see the importance of being digitally
smart and safe.
I think that teachers should provide examples of giving credit to an author to show
students how they should properly cite. I particularly remember being in high
Reflection on school and being confused on how to properly do a works cited page because
experience some teachers called it "Works Cited" and others "Bibliography," so I think in that
particular instance that students would greatly benefit from having an example to
work with.
I try to model appropriate Internet skills. When I make powerpoint presentations I
Reflection on show the students my reference page and we talk about how I got my information
awareness from different sources. I could expand this into an activity and allow the students to
create a slide of what they have learned about each topic at the end of each unit.

14. Levels...with more sample posts
I feel strongly in agreement toward this article. It is good to be offered the wide
variety of viewpoints on 21st Century technology and learning through all of these
Non-reflective
different discussion articles. It goes to show that there are an infinite array of
viewpoints, which can all be argued (for the most part) equally well.
I thought that I would be totally against what McLeod would claim, but after I did the
reading, I feel rather torn about a few points. I can definitely see how the Wireless
Response System can be compared to multiple choice formative assessments, or
Reflection on
how smartboards are comparable to white boards. I do not agree with him about
experience
YouTube videos being the same as VHS tapes or DVD's. .... I understand where he
is coming from when he lists these issues, but I would have to say that the glaring
difference is the use of technology.
I also think that this article was so interesting for me because I am currently
struggling with a group of students who are extremely reticent to read and discuss
books. I have been brainstorming ways to engage them more and I realize that
traditional approaches wont work for them. I was considering something like
Reflection on edmodo and having them share and create things on that but I wonder if the wiki
awareness would be even more efficient. I especially like the ability to connect to new
information and would love for my kids who are reading a book filled with allusions
and dated pop culture references to have a way to clear them up and research them
right away.

15. Remember - we‟d be grading these in SpeedGradertm in Canvas
so the criteria would be right there as we read the student work.

16. Let‟s look at a few posts
How would you assess the level of reflection in these samples?

17. This is example of
a course module
with topic
introduction (the
Instruction page
includes learning
objectives and the
„didactic‟ or
instruction as if I
was giving a
lecture) plus the
links to selected
articles, the
discussion and
that week‟s
assignment.

18. 1. DB from 21st Century Teaching and Student summarizes information,
Learning Non-reflective
course content.
“It was that personalization for teaching a
Student connects information,
lesson that I will always be grateful for. course content with personal
He taught me that there is more to experience and demonstrates
Reflection on awareness that what has been
teaching than becoming, like I have said experience learned is meaningful. Becomes
before, Jeopardy Monkeys. Just as a aware of thought processes and
reasoning in relation to new
rubric is a guide to grading but not the be information.
all and end all, I think MCAS is the same
thing. It can be a powerful assessment
tool, but it is still a tool. When
standardized testing does not take into Student identifies ways in which
new knowledge may impact
consideration multiple learning styles and behavior, decisions, or performance
the fact that due to our varying life in the future. Student is aware of
Reflection on his/her own learning and concludes
experience and diverse cultural history, awareness the new knowledge can be used in
how can it possibly hope to accurately a meaningful way. May question
methods because of new
assess whether or not true learning has knowledge and begin to develop
taken place!? It just boggles my mind.” new ideas.

19. 2. MT from 21st Century Teaching and Student summarizes information,
Learning Non-reflective
course content.
“I know that adding videos of students
might have some legal entanglements Student connects information,
course content with personal
but adding videos on a blog for kids to experience and demonstrates
discuss, like how we use the lecture Reflection on awareness that what has been
experience learned is meaningful. Becomes
from TED, would be a similar way to aware of thought processes and
reasoning in relation to new
engage students. I think that is also why information.
I enjoyed his blog and his story so much
because I really believe that
engagement is the first step and once
Student identifies ways in which
they are in to it you can ask them to new knowledge may impact
learn and try more and more challenging behavior, decisions, or performance
in the future. Student is aware of
things. It made me excited to think about Reflection on his/her own learning and concludes
awareness the new knowledge can be used in
the different possibilities and I am kind of a meaningful way. May question
kicking myself for not initiating methods because of new
knowledge and begin to develop
something like this earlier.” new ideas.

20. This is an example of
the „open ended‟
Discussion topics
posed in the class.
Topics varied week to
week, and varied in
format depending on
what was happening
in the class (module).
This was a fully online
semester-long course.

21. 3. BN from Are we just replicating Student summarizes information,
classroom basics? Non-reflective
course content.
“When I use technology in my own Student connects information,
classroom I am always weary that I am course content with personal
experience and demonstrates
not using technology appropriately. I Reflection on awareness that what has been
often look back and ask myself could I experience learned is meaningful. Becomes
aware of thought processes and
have done that lesson without using reasoning in relation to new
information.
technology? I don‟t want to fall into the
current trend of using technology just to
add bells and whistles to my lessons.
Teachers need more professional Student identifies ways in which
new knowledge may impact
development on programs to help them behavior, decisions, or performance
create lessons that technology adds to in the future. Student is aware of
Reflection on his/her own learning and concludes
student learning. I have recently awareness the new knowledge can be used in
a meaningful way. May question
stumbled on RmEasiteach.... ” methods because of new
knowledge and begin to develop
new ideas.

22. 4. LH from Are we just replicating Student summarizes information,
classroom basics? Non-reflective
course content.
“I think partly because (when I have Student connects information,
course content with personal
been able to incorporate technology in experience and demonstrates
my classroom) I have wondered if it is Reflection on awareness that what has been
experience learned is meaningful. Becomes
enough. Am I really taking full aware of thought processes and
reasoning in relation to new
advantage of this resource? Am I just information.
claiming to use a Smart Board, but
really I'm just using the white board
feature? Am I pretending to use the
Student identifies ways in which
various capabilities, but really I'm just new knowledge may impact
using a calculator that happens to be behavior, decisions, or performance
in the future. Student is aware of
projected up on a Reflection on his/her own learning and concludes
awareness the new knowledge can be used in
screen? Yes. Sometimes the answer a meaningful way. May question
is yes.” methods because of new
knowledge and begin to develop
new ideas.

23. 5. BD from Are we just replicating Student summarizes information,
classroom basics? Non-reflective
course content.
“If our students are hungry for new
methods of teaching and relevant Student connects information,
course content with personal
materials to help them in this ever- experience and demonstrates
Reflection on awareness that what has been
changing world, then we, as their experience learned is meaningful. Becomes
educators are simply starving for the aware of thought processes and
reasoning in relation to new
means to relay that information. I would information.
see my colleagues trying to incorporate
the tech we had, but were not doing
anything to change the dynamic of the Student identifies ways in which
new knowledge may impact
information we were doling out. Luckily behavior, decisions, or performance
in the future. Student is aware of
there are sites like Canvas Reflection on his/her own learning and concludes
K12…http://www.instructure.com/k12 to awareness the new knowledge can be used in
a meaningful way. May question
help educate us on what is possible. The methods because of new
knowledge and begin to develop
children can access it as well.” new ideas.

24. Canvas makes this type of assessment easy - use the Rubric and
Outcomes and the Speedgradertm. Provide feedback right there -
in a place where the student can also see the rubric!

25. carrie l saarinen
clsaarinen@brown.edu
@clsaarinen
save the date!
The NERCOMP Canvas User Group will provide a
forum for community members to share, learn and September 13, 2012
inquire about Canvas tools, services and support.
Attendees will have an opportunity to learn about Canvas User Group
different approaches to using Canvas in online,
hybrid and traditional teaching and learning models.
(a NERCOMP event)
Attendees will have the opportunity to network with
other Canvas school representatives. Non-Canvas
Norwood MA
schools may wish to register participants to learn
about Canvas or to network with others interested in
doing more with learning management systems. A
representative from Instructure will present news
Registration is OPEN
and information from the engineering and
development team behind this groundbreaking LMS.
nercomp.org > Events