Barbour, M. K. (2011, March). Making a difference with educational research: A new methodological paradigm. An invited presentation to the College of Education at the University of Auckland, Auckland, New Zealand.

1. Making a Difference with
Educational Research - A
New Methodological
Paradigm
Michael K. Barbour
Assistant Professor
Wayne State University

2. It all began last spring when I read two queries from doctoral students on the
Qualitative Research for the Human Sciences listserv. Both students came from
large public institutions of higher education, one in the USA and the other in
Canada. The first student wrote that she intended to focus her dissertation
research on the quality of "discourse" that takes place in cafes and coffee shops
located inside bookstores. She complained that she had found no "literature" on
this topic and asked the listserv participants for some guidance. The second
student announced that he was preparing a dissertation prospectus centered on the
question of how people learned about opportunities to take SCUBA diving
lessons and what motivated them to register for such courses. He also sought
directions to relevant literature and advice from the listserv membership.
After pondering these queries, I posted a message asking whether faculty
members at taxpayer-supported universities have a moral responsibility to guide
their students toward "socially responsible" research questions. In my posting, I
suggested that in the face of problems such as adult illiteracy, attacks on public
education, "at-risk" students, homelessness, AIDS, and the like, faculty members
should attempt to inspire in students a dedication to research that would "make a
difference."
Thomas Reeves, University of Georgia
Peter Dean Lecture at the 1995 Association for Educational Communications

3. Problems with Educational
Technology Research
1.Misunderstanding about basic and applied
research.
2.Poor quality of educational technology
research.
3.Disappointing research synthesis.

4. Problems with Educational
Technology Research
1.Misunderstanding about basic and applied
research.
2.Poor quality of educational technology
research.
3.Disappointing research synthesis.

5. Misunderstanding About Basic and
Applied Research.

6. Problems with Educational
Technology Research
1.Misunderstanding about basic and applied
research.
2.Poor quality of educational technology
research.
3.Disappointing research synthesis.

7. Poor Quality of Educational
Technology Research
“The best current evidence is that
media are mere vehicles that
deliver instruction but do not
influence student achievement
any more than the truck that
delivers our groceries causes
changes in our nutrition.”
Richard Clark
Review of Educational Research
1983

8. What Does the Secondary E-Learning
Research Say About Student Performance?

9. Student Performance
• performance of virtual
and classroom students
in Alberta were similar
in English and Social
Studies courses, but
that classroom students
performed better
overall in all other
subject areas (Ballas &
Belyk, 2000)

10. Student Performance
• over half of the students who
completed FLVS courses
scored an A in their course and
only 7% received a failing
grade (Bigbie & McCarroll,
2000)
• students in the six virtual
schools in three different
provinces performed no worse
than the students from the
three conventional schools
(Barker & Wendel, 2001)

11. Student Performance
• FLVS students performed
better on a non-mandatory
assessment tool than students
from the traditional classroom
(Cavanaugh et al., 2005)
• FLVS students performed
better on an assessment of
algebraic understanding than
their classroom counterparts
(McLeod et al., 2005)

12. Let’s look a little closer...

13. Students and Student Performance
Ballas & performance of virtual and participation rate in the
Belyk, 2000 classroom students similar assessment among virtual
in English & Social Studies students ranged from 65% to
courses, but classroom 75% compared to 90% to
students performed better 96% for the classroom-based
in all other subject areas students
Bigbie & over half of the students between 25% and 50% of
McCarroll, who completed FLVS students had dropped out
2000 courses scored an A in of their FLVS courses over
their course and only 7% the previous two-year
received a failing grade period

14. Students and Student Performance
Cavanaugh et FLVS students performed speculated that the virtual
al., 2005 better on a non- school students who did
mandatory assessment take the assessment may
tool than students from have been more
the traditional classroom academically motivated and
naturally higher achieving
students
McLeod et FLVS students performed results of the student
al., 2005 better on an assessment performance were due to
of algebraic understanding the high dropout rate in
than their classroom virtual school courses
counterparts

15. Student Performance and Students
But are we really
comparing apples to
apples?

16. The Students
• the vast majority of VHS
Global Consortium students
in their courses were
planning to attend a
four-year college (Kozma,
Zucker & Espinoza, 1998)
• “VHS courses are
predominantly designated as
‘honors,’ and students
enrolled are mostly college
bound” (Espinoza et al., 1999)

17. The Students
The preferred characteristics
include the highly motivated,
self-directed, self-disciplined,
independent learner who
could read and write well,
and who also had a strong
interest in or ability with
technology (Haughey &
Muirhead, 1999)

18. The Students
• “only students with a high
need to control and structure
their own learning may
choose distance formats
freely” (Roblyer & Elbaum,
2000)
• IVHS students were “highly
motivated, high achieving,
self-directed and/or who liked
to work independently” (Clark
et al., 2002)

19. The Students
• the typical online student
was an A or B student
(Mills, 2003)
• 45% of the students who
participated in e-learning
opportunities in Michigan
were “either advanced
placement or
academically advanced”
students (Watkins, 2005)

20. Problems with Educational
Technology Research
1.Misunderstanding about basic and applied
research.
2.Poor quality of educational technology
research.
3.Disappointing research synthesis.

21. Disappointing Research Synthesis
Kannapel and DeYoung (1999) found that rural schools
contained a strong sense of community and were
regularly the focus of the cultural and social aspects of
the community. They also found that “extracurricular
and non-academic activities are often valued as much or
more than academics, and a higher proportion of
students participate in extracurricular activities than in
urban schools” (p. 170).

22. Disappointing Research Synthesis
In their review of key literature over the past 25 years,
Kannapel and DeYoung (1999) found that rural schools
contained a strong sense of community and were
regularly the focus of the cultural and social aspects of
the community. They also found that “extracurricular
and non-academic activities are often valued as much or
more than academics, and a higher proportion of
students participate in extracurricular activities than in
urban schools” (p. 170).

23. Disappointing Research Synthesis
In their review of key literature over the past 25 years,
Kannapel and DeYoung (1999) found that rural schools
contained a strong sense of community and were
regularly the focus of the cultural and social aspects of
the community. They also found that “extracurricular
and non-academic activities are often valued as much or
more than academics, and a higher proportion of
students participate in extracurricular activities than in
urban schools” (p. 170). It should be noted that three of
the eleven “key” pieces of literature were written by
DeYoung, one of the two authors of this review.

26. Examining Effect Sizes
Teacher
Effects Zone of
Desired Effects
Developmental
Effects
Reverse
Effects

27. Primary & Secondary E-Learning
Meta-Analysis
• Cavanaugh (2001) - 16 studies
– +0.147 in favor of K-12 distance education
• Cavanaugh et al. (2004) - 14 studies
– -0.028 for K-12 distance education
• Means et al. (2009) - 46 studies (5 on K-12)
– +0.24 favoring online over face-to-face*
– +0.35 favoring blended over face-to-face*

28. Results of Interest
• Second and third chance programs (d=0.50)
• Matching style of learning (d=0.40)
• Computer assisted instruction (d=0.37)
• Decreasing disruptive behavior (d=0.34)
• Programmed instruction (d=0.24)
• Individualized instruction (d=0.23)
• Class size (d=0.21)
• Charter schools (d=0.20)
• Web-based learning (d=0.18)
• Home-school programs (d=0.16)
• Teacher training (d=0.11)
• Teacher subject matter knowledge (d=0.09)
• Distance education (d=0.09)
• Student control over learning (d=0.04)
28
28

29. Results to Consider
• Providing formative evaluation (d=0.90)
• Micro teaching (d=0.88)
• Teacher clarity (d=0.75)
• Providing feedback (d=0.73)
• Teacher-student relationships (d=0.72)
• Teaching strategies (d=0.60)
• Cooperative vs. individualistic learning (d=0.59)
• Study skills (d=0.59)
• Direct instruction (d=0.59)
• Mastery learning (d=0.58)
• Worked examples (d=0.57)
• Concept mapping (d=0.57)
• Goals (d=0.56)
• Peer tutoring (d=0.55)
• Cooperative vs. competitive learning (d=0.54)

30. Maybe The Problem Is How We
Conduct Research?

31. Randomized Controlled Trials
1. Is there a control group?
2. Are the control and
experimental groups
assigned randomly?
3. If it is a matched study,
are the groups extremely
similar?
4. Is the sample size large enough?
5. Are the results statistically significant?

32. http://ies.ed.gov/ncee/wwc/

33. What Works Clearinghouse
Ironically, the WWC personnel have been able to identify
very few educational programs and practices that have the
evidence that is sufficiently rigorous according to their
own criteria to warrant their inclusion in the What Works
database. For example, a review of over 1,300 studies that
examined the effect of teacher professional development
on student achievement found that only nine met WWC
standards for rigorous evidence (
Yoon, Duncan, Lee, Scarloss, & Shapley, 2007).
Can Educational Research Be Both Rigorous and Relevant?Thomas Reeves,
2011
http://www.educationaldesigner.org/ed/volume1/issue4/article13/index.htm

34. More Problematic Research
Online 7 principles of Interviews with teachers and course
Course effective online developers at a single virtual school,
Design course content with no verification of whether the
for adolescent interviewees’ perceptions were actually
Barbour learners effective (or any student input for that
(2005; 2007) matter)
Online 37 best Interviews with teachers at a single
Teaching practices in virtual school selected by the virtual
asynchronous school itself. Their teachers’ beliefs
DiPietro et online teaching were not validated through observation
al. (2008) of the teaching or student performance.

35. There Must Be A Better Way!

36. Design-Based Research
Reeves (2006)

38. Virtual High School Global Consortium
• annual evaluations
– e.g., Espinoza, Dove, Zucker & Kozma, 1999;
Kozma, Zucker & Espinoza, 1998; Kozma, Zucker,
Espinoza, McGhee, Yarnall & Zalles, 2000
• content-specific investigations
– e.g., Elbaum, McIntyre & Smith, 2002; Yamashiro
& Zucker, 1999
• final evaluation
– e.g., Zucker & Kozma, 2003

39. • Total Student Enrollment - 15,237
• Number of Course Sections - 412
• Number of Member Schools - 770
• Number of International Schools - 51
• Number of Participating States - 34
• Number of Countries - 33

40. Your
Questions
and
Comments

41. Assistant Professor
Wayne State University, USA
mkbarbour@gmail.com
http://www.michaelbarbour.com
http://virtualschooling.wordpress.com