Barbour, M. K., & Huerta, L. (2015, April). Designing data systems for the hard questions: Data elements and structures for effectively assessing and improving VirtualSC. A presentation to the Blended and Online Research Alliance's South Carolina Workshop, Columbia, SC.

1. Designing Data Systems
for the Hard Questions:
Data Elements and Structures
for Effectively Assessing and
Improving VirtualSC
April 2, 2015

2. This information is being provided as part of a Research to Practice Workshop
administered by the Regional Educational Laboratory Southeast. Information
and materials mentioned or shown during this presentation are provided as
resources and examples for the viewer's convenience. Their inclusion is not
intended as an endorsement by the Regional Educational Laboratory
Southeast or its funding source, the Institute of Education Sciences (Contract
ED-IES-12-C-0011).
In addition, the instructional practices and assessments discussed or shown in
these presentations are not intended to mandate, direct, or control a State’s,
local educational agency’s, or school’s specific instructional content, academic
achievement system and assessments, curriculum, or program of instruction.
State and local programs may use any instructional content, achievement
system and assessments, curriculum, or program of instruction they wish.
2

3. Policy, Finance, and
Governance Issues for
Designing Effective Data
Systems for Online Learning

4. Michael K. Barbour
Sacred Heart University

5. http://nepc.colorado.edu/publication/virtual-schools-annual-2015

6. 1991 – first K-12 online learning program
1994 – first supplemental program
– first full-time program
1996-97 – creation of FLVS & VHS
2001 – K12, Inc. begins first program

7. 2000-01 – between 40,000-50,000 students
(Clark, 2001)
2010-11 – between one and four million
(Ambient Insights, 2011; Watson et al., 2011)
– K-12 online learning activity in all 50
states and DC (Watson et al., 2011)
Today – between one and six million (Ambient
Insights, 2014; Watson et al., 2014)

8. • A number of scholars have documented the
absence of rigorous reviews of virtual schools
(Barbour & Reeves, 2009).
• “based upon the personal experiences of those
involved in the practice of virtual schooling”
(Cavanaugh, Barbour & Clark , 2009)
• “a paucity of research exists when examining
high school students enrolled in virtual schools,
and the research base is smaller still when the
population of students is further narrowed to the
elementary grades” (Rice, 2006)

9. • Cavanaugh, Barbour and Clark (2009)
defended this state of affairs, writing
that “in many ways, this [was] indicative
of the foundational descriptive work that
often precedes experimentation in any
scientific field.”
• We can ask, however, how long must we
wait? (Barbour, 2011).

10. 1. Comparisons of student performance based upon
delivery model (i.e., classroom vs. online)
2. Studies examining the qualities and
characteristics of the teaching/learning
experience
 characteristics of
 supports provided to
 issues related to isolation of online learners
(Rice, 2006)
1 Effectiveness of virtual schooling
2 Student readiness and retention issues
(Cavanaugh et al., 2009)

11. Literature Finding
Bigbie &
McCarroll (2000)
…over half of students who completed FLVS courses
scored an A in their course & only 7% received a failing
grade.
Cavanaugh (2001) …effect size slightly in favor of K-12 distance education.
Cavanaught et al.
(2004)
…negative effect size for K-12 distance education.
Cavanaugh et al.
(2005)
FLVS students performed better on a non-mandatory
assessment tool than students from the traditional
classroom.
McLeod et al.
(2005)
FLVS students performed better on an algebraic
assessment than their classroom counterparts.
Means et al. (2009) …small effect size favoring online cohorts over face-to-
face cohorts based on limited K-12 studies.
Chingos &
Schwerdt (2014)
FLVS students perform about the same or somewhat
better on state tests once their pre-high-school
characteristics are taken into account.

12. Ballas & Belyk
(2000)
participation rate in the assessment among
virtual students ranged from 65% to 75%
compared to 90% to 96% for the classroom-
based students
Bigbie &
McCarroll (2000)
between 25% and 50% of students had dropped
out of their FLVS courses over the previous two-
year period
Cavanaugh et al.
(2005)
speculated that the virtual school students who
did take the assessment may have been more
academically motivated and naturally higher
achieving students
McLeod et al.
(2005)
results of the student performance were due to
the high dropout rate in virtual school courses

13. Literature Finding
Kozma et al.
(1998)
“…vast majority of VHS students in their courses
were planning to attend a four-year college.”
Espinoza et al.
(1999)
“VHS courses are predominantly designated as
‘honors,’ and students enrolled are mostly college
bound.”
Roblyer &
Elbaum (2000)
“…only students with a high need to control and
structure their own learning may choose distance
formats freely.”
Clark et al.
(2002)
“IVHS students were highly motivated, high
achieving, self-directed and/or who liked to work
independently.”
Mills (2003) “…typical online student was an A or B student.”
Watkins (2005) “…45% of the students who participated in e-
learning opportunities in Michigan were either
advanced placement or academically advanced
students.”

14. Literature Finding
CO (2006) “Online student scores in math, reading, and writing have been
lower than scores for students statewide over the last three years.”
OH (2009) …online charter school students experienced significantly lower
achievement gains compared to brick-and-mortar charter schools
in the state.
OH (2009) Online charter schools “rank higher when looking at their ‘value-
added’ progress over one year rather than simply measuring their
one-time testing performance.”
WI (2010) “Virtual charter school pupils’ median scores on the mathematics
section of the Wisconsin Knowledge and Concepts Examination
were almost always lower than statewide medians during the 2005-
06 and 2006-07 school years.”
CO (2011) “Half of the online students wind up leaving within a year. When
they do, they’re often further behind academically then when they
started.”
MN
(2011)
“Compared with all students statewide, full-time online students
had significantly lower proficiency rates on the math MCA-II but
similar proficiency rates in reading.”

15. Literature Finding
AZ (2011) “[N]early nine of every 10 students enrolled in at least one statewide
online course, all had graduation rates and AIMS math passing rates below
the state average”
OH (2011) “[N]early 97 percent of Ohio's traditional school districts have a higher
score than the average score of the seven statewide” online charter schools.
Those schools in Ohio also underperformed brick-and-mortar schools in
graduation rates.
PA (2011) 100% of these online charter schools performed significantly worse than
feeder schools in both reading and math.
AR (2012) …online students performed at levels comparable to their face-to-face
counterparts in six out of eight measures, and on the remaining two
measures online students outperformed their face-to-face counterparts at a
0.10 statistically significant level.
National
(2012)
“…students at K12 Inc., the nation’s largest virtual school company, are
falling further behind in reading and math scores than students in brick-
and-mortar schools.”
KS (2015) “Virtual school students perform similarly to traditional school students in
reading before and after controlling for student demographics. After
controlling for demographic differences, virtual school students’
performance in math was similar to that of traditional school students.”

16. Virtual Public Education In
California: A Study of
Student Performance,
Management Practices and
Oversight Mechanisms at
California Virtual
Academies, a K12 Inc.
Managed School System
http://www.inthepublicinterest.org/sites/default/files/Virtual_Public_Education_In_California.pdf

17. Several findings suggest that the virtual
education model advanced by K12 Inc. in
California does not adequately serve many of its
students. In every year since it began graduating
students, except 2013, CAVA has had more
dropouts than graduates. Its academic growth
was negative for most of its history and it did
not keep up with other demographically similar
schools after 2005. Its Academic Performance
Index scores consistently ranked poorly against
other demographically similar schools and the
state as a whole.

18. • Understanding that K¹²-managed schools are
serving large numbers of students who enter
behind grade level in math and reading
K12 Inc. Public Affairs. (2012). Response to NEPC report on K12 Inc.. Herndon, VA: K12,
Inc.. Retrieved from http://www.k12.com/response-to-nepc#.VPfKu2TF_Kk

19. • “K12 Inc. virtual schools enroll approximately the same
percentages of black students but substantially more white
students and fewer Hispanic students relative to public schools
in the states in which the company operates”
• “39.9% of K12 students qualify for free or reduced lunch,
compared with 47.2% for the same-state comparison group.”
• “K12 virtual schools enroll a slightly smaller proportion of
students with disabilities than schools in their states and in the
nation as a whole (9.4% for K12 schools, 11.5% for same-state
comparisons, and 13.1% in the nation).”
• “Students classified as English language learners are
significantly under-represented in K12 schools; on average the
K12 schools enroll 0.3% ELL students compared with 13.8% in
the same-state comparison group and 9.6% in the nation.”
Miron, G. & Urschel, J. (2012). Understanding and improving full-time virtual schools. Denver, CO: National
Education Policy Center.

20. “AYP is not a reliable measure of school
performance…. There is an emerging
consensus to scrap AYP and replace it with a
better system that measures academic
progress and growth. K12 has been
measuring student academic growth on
behalf of its partner schools, and the results
are strong with academic gains above the
national average.”
Jeff Kwitowski - K12, Inc. Vice President of Public Affairs

21. But What Else Do We Have?

26. • English language arts
• mathematics
academic stream - graduation,
college, university, etc.
basic stream - graduation, trade
school
virtual school program only offers
academic streamed courses

29. • study of rural schooling in three schools on the south coast of
the Labrador
• found two had a higher percentage of students enrolled in
basic-level courses
• speculated because the only way students could do academic
course at their school was online, some students specifically
chose the basic stream to avoid taking an online course
Students who enroll in the basic stream are not eligible for post-
secondary admittance!

30. • One area where existing, if limited,
research can provide some guidance to
policymakers is how to approach funding
for online learning
• As one of the few areas where a reasonable
body of literature exists, and where both
progressive and neo-liberal groups have
reached relative agreement.

31. Literature Finding
Hausner (2004)
- OH
“…cost per student [of cyber schooling] is not
enormously higher than for in-class students.
Over time, cyber education will become
substantially more cost-efficient.”
Ohio Legislative
Committee on
Education Oversight
(2005)
…actual cost of the five existing full-time online
charter schools was $5382/student, compared to
$8437/student for traditional public brick-and-
mortar schools.
Florida Tax Watch
Center for
Educational
Performance &
Accountability
(2007)
…Florida Virtual School to be $284 more cost
effective than brick-and-mortar education in
2003-04, and $1048 more cost effective by 2006-
07.
Dodd (2010) - GA …able to meet Annual Yearly Progress in 2009-
10 with 65% of the funding provided to
traditional schools, or $3500/student.

32. Literature Finding
Gillis (2010) - WI …able to operate its full-time online charter
schools at 65% of traditional funding, or
$6,480/student.
Barbour (2012) -
MI
…it cost 16% less in 2009-10 and was projected to
cost 7% less in 2010-11 to provide full-time online
learning than to provide traditional schooling.
Hassel et al.
(2012) - National
…full-time K-12 online learning costs between
$5,100/student and $7,700/student—or between
51% and 77% of the cost of traditional brick-and-
mortar schooling.
iNACOL (2013) –
National
many states funded virtual schools at 30-50% less
than brick-and-mortar schools, specifically
finding that the national average per pupil
funding for virtual schooling was approximately
$6,400, while per pupil funding for brick-and-
mortar schools averaged $11,282

33. • “…the operating costs of online programs are
about the same as the operating costs of a
regular brick-and-mortar program.”
Anderson, Augenblick, DeCescre & Conrad (2006)

34. • “…the operating costs of online programs are
about the same as the operating costs of a
regular brick-and-mortar program.”
• “…[analysis] excluded schools’ capital expenses
and transportation costs from their brick-and-
mortar estimates; had those costs been included
“the costs of operating virtual schools would
have been less per pupil than brick-and-mortar
schools.”
Anderson, Augenblick, DeCescre & Conrad (2006)

36. Director of Doctoral Studies
Sacred Heart University
mkbarbour@gmail.com
http://www.michaelbarbour.com
http://virtualschooling.wordpress.com

37. The Virtual Evidence Base for
Virtual Education
and the
Implications for Policy
Luis Huerta, Teachers College-Columbia University
Jennifer King Rice, University of Maryland
Sheryl Shafer, Teachers College-Columbia University

38. The “Promise” of
Virtual Schools
 Expand choice options
 Increase effectiveness
 Decrease costs / Increase efficiency
 Facilities
 Personnel
 Materials

39. Fast Growing Movement
 The majority of states and the District of Columbia
allow full-time virtual schools to operate
 Estimates of student enrollment in virtual schools
nationwide range between 200,000 to more than
250,000 in 311 full-time virtual schools
 The majority of these students (67%) are enrolled in a
small number of large for-profit EMOs (K12Inc. enrolls
over 77,000 students).

40. What is the Evidence?
Scarce credible research to justify ongoing calls for ever-
greater expansion (Miron et al, 2014;Barbour, 2014)
 30% of the virtual schools in 2012-13 did not receive any
state accountability/performance ratings
 Of the 231 schools with ratings, only 33.76% had academically
acceptable ratings
 On average, virtual schools’ AYP results were 22 percentage
points lower than those of brick-and-mortar schools
 On-time graduation rates for full-time virtual schools were
close to half the national average: 43.8% and 78.6%,
respectively

41. Our Policy Framework
 Examined the current policy terrain in three areas
 Governance and Finance
 Instructional Program Quality
 Teacher Quality
 Identified
 Policy problems
 Assumptions
 Empirical questions

42. State Legislative Analysis
 Provides a baseline representation of how legislators are
promoting, revising and curbing evolving virtual school models
 2012: 128 bills considered in 31 states
 41 enacted
 87 failed
 2013: 127 bills considered in 25 states
 29 enacted
 7 failed
 92 pending
 Seven states (AZ, FL, PA, TN, UT, NC, WA) showed the most
legislative activity, with eight or more bills proposed in each.

43. Accountability Challenges:
Finance and Governance

44. Recommendations
 Develop new funding formulas based on the actual costs of
operating virtual schools.
 Develop new accountability structures for virtual schools,
calculate the revenue needed to sustain such structures,
and provide adequate support for them.
 Establish geographic boundaries and manageable
enrollment zones for virtual schools by implementing state-
centered funding and accountability systems.
 Develop guidelines and governance mechanisms to ensure
that virtual schools do not prioritize profit over student
performance.

45. Accountability Challenges:
Instructional Program Quality
Policy Problem Assumptions Empirical Questions
Requiring high-quality
curricula
Course content offered through
online curricula is an effective
means for meeting individualized
education goals.
How is the quality of course content best evaluated?
How will the Common Core impact virtual school
content and instruction?
Ensuring both quality and
quantity of instruction
Instructional seat time is not an
accurate measure of learning.
What is the best method of determining learning?
What learning-related factors are different in an
online environment?
Should outcomes beyond subject-matter mastery be
assessed?
Monitoring student
achievement
Students in virtual schools perform
equal to or better than traditional
peers and existing empirical work
has adequately measured student
achievement.
Modest gains can be taken to scale.
As some states move to student choice at the course
level, what do they need to implement quality
assurance from multiple providers?
What are effective measures of student achievement?
How does course content affect student
achievement?

46. Recommendations
 Require high-quality curricula, aligned with applicable
state and district standards, and monitor changes to
digital content.
 Develop a comprehensive system of summative and
formative assessments of student achievement, shifting
assessment from a focus on time- and place-related
requirements to a focus on student mastery of
curricular objectives.
 Assess the contributions of various providers to student
achievement, and close virtual schools and programs
that do not contribute to student growth.

47. Accountability Challenges:
Teacher Quality

48. Recommendations
 Define new certification training and relevant teacher
licensure requirements and continually improve online
teaching models through comprehensive professional
development.
 Address retention issues by developing guidelines for
appropriate student –teacher ratios.
 Work with emerging research to create effective and
comprehensive teacher evaluation rubrics.

49. For More Information
Dr. Diana Sharp
Blended and Online Learning
Research Alliance Manager
REL Southeast
(800) 762-5001
dsharp@rmcres.com
49