Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
Examining the Validity of the
International Association for K–12
Online Learning (iNACOL) Standards:
Improving K–12 Online...
Background
• Only two of the most commonly used standards have published any
research testing validity
o Virtual High Scho...
Background
• iNACOL National Standards for Quality Online Courses were
created in 2007 and updated in 2011
• Based on SREB...
Purpose
The ultimate purpose of the research was
to create a revised instrument for designers
based off the iNACOL standar...
Methodology
• Research followed a commonly used multi-step
approach of literature review, expert review, and real
world fi...
Phase One: Literature Review
• Compared the 52 areas of measurement of the iNACOL standards to
current K-12 and other rele...
Phase One Results
• Results showed that the elements were either fully or partially supported by
current research and lite...
Phase Two: Expert Review
• Designed a revised rubric through three rounds of expert review
• Content validity of the revis...
Phase Two Results
• Results created a revised 40 element rubric with a focus solely
on K-12 online course design
• Element...
Phase Three: Field Test
• Four pairs of K-12 online educators reviewed the revised rubric
against current K-12 online cour...
Phase Three Results
• Results of the overall rubric were below the acceptable percentage for
reliability (i.e. at least 80...
Conclusions
• Research was conducted in three phases, taking almost two years
to fully complete
• Phase one showed the iNA...
Implications
• Literature review provides minor support for the iNACOL
standards
• A revised and focused design rubric all...
Limitations
• There was a lack of K-12 online learning literature and
research, leading to the use of higher education lit...
Post-Study
Adelstein, D. & Barbour, M. (2018). Redesigning The iNACOL
Standards For K-12 Online Course Design. Journal of ...
Your
Questions
and
Comments
Associate Professor of Instructional Design
Touro University, California
mkbarbour@gmail.com
http://www.michaelbarbour.com
Próxima SlideShare
Cargando en…5
×

AERA 2019 - Examining the Validity of the International Association for K–12 Online Learning (iNACOL) Standards: Improving K–12 Online Course Design

58 visualizaciones

Publicado el

Adelstein, D., & Barbour, M. K. (2019, April). Examining the validity of the International Association for K–12 Online Learning (iNACOL) Standards: Improving K–12 online course design. A paper presentation at the annual meeting of the American Education Research Association, Toronto, ON.

Publicado en: Educación
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

AERA 2019 - Examining the Validity of the International Association for K–12 Online Learning (iNACOL) Standards: Improving K–12 Online Course Design

  1. 1. Examining the Validity of the International Association for K–12 Online Learning (iNACOL) Standards: Improving K–12 Online Course Design David Adelstein Manager, Global Leadership Curriculum Development Tiffany & Co. Michael K. Barbour Associate Professor of Instructional Design Touro University California
  2. 2. Background • Only two of the most commonly used standards have published any research testing validity o Virtual High School (VHS) o Quality Matters (QM) • VHS standards were developed in the early days of K-12 online learning (late 1990s) • QM has significant research, however, it is primarily focused on higher education and also proprietary • Most schools, institutions, and programs look for free, non- proprietary standards o International Association for K-12 Online Learning (iNACOL) standards are starting to be used by several states
  3. 3. Background • iNACOL National Standards for Quality Online Courses were created in 2007 and updated in 2011 • Based on SREB and iNACOL’s participation in the Partnership for Twenty-First Century Skills Initiative • The standards are set up as a design instrument on a 0-4 point scale • Five areas of content with 52 unique areas of measurement
  4. 4. Purpose The ultimate purpose of the research was to create a revised instrument for designers based off the iNACOL standards, with a focus strictly on design. The desire was for the revised instrument to improve the design process and the structure of K-12 online courses.
  5. 5. Methodology • Research followed a commonly used multi-step approach of literature review, expert review, and real world field test o (Alad-wani & Palvia, 2002; Dray et al., 2011; Fitzpatrick, 1983; Gandek & Ware, 1998; Haynes et al., 1995; Legon & Runyon, 2007; Roblyer & Wiencke 2003; Stellmack et al., 2009; Taggart et al., 2001; Thaler et al., 2009; Walker & Fraser, 2005; Yang et al., 2013) 1. Literature Review: Content validity of iNACOL standards 2. Expert Review: Content validity of the revised rubric 3. Field Test: Inter-rater reliability of the revised rubric
  6. 6. Phase One: Literature Review • Compared the 52 areas of measurement of the iNACOL standards to current K-12 and other relevant online learning research • Examined the content validity of the standards (Haynes, Richard, & Kubany, 1995; Fitzpatrick, 1983) o Similar to the process that Ferdig, Cavanaugh, DiPietro, Black, and Dawson (2009) undertook with the iNACOL National Standards for Quality Online Teaching. • Over one year was spent compiling data through Wayne State University’s library and subscribed databases • K-12 literature was primarily used, with higher education and other relevant literature supplemented in where appropriate
  7. 7. Phase One Results • Results showed that the elements were either fully or partially supported by current research and literature • It was noted that student motivation were not directly addressed in the standards o Motivation is integral to education (McCombs & Vakili, 2005; Chen & Jang, 2010) • Traditional journal length constrains limited the depth allowed to each element • The manuscript was published in the online Journal of Online Learning Research Adelstein, D., & Barbour, M. K. (2016). Building better courses: Examining the content validity of the iNACOL national standards for quality online courses. Journal of Online Learning Research, 2(1), 41-73. Retrieved from http://www.editlib.org/p/171515)
  8. 8. Phase Two: Expert Review • Designed a revised rubric through three rounds of expert review • Content validity of the revised rubric (Roblyer & Wiencke 2003; Taggart, Phifer, Nixon, & Wood, 2001) • Eight experts, two from each of the following roles: o Researcher/evaluator, online educator, online administrator, online course designer • Multiple years of experience in area of expertise required, as well as a recommendation • Round one – Each element and suggestion was reviewed using a 1-3 Likert scale, plus expert suggestions • Round two – Experts were asked to mark each poorly rated element as (K)eep, (D)elete, (C)ombine, or (R)evise • Round three – Google Hangouts to discuss all ratings and reach consensus
  9. 9. Phase Two Results • Results created a revised 40 element rubric with a focus solely on K-12 online course design • Elements were eliminated if they did not pertain to online course design • It is recommended to increase the face-to-face discussion time, which is where significant refinement took place Adelstein, D., & Barbour, M. K. (2017). Improving the K-12 online course design review process: Experts weigh in on iNACOL National Standards for Quality Online Courses. International Review of Research in Open and Distance Learning, 18(3). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/2800
  10. 10. Phase Three: Field Test • Four pairs of K-12 online educators reviewed the revised rubric against current K-12 online courses • Inter-rater reliability of the revised rubric o Taggart, Phifer, Nixon, & Wood, 2001; Legon & Runyon, 2007) • Recruitment through: o Edgenuity, Michigan Virtual University, Association for Educational Communication & Technology, Huron Valley Schools, Dissertation Advisor • K-12 online teaching experience required for reviewers • A sample course was sent to each reviewer and discussed via Google Hangouts • Each pair was assigned five online courses from two different content providers, which were reviewed individually using a 1-3 Likert scale
  11. 11. Phase Three Results • Results of the overall rubric were below the acceptable percentage for reliability (i.e. at least 80% exact agreement, (Neuendorf, 2002)) • However, numerous individual elements considered reliable • The small number of reviewers and courses reviewed was limiting to the research • Unable to use kappa coefficient, data shared was shared through percentage agreement Adelstein, D., & Barbour, M. K. (2016). Redesigning design: Field testing a revised design rubric based of iNACOL quality course standards. International Journal of E-Learning & Distance Education, 31(2). Retrieved from http://www.ijede.ca/index.php/jde/article/view/976
  12. 12. Conclusions • Research was conducted in three phases, taking almost two years to fully complete • Phase one showed the iNACOL standards aligned with current literature, but not necessarily K-12 research • Phase two created a revised rubric with a focus solely on design by combining, deleting and revising the iNACOL standards • Phase three tested the revised rubric against current K-12 online courses, with overall results not meeting the reliability threshold. However, 10 elements meet the 80% threshold and 15 meet the 70% threshold for exact match. If combined with one point difference, the number greatly increases
  13. 13. Implications • Literature review provides minor support for the iNACOL standards • A revised and focused design rubric allows for a streamlined creation process, which in turn can save time and money • Researchers and designers have a stronger platform from which to build from • Allows educators to judge their current courses with an instrument specifically created to look solely at design • Ultimately, better design standards impact student engagement and comprehension
  14. 14. Limitations • There was a lack of K-12 online learning literature and research, leading to the use of higher education literature to supplement • The expert panel was limited by volunteers and time, with the video conference limited by schedules • The field test was impacted by the number of reviewers and number of courses offered up for review o This ultimately hampered the ability to use kappa and other statistical procedures
  15. 15. Post-Study Adelstein, D. & Barbour, M. (2018). Redesigning The iNACOL Standards For K-12 Online Course Design. Journal of Online Learning Research, 4(3), 233-261. Retrieved from https://www.learntechlib.org/primary/p/178229/ • QM and the Virtual Learning Leadership Alliance are collaborating to refresh the iNACOL standards under the banner of “THE National Standards for Quality Online Learning” o https://www.nsqol.org/
  16. 16. Your Questions and Comments
  17. 17. Associate Professor of Instructional Design Touro University, California mkbarbour@gmail.com http://www.michaelbarbour.com

×