Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a MOOCs and Quality Issues
Similar a MOOCs and Quality Issues (20)
Más de Jon Rosewell
Más de Jon Rosewell (19)
Último
Último (20)
MOOCs and Quality Issues
- 1. MOOCs and Quality Issues - QAQE Steering Group Quality Assurance and Quality Enhancement in E-learning Special Interest Group (QE-SIG) Jon Rosewell The Open University Helen Barefoot University of Hertfordshire
- 2. Session Overview • Definition of MOOCs • Online learning • The Quality Code
- 3. 3 MOOCs and Quality Issues So what is a MOOC? • • • • Massive Open Online Course
- 4. MOOC background information • Term coined in 2008 • Connectivism and Connective Knowledge ▫ 25 tuition paying students ▫ 2300 other students from the general public taking course online and free of charge
- 5. Providers
- 6. Context
- 7. What are the key challenges associated with assuring the quality of MOOCs?
- 8. 8 MOOCs and Quality Issues Why bother with quality? • • • • • Students – know what they are committing to Employers – recognition of content and skills Authors – personal reputation, 'glow' of success Institutions – brand reputation Funders – philanthropic, venture caps, governments • Quality agencies – on behalf of above
- 9. 9 MOOCs and Quality Issues Tensions Delivery: Accreditation: Price: Entry: Scale: Support: Pedagogy: Teacher: face-to-face . distance credit . none cost . free selective . open personal . massive intensive . not supported constructivism . transmission star . anonymous
- 10. Massive • Issue is not large size, but scale independence • Aspects: ▫ Financial ▫ Technical ▫ Pedagogical
- 11. Learning design at scale • Individual learning • Small group collaboration • Large communities
- 12. Open • Different meanings: ▫ ▫ ▫ ▫ Open = free = gratis Open = free = libre Open entry Open = transparency
- 13. Course • What distinguishes MOOC from OER or learning object? ▫ ▫ ▫ ▫ Size? Goal, learning outcomes? Measure of completion / achievement / attainment? Structure As narrative (xMOOC)? Negotiated experience (cMOOC)? ▫ Time?
- 14. Quality and learners “What are MOOCs actually aiming at? “Can the quality of MOOCs be assessed in the same way as any defined university course with traditional degree awarding processes? “Or do we have to take into account a different type of objective with MOOC learners? Are the learners mostly interested in only small sequences of learning, tailored to their own individual purpose, and then sign off and move to other MOOCs because their own learning objective was fulfilled?” Ulf-Daniel Ehlers, Ebba Ossiannilsson, Alastair Creelman http://mooc.efquel.org/
- 15. Quality Assurance Agency “Our job is to safeguard quality and standards in UK universities and colleges, so that students have the best possible learning experience” Quality Code for Higher Education sets out the expectations all providers of UK higher education are required to meet. It gives all higher education providers a shared starting point for setting, describing and assuring the academic standards of their higher education awards and programmes and the quality of the learning opportunities they provide.
- 16. QAA event • Quality Code “Factors which apply to all learning opportunities regardless of location, mode of delivery, academic subject; MOOCs are no exception to that” Anthony McClaran, QAA event (July, 2013)
- 17. The Quality Code The Quality Code Part A: Setting and maintaining threshold academic standards Part B: Assuring and enhancing academic quality Part C: Information about higher education provision
- 18. The Quality Code Part A: Setting and maintaining threshold academic standards • A1: The national level • A2: The subject level and qualification level • A3: The programme level • A4: Approval and review • A5: Externality • A6: Assessment of achievement of learning outcomes Part C: Information about higher education provision Part B: Assuring and enhancing academic quality • • • • • • • • • • • B1: Programme design and approval B2: Admissions B3: Learning and Teaching B4: Enabling student development and achievement B5: Student engagement B6: Assessment of students and the recognition of prior learning B7: External examining B8: Programme monitoring and review B9: Academic appeals and student complaints B10: Managing HE provision with others B11: Research degrees
- 19. Considering MOOCs through the lens of the Quality Code B1: Programme design and approval • Procedures for design and approval for MOOC courses? • Are external partners involved in the design and approval? • Assessing the effectiveness of the course and processes for monitoring and reviewing the course?
- 20. Course approval
- 21. Considering MOOCs through the lens of the Quality Code B3; Learning and Teaching • Constructive alignment • Levelness • Teaching methods ▫ ▫ ▫ ▫ ▫ ▫ Videos Embedded questions Tutorials Discussion forums Wikis Assessments
- 23. Considering MOOCs through the lens of the Quality Code • B5; Student engagement • “improving the motivation of students to engage in learning and to learn independently ▫ data • the participation of students in quality enhancement and quality assurance processes, resulting in the improvement of their educational experience”. Source: edX – Presented by Lori Breslow – Enhancement Themes conference, June 2013
- 24. “the participation of students in quality enhancement and quality assurance processes, resulting in the improvement of their educational experience” Traditional courses MOOCs
- 25. Considering MOOCs through the lens of the Quality Code • B6; Assessment of students and the recognition of prior learning
- 26. Mozilla badge infrastructure http://openbadges.org http://scouts.org.uk/ A badge is a validated indicator of accomplishment, skill, quality or interest that can be earned in any environment http://www.dmlcompetition.net/
- 28. Back to basics… • Are students learning? ▫ ▫ ▫ ▫ ▫ Self assessment Quizzes Peer assessment Analytics Survey evaluation
- 30. Completion rates • Reported completion may be very low (1-10%) • Does that matter? ▫ With very large starting numbers, there are still many learners completing ▫ Maybe learners achieve personal goals even if they don’t complete ▫ Can MOOCs encourage access to HE if >90% have an experience which is a ‘failure’?
- 31. Quality points Provenance Reputation Brand checking MOOC creation peer review use user recommendation
- 32. Quality Dimensions Content Accuracy Currency Relevance Ease of use Clarity Visual attractiveness, engaging Clear navigation Functional! Pedagogic Effectiveness Learning objectives Prerequisites Learning design Learning styles Assessment Reusability & openness Format & interoperability Localisation Discoverability: metadata Digital preservation Accessibility
- 33. Benchmarks / indicators (NB: an unashamed plug!) • Conventional HE benchmarks not be appropriate • E-learning benchmarks a better starting point ▫ E-xcellence NEXT http://e-xcellencelabel.eadtu.eu/ • MOOC benchmarks better! ▫ OpenupEd / E-xcellence MOOC (to come…) http://openuped.eu/
- 34. OpenupEd www.openuped.eu • ‘Reflecting European values such as equity, equality and diversity’ • Courses should show features: ▫ ▫ ▫ ▫ ▫ ▫ ▫ Openness to learners Digital openness Learner-centred approach Media supported interaction Recognition options Quality focus Spectrum of diversity
- 35. UK’s first credit rated MOOC • Level 4 • 20 credit points • Taught as MOOC for undergraduate students and as open access • 1000 students • 200 students going for credit • Assessed by a • 10-minute podcast presentation (40%) • 1500-word critical blog (60%)
- 36. And the latest… • University of California, Irvine, • “Tackle serious scientific issues, related to events in the show” ▫ Physics- "science behind decay“ ▫ Public health department- "What can we learn from past epidemics?“ ▫ Maths - "post-calculus maths can be used to model population and epidemic dynamics" • TV series - audience of 10 million • Student numbers for online course expected to be in hundreds of thousands
Notas del editor
- The first MOOCs emerged from the open educational resources (or OER) movement. The term MOOC was coined in 2008 during a course called "Connectivism and Connective Knowledge". This course consisted of 25 tuition-paying students in Extended Education at the University of Manitoba, as well as 2,300 other students from the general public who took the course online free of charge. All course content was available through RSS feeds, and learners could participate with their choice of tools: threaded discussions in Moodle, blog posts, Second Life, and synchronous online meetings. The term was coined by Dave Cormier of the University of Prince Edward Island, and Senior Research Fellow Bryan Alexander of the National Institute for Technology in Liberal Education in response to the course designed and led by George Siemens of Athabasca University and Stephen Downes of the National Research Council (Canada).[17]
- Udacity: 2011 through Stanford University(computer science courses)In January 2013, Udacity launched MOOCs-for-credit, in collaboration with San Jose State University. – initially had poor student performance rates and suspended the partnership but this summer the pass rates are improved and for some courses, students on the MOOCs outperformed on-campus studentsMay 2013 first-ever entirely MOOC-based Master's Degree, a collaboration between Udacity, AT&T and the Georgia Institute of Technology, costing $7,000.Coursera: 2012 through Stanford Computer Science professorsCoursera started in 2012 working with Stanford University, Princeton, the University of Michigan, and the University of Pennsylvania. 12 partners were added in July 2012followed by 17 more in September 2012. In February 2013, the company announced another 29 partner universities. The current total number of partners is 83. Over 4 million usersEdX: May 2012 Massachusetts Institute of Technology and Harvard University wide range of disciplines to a and to conduct research into learning. EdX has 1.2 million users from 192 countries. Collaborations with 27 institutions (includingBerkely) and offers 60 coursesEliademy.comFebruary 2012, ex-Nokia employees in Finland based CBTec launched based on the MoodleVirtual learning environment. more than 19 languages (including Latin)October 2012: The University of New South Wales in Australia launched UNSW Computing 1, the first MOOC by an Australian University running though the OpenLearning platform.March 2013 Australia set up Open2StudyLate 2012 – OU launched FutureLearn as a separate companySeeing many more collaborations and new initiatives being set up including providers in Germany and Brazil.
- Open software -- GNU, Linux, ApacheOpen distance learning -- UK OU, but others worldwideOpen content Open educational resources -- MIT OpenCourseware, OpenLearnetcAnd general rise of e-learningFor-profit providers? -- publishers -- Phoenix (numbers reached 600K in 2010, down to ~250K?)
- Stakeholders:for students –but investment low so no cost for customers / employers – informal credit for authors – personal reputation, 'glow' of success for institutions – brand reputation for funders – philanthropic, venture caps, governments -- currently done on margins but cost has to met somewhereQuality agencies have responsibilities on behalf of others, esp students
- MassiveDon’t forget that some ODL works at large scale -- OU has 250,000 students, student numbers hundreds to thousands per course, sometimes 10,000 -- successful, ranked highly by students -- Indira Gandhi ~4millionAnd issue is not so much working at large scale but finding teaching and learning that is successful at any scale-- can the course work successfully with 100 or 10,000 students?Business model for moocs not clear, but authors rediscovering lessons of 90s – e-learning requires investment in time/effort for qualityTechnical challenges to operating reliably at scale, but not majorBig issue wrt quality is pedagogy – does learning design scale?
- How to scale learning design?Too easy to think that video and powerpoint make a course – simple transmission modelTraditional ODL experience:-- individual learning from high quality materials – scales well once past initial investment -- but for success, ODL requires support eg traditional OU tutor/AL who provides personal support and feedback -- can be scaled (at cost), but hidden cost of scaling up is needing to recruit, train, monitor -- could be computer supported – eg computer marking, adaptive feedback, intelligent tutoring-- small groups -- OU model is to use tutor groups or smaller, but can also construct peer review support, feedback, assessment -- OpenStudio example? T189 has 500-1500 students each presentation Peer review groups created on the fly each week – creates small groups, students who are active at same time, shuffles people between weeks Here photography, but used for other artefacts – design, computer code-- large communities -- eg in large scale forums, experience can be poor for many students -- possible to reduce into smaller groups to keep traffic lower – OU regions, or ‘colour’ groups ~few hundred
- Open– Open as in gratis free beer -- or at least low cost? Or free for content but cost for assessment / support?– Open as in freely licenced, esp reuse, reversion, repurpose -- control by creative commons eg attribution, share-alike, non-commercial -- not clear what rights are for design? Assessment? Student work?– Open entry in ODL sense, ie noprerequisites for registration / enrollment– but should state level and expected background -- All traditional HE filters on the way in, so quality partly determined ‘good pigs make good bacon’ -- but ODL (and MOOC) filters on the way out -- Cynically, prestige universities have scarcity at heart of business, so prestige increased by number they do not admit. – Open data = transparency
- Assumption that course is of sufficient size that can talk of syllabus Course aims, learning outcomes -- but do students’ aims/goals/intentions match? Difficult o measure success if their goals don’t align
- Anthony McClaran – Chief Executive of QAA8th July We need to talk about MOOCs“QAA doesn’t have a policy or an agency position on MOOCs” but they do have a framework – Quality Code
- The UK Quality Code for Higher Education (the Quality Code) sets out the Expectations that all providers of UK higher education are required to meet. We work closely with the UK higher education sector to develop, maintain and update the Quality Code. Higher education providers apply it in designing and delivering programmes of study. Our reviewers use it as the main reference point for their review work.ChaptersA1: The national levelA2: The subject and qualification levelA3: The programme levelA4: Approval and reviewA5: ExternalityA6: Assessment of achievement of learning outcomesChaptersB1: Programme design and approvalB2: Admissions B3: Learning and teaching (The previous version of this Chapter, which will be used in reviews until August 2013, is also available.)B4: Student support, learning resources and careers education, information, advice and guidance (note this is the existing version of this Chapter; a revised version is currently available for consultation).B5: Student engagementB6: Assessment of students and accreditation of prior learningB7: External examining B8: Programme monitoring and reviewB9: Complaints and appealsB10: Management of collaborative arrangements (note this is the existing version of this Chapter; a revised version is currently available for consultation)B11: Research degreesPart C:Public confidence in higher education relies on public understanding of the achievement represented by higher education qualifications. The Quality Code sets out an Expectation that higher education providers make available valid, reliable useful and accessible information about their provision
- In the UK – sense that providers considering MOOCs have generally approached the developments in similar ways to other programmes; used established processes for approval, considered the assessments and viability of programmes and considered how it fits with Institutional Mission and strategy. When working with partner providers it is obviously important to be aware of their quality processes and we will have a look at the Coursera requirements in the discussion session, but there is more to external partnership work than purely approval processes – how are external partners involved in the design, are external members included on validation panels? There is a lot of discussion about MOOCs being examples of evidence of understanding for jobs – are employers being included in the design and approval of courses. Also, how are students used in the deign and approval processes?Indicator 5 within the Quality code discusses the coherence of the programme to the overall experience of the student. It is interesting to consider how MOOCS will fit together. Should there be structured routs
- Assumption in HE is that courses go through an approval process like this – details don’t matter, but note opportunities for review stages and formal governance decisions. Documentation partly for admin, audit, business needs (eg provides transparency, clarifies implications for other stakeholders) but also prompts reflection, provides evidence for QE over time.Also note that process is not only quality assurance, but quality enhancementShould MOOC go through same process? -- Lightweight version of same? -- Or will they be skunkworks, experimental?
- In many cases, learning outcomes as would be written in traditional courses don’t seem to be explicit however there are statements of course contents, statement of expectation and general aims of the course.Most MOOC courses have been designed as purely stand alone courses and as such may not indicate a sense of levelness. Most have guidance on background requirements and we are beginning to see some courses specifically designed at specific levels e.g. level 7 or level 4.In the future we will see some MOOC s having specific MOOC prerequisites?How is the course going to be taught;Andrew Bollington – Chief operating officer, Uni of London International Programme“Quality has to be good because anything else will be stunningly embarrassing “Is the pure fact that the courses are open to everyone enough to ensure high quality teaching materials?Choice of disciplines – some far more vocational some platforms focusing on specific disciplines – June 2013, the University of North Carolina at Chapel Hill launched Skynet University,[30] which offers MOOCs on the sole topic of introductory astronomy. Participants gain access to the university's global network of robotic telescopes, including telescopes in the Chilean Andes and Australia. Skynet University incorporates popular social media platforms, including YouTube,[31] Facebook,[32] and Twitter.[33]However some do specifically indicate some degree of expertise or have any cohesion with other courses first-ever entirely MOOC-based Master's Degree, a collaboration May 2013 =Udacity, AT&T and the Georgia Institute of Technology, costing $7,000.
- Inclusivity?
- One of the huge benefits we are seeing with MOOCs is the level of analytics which is helping our understanding of personalised learning. How can we use this data, how are we going to engage with it and how will feedback loops be created to ensure continuous enhancement?Lori Breslow, Director of the MIT teaching and learning laboratory presented at the Enhancement Themes conference in June in Glasgow and shared some of their data analysis on MITs first MOOC – Circuits and ElectronicsThey had 154,000+ registrants and 7000+ certificate earners (4.5%) Similar to other MOOCs where completion rates range from 5-10%From clickstream data they know that there are 230 million interactions (IP addresses, interactions with course component, assignments and exams.12,696 discussion threads – 96,000+ posts (questions, answers or comments)End of course survey – 7000+
- One of the biggest challenges for MOOCs is assessment - How can you effectively assess 100K people?Some course have an honor code, but there is still the issues about authenticity of submission. All the considerations regarding plagiarism and one’s own submission must still apply especially when considering credits for MOOCs. Some courses have exam centres as is the case in some established distance learning courses but again this doesn’t cater for the large numbers of people signing up for the MOOCMOOC assessment methods include scores for contributions to discussions/articles (but these are based on some submission, not on the quality of the contribution)Online quizzesMathematical testsPeer assessment – some courses have 4-6 peers assessing the work but where is the information about how to assess and guidance on peer assessment – anyone who has done peer assessment in face to face classes knows that it isn’t an easy or simple processIn many online courses and distance learning courses – students will have a named person (perhaps a personal or programme tutor) who they can contact with any questions about the assessments – presumably this isn’t an option within the MOOC and any questions would have to be posted on the discussion site for peer response or tutor responseInclusive assessments – extra time allowances?
- Focus on outputs rather than process?Competency / mastery cf journey / development? -- is that always appropriate?
- Course in presentationneeds monitoring and mechanisms for identifying and fixing issuesEvaluation at end of presentationThis shows course dashboard -- note some red bars: students feel not given sufficient support (course uses peer review), workload higher than expected
- And in any case, are a large proportion of ‘learners’ just curious about MOOCs, not subject?(Survey of by one MOOC platform provider: 77% already have HE qual, 42% are educators!)
- Fairly conventional aspects re contentOpen to review before / after presentationSome aspects apply to Open in OER / Open Source sense-- is it better to have open = reuseable = reversionable of any quality than high quality closed material?
- Key performance indicatorsStudent goals – likely to be very different for MOOC student than for-credit student
- Synchronous and asynchronous deliveryNo prerequisiteRegular readingRunning through BlackboardDeveloped from having a blended approach to current delivery (including webinars and a blog assessment)Current student were involved in development (interestingly Dr Ben Brabon who is running the MOOC suggested that the current students who are paying £9000 for their course weren’t at all disgruntled about other people having access for free) and went through standard Edge Hill quality proceduresMapped levelness and looked at subject benchmarks for thoughts about criticalityLearning outcomes clearly related to level 4 Divided into two six week blocks – runs over 12 weeksWeek dedicated to assessment
- Academics from the University of California, Irvine, who are setting up the online courses, say these will be academically rigorous and tackle serious scientific issues, related to events in the show.A physicist will look at the "science behind decay"; the public health department will use the series to study questions such as "What can we learn from past epidemics?" and maths lecturers will show how "post-calculus maths can be used to model population and epidemic dynamics".The TV series has an audience of 10 million and student numbers for the online course are expected to be in the hundreds of thousands. If this experiment proves successful, the online platform provider, Instructure, suggests there could be further TV show and university hybrids.This eight-module course will begin in mid-October and will be accessible without charge to anyone with an internet connection.