Keynote – Timothy McKay – Learning Analytics: Harnessing Data Science to Transform Education – OWD17
•
1 like•1,369 views
The 21st century opened amidst an information revolution which promises to transform higher education as dramatically in this century as industrialization did in the last. Many things have already changed, but the real revolution will come when we harness information technology to personalize education: optimizing our education of an increasingly diverse student body, creating much greater student motivation and engagement, and accomplishing more with less. This talk describes how research using data about the processes and products of education led the University of Michigan to discover patterns of inequity in STEM education, develop an array of new student support technologies, and launch a major new Foundational Course Initiative.
Recommended
Recommended
More Related Content
Similar to Keynote – Timothy McKay – Learning Analytics: Harnessing Data Science to Transform Education – OWD17
Similar to Keynote – Timothy McKay – Learning Analytics: Harnessing Data Science to Transform Education – OWD17 (20)
More from SURF Events
More from SURF Events (20)
Recently uploaded
Recently uploaded (20)
Keynote – Timothy McKay – Learning Analytics: Harnessing Data Science to Transform Education – OWD17
- 2. The University of Michigan • 200 yr old public research intensive university with 19 Schools and Colleges & 6,800 faculty • Highly selective group of 29,000 undergraduate and 16,000 graduate students • Annual budget of $7 billion, including $1.4 billion in federally funded research (#1 public)
- 9. • >98% of these are: • Small & changeable • Taught in idiosyncratic, engaged, and creative ways • <2% of them are: • Large & relatively stable • Taught in industrial, remote, and tradition bound ways One reason? There are two ways to teach a large number of students… We have 9200 courses at UM
- 11. Focus on Education @ Scale • Even at big Universities, most courses have what they need to be excellent. Large introductory and many online courses don’t – They could be dramatically better with a different approach to instruction and professional support • Improving education at scale is a major sociotechnical challenge, requiring both: – New information technology for personalization – New social norms for course design & delivery It’s hard to beat Bloom’s tutor where she can act. Learning analytics should focus on improving things where she cannot.
- 12. Five themes of our work Learning analytics for personalization 1. Ethics: what are we doing and why 2. Measurement: data collection and management 3. Analysis: modeling, extraction of meaning, learning from the experience of all 4. Action: decision making, storytelling, creating the motivation for change 5. Synthesis: Building a learning laboratory
- 14. Information ethics: a grand challenge • What principles should govern collection and use of data about individuals in education? – What data is relevant for education? – Norms of consent, privacy, autonomy? How are they different within an academic community? – How are experiments in educational practice related to research norms? • We need to ensure that commercial Ed-Tech is a part of (& constrained by) this conversation
- 15. Six Asilomar principles 1. Respect for the rights and dignity of learners: transparency, consent, protection of privacy 2. Beneficence: maximize benefits, minimize harm 3. Justice: benefit all, reduce inequalities 4. Openness: learning and research are public goods 5. The humanity of learning: insight, judgment, & discretion are essential, we should keep learning humane 6. Continuous consideration: ongoing, inclusive discussion of changing ethical circumstances http://asilomar-highered.info/ Where to have this conversation?
- 16. Example 1: Predictive Modeling As educators, we learn from the past in order to change the future…
- 18. Example 3: Measureable Types • Data often used to categorize, collecting individuals into groups • These labels are often reductive and invisible to learners • Always incomplete, substituting a category for individuals • Cheney-Lippold (2017) ‘measureable types’ ≠ complex socially constructed classes – gender ≠ ‘gender’ – text ≠ ‘positive’ • Categorizations too often one dimensional, excluding intersections of identity How to limit the impact of reflexively reductive data representation?
- 19. Intersectionality in LA • Learn methodology of intersectionality from feminist scholarship*: use multiple approaches – Intercategorical: focus on variation across socially constructed provisional categories – Intracategorical: analyze variation w/in categories – Anticategorical: real personalization, no categories • Be transparent, allow for agency: resist labelling individuals w/o understanding and consent *McCall, L. (2005). The complexity of intersectionality. Signs: Journal of women in culture and society, 30(3), 1771-1800.
- 21. What do we measure? • What we measure now: – Admissions information – Course taking & grades – Degrees & honors • What we’re starting to record (explosive growth) – Process of learning: clickstreams, discussions, video, course structures – Products of learning: forum posts, essays, papers, presentations, theses • What we want to have: Detailed, relevant, evolving portraits of every student's background, interests, goals, and accomplishments • These portraits should be used to help students, faculty, administrators, staff better understand higher education
- 24. How to release information while protecting privacy? • As much as possible, we should let everyone learn from the experience of all – Restricted reporting tools – access to digested information, within tools (Ex: ART 2.0, ECoach) – Existing research protocols – IRB oversight, anonymization => the LARC approach • New approaches are emerging in data science: synthetic data contain all the information but none of the details • Personal privacy can (& must) be protected well. We should rethink institutional privacy…
- 25. Better measures of learning • Grades: performance measures of unrecorded tasks, meant to estimate unknown outcomes, quantified on ill-defined scales • We should be measuring learning – increases in well defined knowledge and skills – and focusing on individual growth over time • Direct: pre and post testing aligned with learning goals. Good for foundational courses? • Indirect: Data Science tools for extracting meaning from products – Simple: IRT, topic modeling and beyond – Complex: peer evaluation, NLP, direct representation rather than data reduction
- 26. Intellectual Breadth Disciplinary Depth Range of Experience Engagement & Effort Social & Professional Networks Academic Performance Measuring what matters: the Transcript of the Future
- 27. Students connect through courses Courses connect through students Can we quantify intellectual breadth? Explore each student’s network of connection • Course co-enrollment: well measured, large bipartite network • Better representations of interaction coming Compare measured network structures to appropriate random graphs – measure diversity of connection Exposes isolation of majors, allows comparison of individuals within a major
- 30. Three examples using different methodologies: 1. Are our classrooms equitable? 2. Do learning communities work? 3. Are placement exams used well?
- 34. Data from 2000 – 2012 for all large STEM lecture and lab courses Lab courses Lecture courses Details, including tests of many other possible performance predictors: arXiv 1608:07565 Intercategorical complexity Classroom equity
- 35. Biology Chemistry Math & Stats Physics Data from five Big 10 Schools: Similar GPD patterns across lecture & lab STEM courses. Matz et al. AERA Open in press This analysis uses both hierarchical linear modeling and quasi-experimental matching methods
- 36. #2: Brooks/Morgan/Maltby - HSSP Impact Living Learning Quasiexperimental design Health Science Scholars Program
- 38. Michigan 1:2:1 Introductory Chemistry Curriculum Model: Traditional 2:2 Introductory Chemistry Curriculum Model: 2 Semesters General Chemistry 2 Semesters Organic Chemistry Chemistry 130 Chemistry 210 & 215 Chemistry 230 #3: Shultz/Gottfried/Winschel Chemistry Placement Analysis Chem Placement Regression discontinuity
- 39. How does taking Gen Chem first matter? Chem Placement Shultz, Ginger V., Amy C. Gottfried, and Grace A. Winschel. Journal of Chemical Education 92.9 (2015): 1449-1455. Regression discontinuity
- 46. DIG team connects faculty/staff FACULTY DIRECTOR Tim McKay OPERATIONS DIRECTOR Mike Daniel FACULTY CHAMPIONS Gus Evrard (LSA) Barry Fishman (SI) Elisabeth Gerber (Ford) Anne Gere (Sweetland) Tim McKay (LSA) Perry Samson (ENG) Ginger Schultz (LSA) LEAD BEHAVIORAL SCIENTIST Holly Derry LEAD DEVELOPERS Ben Hayward Cait Holman Kris Steinhoff Chris Teplovs LEAD INNOVATION ADVOCATE Amy Homkes-Hayes BEHAVIOR SCIENTIST Carly Thanhouser DATA SCIENTIST Kyle Schulz DEVELOPERS Dave Harlan Kushank Raghav Oliver Saunders Ke Ye UX & DESIGN Marie Hooper Kristin Miller Mike Wojan Plus 15-20 student fellows drawn from Computer Science, Social Psychology, Art & Design, UI Design, Behavioral Science, Education, & more….
- 52. Expert tailored communication • Built on 20+ yrs of digital health coaching • Aggregates rich student info from many sources to tailor feedback, encouragement, & advice • Tailoring on both what to say, how to say it, who speaks: w/testimonials from peers, etc. • All content written & tested by behavior change experts, faculty from disciplines, students • A tool for humane personalization w/student agency: allowing us to speak, share data, connect
- 54. ECoach is expanding to more classes, launching at other institutions, and supporting rich array of research projects w/external funding. This fall: 8000 students Stats 250 EECS 183, 280 Chem 130 Physics 140 Bio 171 Econ 101 Engr 100, 101 ALA 125 FirstYear (6800 more…) UC Santa Barbara ECoach Future ECoach supports research addressing widespread gendered performance differences in STEM lecture courses
- 57. Focus on Foundational Courses • Large, relatively stable, mostly introductory courses • Serve students with especially various backgrounds • Serve students with especially various interests and goals • Foundational courses where we educate at scale are ideal environments for the application of analytics • Best large courses taught in multigenerational teams with role specialization • Roles include: – Course management – Delivery of instruction on large and small scales – Instructional design – Technology – Assessment & analytics – Student support • Courses should be broadly ‘instrumented’ for study These ‘foundational’ courses exist across many disciplines, most of which are outside the natural sciences, so this initiative is campus-wide.
- 59. Five years from now… • Carefully designed and instrumented foundational courses established as one of the key elements of a learning laboratory • A well established program, with ~20-30 courses established as part of this laboratory • We aim to play an important role in establishing a robust evidence-basis for learning analytics & personalization at scale
- 60. Educating @ scale in 21st century • Teaching at scale in the information age affords unprecedented opportunities for personalization • Realizing these is a major sociotechnical challenge • We are addressing both the social and technical challenges associated with this task at Michigan • Our campus is creating a laboratory for learning at scale, connecting education research & practice • This will become a “learning higher education” community in which we learn continuously from experience in context • It can all be done in a student-centered way while protecting privacy