Presentation at Open Education Conference 2012. The ALMS rubric measuring the technical difficulty of reusing OER.

1. Establishing inter-
rater reliability with
the ALMS rubric

2. Credit
• David Wiley
• Utah Valley University
• Brigham Young University

3. Purpose of study
Three barriers to reuse:
• Content
• Pedagogical
• Technical

4. Study Design
• Delphi study (n=5)
• Inter-rater reliability (n=4)

5. Technical barriers
ALMS rubric:
• Access to editing tools
• Level of expertise
• Meaningfully Editable
• Source Files

6. Access to Editing Tools
• Most difficult to measure
• Context-dependent
• Delphi study participants were least confident

7. Solution: Access to Editing Tools
MP3 WAV OGG AIFF WMA RM
Is an appropriate
software application
pre-installed with the 2 2 2 1 1 1
operating system?
Mean 3.00 2.88 2.88 2.00 1.61 1.39

8. Findings
• Operating system didn’t matter as much
• Why?
• Web services
• Ubiquity of editing services
• Popularity of file types

9. Level of Expertise
• Set to three levels of editing (like beginner, intermediate, advanced)
• What degree of _____ editing expertise is required?
• Text
• Image
• Audio
• Video

10. Meaningfully Editable
• What portion of the ______ in the OER is editable?
• Text
• Image
• Audio
• Video

11. Source Files
• What portion of _____ are editable?
• Text
• Image
• Audio
• Video

12. Establishing Reliability: Raters
• 4 raters
• Taken from grad. Program at BYU

13. Inter-rater sample
Type NROC MIT OCW WikiEducator
Humanities (e.g. Art,
Literature) 3 samples 3 samples 3 samples
Social Sciences (e.g.
Education, Psychology) 3 samples 3 samples 3 samples
Sciences (e.g. Biology, Math)
3 samples 3 samples 3 samples

14. Results
• ICC(2,1) =.655, df=376, 95% CI [.609, .699]
• Could be better

15. Results: Order
Order Mean Std. Dev.
1 2.91 .17
2 2.81 .25
3 2.80 .27
Mean 2.84 .23

16. Results: Disciplines
Repository Mean Std. Dev.
Humanities 3.02 .16
Social Sciences 2.83 .19
Sciences 2.67 .32
Mean 2.84 .22

17. Results: Repository
Repository Mean Std. Dev.
NROC 2.36 .33
MIT OCW 2.99 .2
WikiEducator 3.16 .15
Mean 2.84 .23

18. Conclusions
• ALMS rubric has some agreement, but could be refined
• There are differences in repositories with respect to reuse
• There may be differences among disciplines