This is the keynote address from the 2016 midwest regional meeting of the Association for Authentic, Experiential, and Evidence Based Learning at Notre Dame University on May 12
7. Why Badges?
• Badges contain specific claims of competency,
achievement, completion, etc.
• Badges can contain evidence supporting those claims.
– The optional evidence field can contain links to additional
evidence (e.g., artifacts) supporting those claims
• These claims and evidence can readily circulate in social
networks, email, etc.
– Thus they gain additional information including endorsements
and (potential) rejection.
– Thus they can help potential earners find opportunities.
• Badges are interoperable and extensible.
– Earners can curate and annotate their collections.
– OBI Badges should be displayable in all future platforms
7
8. • MacArthur launches DML
initiative in 2006
• P2PU and Mozilla define
badges in 2010
• 2012-2014 Badges for
Lifelong Learning Initiative
• 2012-2015 Badges DPD
Project
• Credly and other startups
start emerging ~2013
• Badge Alliance launched in
2014
• Open Badges in Higher Ed
Project 2014-2016 8
11. • MacArthur’s $25M start
up & Project LRNG 2015
• Badge Alliance
reorganization in 2016
• LTI-compliant badges for
major LMSs
• IMS Global Open Badge
in Ed Extensions
• New JSON-LD (linked
data) standards
• DML 2016 Competition11
15. • Project Zero launches in
1967, portfolios in 1980s
• High-stakes K-12 portfolios
rise and fall 1990-1995
• ePortfolios emerge in higher
ed late 1990s
• Commercial platforms start
emerging ~2000
• INCEPR in 2003, AAEEBL in
2009
• Cambridge et. al, 2009,
Penny Light and Chen, 2011
• MCNRC/Catalyst in 201415
16. • Project Zero launches in
1967, portfolios in 1980s
• High-stakes K-12 portfolios
rise and fall 1990-1995
• ePortfolios emerge in higher
ed late 1990s
• Commercial platforms start
emerging ~2000
• INCEPR in 2003, AAEEBL in
2009
• Cambridge et. al, 2009,
Penny Light and Chen, 2011
• MCNRC/Catalyst in 201416
17. The Promise of ePortfolios
• Puts learners in charge of learning and displaying
• ePortfolio is the “common denominator” in the
move from teaching to learning
– Abundance of knowledge
– Knowledge is rapidly changing
– Economy that demands documented competency
– Dismal evidence of learning from legacy methods
– Changing nature of college students
• Complements other trends in higher education
– CBE, SRL, authentic assessment, personalization, self-
pacing, service learning, etc.
17
21. • Allow credit outside of formal credential context
– Give credit for things that are hard to grade
• Can provide additional useful information
– Specific claims and detailed evidence
– Context in which content was created
– Standardize the inclusion of additional information without
cluttering eportfolios
• May simplify the process of defining competencies
• May offload summative credentialing functions
– Allows more formative and transformative functions
• Can connect eportfolio content to competencies and grades
– Learners stack badge URLs in maps or gradebook
• Can increase value of portfolio content by circulating
independently
21
23. ev·i·dence /ˈevədəns/
• the available body of facts or information
indicating whether a belief or proposition is true
or valid
• In education, evidence usually concerns claims
about what someone learned and/or what they
will be able to do in the future.
• Validity is a property of claims not assessments
– Assessments can be reliable (but not valid)
– Validity is an argument
23
24. Standards for Educational and
Psychological Testing (NCME/APA/AERA)
1999
19851966
2014
• Different functions of assessment
– Summative vs. formative vs.
transformative.
• Different types of evidence
– Behavior vs. cognition vs.
participation.
• Different units of analysis
– Practices vs. people vs. programs
vs. institutions.
• Competing purposes create
tensions
– Many unintended consequences
– Focus on functions instead
24
25. Validity is Contentious
• Traditional Conception
– Criterion, content, and construct-related evidence
• Modern “aspects of construct validity” from
Messick
– Content, substantive, structural, generalizability,
external, and consequential
– Shepard, Moss, Hickey, and others argue that
consequences should be considered first.
• Primary threats to validity
– Construct under-representation
– Construct-irrelevant variance
25
26. • Boundary objects are particular
kinds of cultural tools (Star and
Grisemer, 1989)
– “both plastic enough to adapt to local
needs and constraints of the several
parties employing them, yet robust
enough to maintain a common
identity across sites.
– “weakly structured in common use,
and become strongly structured in
individual-site use.
– “They may be abstract or concrete.
They have different meanings in
different social worlds but their
structure is common enough to more
than one world to make them
recognizable, a means of translation.”26
27. • Six dimensions in which evidence crosses
boundaries:
1. Types of evidence (artifacts vs. scores)
2. Explicitness of evidence (systematic?)
3. Extent to which actors from the local context
are available in the new context
4. Comprehensiveness of representation of local
context (foreground vs. background)
5. The interpretive norms and routines that
accompany the evidence
6. Roles of local and external actors in shaping
representations of practice. 27
28. • “When evidence crosses boundaries, it
brings far more than information.”
– Cultural tools (e.g., artifacts & concepts).
– Norms and routines mediate understanding
and (inter)action in sending and receiving
contexts.
• “Positions the different actors with different
authority, accountability, and agency for
making decisions about how their practices
are represented and how those
representations should be interpreted and
used”
28
29. • Students do meaningful work in ePortfolios
– Including curricular and co-curricular
– Artifacts are shared, discussed and endorsed.
• Students earn badges for the work
– Highlights competencies and presents context
– Competencies are shared, discussed, and endorsed
• Students “stack” badges into competency maps
or gradebooks by simply pasting URLs
– Instructors assign grades and private feedback
– Information is FERPA-protected
– Same badge can be in a gradebook or a competency
map
29
32. PLA DESIGN PRINCIPLES
Use public contexts give meaning to
knowledge tools
Publically support and reward productive
disciplinary engagement
Grade artifacts through local reflections
Let individuals assess their understanding
privately
Measure aggregated achievement discreetly 32
33. 2014 2016
Suspended None
Suspended None
Partial Existing
Implemented Existing
Implemented Thriving
Implemented Thriving
Status of 6 (of 29) Projects
from 2012 Badges for Lifelong Learning Initiative
33
34. Open Questions and Discussion
• Are badges even going to endure?
– If it does not what micro-credentialing practice will endure
• Is carpetbadging really a problem?
– Maybe “bro-badges” have a useful function
– Democratization of credentialing?
• Are badges redundant with ePortfolio functions?
• Might the OBI standards be “IMS-OBEE lite”
• What is the financial model for badges?
– e.g., Achievery shut down in 2015
– Registrars charge for copies, platforms and assessment firms charge
for licenses.
• Sustainability of badges
• Unintended consequences?
• Other issues?
34
Editor's Notes
This is my team
I did my PhD in psychology at Vanderbilt and have spent most of my career doing assessment and evaluation with cutting edge educational technology. Jasper Woodbury videodisks at Vanderbilt, GenScope Multimedia software as a postdoc at the ETS center for performance assessment and on the faculty at Georgia, NASA Classroom of the Future as a faculty member in Georgia, and Quest Atlantis, Sakai, and Project NML at Indiana. So of course I am thrilled to be working with what I think will be the most important technology of all.
I also study motivation and incentives, and have thought about badges a lot. I have done empirical studies of them in both Quest Atlantis and Sakai and hope to have them published soon.
I focus on the implications of sociocultural theories of knowing and learning for assessment, motivation, research, and evaluation. There is a research literature in each area that is directly relevant to the goals of the DML initiaive
This is my team,
Happy to be working with Shery Grant and others at HASTAC—you are in the hands of a very capable team there and we have had lots of discussion about how we can help serve both the initiative and individual projects
Also thrilled to be working with Mozilla. Erin Knight has been awesome. As I will show you Carla Casilli is introducing some very important ideas, and I understand that Peter Rawsthorne form England is joining the team as well.
Technically most of my work will be through HASTAC and focused on awardees. But there is an enourmous community of finalists out there who have been inspired by the compeition and are forging ahead as we speak.
1. Knowledge is no longer scarce but abundant (learning to discover sources of knowledge is therefore important)
2. Knowledge is in a period of rapid change (and therefore learning how to learn is most important)
3. The economy demands graduates with a documented record of competency or achievement
4. Learning researchers have suggested that legacy learning designs are not as effective as they need to be for the learner of today
5. Students in higher education are, on average, older than traditional college-age students and their demands and needs tend toward real-world application.
types of evidence (narratives of practice, samples of student work, videotapes of classroom inter-action, interviews and observations, and surveys, tests, etc)
the extent to which the evidence is explicitly and systematically documented,
whether and how actors from the local context accompany the evidence into the new context and can thus recontextualize it,
the comprehensiveness of the representation of the local context (what is made visible and what remains in the background),
what norms and routines accompany the providing and interpretation of evidence, and
what roles the actors in the local and external contexts play in shaping the way in which practice is represented.
types of evidence (narratives of practice, samples of student work, videotapes of classroom inter-action, interviews and observations, and surveys, tests, etc)
the extent to which the evidence is explicitly and systematically documented,
whether and how actors from the local context accompany the evidence into the new context and can thus recontextualize it,
the comprehensiveness of the representation of the local context (what is made visible and what remains in the background),
what norms and routines accompany the providing and interpretation of evidence, and
what roles the actors in the local and external contexts play in shaping the way in which practice is represented.