LAK19 Full Paper. Abstract: To support ethical considerations and system integrity in learning analytics, this paper introduces two cases of applying the Value Sensitive Design methodology to learning analytics design. The first study applied two methods of Value Sensitive Design, namely stakeholder analysis and value analysis, to a conceptual investigation of an existing learning analytics tool. This investigation uncovered a number of values and value tensions, leading to design trade-offs to be considered in future tool refinements. The second study holistically applied Value Sensitive Design to the design of a recommendation system for the Wikipedia WikiProjects. To proactively consider values among stakeholders, we derived a multi-stage design process that included literature analysis, empirical investigations, prototype development, community engagement, iterative testing and refinement, and continuous evaluation. By reporting on these two cases, this paper responds to a need of practical means to support ethical considerations and human values in learning analytics systems. These two cases demonstrate that Value Sensitive Design could be a viable approach for balancing a wide range of human values, which tend to encompass and surpass ethical issues, in learning analytics design.

1. Towards Value-Sensitive
Learning Analytics Design
Bodong Chen, Curriculum & Instruction
Haiyi Zhu, Computer Science & Engineering
LAK Conference | 2019-03-08
1 / 37

2. Ethical considerations
in learning analytics
(Slade & Prinsloo, 2013; Ferguson et al., 2016)
2 / 37

3. Ethical frameworks
Codes of ethical practice
Ethical consideration
checklists
Empirical work of
stakeholders
(Drachsler & Greller, 2016; Ifenthaler &
Tracey, 2016; Sclater, 2016; Willis, Slade, &
Prinsloo, 2016)
3 / 37

4. 2017 - The ACM Statement on Algorithmic
Transparency and Accountability
2018– - The ACM FAT* Conference
2018 - #LAK18 & JLA Special Section on
Human-Centered Learning Analytics
2019 - #LAK19 FairLAK workshop;
papers on evaluating fairness
(Photo Credit)
Transparency, Accountability, Fairness
4 / 37

5. Learning Analytics System Integrity
(Buckingham Shum, 2017, Black box learning analytics? Beyond algorithmic transparency)
5 / 37

6. Ethical Design Critique (EDC)
(Buckingham Shum, 2018, Ethical Design Critique for Learning Analytics Dashboards)
6 / 37

7. Value-Sensitve Design
Another toolkit for a very hard problem
7 / 37

8. "Something important, worthwhile, or
useful;" "one's judgment of what is
important in life" (Oxford English
Dictionaries, 2018).
"What a person or group of people
consider important in life" (Borning &
Muller 2012).
"What is important to people in their lives,
with a focus on ethics and morality"
(Friedman, Hendry, & Borning, 2017)
(Source: "It's about power", ACM Communications)
Values (vs. Ethics)
8 / 37

9. Value Sensitive Design (VSD) is a
methodology and an established set of
methods for addressing values
a pioneering endeavor to proactively
consider human values throughout
the process of technology design
(Davis & Nathan. 2013)
offers a systemic approach with
speci c strategies and methods to
explicitly incorporate the
consideration of human values into
design (Friedman et al., 2017)
Value-Sensitve Design
9 / 37

10. Tripartite Methodology
(Boston, 2016; Friedman et al., 2017)
10 / 37

11. Direct and Indirect Stakeholder
Analysis
Value Source Analysis
Co-evolution of Technology and Social
Structure
Value Scenario
Value Sketch
Value-oriented Semi-structured
Interview
Value-oriented Coding Manual
Value-oriented Mock-up, Prototype or
Field Deployment
Value "Dams" and "Flows"
(Friedman et al., 2017, A Survey of Value
Sensitive Design Methods)
VSD Methods
11 / 37

12. Study 1: A conceptual
investigation
of a social learning analytics tool
Study 2: A holistic
application
to the design of a recommendation system
for WikiProjects
12 / 37

13. Social Media for Learning
(Greenhow & Lewin, 2016; Haythornthwaite, Priya, et al., 2018)
13 / 37

14. Yellowdig
(Photo Credit)
14 / 37

15. (Photo Credit)
Provide another way of engaging with
the class discussion,
Enhance student learning and social
interaction, and
Assist the instructor to grasp
discussion content in order to make
informed instructional decisions
Yellowdig Visualization Tool
15 / 37

16. Tool Features (demo)
Multi-mode, interactive
Temporal
Lexical (instrutor only)
16 / 37

17. Methods
1. Stakeholder analysis
2. Value analysis
17 / 37

18. Direct
Instructors
Students
Active
In-active
Indirect
Future cohorts of the class
Academic advisors
1. The Stakeholders
18 / 37

19. 2. Values
Design Choices Instructors Active Students Inactive Students
What to show Richer data
Utility: navigating, sense­making
Utility: assessing
Freedom from bias
Autonomy
Privacy
Usability: cognitive load
(Bereiter, 2014, Principled Practical Knowledge) 19 / 37

20. 2. Values
Design Choices Instructors Active Students Inactive Students
How to show
Node highlighting
(student or post)
Utility: navigating, sense­making
Utility: intervening
Social interaction
Self­image Self­image
Privacy
Force­directed layout Utility: intervening
Sense of community Self­image
Freedom from bias
Content filter
(instructor only)
Utility: sense­making
Utility: sense­making
Fair access
20 / 37

21. Design trade-offs
Value tensions Analysis Representation Action
utility <­­> autonomy;
privacy; self­image
active <­­> inactive
students
make student nodes not
highlightable
provide the option of not
revealing one's name
freedom from bias
<­­> self­image
consider other graph
layout algorithms
study student
perceptions of layouts
fair access
<­­> utility: sense­
making
<­­> privacy
give student access to
text mining features
reserve the student
filter to the instructor
(Siemens, 2013) 21 / 37

22. Study 1: A conceptual
investigation
of a social learning analytics tool
Study 2: A holistic
application
to the design of a recommendation system
for WikiProjects
22 / 37

23. WikiProjects
Self-organized groups of contributors within the English Wikipedia community.
23 / 37

24. Problem and Goal
 WikiProjects have been declining
Only 12% projects have 10+ active members
 To build a recommendation system
to help WikiProjects identify and recruit suitable new members
24 / 37

25. A Five-Stage Approach
(Zhu et al., 2018, CSCW) 25 / 37

26. Stage 1. Understand Stakeholders
26 / 37

27. Stage 1. Understand Stakeholders
A literature review + A survey study --> Stakeholders' values
Design Choices
New project
comers
Current project
members
Wikipedia
community
Who to
recruit
Experienced
editors
Collaboration Productivity
Member retention
Brand­new
editors
Mentorship Productivity
New member
retention
How to
define fit
Interest­based Mutual interest Mutual interest
Relation­based
Personal
connection
Productivity
How to
decide
Automation Control
Human­in­the­
loop
Control
27 / 37

28. Stage 2. Prototyping - "Who to Recruit"
Design Choices
New project
comers
Current project
members
Wikipedia
community
Who to
recruit
Experienced
editors
Collaboration Productivity
Member retention
Brand­new
editors
Mentorship Productivity
New member
retention
Value Tension -> Design Choice
Evaluateing both brand-new and experienced editors
Ranking the two types of editors separately
28 / 37

29. Interest-based
Rule-based
Category-based
Relationship-based
Bond-based
Co-edit-based
Stage 2. Prototyping - "How to de ne t"
Design Choices
New project
comers
Current project
members
Wikipedia
community
How to define
fit
Interest­
based
Mutual interest Mutual interest
Relation­
based
Personal connection Productivity
Four algorithms
(see Zhu et al., 2018, CSCW) 29 / 37

30. Stage 2. Prototyping - "How to decide"
Design Choices
New project
comers
Current project
members
Wikipedia
community
How to
decide
Automation Control
Human­in­the­
loop
Control
Pitfall -> Design choice
Design a dashboard to communicate recommendations to project members
30 / 37

31. Stage 2. Prototyping - "How to decide"
Analytic results communicated to current WikiProjects members:
31 / 37

32. Stage 3-4. Deploy, Iterate and Re ne
Over a six-month period:
The analytic system
analyzed information from 16,000 editors
delivered 4 distinct batches of 385 recommendations to 18 Wikiprojects
Re ned the system prototype based on stakeholder feedback
32 / 37

33. Stage 5: Evaluate
Messages from current WikiProjects members
"This puts some science behind recommendations, and will be a great
supplement to the current processes. "
Messages from invited new members
"Thank you for reaching out to me and thank you for informing me about the
WikiProject Africa talk page... I appreciate it. "
(see Zhu et al., 2018, CSCW) 33 / 37

34. Stage 5: Evaluate
Response from the Wikipedia community: a Signpost (news story) about the design
34 / 37

35. Summary
Study 1: A conceptual investigation of a social
learning analytics tool
Identi ed stakeholders and their values
Elicited value tensions
Suggested design tradeoffs to address value tensions
Study 2: A holistic application to the design of a
recomendation system
Developed a ve-stage approach that integrate multiple methods
Designed with end-users
Involved the user community in the design process
Iterated over an extended period of time
35 / 37

36. Implications and Future Directions
Value Sensitive Design provides a toolkit to:
1. guide the evaluation and re nement of existing learning analytics applications
2. help holistically addressing values in the design of new learning analytics
Currently applying VSD to the design of the Personal Learning Compass tool ...
36 / 37

37. Thank You!
Full Paper: https://doi.org/10.1145/3303772.3303798
Preprint: https://arxiv.org/abs/1812.08335
 chenbd@umn.edu
 @bod0ng
 Personal website: http://meefen.github.io/
 Research group: https://colig.github.io/
37 / 37