“We Don’t Do That Here”: How Collaborative Editing with Mentors Improves Engagement in Social Q&A Communities
•
2 recomendaciones•573 vistas
Slides from CHI 2018 paper presentation describing the pilot Stack Overflow just-in-time mentorship program. Abstract: Online question-and-answer (Q&A) communities like Stack Overflow have norms that are not obvious to novice users. Novices create and post programming questions without feedback, and the community enforces site norms through public downvoting and commenting. This can leave novices discouraged from further participation. We deployed a month long, just-in-time mentorship program to Stack Overflow in which we redirected novices in the process of asking a question to an on-site Help Room. There, novices received feedback on their question drafts from experienced Stack Overflow mentors. We present examples and discussion of various question improvements including: question context, code formatting, and wording that adheres to on-site cultural norms. We find that mentored questions are substantially improved over non-mentored questions, with average scores increasing by 50%. We provide design implications that challenge how socio-technical communities onboard novices across domains.
Recomendados
Más contenido relacionado
Similar a “We Don’t Do That Here”: How Collaborative Editing with Mentors Improves Engagement in Social Q&A Communities
Similar a “We Don’t Do That Here”: How Collaborative Editing with Mentors Improves Engagement in Social Q&A Communities (20)
Más de Denae Ford
Más de Denae Ford (9)
Último
Último (20)
“We Don’t Do That Here”: How Collaborative Editing with Mentors Improves Engagement in Social Q&A Communities
- 1. “We Don’t Do That Here”: How Collaborative Editing with Mentors Improves Engagement in Social Q&A Communities Denae Ford North Carolina State University Kristina Lustig Stack Exchange, Inc. Jeremy Banks Stack Exchange, Inc. Chris Parnin North Carolina State University @DenaeFord
- 3. 3 source: https://media.giphy.com/media/xT9IgHKDptUFWWMkve/giphy.gif source: https://media.tumblr.com/3d150f8da60feb45ed3254b5721e5c44/tumblr_inline_mjw0l2etMI1qz4rgp.gif What to ask? How to ask? Where to get that info?
- 4. 4 [Burnett et al., 2016]
- 5. 5 What to ask? Question may be a duplicate since list of existing questions appear
- 6. 6 How to ask? Unfamiliar with community expectations and feel like question is unfit
- 7. 7 Where to get that info? Don’t see other users with a similar new experience
- 8. 8 [Stack Overflow 2018 Developer Survey] Even though, about 50 million people visit Stack Overflow monthly…
- 9. 9 [Stack Overflow 2017 Developer Survey] …not everyone contributes. [Stack Overflow 2018 Developer Survey] 7.6% v.s. 88.6% ~87% have accounts—but 20% have never posted
- 10. Collaborative Editing Two core components: Collaborative question draft Help Room to discuss and edit the draft 10 2 1
- 11. Design Principles Provide formative and timely feedback Allow mistakes in private space Do not answer questions, help others ask better questions 11 [Epstein et al., 2002] [Baron 1993] [Wenger, 1998] [Kraut et al., 2012] [Kraut et al., 2012][Fedor et al., 1993] [Ford et al., 2016] [Kraut et al., 2012]
- 13. Study Design Goals To observe and measure changes in question quality To understand feedback provided by mentors To improve experience for new question askers To learn how to scale these benefits to the whole community 13
- 14. How it works 14
- 16. Novice writes question Novice selects to post their question
- 17. Novice writes question Novice selects to post their question Novice selects to join Help Room
- 18. Novice writes question Novice selects to post their question Novice selects to join Help Room
- 19. Novice receives feedback on question Novice writes question Novice edits question draft Novice selects to post their question Novice selects to join Help Room
- 20. Novice receives feedback on question Novice writes question Novice selects to post their question Novice selects to join Help Room
- 21. Novice receives feedback on question Novice writes question Novice selects to post their question Novice selects to join Help Room Petter Friber 2 days later… Mason 3 hours later… Jerem y The last message was posted 5 days ago. }; edit copy draft back to Stack Overflow Hey @KristinaLustig, thanks for coming. :) This is the Stack Overflow chat room we're using to help people improve their questions. This message is automated but everything else is real! Before we start: do you have any particular concerns, or are you just doublechecking before you post? Welcome @KristinaLustig I'm here to help you with the question, are you in chat? edited post draft. room mode changed to Private: only approved users may enter this room (this should only be used for moderation purposes) Wed 15:19 chat they will b leave send help | f
- 22. Novice receives feedback on question Novice writes question Novice edits question draft Novice selects to post their question Novice selects to join Help Room Novice posts question
- 23. 23 [Burnett et al., 2016]
- 24. 343 conversations Data Collected 24 33 day study 71,068 Eligible Novices 271 Novices 343 conversations Transcripts + Question Scores 63 Mentors Interview Mentors Survey Novices 4 Help Rooms 1 Private Mentor Room 520 Entered Room
- 25. Findings 25
- 26. Mentored questions have higher quality. MENTORED QUESTIONS HAD A 50% SCORE INCREASE 26 GOOD NEUTRAL BAD mentored 25% 49% 25% not mentored (control) 18% 51% 30% 6 -30 p <0.05 p <0.05
- 27. Mentors suggest high-fidelity improvements. Paraphrasing title and content (Question Phrasing) Formatting code to make question scannable Community triage Including all the pieces of a question (Question Framing) Community culture of asking 27
- 28. Mentors suggest high-fidelity improvements. Paraphrasing title and content (Question Phrasing) Formatting code to make question scannable Community triage Including all the pieces of a question (Question Framing) Community culture of asking 28
- 30. Community Triage 30 “ok the fact is that on SO you can’t ask for libs you would need to do that on another site [link] (with some [rules] that we can check if you like), instead if you like some code it would be really great if you tried something, do you have some code, do you have some post that you already checked?” (A30 -Mentor)
- 32. Community Triage 32 “Your question is off topic here....We’re not a hacking service.” (C72 -Mentor) "ok the fact is that on SO you can’t ask for libs you would need to do that on another site [link] (with some [rules] that we can check if you like), instead if you like some code it would be really great if you tried something, do you have some code, do you have some post that you already checked?” (A30 -Mentor)
- 33. Community Culture of Asking 33
- 34. Community Culture of Asking 34 “You can probably remove ‘Hello’ and ‘Problem’ from the top of the question. While it’s good to be social, it’s kind of just fluff on a Q&A site.” (D1 -Mentor)
- 35. Community Culture of Asking 35 “You can probably remove ‘Hello’ and ‘Problem’ from the top of the question. While it’s good to be social, it’s kind of just fluff on a Q&A site.” (D1 -Mentor) “You also might want to edit out the “Thank you!” at the end. I know it seems polite, but people object to it on Stack Overflow.” (D5 -Mentor) “Ok thanks for you help. I hope this time people won’t attack me.” (D62 -Novice)
- 36. Participants are satisfied with their interactions. 36 Likert Statement Median I feel that I am a part of the Stack Overflow community. 4 I feel more comfortable posting on Stack Overflow. 4 The help that I received from this program was useful to me. 5 I would recommend this program to other SO users. 5 I would like to participate in this program again. 5 NOVICES SATISFIEDWITH THEIR EXPERIENCE “If we can get the [original poster] through the first question with a positive experience and they can see how this site really works, then we should get more good questions which feeds in to having more good answers.” (M5 -Mentor) MENTORS INTERESTEDIN OFFERING GUIDANCE
- 38. Having a human mentor surprised novices and made them feel welcomed Interpret dialogue for non-native English speaker Organic conversations to work through questions Advantages of humans helping humans 38 Source: https://www.businessinsider.com.au/why-the-iraq-war-cost-2-trillion-2013-3
- 39. Identity-Based Mentoring 39 Novice: Yeah, actually I am! I thought I picked up on your name too! Mentor: Hey! are you from Egypt by chance? Identity-Based Mentoring [Stack Overflow 2018 Developer Survey] Egypt Yossef
- 40. 40 Implications for community-based mentorship Asking community members about their interest Mentor many-to-many, one-to-many, and one-to-one Integrate mentorship functionality precisely when needed Source: http://humansofthearsenal.com/post/143654160037/im-taking-part-because-we-want-things-to-change
- 41. 41 Implications for community-based mentorship http://community.rstudio.com/ https://www.askquestions.tech/faq http://pythontutor.com/visualize.html
- 42. Online Q&A communities are widely used, but unfamiliar norms can leave novices discouraged from participating. Just-in-time mentoring provided novices with help to ask a well- received question, precisely when they needed it. Identifying novice-mentor interactions can define new onboarding and feedback experiences in other online communities. http://bit.ly/StackMentorPodcast http://bit.ly/StackMentorPaper dford3@ncsu.edu Summary
- 43. http://bit.ly/StackMentorPodcast http://bit.ly/StackMentorPaper dford3@ncsu.edu @DenaeFord @KristinaLustig @JeremyBanks @ChrisParnin Thank You! Online Q&A communities are widely used, but unfamiliar norms can leave novices discouraged from participating. Just-in-time mentoring provided novices with help to ask a well- received question, precisely when they needed it. Identifying novice-mentor interactions can define new onboarding and feedback experiences in other online communities.
- 44. ++ 44
- 45. Feature Implementation 45 Expanded existing Stack Overflow chat rooms Novices: eligible novice drafting a question Mentors: Experienced users