SlideShare una empresa de Scribd logo
1 de 11
Toward Effective & Sustainable
Online Crowd Work
Matt Lease
School of Information @mattlease
University of Texas at Austin ml@utexas.edu
Matt Lease <ml@utexas.edu>
2
New Capabilities Enable Innovation
But also new “Dirty” Digital Jobs
• Tasks
– Content moderation (Catch-22)
– Solving captchas, SEO, account creation, …
• Goals
– “Legal” fraud, cyber-crime, or cyber-warfare
• Workers
– Unknown or at-risk: refugees, prisoners, children, …
– Typical lack of informed consent
– How do we know work is freely chosen?
• Part of larger, online ecosystem (cf., Robert Sim)
Matt Lease <ml@utexas.edu>
3
Matt Lease <ml@utexas.edu>
4
Debate in ACM
5
Paul Hyman. Communications of the ACM, Vol. 56 No. 8, Pages 19-21, August 2013.
Matt Lease <ml@utexas.edu>
• Contribute to society and human well-being
• Avoid harm to others
• Be honest and trustworthy
• Be fair and take action not to discriminate
• Respect the privacy of others
COMPLIANCE WITH THE CODE. As an ACM member I will
– Uphold and promote the principles of this Code
– Treat violations of this code as inconsistent with membership in the ACM
Financial and economic imperatives …Which approaches are less
expensive and is this sensible? With the advent of outsourcing
and off-shoring these matters become more complex… there
are often related ethical issues concerning exploitation…
6
Matt Lease <ml@utexas.edu>
What about regulation?
• Wolfson & Lease (ASIS&T 2011)
• As usual, technology ahead of the law
– employment law & independent contractors
– patent inventorship
– data security and the Federal Trade Commission
– copyright ownership
– securities regulation of crowdfunding
• Understand risks of “just in-time compliance”
7
Matt Lease <ml@utexas.edu>
Seek Win-Win Solutions
• Pay debate has centered on quality vs. cost tradeoff
• What is “fair” and sustainable in a global economy?
– Is it better to pay nothing? (i.e., volunteers, gamification)
• A meta-approach
– Learn: Find information (e.g., Internet, surveys)
– Share: Publicize information for others
– Act: Find ways to make the “right” thing preferable
Matt Lease <ml@utexas.edu>
8
The Future of Crowd Work, CSCW’13
Kittur, Nickerson, Bernstein, Gerber,
Shaw, Zimmerman, Lease, and Horton
9
Matt Lease <ml@utexas.edu>
Join the conversation!
Crowdwork-ethics, by Six Silberman
http://crowdwork-ethics.wtf.tw
an informal, occasional blog for researchers
interested in ethical issues in crowd work
ir.ischool.utexas.edu
Unanticipated
challenges & risks with
emerging technologies
Lease et al., SSRN’13
11

Más contenido relacionado

La actualidad más candente

The Rise of Crowd Computing (December 2015)
The Rise of Crowd Computing (December 2015)The Rise of Crowd Computing (December 2015)
The Rise of Crowd Computing (December 2015)Matthew Lease
 
The Rise of Crowd Computing - 2016
The Rise of Crowd Computing - 2016The Rise of Crowd Computing - 2016
The Rise of Crowd Computing - 2016Matthew Lease
 
Data Standards and Linked Data: Challenges & Use Cases in Europe and the Unit...
Data Standards and Linked Data: Challenges & Use Cases in Europe and the Unit...Data Standards and Linked Data: Challenges & Use Cases in Europe and the Unit...
Data Standards and Linked Data: Challenges & Use Cases in Europe and the Unit...Jonathan Pichot
 
Ralph schroeder and eric meyer
Ralph schroeder and eric meyerRalph schroeder and eric meyer
Ralph schroeder and eric meyeroiisdp
 
Joe keating - world legal summit - ethical data science
Joe keating  - world legal summit - ethical data scienceJoe keating  - world legal summit - ethical data science
Joe keating - world legal summit - ethical data scienceJoe Keating
 
Privacy-driven design of Learning Analytics applications – exploring the desi...
Privacy-driven design of Learning Analytics applications – exploring the desi...Privacy-driven design of Learning Analytics applications – exploring the desi...
Privacy-driven design of Learning Analytics applications – exploring the desi...Tore Hoel
 
AI and Legal Tech in Context: Privacy and Security Commons
AI and Legal Tech in Context: Privacy and Security CommonsAI and Legal Tech in Context: Privacy and Security Commons
AI and Legal Tech in Context: Privacy and Security Commonsprofessormadison
 
The Rise of Crowd Computing (July 7, 2016)
The Rise of Crowd Computing (July 7, 2016)The Rise of Crowd Computing (July 7, 2016)
The Rise of Crowd Computing (July 7, 2016)Matthew Lease
 
Student vulnerability, agency and learning analytics: an exploration
Student vulnerability, agency and learning analytics: an explorationStudent vulnerability, agency and learning analytics: an exploration
Student vulnerability, agency and learning analytics: an explorationUniversity of South Africa (Unisa)
 
Crowdsourcing for Search Evaluation and Social-Algorithmic Search
Crowdsourcing for Search Evaluation and Social-Algorithmic SearchCrowdsourcing for Search Evaluation and Social-Algorithmic Search
Crowdsourcing for Search Evaluation and Social-Algorithmic SearchMatthew Lease
 
Feedback on Technology Governance, Strategy, and Funding Proposal: Executive ...
Feedback on Technology Governance, Strategy, and Funding Proposal: Executive ...Feedback on Technology Governance, Strategy, and Funding Proposal: Executive ...
Feedback on Technology Governance, Strategy, and Funding Proposal: Executive ...Ron Dolin
 
Crowd Computing: Opportunities & Challenges (IJCNLP 2011 Keynote)
Crowd Computing: Opportunities & Challenges (IJCNLP 2011 Keynote)Crowd Computing: Opportunities & Challenges (IJCNLP 2011 Keynote)
Crowd Computing: Opportunities & Challenges (IJCNLP 2011 Keynote)Matthew Lease
 
INSPIRE Conference 2013 - Environmental Data on the Web
INSPIRE Conference 2013 - Environmental Data on the WebINSPIRE Conference 2013 - Environmental Data on the Web
INSPIRE Conference 2013 - Environmental Data on the WebAlex Coley
 

La actualidad más candente (17)

The Rise of Crowd Computing (December 2015)
The Rise of Crowd Computing (December 2015)The Rise of Crowd Computing (December 2015)
The Rise of Crowd Computing (December 2015)
 
The Rise of Crowd Computing - 2016
The Rise of Crowd Computing - 2016The Rise of Crowd Computing - 2016
The Rise of Crowd Computing - 2016
 
Data Standards and Linked Data: Challenges & Use Cases in Europe and the Unit...
Data Standards and Linked Data: Challenges & Use Cases in Europe and the Unit...Data Standards and Linked Data: Challenges & Use Cases in Europe and the Unit...
Data Standards and Linked Data: Challenges & Use Cases in Europe and the Unit...
 
Ralph schroeder and eric meyer
Ralph schroeder and eric meyerRalph schroeder and eric meyer
Ralph schroeder and eric meyer
 
Joe keating - world legal summit - ethical data science
Joe keating  - world legal summit - ethical data scienceJoe keating  - world legal summit - ethical data science
Joe keating - world legal summit - ethical data science
 
Privacy-driven design of Learning Analytics applications – exploring the desi...
Privacy-driven design of Learning Analytics applications – exploring the desi...Privacy-driven design of Learning Analytics applications – exploring the desi...
Privacy-driven design of Learning Analytics applications – exploring the desi...
 
AI and Legal Tech in Context: Privacy and Security Commons
AI and Legal Tech in Context: Privacy and Security CommonsAI and Legal Tech in Context: Privacy and Security Commons
AI and Legal Tech in Context: Privacy and Security Commons
 
The Rise of Crowd Computing (July 7, 2016)
The Rise of Crowd Computing (July 7, 2016)The Rise of Crowd Computing (July 7, 2016)
The Rise of Crowd Computing (July 7, 2016)
 
Data Science and its impact on society
Data Science and its impact on societyData Science and its impact on society
Data Science and its impact on society
 
Student vulnerability, agency and learning analytics: an exploration
Student vulnerability, agency and learning analytics: an explorationStudent vulnerability, agency and learning analytics: an exploration
Student vulnerability, agency and learning analytics: an exploration
 
Crowdsourcing for Search Evaluation and Social-Algorithmic Search
Crowdsourcing for Search Evaluation and Social-Algorithmic SearchCrowdsourcing for Search Evaluation and Social-Algorithmic Search
Crowdsourcing for Search Evaluation and Social-Algorithmic Search
 
Feedback on Technology Governance, Strategy, and Funding Proposal: Executive ...
Feedback on Technology Governance, Strategy, and Funding Proposal: Executive ...Feedback on Technology Governance, Strategy, and Funding Proposal: Executive ...
Feedback on Technology Governance, Strategy, and Funding Proposal: Executive ...
 
Crowd Computing: Opportunities & Challenges (IJCNLP 2011 Keynote)
Crowd Computing: Opportunities & Challenges (IJCNLP 2011 Keynote)Crowd Computing: Opportunities & Challenges (IJCNLP 2011 Keynote)
Crowd Computing: Opportunities & Challenges (IJCNLP 2011 Keynote)
 
INSPIRE Conference 2013 - Environmental Data on the Web
INSPIRE Conference 2013 - Environmental Data on the WebINSPIRE Conference 2013 - Environmental Data on the Web
INSPIRE Conference 2013 - Environmental Data on the Web
 
Knoesis Student Achievement
Knoesis Student AchievementKnoesis Student Achievement
Knoesis Student Achievement
 
Introduction to data science
Introduction to data scienceIntroduction to data science
Introduction to data science
 
Broad Data
Broad DataBroad Data
Broad Data
 

Similar a Toward Effective and Sustainable Online Crowd Work

Crowdsourcing: From Aggregation to Search Engine Evaluation
Crowdsourcing: From Aggregation to Search Engine EvaluationCrowdsourcing: From Aggregation to Search Engine Evaluation
Crowdsourcing: From Aggregation to Search Engine EvaluationMatthew Lease
 
Crowdsourcing & ethics: a few thoughts and refences.
Crowdsourcing & ethics: a few thoughts and refences. Crowdsourcing & ethics: a few thoughts and refences.
Crowdsourcing & ethics: a few thoughts and refences. Matthew Lease
 
AI & Work, with Transparency & the Crowd
AI & Work, with Transparency & the Crowd AI & Work, with Transparency & the Crowd
AI & Work, with Transparency & the Crowd Matthew Lease
 
The Challenge of Benefit-Cost Analysis As Applied to Online Safety & Digital ...
The Challenge of Benefit-Cost Analysis As Applied to Online Safety & Digital ...The Challenge of Benefit-Cost Analysis As Applied to Online Safety & Digital ...
The Challenge of Benefit-Cost Analysis As Applied to Online Safety & Digital ...Adam Thierer
 
Explainable Fact Checking with Humans in-the-loop
Explainable Fact Checking with Humans in-the-loopExplainable Fact Checking with Humans in-the-loop
Explainable Fact Checking with Humans in-the-loopMatthew Lease
 
A Lifecycle Approach to Information Privacy
A Lifecycle Approach to Information PrivacyA Lifecycle Approach to Information Privacy
A Lifecycle Approach to Information PrivacyMicah Altman
 
Thierer Internet Privacy Regulation
Thierer Internet Privacy RegulationThierer Internet Privacy Regulation
Thierer Internet Privacy RegulationMercatus Center
 
Governing Privacy Commons at Pitt Science 2017 - Madison
Governing Privacy Commons at Pitt Science 2017 - MadisonGoverning Privacy Commons at Pitt Science 2017 - Madison
Governing Privacy Commons at Pitt Science 2017 - Madisonprofessormadison
 
AI and Robotics policy overview - Adam Thierer (Aug 2022)
AI and Robotics policy overview - Adam Thierer (Aug 2022)AI and Robotics policy overview - Adam Thierer (Aug 2022)
AI and Robotics policy overview - Adam Thierer (Aug 2022)Adam Thierer
 
“Permissionless Innovation” & the Grand Tech Policy Clash of Visions to Come
“Permissionless Innovation” & the Grand Tech Policy Clash of Visions to Come“Permissionless Innovation” & the Grand Tech Policy Clash of Visions to Come
“Permissionless Innovation” & the Grand Tech Policy Clash of Visions to ComeMercatus Center
 
Privacy & the Internet: An Overview of Key Issues
Privacy & the Internet: An Overview of Key IssuesPrivacy & the Internet: An Overview of Key Issues
Privacy & the Internet: An Overview of Key IssuesAdam Thierer
 
UT Dallas CS - Rise of Crowd Computing
UT Dallas CS - Rise of Crowd ComputingUT Dallas CS - Rise of Crowd Computing
UT Dallas CS - Rise of Crowd ComputingMatthew Lease
 
Age Friendly Economy - Legislation and Ethics of Data Use
Age Friendly Economy - Legislation and Ethics of Data UseAge Friendly Economy - Legislation and Ethics of Data Use
Age Friendly Economy - Legislation and Ethics of Data UseAgeFriendlyEconomy
 
Regulating code
Regulating codeRegulating code
Regulating codeblogzilla
 
Forging Self-Sovereign Identity in the Age of the Blockchain - Christopher Al...
Forging Self-Sovereign Identity in the Age of the Blockchain - Christopher Al...Forging Self-Sovereign Identity in the Age of the Blockchain - Christopher Al...
Forging Self-Sovereign Identity in the Age of the Blockchain - Christopher Al...Christopher Allen
 
Blockchain law and governance: General Conclusion -- Milan, October 2019
Blockchain law and governance: General Conclusion -- Milan, October 2019Blockchain law and governance: General Conclusion -- Milan, October 2019
Blockchain law and governance: General Conclusion -- Milan, October 2019Tony Lai
 
Mathews blockchain presentation
Mathews blockchain presentationMathews blockchain presentation
Mathews blockchain presentationMichael Mathews
 
3 - Social Media and Enterprise
3 - Social Media and Enterprise3 - Social Media and Enterprise
3 - Social Media and EnterpriseRaymond Gao
 

Similar a Toward Effective and Sustainable Online Crowd Work (20)

Crowdsourcing: From Aggregation to Search Engine Evaluation
Crowdsourcing: From Aggregation to Search Engine EvaluationCrowdsourcing: From Aggregation to Search Engine Evaluation
Crowdsourcing: From Aggregation to Search Engine Evaluation
 
Crowdsourcing & ethics: a few thoughts and refences.
Crowdsourcing & ethics: a few thoughts and refences. Crowdsourcing & ethics: a few thoughts and refences.
Crowdsourcing & ethics: a few thoughts and refences.
 
AI & Work, with Transparency & the Crowd
AI & Work, with Transparency & the Crowd AI & Work, with Transparency & the Crowd
AI & Work, with Transparency & the Crowd
 
The Challenge of Benefit-Cost Analysis As Applied to Online Safety & Digital ...
The Challenge of Benefit-Cost Analysis As Applied to Online Safety & Digital ...The Challenge of Benefit-Cost Analysis As Applied to Online Safety & Digital ...
The Challenge of Benefit-Cost Analysis As Applied to Online Safety & Digital ...
 
Explainable Fact Checking with Humans in-the-loop
Explainable Fact Checking with Humans in-the-loopExplainable Fact Checking with Humans in-the-loop
Explainable Fact Checking with Humans in-the-loop
 
A Lifecycle Approach to Information Privacy
A Lifecycle Approach to Information PrivacyA Lifecycle Approach to Information Privacy
A Lifecycle Approach to Information Privacy
 
Thierer Internet Privacy Regulation
Thierer Internet Privacy RegulationThierer Internet Privacy Regulation
Thierer Internet Privacy Regulation
 
Governing Privacy Commons at Pitt Science 2017 - Madison
Governing Privacy Commons at Pitt Science 2017 - MadisonGoverning Privacy Commons at Pitt Science 2017 - Madison
Governing Privacy Commons at Pitt Science 2017 - Madison
 
AI and Robotics policy overview - Adam Thierer (Aug 2022)
AI and Robotics policy overview - Adam Thierer (Aug 2022)AI and Robotics policy overview - Adam Thierer (Aug 2022)
AI and Robotics policy overview - Adam Thierer (Aug 2022)
 
“Permissionless Innovation” & the Grand Tech Policy Clash of Visions to Come
“Permissionless Innovation” & the Grand Tech Policy Clash of Visions to Come“Permissionless Innovation” & the Grand Tech Policy Clash of Visions to Come
“Permissionless Innovation” & the Grand Tech Policy Clash of Visions to Come
 
Ethical and social issues in information systems
Ethical and social issues in information systemsEthical and social issues in information systems
Ethical and social issues in information systems
 
Privacy & the Internet: An Overview of Key Issues
Privacy & the Internet: An Overview of Key IssuesPrivacy & the Internet: An Overview of Key Issues
Privacy & the Internet: An Overview of Key Issues
 
UT Dallas CS - Rise of Crowd Computing
UT Dallas CS - Rise of Crowd ComputingUT Dallas CS - Rise of Crowd Computing
UT Dallas CS - Rise of Crowd Computing
 
Age Friendly Economy - Legislation and Ethics of Data Use
Age Friendly Economy - Legislation and Ethics of Data UseAge Friendly Economy - Legislation and Ethics of Data Use
Age Friendly Economy - Legislation and Ethics of Data Use
 
Regulating code
Regulating codeRegulating code
Regulating code
 
Forging Self-Sovereign Identity in the Age of the Blockchain - Christopher Al...
Forging Self-Sovereign Identity in the Age of the Blockchain - Christopher Al...Forging Self-Sovereign Identity in the Age of the Blockchain - Christopher Al...
Forging Self-Sovereign Identity in the Age of the Blockchain - Christopher Al...
 
Ethics in it
Ethics in itEthics in it
Ethics in it
 
Blockchain law and governance: General Conclusion -- Milan, October 2019
Blockchain law and governance: General Conclusion -- Milan, October 2019Blockchain law and governance: General Conclusion -- Milan, October 2019
Blockchain law and governance: General Conclusion -- Milan, October 2019
 
Mathews blockchain presentation
Mathews blockchain presentationMathews blockchain presentation
Mathews blockchain presentation
 
3 - Social Media and Enterprise
3 - Social Media and Enterprise3 - Social Media and Enterprise
3 - Social Media and Enterprise
 

Más de Matthew Lease

Automated Models for Quantifying Centrality of Survey Responses
Automated Models for Quantifying Centrality of Survey ResponsesAutomated Models for Quantifying Centrality of Survey Responses
Automated Models for Quantifying Centrality of Survey ResponsesMatthew Lease
 
Key Challenges in Moderating Social Media: Accuracy, Cost, Scalability, and S...
Key Challenges in Moderating Social Media: Accuracy, Cost, Scalability, and S...Key Challenges in Moderating Social Media: Accuracy, Cost, Scalability, and S...
Key Challenges in Moderating Social Media: Accuracy, Cost, Scalability, and S...Matthew Lease
 
Adventures in Crowdsourcing : Toward Safer Content Moderation & Better Suppor...
Adventures in Crowdsourcing : Toward Safer Content Moderation & Better Suppor...Adventures in Crowdsourcing : Toward Safer Content Moderation & Better Suppor...
Adventures in Crowdsourcing : Toward Safer Content Moderation & Better Suppor...Matthew Lease
 
Designing Human-AI Partnerships to Combat Misinfomation
Designing Human-AI Partnerships to Combat Misinfomation Designing Human-AI Partnerships to Combat Misinfomation
Designing Human-AI Partnerships to Combat Misinfomation Matthew Lease
 
But Who Protects the Moderators?
But Who Protects the Moderators?But Who Protects the Moderators?
But Who Protects the Moderators?Matthew Lease
 
Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact...
Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact...Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact...
Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact...Matthew Lease
 
Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collectio...
Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collectio...Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collectio...
Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collectio...Matthew Lease
 
Fact Checking & Information Retrieval
Fact Checking & Information RetrievalFact Checking & Information Retrieval
Fact Checking & Information RetrievalMatthew Lease
 
Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to E...
Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to E...Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to E...
Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to E...Matthew Lease
 
What Can Machine Learning & Crowdsourcing Do for You? Exploring New Tools for...
What Can Machine Learning & Crowdsourcing Do for You? Exploring New Tools for...What Can Machine Learning & Crowdsourcing Do for You? Exploring New Tools for...
What Can Machine Learning & Crowdsourcing Do for You? Exploring New Tools for...Matthew Lease
 
Deep Learning for Information Retrieval: Models, Progress, & Opportunities
Deep Learning for Information Retrieval: Models, Progress, & OpportunitiesDeep Learning for Information Retrieval: Models, Progress, & Opportunities
Deep Learning for Information Retrieval: Models, Progress, & OpportunitiesMatthew Lease
 
Systematic Review is e-Discovery in Doctor’s Clothing
Systematic Review is e-Discovery in Doctor’s ClothingSystematic Review is e-Discovery in Doctor’s Clothing
Systematic Review is e-Discovery in Doctor’s ClothingMatthew Lease
 
Multidimensional Relevance Modeling via Psychometrics & Crowdsourcing: ACM SI...
Multidimensional Relevance Modeling via Psychometrics & Crowdsourcing: ACM SI...Multidimensional Relevance Modeling via Psychometrics & Crowdsourcing: ACM SI...
Multidimensional Relevance Modeling via Psychometrics & Crowdsourcing: ACM SI...Matthew Lease
 
Crowdsourcing Transcription Beyond Mechanical Turk
Crowdsourcing Transcription Beyond Mechanical TurkCrowdsourcing Transcription Beyond Mechanical Turk
Crowdsourcing Transcription Beyond Mechanical TurkMatthew Lease
 
Crowdsourcing for Information Retrieval: From Statistics to Ethics
Crowdsourcing for Information Retrieval: From Statistics to EthicsCrowdsourcing for Information Retrieval: From Statistics to Ethics
Crowdsourcing for Information Retrieval: From Statistics to EthicsMatthew Lease
 
Crowdsourcing & Human Computation Labeling Data & Building Hybrid Systems
Crowdsourcing & Human Computation Labeling Data & Building Hybrid SystemsCrowdsourcing & Human Computation Labeling Data & Building Hybrid Systems
Crowdsourcing & Human Computation Labeling Data & Building Hybrid SystemsMatthew Lease
 
Mechanical Turk is Not Anonymous
Mechanical Turk is Not AnonymousMechanical Turk is Not Anonymous
Mechanical Turk is Not AnonymousMatthew Lease
 
Rise of Crowd Computing (December 2012)
Rise of Crowd Computing (December 2012)Rise of Crowd Computing (December 2012)
Rise of Crowd Computing (December 2012)Matthew Lease
 
UT Austin @ TREC 2012 Crowdsourcing Track: Image Relevance Assessment Task (I...
UT Austin @ TREC 2012 Crowdsourcing Track: Image Relevance Assessment Task (I...UT Austin @ TREC 2012 Crowdsourcing Track: Image Relevance Assessment Task (I...
UT Austin @ TREC 2012 Crowdsourcing Track: Image Relevance Assessment Task (I...Matthew Lease
 

Más de Matthew Lease (19)

Automated Models for Quantifying Centrality of Survey Responses
Automated Models for Quantifying Centrality of Survey ResponsesAutomated Models for Quantifying Centrality of Survey Responses
Automated Models for Quantifying Centrality of Survey Responses
 
Key Challenges in Moderating Social Media: Accuracy, Cost, Scalability, and S...
Key Challenges in Moderating Social Media: Accuracy, Cost, Scalability, and S...Key Challenges in Moderating Social Media: Accuracy, Cost, Scalability, and S...
Key Challenges in Moderating Social Media: Accuracy, Cost, Scalability, and S...
 
Adventures in Crowdsourcing : Toward Safer Content Moderation & Better Suppor...
Adventures in Crowdsourcing : Toward Safer Content Moderation & Better Suppor...Adventures in Crowdsourcing : Toward Safer Content Moderation & Better Suppor...
Adventures in Crowdsourcing : Toward Safer Content Moderation & Better Suppor...
 
Designing Human-AI Partnerships to Combat Misinfomation
Designing Human-AI Partnerships to Combat Misinfomation Designing Human-AI Partnerships to Combat Misinfomation
Designing Human-AI Partnerships to Combat Misinfomation
 
But Who Protects the Moderators?
But Who Protects the Moderators?But Who Protects the Moderators?
But Who Protects the Moderators?
 
Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact...
Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact...Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact...
Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact...
 
Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collectio...
Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collectio...Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collectio...
Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collectio...
 
Fact Checking & Information Retrieval
Fact Checking & Information RetrievalFact Checking & Information Retrieval
Fact Checking & Information Retrieval
 
Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to E...
Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to E...Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to E...
Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to E...
 
What Can Machine Learning & Crowdsourcing Do for You? Exploring New Tools for...
What Can Machine Learning & Crowdsourcing Do for You? Exploring New Tools for...What Can Machine Learning & Crowdsourcing Do for You? Exploring New Tools for...
What Can Machine Learning & Crowdsourcing Do for You? Exploring New Tools for...
 
Deep Learning for Information Retrieval: Models, Progress, & Opportunities
Deep Learning for Information Retrieval: Models, Progress, & OpportunitiesDeep Learning for Information Retrieval: Models, Progress, & Opportunities
Deep Learning for Information Retrieval: Models, Progress, & Opportunities
 
Systematic Review is e-Discovery in Doctor’s Clothing
Systematic Review is e-Discovery in Doctor’s ClothingSystematic Review is e-Discovery in Doctor’s Clothing
Systematic Review is e-Discovery in Doctor’s Clothing
 
Multidimensional Relevance Modeling via Psychometrics & Crowdsourcing: ACM SI...
Multidimensional Relevance Modeling via Psychometrics & Crowdsourcing: ACM SI...Multidimensional Relevance Modeling via Psychometrics & Crowdsourcing: ACM SI...
Multidimensional Relevance Modeling via Psychometrics & Crowdsourcing: ACM SI...
 
Crowdsourcing Transcription Beyond Mechanical Turk
Crowdsourcing Transcription Beyond Mechanical TurkCrowdsourcing Transcription Beyond Mechanical Turk
Crowdsourcing Transcription Beyond Mechanical Turk
 
Crowdsourcing for Information Retrieval: From Statistics to Ethics
Crowdsourcing for Information Retrieval: From Statistics to EthicsCrowdsourcing for Information Retrieval: From Statistics to Ethics
Crowdsourcing for Information Retrieval: From Statistics to Ethics
 
Crowdsourcing & Human Computation Labeling Data & Building Hybrid Systems
Crowdsourcing & Human Computation Labeling Data & Building Hybrid SystemsCrowdsourcing & Human Computation Labeling Data & Building Hybrid Systems
Crowdsourcing & Human Computation Labeling Data & Building Hybrid Systems
 
Mechanical Turk is Not Anonymous
Mechanical Turk is Not AnonymousMechanical Turk is Not Anonymous
Mechanical Turk is Not Anonymous
 
Rise of Crowd Computing (December 2012)
Rise of Crowd Computing (December 2012)Rise of Crowd Computing (December 2012)
Rise of Crowd Computing (December 2012)
 
UT Austin @ TREC 2012 Crowdsourcing Track: Image Relevance Assessment Task (I...
UT Austin @ TREC 2012 Crowdsourcing Track: Image Relevance Assessment Task (I...UT Austin @ TREC 2012 Crowdsourcing Track: Image Relevance Assessment Task (I...
UT Austin @ TREC 2012 Crowdsourcing Track: Image Relevance Assessment Task (I...
 

Último

Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyKhushali Kathiriya
 
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...apidays
 
Cyberprint. Dark Pink Apt Group [EN].pdf
Cyberprint. Dark Pink Apt Group [EN].pdfCyberprint. Dark Pink Apt Group [EN].pdf
Cyberprint. Dark Pink Apt Group [EN].pdfOverkill Security
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAndrey Devyatkin
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...apidays
 
Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024The Digital Insurer
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Jeffrey Haguewood
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistandanishmna97
 
Spring Boot vs Quarkus the ultimate battle - DevoxxUK
Spring Boot vs Quarkus the ultimate battle - DevoxxUKSpring Boot vs Quarkus the ultimate battle - DevoxxUK
Spring Boot vs Quarkus the ultimate battle - DevoxxUKJago de Vreede
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobeapidays
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxRustici Software
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...DianaGray10
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...apidays
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businesspanagenda
 
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesrafiqahmad00786416
 
Ransomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdfRansomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdfOverkill Security
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024The Digital Insurer
 

Último (20)

Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
 
Cyberprint. Dark Pink Apt Group [EN].pdf
Cyberprint. Dark Pink Apt Group [EN].pdfCyberprint. Dark Pink Apt Group [EN].pdf
Cyberprint. Dark Pink Apt Group [EN].pdf
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistan
 
Spring Boot vs Quarkus the ultimate battle - DevoxxUK
Spring Boot vs Quarkus the ultimate battle - DevoxxUKSpring Boot vs Quarkus the ultimate battle - DevoxxUK
Spring Boot vs Quarkus the ultimate battle - DevoxxUK
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
 
Ransomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdfRansomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdf
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 

Toward Effective and Sustainable Online Crowd Work

  • 1. Toward Effective & Sustainable Online Crowd Work Matt Lease School of Information @mattlease University of Texas at Austin ml@utexas.edu
  • 2. Matt Lease <ml@utexas.edu> 2 New Capabilities Enable Innovation
  • 3. But also new “Dirty” Digital Jobs • Tasks – Content moderation (Catch-22) – Solving captchas, SEO, account creation, … • Goals – “Legal” fraud, cyber-crime, or cyber-warfare • Workers – Unknown or at-risk: refugees, prisoners, children, … – Typical lack of informed consent – How do we know work is freely chosen? • Part of larger, online ecosystem (cf., Robert Sim) Matt Lease <ml@utexas.edu> 3
  • 5. Debate in ACM 5 Paul Hyman. Communications of the ACM, Vol. 56 No. 8, Pages 19-21, August 2013. Matt Lease <ml@utexas.edu>
  • 6. • Contribute to society and human well-being • Avoid harm to others • Be honest and trustworthy • Be fair and take action not to discriminate • Respect the privacy of others COMPLIANCE WITH THE CODE. As an ACM member I will – Uphold and promote the principles of this Code – Treat violations of this code as inconsistent with membership in the ACM Financial and economic imperatives …Which approaches are less expensive and is this sensible? With the advent of outsourcing and off-shoring these matters become more complex… there are often related ethical issues concerning exploitation… 6 Matt Lease <ml@utexas.edu>
  • 7. What about regulation? • Wolfson & Lease (ASIS&T 2011) • As usual, technology ahead of the law – employment law & independent contractors – patent inventorship – data security and the Federal Trade Commission – copyright ownership – securities regulation of crowdfunding • Understand risks of “just in-time compliance” 7 Matt Lease <ml@utexas.edu>
  • 8. Seek Win-Win Solutions • Pay debate has centered on quality vs. cost tradeoff • What is “fair” and sustainable in a global economy? – Is it better to pay nothing? (i.e., volunteers, gamification) • A meta-approach – Learn: Find information (e.g., Internet, surveys) – Share: Publicize information for others – Act: Find ways to make the “right” thing preferable Matt Lease <ml@utexas.edu> 8
  • 9. The Future of Crowd Work, CSCW’13 Kittur, Nickerson, Bernstein, Gerber, Shaw, Zimmerman, Lease, and Horton 9 Matt Lease <ml@utexas.edu>
  • 10. Join the conversation! Crowdwork-ethics, by Six Silberman http://crowdwork-ethics.wtf.tw an informal, occasional blog for researchers interested in ethical issues in crowd work ir.ischool.utexas.edu
  • 11. Unanticipated challenges & risks with emerging technologies Lease et al., SSRN’13 11