SlideShare una empresa de Scribd logo
1 de 13
Descargar para leer sin conexión
Weaving Relations of Trust in Crowd Work:
Transparency and Reputation across
Platforms
Reviews and Awards
Can we crowdsource paper reviews?
We did
● Ask 3 CrowdFlower contributors to read your
papers and provide a textual review +
recommendation in [-3,3] and gave them $2
● Ask 3 PC members to read your papers and
provide a textual review + recommendation in
[-3,3] and thanked them
● Not include quality checks: some short reviews
● Decide paper acceptance informed solely by PC
member feedback
Disclaimer
● Very few data points
● No statistical significance!
● Crowd members know the domain
● See the NIPS Experiment: Disagreement over
25.9% papers
Results
True
Positives
Best Paper!
Pearson correlation: 0.37 Disagreement over 22% papers
● Positive:
○ “Some of the stats were pretty interesting. I'm actually
going to look up some the papers they got their
information from. I enjoyed it.”
○ “[...] is just perfect.”
● Recommendations:
○ “Maybe some more explicit examples might be needed
on the paper in order to make the point clearer.”
○ “Unfortunately for me the article a lot of obscure
technical terms.”
Example Feedback from the Crowd
Example Feedback from the Crowd
● Critical:
○ “... also without offending anyone tell them that it is
extremely unfair discrimination made the authors as
the best tasks are carried up US, ENGLAND,
AUSTRALIA and the worst and the less you pay goes
to the underdeveloped countries like mine [Argentina]
that must be solved is immoral and unjust”
○ “An academic workshop on crowdsourcing should
have papers talking about crowdsourcing all over the
world and not focused only in one ethnic group, in my
opinion that won't be positive.”
Best Paper Award
● Ranked top by both crowd workers and peer reviewers
● Workers said
○ “[we] must devise novel ways to support trustworthy
actions”
○ “Very correct article and the questions raised.”
○ “The article outlines the basic and necessary questions, it's
good!”
● PC Members said
○ “it would produce some interesting discussions not only
within the TRUSTINCW workshop, but within the greater
community as well.”
Best Paper
The best paper award goes to…
On the Improvement of Quality and
Reliability of Trust Cues in Micro-task
Crowdsourcing
(Position paper)
Jie Yang and Alessandro Bozzon
Workshop Web Site
http://trustincrowdwork.west.uni-koblenz.de/
Workshop Twitter Hashtag
#trustincrowdwork
Workshop Twitter Account
https://twitter.com/trustincw
Conference Twitter Hashtag
#websci16
Agenda
Time Session
9:00 - 9:15 Hello! session
9:15 - 10:30 Invited talk
10:30 - 10:45 Coffee break
10:45- 11:30 Paper presentations (part I - studies on
workers)
11:30-12:00 Crowd Statement Marathon
12:00-13:00 Lunch
13:00-14:00 Paper presentations (part II - platforms)
14:00-15:00 Panel & discussions
15:00 -15:15 Coffee break
15:15 - 16:00 Crowdsourced reviews & awards
16:00 -16:30 Closing session

Más contenido relacionado

Destacado

проект вышгород
проект вышгородпроект вышгород
проект вышгородSergey Scherbak
 
Vol. 1. no. 3. the first regular session of the 14th congress. (2008)
Vol. 1. no. 3. the first regular session of the 14th congress. (2008)Vol. 1. no. 3. the first regular session of the 14th congress. (2008)
Vol. 1. no. 3. the first regular session of the 14th congress. (2008)StratbaseResearchInstitute
 
Vol. 1. no. 2. the escalade of power rates. (2008)
Vol. 1. no. 2. the escalade of power rates. (2008)Vol. 1. no. 2. the escalade of power rates. (2008)
Vol. 1. no. 2. the escalade of power rates. (2008)StratbaseResearchInstitute
 
A Short Guide to Bed Bugs
A Short Guide to Bed BugsA Short Guide to Bed Bugs
A Short Guide to Bed Bugsbrown57
 
Vol 1. no. 4. will the federal solution fit the puzzle (2008)
Vol 1. no. 4. will the federal solution fit the puzzle (2008)Vol 1. no. 4. will the federal solution fit the puzzle (2008)
Vol 1. no. 4. will the federal solution fit the puzzle (2008)StratbaseResearchInstitute
 
Electrical Muscle Stimulation for Speed and Strength Development - Derek Hansen
Electrical Muscle Stimulation for Speed and Strength Development - Derek HansenElectrical Muscle Stimulation for Speed and Strength Development - Derek Hansen
Electrical Muscle Stimulation for Speed and Strength Development - Derek HansenDerek Hansen
 
The Basic Organization and Implementation of Training - Derek Hansen
The Basic Organization and Implementation of Training - Derek HansenThe Basic Organization and Implementation of Training - Derek Hansen
The Basic Organization and Implementation of Training - Derek HansenDerek Hansen
 

Destacado (11)

Paper presentations1
Paper presentations1Paper presentations1
Paper presentations1
 
Role point
Role pointRole point
Role point
 
проект вышгород
проект вышгородпроект вышгород
проект вышгород
 
Vol. 1. no. 3. the first regular session of the 14th congress. (2008)
Vol. 1. no. 3. the first regular session of the 14th congress. (2008)Vol. 1. no. 3. the first regular session of the 14th congress. (2008)
Vol. 1. no. 3. the first regular session of the 14th congress. (2008)
 
Vol. 1. no. 2. the escalade of power rates. (2008)
Vol. 1. no. 2. the escalade of power rates. (2008)Vol. 1. no. 2. the escalade of power rates. (2008)
Vol. 1. no. 2. the escalade of power rates. (2008)
 
CurriculimVitae
CurriculimVitaeCurriculimVitae
CurriculimVitae
 
A Short Guide to Bed Bugs
A Short Guide to Bed BugsA Short Guide to Bed Bugs
A Short Guide to Bed Bugs
 
Vol 1. no. 4. will the federal solution fit the puzzle (2008)
Vol 1. no. 4. will the federal solution fit the puzzle (2008)Vol 1. no. 4. will the federal solution fit the puzzle (2008)
Vol 1. no. 4. will the federal solution fit the puzzle (2008)
 
Mitologija
MitologijaMitologija
Mitologija
 
Electrical Muscle Stimulation for Speed and Strength Development - Derek Hansen
Electrical Muscle Stimulation for Speed and Strength Development - Derek HansenElectrical Muscle Stimulation for Speed and Strength Development - Derek Hansen
Electrical Muscle Stimulation for Speed and Strength Development - Derek Hansen
 
The Basic Organization and Implementation of Training - Derek Hansen
The Basic Organization and Implementation of Training - Derek HansenThe Basic Organization and Implementation of Training - Derek Hansen
The Basic Organization and Implementation of Training - Derek Hansen
 

Similar a Reviews and awards

The Engaged Scholar ANZCA 2018 Auckland
The Engaged Scholar ANZCA 2018 AucklandThe Engaged Scholar ANZCA 2018 Auckland
The Engaged Scholar ANZCA 2018 AucklandKim Osman
 
Writing for Publication: What We Can Learn from Other People's Writing
Writing for Publication: What We Can Learn from Other People's WritingWriting for Publication: What We Can Learn from Other People's Writing
Writing for Publication: What We Can Learn from Other People's WritingSelf Employed
 
LIB300 Week 9 finding, analyzing, and documenting information
LIB300 Week 9 finding, analyzing, and documenting informationLIB300 Week 9 finding, analyzing, and documenting information
LIB300 Week 9 finding, analyzing, and documenting informationDr. Russell Rodrigo
 
Scholarly Requirements for Large Scale Text Analysis
Scholarly Requirements for Large Scale Text AnalysisScholarly Requirements for Large Scale Text Analysis
Scholarly Requirements for Large Scale Text AnalysisHarriett Green
 
LibQUAL+ and ClimateQUAL at York
LibQUAL+ and ClimateQUAL at YorkLibQUAL+ and ClimateQUAL at York
LibQUAL+ and ClimateQUAL at YorkStephen Town
 
Engaging News Project - NYC Media Lab 2015 Annual Summit (workshop)
Engaging News Project - NYC Media Lab 2015 Annual Summit (workshop)Engaging News Project - NYC Media Lab 2015 Annual Summit (workshop)
Engaging News Project - NYC Media Lab 2015 Annual Summit (workshop)Katie Steiner
 
How to write a CHI paper
How to write a CHI paperHow to write a CHI paper
How to write a CHI paperMaria Wolters
 
Better UX = Better Web Conversion
Better UX = Better Web ConversionBetter UX = Better Web Conversion
Better UX = Better Web ConversionClementYo
 
Humans, AI and Decisions Making - 3 - What are the editorial questions AI can...
Humans, AI and Decisions Making - 3 - What are the editorial questions AI can...Humans, AI and Decisions Making - 3 - What are the editorial questions AI can...
Humans, AI and Decisions Making - 3 - What are the editorial questions AI can...phillbjones
 
Tools that Encourage Criticism - Leiden University Symposium on Tools Criticism
Tools that Encourage Criticism - Leiden University Symposium on Tools CriticismTools that Encourage Criticism - Leiden University Symposium on Tools Criticism
Tools that Encourage Criticism - Leiden University Symposium on Tools CriticismMarijn Koolen
 
Toward best practices for research assessment: Effects of indicators and the...
Toward best practices for research assessment: Effects of indicators and the...Toward best practices for research assessment: Effects of indicators and the...
Toward best practices for research assessment: Effects of indicators and the...Sarah de Rijcke
 
Measuring Organization Engagement
Measuring Organization EngagementMeasuring Organization Engagement
Measuring Organization EngagementRonald Widha
 
7 15-16 digging deeper-online-inquiry #digiuri
7 15-16 digging deeper-online-inquiry #digiuri7 15-16 digging deeper-online-inquiry #digiuri
7 15-16 digging deeper-online-inquiry #digiurimschirahagerman
 
NEP teleconference
NEP teleconferenceNEP teleconference
NEP teleconferencegkgwebs
 
Toronto Public Relation is a Discipline of Depth
Toronto Public Relation is a Discipline of DepthToronto Public Relation is a Discipline of Depth
Toronto Public Relation is a Discipline of Depthrobinmethew
 
General guidelines for writing reaction papers (Read this docume
General guidelines for writing reaction papers (Read this documeGeneral guidelines for writing reaction papers (Read this docume
General guidelines for writing reaction papers (Read this documeJeanmarieColbert3
 
Survey Research in Design
Survey Research in DesignSurvey Research in Design
Survey Research in DesignSam Ladner
 

Similar a Reviews and awards (20)

The Engaged Scholar ANZCA 2018 Auckland
The Engaged Scholar ANZCA 2018 AucklandThe Engaged Scholar ANZCA 2018 Auckland
The Engaged Scholar ANZCA 2018 Auckland
 
Writing for Publication: What We Can Learn from Other People's Writing
Writing for Publication: What We Can Learn from Other People's WritingWriting for Publication: What We Can Learn from Other People's Writing
Writing for Publication: What We Can Learn from Other People's Writing
 
LIB300 Week 9 finding, analyzing, and documenting information
LIB300 Week 9 finding, analyzing, and documenting informationLIB300 Week 9 finding, analyzing, and documenting information
LIB300 Week 9 finding, analyzing, and documenting information
 
Scholarly Requirements for Large Scale Text Analysis
Scholarly Requirements for Large Scale Text AnalysisScholarly Requirements for Large Scale Text Analysis
Scholarly Requirements for Large Scale Text Analysis
 
LibQUAL+ and ClimateQUAL at York
LibQUAL+ and ClimateQUAL at YorkLibQUAL+ and ClimateQUAL at York
LibQUAL+ and ClimateQUAL at York
 
Engaging News Project - NYC Media Lab 2015 Annual Summit (workshop)
Engaging News Project - NYC Media Lab 2015 Annual Summit (workshop)Engaging News Project - NYC Media Lab 2015 Annual Summit (workshop)
Engaging News Project - NYC Media Lab 2015 Annual Summit (workshop)
 
How to write a CHI paper
How to write a CHI paperHow to write a CHI paper
How to write a CHI paper
 
Better UX = Better Web Conversion
Better UX = Better Web ConversionBetter UX = Better Web Conversion
Better UX = Better Web Conversion
 
Humans, AI and Decisions Making - 3 - What are the editorial questions AI can...
Humans, AI and Decisions Making - 3 - What are the editorial questions AI can...Humans, AI and Decisions Making - 3 - What are the editorial questions AI can...
Humans, AI and Decisions Making - 3 - What are the editorial questions AI can...
 
Tools that Encourage Criticism - Leiden University Symposium on Tools Criticism
Tools that Encourage Criticism - Leiden University Symposium on Tools CriticismTools that Encourage Criticism - Leiden University Symposium on Tools Criticism
Tools that Encourage Criticism - Leiden University Symposium on Tools Criticism
 
Toward best practices for research assessment: Effects of indicators and the...
Toward best practices for research assessment: Effects of indicators and the...Toward best practices for research assessment: Effects of indicators and the...
Toward best practices for research assessment: Effects of indicators and the...
 
Measuring Organization Engagement
Measuring Organization EngagementMeasuring Organization Engagement
Measuring Organization Engagement
 
7 15-16 digging deeper-online-inquiry #digiuri
7 15-16 digging deeper-online-inquiry #digiuri7 15-16 digging deeper-online-inquiry #digiuri
7 15-16 digging deeper-online-inquiry #digiuri
 
NEP teleconference
NEP teleconferenceNEP teleconference
NEP teleconference
 
Toronto Public Relation is a Discipline of Depth
Toronto Public Relation is a Discipline of DepthToronto Public Relation is a Discipline of Depth
Toronto Public Relation is a Discipline of Depth
 
Royal
RoyalRoyal
Royal
 
General guidelines for writing reaction papers (Read this docume
General guidelines for writing reaction papers (Read this documeGeneral guidelines for writing reaction papers (Read this docume
General guidelines for writing reaction papers (Read this docume
 
Chalk dust to star dust
Chalk dust to star dustChalk dust to star dust
Chalk dust to star dust
 
Publish perish and social media
Publish perish and social mediaPublish perish and social media
Publish perish and social media
 
Survey Research in Design
Survey Research in DesignSurvey Research in Design
Survey Research in Design
 

Más de Cristina Sarasua

Editing Behavior over Time Power vs. Standard Wikidata Editors
Editing Behavior over Time  Power vs. Standard Wikidata EditorsEditing Behavior over Time  Power vs. Standard Wikidata Editors
Editing Behavior over Time Power vs. Standard Wikidata EditorsCristina Sarasua
 
Methods for Intrinsic Evaluation of Links in the Web of Data
Methods for Intrinsic Evaluation of Links in the Web of DataMethods for Intrinsic Evaluation of Links in the Web of Data
Methods for Intrinsic Evaluation of Links in the Web of DataCristina Sarasua
 
How links can make your open data even greater
How links can make your open data even greaterHow links can make your open data even greater
How links can make your open data even greaterCristina Sarasua
 
Introduccion a Wikidata DSS Wikidata Editathon
Introduccion a Wikidata DSS Wikidata EditathonIntroduccion a Wikidata DSS Wikidata Editathon
Introduccion a Wikidata DSS Wikidata EditathonCristina Sarasua
 
Interlinking Is More Than owl:sameAs
Interlinking Is More Than owl:sameAsInterlinking Is More Than owl:sameAs
Interlinking Is More Than owl:sameAsCristina Sarasua
 
Programmatic Access to Crowdsourced Human Computation for Designing and Enhan...
Programmatic Access to Crowdsourced Human Computation for Designing and Enhan...Programmatic Access to Crowdsourced Human Computation for Designing and Enhan...
Programmatic Access to Crowdsourced Human Computation for Designing and Enhan...Cristina Sarasua
 
Crowd Work CV: Recognition for Micro Work
Crowd Work CV: Recognition for Micro WorkCrowd Work CV: Recognition for Micro Work
Crowd Work CV: Recognition for Micro WorkCristina Sarasua
 
Dbpedia leipzig2014 csarasua_open
Dbpedia leipzig2014 csarasua_openDbpedia leipzig2014 csarasua_open
Dbpedia leipzig2014 csarasua_openCristina Sarasua
 
Exploring the challenge of linking scientific publications and studies with c...
Exploring the challenge of linking scientific publications and studies with c...Exploring the challenge of linking scientific publications and studies with c...
Exploring the challenge of linking scientific publications and studies with c...Cristina Sarasua
 

Más de Cristina Sarasua (11)

Editing Behavior over Time Power vs. Standard Wikidata Editors
Editing Behavior over Time  Power vs. Standard Wikidata EditorsEditing Behavior over Time  Power vs. Standard Wikidata Editors
Editing Behavior over Time Power vs. Standard Wikidata Editors
 
Methods for Intrinsic Evaluation of Links in the Web of Data
Methods for Intrinsic Evaluation of Links in the Web of DataMethods for Intrinsic Evaluation of Links in the Web of Data
Methods for Intrinsic Evaluation of Links in the Web of Data
 
How links can make your open data even greater
How links can make your open data even greaterHow links can make your open data even greater
How links can make your open data even greater
 
Tecnología e Igualdad
Tecnología e IgualdadTecnología e Igualdad
Tecnología e Igualdad
 
Introduccion a Wikidata DSS Wikidata Editathon
Introduccion a Wikidata DSS Wikidata EditathonIntroduccion a Wikidata DSS Wikidata Editathon
Introduccion a Wikidata DSS Wikidata Editathon
 
Interlinking Is More Than owl:sameAs
Interlinking Is More Than owl:sameAsInterlinking Is More Than owl:sameAs
Interlinking Is More Than owl:sameAs
 
Programmatic Access to Crowdsourced Human Computation for Designing and Enhan...
Programmatic Access to Crowdsourced Human Computation for Designing and Enhan...Programmatic Access to Crowdsourced Human Computation for Designing and Enhan...
Programmatic Access to Crowdsourced Human Computation for Designing and Enhan...
 
Swib2014csarasua
Swib2014csarasuaSwib2014csarasua
Swib2014csarasua
 
Crowd Work CV: Recognition for Micro Work
Crowd Work CV: Recognition for Micro WorkCrowd Work CV: Recognition for Micro Work
Crowd Work CV: Recognition for Micro Work
 
Dbpedia leipzig2014 csarasua_open
Dbpedia leipzig2014 csarasua_openDbpedia leipzig2014 csarasua_open
Dbpedia leipzig2014 csarasua_open
 
Exploring the challenge of linking scientific publications and studies with c...
Exploring the challenge of linking scientific publications and studies with c...Exploring the challenge of linking scientific publications and studies with c...
Exploring the challenge of linking scientific publications and studies with c...
 

Último

Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024The Digital Insurer
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MIND CTI
 
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 AmsterdamDEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 AmsterdamUiPathCommunity
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherRemote DBA Services
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Victor Rentea
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyKhushali Kathiriya
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Jeffrey Haguewood
 
Ransomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdfRansomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdfOverkill Security
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoffsammart93
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAndrey Devyatkin
 
Exploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with MilvusExploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with MilvusZilliz
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProduct Anonymous
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...apidays
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDropbox
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...DianaGray10
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businesspanagenda
 

Último (20)

Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 AmsterdamDEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
Ransomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdfRansomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdf
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Exploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with MilvusExploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with Milvus
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 

Reviews and awards

  • 1. Weaving Relations of Trust in Crowd Work: Transparency and Reputation across Platforms Reviews and Awards
  • 2. Can we crowdsource paper reviews?
  • 3. We did ● Ask 3 CrowdFlower contributors to read your papers and provide a textual review + recommendation in [-3,3] and gave them $2 ● Ask 3 PC members to read your papers and provide a textual review + recommendation in [-3,3] and thanked them ● Not include quality checks: some short reviews ● Decide paper acceptance informed solely by PC member feedback
  • 4. Disclaimer ● Very few data points ● No statistical significance! ● Crowd members know the domain ● See the NIPS Experiment: Disagreement over 25.9% papers
  • 6. ● Positive: ○ “Some of the stats were pretty interesting. I'm actually going to look up some the papers they got their information from. I enjoyed it.” ○ “[...] is just perfect.” ● Recommendations: ○ “Maybe some more explicit examples might be needed on the paper in order to make the point clearer.” ○ “Unfortunately for me the article a lot of obscure technical terms.” Example Feedback from the Crowd
  • 7. Example Feedback from the Crowd ● Critical: ○ “... also without offending anyone tell them that it is extremely unfair discrimination made the authors as the best tasks are carried up US, ENGLAND, AUSTRALIA and the worst and the less you pay goes to the underdeveloped countries like mine [Argentina] that must be solved is immoral and unjust” ○ “An academic workshop on crowdsourcing should have papers talking about crowdsourcing all over the world and not focused only in one ethnic group, in my opinion that won't be positive.”
  • 9. ● Ranked top by both crowd workers and peer reviewers ● Workers said ○ “[we] must devise novel ways to support trustworthy actions” ○ “Very correct article and the questions raised.” ○ “The article outlines the basic and necessary questions, it's good!” ● PC Members said ○ “it would produce some interesting discussions not only within the TRUSTINCW workshop, but within the greater community as well.” Best Paper
  • 10. The best paper award goes to…
  • 11. On the Improvement of Quality and Reliability of Trust Cues in Micro-task Crowdsourcing (Position paper) Jie Yang and Alessandro Bozzon
  • 12. Workshop Web Site http://trustincrowdwork.west.uni-koblenz.de/ Workshop Twitter Hashtag #trustincrowdwork Workshop Twitter Account https://twitter.com/trustincw Conference Twitter Hashtag #websci16
  • 13. Agenda Time Session 9:00 - 9:15 Hello! session 9:15 - 10:30 Invited talk 10:30 - 10:45 Coffee break 10:45- 11:30 Paper presentations (part I - studies on workers) 11:30-12:00 Crowd Statement Marathon 12:00-13:00 Lunch 13:00-14:00 Paper presentations (part II - platforms) 14:00-15:00 Panel & discussions 15:00 -15:15 Coffee break 15:15 - 16:00 Crowdsourced reviews & awards 16:00 -16:30 Closing session