Slide deck of the workshop at the Computational Social Science Symposium in Cologne, 5/12/2018
More details on the nodegame website: https://nodegame.org/css_tutorial.htm
2. Postdoc Researcher
Microsoft Research - Computational Social Science New York City
Previously
Postdoc Network Science Institute, Northeastern University
Fellow IQSS, Harvard University
PhD Computational Social Science, ETH Zurich
http://stefanobalietti.com
Your Instructor: Stefano Balietti
3. Main Research Interests:
Incentives schemes for peer review systems
Consensus formation and social influence
Inequality, redistribution, and preferences thereof
Optimal experimental design
Philosophy of science, in particular Paul Feyerabend
http://stefanobalietti.com
Your Instructor: Stefano Balietti
4. Some Stats:
• 5 Undergrad / 3 PhD / 3 Postdoc / 4 Professor / 1 Other
• 8 Female / 8 Male
• 5 Computer Science / 4 Media&Comm. / 3 Sociology / 1 Econ / 1
Political Science / 1 Other
Your Fellow Participants
5. 1. Review fundamental concepts of experimental design
2. Understand challenges of opportunities for online
synchronous experiments
3. Acquire enough knowledge of JavaScript programming
4. Learn the how to design, implement, and run a basic
experiment with nodeGame (and contribute to it?)
5. Learn more advanced nodeGame features OR
implement your own game OR Amazon Mechanical Turk
Goals of the Workshop
6. 1. Review fundamental concepts of experimental design
2. Understand challenges of opportunities for online
synchronous experiments
3. Acquire enough knowledge of JavaScript programming
4. Learn the how to design, implement, and run a basic
experiment with nodeGame (and contribute to it?)
5. EXTRA: advanced nodeGame features, implement own
game, Amazon Mechanical Turk, JavaScript, …
Goals of the Workshop
7. 1. Git Latest https://git-scm.com (Windows users install and use Git Bash)
2. Node.JS 10.x https://nodejs.org/en
3. NodeGame v4.x https://nodegame.org
Additionally,
• Make sure to have a Text Editor with syntax highlighting associated with .js files
• Simple editor: Notepad++ (Windows), TextMate, TextWrangler (Mac)
• More advanced: Atom, Sublime, Emacs, Vim
• Might want to setup a free account on Heroku https://www.heroku.com
• Might want to signup to the nodeGame’s forum
For Hands On: Install
8. 1. Git Latest https://git-scm.com (Windows users install and use Git Bash)
2. Node.JS 10.x https://nodejs.org/en
3. NodeGame v4.x https://nodegame.org
Additionally,
• Make sure to have a Text Editor with syntax highlighting associated with .js files
• Simple editor: Notepad++ (Windows), TextMate, TextWrangler (Mac)
• More advanced: Atom, Sublime, Emacs, Vim
• Might want to setup a free account on Heroku https://www.heroku.com
• Might want to signup to the nodeGame’s forum
For Hands On: Install
10. • An experiment is a methodological procedure carried out with the
goal of verifying, falsifying, or establishing the validity of a
hypothesis.
• A test under controlled conditions that is made to demonstrate a
known truth, examine the validity of a hypothesis, or determine
the efficacy of something previously untried.
• An experiment is an empirical method that arbitrates between
competing hypotheses.
What Is an Experiment?
11. • It is an experiment where participants
• (i) interact in a common environment,
• (ii) at the same time,
• (iii) and their actions have an immediate effect on the decision
process, and experimental outcome (e.g., monetary payoff)
• Synchronous experiments can be turn-based or real-time
What Is a Synchronous Experiment?
12. The extent to which a causal
inference based on a study is
warranted by an experiment.
Internal Validity
13. 3 Pillars of Experimental Design
Control RandomizationIndependence
14. 3 Pillars of Experimental Design
Control
Deliberate choice of the variables that will be kept constants
and those that will be varied: treatment variables
15. 3 Pillars of Experimental Design
Independence
Treatment variables must not be correlated
16. 3 Pillars of Experimental Design
• Participants are randomly assigned to treatments
• Treatments are executed in controlled order
• Stratification might be required
Randomization
17. Short History of Experimental Research
“There is a property common to almost all
the moral sciences, and by which they are
distinguished from many of the physical;
that is, that it is seldom in our power to
make experiments with them”
John Stuart Mill, 1836
18. Short History of Experimental Research
Friedman, Cassar (2004) Economists go to the laboratory. Who, what, when and why.
19. Short History of Experimental Research
“Economics unfortunately cannot perform the controlled
experiments of chemists or biologists because [it] cannot
easily control other important factors. Like astronomers or
meteorologists, [it] generally must be content largely to
observe.”
Samuelson and Nordhaus 1985, Economics Textbook (italics added)
“There is no laboratory in which
economists can test their hypotheses”
1993
20. Short History of Experimental Research
Nobel prize for Vernon Smith
“for having established laboratory
experiments as a tool in empirical
economic analysis, especially in the
study of alternative market mechanisms”
2002
21. Induced Value Theory
“The key idea of induced value theory is that proper use of a reward
medium allows an experimenter to induce pre-specified characteristics,
and the subjects innate characteristics become largely irrelevant. ”
(Vernon Smith, 1976)
Monotonicity: Participants always prefer more of the reward medium (m)
Salience: Changes in m are directly related to a participant’s actions
Dominance: changes in a subject’s utility from participating in the
experiment are primarily due to changes in m.
22. Induced Value Theory
“The key idea of induced value theory is that proper use of a reward
medium allows an experimenter to induce pre-specified characteristics,
and the subjects innate characteristics become largely irrelevant. ”
(Vernon Smith, 1976)
Monotonicity: Participants always prefer more of the reward medium (m)
Salience: Changes in m are directly related to a participant’s actions
Dominance: changes in a subject’s utility from participating in the
experiment are primarily due to changes in m.
23. Induced Value Theory
“The key idea of induced value theory is that proper use of a reward
medium allows an experimenter to induce pre-specified characteristics,
and the subjects innate characteristics become largely irrelevant. ”
(Vernon Smith, 1976)
Monotonicity: Participants always prefer more of the reward medium (m)
Salience: Changes in m are directly related to a participant’s actions
Dominance: changes in a subject’s utility from participating in the
experiment are primarily due to changes in m.
24. Induced Value Theory
“The key idea of induced value theory is that proper use of a reward
medium allows an experimenter to induce pre-specified characteristics,
and the subjects innate characteristics become largely irrelevant. ”
(Vernon Smith, 1976)
Monotonicity: Participants always prefer more of the reward medium (m)
Salience: Changes in m are directly related to a participant’s actions
Dominance: changes in a subject’s utility from participating in the
experiment are primarily due to changes in m.
Can you think of examples where failing to induce values can affect experimental results?
26. Some Limitations of Traditional Experimental Research
• Small groups of individuals, generally 4-12 (bounded by size uni. Lab)
• Behavior of individuals in social networks under-researched
• Limited in the number of hypotheses that can be tested
• Observations are limited in time
• Generally high costs for an academic budget
• Limited external validity:
• Sample WEIRD student
• Artificial conditions
30. Survey Experiments
Examples of Online Experiments
Kuziemko et al. (2015) How Elastic Are Preferences for Redistribution
Insert
a new block of questions, or
provide additional info
Measure preferences for outcome
variable at the individual level
Relatively simple to implement
31. Game-Based Asynchronous
Experiments
Examples of Online Experiments
Create a common environment
where a single variable is changed (e.g
network structure)
Observe variation on outcome variable at the
group level
Generally complex design
Can extend for long periods of time Centola (2010) The Spread of Behavior in Online Social Networks
32. Game-Based Synchronous
Experiments
Examples of Online Experiments
Create a common environment
where a single variable is changed
(e.g., group size or payoff)
Observe variation on outcome
variable at the group level
Can be really complex or stylized
Mao et al. (2016) An Experimental Study of Team Size and Performance on a Complex Task
33. Examples of Online Experiments
COMPLEXITY OF EXECUTION
Survey
Experiments
Game-Based
Asynchronous
Experiments
Game-Based
Synchronous
Experiments
43. Challenges in Online Experiments
• The same person uses the same computer to participate repeatedly.
• The same person uses different computers to participate repeatedly.
• Different persons use the same computer to participate.
• Anonymity might increase deviant behavior
• (but get access to specialized populations)
46. Improve Data Quality and Attention
Content responsive faking:
(a) responses influenced by the item content but (b) are not completely
accurate.
purposeful faking (e.g., malingering in medical surveys)
socially desirable response of both intentional and nonintentional varieties
Content nonresponsivity:
responding without regard to item content.
Large variance estimation (2.5 – 50%; 10.6 for students)
Inattentive only on some questions
Meade, A. W. and S. B. Craig (2012). Identifying Careless Responses in Survey Data. Psychological methods 17(3), 437.
47. Improve Data Quality and Attention
• Engagement is key
ui design, interactive elements, colors, length, feedback…
• Nonsensical or “bogus” items
“30th February”
• Instructed response items
“To monitor quality, please respond with a two for this item”
• Unlikely sets
“Select all the good qualities you possess”
• Consistency indexes
Evaluating differences among responses to items highly similar in content
• Too quick response times and other obvious patterns in responses
Meade, A. W. and S. B. Craig (2012). Identifying Careless Responses in Survey Data. Psychological methods 17(3), 437.
48. Commitment Device
• Ask participants if they paid attention to the previous set of
questions
• It prompts participants to pay attention to the subsequent set
of questions
• Its purpose is fulfilled regardless of whether the respondents
answer honestly
49. Commitment Device Example
Before proceeding to the next set of questions, we want to ask for your feedback
about the responses you provided so far. It is vital to our study that we only include
responses from people who devoted their full attention to this study. This will not
affect in any way the payment you will receive for taking this survey. In your honest
opinion, should we use your responses, or should we discard your responses
since you did not devote your full attention to the questions so far?
- Yes, I have devoted full attention to the questions so far and I think you should
use my responses for your study
- No, I have not devoted full attention to the questions so far and I think you should
not use my responses for your study
Alesina, Miano, and Stancheva (2018) “Immigration and Redistribution” NBER 24733
50. AMT Bot Scare
TurkPrime Blog
https://blog.turkprime.com/after-the-bot-scare-understanding-whats-been-happening-with-data-collection-on-mturk-and-how-to-stop-it
https://blog.turkprime.com/moving-beyond-bots-mturk-as-a-source-of-high-quality-data
https://github.com/MAHDLab/rIP
https://itaysisso.shinyapps.io/Bots
Tools to filter Farmers’ IP addresses
Dennis et al. (2018)
“MTurk Workers' Use of Low-Cost 'Virtual Private Servers' to Circumvent Screening Methods: A Research Note”
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3233954
Ahler et al. (2018)
The Micro-Task Market for “Lemons”: Collecting Data on Amazon’s Mechanical Turk
http://gsood.com/research/papers/turk.pdf
52. Fairness for Crowd Workers
• IRB, IRB, IRB (Institutional Review Board)
• Crowd worker have flexibility, but almost no
rights
• Low pay, menial tasks, no accrual of sick days,
or contributions to pension, no unions, etc.
http://www.ghostwork.info
http://www.inthecrowd.org Mary Gray & Sid Suri
56. Dropouts (Attrition)
Arechar et al. (2018) “Conducting interactive experiments online”
“The most severe problem for online interactive studies, and the largest
discrepancy with laboratory experiments, is attrition (participant dropout).”
Major challenge to internal validity, if dropout rates vary with treatment, selection bias
may arise
57. Dropouts (Attrition)
Arechar et al. (2018) “Conducting interactive experiments online”
Just a little bit
more than
50% of the
groups that
started
finished
without
dropouts
(18%) overall dropout rate
As dropouts are spread across experimental groups, the fraction of
the data set affected by these dropouts is considerably larger.
58. What Keeps Participants In?
Intrinsic Motivation
Volunteer Science, Galaxy Zoo, Science@Home (Deci, 2005)
Monetary incentives
about 75% of US Turkers reported that MTurk was their primary or secondary
source of income (Paolacci et al, 2009)
Curiosity
Solving and unrelated, but intriguing puzzle (Law et al., 2016)
Personalized Feedback
Lab-in-the-Wild (Ye et al., 2017)
59. Two Simple Techniques To Limit Dropouts
• Implement warm-up phase (nodeGame levels)
• Implement seriousness checks
Reips (2002) Standards for Internet-Based Experimenting
61. What to Do When Dropouts Happen?
• Terminate experiment immediately
• Continue with less participants (notify? Does behavior change?)
• Replace with bots (notify? Behavior is different)
• Replace with other humans (difficult to implement technically and humanly)
62. What to Do When Dropouts Happen?
• Terminate experiment immediately
• Continue with less participants (notify? Does behavior change?)
• Replace with bots (notify? Behavior is different)
• Replace with other humans (difficult to implement technically and humanly)
Try to design games that are dropout-robust, Make Leaving Harder, Inform
about total time for experiment, Inform about waiting times during the experiment,
Wait for reconnections, Notify
65. nodeGame: the Good Parts
• Powerful API to customize experiments
• Access to low-level details
• Integrated JS database
• Fast and highly scalable
• Game Levels
• Modular design (games, widgets, window)
• Well-documented (and active Forum)
• Integration with Amazon Mechanical Turk
66. Who Develops NodeGame?
Me! (shakty) B. Vartok, L. Miserez, J. Dorrie,
P.Kueng, Ewen Wang, and others.
(thanks!)
67. It comes from Node.JS, in which it is implemented, particularly
suited for real-time interactive games.
Why Is It Called NodeGame?
Why Node.JS ?
Built on Chrome's V8 JavaScript engine, it uses an event-driven, non-
blocking I/O model that makes it lightweight and efficient.
Node.js' package ecosystem, npm, is the largest ecosystem of open
source libraries in the world.
68. JavaScript was developed in May 1995 by
Brendan Eich for Netscape Communications Corp
Was created in 10 days in order to
accommodate the Navigator 2.0 Beta release
Initially called Mocha, later renamed LiveScript in
September, and later JavaScript in the same month
JavaScript
69. Microsoft introduced JScript as reverse-engineered implementation of
Netscape's JavaScript in 1996 in Internet Explorer 3
In 1996 Netscape submitted JavaScript to European Computer Manufacturers
Association (ECMA) to create and industry standard
In 1997 ECMAScript was released
Between 1997 and 2009 5 standard have been released.
July 2015 ECMASCRIPT V6 released.
JavaScript
70. Node.JS was invented in 2009 by Ryan Dahl and
other developers working at Joyent
Combination of Google's V8 JavaScript engine, an
event loop, and a low-level I/O API
npm, the node package manager, in 2011
Versions: 0.10, 0.12, 4.0 … 10.0!
Node.JS
72. Hands On 1
Go to Google.com and open the JavaScript console of your browser:
(ctrl+shift+I or Right Click/Inspect Element)
73. Some methods are immediately available to manipulate the page elements:
Hands On 1
// Fetches the HTML element with given id.
var a = document.getElementById("hplogo");
// Changes one of its attributes.
a.src = "https://upload.wikimedia.org/wikipedia/commons/e/e9/Bing_logo.svg";
// Defines an onclick event-handler.
a.onclick = function() {
// Redirect to a new page.
window.location.href = "http://bing.com";
};
74. var a = 1; // numeric
var b = 'Hello world!’; // string
var c = { key: 'value’ }; // object
var d = [ 'value1’, “value2”, 3 ]; // array
var e = function(p) { return p+1; }; // function
Variables are loosely typed
Variables are scoped within the function in which they are declared
Functions and array are objects
All the variables declared above are global
Variable Declaration in JS
• ES6 introduced let const
75. How to make variables private ?
Declare them inside an own function!
This technique is called: closure
function foo(value) {
var a = value;
}
foo(10);
console.log(a); // undefined
Variable Declaration in JS
78. Self-executing anonymous function
(function(value) {
var a = value;
})(10);
console.log(a); // undefined
Variable Declaration in JS
What is an advantage of closures?
• They do not pollute the
global space
• Needed when you create
functions in a loop
79. Variable Declaration in JS
function foo() {
var a = 1;
}
foo();
console.log(a); // undefined
What happens is we do
not use the var keyword?
80. Variable Declaration in JS
function foo() {
var a = 1;
}
foo();
console.log(a); // undefined
JS will try to access the global variable a
What happens is we do
not use the var keyword?
81. Variable Declaration in JS
function foo() {
var a = 1;
}
foo();
console.log(a); // undefined
JS will try to access the global variable a
What if there is no global variable a?
What happens is we do
not use the var keyword?
82. Variable Declaration in JS
function foo() {
var a = 1;
}
foo();
console.log(a); // undefined 1
What happens is we do
not use the var keyword?
JS will try to access the global variable a
What if there is no global variable a?
Variable leaking into the global scope
83. Avoid Programming Mistakes
There are ways to avoid these types of mistakes in JS
Use a linting tool: JSHint/JSLint/ESLint/JSCS
https://www.sitepoint.com/comparison-javascript-linting-tools
JS Strict Mode
“use strict”;
84. Open the Node.JS terminal and launch nodeGame
Hands On 2: nodeGame
85. Open the Node.JS terminal and launch nodeGame
Hands On 2: nodeGame
cd = change directory (cd .. = dir up)
ls = list
node = type to launch Node.JS programs
Hint! Use TAB to autocomplete the name of files
86. Open the Node.JS terminal and launch nodeGame
Take a look at the different components of nodeGame
Home Page, Game Interface, Requirement Room, Waiting Room, Game,
EndScreen Widget, The Monitor Interface, Get Saved Data
Hands On 2: nodeGame
88. Configuration and game files are separated in different folder
Game’s Anatomy
• Define treatments
• Define the sequence of stages and steps of the experiment
• Implement the rules of interactions among participants
• General setup
• Define the graphical contents of the game
• Define the rules about how to assign treatments to
game rooms, and participants to treatments
89. Hands On 3: Folder waitroom/
// Display buttons to play with bots and select treatment
ALLOW_PLAY_WITH_BOTS: true
ALLOW_SELECT_TREATMENT: true
// How many clients must connect before groups are formed
POOL_SIZE: 2
// The size of each group
GROUP_SIZE: 2
Waiting-Room-v4
Change waiting room options
Start 2 groups simultaneously with random treatment assignment (hint: POOL_SIZE),
Remove Select Treatments button
90. Folder game/
Define the sequence of stages
and steps of the experiment
Define game variables and
group them into treatments
93. Hands On 4
// Skip stages from the sequence.
stager.skip('precache');
stager.skip('selectLanguage');
stager.skip('quiz');
stager.skip('instructions');
stager.skip('mood');
Skip some stages of the game sequence (very useful for debugging)
95. Settings and Treatments Settings-and-Treatments-v4
Multiline Comments
Game variables (these two are game-specific)
Node.JS makes the content of this object available outside of the file
96. Settings and Treatments Settings-and-Treatments-v4
• Variable TIMER is read by nodeGame’s engine
• It defines the max duration (in milliseconds) of each step
• The names of the properties are the names of the steps in
the sequence
97. Settings and Treatments Settings-and-Treatments-v4
• Variable treatments is read by nodeGame’s engine
• It is an object indexed by the names of the treatments
• Each treatment contains all the variables defined outside of
the treatments object, plus what it is defined inside
98. Hands On 5
// Change the number of repetition to 1.
REPEAT: 1,
• Change the number of repetition of the ultimatum stage
99. Hands On 5
// Change the number of repetition to 1.
REPEAT: 1,
module.exports = function(stager, settings) {
stager
.next('precache')
.next('selectLanguage')
.next('instructions')
.next('quiz')
.next('mood')
.repeat('ultimatum', settings.REPEAT)
.next('questionnaire')
.next('endgame')
.gameover();
100. Hands On 6: Create a New Game
cd bin
node nodegame create-game gamename
Open a terminal and start the utility to create new games
Choose a name and follow instructions on screen
101. Hands On 6: Create a New Game
cd bin
node nodegame create-game gamename
Open a terminal and start the utility to create new games
Choose a name and follow instructions on screen
• Personalize the name of the game as you wish
102. Hands On 6: Create a New Game
Open a terminal and start the utility to create new games
Choose a name and follow instructions on screen
103. Folder game/client_types/
• Implement the sequence
• The same sequence can be implemented differently,
depending by who is playing or where the code is
going to be executed
Human, web browser
Computer (orchestrator), server
104. Client Type player.js
Contains code that will be executed on the client (web browser)
Has access to all standard JS objects AND nodeGame client API
However, it is “compiled” on the server, so it also has access to some server
objects
105. Information Flow for Client Type
• Decide a treatment and
• Load a client type with correct settings
Read
3
1 2
Read
1 23
106. How To Implement a Sequence
Use Stager API stager.[method]
To initialize game
stager.setOnInit(func);
To add properties to steps and stages
stager.extendStep("stepId", { property: value });
stager.extendStage(…)
frame What to show (interface)
cb How to handle user clicks
… and many others
107. Implement Game Sequence: change contents of header
Hands On 6
108.
109. Header
• It is created at initialization
• It stays throughout the whole game
110. Header Widget
Widget Widget
• It is created at initialization
• It stays throughout the whole game
• Widgets such as timers, stage counters, and done button are generally added here
112. Frame
• It is created at initialization
• Its content is updated at every step
113. Hands On 7
module.exports = function(treatmentName, settings, stager, setup, gameRoom) {
stager.setOnInit(function() {
// Initialize the client.
var header, frame;
header = W.generateHeader();
frame = W.generateFrame();
// …
• Anonymous initialization function passed as
parameter: it will be executed on the client
• Variable W (stands for Window) contains
methods to manipulate the interface of the game
• Function setOnInit is executed on the server
• It adds a property to the stager
• At the end, the stager is sent to the client
Implement Game Sequence: change contents of header
114. Hands On 7
• Variable node contains the entire nodeGame client API
• Variable node.game contains all game-related methods and
objects, including sequence, database, player list, etc.
• You can add your own variables to node.game
• Variable node.widgets contains methods to create and manipulate widgets
• Widgets are reusable code portions, usually something visible on the interface
node.game.visualTimer = node.widgets.append('VisualTimer', header);
Implement Game Sequence: change contents of header
115. Implement Game Sequence: Change contents of frame
Go inside folder public/
Copy public/instructions.html into public/instructions_mod.html
Make a change to the HTML content
Hands On 8 A
116. Implement Game Sequence: Change contents of frame
Rename to instructions_mod.htm
Extend instruction step to load new page
Hands On 8 B
stager.extendStep('instructions', {
frame: 'instructions_mod.htm'
});
• Method extendStep adds/overwrites a property into a
specific step
• Property frame controls which page is loaded from public/
• Property frame can also be a function or an object
• Property frame could also load a template from views/
117. Executed immediately after the frame has fully loaded
The scope is the main body (not the frame, which must be accessed via W)
1. Bind page element to actions (W.gid, W.setInnerHTML, onclick)
2. Send messages to server or other players (node.say)
3. Handle incoming messages from server or other players (node.on.data)
4. Decide when the step is finished (node.done)
Step Callbacks Step-Callback-Functions-v4
stager.extendStep('instructions', {
cb: function() {
// Do something.
}
});
118. Step Callbacks Step-Callback-Functions-v4
node.on.data("offer", function(msg) {
// Display offer using W.
// msg.data = 1
W.setInnerHTML("decision",
"The dictator offered: " + msg.data);
// End step.
node.done();
});
var submit = W.getElementById("submitOffer");
var v = W.gid("offer").value; // 1
submit.onclick = function() {
// Send offer to server, it will store it
// and forward it to observer.
node.done({ offer: v });
};
I say 99 for me
and 1 for you.
Dictator Observer
119. Step Callbacks Step-Callback-Functions-v4
var submit = W.getElementById("submitOffer");
var v = W.gid("offer").value; // 1
submit.onclick = function() {
// Send offer to server, it will store it
// and forward it to observer.
node.done({ offer: v });
// Send offer to observer.
node.say("offer", "partner_id", v);
};
I say 99 for me
and 1 for you.
Dictator Observer
• This is what server
does, we will see it
later in details
node.on.data("offer", function(msg) {
// Display offer using W.
// msg.data = 1
W.setInnerHTML("decision",
"The dictator offered: " + msg.data);
// End step.
node.done();
});
120. Step Callbacks Step-Callback-Functions-v4
var submit = W.getElementById("submitOffer");
var v = W.gid("offer").value; // 1
submit.onclick = function() {
// Send offer to server, it will store it
// and forward it to observer.
node.done({ offer: v });
};
I say 99 for me
and 1 for you.
Dictator Observer
• W.gid = W.getElementById
• Returns HTML element from frame
node.on.data("offer", function(msg) {
// Display offer using W.
// msg.data = 1
W.setInnerHTML("decision",
"The dictator offered: " + msg.data);
// End step.
node.done();
});
121. Step Callbacks Step-Callback-Functions-v4
var submit = W.getElementById("submitOffer");
var v = W.gid("offer").value; // 1
submit.onclick = function() {
// Send offer to server, it will store it
// and forward it to observer.
node.done({ offer: v });
};
I say 99 for me
and 1 for you.
Dictator Observer
• W.gid = W.getElementById
• Returns HTML element from frame
When both players
are DONE the
game advances to
next step
node.on.data("offer", function(msg) {
// Display offer using W.
// msg.data = 1
W.setInnerHTML("decision",
"The dictator offered: " + msg.data);
// End step.
node.done();
});
122. Modify game step
Observer replies a message to the split to approve or disapprove (need to change HTML also)
Hands On 9
123. Modify game step
Observer replies a message to the split to approve or disapprove (need to change HTML also)
Hands On 9
<div id="observer" style="display: none">
<h4 id="waitingFor">
Waiting for the decision of the dictator<span id="dots"></span>
</h4>
<span id="decision"></span>
<p style="display: none" id="reply_p">
Tell the dictator what you think
<input type="text" id="observer_reply" />
<button id="submit_reply">Submit</button>
</p>
<br/>
</div>
124. Hands On 9
Observer
node.on.data("offer", function(msg) {
// Display offer using W.
W.setInnerHTML("decision", "The dictator offered: " + msg.data);
// Show hidden part.
W.show("reply_p");
// Add listener that sends the text back to dictator.
W.gid("submit_reply").onclick = function() {
// Say message directly to dictator.
node.say("reply", "dictator_id", W.gid("observer_reply");
// End step.
node.done();
});
});
125. Hands On 9
Dictator
submit.onclick = function() {
// We send offer directly to observer.
node.say("offer", "partner_id", v);
// End step,
// store offer in server.
node.on.data("reply", function(msg) {
W.writeln("Your partner thinks: " + msg.data || "nothing");
node.timer.randomDone({ offer: v });
});
};
126. Hands On 9
Dictator
submit.onclick = function() {
// We send offer directly to observer.
node.say("offer", "partner_id", v);
// End step,
// store offer in server.
node.on.data("reply", function(msg) {
W.writeln("Your partner thinks: " + msg.data || "nothing");
node.timer.randomDone({ offer: v });
});
};
Same labels as in Observer: “offer” and “reply”
127. Hands On 9
Dictator
submit.onclick = function() {
// We send offer directly to observer.
node.say("offer", "partner_id", v);
// End step,
// store offer in server.
node.on.data("reply", function(msg) {
W.writeln("Your partner thinks: " + msg.data || "nothing");
node.timer.randomDone({ offer: v });
});
};
Same labels as in Observer: “offer” and “reply”
Randomly ends step
128. Hands On 9
Dictator
submit.onclick = function() {
// We send offer directly to observer.
node.say("offer", "partner_id", v);
// End step,
// store offer in server.
node.on.data("reply", function(msg) {
W.writeln("Your partner thinks: " + msg.data || "nothing");
node.timer.randomDone({ offer: v });
});
};
Same labels as in Observer: “offer” and “reply”
Randomly ends step
Logical OR, shortcut to assign a default
value, if the first term is falsy
130. Folder game/client_types/
• Implement the sequence
• The same sequence can be implemented differently,
depending by who is playing or where the code is
going to be executed
Human, web browser
Computer (orchestrator), server
131. Client Type logic.js
Contains code that will be executed on the server
Has access to the nodeGame client API AND to all server objects
What does it do?
It waits for all players to finish a step, if there is nothing better to do
Make sure the timers are respected
Can push/kick out non-responsive players (Matcher API)
It receives incoming messages from players and stores them (node.game.db)
It matches partners, and assigns roles
132. Client Type logic.js
Contains code that will be executed on the server
Has access to the nodeGame client API AND to all server objects
What does it do?
It waits for all players to finish a step, if there is nothing better to do
Make sure the timers are respected
Can push/kick out non-responsive players (Matcher API)
It receives incoming messages from players and stores them (node.game.memory)
It matches partners, and assigns roles
133. Client Type logic.js
stager.extendStep('bidder', {
matcher: {
// Available roles.
roles: [ 'BIDDER', 'RESPONDENT' ],
// Each player is matched with a
// partner and is assigned a role.
match: 'roundrobin',
// How to continue matching after
// all combinations are exhausted
cycle: 'repeat_invert'
}
});
Logic Client Type
Matching-Roles-Partners-v4
134. Client Type logic.js
stager.extendStep('bidder', {
roles: {
BIDDER: {
frame: 'bidder.htm',
cb: function() {
// Bidder stuff.
}
},
RESPONDENT: {
frame: 'resp.htm',
cb: function() {
// Respondent stuff.
}}}
});
Player Client Type
stager.extendStep('bidder', {
matcher: {
// Available roles.
roles: [ 'BIDDER', 'RESPONDENT' ],
// Each player is matched with a
// partner and is assigned a role.
match: 'roundrobin',
// How to continue matching after
// all combinations are exhausted
cycle: 'repeat_invert'
}
});
Logic Client Type
Matching-Roles-Partners-v4
135. Client Type logic.js
stager.extendStep('bidder', {
roles: {
BIDDER: {
frame: 'bidder.htm',
cb: function() {
// Bidder stuff.
}
},
RESPONDENT: {
frame: 'resp.htm',
cb: function() {
// Respondent stuff.
}}}
});
Player Client Type
stager.extendStep('bidder', {
matcher: {
// Available roles.
roles: [ 'BIDDER', 'RESPONDENT' ],
// Each player is matched with a
// partner and is assigned a role.
match: 'roundrobin',
// How to continue matching after
// all combinations are exhausted
cycle: 'repeat_invert'
}
});
Logic Client Type
Logic lists roles
Player implements
Matching-Roles-Partners-v4
139. Memory Database Memory-Database-v4
var m = node.game.memory;
// Lazy evaluation: add a bunch of statements
// and evaluates all of them at the end.
m.select("player", "=", "id12345678");
m.and("stage.step", "=", 1)
.and("offer") // Must have property “offer”
.fetch()[0]; // Returns array, get first element
// Use indexes.
// By sender.
m.player["id12345678"].last();
// By stage.
m.stage["2.1.1"].select("offer").fetch()[0];
140. Explore this RunKit on selecting and fetching items from node.game.memory
https://runkit.com/shakty/gamedb
Hands On 10
141. What’s Left Out
Levels
Widget Steps
Authorization
Get Help / Contribute
Dropout & Reconnections
Upload Your Game to the Cloud
Amazon Mechanical Turk
142. What More?
Technical Topics
Levels
Widget Steps
Authorization
Requirements
Get Help / Contribute
Dropout & Reconnections
Go Live
Upload Your Game to the Cloud
Amazon Mechanical Turk
145. Extend
Core components for the client
Sent to client as part nodegame-full.js
After editing a core component, rebuild
nodegame-server/
public/javascripts/nodegame-full.js
node launcher -b widgets,window
(nodegame-client automatically rebuilt with –b)
148. AMT
1. Upload/install nodeGame + game to the cloud
e.g., Digital Ocean or Heroku cloud: https://github.com/nodeGame/nodegame-
heroku
2. Create a new HIT on Amazon Mechanical Turk (AMT), e.g. using the requester
interface or other service API.
3. Use the nodegame-mturk package
https://github.com/nodeGame/nodegame-mturk/
to handle:
- Connection and authentication from AMT HIT page to nodeGame server.
- Opening/expiring/extending HIT while running it.
- Payments, bonuses and qualifications after HIT is done.
149. AMT HIT Page
Check the boilerplate in the public folder of the nodegame-mturk
package
150. AMT HIT Page
• The important things:
• Inform about the task (in general terms)
• Make it really clear it is a SYNCHRONOUS task
• Explain expected duration and payoff structure
• Explain technical requirements, if any
• Optionally, have a Trouble Ticket
• Exit code input
151. AMT HIT Page
• Display link to game server only after acceptance
• Get turker information, upon acceptance, from the page and send
it to the server along the query string or via a separate AJAX request
HITID = turkGetParam('hitId’);
WORKER_ID = turkGetParam('workerId’);
ASSIGNMENT_ID = turkGetParam('assignmentId');
• These functions are not
officially supported by
AMT, so they might
disappear in the future
• These info is stored in
the page as hidden input
152. AMT Issues for Synchronous Games
1. Not a survey or classification task, need to have DIFFERENT
people at the same time on your platform
2. Btw, not too few, not too many
3. Waiting time in the waiting room should be limited and
homogenous across participants
4. Demographics should be stratified as much as possible
5. Participants should not re-take the same experiment again
6. Dropouts rate (non-completing the HIT) significantly affects the
whole experiment, increasing costs $$ and time to complete it
153. AMT Issues for Synchronous Games
1. Not a survey or classification task, need to have DIFFERENT
people at the same time on your platform
Pay minimum wage (at least). Make bonus dependent on performance
154. AMT Issues for Synchronous Games
1. Not a survey or classification task, need to have DIFFERENT
people at the same time on your platform
2. Btw, not too few, not too many
Rule of thumb: 2.8 times desired number of participants
Expire HIT as soon as desired number of participant entered the
experimental pipeline (+small buffer)
3x
155. AMT Issues for Synchronous Games
1. Not a survey or classification task, need to have DIFFERENT
people at the same time on your platform
2. Btw, not too few, not too many
Rule of thumb: 2.8 times desired number of participants
Expire HIT as soon as desired number of participant entered the
experimental pipeline (+small buffer)
Beware of “HIT Grabbers”: use scripts to accept HITs very quickly, and
then take it when they have time;
Often, they only see the title, so make sure to write in title something
like: “MUST TAKE IMMEDIATELY”
156. AMT Issues for Synchronous Games
1. Not a survey or classification task, need to have DIFFERENT
people at the same time on your platform
2. Btw, not too few, not too many
3. Waiting time in the waiting room should be limited and
homogenous across participants
Dispose participants who waited too long (compensated)
157. AMT Issues for Synchronous Games
1. Not a survey or classification task, need to have DIFFERENT
people at the same time on your platform
2. Btw, not too few, not too many
3. Waiting time in the waiting room should be limited and
homogenous across participants
4. Demographics should be stratified as much as possible
Waiting room can group players based on features
Collect some demographics features in a pre-experimental part (level 1)
Start 2 (or more) groups at the same time
158. AMT Issues for Synchronous Games
1. Not a survey or classification task, need to have DIFFERENT
people at the same time on your platform
2. Btw, not too few, not too many
3. Waiting time in the waiting room should be limited and
homogenous across participants
4. Demographics should be stratified as much as possible
5. Participants should not re-take the same experiment again
Create one HIT (no group) with 3x rule
Keep track of turker ids (nodegame-mturk)
Assign qualification via AMT API (nodegame-mturk)
159. AMT Issues for Synchronous Games
1. Not a survey or classification task, need to have DIFFERENT
people at the same time on your platform
2. Btw, not too few, not too many
3. Waiting time in the waiting room should be limited and
homogenous across participants
4. Demographics should be stratified as much as possible
5. Participants should not re-take the same experiment again
6. Dropouts rate (non-completing the HIT) signigicantly affects the
whole experiment, increasing costs $$ and time to complete it
• Hey, we talked about this already! (see previous slides)
160. AMT Dropouts and Relative Cumulative Payoff
• Tested combinations of
demographics and other
game characteristics
• The only significant difference
is between dropouts and
participants who stay is their
relative payoff.
161. Keep a Good Requester Score
https://turkerview.com
https://turkopticon.ucsd.edu (will be discontinued)
167. External Validity
Galileo's hand-made drawing of the moon surface. Sidereus Nuncius (1610)
Telescope:
From Land to Sky
Experiments:
From Lab to Real Life
169. To access variables between files must be explicitly exported and required
// File A.
var a = 1;
module.exports = a;
// File B.
var a = require('./A.js')
How to export
multiple variables?
// File A.
module.exports = { a: 1, b: 2 };
// File B.
var a = require('./A.js').a;
Node.JS Export Variables
170. JS functions are objects
Can be passed as parameters to other functions
Treat differently different input parameters
Can have properties
Can be executed with different contexts
Two types exists: declaration and expressions
Always remember the scope of creation
JS Functions
171. JS Functions
Create an array of 10 functions returning the index in which they are
inserted in the array.
Is the code below correct?
var i, len, obj;
len = 10, obj = [];
for (i = 0 ; i < len ; i++) {
obj[i] = function() { return i; }
}
172. JS Functions
Create an array of 10 functions returning the index in which they are
inserted in the array.
Is the code below correct?
var i, len, obj;
len = 10, obj = [];
for (i = 0 ; i < len ; i++) {
obj[i] = function() { return i; }
}
• Creates 10 functions all
returning the value 10,
because they are all
referencing variable i, which
has value 10 when the loop
ends
173. JS Functions
Create an array of 10 functions returning the index in which they are
inserted in the array.
You need a closure! for (i = 0 ; i < len ; i++) {
obj[i] = (function(i) {
return function() {
return i;
}
})(i);
}
174. JS Function Input Parameters
// Internally modifies input.
function a(obj, num, str) {
obj.a = 10;
num = 1;
str = 'a';
}
var obj = {}, num = 0, str = '';
a(obj, num, str);
console.log(obj);
console.log(num);
console.log(str); ?
175. JS Function Input Parameters
// Internally modifies input.
function a(obj, num, str) {
obj.a = 10;
num = 1;
str = 'a';
}
var obj = {}, num = 0, str = '';
a(obj, num, str);
console.log(obj); // { a: 10 }
console.log(num); // 0
console.log(str); // ''
176. JS Function Input Parameters
console.log(obj); // { a: 10 }
console.log(num); // 0
console.log(str); // ''
Objects are passed as reference, while numbers and strings as
copy
Modifying a copy does not affect the value outside the function,
modifying the reference does.
177. JS this
function a() { return this.a; }
a(); // undefined
// Create a context.
var foo = { a: 1};
// call and apply set the this value
a.call(foo, 1, 2, 3); // 1;
a.apply(foo, [1, 2, 3]); // 1;
The value of this is dynamic in JavaScript
It is determined when function is called, not when it is declared.
178. JS this
What will be printed to console?
function A() {
this.a = 1;
(function() {
console.log(this.a);
})();
}
// Create a new object.
var a = new A();
179. JS this
What will be printed to console?
function A() {
this.a = 1;
(function() {
console.log(this.a);
})();
}
// Create a new object.
var a = new A();
• It will print undefined
• How to adapt to print 1?
180. JS this
What will be printed to console?
function A() {
this.a = 1;
var that = this;
(function() {
console.log(that.a);
})();
}
• It will print 1
• The reference to this is
stored in another variable
181. JS this
What will be printed to console?
function A() {
this.a = 1;
var that = this;
(function() {
console.log(that.a);
})();
}
• It will print 1
• The reference to this is
stored in another variable
Why is “this” solution better than call or apply?
182. JS this
What will be printed to console?
function A() {
this.a = 1;
var that = this;
(function() {
console.log(that.a);
})();
}
• It will print 1
• The reference to this is
stored in another variable
because you can reuse that multiple times!
Why is “this” solution better than call or apply?
183. In JS, each object inherits methods and properties from a parent object
called prototype
In turn, also the prototype object can have an own prototype, and all the
properties are are inherited through the prototype chain
It is possible to extend an object by extending its prototype or the
prototype of its prototype...
This pattern is called prototypical inheritance, and it is extremely
powerful – if well understood
JS Inheritance
184. function A() {
this.a = 1;
}
A.prototype.printA = function() {
console.log(this.a);
}
var a = new A();
a.printA(); // 1;
JS Prototypical Inheritance
185. function A() {
this.a = 1;
}
A.prototype.printA = function() {
console.log(this.a);
}
var a = new A();
a.printA(); // 1;
JS Prototypical Inheritance
What is the difference with defining
define the method printA inside the
constructor? (this.printA = function … )
186. // Create a second object.
var a2 = new A();
// Assign property to method printA.
a2.printA.foo = 1;
// Property is also on object a
// because it is the prototype
// to be modified.
console.log(a.print.foo); // 1
JS Prototypical Inheritance
187. JS Prototypical Inheritance
Extending the prototype of the function leads to faster
object creations because all the methods are already
existing and only need to be referenced instead of being
created
However sometimes you need to have a clear separation
between methods of objects of the same class
188. Looping in Objects (For In)
Javascript does not guarantee clear separation between
variables of the prototype and of the object itself
Therefore, when looping through the properties of an
object it is necessary to invoke the method
.hasOwnProperty
189. Looping in Objects (For In)
var triangle = { a: 1, b: 2, c: 3 };
function ColoredTriangle() {
this.color = "red";
}
ColoredTriangle.prototype = triangle;
var obj = new ColoredTriangle();
for (var prop in obj) {
if (obj.hasOwnProperty(prop)) {
console.log(obj[prop]);
}
}
191. Really easy to install and use
Install with the node package manager
What is the option -g doing?
Linting Tool: JSHint
You might be interested in one of the plugins for editors (vim, emacs, sublime...)
jshint your_file.js --show-non-errors
npm install -g jshint
192. "use strict";
(function(value) {
a = value;
})(10)
a = value;
^
ReferenceError: a is not defined
at /home/stefano/Documents/myworkshops/nodegame_CESS/scripts/A.js:4:7
at Object.<anonymous> (/home/stefano/Documents/myworkshops/nodegame_CESS/scripts/A.js:5:3)
at Module._compile (module.js:460:26)
at Object.Module._extensions..js (module.js:478:10)
at Module.load (module.js:355:32)
at Function.Module._load (module.js:310:12)
at Function.Module.runMain (module.js:501:10)
at startup (node.js:129:16)
at node.js:814:3
Strict Mode on