How metacognition was trained in ELEKTRA project of game based learning of physics

1. Metacognition in ELEKTRA∗
Jean-Loup Castaigne
LabSET — IFRES — University of Liege
January 24, 2007
1 Why certitude degrees in learning?
In an evaluation based only on identifying correct and incorrect answers there
is little information available for both teacher and learner other than right or
wrong. Adding certitude degree to evaluation brings a new dimension, allowing
more precise conclusions.
For instance what conclusion should raise a teacher when 95 % of his students
succeed answering a question? What other conclusion if those 95 % successful
students only have a mean confidence of 10% in their correct answer? Maybe
the teacher will consider this task as not achieved by his students despite 95%
of correct answer.
The way a test will be corrected will also be different if a majority of students
are failing a question with a mean certitude degree of 10% for that question or
with a mean certitude degree of 90%. In the first case doubt can be expressed
regarding the question itself. In the second case it appears students are confident
in their believe which is an incorrect information. This last situation may be
considered as dangerous as students will trust what they think they know.
Learning does not move someone from total ignorance to perfect knowledge.
Often people will already have some knowledge or representation about what is
teached, even if these representation or knowledge might be erroneous. Evalua-
tion should not be limited to either knowledge or correct answer and ignorance
or incorrect answer. Information is what reduces doubt. “Partial information
exists. To detect it is necessary and feasible” (De Finetti, 1965)1 . certitude
degrees give a teacher a more detailed image of the knowledge of a student:
they reveal partial information.
∗ ELEKTRA : “Enhanced Learning Experience and Knowledge TRAnsfer” is an European
research project that aims at revolutionising technology-enhanced learning.
1 De Finetti, B. (1965). Methods of discriminating levels of partial knowledge concerning
a test item. Bristish Journal of Math. & Statist. Psychol., 18, 87-123.
1

2. 2 What is metacognition?
2.1 Metacognition in general
Metacognition has been defined by Flavell (1976)2 : “Metacognition refers to
one’s knowledge concerning one’s own cognitive processes or anything related
to them, e.g. , the learning-relevant properties of information or data” . Many
interpretations were made from the original definition of metacognition but
metacognition is agreed to involve both
• knowledge about one’s own knowledge and
• knowledge about one’s own cognitive processes.
Metacognition has to do with the active monitoring and regulation of cognitive
processes. Metacognition is also relevant to work on cognitive styles and learning
strategies in so far as the individual has some awareness of their thinking or
learning processes.
Within metacognition “knowledge about memory” is defined as metamem-
ory (Flavell & Wellman, 1977)3 . Metamemory itself is declined into declar-
ative metamemory (e.g. knowledge about strategy variables) and procedural
metamemory with its two components: the monitoring component and the con-
trol and self-regulated component.
Metacognition
• Metamemory
1. Declarative metamemory
2. Procedural metamemory
(a) monitoring component
(b) control and self-regulated component
Schraw and Dennison (1994)4 separate the knowledge and the regulation of
cognition, each one being defined into different parts.
1. The knowledge of cognition is composed of:
(a) Declarative knowledge: knowledge about one’s skills, intellectual re-
sources, and abilities as a learner.
(b) Procedural knowledge: knowledge about how to implement learning
procedures (e.g., strategies).
(c) Conditional knowledge: knowledge about when and why to use learn-
ing procedures.
2. The regulation of cognition refers to:
2 Flavell, J.H. (1976). Metacognitive aspects of problem solving. In Resnick (Ed.), The
nature of intelligence. (pp. 231-235). New Jersey : Lawrence Erlbaum Associates.
3 Flavell, J.H., & Wellman, H. (1977). Metamemory. In R. V. Kail Jr. & J. Hagen (Eds.),
Perspectives on the development of memory and cognition (pp. 3–33). Hillsdale, NJ: Erlbaum
4 Schraw, Gregory, and Rayne Sperling Dennison, 1994. Assessing Metacognitive Aware-
ness? In Contemporary Educational Psychology 19, 460-475.
2

3. (a) Planning: planning, goal setting, and allocating resources prior to
learning.
(b) Information management: skills and strategy sequences used on-line
to process information more efficiently (e.g., organizing, elaborating,
summarizing, selective focusing).
(c) Monitoring: assessment of one’s learning or strategy use.
(d) Debugging: strategies used to correct comprehension and perfor-
mance errors.
(e) Evaluation: analysis of performance and strategy effectiveness after
a learning episode.
The EARLI 5 metacognition special interest group often refers to the main
components of metacognitive monitoring and metacognitive control that was
defined from the works of Nelson (1996) 6 and Nelson & Narens (1994) 7 – see
figure 1.
Figure 1: Main components of metacognitive monitoring and metacognitive control defined
by Nelson (1996) and Nelson & Narens (1994)
Leclercq & Poumay (2004) 8 propose an “operational definition” of metacog-
nition: “Observable (or not) judgments, analysis and /or regulations effectuated
5 EARLI: European Association for Research on Learning and Instruction
6 Nelson, T. O. (1996). Gamma is a measure of the accuracy of predicting performance
on one item relative to another item, not of the absolute performance on an individual item.
Applied Cognitive Psychology, 10, 257–260.
7 Nelson, T. O., & Narens, L. (1994). Why investigate metacognition? In J. Metcalfe &
A. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 1–25). Cambridge, MA:
Bradford Books.
8 personal translation of “les jugements, analyses et/ou r´gulations observables effectu´s par
e e
l’apprenant sur ses propres performances (processus ou produits d’apprentissages), ceci dans
des situations de PRE, PER ou POST performance” from Leclercq, D. & Poumay, M. (2004).
Une d´finition op´rationnelle de la m´tacognition et ses mises en oeuvre. 21e Conf´rence
e e e e
internationale de l’AIPU, Morocco: Marrakech.
3

4. by a learner on his/her own performances (learning processes or products), in
situations of pre, per or post performance (mainly testing or learning).”
2.2 Certitude degrees and metacognition
Ebel (1965)9 defined confidence weighting as “a special mode of responding to
objective test items, and special mode of scoring those responses. In general
terms, the examinee is asked to indicate not only what he believes to be the
correct answer to a question, but also how certain he is of the correctness of his
answer”. certitude degrees are confidence weighting. In Nelson & Narens’s main
components of metacognitive monitoring and metacognitive control certitude
degrees with testing take place in the monitoring side, retrieval of knowledge
category, in the Confidence in retrieved answers box. In Leclercq & Poumay’s
operational definition certitude degree with testing will be observable judgments
effectuated by a learner on his/her own performances (products), in situations
of per performance (testing).
2.3 Metacognition in the demonstrator
One can ask if metacognition is usable as such for teenagers 14 years old. In
brief, publications that were founded consider tested components of metacogni-
tive monitoring and control not significantly different between adults and as low
as 10 years old children. Below the age of 10, components significance evolves
with age. Flavell, Friedrichs & Hoyt (1970)10 have showed significant correla-
tion between predicted and actual memory span in children from the 4th grade
and no significant correlation was founded below that age, including at nursery
and kindergarden. Young children show unrealistic performances prediction.
Schneider (2006)11 offers three theoretical reasons: 1) insufficient metacognitive
knowledge (young children not monitoring their memory activities or lacking in
understanding about the interplay among relevant factors), 2) predominance of
wishful thinking over expectations thus predictions reflecting their desires) and
3) belief in the power of effort. Duell (1986)12 confirm that as children get older
they demonstrate more awareness of their thinking processes.
In the Elektra demonstrator the use of certitude degrees so far is neither
relevant of a “judgment of learning” nor a confidence in retrieved answer. Judg-
ments of learning (jols), are defined as judgments that “occur during or after
acquisition and are predictions about future test performance on recently stud-
ied items” (Nelson & Narens, 1994).
9 Ebel, R. (1965). Confidence weighting and test reliability. Journal of Educational Mea-
surement, pages 49–57.
10 Flavell, J. H., Friedrichs, A., & Hoyt, J. (1970). Developmental changes in memorization
processes. Cognitive Psychology, 1, 324–340.
11 Schneider 2006, The development of metacognition in childhood and adolescence, keyn-
note of the 2nd international biennial conference of the Metacognition Special Interest Group
(SIG 16) of the European Association of Research in Learning and Instruction (EARLI), July
2006, Cambridge
12 Duell, O.K. (1986). Metacognitive skills. In G. Phye & T. Andre (Eds.), Cognitive
Classroom Learning. Orlando, FL: Academic Press.
4

5. 2.3.1 What is a JOL?
Imagine a learner studying pairs of linked words13 such as game–elektra.
The learner receives the instructions that each pair should be learned so that
subsequently when prompted with the cue (e.g., “game–?”), the learner would
recall the target (e.g., “elektra”). An interval of time elapses between the
termination of studying a given item and the onset of the jol for that item.
This interval could be
• extremely brief (e.g., as in an immediate jol that occurs as close in time
as possible to the offset of the studied item) or
• could be delayed for a more lengthy amount of time (as in a delayed jol)
filled with other activity and/or other to-be-learned items.
Then the first jol occurs, prompted by a cue that usually consisted of only
the cue from the studied item (e.g., “game–?”). The learner generates a jol
by choosing the predicted likelihood (e.g., on a Likert-type rating scale or on a
scale ranging from 0% to 100% in steps of 20%) of remembering the item on an
eventual criterion test that might occur later (e.g., 10–20 minutes later). Other
pairs of words are studied, and jols are made for each of them, until every
pair had been studied and had received a jol. Finally, following an interval
of perhaps 10–20 minutes (filled with other items) from the time of studying a
given item, the person received the eventual memory test on that item. The
memory test is self-paced, usually asking the person to recall the target when
prompted by the cue (e.g., when prompted by “game–?”, the person attempt
to recall “elektra”).
With this example we demonstrate that what is asked in the demonstrator
is neither true jols nor confidence in retrieved answers.
2.3.2 JOL or confidence in retrieved answer
We suggest that metacognition in elektra is in a first time limited to the use
of certitude degrees with the objective of improving the learner metacognitive
self assessment. Kruger & Dunning (1999) 14 showed that “learners whose skills
or knowledge bases are weak in a particular area tend to overestimate their
ability in that area”. In other words, they don’t know enough to recognize
that they lack sufficient knowledge for accurate self-assessment. In contrast,
learners whose knowledge or skills are strong may underestimate their ability.
These high-ability learners don’t recognize the extent of their knowledge or skills.
Kruger and Dunning’s research also shows that “it is possible to teach learners at
all ability levels to assess their own performance more accurately”. It is in that
perspective that certitude degrees will be implemented in elektra. With the
use of certitude degree in elektra we can at least give the learner/player some
feedback about their cognitive strategies with the confidence in their retrieved
answers.
13 adapted form Nelson, Narens & Dunlosky (2004). A Revised Methodology for Research on
Metamemory: Pre-judgment Recall And Monitoring (PRAM). Psychological Methods, Vol.
9, No. 1, 53–69.
14 Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in
recognizing one’s own incompetence lead to inflated self-assessment. Journal of Personality
and Social Psychology, 77, 1121-1134.
5

6. Consolidated Research Report M1-6 specifies that metacognition within elek-
tra should:
• support and facilitate the development of metacognitive abilities and strate-
gies and
• support the assessment of knowledge.
We recommend to be very cautious in identifying what is assessment of knowl-
edge and what is support to metacognition. Feedbacks should always be very
clear in regard to what they refer to, either cognition–knowledge or metacognition–
knowledge about knowledge. With elektra we can provide teachers a tool to
begin a reflection with their students about metacognition. The distinction
between cognitive and metacognitive strategies is important, partly because it
gives some indication of which strategies are the most crucial in determining
the effectiveness of learning. It seems that “metacognitive strategies, that allow
students to plan, control, and evaluate their learning, have the most central role
to play in this respect, rather than those that merely maximize interaction and
input . . . Thus the ability to choose and evaluate one’s strategies is of central
importance” (Graham, 1997) 15 . Learners who are metacognitively aware know
what to do when they don’t know what to do; that is, they have strategies for
finding out or figuring out what they need to do. “The use of metacognitive
strategies ignites one’s thinking and can lead to more profound learning and im-
proved performance, especially among learners who are struggling” (Anderson,
2002) 16 .
3 Metacognition in games
Prakash (1999) 17 reports that he heard on the radio a sixth grader explaining
what she was learning from playing the Stock Market Game. This game was an
activity designed to help children become familiar with how the stock market
functions. She said, “This game makes me think how to think”. What this
statement reveals is that this young learner was beginning to understand the
real key to learning; she was engaged in metacognition using a game.
Salda˜a (2004) 18 reports using a Master Mind c modified (order was not
n
important) to assess metacognitive processes use to solve the enigma. The
solution could be found independently or with three levels of assistance
1. focusing on the metacognitive processes present in the task such as
• planning aims / strategies,
• supervision and control of aims / strategies,
• revision of aims / strategies,
15 Graham, S. (1997). Effective language learning. Clevedon, England: Multilingual Mat-
ters.
16 Neil J. Anderson, The Role of Metacognition in Second Language Teaching and Learning,
Brigham Young University
17 Prakash, S. (Reporter). (1999, March 19). Market games [Radio series episode]. All
things considered. Washington: National Public Radio.
18 Salda˜ a, D. (2004) Dynamic Master Mind: interactive use of a game for testing metacog-
n
nition. School Psychology International, Vol 25(4): 422-438.
6

7. • meta knowledge of possibilities and limitations
• meta knowledge of tasks / strategies;
2. scaffolding each and every one of the selfregulatory steps included in the
task;
3. modelling of the task solution process
On top of working metacognitive process this experiment also showed diversifi-
cation in methods for adaptivity.
Additional literature research is still under process. Please keep this docu-
ment for private use within elektra.
4 About the demonstrator, 2006 version
These comments only cover some metacognitive aspects of the demonstrator.
4.1 A surprise
When Galileo says that the most important thing is to first define what you
want to do, he is asking George to identify the objective. This is a metacognitive
activity and looks like what is described by Salda˜a (2004) as “planning aims”
n
in his Master Mind c experiment. It is a pleasant surprise to see that the
formulation of that question turned it into a metacognitive activity.
Figure 2: Cut scene from the elektra demonstrator, 2006
7

8. 4.2 General comments
1. The asked question should not have been ‘Are you sure?” but something
like “With what you know and what you have learned, how confident are
you that you will succeed opening the door in this try?” Asked the way it
was, it was confusing as we mentioned as soon as we had the opportunity
to look at the demonstrator. But it was too late to change as the validation
in ORT Marseilles had already begun. UG document about metacognition
19
mentioned that “As shown for example in the first validation results,
the current formulation of the confidence rating is not understood by most
individuals of the target group”. We argue that
(a) the formulation of the question, especially that lack of “sure of what?”
(b) the fact that the question was neither a jol nor a confidence in
retrieved answer,
(c) the lack of explanations related to the metacognitive aims and the
absence of tutorial explaining how to use certitude degree,
(d) the lack of training of students in general regarding expression of
their confidence,
(e) the use of verbal expression e.g., “sure” is confusing by itself and
should not be used when retrieving certitude (see Shuford, Albert &
Massengill (1966)20 recommendations for the use of subjective prob-
abilities instead of verbal expressions)
are all explanations of the confusion. As mentioned here above, in collect-
ing jol Nelson & Narens themselves are using a probabilistic scale from
0% to 100% by steps of 20%. In the validation report21 ORT mentions
that: “It seemed that most pupils didn’t really understand the meaning
behind the confidence degree, or the questions that was asked.” We to-
tally agree with their recommendation that suggest: “It seems that the
question related to the confidence degree wasn’t very clear, it is therefore
recommended to formulate the question in a clearer way (adapted to 13-14
years old pupils)”. To avoid confusion we propose section 7 some sugges-
tion about what to explain in the tutorial. Shuford (1966) and Leclercq
(198322 , 198623 , 199324 ) defined 4 rules when using certitude degrees:
(a) the use of probabilistic value instead of words;
(b) if points are awarded tariff should respect the decision theory;
(c) feedbacks concerning the good or not good use of certitude degree
provided to students;
19 Kickmeier-Rust, Albert & Linek (2007) A psychological perspective on metacognition and
the usage of confidence ratings. Draft for internal use of elektra consortium
20 Shuford, E. H., Albert, A., and Massengill, H. E. (1966). Admissible probability mea-
surement procedures. Psychometrika, 31(2):125–45.
21 Hirschberg, G. (2007) DELIVERABLE Name: Phase 1 – Validation Report
22 Leclercq, D. (1983). Confidence marking, its use in testing. Postlethwaite, Choppin (eds)
Evaluation in Education, Oxford : Pergamon, 1982, vol. 6, 2, pp. 161-287.
23 Leclercq, D. (1986). La conception des questions ` choix multiple, Bruxelles, Ed. Labor.
a
24 Leclercq, D. (1993). Validity, reliability and acuity of self-assessment in educational test-
ing. In Leclercq, D. and Bruno, J., editors, Item Banking : Interactive Testing and Self-
Assessment, number F 112 in NATO ASI Series, pages 114–131. Berlin: Springer Verlag.
8

9. (d) students must be trained to use certitude degrees.
The respect of these rules is mandatory for anyone willing to use certitude
degrees and avoid confusion or deception.
2. Chosen certitude and chosen answer are not recorded into the game. Both
teacher and learner should be able to view the results, success and failure,
during tries and tasks. These data should influence the feedbacks. As
such they can be part of micro-adaptivity.
3. The feedback should be more precise. Galileo might say something like:
When I asked you how confident you were about succeeding this task, you
told me you were XX% confident succeeding it.
If (FAILURE) then
If (chosen certitude degree > 50%) then
You did not succeed. But you were very confident you would
succeed. So you’ve made a bad assumption. You should be more
careful in your estimations. Try again and be more modest with
your confidence.
else
You did not succeed. But you confidence in your ability to
succeed was low. So your assumption was good: you have failed
but you knew it was risky so you associated doubt with your
try. I congratulate you for that. Give it another try.
end If
else
If (chosen certitude degree ≤ 50%) then
You’ve succeeded. Congratulations. But your confidence in
your ability was too low. You should be more confident in your
estimations.
else
You’ve succeeded. Congratulations. And your estimation was a
good assumption: you were sure and you’ve succeeded. That’s
great.
end If
end If
Notice how feedbacks about cognition and metacognition are in different sen-
tences as suggested in section 2.3.2, page 5. Knowledge feedbacks are clearly
different than support of metacognition. When a few tries have been made, the
feedback can even turn to be more precise. This will be described in the last
part of this document.
4.3 Experiment in world 2 and metacognition
We also suggest that, in world 2, the player should be able to turn on and off
the flashlight. He should only be allowed to move the blinds and the screen
9

10. when the torch is turned off. After he has placed all the objects the way he
wants, he will have to turn the flashlight on to see the results.
Even if the player does not find the correct solution he will see the result of
what he built when turning the flashlight on. He will see it in 3D and will also see
the associated top view. When the player turns on the flashlight Galileo might
also ask a question like “How sure are you that this construction will produce a
narrow beam of light?” When turning it off Galileo will give a feedback (see the
previous algorithm). Instead of Galileo always asking the same question about
confidence, we suggest that the switch of the light might be something like
off on with certitude degree of
0% 20% 40% 60% 80% 100%
Table 1: Flashlight on–off switch enhanced with certitude degree
4.4 The metal-wooden door and the blinds
Imagine that the player succeed placing the blinds and screen in LeS1.2a in
world 2 with a very good confidence. Then he/she fails in the real world with
the metal-wooden door. What kind of feedback can we give about what he/she
has learned in world 2? What kind of feedback can we give about his/her
confidence in succeeding the task based on what he/she has experimented in
world 2? We have no answer for these questions because the problem, from our
point of view, is that the metal-wooden door does not evaluate what might have
been learned with the blinds and screen in world2. This seems impossible to be
changed. So we recommend not to use certitude degrees at the moment with
the opening of the metal–wooden door situation.
5 Deeper with certitude degree
The demonstrator is working on one single expression of the player’s jol. It is
difficult for him to gain any metacognitive knowledge with a single measurement.
The only feedback we can actually give with the actual state of development of
the demonstrator is a goodbad performance associated with a confidentnon
confident jol as showed in table 2.
good bad
performance performance
confident Best Worst
good knowledge wrong knowledge
good confidence bad confidence
non confident good knowledge lack of knowledge
but but
lack of confidence good confidence
Table 2: Feedback class associated with the combination of true-false and confidence
10

11. 5.1 Theoretical background
The following is based on the works of Hunt (1993), Jans & Leclercq (1999) and
Leclercq (2003) and gives a much more detailed measure of knowledge with the
association of a confidence chosen out of a scale of 6 degrees of certitude (see
figures 3 to 5, pages 11 et 12). All figures are spectral distributions of knowledge.
On the left hand side are the incorrect answers. They are distributed by the
learner chosen confidence, ranking from left to right from 100% down to 0%. In
the middle is the grey area of not answered questions. On the right hand side
are the correct answer, also distributed by confidence but ranking from left to
right from 0% to 100%. Figure 3 showed Darwin Hunt (1993) 25 suggestion to
distinguish between three types of knowledge situations in which a person can
be in relation to a piece of content : “misinformed, uninformed, informed”.
Figure 3: Spectral distributions of Hunt’s three situations of knowledge (1993)
These situations were redefined –see figure 4– by Jans & Leclercq (1999) 26
as: “dangerous knowledge, unusable knowledge, usable knowledge”.
Unhappy with the “unusable knowledge” unable to make the difference be-
tween a student making a mistake with a low confidence and a student hav-
ing a low confidence in a correct answer, Leclercq (2003) 27 divided it into
“unawareness” and “mid knowledge” as showed in figure 5. This is an impor-
tant distinction as mid knowledge needs working on metacognitive judgment, a
25 Hunt, D. (1993). Human self-assessment : Theory and application to learning and testing.
In Leclercq, D. and Bruno, J., editors, Item Banking : Interactive Testing and Self-Assessment,
volume F 112 of NATO ASI Series, pages 177–189. Berlin: Springer Verlag
26 Jans, V. and Leclercq, D. (1999) Mesurer l’effet de l’apprentissage ` l’aide de l’analyse
a
spectrale des performances. In Depover, C. and No¨l, B., editors, L’´valuation des
e e
comp´tences et des processus cognitifs. Mod`les pratiques et contextes, pages 303-317. Brux-
e e
elles : De Boeck Universit´e
27 Leclercq, D. (2003). Un diagnostic cognitif et m´tacognitif au seuil de l’universit´. Le
e e
projet MOHICAN men´ par les 9 universit´s de la Communaut´ Fran¸aise Wallonie Bruxelles.
e e e c
Li`ge : Editions de l’Universit´ de Li`ge.
e e e
11

12. Figure 4: Spectral distribution of knowledge kinds by Jans & Leclercq (1999) based on Hunt’s
three situation of knowledge
metacognitive part, and unawareness needs to first work on knowledge, a cogni-
tive part. Again, as mentioned in section2.3.2 page 5, do not confuse assessment
of knowledge and support of metacognition with certitude degrees. It is already
Figure 5: Leclercq’s (2003) spectral splitting of Jans & Leclercq’s (1999) unusable knowledge
much better than a simple correct or not correct feedback. But certitude degrees
allows to give much more refined feedbacks.
12

13. 5.2 Application in the game
Certitude degrees will add a dimension in storytelling as we define a new “socio-
affective” goal to the game: George is to gain the trust of Galileo Galilei to
become the new trustfully leader of the Galileans. Galileo’s trust in George will
be based on his good use on certitude degrees, mainly based on confidence in
retrieved answers. The point is to give a general feedback on how the player is
using certitude degree. Not only immediatly after after a single judgment as it
is in the actual demonstrator but with enough measure to pin tendencies like
over– and under–estimation, see section 2.3, page 4.
5.3 Technical implementation
So on top of what was suggested here above, the program should compute
two values and communicate with the player about confidence (mcdca) and
imprudence (mcdia) as defined by Leclercq & Poumay (2004):
1. mcdca is the mean certitude degree when the tries are successful or
n (n · pn )
Conf idencce = CDcA = (1)
NCDcA
where
• CDcA stands for mean certitude degree for correct answers;
• n is a certitude degree (0%, 20%, 40%, 60%, 80% or 100%)
• pn is the number of use of the n certitude degree associated with
correct answers;
• NCDcA is the number of use of the certitude degrees in correct an-
swers;
2. mcdia is the mean certitude degree when the tries are unsuccessful or
n (n · qn )
Imprudence = CDiA = (2)
NCDiA
• CDiA stands for mean certitude degree for incorrect answers;
• n is a certitude degree (0%, 20%, 40%, 60%, 80% or 100%)
• qn is the number of use of the n certitude degree associated with
incorrect answers;
• NCDiA is the number of use of the certitude degrees in incorrect
answers.
Exemple : Out of a test of 7 questions a student has 5 correct answers and
2 incorrect answers. All 7 answers were associated with a certitude degree as
showed in table 3 page 14.
To compute confidence we focus on the correct answers only: first we count
the number of use of certitude degree 0% (only for correct answer!). So n is 0%.
In our exemple, there is 1 use of certitude degree 0% with a correct answer. So
p0 = 1. We multiply those two numbers by each other: n · pn giving 0 · 1 = 0.
13

14. Question Answer Certitude degree
1 C 80
2 C 20
3 C 100
4 C 0
5 C 80
6 I 60
7 I 0
Table 3: Exemple for the computation of confidence and imprudence (C for correct answer
and I for Incorrect answer).
Then we do the same for n = 20% which means that we count the number of
use of certitude degree 20%, only for correct answer! In our exemple there is 1
use of certitude 20% within all correct answers: p20 = 1. We multiply those two
numbers by each other: n · pn giving 20 · 1 = 20. Same thing for n = 40% which
means that we count the number of use of certitude degree 40%, remember only
for correct answer! In our exemple there is no use of certitude 40% within all
correct answers: p40 = 0. We multiply those two numbers by each other: n · pn
giving 40·0 = 0. Same thing for each certitude degree (60%, 80%) until we reach
certitude of 100% (n = 100%) with one use of certitude 100% making p100 = 1.
We multiply those two numbers by each other: n · pn giving 100 · 1 = 100. We
sum all n · pn into [(0 · 1) + (20 · 1) + (40 · 0) + (60 · 0) + (80 · 2) + (100 · 1)]
and divide by NCDcA or the number of use of certitude degree associated with
correct answers, in our case 5. Final equation being [0+20+0+0+160+100] = 56
5
To compute Imprudence, it is the same process but focusing only on incorrect
answers. So we finish with [(0 · 1) + (20 · 0) + (40 · 0) + (60 · 1) + (80 · 0) + (100 · 0)]
and divide by NCDcA or the number of use of certitude degree associated with
incorrect answers, in our case 2. Final equation being [0+0+0+60+0+0] = 30. So
2
in our exemple confidence is 56% and imprudence 30%.
It is in the computation of those mean values that the use of a scale with
6 levels makes sense. First Miller’s (1956) 28 works demonstrated that people
are able to work with 7 plus or minus two certitude degrees to choose from.
And the 6 levels are very easy to remember as they are from 0% to 100% step
of 20%. Shuford et al.(1966)29 demonstrated the necessity of using subjective
probabilities instead of verbal expressions. Finally it is after a number of answers
that mean values are computed taking into account every choice made by the
student. In the case of elektra, after collecting certitude degrees during the
student’s experimentations it is expected that the mean certitude of correct
answers (mcdca) –that is to say for the successful tries, should above 50%.
Mixing well the use of all 6 levels of the certitude degree scale, students should
learn to weight their certitude in a correct way when they are about to succeed.
A student will be considered as confident when his confidence is above 50%.
On the other side the mean certitude of incorrect answers (mcdia) –that is to
28 Miller, G. (1956). The magical number seven, plus or minus two: Some limits on our
capacity for processing information. Psychological Review, (63):81–97
29 Shuford, E. H., Albert, A., and Massengill, H. E. (1966). Admissible probability mea-
surement procedures. Psychometrika, 31(2):125–45
14

15. say for the unsuccessful tries, should be below 50%. Students should learn to
be modest in confidence using low value on the 6 level scale. An accident may
always occur like using a high certitude degree and failing but the tendency
should be that when about to fail student learn to be prudent. An imprudent
student will be a student with an imprudence above 50%. The best performance
is
• a confidence of 100% or a mean certitude degree for the successful answers
of 100% and
• an imprudence of 0% or a mean certitude degree for the unsuccessful
answers of 0%
For gaming reasons, LMR asked that instead of trying to reach 100% on one
scale and 0% on another scale the student should always try to maximize his
score (like gaining points) or to minimize his level (like note losing live points
when fighting). To comply we suggest to use prudence instead of imprudence
prudence = 100 − imprudence (3)
Imprudence or mean certitude degree of incorrect answer should ideally be as
low as possible. Over 50% is the limit when the student becomes imprudent.
With the inversion presented in equation (3) goals are exactly the opposite.
Prudence should ideally be as close to 100% as possible. A student will be
qualified as prudent if his prudence is over 50%.
5.4 Graphic interface
Prudence and confidence will be the indicators of Galileo’s trust gained by the
player. We suggest to represent prudence and confidence to use the certimeter
that was invented by Galileo and suggested by Frederic Pourbaix. There will
be 2 certimeters, one for the confidence (mean certitude of correct answers) and
one for prudence (100 − mean certitude of incorrect answers), see figure 6. Both
certimeter will be graduated from 0% to 100%. Suggested colors might be from
red to light red between 0% and 50% and from light green to green from 50%
to 100%.
Including prudence and confidence we offer a possible answer to ORT rec-
ommendation30 : “The idea of confidence degree [. . . ] and may have the form
of points in the game, which will bring the pupils to think further about their
answer and will bring additional anticipation.” The aim of the player will be to
reach green level score on both certimeter meaning that Galileo can trust him
when he says that he is confident or when he says that he doubt. Of course to
make the “eclipse machine” work the player will have to gain knowledge and
comprehension of optic physics and astronomy. But the player should also pay
attention to some metacognitive aspects: confidence and prudence. If the player
fails to gain Galileo’s trust but manage to get the “eclipse machine” working
he will gain access to the Galilean’s secret headquarters and to the book. But
maybe Galileo will be a bit reluctant to help the player using the book. . . and
give him a feedback in that way, unless the player starts the game all over again
and tries to improve his score on both certimeter. Success in the game could
30 Hirschberg, G. (2007) DELIVERABLE Name: Phase 1 – Validation Report
15

16. Figure 6: Rough design of a possible way to communicate to the player the level of Galileo’s
trust gained: the cursor on each certimeter takes the color of the position in the color grada-
tion. Both cursor must be at least light green to gain Galileo’s trust.
then be achieved by reaching the Galilean’s HQ. But another level of success
will be to reach HQ and to gain Galileo’s trust.
5.5 Realistic use of certitude degrees
The feedbacks could take the form described in this algorithm here after. The
limit values of 50% used in this alogrithm should be variables. Four limits are
to be created and defined at a default level of 50
conf-lim-ccd the limit to judge a single judgement with a correct answer or the limit
to be declared confident in a single judgment
prud-lim-ccd the limit to judge a single judgment with an incorrect answer or the limit
to be declared prudent in a single judgment
mcdca-lim the limit to judge the mean certitude degree for correct answer as good or
not or when is the player confident or not in the game so far
mcdia-lim the limit to judge the mean certitude degree for incorrect answer as good
or not or when is the player prudent or not in the game so far
Every situation is symbolized by a group in between brackets (e.g., F + -).
• First with a letter
F for failure in case of an unsuccessful try
S for success in case of a successful try
16

17. • a plus (+) or (-) sign
+ if the chosen confidence degree is above 50%
- if the chosen confidence degree is below 50%
• a plus (+) or (-) sign
+ if the mean confidence degree is above 50%
- if the mean confidence degree is below 50%
ccd (chosen certitude degree) : matrix;
mcdca (mean certitude degree of correct answers) : integer;
mcdia (mean certitude degree of incorrect answers) : integer;
print When I asked you how confident you were about succeeding
this task, you told me you were {ccd}% confident succeeding it.
If (FAILURE) then
If (ccd > 50) then
If (mcdia > 50) then
print Situation A (F + +)
else
print Situation B (F + -)
end If
else
If (mcdia ≤ 50) then
print Situation C (F - -)
else
print Situation D (F - +)
end If
end If
else
If (ccd ≤ 50) then
If (mcdca > 50) then
print Situation E (S - +)
else
print Situation F (S - -)
end If
else
If (mcdca ≤ 50) then
print Situation G (S + -)
else
print Situation H (S + +)
end If
end If
end If
17

18. 5.5.1 Situation A (F + +)
Situation: Test unsuccessful, confident in answer, tend to be overconfident
when wrong.
Feedback: You did not succeed. And you were very confident you would
succeed. So you’ve made a bad assumption. You should be more careful in your
estimations. When you failed you tend to choose a too high certitude degree.
So when you fail your average prudence is {100-mcdia}%. I consider that
it should be over {mcdia-lim}%. You should try to be less confident when
you choose a certitude degree. You should learn to be a bit more modest and
humble. You tend to believe you know when you’re still learning. So give it
another try. A try to be more modest if you doubt, even a little.
5.5.2 Situation B (F + -)
Situation: Test unsuccessful, confident in answer, tend to be prudent when
wrong.
Feedback: You did not succeed. You were very confident you would succeed.
So you made a bad assumption but I consider this assumption as an accident.
Usually you seem prudent in your own estimations. Every time you’ve failed
you’ve chosen before a low certitude degree. So when you fail your prudence
is {100-mcdia}%. I consider that it should be over {mcdia-lim}% so I trust
your judgement. Keep going like that when you choose a certitude degree and
maybe go back to the table for some more experimentation.
5.5.3 Situation C (F - -)
Situation: Test unsuccessful, prudent in answer, tend to be prudent when
wrong.
Feedback: You did not succeed. But you certitude in your ability to succeed
was low: you were prudent. So your assumption was good: you have failed but
you knew it was risky so you associated doubt with your try. I congratulate
you for that. It is important for me to know if I can trust you. In case case of
course you’ve failed but before failing you told me that you were not sure. And
as usual when you say that you’re not sure you do not succeed. So this failure
is not important at all. It seems you’re quite good at being prudent: usually
when you’ve failed you’ve chosen before a low certitude degree. So when you
fail your overall prudence is {100-mcdia}%. I consider that it should be over
{mcdia-lim}%. That is a good performance. Continue to do so and I will keep
my trust in your judgments. But you still need to succeed this task: give it
another try.
5.5.4 Situation D (F - +)
Situation: Test unsuccessful, prudent in answer, tend to be overconfident when
wrong.
Feedback: You did not succeed. But this time your assumption was good: you
have failed but you knew it was risky so you associated doubt with your try. I
congratulate you for that. You should continue to behave like that because it
does not appear you’re very skilled at this on the long run. For instance, when
you fail your average prudence is {100-mcdia}%. I consider that it should be
18

19. over {mcdia-lim}%. That is why I said you’re not very skilled. With such a
value I will wait for you to improve before fully trusting your judgement. Maybe
you should go back to the table for some more experimentation before giving it
another try. And do not forget to keep going like you just did in giving your
confidence.
5.5.5 Situation E (S - +)
Situation: Test successful, prudent in answer, tend to be confident when right.
Feedback: To be completed
5.5.6 Situation F (S - -)
Situation: Test successful, prudent in answer, tend to be under confident when
right.
Feedback: To be completed
5.5.7 Situation G (S + -)
Situation: Test successful, confident in answer, tend to be under confident
when right.
Feedback: To be completed
5.5.8 Situation H (S + +)
Situation: Test successful, confident in answer, tend to be confident when
right.
Feedback: To be completed
5.6 Even more complex feedbacks
It will really be enjoying for the player to hear the same long feedback. We
suggest that all these information should be delivered not at once but over a
number of tries. Counter of try and counters of times being in the same situation
(e.g., F + - or S + +) should be implemented into the if–then–else loops. So the
first time the player is in situation A he will received a more simple feedback
but the second time he will receive a different feedback. It will be only after
number of correct or incorrect answers that the computed mean values will have
any meaning and so can be given to the player. Prudence and confidence can
already have significant values computed on 55 value of certitude chosen by the
player if we had the suggested game situation into the tutorial.
6 Certitude degrees in LU1.2b
The Labset proposition for Elektra learning situation 1.2b (the slope) is similar
to what we propose for learning situation 1.2a (the blinds) : every time George
wants to release a marble he is asked his confidence in reaching the target on a
scale from 0% to 100% by step of 20%. We suggest that beneath to the cursors
for the magnet and the fan a third cursor with “confidence to reach the target”
record the confidence associated with each try. Feedbacks will be less systematic
19

20. as the player/learner should be more aware of his confidence score showed in the
certimeter. When the player successfully dropped all 3 balls in the basket Galileo
will ask him a transfer question. The player will have to chose out of a number
a drawings the one that will represent the trajectory of light when turned on.
He will also be asked about his confidence. Imagine they are only 4 drawing to
choose from and the player knows that if he fails he can immediately try again.
Without confidence degrees the player may try any of the 4 propositions until
he finds the right one. But with confidence degrees he knows that if he chooses
a wrong one and is confident in his guessing he will loose some of Galileo’s trust.
So he might know the right solution, be confident and then will answer with
a high confidence degree. He also may guess the right solution but then the
chosen certitude should be lower. If guessing all proposition until he finds the
correct one, all confidence will be low31 . So teacher will have information on
how did the student reached transfer.
7 The tutorial for certitude degrees
ORT recommended32 that “The idea of confidence degree should be explained in
the beginning of the game (tutorial)”. The player will receive some explications
about the Galileans. It is at that moment that certitude degrees should be ex-
plained. We propose a theoretical information based on transmission–reception
and a gaming situation similar to experimentation–reactivity.
The tutorial should include
1. the player should gain the trust of Galileo Galilei
2. Galileo’s trust is based on the fact the when someone says “I know” he
can trust that knowledge. And when someone doubt that person is right
to doubt.
3. To gain Galileo trust everytime the player will experiment to solve an
enigma he will be asked how confident he his that he will succeed resolving
the enigma at this exact moment, on this precise try.
4. To indicate his self-confidence the player will chose one certitude degree
between 0%, 20%, 40%, 60%, 80% and 100%. Points will be gained and
lost based on
• if what the player just tried is successful and
– if he has chosen a high certitude degree he will gain confidence
points
– if he has chosen a low confidence degree he will loose confidence
points
• else if what the player just tried is unsuccessful and
– if he has chosen a high certitude degree he will loose prudence
points
– if he has chosen a low confidence degree he will gain prudence
points
31 with the limitation that certitude will go up as the number of possible answers goes down.
32 Hirschberg, G. (2007) DELIVERABLE Name: Phase 1 – Validation Report
20

21. 5. In other word if the player bet 100% on a try and if that try is
• successful then he will gain a lot of confidence points
• unsuccessful then he will loose a lot of prudence points
6. On the other hand if the player bet 0% on a try and if that try is
• successful then he will loose a lot of confidence points
• unsuccessful then he will gain a lot of prudence points
We suggest that this “theoretical” explanation is to be followed with a Gaming
Situation based on betting certitude degrees. This might be something like the
Shannon Guessing Game (1951)33 or another gaming situation that UL will
provide.
7.1 Shannon’s guessing game
“The new method of estimating entropy exploits the fact that anyone speaking
a language possesses, implicitly, an enormous knowledge of the statistics of the
language. Familiarity with the words, idioms, cliches and grammar enables
him to fill in missing or incorrect letters in proof-reading, or to complete an
unfinished phrase in conversation. An experimental demonstration of the extent
to which English is predictable can be given as follows; Select a short passage
unfamiliar to the person who is to do the predicting, He is then asked to guess
the first letter in the passage. If the guess is correct he is so informed, and
proceeds to guess the second letter. If not, he is told the correct first letter
and proceeds to his next guess. This is continued through the text. As the
experiment progresses, the subject writes down the correct text up to the current
point for use in predicting future letters.” Gilles & Leclercq (1994) 34 have
adapted this game to train student to use certitude degrees: each time a letter
is chosen a certitude for it being the right next letter in the sentence is asked.
Metacognitive indices like realism are computed and a graphical representation
of the confidence in retrieved answer is drawn. The student tries to find the
text and to score well in metacognitive judgment.
In elektra the passage might be a simple phrase like The secret book of
the Galileans is in the third drawer of the desk. The player will face an empty
parchment that will be filled with letters written by a 3D feather when the
player types a letter and a certitude degree. If the letter is incorrect it will
burn, disappear and be replaced by the correct letter. Prudence and confidence
will be computed and a metacognitive feedback will be generated. Then the
player is to guess the next letter and so on.
7.2 Tutorial for teacher
We also suggest that informations about metacognition and certitude degrees
should be available for teachers. These informations should cover a wide domain,
from objectives in the elektra game to proposition of reflection path about
33 Shannon, C. E. (1951) Prediction and entropy in printed English, Bell Syst. Tech. J., 30,
1951, 50-64
34 Leclercq, D. & Gilles, J.-L. (1994). GUESS, un logiciel pour entraˆ
ıner ` l’auto-estimation
a
de sa comp´tence cognitive. Actes du colloque QCM et questionnaires ferm´s, Paris: ISIEE
e e
21

22. metacognition in classroom and support to further metacognitive activities for
the students.
8 Conclusion
Knowing how to learn, and knowing which strategies work best, are valuable
skills that differentiate expert learners from novice learners. Metacognition, or
awareness of the process of learning, is a critical ingredient to successful learning.
“Metacognition is an important concept in cognitive theory. It consists of two
basic processes occurring simultaneously: monitoring your progress as you learn,
and making changes and adapting your strategies if you perceive you are not
doing so well” (Winn, W. & Snyder, D., 1998).35
Contents
1 Why certitude degrees in learning? 1
2 What is metacognition? 2
2.1 Metacognition in general . . . . . . . . . . . . . . . . . . . . . . . 2
2.2 Certitude degrees and metacognition . . . . . . . . . . . . . . . . 4
2.3 Metacognition in the demonstrator . . . . . . . . . . . . . . . . . 4
2.3.1 What is a JOL? . . . . . . . . . . . . . . . . . . . . . . . 5
2.3.2 JOL or confidence in retrieved answer . . . . . . . . . . . 5
3 Metacognition in games 6
4 About the demonstrator, 2006 version 7
4.1 A surprise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4.2 General comments . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4.3 Experiment in world 2 and metacognition . . . . . . . . . . . . . 9
4.4 The metal-wooden door and the blinds . . . . . . . . . . . . . . . 10
5 Deeper with certitude degree 10
5.1 Theoretical background . . . . . . . . . . . . . . . . . . . . . . . 11
5.2 Application in the game . . . . . . . . . . . . . . . . . . . . . . . 13
5.3 Technical implementation . . . . . . . . . . . . . . . . . . . . . . 13
5.4 Graphic interface . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
5.5 Realistic use of certitude degrees . . . . . . . . . . . . . . . . . . 16
5.5.1 Situation A (F + +) . . . . . . . . . . . . . . . . . . . . . 18
35 Winn, W. & Snyder D. (1996). Cognitive perspectives in pyschology. In D.H. Jonassen,
ed. Handbook of research for educational communications and technology, 112-142. New
York: Simon & Schuster Macmillan
22

23. 5.5.2 Situation B (F + -) . . . . . . . . . . . . . . . . . . . . . 18
5.5.3 Situation C (F - -) . . . . . . . . . . . . . . . . . . . . . . 18
5.5.4 Situation D (F - +) . . . . . . . . . . . . . . . . . . . . . 18
5.5.5 Situation E (S - +) . . . . . . . . . . . . . . . . . . . . . . 19
5.5.6 Situation F (S - -) . . . . . . . . . . . . . . . . . . . . . . 19
5.5.7 Situation G (S + -) . . . . . . . . . . . . . . . . . . . . . 19
5.5.8 Situation H (S + +) . . . . . . . . . . . . . . . . . . . . . 19
5.6 Even more complex feedbacks . . . . . . . . . . . . . . . . . . . . 19
6 Certitude degrees in LU1.2b 19
7 The tutorial for certitude degrees 20
7.1 Shannon’s guessing game . . . . . . . . . . . . . . . . . . . . . . 21
7.2 Tutorial for teacher . . . . . . . . . . . . . . . . . . . . . . . . . . 21
8 Conclusion 22
List of Tables
1 Flashlight on–off switch enhanced with certitude degree . . . . 10
2 Feedback class associated with the combination of true-false and
confidence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3 Exemple for the computation of confidence and imprudence (C
for correct answer and I for Incorrect answer). . . . . . . . . . . . 14
List of Figures
1 Main components of metacognitive monitoring and metacognitive
control defined by Nelson (1996) and Nelson & Narens (1994) . . 3
2 Cut scene from the elektra demonstrator, 2006 . . . . . . . . . 7
3 Spectral distributions of Hunt’s three situations of knowledge (1993) 11
4 Spectral distribution of knowledge kinds by Jans & Leclercq (1999)
based on Hunt’s three situation of knowledge . . . . . . . . . . . 12
5 Leclercq’s (2003) spectral splitting of Jans & Leclercq’s (1999)
unusable knowledge . . . . . . . . . . . . . . . . . . . . . . . . . 12
6 Rough design of a possible way to communicate to the player the
level of Galileo’s trust gained: the cursor on each certimeter takes
the color of the position in the color gradation. Both cursor must
be at least light green to gain Galileo’s trust. . . . . . . . . . . . 16
23