http://oro.open.ac.uk/41711/
The LAK15 theme “shifts the focus from data to impact”, noting the potential for Learning Analytics based on existing technologies to have scalable impact on learning for people of all ages. For such demand and potential in scalability to be met the challenges of addressing higher-order thinking skills should be addressed. This paper discuses one such approach – the creation of an analytic and task model to probe epistemic cognition in complex literacy tasks. The research uses existing technologies in novel ways to build a conceptually grounded model of trace-indicators for epistemic-commitments in information seeking behaviors. We argue that such an evidence centered approach is fundamental to realizing the potential of analytics, which should maintain a strong association with learning theory.
2. Developing a multiple-document-
processing performance assessment
for epistemic literacy
Simon Knight
Karen Littleton
Dirk Tempelaar (& team), Matt Mitsui, Chirag Shah,
Simon Buckingham Shum, Bart Rienties, Fridolin Wild
http://sjgknight.com
@sjgknight
3. Introduction
• New technologies:
– Foreground issues in
literacy – finding &
comprehending multi-
media
– Afford new opportunity
to explore & tackle these
issues
4. What is [Digital] literacy?
• [Digital] literacy –
– Comprehension in context of selecting & processing rich
online multimedia content with diverse sources (OECD,
2013)
5.
6. Tasks
• Two collaborative tasks facilitated by a browser
addon
• One group provided with documents; the second
group searches on the web
• “A review of the best supported claims around
the risks” of a substance (herbicide or food
supplement)
7. 7
Friends of
the Earth:
Press
Release
(Urine
presence)
FoE
Commission
ed report
(‘scientific’)
(-ve)
Science-
Literacy
website:
Refutation
(+ve)
Farmer’s
Weekly
Reprints
(+ve)
Related
peer-review
publication
(Limited
risk)
Peer-review
publication
Health
danger
Reuters
Reprints
main claims
Blogger
Critiques
journal &
author
Peer-review
publication
(Limited
health risks)
Peer-review
review of
literature
(Limited risk
to health or
plants)
Peer-review
of lit
(Limited
risk; control
suggestions)
Urine
Health
Agricu
lture
9. Skills of literacy
Rouet [39] – students should be taught:
• Skill of integration: the ability to
connect prior and new information,
including across documents, and
including where claims are
inconsistent or contradictory
• Skill of sourcing: the ability to identify
parameters that characterize the
author and conditions of production
of the information
• Skill of corroboration: the ability to
check information against multiple
sources for its accuracy
10. Skills of literacy
• Communication – particularly meta-discourse
involving exploratory discussion of credentials,
sources, etc. – is key to collectively authored
written outputs
(Goldman and Scardamalia [17, p.260]).
16. Source features
• Participants given 2 texts of varying quality are
more likely to favor high quality sources,
• BUT they make little reference to source
features (on average only 1.85 out of 10
features);
• and rarely (<6% of the 154 participants)
explicitly use source information for justifying
their evaluation of the text’s explanation
Kobayashi [26](2014)
17. Epistemic commitments
• Multiple Documents—Task-based Relevance
Assessment and Content Extraction (MD-TRACE)
model (Rouet/Rouet & Britt)
• 5 Steps:
1. Model construction
2. information need assessment;
3. document processing; - relevance, update needs,
seek new information
4. task product creation;
5. task product assessment.
18. “exploring students’ thought processes during online
searching allows examination of personal epistemology
not as a decontextualized set of beliefs, but as an
activated, situated aspect of cognition that influences the
knowledge construction process” (Hofer, 2004, p. 43).
The Lens of Epistemic Beliefs
Commitments
19. Certainty
19
A
C
By x
(2002)
……
……
……
…
Source
A B C
A
B
C
Simplicity
Justification
1.How simple or complex knowledge is – can it be compartamentalised, or is it interconnected?
2.The justification for knowledge – does knowledge come from observation, or from rules of
enquiry?
3.The source of knowledge – Is knowledge ‘given’ & functional, or is knowledge constructed in
interaction with others?
4.The certainty of knowledge – is knowledge tentative and evolving, or static and absolute
25. ?
• ISEQ - psychometrics, etc.
• Outcome data (IRR)
• Trust
• Predictive value of summary data
(e.g. pages->Topics)
• Sequences
• Deep dive – key patterns, proof of concept of
those patterns. Critical incidents.
26. Thank you
Simon Knight
http://sjgknight.com
@sjgknight
Knight, S., Littleton, K., (2015). Developing a multiple-
document-processing performance assessment for epistemic
literacy. In The 5th International Learning Analytics &
Knowledge Conference (LAK15): Scaling Up: Big Data to Big
Impact, Poughkeepsie, NY. http://oro.open.ac.uk/41711/
28. @sjgknight www.sjgknight.com
Thank you!• Knight, S. (2013). Finding Knowledge: What it means to ‘know’ in the age of search,
Presented at the 2nd Society of the Query Conference, Amsterdam
• Knight, S., Arastoopour, G., Williamson Shaffer, D., Buckingham Shum, S., Littleton, K.,
(2014). Epistemic Networks for Epistemic Commitments. Presented at the International
Conference of the Learning Sciences (ICLS), Boulder, Colorado. Available via The Open
University Eprint Archive: http://oro.open.ac.uk/39254/
• Knight, S., Buckingham Shum, S., & Littleton, K. (In Press, 2014). Epistemology,
Assessment, Pedagogy: Where Learning Meets Analytics in the Middle Space. Journal of
Learning Analytics. Available via The Open University Eprint Archive:
http://oro.open.ac.uk/39226 (HEA presentation version Blog/presentation)
• Knight, S., Mercer, N. (Forthcoming, 2014). The role of exploratory talk in classroom
search engine tasks. Technology, Pedagogy and Education. Available via The Open
University Eprint Archive: http://oro.open.ac.uk/39181/
• Knight, S., & Littleton, K. (2015). Thinking, Interthinking, and Technological Tools. In R.
Wegerif, J. C. Kaufman, & L. Li (Eds.), The Routledge International Handbook of Research
on Teaching Thinking. Routledge.
• In draft publications on computational approaches to discourse analysis + social account
of epistemic cognition
29. References
Anmarkrud, Ø., Bråten, I., & Strømsø, H. I. (In Press). Multiple-documents literacy: Strategic processing, source awareness, and argumentation
when reading multiple conflicting documents. Learning and Individual Differences. doi:10.1016/j.lindif.2013.01.007
Bromme, R., Pieschl, S., & Stahl, E. (2009). Epistemological beliefs are standards for adaptive learning: a functional theory about epistemological
beliefs and metacognition. Metacognition and Learning, 5(1), 7–26. doi:10.1007/s11409-009-9053-5
Ferguson, R. (2012). The State of Learning Analytics in 2012: A Review and Future Challenges (Technical Report No. kmi-12-01). Knowledge Media
Institute: The Open University, UK. Retrieved from http://kmi.open.ac.uk/publications/pdf/kmi-12-01.pdf
Goldman, S. R., Braasch, J. L. G., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and Learning From Internet Sources:
Processing Patterns of Better and Poorer Learners. Reading Research Quarterly, 47(4), 356–381. doi:10.1002/RRQ.027
Goldman, S. R., Lawless, K., Pellegrino, J. W., & Gomez, K. (2012). A Technology for Assessing Multiple Source Comprehension: An Essential Skill of
the 21st Century. In M. Mayrath (Ed.), Technology-based assessments for 21st century skills: Theoretical and practical implications from modern
research. Information Age Publishing (IAP).
Gress, C. L. Z., Fior, M., Hadwin, A. F., & Winne, P. H. (2010). Measurement and assessment in computer-supported collaborative learning.
Computers in Human Behavior, 26(5), 806–814. doi:10.1016/j.chb.2007.05.012
Hastings, P., Hughes, S., Magliano, J. P., Goldman, S. R., & Lawless, K. (2012). Assessing the use of multiple sources in student essays. Behavior
Research Methods, 44(3), 622–633. doi:10.3758/s13428-012-0214-0
Lawless, K. A., Goldman, S. R., Gomez, K., Manning, F., & Braasch, J. (2012). Assessing multiple source comprehension through evidence-centered
design. In J. Sabatini P. .., T. O’Reilly, & E. Albro (Eds.), Reaching an understanding: Innovations in how we view reading assessment (pp. 3–17).
Rowman & Littlefield.
Mason, L., Ariasi, N., & Boldrin, A. (2011). Epistemic beliefs in action: Spontaneous reflections about knowledge and knowing during online
information searching and their influence on learning. Learning and Instruction, 21(1), 137–151. doi:10.1016/j.learninstruc.2010.01.001
Mason, L., Boldrin, A., & Ariasi, N. (2009). Epistemic metacognition in context: evaluating and learning online information.Metacognition and
Learning, 5(1), 67–90. doi:10.1007/s11409-009-9048-2
OECD. (2013). PISA 2015: Draft reading literacy framework. OECD Publishing. Retrieved from
http://www.oecd.org/pisa/pisaproducts/Draft%20PISA%202015%20Reading%20Framework%20.pdf
Rouet, J.-F. (2006). The Skills of Document Use: From Text Comprehension to Web-Based Learning (First Edition edition.). Mahwah, NJ: Routledge.
Strømsø, H. I., Bråten, I., & Britt, M. A. (2011). Do students’ beliefs about knowledge and knowing predict their judgement of texts’
trustworthiness? Educational Psychology, 31(2), 177–206. doi:10.1080/01443410.2010.538039
Tsai, P.-S., Tsai, C.-C., & Hwang, G.-J. (2011). The correlates of Taiwan teachers’ epistemological beliefs concerning Internet environments, online
search strategies, and search outcomes. The Internet and Higher Education, 14(1), 54–63. doi:10.1016/j.iheduc.2010.03.003
Van Strien, J., Brand-Gruwel, S., & Boshuizen, E. (2012). Do prior attitudes influence epistemic cognition while reading conflicting information?
Retrieved from http://dspace.ou.nl/handle/1820/4390
Walraven, A., Brand-Gruwel, S., & Boshuizen, H. (2008). Information-problem solving: A review of problems students encounter and instructional
solutions. Computers in Human Behavior, 24(3), 623–648. doi:10.1016/j.chb.2007.01.030
30. Task1 (warmup)
Please type the answers to the following three prompts in your Task Pad (click in
the bar at the top of the browser). You may use the internet to find the answers.
• In 2010 what was the educational expenditure per primary student in
[XXX] as a % of GDP?
• In 2010 what was the total health expenditure as a % of GDP in [XXX]?
• How much (in US dollars) does a big mac cost in [ZZZ]?
Warmup – variables,
– XXX= France, Finland, Ethiopia, Estonia, The Gambia, Guinea, Guyana,
Guatemala, El Salvador, Ecuador (list of countries not too far from top of
alphabet, with data for 2010)
– Yyy = France, Norway, Switzerland, Sweden, Finland, Italy, Ireland, Canada,
Greece, Turkey, Japan, Egypt, Russia, (sublist of countries on big-mac index)
If you find the warmup taking too long (over 10 minutes) but you feel you’re
now comfortable with using Coagmento, you should move on to Task 1.
• Task Submission Procedure [Slide 9]
31. MDPFor this task, you will be researching a chemical used in herbicide (Roundup) called Glyphosate.
Your task is to act as an advisor to an official within the science ministry. You are advising an
official on the issues below. The official is not an expert in the area, but you can assume they are
a generally informed reader. They are interested in the best supported claims in the
documents. Produce a summary of the best supported claims you find and explain why you
think they are. Note you are not being asked to “create your own argument” or “summarise
everything you find” but rather, make a judgement about which claims have the strongest
support.
A colleague has already found a number of documents for you to process with your partner, you
should use these to extract the best supported claims (without using the internet to find further
material).
You should:
Read the questions/topic areas provided, these will require you to find information and
arguments in the documents to present the best supported of these, you should decide with
your partner which are best as you read.
Group information together by using headings in the Editor
You should work with your partner to explain why the claims you’ve found are the best available
You should spend about 45 minutes on this task
A review is coming up for the license of Glyphosate, the official would like to know the best
supported claims around its risks.
A colleague has collected some documents, available from the
the searching, selecting, and processing of complex documents and multi-media on the web can be seen as a component of literacy (OECD, 2013) and be related to the ways in which students source, corroborate, and integrate claims – key facets of literacy for mature internet use (Rouet, 2006, p. 177). There is thus a growing interest in the learning sciences around how students find information, and how they treat the information that they do find (Barzilai & Zohar, 2009, 2012; Bråten & Samuelstuen, 2007; Bråten & Strømsø, 2006; Chiu, Liang, & Tsai, 2013; Ferguson, Bråten, Strømsø, & Anmarkrud, 2013; Hofer, 2004; C.-Y. Hsu, Tsai, Hou, & Tsai, 2013; L. Hsu, 2014; Lin & Tsai, 2008; Singh, Hsu, & Moon, 2013; Tsai, Hsu, & Tsai, 2012).
Not ‘digital’ in sense of coding, etc.
Interest is in rich understanding of literacy
Internet use prevalent, as are difficulties in information seeking using the internet
the searching, selecting, and processing of complex documents and multi-media on the web can be seen as a component of literacy (OECD, 2013) and be related to the ways in which students source, corroborate, and integrate claims – key facets of literacy for mature internet use (Rouet, 2006, p. 177). There is thus a growing interest in the learning sciences around how students find information, and how they treat the information that they do find (Barzilai & Zohar, 2009, 2012; Bråten & Samuelstuen, 2007; Bråten & Strømsø, 2006; Chiu, Liang, & Tsai, 2013; Ferguson, Bråten, Strømsø, & Anmarkrud, 2013; Hofer, 2004; C.-Y. Hsu, Tsai, Hou, & Tsai, 2013; L. Hsu, 2014; Lin & Tsai, 2008; Singh, Hsu, & Moon, 2013; Tsai, Hsu, & Tsai, 2012).
They argue that we need two foci:
Productive use of metadata and meta-discourse – credentials, dates, source locations, quote v paraphrase, citations, primary/secondary source, etc. are all important parts of the discourse, and the discourse around this becomes an object for discourse (meta-discourse) too
Use of authoritative sources (i.e. stating claims, and citing sources), with a focus on discourse for idea improvement and knowledge-creation
So, in my context, we might imagine a student saying “we should try looking on W for that” – so we know something about how they’re thinking about seeking/search already
We can see their query (and where it’s conducted)
Images public domain
Images public domain
Participants given 2 texts of varying quality are more likely to favor high quality sources,
BUT they make little reference to source features (on average only 1.85 out of 10 features); and rarely (<6% of the 154 participants) explicitly use source information for justifying their evaluation of the text’s explanation
Kobayashi [26] indicated through a controlled experimental design, while:
the searching, selecting, and processing of complex documents and multi-media on the web can be seen as a component of literacy (OECD, 2013) and be related to the ways in which students source, corroborate, and integrate claims – key facets of literacy for mature internet use (Rouet, 2006, p. 177). There is thus a growing interest in the learning sciences around how students find information, and how they treat the information that they do find (Barzilai & Zohar, 2009, 2012; Bråten & Samuelstuen, 2007; Bråten & Strømsø, 2006; Chiu, Liang, & Tsai, 2013; Ferguson, Bråten, Strømsø, & Anmarkrud, 2013; Hofer, 2004; C.-Y. Hsu, Tsai, Hou, & Tsai, 2013; L. Hsu, 2014; Lin & Tsai, 2008; Singh, Hsu, & Moon, 2013; Tsai, Hsu, & Tsai, 2012).
And particularly interested in this in the context of information seeking, but rejecting the ‘beliefs’ element, instead focussing on the commitments to epistemic standards implicit (or explicit) in student’s seeking processes
Lower image from http://commons.wikimedia.org/wiki/File:LensFilter-001.jpg (cc-by)
http://commons.wikimedia.org/wiki/File:Dichroic_filters.jpg public domain (by NASA)
How simple or complex knowledge is – can it be compartamentalised, or is it interconnected?
The justification for knowledge – does knowledge come from observation, or from rules of enquiry?
The source of knowledge – Is knowledge ‘given’ & functional, or is knowledge constructed in interaction with others?
The certainty of knowledge – is knowledge tentative and evolving, or static and absolute – I conceptualise this as a function of justification, source and simplicity – it relates to whether people consider what might be called meta-knowledge (dates, location, etc. of knowledge) in their understanding of the complexity (simplicity), interactive-nature (source) and rules of enquiry (justification) for knowledge. Specific issues could be taught and drawn out (e.g., around historiography) but conceptually certainty might not be distinct.
https://openclipart.org/detail/28499/purzen-Clock-face-by-purzen
"Scale-free network sample". Licensed under CC BY-SA 3.0 via Wikimedia Commons - https://commons.wikimedia.org/wiki/File:Scale-free_network_sample.png#mediaviewer/File:Scale-free_network_sample.png
https://openclipart.org/detail/166341/People-044-Talking-Bubble-by-palomaironique
https://openclipart.org/detail/62989/scales-of-justice-by-laobc
https://openclipart.org/detail/21703/tom-Bunsen-Burner-lineart-by-tom
https://openclipart.org/detail/203314/chemical-by-Minduka
https://openclipart.org/detail/202707/1412278291-by-pnx
https://openclipart.org/detail/202678/1412242970
How simple or complex knowledge is – can it be compartamentalised, or is it interconnected?
The justification for knowledge – does knowledge come from observation, or from rules of enquiry?
The source of knowledge – Is knowledge ‘given’ & functional, or is knowledge constructed in interaction with others?
The certainty of knowledge – is knowledge tentative and evolving, or static and absolute – I conceptualise this as a function of justification, source and simplicity – it relates to whether people consider what might be called meta-knowledge (dates, location, etc. of knowledge) in their understanding of the complexity (simplicity), interactive-nature (source) and rules of enquiry (justification) for knowledge. Specific issues could be taught and drawn out (e.g., around historiography) but conceptually certainty might not be distinct.
https://openclipart.org/detail/28499/purzen-Clock-face-by-purzen
"Scale-free network sample". Licensed under CC BY-SA 3.0 via Wikimedia Commons - https://commons.wikimedia.org/wiki/File:Scale-free_network_sample.png#mediaviewer/File:Scale-free_network_sample.png
https://openclipart.org/detail/166341/People-044-Talking-Bubble-by-palomaironique
https://openclipart.org/detail/62989/scales-of-justice-by-laobc
https://openclipart.org/detail/21703/tom-Bunsen-Burner-lineart-by-tom
https://openclipart.org/detail/203314/chemical-by-Minduka
https://openclipart.org/detail/202707/1412278291-by-pnx
https://openclipart.org/detail/202678/1412242970
http://pixabay.com/en/search-box-optic-http-www-13476/
break point analysis -e.g. shifts in language re: Britte
Source v I think - look for sequences of source->i think, i think->source
What behavioural patterns would we hypothesise? E.g.
page->copy->etherpad (Synth-list)
Chat co-occurrence data (
Query->page->query->page (SDiv/topic)
SDiv
Topic
SQual
Synth
page-vis
CIS
CIS MDP
page-use
MDP (Hyp)
MDP
Chat-Explor
CIS MDP
MDP
CIS MDP (Hyp)
Chat-sourQual
(Hyp)
MDP (Hyp)
MDP
Chat-Topic
CIS MDP
CIS MDP (Hyp)
CIS MDP
CIS
TextLen
CIS MDP
CIS MDP
CIS
How simple or complex knowledge is – can it be compartamentalised, or is it interconnected?
The justification for knowledge – does knowledge come from observation, or from rules of enquiry?
The source of knowledge – Is knowledge ‘given’ & functional, or is knowledge constructed in interaction with others?
The certainty of knowledge – is knowledge tentative and evolving, or static and absolute – I conceptualise this as a function of justification, source and simplicity – it relates to whether people consider what might be called meta-knowledge (dates, location, etc. of knowledge) in their understanding of the complexity (simplicity), interactive-nature (source) and rules of enquiry (justification) for knowledge. Specific issues could be taught and drawn out (e.g., around historiography) but conceptually certainty might not be distinct.
https://openclipart.org/detail/28499/purzen-Clock-face-by-purzen
"Scale-free network sample". Licensed under CC BY-SA 3.0 via Wikimedia Commons - https://commons.wikimedia.org/wiki/File:Scale-free_network_sample.png#mediaviewer/File:Scale-free_network_sample.png
https://openclipart.org/detail/166341/People-044-Talking-Bubble-by-palomaironique
https://openclipart.org/detail/62989/scales-of-justice-by-laobc
https://openclipart.org/detail/21703/tom-Bunsen-Burner-lineart-by-tom
https://openclipart.org/detail/203314/chemical-by-Minduka
https://openclipart.org/detail/202707/1412278291-by-pnx
https://openclipart.org/detail/202678/1412242970
http://pixabay.com/en/search-box-optic-http-www-13476/
NOTE TO US: For 1, building off http://searchresearch1.blogspot.co.uk/2014/05/wednesday-search-challenge-52014-how.html, best way to answer ‘2’ is to just use the same website. For 3 expect to see ‘big mac index’ or/and www.numbeo.com http://www.statista.com/statistics/274326/big-mac-index-global-prices-for-a-big-mac/