This presentation is prepared for a seminar of The Open University's Academic & Professional Literacies Forum

1. USING LEARNER ANALYTICS
TO SUPPORT THE ACADEMIC WRITING
IN HIGHER EDUCATION
DUYGU SIMSEK
Academic and Professional Literacies Forum, The Open University, UK 07th May, 2014
people.kmi.open.ac.uk/simsek
duygu.simsek@open.ac.uk
simsekduygu_
Supervisors: Prof. Simon Buckingham Shum, Dr. Rebecca Ferguson, & Dr. Anna De Liddo
Dr. Ágnes Sándor, Xerox Research Centre Europe

2. ABOUT ME
 Ankara, Turkey
 BA&MA in Computer and Instructional Technologies
Bilkent University, Turkey
 MSc. Software Engineering
University of Southampton, UK
 2nd year PhD Research Student KMi
The Open University, UK
Duygu Simsek, APL Forum Talk
2

3. OUTLINE –
Using Learner Analytics to Support
the Academic Writing in Higher Education
 Research Aim
 Where this research sits?
 Academic Writing
 Learning Analytics
 Computational Text Analysis
 Research Questions
 Research Methods
Duygu Simsek, APL Forum Talk
3

4. RESEARCH AIM
To investigate
whether computational techniques can automatically identify
the attributes of good academic writing in as correlated with
grades of the essay and as identified in the literature
if this proves possible, how best to feed back actionable
analytics to support students and educators
whether this feedback has any demonstrable benefits
Duygu Simsek, APL Forum Talk
4

5. WHERE THIS RESEARCH SITS?
Duygu Simsek, APL Forum Talk
5
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parsers
Discourse
Centric
Learning
Analytics
Meta-
discourse
in Student
writing

6. WHERE THIS RESEARCH SITS?-
ACADEMIC WRITING
Key aim of academic
writing is to convince
readers about the validity
of the claims and
arguments put forward
through an effective
narrative.
Duygu Simsek, APL Forum Talk
6
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parsers
Discourse
Centric
Learning
Analytics
Meta-
discourse
in Student
writing

7. WHERE THIS RESEARCH SITS?-
META-DISCOURSE
This effective narrative is
signalled through meta-
discourse!
Duygu Simsek, APL Forum Talk
7
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parsers
Discourse
Centric
Learning
Analytics
Meta-
discourse
in Student
writing

8. META-DISCOURSE
Meta-discourse refers to the features of text that convey the author’s intended
meaning and intention. It provides cues to the reader which explicitly express a
viewpoint, argument and claim, and signals the writer's stance.
Duygu Simsek, APL Forum Talk
8
Fig. 1 Meta-discourse that convey summary statements
CuestoSummary
statements

9. EXAMPLES OF META-DISCOURSE CUES THAT
SIGNAL ACADEMIC/ANALYTICAL RHETORICAL MOVES
BACKGROUND KNOWLEDGE:
Recent studies indicate …
the previously proposed …
… is universally accepted
Duygu Simsek, APL Forum Talk
9
NOVELTY:
New insights provide direct
evidence…
…suggest a new approach…
Results define a novel role ...
OPEN QUESTION:
Little is known …
… role … has been elusive
Current data is insufficient…
TENDENCY:
... emerging as a promising
approach
Our understanding ... has
grown exponentially ...
Growing recognition of the
importance ...
CONTRASTING IDEAS:
In contrast with previous
hypotheses ...
... inconsistent with past
findings ...
SIGNIFICANCE:
studies ... have provided
important advances
... is crucial for ... understanding
valuable information ... from
SURPRISE:
We have recently observed ...
surprisingly
We have identified ... unusual
The recent discovery ... suggests
intriguing roles
SUMMARISING:
The goal of this study ...
Here, we show ...
Our results ... indicate

10. WHERE THIS RESEARCH SITS?-
META-DISCOURSE
 In order to assess students’
writing therefore, educators
will be examining students’
use of meta-discourse which
make their students’ thinking
visible.
 However, students find it
challenging to learn to write
in an academically sound
way.
 They need to learn how to
make their thinking visible by
recognising and deploying
meta-discourse.
Duygu Simsek, APL Forum Talk
10
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parsers
Discourse
Centric
Learning
Analytics
Meta-
discourse
in Student
writing

11. WHERE THIS RESEARCH SITS?-
COMPUTATIONAL TEXT ANALYSIS
 Meta-discourse cues
are automatically
identifiable.
 This PhD investigates
whether it is possible to
provide automatic
meta-discourse analysis
of student writing
through the use of a
particular rhetorical
parser, XIP.
Duygu Simsek, APL Forum Talk
11
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parsers
(XIP)
Discourse
Centric
Learning
Analytics
Meta-
discourse
in Student
writing

12. EXAMPLE OF A RHETORICAL PARSER:
INCREMENTAL PARSER (XIP)
 Natural Language Processing (NLP)* product which includes a
rhetorical parser detecting meta-discourse in academic texts.
 XIP extracts salient sentences based on their rhetorical functions:
 Background Knowledge
 Summarising
 Tendency
 Novelty
 Significance
 Surprise
 Open Question
 Contrasting Ideas
Duygu Simsek, APL Forum Talk
12
*language processing by computers that enable computers to derive meaning from natural language input

13. Student Writing Analysed by XIP
Duygu Simsek, APL Forum Talk
13
CONTRAST
SUMMARY

14. RHETORICAL FUNCTIONS CLASSIFIED BY XIP
BACKGROUND KNOWLEDGE:
Recent studies indicate …
the previously proposed …
… is universally accepted
Duygu Simsek, APL Forum Talk
14
NOVELTY:
New insights provide direct
evidence…
…suggest a new approach…
Results define a novel role ...
OPEN QUESTION:
Little is known …
… role … has been elusive
Current data is insufficient…
TENDENCY:
... emerging as a promising
approach
Our understanding ... has
grown exponentially ...
Growing recognition of the
importance ...
CONTRASTING IDEAS:
In contrast with previous
hypotheses ...
... inconsistent with past
findings ...
SIGNIFICANCE:
studies ... have provided
important advances
... is crucial for ... understanding
valuable information ... from
SURPRISE:
We have recently observed ...
surprisingly
We have identified ... unusual
The recent discovery ... suggests
intriguing roles
SUMMARISING:
The goal of this study ...
Here, we show ...
Our results ... indicate

15. Fine for
researchers or
machines but it is
not
learner/educator
friendly
XIP’s Output
Duygu Simsek, APL Forum Talk
15

16. WHY XIP? –
KEY FEATURES OF ACADEMIC WRITING?
Duygu Simsek, APL Forum Talk
16
Relevance
Understanding & Knowledge
Structure & Organisation
Linguistic Accuracy
Illustrations
Referencing
Argumentation

17. WHY XIP?
There is a mapping between good and strong features of academic
writing and the XIP’s rhetorical functions.
Duygu Simsek, APL Forum Talk
17

18. WHERE THIS RESEARCH SITS?-
Learning Analytics
XIP is a parser with
potential, if it can be
embedded in a more
complete learning
analytics (LA) approach.
Duygu Simsek, APL Forum Talk
18
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parsers
(XIP)
Discourse
Centric
Learning
Analytics
Meta-
discourse
in Student
writing

19. WHERE THIS RESEARCH SITS?-
Learning Analytics (LA)
 Learning moves online (i.e. pure online courses –OU, MOOCs –
FutureLearn, Coursera)
 Digital learners leave data trails in online activities
 Learner-produced online student data (i.e. data from
Learning Management Systems)
 Collecting traces left behind to improve learning
 LA is “measurement, collection, analysis and reporting of data
about learners and their contexts, for purposes of understanding
and optimising learning and the environments in which it
occurs”.[1]
Duygu Simsek, APL Forum Talk
19
[1] Call for Papers of the 1st International Conference on Learning Analytics & Knowledge (LAK 2011). Retrieved 12 February 2014.

20. WHERE THIS RESEARCH SITS?-
Learning Analytics (LA)
 LA stakeholders can be educators, learners and administrators.
 educators can learn about student activities and progress
 learners can get feedback about their own progress,
understand their own learning habits and clearly see the
impact of the activities helping them more
 administrators can make departmental and institutional
level adjustments based on the data collected and
analysed
 The most common use of LA is to spot learners who appear to
fail and to make interventions in order to help those students.
Duygu Simsek, APL Forum Talk
20

21. WHERE THIS RESEARCH SITS?-
Discourse-centric Learning Analytics
How should a DCLA
approach be validated?
Duygu Simsek, APL Forum Talk
21
ACADEMIC
WRITING
LEARNING
ANALYTICS
COMPUTATIONAL
TEXT ANALYSIS
Rhetorical
Parsers
(XIP)
Discourse
Centric
Learning
Analytics
(DCLA)
Meta-
discourse
in Student
writing

22. MAIN RESEARCH QUESTION
Duygu Simsek, APL Forum Talk
22
To what degree can computational text analysis
and visual analytics be used to
support the academic writing of
students in higher education?

23. TO WHAT EXTENT IS THE RHETORICAL PARSER XIP ACCURATE
AND SUFFICIENT FOR IDENTIFYING THE ATTRIBUTES OF GOOD
ACADEMIC WRITING WITHIN STUDENT WRITING, AS JUDGED BY
THE GRADE, AND BY EDUCATORS?
Duygu Simsek, APL Forum Talk
23
XIP
Evaluates Accuracy
& Sufficiency
Any correlation
between Grades &
XIP output?
XIP’s Highlights vs.
Marker’s
RQ1

24. TO WHAT EXTENT IS THE RHETORICAL PARSER XIP ACCURATE
AND SUFFICIENT FOR IDENTIFYING THE ATTRIBUTES OF GOOD
ACADEMIC WRITING WITHIN STUDENT WRITING, AS JUDGED BY
THE GRADE, AND BY EDUCATORS?
Duygu Simsek, APL Forum Talk
24
RQ1
XIP Highlighted Student Writing
Any correlation between
the final grade of writing & XIP findings?
Pearson for
Total number of salient sentences vs. Grade
Generalised Multiple Regression
How strongly each rhetorical sentence type
influences the final grade
Grades

25.  The OU’s S288 Practical Science
 1st year undergraduates
 5 strands:
 Physics and Astronomy
 Chemistry and Analysis
 Environmental Sciences
 Earth and Environment
 Biology and Health
 Collaborative scientific writing assignment
Learning Analytics Summer Institute (LASI), UK, Informatics Forum, Edinburgh
July 5, 2013
DATA
Duygu Simsek, APL Forum Talk
25

26.  ~300 1st year undergraduate students in 2012 (33 reports)
 ~600 in 2013 (69 reports)
 Not a meaningful correlation between total number of
sentences found and grades.
 Weak negative correlation between summary & grades.
Pearson: -0.28
Learning Analytics Summer Institute (LASI), UK, Informatics Forum, Edinburgh
July 5, 2013
PRELIMINARY RESULTS
Duygu Simsek, APL Forum Talk
26

27. TO WHAT EXTENT IS THE RHETORICAL PARSER XIP ACCURATE
AND SUFFICIENT FOR IDENTIFYING THE ATTRIBUTES OF GOOD
ACADEMIC WRITING WITHIN STUDENT WRITING, AS JUDGED BY
THE GRADE, AND BY EDUCATORS?
Duygu Simsek, APL Forum Talk
27
RQ1
What is the overlap
between XIP’s output and
how tutors judge quality?
Tutor Highlighted Student WritingXIP Highlighted Student Writing

28. IN WHAT WAYS SHOULD XIP OUTPUT BE DELIVERED TO END
USERS (STUDENTS AND EDUCATORS)?
Duygu Simsek, APL Forum Talk
28
XIP
Evaluates Accuracy
& Sufficiency
Any correlation
between Grades &
XIP output?
XIP’s Highlights vs.
Marker’s
Output
RQ2

29. IN WHAT WAYS SHOULD XIP OUTPUT BE DELIVERED TO END
USERS (STUDENTS AND EDUCATORS)?
1st Year
Pilot study
Duygu Simsek, APL Forum Talk
29
RQ2
1 min. Intro. Video: http://goo.gl/WGIkDs
5 mins. Demo: http://goo.gl/Km1di5

30. TO WHAT EXTENT DO EDUCATORS VALUE THE RESULTS OF XIP’S
ANALYSIS OF AN INDIVIDUAL STUDENT OR COHORT’S WORK
WHEN THE PRIMARY FOCUS IS ON ASSESSMENT?
Duygu Simsek, APL Forum Talk
30
XIP
Evaluates Accuracy
& Sufficiency
Any correlation
between Grades &
XIP output?
XIP’s Highlights vs.
Marker’s
Output
What educators
think
RQ3

31. TO WHAT EXTENT DO EDUCATORS VALUE THE RESULTS OF XIP’S
ANALYSIS OF AN INDIVIDUAL STUDENT OR COHORT’S WORK
WHEN THE PRIMARY FOCUS IS ON ASSESSMENT?
Duygu Simsek, APL Forum Talk
31
XIP
Evaluates Accuracy
& Sufficiency
Any correlation
between Grades &
XIP output?
XIP’s Highlights vs.
Marker’s
Output
What educators
think
RQ3

32. TO WHAT EXTENT DO STUDENTS VALUE THE RESULTS OF XIP’S
ANALYSIS AS FORMATIVE FEEDBACK ON THEIR WRITING?
Duygu Simsek, APL Forum Talk
32
XIP
Evaluates Accuracy
& Sufficiency
Any correlation
between Grades &
XIP output?
XIP’s Highlights vs.
Marker’s
Output
What educators
think
What students think
RQ4

33. YOUR VIEW ON…
1. Your view on the distilled summary of key features of academic
writing & argumentation.
2. Any graded student writing you have or any OU module that you
can suggest me to contact for data collection?
3. Any feedback on my studies with ALs?
Duygu Simsek, APL Forum Talk
33