This is the presentation of my mini viva talk given to examiners who assess my PhD's 1st year following the probationary report. It is a summary of my research aims, what I have been doing since the beginning of my 1st year and my plans for the following years of the PhD
2. Research Aim
• To investigate
• Whether or not computational techniques can automatically
identify attributes of good scholarly writing
• What is the potential of these techniques for student essay
analysis?
• How we can best feedback the results of such analysis in a way
that learners can value to improve the quality of their writing.
• How educators can use these results for automatic or semi-
automatic assessment of their students writing.
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
3. Good Scholarly Writing?
Quality of Writing?
• Signalled by the use of
metadiscourse markers in
the text.
• Metadiscourse:
• Linguistic cues in the text
• Expresses a viewpoint, the problem, claim, argument, the
evidence and the implications
• Engages the readers, and signals the writer's stance.
Italicised words are example metadiscourse markers
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
4. Xerox Incremental Parser (XIP)
• Automatic processing of scientific documents
• Recognition of the rhetorically significant sentences
• 8 categories of Rhetorical Moves
• Background Knowledge
• Summarising
• Tendency
• Novelty
• Significance
• Surprise
• Open Question
• Contrasting Ideas
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
6. Original Contribution of PhD
• Carrying XIP into the education field
• For professional scientific articles written by experienced
researchers
• But now for analysis of student essays
• Hypothesis: An outcome of the XIP processed scientific
documents can demonstrate the quality of the author’s written
discourse; and therefore can be used to scaffold and assess
scholarly writing.
• First in depth opportunity to
• Assess a state of the art language technology
• Integrate its services into software tools for academic writing
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
7. Research Questions
• Main Research Question
• How can we support students’ scholarly writing skills to improve the
quality of their writing through automated metadiscourse analysis?
• Sub-Question 1
• How reliable and sufficient are the automated discourse analysis tools
for finding good attributes of scholarly writing within student essays?
• Sub-Question 2
• To what extent is there a relation between the existences of various
kinds of argumentative discourse moves in student essays with final
grades?
• Sub-Question 3
• To what extent automated metadiscourse analysis of discipline-
independent student essays can be used to provide formative feedback?
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
20. • Learning Analytics: Promising
potential of automated, timely
& formative feedback.
• Unresolved Question: “What does
analytics look like for a higher
order skill such as communicating
ideas in scientific writing, with a
primary focus on assessing
students?”
Literature Review Journey
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
21. • Learning Analytics: Promising
potential of automated, timely
& formative feedback.
• Unresolved Question: “What does
analytics look like for a higher
order skill such as communicating
ideas in scientific writing, with a
primary focus on assessing
students?”
Literature Review Journey
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
22. • Learning Analytics: Promising
potential of automated, timely
& formative feedback.
• Unresolved Question: “What does
analytics look like for a higher
order skill such as communicating
ideas in scientific writing, with a
primary focus on assessing
students?”
Literature Review Journey
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
23. • Learning Analytics: Promising
potential of automated, timely
& formative feedback.
• Unresolved Question: “What does
analytics look like for a higher
order skill such as communicating
ideas in scientific writing, with a
primary focus on assessing
students?”
Literature Review Journey
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
24. • Learning Analytics: Promising
potential of automated, timely
& formative feedback.
• Unresolved Question: “What does
analytics look like for a higher
order skill such as communicating
ideas in scientific writing, with a
primary focus on assessing
students?”
Literature Review Journey
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
25. • Learning Analytics: Promising
potential of automated, timely
& formative feedback.
• Unresolved Question: “What does
analytics look like for a higher
order skill such as communicating
ideas in scientific writing, with a
primary focus on assessing
students?”
Literature Review Journey
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
• Academic Writing: Writers
signal argumentative moves by
using well-established patterns
• Debate on whether these
patterns are discipline
independent or not
26. • Learning Analytics: Promising
potential of automated, timely
& formative feedback.
• Unresolved Question: “What does
analytics look like for a higher
order skill such as communicating
ideas in scientific writing, with a
primary focus on assessing
students?”
Literature Review Journey
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
• Academic Writing: Writers
signal argumentative moves by
using well-established patterns
• Debate on whether these
patterns are discipline
independent or not
27. • Learning Analytics: Promising
potential of automated, timely
& formative feedback.
• Unresolved Question: “What does
analytics look like for a higher
order skill such as communicating
ideas in scientific writing, with a
primary focus on assessing
students?”
Literature Review Journey
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
• Academic Writing: Writers
signal argumentative moves by
using well-established patterns
• Debate on whether these
patterns are discipline
independent or not
28. • Learning Analytics: Promising
potential of automated, timely
& formative feedback.
• Unresolved Question: “What does
analytics look like for a higher
order skill such as communicating
ideas in scientific writing, with a
primary focus on assessing
students?”
Literature Review Journey
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
• Academic Writing: Writers
signal argumentative moves by
using well-established patterns
• Debate on whether these
patterns are discipline
independent or not
29. • Learning Analytics: Promising
potential of automated, timely
& formative feedback.
• Unresolved Question: “What does
analytics look like for a higher
order skill such as communicating
ideas in scientific writing, with a
primary focus on assessing
students?”
Literature Review Journey
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
• Academic Writing: Writers
signal argumentative moves by
using well-established patterns
• Debate on whether these
patterns are discipline
independent or not
30. • Learning Analytics: Promising
potential of automated, timely
& formative feedback.
• Unresolved Question: “What does
analytics look like for a higher
order skill such as communicating
ideas in scientific writing, with a
primary focus on assessing
students?”
Literature Review Journey
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
• Academic Writing: Writers
signal argumentative moves by
using well-established patterns
• Debate on whether these
patterns are discipline
independent or not
• Computational Linguistics:
Possible automated analysis of
scientific & technical writing but
barely deployed in educational
context!
• Need: XIP output is not
educator/learner friendly.
31. • Learning Analytics: Promising
potential of automated, timely
& formative feedback.
• Unresolved Question: “What does
analytics look like for a higher
order skill such as communicating
ideas in scientific writing, with a
primary focus on assessing
students?”
Literature Review Journey
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
• Academic Writing: Writers
signal argumentative moves by
using well-established patterns
• Debate on whether these
patterns are discipline
independent or not
• Computational Linguistics:
Possible automated analysis of
scientific & technical writing but
barely deployed in educational
context!
• Need: XIP output is not
educator/learner friendly.
32. • Learning Analytics: Promising
potential of automated, timely
& formative feedback.
• Unresolved Question: “What does
analytics look like for a higher
order skill such as communicating
ideas in scientific writing, with a
primary focus on assessing
students?”
Literature Review Journey
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
• Academic Writing: Writers
signal argumentative moves by
using well-established patterns
• Debate on whether these
patterns are discipline
independent or not
• Computational Linguistics:
Possible automated analysis of
scientific & technical writing but
barely deployed in educational
context!
• Need: XIP output is not
educator/learner friendly.
33. • Learning Analytics: Promising
potential of automated, timely
& formative feedback.
• Unresolved Question: “What does
analytics look like for a higher
order skill such as communicating
ideas in scientific writing, with a
primary focus on assessing
students?”
Literature Review Journey
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
• Academic Writing: Writers
signal argumentative moves by
using well-established patterns
• Debate on whether these
patterns are discipline
independent or not
• Computational Linguistics:
Possible automated analysis of
scientific & technical writing but
barely deployed in educational
context!
• Need: XIP output is not
educator/learner friendly.
34. • Learning Analytics: Promising
potential of automated, timely
& formative feedback.
• Unresolved Question: “What does
analytics look like for a higher
order skill such as communicating
ideas in scientific writing, with a
primary focus on assessing
students?”
Literature Review Journey
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
• Academic Writing: Writers
signal argumentative moves by
using well-established patterns
• Debate on whether these
patterns are discipline
independent or not
• Computational Linguistics:
Possible automated analysis of
scientific & technical writing but
barely deployed in educational
context!
• Need: XIP output is not
educator/learner friendly.
35. • Learning Analytics: Promising
potential of automated, timely
& formative feedback.
• Unresolved Question: “What does
analytics look like for a higher
order skill such as communicating
ideas in scientific writing, with a
primary focus on assessing
students?”
Literature Review Journey
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
• Academic Writing: Writers
signal argumentative moves by
using well-established patterns
• Debate on whether these
patterns are discipline
independent or not
• Computational Linguistics:
Possible automated analysis of
scientific & technical writing but
barely deployed in educational
context!
• Need: XIP output is not
educator/learner friendly.
36. • Learning Analytics: Promising
potential of automated, timely
& formative feedback.
• Unresolved Question: “What does
analytics look like for a higher
order skill such as communicating
ideas in scientific writing, with a
primary focus on assessing
students?”
Literature Review Journey
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
• Academic Writing: Writers
signal argumentative moves by
using well-established patterns
• Debate on whether these
patterns are discipline
independent or not
• Computational Linguistics:
Possible automated analysis of
scientific & technical writing but
barely deployed in educational
context!
• Need: XIP output is not
educator/learner friendly.
• Run XIP on essays from different
disciplines
• Validate XIP in educational
context
• If we can show there is a value
for learners & educators then it
has a potential for formative
assessment of writing.
37. Pilot Work: XIP Dashboard
• Aim: Visualise XIP output
• Question in mind: Can visual XIP analysis of literature help
users to assess current state of the art in terms of trends,
patterns, gaps, and connections?
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
38. Pilot Work: XIP Dashboard
• Aim: Visualise XIP output
• Question in mind: Can visual XIP analysis of literature help
users to assess current state of the art in terms of trends,
patterns, gaps, and connections?
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
Data Set
Proceedings of the Learning Analytics and
Knowledge (LAK) conferences and a journal
special issue, and the Educational Data
Mining (EDM) conferences and journal.
39. Pilot Work: XIP Dashboard
• Aim: Visualise XIP output
• Question in mind: Can visual XIP analysis of literature help
users to assess current state of the art in terms of trends,
patterns, gaps, and connections?
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
Data Set
Proceedings of the Learning Analytics and
Knowledge (LAK) conferences and a journal
special issue, and the Educational Data
Mining (EDM) conferences and journal.
40. Pilot Work: XIP Dashboard
• Aim: Visualise XIP output
• Question in mind: Can visual XIP analysis of literature help
users to assess current state of the art in terms of trends,
patterns, gaps, and connections?
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
XIP Analysis
All of the papers (66 LAK and 239 EDM
papers 305 in total) were analysed
using XIP, extracting 7847 sentences
and 40163 concepts.
Data Set
Proceedings of the Learning Analytics and
Knowledge (LAK) conferences and a journal
special issue, and the Educational Data
Mining (EDM) conferences and journal.
41. Pilot Work: XIP Dashboard
• Aim: Visualise XIP output
• Question in mind: Can visual XIP analysis of literature help
users to assess current state of the art in terms of trends,
patterns, gaps, and connections?
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
XIP Analysis
All of the papers (66 LAK and 239 EDM
papers 305 in total) were analysed
using XIP, extracting 7847 sentences
and 40163 concepts.
Data Set
Proceedings of the Learning Analytics and
Knowledge (LAK) conferences and a journal
special issue, and the Educational Data
Mining (EDM) conferences and journal.
42. Pilot Work: XIP Dashboard
• Aim: Visualise XIP output
• Question in mind: Can visual XIP analysis of literature help
users to assess current state of the art in terms of trends,
patterns, gaps, and connections?
XIP Output: Not learner friendly
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
XIP Analysis
All of the papers (66 LAK and 239 EDM
papers 305 in total) were analysed
using XIP, extracting 7847 sentences
and 40163 concepts.
Data Set
Proceedings of the Learning Analytics and
Knowledge (LAK) conferences and a journal
special issue, and the Educational Data
Mining (EDM) conferences and journal.
43. Pilot Work: XIP Dashboard
• XIP Dashboard is a set of visual analytics modules built on XIP
output.
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
44. Pilot Work: XIP Dashboard
• XIP Dashboard is a set of visual analytics modules built on XIP
output.
Paper Prototype & User Evaluation
6 user sessions were conducted with
1st year PhD students from OU
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
45. Pilot Work: XIP Dashboard
• XIP Dashboard is a set of visual analytics modules built on XIP
output.
Paper Prototype & User Evaluation
6 user sessions were conducted with
1st year PhD students from OU
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
46. Pilot Work: XIP Dashboard
• XIP Dashboard is a set of visual analytics modules built on XIP
output.
Paper Prototype & User Evaluation
6 user sessions were conducted with
1st year PhD students from OU
XIP Dashboard
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
47. Dissemination of Work
• Various poster presentations & talks.
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
48. Dissemination of Work
• Various poster presentations & talks.
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
49. Dissemination of Work
• Various poster presentations & talks.
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
50. Dissemination of Work
• Various poster presentations & talks.
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
51. Dissemination of Work
• Various poster presentations & talks.
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
52. Dissemination of Work
• Various poster presentations & talks.
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
53. Dissemination of Work
• Various poster presentations & talks.
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
54. Dissemination of Work
• Various poster presentations & talks.
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
55. What are the Next Plans?
• Design refinements to the XIP Dashboard
• User evaluations
• XIP as an API, Web Service
• Integrate to software tools, XIP Dashboard
• Test XIP’s power on student essays
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
56. Data Collection & Analysis
What? When? How? Why?
Student Essays
of OU’s S288
S288-12B (2012
Course)
• Analysis of student essays
through XIP
• Comparison of XIP findings
with final grades
• To see whether XIP can identify
important parts of student
essays
• To see whether or not we can
correlate XIP results with final
grades
Student Essays
of OU’s S288
S288-13B (2013
Course)
Same as two above
• Use of Google Docs for
collaboratively written report
where we back up the
revision history & analyse
through XIP
Same as two above
• To see whether XIP can reveal
interesting predictive patterns
about the quality of the end
document and the final grade.
Student Essays
of OU’s S288
S288-14B (2014
Course)
Same as above
• Develop software with XIP
Visual Analytics Dashboard
integrated
• Get users’ reactions
Same as above
• Analyses student essays &
provide real-time analytics of
students’ essays as a feedback to
students.
Student Essays
(soft domains)
N/A Same as above Same as above
• Test the discipline independency
of XIP
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
57. Data Collection & Analysis
What? When? How? Why?
Student Essays
of OU’s S288
S288-12B (2012
Course)
• Analysis of student essays
through XIP
• Comparison of XIP findings
with final grades
• To see whether XIP can identify
important parts of student
essays
• To see whether or not we can
correlate XIP results with final
grades
Student Essays
of OU’s S288
S288-13B (2013
Course)
Same as two above
• Use of Google Docs for
collaboratively written report
where we back up the
revision history & analyse
through XIP
Same as two above
• To see whether XIP can reveal
interesting predictive patterns
about the quality of the end
document and the final grade.
Student Essays
of OU’s S288
S288-14B (2014
Course)
Same as above
• Develop software with XIP
Visual Analytics Dashboard
integrated
• Get users’ reactions
Same as above
• Analyses student essays &
provide real-time analytics of
students’ essays as a feedback to
students.
Student Essays
(soft domains)
N/A Same as above Same as above
• Test the discipline independency
of XIP
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
RQ1: How reliable and
sufficient are the automated
discourse analysis tools for
finding good attributes of
scholarly writing within
student essays?
58. Data Collection & Analysis
What? When? How? Why?
Student Essays
of OU’s S288
S288-12B (2012
Course)
• Analysis of student essays
through XIP
• Comparison of XIP findings
with final grades
• To see whether XIP can identify
important parts of student
essays
• To see whether or not we can
correlate XIP results with final
grades
Student Essays
of OU’s S288
S288-13B (2013
Course)
Same as two above
• Use of Google Docs for
collaboratively written report
where we back up the
revision history & analyse
through XIP
Same as two above
• To see whether XIP can reveal
interesting predictive patterns
about the quality of the end
document and the final grade.
Student Essays
of OU’s S288
S288-14B (2014
Course)
Same as above
• Develop software with XIP
Visual Analytics Dashboard
integrated
• Get users’ reactions
Same as above
• Analyses student essays &
provide real-time analytics of
students’ essays as a feedback to
students.
Student Essays
(soft domains)
N/A Same as above Same as above
• Test the discipline independency
of XIP
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
RQ1: How reliable and
sufficient are the automated
discourse analysis tools for
finding good attributes of
scholarly writing within
student essays?
RQ2: To what extent is there
a relation between the
existences of various kinds
of argumentative discourse
moves in student essays
with final grades?
59. Data Collection & Analysis
What? When? How? Why?
Student Essays
of OU’s S288
S288-12B (2012
Course)
• Analysis of student essays
through XIP
• Comparison of XIP findings
with final grades
• To see whether XIP can identify
important parts of student
essays
• To see whether or not we can
correlate XIP results with final
grades
Student Essays
of OU’s S288
S288-13B (2013
Course)
Same as two above
• Use of Google Docs for
collaboratively written report
where we back up the
revision history & analyse
through XIP
Same as two above
• To see whether XIP can reveal
interesting predictive patterns
about the quality of the end
document and the final grade.
Student Essays
of OU’s S288
S288-14B (2014
Course)
Same as above
• Develop software with XIP
Visual Analytics Dashboard
integrated
• Get users’ reactions
Same as above
• Analyses student essays &
provide real-time analytics of
students’ essays as a feedback to
students.
Student Essays
(soft domains)
N/A Same as above Same as above
• Test the discipline independency
of XIP
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
RQ1: How reliable and
sufficient are the automated
discourse analysis tools for
finding good attributes of
scholarly writing within
student essays?
RQ2: To what extent is there
a relation between the
existences of various kinds
of argumentative discourse
moves in student essays
with final grades?
RQ3: To what extent
automated metadiscourse
analysis of discipline-
independent student
essays can be used to
provide formative
feedback?
60. Data Collection & Analysis
What? When? How? Why?
Student Essays
of OU’s S288
S288-12B (2012
Course)
• Analysis of student essays
through XIP
• Comparison of XIP findings
with final grades
• To see whether XIP can identify
important parts of student
essays
• To see whether or not we can
correlate XIP results with final
grades
Student Essays
of OU’s S288
S288-13B (2013
Course)
Same as two above
• Use of Google Docs for
collaboratively written report
where we back up the
revision history & analyse
through XIP
Same as two above
• To see whether XIP can reveal
interesting predictive patterns
about the quality of the end
document and the final grade.
Student Essays
of OU’s S288
S288-14B (2014
Course)
Same as above
• Develop software with XIP
Visual Analytics Dashboard
integrated
• Get users’ reactions
Same as above
• Analyses student essays &
provide real-time analytics of
students’ essays as a feedback to
students.
Student Essays
(soft domains)
N/A Same as above Same as above
• Test the discipline independency
of XIP
10/09/2013,Tuesday,TheOpen
University
MiniVivaPresentation
RQ1: How reliable and
sufficient are the automated
discourse analysis tools for
finding good attributes of
scholarly writing within
student essays?
RQ2: To what extent is there
a relation between the
existences of various kinds
of argumentative discourse
moves in student essays
with final grades?
RQ3: To what extent
automated metadiscourse
analysis of discipline-
independent student
essays can be used to
provide formative
feedback?
How can we support
students’ scholarly
writing skills to
improve the quality
of their writing
through automated
metadiscourse
analysis?