Slides from my Koli Calling 2010 presentation: Petri Ihantola, Tuukka Ahoniemi, Ville Karavirta, Otto Seppälä (2010). Review of recent systems for automatic assessment of programming assignments. In: Koli Calling ’10: Proceedings of the 10th Koli Calling International Conference on Computing Education Research. Koli, Finland: ACM, pp. 86–93. ISBN: 978-1-4503-0520-4. http://dx.doi.org/10.1145/1930464.1930480

1. Back to programming teaching tools…
Petri Ihantola, Koli Calling 2010

3. Surveyed Before
e.g. a survey from 2005 by Kirst Ala-Mutka
describes what features of programming
assignments are automatcally assessed.
Dynamic analysis is ofen used to assess
functonality, efciency, and testng skills.
Statc checks are used to provide feedback
from style, programming errors, sofware
metrics, and even design. In additon, there
are special features like plagiarism detecton.
Petri Ihantola, Koli Calling 2010

4. Surveyed Before
e.g. a survey from 2005 by Kirst Ala-Mutka
describes what features of programming
assignments are automatcally assessed.
Dynamic analysis is ofen used to assess
functonality, efciency, and testng skills.
Statc checks are used to provide feedback
from style, programming errors, sofware
metrics, and even design. In additon, there
are special features like plagiarism detecton.
Petri Ihantola, Koli Calling 2010

5. Surveyed Before
e.g. a survey from 2005 by Kirst Ala-Mutka
describes what features of programming
assignments are automatcally assessed.
Dynamic analysis is ofen used to assess
functonality, efciency, and testng skills.
Statc checks are used to provide feedback
from style, programming errors, sofware
metrics, and even design. In additon, there
are special features like plagiarism detecton.
Petri Ihantola, Koli Calling 2010

6. Surveyed Before
e.g. a survey from 2005 by Kirst Ala-Mutka
describes what features of programming
assignments are automatcally assessed.
Dynamic analysis is ofen used to assess
functonality, efciency, and testng skills.
Statc checks are used to provide feedback
from style, programming errors, sofware
metrics, and even design. In additon, there
are special features like plagiarism detecton.
Petri Ihantola, Koli Calling 2010

7. Surveyed Before
e.g. a survey from 2005 by Kirst Ala-Mutka
describes what features of programming
assignments are automatcally assessed.
Dynamic analysis is ofen used to assess
functonality, efciency, and testng skills.
Statc checks are used to provide feedback
from style, programming errors, sofware
metrics, and even design. In additon, there
are special features like plagiarism detecton.
Petri Ihantola, Koli Calling 2010

8. Surveyed Before
e.g. a survey from 2005 by Kirst Ala-Mutka
describes what features of programming
assignments are automatcally assessed.
Dynamic analysis is ofen used to assess
functonality, efciency, and testng skills.
Statc checks are used to provide feedback
from style, programming errors, sofware
metrics, and even design. In additon, there
are special features like plagiarism detecton.
Petri Ihantola, Koli Calling 2010

9. Surveyed Before
e.g. a survey from 2005 by Kirst Ala-Mutka
describes what features of programming
assignments are automatcally assessed.
Dynamic analysis is ofen used to assess
functonality, efciency, and testng skills.
Statc checks are used to provide feedback
from style, programming errors, sofware
metrics, and even design. In additon, there
are special features like plagiarism detecton.
Petri Ihantola, Koli Calling 2010

10. Surveyed Before
e.g. a survey from 2005 by Kirst Ala-Mutka
describes what features of programming
assignments are automatcally assessed.
Dynamic analysis is ofen used to assess
functonality, efciency, and testng skills.
Statc checks are used to provide feedback
from style, programming errors, sofware
metrics, and even design. In additon, there
are special features like plagiarism detecton.
Petri Ihantola, Koli Calling 2010

11. Our Method
Systematc literature review:
We collected the data by searching for phrases
('automatc' OR 'automated) AND ('assessment' OR
'grading’) AND 'programming' from the conference
proceedings and journals through ACM Digital Library
and IEEE Xplore published in between 2006 and 2010.
We then applied the inclusion criteria to the abstracts
(a paper must have presented an system providing
summatve, numerical, automatc feedback from
programming assignments or described results from
using such system.) and fnally read through the
remaining papers to do qualitatve data analysis.
Petri Ihantola, Koli Calling 2010

12. Our Method
Systematc literature review:
We collected the data by searching for phrases
('automatc' OR 'automated) AND ('assessment' OR
'grading’) AND 'programming' from the conference
proceedings and journals through ACM Digital Library
and IEEE Xplore published in between 2006 and 2010.
We then applied the inclusion criteria to the abstracts
(a paper must have presented an system providing
summatve, numerical, automatc feedback from
programming assignments or described results from
using such system.) and fnally read through the
remaining papers to do qualitatve data analysis.
Petri Ihantola, Koli Calling 2010

13. Our Method
Systematc literature review:
We collected the data by searching for phrases
('automatc' OR 'automated) AND ('assessment' OR
'grading’) AND 'programming' from the conference
proceedings and journals through ACM Digital Library
and IEEE Xplore published in between 2006 and 2010.
We then applied the inclusion criteria to the abstracts
(a paper must have presented an system providing
summatve, numerical, automatc feedback from
programming assignments or described results from
using such system.) and fnally read through the
remaining papers to do qualitatve data analysis.
Petri Ihantola, Koli Calling 2010

14. Our Method
Systematc literature review:
We collected the data by searching for phrases
('automatc' OR 'automated) AND ('assessment' OR
'grading’) AND 'programming' from the conference
proceedings and journals through ACM Digital Library
and IEEE Xplore published in between 2006 and 2010.
We then applied the inclusion criteria to the abstracts
(a paper must have presented an system providing
summatve, numerical, automatc feedback from
programming assignments or described results from
using such system.) and fnally read through the
remaining papers to do qualitatve data analysis.
Petri Ihantola, Koli Calling 2010

16. Languages
Petri Ihantola, Koli Calling 2010

17. LanguagesJava, Python, Pascal, C/C++,
assembler, shell scripts,
don’t care, ….
Petri Ihantola, Koli Calling 2010

36. Summary
• Programming languages
• Integraton into LMSs (e.g. Moodle)
• Diferent ways to defne tests
– Xunit, acceptance testng, web testng frameworks
– Something not widely applied in industry
• Manual assessment
• Resubmissions
– Limitng tme, feedback
– Customizaton
– Interestng hybrid
• Sandboxing
• Availability
• Specialtes
– GUIs, data bases, concurrency, web programming, students’ own testng skills
Petri Ihantola, Koli Calling 2010

38. Thanks to the authors of the images I used:
• htp://www.fickr.com/photos/smanography/3073414449
• htp://www.fickr.com/photos/stephenpof/2920198159/
• htp://www.fickr.com/photos/kaptainkobold/3203311346/
• htp://www.fickr.com/photos/superk8/3868056064/
• htp://www.fickr.com/photos/naomi_pincher/2691787431/
• htp://www.fickr.com/photos/secondtoughest/347049865/
• htp://www.fickr.com/photos/terrio/4920670944/
• htp://www.fickr.com/photos/pranavsingh/1946245078
• htp://www.fickr.com/photos/arteurbana/496777749/
• htp://www.fickr.com/photos/litle_lushie/28955776/
• htp://www.fickr.com/photos/12392252@N03/2037540230/
• htp://www.fickr.com/photos/markjsebastan/324425852
• htp://www.fickr.com/photos/benheine/5044969981/
• htp://www.fickr.com/photos/piblet/1053194618/
• htp://www.fickr.com/photos/ogil/1507585665/
• htp://www.fickr.com/photos/brandoncripps/2308741771/
• htp://www.fickr.com/photos/clydeorama/2442172268/
• htp://www.fickr.com/photos/imutoo/2631466945/
• htp://www.fickr.com/photos/beatkueng/1350250361/
• htp://www.fickr.com/photos/pagedooley/3227848591/
• htp://www.fickr.com/photos/levork/4509401065/
All these images are licensed under the creatve commons.
Petri Ihantola, Koli Calling 2010