The doctoral dissertations of the former Helsinki University of Technology (TKK) and Aalto University Schools of Technology (CHEM, ELEC, ENG, SCI) published in electronic format are available in the electronic publications archive of Aalto University - Aaltodoc.
Aalto

Automated Assessment of Programming Assignments: Visual Feedback, Assignment Mobility, and Assessment of Students' Testing Skills

Petri Ihantola (né Tenhunen)

Doctoral dissertation for the degree of Doctor of Science in Technology to be presented with due permission of the School of Science for public examination and debate in Auditorium T2 at the Aalto University School of Science (Espoo, Finland) on the 9th of December 2011 at 12 noon.

Overview in PDF format (ISBN 978-952-60-4398-2)   [1771 KB]
Dissertation is also available in print (ISBN 978-952-60-4397-5)

Abstract

The main objective of this thesis is to improve the automated assessment of programming assignments from the perspective of assessment tool developers.

We have developed visual feedback on functionality of students' programs and explored methods to control the level of detail in visual feedback. We have found that visual feedback does not require major changes to existing assessment platforms. Most modern platforms are web based, creating an opportunity to describe visualizations in JavaScript and HTML embedded into textual feedback. Our preliminary results on the effectiveness of automatic visual feedback indicate that students perform equally well with visual and textual feedback. However, visual feedback based on automatically extracted object graphs can take less time to prepare than textual feedback of good quality.

We have also developed programming assignments that are easier to port from one server environment to another by performing assessment on the client-side. This not only makes it easier to use the same assignments in different server environments but also removes the need for sandboxing the execution of students' programs. The approach will likely become more important in the future together with interactive study materials becoming more popular. Client-side assessment is more suitable for self-studying material than for grading because assessment results sent by a client are often too easy to falsify.

Testing is an important part of programming and automated assessment should also cover students' self-written tests. We have analyzed how students behave when they are rewarded for structural test coverage (e.g. line coverage) and found that this can lead students to write tests with good coverage but with poor ability to detect faulty programs. Mutation analysis, where a large number of (faulty) programs are automatically derived from the program under test, turns out to be an effective way to detect tests otherwise fooling our assessment systems. Applying mutation analysis directly for grading is problematic because some of the derived programs are equivalent with the original and some assignments or solution strategies generate more equivalent mutants than others.

This thesis consists of an overview and of the following 9 publications:

  1. Ari Korhonen, Lauri Malmi, Jussi Nikander, and Petri Tenhunen. Interaction and feedback in automatically assessed algorithm simulation exercises. Journal of Information Technology Education, pages 241-255, vol 2, 2003.
  2. Petri Ihantola, Tuukka Ahoniemi, Ville Karavirta, and Otto Seppälä. Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research (Koli Calling '10), ACM, New York, NY, USA, pages 86-93, 2010.
  3. Petri Ihantola. Creating and visualizing test data from programming exercises. Informatics in education 6(1), pages 81-102, January 2007.
  4. Petri Ihantola, Ville Karavirta and Otto Seppälä. Automated visual feedback from programming assignments. In Proceedings of the 6th Program Visualization Workshop (PVW '11), Technische Universität Darmstadt, Darmstadt, pages 87-95, 2011. © 2011 by authors.
  5. Petri Ihantola, Ville Karavirta, Ari Korhonen, and Jussi Nikander. Taxonomy of effortless creation of algorithm visualizations. Proceedings of the 2005 International workshop on Computing Education Research (ICER '05), ACM, New York, NY, USA, pages 123-133, 2005.
  6. Guido Rößling, Myles McNally, Pierluigi Crescenzi, Atanas Radenski, Petri Ihantola, and M. Gloria Sánchez-Torrubia. Adapting moodle to better support CS education. In Proceedings of the 2010 ITiCSE working group reports (ITiCSE-WGR '10), ACM, New York, NY, USA, pages 15-27, 2010.
  7. Petri Ihantola and Ville Karavirta. Two-dimensional Parson's puzzles: the concept, tools, and first observations. Journal of Information Technology Education: Innovations in Practice, vol. 10, pages 119-132, 2011.
  8. Ville Karavirta and Petri Ihantola. Automatic assessment of JavaScript exercises. In Proceedings of 1st Educators' Day on Web Engineering Curricula (WECU 2010), Volume 607 of CEUR-WS, Vienna, Austria, pages P9:1-P9:10, 2010. © 2010 by authors.
  9. Kalle Aaltonen, Petri Ihantola, and Otto Seppälä. Mutation analysis vs. code coverage in automated assessment of students' testing skills. Proceedings of the ACM international conference companion on Object oriented programming systems languages and applications companion (SPLASH '10), ACM, New York, NY, USA, 153-160, 2010.

Errata of publication 3

Keywords: teaching programming, automated assessment, visual feedback, software visualization, portability, mobility, testing, mutation analysis

This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.

© 2011 Aalto University

Last update 2012-07-25