The doctoral dissertations of the former Helsinki University of Technology (TKK) and Aalto University Schools of Technology (CHEM, ELEC, ENG, SCI) published in electronic format are available in the electronic publications archive of Aalto University - Aaltodoc.
Aalto

Novel Applications of Real-Time Audiovisual Signal Processing Technology for Art and Sports Education and Entertainment

Perttu Hämäläinen

Dissertation for the degree of Doctor of Science in Technology to be presented with due permission of the Department of Computer Science and Engineering for public examination and debate in Auditorium T2 at Helsinki University of Technology (Espoo, Finland) on the 30th of March, 2007, at 12 noon.

Overview in PDF format (ISBN 978-951-22-8705-5)   [2935 KB]
Dissertation is also available in print (ISBN 978-951-22-8704-8)

Abstract

This thesis explores the possibilities of real-time audiovisual signal processing in the context of human-computer interfaces. Novel embodied interaction applications are presented, including two computer games controlled by body movements and voice, two computer games played by singing, software and an installation for creating clay animation, and interactive video "magic" mirrors for martial arts training. The common denominators for the case studies are 1) the use of cameras and microphones as the input devices, 2) processing and analyzing the audiovisual input to generate visual feedback that can make it easier for the users to detect and correct errors in their performance, and 3) a holistic approach to interaction design, including not only the user and a computer, but the context of interaction as well, e.g., the social setting and the differences between the real and the virtual environments.

In edutainment (education and entertainment) software, the interface sets the limits for the user's actions. The thesis demonstrates how embodied interaction can widen the scope of edutainment to encompass new skills, in this case martial arts, singing, and modelling and animating clay. The thesis also contributes to computer vision technology in the form of practical vision systems that interpret low-level image features according to the application context. For example, in Kick Ass Kung-Fu, an embodied martial arts game, several simultaneous users can fight against computer-generated virtual enemies. The players can wield weapons such as swords, and no specific clothing or markers are needed.

This thesis consists of an overview and of the following 6 publications:

  1. Perttu Hämäläinen and Johanna Höysniemi. 2003. A Computer Vision and Hearing Based User Interface for a Computer Game for Children. In N. Carbonell, C. Stephanidis (Eds.): User Interfaces for All, Lecture Notes in Computer Science 2615, 299-318, Springer-Verlag Berlin Heidelberg. First appeared in Proceedings of the 7th ERCIM International Workshop on User Interfaces for All, Paris, France, October 23-25, 2002.
  2. Perttu Hämäläinen, Mikko Lindholm, Ari Nykänen and Johanna Höysniemi. 2004. Animaatiokone: an Installation for Creating Clay Animation. Proceedings of ACM Conference on Human Factors in Computing Systems (CHI 2004), Vienna, Austria, April 24-29, 2004, 17-24.
  3. Perttu Hämäläinen, Teemu Mäki-Patola, Ville Pulkki and Matti Airas. 2004. Musical Computer Games Played by Singing. Proceedings of the 7th International Conference on Digital Audio Effects (DAFx'04), Naples, Italy, October 5-8, 2004, 367-371.
  4. Perttu Hämäläinen. 2004. Interactive Video Mirrors for Sports Training. Proceedings of the Third Nordic Conference on Human-Computer Interaction, Tampere, Finland, October 23-27, 2004, 199-202.
  5. Perttu Hämäläinen, Johanna Höysniemi, Tommi Ilmonen, Mikko Lindholm and Ari Nykänen. 2005. Martial Arts in Artificial Reality. Proceedings of ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, Oregon, USA, April 2-7, 2005, 781-790.
  6. Johanna Höysniemi and Perttu Hämäläinen. 2006. Children's and Parents' Perception of Full-Body Interaction and Violence in a Martial Arts Game. GAIN: AIGA Journal of Business and Design, March 7, 2006. First appeared in Proceedings of the 2005 Conference on Designing for User eXperience (DUX 2005), San Francisco, CA, USA, November 3-5, 2005.

Keywords: computer vision, interaction design, signal processing, mixed reality, sports, animation, music, computer games

This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.

© 2007 Helsinki University of Technology

Last update 2011-05-26