The doctoral dissertations of the former Helsinki University of Technology (TKK) and Aalto University Schools of Technology (CHEM, ELEC, ENG, SCI) published in electronic format are available in the electronic publications archive of Aalto University - Aaltodoc.
Aalto

Neurocognitive Mechanisms of Audiovisual Speech Perception

Ville Ojanen

Dissertation for the degree of Doctor of Philosophy to be presented with due permission of the Department of Electrical and Communications Engineering for public examination and debate in Auditorium S2 at Helsinki University of Technology (Espoo, Finland) on the 25th of May, 2005, at 12 noon.

Overview in PDF format (ISBN 951-22-7681-X)   [1132 KB]
Dissertation is also available in print (ISBN 951-22-7680-1)

Abstract

Face-to-face communication involves both hearing and seeing speech. Heard and seen speech inputs interact during audiovisual speech perception. Specifically, seeing the speaker's mouth and lip movements improves identification of acoustic speech stimuli, especially in noisy conditions. In addition, visual speech may even change the auditory percept. This occurs when mismatching auditory speech is dubbed onto visual articulation.

Research on the brain mechanisms of audiovisual perception aims at revealing where, when and how inputs from different modalities interact. In this thesis, functional magnetic resonance imaging (fMRI), magnetoencephalography (MEG) and behavioral methods were used to study the neurocognitive mechanisms of audiovisual speech perception.

The results suggest that interactions during audiovisual and visual speech perception have an effect on auditory speech processing at early levels of processing hierarchy. The results also suggest that auditory and visual speech inputs interact in the motor cortical areas involved in speech production. Some of these regions are part of the "mirror neuron system" (MNS). The MNS performs a specialized primate cerebral function of coupling two fundamental processes – motor action execution and perception – together. It is suggested that this action-perception coupling mechanism might be involved in audiovisual integration of speech.

This thesis consists of an overview and of the following 5 publications:

  1. Ojanen, V., Tuomainen, J., and Sams, M., 2003. Effect of audiovisual primes on identification of auditory target syllables. In: Schwartz, J. L., Berthommier, F., Cathiard, M. A., and Sodoyer, D. (eds.), Proceedings of the International Conference on Audio-Visual Speech Processing (AVSP 2003). St. Jorioz, France, 4-7 September 2003, pages 71-75.
  2. Jääskeläinen, I. P., Ojanen, V., Ahveninen, J., Auranen, T., Levänen, S., Möttönen, R., Tarnanen, I., and Sams, M, 2004. Adaptation of neuromagnetic N1 responses to phonetic stimuli by visual speech in humans. NeuroReport 15 (18), pages 2741-2744.
  3. Pekkola, J., Ojanen, V., Autti, T., Jääskeläinen, I. P., Möttönen, R., Tarkiainen, A., and Sams, M., 2005. Primary auditory cortex activation by visual speech: an fMRI study at 3 T. NeuroReport 16 (2), pages 125-128.
  4. Ojanen, V., Möttönen, R., Pekkola, J., Jääskeläinen, I. P., Joensuu, R., Autti, T., and Sams, M., 2005. Processing of audiovisual speech in Broca's area. NeuroImage 25 (2), pages 333-338.
  5. Ojanen, V., Pekkola, J., Jääskeläinen, I. P., Möttönen, R., Autti, T., Jousmäki, V., and Sams, M., 2005. Common brain areas activated by hearing and seeing speech. Helsinki University of Technology, Laboratory of Computational Engineering publications, Technical Report B49. ISBN 951-22-7679-8.

Keywords: auditory cortex, functional magnetic resonance imaging, magnetoencephalography, multisensory, audiovisual speech perception, lipreading, Broca, motor cortex, superior temporal sulcus

This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.

© 2005 Helsinki University of Technology

Last update 2011-05-26