The doctoral dissertations of the former Helsinki University of Technology (TKK) and Aalto University Schools of Technology (CHEM, ELEC, ENG, SCI) published in electronic format are available in the electronic publications archive of Aalto University - Aaltodoc.

Cortical Mechanisms of Seeing and Hearing Speech

Riikka Möttönen

Dissertation for the degree of Doctor of Philosophy to be presented with due permission of the Department of Electrical and Communications Engineering for public examination and debate in Auditorium S1 at Helsinki University of Technology (Espoo, Finland) on the 10th of December, 2004, at 12 o'clock noon.

Overview in PDF format (ISBN 951-22-7427-2)   [2489 KB]
Dissertation is also available in print (ISBN 951-22-7426-4)


In face-to-face communication speech is perceived through eyes and ears. The talker's articulatory gestures are seen and the speech sounds are heard simultaneously. Whilst acoustic speech can be often understood without visual information, viewing articulatory gestures aids hearing substantially in noisy conditions. On the other hand, speech can be understood, to some extent, by solely viewing articulatory gestures (i.e., by speechreading).

In this thesis, electroencephalography (EEG), magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI) were utilized to disclose cortical mechanisms of seeing and hearing speech.

One of the major challenges of modern cognitive neuroscience is to find out how the brain integrates inputs from different senses. In this thesis, integration of seen and heard speech was investigated using EEG and MEG. Multisensory interactions were found in the sensory-specific cortices at early latencies and in the multisensory regions at late latencies.

Viewing other person's actions activate regions belonging to the human mirror neuron system (MNS) which are also activated when subjects themselves perform actions. Possibly, the human MNS enables simulation of other person's actions, which might be important also for speech recognition. In this thesis, it was demonstrated with MEG that seeing speech modulates activity in the mouth region of the primary somatosensory cortex (SI), suggesting that also the SI cortex is involved in simulation of other person's articulatory gestures during speechreading.

The question whether there are speech-specific mechanisms in the human brain has been under scientific debate for decades. In this thesis, evidence for the speech-specific neural substrate in the left posterior superior temporal sulcus (STS) was obtained using fMRI. Activity in this region was found to be greater when subjects heard acoustic sine wave speech stimuli as speech than when they heard the same stimuli as non-speech.

This thesis consists of an overview and of the following 5 publications:

  1. Möttönen, R., Krause, C. M., Tiippana, K., and Sams, M., 2002. Processing of changes in visual speech in the human auditory cortex. Cognitive Brain Research 13, number 3, pages 417-425.
  2. Klucharev, V., Möttönen, R., and Sams, M., 2003. Electrophysiological indicators of phonetic and non-phonetic multisensory interactions during audiovisual speech perception. Cognitive Brain Research 18, number 1, pages 65-75.
  3. Möttönen, R., Schürmann, M., and Sams, M., 2004. Time course of multisensory interactions during audiovisual speech perception in humans: a magnetoencephalographic study. Neuroscience Letters 363, number 2, pages 112-115.
  4. Möttönen, R., Järveläinen, J., Sams, M., and Hari, R., Viewing speech modulates activity in the left SI mouth cortex. NeuroImage, in press.
  5. Möttönen, R., Calvert, G. A., Jääskeläinen, I. P., Matthews, P. M., Thesen, T., Tuomainen, J., and Sams, M., 2004. Perception of identical acoustic stimuli as speech or non-speech modifies activity in left posterior superior temporal sulcus. Helsinki University of Technology, Laboratory of Computational Engineering publications, Technical Report B42.

Keywords: auditory cortex, electroencephalography, functional magnetic resonance imaging, magnetoencephalography, multisensory, speech, SI, somatosensory cortex, superior temporal sulcus

This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.

© 2004 Helsinki University of Technology

Last update 2011-05-26