Cerebral Cortex Advance Access originally published online on June 19, 2006
Cerebral Cortex 2007 17(5):1147-1153; doi:10.1093/cercor/bhl024
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Do You See What I Am Saying? Exploring Visual Enhancement of Speech Comprehension in Noisy Environments
1 Program in Cognitive Neuroscience, Department of Psychology, The City College of the City University of New York, New York, NY 10031, USA, 2 The Cognitive Neurophysiology Laboratory, Nathan S. Kline Institute for Psychiatric Research, Program in Cognitive Neuroscience and Schizophrenia, Orangeburg, NY 10962, USA, 3 Program in Neuropsychology, Department of Psychology, Queens College of the City University of New York, Flushing, NY 11367, USA
Address correspondence to John J. Foxe, PhD, Cognitive Neurophysiology Laboratory, Nathan S. Kline Institute for Psychiatric Research, Program in Cognitive Neuroscience and Schizophrenia, 140 Old Orangeburg Road, Orangeburg, NY 10962, USA. Email: foxe{at}nki.rfmh.org.
Viewing a speaker's articulatory movements substantially improves a listener's ability to understand spoken words, especially under noisy environmental conditions. It has been claimed that this gain is most pronounced when auditory input is weakest, an effect that has been related to a well-known principle of multisensory integration—"inverse effectiveness." In keeping with the predictions of this principle, the present study showed substantial gain in multisensory speech enhancement at even the lowest signal-to-noise ratios (SNRs) used (–24 dB), but it was also evident that there was a "special zone" at a more intermediate SNR of –12 dB where multisensory integration was additionally enhanced beyond the predictions of this principle. As such, we show that inverse effectiveness does not strictly apply to the multisensory enhancements seen during audiovisual speech perception. Rather, the gain from viewing visual articulations is maximal at intermediate SNRs, well above the lowest auditory SNR where the recognition of whole words is significantly different from zero. We contend that the multisensory speech system is maximally tuned for SNRs between extremes, where the system relies on either the visual (speech-reading) or the auditory modality alone, forming a window of maximal integration at intermediate SNR levels. At these intermediate levels, the extent of multisensory enhancement of speech recognition is considerable, amounting to more than a 3-fold performance improvement relative to an auditory-alone condition.
Key Words: audiovisual • crossmodal • inverse effectiveness • lip-reading • multisensory • speech perception • speech-reading
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  M. Plaza, P. Gatignol, H. Cohen, B. Berger, and H. Duffau A Discrete Area within the Left Dorsolateral Prefrontal Cortex Involved in Visual-Verbal Incongruence Judgment Cereb Cortex, June 1, 2008; 18(6): 1253 - 1259. [Abstract] [Full Text] [PDF]
|
|