Researchers at the Del Monte Institute for Neuroscience at the University of Rochester have expanded the understanding of how the brain is engaged during complex audiovisual speech perception. The study now out in NeuroImage, describes how listening and watching a narrator tell a story activates an extensive network of brain regions involved in sensory processing, multisensory integration, and cognitive functions associated with the comprehension of the story content. Understanding the involvement of this larger network has the potential to give researchers new ways to investigate neurodevelopmental disorders.

“Multisensory integration is an important function of our nervous system as it can substantially enhance our ability to detect and identify objects in our environment,” said Lars Ross, Ph.D., research assistant professor of Imaging Sciences and Neuroscience and first author of the study. “A failure of this function may lead to a sensory environment that is perceived as overwhelming and can cause a person to have difficulty adapting to their surroundings, a problem we believe underlies symptoms of some neurodevelopmental disorders such as autism.”

Using fMRI, researchers examined the brain activity of 53 participants as they watched a video recording of a speaker reading “The Lorax.” How the story was presented would change randomly in one of four ways – audio only, visual only, synchronized audiovisual, or unsynchronized audiovisual. Researchers also monitored the participants’ eye movements. They found that along with the previously identified sites of multisensory integration, viewing the speaker’s facial movements also enhanced brain activity in the broader semantic network and extralinguistic regions not usually associated with multisensory integration, such as the amygdala and primary visual cortex. Researchers also found activity in thalamic brain regions which are known to be very early stages at which sensory information from our eyes and ears interact.

“This suggests many regions beyond multisensory integration play a role in how the brain processes complex multisensory speech – including those associated with extralinguistic perceptual and cognitive processing,” said Ross.

Researchers designed this experiment with children in mind, according to the investigators who have already begun working with both children and adults on the autism spectrum in an effort to gain insight into how their ability to process audiovisual speech develops over time.

“Our lab is profoundly interested in this network because it goes awry in a number of neurodevelopmental disorders,” said John Foxe, Ph.D., lead author of this study. “Now that we have designed this detailed map of the multisensory speech integration network, we can ask much more pointed questions about multisensory speech in neurodevelopmental disorders, like autism and dyslexia, and get at the specific brain circuits that might be impacted.”