1
|
Salmi J, Koistinen OP, Glerean E, Jylänki P, Vehtari A, Jääskeläinen IP, Mäkelä S, Nummenmaa L, Nummi-Kuisma K, Nummi I, Sams M. Distributed neural signatures of natural audiovisual speech and music in the human auditory cortex. Neuroimage 2016; 157:108-117. [PMID: 27932074 DOI: 10.1016/j.neuroimage.2016.12.005] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2016] [Revised: 11/02/2016] [Accepted: 12/03/2016] [Indexed: 11/25/2022] Open
Abstract
During a conversation or when listening to music, auditory and visual information are combined automatically into audiovisual objects. However, it is still poorly understood how specific type of visual information shapes neural processing of sounds in lifelike stimulus environments. Here we applied multi-voxel pattern analysis to investigate how naturally matching visual input modulates supratemporal cortex activity during processing of naturalistic acoustic speech, singing and instrumental music. Bayesian logistic regression classifiers with sparsity-promoting priors were trained to predict whether the stimulus was audiovisual or auditory, and whether it contained piano playing, speech, or singing. The predictive performances of the classifiers were tested by leaving one participant at a time for testing and training the model using the remaining 15 participants. The signature patterns associated with unimodal auditory stimuli encompassed distributed locations mostly in the middle and superior temporal gyrus (STG/MTG). A pattern regression analysis, based on a continuous acoustic model, revealed that activity in some of these MTG and STG areas were associated with acoustic features present in speech and music stimuli. Concurrent visual stimulus modulated activity in bilateral MTG (speech), lateral aspect of right anterior STG (singing), and bilateral parietal opercular cortex (piano). Our results suggest that specific supratemporal brain areas are involved in processing complex natural speech, singing, and piano playing, and other brain areas located in anterior (facial speech) and posterior (music-related hand actions) supratemporal cortex are influenced by related visual information. Those anterior and posterior supratemporal areas have been linked to stimulus identification and sensory-motor integration, respectively.
Collapse
Affiliation(s)
- Juha Salmi
- Department of Neuroscience and Biomedical Engineering (NBE), School of Science, Aalto University, Finland; Advanced Magnetic Imaging (AMI) Centre, School of Science, Aalto University, Finland; Institute of Behavioural Sciences, Division of Cognitive and Neuropsychology, University of Helsinki, Finland
| | - Olli-Pekka Koistinen
- Department of Neuroscience and Biomedical Engineering (NBE), School of Science, Aalto University, Finland
| | - Enrico Glerean
- Department of Neuroscience and Biomedical Engineering (NBE), School of Science, Aalto University, Finland
| | - Pasi Jylänki
- Department of Neuroscience and Biomedical Engineering (NBE), School of Science, Aalto University, Finland
| | - Aki Vehtari
- Department of Neuroscience and Biomedical Engineering (NBE), School of Science, Aalto University, Finland
| | - Iiro P Jääskeläinen
- Department of Neuroscience and Biomedical Engineering (NBE), School of Science, Aalto University, Finland
| | - Sasu Mäkelä
- Department of Neuroscience and Biomedical Engineering (NBE), School of Science, Aalto University, Finland
| | - Lauri Nummenmaa
- Department of Neuroscience and Biomedical Engineering (NBE), School of Science, Aalto University, Finland; Turku PET Centre, University of Turku, Finland
| | | | - Ilari Nummi
- Department of Neuroscience and Biomedical Engineering (NBE), School of Science, Aalto University, Finland
| | - Mikko Sams
- Department of Neuroscience and Biomedical Engineering (NBE), School of Science, Aalto University, Finland.
| |
Collapse
|