Using Complex Networks in the Hearing Sciences

Emotional content and semantic structure of dialogues are associated with Interpersonal Neural Synchrony in the Prefrontal Cortex

A fundamental characteristic of social exchanges is the synchronization of individuals' behaviors, physiological responses, and neural activity. However, the association between how individuals communicate in terms of emotional content and expressed associative knowledge and interpersonal synchrony has been scarcely investigated so far. This study addresses this research gap by bridging recent advances in cognitive neuroscience data, affective computing, and cognitive data science frameworks. Using functional near-infrared spectroscopy (fNIRS) hyperscanning, prefrontal neural data were collected during social interactions involving 84 participants (i.e., 42 dyads) aged 18-35 years. Wavelet transform coherence was used to assess interpersonal neural synchrony between participants. We used manual transcription of dialogues and automated methods to codify transcriptions as emotional levels and syntactic/semantic networks. Our quantitative findings reveal higher than random expectations levels of interpersonal neural synchrony in the superior frontal gyrus (q = .038) and the bilateral middle frontal gyri (q< .001, q< .001). Linear mixed models based on dialogues' emotional content only significantly predicted interpersonal neural synchrony across the prefrontal cortex (Rmarginal2=3.62%). Conversely, models relying on syntactic/semantic features were more effective at the local level, for predicting brain synchrony in the right middle frontal gyrus (Rmarginal2=9.97%). Generally, models based on the emotional content of dialogues were not effective when limited to data from one region of interest at a time, whereas models based on syntactic/semantic features show the opposite trend, losing predictive power when incorporating data from all regions of interest. Moreover, we found an interplay between emotions and associative knowledge in predicting brain synchrony, providing quantitative support to the major role played by these linguistic components in social interactions and in prefrontal processes. Our study identifies a mind-brain duality in emotions and associative knowledge reflecting neural synchrony levels, opening new ways for investigating human interactions.

Using network science to examine audio-visual speech perception with a multi-layer graph

To examine visual speech perception (i.e., lip-reading), we created a multi-layer network (the AV-net) that contained: (1) an auditory layer with nodes representing phonological word-forms and edges connecting words that were phonologically related, and (2) a visual layer with nodes representing the viseme representations of words and edges connecting viseme representations that differed by a single viseme (and additional edges to connect related nodes in the two layers). The results of several computer simulations (in which activation diffused across the network to simulate word identification) are reported and compared to the performance of human participants who identified the same words in a condition in which audio and visual information were both presented (Simulation 1), in an audio-only presentation condition (Simulation 2), and a visual-only presentation condition (Simulation 3). Another simulation (Simulation 4) examined the influence of phonological information on visual speech perception by comparing performance in the multi-layer AV-net to a single-layer network that contained only a visual layer with nodes representing the viseme representations of words and edges connecting viseme representations that differed by a single viseme. We also report the results of several analyses of the errors made by human participants in the visual-only presentation condition. The results of our analyses have implications for future research and training of lip-reading, and for the development of automatic lip-reading devices and software for individuals with certain developmental or acquired disorders or for listeners with normal hearing in noisy conditions.