Audiovisual Processing Skills Before Cochlear Implantation Predict Postoperative Speech Recognition in Adults

OBJECTIVES

Adults with hearing loss (HL) demonstrate greater benefits of adding visual cues to auditory cues (i.e., "visual enhancement" [VE]) during recognition of speech presented in a combined audiovisual (AV) fashion when compared with normal-hearing peers. For patients with moderate-to-profound sensorineural HL who receive cochlear implants (CIs), it is unclear whether the restoration of audibility results in a decrease in the VE provided by visual cues during AV speech recognition. Moreover, it is unclear whether increased VE during the experience of HL before CI is beneficial or maladaptive to ultimate speech recognition abilities after implantation. It is conceivable that greater VE before implantation contributes to the enormous variability in speech recognition outcomes demonstrated among patients with CIs. This study took a longitudinal approach to test two hypotheses: (H1) Adult listeners with HL who receive CIs would demonstrate a decrease in VE after implantation; and (H2) The magnitude of pre-CI VE would predict post-CI auditory-only speech recognition abilities 6 months after implantation, with the direction of that relation supporting a beneficial, redundant, or maladaptive effect on outcomes.

DESIGN

Data were collected from 30 adults at two time points: immediately before CI surgery and 6 months after device activation. Pre-CI speech recognition performance was measured in auditory-only (A-only), visual-only, and combined AV fashion for City University of New York (CUNY) sentences. Scores of VE during AV sentence recognition were computed. At 6 months after CI activation, participants were again tested on CUNY sentence recognition in the same conditions as pre-CI. H1 was tested by comparing post- versus pre-CI VE scores. At 6 months of CI use, additional open-set speech recognition measures were also obtained in the A-only condition, including isolated words, words in meaningful AzBio sentences, and words in AzBio sentences in multitalker babble. To test H2, correlation analyses were performed to assess the relation between post-CI A-only speech recognition scores and pre-CI VE scores.

RESULTS

Inconsistent with H1, after CI, participants did not demonstrate a significant decrease in VE scores. Consistent with H2, preoperative VE scores positively predicted postoperative scores of A-only sentence recognition for both sentences in quiet and in babble (rho = 0.40 to 0.45, p < 0.05), supporting a beneficial effect of pre-CI VE on post-CI auditory outcomes. Pre-CI VE was not significantly related to post-CI isolated word recognition. The raw pre-CI CUNY AV scores also predicted post-CI A-only speech recognition scores to a similar degree as VE scores.

CONCLUSIONS

After implantation, CI users do not demonstrate a decrease in VE from before surgery. The degree of VE during AV speech recognition before CI positively predicts A-only sentence recognition outcomes after implantation, suggesting the potential value of AV testing of CI patients preoperatively to help predict and set expectations for postoperative outcomes.

Interpretable Deep Learning System for Identifying Critical Patients Through the Prediction of Triage Level, Hospitalization, and Length of Stay: Prospective Study

BACKGROUND

Triage is the process of accurately assessing patients' symptoms and providing them with proper clinical treatment in the emergency department (ED). While many countries have developed their triage process to stratify patients' clinical severity and thus distribute medical resources, there are still some limitations of the current triage process. Since the triage level is mainly identified by experienced nurses based on a mix of subjective and objective criteria, mis-triage often occurs in the ED. It can not only cause adverse effects on patients, but also impose an undue burden on the health care delivery system.

OBJECTIVE

Our study aimed to design a prediction system based on triage information, including demographics, vital signs, and chief complaints. The proposed system can not only handle heterogeneous data, including tabular data and free-text data, but also provide interpretability for better acceptance by the ED staff in the hospital.

METHODS

In this study, we proposed a system comprising 3 subsystems, with each of them handling a single task, including triage level prediction, hospitalization prediction, and length of stay prediction. We used a large amount of retrospective data to pretrain the model, and then, we fine-tuned the model on a prospective data set with a golden label. The proposed deep learning framework was built with TabNet and MacBERT (Chinese version of bidirectional encoder representations from transformers [BERT]).

RESULTS

The performance of our proposed model was evaluated on data collected from the National Taiwan University Hospital (901 patients were included). The model achieved promising results on the collected data set, with accuracy values of 63%, 82%, and 71% for triage level prediction, hospitalization prediction, and length of stay prediction, respectively.

CONCLUSIONS

Our system improved the prediction of 3 different medical outcomes when compared with other machine learning methods. With the pretrained vital sign encoder and repretrained mask language modeling MacBERT encoder, our multimodality model can provide a deeper insight into the characteristics of electronic health records. Additionally, by providing interpretability, we believe that the proposed system can assist nursing staff and physicians in taking appropriate medical decisions.

Early or synchronized gestures facilitate speech recall-a study based on motion capture data

Introduction

Temporal co-ordination between speech and gestures has been thoroughly studied in natural production. In most cases gesture strokes precede or coincide with the stressed syllable in words that they are semantically associated with.

Methods

To understand whether processing of speech and gestures is attuned to such temporal coordination, we investigated the effect of delaying, preposing or eliminating individual gestures on the memory for words in an experimental study in which 83 participants watched video sequences of naturalistic 3D-animated speakers generated based on motion capture data. A target word in the sequence appeared (a) with a gesture presented in its original position synchronized with speech, (b) temporally shifted 500 ms before or (c) after the original position, or (d) with the gesture eliminated. Participants were asked to retell the videos in a free recall task. The strength of recall was operationalized as the inclusion of the target word in the free recall.

Results

Both eliminated and delayed gesture strokes resulted in reduced recall rates compared to synchronized strokes, whereas there was no difference between advanced (preposed) and synchronized strokes. An item-level analysis also showed that the greater the interval between the onsets of delayed strokes and stressed syllables in target words, the greater the negative effect was on recall.

Discussion

These results indicate that speech-gesture synchrony affects memory for speech, and that temporal patterns that are common in production lead to the best recall. Importantly, the study also showcases a procedure for using motion capture-based 3D-animated speakers to create an experimental paradigm for the study of speech-gesture comprehension.

Combining electrodermal activity analysis and dynamic causal modeling to investigate the visual-odor multimodal integration during face perception

Objective. This study presents a novel methodological approach for incorporating information related to the peripheral sympathetic response into the investigation of neural dynamics. Particularly, we explore how hedonic contextual olfactory stimuli influence the processing of neutral faces in terms of sympathetic response, event-related potentials and effective connectivity analysis. The objective is to investigate how the emotional valence of odors influences the cortical connectivity underlying face processing and the role of face-induced sympathetic arousal in this visual-olfactory multimodal integration.Approach. To this aim, we combine electrodermal activity (EDA) analysis and dynamic causal modeling to examine changes in cortico-cortical interactions.Results. The results reveal that stimuli arising sympathetic EDA responses are associated with a more negative N170 amplitude, which may be a marker of heightened arousal in response to faces. Hedonic odors, on the other hand, lead to a more negative N1 component and a reduced the vertex positive potential when they are unpleasant or pleasant. Concerning connectivity, unpleasant odors strengthen the forward connection from the inferior temporal gyrus (ITG) to the middle temporal gyrus, which is involved in processing changeable facial features. Conversely, the occurrence of sympathetic responses after a stimulus is correlated with an inhibition of this same connection and an enhancement of the backward connection from ITG to the fusiform face gyrus.Significance. These findings suggest that unpleasant odors may enhance the interpretation of emotional expressions and mental states, while faces capable of eliciting sympathetic arousal prioritize identity processing.

Mechanism-aware and multimodal AI: beyond model-agnostic interpretation

Artificial intelligence (AI) is widely used for exploiting multimodal biomedical data, with increasingly accurate predictions and model-agnostic interpretations, which are however also agnostic to biological mechanisms. Combining metabolic modelling, 'omics, and imaging data via multimodal AI can generate predictions that can be interpreted mechanistically and transparently, therefore with significantly higher therapeutic potential.

Children benefit from gestures to understand degraded speech but to a lesser extent than adults

The present study investigated to what extent children, compared to adults, benefit from gestures to disambiguate degraded speech by manipulating speech signals and manual modality. Dutch-speaking adults (N = 20) and 6- and 7-year-old children (N = 15) were presented with a series of video clips in which an actor produced a Dutch action verb with or without an accompanying iconic gesture. Participants were then asked to repeat what they had heard. The speech signal was either clear or altered into 4- or 8-band noise-vocoded speech. Children had more difficulty than adults in disambiguating degraded speech in the speech-only condition. However, when presented with both speech and gestures, children reached a comparable level of accuracy to that of adults in the degraded-speech-only condition. Furthermore, for adults, the enhancement of gestures was greater in the 4-band condition than in the 8-band condition, whereas children showed the opposite pattern. Gestures help children to disambiguate degraded speech, but children need more phonological information than adults to benefit from use of gestures. Children's multimodal language integration needs to further develop to adapt flexibly to challenging situations such as degraded speech, as tested in our study, or instances where speech is heard with environmental noise or through a face mask.

Multi-region calcium imaging in freely behaving mice with ultra-compact head-mounted fluorescence microscopes

To investigate the circuit-level neural mechanisms of behavior, simultaneous imaging of neuronal activity in multiple cortical and subcortical regions is highly desired. Miniature head-mounted microscopes offer the capability of calcium imaging in freely behaving animals. However, implanting multiple microscopes on a mouse brain remains challenging due to space constraints and the cumbersome weight of the equipment. Here, we present TINIscope, a Tightly Integrated Neuronal Imaging microscope optimized for electronic and opto-mechanical design. With its compact and lightweight design of 0.43 g, TINIscope enables unprecedented simultaneous imaging of behavior-relevant activity in up to four brain regions in mice. Proof-of-concept experiments with TINIscope recorded over 1000 neurons in four hippocampal subregions and revealed concurrent activity patterns spanning across these regions. Moreover, we explored potential multi-modal experimental designs by integrating additional modules for optogenetics, electrical stimulation or local field potential recordings. Overall, TINIscope represents a timely and indispensable tool for studying the brain-wide interregional coordination that underlies unrestrained behaviors.

Caresses, whispers and affective faces: A theoretical framework for a multimodal interoceptive mechanism underlying ASMR and affective touch: An evolutionary and developmental perspective for understanding ASMR and affective touch as complementary processes within affiliative interactions

Autonomous sensory meridian response (ASMR) and affective touch (AT) are two phenomena that have been independently investigated from separate lines of research. In this article, I provide a unified theoretical framework for understanding and studying them as complementary processes. I highlight their shared biological basis and positive effects on emotional and psychophysiological regulation. Drawing from evolutionary and developmental theories, I propose that ASMR results from the development of biological mechanisms associated with early affiliative behaviour and self-regulation, similar to AT. I also propose a multimodal interoceptive mechanism underlying both phenomena, suggesting that different sensory systems could specifically respond to affective stimulation (caresses, whispers and affective faces), where the integration of those inputs occurs in the brain's interoceptive hubs, allowing physiological regulation. The implications of this proposal are discussed with a view to future research that jointly examines ASMR and AT, and their potential impact on improving emotional well-being and mental health.

Integration of peripheral blood- and tissue-based biomarkers of response to immune checkpoint blockade in urothelial carcinoma

As predictive biomarkers of response to immune checkpoint inhibitors (ICIs) remain a major unmet clinical need in patients with urothelial carcinoma (UC), we sought to identify tissue-based immune biomarkers of clinical benefit to ICIs using multiplex immunofluorescence and to integrate these findings with previously identified peripheral blood biomarkers of response. Fifty-five pretreatment and 12 paired on-treatment UC specimens were identified from patients treated with nivolumab with or without ipilimumab. Whole tissue sections were stained with a 12-plex mIF panel, including CD8, PD-1/CD279, PD-L1/CD274, CD68, CD3, CD4, FoxP3, TCF1/7, Ki67, LAG-3, MHC-II/HLA-DR, and pancytokeratin+SOX10 to identify over three million cells. Immune tissue densities were compared to progression-free survival (PFS) and best overall response (BOR) by RECIST version 1.1. Correlation coefficients were calculated between tissue-based and circulating immune populations. The frequency of intratumoral CD3+ LAG-3+ cells was higher in responders compared to nonresponders (p = 0.0001). LAG-3+ cellular aggregates were associated with response, including CD3+ LAG-3+ in proximity to CD3+ (p = 0.01). Exploratory multivariate modeling showed an association between intratumoral CD3+ LAG-3+ cells and improved PFS independent of prognostic clinical factors (log HR -7.0; 95% confidence interval [CI] -12.7 to -1.4), as well as established biomarkers predictive of ICI response (log HR -5.0; 95% CI -9.8 to -0.2). Intratumoral LAG-3+ immune cell populations warrant further study as a predictive biomarker of clinical benefit to ICIs. Differences in LAG-3+ lymphocyte populations across the intratumoral and peripheral compartments may provide complementary information that could inform the future development of multimodal composite biomarkers of ICI response. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

Visual accelerated and olfactory decelerated responses during multimodal learning in honeybees

To obtain accurate information about the outside world and to make appropriate decisions, animals often combine information from different sensory pathways to form a comprehensive representation of their environment. This process of multimodal integration is poorly understood, but it is common view that the single elements of a multimodal stimulus influence each other's perception by enhancing or suppressing their neural representation. The neuronal level of interference might be manifold, for instance, an enhancement might increase, whereas suppression might decrease behavioural response times. In order to investigate this in an insect behavioural model, the Western honeybee, we trained individual bees to associate a sugar reward with an odour, a light, or a combined olfactory-visual stimulus, using the proboscis extension response (PER). We precisely monitored the PER latency (the time between stimulus onset and the first response of the proboscis) by recording the muscle M17, which innervates the proboscis. We found that odours evoked a fast response, whereas visual stimuli elicited a delayed PER. Interestingly, the combined stimulus showed a response time in between the unimodal stimuli, suggesting that olfactory-visual integration accelerates visual responses but decelerates the olfactory response time.

3D multiplexed tissue imaging reconstruction and optimized region of interest (ROI) selection through deep learning model of channels embedding

Introduction: Tissue-based sampling and diagnosis are defined as the extraction of information from certain limited spaces and its diagnostic significance of a certain object. Pathologists deal with issues related to tumor heterogeneity since analyzing a single sample does not necessarily capture a representative depiction of cancer, and a tissue biopsy usually only presents a small fraction of the tumor. Many multiplex tissue imaging platforms (MTIs) make the assumption that tissue microarrays (TMAs) containing small core samples of 2-dimensional (2D) tissue sections are a good approximation of bulk tumors although tumors are not 2D. However, emerging whole slide imaging (WSI) or 3D tumor atlases that use MTIs like cyclic immunofluorescence (CyCIF) strongly challenge this assumption. In spite of the additional insight gathered by measuring the tumor microenvironment in WSI or 3D, it can be prohibitively expensive and time-consuming to process tens or hundreds of tissue sections with CyCIF. Even when resources are not limited, the criteria for region of interest (ROI) selection in tissues for downstream analysis remain largely qualitative and subjective as stratified sampling requires the knowledge of objects and evaluates their features. Despite the fact TMAs fail to adequately approximate whole tissue features, a theoretical subsampling of tissue exists that can best represent the tumor in the whole slide image. Methods: To address these challenges, we propose deep learning approaches to learn multi-modal image translation tasks from two aspects: 1) generative modeling approach to reconstruct 3D CyCIF representation and 2) co-embedding CyCIF image and Hematoxylin and Eosin (H&E) section to learn multi-modal mappings by a cross-domain translation for minimum representative ROI selection. Results and discussion: We demonstrate that generative modeling enables a 3D virtual CyCIF reconstruction of a colorectal cancer specimen given a small subset of the imaging data at training time. By co-embedding histology and MTI features, we propose a simple convex optimization for objective ROI selection. We demonstrate the potential application of ROI selection and the efficiency of its performance with respect to cellular heterogeneity.

Epigenomic dissection of Alzheimer's disease pinpoints causal variants and reveals epigenome erosion

Recent work has identified dozens of non-coding loci for Alzheimer's disease (AD) risk, but their mechanisms and AD transcriptional regulatory circuitry are poorly understood. Here, we profile epigenomic and transcriptomic landscapes of 850,000 nuclei from prefrontal cortexes of 92 individuals with and without AD to build a map of the brain regulome, including epigenomic profiles, transcriptional regulators, co-accessibility modules, and peak-to-gene links in a cell-type-specific manner. We develop methods for multimodal integration and detecting regulatory modules using peak-to-gene linking. We show AD risk loci are enriched in microglial enhancers and for specific TFs including SPI1, ELF2, and RUNX1. We detect 9,628 cell-type-specific ATAC-QTL loci, which we integrate alongside peak-to-gene links to prioritize AD variant regulatory circuits. We report differential accessibility of regulatory modules in late AD in glia and in early AD in neurons. Strikingly, late-stage AD brains show global epigenome dysregulation indicative of epigenome erosion and cell identity loss.

Brown goat yogurt: Metabolomics, peptidomics, and sensory changes during production

Brown goat milk products have gained popularity for their unique taste and flavor. The emergence of chain-reversal phenomenon makes the design and development of goat milk products gradually tend to a consumer-oriented model. However, the precise mechanism of how browning and fermentation process causes characteristics is not clear. In an effort to understand how the treatments potentially lead to certain metabolite profile changes in goat milk, comprehensive, quantitative metabolomics and peptidomics analysis of goat milk samples after browning and fermentation were undertaken. An intelligent hybrid z-score standardization-principal components algorithm-multimodal denoizing autoencoder was used for feature fusion and hidden layer fusion in high-dimensional variable space. The fermentation process significantly improved the flavor of brown goat yogurt through the tricarboxylic acid-urea-glycolysis composite pathway. Bitter peptides HPFLEWAR, PPGLPDKY, and PPPPPKK have strong interactions with both putative dipeptidyl peptidase IV and angiotensin-converting enzyme, proving that brown goat yogurt can be considered as effective provider of potential putative dipeptidyl peptidase IV and angiotensin-converting enzyme inhibitors. The level of health-promoting bioactive components and sensory contributed to consumer selection. The proposed multimodal data integrative analysis platform was applicable to explain the effect of the dynamic changes of metabolites and peptides on consumer preferences.

Small-amplitude head oscillations result from a multimodal head stabilization reflex in hawkmoths

In flying insects, head stabilization is an essential reflex that helps to reduce motion blur during fast aerial manoeuvres. This reflex is multimodal and requires the integration of visual and antennal mechanosensory feedback in hawkmoths, each operating as a negative-feedback-control loop. As in any negative-feedback system, the head stabilization system possesses inherent oscillatory dynamics that depend on the rate at which the sensorimotor components of the reflex operate. Consistent with this expectation, we observed small-amplitude oscillations in the head motion (or head wobble) of the oleander hawkmoth, Daphnis nerii, which are accentuated when sensory feedback is aberrant. Here, we show that these oscillations emerge from the inherent dynamics of the multimodal reflex underlying gaze stabilization, and that the amplitude of head wobble is a function of both the visual feedback and antennal mechanosensory feedback from the Johnston's organs. Our data support the hypothesis that head wobble results from a multimodal, dynamically stabilized reflex loop that mediates head positioning.

Artificial intelligence for multimodal data integration in oncology

In oncology, the patient state is characterized by a whole spectrum of modalities, ranging from radiology, histology, and genomics to electronic health records. Current artificial intelligence (AI) models operate mainly in the realm of a single modality, neglecting the broader clinical context, which inevitably diminishes their potential. Integration of different data modalities provides opportunities to increase robustness and accuracy of diagnostic and prognostic models, bringing AI closer to clinical practice. AI models are also capable of discovering novel patterns within and across modalities suitable for explaining differences in patient outcomes or treatment resistance. The insights gleaned from such models can guide exploration studies and contribute to the discovery of novel biomarkers and therapeutic targets. To support these advances, here we present a synopsis of AI methods and strategies for multimodal data fusion and association discovery. We outline approaches for AI interpretability and directions for AI-driven exploration through multimodal data interconnections. We examine challenges in clinical adoption and discuss emerging solutions.

Pan-cancer integrative histology-genomic analysis via multimodal deep learning

The rapidly emerging field of computational pathology has demonstrated promise in developing objective prognostic models from histology images. However, most prognostic models are either based on histology or genomics alone and do not address how these data sources can be integrated to develop joint image-omic prognostic models. Additionally, identifying explainable morphological and molecular descriptors from these models that govern such prognosis is of interest. We use multimodal deep learning to jointly examine pathology whole-slide images and molecular profile data from 14 cancer types. Our weakly supervised, multimodal deep-learning algorithm is able to fuse these heterogeneous modalities to predict outcomes and discover prognostic features that correlate with poor and favorable outcomes. We present all analyses for morphological and molecular correlates of patient prognosis across the 14 cancer types at both a disease and a patient level in an interactive open-access database to allow for further exploration, biomarker discovery, and feature assessment.

Combining electro- and magnetoencephalography data using directional archetypal analysis

Metastable microstates in electro- and magnetoencephalographic (EEG and MEG) measurements are usually determined using modified k-means accounting for polarity invariant states. However, hard state assignment approaches assume that the brain traverses microstates in a discrete rather than continuous fashion. We present multimodal, multisubject directional archetypal analysis as a scale and polarity invariant extension to archetypal analysis using a loss function based on the Watson distribution. With this method, EEG/MEG microstates are modeled using subject- and modality-specific archetypes that are representative, distinct topographic maps between which the brain continuously traverses. Archetypes are specified as convex combinations of unit norm input data based on a shared generator matrix, thus assuming that the timing of neural responses to stimuli is consistent across subjects and modalities. The input data is reconstructed as convex combinations of archetypes using a subject- and modality-specific continuous archetypal mixing matrix. We showcase the model on synthetic data and an openly available face perception event-related potential data set with concurrently recorded EEG and MEG. In synthetic and unimodal experiments, we compare our model to conventional Euclidean multisubject archetypal analysis. We also contrast our model to a directional clustering model with discrete state assignments to highlight the advantages of modeling state trajectories rather than hard assignments. We find that our approach successfully models scale and polarity invariant data, such as microstates, accounting for intersubject and intermodal variability. The model is readily extendable to other modalities ensuring component correspondence while elucidating spatiotemporal signal variability.

Non-linear multimodal integration in a distributed premotor network controls proprioceptive reflex gain in the insect leg

Producing context-appropriate motor acts requires integrating multiple sensory modalities. Presynaptic inhibition of proprioceptive afferent neurons^1-4 and afferents of different modalities targeting the same motor neurons (MNs)^5-7 underlies some of this integration. However, in most systems, an interneuronal network is interposed between sensory afferents and MNs. How these networks contribute to this integration, particularly at single-neuron resolution, is little understood. Context-specific integration of load and movement sensory inputs occurs in the stick insect locomotory system,^6,8-12 and both inputs feed into a network of premotor nonspiking interneurons (NSIs).⁸ We analyzed how load altered movement signal processing in the stick insect femur-tibia (FTi) joint control system by tracing the interaction of FTi movement^13-15 (femoral chordotonal organ [fCO]) and load^13,15,16 (tibial campaniform sensilla [CS]) signals through the NSI network to the slow extensor tibiae (SETi) MN, the extensor MN primarily active in non-walking animals.^17-19 On the afferent level, load reduced movement signal gain by presynaptic inhibition. In the NSI network, graded responses to movement and load inputs summed nonlinearly, increasing the gain of NSIs opposing movement-induced reflexes and thus decreasing the SETi and extensor tibiae muscle movement reflex responses. Gain modulation was movement-parameter specific and required presynaptic inhibition. These data suggest that gain changes in distributed premotor networks, specifically the relative weighting of antagonistic pathways, could be a general mechanism by which multiple sensory modalities are integrated to generate context-appropriate motor activity.

Tingles down the spinal cord: A spinal functional magnetic resonance imaging investigation of the autonomous sensory meridian response

Autonomous sensory meridian response (ASMR) is a perceptual and emotional phenomenon in which specific sensory stimuli elicit a feeling of calm as well as tingling sensations on the scalp, neck, and shoulders. In the current study, we use fMRI to examine whether the motoric and sensory regions of the spinal cord segments associated with these body parts show increased activity during ASMR experiences. Nine individuals with ASMR completed six spinal functional magnetic resonance imaging runs while passively viewing videos. Three of the videos were shown (through pre-testing) to elicit ASMR tingles and three videos did not (i.e., control videos). The results demonstrated that ASMR-related stimuli elicited activity in dorsal (sensory) regions of spinal cord segments C1, C5, and C6; activity was observed in ventral (motoric) regions of segments C2-C8. Similar activity was not detected in response to control videos.

Faces and Voices Processing in Human and Primate Brains: Rhythmic and Multimodal Mechanisms Underlying the Evolution and Development of Speech

While influential works since the 1970s have widely assumed that imitation is an innate skill in both human and non-human primate neonates, recent empirical studies and meta-analyses have challenged this view, indicating other forms of reward-based learning as relevant factors in the development of social behavior. The visual input translation into matching motor output that underlies imitation abilities instead seems to develop along with social interactions and sensorimotor experience during infancy and childhood. Recently, a new visual stream has been identified in both human and non-human primate brains, updating the dual visual stream model. This third pathway is thought to be specialized for dynamics aspects of social perceptions such as eye-gaze, facial expression and crucially for audio-visual integration of speech. Here, we review empirical studies addressing an understudied but crucial aspect of speech and communication, namely the processing of visual orofacial cues (i.e., the perception of a speaker's lips and tongue movements) and its integration with vocal auditory cues. Along this review, we offer new insights from our understanding of speech as the product of evolution and development of a rhythmic and multimodal organization of sensorimotor brain networks, supporting volitional motor control of the upper vocal tract and audio-visual voices-faces integration.

Temporal differences between load and movement signal integration in the sensorimotor network of an insect leg

Nervous systems face a torrent of sensory inputs, including proprioceptive feedback. Signal integration depends on spatially and temporally coinciding signals. It is unclear how relative time delays affect multimodal signal integration from spatially distant sense organs. We measured transmission times and latencies along all processing stages of sensorimotor pathways in the stick insect leg muscle control system, using intra- and extracellular recordings. Transmission times of signals from load-sensing tibial and trochanterofemoral campaniform sensilla (tiCS, tr/fCS) to the premotor network were longer than from the movement-sensing femoral chordotonal organ (fCO). We characterized connectivity patterns from tiCS, tr/fCS, and fCO afferents to identified premotor nonspiking interneurons (NSIs) and motor neurons (MNs) by distinguishing short- and long-latency responses to sensory stimuli. Functional NSI connectivity depended on sensory context. The timeline of multisensory integration in the NSI network showed an early phase of movement signal processing and a delayed phase of load signal integration. The temporal delay of load signals relative to movement feedback persisted into MN activity and muscle force development. We demonstrate differential delays in the processing of two distinct sensory modalities generated by the sensorimotor network and affecting motor output. The reported temporal differences in sensory processing and signal integration improve our understanding of sensory network computation and function in motor control.NEW & NOTEWORTHY Networks integrating multisensory input face the challenge of not only spatial but also temporal integration. In the local network controlling insect leg movements, proprioceptive signal delays differ between sensory modalities. Specifically, signal transmission times to and neuronal connectivity within the sensorimotor network lead to delayed information about leg loading relative to movement signals. Temporal delays persist up to the level of the motor output, demonstrating its relevance for motor control.

Automated Machine-Learning Framework Integrating Histopathological and Radiological Information for Predicting IDH1 Mutation Status in Glioma

Diffuse gliomas are the most common malignant primary brain tumors. Identification of isocitrate dehydrogenase 1 (IDH1) mutations aids the diagnostic classification of these tumors and the prediction of their clinical outcomes. While histology continues to play a key role in frozen section diagnosis, as a diagnostic reference and as a method for monitoring disease progression, recent research has demonstrated the ability of multi-parametric magnetic resonance imaging (MRI) sequences for predicting IDH genotypes. In this paper, we aim to improve the prediction accuracy of IDH1 genotypes by integrating multi-modal imaging information from digitized histopathological data derived from routine histological slide scans and the MRI sequences including T1-contrast (T1) and Fluid-attenuated inversion recovery imaging (T2-FLAIR). In this research, we have established an automated framework to process, analyze and integrate the histopathological and radiological information from high-resolution pathology slides and multi-sequence MRI scans. Our machine-learning framework comprehensively computed multi-level information including molecular level, cellular level, and texture level information to reflect predictive IDH genotypes. Firstly, an automated pre-processing was developed to select the regions of interest (ROIs) from pathology slides. Secondly, to interactively fuse the multimodal complementary information, comprehensive feature information was extracted from the pathology ROIs and segmented tumor regions (enhanced tumor, edema and non-enhanced tumor) from MRI sequences. Thirdly, a Random Forest (RF)-based algorithm was employed to identify and quantitatively characterize histopathological and radiological imaging origins, respectively. Finally, we integrated multi-modal imaging features with a machine-learning algorithm and tested the performance of the framework for IDH1 genotyping, we also provided visual and statistical explanation to support the understanding on prediction outcomes. The training and testing experiments on 217 pathologically verified IDH1 genotyped glioma cases from multi-resource validated that our fully automated machine-learning model predicted IDH1 genotypes with greater accuracy and reliability than models that were based on radiological imaging data only. The accuracy of IDH1 genotype prediction was 0.90 compared to 0.82 for radiomic result. Thus, the integration of multi-parametric imaging features for automated analysis of cross-modal biomedical data improved the prediction accuracy of glioma IDH1 genotypes.

Multimodal Image Integration for Epilepsy Presurgical Evaluation: A Clinical Workflow

Multimodal image integration (MMII) is a promising tool to help delineate the epileptogenic zone (EZ) in patients with medically intractable focal epilepsies undergoing presurgical evaluation. We report here the detailed methodology of MMII and an overview of the utility of MMII at the Cleveland Clinic Epilepsy Center from 2014 to 2018, exemplified by illustrative cases. The image integration was performed using the Curry platform (Compumedics Neuroscan^™, Charlotte, NC, USA), including all available diagnostic modalities such as Magnetic resonance imaging (MRI), Positron Emission Tomography (PET), single-photon emission computed tomography (SPECT) and Magnetoencephalography (MEG), with additional capability of trajectory planning for intracranial EEG (ICEEG), particularly stereo-EEG (SEEG), as well as surgical resection planning. In the 5-year time span, 467 patients underwent MMII; of them, 98 patients (21%) had a history of prior neurosurgery and recurring seizures. Of the 467 patients, 425 patients underwent ICEEG implantation with further CT co-registration to identify the electrode locations. A total of 351 patients eventually underwent surgery after MMII, including 197 patients (56%) with non-lesional MRI and 223 patients (64%) with extra-temporal lobe epilepsy. Among 269 patients with 1-year post-operative follow up, 134 patients (50%) had remained completely seizure-free. The most common histopathological finding is focal cortical dysplasia. Our study illustrates the usefulness of MMII to enhance SEEG electrode trajectory planning, assist non-invasive/invasive data interpretation, plan resection strategy, and re-evaluate surgical failures. Information presented by MMII is essential to the understanding of the anatomo-functional-electro-clinical correlations in individual cases, which leads to the ultimate success of presurgical evaluation of patients with medically intractable focal epilepsies.

The central nervous system of whip spiders (Amblypygi): Large mushroom bodies receive olfactory and visual input

Whip spiders (Amblypygi) are known for their nocturnal navigational abilities, which rely on chemosensory and tactile cues and, to a lesser degree, on vision. Unlike true spiders, the first pair of legs in whip spiders is modified into extraordinarily long sensory organs (antenniform legs) covered with thousands of mechanosensory, olfactory, and gustatory sensilla. Olfactory neurons send their axons through the leg nerve into the corresponding neuromere of the central nervous system, where they terminate on a particularly large number (about 460) of primary olfactory glomeruli, suggesting an advanced sense of smell. From the primary glomeruli, olfactory projection neurons ascend to the brain and terminate in the mushroom body calyx on a set of secondary olfactory glomeruli, a feature that is not known from olfactory pathways of other animals. Another part of the calyx receives visual input from the secondary visual neuropil (the medulla). This calyx region is composed of much smaller glomeruli ("microglomeruli"). The bimodal input and the exceptional size of their mushroom bodies may support the navigational capabilities of whip spiders. In addition to input to the mushroom body, we describe other general anatomical features of the whip spiders' central nervous system.

Distributed processing of load and movement feedback in the premotor network controlling an insect leg joint

In legged animals, integration of information from various proprioceptors in and on the appendages by local premotor networks in the central nervous system is crucial for controlling motor output. To ensure posture maintenance and precise active movements, information about limb loading and movement is required. In insects, various groups of campaniform sensilla (CS) measure forces and loads acting in different directions on the leg, and the femoral chordotonal organ (fCO) provides information about movement of the femur-tibia (FTi) joint. In this study, we used extra- and intracellular recordings of extensor tibiae (ExtTi) and retractor coxae (RetCx) motor neurons (MNs) and identified local premotor nonspiking interneurons (NSIs) and mechanical stimulation of the fCO and tibial or trochanterofemoral CS (tiCS, tr/fCS), to investigate the premotor network architecture underlying multimodal proprioceptive integration. We found that load feedback from tiCS altered the strength of movement-elicited resistance reflexes and determined the specificity of ExtTi and RetCx MN responses to various load and movement stimuli. These responses were mediated by a common population of identified NSIs into which synaptic inputs from the fCO, tiCS, and tr/fCS are distributed, and whose effects onto ExtTi MNs can be antagonistic for both stimulus modalities. Multimodal sensory signal interaction was found at the level of single NSIs and MNs. The results provide evidence that load and movement feedback are integrated in a multimodal, distributed local premotor network consisting of antagonistic elements controlling movements of the FTi joint, thus substantially extending current knowledge on how legged motor systems achieve fine-tuned motor control.NEW & NOTEWORTHY Proprioception is crucial for motor control in legged animals. We show the extent to which processing of movement (fCO) and load (CS) signals overlaps in the local premotor network of an insect leg. Multimodal signals converge onto the same set of interneurons, and our knowledge about distributed, antagonistic processing is extended to incorporate multiple modalities within one perceptual neuronal framework.

Rapid invisible frequency tagging reveals nonlinear integration of auditory and visual information

During communication in real-life settings, the brain integrates information from auditory and visual modalities to form a unified percept of our environment. In the current magnetoencephalography (MEG) study, we used rapid invisible frequency tagging (RIFT) to generate steady-state evoked fields and investigated the integration of audiovisual information in a semantic context. We presented participants with videos of an actress uttering action verbs (auditory; tagged at 61 Hz) accompanied by a gesture (visual; tagged at 68 Hz, using a projector with a 1,440 Hz refresh rate). Integration difficulty was manipulated by lower-order auditory factors (clear/degraded speech) and higher-order visual factors (congruent/incongruent gesture). We identified MEG spectral peaks at the individual (61/68 Hz) tagging frequencies. We furthermore observed a peak at the intermodulation frequency of the auditory and visually tagged signals (fvisual - fauditory  = 7 Hz), specifically when lower-order integration was easiest because signal quality was optimal. This intermodulation peak is a signature of nonlinear audiovisual integration, and was strongest in left inferior frontal gyrus and left temporal regions; areas known to be involved in speech-gesture integration. The enhanced power at the intermodulation frequency thus reflects the ease of lower-order audiovisual integration and demonstrates that speech-gesture information interacts in higher-order language areas. Furthermore, we provide a proof-of-principle of the use of RIFT to study the integration of audiovisual stimuli, in relation to, for instance, semantic context.

Multimodal Sensorimotor Integration of Visual and Kinaesthetic Afferents Modulates Motor Circuits in Humans

Optimal motor control requires the effective integration of multi-modal information. Visual information of movement performed by others even enhances potentials in the upper motor neurons through the mirror-neuron system. On the other hand, it is known that motor control is intimately associated with afferent proprioceptive information. Kinaesthetic information is also generated by passive, external-driven movements. In the context of sensory integration, it is an important question how such passive kinaesthetic information and visually perceived movements are integrated. We studied the effects of visual and kinaesthetic information in combination, as well as isolated, on sensorimotor integration, compared to a control condition. For this, we measured the change in the excitability of the motor cortex (M1) using low-intensity Transcranial magnetic stimulation (TMS). We hypothesised that both visual motoneurons and kinaesthetic motoneurons enhance the excitability of motor responses. We found that passive wrist movements increase the motor excitability, suggesting that kinaesthetic motoneurons do exist. The kinaesthetic influence on the motor threshold was even stronger than the visual information. Moreover, the simultaneous visual and passive kinaesthetic information increased the cortical excitability more than each of them independently. Thus, for the first time, we found evidence for the integration of passive kinaesthetic- and visual-sensory stimuli.

Investigating the Hand Ownership Illusion With Two Views Merged in

Researchers investigating virtual/augmented reality have shown humans' marked adaptability, especially regarding our sense of body ownership; their cumulative findings have expanded the concept of what it means to have a body. Herein, we report the hand ownership illusion during "two views merged in." In our experiment, participants were presented two first-person perspective views of their arm overlapped, one was the live feed from a camera and the other was a playback video of the same situation, slightly shifted toward one side. The relative visibility of these two views and synchrony of tactile stimulation were manipulated. Participants' level of embodiment was evaluated using a questionnaire and proprioceptive drift. The results show that the likelihood of embodying the virtual hand is affected by the relative visibility of the two views and synchrony of the tactile events. We observed especially strong hand ownership of the virtual hand in the context of high virtual hand visibility with synchronous tactile stimulation.

Effects of an Audiovisual Emotion Perception Training for Schizophrenia: A Preliminary Study

Individuals with schizophrenia show a reduced ability to integrate facial and vocal information in emotion perception. Although emotion perception has been a target for treatment, no study has yet examined the effect of multimodal training on emotion perception in schizophrenia. In the present study, we developed an audiovisual emotion perception training and test in which a voice and a face were simultaneously presented, and subjects were asked to judge whether the emotions of the voice and the face matched. The voices were either angry or happy, and the faces were morphed on a continuum ranging from angry to happy. Sixteen patients with schizophrenia participated in six training sessions and three test sessions (i.e., pre-training, post-training, and generalization). Eighteen healthy controls participated only in pre-training test session. Prior to training, the patients with schizophrenia performed significantly worse than did the controls in the recognition of anger; however, following the training, the patients showed a significant improvement in recognizing anger, which was maintained and generalized to a new set of stimuli. The patients also improved the recognition of happiness following the training, but this effect was not maintained or generalized. These results provide preliminary evidence that a multimodal, audiovisual training may yield improvements in anger perception for patients with schizophrenia.

High-Expanding Regions in Primate Cortical Brain Evolution Support Supramodal Cognitive Flexibility

Primate cortical evolution has been characterized by massive and disproportionate expansion of a set of specific regions in the neocortex. The associated increase in neocortical neurons comes with a high metabolic cost, thus the functions served by these regions must have conferred significant evolutionary advantage. In the present series of analyses, we show that evolutionary high-expanding cortex - as estimated from patterns of surface growth from several primate species - shares functional connections with different brain networks in a context-dependent manner. Specifically, we demonstrate that high-expanding cortex is characterized by high internetwork functional connectivity; is recruited flexibly over many different cognitive tasks; and changes its functional coupling pattern between rest and a multimodal task-state. The capacity of high-expanding cortex to connect flexibly with various specialized brain networks depending on particular cognitive requirements suggests that its selective growth and sustainment in evolution may have been linked to an involvement in supramodal cognition. In accordance with an evolutionary-developmental view, we find that this observed ability of high-expanding regions - to flexibly modulate functional connections as a function of cognitive state - emerges gradually through childhood, with a prolonged developmental trajectory plateauing in young adulthood.

Not Just from Blood: Mosquito Nutrient Acquisition from Nectar Sources

Anthropophilic female mosquitoes are well known for their strong attraction to human hosts, but plant nectar is a common energy source in their diets. When sugar sources are scarce, female mosquitoes of some species can compensate by taking larger and more frequent blood meals. Male mosquitoes are exclusively dependent on plant nectar or alternative sugar sources. Plant preference is likely driven by an innate attraction that may be enhanced by experience, as mosquitoes learn to recognize available sugar rewards. Nectar-seeking involves the integration of at least three sensory systems: olfaction, vision and taste. The prevention of vector-borne illnesses, the determination of the mosquitoes' ecological role, and the design of efficient sugar-baited traps will all benefit from understanding the molecular basis of nectar-seeking.

Multimodal Integration of Brain Images for MRI-Based Diagnosis in Schizophrenia

Magnetic resonance imaging (MRI) has been proposed as a source of information for automatic prediction of individual diagnosis in schizophrenia. Optimal integration of data from different MRI modalities is an active area of research aimed at increasing diagnostic accuracy. Based on a sample of 96 patients with schizophrenia and a matched sample of 115 healthy controls that had undergone a single multimodal MRI session, we generated individual brain maps of gray matter vbm, 1back, and 2back levels of activation (nback fMRI), maps of amplitude of low-frequency fluctuations (resting-state fMRI), and maps of weighted global brain connectivity (resting-state fMRI). Four unimodal classifiers (Ridge, Lasso, Random Forests, and Gradient boosting) were applied to these maps to evaluate their classification accuracies. Based on the assignments made by the algorithms on test individuals, we quantified the amount of predictive information shared between maps (what we call redundancy analysis). Finally, we explored the added accuracy provided by a set of multimodal strategies that included post-classification integration based on probabilities, two-step sequential integration, and voxel-level multimodal integration through one-dimensional-convolutional neural networks (1D-CNNs). All four unimodal classifiers showed the highest test accuracies with the 2back maps (80% on average) achieving a maximum of 84% with the Lasso. Redundancy levels between brain maps were generally low (overall mean redundancy score of 0.14 in a 0–1 range), indicating that each brain map contained differential predictive information. The highest multimodal accuracy was delivered by the two-step Ridge classifier (87%) followed by the Ridge maximum and mean probability classifiers (both with 85% accuracy) and by the 1D-CNN, which achieved the same accuracy as the best unimodal classifier (84%). From these results, we conclude that from all MRI modalities evaluated task-based fMRI may be the best unimodal diagnostic option in schizophrenia. Low redundancy values point to ample potential for accuracy improvements through multimodal integration, with the two-step Ridge emerging as a suitable strategy.

Anatomy and Physiology of Neurons in Layer 9 of the Chicken Optic Tectum

Visual information in birds is to great extent processed in the optic tectum (TeO), a prominent laminated midbrain structure. Retinal input enters the TeO in its superficial layers, while output is limited to intermediate and deeper layers. In addition to visual information, the TeO receives multimodal input from the auditory and somatosensory pathway. The TeO gives rise to a major ascending tectofugal projection where neurons of tectal layer 13 project to the thalamic nucleus rotundus, which then projects to the entopallium. A second tectofugal projection system, called the accessory pathway, has however not been studied as thoroughly. Again, cells of tectal layer 13 form an ascending projection that targets a nucleus known as either the caudal part of the nucleus dorsolateralis posterior of the thalamus (DLPc) or nucleus uveaformis (Uva). This nucleus is known for multimodal integration and receives additional input from the lateral pontine nucleus (PL), which in turn receives projections from layer 8–15 of the TeO. Here, we studied a particular cell type afferent to the PL that consists of radially oriented neurons in layer 9. We characterized these neurons with respect to their anatomy, their retinal input, and the modulation of retinal input by local circuits. We found that comparable to other radial neurons in the tectum, cells of layer 9 have columnar dendritic fields and reach up to layer 2. Sholl analysis demonstrated that dendritic arborization concentrates on retinorecipient layers 2 and 4, with additional arborization in layers 9 and 10. All neurons recorded in layer 9 received retinal input via glutamatergic synapses. We analyzed the influence of modulatory circuits of the TeO by application of antagonists to γ-aminobutyric acid (GABA) and acetylcholine (ACh). Our data show that the neurons of layer 9 are integrated in a network under strong GABAergic inhibition, which is controlled by local cholinergic activation. Output to the PL and to the accessory tectofugal pathway thus appears to be under strict control of local tectal networks, the relevance of which for multimodal integration is discussed.

A Tablet-Based Assessment of Rhythmic Ability

The exponential rise in use of mobile consumer electronics has presented a great potential for research to be conducted remotely, with participants numbering several orders of magnitude greater than a typical research paradigm. Here, we attempt to demonstrate the validity and reliability of using a consumer game-engine to create software presented on a mobile tablet to assess sensorimotor synchronization, a proxy of rhythmic ability. Our goal was to ascertain whether previously observed research results can be replicated, rather than assess whether a mobile tablet achieves comparable performance to a desktop computer. To achieve this, younger (aged 18–35 years) and older (aged 60–80 years) adult musicians and non-musicians were recruited to play a custom-designed sensorimotor synchronization assessment on a mobile tablet in a controlled laboratory environment. To assess reliability, participants performed the assessment twice, separated by a week, and an intra-class correlation coefficient (ICC) was calculated. Results supported the validity of this approach to assessing rhythmic abilities by replicating previously observed results. Specifically, musicians performed better than non-musicians, and younger adults performed better than older adults. Participants also performed best when the tempo was in the range of previously-identified preferred tempos, when the stimuli included both audio and visual information, and when synchronizing on-beat compared to off-beat or continuation (self-paced) synchronization. Additionally, high ICC values (>0.75) suggested excellent test–retest reliability. Together, these results support the notion that consumer electronics running software built with a game engine may serve as a valuable resource for remote, mobile-based data collection of rhythmic abilities.

Cortical Regions Encoding Hardness Perception Modulated by Visual Information Identified by Functional Magnetic Resonance Imaging With Multivoxel Pattern Analysis

Recent studies have revealed that hardness perception is determined by visual information along with the haptic input. This study investigated the cortical regions involved in hardness perception modulated by visual information using functional magnetic resonance imaging (fMRI) and multivoxel pattern analysis (MVPA). Twenty-two healthy participants were enrolled. They were required to place their left and right hands at the front and back, respectively, of a mirror attached to a platform placed above them while lying in a magnetic resonance scanner. In conditions SFT, MED, and HRD, one of three polyurethane foam pads of varying hardness (soft, medium, and hard, respectively) was presented to the left hand in a given trial, while only the medium pad was presented to the right hand in all trials. MED was defined as the control condition, because the visual and haptic information was congruent. During the scan, the participants were required to push the pad with the both hands while observing the reflection of the left hand and estimate the hardness of the pad perceived by the right (hidden) hand based on magnitude estimation. Behavioral results showed that the perceived hardness was significantly biased toward softer or harder in >73% of the trials in conditions SFT and HRD; we designated these trials as visually modulated (SFTvm and HRDvm, respectively). The accuracy map was calculated individually for each of the pair-wise comparisons of (SFTvm vs. MED), (HRDvm vs. MED), and (SFTvm vs. HRDvm) by a searchlight MVPA, and the cortical regions encoding the perceived hardness with visual modulation were identified by conjunction of the three accuracy maps in group analysis. The cluster was observed in the right sensory motor cortex, left anterior intraparietal sulcus (aIPS), bilateral parietal operculum (PO), and occipito-temporal cortex (OTC). Together with previous findings on such cortical regions, we conclude that the visual information of finger movements processed in the OTC may be integrated with haptic input in the left aIPS, and the subjective hardness perceived by the right hand with visual modulation may be processed in the cortical network between the left PO and aIPS.

Why Is There So Much More Research on Vision Than on Any Other Sensory Modality?

Why is there so much more research on vision than on any other sensory modality? There is a seemingly easy answer to this question: It is because vision is our most important and most complex sense. Although there are arguments in favor of this explanation, it can be challenged in two ways: by showing that the arguments regarding the importance and complexity of vision are debatable and by demonstrating that there are other aspects that need to be taken into account. Here, I argue that the explanation is debatable, as there are various ways of defining “importance” and “complexity” and, as there is no clear consensus that vision is indeed the most important and most complex of our senses. Hence, I propose two additional explanations: According to the methodological-structural explanation, there is more research on vision because the available, present-day technology is better suited for studying vision than for studying other modalities – an advantage which most likely is the result of an initial bias toward vision, which reinforces itself. Possible reasons for such an initial bias are discussed. The cultural explanation emphasizes that the dominance of the visual is not an unchangeable constant, but rather the result of the way our societies are designed and thus heavily influenced by human decision-making. As it turns out, there is no universal hierarchy of the senses, but great historical and cross-cultural variation. Realizing that the dominance of the visual is socially and culturally reinforced and not simply a law of nature, gives us the opportunity to take a step back and to think about the kind of sensory environments we want to create and about the kinds of theories that need to be developed in research.

General Visual and Contingent Thermal Cues Interact to Elicit Attraction in Female Aedes aegypti Mosquitoes

Female Aedes aegypti mosquitoes use multiple sensory modalities to hunt human hosts and obtain a blood meal for egg production. Attractive cues include carbon dioxide (CO₂), a major component of exhaled breath [1, 2]; heat elevated above ambient temperature, signifying warm-blooded skin [3, 4]; and dark visual contrast [5, 6], proposed to bridge long-range olfactory and short-range thermal cues [7]. Any of these sensory cues in isolation is an incomplete signal of a human host, and so a mosquito must integrate multimodal sensory information before committing to approaching and biting a person [8]. Here, we study the interaction of visual cues, heat, and CO₂ to investigate the contributions of human-associated stimuli to host-seeking decisions. We show that tethered flying mosquitoes strongly orient toward dark visual contrast, regardless of CO₂ stimulation and internal host-seeking status. This suggests that attraction to visual contrast is general and not contingent on other host cues. In free-flight experiments with CO₂, adding a dark contrasting visual cue to a warmed surface enhanced attraction. Moderate warmth became more attractive to mosquitoes, and mosquitoes aggregated on the cue at all non-noxious temperatures. Gr3 mutants, unable to detect CO₂, were lured to the visual cue at ambient temperatures but fled and did not return when the surface was warmed to host-like temperatures. This suggests that attraction to thermal cues is contingent on the presence of the additional sensory cue CO₂. Our results illustrate that mosquitoes integrate general attractive visual stimuli with context-dependent thermal stimuli to seek promising sites for blood feeding.

Representation of Haltere Oscillations and Integration with Visual Inputs in the Fly Central Complex

The reduced hindwings of flies, known as halteres, are specialized mechanosensory organs that detect body rotations during flight. Primary afferents of the haltere encode its oscillation frequency linearly over a wide bandwidth and with precise phase-dependent spiking. However, it is not currently known whether information from haltere primary afferent neurons is sent to higher brain centers where sensory information about body position could be used in decision making, or whether precise spike timing is useful beyond the peripheral circuits that drive wing movements. We show that in cells in the central brain, the timing and rates of neural spiking can be modulated by sensory input from experimental haltere movements (driven by a servomotor). Using multichannel extracellular recording in restrained flesh flies (Sarcophaga bullata of both sexes), we examined responses of central complex cells to a range of haltere oscillation frequencies alone, and in combination with visual motion speeds and directions. Haltere-responsive units fell into multiple response classes, including those responding to any haltere motion and others with firing rates linearly related to the haltere frequency. Cells with multisensory responses showed higher firing rates than the sum of the unisensory responses at higher haltere frequencies. They also maintained visual properties, such as directional selectivity, while increasing response gain nonlinearly with haltere frequency. Although haltere inputs have been described extensively in the context of rapid locomotion control, we find haltere sensory information in a brain region known to be involved in slower, higher-order behaviors, such as navigation.SIGNIFICANCE STATEMENT Many animals use vision for navigation; however, these cues must be interpreted in the context of the body's position. In mammalian brains, hippocampal cells combine visual and vestibular information to encode head direction. A region of the arthropod brain, known as the central complex (CX), similarly encodes heading information, but it is unknown whether proprioceptive information is integrated here as well. We show that CX neurons respond to input from halteres, specialized proprioceptors in flies that detect body rotations. These neurons also respond to visual input, providing one of the few examples of multiple sensory modalities represented in individual CX cells. Haltere stimulation modifies neural responses to visual signals, providing a mechanism for integrating vision with proprioception.

Brain-Inspired Coding of Robot Body Schema Through Visuo-Motor Integration of Touched Events

Representing objects in space is difficult because sensorimotor events are anchored in different reference frames, which can be either eye-, arm-, or target-centered. In the brain, Gain-Field (GF) neurons in the parietal cortex are involved in computing the necessary spatial transformations for aligning the tactile, visual and proprioceptive signals. In reaching tasks, these GF neurons exploit a mechanism based on multiplicative interaction for binding simultaneously touched events from the hand with visual and proprioception information.By doing so, they can infer new reference frames to represent dynamically the location of the body parts in the visual space (i.e., the body schema) and nearby targets (i.e., its peripersonal space). In this line, we propose a neural model based on GF neurons for integrating tactile events with arm postures and visual locations for constructing hand- and target-centered receptive fields in the visual space. In robotic experiments using an artificial skin, we show how our neural architecture reproduces the behaviors of parietal neurons (1) for encoding dynamically the body schema of our robotic arm without any visual tags on it and (2) for estimating the relative orientation and distance of targets to it. We demonstrate how tactile information facilitates the integration of visual and proprioceptive signals in order to construct the body space.

Multimodal Integration and Vividness in the Angular Gyrus During Episodic Encoding and Retrieval

Much evidence suggests that the angular gyrus (AnG) is involved in episodic memory, but its precise role has yet to be determined. We examined two possible accounts within the same experimental paradigm: the “cortical binding of relational activity” (CoBRA) account (Shimamura, 2011), which suggests that the AnG acts as a convergence zone that binds multimodal episodic features, and the subjectivity account (Yazar et al., 2012), which implicates AnG involvement in subjective mnemonic experience (such as vividness or confidence). fMRI was used during both encoding and retrieval of paired associates. During study, female and male human participants memorized picture-pairs of common objects (in the unimodal task) or of an object-picture and an environmental sound (in the crossmodal task). At test, they performed a cued-recall task and further indicated the vividness of their memory. During retrieval, BOLD activation in the AnG was greatest for vividly remembered associates, consistent with the subjectivity account. During encoding, the same effect of vividness was found, but this was further modulated by task: greater activations were associated with subsequent recall in the crossmodal than the unimodal task. Therefore, encoding data suggest an additional role to the AnG in crossmodal integration, consistent with its role at retrieval proposed by CoBRA. These results resolve some of the puzzles in the literature and indicate that the AnG can play different roles during encoding and retrieval as determined by the cognitive demands posed by different mnemonic tasks.

SIGNIFICANCE STATEMENT We offer new insights into the multiplicity of processes that are associated with angular gyrus (AnG) activation during encoding and retrieval of newly formed memories. We used fMRI while human participants learned and subsequently recalled pairs of objects presented to the same sensory modality or to different modalities. We were able to show that the AnG is involved when vivid memories are created and retrieved, as well as when encoded information is integrated across different sensory modalities. These findings provide novel evidence for the contribution of the AnG to our subjective experience of remembering alongside its role in integrative processes that promote subsequent memory.

Influence of Stimulus Intensity on Multimodal Integration in the Startle Escape System of Goldfish

Processing of multimodal information is essential for an organism to respond to environmental events. However, how multimodal integration in neurons translates into behavior is far from clear. Here, we investigate integration of biologically relevant visual and auditory information in the goldfish startle escape system in which paired Mauthner-cells (M-cells) initiate the behavior. Sound pips and visual looms as well as multimodal combinations of these stimuli were tested for their effectiveness of evoking the startle response. Results showed that adding a low intensity sound early during a visual loom (low visual effectiveness) produced a supralinear increase in startle responsiveness as compared to an increase expected from a linear summation of the two unimodal stimuli. In contrast, adding a sound pip late during the loom (high visual effectiveness) increased responsiveness consistent with a linear multimodal integration of the two stimuli. Together the results confirm the Inverse Effectiveness Principle (IEP) of multimodal integration proposed in other species. Given the well-established role of the M-cell as a multimodal integrator, these results suggest that IEP is computed in individual neurons that initiate vital behavioral decisions.

Multimodal Integration and Phenomenal Spatiotemporal Binding: A Perspective From the Default Space Theory

How does the integrated and unified conscious experience arise from the vastly distributed activities of the nervous system? How is the information from the many cones of the retina bound with information coming from the cochlea to create the association of sounds with objects in visual space? In this perspective article, we assert a novel viewpoint on the "binding problem" in which we explain a metastable operation of the brain and body that may provide insight into this problem. In our view which is a component of the Default Space Theory (DST), consciousness arises from a metastable synchronization of local computations into a global coherence by a framework of widespread slow and ultraslow oscillations coordinated by the thalamus. We reinforce a notion shared by some consciousness researchers such as Revonsuo and the Fingelkurts that a spatiotemporal matrix is the foundation of phenomenological experience and that this phenomenology is directly tied to bioelectric operations of the nervous system. Through the oscillatory binding system we describe, cognitive neuroscientists may be able to more accurately correlate bioelectric activity of the brain and body with the phenomenology of human experience.

Effect of Visual Information on Active Touch During Mirror Visual Feedback

Several studies have demonstrated that observation of a dummy or mirror-reflected hand being stroked or moving at the same time as the hidden hand evokes a feeling that the dummy hand is one’s own, such as the rubber hand illusion (RHI) and mirror visual feedback (MVF). Under these conditions, participants also report sensing the tactile stimulation applied to the fake hands, suggesting that tactile perception is modulated by visual information during the RHI and MVF. Previous studies have utilized passive stimulation conditions; however, active touch is more common in real-world settings. Therefore, we investigated whether active touch is also modulated by visual information during an MVF scenario. Twenty-three participants (13 men and 10 women; mean age ± SD: 21.6 ± 2.0 years) were required to touch a polyurethane pad with both hands synchronously, and estimate the hardness of the pad while observing the mirror reflection. When participants observed the mirror reflection of the other hand pushing a softer or harder pad, perceived hardness estimates were significantly biased toward softer or harder, respectively, even though the physical hardness of the pad remained constant. Furthermore, perceived hardness exhibited a strong correlation with finger displacement of the mirrored, but not hidden, hand. The modulatory effects on perceived hardness diminished when participants touched the pad with both hands asynchronously or with their eyes closed. Moreover, participants experienced ownership of the mirrored hand when they touched the pad with both hands synchronously but not asynchronously. These results indicate that hardness estimates were modulated by observation of the mirrored hand during synchronous touch conditions. The present study demonstrates that, similar to passive touch, active touch is also modulated by visual input.

Empathic Cognitions Affected by Undetectable Social Chemosignals: An EEG Study on Visually Evoked Empathy for Pain in an Auditory and Chemosensory Context

Reduction of mu activity within the EEG is an indicator of cognitive empathy and can be generated in response to visual depictions of others in pain. The current study tested whether this brain response can be modulated by an auditory and a chemosensory context. Participants observed pictures of painful and non-painful actions while pain associated and neutral exclamations were presented (Study 1, N = 30) or while chemosensory stimuli were presented via a constant flow olfactometer (Study 2, N = 22). Chemosensory stimuli were sampled on cotton pads while donors participated in a simulated job interview (stress condition) or cycled on a stationary bike (sport condition). Pure cotton was used as a control. The social chemosignals could not be detected as odors. Activity within the 8–13 Hz band at electrodes C3, C4 (mu activity) and electrodes O1, O2 (alpha-activity) was calculated using Fast-Fourier-Transformation (FFT). As expected, suppression of power in the 8–13 Hz band was stronger when painful as compared to non-painful actions were observed (Study 1, p = 0.020; Study 2, p = 0.005). In addition, as compared to the neutral auditory and chemosensory context, painful exclamations (Study 1, p = 0.039) and chemosensory stress signals (Study 2, p = 0.014) augmented mu-/alpha suppression also in response to non-painful pictures. The studies show that processing of social threat-related information is not dominated by visual information. Rather, cognitive appraisal related to empathy can be affected by painful exclamations and subthreshold chemosensory social information.

Multiplex core-periphery organization of the human connectome

What is the core of the human brain is a fundamental question that has been mainly addressed by studying the anatomical connections between differently specialized areas, thus neglecting the possible contributions from their functional interactions. While many methods are available to identify the core of a network when connections between nodes are all of the same type, a principled approach to define the core when multiple types of connectivity are allowed is still lacking. Here, we introduce a general framework to define and extract the core-periphery structure of multi-layer networks by explicitly taking into account the connectivity patterns at each layer. We first validate our algorithm on synthetic networks of different size and density, and with tunable overlap between the cores at different layers. We then use our method to merge information from structural and functional brain networks, obtaining in this way an integrated description of the core of the human connectome. Results confirm the role of the main known cortical and subcortical hubs, but also suggest the presence of new areas in the sensori-motor cortex that are crucial for intrinsic brain functioning. Taken together these findings provide fresh evidence on a fundamental question in modern neuroscience and offer new opportunities to explore the mesoscale properties of multimodal brain networks.

Long-Range, Border-Crossing, Horizontal Axon Radiations Are a Common Feature of Rat Neocortical Regions That Differ in Cytoarchitecture

Employing wide-field optical imaging techniques supported by electrophysiological recordings, previous studies have demonstrated that stimulation of a spatially restricted area (point) in the sensory periphery results in a large evoked neuronal activity spread in mammalian primary cortices. In rats' primary cortices, such large evoked spreads extend diffusely in multiple directions, cross cortical cytoarchitectural borders and can trespass into other unimodal sensory areas. These point spreads are supported by a spatially matching, diffuse set of long-range horizontal projections within gray matter that extend in multiple directions and cross borders to interconnect different cortical areas. This horizontal projection system is in addition to well-known area-to-area clustered projections to defined targets through white matter. Could similar two-projection cortical systems also be found in cortical regions that differ in their cytoarchitectural structure? To address this question, an adeno-associated viral vector expressing green fluorescent protein (GFP) was injected as an anterograde tract tracer into granular somatosensory cortex (trunk area), dysgranular cortex (somatosensory dysgranular zone and extrastriate cortex) and agranular motor cortex (MCx). Irrespective of the injection site the same two projection systems were found, and their quantification revealed a close similarity to findings in primary sensory cortices. Following detailed reconstruction, the diffuse horizontal axon radiation was found to possess numerous varicosities and to include short, medium and long axons, the latter extending up to 5.2 mm. These "proof of concept" findings suggest that the similarity of the two projection systems among different cortical areas could potentially constitute a canonical motif of neocortical organization.

Potters Make Shorter Pots Under Conditions of Reduced Sensory Input

Although people have made clay pots for millennia, little behavioral research has explored how they do so. We were specifically interested in potters' use of auditory, haptic, and visual feedback. We asked what would happen if one or two of these sources of feedback were removed and potters tried to create pots of a given height, stopping when they thought they had reached that height. We asked students in a pottery class to build simple clay vessels either when they had full sensory feedback (in the control condition for all participants) or when they had reduced input from one modality (in Experiment 1) or two modalities (in Experiment 2). Participants were asked to stop building the vessels when they thought the vessels were 5 in. high. We found that participants produced shorter vessels when one or more forms of sensory feedback was reduced. The degree of shortening did not depend on the type or number of reduced sensory channels. The results are consistent with a control hypothesis where potters must have learned how to use sensory feedback from the modalities to help them control their ceramic creations. The results help highlight the importance of the intimate connections between perception and action.

Hearing and seeing meaning in noise: Alpha, beta, and gamma oscillations predict gestural enhancement of degraded speech comprehension

During face‐to‐face communication, listeners integrate speech with gestures. The semantic information conveyed by iconic gestures (e.g., a drinking gesture) can aid speech comprehension in adverse listening conditions. In this magnetoencephalography (MEG) study, we investigated the spatiotemporal neural oscillatory activity associated with gestural enhancement of degraded speech comprehension. Participants watched videos of an actress uttering clear or degraded speech, accompanied by a gesture or not and completed a cued‐recall task after watching every video. When gestures semantically disambiguated degraded speech comprehension, an alpha and beta power suppression and a gamma power increase revealed engagement and active processing in the hand‐area of the motor cortex, the extended language network (LIFG/pSTS/STG/MTG), medial temporal lobe, and occipital regions. These observed low‐ and high‐frequency oscillatory modulations in these areas support general unification, integration and lexical access processes during online language comprehension, and simulation of and increased visual attention to manual gestures over time. All individual oscillatory power modulations associated with gestural enhancement of degraded speech comprehension predicted a listener's correct disambiguation of the degraded verb after watching the videos. Our results thus go beyond the previously proposed role of oscillatory dynamics in unimodal degraded speech comprehension and provide first evidence for the role of low‐ and high‐frequency oscillations in predicting the integration of auditory and visual information at a semantic level.

Supralinear and Supramodal Integration of Visual and Tactile Signals in Rats: Psychophysics and Neuronal Mechanisms

To better understand how object recognition can be triggered independently of the sensory channel through which information is acquired, we devised a task in which rats judged the orientation of a raised, black and white grating. They learned to recognize two categories of orientation: 0° ± 45° (“horizontal”) and 90° ± 45° (“vertical”). Each trial required a visual (V), a tactile (T), or a visual-tactile (VT) discrimination; VT performance was better than that predicted by optimal linear combination of V and T signals, indicating synergy between sensory channels. We examined posterior parietal cortex (PPC) and uncovered key neuronal correlates of the behavioral findings: PPC carried both graded information about object orientation and categorical information about the rat’s upcoming choice; single neurons exhibited identical responses under the three modality conditions. Finally, a linear classifier of neuronal population firing replicated the behavioral findings. Taken together, these findings suggest that PPC is involved in the supramodal processing of shape.

•

Rats combine vision and touch to distinguish two grating orientation categories

•

Performance with vision and touch together reveals synergy between the two channels

•

Posterior parietal cortex (PPC) neuronal responses are invariant to modality

•

PPC neurons carry information about object orientation and the rat’s categorization