Multimodal fall detection for solitary individuals based on audio-video decision fusion processing

Falls often pose significant safety risks to solitary individuals, especially the elderly. Implementing a fast and efficient fall detection system is an effective strategy to address this hidden danger. We propose a multimodal method based on audio and video. On the basis of using non-intrusive equipment, it reduces to a certain extent the false negative situation that the most commonly used video-based methods may face due to insufficient lighting conditions, exceeding the monitoring range, etc. Therefore, in the foreseeable future, methods based on audio and video fusion are expected to become the best solution for fall detection. Specifically, this article outlines the following methodology: the video-based model utilizes YOLOv7-Pose to extract key skeleton joints, which are then fed into a two stream Spatial Temporal Graph Convolutional Network (ST-GCN) for classification. Meanwhile, the audio-based model employs log-scaled mel spectrograms to capture different features, which are processed through the MobileNetV2 architecture for detection. The final decision fusion of the two results is achieved through linear weighting and Dempster-Shafer (D-S) theory. After evaluation, our multimodal fall detection method significantly outperforms the single modality method, especially the evaluation metric sensitivity increased from 81.67% in single video modality to 96.67% (linear weighting) and 97.50% (D-S theory), which emphasizing the effectiveness of integrating video and audio data to achieve more powerful and reliable fall detection in complex and diverse daily life environments.

A computational pipeline towards large-scale and multiscale modeling of traumatic axonal injury

Contemporary biomechanical modeling of traumatic brain injury (TBI) focuses on either the global brain as an organ or a representative tiny section of a single axon. In addition, while it is common for a global brain model to employ real-world impacts as input, axonal injury models have largely been limited to inputs of either tension or compression with assumed peak strain and strain rate. These major gaps between global and microscale modeling preclude a systematic and mechanistic investigation of how tissue strain from impact leads to downstream axonal damage throughout the white matter. In this study, a unique subject-specific multimodality dataset from a male ice-hockey player sustaining a diagnosed concussion is used to establish an efficient and scalable computational pipeline. It is then employed to derive voxelized brain deformation, maximum principal strains and white matter fiber strains, and finally, to produce diverse fiber strain profiles of various shapes in temporal history necessary for the development and application of a deep learning axonal injury model in the future. The pipeline employs a structured, voxelized representation of brain deformation with adjustable spatial resolution independent of model mesh resolution. The method can be easily extended to other head impacts or individuals. The framework established in this work is critical for enabling large-scale (i.e., across the entire white matter region, head impacts, and individuals) and multiscale (i.e., from organ to cell length scales) modeling for the investigation of traumatic axonal injury (TAI) triggering mechanisms. Ultimately, these efforts could enhance the assessment of concussion risks and design of protective headgear. Therefore, this work contributes to improved strategies for concussion detection, mitigation, and prevention.

Chiral high-performance liquid chromatography analysis of mono-, di-, and triacylglycerols with amylose- and cellulose-phenylcarbamate-based stationary phases

The ever-increasing technological advancement in the (ultra)high-performance liquid chromatography tandem (high-resolution) mass spectrometry platforms have largely contributed to steeply intensify the interest towards lipidomics research. However, mass spectrometers alone are unable to distinguish between enantiomers. This obstacle is especially evident in the case of glycerolipids analysis due the prochiral nature of glycerol. Until a couple of decades ago, the stereoselective analysis of triacylglycerols (TAGs) was performed on the end products generated either from their enzymatic or chemical hydrolysis, namely on mono- or diacyl-sn-glycerols (MAGs and DAGs, respectively). These were then mostly analyzed with Pirkle-type chiral stationary phases (CSPs) after dedicated multi-step derivatization procedures. One of the most significant drawbacks of these traditional methods for enantioselective TAGs analysis (actually of the produced MAGs and DAGs, often investigated as target species per se) was the difficulty to totally abolish the migration of fatty acyls between glycerol positions. This made difficult to control and keep unaltered the stereochemistry of the original molecules. Over the last two decades, it has been widely demonstrated that the enantioselective analysis of intact TAGs as well as of non-derivatized MAGs and DAGs can be efficiently obtained using polysaccharide-based CSPs incorporating either amylose- or cellulose-phenylcarbamate derivatives chiral selectors. In this paper, the enantioselective methods developed with these CSPs for the enantioselective direct LC analysis of MAGs, DAGs and TAGs embedding different types of fatty acid residues are comprehensively reviewed.

Testing anxiety and reward processing in anorexia nervosa as predictors of longitudinal clinical outcomes

Anorexia nervosa (AN) is a psychiatric disorder with a tenuous longitudinal course marked by a high risk of relapse. Previous studies suggest that aberrant threat perception and reward processing operate in many with AN, and may produce obstacles to treatment engagement; therefore, these could potentially represent predictors for longitudinal clinical outcomes. In this study, anxiety and reward symptoms, behaviors, and neural circuit connectivity were measured in intensively treated AN-restrictive subtype patients (n = 33) and healthy controls (n = 31). Participants underwent an fMRI experiment using a monetary reward task in combination with either overlapping individually tailored anxiety-provoking words or neutral words. Behavioral/psychometric measures consisted of reaction times on the monetary reward task and self-ratings on anxiety symptoms at study entry. We tested multimodal, multivariate models based on neural, behavioral, and psychometric measures of reward and anxiety to predict physiological (Body Mass Index; BMI) and psychological (eating disorder symptom severity) longitudinal outcomes in AN over six months. Our results indicated that higher anxiety symptom psychometric scores significantly predicted BMI reductions at follow-up. Untreated anxiety after intensive treatment could put individuals with AN at heightened risk for weight loss. This represents a potentially modifiable risk factor that could be targeted more aggressively to help reduce the chance of future clinical worsening.

Autism Is Associated With Interindividual Variations of Gray and White Matter Morphology

BACKGROUND

Although many studies have explored atypicalities in gray matter (GM) and white matter (WM) morphology of autism, most of them relied on unimodal analyses that did not benefit from the likelihood that different imaging modalities may reflect common neurobiology. We aimed to establish brain patterns of modalities that differentiate between individuals with and without autism and explore associations between these brain patterns and clinical measures in the autism group.

METHODS

We studied 183 individuals with autism and 157 nonautistic individuals (age range, 6-30 years) in a large, deeply phenotyped autism dataset (EU-AIMS LEAP [European Autism Interventions-A Multicentre Study for Developing New Medications Longitudinal European Autism Project]). Linked independent component analysis was used to link all participants' GM volume and WM diffusion tensor images, and group comparisons of modality shared variances were examined. Subsequently, we performed univariate and multivariate brain-behavior correlation analyses to separately explore the relationships between brain patterns and clinical profiles.

RESULTS

One multimodal pattern was significantly related to autism. This pattern was primarily associated with GM volume in bilateral insula and frontal, precentral and postcentral, cingulate, and caudate areas and co-occurred with altered WM features in the superior longitudinal fasciculus. The brain-behavior correlation analyses showed a significant multivariate association primarily between brain patterns that involved variation of WM and symptoms of restricted and repetitive behavior in the autism group.

CONCLUSIONS

Our findings demonstrate the assets of integrated analyses of GM and WM alterations to study the brain mechanisms that underpin autism and show that the complex clinical autism phenotype can be interpreted by brain covariation patterns that are spread across the brain involving both cortical and subcortical areas.

Predicting survival of Iranian COVID-19 patients infected by various variants including omicron from CT Scan images and clinical data using deep neural networks

Purpose: The rapid spread of the COVID-19 omicron variant virus has resulted in an overload of hospitals around the globe. As a result, many patients are deprived of hospital facilities, increasing mortality rates. Therefore, mortality rates can be reduced by efficiently assigning facilities to higher-risk patients. Therefore, it is crucial to estimate patients' survival probability based on their conditions at the time of admission so that the minimum required facilities can be provided, allowing more opportunities to be available for those who need them. Although radiologic findings in chest computerized tomography scans show various patterns, considering the individual risk factors and other underlying diseases, it is difficult to predict patient prognosis through routine clinical or statistical analysis. Method: In this study, a deep neural network model is proposed for predicting survival based on simple clinical features, blood tests, axial computerized tomography scan images of lungs, and the patients' planned treatment. The model's architecture combines a Convolutional Neural Network and a Long Short Term Memory network. The model was trained using 390 survivors and 108 deceased patients from the Rasoul Akram Hospital and evaluated 109 surviving and 36 deceased patients infected by the omicron variant. Results: The proposed model reached an accuracy of 87.5% on the test data, indicating survival prediction possibility. The accuracy was significantly higher than the accuracy achieved by classical machine learning methods without considering computerized tomography scan images (p-value <= 4E-5). The images were also replaced with hand-crafted features related to the ratio of infected lung lobes used in classical machine-learning models. The highest-performing model reached an accuracy of 84.5%, which was considerably higher than the models trained on mere clinical information (p-value <= 0.006). However, the performance was still significantly less than the deep model (p-value <= 0.016). Conclusion: The proposed deep model achieved a higher accuracy than classical machine learning methods trained on features other than computerized tomography scan images. This proves the images contain extra information. Meanwhile, Artificial Intelligence methods with multimodal inputs can be more reliable and accurate than computerized tomography severity scores.

Multimodal assessment of brain fluid clearance is associated with amyloid-beta deposition in humans

PURPOSE

The present study investigates a multimodal imaging assessment of glymphatic function and its association with brain amyloid-beta deposition.

METHODS

Two brain CSF clearance measures (vCSF and DTI-ALPS) were derived from dynamic PET and MR diffusion tensor imaging (DTI) for 50 subjects, 24/50 were Aβ positive (Aβ+). T1W, T2W, DTI, T2FLAIR, and 11C-PiB and 18F-MK-6240 PET were acquired. Multivariate linear regression models were assessed with both vCSF and DTI-ALPS as independent variables and brain Aβ as the dependent variable. Three types of models were evaluated, including the vCSF-only model, the ALPS-only model and the vCSF+ALPS combined model. Models were applied to the whole group, and Aβ subgroups. All analyses were controlled for age, gender, and intracranial volume.

RESULTS

Sample demographics (N=50) include 20 males and 30 females with a mean age of 69.30 (sd=8.55). Our results show that the combination of vCSF and ALPS associates with Aβ deposition (p < 0.05, R2 = 0.575) better than either vCSF (p < 0.05, R2 = 0.431) or ALPS (p < 0.05, R2 = 0.372) alone in the Aβ+ group. We observed similar results in whole-group analyses (combined model: p < 0.05, R2 = 0.287; vCSF model: p <0.05, R2 = 0.175; ALPS model: p < 0.05, R2 = 0.196) with less significance. Our data also showed that vCSF has higher correlation (r = -0.548) in subjects with mild Aβ deposition and DTI-ALPS has higher correlation (r=-0.451) with severe Aβ deposition subjects.

CONCLUSION

The regression model with both vCSF and DTI-ALPS is better associated with brain Aβ deposition. These two independent brain clearance measures may better explain the variation in Aβ deposition than either term individually. Our results suggest that vCSF and DTI-ALPS reflect complementary aspects of brain clearance functions.

Copy Number Variation Informs fMRI-based Prediction of Autism Spectrum Disorder

The multifactorial etiology of autism spectrum disorder (ASD) suggests that its study would benefit greatly from multimodal approaches that combine data from widely varying platforms, e.g., neuroimaging, genetics, and clinical characterization. Prior neuroimaging-genetic analyses often apply naive feature concatenation approaches in data-driven work or use the findings from one modality to guide posthoc analysis of another, missing the opportunity to analyze the paired multimodal data in a truly unified approach. In this paper, we develop a more integrative model for combining genetic, demographic, and neuroimaging data. Inspired by the influence of genotype on phenotype, we propose using an attention-based approach where the genetic data guides attention to neuroimaging features of importance for model prediction. The genetic data is derived from copy number variation parameters, while the neuroimaging data is from functional magnetic resonance imaging. We evaluate the proposed approach on ASD classification and severity prediction tasks, using a sex-balanced dataset of 228 ASD and typically developing subjects in a 10-fold cross-validation framework. We demonstrate that our attention-based model combining genetic information, demographic data, and functional magnetic resonance imaging results in superior prediction performance compared to other multimodal approaches.

Integrative deep learning analysis improves colon adenocarcinoma patient stratification at risk for mortality

BACKGROUND

Colorectal cancers are the fourth most diagnosed cancer and the second leading cancer in number of deaths. Many clinical variables, pathological features, and genomic signatures are associated with patient risk, but reliable patient stratification in the clinic remains a challenging task. Here we assess how image, clinical, and genomic features can be combined to predict risk.

METHODS

We developed and evaluated integrative deep learning models combining formalin-fixed, paraffin-embedded (FFPE) whole slide images (WSIs), clinical variables, and mutation signatures to stratify colon adenocarcinoma (COAD) patients based on their risk of mortality. Our models were trained using a dataset of 108 patients from The Cancer Genome Atlas (TCGA), and were externally validated on newly generated dataset from Wayne State University (WSU) of 123 COAD patients and rectal adenocarcinoma (READ) patients in TCGA (N = 52).

FINDINGS

We first observe that deep learning models trained on FFPE WSIs of TCGA-COAD separate high-risk (OS < 3 years, N = 38) and low-risk (OS > 5 years, N = 25) patients (AUC = 0.81 ± 0.08, 5 year survival p < 0.0001, 5 year relative risk = 1.83 ± 0.04) though such models are less effective at predicting overall survival (OS) for moderate-risk (3 years < OS < 5 years, N = 45) patients (5 year survival p-value = 0.5, 5 year relative risk = 1.05 ± 0.09). We find that our integrative models combining WSIs, clinical variables, and mutation signatures can improve patient stratification for moderate-risk patients (5 year survival p < 0.0001, 5 year relative risk = 1.87 ± 0.07). Our integrative model combining image and clinical variables is also effective on an independent pathology dataset (WSU-COAD, N = 123) generated by our team (5 year survival p < 0.0001, 5 year relative risk = 1.52 ± 0.08), and the TCGA-READ data (5 year survival p < 0.0001, 5 year relative risk = 1.18 ± 0.17). Our multicenter integrative image and clinical model trained on combined TCGA-COAD and WSU-COAD is effective in predicting risk on TCGA-READ (5 year survival p < 0.0001, 5 year relative risk = 1.82 ± 0.13) data. Pathologist review of image-based heatmaps suggests that nuclear size pleomorphism, intense cellularity, and abnormal structures are associated with high-risk, while low-risk regions have more regular and small cells. Quantitative analysis shows high cellularity, high ratios of tumor cells, large tumor nuclei, and low immune infiltration are indicators of high-risk tiles.

INTERPRETATION

The improved stratification of colorectal cancer patients from our computational methods can be beneficial for treatment plans and enrollment of patients in clinical trials.

FUNDING

This study was supported by the National Cancer Institutes (Grant No. R01CA230031 and P30CA034196). The funders had no roles in study design, data collection and analysis or preparation of the manuscript.

Interaction and psychological characteristics of art teaching based on Openpose and Long Short-Term Memory network

As living standards improve, people's demand for appreciation and learning of art is growing gradually. Unlike the traditional learning model, art teaching requires a specific understanding of learners' psychology and controlling what they have learned so that they can create new ideas. This article combines the current deep learning technology with heart rate to complete the action recognition of art dance teaching. The video data processing and recognition are conducted through the Openpose network and graph convolution network. The heart rate data recognition is completed through the Long Short-Term Memory (LSTM) network. The optimal recognition model is established through the data fusion of the two decision levels through the adaptive weight analysis method. The experimental results show that the accuracy of the classification fusion model is better than that of the single-mode recognition method, which is improved from 85.0% to 97.5%. The proposed method can evaluate the heart rate while ensuring high accuracy recognition. The proposed research can help analyze dance teaching and provide a new idea for future combined research on teaching interaction.

Multimodal-neuroimaging machine-learning analysis of motor disability in multiple sclerosis

Motor disability is a dominant and restricting symptom in multiple sclerosis, yet its neuroimaging correlates are not fully understood. We apply statistical and machine learning techniques on multimodal neuroimaging data to discriminate between multiple sclerosis patients and healthy controls and to predict motor disability scores in the patients. We examine the data of sixty-four multiple sclerosis patients and sixty-five controls, who underwent the MRI examination and the evaluation of motor disability scales. The modalities used comprised regional fractional anisotropy, regional grey matter volumes, and functional connectivity. For analysis, we employ two approaches: high-dimensional support vector machines run on features selected by Fisher Score (aiming for maximal classification accuracy), and low-dimensional logistic regression on the principal components of data (aiming for increased interpretability). We apply analogous regression methods to predict symptom severity. While fractional anisotropy provides the classification accuracy of 96.1% and 89.9% with both approaches respectively, including other modalities did not bring further improvement. Concerning the prediction of motor impairment, the low-dimensional approach performed more reliably. The first grey matter volume component was significantly correlated (R = 0.28-0.46, p < 0.05) with most clinical scales. In summary, we identified the relationship between both white and grey matter changes and motor impairment in multiple sclerosis. Furthermore, we were able to achieve the highest classification accuracy based on quantitative MRI measures of tissue integrity between patients and controls yet reported, while also providing a low-dimensional classification approach with comparable results, paving the way to interpretable machine learning models of brain changes in multiple sclerosis.

Single-cell spatial explorer: easy exploration of spatial and multimodal transcriptomics

BACKGROUND

The development of single-cell technologies yields large datasets of information as diverse and multimodal as transcriptomes, immunophenotypes, and spatial position from tissue sections in the so-called 'spatial transcriptomics'. Currently however, user-friendly, powerful, and free algorithmic tools for straightforward analysis of spatial transcriptomic datasets are scarce.

RESULTS

Here, we introduce Single-Cell Spatial Explorer, an open-source software for multimodal exploration of spatial transcriptomics, examplified with 9 human and murine tissues datasets from 4 different technologies.

CONCLUSIONS

Single-Cell Spatial Explorer is a very powerful, versatile, and interoperable tool for spatial transcriptomics analysis.

Transformer-based personalized attention mechanism for medical images with clinical records

In medical image diagnosis, identifying the attention region, i.e., the region of interest for which the diagnosis is made, is an important task. Various methods have been developed to automatically identify target regions from given medical images. However, in actual medical practice, the diagnosis is made based on both the images and various clinical records. Consequently, pathologists examine medical images with prior knowledge of the patients and the attention regions may change depending on the clinical records. In this study, we propose a method, called the Personalized Attention Mechanism (PersAM) method, by which the attention regions in medical images according to the clinical records. The primary idea underlying the PersAM method is the encoding of the relationships between medical images and clinical records using a variant of the Transformer architecture. To demonstrate the effectiveness of the PersAM method, we applied it to a large-scale digital pathology problem involving identifying the subtypes of 842 malignant lymphoma patients based on their gigapixel whole-slide images and clinical records.

Exploring EFL Learners' Metaphorical Conceptions of Language Learning: A Multimodal Analysis

Owing to the limitations of linguistic modes to portray aptly L2 learners' metaphors of language learning experience, growing attention has been paid to taking advantage of other modes like visual ones to ameliorate this concern. Hence, the present study sought to explore images and metaphors Iranian EFL learners may have in mind about the essence of English language as a foreign learning. To this end, Iranian EFL learners' verbal and non-verbal forms of metaphorical depiction were examined. One intact class, including intermediate male and female learners (n = 11) at a non-profit language institute was selected randomly. The data were collected through both verbal (a single-item questionnaire) and non-verbal (drawings) tools. The learners' drawings and written descriptions were examined so as to both tap into their mental representations of what 'English learning' means to them and get closer insights into the learners' belief system. The study's conceptual framework was mainly built on Oxford et al.'s (System 26:3-50, 1998) perspectives on education and Vygotsky's (Mind in society: the development of higher psychological processes, Harvard University Press, 1978) Socio-cultural theory (SCT) of learning. The multimodal analysis of the metaphors evidenced that the learners' verbal and visual metaphorical representations shared the common perspectives towards English learning. Additionally, the extracted metaphorical concepts disclosed the learners' positive attitudes, enthusiasm, and desire for English learning. Results unveiled that most of the visual and verbal metaphorical depictions portray language learning as a joyful, dynamic and discovery individual process. The study ends with presenting some implications and offering some avenues for further research.

Displacement voxelization to resolve mesh-image mismatch: Application in deriving dense white matter fiber strains

BACKGROUND AND OBJECTIVE

It is common to combine biomechanical modeling and medical images for multimodal analyses. However, mesh-image mismatch may occur that prevents direct information exchange. To eliminate mesh-image mismatch, we develop a simple but elegant displacement voxelization technique based on image voxel corner nodes to achieve voxel-wise strain. We then apply the technique to derive dense white matter fiber strains along whole-brain tractography (∼35 k fiber tracts consisting of ∼3.3 million sampling points) resulting from head impact.

METHODS

Displacements at image voxel corner nodes are first obtained from model simulation via scattered interpolation. Each voxel is then scaled linearly to form a unit hexahedral element. This allows convenient and efficient voxel-wise strain tensor calculation and displacement interpolation at arbitrary fiber sampling points via shape functions. Fiber strains from displacement interpolation are then compared with those from the commonly used strain tensor projection using either voxel- or element-wise strain tensors.

RESULTS

Based on a synthetic displacement field, fiber strains interpolated from voxelized displacement are considerably more accurate than those from strain tensor projection relative to the prescribed ground-truth (determinant of coefficient (R²) of 1.00 and root mean squared error (RMSE) of 0.01 vs. 0.87 and 0.10, respectively). For a set of real-world reconstructed head impacts (N = 53), the strain tensor projection method performs similarly poorly (R² of 0.80-0.90 and RMSE of 0.03-0.07), with overestimation strongly correlated with strain magnitude (Pearson correlation coefficient >0.9). Up to ∼15% of the fiber strains are overestimated by more than the lower bound of a conservative injury threshold of 0.09. The percentage increases to ∼37% when halving the threshold. Voxel interpolation is also significantly more efficient (15 s vs. 40 s for element strain tensor projection, without parallelization).

CONCLUSIONS

Voxelized displacement interpolation is considerably more accurate and efficient in deriving dense white matter fiber strains than strain tensor projection. The latter generally overestimates with overestimation magnitude strongly correlating with fiber strain magnitude. Displacement voxelization is an effective technique to eliminate mesh-image mismatch and generates a convenient image representation of tissue deformation. This technique can be generalized to broadly facilitate a diverse range of image-related biomechanical problems for multimodal analyses. The convenient image format may also promote and facilitate biomechanical data sharing in the future.

Use of multimodal analysis in the diagnosis and follow-up of a case of Acute Zonal Occult Outer Retinopathy (AZOOR)

A clinical case is presented in order to show the usefulness of multimodal analysis in the diagnosis and monitoring of patients with Acute Zonal Occult Outer Retinopathy (AZOOR). A 22 year-old patient was seen in the emergency department complaining of photopsia and paracentral scotoma of the left eye. Several structural and functional tests were performed and the patient was diagnosed with AZOOR. The evolution of the case was towards an initial structural worsening, followed by the almost complete resolution of the lesions identified in the different tests carried out, with an obvious symptomatic improvement. Multimodal analysis of AZOOR cases allows a fairly accurate diagnosis of this condition, and its differentiation from others with a similar appearance, such as multiple white point syndromes, or multifocal choroiditis.

Video remote interpreting in clinical communication: A multimodal analysis

OBJECTIVE

Investigating how the spatial and audiovisual conditions in video remote interpreting (VRI) shape communicative interaction in a language-discordant clinical consultation.

METHODS

We conducted a multimodal analysis of an authentic VRI-mediated consultation with special reference to spatial arrangements, audiovisual conditions, and the healthcare professional's use of embodied communicative resources (body orientation, eye gaze, gestures).

RESULTS

The physician is found to pursue his communicative goals for the consultation by first creating an appropriate spatial and technical environment and then supporting his information-giving and relationship-building actions through the use of nonverbal (embodied) resources like body orientation, gaze and gestures as well as specific turn-management behaviour.

CONCLUSION

VRI allows healthcare professionals to access professional interpreters for language-discordant consultations but requires appropriate technical and spatial arrangements as well as users capable of adapting their communicative behaviour to spatial and audiovisual constraints.

PRACTICE IMPLICATIONS

Alongside telephone interpreting, VRI is the solution of choice for language-discordant clinical encounters in times of the Covid-19 pandemic. Its use requires appropriate technical and spatial arrangements as well as specific skills on the part of healthcare professionals to cope with inherent audiovisual constraints.

Multimodal ex vivo methods reveal that Gd-rich corrosion byproducts remain at the implant site of biodegradable Mg-Gd screws

Extensive research is being conducted on magnesium (Mg) alloys for bone implant manufacturing, due to their biocompatibility, biodegradability and mechanical properties. Gadolinium (Gd) is among the most promising alloying elements for property control in Mg alloy implants; however, its toxicity is controversial. Investigating Gd behavior during implant corrosion is thus of utmost importance. In this study, we analyzed the degradation byproducts at the implant site of biodegradable Mg-5Gd and Mg-10Gd implants after 12 weeks healing time, using a combination of different imaging techniques: histology, energy-dispersive x-ray spectroscopy (EDX), x-ray microcomputed tomography (µCT) and neutron µCT. The main finding has been that, at the healing time in exam, the corrosion appears to have involved only the Mg component, which has been substituted by calcium and phosphorus, while the Gd remains localized at the implant site. This was observed in 2D by means of EDX maps and extended to 3D with a novel application of neutron tomography. X-ray fluorescence analysis of the main excretory organs also did not reveal any measurable accumulation of Gd, further reinforcing the conclusion that very limited or no removal at all of Gd-alloy happened during degradation. STATEMENT OF SIGNIFICANCE: Gadolinium is among the most promising alloying elements for property control in biodegradable magnesium alloy implants, but its toxicity is controversial and its behavior during corrosion needs to be investigated. We combine 2D energy dispersive x-ray spectroscopy and 3D neutron and x-ray tomography to image the degradation of magnesium-gadolinium implants after 12 weeks of healing time. We find that, at the time in exam, the corrosion has involved only the magnesium component, while the gadolinium remains localized at the implant site. X-ray fluorescence analysis of the main excretory organs also does not reveal any measurable accumulation of Gd, further reinforcing the conclusion that very limited or no removal at all of Gd-alloy has happened during degradation.

Generalized tonic-clonic seizures are accompanied by changes of interrelations within the autonomic nervous system

PURPOSE

A seizure is a strong central stimulus that affects multiple subsystems of the autonomic nervous system (ANS), and results in different interactions across ANS modalities. Here, we aimed to evaluate whether multimodal peripheral ANS measures demonstrate interactions before and after seizures as compared to controls to provide the basis for seizure detection and forecasting based on peripheral ANS signals.

METHODS

Continuous electrodermal activity (EDA), heart rate (HR), peripheral body temperature (TEMP), and respiratory rate (RR) calculated based on blood volume pulse were acquired by a wireless multi-sensor device. We selected 45 min of preictal and 60 min of postictal data and time-matched segments for controls. Data were analyzed over 15-min windows. For unimodal analysis, mean values over each time window were calculated for all modalities and analyzed by Friedman's two-way analysis of variance.

RESULTS

Twenty-one children with recorded generalized tonic-clonic seizures (GTCS), and 21 age- and gender-matched controls were included. Unimodal results revealed no significant effect for RR and TEMP, but EDA (p = 0.002) and HR (p < 0.001) were elevated 0-15 min after seizures. The averaged bimodal correlation across all pairs of modalities changed for 15-min windows in patients with seizures. The highest correlations were observed immediately before (0.85) and the lowest correlation immediately after seizures. Overall, average correlations for controls were higher.

SIGNIFICANCE

Multimodal ANS changes related to GTCS occur within and across autonomic nervous system modalities. While unimodal changes were most prominent during postictal segments, bimodal correlations increased before seizures and decreased postictally. This offers a promising avenue for further research on seizure detection, and potentially risk assessment for seizure recurrence and sudden unexplained death in epilepsy.

Multimodal tensor-based method for integrative and continuous patient monitoring during postoperative cardiac care

Patients recovering from cardiovascular surgeries may develop life-threatening complications such as hemodynamic decompensation, making the monitoring of patients for such complications an essential component of postoperative care. However, this need has given rise to an inexorable increase in the number and modalities of data points collected, making it challenging to effectively analyze in real time. While many algorithms exist to assist in monitoring these patients, they often lack accuracy and specificity, leading to alarm fatigue among healthcare practitioners. In this study we propose a multimodal approach that incorporates salient physiological signals and EHR data to predict the onset of hemodynamic decompensation. A retrospective dataset of patients recovering from cardiac surgery was created and used to train predictive models. Advanced signal processing techniques were employed to extract complex features from physiological waveforms, while a novel tensor-based dimensionality reduction method was used to reduce the size of the feature space. These methods were evaluated for predicting the onset of decompensation at varying time intervals, ranging from a half-hour to 12 h prior to a decompensation event. The best performing models achieved AUCs of 0.87 and 0.80 for the half-hour and 12-h intervals respectively. These analyses evince that a multimodal approach can be used to develop clinical decision support systems that predict adverse events several hours in advance.

Use of multimodal analysis in the diagnosis and follow-up of a case of Acute Zonal Occult Outer Retinopathy (AZOOR)

A clinical case is presented in order to show the usefulness of multimodal analysis in the diagnosis and monitoring of patients with Acute Zonal Occult Outer Retinopathy (AZOOR). A 22 year-old patient was seen in the emergency department complaining of photopsia and paracentral scotoma of the left eye. Several structural and functional tests were performed and the patient was diagnosed with AZOOR. The evolution of the case was towards an initial structural worsening, followed by the almost complete resolution of the lesions identified in the different tests carried out, with an obvious symptomatic improvement. Multimodal analysis of AZOOR cases allows a fairly accurate diagnosis of this condition, and its differentiation from others with a similar appearance, such as multiple white point syndromes, or multifocal choroiditis.

More than words: how multimodal analysis can inform health professions education

The contexts and methods for communicating in healthcare and health professions education (HPE) profoundly affect how we understand information, relate to others, and construct our identities. Multimodal analysis provides a method for exploring how we communicate using multiple modes-e.g., language, gestures, images-in concert with each other and within specific contexts. In this paper, we demonstrate how multimodal analysis helps us investigate the ways our communication practices shape healthcare and HPE. We provide an overview of the theoretical underpinnings, traditions, and methodologies of multimodal analysis. Then, we illustrate how to design and conduct a study using one particular approach to multimodal analysis, multimodal (inter)action analysis, using examples from a study focused on clinical reasoning and patient documentation. Finally, we suggest how multimodal analysis can be used to address a variety of HPE topics and contexts, highlighting the unique contributions multimodal analysis can offer to our field.

Multimodal analysis using [11C]PiB-PET/MRI for functional evaluation of patients with Alzheimer's disease

BACKGROUND

Multimodal PET/MRI image data simultaneously obtained from patients with early-stage of Alzheimer's disease (eAD) were assessed in order to observe pathophysiologic and functional changes, as well as alterations of morphology and connectivity in the brain. Fifty-eight patients with mild cognitive impairment and early dementia (29 males, 69 ± 12 years) underwent [¹¹C]Pittsburgh compound-B (PiB) PET/MRI with 70-min PET and MRI scans. Sixteen age-matched healthy controls (CTL) (9 males, 68 ± 11 years) were also studied with the same scanning protocol. Cerebral blood flow (CBF) was calculated from the early phase PET images using the image-derived input function method. A standardized uptake value ratio (SUVr) was calculated from 50 to 70 min PET data with a reference region of the cerebellar cortex. MR images such as 3D-T1WI, resting-state functional MRI (RS-fMRI), diffusion tensor image (DTI), and perfusion MRI acquired during the dynamic PET scan were also analyzed to evaluate various brain functions on MRI.

RESULTS

Twenty-seven of the 58 patients were determined as eAD based on the results of PiB-PET and clinical findings, and a total of 43 subjects' data including CTL were analyzed in this study. PiB SUVr values in all cortical regions of eAD were significantly greater than those of CTL. The PiB accumulation intensity was negatively correlated with cognitive scores. The regional PET-CBF values of eAD were significantly lower in the bilateral parietal lobes and right temporal lobe compared with CTL, but not in MRI perfusion; however, SPM showed regional differences on both PET- and MRI-CBF. SPM analysis of RS-fMRI delineated regional differences between the groups in the anterior cingulate cortex and the left precuneus. VBM analysis showed atrophic changes in the AD group in a part of the bilateral hippocampus; however, analysis of fractional anisotropy calculated from DTI data did not show differences between the two groups.

CONCLUSION

Multimodal analysis conducted with various image data from PiB-PET/MRI scans showed differences in regional CBF, cortical volume, and neuronal networks in different regions, indicating that pathophysiologic and functional changes in the AD brain can be observed from various aspects of neurophysiologic parameters. Application of multimodal brain images using PET/MRI would be ideal for investigating pathophysiologic changes in patients with dementia and other neurodegenerative diseases.

A framework for linking resting-state chronnectome/genome features in schizophrenia: A pilot study

Multimodal, imaging-genomics techniques offer a platform for understanding genetic influences on brain abnormalities in psychiatric disorders. Such approaches utilize the information available from both imaging and genomics data and identify their association. Particularly for complex disorders such as schizophrenia, the relationship between imaging and genomic features may be better understood by incorporating additional information provided by advanced multimodal modeling. In this study, we propose a novel framework to combine features corresponding to functional magnetic resonance imaging (functional) and single nucleotide polymorphism (SNP) data from 61 schizophrenia (SZ) patients and 87 healthy controls (HC). In particular, the features for the functional and genetic modalities include dynamic (i.e., time-varying) functional network connectivity (dFNC) features and the SNP data, respectively. The dFNC features are estimated from component time-courses, obtained using group independent component analysis (ICA), by computing sliding-window functional network connectivity, and then estimating subject specific states from this dFNC data using a k-means clustering approach. For each subject, both the functional (dFNC states) and SNP data are selected as features for a parallel ICA (pICA) based imaging-genomic framework. This analysis identified a significant association between a SNP component (defined by large clusters of functionally related SNPs statistically correlated with phenotype components) and time-varying or dFNC component (defined by clusters of related connectivity links among distant brain regions distributed across discrete dynamic states, and statistically correlated with genomic components) in schizophrenia. Importantly, the polygenetic risk score (PRS) for SZ (computed as a linearly weighted sum of the genotype profiles with weights derived from the odds ratios of the psychiatric genomics consortium (PGC)) was negatively correlated with the significant dFNC component, which were mostly present within a state that exhibited a lower occupancy rate in individuals with SZ compared with HC, hence identifying a potential dFNC imaging biomarker for schizophrenia. Taken together, the current findings provide preliminary evidence for a link between dFNC measures and genetic risk, suggesting the application of dFNC patterns as biomarkers in imaging genetic association study.

Measuring acute stress response through physiological signals: towards a quantitative assessment of stress

Social and medical problems associated with stress are increasing globally and seriously affect mental health and well-being. However, an effective stress-level monitoring method is still not available. This paper presents a quantitative method for monitoring acute stress levels in healthy young people using biomarkers from physiological signals that can be unobtrusively monitored. Two states were induced to 40 volunteers, a basal state generated with a relaxation task and an acute stress state generated by applying a standard stress test that includes five different tasks. Standard psychological questionnaires and biochemical markers were utilized as ground truth of stress levels. A multivariable approach to comprehensively measure the physiological stress response is proposed using stress biomarkers derived from skin temperature, heart rate, and pulse wave signals. Acute physiological stress levels (total-range 0-100 au) were continuously estimated every 1 min showing medians of 29.06 au in the relaxation tasks, while rising from 34.58 to 47.55 au in the stress tasks. Moreover, using the proposed method, five statistically different stress levels induced by the performed tasks were also measured. Results obtained show that, in these experimental conditions, stress can be monitored from unobtrusive biomarkers. Thus, a more general stress monitoring method could be derived based on this approach. Graphical abstract Stress measurements of different healthy young people throughout a Stress Session that includes a pre-relax stage (BL_s), memory test (ST and MT), stress anticipation time (SA), video display (VD) and arithmetic task.

Integration of transcriptomic and cytoarchitectonic data implicates a role for MAOA and TAC1 in the limbic-cortical network

Decoding the chain from genes to cognition requires detailed insights how areas with specific gene activities and microanatomical architectures contribute to brain function and dysfunction. The Allen Human Brain Atlas contains regional gene expression data, while the JuBrain Atlas offers three-dimensional cytoarchitectonic maps reflecting interindividual variability. To date, an integrated framework that combines the analytical benefits of both scientific platforms towards a multi-level brain atlas of adult humans was not available. We have, therefore, developed JuGEx, a new method for integrating tissue transcriptome and cytoarchitectonic segregation. We investigated differential gene expression in two JuBrain areas of the frontal pole that we have structurally and functionally characterized in previous studies. Our results show a significant upregulation of MAOA and TAC1 in the medial area frontopolaris which is a node in the limbic-cortical network and known to be susceptible for gray matter loss and behavioral dysfunction in patients with depression. The MAOA gene encodes an enzyme which is involved in the catabolism of dopamine, norepinephrine, serotonin, and other monoaminergic neurotransmitters. The TAC1 locus generates hormones that play a role in neuron excitations and behavioral responses. Overall, JuGEx provides a new tool for the scientific community that empowers research from basic, cognitive and clinical neuroscience in brain regions and disease models with regard to gene expression.

[Contribution of multimodal imaging in the various stages of Stargardt disease]

PURPOSE

To describe the contribution of multimodal imaging in the various stages of Stargardt disease (STGD).

PATIENTS AND METHODS

We retrospectively reviewed 46 eyes of 23 STGD patients with identified ABCA4 mutations. All patients underwent a complete ophthalmic examination, spectral-domain optical coherence tomography (SD-OCT), fundus autofluorescence (FAF), fluorescein angiography (FA) and Indocyanine green angiography (ICGA).

RESULTS

The mean age of patients was 25.5 years (range 8-56). Fundus examination was normal in 2 patients (subclinical stage), where SD-OCT showed localized retrofoveolar retinal pigment epithelium (RPE) thickening. FAF was normal in 1 eye and showed mild heterogeneous hyper-FAF in 3 eyes. Twelve eyes had mild salt and pepper changes in the macula (early stage) with diffuse retinal atrophy on SD-OCT and mixed hyper and hypoautofluorescence on FAF. Nine patients showed central atrophy with white-yellow flecks distributed in the posterior pole and mid-periphery. This phenotype showed total foveal atrophy on SD-OCT and normal peripapillary area on FAF. Twelve eyes had a large demarcated area of RPE atrophy, pigment clumping and migration extending to the peripheral retina associated with peripapillary atrophy. These eyes showed diffuse retinochoroidal atrophy on OCT with diffuse alterations reaching the peripapillary area on FAF. On FA, it was difficult to analyze the choroidal silence sign in patients with advanced stages of the disease. A hyperfluorescent window defect pattern was also found in patients with white-yellow flecks and did not correspond exactly to them, or to the areas of peripheral autofluorescent lesions. ICGA showed hypocyanescent areas seen at intermediate and late phases with multiple cyanescent points adjacent to them. On ICGA, hypocyanescent areas were more extensive than lesions observed on FAF.

CONCLUSIONS

Multimodal imaging is helpful for the diagnosis of early stages of STGD disease and to better understand its pathophysiology. FAF and mostly SD-OCT have supplanted FA in the early, especially subclinical, stages. Over all, ICGA shows more extensive damage, making this tool useful for better understanding STGD and suggesting possible direct damage to the choriocapillaris associated with RPE lesions. In advanced stages, only DNA testing can confirm the diagnosis of STGD.

Characterization of brain blood flow and the amplitude of low-frequency fluctuations in major depressive disorder: A multimodal meta-analysis

BACKGROUND

In healthy subjects, there is an association between amplitude of low-frequency fluctuations (ALFF) and regional cerebral blood flow (rCBF). To date, no published meta-analysis has investigated changes in the regional ALFF in medication-free depressed patients.

METHODS

In this study, we aimed to explore whether resting-state rCBF and ALFF changes co-occur in the depressed brain without the potential confound of medication. Using signed differential mapping (SDM), we conducted two meta-analyses, one of rCBF studies and one of ALFF studies, involving medication-free patients with major depressive disorder (MDD). In addition, we conducted a multimodal meta-analysis to identify brain regions that showed abnormalities in both rCBF and ALFF.

RESULTS

A total of 16 studies were included in this series. We identified abnormalities in resting-state rCBF and ALFF in the left insula in medication-free MDD patients compared with healthy controls (HC). In addition, we observed altered resting-state rCBF in the limbic-subcortical-cortical circuit and altered ALFF in the default mode network (DMN) and some motor-related brain regions.

LIMITATIONS

The analysis techniques, patient characteristics and clinical variables of the included studies were heterogeneous.

CONCLUSIONS

The conjoint alterations in ALFF and rCBF in the left insula may represent core neuropathological changes in medication-free patients with MDD and merit further studying.

Multimodal analysis of startle type responses

BACKGROUND AND OBJECTIVE

This article presents a multimodal analysis of startle type responses using a variety of physiological, facial, and speech features. These multimodal components of the startle type response reflect complex brain-body reactions to a sudden and intense stimulus. Additionally, the proposed multimodal evaluation of reflexive and emotional reactions associated with the startle eliciting stimuli and underlying neural networks and pathways could be applied in diagnostics of different psychiatric and neurological diseases. Different startle type stimuli can be compared in the strength of their elicitation of startle responses, i.e. their potential to activate stress-related neural pathways, underlying biomarkers and corresponding behavioral reactions.

METHODS

An innovative method for measuring startle type responses using multimodal stimuli and multimodal feature analysis has been introduced. Individual's multimodal reflexive and emotional expressions during startle type elicitation have been assessed by corresponding physiological, speech and facial features on ten female students of psychology. Different startle eliciting stimuli like noise and airblast probes, as well as a variety of visual and auditory stimuli of different valence and arousal levels, based on International Affective Picture System (IAPS) images and/or sounds from International Affective Digitized Sounds (IADS) database, have been designed and tested. Combined together into more complex startle type stimuli, such composite stimuli can potentiate the evoked response of underlying neural networks, and corresponding neurotransmitters and neuromodulators as well; this is referred to as increased power of response elicitation. The intensity and magnitude of multimodal responses to selected startle type stimuli have been analyzed using effect sizes and medians of dominant multimodal features, i.e. skin conductance, eye blink, head movement, speech fundamental frequency and energy. The significance of the observed effects and comparisons between paradigms were evaluated using one-tailed t-tests and ANOVA methods, respectively. Skin conductance response habituation was analyzed using ANOVA and post hoc multiple comparison tests with the Dunn-Šidák correction.

RESULTS

The results revealed specific physiological, facial and vocal reflexive and emotional responses on selected five stimuli paradigms which included: (1) acoustic startle probes, (2) airblasts, (3) IAPS images, (4) IADS sounds, and (5) image-sound-airblast composite stimuli. Overall, composite and airblast paradigms resulted in the largest responses across all analyzed features, followed by sound and acoustic startle paradigms, while paradigm using images consistently elicited the smallest responses. In this context, power of response elicitation of the selected stimuli paradigms can be described according to the aggregated magnitude of the participants' multimodal responses. We also observed a habituation effect only in skin conductance response to acoustic startle, airblast and sound paradigms.

CONCLUSIONS

This study developed a system for paradigm design and stimuli generation, as well as real-time multimodal signal processing and feature calculation. Experimental paradigms for monitoring individual responses to stressful startle type stimuli were designed in order to compare the response elicitation power across various stimuli. The developed system, applied paradigms and obtained results might be useful in further research for evaluation of individuals' multimodal responses when they are faced with a variety of aversive emotional distractors and stressful situations.

Integrated analysis of gray and white matter alterations in attention-deficit/hyperactivity disorder

BACKGROUND

Magnetic resonance imaging (MRI) is able to provide detailed insights into the structural organization of the brain, e.g., by means of mapping brain anatomy and white matter microstructure. Understanding interrelations between MRI modalities, rather than mapping modalities in isolation, will contribute to unraveling the complex neural mechanisms associated with neuropsychiatric disorders as deficits detected across modalities suggest common underlying mechanisms. Here, we conduct a multimodal analysis of structural MRI modalities in the context of attention-deficit/hyperactivity disorder (ADHD).

METHODS

Gray matter volume, cortical thickness, surface areal expansion estimates, and white matter diffusion indices of 129 participants with ADHD and 204 participants without ADHD were entered into a linked independent component analysis. This data-driven analysis decomposes the data into multimodal independent components reflecting common inter-subject variation across imaging modalities.

RESULTS

ADHD severity was related to two multimodal components. The first component revealed smaller prefrontal volumes in participants with more symptoms, co-occurring with abnormal white matter indices in prefrontal cortex. The second component demonstrated decreased orbitofrontal volume as well as abnormalities in insula, occipital, and somato-sensory areas in participants with more ADHD symptoms.

CONCLUSIONS

Our results replicate and extend previous unimodal structural MRI findings by demonstrating that prefrontal, parietal, and occipital areas, as well as fronto-striatal and fronto-limbic systems are implicated in ADHD. By including multiple modalities, sensitivity for between-participant effects is increased, as shared variance across modalities is modeled. The convergence of modality-specific findings in our results suggests that different aspects of brain structure share underlying pathophysiology and brings us closer to a biological characterization of ADHD.

Morphometric hemispheric asymmetry of orbitofrontal cortex in women with borderline personality disorder: a multi-parameter approach

Functional imaging studies have implicated the orbitofrontal cortex (OFC) in the pathophysiology of borderline personality disorder (BPD). To date, however, volume-based magnetic resonance imaging (MRI) studies have yielded mixed results. We used a surface-based processing approach that allowed us to measure five morphometric cortical features of the OFC, including volumetric (cortical thickness and surface area) and geometric (mean curvature, depth of sulcus, and metric distortion - three indicators of cortical folding) parameters. Participants comprised 25 female BPD patients with no other current psychiatric comorbidity and 25 age- and gender-matched healthy controls who received structural MRI scans. Images were processed using the Freesurfer package. All BPD patients had a history of comorbid psychiatric disorder(s) and were currently on medications. Compared with controls, the BPD group showed reduced cortical thickness, surface area, mean curvature, depth of sulcus, and metric distortion in the right medial OFC. In the left medial OFC, the BPD group had reduced cortical thickness and mean curvature, but increased metric distortion. This study confirmed the utility of surface-based analysis in the study of BPD cortical structures. In addition, we observed extensive structural abnormalities in the medial OFC of female subjects with BPD, findings that were most pronounced in the right OFC, with preliminary data suggesting hemispheric asymmetry.