Deep Multimodal Learning From MRI and Clinical Data for Early Prediction of Neurodevelopmental Deficits in Very Preterm Infants

Supervised contrastive learning enhances graph convolutional networks for predicting neurodevelopmental deficits in very preterm infants using brain structural connectome

Very preterm (VPT) infants (born at less than 32 weeks gestational age) are at high risk for various adverse neurodevelopmental deficits. Unfortunately, most of these deficits cannot be accurately diagnosed until the age of 2-5 years old. Given the benefits of early interventions, accurate diagnosis and prediction soon after birth are urgently needed for VPT infants. Previous studies have applied deep learning models to learn the brain structural connectome (SC) to predict neurodevelopmental deficits in the preterm population. However, none of these models are specifically designed for graph-structured data, and thus may potentially miss certain topological information conveyed in the brain SC. In this study, we aim to develop deep learning models to learn the SC acquired at term-equivalent age for early prediction of neurodevelopmental deficits at 2 years corrected age in VPT infants. We directly treated the brain SC as a graph, and applied graph convolutional network (GCN) models to capture complex topological information of the SC. In addition, we applied the supervised contrastive learning (SCL) technique to mitigate the effects of the data scarcity problem, and enable robust training of GCN models. We hypothesize that SCL will enhance GCN models for early prediction of neurodevelopmental deficits in VPT infants using the SC. We used a regional prospective cohort of ∼280 VPT infants who underwent MRI examinations at term-equivalent age from the Cincinnati Infant Neurodevelopment Early Prediction Study (CINEPS). These VPT infants completed neurodevelopmental assessment at 2 years corrected age to evaluate cognition, language, and motor skills. Using the SCL technique, the GCN model achieved mean areas under the receiver operating characteristic curve (AUCs) in the range of 0.72∼0.75 for predicting three neurodevelopmental deficits, outperforming several competing models. Our results support our hypothesis that the SCL technique is able to enhance the GCN model in our prediction tasks.

Predicting 2-year neurodevelopmental outcomes in preterm infants using multimodal structural brain magnetic resonance imaging with local connectivity

The neurodevelopmental outcomes of preterm infants can be stratified based on the level of prematurity. We explored brain structural networks in extremely preterm (EP; < 28 weeks of gestation) and very-to-late (V-LP; ≥ 28 and < 37 weeks of gestation) preterm infants at term-equivalent age to predict 2-year neurodevelopmental outcomes. Using MRI and diffusion MRI on 62 EP and 131 V-LP infants, we built a multimodal feature set for volumetric and structural network analysis. We employed linear and nonlinear machine learning models to predict the Bayley Scales of Infant and Toddler Development, Third Edition (BSID-III) scores, assessing predictive accuracy and feature importance. Our findings revealed that models incorporating local connectivity features demonstrated high predictive performance for BSID-III subsets in preterm infants. Specifically, for cognitive scores in preterm (variance explained, 17%) and V-LP infants (variance explained, 17%), and for motor scores in EP infants (variance explained, 15%), models with local connectivity features outperformed others. Additionally, a model using only local connectivity features effectively predicted language scores in preterm infants (variance explained, 15%). This study underscores the value of multimodal feature sets, particularly local connectivity, in predicting neurodevelopmental outcomes, highlighting the utility of machine learning in understanding microstructural changes and their implications for early intervention.

A novel generative adversarial networks modelling for the class imbalance problem in high dimensional omics data

Class imbalance remains a large problem in high-throughput omics analyses, causing bias towards the over-represented class when training machine learning-based classifiers. Oversampling is a common method used to balance classes, allowing for better generalization of the training data. More naive approaches can introduce other biases into the data, being especially sensitive to inaccuracies in the training data, a problem considering the characteristically noisy data obtained in healthcare. This is especially a problem with high-dimensional data. A generative adversarial network-based method is proposed for creating synthetic samples from small, high-dimensional data, to improve upon other more naive generative approaches. The method was compared with 'synthetic minority over-sampling technique' (SMOTE) and 'random oversampling' (RO). Generative methods were validated by training classifiers on the balanced data.

Corpus Callosum Abnormalities at Term-Equivalent Age Are Associated with Language Development at 2 Years' Corrected Age in Infants Born Very Preterm

We studied the effect of microstructural abnormalities in the corpus callosum on language development in 348 infants born very prematurely. We discovered that the fractional anisotropy of the corpus callosum anterior midbody was a significant predictor of standardized language scores at 2 years, independent of clinical and social risk factors.

Global trends in neonatal MRI brain neuroimaging research over the last decade: a bibliometric analysis

Background

Neuroimaging plays a central role in the evaluation, treatment, and prognosis of neonates. In recent years, the exploration of the developing brain has been a major focus of research for researchers and clinicians. In this study, we conducted bibliometric and visualization analyses of the related studies in the field of neonatal magnetic resonance imaging (MRI) brain neuroimaging from the past 10 years, and summarized its research status, hotspots, and frontier development trends.

Methods

The Web of Science core collection database was used as the literature source from which to retrieve the relevant papers and reviews in the field of neonatal MRI brain neuroimaging published in the Science Citation Index-Expanded from 2013 to 2022. VOSviewer and CiteSpace were used to conduct bibliometric and visualization analyses of the annual publication volume, countries, institutions, journals, authors, co-cited literature, and the overall distribution of keywords.

Results

We retrieved 3,568 papers and reviews published from 2013 to 2022. The number of publications increased during this period. The United States (US) and the United Kingdom were the largest contributors, with the US receiving the highest H-index and number of citations. The institutions that published the most were the University of London and Harvard University. The research mainly focused on cerebral cortex, brain tissue, brain structure network, artificial intelligence algorithm, automatic image segmentation, and premature infants.

Conclusions

This study reveals the research status and hotspots of magnetic resonance imaging in the field of neonatal brain neuroimaging in the past decade, which helps researchers to better understand the research status, hotspots, and frontier development trends.

Efficacy of MRI data harmonization in the age of machine learning: a multicenter study across 36 datasets

Pooling publicly-available MRI data from multiple sites allows to assemble extensive groups of subjects, increase statistical power, and promote data reuse with machine learning techniques. The harmonization of multicenter data is necessary to reduce the confounding effect associated with non-biological sources of variability in the data. However, when applied to the entire dataset before machine learning, the harmonization leads to data leakage, because information outside the training set may affect model building, and potentially falsely overestimate performance. We propose a 1) measurement of the efficacy of data harmonization; 2) harmonizer transformer, i.e., an implementation of the ComBat harmonization allowing its encapsulation among the preprocessing steps of a machine learning pipeline, avoiding data leakage by design. We tested these tools using brain T1-weighted MRI data from 1740 healthy subjects acquired at 36 sites. After harmonization, the site effect was removed or reduced, and we showed the data leakage effect in predicting individual age from MRI data, highlighting that introducing the harmonizer transformer into a machine learning pipeline allows for avoiding data leakage by design.

The Quartile Levels of Thyroid-Stimulating Hormone by Newborn Screening Stratified Risks of Neurodevelopmental Impairment in Extremely Preterm Infants - A Population Cohort Study

BACKGROUND

To evaluate whether thyroid-stimulating hormone (TSH) by newborn screening (NBS) at birth and at discharge can be surrogate markers for neurodevelopmental impairment (NDI) in extremely preterm infants.

METHODS

The population cohort enrolled infants born <29 weeks' gestation in 2008 - 2020 in southern Taiwan. Infants with a maternal history of thyroid disorders and infants who required thyroxine supplementation during hospitalization were excluded. TSH levels by NBS at birth and at term-equivalent age (TEA)/discharge were respectively categorized into the lowest quartile, the interquartile range, and the highest quartile, which were correlated to NDI outcomes.

RESULTS

Among 392 patients with paired TSH data, 358 (91%) were prospectively followed until corrected age 24 months. At birth, infants with lowest-quartile TSH had higher NDI risks (OR 2.3, 95% CI 1.3 - 4.1, P = 0.004) compared to infants with interquartile-range TSH. Conversely, by TEA/discharge, infants with highest-quartile TSH had increased NDI (OR 1.9, 1.0 - 3.4, P = 0.03). By paired TSH categories, infants persistently in the lowest TSH quartile (48%, aOR 4.4, 1.4 - 14.5, P = 0.01) and those with a shift from interquartile range to the highest quartile (32%, aOR 2.7, 1.0 - 7.4, P = 0.046) had increased NDI risks compared with the reference with consistent interquartile-range TSH.

CONCLUSIONS

Extremely preterm infants persistently in the lowest-quartile TSH level at birth and at discharge had the highest NDI risk. TSH quartile levels by NBS may serve as a population surrogate biomarker for assessing NDI risks in infants born extremely preterm.

Corpus callosum abnormalities at term-equivalent age are associated with language development at two years corrected age in infants born very preterm

We studied the impact of microstructural abnormalities in the corpus callosum on language development in 348 infants born very prematurely. We discovered that the fractional anisotropy of the corpus callosum anterior midbody was a significant predictor of standardized language scores at two years, independent of clinical and social risk factors.

A Semi-Supervised Graph Convolutional Network for Early Prediction of Motor Abnormalities in Very Preterm Infants

Approximately 32-42% of very preterm infants develop minor motor abnormalities. Earlier diagnosis soon after birth is urgently needed because the first two years of life represent a critical window of opportunity for early neuroplasticity in infants. In this study, we developed a semi-supervised graph convolutional network (GCN) model that is able to simultaneously learn the neuroimaging features of subjects and consider the pairwise similarity between them. The semi-supervised GCN model also allows us to combine labeled data with additional unlabeled data to facilitate model training. We conducted our experiments on a multisite regional cohort of 224 preterm infants (119 labeled subjects and 105 unlabeled subjects) who were born at 32 weeks or earlier from the Cincinnati Infant Neurodevelopment Early Prediction Study. A weighted loss function was applied to mitigate the impact of an imbalanced positive:negative (~1:2) subject ratio in our cohort. With only labeled data, our GCN model achieved an accuracy of 66.4% and an AUC of 0.67 in the early prediction of motor abnormalities, outperforming prior supervised learning models. By taking advantage of additional unlabeled data, the GCN model had significantly better accuracy (68.0%, p = 0.016) and a higher AUC (0.69, p = 0.029). This pilot work suggests that the semi-supervised GCN model can be utilized to aid early prediction of neurodevelopmental deficits in preterm infants.

Multilayer perceptron-based self-care early prediction of children with disabilities

Early identification of children with self-care impairments is one of the key challenges professional therapists face due to the complex and time-consuming detection process using relevant self-care activities. Due to the complex nature of the problem, machine-learning methods have been widely applied in this area. In this study, a feed-forward artificial neural network (ANN)-based self-care prediction methodology, called multilayer perceptron (MLP)-progressive, has been proposed. The proposed methodology integrates unsupervised instance-based resampling and randomizing preprocessing techniques to MLP for improved early detection of self-care disabilities in children. Preprocessing of the dataset affects the MLP performance; hence, randomization and resampling of the dataset improves the performance of the MLP model. To confirm the usefulness of MLP-progressive, three experiments were conducted, including validating MLP-progressive methodology over multi-class and binary-class datasets, impact analysis of the proposed preprocessing filters on the model performance, and comparing the MLP-progressive results with state-of-the-art studies. The evaluation metrics accuracy, precision, recall, F-measure, TP rate, FP rate, and ROC were used to measure performance of the proposed disability detection model. The proposed MLP-progressive model outperforms existing methods and attains a classification accuracy of 97.14% and 98.57% on multi-class and binary-class datasets, respectively. Additionally, when evaluated on the multi-class dataset, significant improvements in accuracies ranging from 90.00% to 97.14% were observed when compared to state-of-the-art methods.

Machine learning for understanding and predicting neurodevelopmental outcomes in premature infants: a systematic review

BACKGROUND

Machine learning has been attracting increasing attention for use in healthcare applications, including neonatal medicine. One application for this tool is in understanding and predicting neurodevelopmental outcomes in preterm infants. In this study, we have carried out a systematic review to identify findings and challenges to date.

METHODS

This systematic review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis guidelines. Four databases were searched in February 2022, with articles then screened in a non-blinded manner by two authors.

RESULTS

The literature search returned 278 studies, with 11 meeting the eligibility criteria for inclusion. Convolutional neural networks were the most common machine learning approach, with most studies seeking to predict neurodevelopmental outcomes from images and connectomes describing brain structure and function. Studies to date also sought to identify features predictive of outcomes; however, results varied greatly.

CONCLUSIONS

Initial studies in this field have achieved promising results; however, many machine learning techniques remain to be explored, and the consensus is yet to be reached on which clinical and brain features are most predictive of neurodevelopmental outcomes.

IMPACT

This systematic review looks at the question of whether machine learning can be used to predict and understand neurodevelopmental outcomes in preterm infants. Our review finds that promising initial works have been conducted in this field, but many challenges and opportunities remain. Quality assessment of relevant articles is conducted using the Newcastle-Ottawa Scale. This work identifies challenges that remain and suggests several key directions for future research. To the best of the authors' knowledge, this is the first systematic review to explore this topic.

Diffuse excessive high signal intensity in the preterm brain on advanced MRI represents widespread neuropathology

Preterm brains commonly exhibit elevated signal intensity in the white matter on T2-weighted MRI at term-equivalent age. This signal, known as diffuse excessive high signal intensity (DEHSI) or diffuse white matter abnormality (DWMA) when quantitatively assessed, is associated with abnormal microstructure on diffusion tensor imaging. However, postmortem data are largely lacking and difficult to obtain, and the pathological significance of DEHSI remains in question. In a cohort of 202 infants born preterm at ≤32 weeks gestational age, we leveraged two newer diffusion MRI models - Constrained Spherical Deconvolution (CSD) and neurite orientation dispersion and density index (NODDI) - to better characterize the macro and microstructural properties of DWMA and inform the ongoing debate around the clinical significance of DWMA. With increasing DWMA volume, fiber density broadly decreased throughout the white matter and fiber cross-section decreased in the major sensorimotor tracts. Neurite orientation dispersion decreased in the centrum semiovale, corona radiata, and temporal lobe. These findings provide insight into DWMA's biological underpinnings and demonstrate that it is a serious pathology.

Artificial intelligence-based methods for fusion of electronic health records and imaging data

Healthcare data are inherently multimodal, including electronic health records (EHR), medical images, and multi-omics data. Combining these multimodal data sources contributes to a better understanding of human health and provides optimal personalized healthcare. The most important question when using multimodal data is how to fuse them-a field of growing interest among researchers. Advances in artificial intelligence (AI) technologies, particularly machine learning (ML), enable the fusion of these different data modalities to provide multimodal insights. To this end, in this scoping review, we focus on synthesizing and analyzing the literature that uses AI techniques to fuse multimodal medical data for different clinical applications. More specifically, we focus on studies that only fused EHR with medical imaging data to develop various AI methods for clinical applications. We present a comprehensive analysis of the various fusion strategies, the diseases and clinical outcomes for which multimodal fusion was used, the ML algorithms used to perform multimodal fusion for each clinical application, and the available multimodal medical datasets. We followed the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines. We searched Embase, PubMed, Scopus, and Google Scholar to retrieve relevant studies. After pre-processing and screening, we extracted data from 34 studies that fulfilled the inclusion criteria. We found that studies fusing imaging data with EHR are increasing and doubling from 2020 to 2021. In our analysis, a typical workflow was observed: feeding raw data, fusing different data modalities by applying conventional machine learning (ML) or deep learning (DL) algorithms, and finally, evaluating the multimodal fusion through clinical outcome predictions. Specifically, early fusion was the most used technique in most applications for multimodal learning (22 out of 34 studies). We found that multimodality fusion models outperformed traditional single-modality models for the same task. Disease diagnosis and prediction were the most common clinical outcomes (reported in 20 and 10 studies, respectively) from a clinical outcome perspective. Neurological disorders were the dominant category (16 studies). From an AI perspective, conventional ML models were the most used (19 studies), followed by DL models (16 studies). Multimodal data used in the included studies were mostly from private repositories (21 studies). Through this scoping review, we offer new insights for researchers interested in knowing the current state of knowledge within this research field.

A self-training deep neural network for early prediction of cognitive deficits in very preterm infants using brain functional connectome data

BACKGROUND

Deep learning has been employed using brain functional connectome data for evaluating the risk of cognitive deficits in very preterm infants. Although promising, training these deep learning models typically requires a large amount of labeled data, and labeled medical data are often very difficult and expensive to obtain.

OBJECTIVE

This study aimed to develop a self-training deep neural network (DNN) model for early prediction of cognitive deficits at 2 years of corrected age in very preterm infants (gestational age ≤32 weeks) using both labeled and unlabeled brain functional connectome data.

MATERIALS AND METHODS

We collected brain functional connectome data from 343 very preterm infants at a mean (standard deviation) postmenstrual age of 42.7 (2.5) weeks, among whom 103 children had a cognitive assessment at 2 years (i.e. labeled data), and the remaining 240 children had not received 2-year assessments at the time this study was conducted (i.e. unlabeled data). To develop a self-training DNN model, we built an initial student model using labeled brain functional connectome data. Then, we applied the trained model as a teacher model to generate pseudo-labels for unlabeled brain functional connectome data. Next, we combined labeled and pseudo-labeled data to train a new student model. We iterated this procedure to obtain the best student model for the early prediction task in very preterm infants.

RESULTS

In our cross-validation experiments, the proposed self-training DNN model achieved an accuracy of 71.0%, a specificity of 71.5%, a sensitivity of 70.4% and an area under the curve of 0.75, significantly outperforming transfer learning models through pre-training approaches.

CONCLUSION

We report the first self-training prognostic study in very preterm infants, efficiently utilizing a small amount of labeled data with a larger share of unlabeled data to aid the model training. The proposed technique is expected to facilitate deep learning with insufficient training data.