Multimodal machine learning in precision health: A scoping review

From challenges and pitfalls to recommendations and opportunities: Implementing federated learning in healthcare

Federated learning holds great potential for enabling large-scale healthcare research and collaboration across multiple centers while ensuring data privacy and security are not compromised. Although numerous recent studies suggest or utilize federated learning based methods in healthcare, it remains unclear which ones have potential clinical utility. This review paper considers and analyzes the most recent studies up to May 2024 that describe federated learning based methods in healthcare. After a thorough review, we find that the vast majority are not appropriate for clinical use due to their methodological flaws and/or underlying biases which include but are not limited to privacy concerns, generalization issues, and communication costs. As a result, the effectiveness of federated learning in healthcare is significantly compromised. To overcome these challenges, we provide recommendations and promising opportunities that might be implemented to resolve these problems and improve the quality of model development in federated learning with healthcare.

Biosensors integrated within wearable devices for monitoring chronic wound status

Slowly healing wounds significantly affect the life quality of patients in different ways, due to constant pain, unpleasant odor, reduced mobility up to social isolation, and personal frustration. While remote wound management has become more widely accepted since the COVID-19 pandemic, delayed treatment remains frequent and results in several wound healing related complications. As inappropriate management of notably diabetic foot ulcers is linked to a high risk of amputation, effective management of wounds in a patient-centered manner remains important to be implemented. The integration of diagnostic devices into wound bandages is under way, owing to advancements in materials science and nanofabrication strategies as well as innovation in communication technologies together with machine learning and data-driven assessment tools. Leveraging advanced analytical approaches around local pH, temperature, pressure, and wound biomarker sensing is expected to facilitate adequate wound treatment. The state-of-the-art of time-resolved monitoring of the wound status by quantifying key physiological parameters as well as wound biomarkers' concentration is presented herewith. A special focus will be given to smart bandages with on-demand delivery capabilities for improved wound management.

Robust multi-modal fusion architecture for medical data with knowledge distillation

BACKGROUND

The fusion of multi-modal data has been shown to significantly enhance the performance of deep learning models, particularly on medical data. However, missing modalities are common in medical data due to patient specificity, which poses a substantial challenge to the application of these models.

OBJECTIVE

This study aimed to develop a novel and efficient multi-modal fusion framework for medical datasets that maintains consistent performance, even in the absence of one or more modalities.

METHODS

In this paper, we fused three modalities: chest X-ray radiographs, history of present illness text, and tabular data such as demographics and laboratory tests. A multi-modal fusion module based on pooled bottleneck (PB) attention was proposed in conjunction with knowledge distillation (KD) for enhancing model inference in the case of missing modalities. In addition, we introduced a gradient modulation (GM) method to deal with the unbalanced optimization in multi-modal model training. Finally, we designed comparison and ablation experiments to evaluate the fusion effect, the model robustness to missing modalities, and the contribution of each component (PB, KD, and GM). The evaluation experiments were performed on the MIMIC-IV datasets with the task of predicting in-hospital mortality risk. Model performance was assessed using the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC).

RESULTS

The proposed multi-modal fusion framework achieved an AUROC of 0.886 and AUPRC of 0.459, significantly surpassing the performance of baseline models. Even when one or two modalities were missing, our model consistently outperformed the reference models. Ablation of each of the three components resulted in varying degrees of performance degradation, highlighting their distinct contributions to the model's overall effectiveness.

CONCLUSIONS

This innovative multi-modal fusion architecture has demonstrated robustness to missing modalities, and has shown excellent performance in fusing three medical modalities for patient outcome prediction. This study provides a novel idea for addressing the challenge of missing modalities and has the potential be scaled to additional modalities.

Missing-modality enabled multi-modal fusion architecture for medical data

BACKGROUND

Fusion of multi-modal data can improve the performance of deep learning models. However, missing modalities are common in medical data due to patient specificity, which is detrimental to the performance of multi-modal models in applications. Therefore, it is critical to adapt the models to missing modalities.

OBJECTIVE

This study aimed to develop an effective multi-modal fusion architecture for medical data that was robust to missing modalities and further improved the performance for clinical tasks.

METHODS

X-ray chest radiographs for the image modality, radiology reports for the text modality, and structured value data for the tabular data modality were fused in this study. Each modality pair was fused with a Transformer-based bi-modal fusion module, and the three bi-modal fusion modules were then combined into a tri-modal fusion framework. Additionally, multivariate loss functions were introduced into the training process to improve models' robustness to missing modalities during the inference process. Finally, we designed comparison and ablation experiments to validate the effectiveness of the fusion, the robustness to missing modalities, and the enhancements from each key component. Experiments were conducted on MIMIC-IV and MIMIC-CXR datasets with the 14-label disease diagnosis and patient in-hospital mortality prediction task The area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC) were used to evaluate models' performance.

RESULTS

Our proposed architecture showed superior predictive performance, achieving the average AUROC and AUPRC of 0.916 and 0.551 in the 14-label classification task, 0.816 and 0.392 in the mortality prediction task. while the best average AUROC and AUPRC among the comparison methods were 0.876, 0.492 in the 14-label classification task and 0.806, 0.366 in the mortality prediction task. Both metrics decreased only slightly when tested with modal-incomplete data. Different levels of enhancements were achieved through three key components.

CONCLUSIONS

The proposed multi-modal fusion architecture effectively fused three modalities and showed strong robustness to missing modalities. This architecture holds promise for scaling up to more modalities to enhance the clinical practicality of the model.

A New Method Using Deep Learning to Predict the Response to Cardiac Resynchronization Therapy

Clinical parameters measured from gated single-photon emission computed tomography myocardial perfusion imaging (SPECT MPI) have value in predicting cardiac resynchronization therapy (CRT) patient outcomes, but still show limitations. The purpose of this study is to combine clinical variables, features from electrocardiogram (ECG), and parameters from assessment of cardiac function with polar maps from gated SPECT MPI through deep learning (DL) to predict CRT response. A total of 218 patients who underwent rest-gated SPECT MPI were enrolled in this study. CRT response was defined as an increase in left ventricular ejection fraction (LVEF) > 5% at a 6-month follow-up. A DL model was constructed by combining a pre-trained VGG16 model and a multilayer perceptron. Two modalities of data were input to the model: polar map images from SPECT MPI and tabular data from clinical features, ECG parameters, and SPECT-MPI-derived parameters. Gradient-weighted class activation mapping (Grad-CAM) was applied to the VGG16 model to provide explainability for the polar maps. For comparison, four machine learning (ML) models were trained using only the tabular features. Modeling was performed on 218 patients who underwent CRT implantation with a response rate of 55.5% (n = 121). The DL model demonstrated average AUC (0.83), accuracy (0.73), sensitivity (0.76), and specificity (0.69) surpassing ML models and guideline criteria. Guideline recommendations achieved accuracy (0.53), sensitivity (0.75), and specificity (0.26). The DL model trended towards improvement over the ML models, showcasing the additional predictive benefit of utilizing SPECT MPI polar maps. Incorporating additional patient data directly in the form of medical imagery can improve CRT response prediction.

Data-Driven Early Prediction of Cerebral Palsy Using AutoML and interpretable kinematic features

Early identification of cerebral palsy (CP) remains a major challenge due to the reliance on expert assessments that are time-intensive and not scalable. Consequently, a range of studies have aimed at using machine learning to predict CP scores based on motion tracking,e.g. from video data. These studies generally predict clinical scores which are a proxy for CP risk. However, clinicians do not REALLY want to estimate scores, they want to estimate the patients' risk of developing clinical symptoms. Here we present a data-driven machine-learning (ML) pipeline that extracts movement features from infant video based motion tracking and estimates CP risk using AutoML. Using AutoSklearn, our framework minimizes risk of overfitting by abstracting away researcher-driver hyperparameter optimization. Trained on movement data from 3- to 4-month-old infants, our classifier predicts a highly indicative clinical score (General Movements Assessment [GMA]) with an ROC-AUC of 0.78 on a held-out test set, indicating that kinematic movement features capture clinically relevant variability. Without retraining, the same model predicts the risk of cerebral palsy outcomes at later clinical follow-ups with an ROC-AUC of 0.74, demonstrating that early motor representations generalize to long-term neurodevelopmental risk. We employ pre-registered lock-box validation to ensure rig-orous performance evaluation. This study highlights the potential of AutoML-powered movement analytics for neurodevelopmental screening, demonstrating that data-driven feature extraction from movement trajectories can provide an interpretable and scalable approach to early risk assessment. By integrating pre-trained vision transformers, AutoML-driven model selection, and rigorous validation protocols, this work advances the use of video-derived movement features for scalable, data-driven clinical assessment, demonstrating how computational methods based on readily available data like infant videos can enhance early risk detection in neurodevelopmental disorders.

CCS Concepts

Computing methodologies → Machine learning approaches ; Applied computing → Health informatics .

ACM Reference Format

Melanie Segado, Laura Prosser, Andrea F. Duncan, Michelle J. Johnson, and Konrad P. Kording.. Data-Driven Early Prediction of Cerebral Palsy Using AutoML and interpretable kinematic features. In. ACM, New York, NY, USA, 8 pages.

Developing Trustworthy Artificial Intelligence Models to Predict Vascular Disease Progression: the VASCUL-AID-RETRO Study Protocol

INTRODUCTION

Abdominal aortic aneurysms (AAAs) and peripheral artery disease (PAD) are two vascular diseases with a significant risk of major adverse cardiovascular events and mortality. A challenge in current disease management is the unpredictable disease progression in individual patients. The VASCUL-AID-RETRO study aims to develop trustworthy multimodal predictive artificial intelligence (AI) models for multiple tasks including risk stratification of disease progression and cardiovascular events in patients with AAA and PAD.

METHODS

The VASCUL-AID-RETRO study will collect data from 5000 AAA and 6000 PAD patients across multiple European centers of the VASCUL-AID consortium using electronic health records from 2015 to 2024. This retrospectively-collected data will be enriched with additional data from existing biobanks and registries. Multimodal data, including clinical records, radiological imaging, proteomics, and genomics, will be collected to develop AI models predicting disease progression and cardiovascular risks. This will be done while integrating the international ethics guidelines and legal standards for trustworthy AI, to ensure a socially-responsible data integration and analysis.

PROPOSED ANALYSES

A consensus-based variable list of clinical parameters and core outcome set for both diseases will be developed through meetings with key opinion leaders. Blood, plasma, and tissue samples from existing biobanks will be analyzed for proteomic and genomic variations. AI models will be trained on segmented AAA and PAD artery geometries for estimation of hemodynamic parameters to quantify disease progression. Initially, risk prediction models will be developed for each modality separately, and subsequently, all data will be combined to be used as input to multimodal prediction models. During all processes, data security, data quality, and ethical guidelines and legal standards will be carefully considered. As a next step, the developed models will be further adjusted with prospective data and internally validated in a prospective cohort (VASCUL-AID-PRO study).

CONCLUSION

The VASCUL-AID-RETRO study will utilize advanced AI techniques and integrate clinical, imaging, and multi-omics data to predict AAA and PAD progression and cardiovascular events.

CLINICAL TRIAL REGISTRATION

The VASCUL-AID-RETRO study is registered at www.clinicaltrials.gov under the identification number NCT06206369.

CLINICAL IMPACT

The VASCUL-AID-RETRO study aims to improve clinical practice of vascular surgery by developing artificial intelligence-driven multimodal predictive models for patients with abdominal aortic aneurysms or peripheral artery disease, enhancing personalized medicine. By integrating comprehensive data sets including clinical, imaging, and multi-omics data, these models have the potential to provide accurate risk stratification for disease progression and cardiovascular events. An innovation lies in the extensive European data set in combination with multimodal analyses approaches, which enables the development of advanced models to facilitate better understanding of disease mechanisms and progression. For clinicians, this means that more precise, individualized treatment plans can be established, ultimately aiming to improve patient outcomes.

A Deep-Learning Model for Multi-class Audio Classification of Vocal Fold Pathologies in Office Stroboscopy

OBJECTIVE

To develop and validate a deep-learning classifier trained on voice data extracted from videolaryngostroboscopy recordings, differentiating between three different vocal fold (VF) states: healthy (HVF), unilateral paralysis (UVFP), and VF lesions, including benign and malignant pathologies.

METHODS

Patients with UVFP (n = 105), VF lesions (n = 63), and HVF (n = 41) were retrospectively identified. Voice samples were extracted from stroboscopic videos (Pentax Laryngeal Strobe Model 9400), including sustained /i/ phonation, pitch glide, and /i/ sniff task. Extracted audio files were converted into Mel-spectrograms. Voice samples were independently divided into training (80%), validation (10%), and test (10%) by patient. Pretrained ResNet18 models were trained to classify (1) HVF and pathological VF (lesions and UVFP), and (2) HVF, UVFP, and VF lesions. Both classifiers were further validated on an external dataset consisting of 12 UVFP, 13 VF lesions, and 15 HVF patients. Model performances were evaluated by accuracy and F1-score.

RESULTS

When evaluated on a hold-out test set, the binary classifier demonstrated stronger performance compared to the multi-class classifier (accuracy 83% vs. 40%; F1-score 0.90 vs. 0.36). When evaluated on an external dataset, the binary classifier achieved an accuracy of 63% and F1-score of 0.48, compared to 35% and 0.25 for the multi-class classifier.

CONCLUSIONS

Deep-learning classifiers differentiating HVF, UVFP, and VF lesions were developed using voice data from stroboscopic videos. Although healthy and pathological voice were differentiated with moderate accuracy, multi-class classification lowered model performance. The model performed poorly on an external dataset. Voice captured in stroboscopic videos may have limited diagnostic value, though further studies are needed.

LEVEL OF EVIDENCE

4 Laryngoscope, 2025.

The Social Construction of Categorical Data: Mixed Methods Approach to Assessing Data Features in Publicly Available Datasets

BACKGROUND

In data-sparse areas such as health care, computer scientists aim to leverage as much available information as possible to increase the accuracy of their machine learning models' outputs. As a standard, categorical data, such as patients' gender, socioeconomic status, or skin color, are used to train models in fusion with other data types, such as medical images and text-based medical information. However, the effects of including categorical data features for model training in such data-scarce areas are underexamined, particularly regarding models intended to serve individuals equitably in a diverse population.

OBJECTIVE

This study aimed to explore categorical data's effects on machine learning model outputs, rooted the effects in the data collection and dataset publication processes, and proposed a mixed methods approach to examining datasets' data categories before using them for machine learning training.

METHODS

Against the theoretical background of the social construction of categories, we suggest a mixed methods approach to assess categorical data's utility for machine learning model training. As an example, we applied our approach to a Brazilian dermatological dataset (Dermatological and Surgical Assistance Program at the Federal University of Espírito Santo [PAD-UFES] 20). We first present an exploratory, quantitative study that assesses the effects when including or excluding each of the unique categorical data features of the PAD-UFES 20 dataset for training a transformer-based model using a data fusion algorithm. We then pair our quantitative analysis with a qualitative examination of the data categories based on interviews with the dataset authors.

RESULTS

Our quantitative study suggests scattered effects of including categorical data for machine learning model training across predictive classes. Our qualitative analysis gives insights into how the categorical data were collected and why they were published, explaining some of the quantitative effects that we observed. Our findings highlight the social constructedness of categorical data in publicly available datasets, meaning that the data in a category heavily depend on both how these categories are defined by the dataset creators and the sociomedico context in which the data are collected. This reveals relevant limitations of using publicly available datasets in contexts different from those of the collection of their data.

CONCLUSIONS

We caution against using data features of publicly available datasets without reflection on the social construction and context dependency of their categorical data features, particularly in data-sparse areas. We conclude that social scientific, context-dependent analysis of available data features using both quantitative and qualitative methods is helpful in judging the utility of categorical data for the population for which a model is intended.

Multimodal learning for mapping genotype-phenotype dynamics

How complex phenotypes emerge from intricate gene expression patterns is a fundamental question in biology. Integrating high-content genotyping approaches such as single-cell RNA sequencing and advanced learning methods such as language models offers an opportunity for dissecting this complex relationship. Here we present a computational integrated genetics framework designed to analyze and interpret the high-dimensional landscape of genotypes and their associated phenotypes simultaneously. We applied this approach to develop a multimodal foundation model to explore the genotype-phenotype relationship manifold for human transcriptomics at the cellular level. Analyzing this joint manifold showed a refined resolution of cellular heterogeneity, uncovered potential cross-tissue biomarkers and provided contextualized embeddings to investigate the polyfunctionality of genes shown for the von Willebrand factor (VWF) gene in endothelial cells. Overall, this study advances our understanding of the dynamic interplay between gene expression and phenotypic manifestation and demonstrates the potential of integrated genetics in uncovering new dimensions of cellular function and complexity.

The association of lifestyle with cardiovascular and all-cause mortality based on machine learning: a prospective study from the NHANES

BACKGROUND

Lifestyle and cardiovascular mortality and all-cause mortality have been exhaustively explored by traditional methods, but the advantages of machine learning (ML) over traditional methods may lead to different or more precise conclusions. The aim of this study was to evaluate the effectiveness of machine learning-based lifestyle factors in predicting cardiovascular and all-cause mortality and compare the results obtained by traditional methods.

METHOD

A prospective cohort study was conducted using a nationally representative sample of adults aged 40 years or older, drawn from the US National Health and Nutrition Examination Survey from 2007 to 2010. The participants underwent a comprehensive in-person interview and medical laboratory examinations, and subsequently, their records were linked with the National Death Index for further analysis. Extreme gradient enhancement, random forest, support vector machine and other machine learning methods are used to build the prediction model.

RESULT

Within a cohort comprising 7921 participants, spanning an average follow-up duration of 9.75 years, a total of 1911 deaths, including 585 cardiovascular-related deaths, were recorded. The model predicted mortality with an area under the receiver operating characteristic curve (AUC) of 0.862 and 0.836. Stratifying participants into distinct risk groups based on ML scores proved effective. All lifestyle behaviors were associated with a reduced risk of all-cause and cardiovascular mortality. As age increases, the effects of dietary scores and sedentary time on mortality risk become more pronounced, while the influence of physical activity tends to diminish.

CONCLUSION

We develop a ML model based on lifestyle behaviors to predict all-cause and cardiovascular mortality. The developed model offers valuable insights for the assessment of individual lifestyle-related risks. It applies to individuals, healthcare professionals, and policymakers to make informed decisions.

Use of Multimodal Artificial Intelligence in Surgical Instrument Recognition

Accurate identification of surgical instruments is crucial for efficient workflows and patient safety within the operating room, particularly in preventing complications such as retained surgical instruments. Artificial Intelligence (AI) models have shown the potential to automate this process. This study evaluates the accuracy of publicly available Large Language Models (LLMs)-ChatGPT-4, ChatGPT-4o, and Gemini-and a specialized commercial mobile application, Surgical-Instrument Directory (SID 2.0), in identifying surgical instruments from images. The study utilized a dataset of 92 high-resolution images of 25 surgical instruments (retractors, forceps, scissors, and trocars) photographed from multiple angles. Model performance was evaluated using accuracy, weighted precision, recall, and F1 score. ChatGPT-4o exhibited the highest accuracy (89.1%) in categorizing instruments (e.g., scissors, forceps). SID 2.0 (77.2%) and ChatGPT-4 (76.1%) achieved comparable accuracy, while Gemini (44.6%) demonstrated lower accuracy in this task. For precise subtype identification of instrument names (like "Mayo scissors" or "Kelly forceps"), all models had low accuracy, with SID 2.0 having an accuracy of 39.1%, followed by ChatGPT-4o (33.69%). Subgroup analysis revealed ChatGPT-4 and 4o recognized trocars in all instances. Similarly, Gemini identified surgical scissors in all instances. In conclusion, publicly available LLMs can reliably identify surgical instruments at the category level, with ChatGPT-4o demonstrating an overall edge. However, precise subtype identification remains a challenge for all models. These findings highlight the potential of AI-driven solutions to enhance surgical-instrument management and underscore the need for further refinements to improve accuracy and support patient safety.

Leveraging Natural Language Processing and Machine Learning Methods for Adverse Drug Event Detection in Electronic Health/Medical Records: A Scoping Review

BACKGROUND

Natural language processing (NLP) and machine learning (ML) techniques may help harness unstructured free-text electronic health record (EHR) data to detect adverse drug events (ADEs) and thus improve pharmacovigilance. However, evidence of their real-world effectiveness remains unclear.

OBJECTIVE

To summarise the evidence on the effectiveness of NLP/ML in detecting ADEs from unstructured EHR data and ultimately improve pharmacovigilance in comparison to other data sources.

METHODS

A scoping review was conducted by searching six databases in July 2023. Studies leveraging NLP/ML to identify ADEs from EHR were included. Titles/abstracts were screened by two independent researchers as were full-text articles. Data extraction was conducted by one researcher and checked by another. A narrative synthesis summarises the research techniques, ADEs analysed, model performance and pharmacovigilance impacts.

RESULTS

Seven studies met the inclusion criteria covering a wide range of ADEs and medications. The utilisation of rule-based NLP, statistical models, and deep learning approaches was observed. Natural language processing/ML techniques with unstructured data improved the detection of under-reported adverse events and safety signals. However, substantial variability was noted in the techniques and evaluation methods employed across the different studies and limitations exist in integrating the findings into practice.

CONCLUSIONS

Natural language processing (NLP) and machine learning (ML) have promising possibilities in extracting valuable insights with regard to pharmacovigilance from unstructured EHR data. These approaches have demonstrated proficiency in identifying specific adverse events and uncovering previously unknown safety signals that would not have been apparent through structured data alone. Nevertheless, challenges such as the absence of standardised methodologies and validation criteria obstruct the widespread adoption of NLP/ML for pharmacovigilance leveraging of unstructured EHR data.

Machine learning prediction of anxiety symptoms in social anxiety disorder: utilizing multimodal data from virtual reality sessions

Introduction

Machine learning (ML) is an effective tool for predicting mental states and is a key technology in digital psychiatry. This study aimed to develop ML algorithms to predict the upper tertile group of various anxiety symptoms based on multimodal data from virtual reality (VR) therapy sessions for social anxiety disorder (SAD) patients and to evaluate their predictive performance across each data type.

Methods

This study included 32 SAD-diagnosed individuals, and finalized a dataset of 132 samples from 25 participants. It utilized multimodal (physiological and acoustic) data from VR sessions to simulate social anxiety scenarios. This study employed extended Geneva minimalistic acoustic parameter set for acoustic feature extraction and extracted statistical attributes from time series-based physiological responses. We developed ML models that predict the upper tertile group for various anxiety symptoms in SAD using Random Forest, extreme gradient boosting (XGBoost), light gradient boosting machine (LightGBM), and categorical boosting (CatBoost) models. The best parameters were explored through grid search or random search, and the models were validated using stratified cross-validation and leave-one-out cross-validation.

Results

The CatBoost, using multimodal features, exhibited high performance, particularly for the Social Phobia Scale with an area under the receiver operating characteristics curve (AUROC) of 0.852. It also showed strong performance in predicting cognitive symptoms, with the highest AUROC of 0.866 for the Post-Event Rumination Scale. For generalized anxiety, the LightGBM's prediction for the State-Trait Anxiety Inventory-trait led to an AUROC of 0.819. In the same analysis, models using only physiological features had AUROCs of 0.626, 0.744, and 0.671, whereas models using only acoustic features had AUROCs of 0.788, 0.823, and 0.754.

Conclusions

This study showed that a ML algorithm using integrated multimodal data can predict upper tertile anxiety symptoms in patients with SAD with higher performance than acoustic or physiological data obtained during a VR session. The results of this study can be used as evidence for personalized VR sessions and to demonstrate the strength of the clinical use of multimodal data.

A machine learning model to predict neurological deterioration after mild traumatic brain injury in older adults

Objective

Neurological deterioration after mild traumatic brain injury (TBI) has been recognized as a poor prognostic factor. Early detection of neurological deterioration would allow appropriate monitoring and timely therapeutic interventions to improve patient outcomes. In this study, we developed a machine learning model to predict the occurrence of neurological deterioration after mild TBI using information obtained on admission.

Methods

This was a retrospective cohort study of data from the Think FAST registry, a multicenter prospective observational study of elderly TBI patients in Japan. Patients with an admission Glasgow Coma Scale (GCS) score of 12 or below or who underwent surgical treatment immediately upon admission were excluded. Neurological deterioration was defined as a decrease of 2 or more points from a GCS score of 13 or more within 24 h of hospital admission. The model predictive accuracy was judged with the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC), and the Youden index was used to determine the cutoff value.

Results

A total of 421 of 721 patients registered in the Think FAST registry between December 2019 and May 2021 were included in our study, among whom 25 demonstrated neurological deterioration. Among several machine learning algorithms, eXtreme Gradient Boosting (XGBoost) demonstrated the highest predictive accuracy in cross-validation, with an AUROC of 0.81 (±0.07) and an AUPRC of 0.33 (±0.08). Through SHapley Additive exPlanations (SHAP) analysis, five important features (D-dimer, fibrinogen, acute subdural hematoma thickness, cerebral contusion size, and systolic blood pressure) were identified and used to construct a better performing model (cross-validation AUROC of 0.84 and AUPRC of 0.34; testing data AUROC of 0.77 and AUPRC of 0.19). At the cutoff value from the Youden index, the model showed a sensitivity, specificity, and positive predictive value of 60, 96, and 38%, respectively. When neurosurgeons attempted to predict neurological deterioration using the same testing data, their values were 20, 94, and 19%, respectively.

Conclusion

In this study, our predictive model showed an acceptable performance in detecting neurological deterioration after mild TBI. Further validation through prospective studies is necessary to confirm these results.

Efficacy and Safety of Herbal Medicine Bojungikki-Tang in Combination with Pembrolizumab versus Pembrolizumab Monotherapy for Stage IV Non-Small Cell Lung Cancer: Study Protocol for a Randomized, Open-Label, Double-Arm, Multicenter Trial

BACKGROUND

Non-small cell lung cancer (NSCLC) exhibits low survival rates. Although immune checkpoint inhibitors (ICIs) have become first-line treatment for NSCLC, their limited response to ICI monotherapy has led to exploration of combination treatments. However, the high incidence of treatment-related adverse events associated with conventional drug combinations has highlighted the need for alternative herbal therapy. Bojungikki-tang (BJIKT), a traditional herbal medicine, has been used to treat gastrointestinal disorders and enhance immune function. Our preclinical studies have demonstrated that BJIKT combined with anti-PD-1 or anti-PD-L1 antibodies exhibits significant efficacy in suppressing tumor growth by modulating the immunosuppressive tumor microenvironment. Building on these preclinical findings, this study aims to evaluate the efficacy and safety of BJIKT with pembrolizumab combination therapy compared to pembrolizumab monotherapy in advanced NSCLC patients.

METHODS

70 individuals with stage IV NSCLC scheduled for first-line pembrolizumab monotherapy will be randomly assigned to intervention or control groups. The primary outcome will be progression-free survival, with secondary outcomes including disease control rate, overall survival, and quality of life assessment. Adverse events will be monitored for safety. This study will explore the synergistic mechanism of combinatorial therapy using immune profiling and multi-omics analysis, and the possibility for personalized integrative therapy based on cold-heat syndrome differentiation (SD) types in East Asian medicine.

DISCUSSION

This study will provide novel evidence regarding survival outcomes, quality of life, and safety profiles of combined ICI and BJIKT therapy for advanced NSCLC. The exploratory data will contribute to tailoring treatments to immune-based SD types in NSCLC patients.

The Transition From Homogeneous to Heterogeneous Machine Learning in Neuropsychiatric Research

Despite the advantage of neuroimaging-based machine learning (ML) models as pivotal tools for investigating brain-behavior relationships in neuropsychiatric studies, these data-driven predictive approaches have yet to yield substantial, clinically actionable insights for mental health care. A notable impediment lies in the inadequate accommodation of most ML research to the natural heterogeneity within large samples. Although commonly thought of as individual-level analyses, many ML algorithms are unimodal and homogeneous and thus incapable of capturing the potentially heterogeneous relationships between biology and psychopathology. We review the current landscape of computational research targeting population heterogeneity and argue that there is a need to expand from brain subtyping and behavioral phenotyping to analyses that focus on heterogeneity at the relational level. To this end, we review and suggest several existing ML models with the capacity to discern how external environmental and sociodemographic factors moderate the brain-behavior mapping function in a data-driven fashion. These heterogeneous ML models hold promise for enhancing the discovery of individualized brain-behavior associations and advancing precision psychiatry.

A Modality-Flexible Framework for Alzheimer's Disease Diagnosis Following Clinical Routine

Dementia has high incidence among the elderly, and Alzheimer's disease (AD) is the most common dementia. The procedure of AD diagnosis in clinics usually follows a standard routine consisting of different phases, from acquiring non-imaging tabular data in the screening phase to MR imaging and ultimately to PET imaging. Most of the existing AD diagnosis studies are dedicated to a specific phase using either single or multi-modal data. In this paper, we introduce a modality-flexible classification framework, which is applicable for different AD diagnosis phases following the clinical routine. Specifically, our framework consists of three branches corresponding to three diagnosis phases: 1) a tabular branch using only tabular data for screening phase, 2) an MRI branch using both MRI and tabular data for uncertain cases in screening phase, and 3) ultimately a PET branch for the challenging cases using all the modalities including PET, MRI, and tabular data. To achieve effective fusion of imaging and non-imaging modalities, we introduce an image-tabular transformer block to adaptively scale and shift the image and tabular features according to modality importance determined by the network. The proposed framework is extensively validated on four cohorts containing 6495 subjects. Experiments demonstrate that our framework achieves superior diagnostic performance than the other representative methods across various AD diagnosis tasks, and shows promising performance for all the diagnosis phases, which exhibits great potential for clinical application.

The Role of Machine Learning Models in Predicting Cirrhosis Mortality: A Systematic Review

Liver cirrhosis affects millions of individuals worldwide and is one of the primary causes of mortality. Early mortality prediction for cirrhosis patients may increase the possibility for medical professionals to treat the illness successfully. This study assesses the ability of machine learning (ML) models to predict cirrhosis mortality. We followed Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines to search for relevant literature across four different databases. We found 379 studies of which 10 were eligible for inclusion in the current study. We analyzed 10 retrospective studies that showed that ML models outperformed conventional scores in predicting the death rate from end-stage liver disease (ESLD). Interestingly, models that used more parameters, such as patient demographics and extensive laboratory testing, exhibited higher prediction accuracy. With an area under the receiver operating characteristic (AUROC) ranging from 0.71 to 0.96, ML models showed consistently significant gains over traditional prognostic ratings. This review emphasizes how ML models might improve ESLD patient death prediction. Because machine learning models are more accurate than conventional approaches, it is important to incorporate data-driven informatics technologies into clinical settings. Additional validation and openness are required to guarantee model dependability and interpretability before ML may be used in clinical practice. The goal of future research should be to create reliable, interpretable models that may be used successfully in a variety of clinical contexts, enhancing ESLD patient treatment and results.

Machine learning-based infection diagnostic and prognostic models in post-acute care settings: a systematic review

OBJECTIVES

This study aims to (1) review machine learning (ML)-based models for early infection diagnostic and prognosis prediction in post-acute care (PAC) settings, (2) identify key risk predictors influencing infection-related outcomes, and (3) examine the quality and limitations of these models.

MATERIALS AND METHODS

PubMed, Web of Science, Scopus, IEEE Xplore, CINAHL, and ACM digital library were searched in February 2024. Eligible studies leveraged PAC data to develop and evaluate ML models for infection-related risks. Data extraction followed the CHARMS checklist. Quality appraisal followed the PROBAST tool. Data synthesis was guided by the socio-ecological conceptual framework.

RESULTS

Thirteen studies were included, mainly focusing on respiratory infections and nursing homes. Most used regression models with structured electronic health record data. Since 2020, there has been a shift toward advanced ML algorithms and multimodal data, biosensors, and clinical notes being significant sources of unstructured data. Despite these advances, there is insufficient evidence to support performance improvements over traditional models. Individual-level risk predictors, like impaired cognition, declined function, and tachycardia, were commonly used, while contextual-level predictors were barely utilized, consequently limiting model fairness. Major sources of bias included lack of external validation, inadequate model calibration, and insufficient consideration of data complexity.

DISCUSSION AND CONCLUSION

Despite the growth of advanced modeling approaches in infection-related models in PAC settings, evidence supporting their superiority remains limited. Future research should leverage a socio-ecological lens for predictor selection and model construction, exploring optimal data modalities and ML model usage in PAC, while ensuring rigorous methodologies and fairness considerations.

Multimodality Fusion Aspects of Medical Diagnosis: A Comprehensive Review

Utilizing information from multiple sources is a preferred and more precise method for medical experts to confirm a diagnosis. Each source provides critical information about the disease that might otherwise be absent in other modalities. Combining information from various medical sources boosts confidence in the diagnosis process, enabling the creation of an effective treatment plan for the patient. The scarcity of medical experts to diagnose diseases motivates the development of automatic diagnoses relying on multimodal data. With the progress in artificial intelligence technology, automated diagnosis using multimodal fusion techniques is now possible. Nevertheless, the concept of multimodal medical diagnosis is still new and requires an understanding of the diverse aspects of multimodal data and its related challenges. This review article examines the various aspects of multimodal medical diagnosis to equip readers, academicians, and researchers with necessary knowledge to advance multimodal medical research. The chosen articles in the study underwent thorough screening from reputable journals and publishers to offer high-quality content to readers, who can then apply the knowledge to produce quality research. Besides, the need for multimodal information and the associated challenges are discussed with solutions. Additionally, ethical issues of using artificial intelligence in medical diagnosis is also discussed.

Seeing the random forest through the decision trees. Supporting learning health systems from histopathology with machine learning models: Challenges and opportunities

This paper discusses some overlooked challenges faced when working with machine learning models for histopathology and presents a novel opportunity to support "Learning Health Systems" with them. Initially, the authors elaborate on these challenges after separating them according to their mitigation strategies: those that need innovative approaches, time, or future technological capabilities and those that require a conceptual reappraisal from a critical perspective. Then, a novel opportunity to support "Learning Health Systems" by integrating hidden information extracted by ML models from digitalized histopathology slides with other healthcare big data is presented.

Session-by-Session Prediction of Anti-Endothelial Growth Factor Injection Needs in Neovascular Age-Related Macular Degeneration Using Optical-Coherence-Tomography-Derived Features and Machine Learning

BACKGROUND/OBJECTIVES

Neovascular age-related macular degeneration (nAMD) is a retinal disorder leading to irreversible central vision loss. The pro-re-nata (PRN) treatment for nAMD involves frequent intravitreal injections of anti-VEGF medications, placing a burden on patients and healthcare systems. Predicting injections needs at each monitoring session could optimize treatment outcomes and reduce unnecessary interventions.

METHODS

To achieve these aims, machine learning (ML) models were evaluated using different combinations of clinical variables, including retinal thickness and volume, best-corrected visual acuity, and features derived from macular optical coherence tomography (OCT). A "Leave Some Subjects Out" (LSSO) nested cross-validation approach ensured robust evaluation. Moreover, the SHapley Additive exPlanations (SHAP) analysis was employed to quantify the contribution of each feature to model predictions.

RESULTS

Results demonstrated that models incorporating both structural and functional features achieved high classification accuracy in predicting injection necessity (AUC = 0.747 ± 0.046, MCC = 0.541 ± 0.073). Moreover, the explainability analysis identified as key predictors both subretinal and intraretinal fluid, alongside central retinal thickness.

CONCLUSIONS

These findings suggest that session-by-session prediction of injection needs in nAMD patients is feasible, even without processing the entire OCT image. The proposed ML framework has the potential to be integrated into routine clinical workflows, thereby optimizing nAMD therapeutic management.

HELMET: A Hybrid Machine Learning Framework for Real-Time Prediction of Edema Trajectory in Large Middle Cerebral Artery Stroke

Malignant cerebral edema occurs when brain swelling displaces and compresses vital midline structures within the first week of a large middle cerebral artery stroke. Early interventions such as hyperosmolar therapy or surgical decompression may reverse secondary injury but must be administered judiciously. To optimize treatment and reduce secondary damage, clinicians need strategies to frequently and quantitatively assess the trajectory of edema using updated, relevant information. However, existing risk assessment tools are limited by the absence of structured records capturing the evolution of edema and typically estimate risk at a single time point early in the admission, therefore failing to account for changes in variables over the following hours or days. To address this, we developed and validated dynamic machine learning models capable of accurately predicting the severity of midline structure displacement, an established indicator of malignant edema, in real-time. Our models can provide updated estimations as frequently as every hour, using data from structured time-varying patient records, radiographic text, and human-curated neurological characteristics. Our work resulted in two novel multi-class classification models, collectively named Hybrid Ensemble Learning Models for Edema Trajectory (HELMET), predicting the progression of midline shift over 8-hour (HELMET-8) and 24-hour windows (HELMET-24), respectively. HELMET combines transformer-based large language models with supervised ensemble learning, demonstrating the value of merging human expertise and multimodal health records in developing clinical risk scores. Both models were trained on a retrospective cohort of 15,696 observations from 623 patients hospitalized with large middle cerebral artery ischemic stroke and were externally validated using 3,713 observations from 60 patients at a separate hospital system. Our HELMET models are accurate and generalize effectively to diverse populations, achieving a cross-validated mean area under the receiver operating characteristic score of 96.6% in the derivation cohort and 92.5% in the external validation cohort. Moreover, our approach provides a framework for developing hybrid risk prediction models that integrate both human-extracted and algorithm-derived multi-modal inputs. Our work enables accurate estimation of complex, dynamic, and highly specific clinical targets, such as midline shift, in real-time, even when relevant structured information is limited in electronic health record databases.

AI-Reinforced Wearable Sensors and Intelligent Point-of-Care Tests

Artificial intelligence (AI) techniques offer great potential to advance point-of-care testing (POCT) and wearable sensors for personalized medicine applications. This review explores the recent advances and the transformative potential of the use of AI in improving wearables and POCT. The integration of AI significantly contributes to empowering these tools and enables continuous monitoring, real-time analysis, and rapid diagnostics, thus enhancing patient outcomes and healthcare efficiency. Wearable sensors powered by AI models offer tremendous opportunities for precise and non-invasive tracking of physiological conditions that are essential for early disease detection and personalized treatments. AI-empowered POCT facilitates rapid, accurate diagnostics, making these medical testing kits accessible and available even in resource-limited settings. This review discusses the key advances in AI applications for data processing, sensor fusion, and multivariate analytics, highlighting case examples that exhibit their impact in different medical scenarios. In addition, the challenges associated with data privacy, regulatory approvals, and technology integrations into the existing healthcare system have been overviewed. The outlook emphasizes the urgent need for continued innovation in AI-driven health technologies to overcome these challenges and to fully achieve the potential of these techniques to revolutionize personalized medicine.

Machine learning for anxiety and depression profiling and risk assessment in the aftermath of an emergency

BACKGROUND & OBJECTIVES

Mental health disorders pose an increasing public health challenge worsened by the COVID-19 pandemic. The pandemic highlighted gaps in preparedness, emphasizing the need for early identification of at-risk groups and targeted interventions. This study aims to develop a risk assessment tool for anxiety, depression, and self-perceived stress using machine learning (ML) and explainable AI to identify key risk factors and stratify the population into meaningful risk profiles.

METHODS

We utilized a cohort of 9291 individuals from Northern Spain, with extensive post-COVID-19 mental health surveys. ML classification algorithms predicted depression, anxiety, and self-reported stress in three classes: healthy, mild, and severe outcomes. A novel combination of SHAP (SHapley Additive exPlanations) and UMAP (Uniform Manifold Approximation and Projection) was employed to interpret model predictions and facilitate the identification of high-risk phenotypic clusters.

RESULTS

The mean macro-averaged one-vs-one AUROC was 0.77 (± 0.01) for depression, 0.72 (± 0.01) for anxiety, and 0.73 (± 0.02) for self-perceived stress. Key risk factors included poor self-reported health, chronic mental health conditions, and poor social support. High-risk profiles, such as women with reduced sleep hours, were identified for self-perceived stress. Binary classification of healthy vs. at-risk classes yielded F1-Scores over 0.70.

CONCLUSIONS

Combining SHAP with UMAP for risk profile stratification offers valuable insights for developing effective interventions and shaping public health policies. This data-driven approach to mental health preparedness, when validated in real-world scenarios, can significantly address the mental health impact of public health crises like COVID-19.

HyMNet: A Multimodal Deep Learning System for Hypertension Prediction Using Fundus Images and Cardiometabolic Risk Factors

STUDY OBJECTIVES

This study aimed to develop a multimodal deep learning (MMDL) system called HyMNet, integrating fundus images and cardiometabolic factors (age and sex) to enhance hypertension (HTN) detection.

METHODS

HyMNet employed RETFound, a model pretrained on 1.6 million retinal images, for the fundus data, in conjunction with a fully connected neural network for age and sex. The two pathways were jointly trained by joining their feature vectors into a fusion network. The system was trained on 5016 retinal images from 1243 individuals provided by the Saudi Ministry of National Guard Health Affairs. The influence of diabetes on HTN detection was also assessed.

RESULTS

HyMNet surpassed the unimodal system, achieving an F1 score of 0.771 compared to 0.745 for the unimodal model. For diabetic patients, the F1 score was 0.796, while it was 0.466 for non-diabetic patients.

CONCLUSIONS

HyMNet exhibited superior performance relative to unimodal approaches, with an F1 score of 0.771 for HyMNet compared to 0.752 for models trained on demographic data alone, underscoring the advantages of MMDL systems in HTN detection. The findings indicate that diabetes significantly impacts HTN prediction, enhancing detection accuracy among diabetic patients. Utilizing MMDL with diverse data sources could improve clinical applicability and generalization.

Video-audio neural network ensemble for comprehensive screening of autism spectrum disorder in young children

A timely diagnosis of autism is paramount to allow early therapeutic intervention in preschoolers. Deep Learning tools have been increasingly used to identify specific autistic symptoms. But they also offer opportunities for broad automated detection of autism at an early age. Here, we leverage a multi-modal approach by combining two neural networks trained on video and audio features of semi-standardized social interactions in a sample of 160 children aged 1 to 5 years old. Our ensemble model performs with an accuracy of 82.5% (F1 score: 0.816, Precision: 0.775, Recall: 0.861) for screening Autism Spectrum Disorders (ASD). Additional combinations of our model were developed to achieve higher specificity (92.5%, i.e., few false negatives) or sensitivity (90%, i.e. few false positives). Finally, we found a relationship between the neural network modalities and specific audio versus video ASD characteristics, bringing evidence that our neural network implementation was effective in taking into account different features that are currently standardized under the gold standard ASD assessment.

The Emerging and Important Role of Artificial Intelligence in Cardiac Surgery

Artificial Intelligence (AI) has greatly affected our everyday lives and holds great promise to change the landscape of medicine. AI is particularly positioned to improve care for the increasingly complex patients undergoing cardiac surgery using the immense amount of data generated in the course of their care. When deployed, AI can be used to analyze this information at the patient's bedside more expediently and accurately, all while providing new insights. This review summarizes the current applications of AI in cardiac surgery from the vantage point of a patient's journey. Applications of AI include preoperative risk assessment, intraoperative planning, postoperative patient care, and outpatient telemonitoring, encompassing the spectrum of cardiac surgical care. Offloading of administrative processes and enhanced experience with information gathering also represent a unique and under-represented avenue for future use of AI. As clinicians, understanding the nomenclature and applications of AI is important to contextualize issues, to ensure problem-driven solutions, and for clinical benefit. Precision medicine, and thus clinically relevant AI, remains dependent on data curation and warehousing to gather insights from large multicentre repositories while treating privacy with the utmost importance. AI tasks should not be siloed but rather holistically integrated into clinical workflow to retain context and relevance. As cardiac surgeons, AI allows us to look forward to a bright future of more efficient use of our clinical expertise toward high-level decision making and technical prowess.

Emerging Analytical Approaches for Personalized Medicine Using Machine Learning In Pediatric and Congenital Heart Disease

Precision and personalized medicine, the process by which patient management is tailored to individual circumstances, are now terms that are familiar to cardiologists, despite it still being an emerging field. Although precision medicine relies most often on the underlying biology and pathophysiology of a patient's condition, personalized medicine relies on digital biomarkers generated through algorithms. Given the complexity of the underlying data, these digital biomarkers are most often generated through machine-learning algorithms. There are a number of analytic considerations regarding the creation of digital biomarkers that are discussed in this review, including data preprocessing, time dependency and gating, dimensionality reduction, and novel methods, both in the realm of supervised and unsupervised machine learning. Some of these considerations, such as sample size requirements and measurements of model performance, are particularly challenging in small and heterogeneous populations with rare outcomes such as children with congenital heart disease. Finally, we review analytic considerations for the deployment of digital biomarkers in clinical settings, including the emerging field of clinical artificial intelligence (AI) operations, computational needs for deployment, efforts to increase the explainability of AI, algorithmic drift, and the needs for distributed surveillance and federated learning. We conclude this review by discussing a recent simulation study that shows that, despite these analytic challenges and complications, the use of digital biomarkers in managing clinical care might have substantial benefits regarding individual patient outcomes.

Multimodal masked siamese network improves chest X-ray representation learning

Self-supervised learning methods for medical images primarily rely on the imaging modality during pretraining. Although such approaches deliver promising results, they do not take advantage of the associated patient or scan information collected within Electronic Health Records (EHR). This study aims to develop a multimodal pretraining approach for chest radiographs that considers EHR data incorporation as an additional modality that during training. We propose to incorporate EHR data during self-supervised pretraining with a Masked Siamese Network (MSN) to enhance the quality of chest radiograph representations. We investigate three types of EHR data, including demographic, scan metadata, and inpatient stay information. We evaluate the multimodal MSN on three publicly available chest X-ray datasets, MIMIC-CXR, CheXpert, and NIH-14, using two vision transformer (ViT) backbones, specifically ViT-Tiny and ViT-Small. In assessing the quality of the representations through linear evaluation, our proposed method demonstrates significant improvement compared to vanilla MSN and state-of-the-art self-supervised learning baselines. In particular, our proposed method achieves an improvement of of 2%  in the Area Under the Receiver Operating Characteristic Curve (AUROC) compared to vanilla MSN and 5% to 8% compared to other baselines, including uni-modal ones. Furthermore, our findings reveal that demographic features provide the most significant performance improvement. Our work highlights the potential of EHR-enhanced self-supervised pretraining for medical imaging and opens opportunities for future research to address limitations in existing representation learning methods for other medical imaging modalities, such as neuro-, ophthalmic, and sonar imaging.

Can Machine Learning Overcome the 95% Failure Rate and Reality that Only 30% of Approved Cancer Drugs Meaningfully Extend Patient Survival?

Despite implementing hundreds of strategies, cancer drug development suffers from a 95% failure rate over 30 years, with only 30% of approved cancer drugs extending patient survival beyond 2.5 months. Adding more criteria without eliminating nonessential ones is impractical and may fall into the "survivorship bias" trap. Machine learning (ML) models may enhance efficiency by saving time and cost. Yet, they may not improve success rate without identifying the root causes of failure. We propose a "STAR-guided ML system" (structure-tissue/cell selectivity-activity relationship) to enhance success rate and efficiency by addressing three overlooked interdependent factors: potency/specificity to the on/off-targets determining efficacy in tumors at clinical doses, on/off-target-driven tissue/cell selectivity influencing adverse effects in the normal organs at clinical doses, and optimal clinical doses balancing efficacy/safety as determined by potency/specificity and tissue/cell selectivity. STAR-guided ML models can directly predict clinical dose/efficacy/safety from five features to design/select the best drugs, enhancing success and efficiency of cancer drug development.

Cardiovascular care with digital twin technology in the era of generative artificial intelligence

Digital twins, which are in silico replications of an individual and its environment, have advanced clinical decision-making and prognostication in cardiovascular medicine. The technology enables personalized simulations of clinical scenarios, prediction of disease risk, and strategies for clinical trial augmentation. Current applications of cardiovascular digital twins have integrated multi-modal data into mechanistic and statistical models to build physiologically accurate cardiac replicas to enhance disease phenotyping, enrich diagnostic workflows, and optimize procedural planning. Digital twin technology is rapidly evolving in the setting of newly available data modalities and advances in generative artificial intelligence, enabling dynamic and comprehensive simulations unique to an individual. These twins fuse physiologic, environmental, and healthcare data into machine learning and generative models to build real-time patient predictions that can model interactions with the clinical environment to accelerate personalized patient care. This review summarizes digital twins in cardiovascular medicine and their potential future applications by incorporating new personalized data modalities. It examines the technical advances in deep learning and generative artificial intelligence that broaden the scope and predictive power of digital twins. Finally, it highlights the individual and societal challenges as well as ethical considerations that are essential to realizing the future vision of incorporating cardiology digital twins into personalized cardiovascular care.

Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook

In the complex and multidimensional field of medicine, multimodal data are prevalent and crucial for informed clinical decisions. Multimodal data span a broad spectrum of data types, including medical images (eg, MRI and CT scans), time-series data (eg, sensor data from wearable devices and electronic health records), audio recordings (eg, heart and respiratory sounds and patient interviews), text (eg, clinical notes and research articles), videos (eg, surgical procedures), and omics data (eg, genomics and proteomics). While advancements in large language models (LLMs) have enabled new applications for knowledge retrieval and processing in the medical field, most LLMs remain limited to processing unimodal data, typically text-based content, and often overlook the importance of integrating the diverse data modalities encountered in clinical practice. This paper aims to present a detailed, practical, and solution-oriented perspective on the use of multimodal LLMs (M-LLMs) in the medical field. Our investigation spanned M-LLM foundational principles, current and potential applications, technical and ethical challenges, and future research directions. By connecting these elements, we aimed to provide a comprehensive framework that links diverse aspects of M-LLMs, offering a unified vision for their future in health care. This approach aims to guide both future research and practical implementations of M-LLMs in health care, positioning them as a paradigm shift toward integrated, multimodal data-driven medical practice. We anticipate that this work will spark further discussion and inspire the development of innovative approaches in the next generation of medical M-LLM systems.

Precision prevention in occupational health: a conceptual analysis and development of a unified understanding and an integrative framework

Introduction

Precision prevention implements highly precise, tailored health interventions for individuals by directly addressing personal and environmental determinants of health. However, precision prevention does not yet appear to be fully established in occupational health. There are numerous understandings and conceptual approaches, but these have not yet been systematically presented or synthesized. Therefore, this conceptual analysis aims to propose a unified understanding and develop an integrative conceptual framework for precision prevention in occupational health.

Methods

Firstly, to systematically present definitions and frameworks of precision prevention in occupational health, six international databases were searched for studies published between January 2010 and January 2024 that used the term precision prevention or its synonyms in the context of occupational health. Secondly, a qualitative content analysis was conducted to analyze the existing definitions and propose a unified understanding. Thirdly, based on the identified frameworks, a multi-stage exploratory development process was applied to develop and propose an integrative conceptual framework for precision prevention in occupational health.

Results

After screening 3,681 articles, 154 publications were reviewed, wherein 29 definitions of precision prevention and 64 different frameworks were found, which can be summarized in eight higher-order categories. The qualitative content analysis revealed seven themes and illustrated many different wordings. The proposed unified understanding of precision prevention in occupational health takes up the identified themes. It includes, among other things, a contrast to a "one-size-fits-all approach" with a risk- and resource-oriented data collection and innovative data analytics with profiling to provide and improve tailored interventions. The developed and proposed integrative conceptual framework comprises three overarching stages: (1) data generation, (2) data management lifecycle and (3) interventions (development, implementation and adaptation).

Discussion

Although there are already numerous studies on precision prevention in occupational health, this conceptual analysis offers, for the first time, a proposal for a unified understanding and an integrative conceptual framework. However, the proposed unified understanding and the developed integrative conceptual framework should only be seen as an initial proposal that should be critically discussed and further developed to expand and strengthen both research on precision prevention in occupational health and its practical application in the workplace.

Prediction of future dementia among patients with mild cognitive impairment (MCI) by integrating multimodal clinical data

Efficiently and objectively analyzing the complex, diverse multimodal data collected from patients at risk for dementia can be difficult in the clinical setting, contributing to high rates of underdiagnosis or misdiagnosis of this serious disorder. Patients with mild cognitive impairment (MCI) are especially at risk of developing dementia in the future. This study evaluated the ability of multi-modal machine learning (ML) methods, especially the Ensemble Integration (EI) framework, to predict future dementia development among patients with MCI. EI is a machine learning framework designed to leverage complementarity and consensus in multimodal data, which may not be adequately captured by methods used by prior dementia-related prediction studies. We tested EI's ability to predict future dementia development among MCI patients using multimodal clinical and imaging data, such as neuroanatomical measurements from structural magnetic resonance imaging (MRI) and positron emission tomography (PET) scans, from The Alzheimer's Disease Prediction of Longitudinal Evolution (TADPOLE) challenge. For predicting future dementia development among MCI patients, on a held out test set, the EI-based model performed better (AUC = 0.81, F-measure = 0.68) than the more commonly used XGBoost (AUC = 0.68, F-measure = 0.57) and deep learning (AUC = 0.79, F-measure = 0.61) approaches. This EI-based model also suggested MRI-derived volumes of regions in the middle temporal gyrus, posterior cingulate gyrus and inferior lateral ventricle brain regions to be predictive of progression to dementia.

Systematic Review of IoT-Based Solutions for User Tracking: Towards Smarter Lifestyle, Wellness and Health Management

The Internet of Things (IoT) base has grown to over 20 billion devices currently operational worldwide. As they greatly extend the applicability and use of biosensors, IoT developments are transformative. Recent studies show that IoT, coupled with advanced communication frameworks, such as machine-to-machine (M2M) interactions, can lead to (1) improved efficiency in data exchange, (2) accurate and timely health monitoring, and (3) enhanced user engagement and compliance through advancements in human-computer interaction. This systematic review of the 19 most relevant studies examines the potential of IoT in health and lifestyle management by conducting detailed analyses and quality assessments of each study. Findings indicate that IoT-based systems effectively monitor various health parameters using biosensors, facilitate real-time feedback, and support personalized health recommendations. Key limitations include small sample sizes, insufficient security measures, practical issues with wearable sensors, and reliance on internet connectivity in areas with poor network infrastructure. The reviewed studies demonstrated innovative applications of IoT, focusing on M2M interactions, edge devices, multimodality health monitoring, intelligent decision-making, and automated health management systems. These insights offer valuable recommendations for optimizing IoT technologies in health and wellness management.

Perspectives on Advancing Multimodal Learning in Environmental Science and Engineering Studies

The environment faces increasing anthropogenic impacts, resulting in a rapid increase in environmental issues that undermine the natural capital essential for human wellbeing. These issues are complex and often influenced by various factors represented by data with different modalities. While machine learning (ML) provides data-driven tools for addressing the environmental issues, the current ML models in environmental science and engineering (ES&E) often neglect the utilization of multimodal data. With the advancement in deep learning, multimodal learning (MML) holds promise for comprehensive descriptions of the environmental issues by harnessing data from diverse modalities. This advancement has the potential to significantly elevate the accuracy and robustness of prediction models in ES&E studies, providing enhanced solutions for various environmental modeling tasks. This perspective summarizes MML methodologies and proposes potential applications of MML models in ES&E studies, including environmental quality assessment, prediction of chemical hazards, and optimization of pollution control techniques. Additionally, we discuss the challenges associated with implementing MML in ES&E and propose future research directions in this domain.

Advancing Chinese biomedical text mining with community challenges

OBJECTIVE

This study aims to review the recent advances in community challenges for biomedical text mining in China.

METHODS

We collected information of evaluation tasks released in community challenges of biomedical text mining, including task description, dataset description, data source, task type and related links. A systematic summary and comparative analysis were conducted on various biomedical natural language processing tasks, such as named entity recognition, entity normalization, attribute extraction, relation extraction, event extraction, text classification, text similarity, knowledge graph construction, question answering, text generation, and large language model evaluation.

RESULTS

We identified 39 evaluation tasks from 6 community challenges that spanned from 2017 to 2023. Our analysis revealed the diverse range of evaluation task types and data sources in biomedical text mining. We explored the potential clinical applications of these community challenge tasks from a translational biomedical informatics perspective. We compared with their English counterparts, and discussed the contributions, limitations, lessons and guidelines of these community challenges, while highlighting future directions in the era of large language models.

CONCLUSION

Community challenge evaluation competitions have played a crucial role in promoting technology innovation and fostering interdisciplinary collaboration in the field of biomedical text mining. These challenges provide valuable platforms for researchers to develop state-of-the-art solutions.

Applications of Multimodal Artificial Intelligence in Non-Hodgkin Lymphoma B Cells

Given advancements in large-scale data and AI, integrating multimodal artificial intelligence into cancer research can enhance our understanding of tumor behavior by simultaneously processing diverse biomedical data types. In this review, we explore the potential of multimodal AI in comprehending B-cell non-Hodgkin lymphomas (B-NHLs). B-cell non-Hodgkin lymphomas (B-NHLs) represent a particular challenge in oncology due to tumor heterogeneity and the intricate ecosystem in which tumors develop. These complexities complicate diagnosis, prognosis, and therapy response, emphasizing the need to use sophisticated approaches to enhance personalized treatment strategies for better patient outcomes. Therefore, multimodal AI can be leveraged to synthesize critical information from available biomedical data such as clinical record, imaging, pathology and omics data, to picture the whole tumor. In this review, we first define various types of modalities, multimodal AI frameworks, and several applications in precision medicine. Then, we provide several examples of its usage in B-NHLs, for analyzing the complexity of the ecosystem, identifying immune biomarkers, optimizing therapy strategy, and its clinical applications. Lastly, we address the limitations and future directions of multimodal AI, highlighting the need to overcome these challenges for better clinical practice and application in healthcare.

Large Language Models for Wearable Sensor-Based Human Activity Recognition, Health Monitoring, and Behavioral Modeling: A Survey of Early Trends, Datasets, and Challenges

The proliferation of wearable technology enables the generation of vast amounts of sensor data, offering significant opportunities for advancements in health monitoring, activity recognition, and personalized medicine. However, the complexity and volume of these data present substantial challenges in data modeling and analysis, which have been addressed with approaches spanning time series modeling to deep learning techniques. The latest frontier in this domain is the adoption of large language models (LLMs), such as GPT-4 and Llama, for data analysis, modeling, understanding, and human behavior monitoring through the lens of wearable sensor data. This survey explores the current trends and challenges in applying LLMs for sensor-based human activity recognition and behavior modeling. We discuss the nature of wearable sensor data, the capabilities and limitations of LLMs in modeling them, and their integration with traditional machine learning techniques. We also identify key challenges, including data quality, computational requirements, interpretability, and privacy concerns. By examining case studies and successful applications, we highlight the potential of LLMs in enhancing the analysis and interpretation of wearable sensor data. Finally, we propose future directions for research, emphasizing the need for improved preprocessing techniques, more efficient and scalable models, and interdisciplinary collaboration. This survey aims to provide a comprehensive overview of the intersection between wearable sensor data and LLMs, offering insights into the current state and future prospects of this emerging field.

Graph Artificial Intelligence in Medicine

In clinical artificial intelligence (AI), graph representation learning, mainly through graph neural networks and graph transformer architectures, stands out for its capability to capture intricate relationships and structures within clinical datasets. With diverse data-from patient records to imaging-graph AI models process data holistically by viewing modalities and entities within them as nodes interconnected by their relationships. Graph AI facilitates model transfer across clinical tasks, enabling models to generalize across patient populations without additional parameters and with minimal to no retraining. However, the importance of human-centered design and model interpretability in clinical decision-making cannot be overstated. Since graph AI models capture information through localized neural transformations defined on relational datasets, they offer both an opportunity and a challenge in elucidating model rationale. Knowledge graphs can enhance interpretability by aligning model-driven insights with medical knowledge. Emerging graph AI models integrate diverse data modalities through pretraining, facilitate interactive feedback loops, and foster human-AI collaboration, paving the way toward clinically meaningful predictions.

Non-Invasive Detection of Early-Stage Fatty Liver Disease via an On-Skin Impedance Sensor and Attention-Based Deep Learning

Early-stage nonalcoholic fatty liver disease (NAFLD) is a silent condition, with most cases going undiagnosed, potentially progressing to liver cirrhosis and cancer. A non-invasive and cost-effective detection method for early-stage NAFLD detection is a public health priority but challenging. In this study, an adhesive, soft on-skin sensor with low electrode-skin contact impedance for early-stage NAFLD detection is fabricated. A method is developed to synthesize platinum nanoparticles and reduced graphene quantum dots onto the on-skin sensor to reduce electrode-skin contact impedance by increasing double-layer capacitance, thereby enhancing detection accuracy. Furthermore, an attention-based deep learning algorithm is introduced to differentiate impedance signals associated with early-stage NAFLD in high-fat-diet-fed low-density lipoprotein receptor knockout (Ldlr-/-) mice compared to healthy controls. The integration of an adhesive, soft on-skin sensor with low electrode-skin contact impedance and the attention-based deep learning algorithm significantly enhances the detection accuracy for early-stage NAFLD, achieving a rate above 97.5% with an area under the receiver operating characteristic curve (AUC) of 1.0. The findings present a non-invasive approach for early-stage NAFLD detection and display a strategy for improved early detection through on-skin electronics and deep learning.

A review of deep learning-based information fusion techniques for multimodal medical image classification

Multimodal medical imaging plays a pivotal role in clinical diagnosis and research, as it combines information from various imaging modalities to provide a more comprehensive understanding of the underlying pathology. Recently, deep learning-based multimodal fusion techniques have emerged as powerful tools for improving medical image classification. This review offers a thorough analysis of the developments in deep learning-based multimodal fusion for medical classification tasks. We explore the complementary relationships among prevalent clinical modalities and outline three main fusion schemes for multimodal classification networks: input fusion, intermediate fusion (encompassing single-level fusion, hierarchical fusion, and attention-based fusion), and output fusion. By evaluating the performance of these fusion techniques, we provide insight into the suitability of different network architectures for various multimodal fusion scenarios and application domains. Furthermore, we delve into challenges related to network architecture selection, handling incomplete multimodal data management, and the potential limitations of multimodal fusion. Finally, we spotlight the promising future of Transformer-based multimodal fusion techniques and give recommendations for future research in this rapidly evolving field.

Northwestern University resource and education development initiatives to advance collaborative artificial intelligence across the learning health system

Introduction

The rapid development of artificial intelligence (AI) in healthcare has exposed the unmet need for growing a multidisciplinary workforce that can collaborate effectively in the learning health systems. Maximizing the synergy among multiple teams is critical for Collaborative AI in Healthcare.

Methods

We have developed a series of data, tools, and educational resources for cultivating the next generation of multidisciplinary workforce for Collaborative AI in Healthcare. We built bulk-natural language processing pipelines to extract structured information from clinical notes and stored them in common data models. We developed multimodal AI/machine learning (ML) tools and tutorials to enrich the toolbox of the multidisciplinary workforce to analyze multimodal healthcare data. We have created a fertile ground to cross-pollinate clinicians and AI scientists and train the next generation of AI health workforce to collaborate effectively.

Results

Our work has democratized access to unstructured health information, AI/ML tools and resources for healthcare, and collaborative education resources. From 2017 to 2022, this has enabled studies in multiple clinical specialties resulting in 68 peer-reviewed publications. In 2022, our cross-discipline efforts converged and institutionalized into the Center for Collaborative AI in Healthcare.

Conclusions

Our Collaborative AI in Healthcare initiatives has created valuable educational and practical resources. They have enabled more clinicians, scientists, and hospital administrators to successfully apply AI methods in their daily research and practice, develop closer collaborations, and advanced the institution-level learning health system.

Converting OMOP CDM to phenopackets: A model alignment and patient data representation evaluation

OBJECTIVE

This study aims to promote interoperability in precision medicine and translational research by aligning the Observational Medical Outcomes Partnership (OMOP) and Phenopackets data models. Phenopackets is an expert knowledge-driven schema designed to facilitate the storage and exchange of multimodal patient data, and support downstream analysis. The first goal of this paper is to explore model alignment by characterizing the common data models using a newly developed data transformation process and evaluation method. Second, using OMOP normalized clinical data, we evaluate the mapping of real-world patient data to Phenopackets. We evaluate the suitability of Phenopackets as a patient data representation for real-world clinical cases.

METHODS

We identified mappings between OMOP and Phenopackets and applied them to a real patient dataset to assess the transformation's success. We analyzed gaps between the models and identified key considerations for transforming data between them. Further, to improve ambiguous alignment, we incorporated Unified Medical Language System (UMLS) semantic type-based filtering to direct individual concepts to their most appropriate domain and conducted a domain-expert evaluation of the mapping's clinical utility.

RESULTS

The OMOP to Phenopacket transformation pipeline was executed for 1,000 Alzheimer's disease patients and successfully mapped all required entities. However, due to missing values in OMOP for required Phenopacket attributes, 10.2 % of records were lost. The use of UMLS-semantic type filtering for ambiguous alignment of individual concepts resulted in 96 % agreement with clinical thinking, increased from 68 % when mapping exclusively by domain correspondence.

CONCLUSION

This study presents a pipeline to transform data from OMOP to Phenopackets. We identified considerations for the transformation to ensure data quality, handling restrictions for successful Phenopacket validation and discrepant data formats. We identified unmappable Phenopacket attributes that focus on specialty use cases, such as genomics or oncology, which OMOP does not currently support. We introduce UMLS semantic type filtering to resolve ambiguous alignment to Phenopacket entities to be most appropriate for real-world interpretation. We provide a systematic approach to align OMOP and Phenopackets schemas. Our work facilitates future use of Phenopackets in clinical applications by addressing key barriers to interoperability when deriving a Phenopacket from real-world patient data.

Multimodal Machine Learning for Prediction of 30-Day Readmission Risk in Elderly Population

BACKGROUND

Readmission within 30 days is a prevalent issue among elderly patients, linked to unfavorable health outcomes. Our objective was to develop and validate multimodal machine learning models for predicting 30-day readmission risk in elderly patients discharged from internal medicine departments.

METHODS

This was a retrospective cohort study which included elderly patients aged 75 or older, who were hospitalized at the Hadassah Medical Center internal medicine departments between 2014 and 2020. Three machine learning algorithms were developed and employed to predict 30-day readmission risk. The primary measures were predictive model performance scores, specifically area under the receiver operator curve (AUROC), and average precision.

RESULTS

This study included 19,569 admissions. Of them, 3258 (16.65%) resulted in 30-day readmission. Our 3 proposed models demonstrated high accuracy and precision on an unseen test set, with AUROC values of 0.87, 0.89, and 0.93, respectively, and average precision values of 0.76, 0.78, and 0.81. Feature importance analysis revealed that the number of admissions in the past year, history of 30-day readmission, Charlson score, and admission length were the most influential variables. Notably, the natural language processing score, representing the probability of readmission according to a textual-based model trained on social workers' assessment letters during hospitalization, ranked among the top 10 contributing factors.

CONCLUSIONS

Leveraging multimodal machine learning offers a promising strategy for identifying elderly patients who are at high risk for 30-day readmission. By identifying these patients, machine learning models may facilitate the effective execution of preventive actions to reduce avoidable readmission incidents.