Multimodal machine learning in precision health: A scoping review

Seeing the random forest through the decision trees. Supporting learning health systems from histopathology with machine learning models: Challenges and opportunities

This paper discusses some overlooked challenges faced when working with machine learning models for histopathology and presents a novel opportunity to support "Learning Health Systems" with them. Initially, the authors elaborate on these challenges after separating them according to their mitigation strategies: those that need innovative approaches, time, or future technological capabilities and those that require a conceptual reappraisal from a critical perspective. Then, a novel opportunity to support "Learning Health Systems" by integrating hidden information extracted by ML models from digitalized histopathology slides with other healthcare big data is presented.

Perspectives on Advancing Multimodal Learning in Environmental Science and Engineering Studies

The environment faces increasing anthropogenic impacts, resulting in a rapid increase in environmental issues that undermine the natural capital essential for human wellbeing. These issues are complex and often influenced by various factors represented by data with different modalities. While machine learning (ML) provides data-driven tools for addressing the environmental issues, the current ML models in environmental science and engineering (ES&E) often neglect the utilization of multimodal data. With the advancement in deep learning, multimodal learning (MML) holds promise for comprehensive descriptions of the environmental issues by harnessing data from diverse modalities. This advancement has the potential to significantly elevate the accuracy and robustness of prediction models in ES&E studies, providing enhanced solutions for various environmental modeling tasks. This perspective summarizes MML methodologies and proposes potential applications of MML models in ES&E studies, including environmental quality assessment, prediction of chemical hazards, and optimization of pollution control techniques. Additionally, we discuss the challenges associated with implementing MML in ES&E and propose future research directions in this domain.

Advancing Chinese biomedical text mining with community challenges

OBJECTIVE

This study aims to review the recent advances in community challenges for biomedical text mining in China.

METHODS

We collected information of evaluation tasks released in community challenges of biomedical text mining, including task description, dataset description, data source, task type and related links. A systematic summary and comparative analysis were conducted on various biomedical natural language processing tasks, such as named entity recognition, entity normalization, attribute extraction, relation extraction, event extraction, text classification, text similarity, knowledge graph construction, question answering, text generation, and large language model evaluation.

RESULTS

We identified 39 evaluation tasks from 6 community challenges that spanned from 2017 to 2023. Our analysis revealed the diverse range of evaluation task types and data sources in biomedical text mining. We explored the potential clinical applications of these community challenge tasks from a translational biomedical informatics perspective. We compared with their English counterparts, and discussed the contributions, limitations, lessons and guidelines of these community challenges, while highlighting future directions in the era of large language models.

CONCLUSION

Community challenge evaluation competitions have played a crucial role in promoting technology innovation and fostering interdisciplinary collaboration in the field of biomedical text mining. These challenges provide valuable platforms for researchers to develop state-of-the-art solutions.

Emerging Analytical Approaches for Personalized Medicine Using Machine Learning In Pediatric and Congenital Heart Disease

Precision and personalized medicine, the process by which patient management is tailored to individual circumstances, are now terms that are familiar to cardiologists, despite it still being an emerging field. Although precision medicine relies most often on the underlying biology and pathophysiology of a patient's condition, personalized medicine relies on digital biomarkers generated through algorithms. Given the complexity of the underlying data, these digital biomarkers are most often generated through machine-learning algorithms. There are a number of analytic considerations regarding the creation of digital biomarkers that are discussed in this review, including data preprocessing, time dependency and gating, dimensionality reduction, and novel methods, both in the realm of supervised and unsupervised machine learning. Some of these considerations, such as sample size requirements and measurements of model performance, are particularly challenging in small and heterogeneous populations with rare outcomes such as children with congenital heart disease. Finally, we review analytic considerations for the deployment of digital biomarkers in clinical settings, including the emerging field of clinical artificial intelligence (AI) operations, computational needs for deployment, efforts to increase the explainability of AI, algorithmic drift, and the needs for distributed surveillance and federated learning. We conclude this review by discussing a recent simulation study that shows that, despite these analytic challenges and complications, the use of digital biomarkers in managing clinical care might have substantial benefits regarding individual patient outcomes.

Applications of Multimodal Artificial Intelligence in Non-Hodgkin Lymphoma B Cells

Given advancements in large-scale data and AI, integrating multimodal artificial intelligence into cancer research can enhance our understanding of tumor behavior by simultaneously processing diverse biomedical data types. In this review, we explore the potential of multimodal AI in comprehending B-cell non-Hodgkin lymphomas (B-NHLs). B-cell non-Hodgkin lymphomas (B-NHLs) represent a particular challenge in oncology due to tumor heterogeneity and the intricate ecosystem in which tumors develop. These complexities complicate diagnosis, prognosis, and therapy response, emphasizing the need to use sophisticated approaches to enhance personalized treatment strategies for better patient outcomes. Therefore, multimodal AI can be leveraged to synthesize critical information from available biomedical data such as clinical record, imaging, pathology and omics data, to picture the whole tumor. In this review, we first define various types of modalities, multimodal AI frameworks, and several applications in precision medicine. Then, we provide several examples of its usage in B-NHLs, for analyzing the complexity of the ecosystem, identifying immune biomarkers, optimizing therapy strategy, and its clinical applications. Lastly, we address the limitations and future directions of multimodal AI, highlighting the need to overcome these challenges for better clinical practice and application in healthcare.

Large Language Models for Wearable Sensor-Based Human Activity Recognition, Health Monitoring, and Behavioral Modeling: A Survey of Early Trends, Datasets, and Challenges

The proliferation of wearable technology enables the generation of vast amounts of sensor data, offering significant opportunities for advancements in health monitoring, activity recognition, and personalized medicine. However, the complexity and volume of these data present substantial challenges in data modeling and analysis, which have been addressed with approaches spanning time series modeling to deep learning techniques. The latest frontier in this domain is the adoption of large language models (LLMs), such as GPT-4 and Llama, for data analysis, modeling, understanding, and human behavior monitoring through the lens of wearable sensor data. This survey explores the current trends and challenges in applying LLMs for sensor-based human activity recognition and behavior modeling. We discuss the nature of wearable sensor data, the capabilities and limitations of LLMs in modeling them, and their integration with traditional machine learning techniques. We also identify key challenges, including data quality, computational requirements, interpretability, and privacy concerns. By examining case studies and successful applications, we highlight the potential of LLMs in enhancing the analysis and interpretation of wearable sensor data. Finally, we propose future directions for research, emphasizing the need for improved preprocessing techniques, more efficient and scalable models, and interdisciplinary collaboration. This survey aims to provide a comprehensive overview of the intersection between wearable sensor data and LLMs, offering insights into the current state and future prospects of this emerging field.

The Emerging and Important Role of Artificial Intelligence in Cardiac Surgery

Artificial Intelligence (AI) has greatly affected our everyday lives and holds great promise to change the landscape of medicine. AI is particularly positioned to improve care for the increasingly complex patients undergoing cardiac surgery utilizing immense amount of data generated in the course of their care. When deployed, AI can be used to analyze this information at the patient's bedside more expediently and accurately, all while providing new insights. This review summarizes the current applications of AI in cardiac surgery, from the vantage point of a patient's journey. Applications of AI include pre-operative risk assessment, intraoperative planning, post-operative patient care and out-patient telemonitoring, encompassing the spectrum of cardiac surgical care. Offloading of administrative processes and enhanced experience with information gathering also represent a unique and underrepresented avenue for future utilization of AI. As clinicians, understanding the nomenclature and applications of AI is important to contextualize problems, to ensure problem-driven solutions and for clinical benefit. Precision medicine, and thus clinically relevant AI, remains dependent on data curation and warehousing to gather insights from large multicenter repositories while treating privacy with the utmost importance. AI tasks should not be siloed but rather holistically integrated into clinical workflow to retain context and relevance. As cardiac surgeons, AI allows us to look forward to a bright future of more efficient utilization of our clinical expertise toward high-level decision making and technical prowess.

Graph Artificial Intelligence in Medicine

In clinical artificial intelligence (AI), graph representation learning, mainly through graph neural networks and graph transformer architectures, stands out for its capability to capture intricate relationships and structures within clinical datasets. With diverse data-from patient records to imaging-graph AI models process data holistically by viewing modalities and entities within them as nodes interconnected by their relationships. Graph AI facilitates model transfer across clinical tasks, enabling models to generalize across patient populations without additional parameters and with minimal to no retraining. However, the importance of human-centered design and model interpretability in clinical decision-making cannot be overstated. Since graph AI models capture information through localized neural transformations defined on relational datasets, they offer both an opportunity and a challenge in elucidating model rationale. Knowledge graphs can enhance interpretability by aligning model-driven insights with medical knowledge. Emerging graph AI models integrate diverse data modalities through pretraining, facilitate interactive feedback loops, and foster human-AI collaboration, paving the way toward clinically meaningful predictions.

Non-Invasive Detection of Early-Stage Fatty Liver Disease via an On-Skin Impedance Sensor and Attention-Based Deep Learning

Early-stage nonalcoholic fatty liver disease (NAFLD) is a silent condition, with most cases going undiagnosed, potentially progressing to liver cirrhosis and cancer. A non-invasive and cost-effective detection method for early-stage NAFLD detection is a public health priority but challenging. In this study, an adhesive, soft on-skin sensor with low electrode-skin contact impedance for early-stage NAFLD detection is fabricated. A method is developed to synthesize platinum nanoparticles and reduced graphene quantum dots onto the on-skin sensor to reduce electrode-skin contact impedance by increasing double-layer capacitance, thereby enhancing detection accuracy. Furthermore, an attention-based deep learning algorithm is introduced to differentiate impedance signals associated with early-stage NAFLD in high-fat-diet-fed low-density lipoprotein receptor knockout (Ldlr-/-) mice compared to healthy controls. The integration of an adhesive, soft on-skin sensor with low electrode-skin contact impedance and the attention-based deep learning algorithm significantly enhances the detection accuracy for early-stage NAFLD, achieving a rate above 97.5% with an area under the receiver operating characteristic curve (AUC) of 1.0. The findings present a non-invasive approach for early-stage NAFLD detection and display a strategy for improved early detection through on-skin electronics and deep learning.

A review of deep learning-based information fusion techniques for multimodal medical image classification

Multimodal medical imaging plays a pivotal role in clinical diagnosis and research, as it combines information from various imaging modalities to provide a more comprehensive understanding of the underlying pathology. Recently, deep learning-based multimodal fusion techniques have emerged as powerful tools for improving medical image classification. This review offers a thorough analysis of the developments in deep learning-based multimodal fusion for medical classification tasks. We explore the complementary relationships among prevalent clinical modalities and outline three main fusion schemes for multimodal classification networks: input fusion, intermediate fusion (encompassing single-level fusion, hierarchical fusion, and attention-based fusion), and output fusion. By evaluating the performance of these fusion techniques, we provide insight into the suitability of different network architectures for various multimodal fusion scenarios and application domains. Furthermore, we delve into challenges related to network architecture selection, handling incomplete multimodal data management, and the potential limitations of multimodal fusion. Finally, we spotlight the promising future of Transformer-based multimodal fusion techniques and give recommendations for future research in this rapidly evolving field.

Northwestern University resource and education development initiatives to advance collaborative artificial intelligence across the learning health system

Introduction

The rapid development of artificial intelligence (AI) in healthcare has exposed the unmet need for growing a multidisciplinary workforce that can collaborate effectively in the learning health systems. Maximizing the synergy among multiple teams is critical for Collaborative AI in Healthcare.

Methods

We have developed a series of data, tools, and educational resources for cultivating the next generation of multidisciplinary workforce for Collaborative AI in Healthcare. We built bulk-natural language processing pipelines to extract structured information from clinical notes and stored them in common data models. We developed multimodal AI/machine learning (ML) tools and tutorials to enrich the toolbox of the multidisciplinary workforce to analyze multimodal healthcare data. We have created a fertile ground to cross-pollinate clinicians and AI scientists and train the next generation of AI health workforce to collaborate effectively.

Results

Our work has democratized access to unstructured health information, AI/ML tools and resources for healthcare, and collaborative education resources. From 2017 to 2022, this has enabled studies in multiple clinical specialties resulting in 68 peer-reviewed publications. In 2022, our cross-discipline efforts converged and institutionalized into the Center for Collaborative AI in Healthcare.

Conclusions

Our Collaborative AI in Healthcare initiatives has created valuable educational and practical resources. They have enabled more clinicians, scientists, and hospital administrators to successfully apply AI methods in their daily research and practice, develop closer collaborations, and advanced the institution-level learning health system.

Converting OMOP CDM to phenopackets: A model alignment and patient data representation evaluation

OBJECTIVE

This study aims to promote interoperability in precision medicine and translational research by aligning the Observational Medical Outcomes Partnership (OMOP) and Phenopackets data models. Phenopackets is an expert knowledge-driven schema designed to facilitate the storage and exchange of multimodal patient data, and support downstream analysis. The first goal of this paper is to explore model alignment by characterizing the common data models using a newly developed data transformation process and evaluation method. Second, using OMOP normalized clinical data, we evaluate the mapping of real-world patient data to Phenopackets. We evaluate the suitability of Phenopackets as a patient data representation for real-world clinical cases.

METHODS

We identified mappings between OMOP and Phenopackets and applied them to a real patient dataset to assess the transformation's success. We analyzed gaps between the models and identified key considerations for transforming data between them. Further, to improve ambiguous alignment, we incorporated Unified Medical Language System (UMLS) semantic type-based filtering to direct individual concepts to their most appropriate domain and conducted a domain-expert evaluation of the mapping's clinical utility.

RESULTS

The OMOP to Phenopacket transformation pipeline was executed for 1,000 Alzheimer's disease patients and successfully mapped all required entities. However, due to missing values in OMOP for required Phenopacket attributes, 10.2 % of records were lost. The use of UMLS-semantic type filtering for ambiguous alignment of individual concepts resulted in 96 % agreement with clinical thinking, increased from 68 % when mapping exclusively by domain correspondence.

CONCLUSION

This study presents a pipeline to transform data from OMOP to Phenopackets. We identified considerations for the transformation to ensure data quality, handling restrictions for successful Phenopacket validation and discrepant data formats. We identified unmappable Phenopacket attributes that focus on specialty use cases, such as genomics or oncology, which OMOP does not currently support. We introduce UMLS semantic type filtering to resolve ambiguous alignment to Phenopacket entities to be most appropriate for real-world interpretation. We provide a systematic approach to align OMOP and Phenopackets schemas. Our work facilitates future use of Phenopackets in clinical applications by addressing key barriers to interoperability when deriving a Phenopacket from real-world patient data.

Multimodal Machine Learning for Prediction of 30-Day Readmission Risk in Elderly Population

BACKGROUND

Readmission within 30 days is a prevalent issue among elderly patients, linked to unfavorable health outcomes. Our objective was to develop and validate multimodal machine learning models for predicting 30-day readmission risk in elderly patients discharged from internal medicine departments.

METHODS

This was a retrospective cohort study which included elderly patients aged 75 or older, who were hospitalized at the Hadassah Medical Center internal medicine departments between 2014 and 2020. Three machine learning algorithms were developed and employed to predict 30-day readmission risk. The primary measures were predictive model performance scores, specifically area under the receiver operator curve (AUROC), and average precision.

RESULTS

This study included 19,569 admissions. Of them, 3258 (16.65%) resulted in 30-day readmission. Our 3 proposed models demonstrated high accuracy and precision on an unseen test set, with AUROC values of 0.87, 0.89, and 0.93, respectively, and average precision values of 0.76, 0.78, and 0.81. Feature importance analysis revealed that the number of admissions in the past year, history of 30-day readmission, Charlson score, and admission length were the most influential variables. Notably, the natural language processing score, representing the probability of readmission according to a textual-based model trained on social workers' assessment letters during hospitalization, ranked among the top 10 contributing factors.

CONCLUSIONS

Leveraging multimodal machine learning offers a promising strategy for identifying elderly patients who are at high risk for 30-day readmission. By identifying these patients, machine learning models may facilitate the effective execution of preventive actions to reduce avoidable readmission incidents.

Multimodal fusion for anticipating human decision performance

Anticipating human decisions while performing complex tasks remains a formidable challenge. This study proposes a multimodal machine-learning approach that leverages image features and electroencephalography (EEG) data to predict human response correctness in a demanding visual searching task. Notably, we extract a novel set of image features pertaining to object relationships using the Segment Anything Model (SAM), which enhances prediction accuracy compared to traditional features. Additionally, our approach effectively utilizes a combination of EEG signals and image features to streamline the feature set required for the Random Forest Classifier (RFC) while maintaining high accuracy. The findings of this research hold substantial potential for developing advanced fault alert systems, particularly in critical decision-making environments such as the medical and defence sectors.

Visualizing Clinical Data Retrieval and Curation in Multimodal Healthcare AI Research: A Technical Note on RIL-workflow

Curating and integrating data from sources are bottlenecks to procuring robust training datasets for artificial intelligence (AI) models in healthcare. While numerous applications can process discrete types of clinical data, it is still time-consuming to integrate heterogenous data types. Therefore, there exists a need for more efficient retrieval and storage of curated patient data from dissimilar sources, such as biobanks, health records, and sensors. We describe a customizable, modular data retrieval application (RIL-workflow), which integrates clinical notes, images, and prescription data, and show its feasibility applied to research at our institution. It uses the workflow automation platform Camunda (Camunda Services GmbH, Berlin, Germany) to collect internal data from Fast Healthcare Interoperability Resources (FHIR) and Digital Imaging and Communications in Medicine (DICOM) sources. Using the web-based graphical user interface (GUI), the workflow runs tasks to completion according to visual representation, retrieving and storing results for patients meeting study inclusion criteria while segregating errors for human review. We showcase RIL-workflow with its library of ready-to-use modules, enabling researchers to specify human input or automation at fixed steps. We validated our workflow by demonstrating its capability to aggregate, curate, and handle errors related to data from multiple sources to generate a multimodal database for clinical AI research. Further, we solicited user feedback to highlight the pros and cons associated with RIL-workflow. The source code is available at github.com/magnooj/RIL-workflow.

Enhancing psychiatric rehabilitation outcomes through a multimodal multitask learning model based on BERT and TabNet: An approach for personalized treatment and improved decision-making

Evaluating the rehabilitation status of individuals with serious mental illnesses (SMI) necessitates a comprehensive analysis of multimodal data, including unstructured text records and structured diagnostic data. However, progress in the effective assessment of rehabilitation status remains limited. Our study develops a deep learning model integrating Bidirectional Encoder Representations from Transformers (BERT) and TabNet through a late fusion strategy to enhance rehabilitation prediction, including referral risk, dangerous behaviors, self-awareness, and medication adherence, in patients with SMI. BERT processes unstructured textual data, such as doctor's notes, whereas TabNet manages structured diagnostic information. The model's interpretability function serves to assist healthcare professionals in understanding the model's predictive decisions, improving patient care. Our model exhibited excellent predictive performance for all four tasks, with an accuracy exceeding 0.78 and an area under the curve of 0.70. In addition, a series of tests proved the model's robustness, fairness, and interpretability. This study combines multimodal and multitask learning strategies into a model and applies it to rehabilitation assessment tasks, offering a promising new tool that can be seamlessly integrated with the clinical workflow to support the provision of optimized patient care.

Computational Approaches for Connecting Maternal Stress to Preterm Birth

Multiple studies have hinted at a complex connection between maternal stress and preterm birth (PTB). This article describes the potential of computational methods to provide new insights into this relationship. For this, we outline existing approaches for stress assessments and various data modalities available for profiling stress responses, and review studies that sought either to establish a connection between stress and PTB or to predict PTB based on stress-related factors. Finally, we summarize the challenges of computational methods, highlighting potential future research directions within this field.

Simple and Fast Prediction of Gestational Diabetes Mellitus Based on Machine Learning and Near-Infrared Spectra of Serum: A Proof of Concept Study at Different Stages of Pregnancy

Gestational diabetes mellitus (GDM) is a hyperglycemic state that is typically diagnosed by an oral glucose tolerance test (OGTT), which is unpleasant, time-consuming, has low reproducibility, and results are tardy. The machine learning (ML) predictive models that have been proposed to improve GDM diagnosis are usually based on instrumental methods that take hours to produce a result. Near-infrared (NIR) spectroscopy is a simple, fast, and low-cost analytical technique that has never been assessed for the prediction of GDM. This study aims to develop ML predictive models for GDM based on NIR spectroscopy, and to evaluate their potential as early detection or alternative screening tools according to their predictive power and duration of analysis. Serum samples from the first trimester (before GDM diagnosis) and the second trimester (at the time of GDM diagnosis) of pregnancy were analyzed by NIR spectroscopy. Four spectral ranges were considered, and 80 mathematical pretreatments were tested for each. NIR data-based models were built with single- and multi-block ML techniques. Every model was subjected to double cross-validation. The best models for first and second trimester achieved areas under the receiver operating characteristic curve of 0.5768 ± 0.0635 and 0.8836 ± 0.0259, respectively. This is the first study reporting NIR-spectroscopy-based methods for the prediction of GDM. The developed methods allow for prediction of GDM from 10 µL of serum in only 32 min. They are simple, fast, and have a great potential for application in clinical practice, especially as alternative screening tools to the OGTT for GDM diagnosis.

Multimodal Learning for Mapping the Genotype-Phenotype Dynamics

How complex phenotypes emerge from intricate gene expression patterns is a fundamental question in biology. Quantitative characterization of this relationship, however, is challenging due to the vast combinatorial possibilities and dynamic interplay between genotype and phenotype landscapes. Integrating high-content genotyping approaches such as single-cell RNA sequencing and advanced learning methods such as language models offers an opportunity for dissecting this complex relationship. Here, we present a computational integrated genetics framework designed to analyze and interpret the high-dimensional landscape of genotypes and their associated phenotypes simultaneously. We applied this approach to develop a multimodal foundation model to explore the genotype-phenotype relationship manifold for human transcriptomics at the cellular level. Analyzing this joint manifold showed a refined resolution of cellular heterogeneity, enhanced precision in phenotype annotating, and uncovered potential cross-tissue biomarkers that are undetectable through conventional gene expression analysis alone. Moreover, our results revealed that the gene networks are characterized by scale-free patterns and show context-dependent gene-gene interactions, both of which result in significant variations in the topology of the gene network, particularly evident during aging. Finally, utilizing contextualized embeddings, we investigated gene polyfunctionality which illustrates the multifaceted roles that genes play in different biological processes, and demonstrated that for VWF gene in endothelial cells. Overall, this study advances our understanding of the dynamic interplay between gene expression and phenotypic manifestation and demonstrates the potential of integrated genetics in uncovering new dimensions of cellular function and complexity.

PheSeq, a Bayesian deep learning model to enhance and interpret the gene-disease association studies

Despite the abundance of genotype-phenotype association studies, the resulting association outcomes often lack robustness and interpretations. To address these challenges, we introduce PheSeq, a Bayesian deep learning model that enhances and interprets association studies through the integration and perception of phenotype descriptions. By implementing the PheSeq model in three case studies on Alzheimer's disease, breast cancer, and lung cancer, we identify 1024 priority genes for Alzheimer's disease and 818 and 566 genes for breast cancer and lung cancer, respectively. Benefiting from data fusion, these findings represent moderate positive rates, high recall rates, and interpretation in gene-disease association studies.

Transforming the cardiometabolic disease landscape: Multimodal AI-powered approaches in prevention and management

The rise of artificial intelligence (AI) has revolutionized various scientific fields, particularly in medicine, where it has enabled the modeling of complex relationships from massive datasets. Initially, AI algorithms focused on improved interpretation of diagnostic studies such as chest X-rays and electrocardiograms in addition to predicting patient outcomes and future disease onset. However, AI has evolved with the introduction of transformer models, allowing analysis of the diverse, multimodal data sources existing in medicine today. Multimodal AI holds great promise in more accurate disease risk assessment and stratification as well as optimizing the key driving factors in cardiometabolic disease: blood pressure, sleep, stress, glucose control, weight, nutrition, and physical activity. In this article we outline the current state of medical AI in cardiometabolic disease, highlighting the potential of multimodal AI to augment personalized prevention and treatment strategies in cardiometabolic disease.

Large language models approach expert-level clinical knowledge and reasoning in ophthalmology: A head-to-head cross-sectional study

Large language models (LLMs) underlie remarkable recent advanced in natural language processing, and they are beginning to be applied in clinical contexts. We aimed to evaluate the clinical potential of state-of-the-art LLMs in ophthalmology using a more robust benchmark than raw examination scores. We trialled GPT-3.5 and GPT-4 on 347 ophthalmology questions before GPT-3.5, GPT-4, PaLM 2, LLaMA, expert ophthalmologists, and doctors in training were trialled on a mock examination of 87 questions. Performance was analysed with respect to question subject and type (first order recall and higher order reasoning). Masked ophthalmologists graded the accuracy, relevance, and overall preference of GPT-3.5 and GPT-4 responses to the same questions. The performance of GPT-4 (69%) was superior to GPT-3.5 (48%), LLaMA (32%), and PaLM 2 (56%). GPT-4 compared favourably with expert ophthalmologists (median 76%, range 64-90%), ophthalmology trainees (median 59%, range 57-63%), and unspecialised junior doctors (median 43%, range 41-44%). Low agreement between LLMs and doctors reflected idiosyncratic differences in knowledge and reasoning with overall consistency across subjects and types (p>0.05). All ophthalmologists preferred GPT-4 responses over GPT-3.5 and rated the accuracy and relevance of GPT-4 as higher (p<0.05). LLMs are approaching expert-level knowledge and reasoning skills in ophthalmology. In view of the comparable or superior performance to trainee-grade ophthalmologists and unspecialised junior doctors, state-of-the-art LLMs such as GPT-4 may provide useful medical advice and assistance where access to expert ophthalmologists is limited. Clinical benchmarks provide useful assays of LLM capabilities in healthcare before clinical trials can be designed and conducted.

Utilizing multimodal AI to improve genetic analyses of cardiovascular traits

Electronic health records, biobanks, and wearable biosensors contain multiple high-dimensional clinical data (HDCD) modalities (e.g., ECG, Photoplethysmography (PPG), and MRI) for each individual. Access to multimodal HDCD provides a unique opportunity for genetic studies of complex traits because different modalities relevant to a single physiological system (e.g., circulatory system) encode complementary and overlapping information. We propose a novel multimodal deep learning method, M-REGLE, for discovering genetic associations from a joint representation of multiple complementary HDCD modalities. We showcase the effectiveness of this model by applying it to several cardiovascular modalities. M-REGLE jointly learns a lower representation (i.e., latent factors) of multimodal HDCD using a convolutional variational autoencoder, performs genome wide association studies (GWAS) on each latent factor, then combines the results to study the genetics of the underlying system. To validate the advantages of M-REGLE and multimodal learning, we apply it to common cardiovascular modalities (PPG and ECG), and compare its results to unimodal learning methods in which representations are learned from each data modality separately, but the downstream genetic analyses are performed on the combined unimodal representations. M-REGLE identifies 19.3% more loci on the 12-lead ECG dataset, 13.0% more loci on the ECG lead I + PPG dataset, and its genetic risk score significantly outperforms the unimodal risk score at predicting cardiac phenotypes, such as atrial fibrillation (Afib), in multiple biobanks.

Generative AI in healthcare: an implementation science informed translational path on application, integration and governance

BACKGROUND

Artificial intelligence (AI), particularly generative AI, has emerged as a transformative tool in healthcare, with the potential to revolutionize clinical decision-making and improve health outcomes. Generative AI, capable of generating new data such as text and images, holds promise in enhancing patient care, revolutionizing disease diagnosis and expanding treatment options. However, the utility and impact of generative AI in healthcare remain poorly understood, with concerns around ethical and medico-legal implications, integration into healthcare service delivery and workforce utilisation. Also, there is not a clear pathway to implement and integrate generative AI in healthcare delivery.

METHODS

This article aims to provide a comprehensive overview of the use of generative AI in healthcare, focusing on the utility of the technology in healthcare and its translational application highlighting the need for careful planning, execution and management of expectations in adopting generative AI in clinical medicine. Key considerations include factors such as data privacy, security and the irreplaceable role of clinicians' expertise. Frameworks like the technology acceptance model (TAM) and the Non-Adoption, Abandonment, Scale-up, Spread and Sustainability (NASSS) model are considered to promote responsible integration. These frameworks allow anticipating and proactively addressing barriers to adoption, facilitating stakeholder participation and responsibly transitioning care systems to harness generative AI's potential.

RESULTS

Generative AI has the potential to transform healthcare through automated systems, enhanced clinical decision-making and democratization of expertise with diagnostic support tools providing timely, personalized suggestions. Generative AI applications across billing, diagnosis, treatment and research can also make healthcare delivery more efficient, equitable and effective. However, integration of generative AI necessitates meticulous change management and risk mitigation strategies. Technological capabilities alone cannot shift complex care ecosystems overnight; rather, structured adoption programs grounded in implementation science are imperative.

CONCLUSIONS

It is strongly argued in this article that generative AI can usher in tremendous healthcare progress, if introduced responsibly. Strategic adoption based on implementation science, incremental deployment and balanced messaging around opportunities versus limitations helps promote safe, ethical generative AI integration. Extensive real-world piloting and iteration aligned to clinical priorities should drive development. With conscientious governance centred on human wellbeing over technological novelty, generative AI can enhance accessibility, affordability and quality of care. As these models continue advancing rapidly, ongoing reassessment and transparent communication around their strengths and weaknesses remain vital to restoring trust, realizing positive potential and, most importantly, improving patient outcomes.

Multimodal risk prediction with physiological signals, medical images and clinical notes

The broad adoption of electronic health record (EHR) systems brings us a tremendous amount of clinical data and thus provides opportunities to conduct data-based healthcare research to solve various clinical problems in the medical domain. Machine learning and deep learning methods are widely used in the medical informatics and healthcare domain due to their power to mine insights from raw data. When adapting deep learning models for EHR data, it is essential to consider its heterogeneous nature: EHR contains patient records from various sources including medical tests (e.g. blood test, microbiology test), medical imaging, diagnosis, medications, procedures, clinical notes, etc. Those modalities together provide a holistic view of patient health status and complement each other. Therefore, combining data from multiple modalities that are intrinsically different is challenging but intuitively promising in deep learning for EHR. To assess the expectations of multimodal data, we introduce a comprehensive fusion framework designed to integrate temporal variables, medical images, and clinical notes in EHR for enhanced performance in clinical risk prediction. Early, joint, and late fusion strategies are employed to combine data from various modalities effectively. We test the model with three predictive tasks: in-hospital mortality, long length of stay, and 30-day readmission. Experimental results show that multimodal models outperform uni-modal models in the tasks involved. Additionally, by training models with different input modality combinations, we calculate the Shapley value for each modality to quantify their contribution to multimodal performance. It is shown that temporal variables tend to be more helpful than CXR images and clinical notes in the three explored predictive tasks.

Deep learning methods improve genomic prediction of wheat breeding

In the field of plant breeding, various machine learning models have been developed and studied to evaluate the genomic prediction (GP) accuracy of unseen phenotypes. Deep learning has shown promise. However, most studies on deep learning in plant breeding have been limited to small datasets, and only a few have explored its application in moderate-sized datasets. In this study, we aimed to address this limitation by utilizing a moderately large dataset. We examined the performance of a deep learning (DL) model and compared it with the widely used and powerful best linear unbiased prediction (GBLUP) model. The goal was to assess the GP accuracy in the context of a five-fold cross-validation strategy and when predicting complete environments using the DL model. The results revealed the DL model outperformed the GBLUP model in terms of GP accuracy for two out of the five included traits in the five-fold cross-validation strategy, with similar results in the other traits. This indicates the superiority of the DL model in predicting these specific traits. Furthermore, when predicting complete environments using the leave-one-environment-out (LOEO) approach, the DL model demonstrated competitive performance. It is worth noting that the DL model employed in this study extends a previously proposed multi-modal DL model, which had been primarily applied to image data but with small datasets. By utilizing a moderately large dataset, we were able to evaluate the performance and potential of the DL model in a context with more information and challenging scenario in plant breeding.

Oxygenation Fluctuations Associated with Severe Retinopathy of Prematurity: Insights from a Multimodal Deep Learning Approach

Purpose

Retinopathy of prematurity (ROP) is one of the leading causes of blindness in children. Although the role of oxygen in the pathophysiology of ROP is well established, a precise understanding of the dynamic relationship between oxygen exposure ROP incidence and severity is lacking. The purpose of this study was to evaluate the correlation between time-dependent oxygen variables and the onset of ROP.

Design

Retrospective cohort study.

Participants

Two hundred thirty infants who were born at a single academic center and met the inclusion criteria were included. Infants are mainly born between January 2011 and October 2022.

Methods

Patient data were extracted from electronic health records (EHRs), with sufficient time-dependent oxygen data. Clinical outcomes for ROP were recorded as none/mild or moderate/severe (defined as type II or worse). Mixed-effects linear models were used to compare the 2 groups in terms of dynamic oxygen variables, such as daily average and the coefficient of variation (COV) fraction of inspired oxygen (FiO2). Support vector machine (SVM) and long-short-term memory (LSTM)-based multimodal models were trained with fivefold cross-validation to predict which infants would develop moderate/severe ROP. Gestational age (GA), birth weight, and time-dependent oxygen variables were used to develop predictive models.

Main Outcome Measures

Model cross-validation performance was evaluated by computing the mean area under the receiver operating characteristic (AUROC) curve, precision, recall, and F1 score.

Results

We found that both daily average and COV of FiO2 were associated with more severe ROP (adjusted P < 0.001). With fivefold cross-validation, the multimodal LSTM models had higher performance than the best static models (SVM using GA and 3 average FiO2 features) and SVM models trained on GA alone (mean AUROC = 0.89 ± 0.04 vs. 0.86 ± 0.05 vs. 0.83 ± 0.04).

Conclusions

The development of severe ROP might not only be influenced by oxygen exposure but also by its fluctuation, which provides direction for future study of pathophysiological factors associated with severe ROP development. Additionally, we demonstrated that multimodal neural networks can be a method to extract useful information from time-series data, which may be a valuable methodology for the investigation of other diseases using EHR data.

Financial Disclosures

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

NIMEQ-SACNet: A novel self-attention precision medicine model for vision-threatening diabetic retinopathy using image data

In the realm of precision medicine, the potential of deep learning is progressively harnessed to facilitate intricate clinical decision-making, especially when navigating multifaceted datasets encompassing Omics, Clinical, image, device, social, and environmental dimensions. This study accentuates the criticality of image data, given its instrumental role in detecting and classifying vision-threatening diabetic retinopathy (VTDR) - a predominant global contributor to vision impairment. The timely identification of VTDR is a linchpin for efficacious interventions and the mitigation of vision loss. Addressing this, This study introduces "NIMEQ-SACNet," a novel hybrid model by the prowess of the Enhanced Quantum-Inspired Binary Grey Wolf Optimizer (EQI-BGWO) with a self-attention capsule network. The proposed approach is characterized by two pivotal advancements: firstly, the augmentation of the Binary Grey Wolf Optimization through Quantum Computing methodologies, and secondly, the deployment of the enhanced EQI-BGWO to adeptly calibrate the SACNet's parameters, culminating in a notable uplift in VTDR classification accuracy. The proposed model's ability to handle binary, 5-stage, and 7-stage VTDR classifications adroitly is noteworthy. Rigorous assessments on the fundus image dataset, underscored by metrics such as Accuracy, Sensitivity, Specificity, Precision, F1-Score, and MCC, bear testament to NIMEQ-SACNet's pre-eminence over prevailing algorithms and classification frameworks.

Systematic review of automatic post-stroke gait classification systems

BACKGROUND

Gait classification is a clinically helpful task performed after a stroke in order to guide rehabilitation therapy. Gait disorders are commonly identified using observational gait analysis in clinical settings, but this approach is limited due to low reliability and accuracy. Data-driven gait classification can quantify gait deviations and categorise gait patterns automatically possibly improving reliability and accuracy; however, the development and clinical utility of current data driven systems has not been reviewed previously.

RESEARCH QUESTION

The purpose of this systematic review is to evaluate the literature surrounding the methodology used to develop automatic gait classification systems, and their potential effectiveness in the clinical management of stroke-affected gait.

METHOD

The database search included PubMed, IEEE Xplore, and Scopus. Twenty-one studies were identified through inclusion and exclusion criteria from 407 available studies published between 2015 and 2022. Development methodology, classification performance, and clinical utility information were extracted for review.

RESULTS AND SIGNIFICANCE

Most of gait classification systems reported a classification accuracy between 80%-100%. However, collated studies presented methodological errors in machine learning (ML) model development. Further, many studies neglected model components such as clinical utility (e.g., predictions don't assist clinicians or therapists in making decisions, interpretability, and generalisability). We provided recommendations to guide development of future post-stroke automatic gait classification systems to better assist clinicians and therapists. Future automatic gait classification systems should emphasise the clinical significance and adopt a standardised development methodology of ML model.

Complication Prediction after Esophagectomy with Machine Learning

Esophageal cancer can be treated effectively with esophagectomy; however, the postoperative complication rate is high. In this paper, we study to what extent machine learning methods can predict anastomotic leakage and pneumonia up to two days in advance. We use a dataset with 417 patients who underwent esophagectomy between 2011 and 2021. The dataset contains multimodal temporal information, specifically, laboratory results, vital signs, thorax images, and preoperative patient characteristics. The best models scored mean test set AUROCs of 0.87 and 0.82 for leakage 1 and 2 days ahead, respectively. For pneumonia, this was 0.74 and 0.61 for 1 and 2 days ahead, respectively. We conclude that machine learning models can effectively predict anastomotic leakage and pneumonia after esophagectomy.

Integrative analysis of multimodal patient data identifies personalized predictors of tuberculosis treatment prognosis

Tuberculosis (TB) afflicted 10.6 million people in 2021, and its global burden is increasing due to multidrug-resistant TB (MDR-TB) and extensively resistant TB (XDR-TB). Here, we analyze multi-domain information from 5,060 TB patients spanning 10 countries with high burden of MDR-TB from the NIAID TB Portals database to determine predictors of TB treatment outcome. Our analysis revealed significant associations between radiological, microbiological, therapeutic, and demographic data modalities. Our machine learning model, built with 203 features across modalities outperforms models built using each modality alone in predicting treatment outcomes, with an accuracy of 83% and area under the curve of 0.84. Notably, our analysis revealed that the drug regimens Bedaquiline-Clofazimine-Cycloserine-Levofloxacin-Linezolid and Bedaquiline-Clofazimine-Linezolid-Moxifloxacin were associated with treatment success and failure, respectively, for MDR non-XDR-TB. Drug combinations predicted to be synergistic by the INDIGO algorithm performed better than antagonistic combinations. Our prioritized set of features predictive of treatment outcomes can ultimately guide the personalized clinical management of TB.

Learning across diverse biomedical data modalities and cohorts: Challenges and opportunities for innovation

In healthcare, machine learning (ML) shows significant potential to augment patient care, improve population health, and streamline healthcare workflows. Realizing its full potential is, however, often hampered by concerns about data privacy, diversity in data sources, and suboptimal utilization of different data modalities. This review studies the utility of cross-cohort cross-category (C4) integration in such contexts: the process of combining information from diverse datasets distributed across distinct, secure sites. We argue that C4 approaches could pave the way for ML models that are both holistic and widely applicable. This paper provides a comprehensive overview of C4 in health care, including its present stage, potential opportunities, and associated challenges.

Multimodal machine learning for modeling infant head circumference, mothers' milk composition, and their shared environment

Links between human milk (HM) and infant development are poorly understood and often focus on individual HM components. Here we apply multi-modal predictive machine learning to study HM and head circumference (a proxy for brain development) among 1022 mother-infant dyads of the CHILD Cohort. We integrated HM data (19 oligosaccharides, 28 fatty acids, 3 hormones, 28 chemokines) with maternal and infant demographic, health, dietary and home environment data. Head circumference was significantly predictable at 3 and 12 months. Two of the most associated features were HM n3-polyunsaturated fatty acid C22:6n3 (docosahexaenoic acid, DHA; p = 9.6e-05) and maternal intake of fish (p = 4.1e-03), a key dietary source of DHA with established relationships to brain function. Thus, using a systems biology approach, we identified meaningful relationships between HM and brain development, which validates our statistical approach, gives credence to the novel associations we observed, and sets the foundation for further research with additional cohorts and HM analytes.

Decoding the exposome: data science methodologies and implications in exposome-wide association studies (ExWASs)

This paper explores the exposome concept and its role in elucidating the interplay between environmental exposures and human health. We introduce two key concepts critical for exposomics research. Firstly, we discuss the joint impact of genetics and environment on phenotypes, emphasizing the variance attributable to shared and nonshared environmental factors, underscoring the complexity of quantifying the exposome's influence on health outcomes. Secondly, we introduce the importance of advanced data-driven methods in large cohort studies for exposomic measurements. Here, we introduce the exposome-wide association study (ExWAS), an approach designed for systematic discovery of relationships between phenotypes and various exposures, identifying significant associations while controlling for multiple comparisons. We advocate for the standardized use of the term "exposome-wide association study, ExWAS," to facilitate clear communication and literature retrieval in this field. The paper aims to guide future health researchers in understanding and evaluating exposomic studies. Our discussion extends to emerging topics, such as FAIR Data Principles, biobanked healthcare datasets, and the functional exposome, outlining the future directions in exposomic research. This abstract provides a succinct overview of our comprehensive approach to understanding the complex dynamics of the exposome and its significant implications for human health.

Machine learning in onco-pharmacogenomics: a path to precision medicine with many challenges

Over the past two decades, Next-Generation Sequencing (NGS) has revolutionized the approach to cancer research. Applications of NGS include the identification of tumor specific alterations that can influence tumor pathobiology and also impact diagnosis, prognosis and therapeutic options. Pharmacogenomics (PGx) studies the role of inheritance of individual genetic patterns in drug response and has taken advantage of NGS technology as it provides access to high-throughput data that can, however, be difficult to manage. Machine learning (ML) has recently been used in the life sciences to discover hidden patterns from complex NGS data and to solve various PGx problems. In this review, we provide a comprehensive overview of the NGS approaches that can be employed and the different PGx studies implicating the use of NGS data. We also provide an excursus of the ML algorithms that can exert a role as fundamental strategies in the PGx field to improve personalized medicine in cancer.

Generating a multimodal artificial intelligence model to differentiate benign and malignant follicular neoplasms of the thyroid: A proof-of-concept study

BACKGROUND

Machine learning has been increasingly used to develop algorithms that can improve medical diagnostics and prognostication and has shown promise in improving the classification of thyroid ultrasound images. This proof-of-concept study aims to develop a multimodal machine-learning model to classify follicular carcinoma from adenoma.

METHODS

This is a retrospective study of patients with follicular adenoma or carcinoma at a single institution between 2010 and 2022. Demographics, imaging, and perioperative variables were collected. The region of interest was annotated on ultrasound and used to perform radiomics analysis. Imaging features and clinical variables were then used to create a random forest classifier to predict malignancy. Leave-one-out cross-validation was conducted to evaluate classifier performance using the area under the receiver operating characteristic curve.

RESULTS

Patients with follicular adenomas (n = 7) and carcinomas (n = 11) with complete imaging and perioperative data were included. A total of 910 features were extracted from each image. The t-distributed stochastic neighbor embedding method reduced the dimension to 2 primary represented components. The random forest classifier achieved an area under the receiver operating characteristic curve of 0.76 (clinical only), 0.29 (image only), and 0.79 (multimodal data).

CONCLUSION

Our multimodal machine learning model demonstrates promising results in classifying follicular carcinoma from adenoma. This approach can potentially be applied in future studies to generate models for preoperative differentiation of follicular thyroid neoplasms.

Recompensation in cirrhosis: unravelling the evolving natural history of nonalcoholic fatty liver disease

Recompensation has gained increasing attention in the field of cirrhosis, particularly in chronic liver disease with a definite aetiology. The current global prevalence of obesity and nonalcoholic fatty liver disease (NAFLD) is increasing, but there is currently a lack of a clear definition for recompensation in NAFLD-related cirrhosis. Here, we provide an up-to-date perspective on the natural history of NAFLD, emphasizing the reversible nature of the disease, summarizing possible mechanisms underlying recompensation in NAFLD, discussing challenges that need to be addressed and outlining future research directions in the field. Recompensation is a promising goal in patients with NAFLD-related cirrhosis, and further studies are needed to explore its underlying mechanisms and uncover its clinical features.

Convert-Pheno: A software toolkit for the interconversion of standard data models for phenotypic data

Efficient sharing and integration of phenotypic data is crucial for advancing biomedical research and enhancing patient outcomes in precision medicine and public health. To achieve this, the health data community has developed standards to promote the harmonization of variable names and values. However, the use of diverse standards across different research centers can hinder progress. Here we present Convert-Pheno, an open-source software toolkit that enables the interconversion of common data models for phenotypic data such as Beacon v2 Models, CDISC-ODM, OMOP-CDM, Phenopackets v2, and REDCap. Along with the software, we have created a detailed documentation that includes information on deployment and installation.

Prediction of High-Risk Donors for Kidney Discard and Nonrecovery Using Structured Donor Characteristics and Unstructured Donor Narratives

Importance

Despite the unmet need, many deceased-donor kidneys are discarded or not recovered. Inefficient allocation and prolonged ischemia time are contributing factors, and early detection of high-risk donors may reduce organ loss.

Objective

To evaluate the feasibility of machine learning (ML) and natural language processing (NLP) classification of donors with kidneys that are used vs not used for organ transplant.

Design, Setting, and Participants

This retrospective cohort study used donor information (structured donor characteristics and unstructured donor narratives) from the United Network for Organ Sharing (UNOS). All donor offers to a single transplant center between January 2015 and December 2020 were used to train and validate ML models to predict donors who had at least 1 kidney transplanted (at our center or another center). The donor data from 2021 were used to test each model.

Exposures

Donor information was provided by UNOS to the transplant centers with potential transplant candidates. Each center evaluated the donor and decided within an allotted time whether to accept the kidney for organ transplant.

Main Outcomes and Measures

Outcome metrics of the test cohort included area under the receiver operating characteristic curve (AUROC), F1 score, accuracy, precision, and recall of each ML classifier. Feature importance and Shapley additive explanation (SHAP) summaries were assessed for model explainability.

Results

The training/validation cohort included 9555 donors (median [IQR] age, 50 [36-58] years; 5571 male [58.3%]), and the test cohort included 2481 donors (median [IQR] age, 52 [40-59] years; 1496 male [60.3%]). Only 20% to 30% of potential donors had at least 1 kidney transplanted. The ML model with a single variable (Kidney Donor Profile Index) showed an AUROC of 0.69, F1 score of 0.42, and accuracy of 0.64. Multivariable ML models based on basic a priori structured donor data showed similar metrics (logistic regression: AUROC = 0.70; F1 score = 0.42; accuracy = 0.62; random forest classifier: AUROC = 0.69; F1 score = 0.42; accuracy = 0.64). The classic NLP model (bag-of-words model) showed its best metrics (AUROC = 0.60; F1 score = 0.35; accuracy = 0.59) by the logistic regression classifier. The advanced Bidirectional Encoder Representations From Transformers model showed comparable metrics (AUROC = 0.62; F1 score = 0.39; accuracy = 0.69) only after appending basic donor information. Feature importance and SHAP detected the variables (and words) that affected the models most.

Conclusions and Relevance

Results of this cohort study suggest that models using ML can be applied to predict donors with high-risk kidneys not used for organ transplant, but the models still need further elaboration. The use of unstructured data is likely to expand the possibilities; further exploration of new approaches will be necessary to develop models with better predictive metrics.

Fusion of electronic health records and radiographic images for a multimodal deep learning prediction model of atypical femur fractures

Atypical femur fractures (AFF) represent a very rare type of fracture that can be difficult to discriminate radiologically from normal femur fractures (NFF). AFFs are associated with drugs that are administered to prevent osteoporosis-related fragility fractures, which are highly prevalent in the elderly population. Given that these fractures are rare and the radiologic changes are subtle currently only 7% of AFFs are correctly identified, which hinders adequate treatment for most patients with AFF. Deep learning models could be trained to classify automatically a fracture as AFF or NFF, thereby assisting radiologists in detecting these rare fractures. Historically, for this classification task, only imaging data have been used, using convolutional neural networks (CNN) or vision transformers applied to radiographs. However, to mimic situations in which all available data are used to arrive at a diagnosis, we adopted an approach of deep learning that is based on the integration of image data and tabular data (from electronic health records) for 159 patients with AFF and 914 patients with NFF. We hypothesized that the combinatorial data, compiled from all the radiology departments of 72 hospitals in Sweden and the Swedish National Patient Register, would improve classification accuracy, as compared to using only one modality. At the patient level, the area under the ROC curve (AUC) increased from 0.966 to 0.987 when using the integrated set of imaging data and seven pre-selected variables, as compared to only using imaging data. More importantly, the sensitivity increased from 0.796 to 0.903. We found a greater impact of data fusion when only a randomly selected subset of available images was used to make the image and tabular data more balanced for each patient. The AUC then increased from 0.949 to 0.984, and the sensitivity increased from 0.727 to 0.849. These AUC improvements are not large, mainly because of the already excellent performance of the CNN (AUC of 0.966) when only images are used. However, the improvement is clinically highly relevant considering the importance of accuracy in medical diagnostics. We expect an even greater effect when imaging data from a clinical workflow, comprising a more diverse set of diagnostic images, are used.

Artificial intelligence differentiates abdominal Henoch-Schönlein purpura from acute appendicitis in children

OBJECTIVE

This study aims to construct an artificial intelligence (AI) model capable of effectively discriminating between abdominal Henoch-Schönlein purpura (AHSP) and acute appendicitis (AA) in pediatric patients.

METHODS

A total of 6965 participants, comprising 2201 individuals with AHSP and 4764 patients with AA, were enrolled in the study. Additionally, 53 laboratory indicators were taken into consideration. Five distinct artificial intelligence (AI) models were developed employing machine learning algorithms, namely XGBoost, AdaBoost, Gaussian Naïve Bayes (GNB), MLPClassifier (MLP), and support vector machine (SVM). The performance of these prediction models was assessed through receiver operating characteristic (ROC) curve analysis, calibration curve assessment, and decision curve analysis (DCA).

RESULTS

We identified 32 discriminative indicators (p < .05) between AHSP and AA. Five indicators, namely the lymphocyte ratio (LYMPH ratio), eosinophil ratio (EO ratio), eosinophil count (EO count), neutrophil ratio (NEUT ratio), and C-reactive protein (CRP), exhibited strong performance in distinguishing AHSP from AA (AUC ≥ 0.80). Among the various prediction models, the XGBoost model displayed superior performance evidenced by the highest AUC (XGBoost = 0.895, other models < 0.89), accuracy (XGBoost = 0.824, other models < 0.81), and Kappa value (XGBoost = 0.621, other models < 0.60) in the validation set. After optimization, the XGBoost model demonstrated remarkable diagnostic performance for AHSP and AA (AUC > 0.95). Both the calibration curve and decision curve analysis suggested the promising clinical utility and net benefits of the XGBoost model.

CONCLUSION

The AI-based machine learning model exhibits high prediction accuracy and can differentiate AHSP and AA from a data-driven perspective.

The Emergence of AI-Based Wearable Sensors for Digital Health Technology: A Review

Disease diagnosis and monitoring using conventional healthcare services is typically expensive and has limited accuracy. Wearable health technology based on flexible electronics has gained tremendous attention in recent years for monitoring patient health owing to attractive features, such as lower medical costs, quick access to patient health data, ability to operate and transmit data in harsh environments, storage at room temperature, non-invasive implementation, mass scaling, etc. This technology provides an opportunity for disease pre-diagnosis and immediate therapy. Wearable sensors have opened a new area of personalized health monitoring by accurately measuring physical states and biochemical signals. Despite the progress to date in the development of wearable sensors, there are still several limitations in the accuracy of the data collected, precise disease diagnosis, and early treatment. This necessitates advances in applied materials and structures and using artificial intelligence (AI)-enabled wearable sensors to extract target signals for accurate clinical decision-making and efficient medical care. In this paper, we review two significant aspects of smart wearable sensors. First, we offer an overview of the most recent progress in improving wearable sensor performance for physical, chemical, and biosensors, focusing on materials, structural configurations, and transduction mechanisms. Next, we review the use of AI technology in combination with wearable technology for big data processing, self-learning, power-efficiency, real-time data acquisition and processing, and personalized health for an intelligent sensing platform. Finally, we present the challenges and future opportunities associated with smart wearable sensors.

Overcoming the Challenges in the Development and Implementation of Artificial Intelligence in Radiology: A Comprehensive Review of Solutions Beyond Supervised Learning

Artificial intelligence (AI) in radiology is a rapidly developing field with several prospective clinical studies demonstrating its benefits in clinical practice. In 2022, the Korean Society of Radiology held a forum to discuss the challenges and drawbacks in AI development and implementation. Various barriers hinder the successful application and widespread adoption of AI in radiology, such as limited annotated data, data privacy and security, data heterogeneity, imbalanced data, model interpretability, overfitting, and integration with clinical workflows. In this review, some of the various possible solutions to these challenges are presented and discussed; these include training with longitudinal and multimodal datasets, dense training with multitask learning and multimodal learning, self-supervised contrastive learning, various image modifications and syntheses using generative models, explainable AI, causal learning, federated learning with large data models, and digital twins.

A scoping review of artificial intelligence-based methods for diabetes risk prediction

The increasing prevalence of type 2 diabetes mellitus (T2DM) and its associated health complications highlight the need to develop predictive models for early diagnosis and intervention. While many artificial intelligence (AI) models for T2DM risk prediction have emerged, a comprehensive review of their advancements and challenges is currently lacking. This scoping review maps out the existing literature on AI-based models for T2DM prediction, adhering to the PRISMA extension for Scoping Reviews guidelines. A systematic search of longitudinal studies was conducted across four databases, including PubMed, Scopus, IEEE-Xplore, and Google Scholar. Forty studies that met our inclusion criteria were reviewed. Classical machine learning (ML) models dominated these studies, with electronic health records (EHR) being the predominant data modality, followed by multi-omics, while medical imaging was the least utilized. Most studies employed unimodal AI models, with only ten adopting multimodal approaches. Both unimodal and multimodal models showed promising results, with the latter being superior. Almost all studies performed internal validation, but only five conducted external validation. Most studies utilized the area under the curve (AUC) for discrimination measures. Notably, only five studies provided insights into the calibration of their models. Half of the studies used interpretability methods to identify key risk predictors revealed by their models. Although a minority highlighted novel risk predictors, the majority reported commonly known ones. Our review provides valuable insights into the current state and limitations of AI-based models for T2DM prediction and highlights the challenges associated with their development and clinical integration.

A scoping review on multimodal deep learning in biomedical images and texts

OBJECTIVE

Computer-assisted diagnostic and prognostic systems of the future should be capable of simultaneously processing multimodal data. Multimodal deep learning (MDL), which involves the integration of multiple sources of data, such as images and text, has the potential to revolutionize the analysis and interpretation of biomedical data. However, it only caught researchers' attention recently. To this end, there is a critical need to conduct a systematic review on this topic, identify the limitations of current work, and explore future directions.

METHODS

In this scoping review, we aim to provide a comprehensive overview of the current state of the field and identify key concepts, types of studies, and research gaps with a focus on biomedical images and texts joint learning, mainly because these two were the most commonly available data types in MDL research.

RESULT

This study reviewed the current uses of multimodal deep learning on five tasks: (1) Report generation, (2) Visual question answering, (3) Cross-modal retrieval, (4) Computer-aided diagnosis, and (5) Semantic segmentation.

CONCLUSION

Our results highlight the diverse applications and potential of MDL and suggest directions for future research in the field. We hope our review will facilitate the collaboration of natural language processing (NLP) and medical imaging communities and support the next generation of decision-making and computer-assisted diagnostic system development.

Advancing Diabetic Retinopathy Diagnosis: Leveraging Optical Coherence Tomography Imaging with Convolutional Neural Networks

Diabetic retinopathy (DR) is a vision-threatening complication of diabetes, necessitating early and accurate diagnosis. The combination of optical coherence tomography (OCT) imaging with convolutional neural networks (CNNs) has emerged as a promising approach for enhancing DR diagnosis. OCT provides detailed retinal morphology information, while CNNs analyze OCT images for automated detection and classification of DR. This paper reviews the current research on OCT imaging and CNNs for DR diagnosis, discussing their technical aspects and suitability. It explores CNN applications in detecting lesions, segmenting microaneurysms, and assessing disease severity, showing high sensitivity and accuracy. CNN models outperform traditional methods and rival expert ophthalmologists' results. However, challenges such as dataset availability and model interpretability remain. Future directions include multimodal imaging integration and real-time, point-of-care CNN systems for DR screening. The integration of OCT imaging with CNNs has transformative potential in DR diagnosis, facilitating early intervention, personalized treatments, and improved patient outcomes. Abbreviations: DR = Diabetic Retinopathy, OCT = Optical Coherence Tomography, CNN = Convolutional Neural Network, CMV = Cytomegalovirus, PDR = Proliferative Diabetic Retinopathy, AMD = Age-Related Macular Degeneration, VEGF = vascular endothelial growth factor, RAP = Retinal Angiomatous Proliferation, OCTA = OCT Angiography, AI = Artificial Intelligence.

Artificial intelligence in psychiatry research, diagnosis, and therapy

Psychiatric disorders are now responsible for the largest proportion of the global burden of disease, and even more challenges have been seen during the COVID-19 pandemic. Artificial intelligence (AI) is commonly used to facilitate the early detection of disease, understand disease progression, and discover new treatments in the fields of both physical and mental health. The present review provides a broad overview of AI methodology and its applications in data acquisition and processing, feature extraction and characterization, psychiatric disorder classification, potential biomarker detection, real-time monitoring, and interventions in psychiatric disorders. We also comprehensively summarize AI applications with regard to the early warning, diagnosis, prognosis, and treatment of specific psychiatric disorders, including depression, schizophrenia, autism spectrum disorder, attention-deficit/hyperactivity disorder, addiction, sleep disorders, and Alzheimer's disease. The advantages and disadvantages of AI in psychiatry are clarified. We foresee a new wave of research opportunities to facilitate and improve AI technology and its long-term implications in psychiatry during and after the COVID-19 era.

AI and machine learning in resuscitation: Ongoing research, new concepts, and key challenges

Aim

Artificial intelligence (AI) and machine learning (ML) are important areas of computer science that have recently attracted attention for their application to medicine. However, as techniques continue to advance and become more complex, it is increasingly challenging for clinicians to stay abreast of the latest research. This overview aims to translate research concepts and potential concerns to healthcare professionals interested in applying AI and ML to resuscitation research but who are not experts in the field.

Main text

We present various research including prediction models using structured and unstructured data, exploring treatment heterogeneity, reinforcement learning, language processing, and large-scale language models. These studies potentially offer valuable insights for optimizing treatment strategies and clinical workflows. However, implementing AI and ML in clinical settings presents its own set of challenges. The availability of high-quality and reliable data is crucial for developing accurate ML models. A rigorous validation process and the integration of ML into clinical practice is essential for practical implementation. We furthermore highlight the potential risks associated with self-fulfilling prophecies and feedback loops, emphasizing the importance of transparency, interpretability, and trustworthiness in AI and ML models. These issues need to be addressed in order to establish reliable and trustworthy AI and ML models.

Conclusion

In this article, we overview concepts and examples of AI and ML research in the resuscitation field. Moving forward, appropriate understanding of ML and collaboration with relevant experts will be essential for researchers and clinicians to overcome the challenges and harness the full potential of AI and ML in resuscitation.