Integrated multimodal artificial intelligence framework for healthcare applications

Artificial Multimodal Neuron with Associative Learning Capabilities: Acquisition, Extinction, and Spontaneous Recovery

Associative multimodal artificial intelligence (AMAI) has gained significant attention across various fields, yet its implementation poses challenges due to the burden on computing and memory resources. To address these challenges, researchers have paid increasing attention to neuromorphic devices based on novel materials and structures, which can implement classical conditioning behaviors with simplified circuitry. Herein, we introduce an artificial multimodal neuron device that shows not only the acquisition behavior but also the extinction and the spontaneous recovery behaviors for the first time. Being composed of an ovonic threshold switch (OTS)-based neuron device, a conductive bridge memristor (CBM)-based synapse device, and a few passive electrical elements, such observed behaviors of this neuron device are explained in terms of the electroforming and the diffusion of metallic ions in the CBM. We believe that the proposed associative learning neuron device will shed light on the way of developing large-scale AMAI systems by providing inspiration to devise an associative learning network with improved energy efficiency.

AI-powered microscopy image analysis for parasitology: integrating human expertise

Microscopy image analysis plays a pivotal role in parasitology research. Deep learning (DL), a subset of artificial intelligence (AI), has garnered significant attention. However, traditional DL-based methods for general purposes are data-driven, often lacking explainability due to their black-box nature and sparse instructional resources. To address these challenges, this article presents a comprehensive review of recent advancements in knowledge-integrated DL models tailored for microscopy image analysis in parasitology. The massive amounts of human expert knowledge from parasitologists can enhance the accuracy and explainability of AI-driven decisions. It is expected that the adoption of knowledge-integrated DL models will open up a wide range of applications in the field of parasitology.

Artificial Intelligence in Cardiovascular Disease Prevention: Is it Ready for Prime Time?

PURPOSE OF REVIEW

This review evaluates how Artificial Intelligence (AI) enhances atherosclerotic cardiovascular disease (ASCVD) risk assessment, allows for opportunistic screening, and improves adherence to guidelines through the analysis of unstructured clinical data and patient-generated data. Additionally, it discusses strategies for integrating AI into clinical practice in preventive cardiology.

RECENT FINDINGS

AI models have shown superior performance in personalized ASCVD risk evaluations compared to traditional risk scores. These models now support automated detection of ASCVD risk markers, including coronary artery calcium (CAC), across various imaging modalities such as dedicated ECG-gated CT scans, chest X-rays, mammograms, coronary angiography, and non-gated chest CT scans. Moreover, large language model (LLM) pipelines are effective in identifying and addressing gaps and disparities in ASCVD preventive care, and can also enhance patient education. AI applications are proving invaluable in preventing and managing ASCVD and are primed for clinical use, provided they are implemented within well-regulated, iterative clinical pathways.

Evaluation of machine learning approach for surgical results of Ahmed valve implantation in patients with glaucoma

BACKGROUND

Ahmed valve implantation demonstrated an increasing proportion in glaucoma surgery, but predicting the successful maintenance of target intraocular pressure remains a challenging task. This study aimed to evaluate the performance of machine learning (ML) in predicting surgical outcomes after Ahmed valve implantation and to assess potential risk factors associated with surgical failure to contribute to improving the success rate.

METHODS

This study used preoperative data of patients who underwent Ahmed valve implantation from 2017 to 2021 at Ajou University Hospital. These datasets included demographic and ophthalmic parameters (dataset A), systemic medical records excluding psychiatric records (dataset B), and psychiatric medications (dataset C). Logistic regression, extreme gradient boosting (XGBoost), and support vector machines were first evaluated using only dataset A. The algorithm with the best performance was selected based on the area under the receiver operating characteristics curve (AUROC). Finally, three additional prediction models were developed using the best performance algorithm, incorporating combinations of multiple datasets to predict surgical outcomes at 1 year.

RESULTS

Among 153 eyes of 133 patients, 131 (85.6%) and 22 (14.4%) eyes were categorized as the success and failure groups, respectively. The XGBoost was shown as the best-performance model with an AUROC value of 0.684, using only dataset A. The final three further prediction models were developed based on the combination of multiple datasets using the XGBoost model. All datasets combinations demonstrated the best performances in terms of AUROC (dataset A + B: 0.782; A + C: 0.773; A + B + C: 0.801). Furthermore, advancing age was a risk factor associated with a higher surgical failure incidence.

CONCLUSIONS

ML provides some predictive value in predicting the outcomes of Ahmed valve implantation at 1 year. ML evaluation revealed advancing age as a common risk factor for surgical failure.

Prognostic stratification of glioblastoma patients by unsupervised clustering of morphology patterns on whole slide images furthering our disease understanding

Introduction

Glioblastoma (GBM) is a highly aggressive malignant tumor of the central nervous system that displays varying molecular and morphological profiles, leading to challenging prognostic assessments. Stratifying GBM patients according to overall survival (OS) from H&E-stained whole slide images (WSI) using advanced computational methods is challenging, but with direct clinical implications.

Methods

This work is focusing on GBM (IDH-wildtype, CNS WHO Gr.4) cases, identified from the TCGA-GBM and TCGA-LGG collections after considering the 2021 WHO classification criteria. The proposed approach starts with patch extraction in each WSI, followed by comprehensive patch-level curation to discard artifactual content, i.e., glass reflections, pen markings, dust on the slide, and tissue tearing. Each patch is then computationally described as a feature vector defined by a pre-trained VGG16 convolutional neural network. Principal component analysis provides a feature representation of reduced dimensionality, further facilitating identification of distinct groups of morphology patterns, via unsupervised k-means clustering.

Results

The optimal number of clusters, according to cluster reproducibility and separability, is automatically determined based on the rand index and silhouette coefficient, respectively. Our proposed approach achieved prognostic stratification accuracy of 83.33% on a multi-institutional independent unseen hold-out test set with sensitivity and specificity of 83.33%.

Discussion

We hypothesize that the quantification of these clusters of morphology patterns, reflect the tumor's spatial heterogeneity and yield prognostic relevant information to distinguish between short and long survivors using a decision tree classifier. The interpretability analysis of the obtained results can contribute to furthering and quantifying our understanding of GBM and potentially improving our diagnostic and prognostic predictions.

Exploring trends and autonomy levels of adaptive business intelligence in healthcare: A systematic review

OBJECTIVE

In order to comprehensively understand the characteristics of Adaptive Business Intelligence (ABI) in Healthcare, this study is structured to provide insights into the common features and evolving patterns within this domain. Applying the Sheridan's Classification as a framework, we aim to assess the degree of autonomy exhibited by various ABI components. Together, these objectives will contribute to a deeper understanding of ABI implementation and its implications within the Healthcare context.

METHODS

A comprehensive search of academic databases was conducted to identify relevant studies, selecting AIS e-library (AISel), Decision Support Systems Journal (DSSJ), Nature, The Lancet Digital Health (TLDH), PubMed, Expert Systems with Application (ESWA) and npj Digital Medicine as information sources. Studies from 2006 to 2022 were included based on predefined eligibility criteria. PRISMA statements were used to report this study.

RESULTS

The outcomes showed that ABI systems present distinct levels of development, autonomy and practical deployment. The high levels of autonomy were essentially associated with predictive components. However, the possibility of completely autonomous decisions by these systems is totally excluded. Lower levels of autonomy are also observed, particularly in connection with prescriptive components, granting users responsibility in the generation of decisions.

CONCLUSION

The study presented emphasizes the vital connection between desired outcomes and the inherent autonomy of these solutions, highlighting the critical need for additional research on the consequences of ABI systems and their constituent elements. Organizations should deploy these systems in a way consistent with their objectives and values, while also being mindful of potential adverse effects. Providing valuable insights for researchers, practitioners, and policymakers aiming to comprehend the diverse levels of ABI systems implementation, it contributes to well-informed decision-making in this dynamic field.

Digitalomics, digital intervention, and designing future: The next frontier in cardiology

The discipline of cardiology stands at a transformative juncture, primarily influenced by the surge in digital health technologies. These innovations hold the promise to redefine the realms of cardiovascular research and patient care, ushering in an era of individualized and data-driven treatments. This review delves into the heart of this evolution, introducing a comprehensive design for the future of cardiology. Emphasizing the emerging domains of "digitalomics" and "digital intervention", it explores how the integration of patient data, artificial intelligence-enabled diagnostics, and telehealth can lead to more streamlined and personalized cardiovascular health. The "digital-twin" model, a highlight of this approach, encapsulates individual patient characteristics, allowing for targeted treatments. The role of interdisciplinary collaboration in cardiovascular medicine is also underlined, emphasizing the importance of merging traditional cardiology with technological advancements. The convergence of traditional cardiology methods and digital health technologies, facilitated by a transdisciplinary approach, is set to chart a new course in cardiovascular health, emphasizing individualized care and improved clinical outcomes.

Enhancing NSCLC recurrence prediction with PET/CT habitat imaging, ctDNA, and integrative radiogenomics-blood insights

While we recognize the prognostic importance of clinicopathological measures and circulating tumor DNA (ctDNA), the independent contribution of quantitative image markers to prognosis in non-small cell lung cancer (NSCLC) remains underexplored. In our multi-institutional study of 394 NSCLC patients, we utilize pre-treatment computed tomography (CT) and 18F-fluorodeoxyglucose positron emission tomography (FDG-PET) to establish a habitat imaging framework for assessing regional heterogeneity within individual tumors. This framework identifies three PET/CT subtypes, which maintain prognostic value after adjusting for clinicopathologic risk factors including tumor volume. Additionally, these subtypes complement ctDNA in predicting disease recurrence. Radiogenomics analysis unveil the molecular underpinnings of these imaging subtypes, highlighting downregulation in interferon alpha and gamma pathways in the high-risk subtype. In summary, our study demonstrates that these habitat imaging subtypes effectively stratify NSCLC patients based on their risk levels for disease recurrence after initial curative surgery or radiotherapy, providing valuable insights for personalized treatment approaches.

Transforming the cardiometabolic disease landscape: Multimodal AI-powered approaches in prevention and management

The rise of artificial intelligence (AI) has revolutionized various scientific fields, particularly in medicine, where it has enabled the modeling of complex relationships from massive datasets. Initially, AI algorithms focused on improved interpretation of diagnostic studies such as chest X-rays and electrocardiograms in addition to predicting patient outcomes and future disease onset. However, AI has evolved with the introduction of transformer models, allowing analysis of the diverse, multimodal data sources existing in medicine today. Multimodal AI holds great promise in more accurate disease risk assessment and stratification as well as optimizing the key driving factors in cardiometabolic disease: blood pressure, sleep, stress, glucose control, weight, nutrition, and physical activity. In this article we outline the current state of medical AI in cardiometabolic disease, highlighting the potential of multimodal AI to augment personalized prevention and treatment strategies in cardiometabolic disease.

Development of Medical Imaging Data Standardization for Imaging-Based Observational Research: OMOP Common Data Model Extension

The rapid growth of artificial intelligence (AI) and deep learning techniques require access to large inter-institutional cohorts of data to enable the development of robust models, e.g., targeting the identification of disease biomarkers and quantifying disease progression and treatment efficacy. The Observational Medical Outcomes Partnership Common Data Model (OMOP CDM) has been designed to accommodate a harmonized representation of observational healthcare data. This study proposes the Medical Imaging CDM (MI-CDM) extension, adding two new tables and two vocabularies to the OMOP CDM to address the structural and semantic requirements to support imaging research. The tables provide the capabilities of linking DICOM data sources as well as tracking the provenance of imaging features derived from those images. The implementation of the extension enables phenotype definitions using imaging features and expanding standardized computable imaging biomarkers. This proposal offers a comprehensive and unified approach for conducting imaging research and outcome studies utilizing imaging features.

Utilizing multimodal AI to improve genetic analyses of cardiovascular traits

Electronic health records, biobanks, and wearable biosensors contain multiple high-dimensional clinical data (HDCD) modalities (e.g., ECG, Photoplethysmography (PPG), and MRI) for each individual. Access to multimodal HDCD provides a unique opportunity for genetic studies of complex traits because different modalities relevant to a single physiological system (e.g., circulatory system) encode complementary and overlapping information. We propose a novel multimodal deep learning method, M-REGLE, for discovering genetic associations from a joint representation of multiple complementary HDCD modalities. We showcase the effectiveness of this model by applying it to several cardiovascular modalities. M-REGLE jointly learns a lower representation (i.e., latent factors) of multimodal HDCD using a convolutional variational autoencoder, performs genome wide association studies (GWAS) on each latent factor, then combines the results to study the genetics of the underlying system. To validate the advantages of M-REGLE and multimodal learning, we apply it to common cardiovascular modalities (PPG and ECG), and compare its results to unimodal learning methods in which representations are learned from each data modality separately, but the downstream genetic analyses are performed on the combined unimodal representations. M-REGLE identifies 19.3% more loci on the 12-lead ECG dataset, 13.0% more loci on the ECG lead I + PPG dataset, and its genetic risk score significantly outperforms the unimodal risk score at predicting cardiac phenotypes, such as atrial fibrillation (Afib), in multiple biobanks.

Multimodal risk prediction with physiological signals, medical images and clinical notes

The broad adoption of electronic health record (EHR) systems brings us a tremendous amount of clinical data and thus provides opportunities to conduct data-based healthcare research to solve various clinical problems in the medical domain. Machine learning and deep learning methods are widely used in the medical informatics and healthcare domain due to their power to mine insights from raw data. When adapting deep learning models for EHR data, it is essential to consider its heterogeneous nature: EHR contains patient records from various sources including medical tests (e.g. blood test, microbiology test), medical imaging, diagnosis, medications, procedures, clinical notes, etc. Those modalities together provide a holistic view of patient health status and complement each other. Therefore, combining data from multiple modalities that are intrinsically different is challenging but intuitively promising in deep learning for EHR. To assess the expectations of multimodal data, we introduce a comprehensive fusion framework designed to integrate temporal variables, medical images, and clinical notes in EHR for enhanced performance in clinical risk prediction. Early, joint, and late fusion strategies are employed to combine data from various modalities effectively. We test the model with three predictive tasks: in-hospital mortality, long length of stay, and 30-day readmission. Experimental results show that multimodal models outperform uni-modal models in the tasks involved. Additionally, by training models with different input modality combinations, we calculate the Shapley value for each modality to quantify their contribution to multimodal performance. It is shown that temporal variables tend to be more helpful than CXR images and clinical notes in the three explored predictive tasks.

Development and Validation of Multimodal Models to Predict the 30-Day Mortality of ICU Patients Based on Clinical Parameters and Chest X-Rays

We aimed to develop and validate multimodal ICU patient prognosis models that combine clinical parameters data and chest X-ray (CXR) images. A total of 3798 subjects with clinical parameters and CXR images were extracted from the Medical Information Mart for Intensive Care IV (MIMIC-IV) database and an external hospital (the test set). The primary outcome was 30-day mortality after ICU admission. Automated machine learning (AutoML) and convolutional neural networks (CNNs) were used to construct single-modal models based on clinical parameters and CXR separately. An early fusion approach was used to integrate both modalities (clinical parameters and CXR) into a multimodal model named PrismICU. Compared to the single-modal models, i.e., the clinical parameter model (AUC = 0.80, F1-score = 0.43) and the CXR model (AUC = 0.76, F1-score = 0.45) and the scoring system APACHE II (AUC = 0.83, F1-score = 0.77), PrismICU (AUC = 0.95, F1 score = 0.95) showed improved performance in predicting the 30-day mortality in the validation set. In the test set, PrismICU (AUC = 0.82, F1-score = 0.61) was also better than the clinical parameters model (AUC = 0.72, F1-score = 0.50), CXR model (AUC = 0.71, F1-score = 0.36), and APACHE II (AUC = 0.62, F1-score = 0.50). PrismICU, which integrated clinical parameters data and CXR images, performed better than single-modal models and the existing scoring system. It supports the potential of multimodal models based on structured data and imaging in clinical management.

Multi-modal learning for inpatient length of stay prediction

Predicting inpatient length of stay (LoS) is important for hospitals aiming to improve service efficiency and enhance management capabilities. Patient medical records are strongly associated with LoS. However, due to diverse modalities, heterogeneity, and complexity of data, it becomes challenging to effectively leverage these heterogeneous data to put forth a predictive model that can accurately predict LoS. To address the challenge, this study aims to establish a novel data-fusion model, termed as DF-Mdl, to integrate heterogeneous clinical data for predicting the LoS of inpatients between hospital discharge and admission. Multi-modal data such as demographic data, clinical notes, laboratory test results, and medical images are utilized in our proposed methodology with individual "basic" sub-models separately applied to each different data modality. Specifically, a convolutional neural network (CNN) model, which we termed CRXMDL, is designed for chest X-ray (CXR) image data, two long short-term memory networks are used to extract features from long text data, and a novel attention-embedded 1D convolutional neural network is developed to extract useful information from numerical data. Finally, these basic models are integrated to form a new data-fusion model (DF-Mdl) for inpatient LoS prediction. The proposed method attains the best R2 and EVAR values of 0.6039 and 0.6042 among competitors for the LoS prediction on the Medical Information Mart for Intensive Care (MIMIC)-IV test dataset. Empirical evidence suggests better performance compared with other state-of-the-art (SOTA) methods, which demonstrates the effectiveness and feasibility of the proposed approach.

Construction and optimization of multi-platform precision pathways for precision medicine

In the enduring challenge against disease, advancements in medical technology have empowered clinicians with novel diagnostic platforms. Whilst in some cases, a single test may provide a confident diagnosis, often additional tests are required. However, to strike a balance between diagnostic accuracy and cost-effectiveness, one must rigorously construct the clinical pathways. Here, we developed a framework to build multi-platform precision pathways in an automated, unbiased way, recommending the key steps a clinician would take to reach a diagnosis. We achieve this by developing a confidence score, used to simulate a clinical scenario, where at each stage, either a confident diagnosis is made, or another test is performed. Our framework provides a range of tools to interpret, visualize and compare the pathways, improving communication and enabling their evaluation on accuracy and cost, specific to different contexts. This framework will guide the development of novel diagnostic pathways for different diseases, accelerating the implementation of precision medicine into clinical practice.

Building digital patient pathways for the management and treatment of multiple sclerosis

Recent advances in the field of artificial intelligence (AI) could yield new insights into the potential causes of multiple sclerosis (MS) and factors influencing its course as the use of AI opens new possibilities regarding the interpretation and use of big data from not only a cross-sectional, but also a longitudinal perspective. For each patient with MS, there is a vast amount of multimodal data being accumulated over time. But for the application of AI and related technologies, these data need to be available in a machine-readable format and need to be collected in a standardized and structured manner. Through the use of mobile electronic devices and the internet it has also become possible to provide healthcare services from remote and collect information on a patient's state of health outside of regular check-ups on site. Against this background, we argue that the concept of pathways in healthcare now could be applied to structure the collection of information across multiple devices and stakeholders in the virtual sphere, enabling us to exploit the full potential of AI technology by e.g., building digital twins. By going digital and using pathways, we can virtually link patients and their caregivers. Stakeholders then could rely on digital pathways for evidence-based guidance in the sequence of procedures and selection of therapy options based on advanced analytics supported by AI as well as for communication and education purposes. As far as we aware of, however, pathway modelling with respect to MS management and treatment has not been thoroughly investigated yet and still needs to be discussed. In this paper, we thus present our ideas for a modular-integrative framework for the development of digital patient pathways for MS treatment.

Learning across diverse biomedical data modalities and cohorts: Challenges and opportunities for innovation

In healthcare, machine learning (ML) shows significant potential to augment patient care, improve population health, and streamline healthcare workflows. Realizing its full potential is, however, often hampered by concerns about data privacy, diversity in data sources, and suboptimal utilization of different data modalities. This review studies the utility of cross-cohort cross-category (C4) integration in such contexts: the process of combining information from diverse datasets distributed across distinct, secure sites. We argue that C4 approaches could pave the way for ML models that are both holistic and widely applicable. This paper provides a comprehensive overview of C4 in health care, including its present stage, potential opportunities, and associated challenges.

Integrating Multi-Omics Data With EHR for Precision Medicine Using Advanced Artificial Intelligence

With the recent advancement of novel biomedical technologies such as high-throughput sequencing and wearable devices, multi-modal biomedical data ranging from multi-omics molecular data to real-time continuous bio-signals are generated at an unprecedented speed and scale every day. For the first time, these multi-modal biomedical data are able to make precision medicine close to a reality. However, due to data volume and the complexity, making good use of these multi-modal biomedical data requires major effort. Researchers and clinicians are actively developing artificial intelligence (AI) approaches for data-driven knowledge discovery and causal inference using a variety of biomedical data modalities. These AI-based approaches have demonstrated promising results in various biomedical and healthcare applications. In this review paper, we summarize the state-of-the-art AI models for integrating multi-omics data and electronic health records (EHRs) for precision medicine. We discuss the challenges and opportunities in integrating multi-omics data with EHRs and future directions. We hope this review can inspire future research and developing in integrating multi-omics data with EHRs for precision medicine.

Applications of AI in multi-modal imaging for cardiovascular disease

Data for healthcare is diverse and includes many different modalities. Traditional approaches to Artificial Intelligence for cardiovascular disease were typically limited to single modalities. With the proliferation of diverse datasets and new methods in AI, we are now able to integrate different modalities, such as magnetic resonance scans, computerized tomography scans, echocardiography, x-rays, and electronic health records. In this paper, we review research from the last 5 years in applications of AI to multi-modal imaging. There have been many promising results in registration, segmentation, and fusion of different magnetic resonance imaging modalities with each other and computer tomography scans, but there are still many challenges that need to be addressed. Only a few papers have addressed modalities such as x-ray, echocardiography, or non-imaging modalities. As for prediction or classification tasks, there have only been a couple of papers that use multiple modalities in the cardiovascular domain. Furthermore, no models have been implemented or tested in real world cardiovascular clinical settings.

A perspective on computer vision in biosensing

Computer vision has become a powerful tool in the field of biosensing, aiding in the development of innovative and precise systems for the analysis and interpretation of biological data. This interdisciplinary approach harnesses the capabilities of computer vision algorithms and techniques to extract valuable information from various biosensing applications, including medical diagnostics, environmental monitoring, and food health. Despite years of development, there is still significant room for improvement in this area. In this perspective, we outline how computer vision is applied to raw sensor data in biosensors and its advantages to biosensing applications. We then discuss ongoing research and developments in the field and subsequently explore the challenges and opportunities that computer vision faces in biosensor applications. We also suggest directions for future work, ultimately underscoring the significant impact of computer vision on advancing biosensing technologies and their applications.

Fusion of electronic health records and radiographic images for a multimodal deep learning prediction model of atypical femur fractures

Atypical femur fractures (AFF) represent a very rare type of fracture that can be difficult to discriminate radiologically from normal femur fractures (NFF). AFFs are associated with drugs that are administered to prevent osteoporosis-related fragility fractures, which are highly prevalent in the elderly population. Given that these fractures are rare and the radiologic changes are subtle currently only 7% of AFFs are correctly identified, which hinders adequate treatment for most patients with AFF. Deep learning models could be trained to classify automatically a fracture as AFF or NFF, thereby assisting radiologists in detecting these rare fractures. Historically, for this classification task, only imaging data have been used, using convolutional neural networks (CNN) or vision transformers applied to radiographs. However, to mimic situations in which all available data are used to arrive at a diagnosis, we adopted an approach of deep learning that is based on the integration of image data and tabular data (from electronic health records) for 159 patients with AFF and 914 patients with NFF. We hypothesized that the combinatorial data, compiled from all the radiology departments of 72 hospitals in Sweden and the Swedish National Patient Register, would improve classification accuracy, as compared to using only one modality. At the patient level, the area under the ROC curve (AUC) increased from 0.966 to 0.987 when using the integrated set of imaging data and seven pre-selected variables, as compared to only using imaging data. More importantly, the sensitivity increased from 0.796 to 0.903. We found a greater impact of data fusion when only a randomly selected subset of available images was used to make the image and tabular data more balanced for each patient. The AUC then increased from 0.949 to 0.984, and the sensitivity increased from 0.727 to 0.849. These AUC improvements are not large, mainly because of the already excellent performance of the CNN (AUC of 0.966) when only images are used. However, the improvement is clinically highly relevant considering the importance of accuracy in medical diagnostics. We expect an even greater effect when imaging data from a clinical workflow, comprising a more diverse set of diagnostic images, are used.

Predicting Postoperative Hospital Stays Using Nursing Narratives and the Reverse Time Attention (RETAIN) Model: Retrospective Cohort Study

Background

Nursing narratives are an intriguing feature in the prediction of short-term clinical outcomes. However, it is unclear which nursing narratives significantly impact the prediction of postoperative length of stay (LOS) in deep learning models.

Objective

Therefore, we applied the Reverse Time Attention (RETAIN) model to predict LOS, entering nursing narratives as the main input.

Methods

A total of 354 patients who underwent ovarian cancer surgery at the Seoul National University Bundang Hospital from 2014 to 2020 were retrospectively enrolled. Nursing narratives collected within 3 postoperative days were used to predict prolonged LOS (≥10 days). The physician's assessment was conducted based on a retrospective review of the physician's note within the same period of the data model used.

Results

The model performed better than the physician's assessment (area under the receiver operating curve of 0.81 vs 0.58; P=.02). Nursing narratives entered on the first day were the most influential predictors in prolonged LOS. The likelihood of prolonged LOS increased if the physician had to check the patient often and if the patient received intravenous fluids or intravenous patient-controlled analgesia late.

Conclusions

The use of the RETAIN model on nursing narratives predicted postoperative LOS effectively for patients who underwent ovarian cancer surgery. These findings suggest that accurate and interpretable deep learning information obtained shortly after surgery may accurately predict prolonged LOS.

Enhancing Prostate Cancer Diagnosis with a Novel Artificial Intelligence-Based Web Application: Synergizing Deep Learning Models, Multimodal Data, and Insights from Usability Study with Pathologists

Prostate cancer remains a significant cause of male cancer mortality in the United States, with an estimated 288,300 new cases in 2023. Accurate grading of prostate cancer is crucial for ascertaining disease severity and shaping treatment strategies. Modern deep learning techniques show promise in grading biopsies, but there is a gap in integrating these advances into clinical practice. Our web platform tackles this challenge by integrating human expertise with AI-driven grading, incorporating diverse data sources. We gathered feedback from four pathologists and one medical practitioner to assess usability and real-world alignment through a survey and the NASA TLX Usability Test. Notably, 60% of users found it easy to navigate, rating it 5.5 out of 7 for ease of understanding. Users appreciated self-explanatory information in popup tabs. For ease of use, all users favored the detailed summary tab, rating it 6.5 out of 7. While 80% felt patient demographics beyond age were unnecessary, high-resolution biopsy images were deemed vital. Acceptability was high, with all users willing to adopt the app, and some believed it could reduce workload. The NASA TLX Usability Test indicated a low-moderate perceived workload, suggesting room for improved explanations and data visualization.

Diagnosing psychiatric disorders from history of present illness using a large-scale linguistic model

AIM

Recent advances in natural language processing models are expected to provide diagnostic assistance in psychiatry from the history of present illness (HPI). However, existing studies have been limited, with the target diseases including only major diseases, small sample sizes, or no comparison with diagnoses made by psychiatrists to ensure accuracy. Therefore, we formulated an accurate diagnostic model that covers all psychiatric disorders.

METHODS

HPIs and diagnoses were extracted from discharge summaries of 2,642 cases at the Nara Medical University Hospital, Japan, from 21 May 2007, to 31 May 31 2021. The diagnoses were classified into 11 classes according to the code from ICD-10 Chapter V. Using UTH-BERT pre-trained on the electronic medical records of the University of Tokyo Hospital, Japan, we predicted the main diagnoses at discharge based on HPIs and compared the concordance rate with the results of psychiatrists. The psychiatrists were divided into two groups: semi-Designated with 3-4 years of experience and Residents with only 2 months of experience.

RESULTS

The model's match rate was 74.3%, compared to 71.5% for the semi-Designated psychiatrists and 69.4% for the Residents. If the cases were limited to those correctly answered by the semi-Designated group, the model and the Residents performed at 84.9% and 83.3%, respectively.

CONCLUSION

We demonstrated that the model matched the diagnosis predicted from the HPI with a high probability to the principal diagnosis at discharge. Hence, the model can provide diagnostic suggestions in actual clinical practice.

Automated Segmentation of Sacral Chordoma and Surrounding Muscles Using Deep Learning Ensemble

PURPOSE

The manual segmentation of organ structures in radiation oncology treatment planning is a time-consuming and highly skilled task, particularly when treating rare tumors like sacral chordomas. This study evaluates the performance of automated deep learning (DL) models in accurately segmenting the gross tumor volume (GTV) and surrounding muscle structures of sacral chordomas.

METHODS AND MATERIALS

An expert radiation oncologist contoured 5 muscle structures (gluteus maximus, gluteus medius, gluteus minimus, paraspinal, piriformis) and sacral chordoma GTV on computed tomography images from 48 patients. We trained 6 DL auto-segmentation models based on 3-dimensional U-Net and residual 3-dimensional U-Net architectures. We then implemented an average and an optimally weighted average ensemble to improve prediction performance. We evaluated algorithms with the average and standard deviation of the volumetric Dice similarity coefficient, surface Dice similarity coefficient with 2- and 3-mm thresholds, and average symmetric surface distance. One independent expert radiation oncologist assessed the clinical viability of the DL contours and determined the necessary amount of editing before they could be used in clinical practice.

RESULTS

Quantitatively, the ensembles performed the best across all structures. The optimal ensemble (volumetric Dice similarity coefficient, average symmetric surface distance) was (85.5 ± 6.4, 2.6 ± 0.8; GTV), (94.4 ± 1.5, 1.0 ± 0.4; gluteus maximus), (92.6 ± 0.9, 0.9 ± 0.1; gluteus medius), (85.0 ± 2.7, 1.1 ± 0.3; gluteus minimus), (92.1 ± 1.5, 0.8 ± 0.2; paraspinal), and (78.3 ± 5.7, 1.5 ± 0.6; piriformis). The qualitative evaluation suggested that the best model could reduce the total muscle and tumor delineation time to a 19-minute average.

CONCLUSIONS

Our methodology produces expert-level muscle and sacral chordoma tumor segmentation using DL and ensemble modeling. It can substantially augment the streamlining and accuracy of treatment planning and represents a critical step toward automated delineation of the clinical target volume in sarcoma and other disease sites.

Unsupervised pre-training of graph transformers on patient population graphs

Pre-training has shown success in different areas of machine learning, such as Computer Vision, Natural Language Processing (NLP), and medical imaging. However, it has not been fully explored for clinical data analysis. An immense amount of clinical records are recorded, but still, data and labels can be scarce for data collected in small hospitals or dealing with rare diseases. In such scenarios, pre-training on a larger set of unlabeled clinical data could improve performance. In this paper, we propose novel unsupervised pre-training techniques designed for heterogeneous, multi-modal clinical data for patient outcome prediction inspired by masked language modeling (MLM), by leveraging graph deep learning over population graphs. To this end, we further propose a graph-transformer-based network, designed to handle heterogeneous clinical data. By combining masking-based pre-training with a transformer-based network, we translate the success of masking-based pre-training in other domains to heterogeneous clinical data. We show the benefit of our pre-training method in a self-supervised and a transfer learning setting, utilizing three medical datasets TADPOLE, MIMIC-III, and a Sepsis Prediction Dataset. We find that our proposed pre-training methods help in modeling the data at a patient and population level and improve performance in different fine-tuning tasks on all datasets.

Maintaining High-Touch in High-Tech Digital Health Monitoring and Multi-Omics Prognostication: Ethical, Equity, and Societal Considerations in Precision Health for Palliative Care

Advances in digital health, systems biology, environmental monitoring, and artificial intelligence (AI) continue to revolutionize health care, ushering a precision health future. More than disease treatment and prevention, precision health aims at maintaining good health throughout the lifespan. However, how can precision health impact care for people with a terminal or life-limiting condition? We examine here the ethical, equity, and societal/relational implications of two precision health modalities, (1) integrated systems biology/multi-omics analysis for disease prognostication and (2) digital health technologies for health status monitoring and communication. We focus on three main ethical and societal considerations: benefits and risks associated with integration of these modalities into the palliative care system; inclusion of underrepresented and marginalized groups in technology development and deployment; and the impact of high-tech modalities on palliative care's highly personalized and "high-touch" practice. We conclude with 10 recommendations for ensuring that precision health technologies, such as multi-omics prognostication and digital health monitoring, for palliative care are developed, tested, and implemented ethically, inclusively, and equitably.

Attention-based multimodal fusion with contrast for robust clinical prediction in the face of missing modalities

OBJECTIVE

With the increasing amount and growing variety of healthcare data, multimodal machine learning supporting integrated modeling of structured and unstructured data is an increasingly important tool for clinical machine learning tasks. However, it is non-trivial to manage the differences in dimensionality, volume, and temporal characteristics of data modalities in the context of a shared target task. Furthermore, patients can have substantial variations in the availability of data, while existing multimodal modeling methods typically assume data completeness and lack a mechanism to handle missing modalities.

METHODS

We propose a Transformer-based fusion model with modality-specific tokens that summarize the corresponding modalities to achieve effective cross-modal interaction accommodating missing modalities in the clinical context. The model is further refined by inter-modal, inter-sample contrastive learning to improve the representations for better predictive performance. We denote the model as Attention-based cRoss-MOdal fUsion with contRast (ARMOUR). We evaluate ARMOUR using two input modalities (structured measurements and unstructured text), six clinical prediction tasks, and two evaluation regimes, either including or excluding samples with missing modalities.

RESULTS

Our model shows improved performances over unimodal or multimodal baselines in both evaluation regimes, including or excluding patients with missing modalities in the input. The contrastive learning improves the representation power and is shown to be essential for better results. The simple setup of modality-specific tokens enables ARMOUR to handle patients with missing modalities and allows comparison with existing unimodal benchmark results.

CONCLUSION

We propose a multimodal model for robust clinical prediction to achieve improved performance while accommodating patients with missing modalities. This work could inspire future research to study the effective incorporation of multiple, more complex modalities of clinical data into a single model.

The shaky foundations of large language models and foundation models for electronic health records

The success of foundation models such as ChatGPT and AlphaFold has spurred significant interest in building similar models for electronic medical records (EMRs) to improve patient care and hospital operations. However, recent hype has obscured critical gaps in our understanding of these models' capabilities. In this narrative review, we examine 84 foundation models trained on non-imaging EMR data (i.e., clinical text and/or structured data) and create a taxonomy delineating their architectures, training data, and potential use cases. We find that most models are trained on small, narrowly-scoped clinical datasets (e.g., MIMIC-III) or broad, public biomedical corpora (e.g., PubMed) and are evaluated on tasks that do not provide meaningful insights on their usefulness to health systems. Considering these findings, we propose an improved evaluation framework for measuring the benefits of clinical foundation models that is more closely grounded to metrics that matter in healthcare.

Integrative toxicogenomics: Advancing precision medicine and toxicology through artificial intelligence and OMICs technology

More information about a person's genetic makeup, drug response, multi-omics response, and genomic response is now available leading to a gradual shift towards personalized treatment. Additionally, the promotion of non-animal testing has fueled the computational toxicogenomics as a pivotal part of the next-gen risk assessment paradigm. Artificial Intelligence (AI) has the potential to provid new ways analyzing the patient data and making predictions about treatment outcomes or toxicity. As personalized medicine and toxicogenomics involve huge data processing, AI can expedite this process by providing powerful data processing, analysis, and interpretation algorithms. AI can process and integrate a multitude of data including genome data, patient records, clinical data and identify patterns to derive predictive models anticipating clinical outcomes and assessing the risk of any personalized medicine approaches. In this article, we have studied the current trends and future perspectives in personalized medicine & toxicology, the role of toxicogenomics in connecting the two fields, and the impact of AI on personalized medicine & toxicology. In this work, we also study the key challenges and limitations in personalized medicine, toxicogenomics, and AI in order to fully realize their potential.

High night-to-night variability in sleep apnea severity is associated with uncontrolled hypertension

Obstructive sleep apnea (OSA) severity can vary markedly from night-to-night. However, the impact of night-to-night variability in OSA severity on key cardiovascular outcomes such as hypertension is unknown. Thus, the primary aim of this study is to determine the effects of night-to-night variability in OSA severity on hypertension likelihood. This study uses in-home monitoring of 15,526 adults with ~180 nights per participant with an under-mattress sleep sensor device, plus ~30 repeat blood pressure measures. OSA severity is defined from the mean estimated apnea-hypopnoea index (AHI) over the ~6-month recording period for each participant. Night-to-night variability in severity is determined from the standard deviation of the estimated AHI across recording nights. Uncontrolled hypertension is defined as mean systolic blood pressure ≥140 mmHg and/or mean diastolic blood pressure ≥90 mmHg. Regression analyses are performed adjusted for age, sex, and body mass index. A total of 12,287 participants (12% female) are included in the analyses. Participants in the highest night-to-night variability quartile within each OSA severity category, have a 50-70% increase in uncontrolled hypertension likelihood versus the lowest variability quartile, independent of OSA severity. This study demonstrates that high night-to-night variability in OSA severity is a predictor of uncontrolled hypertension, independent of OSA severity. These findings have important implications for the identification of which OSA patients are most at risk of cardiovascular harm.

Applications of artificial intelligence in clinical management, research and health administration: imaging perspectives with a focus on hemophilia

INTRODUCTION

Joints of persons with hemophilia are frequently affected by repetitive hemarthrosis. In this paper concepts, perks and quirks of the use of artificial intelligence (AI), machine learning (ML) and deep learning are reviewed within clinical and research contexts of hemophilia and other blood-induced disorders' patient care, targeted to the imaging diagnosis of hemophilic joints, under the perspective of different stakeholders (radiologists, hematologists, nurses, physiotherapists, technologists, researchers, managers and patients/caregivers).

AREAS COVERED

Rubrics that determine the suitability of the utilization of AI in blood-induced disorders' patient care, including diagnosis and follow-up of patients are discussed, focusing on features in which AI can replace or augment the role of radiology in the clinical management and in research of patients. Insights on features in the design and conduct of AI projects in which the human intervention remains critical are provided.

EXPERT OPINION

The author discusses research concepts in radiogenomics, and challenges of the utilization of AI in different healthcare fields such as patient safety, data sharing and privacy regulations, workforce education and future jobs' shortage. Finally, the author proposes alternatives and potential solutions to mitigate challenges in successfully deploying ML algorithms into clinical practice.

Millimeter-wave radar object classification using knowledge-assisted neural network

To improve the cognition and understanding capabilities of artificial intelligence (AI) technology, it is a tendency to explore the human brain learning processing and integrate brain mechanisms or knowledge into neural networks for inspiration and assistance. This paper concentrates on the application of AI technology in advanced driving assistance system. In this field, millimeter-wave radar is essential for elaborate environment perception due to its robustness to adverse conditions. However, it is still challenging for radar object classification in the complex traffic environment. In this paper, a knowledge-assisted neural network (KANN) is proposed for radar object classification. Inspired by the human brain cognition mechanism and algorithms based on human expertise, two kinds of prior knowledge are injected into the neural network to guide its training and improve its classification accuracy. Specifically, image knowledge provides spatial information about samples. It is integrated into an attention mechanism in the early stage of the network to help reassign attention precisely. In the late stage, object knowledge is combined with the deep features extracted from the network. It contains discriminant semantic information about samples. An attention-based injection method is proposed to adaptively allocate weights to the knowledge and deep features, generating more comprehensive and discriminative features. Experimental results on measured data demonstrate that KANN is superior to current methods and the performance is improved with knowledge assistance.