A Review of Fusion Methods for Omics and Imaging Data

Deep learning model based on primary tumor to predict lymph node status in clinical stage IA lung adenocarcinoma: a multicenter study

Objective

To develop a deep learning model to predict lymph node (LN) status in clinical stage IA lung adenocarcinoma patients.

Methods

This diagnostic study included 1,009 patients with pathologically confirmed clinical stage T1N0M0 lung adenocarcinoma from two independent datasets (699 from Cancer Hospital of Chinese Academy of Medical Sciences and 310 from PLA General Hospital) between January 2005 and December 2019. The Cancer Hospital dataset was randomly split into a training cohort (559 patients) and a validation cohort (140 patients) to train and tune a deep learning model based on a deep residual network (ResNet). The PLA Hospital dataset was used as a testing cohort to evaluate the generalization ability of the model. Thoracic radiologists manually segmented tumors and interpreted high-resolution computed tomography (HRCT) features for the model. The predictive performance was assessed by area under the curves (AUCs), accuracy, precision, recall, and F1 score. Subgroup analysis was performed to evaluate the potential bias of the study population.

Results

A total of 1,009 patients were included in this study; 409 (40.5%) were male and 600 (59.5%) were female. The median age was 57.0 years (inter-quartile range, IQR: 50.0-64.0). The deep learning model achieved AUCs of 0.906 (95% CI: 0.873-0.938) and 0.893 (95% CI: 0.857-0.930) for predicting pN0 disease in the testing cohort and a non-pure ground glass nodule (non-pGGN) testing cohort, respectively. No significant difference was detected between the testing cohort and the non-pGGN testing cohort (P = 0.622). The precisions of this model for predicting pN0 disease were 0.979 (95% CI: 0.963-0.995) and 0.983 (95% CI: 0.967-0.998) in the testing cohort and the non-pGGN testing cohort, respectively. The deep learning model achieved AUCs of 0.848 (95% CI: 0.798-0.898) and 0.831 (95% CI: 0.776-0.887) for predicting pN2 disease in the testing cohort and the non-pGGN testing cohort, respectively. No significant difference was detected between the testing cohort and the non-pGGN testing cohort (P = 0.657). The recalls of this model for predicting pN2 disease were 0.903 (95% CI: 0.870-0.936) and 0.931 (95% CI: 0.901-0.961) in the testing cohort and the non-pGGN testing cohort, respectively.

Conclusions

The superior performance of the deep learning model will help to target the extension of lymph node dissection and reduce the ineffective lymph node dissection in early-stage lung adenocarcinoma patients.

Integrative approach of omics and imaging data to discover new insights for understanding brain diseases

Treatments that can completely resolve brain diseases have yet to be discovered. Omics is a novel technology that allows researchers to understand the molecular pathways underlying brain diseases. Multiple omics, including genomics, transcriptomics and proteomics, and brain imaging technologies, such as MRI, PET and EEG, have contributed to brain disease-related therapeutic target detection. However, new treatment discovery remains challenging. We focused on establishing brain multi-molecular maps using an integrative approach of omics and imaging to provide insights into brain disease diagnosis and treatment. This approach requires precise data collection using omics and imaging technologies, data processing and normalization. Incorporating a brain molecular map with the advanced technologies through artificial intelligence will help establish a system for brain disease diagnosis and treatment through regulation at the molecular level.

A machine learning and deep learning-based integrated multi-omics technique for leukemia prediction

In recent years, scientific data on cancer has expanded, providing potential for a better understanding of malignancies and improved tailored care. Advances in Artificial Intelligence (AI) processing power and algorithmic development position Machine Learning (ML) and Deep Learning (DL) as crucial players in predicting Leukemia, a blood cancer, using integrated multi-omics technology. However, realizing these goals demands novel approaches to harness this data deluge. This study introduces a novel Leukemia diagnosis approach, analyzing multi-omics data for accuracy using ML and DL algorithms. ML techniques, including Random Forest (RF), Naive Bayes (NB), Decision Tree (DT), Logistic Regression (LR), Gradient Boosting (GB), and DL methods such as Recurrent Neural Networks (RNN) and Feedforward Neural Networks (FNN) are compared. GB achieved 97 % accuracy in ML, while RNN outperformed by achieving 98 % accuracy in DL. This approach filters unclassified data effectively, demonstrating the significance of DL for leukemia prediction. The testing validation was based on 17 different features such as patient age, sex, mutation type, treatment methods, chromosomes, and others. Our study compares ML and DL techniques and chooses the best technique that gives optimum results. The study emphasizes the implications of high-throughput technology in healthcare, offering improved patient care.

Comprehensive multimodal deep learning survival prediction enabled by a transformer architecture: A multicenter study in glioblastoma

Background

This research aims to improve glioblastoma survival prediction by integrating MR images, clinical, and molecular-pathologic data in a transformer-based deep learning model, addressing data heterogeneity and performance generalizability.

Methods

We propose and evaluate a transformer-based nonlinear and nonproportional survival prediction model. The model employs self-supervised learning techniques to effectively encode the high-dimensional MRI input for integration with nonimaging data using cross-attention. To demonstrate model generalizability, the model is assessed with the time-dependent concordance index (Cdt) in 2 training setups using 3 independent public test sets: UPenn-GBM, UCSF-PDGM, and Rio Hortega University Hospital (RHUH)-GBM, each comprising 378, 366, and 36 cases, respectively.

Results

The proposed transformer model achieved a promising performance for imaging as well as nonimaging data, effectively integrating both modalities for enhanced performance (UCSF-PDGM test-set, imaging Cdt 0.578, multimodal Cdt 0.672) while outperforming state-of-the-art late-fusion 3D-CNN-based models. Consistent performance was observed across the 3 independent multicenter test sets with Cdt values of 0.707 (UPenn-GBM, internal test set), 0.672 (UCSF-PDGM, first external test set), and 0.618 (RHUH-GBM, second external test set). The model achieved significant discrimination between patients with favorable and unfavorable survival for all 3 datasets (log-rank P 1.9 × 10-8, 9.7 × 10-3, and 1.2 × 10-2). Comparable results were obtained in the second setup using UCSF-PDGM for training/internal testing and UPenn-GBM and RHUH-GBM for external testing (Cdt 0.670, 0.638, and 0.621).

Conclusions

The proposed transformer-based survival prediction model integrates complementary information from diverse input modalities, contributing to improved glioblastoma survival prediction compared to state-of-the-art methods. Consistent performance was observed across institutions supporting model generalizability.

Spatial mapping of tumor heterogeneity in whole-body PET-CT: a feasibility study

BACKGROUND

Tumor heterogeneity is recognized as a predictor of treatment response and patient outcome. Quantification of tumor heterogeneity across all scales may therefore provide critical insight that ultimately improves cancer management.

METHODS

An image registration-based framework for the study of tumor heterogeneity in whole-body images was evaluated on a dataset of 490 FDG-PET-CT images of lung cancer, lymphoma, and melanoma patients. Voxel-, lesion- and subject-level features were extracted from the subjects' segmented lesion masks and mapped to female and male template spaces for voxel-wise analysis. Resulting lesion feature maps of the three subsets of cancer patients were studied visually and quantitatively. Lesion volumes and lesion distances in subject spaces were compared with resulting properties in template space. The strength of the association between subject and template space for these properties was evaluated with Pearson's correlation coefficient.

RESULTS

Spatial heterogeneity in terms of lesion frequency distribution in the body, metabolic activity, and lesion volume was seen between the three subsets of cancer patients. Lesion feature maps showed anatomical locations with low versus high mean feature value among lesions sampled in space and also highlighted sites with high variation between lesions in each cancer subset. Spatial properties of the lesion masks in subject space correlated strongly with the same properties measured in template space (lesion volume, R = 0.986, p < 0.001; total metabolic volume, R = 0.988, p < 0.001; maximum within-patient lesion distance, R = 0.997, p < 0.001). Lesion volume and total metabolic volume increased on average from subject to template space (lesion volume, 3.1 ± 52 ml; total metabolic volume, 53.9 ± 229 ml). Pair-wise lesion distance decreased on average by 0.1 ± 1.6 cm and maximum within-patient lesion distance increased on average by 0.5 ± 2.1 cm from subject to template space.

CONCLUSIONS

Spatial tumor heterogeneity between subsets of interest in cancer cohorts can successfully be explored in whole-body PET-CT images within the proposed framework. Whole-body studies are, however, especially prone to suffer from regional variation in lesion frequency, and thus statistical power, due to the non-uniform distribution of lesions across a large field of view.

Development and validation of an artificial intelligence-powered acne grading system incorporating lesion identification

Background

The management of acne requires the consideration of its severity; however, a universally adopted evaluation system for clinical practice is lacking. Artificial intelligence (AI) evaluation systems hold the promise of enhancing the efficiency and reproducibility of assessments. Artificial intelligence (AI) evaluation systems offer the potential to enhance the efficiency and reproducibility of assessments in this domain. While the identification of skin lesions represents a crucial component of acne evaluation, existing AI systems often overlook lesion identification or fail to integrate it with severity assessment. This study aimed to develop an AI-powered acne grading system and compare its performance with physician image-based scoring.

Methods

A total of 1,501 acne patients were included in the study, and standardized pictures were obtained using the VISIA system. The initial evaluation involved 40 stratified sampled frontal photos assessed by seven dermatologists. Subsequently, the three doctors with the highest inter-rater agreement annotated the remaining 1,461 images, which served as the dataset for the development of the AI system. The dataset was randomly divided into two groups: 276 images were allocated for training the acne lesion identification platform, and 1,185 images were used to assess the severity of acne.

Results

The average precision of our model for skin lesion identification was 0.507 and the average recall was 0.775. The AI severity grading system achieved good agreement with the true label (linear weighted kappa = 0.652). After integrating the lesion identification results into the severity assessment with fixed weights and learnable weights, the kappa rose to 0.737 and 0.696, respectively, and the entire evaluation on a Linux workstation with a Tesla K40m GPU took less than 0.1s per picture.

Conclusion

This study developed a system that detects various types of acne lesions and correlates them well with acne severity grading, and the good accuracy and efficiency make this approach potentially an effective clinical decision support tool.

Data-driven support to decision-making in molecular tumour boards for lymphoma: A design science approach

Background

The increasing amount of molecular data and knowledge about genomic alterations from next-generation sequencing processes together allow for a greater understanding of individual patients, thereby advancing precision medicine. Molecular tumour boards feature multidisciplinary teams of clinical experts who meet to discuss complex individual cancer cases. Preparing the meetings is a manual and time-consuming process.

Purpose

To design a clinical decision support system to improve the multimodal data interpretation in molecular tumour board meetings for lymphoma patients at Karolinska University Hospital, Stockholm, Sweden. We investigated user needs and system requirements, explored the employment of artificial intelligence, and evaluated the proposed design with primary stakeholders.

Methods

Design science methodology was used to form and evaluate the proposed artefact. Requirements elicitation was done through a scoping review followed by five semi-structured interviews. We used UML Use Case diagrams to model user interaction and UML Activity diagrams to inform the proposed flow of control in the system. Additionally, we modelled the current and future workflow for MTB meetings and its proposed machine learning pipeline. Interactive sessions with end-users validated the initial requirements based on a fictive patient scenario which helped further refine the system.

Results

The analysis showed that an interactive secure Web-based information system supporting the preparation of the meeting, multidisciplinary discussions, and clinical decision-making could address the identified requirements. Integrating artificial intelligence via continual learning and multimodal data fusion were identified as crucial elements that could provide accurate diagnosis and treatment recommendations.

Impact

Our work is of methodological importance in that using artificial intelligence for molecular tumour boards is novel. We provide a consolidated proof-of-concept system that could support the end-to-end clinical decision-making process and positively and immediately impact patients.

Conclusion

Augmenting a digital decision support system for molecular tumour boards with retrospective patient material is promising. This generates realistic and constructive material for human learning, and also digital data for continual learning by data-driven artificial intelligence approaches. The latter makes the future system adaptable to human bias, improving adequacy and decision quality over time and over tasks, while building and maintaining a digital log.