Deep Learning of Imaging Phenotype and Genotype for Predicting Overall Survival Time of Glioblastoma Patients

Multi-sequence generative adversarial network: better generation for enhanced magnetic resonance imaging images

Introduction

MRI is one of the commonly used diagnostic methods in clinical practice, especially in brain diseases. There are many sequences in MRI, but T1CE images can only be obtained by using contrast agents. Many patients (such as cancer patients) must undergo alignment of multiple MRI sequences for diagnosis, especially the contrast-enhanced magnetic resonance sequence. However, some patients such as pregnant women, children, etc. find it difficult to use contrast agents to obtain enhanced sequences, and contrast agents have many adverse reactions, which can pose a significant risk. With the continuous development of deep learning, the emergence of generative adversarial networks makes it possible to extract features from one type of image to generate another type of image.

Methods

We propose a generative adversarial network model with multimodal inputs and end-to-end decoding based on the pix2pix model. For the pix2pix model, we used four evaluation metrics: NMSE, RMSE, SSIM, and PNSR to assess the effectiveness of our generated model.

Results

Through statistical analysis, we compared our proposed new model with pix2pix and found significant differences between the two. Our model outperformed pix2pix, with higher SSIM and PNSR, lower NMSE and RMSE. We also found that the input of T1W images and T2W images had better effects than other combinations, providing new ideas for subsequent work on generating magnetic resonance enhancement sequence images. By using our model, it is possible to generate magnetic resonance enhanced sequence images based on magnetic resonance non-enhanced sequence images.

Discussion

This has significant implications as it can greatly reduce the use of contrast agents to protect populations such as pregnant women and children who are contraindicated for contrast agents. Additionally, contrast agents are relatively expensive, and this generation method may bring about substantial economic benefits.

A New Automated Prognostic Prediction Method Based on Multi-Sequence Magnetic Resonance Imaging for Hepatic Resection of Colorectal Cancer Liver Metastases

Colorectal cancer is a prevalent and life-threatening disease, where colorectal cancer liver metastasis (CRLM) exhibits the highest mortality rate. Currently, surgery stands as the most effective curative option for eligible patients. However, due to the insufficient performance of traditional methods and the lack of multi-modality MRI feature complementarity in existing deep learning methods, the prognosis of CRLM surgical resection has not been fully explored. This paper proposes a new method, multi-modal guided complementary network (MGCNet), which employs multi-sequence MRI to predict 1-year recurrence and recurrence-free survival in patients after CRLM resection. In light of the complexity and redundancy of features in the liver region, we designed the multi-modal guided local feature fusion module to utilize the tumor features to guide the dynamic fusion of prognostically relevant local features within the liver. On the other hand, to solve the loss of spatial information during multi-sequence MRI fusion, the cross-modal complementary external attention module designed an external mask branch to establish inter-layer correlation. The results show that the model has accuracy (ACC) of 0.79, the area under the curve (AUC) of 0.84, C-Index of 0.73, and hazard ratio (HR) of 4.0, which is a significant improvement over state-of-the-art methods. Additionally, MGCNet exhibits good interpretability.

Convolutional Neural Networks for Glioma Segmentation and Prognosis: A Systematic Review

Deep learning (DL) is poised to redefine the way medical images are processed and analyzed. Convolutional neural networks (CNNs), a specific type of DL architecture, are exceptional for high-throughput processing, allowing for the effective extraction of relevant diagnostic patterns from large volumes of complex visual data. This technology has garnered substantial interest in the field of neuro-oncology as a promising tool to enhance medical imaging throughput and analysis. A multitude of methods harnessing MRI-based CNNs have been proposed for brain tumor segmentation, classification, and prognosis prediction. They are often applied to gliomas, the most common primary brain cancer, to classify subtypes with the goal of guiding therapy decisions. Additionally, the difficulty of repeating brain biopsies to evaluate treatment response in the setting of often confusing imaging findings provides a unique niche for CNNs to help distinguish the treatment response to gliomas. For example, glioblastoma, the most aggressive type of brain cancer, can grow due to poor treatment response, can appear to grow acutely due to treatment-related inflammation as the tumor dies (pseudo-progression), or falsely appear to be regrowing after treatment as a result of brain damage from radiation (radiation necrosis). CNNs are being applied to separate this diagnostic dilemma. This review provides a detailed synthesis of recent DL methods and applications for intratumor segmentation, glioma classification, and prognosis prediction. Furthermore, this review discusses the future direction of MRI-based CNN in the field of neuro-oncology and challenges in model interpretability, data availability, and computation efficiency.

Artificial intelligence in neuroimaging of brain tumors: reality or still promise?

PURPOSE OF REVIEW

To provide an updated overview of artificial intelligence (AI) applications in neuro-oncologic imaging and discuss current barriers to wider clinical adoption.

RECENT FINDINGS

A wide variety of AI applications in neuro-oncologic imaging have been developed and researched, spanning tasks from pretreatment brain tumor classification and segmentation, preoperative planning, radiogenomics, prognostication and survival prediction, posttreatment surveillance, and differentiating between pseudoprogression and true disease progression. While earlier studies were largely based on data from a single institution, more recent studies have demonstrated that the performance of these algorithms are also effective on external data from other institutions. Nevertheless, most of these algorithms have yet to see widespread clinical adoption, given the lack of prospective studies demonstrating their efficacy and the logistical difficulties involved in clinical implementation.

SUMMARY

While there has been significant progress in AI and neuro-oncologic imaging, clinical utility remains to be demonstrated. The next wave of progress in this area will be driven by prospective studies measuring outcomes relevant to clinical practice and go beyond retrospective studies which primarily aim to demonstrate high performance.

Artificial Intelligence Applications in Glioma With 1p/19q Co-Deletion: A Systematic Review

As an important genomic marker for oligodendrogliomas, early determination of 1p/19q co-deletion status is critical for guiding therapy and predicting prognosis in patients with glioma. The purpose of this study is to systematically review the literature concerning the magnetic resonance imaging (MRI) with artificial intelligence (AI) methods for predicting 1p/19q co-deletion status in glioma. PubMed, Scopus, Embase, and IEEE Xplore were searched in accordance with the Preferred Reporting Items for systematic reviews and meta-analyses guidelines. Methodological quality of studies was assessed according to the Quality Assessment of Diagnostic Accuracy Studies-2. Finally, 28 studies were included in the quantitative analysis. Diagnostic test accuracy reached an area under the ROC curve of 0.71-0.98 were reported in 24 studies. The remaining four studies with no available AUC provided an accuracy of 0.75-0. 89. The included studies varied widely in terms of imaging sequences, input features, and modeling methods. The current review highlighted that integrating MRI with AI technology is a potential tool for determination 1p/19q status pre-operatively and noninvasively, which can possibly help clinical decision-making. However, the reliability and feasibility of this approach still need to be further validated and improved in a real clinical setting. EVIDENCE LEVEL: 2. TECHNICAL EFFICACY: 2.

Deep learning in precision medicine and focus on glioma

Deep learning (DL) has been successfully applied to different fields for a range of tasks. In medicine, DL methods have been also used to improve the efficiency of disease diagnosis. In this review, we first summarize the history of the development of artificial intelligence models, demonstrate the features of the subtypes of machine learning and different DL networks, and then explore their application in the different fields of precision medicine, such as cardiology, gastroenterology, ophthalmology, dermatology, and oncology. By digging more information and extracting multilevel features from medical data, we found that DL helps doctors assess diseases automatically and monitor patients' physical health. In gliomas, research regarding application prospect of DL was mainly shown through magnetic resonance imaging and then by pathological slides. However, multi-omics data, such as whole exome sequence, RNA sequence, proteomics, and epigenomics, have not been covered thus far. In general, the quality and quantity of DL datasets still need further improvements, and more fruitful multi-omics characteristics will bring more comprehensive and accurate diagnosis in precision medicine and glioma.

The Efficacy of Pretreatment 18F-FDG PET-CT-Based Deep Learning Network Structure to Predict Survival in Nasopharyngeal Carcinoma

Background

Previous studies have shown that the 5-year survival rates of patients with nasopharyngeal carcinoma (NPC) were still not ideal despite great improvement in NPC treatments. To achieve individualized treatment of NPC, we have been looking for novel models to predict the prognosis of patients with NPC. The objective of this study was to use a novel deep learning network structural model to predict the prognosis of patients with NPC and to compare it with the traditional PET-CT model combining metabolic parameters and clinical factors.

Methods

A total of 173 patients were admitted to 2 institutions between July 2014 and April 2020 for the retrospective study; each received a PET-CT scan before treatment. The least absolute shrinkage and selection operator (LASSO) was employed to select some features, including SUVpeak-P, T3, age, stage II, MTV-P, N1, stage III and pathological type, which were associated with overall survival (OS) of patients. We constructed 2 survival prediction models: an improved optimized adaptive multimodal task (a 3D Coordinate Attention Convolutional Autoencoder and an uncertainty-based jointly Optimizing Cox Model, CACA-UOCM for short) and a clinical model. The predictive power of these models was assessed using the Harrell Consistency Index (C index). Overall survival of patients with NPC was compared by Kaplan-Meier and Log-rank tests.

Results

The results showed that CACA-UOCM model could estimate OS (C index, 0.779 for training, 0.774 for validation, and 0.819 for testing) and divide patients into low and high mortality risk groups, which were significantly associated with OS (P < .001). However, the C-index of the model based only on clinical variables was only 0.42.

Conclusions

The deep learning network model based on ¹⁸F-FDG PET/CT can serve as a reliable and powerful predictive tool for NPC and provide therapeutic strategies for individual treatment.

Emerging role of artificial intelligence in diagnosis, classification and clinical management of glioma

Glioma represents a dominant primary intracranial malignancy in the central nervous system. Artificial intelligence that mainly includes machine learning, and deep learning computational approaches, presents a unique opportunity to enhance clinical management of glioma through improving tumor segmentation, diagnosis, differentiation, grading, treatment, prediction of clinical outcomes (prognosis, and recurrence), molecular features, clinical classification, characterization of the tumor microenvironment, and drug discovery. A growing body of recent studies apply artificial intelligence-based models to disparate data sources of glioma, covering imaging modalities, digital pathology, high-throughput multi-omics data (especially emerging single-cell RNA sequencing and spatial transcriptome), etc. While these early findings are promising, future studies are required to normalize artificial intelligence-based models to improve the generalizability and interpretability of the results. Despite prominent issues, targeted clinical application of artificial intelligence approaches in glioma will facilitate the development of precision medicine of this field. If these challenges can be overcome, artificial intelligence has the potential to profoundly change the way patients with or at risk of glioma are provided with more rational care.

Multi-task deep learning for medical image computing and analysis: A review

The renaissance of deep learning has provided promising solutions to various tasks. While conventional deep learning models are constructed for a single specific task, multi-task deep learning (MTDL) that is capable to simultaneously accomplish at least two tasks has attracted research attention. MTDL is a joint learning paradigm that harnesses the inherent correlation of multiple related tasks to achieve reciprocal benefits in improving performance, enhancing generalizability, and reducing the overall computational cost. This review focuses on the advanced applications of MTDL for medical image computing and analysis. We first summarize four popular MTDL network architectures (i.e., cascaded, parallel, interacted, and hybrid). Then, we review the representative MTDL-based networks for eight application areas, including the brain, eye, chest, cardiac, abdomen, musculoskeletal, pathology, and other human body regions. While MTDL-based medical image processing has been flourishing and demonstrating outstanding performance in many tasks, in the meanwhile, there are performance gaps in some tasks, and accordingly we perceive the open challenges and the perspective trends. For instance, in the 2018 Ischemic Stroke Lesion Segmentation challenge, the reported top dice score of 0.51 and top recall of 0.55 achieved by the cascaded MTDL model indicate further research efforts in high demand to escalate the performance of current models.

Image-based deep learning identifies glioblastoma risk groups with genomic and transcriptomic heterogeneity: a multi-center study

OBJECTIVES

To develop and validate a deep learning imaging signature (DLIS) for risk stratification in patients with multiforme (GBM), and to investigate the biological pathways and genetic alterations underlying the DLIS.

METHODS

The DLIS was developed from multi-parametric MRI based on a training set (n = 600) and validated on an internal validation set (n = 164), an external test set 1 (n = 100), an external test set 2 (n = 161), and a public TCIA set (n = 88). A co-profiling framework based on a radiogenomics analysis dataset (n = 127) using multiscale high-dimensional data, including imaging, transcriptome, and genome, was established to uncover the biological pathways and genetic alterations underpinning the DLIS.

RESULTS

The DLIS was associated with survival (log-rank p < 0.001) and was an independent predictor (p < 0.001). The integrated nomogram incorporating the DLIS achieved improved C indices than the clinicomolecular nomogram (net reclassification improvement 0.39, p < 0.001). DLIS significantly correlated with core pathways of GBM (apoptosis and cell cycle-related P53 and RB pathways, and cell proliferation-related RTK pathway), as well as key genetic alterations (del_CDNK2A). The prognostic value of DLIS-correlated genes was externally confirmed on TCGA/CGGA sets (p < 0.01).

CONCLUSIONS

Our study offers a biologically interpretable deep learning predictor of survival outcomes in patients with GBM, which is crucial for better understanding GBM patient's prognosis and guiding individualized treatment.

KEY POINTS

• MRI-based deep learning imaging signature (DLIS) stratifies GBM into risk groups with distinct molecular characteristics. • DLIS is associated with P53, RB, and RTK pathways and del_CDNK2A mutation. • The prognostic value of DLIS-correlated pathway genes is externally demonstrated.

Multiview Deep Forest for Overall Survival Prediction in Cancer

Overall survival (OS) in cancer is crucial for cancer treatment. Many machine learning methods have been applied to predict OS, but there are still the challenges of dealing with multiview data and overfitting. To overcome these problems, we propose a multiview deep forest (MVDF) in this paper. MVDF can learn the features of each view and fuse them with integrated learning and multiple kernel learning. Then, a gradient boost forest based on the information bottleneck theory is proposed to reduce redundant information and avoid overfitting. In addition, a pruning strategy for a cascaded forest is used to limit the impact of outlier data. Comprehensive experiments have been carried out on a data set from West China Hospital of Sichuan University and two public data sets. Results have demonstrated that our method outperforms the compared methods in predicting overall survival.

Survival prediction for patients with glioblastoma multiforme using a Cox proportional hazards denoising autoencoder network

Objectives

This study aimed to establish and validate a prognostic model based on magnetic resonance imaging and clinical features to predict the survival time of patients with glioblastoma multiforme (GBM).

Methods

In this study, a convolutional denoising autoencoder (DAE) network combined with the loss function of the Cox proportional hazard regression model was used to extract features for survival prediction. In addition, the Kaplan-Meier curve, the Schoenfeld residual analysis, the time-dependent receiver operating characteristic curve, the nomogram, and the calibration curve were performed to assess the survival prediction ability.

Results

The concordance index (C-index) of the survival prediction model, which combines the DAE and the Cox proportional hazard regression model, reached 0.78 in the training set, 0.75 in the validation set, and 0.74 in the test set. Patients were divided into high- and low-risk groups based on the median prognostic index (PI). Kaplan-Meier curve was used for survival analysis (p = < 2e-16 in the training set, p = 3e-04 in the validation set, and p = 0.007 in the test set), which showed that the survival probability of different groups was significantly different, and the PI of the network played an influential role in the prediction of survival probability. In the residual verification of the PI, the fitting curve of the scatter plot was roughly parallel to the x-axis, and the p-value of the test was 0.11, proving that the PI and survival time were independent of each other and the survival prediction ability of the PI was less affected than survival time. The areas under the curve of the training set were 0.843, 0.871, 0.903, and 0.941; those of the validation set were 0.687, 0.895, 1.000, and 0.967; and those of the test set were 0.757, 0.852, 0.683, and 0.898.

Conclusion

The survival prediction model, which combines the DAE and the Cox proportional hazard regression model, can effectively predict the prognosis of patients with GBM.

A multi-perspective information aggregation network for automatedT-staging detection of nasopharyngeal carcinoma

AccurateT-staging is important when planning personalized radiotherapy. However,T-staging via manual slice-by-slice inspection is time-consuming while tumor sizes and shapes are heterogeneous, and junior physicians find such inspection challenging. With inspiration from oncological diagnostics, we developed a multi-perspective aggregation network that incorporated various diagnosis-oriented knowledge which allowed automated nasopharyngeal carcinomaT-staging detection (TSD Net). Specifically, our TSD Net was designed in multi-branch architecture, which can capture tumor size and shape information (basic knowledge), strongly correlated contextual features, and associations between the tumor and surrounding tissues. We defined the association between the tumor and surrounding tissues by a signed distance map which can embed points and tumor contours in higher-dimensional spaces, yielding valuable information regarding the locations of tissue associations. TSD Net finally outputs aT1-T4 stage prediction by aggregating data from the three branches. We evaluated TSD Net by using the T1-weighted contrast-enhanced magnetic resonance imaging database of 320 patients in a three-fold cross-validation manner. The results show that the proposed method achieves a mean area under the curve (AUC) as high as 87.95%. We also compared our method to traditional classifiers and a deep learning-based method. Our TSD Net is efficient and accurate and outperforms other methods.

Ensemble learning for glioma patients overall survival prediction using pre-operative MRIs

Objective: Gliomas are the most common primary brain tumors. Approximately 70% of the glioma patients diagnosed with glioblastoma have an averaged overall survival (OS) of only ∼16 months. Early survival prediction is essential for treatment decision-making in glioma patients. Here we proposed an ensemble learning approach to predict the post-operative OS of glioma patients using only pre-operative MRIs.Approach: Our dataset was from the Medical Image Computing and Computer Assisted Intervention Brain Tumor Segmentation challenge 2020, which consists of multimodal pre-operative MRI scans of 235 glioma patients with survival days recorded. The backbone of our approach was a Siamese network consisting of twinned ResNet-based feature extractors followed by a 3-layer classifier. During training, the feature extractors explored traits of intra and inter-class by minimizing contrastive loss of randomly paired 2D pre-operative MRIs, and the classifier utilized the extracted features to generate labels with cost defined by cross-entropy loss. During testing, the extracted features were also utilized to define distance between the test sample and the reference composed of training data, to generate an additional predictor via K-NN classification. The final label was the ensemble classification from both the Siamese model and the K-NN model.Main results: Our approach classifies the glioma patients into 3 OS classes: long-survivors (>15 months), mid-survivors (between 10 and 15 months) and short-survivors (<10 months). The performance is assessed by the accuracy (ACC) and the area under the curve (AUC) of 3-class classification. The final result achieved an ACC of 65.22% and AUC of 0.81.Significance: Our Siamese network based ensemble learning approach demonstrated promising ability in mining discriminative features with minimal manual processing and generalization requirement. This prediction strategy can be potentially applied to assist timely clinical decision-making.

Deep learning and radiomics to predict the mitotic index of gastrointestinal stromal tumors based on multiparametric MRI

Introduction

Preoperative evaluation of the mitotic index (MI) of gastrointestinal stromal tumors (GISTs) represents the basis of individualized treatment of patients. However, the accuracy of conventional preoperative imaging methods is limited. The aim of this study was to develop a predictive model based on multiparametric MRI for preoperative MI prediction.

Methods

A total of 112 patients who were pathologically diagnosed with GIST were enrolled in this study. The dataset was subdivided into the development (n = 81) and test (n = 31) sets based on the time of diagnosis. With the use of T2-weighted imaging (T2WI) and apparent diffusion coefficient (ADC) map, a convolutional neural network (CNN)-based classifier was developed for MI prediction, which used a hybrid approach based on 2D tumor images and radiomics features from 3D tumor shape. The trained model was tested on an internal test set. Then, the hybrid model was comprehensively tested and compared with the conventional ResNet, shape radiomics classifier, and age plus diameter classifier.

Results

The hybrid model showed good MI prediction ability at the image level; the area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve (AUPRC), and accuracy in the test set were 0.947 (95% confidence interval [CI]: 0.927-0.968), 0.964 (95% CI: 0.930-0.978), and 90.8 (95% CI: 88.0-93.0), respectively. With the average probabilities from multiple samples per patient, good performance was also achieved at the patient level, with AUROC, AUPRC, and accuracy of 0.930 (95% CI: 0.828-1.000), 0.941 (95% CI: 0.792-1.000), and 93.6% (95% CI: 79.3-98.2) in the test set, respectively.

Discussion

The deep learning-based hybrid model demonstrated the potential to be a good tool for the operative and non-invasive prediction of MI in GIST patients.

Predicting Survival in Patients with Brain Tumors: Current State-of-the-Art of AI Methods Applied to MRI

Given growing clinical needs, in recent years Artificial Intelligence (AI) techniques have increasingly been used to define the best approaches for survival assessment and prediction in patients with brain tumors. Advances in computational resources, and the collection of (mainly) public databases, have promoted this rapid development. This narrative review of the current state-of-the-art aimed to survey current applications of AI in predicting survival in patients with brain tumors, with a focus on Magnetic Resonance Imaging (MRI). An extensive search was performed on PubMed and Google Scholar using a Boolean research query based on MeSH terms and restricting the search to the period between 2012 and 2022. Fifty studies were selected, mainly based on Machine Learning (ML), Deep Learning (DL), radiomics-based methods, and methods that exploit traditional imaging techniques for survival assessment. In addition, we focused on two distinct tasks related to survival assessment: the first on the classification of subjects into survival classes (short and long-term or eventually short, mid and long-term) to stratify patients in distinct groups. The second focused on quantification, in days or months, of the individual survival interval. Our survey showed excellent state-of-the-art methods for the first, with accuracy up to ∼98%. The latter task appears to be the most challenging, but state-of-the-art techniques showed promising results, albeit with limitations, with C-Index up to ∼0.91. In conclusion, according to the specific task, the available computational methods perform differently, and the choice of the best one to use is non-univocal and dependent on many aspects. Unequivocally, the use of features derived from quantitative imaging has been shown to be advantageous for AI applications, including survival prediction. This evidence from the literature motivates further research in the field of AI-powered methods for survival prediction in patients with brain tumors, in particular, using the wealth of information provided by quantitative MRI techniques.

Deep Neural Network for the Prediction of KRAS Genotype in Rectal Cancer

BACKGROUND

KRAS mutation can alter the treatment plan after resection of colorectal cancer. Despite its importance, the KRAS status of several patients remains unchecked because of the high cost and limited resources. This study developed a deep neural network (DNN) to predict the KRAS genotype using hematoxylin and eosin (H&E)-stained histopathological images.

STUDY DESIGN

Three DNNs were created (KRAS_Mob, KRAS_Shuff, and KRAS_Ince) using the structural backbone of the MobileNet, ShuffleNet, and Inception networks, respectively. The Cancer Genome Atlas was screened to extract 49,684 image tiles that were used for deep learning and internal validation. An independent cohort of 43,032 image tiles was used for external validation. The performance was compared with humans, and a virtual cost-saving analysis was done.

RESULTS

The KRAS_Mob network (area under the receiver operating curve [AUC] 0.8, 95% CI 0.71 to 0.89) was the best-performing model for predicting the KRAS genotype, followed by the KRAS_Shuff (AUC 0.73, 95% CI 0.62 to 0.84) and KRAS_Ince (AUC 0.71, 95% CI 0.6 to 0.82) networks. Combing the KRAS_Mob and KRAS_Shuff networks as a double prediction approach showed improved performance. KRAS_Mob network accuracy surpassed that of two independent pathologists (AUC 0.79 [95% CI 0.64 to 0.93], 0.51 [95% CI 0.34 to 0.69], and 0.51 (95% CI 0.34 to 0.69]; p < 0.001 for all comparisons).

CONCLUSION

The DNN has the potential to predict the KRAS genotype directly from H&E-stained histopathological slide images. As an algorithmic screening method to prioritize patients for laboratory confirmation, such a model might possibly reduce the number of patients screened, resulting in significant test-related time and economic savings.

Artificial intelligence in multi-parametric magnetic resonance imaging: A review

Multi-parametric magnetic resonance imaging (mpMRI) is an indispensable tool in the clinical workflow for the diagnosis and treatment planning of various diseases. Machine learning-based artificial intelligence (AI) methods, especially those adopting the deep learning technique, have been extensively employed to perform mpMRI image classification, segmentation, registration, detection, reconstruction, and super-resolution. The current availability of increasing computational power and fast-improving AI algorithms have empowered numerous computer-based systems for applying mpMRI to disease diagnosis, imaging-guided radiotherapy, patient risk and overall survival time prediction, and the development of advanced quantitative imaging technology for magnetic resonance fingerprinting. However, the wide application of these developed systems in the clinic is still limited by a number of factors, including robustness, reliability, and interpretability. This survey aims to provide an overview for new researchers in the field as well as radiologists with the hope that they can understand the general concepts, main application scenarios, and remaining challenges of AI in mpMRI. This article is protected by copyright. All rights reserved.

Glioma survival prediction from whole-brain MRI without tumor segmentation using deep attention network: a multicenter study

OBJECTIVES

To develop and validate a deep learning model for predicting overall survival from whole-brain MRI without tumor segmentation in patients with diffuse gliomas.

METHODS

In this multicenter retrospective study, two deep learning models were built for survival prediction from MRI, including a DeepRisk model built from whole-brain MRI, and an original ResNet model built from expert-segmented tumor images. Both models were developed using a training dataset (n = 935) and an internal tuning dataset (n = 156) and tested on two external test datasets (n = 194 and 150) and a TCIA dataset (n = 121). C-index, integrated Brier score (IBS), prediction error curves, and calibration curves were used to assess the model performance.

RESULTS

In total, 1556 patients were enrolled (age, 49.0 ± 13.1 years; 830 male). The DeepRisk score was an independent predictor and can stratify patients in each test dataset into three risk subgroups. The IBS and C-index for DeepRisk were 0.14 and 0.83 in external test dataset 1, 0.15 and 0.80 in external dataset 2, and 0.16 and 0.77 in TCIA dataset, respectively, which were comparable with those for original ResNet. The AUCs at 6, 12, 24, 26, and 48 months for DeepRisk ranged between 0.77 and 0.94. Combining DeepRisk score with clinicomolecular factors resulted in a nomogram with a better calibration and classification accuracy (net reclassification improvement 0.69, p < 0.001) than the clinical nomogram.

CONCLUSIONS

DeepRisk that obviated the need of tumor segmentation can predict glioma survival from whole-brain MRI and offers incremental prognostic value.

KEY POINTS

• DeepRisk can predict overall survival directly from whole-brain MRI without tumor segmentation. • DeepRisk achieves comparable accuracy in survival prediction with deep learning model built using expert-segmented tumor images. • DeepRisk has independent and incremental prognostic value over existing clinical parameters and IDH mutation status.

Overall survival time prediction for glioblastoma using multimodal deep KNN

Glioblastoma (GBM) is a severe malignant brain tumor with bad prognosis, and overall survival (OS) time prediction is of great clinical value for customized treatment. Recently, many deep learning (DL) based methods have been proposed, and most of them build deep networks to directly map pre-operative images of patients to the OS time. However, such end-to-end prediction is sensitive to data inconsistency and noise. In this paper, inspired by the fact that clinicians usually evaluate patient prognosis according to previously encountered similar cases, we propose a novel multimodal deep KNN based OS time prediction method. Specifically, instead of the end-to-end prediction, for each input patient, our method first search its K nearest patients with known OS time in a learned metric space, and the final OS time of the input patient is jointly determined by the K nearest patients, which is robust to data inconsistency and noise. Moreover, to take advantage of multiple imaging modalities, a new inter-modality loss is introduced to encourage learning complementary features from different modalities. The in-house single-center dataset containing multimodal MR brain images of 78 GBM patients is used to evaluate our method. In addition, to demonstrate that our method is not limited to GBM, a public multi-center dataset (BRATS2019) containing 211 patients with low and high grade gliomas is also used in our experiment. As benefiting from the deep KNN and the inter-modality loss, our method outperforms all methods under evaluation in both datasets. To the best of our knowledge, this is the first work, which predicts the OS time of GBM patients in the strategy of KNN under the DL framework.

Artificial Intelligence for Survival Prediction in Brain Tumors on Neuroimaging

Survival prediction of patients affected by brain tumors provides essential information to guide surgical planning, adjuvant treatment selection, and patient counseling. Current reliance on clinical factors, such as Karnofsky Performance Status Scale, and simplistic radiological characteristics are, however, inadequate for survival prediction in tumors such as glioma that demonstrate molecular and clinical heterogeneity with variable survival outcomes. Advances in the domain of artificial intelligence have afforded powerful tools to capture a large number of hidden high-dimensional imaging features that reflect abundant information about tumor structure and physiology. Here, we provide an overview of current literature that apply computational analysis tools such as radiomics and machine learning methods to the pipeline of image preprocessing, tumor segmentation, feature extraction, and construction of classifiers to establish survival prediction models based on neuroimaging. We also discuss challenges relating to the development and evaluation of such models and explore ethical issues surrounding the future use of machine learning predictions.

Overview of radiomics in prostate imaging and future directions

Recent advancements in imaging technology and analysis methods have led to an analytic framework known as radiomics. This framework extracts comprehensive high-dimensional features from imaging data and performs data mining to build analytical models for improved decision-support. Its features include many categories spanning texture and shape; thus, it can provide abundant information for precision medicine. Many studies of prostate radiomics have shown promising results in the assessment of pathological features, prediction of treatment response, and stratification of risk groups. Herein, we aimed to provide a general overview of radiomics procedures, discuss technical issues, explain various clinical applications, and suggest future research directions, especially for prostate imaging.

Deep Learning for Outcome Prediction in Neurosurgery: A Systematic Review of Design, Reporting, and Reproducibility

Deep learning (DL) is a powerful machine learning technique that has increasingly been used to predict surgical outcomes. However, the large quantity of data required and lack of model interpretability represent substantial barriers to the validity and reproducibility of DL models. The objective of this study was to systematically review the characteristics of DL studies involving neurosurgical outcome prediction and to assess their bias and reporting quality. Literature search using the PubMed, Scopus, and Embase databases identified 1949 records of which 35 studies were included. Of these, 32 (91%) developed and validated a DL model while 3 (9%) validated a pre-existing model. The most commonly represented subspecialty areas were oncology (16 of 35, 46%), spine (8 of 35, 23%), and vascular (6 of 35, 17%). Risk of bias was low in 18 studies (51%), unclear in 5 (14%), and high in 12 (34%), most commonly because of data quality deficiencies. Adherence to transparent reporting of a multivariable prediction model for individual prognosis or diagnosis reporting standards was low, with a median of 12 transparent reporting of a multivariable prediction model for individual prognosis or diagnosis items (39%) per study not reported. Model transparency was severely limited because code was provided in only 3 studies (9%) and final models in 2 (6%). With the exception of public databases, no study data sets were readily available. No studies described DL models as ready for clinical use. The use of DL for neurosurgical outcome prediction remains nascent. Lack of appropriate data sets poses a major concern for bias. Although studies have demonstrated promising results, greater transparency in model development and reporting is needed to facilitate reproducibility and validation.

Radiomics-guided deep neural networks stratify lung adenocarcinoma prognosis from CT scans

Deep learning (DL) is a breakthrough technology for medical imaging with high sample size requirements and interpretability issues. Using a pretrained DL model through a radiomics-guided approach, we propose a methodology for stratifying the prognosis of lung adenocarcinomas based on pretreatment CT. Our approach allows us to apply DL with smaller sample size requirements and enhanced interpretability. Baseline radiomics and DL models for the prognosis of lung adenocarcinomas were developed and tested using local (n = 617) cohort. The DL models were further tested in an external validation (n = 70) cohort. The local cohort was divided into training and test cohorts. A radiomics risk score (RRS) was developed using Cox-LASSO. Three pretrained DL networks derived from natural images were used to extract the DL features. The features were further guided using radiomics by retaining those DL features whose correlations with the radiomics features were high and Bonferroni-corrected p-values were low. The retained DL features were subject to a Cox-LASSO when constructing DL risk scores (DRS). The risk groups stratified by the RRS and DRS showed a significant difference in training, testing, and validation cohorts. The DL features were interpreted using existing radiomics features, and the texture features explained the DL features well.

Fusion of 3D lung CT and serum biomarkers for diagnosis of multiple pathological types on pulmonary nodules

BACKGROUND AND OBJECTIVE

Current researches on pulmonary nodules mainly focused on the binary-classification of benign and malignant pulmonary nodules. However, in clinical applications, it is not enough to judge whether pulmonary nodules are benign or malignant. In this paper, we proposed a fusion model based on the Lung Information Dataset Containing 3D CT Images and Serum Biomarkers (LIDCCISB) we constructed to accurately diagnose the types of pulmonary nodules in squamous cell carcinoma, adenocarcinoma, inflammation and other benign diseases.

METHODS

Using single modal information of lung 3D CT images and single modal information of Lung Tumor Biomarkers (LTBs) in LIDCCISB, a Multi-resolution 3D Multi-classification deep learning model (Mr-Mc) and a Multi-Layer Perceptron machine learning model (MLP) were constructed for diagnosing multiple pathological types of pulmonary nodules, respectively. To comprehensively use the double modal information of CT images and LTBs, we used transfer learning to fuse Mr-Mc and MLP, and constructed a multimodal information fusion model that could classify multiple pathological types of benign and malignant pulmonary nodules.

RESULTS

Experiments showed that the constructed Mr-Mc model can achieve an average accuracy of 0.805 and MLP model can achieve an average accuracy of 0.887. The fusion model was verified on a dataset containing 64 samples, and achieved an average accuracy of 0.906.

CONCLUSIONS

This is the first study to simultaneously use CT images and LTBs to diagnose multiple pathological types of benign and malignant pulmonary nodules, and experiments showed that our research was more advanced and more suitable for practical clinical applications.

Computerized Tomography Image Feature under Convolutional Neural Network Algorithm Evaluated for Therapeutic Effect of Clarithromycin Combined with Salmeterol/Fluticasone on Chronic Obstructive Pulmonary Disease

This study was to explore the use of convolutional neural network (CNN) for the classification and recognition of computerized tomography (CT) images of chronic obstructive pulmonary disease (COPD) and the therapeutic effect of clarithromycin combined with salmeterol/fluticasone. First, the clinical data of COPD patients treated in hospital from September 2018 to December 2020 were collected, and CT and X-ray images were also collected. CT-CNN and X ray-CNN single modal models were constructed based on the LeNet-5 model. The randomized fusion algorithm was introduced to construct a fused CNN model for the diagnosis of COPD patients, and the recognition effect of the model was verified. Subsequently, the three-dimensional reconstruction of the patient's bronchus was performed using the classified CT images, and the changes of CT quantitative parameters in COPD patients were compared and analyzed. Finally, COPD patients were treated with salmeterol/fluticasone (COPD-C) and combined with clarithromycin (COPD-T). In addition, the differences between patients' lung function indexes, blood gas indexes, St. George respiratory questionnaire (SGRQ) scores, and the number of acute exacerbations (AECOPD) before and after treatment were evaluated. The results showed that the randomized fusion model under different iteration times and batch sizes always had the highest recognition rate, sensitivity, and specificity compared to the two single modal CNN models, but it also had longer training time. After CT images were used to quantitatively evaluate the changes of the patient's bronchus, it was found that the area of the upper and lower lung lobes of the affected side of COPD patients and the ratio of the area of the tube wall to the bronchus were significantly changed. The lung function, blood gas index, and SGRQ score of COPD-T patients were significantly improved compared with the COPD-C group (P < 0.05), but there was no considerable difference in AECOPD (P > 0.05). In summary, the randomized fusion-based CNN model can improve the recognition rate of COPD, and salmeterol/fluticasone combined with clarithromycin therapy can significantly improve the clinical treatment effect of COPD patients.

DeepPrognosis: Preoperative prediction of pancreatic cancer survival and surgical margin via comprehensive understanding of dynamic contrast-enhanced CT imaging and tumor-vascular contact parsing

Pancreatic ductal adenocarcinoma (PDAC) is one of the most lethal cancers and carries a dismal prognosis of ∼10% in five year survival rate. Surgery remains the best option of a potential cure for patients who are evaluated to be eligible for initial resection of PDAC. However, outcomes vary significantly even among the resected patients who were the same cancer stage and received similar treatments. Accurate quantitative preoperative prediction of primary resectable PDACs for personalized cancer treatment is thus highly desired. Nevertheless, there are a very few automated methods yet to fully exploit the contrast-enhanced computed tomography (CE-CT) imaging for PDAC prognosis assessment. CE-CT plays a critical role in PDAC staging and resectability evaluation. In this work, we propose a novel deep neural network model for the survival prediction of primary resectable PDAC patients, named as 3D Contrast-Enhanced Convolutional Long Short-Term Memory network (CE-ConvLSTM), which can derive the tumor attenuation signatures or patterns from patient CE-CT imaging studies. Tumor-vascular relationships, which might indicate the resection margin status, have also been proven to hold strong relationships with the overall survival of PDAC patients. To capture such relationships, we propose a self-learning approach for automated pancreas and peripancreatic anatomy segmentation without requiring any annotations on our PDAC datasets. We then employ a multi-task convolutional neural network (CNN) to accomplish both tasks of survival outcome and margin prediction where the network benefits from learning the resection margin related image features to improve the survival prediction. Our presented framework can improve overall survival prediction performances compared with existing state-of-the-art survival analysis approaches. The new staging biomarker integrating both the proposed risk signature and margin prediction has evidently added values to be combined with the current clinical staging system.

Weakly supervised deep learning for determining the prognostic value of 18F-FDG PET/CT in extranodal natural killer/T cell lymphoma, nasal type

Purpose

To develop a weakly supervised deep learning (WSDL) method that could utilize incomplete/missing survival data to predict the prognosis of extranodal natural killer/T cell lymphoma, nasal type (ENKTL) based on pretreatment ¹⁸F-FDG PET/CT results.

Methods

One hundred and sixty-seven patients with ENKTL who underwent pretreatment ¹⁸F-FDG PET/CT were retrospectively collected. Eighty-four patients were followed up for at least 2 years (training set = 64, test set = 20). A WSDL method was developed to enable the integration of the remaining 83 patients with incomplete/missing follow-up information in the training set. To test generalization, these data were derived from three types of scanners. Prediction similarity index (PSI) was derived from deep learning features of images. Its discriminative ability was calculated and compared with that of a conventional deep learning (CDL) method. Univariate and multivariate analyses helped explore the significance of PSI and clinical features.

Results

PSI achieved area under the curve scores of 0.9858 and 0.9946 (training set) and 0.8750 and 0.7344 (test set) in the prediction of progression-free survival (PFS) with the WSDL and CDL methods, respectively. PSI threshold of 1.0 could significantly differentiate the prognosis. In the test set, WSDL and CDL achieved prediction sensitivity, specificity, and accuracy of 87.50% and 62.50%, 83.33% and 83.33%, and 85.00% and 75.00%, respectively. Multivariate analysis confirmed PSI to be an independent significant predictor of PFS in both the methods.

Conclusion

The WSDL-based framework was more effective for extracting ¹⁸F-FDG PET/CT features and predicting the prognosis of ENKTL than the CDL method.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00259-021-05232-3.

Clinical correlates for immune checkpoint therapy: significance for CNS malignancies

Immune checkpoint inhibitors (ICIs) have revolutionized the field of cancer immunotherapy. Most commonly, inhibitors of PD-1 and CTLA4 are used having received approval for the treatment of many cancers like melanoma, non-small-cell lung carcinoma, and leukemia. In contrast, to date, clinical studies conducted in patients with CNS malignancies have not demonstrated promising results. However, patients with CNS malignancies have several underlying factors such as treatment with supportive medications like corticosteroids and cancer therapies including radiation and chemotherapy that may negatively impact response to ICIs. Although many clinical trials have been conducted with ICIs, measures that reproducibly and reliably indicate that treatment has evoked an effective immune response have not been fully developed. In this article, we will review the history of ICI therapy and the correlative biology that has been performed in the clinical trials testing these therapies in different cancers. It is our aim to help provide an overview of the assays that may be used to gauge immunologic response. This may be particularly germane for CNS tumors, where there is currently a great need for predictive biomarkers that will allow for the selection of patients with the highest likelihood of responding.

[Application of deep learning in cancer prognosis prediction model]

In recent years, deep learning has provided a new method for cancer prognosis analysis. The literatures related to the application of deep learning in the prognosis of cancer are summarized and their advantages and disadvantages are analyzed, which can be provided for in-depth research. Based on this, this paper systematically reviewed the latest research progress of deep learning in the construction of cancer prognosis model, and made an analysis on the strengths and weaknesses of relevant methods. Firstly, the construction idea and performance evaluation index of deep learning cancer prognosis model were clarified. Secondly, the basic network structure was introduced, and the data type, data amount, and specific network structures and their merits and demerits were discussed. Then, the mainstream method of establishing deep learning cancer prognosis model was verified and the experimental results were analyzed. Finally, the challenges and future research directions in this field were summarized and expected. Compared with the previous models, the deep learning cancer prognosis model can better improve the prognosis prediction ability of cancer patients. In the future, we should continue to explore the research of deep learning in cancer recurrence rate, cancer treatment program and drug efficacy evaluation, and fully explore the application value and potential of deep learning in cancer prognosis model, so as to establish an efficient and accurate cancer prognosis model and realize the goal of precision medicine.

Machine learning-based radiomic, clinical and semantic feature analysis for predicting overall survival and MGMT promoter methylation status in patients with glioblastoma

INTRODUCTION

Survival varies in patients with glioblastoma due to intratumoral heterogeneity and radiomics/imaging biomarkers have potential to demonstrate heterogeneity. The objective was to combine radiomic, semantic and clinical features to improve prediction of overall survival (OS) and O⁶-methylguanine-DNA methyltransferase (MGMT) promoter methylation status from pre-operative MRI in patients with glioblastoma.

METHODS

A retrospective study of 181 MRI studies (mean age 58 ± 13 years, mean OS 497 ± 354 days) performed in patients with histopathology-proven glioblastoma. Tumour mass, contrast-enhancement and necrosis were segmented from volumetric contrast-enhanced T1-weighted imaging (CE-T1WI). 333 radiomic features were extracted and 16 Visually Accessible Rembrandt Images (VASARI) features were evaluated by two experienced neuroradiologists. Top radiomic, VASARI and clinical features were used to build machine learning models to predict MGMT status, and all features including MGMT status were used to build Cox proportional hazards regression (Cox) and random survival forest (RSF) models for OS prediction.

RESULTS

The optimal cut-off value for MGMT promoter methylation index was 12.75%; 42 radiomic features exhibited significant differences between high and low-methylation groups. However, model performance accuracy combining radiomic, VASARI and clinical features for MGMT status prediction varied between 45 and 67%. For OS predication, the RSF model based on clinical, VASARI and CE radiomic features achieved the best performance with an average iAUC of 96.2 ± 1.7 and C-index of 90.0 ± 0.3.

CONCLUSIONS

VASARI features in combination with clinical and radiomic features from the enhancing tumour show promise for predicting OS with a high accuracy in patients with glioblastoma from pre-operative volumetric CE-T1WI.

Multi-Parametric Deep Learning Model for Prediction of Overall Survival after Postoperative Concurrent Chemoradiotherapy in Glioblastoma Patients

This study aimed to investigate the performance of a deep learning-based survival-prediction model, which predicts the overall survival (OS) time of glioblastoma patients who have received surgery followed by concurrent chemoradiotherapy (CCRT). The medical records of glioblastoma patients who had received surgery and CCRT between January 2011 and December 2017 were retrospectively reviewed. Based on our inclusion criteria, 118 patients were selected and semi-randomly allocated to training and test datasets (3:1 ratio, respectively). A convolutional neural network–based deep learning model was trained with magnetic resonance imaging (MRI) data and clinical profiles to predict OS. The MRI was reconstructed by using four pulse sequences (22 slices) and nine images were selected based on the longest slice of glioblastoma by a physician for each pulse sequence. The clinical profiles consist of personal, genetic, and treatment factors. The concordance index (C-index) and integrated area under the curve (iAUC) of the time-dependent area-under-the-curve curves of each model were calculated to evaluate the performance of the survival-prediction models. The model that incorporated clinical and radiomic features showed a higher C-index (0.768 (95% confidence interval (CI): 0.759, 0.776)) and iAUC (0.790 (95% CI: 0.783, 0.797)) than the model using clinical features alone (C-index = 0.693 (95% CI: 0.685, 0.701); iAUC = 0.723 (95% CI: 0.716, 0.731)) and the model using radiomic features alone (C-index = 0.590 (95% CI: 0.579, 0.600); iAUC = 0.614 (95% CI: 0.607, 0.621)). These improvements to the C-indexes and iAUCs were validated using the 1000-times bootstrapping method; all were statistically significant (p < 0.001). This study suggests the synergistic benefits of using both clinical and radiomic parameters. Furthermore, it indicates the potential of multi-parametric deep learning models for the survival prediction of glioblastoma patients.