GMILT: A Novel Transformer Network That Can Noninvasively Predict EGFR Mutation Status

Hybrid deep multi-task learning radiomics approach for predicting EGFR mutation status of non-small cell lung cancer in CT images

Objective.Epidermal growth factor receptor (EGFR) mutation genotyping plays a pivotal role in targeted therapy for non-small cell lung cancer (NSCLC). We aimed to develop a computed tomography (CT) image-based hybrid deep radiomics model to predict EGFR mutation status in NSCLC and investigate the correlations between deep image and quantitative radiomics features.Approach.First, we retrospectively enrolled 818 patients from our centre and 131 patients from The Cancer Imaging Archive database to establish a training cohort (N= 654), an independent internal validation cohort (N= 164) and an external validation cohort (N= 131). Second, to predict EGFR mutation status, we developed three CT image-based models, namely, a multi-task deep neural network (DNN), a radiomics model and a feature fusion model. Third, we proposed a hybrid loss function to train the DNN model. Finally, to evaluate the model performance, we computed the areas under the receiver operating characteristic curves (AUCs) and decision curve analysis curves of the models.Main results.For the two validation cohorts, the feature fusion model achieved AUC values of 0.86 ± 0.03 and 0.80 ± 0.05, which were significantly higher than those of the single-task DNN and radiomics models (allP< 0.05). There was no significant difference between the feature fusion and the multi-task DNN models (P> 0.8). The binary prediction scores showed excellent prognostic value in predicting disease-free survival (P= 0.02) and overall survival (P< 0.005) for validation cohort 2.Significance.The results demonstrate that (1) the feature fusion and multi-task DNN models achieve significantly higher performance than that of the conventional radiomics and single-task DNN models, (2) the feature fusion model can decode the imaging phenotypes representing NSCLC heterogeneity related to both EGFR mutation and patient NSCLC prognosis, and (3) high correlations exist between some deep image and radiomics features.

Deep learning-assisted LI-RADS grading and distinguishing hepatocellular carcinoma (HCC) from non-HCC based on multiphase CT: a two-center study

OBJECTIVES

To develop a deep learning (DL) method that can determine the Liver Imaging Reporting and Data System (LI-RADS) grading of high-risk liver lesions and distinguish hepatocellular carcinoma (HCC) from non-HCC based on multiphase CT.

METHODS

This retrospective study included 1049 patients with 1082 lesions from two independent hospitals that were pathologically confirmed as HCC or non-HCC. All patients underwent a four-phase CT imaging protocol. All lesions were graded (LR 4/5/M) by radiologists and divided into an internal (n = 886) and external cohort (n = 196) based on the examination date. In the internal cohort, Swin-Transformer based on different CT protocols were trained and tested for their ability to LI-RADS grading and distinguish HCC from non-HCC, and then validated in the external cohort. We further developed a combined model with the optimal protocol and clinical information for distinguishing HCC from non-HCC.

RESULTS

In the test and external validation cohorts, the three-phase protocol without pre-contrast showed κ values of 0.6094 and 0.4845 for LI-RADS grading, and its accuracy was 0.8371 and 0.8061, while the accuracy of the radiologist was 0.8596 and 0.8622, respectively. The AUCs in distinguishing HCC from non-HCC were 0.865 and 0.715 in the test and external validation cohorts, while those of the combined model were 0.887 and 0.808.

CONCLUSION

The Swin-Transformer based on three-phase CT protocol without pre-contrast could feasibly simplify LI-RADS grading and distinguish HCC from non-HCC. Furthermore, the DL model have the potential in accurately distinguishing HCC from non-HCC using imaging and highly characteristic clinical data as inputs.

CLINICAL RELEVANCE STATEMENT

The application of deep learning model for multiphase CT has proven to improve the clinical applicability of the Liver Imaging Reporting and Data System and provide support to optimize the management of patients with liver diseases.

KEY POINTS

• Deep learning (DL) simplifies LI-RADS grading and helps distinguish hepatocellular carcinoma (HCC) from non-HCC. • The Swin-Transformer based on the three-phase CT protocol without pre-contrast outperformed other CT protocols. • The Swin-Transformer provide help in distinguishing HCC from non-HCC by using CT and characteristic clinical information as inputs.

Improving diagnosis and prognosis of lung cancer using vision transformers: a scoping review

BACKGROUND

Vision transformer-based methods are advancing the field of medical artificial intelligence and cancer imaging, including lung cancer applications. Recently, many researchers have developed vision transformer-based AI methods for lung cancer diagnosis and prognosis.

OBJECTIVE

This scoping review aims to identify the recent developments on vision transformer-based AI methods for lung cancer imaging applications. It provides key insights into how vision transformers complemented the performance of AI and deep learning methods for lung cancer. Furthermore, the review also identifies the datasets that contributed to advancing the field.

METHODS

In this review, we searched Pubmed, Scopus, IEEEXplore, and Google Scholar online databases. The search terms included intervention terms (vision transformers) and the task (i.e., lung cancer, adenocarcinoma, etc.). Two reviewers independently screened the title and abstract to select relevant studies and performed the data extraction. A third reviewer was consulted to validate the inclusion and exclusion. Finally, the narrative approach was used to synthesize the data.

RESULTS

Of the 314 retrieved studies, this review included 34 studies published from 2020 to 2022. The most commonly addressed task in these studies was the classification of lung cancer types, such as lung squamous cell carcinoma versus lung adenocarcinoma, and identifying benign versus malignant pulmonary nodules. Other applications included survival prediction of lung cancer patients and segmentation of lungs. The studies lacked clear strategies for clinical transformation. SWIN transformer was a popular choice of the researchers; however, many other architectures were also reported where vision transformer was combined with convolutional neural networks or UNet model. Researchers have used the publicly available lung cancer datasets of the lung imaging database consortium and the cancer genome atlas. One study used a cluster of 48 GPUs, while other studies used one, two, or four GPUs.

CONCLUSION

It can be concluded that vision transformer-based models are increasingly in popularity for developing AI methods for lung cancer applications. However, their computational complexity and clinical relevance are important factors to be considered for future research work. This review provides valuable insights for researchers in the field of AI and healthcare to advance the state-of-the-art in lung cancer diagnosis and prognosis. We provide an interactive dashboard on lung-cancer.onrender.com/ .

Deep learning predicts malignancy and metastasis of solid pulmonary nodules from CT scans

In the clinic, it is difficult to distinguish the malignancy and aggressiveness of solid pulmonary nodules (PNs). Incorrect assessments may lead to delayed diagnosis and an increased risk of complications. We developed and validated a deep learning-based model for the prediction of malignancy as well as local or distant metastasis in solid PNs based on CT images of primary lesions during initial diagnosis. In this study, we reviewed the data from multiple patients with solid PNs at our institution from 1 January 2019 to 30 April 2022. The patients were divided into three groups: benign, Ia-stage lung cancer, and T1-stage lung cancer with metastasis. Each cohort was further split into training and testing groups. The deep learning system predicted the malignancy and metastasis status of solid PNs based on CT images, and then we compared the malignancy prediction results among four different levels of clinicians. Experiments confirmed that human-computer collaboration can further enhance diagnostic accuracy. We made a held-out testing set of 134 cases, with 689 cases in total. Our convolutional neural network model reached an area under the ROC (AUC) of 80.37% for malignancy prediction and an AUC of 86.44% for metastasis prediction. In observer studies involving four clinicians, the proposed deep learning method outperformed a junior respiratory clinician and a 5-year respiratory clinician by considerable margins; it was on par with a senior respiratory clinician and was only slightly inferior to a senior radiologist. Our human-computer collaboration experiment showed that by simply adding binary human diagnosis into model prediction probabilities, model AUC scores improved to 81.80-88.70% when combined with three out of four clinicians. In summary, the deep learning method can accurately diagnose the malignancy of solid PNs, improve its performance when collaborating with human experts, predict local or distant metastasis in patients with T1-stage lung cancer, and facilitate the application of precision medicine.

Explainable artificial intelligence (XAI) in radiology and nuclear medicine: a literature review

Rational

Deep learning (DL) has demonstrated a remarkable performance in diagnostic imaging for various diseases and modalities and therefore has a high potential to be used as a clinical tool. However, current practice shows low deployment of these algorithms in clinical practice, because DL algorithms lack transparency and trust due to their underlying black-box mechanism. For successful employment, explainable artificial intelligence (XAI) could be introduced to close the gap between the medical professionals and the DL algorithms. In this literature review, XAI methods available for magnetic resonance (MR), computed tomography (CT), and positron emission tomography (PET) imaging are discussed and future suggestions are made.

Methods

PubMed, Embase.com and Clarivate Analytics/Web of Science Core Collection were screened. Articles were considered eligible for inclusion if XAI was used (and well described) to describe the behavior of a DL model used in MR, CT and PET imaging.

Results

A total of 75 articles were included of which 54 and 17 articles described post and ad hoc XAI methods, respectively, and 4 articles described both XAI methods. Major variations in performance is seen between the methods. Overall, post hoc XAI lacks the ability to provide class-discriminative and target-specific explanation. Ad hoc XAI seems to tackle this because of its intrinsic ability to explain. However, quality control of the XAI methods is rarely applied and therefore systematic comparison between the methods is difficult.

Conclusion

There is currently no clear consensus on how XAI should be deployed in order to close the gap between medical professionals and DL algorithms for clinical implementation. We advocate for systematic technical and clinical quality assessment of XAI methods. Also, to ensure end-to-end unbiased and safe integration of XAI in clinical workflow, (anatomical) data minimization and quality control methods should be included.

Improving the diagnosis of acute ischemic stroke on non-contrast CT using deep learning: a multicenter study

OBJECTIVE

This study aimed to develop a deep learning (DL) model to improve the diagnostic performance of EIC and ASPECTS in acute ischemic stroke (AIS).

METHODS

Acute ischemic stroke patients were retrospectively enrolled from 5 hospitals. We proposed a deep learning model to simultaneously segment the infarct and estimate ASPECTS automatically using baseline CT. The model performance of segmentation and ASPECTS scoring was evaluated using dice similarity coefficient (DSC) and ROC, respectively. Four raters participated in the multi-reader and multicenter (MRMC) experiment to fulfill the region-based ASPECTS reading under the assistance of the model or not. At last, sensitivity, specificity, interpretation time and interrater agreement were used to evaluate the raters' reading performance.

RESULTS

In total, 1391 patients were enrolled for model development and 85 patients for external validation with onset to CT scanning time of 176.4 ± 93.6 min and NIHSS of 5 (IQR 2-10). The model achieved a DSC of 0.600 and 0.762 and an AUC of 0.876 (CI 0.846-0.907) and 0.729 (CI 0.679-0.779), in the internal and external validation set, respectively. The assistance of the DL model improved the raters' average sensitivities and specificities from 0.254 (CI 0.22-0.26) and 0.896 (CI 0.884-0.907), to 0.333 (CI 0.301-0.345) and 0.915 (CI 0.904-0.926), respectively. The average interpretation time of the raters was reduced from 219.0 to 175.7 s (p = 0.035). Meanwhile, the interrater agreement increased from 0.741 to 0.980.

CONCLUSIONS

With the assistance of our proposed DL model, radiologists got better performance in the detection of AIS lesions on NCCT.

Swin Transformer Improves the IDH Mutation Status Prediction of Gliomas Free of MRI-Based Tumor Segmentation

Background: Deep learning (DL) could predict isocitrate dehydrogenase (IDH) mutation status from MRIs. Yet, previous work focused on CNNs with refined tumor segmentation. To bridge the gap, this study aimed to evaluate the feasibility of developing a Transformer-based network to predict the IDH mutation status free of refined tumor segmentation. Methods: A total of 493 glioma patients were recruited from two independent institutions for model development (TCIA; N = 259) and external test (AHXZ; N = 234). IDH mutation status was predicted directly from T2 images with a Swin Transformer and conventional ResNet. Furthermore, to investigate the necessity of refined tumor segmentation, seven strategies for the model input image were explored: (i) whole tumor slice; (ii-iii) tumor mask and/or not edema; (iv-vii) tumor bounding box of 0.8, 1.0, 1.2, 1.5 times. Performance comparison was made among the networks of different architectures along with different image input strategies, using area under the curve (AUC) and accuracy (ACC). Finally, to further boost the performance, a hybrid model was built by incorporating the images with clinical features. Results: With the seven proposed input strategies, seven Swin Transformer models and seven ResNet models were built, respectively. Based on the seven Swin Transformer models, an averaged AUC of 0.965 (internal test) and 0.842 (external test) were achieved, outperforming 0.922 and 0.805 resulting from the seven ResNet models, respectively. When a bounding box of 1.0 times was used, Swin Transformer (AUC = 0.868, ACC = 80.7%), achieved the best results against the one that used tumor segmentation (Tumor + Edema, AUC = 0.862, ACC = 78.5%). The hybrid model that integrated age and location features into images yielded improved performance (AUC = 0.878, Accuracy = 82.0%) over the model that used images only. Conclusions: Swin Transformer outperforms the CNN-based ResNet in IDH prediction. Using bounding box input images benefits the DL networks in IDH prediction and makes the IDH prediction free of refined glioma segmentation feasible.