Comprehensive Computational Pathological Image Analysis Predicts Lung Cancer Prognosis

INTRODUCTION

Pathological examination of histopathological slides is a routine clinical procedure for lung cancer diagnosis and prognosis. Although the classification of lung cancer has been updated to become more specific, only a small subset of the total morphological features are taken into consideration. The vast majority of the detailed morphological features of tumor tissues, particularly tumor cells' surrounding microenvironment, are not fully analyzed. The heterogeneity of tumor cells and close interactions between tumor cells and their microenvironments are closely related to tumor development and progression. The goal of this study is to develop morphological feature-based prediction models for the prognosis of patients with lung cancer.

METHOD

We developed objective and quantitative computational approaches to analyze the morphological features of pathological images for patients with NSCLC. Tissue pathological images were analyzed for 523 patients with adenocarcinoma (ADC) and 511 patients with squamous cell carcinoma (SCC) from The Cancer Genome Atlas lung cancer cohorts. The features extracted from the pathological images were used to develop statistical models that predict patients' survival outcomes in ADC and SCC, respectively.

RESULTS

We extracted 943 morphological features from pathological images of hematoxylin and eosin-stained tissue and identified morphological features that are significantly associated with prognosis in ADC and SCC, respectively. Statistical models based on these extracted features stratified NSCLC patients into high-risk and low-risk groups. The models were developed from training sets and validated in independent testing sets: a predicted high-risk group versus a predicted low-risk group (for patients with ADC: hazard ratio = 2.34, 95% confidence interval: 1.12-4.91, p = 0.024; for patients with SCC: hazard ratio = 2.22, 95% confidence interval: 1.15-4.27, p = 0.017) after adjustment for age, sex, smoking status, and pathologic tumor stage.

CONCLUSIONS

The results suggest that the quantitative morphological features of tumor pathological images predict prognosis in patients with lung cancer.

Integrating genomic data and pathological images to effectively predict breast cancer clinical outcome

BACKGROUND AND OBJECTIVE

Breast cancer is a leading cause of death from cancer for females. The high mortality rate of breast cancer is largely due to the complexity among invasive breast cancer and its significantly varied clinical outcomes. Therefore, improving the accuracy of breast cancer survival prediction has important significance and becomes one of the major research areas. Nowadays many computational models have been proposed for breast cancer survival prediction, however, most of them generate the predictive models by employing only the genomic data information and few of them consider the complementary information from pathological images.

METHODS

In our study, we introduce a novel method called GPMKL based on multiple kernel learning (MKL), which efficiently employs heterogeneous information containing genomic data (gene expression, copy number alteration, gene methylation, protein expression) and pathological images. With above heterogeneous features, GPMKL is proposed to execute feature fusion which is embedded in breast cancer classification.

RESULTS

Performance analysis of the GPMKL model indicates that the pathological image information plays a critical part in accurately predicting the survival time of breast cancer patients. Furthermore, the proposed method is compared with other existing breast cancer survival prediction methods, and the results demonstrate that the proposed framework with pathological images performs remarkably better than the existing survival prediction methods.

CONCLUSIONS

All results performed in our study suggest that the usefulness and superiority of GPMKL in predicting human breast cancer survival.

Richer fusion network for breast cancer classification based on multimodal data

Background

Deep learning algorithms significantly improve the accuracy of pathological image classification, but the accuracy of breast cancer classification using only single-mode pathological images still cannot meet the needs of clinical practice. Inspired by the real scenario of pathologists reading pathological images for diagnosis, we integrate pathological images and structured data extracted from clinical electronic medical record (EMR) to further improve the accuracy of breast cancer classification.

Methods

In this paper, we propose a new richer fusion network for the classification of benign and malignant breast cancer based on multimodal data. To make pathological image can be integrated more sufficient with structured EMR data, we proposed a method to extract richer multilevel feature representation of the pathological image from multiple convolutional layers. Meanwhile, to minimize the information loss for each modality before data fusion, we use the denoising autoencoder as a way to increase the low-dimensional structured EMR data to high-dimensional, instead of reducing the high-dimensional image data to low-dimensional before data fusion. In addition, denoising autoencoder naturally generalizes our method to make the accurate prediction with partially missing structured EMR data.

Results

The experimental results show that the proposed method is superior to the most advanced method in terms of the average classification accuracy (92.9%). In addition, we have released a dataset containing structured data from 185 patients that were extracted from EMR and 3764 paired pathological images of breast cancer, which can be publicly downloaded from http://ear.ict.ac.cn/?page_id=1663.

Conclusions

We utilized a new richer fusion network to integrate highly heterogeneous data to leverage the structured EMR data to improve the accuracy of pathological image classification. Therefore, the application of automatic breast cancer classification algorithms in clinical practice becomes possible. Due to the generality of the proposed fusion method, it can be straightforwardly extended to the fusion of other structured data and unstructured data.

Deformable image registration between pathological images and MR image via an optical macro image

Computed tomography (CT) and magnetic resonance (MR) imaging have been widely used for visualizing the inside of the human body. However, in many cases, pathological diagnosis is conducted through a biopsy or resection of an organ to evaluate the condition of tissues as definitive diagnosis. To provide more advanced information onto CT or MR image, it is necessary to reveal the relationship between tissue information and image signals. We propose a registration scheme for a set of PT images of divided specimens and a 3D-MR image by reference to an optical macro image (OM image) captured by an optical camera. We conducted a fundamental study using a resected human brain after the death of a brain cancer patient. We constructed two kinds of registration processes using the OM image as the base for both registrations to make conversion parameters between the PT and MR images. The aligned PT images had shapes similar to the OM image. On the other hand, the extracted cross-sectional MR image was similar to the OM image. From these resultant conversion parameters, the corresponding region on the PT image could be searched and displayed when an arbitrary pixel on the MR image was selected. The relationship between the PT and MR images of the whole brain can be analyzed using the proposed method. We confirmed that same regions between the PT and MR images could be searched and displayed using resultant information obtained by the proposed method. In terms of the accuracy of proposed method, the TREs were 0.56±0.39mm and 0.87±0.42mm. We can analyze the relationship between tissue information and MR signals using the proposed method.

Fast detection method for prostate cancer cells based on an integrated ResNet50 and YoloV5 framework

PURPOSE

To propose a fast detection method for prostate cancer abnormal cells based on deep learning. The purpose of this method is to quickly and accurately locate and identify abnormal cells, so as to improve the efficiency of prostate precancerous screening and promote the application and popularization of prostate cancer cell assisted screening technology.

METHOD

The method includes two stages: preliminary screening of abnormal cell images and accurate identification of abnormal cells. In the preliminary screening stage of abnormal cell images, ResNet50 model is used as the image classification network to judge whether the local area contains cell clusters. In the another stage, YoloV5 model is used as the target detection network to locate and recognize abnormal cells in the image containing cell clusters.

RESULTS

This detection method aims at the pathological cell images obtained by the membrane method. And the double stage models proposed in this paper are compared with the single stage model method using only the target detection model. The results show that through the image classification network based on deep learning, we can first judge whether there are abnormal cells in the local area. If there are abnormal cells, we can further use the target detection method based on candidate box for analysis, which can reduce the reasoning time by 50% and improve the efficiency of abnormal cell detection under the condition of losing a small amount of accuracy and slightly increasing the complexity of the model.

CONCLUSION

This study proposes a fast detection method for prostate cancer abnormal cells based on deep learning, which can greatly shorten the reasoning time and improve the detection speed. It is able to improve the efficiency of prostate precancerous screening.

Colorectal Cancer Diagnostic Algorithm Based on Sub-Patch Weight Color Histogram in Combination of Improved Least Squares Support Vector Machine for Pathological Image

In order to improve the diagnostic accuracy of colon cancer, a novel classification algorithm based on sub-patch weight color histogram and improved SVM is proposed, which has good approximation ability for complex pathological image. Our proposed algorithm combines wavelet kernel SVM with color histogram to classify pathological image. Firstly, the pathological image is divided into non-overlapping sub-patches, and the features of sub-patch histogram are extracted. Then, the global and local features are fused by the sub-patch weighting algorithm. Then, the RelicfF based forward selection algorithm is used to integrate color features and texture features so as to enhance the characterization capabilities of the tumor cell. Finally, Morlet wavelet kernel-based least squares support vector machine method is adopted to enhance the generalization ability of the model for small sample with non-linear and high-dimensional pattern classification problems. Experimental results show that the proposed pathological diagnostic algorithm can gain higher accuracy compared with existing comparison algorithms.

LSCDFS-MKL: A multiple kernel based method for lung squamous cell carcinomas disease-free survival prediction with pathological and genomic data

Lung squamous cell carcinoma (SCC) is a fatal disease in both male and female, for which current treatments are inadequate. Surgical resection is regarded as the cornerstone of treatment for patients with lung SCC, but even for the same stage patients, the wide spectrum of disease-free survival (DFS) times exits. Therefore, how to improve the DFS prediction performance of lung SCC becomes one major research area. In this study, we proposed a novel method called LSCDFS-MKL, which was on the basis of multiple kernel learning to predict DFS of lung SCC. In LSCDFS-MKL, we first efficiently integrated pathological images and genomic data (copy number aberration, gene expression, protein expression) from lung SCC. The results of LSCDFS-MKL between different types of data show that the features extracted from pathological images play an important role in DFS prediction of lung SCC. Then we compared our method LSCDFS-MKL with other existing methods and performance analysis indicates that LSCDFS-MKL has a significantly better performance than other prediction methods. After that, we applied the proposed method on different stage stratums and the performance demonstrates that LSCDFS-MKL remains efficient in DFS prediction of lung SCC patients. Finally, we performed LSCDFS-MKL on an independent validation dataset and the accuracy of DFS prediction achieves 100%, which is promising.

Matrix metalloproteinase 9 expression and glioblastoma survival prediction using machine learning on digital pathological images

This study aimed to apply pathomics to predict Matrix metalloproteinase 9 (MMP9) expression in glioblastoma (GBM) and investigate the underlying molecular mechanisms associated with pathomics. Here, we included 127 GBM patients, 78 of whom were randomly allocated to the training and test cohorts for pathomics modeling. The prognostic significance of MMP9 was assessed using Kaplan-Meier and Cox regression analyses. PyRadiomics was used to extract the features of H&E-stained whole slide images. Feature selection was performed using the maximum relevance and minimum redundancy (mRMR) and recursive feature elimination (RFE) algorithms. Prediction models were created using support vector machines (SVM) and logistic regression (LR). The performance was assessed using ROC analysis, calibration curve assessment, and decision curve analysis. MMP9 expression was elevated in patients with GBM. This was an independent prognostic factor for GBM. Six features were selected for the pathomics model. The area under the curves (AUCs) of the training and test subsets were 0.828 and 0.808, respectively, for the SVM model and 0.778 and 0.754, respectively, for the LR model. The C-index and calibration plots exhibited effective estimation abilities. The pathomics score calculated using the SVM model was highly correlated with overall survival time. These findings indicate that MMP9 plays a crucial role in GBM development and prognosis. Our pathomics model demonstrated high efficacy for predicting MMP9 expression levels and prognosis of patients with GBM.

PathoDuet: Foundation models for pathological slide analysis of H&E and IHC stains

Large amounts of digitized histopathological data display a promising future for developing pathological foundation models via self-supervised learning methods. Foundation models pretrained with these methods serve as a good basis for downstream tasks. However, the gap between natural and histopathological images hinders the direct application of existing methods. In this work, we present PathoDuet, a series of pretrained models on histopathological images, and a new self-supervised learning framework in histopathology. The framework is featured by a newly-introduced pretext token and later task raisers to explicitly utilize certain relations between images, like multiple magnifications and multiple stains. Based on this, two pretext tasks, cross-scale positioning and cross-stain transferring, are designed to pretrain the model on Hematoxylin and Eosin (H&E) images and transfer the model to immunohistochemistry (IHC) images, respectively. To validate the efficacy of our models, we evaluate the performance over a wide variety of downstream tasks, including patch-level colorectal cancer subtyping and whole slide image (WSI)-level classification in H&E field, together with expression level prediction of IHC marker, tumor identification and slide-level qualitative analysis in IHC field. The experimental results show the superiority of our models over most tasks and the efficacy of proposed pretext tasks. The codes and models are available at https://github.com/openmedlab/PathoDuet.

Tumor-identification method for predicting recurrence of early-stage lung adenocarcinoma using digital pathology images by machine learning

Lung cancer is one of the cancers with the highest morbidity and mortality in the world. Recurrence often occurs even after complete resection of early-stage lung cancer, and prediction of recurrence after resection is clinically important. However, the pathological characteristics of the recurrence of pathological stage IB lung adenocarcinoma (LAIB) have not yet been elucidated. Therefore, the problem is what type of histological image of lung adenocarcinoma recurs, and it is important to examine the histological image of recurrence. We attempted to predict recurrence of early lung adenocarcinoma after resection on the basis of digital pathological images of hematoxylin and eosin-stained specimens and machine learning applying a convolutional neural network. We constructed a model that extracts the features of two-color spaces and a switching model that automatically switches between our extraction model and one that extracts the features of one-color space for each image. We then developed a tumor-identification method for predicting the presence or absence of LAIB recurrence using these models. We conducted an experiment involving 55 patients with LAIB who underwent surgical resection to evaluate the proposed method. The proposed method determined LAIB recurrence with an accuracy of 84.8%. The use of digital pathology and machine learning can be used for highly accurate prediction of LAIB recurrence after surgical resection. The proposed method has the potential for objective postoperative follow-up observation.

Prediction of cardiovascular events after carotid endarterectomy using pathological images and clinical data

PURPOSE

Carotid endarterectomy (CEA) is a surgical treatment for carotid artery stenosis. After CEA, some patients experience cardiovascular events (myocardial infarction, stroke, etc.); however, the prognostic factor has yet to be revealed. Therefore, this study explores the predictive factors in pathological images and predicts cardiovascular events within one year after CEA using pathological images of carotid plaques and patients' clinical data.

METHOD

This paper proposes a two-step method to predict the prognosis of CEA patients. The proposed method first computes the pathological risk score using an anomaly detection model trained using pathological images of patients without cardiovascular events. By concatenating the obtained image-based risk score with a patient's clinical data, a statistical machine learning-based classifier predicts the patient's prognosis.

RESULTS

We evaluate the proposed method on a dataset containing 120 patients without cardiovascular events and 21 patients with events. The combination of autoencoder as the anomaly detection model and XGBoost as the classification model obtained the best results: area under the receiver operating characteristic curve, accuracy, sensitivity, specificity, and F1-score were 81.9%, 84.1%, 79.1%, 86.3%, and 76.6%, respectively. These values were superior to those obtained using pathological images or clinical data alone.

CONCLUSION

We showed the feasibility of predicting CEA patient's long-term prognosis using pathological images and clinical data. Our results revealed some histopathological features related to cardiovascular events: plaque hemorrhage (thrombus), lymphocytic infiltration, and hemosiderin deposition, which will contribute to developing preventive treatment methods for plaque development and progression.

Prognostic and predictive value of pathohistological features in gastric cancer and identification of SLITRK4 as a potential biomarker for gastric cancer

The aim of this study was to develop a quantitative feature-based model from histopathologic images to assess the prognosis of patients with gastric cancer. Whole slide image (WSI) images of H&E-stained histologic specimens of gastric cancer patients from The Cancer Genome Atlas were included and randomly assigned to training and test groups in a 7:3 ratio. A systematic preprocessing approach was employed as well as a non-overlapping segmentation method that combined patch-level prediction with a multi-instance learning approach to integrate features across the slide images. Subjects were categorized into high- or low-risk groups based on the median risk score derived from the model, and the significance of this stratification was assessed using a log-rank test. In addition, combining transcriptomic data from patients and data from other large cohort studies, we further searched for genes associated with pathological features and their prognostic value. A total of 165 gastric cancer patients were included for model training, and a total of 26 features were integrated through multi-instance learning, with each process generating 11 probabilistic features and 2 predictive labeling features. We applied a 10-fold Lasso-Cox regression model to achieve dimensionality reduction of these features. The predictive accuracy of the model was verified using Kaplan-Meyer (KM) curves for stratification with a consistency index of 0.741 for the training set and 0.585 for the test set. Deep learning-based resultant supervised pathohistological features have the potential for superior prognostic stratification of gastric cancer patients, transforming image pixels into an effective and labor-saving tool to optimize the clinical management of gastric cancer patients. Also, SLITRK4 was identified as a prognostic marker for gastric cancer.

Multi-scale feature fusion for prediction of IDH1 mutations in glioma histopathological images

BACKGROUND AND OBJECTIVE

Mutations in isocitrate dehydrogenase 1 (IDH1) play a crucial role in the prognosis, diagnosis, and treatment of gliomas. However, current methods for determining its mutation status, such as immunohistochemistry and gene sequencing, are difficult to implement widely in routine clinical diagnosis. Recent studies have shown that using deep learning methods based on pathological images of glioma can predict the mutation status of the IDH1 gene. However, our research focuses on utilizing multi-scale information in pathological images to improve the accuracy of predicting IDH1 gene mutations, thereby providing an accurate and cost-effective prediction method for routine clinical diagnosis.

METHODS

In this paper, we propose a multi-scale fusion gene identification network (MultiGeneNet). The network first uses two feature extractors to obtain feature maps at different scale images, and then by employing a bilinear pooling layer based on Hadamard product to realize the fusion of multi-scale features. Through fully exploiting the complementarity among features at different scales, we are able to obtain a more comprehensive and rich representation of multi-scale features.

RESULTS

Based on the Hematoxylin and Eosin stained pathological section dataset of 296 patients, our method achieved an accuracy of 83.575 % and an AUC of 0.886, thus significantly outperforming other single-scale methods.

CONCLUSIONS

Our method can be deployed in medical aid systems at very low cost, serving as a diagnostic or prognostic tool for glioma patients in medically underserved areas.

Cancer prediction from few amounts of histology samples through self-attention based multi-routines cross-domains network

OBJECTIVE

Rapid and efficient analysis of cancer has become a focus of research. Artificial intelligence can use histopathological data to quickly determine the cancer situation, but still faces challenges. For example, the convolutional network is limited by the local receptive field, human histopathological information is precious and difficult to be collected in large quantities, and cross-domain data is hard to be used to learn histopathological features. In order to alleviate the above questions, we design a novel network, SMC-Net.
Approach: Feature analysis module and decoupling analysis module designed are the core of the SMC-Net. The feature analysis module base on multi-subspace self-attention mechanism with pathological feature channel embedding. It in charge of learning the interdependence between pathological features to alleviate the problem that the classical convolution model is difficult to learn the impact of joint features on pathological examination results. The decoupling analysis module base on the designed multi-channel and multi-discriminator architecture. Its function is to decouple the features related to the target task in cross-domain samples so that the model has cross-domain learning ability.
Main results: To evaluate the performance of the model more objectively, three datasets are used. Compared with other popular methods, our model achieves better performance without performance imbalance. In this work, a novel network is design. It can use domain-independent data to assist in the learning of target tasks, and can achieve acceptable histopathological diagnosis results even in the absence of data.
Significance: The proposed method has higher clinical embedding potential and provides a viewpoint for the combination of deep learning and histopathological examination.&#xD.

A semi-supervised learning framework for micropapillary adenocarcinoma detection

PURPOSE

Micropapillary adenocarcinoma is a distinctive histological subtype of lung adenocarcinoma with poor prognosis. Computer-aided diagnosis method has the potential to provide help for its early diagnosis. But the implementation of the existing methods largely relies on massive manually labeled data and consumes a lot of time and energy. To tackle these problems, we propose a framework that applies semi-supervised learning method to detect micropapillary adenocarcinoma, which aims to utilize labeled and unlabeled data better.

METHODS

The framework consists of a teacher model and a student model. The teacher model is first obtained by using the labeled data. Then, it makes predictions on unlabeled data as pseudo-labels for students. Finally, high-quality pseudo-labels are selected and associated with the labeled data to train the student model. During the learning process of the student model, augmentation is added so that the student model generalizes better than the teacher model.

RESULTS

Experiments are conducted on our own whole slide micropapillary lung adenocarcinoma histopathology image dataset and we selected 3527 patches for the experiment. In the supervised learning, our detector achieves a precision of 0.762 and recall of 0.884. In the semi-supervised learning, our method achieves a precision of 0.775 and recall of 0.896; it is superior to other methods.

CONCLUSION

We proposed a semi-supervised learning framework for micropapillary adenocarcinoma detection, which has better performance in utilizing both labeled and unlabeled data. In addition, the detector we designed improves the detection accuracy and speed and achieves promising results in detecting micropapillary adenocarcinoma.