New breast cancer prognostic factors identified by computer-aided image analysis of HE stained histopathology images

Colorectal cancer histopathology image analysis: A comparative study of prognostic values of automatically extracted morphometric nuclear features in multispectral and red-blue-green imagery

OBJECTIVES

Multispectral imaging (MSI) has been utilized to predict the prognosis of colorectal cancer (CRC) patients, however, our understanding of the prognostic value of nuclear morphological parameters of bright-field MSI in CRC is still limited. This study was designed to compare the efficiency of MSI and standard red-green-blue (RGB) images in predicting the prognosis of CRC.

METHODS

We compared the efficiency of MS and conventional RGB images on the quantitative assessment of hematoxylin-eosin (HE) stained histopathology images. A pipeline was developed using a pixel-wise support vector machine (SVM) classifier for gland-stroma segmentation, and a marker-controlled watershed algorithm was used for nuclei segmentation. The correlation between extracted morphological parameters and the five-year disease-free survival (5-DFS) was analyzed.

RESULTS

Forty-seven nuclear morphological parameters were extracted in total. Based on Kaplan-Meier analysis, eight features derived from MS images and seven featured derived from RGB images were significantly associated with 5-DFS, respectively. Compared with RGB images, MSI showed higher accuracy, precision, and Dice index in nuclei segmentation. Multivariate analysis indicated that both integrated parameters 1 (factors negatively correlated with CRC prognosis including nuclear number, circularity, eccentricity, major axis length) and 2 (factors positively correlated with CRC prognosis including nuclear average area, area perimeter, total area/total perimeter ratio, average area/perimeter ratio) in MS images were independent prognostic factors of 5-DFS, in contrast with only integrated parameter 1 (P<0.001) in RGB images. More importantly, the quantification of HE-stained MS images displayed higher accuracy in predicting 5-DFS compared with RGB images (76.9% vs 70.9%).

CONCLUSIONS

Quantitative evaluation of HE-stained MS images could yield more information and better predictive performance for CRC prognosis than conventional RGB images, thereby contributing to precision oncology.

From Pixels to Prognosis: A Survey on AI-Driven Cancer Patient Survival Prediction Using Digital Histology Images

Survival analysis is an integral part of medical statistics that is extensively utilized to establish prognostic indices for mortality or disease recurrence, assess treatment efficacy, and tailor effective treatment plans. The identification of prognostic biomarkers capable of predicting patient survival is a primary objective in the field of cancer research. With the recent integration of digital histology images into routine clinical practice, a plethora of Artificial Intelligence (AI)-based methods for digital pathology has emerged in scholarly literature, facilitating patient survival prediction. These methods have demonstrated remarkable proficiency in analyzing and interpreting whole slide images, yielding results comparable to those of expert pathologists. The complexity of AI-driven techniques is magnified by the distinctive characteristics of digital histology images, including their gigapixel size and diverse tissue appearances. Consequently, advanced patch-based methods are employed to effectively extract features that correlate with patient survival. These computational methods significantly enhance survival prediction accuracy and augment prognostic capabilities in cancer patients. The review discusses the methodologies employed in the literature, their performance metrics, ongoing challenges, and potential solutions for future advancements. This paper explains survival analysis and feature extraction methods for analyzing cancer patients. It also compiles essential acronyms related to cancer precision medicine. Furthermore, it is noteworthy that this is the inaugural review paper in the field. The target audience for this interdisciplinary review comprises AI practitioners, medical statisticians, and progressive oncologists who are enthusiastic about translating AI-driven solutions into clinical practice. We expect this comprehensive review article to guide future research directions in the field of cancer research.

Integrative models of histopathological images and multi-omics data predict prognosis in endometrial carcinoma

Objective

This study aimed to predict the molecular features of endometrial carcinoma (EC) and the overall survival (OS) of EC patients using histopathological imaging.

Methods

The patients from The Cancer Genome Atlas (TCGA) were separated into the training set (n = 215) and test set (n = 214) in proportion of 1:1. By analyzing quantitative histological image features and setting up random forest model verified by cross-validation, we constructed prognostic models for OS. The model performance is evaluated with the time-dependent receiver operating characteristics (AUC) over the test set.

Results

Prognostic models based on histopathological imaging features (HIF) predicted OS in the test set (5-year AUC = 0.803). The performance of combining histopathology and omics transcends that of genomics, transcriptomics, or proteomics alone. Additionally, multi-dimensional omics data, including HIF, genomics, transcriptomics, and proteomics, attained the largest AUCs of 0.866, 0.869, and 0.856 at years 1, 3, and 5, respectively, showcasing the highest discrepancy in survival (HR = 18.347, 95% CI [11.09-25.65], p < 0.001).

Conclusions

The results of this experiment indicated that the complementary features of HIF could improve the prognostic performance of EC patients. Moreover, the integration of HIF and multi-dimensional omics data might ameliorate survival prediction and risk stratification in clinical practice.

Deep learning-based morphological feature analysis and the prognostic association study in colon adenocarcinoma histopathological images

Colorectal cancer (CRC) is now the third most common malignancy to cause mortality worldwide, and its prognosis is of great importance. Recent CRC prognostic prediction studies mainly focused on biomarkers, radiometric images, and end-to-end deep learning methods, while only a few works paid attention to exploring the relationship between the quantitative morphological features of patients' tissue slides and their prognosis. However, existing few works in this area suffered from the drawback of choosing the cells randomly from the whole slides, which contain the non-tumor region that lakes information about prognosis. In addition, the existing works, which tried to demonstrate their biological interpretability using patients' transcriptome data, failed to show the biological meaning closely related to cancer. In this study, we proposed and evaluated a prognostic model using morphological features of cells in the tumor region. The features were first extracted by the software CellProfiler from the tumor region selected by Eff-Unet deep learning model. Features from different regions were then averaged for each patient as their representative, and the Lasso-Cox model was used to select the prognosis-related features. The prognostic prediction model was at last constructed using the selected prognosis-related features and was evaluated through KM estimate and cross-validation. In terms of biological meaning, Gene Ontology (GO) enrichment analysis of the expressed genes that correlated with the prognostically significant features was performed to show the biological interpretability of our model.With the help of tumor segmentation, our model achieved better statistical significance and better biological interpretability compared to the results without tumor segmentation. Statistically, the Kaplan Meier (KM) estimate of our model showed that the model using features in the tumor region has a higher C-index, a lower p-value, and a better performance on cross-validation than the model without tumor segmentation. In addition, revealing the pathway of the immune escape and the spread of the tumor, the model with tumor segmentation demonstrated a biological meaning much more related to cancer immunobiology than the model without tumor segmentation. Our prognostic prediction model using quantitive morphological features from tumor regions was almost as good as the TNM tumor staging system as they had a close C-index, and our model can be combined with the TNM tumor stage system to make a better prognostic prediction. And to the best of our knowledge, the biological mechanisms in our study were the most relevant to the immune mechanism of cancer compared to the previous studies.

Artificial intelligence in breast cancer histopathology

This is a review on the use of artificial intelligence for digital breast pathology. A systematic search on PubMed was conducted, identifying 17,324 research papers related to breast cancer pathology. Following a semimanual screening, 664 papers were retrieved and pursued. The papers are grouped into six major tasks performed by pathologists-namely, molecular and hormonal analysis, grading, mitotic figure counting, ki-67 indexing, tumour-infiltrating lymphocyte assessment, and lymph node metastases identification. Under each task, open-source datasets for research to build artificial intelligence (AI) tools are also listed. Many AI tools showed promise and demonstrated feasibility in the automation of routine pathology investigations. We expect continued growth of AI in this field as new algorithms mature.

Artificial intelligence techniques for neuropathological diagnostics and research

Artificial intelligence (AI) research began in theoretical neurophysiology, and the resulting classical paper on the McCulloch-Pitts mathematical neuron was written in a psychiatry department almost 80 years ago. However, the application of AI in digital neuropathology is still in its infancy. Rapid progress is now being made, which prompted this article. Human brain diseases represent distinct system states that fall outside the normal spectrum. Many differ not only in functional but also in structural terms, and the morphology of abnormal nervous tissue forms the traditional basis of neuropathological disease classifications. However, only a few countries have the medical specialty of neuropathology, and, given the sheer number of newly developed histological tools that can be applied to the study of brain diseases, a tremendous shortage of qualified hands and eyes at the microscope is obvious. Similarly, in neuroanatomy, human observers no longer have the capacity to process the vast amounts of connectomics data. Therefore, it is reasonable to assume that advances in AI technology and, especially, whole-slide image (WSI) analysis will greatly aid neuropathological practice. In this paper, we discuss machine learning (ML) techniques that are important for understanding WSI analysis, such as traditional ML and deep learning, introduce a recently developed neuropathological AI termed PathoFusion, and present thoughts on some of the challenges that must be overcome before the full potential of AI in digital neuropathology can be realized.

Improved Image Analysis for Measuring Gastric Ulcer Index in Animal Models and Clinical Diagnostic Data

Gastric ulcers are one of the most common gastrointestinal diseases. In this study, as an attempt to reduce the minimal error in clinical observations during the diagnosis of gastric ulcers, the applicability of improved ImageJ analysis (IA) was investigated by comparing the results of animal experiments and clinical data. As a result, IA exhibited a significantly improved potential for determining the ulcer index (UI) of clinical data sheets compared to those rated directly by conventional clinical observation (CCO). This indicated that IA enhanced the reproducibility of the measurement of gastric UI using a Bland-Altman plot, resulting in a reduced deviation of each UI value. In addition, it was confirmed that errors in gastric UI decisions can be reduced by adjusting RGB values in diagnostic clinical data (i.e., adjusting to 100 is relatively better than adjusting to 50 or 200). Together, these results suggest that the new enhanced IA could be compatible with novel applications for measuring and evaluating gastric ulcers in clinical settings, meaning that the developed method could be used not only as an auxiliary tool for CCO, but also as a pipeline for ulcer diagnosis.

Integration of histopathological images and multi-dimensional omics analyses predicts molecular features and prognosis in high-grade serous ovarian cancer

OBJECTIVE

This study used histopathological image features to predict molecular features, and combined with multi-dimensional omics data to predict overall survival (OS) in high-grade serous ovarian cancer (HGSOC).

METHODS

Patients from The Cancer Genome Atlas (TCGA) were distributed into training set (n = 115) and test set (n = 114). In addition, we collected tissue microarrays of 92 patients as an external validation set. Quantitative features were extracted from histopathological images using CellProfiler, and utilized to establish prediction models by machine learning methods in training set. The prediction performance was assessed in test set and validation set.

RESULTS

The prediction models were able to identify BRCA1 mutation (AUC = 0.952), BRCA2 mutation (AUC = 0.912), microsatellite instability-high (AUC = 0.919), microsatellite stable (AUC = 0.924), and molecular subtypes: proliferative (AUC = 0.961), differentiated (AUC = 0.952), immunoreactive (AUC = 0.941), mesenchymal (AUC = 0.918) in test set. The prognostic model based on histopathological image features could predict OS in test set (5-year AUC = 0.825) and validation set (5-year AUC = 0.703). We next explored the integrative prognostic models of image features, genomics, transcriptomics and proteomics. In test set, the models combining two omics had higher prediction accuracy, such as image features and genomics (5-year AUC = 0.834). The multi-omics model including all features showed the best prediction performance (5-year AUC = 0.911). According to risk score of multi-omics model, the high-risk and low-risk groups had significant survival differences (HR = 18.23, p < 0.001).

CONCLUSIONS

These results indicated the potential ability of histopathological image features to predict above molecular features and survival risk of HGSOC patients. The integration of image features and multi-omics data may improve prognosis prediction in HGSOC patients.

A review and comparison of breast tumor cell nuclei segmentation performances using deep convolutional neural networks

Breast cancer is currently the second most common cause of cancer-related death in women. Presently, the clinical benchmark in cancer diagnosis is tissue biopsy examination. However, the manual process of histopathological analysis is laborious, time-consuming, and limited by the quality of the specimen and the experience of the pathologist. This study's objective was to determine if deep convolutional neural networks can be trained, with transfer learning, on a set of histopathological images independent of breast tissue to segment tumor nuclei of the breast. Various deep convolutional neural networks were evaluated for the study, including U-Net, Mask R-CNN, and a novel network (GB U-Net). The networks were trained on a set of Hematoxylin and Eosin (H&E)-stained images of eight diverse types of tissues. GB U-Net demonstrated superior performance in segmenting sites of invasive diseases (AJI = 0.53, mAP = 0.39 & AJI = 0.54, mAP = 0.38), validated on two hold-out datasets exclusively containing breast tissue images of approximately 7,582 annotated cells. The results of the networks, trained on images independent of breast tissue, demonstrated that tumor nuclei of the breast could be accurately segmented.

Integrative Survival Analysis of Breast Cancer with Gene Expression and DNA Methylation Data

Motivation

Integrative multi-feature fusion analysis on biomedical data has gained much attention recently. In breast cancer, existing studies have demonstrated that combining genomic mRNA data and DNA methylation data can better stratify cancer patients with distinct prognosis than using single signature. However, those existing methods are simply combining these gene features in series and have ignored the correlations between separate omics dimensions over time.

Results

In the present study, we propose an adaptive multi-task learning method, which combines the Cox loss task with the ordinal loss task, for survival prediction of breast cancer patients using multi-modal learning instead of performing survival analysis on each feature dataset. First, we use local maximum quasi-clique merging (lmQCM) algorithm to reduce the mRNA and methylation feature dimensions and extract cluster eigengenes respectively. Then, we add an auxiliary ordinal loss to the original Cox model to improve the ability to optimize the learning process in training and regularization. The auxiliary loss helps to reduce the vanishing gradient problem for earlier layers and helps to decrease the loss of the primary task. Meanwhile, we use an adaptive weights approach to multi-task learning which weighs multiple loss functions by considering the homoscedastic uncertainty of each task. Finally, we build an ordinal cox hazards model for survival analysis and use long short-term memory (LSTM) method to predict patients’ survival risk. We use the cross-validation method and the concordance index (C-index) for assessing the prediction effect. Stringent cross-verification testing processes for the benchmark dataset and two additional datasets demonstrate that the developed approach is effective, achieving very competitive performance with existing approaches.

Availability and implementation

https://github.com/bhioswego/ML_ordCOX.

Machine Learning Methods for Histopathological Image Analysis: A Review

Histopathological images (HIs) are the gold standard for evaluating some types of tumors for cancer diagnosis. The analysis of such images is time and resource-consuming and very challenging even for experienced pathologists, resulting in inter-observer and intra-observer disagreements. One of the ways of accelerating such an analysis is to use computer-aided diagnosis (CAD) systems. This paper presents a review on machine learning methods for histopathological image analysis, including shallow and deep learning methods. We also cover the most common tasks in HI analysis, such as segmentation and feature extraction. Besides, we present a list of publicly available and private datasets that have been used in HI research.

Analysis of tumor nuclear features using artificial intelligence to predict response to neoadjuvant chemotherapy in high-risk breast cancer patients

PURPOSE

Neoadjuvant chemotherapy (NAC) is used to treat patients with high-risk breast cancer. The tumor response to NAC can be classified as either a pathological partial response (pPR) or pathological complete response (pCR), defined as complete eradication of invasive tumor cells, with a pCR conferring a significantly lower risk of recurrence. Predicting the response to NAC, however, remains a significant clinical challenge. The objective of this study was to determine if analysis of nuclear features on core biopsies using artificial intelligence (AI) can predict response to NAC.

METHODS

Fifty-eight HER2-positive or triple-negative breast cancer patients were included in this study (pCR n = 37, pPR n = 21). Multiple deep convolutional neural networks were developed to automate tumor detection and nuclear segmentation. Nuclear count, area, and circularity, as well as image-based first- and second-order features including mean pixel intensity and correlation of the gray-level co-occurrence matrix (GLCM-COR) were determined.

RESULTS

In univariate analysis, the pCR group had fewer multifocal/multicentric tumors, higher nuclear intensity, and lower GLCM-COR compared to the pPR group. In multivariate binary logistic regression, tumor multifocality/multicentricity (OR = 0.14, p = 0.012), nuclear intensity (OR = 1.23, p = 0.018), and GLCM-COR (OR = 0.96, p = 0.043) were each independently associated with likelihood of achieving a pCR, and the model was able to successful classify 79% of cases (62% for pPR and 89% for pCR).

CONCLUSION

Analysis of tumor nuclear features using digital pathology/AI can significantly improve models to predict pathological response to NAC.

Integrative Models of Histopathological Image Features and Omics Data Predict Survival in Head and Neck Squamous Cell Carcinoma

Background

Both histopathological image features and genomics data were associated with survival outcome of cancer patients. However, integrating features of histopathological images, genomics and other omics for improving prognosis prediction has not been reported in head and neck squamous cell carcinoma (HNSCC).

Methods

A dataset of 216 HNSCC patients was derived from the Cancer Genome Atlas (TCGA) with information of clinical characteristics, genetic mutation, RNA sequencing, protein expression and histopathological images. Patients were randomly assigned into training (n = 108) or validation (n = 108) sets. We extracted 593 quantitative image features, and used random forest algorithm with 10-fold cross-validation to build prognostic models for overall survival (OS) in training set, then compared the area under the time-dependent receiver operating characteristic curve (AUC) in validation set.

Results

In validation set, histopathological image features had significant predictive value for OS (5-year AUC = 0.784). The histopathology + omics models showed better predictive performance than genomics, transcriptomics or proteomics alone. Moreover, the multi-omics model incorporating image features, genomics, transcriptomics and proteomics reached the maximal 1-, 3-, and 5-year AUC of 0.871, 0.908, and 0.929, with most significant survival difference (HR = 10.66, 95%CI: 5.06–26.8, p < 0.001). Decision curve analysis also revealed a better net benefit of multi-omics model.

Conclusion

The histopathological images could provide complementary features to improve prognostic performance for HNSCC patients. The integrative model of histopathological image features and omics data might serve as an effective tool for survival prediction and risk stratification in clinical practice.

Review of the current state of digital image analysis in breast pathology

Advances in digital image analysis have the potential to transform the practice of breast pathology. In the near future, a move to a digital workflow offers improvements in efficiency. Coupled with artificial intelligence (AI), digital pathology can assist pathologist interpretation, automate time-consuming tasks, and discover novel morphologic patterns. Opportunities for digital enhancements abound in breast pathology, from increasing reproducibility in grading and biomarker interpretation, to discovering features that correlate with patient outcome and treatment. Our objective is to review the most recent developments in digital pathology with clear impact to breast pathology practice. Although breast pathologists currently undertake limited adoption of digital methods, the field is rapidly evolving. Care is needed to validate emerging technologies for effective patient care.

Computational analysis of morphological and molecular features in gastric cancer tissues

Biological morphologies of cells and tissues represent their physiological and pathological conditions. The importance of quantitative assessment of morphological information has been highly recognized in clinical diagnosis and therapeutic strategies. In this study, we used a supervised machine learning algorithm wndchrm to classify hematoxylin and eosin (H&E)‐stained images of human gastric cancer tissues. This analysis distinguished between noncancer and cancer tissues with different histological grades. We then classified the H&E‐stained images by expression levels of cancer‐associated nuclear ATF7IP/MCAF1 and membranous PD‐L1 proteins using immunohistochemistry of serial sections. Interestingly, classes with low and high expressions of each protein exhibited significant morphological dissimilarity in H&E images. These results indicated that morphological features in cancer tissues are correlated with expression of specific cancer‐associated proteins, suggesting the usefulness of biomolecular‐based morphological classification.

Learning Where to See: A Novel Attention Model for Automated Immunohistochemical Scoring

Estimating over-amplification of human epidermal growth factor receptor 2 (HER2) on invasive breast cancer is regarded as a significant predictive and prognostic marker. We propose a novel deep reinforcement learning (DRL)-based model that treats immunohistochemical (IHC) scoring of HER2 as a sequential learning task. For a given image tile sampled from multi-resolution giga-pixel whole slide image (WSI), the model learns to sequentially identify some of the diagnostically relevant regions of interest (ROIs) by following a parameterized policy. The selected ROIs are processed by recurrent and residual convolution networks to learn the discriminative features for different HER2 scores and predict the next location, without requiring to process all the sub-image patches of a given tile for predicting the HER2 score, mimicking the histopathologist who would not usually analyze every part of the slide at the highest magnification. The proposed model incorporates a task-specific regularization term and inhibition of return mechanism to prevent the model from revisiting the previously attended locations. We evaluated our model on two IHC datasets: a publicly available dataset from the HER2 scoring challenge contest and another dataset consisting of WSIs of gastroenteropancreatic neuroendocrine tumor sections stained with Glo1 marker. We demonstrate that the proposed model outperforms other methods based on state-of-the-art deep convolutional networks. To the best of our knowledge, this is the first study using DRL for IHC scoring and could potentially lead to wider use of DRL in the domain of computational pathology reducing the computational burden of the analysis of large multi-gigapixel histology images.

Quantitative Histopathology of Stained Tissues using Color Spatial Light Interference Microscopy (cSLIM)

Tissue biopsy evaluation in the clinic is in need of quantitative disease markers for diagnosis and, most importantly, prognosis. Among the new technologies, quantitative phase imaging (QPI) has demonstrated promise for histopathology because it reveals intrinsic tissue nanoarchitecture through the refractive index. However, a vast majority of past QPI investigations have relied on imaging unstained tissues, which disrupts the established specimen processing. Here we present color spatial light interference microscopy (cSLIM) as a new whole-slide imaging modality that performs interferometric imaging on stained tissue, with a color detector array. As a result, cSLIM yields in a single scan both the intrinsic tissue phase map and the standard color bright-field image, familiar to the pathologist. Our results on 196 breast cancer patients indicate that cSLIM can provide stain-independent prognostic information from the alignment of collagen fibers in the tumor microenvironment. The effects of staining on the tissue phase maps were corrected by a mathematical normalization. These characteristics are likely to reduce barriers to clinical translation for the new cSLIM technology.

A deep learning algorithm for one-step contour aware nuclei segmentation of histopathology images

This paper addresses the task of nuclei segmentation in high-resolution histopathology images. We propose an automatic end-to-end deep neural network algorithm for segmentation of individual nuclei. A nucleus-boundary model is introduced to predict nuclei and their boundaries simultaneously using a fully convolutional neural network. Given a color-normalized image, the model directly outputs an estimated nuclei map and a boundary map. A simple, fast, and parameter-free post-processing procedure is performed on the estimated nuclei map to produce the final segmented nuclei. An overlapped patch extraction and assembling method is also designed for seamless prediction of nuclei in large whole-slide images. We also show the effectiveness of data augmentation methods for nuclei segmentation task. Our experiments showed our method outperforms prior state-of-the-art methods. Moreover, it is efficient that one 1000×1000 image can be segmented in less than 5 s. This makes it possible to precisely segment the whole-slide image in acceptable time. The source code is available at https://github.com/easycui/nuclei_segmentation . Graphical Abstract The neural network for nuclei segmentation.

Breast cancer outcome prediction with tumour tissue images and machine learning

PURPOSE

Recent advances in machine learning have enabled better understanding of large and complex visual data. Here, we aim to investigate patient outcome prediction with a machine learning method using only an image of tumour sample as an input.

METHODS

Utilising tissue microarray (TMA) samples obtained from the primary tumour of patients (N = 1299) within a nationwide breast cancer series with long-term-follow-up, we train and validate a machine learning method for patient outcome prediction. The prediction is performed by classifying samples into low or high digital risk score (DRS) groups. The outcome classifier is trained using sample images of 868 patients and evaluated and compared with human expert classification in a test set of 431 patients.

RESULTS

In univariate survival analysis, the DRS classification resulted in a hazard ratio of 2.10 (95% CI 1.33-3.32, p = 0.001) for breast cancer-specific survival. The DRS classification remained as an independent predictor of breast cancer-specific survival in a multivariate Cox model with a hazard ratio of 2.04 (95% CI 1.20-3.44, p = 0.007). The accuracy (C-index) of the DRS grouping was 0.60 (95% CI 0.55-0.65), as compared to 0.58 (95% CI 0.53-0.63) for human expert predictions based on the same TMA samples.

CONCLUSIONS

Our findings demonstrate the feasibility of learning prognostic signals in tumour tissue images without domain knowledge. Although further validation is needed, our study suggests that machine learning algorithms can extract prognostically relevant information from tumour histology complementing the currently used prognostic factors in breast cancer.

Nuclear shape, architecture and orientation features from H&E images are able to predict recurrence in node-negative gastric adenocarcinoma

BACKGROUND

Identifying intestinal node-negative gastric adenocarcinoma (INGA) patients with high risk of recurrence could help perceive benefit of adjuvant therapy for INGA patients following surgical resection. This study evaluated whether the computer-extracted image features of nuclear shapes, texture, orientation, and tumor architecture on digital images of hematoxylin and eosin stained tissue, could help to predict recurrence in INGA patients.

METHODS

A tissue microarrays cohort of 160 retrospectively INGA cases were digitally scanned, and randomly selected as training cohort (D1 = 60), validation cohort (D2 = 100 and D3 = 100, D2 and D3 are different tumor TMA spots from the same patient), accompanied with immunohistochemistry data cohort (D3' = 100, a duplicate cohort of D3) and negative controls data cohort (D5 = 100, normal adjacent tissues). After nuclear segmentation by watershed-based method, 189 local nuclear features were captured on each TMA core and the top 5 features were selected by Wilcoxon rank sum test within D1. A morphometric-based image classifier (NGAHIC) was composed across the discriminative features and predicted the recurrence in INGA on D2. The intra-tumor heterogeneity was assessed on D3. Manual nuclear atypia grading was conducted on D1 and D2 by two pathologists. The expression of HER2 and Ki67 were detected by immunohistochemistry on D3 and D3', respectively. The association between manual grading and INGA outcome was analysis.

RESULTS

Independent validation results showed the NGAHIC achieved an AUC of 0.76 for recurrence prediction. NGAHIC-positive patients had poorer overall survival (P = 0.017) by univariate survival analysis. Multivariate survival analysis, controlling for T-stage, histology stage, invasion depth, demonstrated NGAHIC-positive was a reproducible prognostic factor for poorer disease-specific survival (HR = 17.24, 95% CI 3.93-75.60, P < 0.001). In contrast, human grading was only prognostic for one reader on D2. Moreover, significant correlations were observed between NGAHIC-positive patients and positivity of HER2 and Ki67 labeling index.

CONCLUSIONS

The NGAHIC could provide precision oncology, personalized cancer management.

Convolutional neural network initialized active contour model with adaptive ellipse fitting for nuclear segmentation on breast histopathological images

Automated detection and segmentation of nuclei from high-resolution histopathological images is a challenging problem owing to the size and complexity of digitized histopathologic images. In the context of breast cancer, the modified Bloom-Richardson Grading system is highly correlated with the morphological and topological nuclear features are highly correlated with Modified Bloom-Richardson grading. Therefore, to develop a computer-aided prognosis system, automated detection and segmentation of nuclei are critical prerequisite steps. We present a method for automated detection and segmentation of breast cancer nuclei named a convolutional neural network initialized active contour model with adaptive ellipse fitting (CoNNACaeF). The CoNNACaeF model is able to detect and segment nuclei simultaneously, which consist of three different modules: convolutional neural network (CNN) for accurate nuclei detection, (2) region-based active contour (RAC) model for subsequent nuclear segmentation based on the initial CNN-based detection of nuclear patches, and (3) adaptive ellipse fitting for overlapping solution of clumped nuclear regions. The performance of the CoNNACaeF model is evaluated on three different breast histological data sets, comprising a total of 257 H&E-stained images. The model is shown to have improved detection accuracy of F-measure 80.18%, 85.71%, and 80.36% and average area under precision-recall curves (AveP) 77%, 82%, and 74% on a total of 3 million nuclei from 204 whole slide images from three different datasets. Additionally, CoNNACaeF yielded an F-measure at 74.01% and 85.36%, respectively, for two different breast cancer datasets. The CoNNACaeF model also outperformed the three other state-of-the-art nuclear detection and segmentation approaches, which are blue ratio initialized local region active contour, iterative radial voting initialized local region active contour, and maximally stable extremal region initialized local region active contour models.

Identification of topological features in renal tumor microenvironment associated with patient survival

Motivation

As a highly heterogeneous disease, the progression of tumor is not only achieved by unlimited growth of the tumor cells, but also supported, stimulated, and nurtured by the microenvironment around it. However, traditional qualitative and/or semi-quantitative parameters obtained by pathologist’s visual examination have very limited capability to capture this interaction between tumor and its microenvironment. With the advent of digital pathology, computerized image analysis may provide a better tumor characterization and give new insights into this problem.

Results

We propose a novel bioimage informatics pipeline for automatically characterizing the topological organization of different cell patterns in the tumor microenvironment. We apply this pipeline to the only publicly available large histopathology image dataset for a cohort of 190 patients with papillary renal cell carcinoma obtained from The Cancer Genome Atlas project. Experimental results show that the proposed topological features can successfully stratify early- and middle-stage patients with distinct survival, and show superior performance to traditional clinical features and cellular morphological and intensity features. The proposed features not only provide new insights into the topological organizations of cancers, but also can be integrated with genomic data in future studies to develop new integrative biomarkers.

Availability and implementation

https://github.com/chengjun583/KIRP-topological-features

Supplementary information

Supplementary data are available atBioinformatics online.

A NanoFlare-Based Strategy for In Situ Tumor Margin Demarcation and Neoadjuvant Gene/Photothermal Therapy

Accurate tumor margin demarcation in situ remains a paramount challenge. Herein, a NanoFlare (also known as spherical-nucleic-acid technology) based strategy is reported for in situ tumor margin delineation by transforming and amplifying the pathophysiological redox signals of tumor microenvironment. The NanoFlare designed (named AuNS-ASON) is based on gold nanostar (AuNS) coated with a dense shell of disulfide bridge-inserted and cyanine dyes-labeled antisense oligonucleotides (ASON) targeting survivin mRNA. The unique anisotropic ASON-spike nanostructure endows the AuNS-ASON with universal cellular internalization of tumor cells, while the disulfide bridge inserted confers response specificity toward redox activation. In vitro experiments demonstrate that the AuNS-ASON can discriminate tumor cells rapidly with activated fluorescence signals (>100-fold) in 2 h, and further achieve synergistic gene/photothermal tumor cells ablation upon near-infrared laser irradiation. Remarkably, in situ tumor margin delineation with high accuracy and outstanding spatial resolution (<100 µm) in mice bearing different tumors is obtained based on the AuNS-ASON, providing intraoperative guidance for tumor resection. Moreover, the AuNS-ASON can enable efficient neoadjuvant gene/photothermal therapy before surgery to reduce tumor extent and increase resectability. The concept of NanoFlare-based microenvironment signal transformation and amplification could be used as a general strategy to guide the design of activatable nanoprobes for cancer theranostics.

Development of a computer-aided tool for the pattern recognition of facial features in diagnosing Turner syndrome: comparison of diagnostic accuracy with clinical workers

Technologies applied for the recognition of facial features in diagnosing certain disorders seem to be promising in reducing the medical burden and improve the efficiency. This pilot study aimed to develop a computer-assisted tool for the pattern recognition of facial features for diagnosing Turner syndrome (TS). Photographs of 54 patients with TS and 158 female controls were collected from July 2016 to May 2017. Finally, photographs of 32 patients with TS and 96 age-matched controls were included in the study that were further divided equally into training and testing groups. The process of automatic classification consisted of image preprocessing, facial feature extraction, feature reduction and fusion, automatic classification, and result presentation. A total of 27 physicians and 21 medical students completed a web-based test including the same photographs used in computer testing. After training, the automatic facial classification system for diagnosing TS achieved a 68.8% sensitivity and 87.5% specificity (and a 67.6% average sensitivity and 87.9% average specificity after resampling), which was significantly higher than the average sensitivity (57.4%, P < 0.001) and specificity (75.4%, P < 0.001) of 48 participants, respectively. The accuracy of this system was satisfactory and better than the diagnosis by clinicians. However, the system necessitates further improvement for achieving a high diagnostic accuracy in clinical practice.

The significance of tumour microarchitectural features in breast cancer prognosis: a digital image analysis

Background

As only a minor portion of the information present in histological sections is accessible by eye, recognition and quantification of complex patterns and relationships among constituents relies on digital image analysis. In this study, our working hypothesis was that, with the application of digital image analysis technology, visually unquantifiable breast cancer microarchitectural features can be rigorously assessed and tested as prognostic parameters for invasive breast carcinoma of no special type.

Methods

Digital image analysis was performed using public domain software (ImageJ) on tissue microarrays from a cohort of 696 patients, and validated with a commercial platform (Visiopharm). Quantified features included elements defining tumour microarchitecture, with emphasis on the extent of tumour-stroma interface. The differential prognostic impact of tumour nest microarchitecture in the four immunohistochemical surrogates for molecular classification was analysed. Prognostic parameters included axillary lymph node status, breast cancer-specific survival, and time to distant metastasis. Associations of each feature with prognostic parameters were assessed using logistic regression and Cox proportional models adjusting for age at diagnosis, grade, and tumour size.

Results

An arrangement in numerous small nests was associated with axillary lymph node involvement. The association was stronger in luminal tumours (odds ratio (OR) = 1.39, p = 0.003 for a 1-SD increase in nest number, OR = 0.75, p = 0.006 for mean nest area). Nest number was also associated with survival (hazard ratio (HR) = 1.15, p = 0.027), but total nest perimeter was the parameter most significantly associated with survival in luminal tumours (HR = 1.26, p = 0.005). In the relatively small cohort of triple-negative tumours, mean circularity showed association with time to distant metastasis (HR = 1.71, p = 0.027) and survival (HR = 1.8, p = 0.02).

Conclusions

We propose that tumour arrangement in few large nests indicates a decreased metastatic potential. By contrast, organisation in numerous small nests provides the tumour with increased metastatic potential to regional lymph nodes. An outstretched pattern in small nests bestows tumours with a tendency for decreased breast cancer-specific survival. Although further validation studies are required before the argument for routine quantification of microarchitectural features is established, our approach is consistent with the demand for cost-effective methods for triaging breast cancer patients that are more likely to benefit from chemotherapy.

Integrative Analysis of Histopathological Images and Genomic Data Predicts Clear Cell Renal Cell Carcinoma Prognosis

In cancer, both histopathologic images and genomic signatures are used for diagnosis, prognosis, and subtyping. However, combining histopathologic images with genomic data for predicting prognosis, as well as the relationships between them, has rarely been explored. In this study, we present an integrative genomics framework for constructing a prognostic model for clear cell renal cell carcinoma. We used patient data from The Cancer Genome Atlas (n = 410), extracting hundreds of cellular morphologic features from digitized whole-slide images and eigengenes from functional genomics data to predict patient outcome. The risk index generated by our model correlated strongly with survival, outperforming predictions based on considering morphologic features or eigengenes separately. The predicted risk index also effectively stratified patients in early-stage (stage I and stage II) tumors, whereas no significant survival difference was observed using staging alone. The prognostic value of our model was independent of other known clinical and molecular prognostic factors for patients with clear cell renal cell carcinoma. Overall, this workflow and the shared software code provide building blocks for applying similar approaches in other cancers. Cancer Res; 77(21); e91-100. ©2017 AACR.

Emerging Themes in Image Informatics and Molecular Analysis for Digital Pathology

Pathology is essential for research in disease and development, as well as for clinical decision making. For more than 100 years, pathology practice has involved analyzing images of stained, thin tissue sections by a trained human using an optical microscope. Technological advances are now driving major changes in this paradigm toward digital pathology (DP). The digital transformation of pathology goes beyond recording, archiving, and retrieving images, providing new computational tools to inform better decision making for precision medicine. First, we discuss some emerging innovations in both computational image analytics and imaging instrumentation in DP. Second, we discuss molecular contrast in pathology. Molecular DP has traditionally been an extension of pathology with molecularly specific dyes. Label-free, spectroscopic images are rapidly emerging as another important information source, and we describe the benefits and potential of this evolution. Third, we describe multimodal DP, which is enabled by computational algorithms and combines the best characteristics of structural and molecular pathology. Finally, we provide examples of application areas in telepathology, education, and precision medicine. We conclude by discussing challenges and emerging opportunities in this area.

Distribution pattern of tumor associated macrophages predicts the prognosis of gastric cancer

Purpose

As mayor biomarkers in tumor microenvironment (TME), tumor associated macrophages (TAMs) of gastric cancer (GC) still needs further studies in terms of the number and distribution pattern.

Methods

Herein, tissue microarrays (TMA) incorporating 494 GC surgical samples in duplicate were stained for TAMs infiltration analysis. TAMs number was counted according to the locations, including infiltrating macrophages in cancer nest (MC), in invasive front (MF) and in stroma (MS). Correlations between TAMs number, distribution pattern and clinic-pathological features and survival analyses were performed.

Results

Infiltrating macrophages number in GC tissues was much higher than that in peritumoral tissues. TAMs number was not significantly correlated with the overall survival (OS). TAMs distribution pattern could be categorized into MC or MF/MS dominant pattern, and correlated with histological grade (P =0.001). The median OS of MF/MS dominant pattern (22.1, 95%CI: 23.5-28.9) was significantly shorter than that of MC dominant pattern (25.6, 95%CI: 28.5-35.6) (P =0.002). By receiver operating characteristic curve (ROC) analysis, the predictive value of TAMs distribution pattern was superior to histological grade and pM stage, but inferior to pN and TNM stage.

Conclusions

TAMs distribution pattern could be an independent prognostic factor for the OS of GC patients, and patients with MF/MS dominant pattern had worse outcomes.

Automated Classification of Breast Cancer Stroma Maturity From Histological Images

OBJECTIVE

The tumor microenvironment plays a crucial role in regulating tumor progression by a number of different mechanisms, in particular, the remodeling of collagen fibers in tumor-associated stroma, which has been reported to be related to patient survival. The underlying motivation of this work is that remodeling of collagen fibers gives rise to observable patterns in hematoxylin and eosin (H&E) stained slides from clinical cases of invasive breast carcinoma that the pathologist can label as mature or immature stroma. The aim of this paper is to categorise and automatically classify stromal regions according to their maturity and show that this classification agrees with that of skilled observers, hence providing a repeatable and quantitative measure for prognostic studies.

METHODS

We use multiscale basic image features and local binary patterns, in combination with a random decision trees classifier for classification of breast cancer stroma regions-of-interest (ROI).

RESULTS

We present results from a cohort of 55 patients with analysis of 169 ROI. Our multiscale approach achieved a classification accuracy of 84%.

CONCLUSION

This work demonstrates the ability of texture-based image analysis to differentiate breast cancer stroma maturity in clinically acquired H&E-stained slides at least as well as skilled observers.

Multiview boosting digital pathology analysis of prostate cancer

BACKGROUND AND OBJECTIVE

Various digital pathology tools have been developed to aid in analyzing tissues and improving cancer pathology. The multi-resolution nature of cancer pathology, however, has not been fully analyzed and utilized. Here, we develop an automated, cooperative, and multi-resolution method for improving prostate cancer diagnosis.

METHODS

Digitized tissue specimen images are obtained from 5 tissue microarrays (TMAs). The TMAs include 70 benign and 135 cancer samples (TMA1), 74 benign and 89 cancer samples (TMA2), 70 benign and 115 cancer samples (TMA3), 79 benign and 82 cancer samples (TMA4), and 72 benign and 86 cancer samples (TMA5). The tissue specimen images are segmented using intensity- and texture-based features. Using the segmentation results, a number of morphological features from lumens and epithelial nuclei are computed to characterize tissues at different resolutions. Applying a multiview boosting algorithm, tissue characteristics, obtained from differing resolutions, are cooperatively combined to achieve accurate cancer detection.

RESULTS

In segmenting prostate tissues, the multiview boosting method achieved≥ 0.97 AUC using TMA1. For detecting cancers, the multiview boosting method achieved an AUC of 0.98 (95% CI: 0.97-0.99) as trained on TMA2 and tested on TMA3, TMA4, and TMA5. The proposed method was superior to single-view approaches, utilizing features from a single resolution or merging features from all the resolutions. Moreover, the performance of the proposed method was insensitive to the choice of the training dataset. Trained on TMA3, TMA4, and TMA5, the proposed method obtained an AUC of 0.97 (95% CI: 0.96-0.98), 0.98 (95% CI: 0.96-0.99), and 0.97 (95% CI: 0.96-0.98), respectively.

CONCLUSIONS

The multiview boosting method is capable of integrating information from multiple resolutions in an effective and efficient fashion and identifying cancers with high accuracy. The multiview boosting method holds a great potential for improving digital pathology tools and research.

Computer-aided prognosis on breast cancer with hematoxylin and eosin histopathology images: A review

With the advance of digital pathology, image analysis has begun to show its advantages in information analysis of hematoxylin and eosin histopathology images. Generally, histological features in hematoxylin and eosin images are measured to evaluate tumor grade and prognosis for breast cancer. This review summarized recent works in image analysis of hematoxylin and eosin histopathology images for breast cancer prognosis. First, prognostic factors for breast cancer based on hematoxylin and eosin histopathology images were summarized. Then, usual procedures of image analysis for breast cancer prognosis were systematically reviewed, including image acquisition, image preprocessing, image detection and segmentation, and feature extraction. Finally, the prognostic value of image features and image feature–based prognostic models was evaluated. Moreover, we discussed the issues of current analysis, and some directions for future research.

Prognostic biomarker value of binary and grayscale breast tumor histopathology images

AIM

Breast cancer prognosis is in the spotlight owing to its potentially major clinical importance in effective therapeutic management. Following our recent prognostic establishment of the fractal features calculated on binary breast tumor histopathology images, this study aimed to accomplish the first optimization of this methodology by direct comparison of monofractal, multifractal and co-occurrence algorithms in analysis of binary versus grayscale image formats.

PATIENTS & METHODS

The study included 93 patients with invasive breast cancer, without systemic treatment and a long median follow-up of 150 months.

RESULTS

Grayscale images provided a better prognostic source in comparison to binary, while monofractal, multifractal and co-occurrence image analysis algorithms exerted a comparable performance.

CONCLUSION

The critical prognostic importance of the grayscale texture is revealed.

A robust nonlinear tissue-component discrimination method for computational pathology

Advances in digital pathology, specifically imaging instrumentation and data management, have allowed for the development of computational pathology tools with the potential for better, faster, and cheaper diagnosis, prognosis, and prediction of disease. Images of tissue sections frequently vary in color appearance across research laboratories and medical facilities because of differences in tissue fixation, staining protocols, and imaging instrumentation, leading to difficulty in the development of robust computational tools. To address this challenge, we propose a novel nonlinear tissue-component discrimination (NLTD) method to register automatically the color space of histopathology images and visualize individual tissue components, independent of color differences between images. Our results show that the NLTD method could effectively discriminate different tissue components from different types of tissues prepared at different institutions. Further, we demonstrate that NLTD can improve the accuracy of nuclear detection and segmentation algorithms, compared with using conventional color deconvolution methods, and can quantitatively analyze immunohistochemistry images. Together, the NLTD method is objective, robust, and effective, and can be easily implemented in the emerging field of computational pathology.

Quantum dots-based double imaging combined with organic dye imaging to establish an automatic computerized method for cancer Ki67 measurement

As a widely used proliferative marker, Ki67 has important impacts on cancer prognosis, especially for breast cancer (BC). However, variations in analytical practice make it difficult for pathologists to manually measure Ki67 index. This study is to establish quantum dots (QDs)-based double imaging of nuclear Ki67 as red signal by QDs-655, cytoplasmic cytokeratin (CK) as yellow signal by QDs-585, and organic dye imaging of cell nucleus as blue signal by 4′,6-diamidino-2-phenylindole (DAPI), and to develop a computer-aided automatic method for Ki67 index measurement. The newly developed automatic computerized Ki67 measurement could efficiently recognize and count Ki67-positive cancer cell nuclei with red signals and cancer cell nuclei with blue signals within cancer cell cytoplasmic with yellow signals. Comparisons of computerized Ki67 index, visual Ki67 index, and marked Ki67 index for 30 patients of 90 images with Ki67 ≤ 10% (low grade), 10% < Ki67 < 50% (moderate grade), and Ki67 ≥ 50% (high grade) showed computerized Ki67 counting is better than visual Ki67 counting, especially for Ki67 low and moderate grades. Based on QDs-based double imaging and organic dye imaging on BC tissues, this study successfully developed an automatic computerized Ki67 counting method to measure Ki67 index.