Image analysis and machine learning in digital pathology: Challenges and opportunities

SPLICE - Streamlining Digital Pathology Image Processing

Digital pathology and the integration of artificial intelligence (AI) models have revolutionized histopathology, opening new opportunities. With the increasing availability of Whole Slide Images (WSIs), there's a growing demand for efficient retrieval, processing, and analysis of relevant images from vast biomedical archives. However, processing WSIs presents challenges due to their large size and content complexity. Full computer digestion of WSIs is impractical, and processing all patches individually is prohibitively expensive. In this paper, we propose an unsupervised patching algorithm, Sequential Patching Lattice for Image Classification and Enquiry (SPLICE). This novel approach condenses a histopathology WSI into a compact set of representative patches, forming a "collage" of WSI while minimizing redundancy. SPLICE prioritizes patch quality and uniqueness by sequentially analyzing a WSI and selecting nonredundant representative features. We evaluated SPLICE for search and match applications, demonstrating improved accuracy, reduced computation time, and storage requirements compared to existing state-of-the-art methods. As an unsupervised method, SPLICE effectively reduces storage requirements for representing tissue images by 50%. This reduction enables numerous algorithms in computational pathology to operate much more efficiently, paving the way for accelerated adoption of digital pathology.

Advantages of automated immunostain analyses for complex membranous immunostains: An exemplar investigating loss of E-cadherin expression in colorectal cancer

As pathology moves towards digitisation, biomarker profiling through automated image analysis provides potentially objective and time-efficient means of assessment. This study set out to determine how a complex membranous immunostain, E-cadherin, assessed using an automated digital platform fares in comparison to manual evaluation in terms of clinical correlations and prognostication. Tissue microarrays containing 1000 colorectal cancer samples, stained with clinical E-cadherin antibodies were assessed through both manual scoring and automated image analysis. Both manual and automated scores were correlated to clinicopathological and survival data. E-cadherin data generated through digital image analysis was superior to manual evaluation when investigating for clinicopathological correlations in colorectal cancer. Loss of membranous E-cadherin, assessed on automated platforms, correlated with: right sided tumours (p = <0.001), higher T-stage (p = <0.001), higher grade (p = <0.001), N2 nodal stage (p = <0.001), intramural lymphovascular invasion (p = 0.006), perineural invasion (p = 0.028), infiltrative tumour edge (p = 0.001) high tumour budding score (p = 0.038), distant metastasis (p = 0.035), and poorer 5-year (p= 0.042) survival status. Manual assessment was only correlated with higher grade tumours, though other correlations become apparent only when assessed for morphological expression pattern (circumferential, basolateral, parallel) irrespective of intensity. Digital assessment of E-cadherin is effective for prognostication of colorectal cancer and may potentially offer benefits of improved objectivity, accuracy, and economy of time. Incorporating tools to assess patterns of staining may further improve such digital assessment in the future.

CohortFinder: an open-source tool for data-driven partitioning of digital pathology and imaging cohorts to yield robust machine-learning models

Batch effects (BEs) refer to systematic technical differences in data collection unrelated to biological variations whose noise is shown to negatively impact machine learning (ML) model generalizability. Here we release CohortFinder (http://cohortfinder.com), an open-source tool aimed at mitigating BEs via data-driven cohort partitioning. We demonstrate CohortFinder improves ML model performance in downstream digital pathology and medical image processing tasks. CohortFinder is freely available for download at cohortfinder.com.

Automated grading of anatomical objective structured practical examinations using decision trees: An artificial intelligence approach

An Objective Structured Practical Examination (OSPE) is an effective and robust, but resource-intensive, means of evaluating anatomical knowledge. Since most OSPEs employ short answer or fill-in-the-blank style questions, the format requires many people familiar with the content to mark the examinations. However, the increasing prevalence of online delivery for anatomy and physiology courses could result in students losing the OSPE practice that they would receive in face-to-face learning sessions. The purpose of this study was to test the accuracy of Decision Trees (DTs) in marking OSPE questions as a first step to creating an intelligent, online OSPE tutoring system. The study used the results of the winter 2020 semester final OSPE from McMaster University's anatomy and physiology course in the Faculty of Health Sciences (HTHSCI 2FF3/2LL3/1D06) as the data set. Ninety percent of the data set was used in a 10-fold validation algorithm to train a DT for each of the 54 questions. Each DT was comprised of unique words that appeared in correct, student-written answers. The remaining 10% of the data set was marked by the generated DTs. When the answers marked by the DT were compared to the answers marked by staff and faculty, the DT achieved an average accuracy of 94.49% across all 54 questions. This suggests that machine learning algorithms such as DTs are a highly effective option for OSPE grading and are suitable for the development of an intelligent, online OSPE tutoring system.

A systematic comparison of deep learning methods for Gleason grading and scoring

Prostate cancer is the second most frequent cancer in men worldwide after lung cancer. Its diagnosis is based on the identification of the Gleason score that evaluates the abnormality of cells in glands through the analysis of the different Gleason patterns within tissue samples. The recent advancements in computational pathology, a domain aiming at developing algorithms to automatically analyze digitized histopathology images, lead to a large variety and availability of datasets and algorithms for Gleason grading and scoring. However, there is no clear consensus on which methods are best suited for each problem in relation to the characteristics of data and labels. This paper provides a systematic comparison on nine datasets with state-of-the-art training approaches for deep neural networks (including fully-supervised learning, weakly-supervised learning, semi-supervised learning, Additive-MIL, Attention-Based MIL, Dual-Stream MIL, TransMIL and CLAM) applied to Gleason grading and scoring tasks. The nine datasets are collected from pathology institutes and openly accessible repositories. The results show that the best methods for Gleason grading and Gleason scoring tasks are fully supervised learning and CLAM, respectively, guiding researchers to the best practice to adopt depending on the task to solve and the labels that are available.

BiliQML: a supervised machine-learning model to quantify biliary forms from digitized whole slide liver histopathological images

The progress of research focused on cholangiocytes and the biliary tree during development and following injury is hindered by limited available quantitative methodologies. Current techniques include two-dimensional standard histological cell-counting approaches, which are rapidly performed, error prone, and lack architectural context or three-dimensional analysis of the biliary tree in opacified livers, which introduce technical issues along with minimal quantitation. The present study aims to fill these quantitative gaps with a supervised machine-learning model (BiliQML) able to quantify biliary forms in the liver of anti-keratin 19 antibody-stained whole slide images. Training utilized 5,019 researcher-labeled biliary forms, which following feature selection, and algorithm optimization, generated an F score of 0.87. Application of BiliQML on seven separate cholangiopathy models [genetic (Afp-CRE;Pkd1l1null/Fl, Alb-CRE;Rbp-jkfl/fl, and Albumin-CRE;ROSANICD), surgical (bile duct ligation), toxicological (3,5-diethoxycarbonyl-1,4-dihydrocollidine), and therapeutic (Cyp2c70-/- with ileal bile acid transporter inhibition)] allowed for a means to validate the capabilities and utility of this platform. The results from BiliQML quantification revealed biological and pathological differences across these seven diverse models, indicating a highly sensitive, robust, and scalable methodology for the quantification of distinct biliary forms. BiliQML is the first comprehensive machine-learning platform for biliary form analysis, adding much-needed morphologic context to standard immunofluorescence-based histology, and provides clinical and basic science researchers with a novel tool for the characterization of cholangiopathies.NEW & NOTEWORTHY BiliQML is the first comprehensive machine-learning platform for biliary form analysis in whole slide histopathological images. This platform provides clinical and basic science researchers with a novel tool for the improved quantification and characterization of biliary tract disorders.

Assessment of AURKA expression and prognosis prediction in lung adenocarcinoma using machine learning-based pathomics signature

Objective

This study aimed to develop quantitative feature-based models from histopathological images to assess aurora kinase A (AURKA) expression and predict the prognosis of patients with lung adenocarcinoma (LUAD).

Methods

A dataset of patients with LUAD was derived from the cancer genome atlas (TCGA) with information on clinical characteristics, RNA sequencing and histopathological images. The TCGA-LUAD cohort was randomly divided into training (n = 229) and testing (n = 98) sets. We extracted quantitative image features from histopathological slides of patients with LUAD using computational approaches, constructed a predictive model for AURKA expression in the training set, and estimated their predictive performance in the test set. A Cox proportional hazards model was used to assess whether the pathomic scores (PS) generated by the model independently predicted LUAD survival.

Results

High AURKA expression was an independent risk factor for overall survival (OS) in patients with LUAD (hazard ratio = 1.816, 95 % confidence intervals = 1.257-2.623, P = 0.001). The model based on histopathological image features had significant predictive value for AURKA expression: the area under the curve of the receiver operating characteristic curve in the training set and validation set was 0.809 and 0.739, respectively. Decision curve analysis showed that the model had clinical utility. Patients with high PS and low PS had different survival rates (P = 0.019). Multivariate analysis suggested that PS was an independent prognostic factor for LUAD (hazard ratio = 1.615, 95 % confidence intervals = 1.071-2.438, P = 0.022).

Conclusion

Pathomics models based on machine learning can accurately predict AURKA expression and the PS generated by the model can predict LUAD prognosis.

A fully automated and explainable algorithm for predicting malignant transformation in oral epithelial dysplasia

Oral epithelial dysplasia (OED) is a premalignant histopathological diagnosis given to lesions of the oral cavity. Its grading suffers from significant inter-/intra-observer variability, and does not reliably predict malignancy progression, potentially leading to suboptimal treatment decisions. To address this, we developed an artificial intelligence (AI) algorithm, that assigns an Oral Malignant Transformation (OMT) risk score based on the Haematoxylin and Eosin (H&E) stained whole slide images (WSIs). Our AI pipeline leverages an in-house segmentation model to detect and segment both nuclei and epithelium. Subsequently, a shallow neural network utilises interpretable morphological and spatial features, emulating histological markers, to predict progression. We conducted internal cross-validation on our development cohort (Sheffield; n = 193 cases) and independent validation on two external cohorts (Birmingham and Belfast; n = 89 cases). On external validation, the proposed OMTscore achieved an AUROC = 0.75 (Recall = 0.92) in predicting OED progression, outperforming other grading systems (Binary: AUROC = 0.72, Recall = 0.85). Survival analyses showed the prognostic value of our OMTscore (C-index = 0.60, p = 0.02), compared to WHO (C-index = 0.64, p = 0.003) and binary grades (C-index = 0.65, p < 0.001). Nuclear analyses elucidated the presence of peri-epithelial and intra-epithelial lymphocytes in highly predictive patches of transforming cases (p < 0.001). This is the first study to propose a completely automated, explainable, and externally validated algorithm for predicting OED transformation. Our algorithm shows comparable-to-human-level performance, offering a promising solution to the challenges of grading OED in routine clinical practice.

Deep learning-based virtual H& E staining from label-free autofluorescence lifetime images

Label-free autofluorescence lifetime is a unique feature of the inherent fluorescence signals emitted by natural fluorophores in biological samples. Fluorescence lifetime imaging microscopy (FLIM) can capture these signals enabling comprehensive analyses of biological samples. Despite the fundamental importance and wide application of FLIM in biomedical and clinical sciences, existing methods for analysing FLIM images often struggle to provide rapid and precise interpretations without reliable references, such as histology images, which are usually unavailable alongside FLIM images. To address this issue, we propose a deep learning (DL)-based approach for generating virtual Hematoxylin and Eosin (H&E) staining. By combining an advanced DL model with a contemporary image quality metric, we can generate clinical-grade virtual H&E-stained images from label-free FLIM images acquired on unstained tissue samples. Our experiments also show that the inclusion of lifetime information, an extra dimension beyond intensity, results in more accurate reconstructions of virtual staining when compared to using intensity-only images. This advancement allows for the instant and accurate interpretation of FLIM images at the cellular level without the complexities associated with co-registering FLIM and histology images. Consequently, we are able to identify distinct lifetime signatures of seven different cell types commonly found in the tumour microenvironment, opening up new opportunities towards biomarker-free tissue histology using FLIM across multiple cancer types.

Artificial Intelligence in Dermatopathology: updates, strengths, and challenges

Artificial Intelligence (AI) has evolved to become a significant force in various domains, including medicine. We explore the role of AI in pathology, with a specific focus on dermatopathology and neoplastic dermatopathology. AI, encompassing Machine Learning (ML) and Deep Learning (DL), has demonstrated its potential in tasks ranging from diagnostic applications on Whole Slide Imaging (WSI) to predictive and prognostic functions in skin pathology. In dermatopathology, studies have assessed AI's ability to identify skin lesions, classify melanomas, and improve diagnostic accuracy. Results indicate that AI, particularly Convolutional Neural Networks (CNNs), can outperform human pathologists in terms of sensitivity and specificity. Moreover, AI aids in predicting disease outcomes, identifying aggressive tumors, and differentiating between various skin conditions. Neoplastic dermatopathology showcases AI's prowess in classifying melanocytic lesions, discriminating between melanomas and nevi, and aiding dermatopathologists in making accurate diagnoses. Studies emphasize the reproducibility and diagnostic aid that AI provides, especially in challenging cases. In inflammatory and lymphoproliferative dermatopathology, limited research exists, but studies show attempts to use AI to differentiate conditions like Mycosis Fungoides and eczema. While some results are promising, further exploration is needed in these areas. We highlight the extraordinary interest AI has garnered in the scientific community and its potential to assist clinicians and pathologists. Despite the advancements, we have stress edthe importance of collaboration between medical professionals, computer scientists, bioinformaticians, and engineers to harness AI's benefits while acknowledging its limitations and risks. The integration of AI into dermatopathology holds great promise, positioning it as a valuable tool rather than as a replacement for human expertise.

An AI based, open access screening tool for early diagnosis of Burkitt lymphoma

Burkitt Lymphoma (BL) is a highly treatable cancer. However, delayed diagnosis of BL contributes to high mortality in BL endemic regions of Africa. Lack of enough pathologists in the region is a major reason for delayed diagnosis. The work described in this paper is a proof-of-concept study to develop a targeted, open access AI tool for screening of histopathology slides in suspected BL cases. Slides were obtained from a total of 90 BL patients. 70 Tonsillectomy samples were used as controls. We fine-tuned 6 pre-trained models and evaluated the performance of all 6 models across different configurations. An ensemble-based consensus approach ensured a balanced and robust classification. The tool applies novel features to BL diagnosis including use of multiple image magnifications, thus enabling use of different magnifications of images based on the microscope/scanner available in remote clinics, composite scoring of multiple models and utilizing MIL with weak labeling and image augmentation, enabling use of relatively low sample size to achieve good performance on the inference set. The open access model allows free access to the AI tool from anywhere with an internet connection. The ultimate aim of this work is making pathology services accessible, efficient and timely in remote clinics in regions where BL is endemic. New generation of low-cost slide scanners/microscopes is expected to make slide images available immediately for the AI tool for screening and thus accelerate diagnosis by pathologists available locally or online.

Development and deployment of a histopathology-based deep learning algorithm for patient prescreening in a clinical trial

Accurate identification of genetic alterations in tumors, such as Fibroblast Growth Factor Receptor, is crucial for treating with targeted therapies; however, molecular testing can delay patient care due to the time and tissue required. Successful development, validation, and deployment of an AI-based, biomarker-detection algorithm could reduce screening cost and accelerate patient recruitment. Here, we develop a deep-learning algorithm using >3000 H&E-stained whole slide images from patients with advanced urothelial cancers, optimized for high sensitivity to avoid ruling out trial-eligible patients. The algorithm is validated on a dataset of 350 patients, achieving an area under the curve of 0.75, specificity of 31.8% at 88.7% sensitivity, and projected 28.7% reduction in molecular testing. We successfully deploy the system in a non-interventional study comprising 89 global study clinical sites and demonstrate its potential to prioritize/deprioritize molecular testing resources and provide substantial cost savings in the drug development and clinical settings.

Muscle fibre morphometric analysis (MusMA) correlates with muscle function and cardiovascular risk prognosis

Morphometry of striated muscle fibres is critical for monitoring muscle health and function. Here, we evaluated functional parameters of skeletal and cardiac striated muscle in two experimental models using the Morphometric Analysis of Muscle Fibre tool (MusMA). The collagen-induced arthritis model was used to evaluate the function of skeletal striated muscle and the non-alcoholic fatty liver disease model was used for cardiac striated muscle analysis. After euthanasia, we used haeamatoxylin and eosin stained sections of skeletal and cardiac muscle to perform muscle fibre segmentation and morphometric analysis. Morphometric analysis classified muscle fibres into six subpopulations: normal, regular hypertrophic, irregular hypertrophic, irregular, irregular atrophic and regular atrophic. The percentage of atrophic fibres was associated with lower walking speed (p = 0.009) and lower body weight (p = 0.026), respectively. Fibres categorized as normal were associated with maximum grip strength (p < 0.001) and higher march speed (p < 0.001). In the evaluation of cardiac striated muscle fibres, the percentage of normal cardiomyocytes negatively correlated with cardiovascular risk markers such as the presence of abdominal adipose tissue (p = .003), miR-33a expression (p = .001) and the expression of miR-126 (p = .042) Furthermore, the percentage of atrophic cardiomyocytes correlated significantly with the Castelli risk index II (p = .014). MusMA is a simple and objective tool that allows the screening of striated muscle fibre morphometry, which can complement the diagnosis of muscle diseases while providing functional and prognostic information in basic and clinical research.

Investigating cartilage-related diseases by polarization-resolved second harmonic generation (P-SHG) imaging

Establishing quantitative parameters for differentiating between healthy and diseased cartilage tissues by examining collagen fibril degradation patterns facilitates the understanding of tissue characteristics during disease progression. These findings could also complement existing clinical methods used to diagnose cartilage-related diseases. In this study, cartilage samples from normal, osteoarthritis (OA), and rheumatoid arthritis (RA) tissues were prepared and analyzed using polarization-resolved second harmonic generation (P-SHG) imaging and quantitative image texture analysis. The enhanced molecular contrast obtained from this approach is expected to aid in distinguishing between healthy and diseased cartilage tissues. P-SHG image analysis revealed distinct parameters in the cartilage samples, reflecting variations in collagen fibril arrangement and organization across different pathological states. Normal tissues exhibited distinct χ33/χ31 values compared with those of OA and RA, indicating collagen type transition and cartilage erosion with chondrocyte swelling, respectively. Compared with those of normal tissues, OA samples demonstrated a higher degree of linear polarization, suggesting increased tissue birefringence due to the deposition of type-I collagen in the extracellular matrix. The distribution of the planar orientation of collagen fibrils revealed a more directional orientation in the OA samples, associated with increased type-I collagen, while the RA samples exhibited a heterogeneous molecular orientation. This study revealed that the imaging technique, the quantitative analysis of the images, and the derived parameters presented in this study could be used as a reference for disease diagnostics, providing a clear understanding of collagen fibril degradation in cartilage.

The advent of preventive high-resolution structural histopathology by artificial-intelligence-powered cryogenic electron tomography

Advances in cryogenic electron microscopy (cryoEM) single particle analysis have revolutionized structural biology by facilitating the in vitro determination of atomic- and near-atomic-resolution structures for fully hydrated macromolecular complexes exhibiting compositional and conformational heterogeneity across a wide range of sizes. Cryogenic electron tomography (cryoET) and subtomogram averaging are rapidly progressing toward delivering similar insights for macromolecular complexes in situ, without requiring tags or harsh biochemical purification. Furthermore, cryoET enables the visualization of cellular and tissue phenotypes directly at molecular, nanometric resolution without chemical fixation or staining artifacts. This forward-looking review covers recent developments in cryoEM/ET and related technologies such as cryogenic focused ion beam milling scanning electron microscopy and correlative light microscopy, increasingly enhanced and supported by artificial intelligence algorithms. Their potential application to emerging concepts is discussed, primarily the prospect of complementing medical histopathology analysis. Machine learning solutions are poised to address current challenges posed by "big data" in cryoET of tissues, cells, and macromolecules, offering the promise of enabling novel, quantitative insights into disease processes, which may translate into the clinic and lead to improved diagnostics and targeted therapeutics.

Integrating Spatial and Morphological Characteristics into Melanoma Prognosis: A Computational Approach

In this study, the prognostic value of cellular morphology and spatial configurations in melanoma has been examined, aiming to complement traditional prognostic indicators like mitotic activity and tumor thickness. Through a computational pipeline using machine learning and deep learning methods, we quantified nuclei sizes within different spatial regions and analyzed their prognostic significance using univariate and multivariate Cox models. Nuclei sizes in the invasive band demonstrated a significant hazard ratio (HR) of 1.1 (95% CI: 1.03, 1.18). Similarly, the nuclei sizes of tumor cells and Ki67 S100 co-positive cells in the invasive band achieved HRs of 1.07 (95% CI: 1.02, 1.13) and 1.09 (95% CI: 1.04, 1.16), respectively. Our findings reveal that nuclei sizes, particularly in the invasive band, are potentially prognostic factors. Correlation analyses further demonstrated a meaningful relationship between cellular morphology and tumor progression, notably showing that nuclei size within the invasive band correlates substantially with tumor thickness. These results suggest the potential of integrating spatial and morphological analyses into melanoma prognostication.

MurSS: A Multi-Resolution Selective Segmentation Model for Breast Cancer

Accurately segmenting cancer lesions is essential for effective personalized treatment and enhanced patient outcomes. We propose a multi-resolution selective segmentation (MurSS) model to accurately segment breast cancer lesions from hematoxylin and eosin (H&E) stained whole-slide images (WSIs). We used The Cancer Genome Atlas breast invasive carcinoma (BRCA) public dataset for training and validation. We used the Korea University Medical Center, Guro Hospital, BRCA dataset for the final test evaluation. MurSS utilizes both low- and high-resolution patches to leverage multi-resolution features using adaptive instance normalization. This enhances segmentation performance while employing a selective segmentation method to automatically reject ambiguous tissue regions, ensuring stable training. MurSS rejects 5% of WSI regions and achieves a pixel-level accuracy of 96.88% (95% confidence interval (CI): 95.97-97.62%) and mean Intersection over Union of 0.7283 (95% CI: 0.6865-0.7640). In our study, MurSS exhibits superior performance over other deep learning models, showcasing its ability to reject ambiguous areas identified by expert annotations while using multi-resolution inputs.

LMFE: Learning-Based Multiscale Feature Engineering in Partial Discharge Detection

The partial discharge (PD) detection is of critical importance in the stability and continuity of power distribution operations. Although several feature engineering methods have been developed to refine and improve PD detection accuracy, they can be suboptimal due to several major issues: 1) failure in identifying fault-related pulses; 2) the lack of inner-phase temporal representation; and 3) multiscale feature integration. The aim of this article is to develop a learning-based multiscale feature engineering (LMFE) framework for PD detection of each signal in a three-phase power system, while addressing the above issues. The three-phase measurements are first preprocessed to identify the pulses together with the surrounded waveforms. Next, our feature engineering is conducted to extract the global-scale features, i.e., phase-level and measurement-level aggregations of the pulse-level information, and the local-scale features focusing on waveforms and their inner-phase temporal information. A recurrent neural network (RNN) model is trained, and intermediate features are extracted from this trained RNN model. Furthermore, these multiscale features are merged and fed into a classifier to distinguish the different patterns between faulty and nonfaulty signals. Finally, our LMFE is evaluated by analyzing the VSB ENET dataset, which shows that LMFE outperforms existing approaches and provides the state-of-the-art solution in PD detection.

An Optimized and Advanced Algorithm for the Quantification of Immunohistochemical Biomarkers in Keratinocytes

Advancements in pathology have given rise to software applications intended to minimize human error and improve efficacy of image analysis. Still, the subjectivity of image quantification performed manually and the limitations of the most ubiquitous tissue stain analysis software requiring parameters tuned by the observer, reveal the need for a highly accurate, automated nuclear quantification software specific to immunohistochemistry, with improved precision and efficiency compared with the methods currently in use. We present a method for the quantification of immunohistochemical biomarkers in keratinocyte nuclei proposed to overcome these limitations, contributing sensitive shape-focused segmentation, accurate nuclear detection, and automated device-independent color assessment, without observer-dependent analysis parameters.

Prediction of prognosis using artificial intelligence-based histopathological image analysis in patients with soft tissue sarcomas

BACKGROUND

Prompt histopathological diagnosis with accuracy is required for soft tissue sarcomas (STSs) which are still challenging. In addition, the advances in artificial intelligence (AI) along with the development of pathology slides digitization may empower the demand for the prediction of behavior of STSs. In this article, we explored the application of deep learning for prediction of prognosis from histopathological images in patients with STS.

METHODS

Our retrospective study included a total of 35 histopathological slides from patients with STS. We trained Inception v3 which is proposed method of convolutional neural network based survivability estimation. F1 score which identify the accuracy and area under the receiver operating characteristic curve (AUC) served as main outcome measures from a 4-fold validation.

RESULTS

The cohort included 35 patients with a mean age of 64 years, and the mean follow-up period was 34 months (2-66 months). Our deep learning method achieved AUC of 0.974 and an accuracy of 91.9% in predicting overall survival. Concerning with the prediction of metastasis-free survival, the accuracy was 84.2% with the AUC of 0.852.

CONCLUSION

AI might be used to help pathologists with accurate prognosis prediction. This study could substantially improve the clinical management of patients with STS.

Applying image features of proximal paracancerous tissues in predicting prognosis of patients with hepatocellular carcinoma

BACKGROUND

Most of the methods using digital pathological image for predicting Hepatocellular carcinoma (HCC) prognosis have not considered paracancerous tissue microenvironment (PTME), which are potentially important for tumour initiation and metastasis. This study aimed to identify roles of image features of PTME in predicting prognosis and tumour recurrence of HCC patients.

METHODS

We collected whole slide images (WSIs) of 146 HCC patients from Sun Yat-sen Memorial Hospital (SYSM dataset). For each WSI, five types of regions of interests (ROIs) in PTME and tumours were manually annotated. These ROIs were used to construct a Lasso Cox survival model for predicting the prognosis of HCC patients. To make the model broadly useful, we established a deep learning method to automatically segment WSIs, and further used it to construct a prognosis prediction model. This model was tested by the samples of 225 HCC patients from the Cancer Genome Atlas Liver Hepatocellular Carcinoma (TCGA-LIHC).

RESULTS

In predicting prognosis of the HCC patients, using the image features of manually annotated ROIs in PTME achieved C-index 0.668 in the SYSM testing dataset, which is higher than the C-index 0.648 reached by the model only using image features of tumours. Integrating ROIs of PTME and tumours achieved C-index 0.693 in the SYSM testing dataset. The model using automatically segmented ROIs of PTME and tumours achieved C-index of 0.665 (95% CI: 0.556-0.774) in the TCGA-LIHC samples, which is better than the widely used methods, WSISA (0.567), DeepGraphSurv (0.593), and SeTranSurv (0.642). Finally, we found the Texture SumAverage Skew HV on immune cell infiltration and Texture related features on desmoplastic reaction are the most important features of PTME in predicting HCC prognosis. We additionally used the model in prediction HCC recurrence for patients from SYSM-training, SYSM-testing, and TCGA-LIHC datasets, indicating the important roles of PTME in the prediction.

CONCLUSIONS

Our results indicate image features of PTME is critical for improving the prognosis prediction of HCC. Moreover, the image features related with immune cell infiltration and desmoplastic reaction of PTME are the most important factors associated with prognosis of HCC.

A machine learning algorithm improves the diagnostic accuracy of the histologic component of antibody mediated rejection (AMR-H) in cardiac transplant endomyocardial biopsies

BACKGROUND

Pathologic antibody mediated rejection (pAMR) remains a major driver of graft failure in cardiac transplant patients. The endomyocardial biopsy remains the primary diagnostic tool but presents with challenges, particularly in distinguishing the histologic component (pAMR-H) defined by 1) intravascular macrophage accumulation in capillaries and 2) activated endothelial cells that expand the cytoplasm to narrow or occlude the vascular lumen. Frequently, pAMR-H is difficult to distinguish from acute cellular rejection (ACR) and healing injury. With the advent of digital slide scanning and advances in machine deep learning, artificial intelligence technology is widely under investigation in the areas of oncologic pathology, but in its infancy in transplant pathology. For the first time, we determined if a machine learning algorithm could distinguish pAMR-H from normal myocardium, healing injury and ACR.

MATERIALS AND METHODS

A total of 4,212 annotations (1,053 regions of normal, 1,053 pAMR-H, 1,053 healing injury and 1,053 ACR) were completed from 300 hematoxylin and eosin slides scanned using a Leica Aperio GT450 digital whole slide scanner at 40X magnification. All regions of pAMR-H were annotated from patients confirmed with a previous diagnosis of pAMR2 (>50% positive C4d immunofluorescence and/or >10% CD68 positive intravascular macrophages). Annotations were imported into a Python 3.7 development environment using the OpenSlide™ package and a convolutional neural network approach utilizing transfer learning was performed.

RESULTS

The machine learning algorithm showed 98% overall validation accuracy and pAMR-H was correctly distinguished from specific categories with the following accuracies: normal myocardium (99.2%), healing injury (99.5%) and ACR (99.5%).

CONCLUSION

Our novel deep learning algorithm can reach acceptable, and possibly surpass, performance of current diagnostic standards of identifying pAMR-H. Such a tool may serve as an adjunct diagnostic aid for improving the pathologist's accuracy and reproducibility, especially in difficult cases with high inter-observer variability. This is one of the first studies that provides evidence that an artificial intelligence machine learning algorithm can be trained and validated to diagnose pAMR-H in cardiac transplant patients. Ongoing studies include multi-institutional verification testing to ensure generalizability.

Automatic brain-tumor diagnosis using cascaded deep convolutional neural networks with symmetric U-Net and asymmetric residual-blocks

The use of various kinds of magnetic resonance imaging (MRI) techniques for examining brain tissue has increased significantly in recent years, and manual investigation of each of the resulting images can be a time-consuming task. This paper presents an automatic brain-tumor diagnosis system that uses a CNN for detection, classification, and segmentation of glioblastomas; the latter stage seeks to segment tumors inside glioma MRI images. The structure of the developed multi-unit system consists of two stages. The first stage is responsible for tumor detection and classification by categorizing brain MRI images into normal, high-grade glioma (glioblastoma), and low-grade glioma. The uniqueness of the proposed network lies in its use of different levels of features, including local and global paths. The second stage is responsible for tumor segmentation, and skip connections and residual units are used during this step. Using 1800 images extracted from the BraTS 2017 dataset, the detection and classification stage was found to achieve a maximum accuracy of 99%. The segmentation stage was then evaluated using the Dice score, specificity, and sensitivity. The results showed that the suggested deep-learning-based system ranks highest among a variety of different strategies reported in the literature.

WITHDRAWN: Histomorphometric Image Classifier of Different Grades of Oral Squamous Cell Carcinoma Using Transfer Learning and Convolutional Neural Network

This article has been withdrawn at the request of the authors. The Publisher apologizes for any inconvenience this may cause. The full Elsevier Policy on Article Withdrawal can be found at https://www.elsevier.com/about/policies/article-withdrawal

Using Deep Learning to Automate Eosinophil Counting in Pediatric Ulcerative Colitis Histopathological Images

Background

Accurate identification of inflammatory cells from mucosal histopathology images is important in diagnosing ulcerative colitis. The identification of eosinophils in the colonic mucosa has been associated with disease course. Cell counting is not only time-consuming but can also be subjective to human biases. In this study we developed an automatic eosinophilic cell counting tool from mucosal histopathology images, using deep learning.

Method

Four pediatric IBD pathologists from two North American pediatric hospitals annotated 530 crops from 143 standard-of-care hematoxylin and eosin (H & E) rectal mucosal biopsies. A 305/75 split was used for training/validation to develop and optimize a U-Net based deep learning model, and 150 crops were used as a test set. The U-Net model was then compared to SAU-Net, a state-of-the-art U-Net variant. We undertook post-processing steps, namely, (1) the pixel-level probability threshold, (2) the minimum number of clustered pixels to designate a cell, and (3) the connectivity. Experiments were run to optimize model parameters using AUROC and cross-entropy loss as the performance metrics.

Results

The F1-score was 0.86 (95%CI:0.79-0.91) (Precision: 0.77 (95%CI:0.70-0.83), Recall: 0.96 (95%CI:0.93-0.99)) to identify eosinophils as compared to an F1-score of 0.2 (95%CI:0.13-0.26) for SAU-Net (Precision: 0.38 (95%CI:0.31-0.46), Recall: 0.13 (95%CI:0.08-0.19)). The inter-rater reliability was 0.96 (95%CI:0.93-0.97). The correlation between two pathologists and the algorithm was 0.89 (95%CI:0.82-0.94) and 0.88 (95%CI:0.80-0.94) respectively.

Conclusion

Our results indicate that deep learning-based automated eosinophilic cell counting can obtain a robust level of accuracy with a high degree of concordance with manual expert annotations.

Social network analysis of cell networks improves deep learning for prediction of molecular pathways and key mutations in colorectal cancer

Colorectal cancer (CRC) is a primary global health concern, and identifying the molecular pathways, genetic subtypes, and mutations associated with CRC is crucial for precision medicine. However, traditional measurement techniques such as gene sequencing are costly and time-consuming, while most deep learning methods proposed for this task lack interpretability. This study offers a new approach to enhance the state-of-the-art deep learning methods for molecular pathways and key mutation prediction by incorporating cell network information. We build cell graphs with nuclei as nodes and nuclei connections as edges of the network and leverage Social Network Analysis (SNA) measures to extract abstract, perceivable, and interpretable features that explicitly describe the cell network characteristics in an image. Our approach does not rely on precise nuclei segmentation or feature extraction, is computationally efficient, and is easily scalable. In this study, we utilize the TCGA-CRC-DX dataset, comprising 499 patients and 502 diagnostic slides from primary colorectal tumours, sourced from 36 distinct medical centres in the United States. By incorporating the SNA features alongside deep features in two multiple instance learning frameworks, we demonstrate improved performance for chromosomal instability (CIN), hypermutated tumour (HM), TP53 gene, BRAF gene, and Microsatellite instability (MSI) status prediction tasks (2.4%-4% and 7-8.8% improvement in AUROC and AUPRC on average). Additionally, our method achieves outstanding performance on MSI prediction in an external PAIP dataset (99% AUROC and 98% AUPRC), demonstrating its generalizability. Our findings highlight the discrimination power of SNA features and how they can be beneficial to deep learning models' performance and provide insights into the correlation of cell network profiles with molecular pathways and key mutations.

Lusca: FIJI (ImageJ) based tool for automated morphological analysis of cellular and subcellular structures

The human body consists of diverse subcellular, cellular and supracellular structures. Neurons possess varying-sized projections that interact with different cellular structures leading to the development of highly complex morphologies. Aiming to enhance image analysis of complex biological forms including neurons using available FIJI (ImageJ) plugins, Lusca, an advanced open-source tool, was developed. Lusca utilizes machine learning for image segmentation with intensity and size thresholds. It performs particle analysis to ascertain parameters such as area/volume, quantity, and intensity, in addition to skeletonization for determining length, branching, and width. Moreover, in conjunction with colocalization measurements, it provides an extensive set of 29 morphometric parameters for both 2D and 3D analysis. This is a significant enhancement compared to other scripts that offer only 5-15 parameters. Consequently, it ensures quicker and more precise quantification by effectively eliminating noise and discerning subtle details. With three times larger execution speed, fewer false positive and negative results, and the capacity to measure various parameters, Lusca surpasses other existing open-source solutions. Its implementation of machine learning-based segmentation facilitates versatile applications for different cell types and biological structures, including mitochondria, fibres, and vessels. Lusca's automated and precise measurement capability makes it an ideal choice for diverse biological image analyses.

An automatic diagnosis model of otitis media with high accuracy rate using transfer learning

Introduction: Chronic Suppurative Otitis Media (CSOM) and Middle Ear Cholesteatoma are two common chronic otitis media diseases that often cause confusion among physicians due to their similar location and shape in clinical CT images of the internal auditory canal. In this study, we utilized the transfer learning method combined with CT scans of the internal auditory canal to achieve accurate lesion segmentation and automatic diagnosis for patients with CSOM and middle ear cholesteatoma. Methods: We collected 1019 CT scan images and utilized the nnUnet skeleton model along with coarse grained focal segmentation labeling to pre-train on the above CT images for focal segmentation. We then fine-tuned the pre-training model for the downstream three-classification diagnosis task. Results: Our proposed algorithm model achieved a classification accuracy of 92.33% for CSOM and middle ear cholesteatoma, which is approximately 5% higher than the benchmark model. Moreover, our upstream segmentation task training resulted in a mean Intersection of Union (mIoU) of 0.569. Discussion: Our results demonstrate that using coarse-grained contour boundary labeling can significantly enhance the accuracy of downstream classification tasks. The combination of deep learning and automatic diagnosis of CSOM and internal auditory canal CT images of middle ear cholesteatoma exhibits high sensitivity and specificity.

A lightweight xAI approach to cervical cancer classification

Cervical cancer is caused in the vast majority of cases by the human papilloma virus (HPV) through sexual contact and requires a specific molecular-based analysis to be detected. As an HPV vaccine is available, the incidence of cervical cancer is up to ten times higher in areas without adequate healthcare resources. In recent years, liquid cytology has been used to overcome these shortcomings and perform mass screening. In addition, classifiers based on convolutional neural networks can be developed to help pathologists diagnose the disease. However, these systems always require the final verification of a pathologist to make a final diagnosis. For this reason, explainable AI techniques are required to highlight the most significant data to the healthcare professional, as it can be used to determine the confidence in the results and the areas of the image used for classification (allowing the professional to point out the areas he/she thinks are most important and cross-check them against those detected by the system in order to create incremental learning systems). In this work, a 4-phase optimization process is used to obtain a custom deep-learning classifier for distinguishing between 4 severity classes of cervical cancer with liquid-cytology images. The final classifier obtains an accuracy over 97% for 4 classes and 100% for 2 classes with execution times under 1 s (including the final report generation). Compared to previous works, the proposed classifier obtains better accuracy results with a lower computational cost.

An interpretable machine learning system for colorectal cancer diagnosis from pathology slides

Considering the profound transformation affecting pathology practice, we aimed to develop a scalable artificial intelligence (AI) system to diagnose colorectal cancer from whole-slide images (WSI). For this, we propose a deep learning (DL) system that learns from weak labels, a sampling strategy that reduces the number of training samples by a factor of six without compromising performance, an approach to leverage a small subset of fully annotated samples, and a prototype with explainable predictions, active learning features and parallelisation. Noting some problems in the literature, this study is conducted with one of the largest WSI colorectal samples dataset with approximately 10,500 WSIs. Of these samples, 900 are testing samples. Furthermore, the robustness of the proposed method is assessed with two additional external datasets (TCGA and PAIP) and a dataset of samples collected directly from the proposed prototype. Our proposed method predicts, for the patch-based tiles, a class based on the severity of the dysplasia and uses that information to classify the whole slide. It is trained with an interpretable mixed-supervision scheme to leverage the domain knowledge introduced by pathologists through spatial annotations. The mixed-supervision scheme allowed for an intelligent sampling strategy effectively evaluated in several different scenarios without compromising the performance. On the internal dataset, the method shows an accuracy of 93.44% and a sensitivity between positive (low-grade and high-grade dysplasia) and non-neoplastic samples of 0.996. On the external test samples varied with TCGA being the most challenging dataset with an overall accuracy of 84.91% and a sensitivity of 0.996.

Clinical application of machine learning-based pathomics signature of gastric atrophy

Background

The diagnosis of gastric atrophy is highly subjective, and we aimed to establish a model of gastric atrophy based on pathological features to improve diagnostic consistency.

Methods

We retrospectively collected the HE-stained pathological slides of gastric biopsies and used CellProfiler software for image segmentation and feature extraction of ten representative images for each sample. Subsequently, we employed the Least absolute shrinkage and selection operator (LASSO) to select features and different machine learning (ML) algorithms to construct the diagnostic models for gastric atrophy.

Results

We selected 289 gastric biopsy specimens for training, testing, and external validation. We extracted 464 pathological features and screened ten features by LASSO to establish the diagnostic model for moderate-to-severe atrophy. The range of area under the curve (AUC) for various machine learning algorithms was 0.835-1.000 in the training set, 0.786-0.949 in the testing set, and 0.689-0.818 in the external validation set. LR model had the highest AUC value, with 0.900 (95% CI: 0.852-0.947) in the training set, 0.901 (95% CI: 0.807-0.996) in the testing set, and 0.818 (95% CI: 0.714-0.923) in the external validation set. The atrophy pathological score based on the LR model was associated with endoscopic atrophy grading (Z=-2.478, P=0.013) and gastric cancer (GC) (OR=5.70, 95% CI: 2.63-12.33, P<0.001).

Conclusion

The ML model based on pathological features could improve the diagnostic consistency of gastric atrophy, which is also associated with endoscopic atrophy grading and GC.

Virtual histological staining of unlabeled autopsy tissue

Traditional histochemical staining of post-mortem samples often confronts inferior staining quality due to autolysis caused by delayed fixation of cadaver tissue, and such chemical staining procedures covering large tissue areas demand substantial labor, cost and time. Here, we demonstrate virtual staining of autopsy tissue using a trained neural network to rapidly transform autofluorescence images of label-free autopsy tissue sections into brightfield equivalent images, matching hematoxylin and eosin (H&E) stained versions of the same samples. The trained model can effectively accentuate nuclear, cytoplasmic and extracellular features in new autopsy tissue samples that experienced severe autolysis, such as COVID-19 samples never seen before, where the traditional histochemical staining fails to provide consistent staining quality. This virtual autopsy staining technique provides a rapid and resource-efficient solution to generate artifact-free H&E stains despite severe autolysis and cell death, also reducing labor, cost and infrastructure requirements associated with the standard histochemical staining.

Applications of machine learning in time-domain fluorescence lifetime imaging: a review

Many medical imaging modalities have benefited from recent advances in Machine Learning (ML), specifically in deep learning, such as neural networks. Computers can be trained to investigate and enhance medical imaging methods without using valuable human resources. In recent years, Fluorescence Lifetime Imaging (FLIm) has received increasing attention from the ML community. FLIm goes beyond conventional spectral imaging, providing additional lifetime information, and could lead to optical histopathology supporting real-time diagnostics. However, most current studies do not use the full potential of machine/deep learning models. As a developing image modality, FLIm data are not easily obtainable, which, coupled with an absence of standardisation, is pushing back the research to develop models which could advance automated diagnosis and help promote FLIm. In this paper, we describe recent developments that improve FLIm image quality, specifically time-domain systems, and we summarise sensing, signal-to-noise analysis and the advances in registration and low-level tracking. We review the two main applications of ML for FLIm: lifetime estimation and image analysis through classification and segmentation. We suggest a course of action to improve the quality of ML studies applied to FLIm. Our final goal is to promote FLIm and attract more ML practitioners to explore the potential of lifetime imaging.

Algorithm-assisted diagnosis of Hirschsprung's disease - evaluation of robustness and comparative image analysis on data from various labs and slide scanners

BACKGROUND

Differences in the preparation, staining and scanning of digital pathology slides create significant pre-analytic variability. Algorithm-assisted tools must be able to contend with this variability in order to be applicable in clinical practice. In a previous study, a decision support algorithm was developed to assist in the diagnosis of Hirschsprung's disease. In the current study, we tested the robustness of this algorithm while assessing for pre-analytic factors which may affect its performance.

METHODS

The decision support algorithm was used on digital pathology slides obtained from four different medical centers (A-D) and scanned by three different scanner models (by Philips, Hamamatsu and 3DHISTECH). A total of 192 cases and 1782 slides were used in this study. RGB histograms were constructed to compare images from the various medical centers and scanner models and highlight the differences in color and contrast.

RESULTS

The algorithm was able to correctly identify ganglion cells in 99.2% of cases, from all medical centers (All scanned by the Philips slide scanner) as well as 95.5% and 100% of the slides scanned by the 3DHISTECH and Hamamatsu brand slide scanners, respectively. The total error rate for center D was lower than the other medical centers (3.9% vs 7.1%, 10.8% and 6% for centers A-C, respectively), the vast majority of errors being false positives (3.45% vs 0.45% false negatives). The other medical centers showed a higher rate of false negatives in relation to false positives (6.81% vs 0.29%, 9.8% vs 1.2% and 5.37% vs 0.63% for centers A-C, respectively). The total error rates for the Philips, Hamamatsu and 3DHISTECH brand scanners were 3.9%, 3.2% and 9.8%, respectively. RGB histograms demonstrated significant differences in pixel value distribution between the four medical centers, as well as between the 3DHISTECH brand scanner when compared to the Philips and Hamamatsu brand scanners.

CONCLUSIONS

The results reported in this paper suggest that the algorithm-based decision support system has sufficient robustness to be applicable for clinical practice. In addition, the novel method used in its development - Hierarchial-Contexual Analysis (HCA) may be applicable to the development of algorithm-assisted tools in other diseases, for which available datasets are limited. Validation of any given algorithm-assisted support system should nonetheless include data from as many medical centers and scanner models as possible.

The effect of deep learning-based lesion segmentation on failure load calculations of metastatic femurs using finite element analysis

Bone ranks as the third most frequent tissue affected by cancer metastases, following the lung and liver. Bone metastases are often painful and may result in pathological fracture, which is a major cause of morbidity and mortality in cancer patients. To quantify fracture risk, finite element (FE) analysis has shown to be a promising tool, but metastatic lesions are typically not specifically segmented and therefore their mechanical properties may not be represented adequately. Deep learning methods potentially provide the opportunity to automatically segment these lesions and change the mechanical properties more adequately. In this study, our primary focus was to gain insight into the performance of an automatic segmentation algorithm for femoral metastatic lesions using deep learning methods and the subsequent effects on FE outcomes. The aims were to determine the similarity between manual segmentation and automatic segmentation; the differences in predicted failure load between FE models with automatically segmented osteolytic and mixed lesions and the models with CT-based lesion values (the gold standard); and the effect on the BOne Strength (BOS) score (failure load adjusted for body weight) and subsequent fracture risk assessments. From two patient cohorts, a total number of 50 femurs with osteolytic and mixed metastatic lesions were included in this study. The femurs were segmented from CT images and transferred into FE meshes. The material behavior was implemented as non-linear isotropic. These FE models were considered as gold standard (Finite Element no Segmented Lesion: FE-no-SL), whereby the local calcium equivalent density of both femur and metastatic lesion was extracted from CT-values. Lesions in the femur were manually segmented by two biomechanical experts after which final lesion segmentation for each femur was obtained based on consensus of opinions between two observers. Subsequently, a self-configuring variant of the popular deep learning model U-Net known as nnU-Net was used to automatically segment metastatic lesions within the femur. For these models with segmented lesions (Finite Element with Segmented Lesion: FE-with-SL), the calcium equivalent density within the metastatic lesions was set to zero after being segmented by the neural network, simulating absence of load-bearing capacity of these lesions. The models (either with or without automatically segmented lesions) were loaded incrementally in axial direction until failure was simulated. Dice coefficient was used to evaluate the similarity of the manual and automatic segmentation. Mean calcium equivalent density values within the automatically segmented lesions were calculated. Failure loads and patterns were determined. Furthermore, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for both groups by comparing the predictions to the occurrence or absence of actual fracture within the patient cohorts. The automatic segmentation algorithm performed in a none-robust manner. Dice coefficients describing the similarity between consented manual and automatic segmentations were relatively low (mean 0.45 ± standard deviation 0.33, median 0.54). Failure load difference between the FE-no-SL and FE-with-SL groups varied from 0 % to 48 % (mean 6.6 %). Correlation analysis of failure loads between the two groups showed a strong relationship (R2 > 0.9). From the 50 cases, four cases showed clear deviations for which models with automatic lesion segmentation (FE-with-SL) showed considerably lower failure loads. In the whole database including osteolytic and mixed lesions, sensitivity and NPV remained the same, but specificity and PPV decreased from 94 % to 83 %, and from 78 % to 54 % respectively from FE-no-SL to FE-with-SL. This study indicates that the nnU-Net yielded none-robust outcomes in femoral lesion segmentation and that other segmentation algorithms should be considered. However, the difference in failure pattern and failure load between FE models with automatically segmented osteolytic and mixed lesions were relatively small in most cases with a few exceptions. On the other hand, the accuracy of fracture risk assessment using the BOS score was lower compared to the FE-no-SL. In conclusion, this study showed that automatic lesion segmentation is a none-solved issue and therefore, quantifying lesion characteristics and the subsequent effect on the fracture risk using deep learning will remain challenging.

Emerging Landscape of Targeted Therapy of Breast Cancers With Low Human Epidermal Growth Factor Receptor 2 Protein Expression

CONTEXT.—

Human epidermal growth factor receptor 2 (HER2) status in breast cancer is currently classified as negative or positive for selecting patients for anti-HER2 targeted therapy. The evolution of the HER2 status has included a new HER2-low category defined as an HER2 immunohistochemistry score of 1+ or 2+ without gene amplification. This new category opens the door to a targetable HER2-low breast cancer population for which new treatments may be effective.

OBJECTIVE.—

To review the current literature on the emerging category of breast cancers with low HER2 protein expression, including the clinical, histopathologic, and molecular features, and outline the clinical trials and best practice recommendations for identifying HER2-low-expressing breast cancers by immunohistochemistry.

DATA SOURCES.—

We conducted a literature review based on peer-reviewed original articles, review articles, regulatory communications, ongoing and past clinical trials identified through ClinicalTrials.gov, and the authors' practice experience.

CONCLUSIONS.—

The availability of new targeted therapy potentially effective for patients with breast cancers with low HER2 protein expression requires multidisciplinary recognition. In particular, pathologists need to recognize and identify this category to allow the optimal selection of patients for targeted therapy.

A Weakly Supervised Segmentation Network Embedding Cross-Scale Attention Guidance and Noise-Sensitive Constraint for Detecting Tertiary Lymphoid Structures of Pancreatic Tumors

The presence of tertiary lymphoid structures (TLSs) on pancreatic pathological images is an important prognostic indicator of pancreatic tumors. Therefore, TLSs detection on pancreatic pathological images plays a crucial role in diagnosis and treatment for patients with pancreatic tumors. However, fully supervised detection algorithms based on deep learning usually require a large number of manual annotations, which is time-consuming and labor-intensive. In this paper, we aim to detect the TLSs in a manner of few-shot learning by proposing a weakly supervised segmentation network. We firstly obtain the lymphocyte density maps by combining a pretrained model for nuclei segmentation and a domain adversarial network for lymphocyte nuclei recognition. Then, we establish a cross-scale attention guidance mechanism by jointly learning the coarse-scale features from the original histopathology images and fine-scale features from our designed lymphocyte density attention. A noise-sensitive constraint is introduced by an embedding signed distance function loss in the training procedure to reduce tiny prediction errors. Experimental results on two collected datasets demonstrate that our proposed method significantly outperforms the state-of-the-art segmentation-based algorithms in terms of TLSs detection accuracy. Additionally, we apply our method to study the congruent relationship between the density of TLSs and peripancreatic vascular invasion and obtain some clinically statistical results.

Digital image analysis and machine learning-assisted prediction of neoadjuvant chemotherapy response in triple-negative breast cancer

BACKGROUND

Pathological complete response (pCR) is associated with favorable prognosis in patients with triple-negative breast cancer (TNBC). However, only 30-40% of TNBC patients treated with neoadjuvant chemotherapy (NAC) show pCR, while the remaining 60-70% show residual disease (RD). The role of the tumor microenvironment in NAC response in patients with TNBC remains unclear. In this study, we developed a machine learning-based two-step pipeline to distinguish between various histological components in hematoxylin and eosin (H&E)-stained whole slide images (WSIs) of TNBC tissue biopsies and to identify histological features that can predict NAC response.

METHODS

H&E-stained WSIs of treatment-naïve biopsies from 85 patients (51 with pCR and 34 with RD) of the model development cohort and 79 patients (41 with pCR and 38 with RD) of the validation cohort were separated through a stratified eightfold cross-validation strategy for the first step and leave-one-out cross-validation strategy for the second step. A tile-level histology label prediction pipeline and four machine-learning classifiers were used to analyze 468,043 tiles of WSIs. The best-trained classifier used 55 texture features from each tile to produce a probability profile during testing. The predicted histology classes were used to generate a histology classification map of the spatial distributions of different tissue regions. A patient-level NAC response prediction pipeline was trained with features derived from paired histology classification maps. The top graph-based features capturing the relevant spatial information across the different histological classes were provided to the radial basis function kernel support vector machine (rbfSVM) classifier for NAC treatment response prediction.

RESULTS

The tile-level prediction pipeline achieved 86.72% accuracy for histology class classification, while the patient-level pipeline achieved 83.53% NAC response (pCR vs. RD) prediction accuracy of the model development cohort. The model was validated with an independent cohort with tile histology validation accuracy of 83.59% and NAC prediction accuracy of 81.01%. The histological class pairs with the strongest NAC response predictive ability were tumor and tumor tumor-infiltrating lymphocytes for pCR and microvessel density and polyploid giant cancer cells for RD.

CONCLUSION

Our machine learning pipeline can robustly identify clinically relevant histological classes that predict NAC response in TNBC patients and may help guide patient selection for NAC treatment.

AI improves accuracy, agreement and efficiency of pathologists for Ki67 assessments in breast cancer

The Ki-67 proliferation index (PI) guides treatment decisions in breast cancer but suffers from poor inter-rater reproducibility. Although AI tools have been designed for Ki-67 assessment, their impact on pathologists' work remains understudied. 90 international pathologists were recruited to assess the Ki-67 PI of ten breast cancer tissue microarrays with and without AI. Accuracy, agreement, and turnaround time with and without AI were compared. Pathologists' perspectives on AI were collected. Using AI led to a significant decrease in PI error (2.1% with AI vs. 5.9% without AI, p < 0.001), better inter-rater agreement (ICC: 0.70 vs. 0.92; Krippendorff's α: 0.63 vs. 0.89; Fleiss' Kappa: 0.40 vs. 0.86), and an 11.9% overall median reduction in turnaround time. Most pathologists (84%) found the AI reliable. For Ki-67 assessments, 76% of respondents believed AI enhances accuracy, 82% said it improves consistency, and 83% trust it will improve efficiency. This study highlights AI's potential to standardize Ki-67 scoring, especially between 5 and 30% PI-a range with low PI agreement. This could pave the way for a universally accepted PI score to guide treatment decisions, emphasizing the promising role of AI integration into pathologist workflows.

Use of deep learning for the classification of hyperplastic lymph node and common subtypes of canine lymphomas: a preliminary study

Artificial Intelligence has observed significant growth in its ability to classify different types of tumors in humans due to advancements in digital pathology technology. Among these tumors, lymphomas are quite common in dogs, despite studies on the application of AI in domestic species are scarce. This research aims to employ deep learning (DL) through convolutional neural networks (CNNs) to distinguish between normal lymph nodes and 3 WHO common subtypes of canine lymphomas. To train and validate the CNN, 1,530 high-resolution microscopic images derived from whole slide scans (WSIs) were used, including those of background areas, hyperplastic lymph nodes (n = 4), and three different lymphoma subtypes: diffuse large B cell lymphoma (DLBCL; n = 5), lymphoblastic (LBL; n = 5), and marginal zone lymphoma (MZL; n = 3). The CNN was able to correctly identify 456 images of the possible 457 test sets, achieving a maximum accuracy of 99.34%. The results of this study have demonstrated the feasibility of using deep learning to differentiate between hyperplastic lymph nodes and lymphomas, as well as to classify common WHO subtypes. Further research is required to explore the implications of these findings and validate the ability of the network to classify a broader range of lymphomas.

Cancer immunotherapy efficacy and machine learning

INTRODUCTION

Immunotherapy is one of the major breakthroughs in the treatment of cancer, and it has become a powerful clinical strategy, however, not all patients respond to immune checkpoint blockade and other immunotherapy strategies. Applying machine learning (ML) techniques to predict the efficacy of cancer immunotherapy is useful for clinical decision-making.

AREAS COVERED

Applying ML including deep learning (DL) in radiomics, pathomics, tumor microenvironment (TME) and immune-related genes analysis to predict immunotherapy efficacy. The studies in this review were searched from PubMed and ClinicalTrials.gov (January 2023).

EXPERT OPINION

An increasing number of studies indicate that ML has been applied to various aspects of oncology research, with the potential to provide more effective individualized immunotherapy strategies and enhance treatment decisions. With advances in ML technology, more efficient methods of predicting the efficacy of immunotherapy may become available in the future.

Artificial Intelligence in Predicting Microsatellite Instability and KRAS, BRAF Mutations from Whole-Slide Images in Colorectal Cancer: A Systematic Review

Mismatch repair deficiency (d-MMR)/microsatellite instability (MSI), KRAS, and BRAF mutational status are crucial for treating advanced colorectal cancer patients. Traditional methods like immunohistochemistry or polymerase chain reaction (PCR) can be challenged by artificial intelligence (AI) based on whole slide images (WSI) to predict tumor status. In this systematic review, we evaluated the role of AI in predicting MSI status, KRAS, and BRAF mutations in colorectal cancer. Studies published in PubMed up to June 2023 were included (n = 17), and we reported the risk of bias and the performance for each study. Some studies were impacted by the reduced number of slides included in the data set and the lack of external validation cohorts. Deep learning models for the d-MMR/MSI status showed a good performance in training cohorts (mean AUC = 0.89, [0.74-0.97]) but slightly less than expected in the validation cohort when available (mean AUC = 0.82, [0.63-0.98]). Contrary to the MSI status, the prediction of KRAS and BRAF mutations was less explored with a less robust methodology. The performance was lower, with a maximum of 0.77 in the training cohort, 0.58 in the validation cohort for KRAS, and 0.82 AUC in the training cohort for BRAF.

Survey of liver pathologists to assess attitudes towards digital pathology and artificial intelligence

AIMS

A survey of members of the UK Liver Pathology Group (UKLPG) was conducted, comprising consultant histopathologists from across the UK who report liver specimens and participate in the UK National Liver Pathology External Quality Assurance scheme. The aim of this study was to understand attitudes and priorities of liver pathologists towards digital pathology and artificial intelligence (AI).

METHODS

The survey was distributed to all full consultant members of the UKLPG via email. This comprised 50 questions, with 48 multiple choice questions and 2 free-text questions at the end, covering a range of topics and concepts pertaining to the use of digital pathology and AI in liver disease.

RESULTS

Forty-two consultant histopathologists completed the survey, representing 36% of fully registered members of the UKLPG (42/116). Questions examining digital pathology showed respondents agreed with the utility of digital pathology for primary diagnosis 83% (34/41), second opinions 90% (37/41), research 85% (35/41) and training and education 95% (39/41). Fatty liver diseases were an area of demand for AI tools with 80% in agreement (33/41), followed by neoplastic liver diseases with 59% in agreement (24/41). Participants were concerned about AI development without pathologist involvement 73% (30/41), however, 63% (26/41) disagreed when asked whether AI would replace pathologists.

CONCLUSIONS

This study outlines current interest, priorities for research and concerns around digital pathology and AI for liver pathologists. The majority of UK liver pathologists are in favour of the application of digital pathology and AI in clinical practice, research and education.

National resident survey in dermatopathology: The role of slide scanners in resident learning

BACKGROUND

Dermatology residents gain exposure to dermatopathology through a variety of educational modalities. While virtual pathology applications have risen dramatically, resident utilization of digital libraries, slide scanner availability, and comfort with virtual slides are not well-known. This study aims to assess the current landscape of educational resources used by dermatology residents.

METHODS

A 17-question survey was sent to dermatology residents through a national email database. The survey was a self-assessment of their experience in dermatopathology education and the use of departmental slide scanners.

RESULTS

The use of digital dermatopathology is high among trainees, despite only half of respondents reporting slide scanner access. Residents report using virtual images more often in non-clinical dermatopathology didactics and independent studies compared to clinical dermatopathology rotations. Public slide set use was common, while professional society and departmental slide sets may be underutilized. Over half of respondents report being extremely or very comfortable navigating interactive scanned slides.

CONCLUSIONS

Survey data suggests digital slides are currently predominantly used in non-clinical dermatopathology rotations and independent studies. Incorporation of slide scanners into departments may benefit resident education through the development of high-quality, curated departmental slide sets.

Learning with limited target data to detect cells in cross-modality images

Deep neural networks have achieved excellent cell or nucleus quantification performance in microscopy images, but they often suffer from performance degradation when applied to cross-modality imaging data. Unsupervised domain adaptation (UDA) based on generative adversarial networks (GANs) has recently improved the performance of cross-modality medical image quantification. However, current GAN-based UDA methods typically require abundant target data for model training, which is often very expensive or even impossible to obtain for real applications. In this paper, we study a more realistic yet challenging UDA situation, where (unlabeled) target training data is limited and previous work seldom delves into cell identification. We first enhance a dual GAN with task-specific modeling, which provides additional supervision signals to assist with generator learning. We explore both single-directional and bidirectional task-augmented GANs for domain adaptation. Then, we further improve the GAN by introducing a differentiable, stochastic data augmentation module to explicitly reduce discriminator overfitting. We examine source-, target-, and dual-domain data augmentation for GAN enhancement, as well as joint task and data augmentation in a unified GAN-based UDA framework. We evaluate the framework for cell detection on multiple public and in-house microscopy image datasets, which are acquired with different imaging modalities, staining protocols and/or tissue preparations. The experiments demonstrate that our method significantly boosts performance when compared with the reference baseline, and it is superior to or on par with fully supervised models that are trained with real target annotations. In addition, our method outperforms recent state-of-the-art UDA approaches by a large margin on different datasets.

Standardized Classification of Lung Adenocarcinoma Subtypes and Improvement of Grading Assessment Through Deep Learning

The histopathologic distinction of lung adenocarcinoma (LADC) subtypes is subject to high interobserver variability, which can compromise the optimal assessment of patient prognosis. Therefore, this study developed convolutional neural networks capable of distinguishing LADC subtypes and predicting disease-specific survival, according to the recently established LADC tumor grades. Consensus LADC histopathologic images were obtained from 17 expert pulmonary pathologists and one pathologist in training. Two deep learning models (AI-1 and AI-2) were trained to predict eight different LADC classes. Furthermore, the trained models were tested on an independent cohort of 133 patients. The models achieved high precision, recall, and F1 scores exceeding 0.90 for most of the LADC classes. Clear stratification of the three LADC grades was reached in predicting the disease-specific survival by the two models, with both Kaplan-Meier curves showing significance (P = 0.0017 and 0.0003). Moreover, both trained models showed high stability in the segmentation of each pair of predicted grades with low variation in the hazard ratio across 200 bootstrapped samples. These findings indicate that the trained convolutional neural networks improve the diagnostic accuracy of the pathologist and refine LADC grade assessment. Thus, the trained models are promising tools that may assist in the routine evaluation of LADC subtypes and grades in clinical practice.

Hourglass, a rapid analysis framework for heterogeneous bioimaging data, identifies sex disparity in IL-6/STAT3-associated immune phenotypes in pancreatic cancer

Integration and mining of bioimaging data remains a challenge and lags behind the rapidly expanding digital pathology field. We introduce Hourglass, an open-access analytical framework that streamlines biology-driven visualization, interrogation, and statistical assessment of multiparametric datasets. Cognizant of tissue and clinical heterogeneity, Hourglass systematically organizes observations across spatial and global levels and within patient subgroups. Applied to an extensive bioimaging dataset, Hourglass promptly consolidated a breadth of known interleukin-6 (IL-6) functions via its downstream effector STAT3 and uncovered a so-far unknown sexual dimorphism in the IL-6/STAT3-linked intratumoral T-cell response in human pancreatic cancer. As an R package and cross-platform application, Hourglass facilitates knowledge extraction from multi-layered bioimaging datasets for users with or without computational proficiency and provides unique and widely accessible analytical means to harness insights hidden within heterogeneous tissues at the sample and patient level. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

Enhancing diagnosis of Hirschsprung's disease using deep learning from histological sections of post pull-through specimens: preliminary results

PURPOSE

Accurate histological diagnosis in Hirschsprung disease (HD) is challenging, due to its complexity and potential for errors. In this study, we present an artificial intelligence (AI)-based method designed to identify ganglionic cells and hypertrophic nerves in HD histology.

METHODS

Formalin-fixed samples were used and an expert pathologist and a surgeon annotated these slides on a web-based platform, identifying ganglionic cells and nerves. Images were partitioned into square sections, augmented through data manipulation techniques and used to develop two distinct U-net models: one for detecting ganglionic cells and normal nerves; the other to recognise hypertrophic nerves.

RESULTS

The study included 108 annotated samples, resulting in 19,600 images after data augmentation and manually segmentation. Subsequently, 17,655 slides without target elements were excluded. The algorithm was trained using 1945 slides (930 for model 1 and 1015 for model 2) with 1556 slides used for training the supervised network and 389 for validation. The accuracy of model 1 was found to be 92.32%, while model 2 achieved an accuracy of 91.5%.

CONCLUSION

The AI-based U-net technique demonstrates robustness in detecting ganglion cells and nerves in HD. The deep learning approach has the potential to standardise and streamline HD diagnosis, benefiting patients and aiding in training of pathologists.

Trends and hotspots in research on medical images with deep learning: a bibliometric analysis from 2013 to 2023

Background

With the rapid development of the internet, the improvement of computer capabilities, and the continuous advancement of algorithms, deep learning has developed rapidly in recent years and has been widely applied in many fields. Previous studies have shown that deep learning has an excellent performance in image processing, and deep learning-based medical image processing may help solve the difficulties faced by traditional medical image processing. This technology has attracted the attention of many scholars in the fields of computer science and medicine. This study mainly summarizes the knowledge structure of deep learning-based medical image processing research through bibliometric analysis and explores the research hotspots and possible development trends in this field.

Methods

Retrieve the Web of Science Core Collection database using the search terms "deep learning," "medical image processing," and their synonyms. Use CiteSpace for visual analysis of authors, institutions, countries, keywords, co-cited references, co-cited authors, and co-cited journals.

Results

The analysis was conducted on 562 highly cited papers retrieved from the database. The trend chart of the annual publication volume shows an upward trend. Pheng-Ann Heng, Hao Chen, and Klaus Hermann Maier-Hein are among the active authors in this field. Chinese Academy of Sciences has the highest number of publications, while the institution with the highest centrality is Stanford University. The United States has the highest number of publications, followed by China. The most frequent keyword is "Deep Learning," and the highest centrality keyword is "Algorithm." The most cited author is Kaiming He, and the author with the highest centrality is Yoshua Bengio.

Conclusion

The application of deep learning in medical image processing is becoming increasingly common, and there are many active authors, institutions, and countries in this field. Current research in medical image processing mainly focuses on deep learning, convolutional neural networks, classification, diagnosis, segmentation, image, algorithm, and artificial intelligence. The research focus and trends are gradually shifting toward more complex and systematic directions, and deep learning technology will continue to play an important role.

Solution-free and simplified H&E staining using a hydrogel-based stamping technology

Hematoxylin and eosin (H&E) staining has been widely used as a fundamental and essential tool for diagnosing diseases and understanding biological phenomena by observing cellular arrangements and tissue morphological changes. However, conventional staining methods commonly involve solution-based, complex, multistep processes that are susceptible to user-handling errors. Moreover, inconsistent staining results owing to staining artifacts pose real challenges for accurate diagnosis. This study introduces a solution-free H&E staining method based on agarose hydrogel patches that is expected to represent a valuable tool to overcome the limitations of the solution-based approach. Using two agarose gel-based hydrogel patches containing hematoxylin and eosin dyes, H&E staining can be performed through serial stamping processes, minimizing color variation from handling errors. This method allows easy adjustments of the staining color by controlling the stamping time, effectively addressing variations in staining results caused by various artifacts, such as tissue processing and thickness. Moreover, the solution-free approach eliminates the need for water, making it applicable even in environmentally limited middle- and low-income countries, while still achieving a staining quality equivalent to that of the conventional method. In summary, this hydrogel-based H&E staining method can be used by researchers and medical professionals in resource-limited settings as a powerful tool to diagnose and understand biological phenomena.