The impact of pre- and post-image processing techniques on deep learning frameworks: A comprehensive review for digital pathology image analysis

Improving generalization capability of deep learning-based nuclei instance segmentation by non-deterministic train time and deterministic test time stain normalization

With the advent of digital pathology and microscopic systems that can scan and save whole slide histological images automatically, there is a growing trend to use computerized methods to analyze acquired images. Among different histopathological image analysis tasks, nuclei instance segmentation plays a fundamental role in a wide range of clinical and research applications. While many semi- and fully-automatic computerized methods have been proposed for nuclei instance segmentation, deep learning (DL)-based approaches have been shown to deliver the best performances. However, the performance of such approaches usually degrades when tested on unseen datasets. In this work, we propose a novel method to improve the generalization capability of a DL-based automatic segmentation approach. Besides utilizing one of the state-of-the-art DL-based models as a baseline, our method incorporates non-deterministic train time and deterministic test time stain normalization, and ensembling to boost the segmentation performance. We trained the model with one single training set and evaluated its segmentation performance on seven test datasets. Our results show that the proposed method provides up to 4.9%, 5.4%, and 5.9% better average performance in segmenting nuclei based on Dice score, aggregated Jaccard index, and panoptic quality score, respectively, compared to the baseline segmentation model.

Publicly available datasets of breast histopathology H&E whole-slide images: A scoping review

Advancements in digital pathology and computing resources have made a significant impact in the field of computational pathology for breast cancer diagnosis and treatment. However, access to high-quality labeled histopathological images of breast cancer is a big challenge that limits the development of accurate and robust deep learning models. In this scoping review, we identified the publicly available datasets of breast H&E-stained whole-slide images (WSIs) that can be used to develop deep learning algorithms. We systematically searched 9 scientific literature databases and 9 research data repositories and found 17 publicly available datasets containing 10 385 H&E WSIs of breast cancer. Moreover, we reported image metadata and characteristics for each dataset to assist researchers in selecting proper datasets for specific tasks in breast cancer computational pathology. In addition, we compiled 2 lists of breast H&E patches and private datasets as supplementary resources for researchers. Notably, only 28% of the included articles utilized multiple datasets, and only 14% used an external validation set, suggesting that the performance of other developed models may be susceptible to overestimation. The TCGA-BRCA was used in 52% of the selected studies. This dataset has a considerable selection bias that can impact the robustness and generalizability of the trained algorithms. There is also a lack of consistent metadata reporting of breast WSI datasets that can be an issue in developing accurate deep learning models, indicating the necessity of establishing explicit guidelines for documenting breast WSI dataset characteristics and metadata.

Automatic semantic segmentation of EHG recordings by deep learning: An approach to a screening tool for use in clinical practice

BACKGROUND AND OBJECTIVE

Preterm delivery is an important factor in the disease burden of the newborn and infants worldwide. Electrohysterography (EHG) has become a promising technique for predicting this condition, thanks to its high degree of sensitivity. Despite the technological progress made in predicting preterm labor, its use in clinical practice is still limited, one of the main barriers being the lack of tools for automatic signal processing without expert supervision, i.e. automatic screening of motion and respiratory artifacts in EHG records. Our main objective was thus to design and validate an automatic system of segmenting and screening the physiological segments of uterine origin in EHG records for robust characterization of uterine myoelectric activity, predicting preterm labor and help to promote the transferability of the EHG technique to clinical practice.

METHODS

For this, we combined 300 EHG recordings from the TPEHG DS database and 69 EHG recordings from our own database (Ci2B-La Fe) of women with singleton gestations. This dataset was used to train and evaluate U-Net, U-Net++, and U-Net 3+ for semantic segmentation of the physiological and artifacted segments of EHG signals. The model's predictions were then fine-tuned by post-processing.

RESULTS

U-Net 3+ outperformed the other models, achieving an area under the ROC curve of 91.4 % and an average precision of 96.4 % in detecting physiological activity. Thresholds from 0.6 to 0.8 achieved precision from 93.7 % to 97.4 % and specificity from 81.7 % to 94.5 %, detecting high-quality physiological segments while maintaining a trade-off between recall and specificity. Post-processing improved the model's adaptability by fine-tuning both the physiological and corrupted segments, ensuring accurate artifact detection while maintaining physiological segment integrity in EHG signals.

CONCLUSIONS

As automatic segmentation proved to be as effective as double-blind manual segmentation in predicting preterm labor, this automatic segmentation tool fills a crucial gap in the existing preterm delivery prediction system workflow by eliminating the need for double-blind segmentation by experts and facilitates the practical clinical use of EHG. This work potentially contributes to the early detection of authentic preterm labor women and will allow clinicians to design individual patient strategies for maternal health surveillance systems and predict adverse pregnancy outcomes.

Factor-GAN: Enhancing stock price prediction and factor investment with Generative Adversarial Networks

Deep learning, a pivotal branch of artificial intelligence, has increasingly influenced the financial domain with its advanced data processing capabilities. This paper introduces Factor-GAN, an innovative framework that utilizes Generative Adversarial Networks (GAN) technology for factor investing. Leveraging a comprehensive factor database comprising 70 firm characteristics, Factor-GAN integrates deep learning techniques with the multi-factor pricing model, thereby elevating the precision and stability of investment strategies. To explain the economic mechanisms underlying deep learning, we conduct a subsample analysis of the Chinese stock market. The findings reveal that the deep learning-based pricing model significantly enhances return prediction accuracy and factor investment performance in comparison to linear models. Particularly noteworthy is the superior performance of the long-short portfolio under Factor-GAN, demonstrating an annualized return of 23.52% with a Sharpe ratio of 1.29. During the transition from state-owned enterprises (SOEs) to non-SOEs, our study discerns shifts in factor importance, with liquidity and volatility gaining significance while fundamental indicators diminish. Additionally, A-share listed companies display a heightened emphasis on momentum and growth indicators relative to their dual-listed counterparts. This research holds profound implications for the expansion of explainable artificial intelligence research and the exploration of financial technology applications.

All you need is data preparation: A systematic review of image harmonization techniques in Multi-center/device studies for medical support systems

BACKGROUND AND OBJECTIVES

Artificial intelligence (AI) models trained on multi-centric and multi-device studies can provide more robust insights and research findings compared to single-center studies. However, variability in acquisition protocols and equipment can introduce inconsistencies that hamper the effective pooling of multi-source datasets. This systematic review evaluates strategies for image harmonization, which standardizes appearances to enable reliable AI analysis of multi-source medical imaging.

METHODS

A literature search using PRISMA guidelines was conducted to identify relevant papers published between 2013 and 2023 analyzing multi-centric and multi-device medical imaging studies that utilized image harmonization approaches.

RESULTS

Common image harmonization techniques included grayscale normalization (improving classification accuracy by up to 24.42 %), resampling (increasing the percentage of robust radiomics features from 59.5 % to 89.25 %), and color normalization (enhancing AUC by up to 0.25 in external test sets). Initially, mathematical and statistical methods dominated, but machine and deep learning adoption has risen recently. Color imaging modalities like digital pathology and dermatology have remained prominent application areas, though harmonization efforts have expanded to diverse fields including radiology, nuclear medicine, and ultrasound imaging. In all the modalities covered by this review, image harmonization improved AI performance, with increasing of up to 24.42 % in classification accuracy and 47 % in segmentation Dice scores.

CONCLUSIONS

Continued progress in image harmonization represents a promising strategy for advancing healthcare by enabling large-scale, reliable analysis of integrated multi-source datasets using AI. Standardizing imaging data across clinical settings can help realize personalized, evidence-based care supported by data-driven technologies while mitigating biases associated with specific populations or acquisition protocols.

Convolutional neural network with parallel convolution scale attention module and ResCBAM for breast histology image classification

Breast cancer is the most common cause of female morbidity and death worldwide. Compared with other cancers, early detection of breast cancer is more helpful to improve the prognosis of patients. In order to achieve early diagnosis and treatment, clinical treatment requires rapid and accurate diagnosis. Therefore, the development of an automatic detection system for breast cancer suitable for patient imaging is of great significance for assisting clinical treatment. Accurate classification of pathological images plays a key role in computer-aided medical diagnosis and prognosis. However, in the automatic recognition and classification methods of breast cancer pathological images, the scale information, the loss of image information caused by insufficient feature fusion, and the enormous structure of the model may lead to inaccurate or inefficient classification. To minimize the impact, we proposed a lightweight PCSAM-ResCBAM model based on two-stage convolutional neural network. The model included a Parallel Convolution Scale Attention Module network (PCSAM-Net) and a Residual Convolutional Block Attention Module network (ResCBAM-Net). The first-level convolutional network was built through a 4-layer PCSAM module to achieve prediction and classification of patches extracted from images. To optimize the network's ability to represent global features of images, we proposed a tiled feature fusion method to fuse patch features from the same image, and proposed a residual convolutional attention module. Based on the above, the second-level convolutional network was constructed to achieve predictive classification of images. We evaluated the performance of our proposed model on the ICIAR2018 dataset and the BreakHis dataset, respectively. Furthermore, through model ablation studies, we found that scale attention and dilated convolution play an important role in improving model performance. Our proposed model outperforms the existing state-of-the-art models on 200 × and 400 × magnification datasets with a maximum accuracy of 98.74 %.

Eradication of Klebsiella pneumoniae pulmonary infection by silver oxytetracycline nano-structure

Targeted bactericidal nanosystems hold significant promise to improve the efficacy of existing antimicrobials for treatment of severe bacterial infections by minimizing the side effects and lowering the risk of antibiotic resistance development. In this work, Silver Oxytetracycline Nano-structure (Ag-OTC-Ns) was developed for selective and effective eradication of Klebsiella pneumoniae pulmonary infection. Ag-OTC-Ns were prepared by simple homogenization-ultrasonication method and were characterized by DLS, Zeta potential, TEM and FT-IR. The antimicrobial activity of Ag-OTC-Ns was evaluated in vitro using broth micro-dilution technique and time-kill methods. Our study showed that MICs of AgNO3, OTC, AgNPs and Ag-OTC-Ns were 100, 100, 50 and 6.25 µg/ml, respectively. Ag-OTC-Ns demonstrated higher bactericidal efficacy against the targeted Klebsiella pneumoniae at 12.5 µg/ml compared to the free Oxytetracycline, AgNO3 and AgNPs. In vivo results confirmed that, Ag-OTC-Ns could significantly eradicate K. pneumoniae from mice lung in compare with free Oxytetracycline, AgNO3 and AgNPs. In addition, Ag-OTC-Ns could effectually diminish the inflammatory biomarkers levels of Interferon Gamma and IL-12, and as a result it could effectively lower lung damage in K. pneumoniae infected mice. Ag-OTC-Ns has no significant toxicity on tested mice along the experimental period, there was no sign of behavioral abnormality in the surviving mice indicating that the Ag-OTC-Ns is safe at the used concentration. Furthermore, capability of 5 kGy Gamma ray to sterilize Ag-OTC-Ns solution without affecting it stability was proven.

Multi-domain stain normalization for digital pathology: A cycle-consistent adversarial network for whole slide images

The variation in histologic staining between different medical centers is one of the most profound challenges in the field of computer-aided diagnosis. The appearance disparity of pathological whole slide images causes algorithms to become less reliable, which in turn impedes the wide-spread applicability of downstream tasks like cancer diagnosis. Furthermore, different stainings lead to biases in the training which in case of domain shifts negatively affect the test performance. Therefore, in this paper we propose MultiStain-CycleGAN, a multi-domain approach to stain normalization based on CycleGAN. Our modifications to CycleGAN allow us to normalize images of different origins without retraining or using different models. We perform an extensive evaluation of our method using various metrics and compare it to commonly used methods that are multi-domain capable. First, we evaluate how well our method fools a domain classifier that tries to assign a medical center to an image. Then, we test our normalization on the tumor classification performance of a downstream classifier. Furthermore, we evaluate the image quality of the normalized images using the Structural similarity index and the ability to reduce the domain shift using the Fréchet inception distance. We show that our method proves to be multi-domain capable, provides a very high image quality among the compared methods, and can most reliably fool the domain classifier while keeping the tumor classifier performance high. By reducing the domain influence, biases in the data can be removed on the one hand and the origin of the whole slide image can be disguised on the other, thus enhancing patient data privacy.

Don't Fear the Artificial Intelligence: A Systematic Review of Machine Learning for Prostate Cancer Detection in Pathology

CONTEXT

Automated prostate cancer detection using machine learning technology has led to speculation that pathologists will soon be replaced by algorithms. This review covers the development of machine learning algorithms and their reported effectiveness specific to prostate cancer detection and Gleason grading.

OBJECTIVE

To examine current algorithms regarding their accuracy and classification abilities. We provide a general explanation of the technology and how it is being used in clinical practice. The challenges to the application of machine learning algorithms in clinical practice are also discussed.

DATA SOURCES

The literature for this review was identified and collected using a systematic search. Criteria were established prior to the sorting process to effectively direct the selection of studies. A 4-point system was implemented to rank the papers according to their relevancy. For papers accepted as relevant to our metrics, all cited and citing studies were also reviewed. Studies were then categorized based on whether they implemented binary or multi-class classification methods. Data were extracted from papers that contained accuracy, area under the curve (AUC), or κ values in the context of prostate cancer detection. The results were visually summarized to present accuracy trends between classification abilities.

CONCLUSIONS

It is more difficult to achieve high accuracy metrics for multiclassification tasks than for binary tasks. The clinical implementation of an algorithm that can assign a Gleason grade to clinical whole slide images (WSIs) remains elusive. Machine learning technology is currently not able to replace pathologists but can serve as an important safeguard against misdiagnosis.

A feature-enhanced network for stroke lesion segmentation from brain MRI images

Accurate and expeditious segmentation of stroke lesions can greatly assist physicians in making accurate medical diagnoses and administering timely treatments. However, there are two limitations to the current deep learning methods. On the one hand, the attention structure utilizes only local features, which misleads the subsequent segmentation; on the other hand, simple downsampling compromises task-relevant detailed semantic information. To address these challenges, we propose a novel feature refinement and protection network (FRPNet) for stroke lesion segmentation. FRPNet employs a symmetric encoding-decoding structure and incorporates twin attention gate (TAG) and multi-dimension attention pooling (MAP) modules. The TAG module leverages the self-attention mechanism and bi-directional attention to extract both global and local features of the lesion. On the other hand, the MAP module establishes multidimensional pooling attention to effectively mitigate the loss of features during the encoding process. Extensive comparative experiments show that, our method significantly outperforms the state-of-the-art approaches with 60.16% DSC, 36.20px HD and 85.72% DSC, 27.02px HD on two ischemic stroke datasets that contain all stroke stages and several sequences of stroke images. The excellent results that exceed those of existing methods illustrate the efficacy and generalizability of the proposed method. The source code is released on https://github.com/wu2ze2lin2/FRPNet.

Using Deep Learning to Automate Eosinophil Counting in Pediatric Ulcerative Colitis Histopathological Images

Background

Accurate identification of inflammatory cells from mucosal histopathology images is important in diagnosing ulcerative colitis. The identification of eosinophils in the colonic mucosa has been associated with disease course. Cell counting is not only time-consuming but can also be subjective to human biases. In this study we developed an automatic eosinophilic cell counting tool from mucosal histopathology images, using deep learning.

Method

Four pediatric IBD pathologists from two North American pediatric hospitals annotated 530 crops from 143 standard-of-care hematoxylin and eosin (H & E) rectal mucosal biopsies. A 305/75 split was used for training/validation to develop and optimize a U-Net based deep learning model, and 150 crops were used as a test set. The U-Net model was then compared to SAU-Net, a state-of-the-art U-Net variant. We undertook post-processing steps, namely, (1) the pixel-level probability threshold, (2) the minimum number of clustered pixels to designate a cell, and (3) the connectivity. Experiments were run to optimize model parameters using AUROC and cross-entropy loss as the performance metrics.

Results

The F1-score was 0.86 (95%CI:0.79-0.91) (Precision: 0.77 (95%CI:0.70-0.83), Recall: 0.96 (95%CI:0.93-0.99)) to identify eosinophils as compared to an F1-score of 0.2 (95%CI:0.13-0.26) for SAU-Net (Precision: 0.38 (95%CI:0.31-0.46), Recall: 0.13 (95%CI:0.08-0.19)). The inter-rater reliability was 0.96 (95%CI:0.93-0.97). The correlation between two pathologists and the algorithm was 0.89 (95%CI:0.82-0.94) and 0.88 (95%CI:0.80-0.94) respectively.

Conclusion

Our results indicate that deep learning-based automated eosinophilic cell counting can obtain a robust level of accuracy with a high degree of concordance with manual expert annotations.

Advancing Point-of-Care Diagnosis: Digitalizing Combinatorial Biomarker Signals for Lupus Nephritis

To improve the efficiency and patient coverage of the current healthcare system, user-friendly novel homecare devices are urgently needed. In this work, we developed a smartphone-based analyzing and reporting system (SBARS) for biomarker detection in lupus nephritis (LN). This system offers a cost-effective alternative to traditional, expensive large equipment in signal detection and quantification. This innovative approach involves using a portable and affordable microscopic reader to capture biomarker signals. Through smartphone-based image processing techniques, the intensity of each biomarker signal is analyzed. This system exhibited comparable performance to a commercial Genepix scanner in the detection of two potential novel biomarkers of LN, VISG4 and TNFRSF1b. Importantly, this smartphone-based analyzing and reporting system allows for discriminating LN patients with active renal disease from healthy controls with the area-under-the-curve (AUC) value = 0.9 for TNFRSF1b and 1.0 for VSIG4, respectively, indicating high predictive accuracy.

Multimodal artificial intelligence-based pathogenomics improves survival prediction in oral squamous cell carcinoma

In this study, we aimed to develop a novel prognostic algorithm for oral squamous cell carcinoma (OSCC) using a combination of pathogenomics and AI-based techniques. We collected comprehensive clinical, genomic, and pathology data from a cohort of OSCC patients in the TCGA dataset and used machine learning and deep learning algorithms to identify relevant features that are predictive of survival outcomes. Our analyses included 406 OSCC patients. Initial analyses involved gene expression analyses, principal component analyses, gene enrichment analyses, and feature importance analyses. These insights were foundational for subsequent model development. Furthermore, we applied five machine learning/deep learning algorithms (Random Survival Forest, Gradient Boosting Survival Analysis, Cox PH, Fast Survival SVM, and DeepSurv) for survival prediction. Our initial analyses revealed relevant gene expression variations and biological pathways, laying the groundwork for robust feature selection in model building. The results showed that the multimodal model outperformed the unimodal models across all methods, with c-index values of 0.722 for RSF, 0.633 for GBSA, 0.625 for FastSVM, 0.633 for CoxPH, and 0.515 for DeepSurv. When considering only important features, the multimodal model continued to outperform the unimodal models, with c-index values of 0.834 for RSF, 0.747 for GBSA, 0.718 for FastSVM, 0.742 for CoxPH, and 0.635 for DeepSurv. Our results demonstrate the potential of pathogenomics and AI-based techniques in improving the accuracy of prognostic prediction in OSCC, which may ultimately aid in the development of personalized treatment strategies for patients with this devastating disease.

Improved assessment of donor liver steatosis using Banff consensus recommendations and deep learning algorithms

BACKGROUND & AIMS

The Banff Liver Working Group recently published consensus recommendations for steatosis assessment in donor liver biopsy, but few studies reported their use and no automated deep-learning algorithms based on the proposed criteria have been developed so far. We evaluated Banff recommendations on a large monocentric series of donor liver needle biopsies by comparing pathologists' scores with those generated by convolutional neural networks (CNNs) we specifically developed for automated steatosis assessment.

METHODS

We retrospectively retrieved 292 allograft liver needle biopsies collected between January 2016 and January 2020 and performed steatosis assessment using a former intra-institution method (pre-Banff method) and the newly introduced Banff recommendations. Scores provided by pathologists and CNN models were then compared, and the degree of agreement was measured with the intraclass correlation coefficient (ICC).

RESULTS

Regarding the pre-Banff method, poor agreement was observed between the pathologist and CNN models for small droplet macrovesicular steatosis (ICC: 0.38), large droplet macrovesicular steatosis (ICC: 0.08), and the final combined score (ICC: 0.16) evaluation, but none of these reached statistically significance. Interestingly, significantly improved agreement was observed using the Banff approach: ICC was 0.93 for the low-power score (p <0.001), 0.89 for the high-power score (p <0.001), and 0.93 for the final score (p <0.001). Comparing the pre-Banff method with the Banff approach on the same biopsy, pathologist and CNN model assessment showed a mean (±SD) percentage of discrepancy of 26.89 (±22.16) and 1.20 (±5.58), respectively.

CONCLUSIONS

Our findings support the use of Banff recommendations in daily practice and highlight the need for a granular analysis of their effect on liver transplantation outcomes.

IMPACT AND IMPLICATIONS

We developed and validated the first automated deep-learning algorithms for standardized steatosis assessment based on the Banff Liver Working Group consensus recommendations. Our algorithm provides an unbiased automated evaluation of steatosis, which will lay the groundwork for granular analysis of steatosis's short- and long-term effects on organ viability, enabling the identification of clinically relevant steatosis cut-offs for donor organ acceptance. Implementing our algorithm in daily clinical practice will allow for a more efficient and safe allocation of donor organs, improving the post-transplant outcomes of patients.

Are you sure it's an artifact? Artifact detection and uncertainty quantification in histological images

Modern cancer diagnostics involves extracting tissue specimens from suspicious areas and conducting histotechnical procedures to prepare a digitized glass slide, called Whole Slide Image (WSI), for further examination. These procedures frequently introduce different types of artifacts in the obtained WSI, and histological artifacts might influence Computational Pathology (CPATH) systems further down to a diagnostic pipeline if not excluded or handled. Deep Convolutional Neural Networks (DCNNs) have achieved promising results for the detection of some WSI artifacts, however, they do not incorporate uncertainty in their predictions. This paper proposes an uncertainty-aware Deep Kernel Learning (DKL) model to detect blurry areas and folded tissues, two types of artifacts that can appear in WSIs. The proposed probabilistic model combines a CNN feature extractor and a sparse Gaussian Processes (GPs) classifier, which improves the performance of current state-of-the-art artifact detection DCNNs and provides uncertainty estimates. We achieved 0.996 and 0.938 F1 scores for blur and folded tissue detection on unseen data, respectively. In extensive experiments, we validated the DKL model on unseen data from external independent cohorts with different staining and tissue types, where it outperformed DCNNs. Interestingly, the DKL model is more confident in the correct predictions and less in the wrong ones. The proposed DKL model can be integrated into the preprocessing pipeline of CPATH systems to provide reliable predictions and possibly serve as a quality control tool.

Machine Learning and Deep Learning Applications in Magnetic Particle Imaging

In recent years, magnetic particle imaging (MPI) has emerged as a promising imaging technique depicting high sensitivity and spatial resolution. It originated in the early 2000s where it proposed a new approach to challenge the low spatial resolution achieved by using relaxometry in order to measure the magnetic fields. MPI presents 2D and 3D images with high temporal resolution, non-ionizing radiation, and optimal visual contrast due to its lack of background tissue signal. Traditionally, the images were reconstructed by the conversion of signal from the induced voltage by generating system matrix and X-space based methods. Because image reconstruction and analyses play an integral role in obtaining precise information from MPI signals, newer artificial intelligence-based methods are continuously being researched and developed upon. In this work, we summarize and review the significance and employment of machine learning and deep learning models for applications with MPI and the potential they hold for the future. LEVEL OF EVIDENCE: 5 TECHNICAL EFFICACY: Stage 1.

Computational pathology-based weakly supervised prediction model for MGMT promoter methylation status in glioblastoma

Introduction

The methylation status of oxygen 6-methylguanine-DNA methyltransferase (MGMT) is closely related to the treatment and prognosis of glioblastoma. However, there are currently some challenges in detecting the methylation status of MGMT promoters. The hematoxylin and eosin (H&E)-stained histopathological slides have always been the gold standard for tumor diagnosis.

Methods

In this study, based on the TCGA database and H&E-stained Whole slide images (WSI) of Beijing Tiantan Hospital, we constructed a weakly supervised prediction model of MGMT promoter methylation status in glioblastoma by using two Transformer structure models.

Results

The accuracy scores of this model in the TCGA dataset and our independent dataset were 0.79 (AUC = 0.86) and 0.76 (AUC = 0.83), respectively.

Conclusion

The model demonstrates effective prediction of MGMT promoter methylation status in glioblastoma and exhibits some degree of generalization capability. At the same time, our study also shows that adding Patches automatic screening module to the computational pathology research framework of glioma can significantly improve the model effect.

Task design for crowdsourced glioma cell annotation in microscopy images

Crowdsourcing has been used in computational pathology to generate cell and cell nuclei annotations for machine learning. Herein, we broaden its scope to the previously unsolved challenging task of glioma cell detection. This requires multiplexed immunofluorescence microscopy due to diffuse invasiveness and exceptional similarity between glioma cells and reactive astrocytes. In four pilot experiments, we iteratively developed a task design enabling high-quality annotations by crowdworkers on Amazon Mechanical Turk. We applied majority or weighted vote and validated them against ground truth in the final setting. On the base of a YOLO convolutional neural network architecture, we used these consensus labels for training with different image representations regarding colors, intensities, and immmunohistochemical marker combinations. A crowd of 712 workers defined aggregated point annotations in 235 images with an average [Formula: see text] score of 0.627 for majority vote. The networks resulted in acceptable [Formula: see text] scores up to 0.69 for YOLOv8 on average and indicated first evidence for transferability to images lacking tumor markers, especially in IDH-wildtype glioblastoma. Our work confirms feasibility of crowdsourcing to generate labels suitable for training of machine learning tools in the challenging and clinically relevant use case of glioma microenvironment.

Digital pathology in cardiac transplant diagnostics: from biopsies to algorithms

In the field of heart transplantation, the ability to accurately and promptly diagnose cardiac allograft rejection is crucial. This comprehensive review explores the transformative role of digital pathology and computational pathology, especially through machine learning, in this critical domain. These methodologies harness large datasets to extract subtle patterns and valuable information that extend beyond human perceptual capabilities, potentially enhancing diagnostic outcomes. Current research indicates that these computer-based systems could offer accuracy and performance matching, or even exceeding, that of expert pathologists, thereby introducing more objectivity and reducing observer variability. Despite promising results, several challenges such as limited sample sizes, diverse data sources, and the absence of standardized protocols pose significant barriers to the widespread adoption of these techniques. The future of digital pathology in heart transplantation diagnostics depends on utilizing larger, more diverse patient cohorts, standardizing data collection, processing, and evaluation protocols, and fostering collaborative research efforts. The integration of various data types, including clinical, demographic, and imaging information, could further refine diagnostic precision. As researchers address these challenges and promote collaborative efforts, digital pathology has the potential to become an integral part of clinical practice, ultimately improving patient care in heart transplantation.

Biomedical Big Data Technologies, Applications, and Challenges for Precision Medicine: A Review

The explosive growth of biomedical Big Data presents both significant opportunities and challenges in the realm of knowledge discovery and translational applications within precision medicine. Efficient management, analysis, and interpretation of big data can pave the way for groundbreaking advancements in precision medicine. However, the unprecedented strides in the automated collection of large-scale molecular and clinical data have also introduced formidable challenges in terms of data analysis and interpretation, necessitating the development of novel computational approaches. Some potential challenges include the curse of dimensionality, data heterogeneity, missing data, class imbalance, and scalability issues. This overview article focuses on the recent progress and breakthroughs in the application of big data within precision medicine. Key aspects are summarized, including content, data sources, technologies, tools, challenges, and existing gaps. Nine fields-Datawarehouse and data management, electronic medical record, biomedical imaging informatics, Artificial intelligence-aided surgical design and surgery optimization, omics data, health monitoring data, knowledge graph, public health informatics, and security and privacy-are discussed.

TGMIL: A hybrid multi-instance learning model based on the Transformer and the Graph Attention Network for whole-slide images classification of renal cell carcinoma

BACKGROUND AND OBJECTIVES

The pathological diagnosis of renal cell carcinoma is crucial for treatment. Currently, the multi-instance learning method is commonly used for whole-slide image classification of renal cell carcinoma, which is mainly based on the assumption of independent identical distribution. But this is inconsistent with the need to consider the correlation between different instances in the diagnosis process. Furthermore, the problem of high resource consumption of pathology images is still urgent to be solved. Therefore, we propose a new multi-instance learning method to solve this problem.

METHODS

In this study, we proposed a hybrid multi-instance learning model based on the Transformer and the Graph Attention Network, called TGMIL, to achieve whole-slide image of renal cell carcinoma classification without pixel-level annotation or region of interest extraction. Our approach is divided into three steps. First, we designed a feature pyramid with the multiple low magnifications of whole-slide image named MMFP. It makes the model incorporates richer information, and reduces memory consumption as well as training time compared to the highest magnification. Second, TGMIL amalgamates the Transformer and the Graph Attention's capabilities, adeptly addressing the loss of instance contextual and spatial. Within the Graph Attention network stream, an easy and efficient approach employing max pooling and mean pooling yields the graph adjacency matrix, devoid of extra memory consumption. Finally, the outputs of two streams of TGMIL are aggregated to achieve the classification of renal cell carcinoma.

RESULTS

On the TCGA-RCC validation set, a public dataset for renal cell carcinoma, the area under a receiver operating characteristic (ROC) curve (AUC) and accuracy of TGMIL were 0.98±0.0015,0.9191±0.0062, respectively. It showcased remarkable proficiency on the private validation set of renal cell carcinoma pathology images, attaining AUC of 0.9386±0.0162 and ACC of 0.9197±0.0124. Furthermore, on the public breast cancer whole-slide image test dataset, CAMELYON 16, our model showed good classification performance with an accuracy of 0.8792.

CONCLUSIONS

TGMIL models the diagnostic process of pathologists and shows good classification performance on multiple datasets. Concurrently, the MMFP module efficiently diminishes resource requirements, offering a novel angle for exploring computational pathology images.

Performance and limitations of a supervised deep learning approach for the histopathological Oxford Classification of glomeruli with IgA nephropathy

BACKGROUND AND OBJECTIVE

The Oxford Classification for IgA nephropathy is the most successful example of an evidence-based nephropathology classification system. The aim of our study was to replicate the glomerular components of Oxford scoring with an end-to-end deep learning pipeline that involves automatic glomerular segmentation followed by classification for mesangial hypercellularity (M), endocapillary hypercellularity (E), segmental sclerosis (S) and active crescents (C).

METHODS

A total number of 1056 periodic acid-Schiff (PAS) whole slide images (WSIs), coming from 386 kidney biopsies, were annotated. Several detection models for glomeruli, based on the Mask R-CNN architecture, were trained on 587 WSIs, validated on 161 WSIs, and tested on 127 WSIs. For the development of segmentation models, 20,529 glomeruli were annotated, of which 16,571 as training and 3958 as validation set. The test set of the segmentation module comprised of 2948 glomeruli. For the Oxford classification, 6206 expert-annotated glomeruli from 308 PAS WSIs were labelled for M, E, S, C and split into a training set of 4298 glomeruli from 207 WSIs, and a test set of 1908 glomeruli. We chose the best-performing models to construct an end-to-end pipeline, which we named MESCnn (MESC classification by neural network), for the glomerular Oxford classification of WSIs.

RESULTS

Instance segmentation yielded excellent results with an AP50 ranging between 78.2-80.1 % (79.4 ± 0.7 %) on the validation and 75.1-77.7 % (76.5 ± 0.9 %) on the test set. The aggregated Jaccard Index was between 73.4-75.9 % (75.0 ± 0.8 %) on the validation and 69.1-73.4 % (72.2 ± 1.4 %) on the test set. At granular glomerular level, Oxford Classification was best replicated for M with EfficientNetV2-L with a mean ROC-AUC of 90.2 % and a mean precision/recall area under the curve (PR-AUC) of 81.8 %, best for E with MobileNetV2 (ROC-AUC 94.7 %) and ResNet50 (PR-AUC 75.8 %), best for S with EfficientNetV2-M (mean ROC-AUC 92.7 %, mean PR-AUC 87.7 %), best for C with EfficientNetV2-L (ROC-AUC 92.3 %) and EfficientNetV2-S (PR-AUC 54.7 %). At biopsy-level, correlation between expert and deep learning labels fulfilled the demands of the Oxford Classification.

CONCLUSION

We designed an end-to-end pipeline for glomerular Oxford Classification on both a granular glomerular and an entire biopsy level. Both the glomerular segmentation and the classification modules are freely available for further development to the renal medicine community.

Personalizable AI platform for universal access to research and diagnosis in digital pathology

BACKGROUND AND MOTIVATION

Digital pathology has been evolving over the last years, proposing significant workflow advantages that have fostered its adoption in professional environments. Patient clinical and image data are readily available in remote data banks that can be consumed efficiently over standard communication technologies. The appearance of new imaging techniques and advanced artificial intelligence algorithms has significantly reduced the burden on medical professionals by speeding up the screening process. Despite these advancements, the usage of digital pathology in professional environments has been slowed down by poor interoperability between services resulting from a lack of standard interfaces and integrative solutions. This work addresses this issue by proposing a cloud-based digital pathology platform built on standard and open interfaces.

METHODS

The work proposes and describes a vendor-neutral platform that provides interfaces for managing digital slides, and medical reports, and integrating digital image analysis services compatible with existing standards. The solution integrates the open-source plugin-based Dicoogle PACS for interoperability and extensibility, which grants the proposed solution great feature customization.

RESULTS

The solution was developed in collaboration with iPATH research project partners, including the validation by medical pathologists. The result is a pure Web collaborative framework that supports both research and production environments. A total of 566 digital slides from different pathologies were successfully uploaded to the platform. Using the integration interfaces, a mitosis detection algorithm was successfully installed into the platform, and it was trained with 2400 annotations collected from breast carcinoma images.

CONCLUSION

Interoperability is a key factor when discussing digital pathology solutions, as it facilitates their integration into existing institutions' information systems. Moreover, it improves data sharing and integration of third-party services such as image analysis services, which have become relevant in today's digital pathology workflow. The proposed solution fully embraces the DICOM standard for digital pathology, presenting an interoperable cloud-based solution that provides great feature customization thanks to its extensible architecture.

Artifact Detection and Restoration in Histology Images With Stain-Style and Structural Preservation

The artifacts in histology images may encumber the accurate interpretation of medical information and cause misdiagnosis. Accordingly, prepending manual quality control of artifacts considerably decreases the degree of automation. To close this gap, we propose a methodical pre-processing framework to detect and restore artifacts, which minimizes their impact on downstream AI diagnostic tasks. First, the artifact recognition network AR-Classifier first differentiates common artifacts from normal tissues, e.g., tissue folds, marking dye, tattoo pigment, spot, and out-of-focus, and also catalogs artifact patches by their restorability. Then, the succeeding artifact restoration network AR-CycleGAN performs de-artifact processing where stain styles and tissue structures can be maximally retained. We construct a benchmark for performance evaluation, curated from both clinically collected WSIs and public datasets of colorectal and breast cancer. The functional structures are compared with state-of-the-art methods, and also comprehensively evaluated by multiple metrics across multiple tasks, including artifact classification, artifact restoration, downstream diagnostic tasks of tumor classification and nuclei segmentation. The proposed system allows full automation of deep learning based histology image analysis without human intervention. Moreover, the structure-independent characteristic enables its processing with various artifact subtypes. The source code and data in this research are available at https://github.com/yunboer/AR-classifier-and-AR-CycleGAN.

Multi-Layer Preprocessing and U-Net with Residual Attention Block for Retinal Blood Vessel Segmentation

Retinal blood vessel segmentation is a valuable tool for clinicians to diagnose conditions such as atherosclerosis, glaucoma, and age-related macular degeneration. This paper presents a new framework for segmenting blood vessels in retinal images. The framework has two stages: a multi-layer preprocessing stage and a subsequent segmentation stage employing a U-Net with a multi-residual attention block. The multi-layer preprocessing stage has three steps. The first step is noise reduction, employing a U-shaped convolutional neural network with matrix factorization (CNN with MF) and detailed U-shaped U-Net (D_U-Net) to minimize image noise, culminating in the selection of the most suitable image based on the PSNR and SSIM values. The second step is dynamic data imputation, utilizing multiple models for the purpose of filling in missing data. The third step is data augmentation through the utilization of a latent diffusion model (LDM) to expand the training dataset size. The second stage of the framework is segmentation, where the U-Nets with a multi-residual attention block are used to segment the retinal images after they have been preprocessed and noise has been removed. The experiments show that the framework is effective at segmenting retinal blood vessels. It achieved Dice scores of 95.32, accuracy of 93.56, precision of 95.68, and recall of 95.45. It also achieved efficient results in removing noise using CNN with matrix factorization (MF) and D-U-NET according to values of PSNR and SSIM for (0.1, 0.25, 0.5, and 0.75) levels of noise. The LDM achieved an inception score of 13.6 and an FID of 46.2 in the augmentation step.

An Efficient and Effective Framework for Intestinal Parasite Egg Detection Using YOLOv5

Intestinal parasitic infections pose a grave threat to human health, particularly in tropical and subtropical regions. The traditional manual microscopy system of intestinal parasite detection remains the gold standard procedure for diagnosing parasite cysts or eggs. This approach is costly, time-consuming (30 min per sample), highly tedious, and requires a specialist. However, computer vision, based on deep learning, has made great strides in recent years. Despite the significant advances in deep convolutional neural network-based architectures, little research has been conducted to explore these techniques' potential in parasitology, specifically for intestinal parasites. This research presents a novel proposal for state-of-the-art transfer learning architecture for the detection and classification of intestinal parasite eggs from images. The ultimate goal is to ensure prompt treatment for patients while also alleviating the burden on experts. Our approach comprised two main stages: image pre-processing and augmentation in the first stage, and YOLOv5 algorithms for detection and classification in the second stage, followed by performance comparison based on different parameters. Remarkably, our algorithms achieved a mean average precision of approximately 97% and a detection time of only 8.5 ms per sample for a dataset of 5393 intestinal parasite images. This innovative approach holds tremendous potential to form a solid theoretical basis for real-time detection and classification in routine clinical examinations, addressing the increasing demand and accelerating the diagnostic process. Our research contributes to the development of cutting-edge technologies for the efficient and accurate detection of intestinal parasite eggs, advancing the field of medical imaging and diagnosis.

Biomimetic Gland Models with Engineered Stratagems

As extensively distributed tissues throughout the human body, glands play a critical role in various physiological processes. Therefore, the construction of biomimetic gland models in vitro has aroused great interest in multiple disciplines. In the biological field, the researchers focus on optimizing the cell sources and culture techniques to reconstruct the specific structures and functions of glands, such as the emergence of organoid technology. From the perspective of biomedical engineering, the generation of biomimetic gland models depends on the combination of engineered scaffolds and microfluidics, to mimic the in vivo environment of glandular tissues. These engineered stratagems endowed gland models with more biomimetic features, as well as a wide range of application prospects. In this review, we first describe the biomimetic strategies for constructing different in vitro gland models, focusing on the role of microfluidics in promoting the structure and function development of biomimetic glands. After summarizing several common in vitro models of endocrine and exocrine glands, the applications of gland models in disease modelling, drug screening, regenerative medicine, and personalized medicine are enumerated. Finally, we conclude the current challenges and our perspective of these biomimetic gland models.

Applications of discriminative and deep learning feature extraction methods for whole slide image analysis: A survey

Digital pathology technologies, including whole slide imaging (WSI), have significantly improved modern clinical practices by facilitating storing, viewing, processing, and sharing digital scans of tissue glass slides. Researchers have proposed various artificial intelligence (AI) solutions for digital pathology applications, such as automated image analysis, to extract diagnostic information from WSI for improving pathology productivity, accuracy, and reproducibility. Feature extraction methods play a crucial role in transforming raw image data into meaningful representations for analysis, facilitating the characterization of tissue structures, cellular properties, and pathological patterns. These features have diverse applications in several digital pathology applications, such as cancer prognosis and diagnosis. Deep learning-based feature extraction methods have emerged as a promising approach to accurately represent WSI contents and have demonstrated superior performance in histology-related tasks. In this survey, we provide a comprehensive overview of feature extraction methods, including both manual and deep learning-based techniques, for the analysis of WSIs. We review relevant literature, analyze the discriminative and geometric features of WSIs (i.e., features suited to support the diagnostic process and extracted by "engineered" methods as opposed to AI), and explore predictive modeling techniques using AI and deep learning. This survey examines the advances, challenges, and opportunities in this rapidly evolving field, emphasizing the potential for accurate diagnosis, prognosis, and decision-making in digital pathology.

Invasion depth estimation of carcinoma cells using adaptive stain normalization to improve epidermis segmentation accuracy

Submucosal invasion depth is a significant prognostic factor when assessing lymph node metastasis and cancer itself to plan proper treatment for the patient. Conventionally, oncologists measure the invasion depth by hand which is a laborious, subjective, and time-consuming process. The manual pathological examination by measuring accurate carcinoma cell invasion with considerable inter-observer and intra-observer variations is still challenging. The increasing use of medical imaging and artificial intelligence reveals a significant role in clinical medicine and pathology. In this paper, we propose an approach to study invasive behavior and measure the invasion depth of carcinoma from stained histopathology images. Specifically, our model includes adaptive stain normalization, color decomposition, and morphological reconstruction with adaptive thresholding to separate the epithelium with blue ratio image. Our method splits the image into multiple non-overlapping meaningful segments and successfully finds the homogeneous segments to measure accurate invasion depth. The invasion depths are measured from the inner epithelium edge to outermost pixels of the deepest part of particles in image. We conduct our experiments on skin melanoma tissue samples as well as on organotypic invasion model utilizing myoma tissue and oral squamous cell carcinoma. The performance is experimentally compared to three closely related reference methods and our method provides a superior result in measuring invasion depth. This computational technique will be beneficial for the segmentation of epithelium and other particles for the development of novel computer-aided diagnostic tools in biobank applications.

An improved Fourier Ptychography algorithm for ultrasonic array imaging

Inspired by the optical imaging algorithm, the Fourier Ptychography (FP) algorithm is adopted to improve the resolution of ultrasonic array imaging. In the FP algorithm, the steady-state spectrum is utilized to recover the high-resolution ultrasonic images. Meanwhile, the parameters of FP algorithm are empirical, which can affect the imaging quality of ultrasonic array. Then the particle swarm optimization (PSO) algorithm is used to optimize the parameters of FP algorithm to further improve the imaging quality of ultrasonic array. The tungsten imaging experiments and pig eye imaging experiments are conducted to demonstrate the feasibility and effectiveness of the developed algorithm. In addition, the proposed algorithm and the coherent wave superposition (CWS) algorithm are both based on single plane wave (SPW) algorithms and they are then compared. The results show that the CWS algorithm and FP algorithm have good longitudinal and lateral resolutions, respectively. The particle swarm optimization-based FP (PSOFP) imaging algorithm has both excellent lateral and longitudinal resolutions. The average lateral resolution of PSOFP imaging algorithm is improved by 34.47% compared with CWS imaging algorithm in the tungsten wires experiments, and the lateral boundary structure width of the lens is improved by 49.48% in the pig eye experiments. The proposed algorithm can effectively improve the ultrasonic imaging quality for medical application.

Digital histopathological images of biopsy predict response to neoadjuvant chemotherapy for locally advanced gastric cancer

BACKGROUND

Neoadjuvant chemotherapy (NAC) has been recognized as an effective therapeutic option for locally advanced gastric cancer as it is expected to reduce tumor size, increase the resection rate, and improve overall survival. However, for patients who are not responsive to NAC, the best operation timing may be missed together with suffering from side effects. Therefore, it is paramount to differentiate potential respondents from non-respondents. Histopathological images contain rich and complex data that can be exploited to study cancers. We assessed the ability of a novel deep learning (DL)-based biomarker to predict pathological responses from images of hematoxylin and eosin (H&E)-stained tissue.

METHODS

In this multicentre observational study, H&E-stained biopsy sections of patients with gastric cancer were collected from four hospitals. All patients underwent NAC followed by gastrectomy. The Becker tumor regression grading (TRG) system was used to evaluate the pathologic chemotherapy response. Based on H&E-stained slides of biopsies, DL methods (Inception-V3, Xception, EfficientNet-B5, and ensemble CRSNet models) were employed to predict the pathological response by scoring the tumor tissue to obtain a histopathological biomarker, the chemotherapy response score (CRS). The predictive performance of the CRSNet was evaluated.

RESULTS

69,564 patches from 230 whole-slide images of 213 patients with gastric cancer were obtained in this study. Based on the F1 score and area under the curve (AUC), an optimal model was finally chosen, named the CRSNet model. Using the ensemble CRSNet model, the response score derived from H&E staining images reached an AUC of 0.936 in the internal test cohort and 0.923 in the external validation cohort for predicting pathological response. The CRS of major responders was significantly higher than that of minor responders in both internal and external test cohorts (both p < 0.001).

CONCLUSION

In this study, the proposed DL-based biomarker (CRSNet model) derived from histopathological images of the biopsy showed potential as a clinical aid for predicting the response to NAC in patients with locally advanced GC. Therefore, the CRSNet model provides a novel tool for the individualized management of locally advanced gastric cancer.

Deep learning in computational dermatopathology of melanoma: A technical systematic literature review

Deep learning (DL) has become one of the major approaches in computational dermatopathology, evidenced by a significant increase in this topic in the current literature. We aim to provide a structured and comprehensive overview of peer-reviewed publications on DL applied to dermatopathology focused on melanoma. In comparison to well-published DL methods on non-medical images (e.g., classification on ImageNet), this field of application comprises a specific set of challenges, such as staining artifacts, large gigapixel images, and various magnification levels. Thus, we are particularly interested in the pathology-specific technical state-of-the-art. We also aim to summarize the best performances achieved thus far with respect to accuracy, along with an overview of self-reported limitations. Accordingly, we conducted a systematic literature review of peer-reviewed journal and conference articles published between 2012 and 2022 in the databases ACM Digital Library, Embase, IEEE Xplore, PubMed, and Scopus, expanded by forward and backward searches to identify 495 potentially eligible studies. After screening for relevance and quality, a total of 54 studies were included. We qualitatively summarized and analyzed these studies from technical, problem-oriented, and task-oriented perspectives. Our findings suggest that the technical aspects of DL for histopathology in melanoma can be further improved. The DL methodology was adopted later in this field, and still lacks the wider adoption of DL methods already shown to be effective for other applications. We also discuss upcoming trends toward ImageNet-based feature extraction and larger models. While DL has achieved human-competitive accuracy in routine pathological tasks, its performance on advanced tasks is still inferior to wet-lab testing (for example). Finally, we discuss the challenges impeding the translation of DL methods to clinical practice and provide insight into future research directions.

Unleashing the potential of AI for pathology: challenges and recommendations

Computational pathology is currently witnessing a surge in the development of AI techniques, offering promise for achieving breakthroughs and significantly impacting the practices of pathology and oncology. These AI methods bring with them the potential to revolutionize diagnostic pipelines as well as treatment planning and overall patient care. Numerous peer-reviewed studies reporting remarkable performance across diverse tasks serve as a testimony to the potential of AI in the field. However, widespread adoption of these methods in clinical and pre-clinical settings still remains a challenge. In this review article, we present a detailed analysis of the major obstacles encountered during the development of effective models and their deployment in practice. We aim to provide readers with an overview of the latest developments, assist them with insights into identifying some specific challenges that may require resolution, and suggest recommendations and potential future research directions. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

Artificial Intelligence to Predict the BRAF V595E Mutation in Canine Urinary Bladder Urothelial Carcinomas

In dogs, the BRAF mutation (V595E) is common in bladder and prostate cancer and represents a specific diagnostic marker. Recent advantages in artificial intelligence (AI) offer new opportunities in the field of tumour marker detection. While AI histology studies have been conducted in humans to detect BRAF mutation in cancer, comparable studies in animals are lacking. In this study, we used commercially available AI histology software to predict BRAF mutation in whole slide images (WSI) of bladder urothelial carcinomas (UC) stained with haematoxylin and eosin (HE), based on a training (n = 81) and a validation set (n = 96). Among 96 WSI, 57 showed identical PCR and AI-based BRAF predictions, resulting in a sensitivity of 58% and a specificity of 63%. The sensitivity increased substantially to 89% when excluding small or poor-quality tissue sections. Test reliability depended on tumour differentiation (p < 0.01), presence of inflammation (p < 0.01), slide quality (p < 0.02) and sample size (p < 0.02). Based on a small subset of cases with available adjacent non-neoplastic urothelium, AI was able to distinguish malignant from benign epithelium. This is the first study to demonstrate the use of AI histology to predict BRAF mutation status in canine UC. Despite certain limitations, the results highlight the potential of AI in predicting molecular alterations in routine tissue sections.

Multilevel Colonoscopy Histopathology Image Segmentation Using Particle Swarm Optimization Techniques

Histopathology image segmentation is a challenging task in medical image processing. This work aims to segment lesion regions from colonoscopy histopathology images. Initially, the images are preprocessed and then segmented using the multilevel image thresholding technique. Multilevel thresholding is considered an optimization problem. Particle swarm optimization (PSO) and its variants, darwinian particle swarm optimization (DPSO), and fractional order darwinian particle swarm optimization (FODPSO) are used to solve the optimization problem and they generate the threshold values. The threshold values obtained are used to segment the lesion regions from the images of the colonoscopy tissue data set. Segmented images containing the lesion regions are then postprocessed to remove unnecessary regions. Experimental results reveal that the FODPSO algorithm with Otsu's discriminant criterion as the objective function achieves the best accuracy, Dice and Jaccard values of 0.89, 0.68 and 0.52, respectively, for the colonoscopy data set. The FODPSO algorithm also outperforms other optimization methods such as artificial bee colony and the firefly algorithms in terms of the accuracy, Dice and Jaccard values.

The role of unpaired image-to-image translation for stain color normalization in colorectal cancer histology classification

BACKGROUND

Histological assessment of colorectal cancer (CRC) tissue is a crucial and demanding task for pathologists. Unfortunately, manual annotation by trained specialists is a burdensome operation, which suffers from problems like intra- and inter-pathologist variability. Computational models are revolutionizing the Digital Pathology field, offering reliable and fast approaches for challenges like tissue segmentation and classification. With this respect, an important obstacle to overcome consists in stain color variations among different laboratories, which can decrease the performance of classifiers. In this work, we investigated the role of Unpaired Image-to-Image Translation (UI2IT) models for stain color normalization in CRC histology and compared to classical normalization techniques for Hematoxylin-Eosin (H&E) images.

METHODS

Five Deep Learning normalization models based on Generative Adversarial Networks (GANs) belonging to the UI2IT paradigm have been thoroughly compared to realize a robust stain color normalization pipeline. To avoid the need for training a style transfer GAN between each pair of data domains, in this paper we introduce the concept of training by exploiting a meta-domain, which contains data coming from a wide variety of laboratories. The proposed framework enables a huge saving in terms of training time, by allowing to train a single image normalization model for a target laboratory. To prove the applicability of the proposed workflow in the clinical practice, we conceived a novel perceptive quality measure, which we defined as Pathologist Perceptive Quality (PPQ). The second stage involved the classification of tissue types in CRC histology, where deep features extracted from Convolutional Neural Networks have been exploited to realize a Computer-Aided Diagnosis system based on a Support Vector Machine (SVM). To prove the reliability of the system on new data, an external validation set composed of N = 15,857 tiles has been collected at IRCCS Istituto Tumori "Giovanni Paolo II".

RESULTS

The exploitation of a meta-domain consented to train normalization models that allowed achieving better classification results than normalization models explicitly trained on the source domain. PPQ metric has been found correlated to quality of distributions (Fréchet Inception Distance - FID) and to similarity of the transformed image to the original one (Learned Perceptual Image Patch Similarity - LPIPS), thus showing that GAN quality measures introduced in natural image processing tasks can be linked to pathologist evaluation of H&E images. Furthermore, FID has been found correlated to accuracies of the downstream classifiers. The SVM trained on DenseNet201 features allowed to obtain the highest classification results in all configurations. The normalization method based on the fast variant of CUT (Contrastive Unpaired Translation), FastCUT, trained with the meta-domain paradigm, allowed to achieve the best classification result for the downstream task and, correspondingly, showed the highest FID on the classification dataset.

CONCLUSIONS

Stain color normalization is a difficult but fundamental problem in the histopathological setting. Several measures should be considered for properly assessing normalization methods, so that they can be introduced in the clinical practice. UI2IT frameworks offer a powerful and effective way to perform the normalization process, providing realistic images with proper colorization, unlike traditional normalization methods that introduce color artifacts. By adopting the proposed meta-domain framework, the training time can be reduced, and the accuracy of downstream classifiers can be increased.

Geometric and network organization of visceral organ epithelium

Mammalian epithelia form a continuous sheet of cells that line the surface of visceral organs. To analyze the epithelial organization of the heart, lung, liver and bowel, epithelial cells were labeled in situ, isolated as a single layer and imaged as large epithelial digitally combine montages. The stitched epithelial images were analyzed for geometric and network organization. Geometric analysis demonstrated a similar polygon distribution in all organs with the greatest variability in the heart epithelia. Notably, the normal liver and inflated lung demonstrated the largest average cell surface area (p < 0.01). In lung epithelia, characteristic wavy or interdigitated cell boundaries were observed. The prevalence of interdigitations increased with lung inflation. To complement the geometric analyses, the epithelia were converted into a network of cell-to-cell contacts. Using the open-source software EpiGraph, subgraph (graphlet) frequencies were used to characterize epithelial organization and compare to mathematical (Epi-Hexagon), random (Epi-Random) and natural (Epi-Voronoi5) patterns. As expected, the patterns of the lung epithelia were independent of lung volume. In contrast, liver epithelia demonstrated a pattern distinct from lung, heart and bowel epithelia (p < 0.05). We conclude that geometric and network analyses can be useful tools in characterizing fundamental differences in mammalian tissue topology and epithelial organization.

USING CONVOLUTIONAL NEURAL NETWORK-BASED SEGMENTATION FOR IMAGE-BASED COMPUTATIONAL FLUID DYNAMICS SIMULATIONS OF BRAIN ANEURYSMS: INITIAL EXPERIENCE IN AUTOMATED MODEL CREATION

"Image-based" computational fluid dynamics (CFD) simulations provide insights into each patient's hemodynamic environment. However, current standard procedures for creating CFD models start with manual segmentation and are time-consuming, hindering the clinical translation of image-based CFD simulations. This feasibility study adopts deep-learning-based image segmentation (hereafter referred to as Artificial Intelligence (AI) segmentation) to replace manual segmentation to accelerate CFD model creation. Two published convolutional neural network-based AI methods (MIScnn and DeepMedic) were selected to perform CFD model extraction from three-dimensional (3D) rotational angiography data containing intracranial aneurysms. In this study, aneurysm morphological and hemodynamic results using models generated by AI segmentation methods were compared with those obtained by two human users for the same data. Interclass coefficients (ICC), Bland-Altman plots, and Pearson's correlation coefficients (PCC) were combined to assess how well AI-generated CFD models were performed. We found that almost perfect agreement was obtained between the human and AI results for all eleven morphological and five out of eight hemodynamic parameters, while a moderate agreement was obtained from the remaining three hemodynamic parameters. Given this level of agreement, using AI segmentation to create CFD models is feasible, given more developments.

Which data subset should be augmented for deep learning? a simulation study using urothelial cell carcinoma histopathology images

BACKGROUND

Applying deep learning to digital histopathology is hindered by the scarcity of manually annotated datasets. While data augmentation can ameliorate this obstacle, its methods are far from standardized. Our aim was to systematically explore the effects of skipping data augmentation; applying data augmentation to different subsets of the whole dataset (training set, validation set, test set, two of them, or all of them); and applying data augmentation at different time points (before, during, or after dividing the dataset into three subsets). Different combinations of the above possibilities resulted in 11 ways to apply augmentation. The literature contains no such comprehensive systematic comparison of these augmentation ways.

RESULTS

Non-overlapping photographs of all tissues on 90 hematoxylin-and-eosin-stained urinary bladder slides were obtained. Then, they were manually classified as either inflammation (5948 images), urothelial cell carcinoma (5811 images), or invalid (3132 images; excluded). If done, augmentation was eight-fold by flipping and rotation. Four convolutional neural networks (Inception-v3, ResNet-101, GoogLeNet, and SqueezeNet), pre-trained on the ImageNet dataset, were fine-tuned to binary classify images of our dataset. This task was the benchmark for our experiments. Model testing performance was evaluated using accuracy, sensitivity, specificity, and area under the receiver operating characteristic curve. Model validation accuracy was also estimated. The best testing performance was achieved when augmentation was done to the remaining data after test-set separation, but before division into training and validation sets. This leaked information between the training and the validation sets, as evidenced by the optimistic validation accuracy. However, this leakage did not cause the validation set to malfunction. Augmentation before test-set separation led to optimistic results. Test-set augmentation yielded more accurate evaluation metrics with less uncertainty. Inception-v3 had the best overall testing performance.

CONCLUSIONS

In digital histopathology, augmentation should include both the test set (after its allocation), and the remaining combined training/validation set (before being split into separate training and validation sets). Future research should try to generalize our results.

A pathomic approach for tumor-infiltrating lymphocytes classification on breast cancer digital pathology images

Background and objectives

The detection of tumor-infiltrating lymphocytes (TILs) could aid in the development of objective measures of the infiltration grade and can support decision-making in breast cancer (BC). However, manual quantification of TILs in BC histopathological whole slide images (WSI) is currently based on a visual assessment, thus resulting not standardized, not reproducible, and time-consuming for pathologists. In this work, a novel pathomic approach, aimed to apply high-throughput image feature extraction techniques to analyze the microscopic patterns in WSI, is proposed. In fact, pathomic features provide additional information concerning the underlying biological processes compared to the WSI visual interpretation, thus providing more easily interpretable and explainable results than the most frequently investigated Deep Learning based methods in the literature.

Methods

A dataset containing 1037 regions of interest with tissue compartments and TILs annotated on 195 TNBC and HER2+ BC hematoxylin and eosin (H&E)-stained WSI was used. After segmenting nuclei within tumor-associated stroma using a watershed-based approach, 71 pathomic features were extracted from each nucleus and reduced using a Spearman's correlation filter followed by a nonparametric Wilcoxon rank-sum test and least absolute shrinkage and selection operator. The relevant features were used to classify each candidate nucleus as either TILs or non-TILs using 5 multivariable machine learning classification models trained using 5-fold cross-validation (1) without resampling, (2) with the synthetic minority over-sampling technique and (3) with downsampling. The prediction performance of the models was assessed using ROC curves.

Results

21 features were selected, with most of them related to the well-known TILs properties of having regular shape, clearer margins, high peak intensity, more homogeneous enhancement and different textural pattern than other cells. The best performance was obtained by Random-Forest with ROC AUC of 0.86, regardless of resampling technique.

Conclusions

The presented approach holds promise for the classification of TILs in BC H&E-stained WSI and could provide support to pathologists for a reliable, rapid and interpretable clinical assessment of TILs in BC.

Automated analysis of small intestinal lamina propria to distinguish normal, Celiac Disease, and Non-Celiac Duodenitis biopsy images

BACKGROUND AND OBJECTIVE

Celiac Disease (CD) is characterized by gluten intolerance in genetically predisposed individuals. High disease prevalence, absence of a cure, and low diagnosis rates make this disease a public health problem. The diagnosis of CD predominantly relies on recognizing characteristic mucosal alterations of the small intestine, such as villous atrophy, crypt hyperplasia, and intraepithelial lymphocytosis. However, these changes are not entirely specific to CD and overlap with Non-Celiac Duodenitis (NCD) due to various etiologies. We investigated whether Artificial Intelligence (AI) models could assist in distinguishing normal, CD, and NCD (and unaffected individuals) based on the characteristics of small intestinal lamina propria (LP).

METHODS

Our method was developed using a dataset comprising high magnification biopsy images of the duodenal LP compartment of CD patients with different clinical stages of CD, those with NCD, and individuals lacking an intestinal inflammatory disorder (controls). A pre-processing step was used to standardize and enhance the acquired images.

RESULTS

For the normal controls versus CD use case, a Support Vector Machine (SVM) achieved an Accuracy (ACC) of 98.53%. For a second use case, we investigated the ability of the classification algorithm to differentiate between normal controls and NCD. In this use case, the SVM algorithm with linear kernel outperformed all the tested classifiers by achieving 98.55% ACC.

CONCLUSIONS

To the best of our knowledge, this is the first study that documents automated differentiation between normal, NCD, and CD biopsy images. These findings are a stepping stone toward automated biopsy image analysis that can significantly benefit patients and healthcare providers.

An Atrous Convolved Hybrid Seg-Net Model with residual and attention mechanism for gland detection and segmentation in histopathological images

PURPOSE

A clinically compatible computerized segmentation model is presented here that aspires to supply clinical gland informative details by seizing every small and intricate variation in medical images, integrate second opinions, and reduce human errors.

APPROACH

It comprises of enhanced learning capability that extracts denser multi-scale gland-specific features, recover semantic gap during concatenation, and effectively handle resolution-degradation and vanishing gradient problems. It is having three proposed modules namely Atrous Convolved Residual Learning Module in the encoder as well as decoder, Residual Attention Module in the skip connection paths, and Atrous Convolved Transitional Module as the transitional and output layer. Also, pre-processing techniques like patch-sampling, stain-normalization, augmentation, etc. are employed to develop its generalization capability. To verify its robustness and invigorate network invariance against digital variability, extensive experiments are carried out employing three different public datasets i.e., GlaS (Gland Segmentation Challenge), CRAG (Colorectal Adenocarcinoma Gland) and LC-25000 (Lung Colon-25000) dataset and a private HosC (Hospital Colon) dataset.

RESULTS

The presented model accomplished combative gland detection outcomes having F1-score (GlaS(Test A(0.957), Test B(0.926)), CRAG(0.935), LC 25000(0.922), HosC(0.963)); and gland segmentation results having Object-Dice Index (GlaS(Test A(0.961), Test B(0.933)), CRAG(0.961), LC-25000(0.940), HosC(0.929)), and Object-Hausdorff Distance (GlaS(Test A(21.77) and Test B(69.74)), CRAG(87.63), LC-25000(95.85), HosC(83.29)). In addition, validation score (GlaS (Test A(0.945), Test B(0.937)), CRAG(0.934), LC-25000(0.911), HosC(0.928)) supplied by the proficient pathologists is integrated for the end segmentation results to corroborate the applicability and appropriateness for assistance at the clinical level applications.

CONCLUSION

The proposed system will assist pathologists in devising precise diagnoses by offering a referential perspective during morphology assessment of colon histopathology images.

Expectation-maximization algorithm leads to domain adaptation for a perineural invasion and nerve extraction task in whole slide digital pathology images

In addition to lymphatic and vascular channels, tumor cells can also spread via nerves, i.e., perineural invasion (PNI). PNI serves as an independent prognostic indicator in many malignancies. As a result, identifying and determining the extent of PNI is an important yet extremely tedious task in surgical pathology. In this work, we present a computational approach to extract nerves and PNI from whole slide histopathology images. We make manual annotations on selected prostate cancer slides once but then apply the trained model for nerve segmentation to both prostate cancer slides and head and neck cancer slides. For the purpose of multi-domain learning/prediction and investigation on the generalization capability of deep neural network, an expectation-maximization (EM)-based domain adaptation approach is proposed to improve the segmentation performance, in particular for the head and neck cancer slides. Experiments are conducted to demonstrate the segmentation performances. The average Dice coefficient for prostate cancer slides is 0.82 and 0.79 for head and neck cancer slides. Comparisons are then made for segmentations with and without the proposed EM-based domain adaptation on prostate cancer and head and neck cancer whole slide histopathology images from The Cancer Genome Atlas (TCGA) database and significant improvements are observed.

Joint multimodal deep learning-based automatic segmentation of ICGA and OCT images for assessment of PCV biomarkers

Purpose

To develop a fully-automatic hybrid algorithm to jointly segment and quantify biomarkers of polypoidal choroidal vasculopathy (PCV) on indocyanine green angiography (ICGA) and spectral domain-OCT (SD-OCT) images.

Design

Evaluation of diagnostic test or technology.

Participants

Seventy-two participants with PCV enrolled in clinical studies at Singapore National Eye Center.

Methods

The dataset consisted of 2-dimensional (2-D) ICGA and 3-dimensional (3-D) SD-OCT images which were spatially registered and manually segmented by clinicians. A deep learning-based hybrid algorithm called PCV-Net was developed for automatic joint segmentation of biomarkers. The PCV-Net consisted of a 2-D segmentation branch for ICGA and 3-D segmentation branch for SD-OCT. We developed fusion attention modules to connect the 2-D and 3-D branches for effective use of the spatial correspondence between the imaging modalities by sharing learned features. We also used self-supervised pretraining and ensembling to further enhance the performance of the algorithm without the need for additional datasets. We compared the proposed PCV-Net to several alternative model variants.

Main Outcome Measures

The PCV-Net was evaluated based on the Dice similarity coefficient (DSC) of the segmentations and the Pearson's correlation and absolute difference of the clinical measurements obtained from the segmentations. Manual grading was used as the gold standard.

Results

The PCV-Net showed good performance compared to manual grading and alternative model variants based on both quantitative and qualitative analyses. Compared to the baseline variant, PCV-Net improved the DSC by 0.04 to 0.43 across the different biomarkers, increased the correlations, and decreased the absolute differences of clinical measurements of interest. Specifically, the largest average (mean ± standard error) DSC improvement was for intraretinal fluid, from 0.02 ± 0.00 (baseline variant) to 0.45 ± 0.06 (PCV-Net). In general, improving trends were observed across the model variants as more technical specifications were added, demonstrating the importance of each aspect of the proposed method.

Conclusion

The PCV-Net has the potential to aid clinicians in disease assessment and research to improve clinical understanding and management of PCV.

Financial Disclosures

Proprietary or commercial disclosure may be found after the references.

Deep Domain Adversarial Learning for Species-Agnostic Classification of Histologic Subtypes of Osteosarcoma

Osteosarcomas (OSs) are aggressive bone tumors with many divergent histologic patterns. During pathology review, OSs are subtyped based on the predominant histologic pattern; however, tumors often demonstrate multiple patterns. This high tumor heterogeneity coupled with scarcity of samples compared with other tumor types render histology-based prognosis of OSs challenging. To combat lower case numbers in humans, dogs with spontaneous OSs have been suggested as a model species. Herein, a convolutional neural network was adversarially trained to classify distinct histologic patterns of OS in humans using mostly canine OS data during training. Adversarial training improved domain adaption of a histologic subtype classifier from canines to humans, achieving an average multiclass F1 score of 0.77 (95% CI, 0.74-0.79) and 0.80 (95% CI, 0.78-0.81) when compared with the ground truth in canines and humans, respectively. Finally, this trained model, when used to characterize the histologic landscape of 306 canine OSs, uncovered distinct clusters with markedly different clinical responses to standard-of-care therapy.

Longitudinal drug synergy assessment using convolutional neural network image-decoding of glioblastoma single-spheroid cultures

Background

In recent years, drug combinations have become increasingly popular to improve therapeutic outcomes in various diseases, including difficult to cure cancers such as the brain cancer glioblastoma. Assessing the interaction between drugs over time is critical for predicting drug combination effectiveness and minimizing the risk of therapy resistance. However, as viability readouts of drug combination experiments are commonly performed as an endpoint where cells are lysed, longitudinal drug-interaction monitoring is currently only possible through combined endpoint assays.

Methods

We provide a method for massive parallel monitoring of drug interactions for 16 drug combinations in 3 glioblastoma models over a time frame of 18 days. In our assay, viabilities of single neurospheres are to be estimated based on image information taken at different time points. Neurosphere images taken on the final day (day 18) were matched to the respective viability measured by CellTiter-Glo 3D on the same day. This allowed to use of machine learning to decode image information to viability values on day 18 as well as for the earlier time points (on days 8, 11, and 15).

Results

Our study shows that neurosphere images allow us to predict cell viability from extrapolated viabilities. This enables to assess of the drug interactions in a time window of 18 days. Our results show a clear and persistent synergistic interaction for several drug combinations over time.

Conclusions

Our method facilitates longitudinal drug-interaction assessment, providing new insights into the temporal-dynamic effects of drug combinations in 3D neurospheres which can help to identify more effective therapies against glioblastoma.

Further predictive value of lymphovascular invasion explored via supervised deep learning for lymph node metastases in breast cancer

Lymphovascular invasion, specifically lymph-blood vessel invasion (LBVI), is a risk factor for metastases in breast invasive ductal carcinoma (IDC) and is routinely screened using hematoxylin-eosin histopathological images. However, routine reports only describe whether LBVI is present and does not provide other potential prognostic information of LBVI. This study aims to evaluate the clinical significance of LBVI in 685 IDC cases and explore the added predictive value of LBVI on lymph node metastases (LNM) via supervised deep learning (DL), an expert-experience embedded knowledge transfer learning (EEKT) model in 40 LBVI-positive cases signed by the routine report. Multivariate logistic regression and propensity score matching analysis demonstrated that LBVI (OR 4.203, 95% CI 2.809-6.290, P < 0.001) was a significant risk factor for LNM. Then, the EEKT model trained on 5780 image patches automatically segmented LBVI with a patch-wise Dice similarity coefficient of 0.930 in the test set and output counts, location, and morphometric features of the LBVIs. Some morphometric features were beneficial for further stratification within the 40 LBVI-positive cases. The results showed that LBVI in cases with LNM had a higher short-to-long side ratio of the minimum rectangle (MR) (0.686 vs. 0.480, P = 0.001), LBVI-to-MR area ratio (0.774 vs. 0.702, P = 0.002), and solidity (0.983 vs. 0.934, P = 0.029) compared to LBVI in cases without LNM. The results highlight the potential of DL to assist pathologists in quantifying LBVI and, more importantly, in exploring added prognostic information from LBVI.

Artificial intelligence applications in predicting the behavior of gastrointestinal cancers in pathology

Recent research has provided a wealth of data supporting the application of artificial intelligence (AI)-based applications in routine pathology practice. Indeed, it is clear that these methods can significantly support an accurate and rapid diagnosis by eliminating errors, increasing reliability, and improving workflow. In addition, the effectiveness of AI in the pathological evaluation of prognostic parameters associated with behavior, course, and treatment in many types of tumors has also been noted. Regarding gastrointestinal system (GIS) cancers, the contribution of AI methods to pathological diagnosis has been investigated in many studies. On the other hand, studies focusing on AI applications in evaluating parameters to determine tumor behavior are relatively few. For this purpose, the potential of AI models has been studied over a broad spectrum, from tumor subtyping to the identification of new digital biomarkers. The capacity of AI to infer genetic alterations of cancer tissues from digital slides has been demonstrated. Although current data suggest the merit of AI-based approaches in assessing tumor behavior in GIS cancers, a wide range of challenges still need to be solved, from laboratory infrastructure to improving the robustness of algorithms, before incorporating AI applications into real-life GIS pathology practice. This review aims to present data from AI applications in evaluating pathological parameters related to the behavior of GIS cancer with an overview of the opportunities and challenges encountered in implementing AI in pathology.