Glomerulus Classification and Detection Based on Convolutional Neural Networks

Artificial intelligence-assisted quantification and assessment of whole slide images for pediatric kidney disease diagnosis

MOTIVATION

Pediatric kidney disease is a widespread, progressive condition that severely impacts growth and development of children. Chronic kidney disease is often more insidious in children than in adults, usually requiring a renal biopsy for diagnosis. Biopsy evaluation requires copious examination by trained pathologists, which can be tedious and prone to human error. In this study, we propose an artificial intelligence (AI) method to assist pathologists in accurate segmentation and classification of pediatric kidney structures, named as AI-based Pediatric Kidney Diagnosis (APKD).

RESULTS

We collected 2935 pediatric patients diagnosed with kidney disease for the development of APKD. The dataset comprised 93 932 histological structures annotated manually by three skilled nephropathologists. APKD scored an average accuracy of 94% for each kidney structure category, including 99% in the glomerulus. We found strong correlation between the model and manual detection in detected glomeruli (Spearman correlation coefficient r = 0.98, P < .001; intraclass correlation coefficient ICC = 0.98, 95% CI = 0.96-0.98). Compared to manual detection, APKD was approximately 5.5 times faster in segmenting glomeruli. Finally, we show how the pathological features extracted by APKD can identify focal abnormalities of the glomerular capillary wall to aid in the early diagnosis of pediatric kidney disease.

AVAILABILITY AND IMPLEMENTATION

https://github.com/ChunyueFeng/Kidney-DataSet.

Automatic Identification of Glomerular in Whole-Slide Images Using a Modified UNet Model

Glomeruli are interconnected capillaries in the renal cortex that are responsible for blood filtration. Damage to these glomeruli often signifies the presence of kidney disorders like glomerulonephritis and glomerulosclerosis, which can ultimately lead to chronic kidney disease and kidney failure. The timely detection of such conditions is essential for effective treatment. This paper proposes a modified UNet model to accurately detect glomeruli in whole-slide images of kidney tissue. The UNet model was modified by changing the number of filters and feature map dimensions from the first to the last layer to enhance the model's capacity for feature extraction. Moreover, the depth of the UNet model was also improved by adding one more convolution block to both the encoder and decoder sections. The dataset used in the study comprised 20 large whole-side images. Due to their large size, the images were cropped into 512 × 512-pixel patches, resulting in a dataset comprising 50,486 images. The proposed model performed well, with 95.7% accuracy, 97.2% precision, 96.4% recall, and 96.7% F1-score. These results demonstrate the proposed model's superior performance compared to the original UNet model, the UNet model with EfficientNetb3, and the current state-of-the-art. Based on these experimental findings, it has been determined that the proposed model accurately identifies glomeruli in extracted kidney patches.

Quantifying acute kidney injury in an Ischaemia-Reperfusion Injury mouse model using deep-learning-based semantic segmentation in histology

This study focuses on ischaemia-reperfusion injury (IRI) in kidneys, a cause of acute kidney injury (AKI) and end-stage kidney disease (ESKD). Traditional kidney damage assessment methods are semi-quantitative and subjective. This study aims to use a convolutional neural network (CNN) to segment murine kidney structures after IRI, quantify damage via CNN-generated pathological measurements, and compare this to conventional scoring. The CNN was able to accurately segment the different pathological classes, such as Intratubular casts and Tubular necrosis, with an F1 score of over 0.75. Some classes, such as Glomeruli and Proximal tubules, had even higher statistical values with F1 scores over 0.90. The scoring generated based on the segmentation approach statistically correlated with the semiquantitative assessment (Spearman's rank correlation coefficient=0.94). The heatmap approach localised the intratubular necrosis mainly in the outer stripe of the outer medulla, while the tubular casts were also present in more superficial or deeper portions of the cortex and medullary areas. This study presents a CNN model capable of segmenting multiple classes of interest, including acute IRI-specific pathological changes, in a whole mouse kidney section and can provide insights into the distribution of pathological classes within the whole mouse kidney section.

Omni-Seg: A Scale-Aware Dynamic Network for Renal Pathological Image Segmentation

Comprehensive semantic segmentation on renal pathological images is challenging due to the heterogeneous scales of the objects. For example, on a whole slide image (WSI), the cross-sectional areas of glomeruli can be 64 times larger than that of the peritubular capillaries, making it impractical to segment both objects on the same patch, at the same scale. To handle this scaling issue, prior studies have typically trained multiple segmentation networks in order to match the optimal pixel resolution of heterogeneous tissue types. This multi-network solution is resource-intensive and fails to model the spatial relationship between tissue types. In this article, we propose the Omni-Seg network, a scale-aware dynamic neural network that achieves multi-object (six tissue types) and multi-scale (5× to 40× scale) pathological image segmentation via a single neural network. The contribution of this article is three-fold: (1) a novel scale-aware controller is proposed to generalize the dynamic neural network from single-scale to multi-scale; (2) semi-supervised consistency regularization of pseudo-labels is introduced to model the inter-scale correlation of unannotated tissue types into a single end-to-end learning paradigm; and (3) superior scale-aware generalization is evidenced by directly applying a model trained on human kidney images to mouse kidney images, without retraining. By learning from 150,000 human pathological image patches from six tissue types at three different resolutions, our approach achieved superior segmentation performance according to human visual assessment and evaluation of image-omics (i.e., spatial transcriptomics).

Glomerulus Detection Using Segmentation Neural Networks

Digital pathology is vital for the correct diagnosis of kidney before transplantation or kidney disease identification. One of the key challenges in kidney diagnosis is glomerulus detection in kidney tissue segments. In this study, we propose a deep learning-based method for glomerulus detection from digitized kidney slide segments. The proposed method applies models based on convolutional neural networks to detect image segments containing the glomerulus region. We employ various networks such as ResNets, UNet, LinkNet, and EfficientNet to train the models. In our experiments on a network trained on the NIH HuBMAP kidney whole slide image dataset, the proposed method achieves the highest scores with Dice coefficient of 0.942.

Segmentation of human functional tissue units in support of a Human Reference Atlas

The Human BioMolecular Atlas Program (HuBMAP) aims to compile a Human Reference Atlas (HRA) for the healthy adult body at the cellular level. Functional tissue units (FTUs), relevant for HRA construction, are of pathobiological significance. Manual segmentation of FTUs does not scale; highly accurate and performant, open-source machine-learning algorithms are needed. We designed and hosted a Kaggle competition that focused on development of such algorithms and 1200 teams from 60 countries participated. We present the competition outcomes and an expanded analysis of the winning algorithms on additional kidney and colon tissue data, and conduct a pilot study to understand spatial location and density of FTUs across the kidney. The top algorithm from the competition, Tom, outperforms other algorithms in the expanded study, while using fewer computational resources. Tom was added to the HuBMAP infrastructure to run kidney FTU segmentation at scale-showcasing the value of Kaggle competitions for advancing research.

Accurate Kidney Pathological Image Classification Method Based on Deep Learning and Multi-Modal Fusion Method with Application to Membranous Nephropathy

Membranous nephropathy is one of the most prevalent conditions responsible for nephrotic syndrome in adults. It is clinically nonspecific and mainly diagnosed by kidney biopsy pathology, with three prevalent techniques: light microscopy, electron microscopy, and immunofluorescence microscopy. Manual observation of glomeruli one by one under the microscope is very time-consuming, and there are certain differences in the observation results between physicians. This study makes use of whole-slide images scanned by a light microscope as well as immunofluorescence images to classify patients with membranous nephropathy. The framework mainly includes a glomerular segmentation module, a confidence coefficient extraction module, and a multi-modal fusion module. This framework first identifies and segments the glomerulus from whole-slide images and immunofluorescence images, and then a glomerular classifier is trained to extract the features of each glomerulus. The results are then combined to produce the final diagnosis. The results of the experiments show that the F1-score of image classification results obtained by combining two kinds of features, which can reach 97.32%, is higher than those obtained by using only light-microscopy-observed images or immunofluorescent images, which reach 92.76% and 93.20%, respectively. Experiments demonstrate that considering both WSIs and immunofluorescence images is effective in improving the diagnosis of membranous nephropathy.

Computational Pipeline for Glomerular Segmentation and Association of the Quantified Regions with Prognosis of Kidney Function in IgA Nephropathy

The histopathological findings of the glomeruli from whole slide images (WSIs) of a renal biopsy play an important role in diagnosing and grading kidney disease. This study aimed to develop an automated computational pipeline to detect glomeruli and to segment the histopathological regions inside of the glomerulus in a WSI. In order to assess the significance of this pipeline, we conducted a multivariate regression analysis to determine whether the quantified regions were associated with the prognosis of kidney function in 46 cases of immunoglobulin A nephropathy (IgAN). The developed pipelines showed a mean intersection over union (IoU) of 0.670 and 0.693 for five classes (i.e., background, Bowman's space, glomerular tuft, crescentic, and sclerotic regions) against the WSI of its facility, and 0.678 and 0.609 against the WSI of the external facility. The multivariate analysis revealed that the predicted sclerotic regions, even those that were predicted by the external model, had a significant negative impact on the slope of the estimated glomerular filtration rate after biopsy. This is the first study to demonstrate that the quantified sclerotic regions that are predicted by an automated computational pipeline for the segmentation of the histopathological glomerular components on WSIs impact the prognosis of kidney function in patients with IgAN.

Vascular smooth muscle cell dysfunction in neurodegeneration

Vascular smooth muscle cells (VSMCs) are the key moderators of cerebrovascular dynamics in response to the brain's oxygen and nutrient demands. Crucially, VSMCs may provide a sensitive biomarker for neurodegenerative pathologies where vasculature is compromised. An increasing body of research suggests that VSMCs have remarkable plasticity and their pathophysiology may play a key role in the complex process of neurodegeneration. Furthermore, extrinsic risk factors, including environmental conditions and traumatic events can impact vascular function through changes in VSMC morphology. VSMC dysfunction can be characterised at the molecular level both preclinically, and clinically ex vivo. However the identification of VSMC dysfunction in living individuals is important to understand changes in vascular function at the onset and progression of neurological disorders such as dementia, Alzheimer's disease, and Parkinson's disease. A promising technique to identify changes in the state of cerebral smooth muscle is cerebrovascular reactivity (CVR) which reflects the intrinsic dynamic response of blood vessels in the brain to vasoactive stimuli in order to modulate regional cerebral blood flow (CBF). In this work, we review the role of VSMCs in the most common neurodegenerative disorders and identify physiological systems that may contribute to VSMC dysfunction. The evidence collected here identifies VSMC dysfunction as a strong candidate for novel therapeutics to combat the development and progression of neurodegeneration, and highlights the need for more research on the role of VSMCs and cerebrovascular dynamics in healthy and diseased states.

Glo-In-One: holistic glomerular detection, segmentation, and lesion characterization with large-scale web image mining

Purpose: The quantitative detection, segmentation, and characterization of glomeruli from high-resolution whole slide imaging (WSI) play essential roles in the computer-assisted diagnosis and scientific research in digital renal pathology. Historically, such comprehensive quantification requires extensive programming skills to be able to handle heterogeneous and customized computational tools. To bridge the gap of performing glomerular quantification for non-technical users, we develop the Glo-In-One toolkit to achieve holistic glomerular detection, segmentation, and characterization via a single line of command. Additionally, we release a large-scale collection of 30,000 unlabeled glomerular images to further facilitate the algorithmic development of self-supervised deep learning. Approach: The inputs of the Glo-In-One toolkit are WSIs, while the outputs are (1) WSI-level multi-class circle glomerular detection results (which can be directly manipulated with ImageScope), (2) glomerular image patches with segmentation masks, and (3) different lesion types. In the current version, the fine-grained global glomerulosclerosis (GGS) characterization is provided, including assessed-solidified-GSS (associated with hypertension-related injury), disappearing-GSS (a further end result of the SGGS becoming contiguous with fibrotic interstitium), and obsolescent-GSS (nonspecific GGS increasing with aging) glomeruli. To leverage the performance of the Glo-In-One toolkit, we introduce self-supervised deep learning to glomerular quantification via large-scale web image mining. Results: The GGS fine-grained classification model achieved a decent performance compared with baseline supervised methods while only using 10% of the annotated data. The glomerular detection achieved an average precision of 0.627 with circle representations, while the glomerular segmentation achieved a 0.955 patch-wise Dice dimilarity coefficient. Conclusion: We develop and release an open-source Glo-In-One toolkit, a software with holistic glomerular detection, segmentation, and lesion characterization. This toolkit is user-friendly to non-technical users via a single line of command. The toolbox and the 30,000 web mined glomerular images have been made publicly available at https://github.com/hrlblab/Glo-In-One.

Deep Learning Approaches for the Segmentation of Glomeruli in Kidney Histopathological Images

Deep learning is widely applied in bioinformatics and biomedical imaging, due to its ability to perform various clinical tasks automatically and accurately. In particular, the application of deep learning techniques for the automatic identification of glomeruli in histopathological kidney images can play a fundamental role, offering a valid decision support system tool for the automatic evaluation of the Karpinski metric. This will help clinicians in detecting the presence of sclerotic glomeruli in order to decide whether the kidney is transplantable or not. In this work, we implemented a deep learning framework to identify and segment sclerotic and non-sclerotic glomeruli from scanned Whole Slide Images (WSIs) of human kidney biopsies. The experiments were conducted on a new dataset collected by both the Siena and Trieste hospitals. The images were segmented using the DeepLab V2 model, with a pre-trained ResNet101 encoder, applied to 512 × 512 patches extracted from the original WSIs. The results obtained are promising and show a good performance in the segmentation task and a good generalization capacity, despite the different coloring and typology of the histopathological images. Moreover, we present a novel use of the CD10 staining procedure, which gives promising results when applied to the segmentation of sclerotic glomeruli in kidney tissues.

Data augmentation based on spatial deformations for histopathology: An evaluation in the context of glomeruli segmentation

BACKGROUND AND OBJECTIVE

The effective application of deep learning to digital histopathology is hampered by the shortage of high-quality annotated images. In this paper we focus on the supervised segmentation of glomerular structures in patches of whole slide images of renal histopathological slides. Considering a U-Net model employed for segmentation, our goal is to evaluate the impact of augmenting training data with random spatial deformations.

METHODS

The effective application of deep learning to digital histopathology is hampered by the shortage of high-quality annotated images. In this paper we focus on the supervised segmentation of glomerular structures in patches of whole slide images of renal histopathological slides. Considering a U-Net model employed for segmentation, our goal is to evaluate the impact of augmenting training data with random spatial deformations.

RESULTS

We show that augmenting training data with spatially deformed images yields an improvement of up to 0.23 in average Dice score, with respect to training with no augmentation. We demonstrate that deformations with relatively strong distortions yield the best performance increase, while previous work only report the use of deformations with low distortions. The selected deformation models yield similar performance increase, provided that their parameters are properly adjusted. We provide bounds on the optimal parameter values, obtained through parameter sampling, which is achieved in a lower computational complexity with our single-parameter method. The paper is accompanied by a framework for evaluating the impact of random spatial deformations on the performance of any U-Net segmentation model.

CONCLUSION

To our knowledge, this study is the first to evaluate the impact of random spatial deformations on the segmentation of histopathological images. Our study and framework provide tools to help practitioners and researchers to make a better usage of random spatial deformations when training deep models for segmentation.

Large-scale extraction of interpretable features provides new insights into kidney histopathology – a proof-of-concept study

Whole slide images contain a magnitude of quantitative information that may not be fully explored in qualitative visual assessments. We propose: (1) a novel pipeline for extracting a comprehensive set of visual features, which are detectable by a pathologist, as well as sub-visual features, which are not discernible by human experts and (2) perform detailed analyses on renal images from mice with experimental unilateral ureteral obstruction. An important criterion for these features is that they are easy to interpret, as opposed to features obtained from neural networks. We extract and compare features from pathological and healthy control kidneys to learn how the compartments (glomerulus, Bowman's capsule, tubule, interstitium, artery, and arterial lumen) are affected by the pathology. We define feature selection methods to extract the most informative and discriminative features. We perform statistical analyses to understand the relation of the extracted features, both individually, and in combinations, with tissue morphology and pathology. Particularly for the presented case-study, we highlight features that are affected in each compartment. With this, prior biological knowledge, such as the increase in interstitial nuclei, is confirmed and presented in a quantitative way, alongside with novel findings, like color and intensity changes in glomeruli and Bowman's capsule. The proposed approach is therefore an important step towards quantitative, reproducible, and rater-independent analysis in histopathology.

Artificial intelligence applications for pre-implantation kidney biopsy pathology practice: a systematic review

BACKGROUND

Transplant nephropathology is a highly specialized field of pathology comprising both the evaluation of organ donor biopsy for organ allocation and post-transplant graft biopsy for assessment of rejection or graft damage. The introduction of digital pathology with whole-slide imaging (WSI) in clinical research, trials and practice has catalyzed the application of artificial intelligence (AI) for histopathology, with development of novel machine-learning models for tissue interrogation and discovery. We aimed to review the literature for studies specifically applying AI algorithms to WSI-digitized pre-implantation kidney biopsy.

METHODS

A systematic search was carried out in the electronic databases PubMed-MEDLINE and Embase until 25th September, 2021 with a combination of the key terms "kidney", "biopsy", "transplantation" and "artificial intelligence" and their aliases. Studies dealing with the application of AI algorithms coupled with WSI in pre-implantation kidney biopsies were included. The main theme addressed was detection and quantification of tissue components. Extracted data were: author, year and country of the study, type of biopsy features investigated, number of cases, type of algorithm deployed, main results of the study in terms of diagnostic outcome, and the main limitations of the study.

RESULTS

Of 5761 retrieved articles, 7 met our inclusion criteria. All studies focused largely on AI-based detection and classification of glomerular structures and to a lesser extent on tubular and vascular structures. Performance of AI algorithms was excellent and promising.

CONCLUSION

All studies highlighted the importance of expert pathologist annotation to reliably train models and the need to acknowledge clinical nuances of the pre-implantation setting. Close cooperation between computer scientists and practicing as well as expert renal pathologists is needed, helping to refine the performance of AI-based models for routine pre-implantation kidney biopsy clinical practice.

Circle Representation for Medical Object Detection

Box representation has been extensively used for object detection in computer vision. Such representation is efficacious but not necessarily optimized for biomedical objects (e.g., glomeruli), which play an essential role in renal pathology. In this paper, we propose a simple circle representation for medical object detection and introduce CircleNet, an anchor-free detection framework. Compared with the conventional bounding box representation, the proposed bounding circle representation innovates in three-fold: (1) it is optimized for ball-shaped biomedical objects; (2) The circle representation reduced the degree of freedom compared with box representation; (3) It is naturally more rotation invariant. When detecting glomeruli and nuclei on pathological images, the proposed circle representation achieved superior detection performance and be more rotation-invariant, compared with the bounding box. The code has been made publicly available: https://github.com/hrlblab/CircleNet.

Classification of renal biopsy direct immunofluorescence image using multiple attention convolutional neural network

BACKGROUND AND OBJECTIVES

Direct immunofluorescence (DIF) is an important medical evaluation tool for renal pathology. In the DIF images, the deposition appearances and locations of immunoglobulin on glomeruli involve immunological characteristics of glomerulonephritis and thus can be used to aid in the identification of glomerulonephritis disease. Manual classification to such deposition patterns is time consuming and may lead to significant inter and intra operator variances. We wanted to automate the identification and fusion of deposition location and deposition appearance to assist physicians in achieving immunofluorescence reporting.

METHODS

In this paper, we propose a framework that consists of a pre-segmentation module and a classification module for automatically segmenting glomerulus object and classifying the deposition pattern of immunoglobulin on glomerulus object. For the pre-segmentation module, the glomerulus object is segmented out from the acquired DIF images using a segmentation network, which excludes other tissues and makes the classification module focus on the glomerulus. For the classification module, two branches of classifying deposition region and appearance, respectively, are formed by using multiple attentions convolutional neural network (MANet) based on the segmented images, and the classification results of the two pre-trained classification networks are fused with labels.

RESULTS

Experimental results show that the proposed framework achieves a high classification performance with an accuracy of 98% and 95% in terms of deposition region and appearance, respectively. The label fusion of deposition appearance and deposition classification is achieved with high accuracy based on well-trained classification.

CONCLUSIONS

The data show that automated and accurate patterned immunofluorescence report generation is achieved, which can effectively help improve the diagnosis of autoimmune kidney disease.

Explainable Biomarkers for Automated Glomerular and Patient-Level Disease Classification

Pathologists use multiple microscopy modalities to assess renal biopsy specimens. Besides usual diagnostic features, some changes are too subtle to be properly defined. Computational approaches have the potential to systematically quantitate subvisual clues, provide pathogenetic insight, and link to clinical outcomes. To this end, a proof-of-principle study is presented demonstrating that explainable biomarkers through machine learning can distinguish between glomerular disorders at the light-microscopy level. The proposed system used image analysis techniques and extracted 233 explainable biomarkers related to color, morphology, and microstructural texture. Traditional machine learning was then used to classify minimal change disease (MCD), membranous nephropathy (MN), and thin basement membrane nephropathy (TBMN) diseases on a glomerular and patient-level basis. The final model combined the Gini feature importance set and linear discriminant analysis classifier. Six morphologic (nuclei-to-glomerular tuft area, nuclei-to-glomerular area, glomerular tuft thickness greater than ten, glomerular tuft thickness greater than three, total glomerular tuft thickness, and glomerular circularity) and four microstructural texture features (luminal contrast using wavelets, nuclei energy using wavelets, nuclei variance using color vector LBP, and glomerular correlation using GLCM) were, together, the best performing biomarkers. Accuracies of 77% and 87% were obtained for glomerular and patient-level classification, respectively. Computational methods, using explainable glomerular biomarkers, have diagnostic value and are compatible with our existing knowledge of disease pathogenesis. Furthermore, this algorithm can be applied to clinical datasets for novel prognostic and mechanistic biomarker discovery.

Multi-view features integrated 2D\3D Net for glomerulopathy histologic types classification using ultrasound images

BACKGROUND AND OBJECTIVE

Early diagnoses and rational therapeutics of glomerulopathy can control progression and improve prognosis. The gold standard for the diagnosis of glomerulopathy is pathology by renal biopsy, which is invasive and has many contraindications. We aim to use renal ultrasonography for histologic classification of glomerulopathy.

METHODS

Ultrasonography can present multi-view sections of kidney, thus we proposed a multi-view and cross-domain integration strategy (CD-ConcatNet) to obtain more effective features and improve diagnosis accuracy. We creatively apply 2D group convolution and 3D convolution to process multiple 2D ultrasound images and extract multi-view features of renal ultrasound images. Cross-domain concatenation in each spatial resolution of feature maps is applied for more informative feature learning.

RESULTS

A total of 76 adult patients were collected and divided into training dataset (56 cases with 515 images) and validation dataset (20 cases with 180 images). We obtained the best mean accuracy of 0.83 and AUC of 0.8667 in the validation dataset.

CONCLUSION

Comparison experiments demonstrate that our designed CD-ConcatNet achieves the best classification performance and has great superiority on histologic types diagnosis. Results also prove that the integration of multi-view ultrasound images is beneficial for histologic classification and ultrasound images can indeed provide discriminating information for histologic diagnosis.

Glomerular Disease Classification and Lesion Identification by Machine Learning

Background

Classification of glomerular diseases and identification of glomerular lesions require careful morphological examination by experienced nephropathologists, which is labor-intensive, time-consuming, and prone to interobserver variability. In this regard, recent advance in machine learning-based image analysis is promising.

Methods

We combined Mask Region-based Convolutional Neural Networks (Mask R–CNN) with an additional classification step to build a glomerulus detection model using human kidney biopsy samples. A Long Short-Term Memory (LSTM) recurrent neural network was applied for glomerular disease classification, and another two-stage model using ResNeXt-101 was constructed for glomerular lesion identification in cases of lupus nephritis.

Results

The detection model showed state-of-the-art performance on variedly stained slides with F1 scores up to 0.944. The disease classification model showed good accuracies up to 0.940 on recognizing different glomerular diseases based on H&E whole slide images. The lesion identification model demonstrated high discriminating power with area under the receiver operating characteristic curve up to 0.947 for various glomerular lesions. Models showed good generalization on external testing datasets.

Conclusion

This study is the first-of-its-kind showing how each step of kidney biopsy interpretation carried out by nephropathologists can be captured and simulated by machine learning models. The models were integrated into a whole slide image viewing and annotating platform to enable nephropathologists to review, correct, and confirm the inference results. Further improvement on model performances and incorporating inputs from immunofluorescence, electron microscopy, and clinical data might realize actual clinical use.

Automated assessment of glomerulosclerosis and tubular atrophy using deep learning

In kidney transplantations, pathologists evaluate the architecture of both glomeruli, interstitium and tubules to assess the nephron status. An accurate assessment of glomerulosclerosis and tubular atrophy is crucial for determining kidney acceptance, which is currently based on the pathologists' histological evaluations on renal biopsies in addition to clinical data. In this work, we present an automated algorithm, called RENTAG (Robust EvaluatioN of Tubular Atrophy & Glomerulosclerosis), for the segmentation and classification of glomerular and tubular structures in histopathological images. The proposed novel strategy combines the accuracy of a level-set with the semantic segmentation of convolutional neural networks to detect the glomeruli and tubules contours. In the TEST set, our method exhibited excellent performance in both glomeruli (dice score: 0.9529) and tubule (dice score: 0.9174) detection and outperformed all the compared methods. To the best of our knowledge, the RENTAG algorithm is the first fully automated method capable of quantifying glomerulosclerosis and tubular atrophy in digital histological images. The developed software can be employed for the analysis of pre-transplantation biopsies to support the pathologists' diagnostic activity.

Artificial intelligence and algorithmic computational pathology: an introduction with renal allograft examples

Whole slide imaging, which is an important technique in the field of digital pathology, has recently been the subject of increased interest and avenues for utilisation, and with more widespread whole slide image (WSI) utilisation, there will also be increased interest in and implementation of image analysis (IA) techniques. IA includes artificial intelligence (AI) and targeted or hypothesis-driven algorithms. In the overall pathology field, the number of citations related to these topics has increased in recent years. Renal pathology is one anatomical pathology subspecialty that has utilised WSIs and IA algorithms; it can be argued that renal transplant pathology could be particularly suited for whole slide imaging and IA, as renal transplant pathology is frequently classified by use of the semiquantitative Banff classification of renal allograft pathology. Hypothesis-driven/targeted algorithms have been used in the past for the assessment of a variety of features in the kidney (e.g. interstitial fibrosis, tubular atrophy, inflammation); in recent years, the amount of research has particularly increased in the area of AI/machine learning for the identification of glomeruli, for histological segmentation, and for other applications. Deep learning is the form of machine learning that is most often used for such AI approaches to the 'big data' of pathology WSIs, and deep learning methods such as artificial neural networks (ANNs)/convolutional neural networks (CNNs) are utilised. Unsupervised and supervised AI algorithms can be employed to accomplish image or semantic classification. In this review, AI and other IA algorithms applied to WSIs are discussed, and examples from renal pathology are covered, with an emphasis on renal transplant pathology.

Artificial intelligence driven next-generation renal histomorphometry

PURPOSE OF REVIEW

Successful integration of artificial intelligence into extant clinical workflows is contingent upon a number of factors including clinician comprehension and interpretation of computer vision. This article discusses how image analysis and machine learning have enabled comprehensive characterization of kidney morphology for development of automated diagnostic and prognostic renal pathology applications.

RECENT FINDINGS

The primordial digital pathology informatics work employed classical image analysis and machine learning to prognosticate renal disease. Although this classical approach demonstrated tremendous potential, subsequent advancements in hardware technology rendered artificial neural networks '(ANNs) the method of choice for machine vision in computational pathology'. Offering rapid and reproducible detection, characterization and classification of kidney morphology, ANNs have facilitated the development of diagnostic and prognostic applications. In addition, modern machine learning with ANNs has revealed novel biomarkers in kidney disease, demonstrating the potential for machine vision to elucidate novel pathologic mechanisms beyond extant clinical knowledge.

SUMMARY

Despite the revolutionary developments potentiated by modern machine learning, several challenges remain, including data quality control and curation, image annotation and ontology, integration of multimodal data and interpretation of machine vision or 'opening the black box'. Resolution of these challenges will not only revolutionize diagnostic pathology but also pave the way for precision medicine and integration of artificial intelligence in the process of care.

A U-Net based framework to quantify glomerulosclerosis in digitized PAS and H&E stained human tissues

Reliable counting of glomeruli and evaluation of glomerulosclerosis in renal specimens are essential steps to assess morphological changes in kidney and identify individuals requiring treatment. Because microscopic identification of sclerosed glomeruli performed under the microscope is labor intensive, we developed a deep learning (DL) approach to identify and classify glomeruli as normal or sclerosed in digital whole slide images (WSIs). The segmentation and classification of glomeruli was performed by the U-Net model. Subsequently, glomerular classifications were refined based on glomerular histomorphometry. The U-Net model was trained using patches from Periodic Acid-Schiff (PAS) stained WSIs (n=31) from the AIDPATH - a multi-center dataset, and then tested on an independent set of WSIs (n=20) including PAS (n=6), and hematoxylin and eosin (H&E) stained WSIs (n=14) from four other institutions. The training and test WSIs were obtained from formalin fixed and paraffin embedded blocks with of human kidney specimens each presenting various proportions of normal and sclerosed glomeruli. In the PAS stained WSIs, normal and sclerosed glomeruli were respectively classified with the F1-score of 97.5% and 68.8%. In the H&E stained WSIs, the F1-scores of 90.8% and 78.1% were achieved. Regardless the tissue staining, the glomeruli in the test WSIs were classified with the F1-score of 94.5% (n=923, normal) and 76.8% for (n=261, sclerosed). These results demonstrate for the first time that a framework based on the U-Net model trained with glomerular patches from PAS stained WSIs can reliably segment and classify normal and sclerosed glomeruli in PAS and also H&E stained WSIs. Our approach yielded higher accuracy of glomerular classifications than some of the recently published methods. Additionally, our test set of images with ground truth is publicly available.

Digital pathology and computational image analysis in nephropathology

The emergence of digital pathology - an image-based environment for the acquisition, management and interpretation of pathology information supported by computational techniques for data extraction and analysis - is changing the pathology ecosystem. In particular, by virtue of our new-found ability to generate and curate digital libraries, the field of machine vision can now be effectively applied to histopathological subject matter by individuals who do not have deep expertise in machine vision techniques. Although these novel approaches have already advanced the detection, classification, and prognostication of diseases in the fields of radiology and oncology, renal pathology is just entering the digital era, with the establishment of consortia and digital pathology repositories for the collection, analysis and integration of pathology data with other domains. The development of machine-learning approaches for the extraction of information from image data, allows for tissue interrogation in a way that was not previously possible. The application of these novel tools are placing pathology centre stage in the process of defining new, integrated, biologically and clinically homogeneous disease categories, to identify patients at risk of progression, and shifting current paradigms for the treatment and prevention of kidney diseases.

A Deep Learning Instance Segmentation Approach for Global Glomerulosclerosis Assessment in Donor Kidney Biopsies

The histological assessment of glomeruli is fundamental for determining if a kidney is suitable for transplantation. The Karpinski score is essential to evaluate the need for a single or dual kidney transplant and includes the ratio between the number of sclerotic glomeruli and the overall number of glomeruli in a kidney section. The manual evaluation of kidney biopsies performed by pathologists is time-consuming and error-prone, so an automatic framework to delineate all the glomeruli present in a kidney section can be very useful. Our experiments have been conducted on a dataset provided by the Department of Emergency and Organ Transplantations (DETO) of Bari University Hospital. This dataset is composed of 26 kidney biopsies coming from 19 donors. The rise of Convolutional Neural Networks (CNNs) has led to a realm of methods which are widely applied in Medical Imaging. Deep learning techniques are also very promising for the segmentation of glomeruli, with a variety of existing approaches. Many methods only focus on semantic segmentation—which consists in segmentation of individual pixels—or ignore the problem of discriminating between non-sclerotic and sclerotic glomeruli, so these approaches are not optimal or inadequate for transplantation assessment. In this work, we employed an end-to-end fully automatic approach based on Mask R-CNN for instance segmentation and classification of glomeruli. We also compared the results obtained with a baseline based on Faster R-CNN, which only allows detection at bounding boxes level. With respect to the existing literature, we improved the Mask R-CNN approach in sliding window contexts, by employing a variant of the Non-Maximum Suppression (NMS) algorithm, which we called Non-Maximum-Area Suppression (NMAS). The obtained results are very promising, leading to improvements over existing literature. The baseline Faster R-CNN-based approach obtained an F-Measure of 0.904 and 0.667 for non-sclerotic and sclerotic glomeruli, respectively. The Mask R-CNN approach has a significant improvement over the baseline, obtaining an F-Measure of 0.925 and 0.777 for non-sclerotic and sclerotic glomeruli, respectively. The proposed method is very promising for the instance segmentation and classification of glomeruli, and allows to make a robust evaluation of global glomerulosclerosis. We also compared Karpinski score obtained with our algorithm to that obtained with pathologists’ annotations to show the soundness of the proposed workflow from a clinical point of view.

CircleNet: Anchor-free Glomerulus Detection with Circle Representation

Object detection networks are powerful in computer vision, but not necessarily optimized for biomedical object detection. In this work, we propose CircleNet, a simple anchor-free detection method with circle representation for detection of the ball-shaped glomerulus. Different from the traditional bounding box based detection method, the bounding circle (1) reduces the degrees of freedom of detection representation, (2) is naturally rotation invariant, (3) and optimized for ball-shaped objects. The key innovation to enable this representation is the anchor-free framework with the circle detection head. We evaluate CircleNet in the context of detection of glomerulus. CircleNet increases average precision of the glomerulus detection from 0.598 to 0.647. Another key advantage is that CircleNet achieves better rotation consistency compared with bounding box representations.

Evaluation of the Classification Accuracy of the Kidney Biopsy Direct Immunofluorescence through Convolutional Neural Networks

BACKGROUND AND OBJECTIVES

Immunohistopathology is an essential technique in the diagnostic workflow of a kidney biopsy. Deep learning is an effective tool in the elaboration of medical imaging. We wanted to evaluate the role of a convolutional neural network as a support tool for kidney immunofluorescence reporting.

DESIGN, SETTING, PARTICIPANTS, & MEASUREMENTS

High-magnification (×400) immunofluorescence images of kidney biopsies performed from the year 2001 to 2018 were collected. The report, adopted at the Division of Nephrology of the AOU Policlinico di Modena, describes the specimen in terms of "appearance," "distribution," "location," and "intensity" of the glomerular deposits identified with fluorescent antibodies against IgG, IgA, IgM, C1q and C3 complement fractions, fibrinogen, and κ- and λ-light chains. The report was used as ground truth for the training of the convolutional neural networks.

RESULTS

In total, 12,259 immunofluorescence images of 2542 subjects undergoing kidney biopsy were collected. The test set analysis showed accuracy values between 0.79 ("irregular capillary wall" feature) and 0.94 ("fine granular" feature). The agreement test of the results obtained by the convolutional neural networks with respect to the ground truth showed similar values to three pathologists of our center. Convolutional neural networks were 117 times faster than human evaluators in analyzing 180 test images. A web platform, where it is possible to upload digitized images of immunofluorescence specimens, is available to evaluate the potential of our approach.

CONCLUSIONS

The data showed that the accuracy of convolutional neural networks is comparable with that of pathologists experienced in the field.

Classification of glomerular pathological findings using deep learning and nephrologist-AI collective intelligence approach

BACKGROUND

Automated classification of glomerular pathological findings is potentially beneficial in establishing an efficient and objective diagnosis in renal pathology. While previous studies have verified the artificial intelligence (AI) models for the classification of global sclerosis and glomerular cell proliferation, there are several other glomerular pathological findings required for diagnosis, and the comprehensive models for the classification of these major findings have not yet been reported. Whether the cooperation between these AI models and clinicians improves diagnostic performance also remains unknown. Here, we developed AI models to classify glomerular images for major findings required for pathological diagnosis and investigated whether those models could improve the diagnostic performance of nephrologists.

METHODS

We used a dataset of 283 kidney biopsy cases comprising 15,888 glomerular images that were annotated by a total of 25 nephrologists. AI models to classify seven pathological findings: global sclerosis, segmental sclerosis, endocapillary proliferation, mesangial matrix accumulation, mesangial cell proliferation, crescent, and basement membrane structural changes, were constructed using deep learning by fine-tuning of InceptionV3 convolutional neural network. Subsequently, we compared the agreement to truth labels between majority decision among nephrologists with or without the AI model as a voter.

RESULTS

Our model for global sclerosis showed high performance (area under the curve: periodic acid-Schiff, 0.986; periodic acid methenamine silver, 0.983); the models for the other findings also showed performance close to those of nephrologists. By adding the AI model output to majority decision among nephrologists, out of the 14 constructed models, the results of the majority decision showed improvement in sensitivity for 10 models (four of them were statistically significant) and specificity for eight models (five significant).

CONCLUSION

Our study showed a proof-of-concept for the classification of multiple glomerular findings in a comprehensive method of deep learning and suggested its potential effectiveness in improving diagnostic accuracy of clinicians.

Identification of glomerular lesions and intrinsic glomerular cell types in kidney diseases via deep learning

Identification of glomerular lesions and structures is a key point for pathological diagnosis, treatment instructions, and prognosis evaluation in kidney diseases. These time‐consuming tasks require a more accurate and reproducible quantitative analysis method. We established derivation and validation cohorts composed of 400 Chinese patients with immunoglobulin A nephropathy (IgAN) retrospectively. Deep convolutional neural networks and biomedical image processing algorithms were implemented to locate glomeruli, identify glomerular lesions (global and segmental glomerular sclerosis, crescent, and none of the above), identify and quantify different intrinsic glomerular cells, and assess a network‐based mesangial hypercellularity score in periodic acid–Schiff (PAS)‐stained slides. Our framework achieved 93.1% average precision and 94.9% average recall for location of glomeruli, and a total Cohen's kappa of 0.912 [95% confidence interval (CI), 0.892–0.932] for glomerular lesion classification. The evaluation of global, segmental glomerular sclerosis, and crescents achieved Cohen's kappa values of 1.0, 0.776, 0.861, and 95% CI of (1.0, 1.0), (0.727, 0.825), (0.824, 0.898), respectively. The well‐designed neural network can identify three kinds of intrinsic glomerular cells with 92.2% accuracy, surpassing the about 5–11% average accuracy of junior pathologists. Statistical interpretation shows that there was a significant difference (P value < 0.0001) between this analytic renal pathology system (ARPS) and four junior pathologists for identifying mesangial and endothelial cells, while that for podocytes was similar, with P value = 0.0602. In addition, this study indicated that the ratio of mesangial cells, endothelial cells, and podocytes within glomeruli from IgAN was 0.41:0.36:0.23, and the performance of mesangial score assessment reached a Cohen's kappa of 0.42 and 95% CI (0.18, 0.69). The proposed computer‐aided diagnosis system has feasibility for quantitative analysis and auxiliary recognition of glomerular pathological features. © 2020 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of Pathological Society of Great Britain and Ireland.

Semantic Segmentation Framework for Glomeruli Detection and Classification in Kidney Histological Sections

The evaluation of kidney biopsies performed by expert pathologists is a crucial process for assessing if a kidney is eligible for transplantation. In this evaluation process, an important step consists of the quantification of global glomerulosclerosis, which is the ratio between sclerotic glomeruli and the overall number of glomeruli. Since there is a shortage of organs available for transplantation, a quick and accurate assessment of global glomerulosclerosis is essential for retaining the largest number of eligible kidneys. In the present paper, the authors introduce a Computer-Aided Diagnosis (CAD) system to assess global glomerulosclerosis. The proposed tool is based on Convolutional Neural Networks (CNNs). In particular, the authors considered approaches based on Semantic Segmentation networks, such as SegNet and DeepLab v3+. The dataset has been provided by the Department of Emergency and Organ Transplantations (DETO) of Bari University Hospital, and it is composed of 26 kidney biopsies coming from 19 donors. The dataset contains 2344 non-sclerotic glomeruli and 428 sclerotic glomeruli. The proposed model consents to achieve promising results in the task of automatically detecting and classifying glomeruli, thus easing the burden of pathologists. We get high performance both at pixel-level, achieving mean F-score higher than 0.81, and Weighted Intersection over Union (IoU) higher than 0.97 for both SegNet and Deeplab v3+ approaches, and at object detection level, achieving 0.924 as best F-score for non-sclerotic glomeruli and 0.730 as best F-score for sclerotic glomeruli.

Multi-template matching: a versatile tool for object-localization in microscopy images

BACKGROUND

The localization of objects of interest is a key initial step in most image analysis workflows. For biomedical image data, classical image-segmentation methods like thresholding or edge detection are typically used. While those methods perform well for labelled objects, they are reaching a limit when samples are poorly contrasted with the background, or when only parts of larger structures should be detected. Furthermore, the development of such pipelines requires substantial engineering of analysis workflows and often results in case-specific solutions. Therefore, we propose a new straightforward and generic approach for object-localization by template matching that utilizes multiple template images to improve the detection capacity.

RESULTS

We provide a new implementation of template matching that offers higher detection capacity than single template approach, by enabling the detection of multiple template images. To provide an easy-to-use method for the automatic localization of objects of interest in microscopy images, we implemented multi-template matching as a Fiji plugin, a KNIME workflow and a python package. We demonstrate its application for the localization of entire, partial and multiple biological objects in zebrafish and medaka high-content screening datasets. The Fiji plugin can be installed by activating the Multi-Template-Matching and IJ-OpenCV update sites. The KNIME workflow is available on nodepit and KNIME Hub. Source codes and documentations are available on GitHub (https://github.com/multi-template-matching).

CONCLUSION

The novel multi-template matching is a simple yet powerful object-localization algorithm, that requires no data-pre-processing or annotation. Our implementation can be used out-of-the-box by non-expert users for any type of 2D-image. It is compatible with a large variety of applications including, for instance, analysis of large-scale datasets originating from automated microscopy, detection and tracking of objects in time-lapse assays, or as a general image-analysis step in any custom processing pipelines. Using different templates corresponding to distinct object categories, the tool can also be used for classification of the detected regions.

Glomerulosclerosis identification in whole slide images using semantic segmentation

BACKGROUND AND OBJECTIVE

Glomeruli identification, i.e., detection and characterization, is a key procedure in many nephropathology studies. In this paper, semantic segmentation based on convolutional neural networks (CNN) is proposed to detect glomeruli using Whole Slide Imaging (WSI) follows by a classification CNN to divide the glomeruli into normal and sclerosed.

METHODS

Comparison between U-Net and SegNet CNNs is performed for pixel-level segmentation considering both a two and three class problem, that is, a) non-glomerular and glomerular structures and b) non-glomerular normal glomerular and sclerotic structures. The two class semantic segmentation result is then used for a CNN classification where glomerular regions are divided into normal and global sclerosed glomeruli.

RESULTS

These methods were tested on a dataset composed of 47 WSIs belonging to human kidney sections stained with Periodic Acid Schiff (PAS). The best approach was the SegNet for two class segmentation follows by a fine-tuned AlexNet network to characterize the glomeruli. 98.16% of accuracy was obtained with this process of consecutive CNNs (SegNet-AlexNet) for segmentation and classification.

CONCLUSION

The results obtained demonstrate that the sequential CNN segmentation-classification strategy achieves higher accuracy reducing misclassified cases and therefore being the methodology proposed for glomerulosclerosis detection.

Generative Adversarial Networks for Facilitating Stain-Independent Supervised and Unsupervised Segmentation: A Study on Kidney Histology

A major challenge in the field of segmentation in digital pathology is given by the high effort for manual data annotations in combination with many sources introducing variability in the image domain. This requires methods that are able to cope with variability without requiring to annotate a large amount of samples for each characteristic. In this paper, we develop approaches based on adversarial models for image-to-image translation relying on unpaired training. Specifically, we propose approaches for stain-independent supervised segmentation relying on image-to-image translation for obtaining an intermediate representation. Furthermore, we develop a fully-unsupervised segmentation approach exploiting image-to-image translation to convert from the image to the label domain. Finally, both approaches are combined to obtain optimum performance in unsupervised segmentation independent of the characteristics of the underlying stain. Experiments on patches showing kidney histology proof that stain-translation can be performed highly effectively and can be used for domain adaptation to obtain independence of the underlying stain. It is even capable of facilitating the underlying segmentation task, thereby boosting the accuracy if an appropriate intermediate stain is selected. Combining domain adaptation with unsupervised segmentation finally showed the most significant improvements.