Segmentation of Heavily Clustered Nuclei from Histopathological Images

Enhancing AI Research for Breast Cancer: A Comprehensive Review of Tumor-Infiltrating Lymphocyte Datasets

The field of immunology is fundamental to our understanding of the intricate dynamics of the tumor microenvironment. In particular, tumor-infiltrating lymphocyte (TIL) assessment emerges as essential aspect in breast cancer cases. To gain comprehensive insights, the quantification of TILs through computer-assisted pathology (CAP) tools has become a prominent approach, employing advanced artificial intelligence models based on deep learning techniques. The successful recognition of TILs requires the models to be trained, a process that demands access to annotated datasets. Unfortunately, this task is hampered not only by the scarcity of such datasets, but also by the time-consuming nature of the annotation phase required to create them. Our review endeavors to examine publicly accessible datasets pertaining to the TIL domain and thereby become a valuable resource for the TIL community. The overall aim of the present review is thus to make it easier to train and validate current and upcoming CAP tools for TIL assessment by inspecting and evaluating existing publicly available online datasets.

Optimizing deep learning-based segmentation of densely packed cells using cell surface markers

BACKGROUND

Spatial molecular profiling depends on accurate cell segmentation. Identification and quantitation of individual cells in dense tissues, e.g. highly inflamed tissue caused by viral infection or immune reaction, remains a challenge.

METHODS

We first assess the performance of 18 deep learning-based cell segmentation models, either pre-trained or trained by us using two public image sets, on a set of immunofluorescence images stained with immune cell surface markers in skin tissue obtained during human herpes simplex virus (HSV) infection. We then further train eight of these models using up to 10,000+ training instances from the current image set. Finally, we seek to improve performance by tuning parameters of the most successful method from the previous step.

RESULTS

The best model before fine-tuning achieves a mean Average Precision (mAP) of 0.516. Prediction performance improves substantially after training. The best model is the cyto model from Cellpose. After training, it achieves an mAP of 0.694; with further parameter tuning, the mAP reaches 0.711.

CONCLUSION

Selecting the best model among the existing approaches and further training the model with images of interest produce the most gain in prediction performance. The performance of the resulting model compares favorably to human performance. The imperfection of the final model performance can be attributed to the moderate signal-to-noise ratio in the imageset.

High-Plex Assessment of Biomarkers in Tumors

The assessment of biomarkers plays a critical role in the diagnosis and treatment of many cancers. Biomarkers not only provide diagnostic, prognostic, or predictive information but also can act as effective targets for new pharmaceutical therapies. As the utility of biomarkers increases, it becomes more important to utilize accurate and efficient methods for biomarker discovery and, ultimately, clinical assessment. High-plex imaging studies, defined here as assessment of 8 or more biomarkers on a single slide, have become the method of choice for biomarker discovery and assessment of biomarker spatial context. In this review, we discuss methods of measuring biomarkers in slide-mounted tissue samples, detail the various high-plex methods that allow for the simultaneous assessment of multiple biomarkers in situ, and describe the impact of high-plex biomarker assessment on the future of anatomic pathology.

High resolution spatial profiling of kidney injury and repair using RNA hybridization-based in situ sequencing

Emerging spatially resolved transcriptomics technologies allow for the measurement of gene expression in situ at cellular resolution. We apply direct RNA hybridization-based in situ sequencing (dRNA HybISS, Cartana part of 10xGenomics) to compare male and female healthy mouse kidneys and the male kidney injury and repair timecourse. A pre-selected panel of 200 genes is used to identify cell state dynamics patterns during injury and repair. We develop a new computational pipeline, CellScopes, for the rapid analysis, multi-omic integration and visualization of spatially resolved transcriptomic datasets. The resulting dataset allows us to resolve 13 kidney cell types within distinct kidney niches, dynamic alterations in cell state over the course of injury and repair and cell-cell interactions between leukocytes and kidney parenchyma. At late timepoints after injury, C3+ leukocytes are enriched near pro-inflammatory, failed-repair proximal tubule cells. Integration of snRNA-seq dataset from the same injury and repair samples also allows us to impute the spatial localization of genes not directly measured by dRNA HybISS.

Software BreastAnalyser for the semi-automatic analysis of breast cancer immunohistochemical images

Breast cancer is the most diagnosed cancer worldwide and represents the fifth cause of cancer mortality globally. It is a highly heterogeneous disease, that comprises various molecular subtypes, often diagnosed by immunohistochemistry. This technique is widely employed in basic, translational and pathological anatomy research, where it can support the oncological diagnosis, therapeutic decisions and biomarker discovery. Nevertheless, its evaluation is often qualitative, raising the need for accurate quantitation methodologies. We present the software BreastAnalyser, a valuable and reliable tool to automatically measure the area of 3,3'-diaminobenzidine tetrahydrocholoride (DAB)-brown-stained proteins detected by immunohistochemistry. BreastAnalyser also automatically counts cell nuclei and classifies them according to their DAB-brown-staining level. This is performed using sophisticated segmentation algorithms that consider intrinsic image variability and save image normalization time. BreastAnalyser has a clean, friendly and intuitive interface that allows to supervise the quantitations performed by the user, to annotate images and to unify the experts' criteria. BreastAnalyser was validated in representative human breast cancer immunohistochemistry images detecting various antigens. According to the automatic processing, the DAB-brown area was almost perfectly recognized, being the average difference between true and computer DAB-brown percentage lower than 0.7 points for all sets. The detection of nuclei allowed proper cell density relativization of the brown signal for comparison purposes between the different patients. BreastAnalyser obtained a score of 85.5 using the system usability scale questionnaire, which means that the tool is perceived as excellent by the experts. In the biomedical context, the connexin43 (Cx43) protein was found to be significantly downregulated in human core needle invasive breast cancer samples when compared to normal breast, with a trend to decrease as the subtype malignancy increased. Higher Cx43 protein levels were significantly associated to lower cancer recurrence risk in Oncotype DX-tested luminal B HER2- breast cancer tissues. BreastAnalyser and the annotated images are publically available https://citius.usc.es/transferencia/software/breastanalyser for research purposes.

Recurrence risk stratification of hepatocellular carcinomas based on immune gene expression and features extracted from pathological images

BACKGROUND

Immune-based therapy is a promising type of treatment for hepatocellular carcinoma (HCC) but has only been partially successful due to the high heterogeneity in HCC tumor. The differences in the degree of tumor cell progression and in the activity of tumor immune microenvironment could lead to varied clinical outcome. Accurate subgrouping for recurrence risk is an approach to address the issue of such heterogeneity. It remains under investigation as whether integrating quantitative whole slide image (WSI) features with the expression profile of immune marker genes can improve the risk stratification, and whether clinical outcome prediction can assist in understanding molecular biology that drives the outcome.

METHODS

We included a total of 231 patients from the Cancer Genome Atlas Liver Hepatocellular Carcinoma (TCGA-LIHC) project. For each patient, we extracted 18 statistical metrics corresponding to a global region of interest and 135 features regarding nucleus shape from WSI. A risk score was developed using these image features with high-dimensional survival modeling. We also introduced into the model the expression profile of 66 representative marker genes relevant to currently available immunotherapies. We stratified all patients into higher and lower-risk subgroup based on the final risk score selected from multiple models generated, and further investigated underlying molecular mechanisms associated with the risk stratification.

RESULTS

One WSI feature and three immune marker genes were selected into the final recurrence-free survival (RFS) prediction model following the best integrated modeling framework. The resultant score showed a significantly improved prediction performance on the test dataset (mean time-dependent AUCs = 0.707) as compared to those of other types (e.g: mean time-dependent AUCs of AJCC tumor stage = 0.525) of input data integration. To assess that the risk score could provide a higher-resolution risk stratification, a lower-risk subgroup (or a higher-risk subgroup) was arbitrarily assigned according to score falling below (or above) the median score. The lower risk subgroup had significantly longer median RFS time than that of the higher-risk patients (median RFS = 903 vs. 265 days, log-rank test p-value< 0.0001). Additionally, the higher-risk subgroup, in contrast to the lower-risk patients were characterized with a significant downregulation of immune checkpoint genes, suppressive signal in tumor immune response pathways, and depletion of CD8 T cells. These observations for the higher-risk subgroup suggest that new targets for adoptive or checkpoint-based combined systemic therapies may be useful.

CONCLUSION

We developed a novel prognostic model to predict RFS for HCC patients, using one feature that can be automatically extracted from routine histopathological images, as well as the expression profiles of three immune marker genes. The methodology used in this paper demonstrates the feasibility of developing prognostic models that provide both useful risk stratification along with valuable biological insights into the underlying characteristics of the subgroups identified.

Optimizing Deep Learning-Based Segmentation of Densely Packed Cells using Cell Surface Markers

Background

Spatial molecular profiling depends on accurate cell segmentation. Identification and quantitation of individual cells in dense tissues, e.g. highly inflamed tissue caused by viral infection or immune reaction, remains a challenge.

Methods

We first assess the performance of 18 deep learning-based cell segmentation models, either pre-trained or trained by us using two public image sets, on a set of immunofluorescence images stained with immune cell surface markers in skin tissue obtained during human herpes simplex virus (HSV) infection. We then further train eight of these models using up to 10,000+ training instances from the current image set. Finally, we seek to improve performance by tuning parameters of the most successful method from the previous step.

Results

The best model before fine-tuning achieves a mean Average Precision (mAP) of 0.516. Prediction performance improves substantially after training. The best model is the cyto model from Cellpose. After training, it achieves an mAP of 0.694; with further parameter tuning, the mAP reaches 0.711.

Conclusion

Selecting the best model among the existing approaches and further training the model with images of interest produce the most gain in prediction performance. The performance of the resulting model compares favorably to human performance. The imperfection of the final model performance can be attributed to the moderate signal-to-noise ratio i the imageset.

Breast Cancer Histopathological Images Segmentation Using Deep Learning

Hospitals generate a significant amount of medical data every day, which constitute a very rich database for research. Today, this database is still not exploitable because to make its valorization possible, the images require an annotation which remains a costly and difficult task. Thus, the use of an unsupervised segmentation method could facilitate the process. In this article, we propose two approaches for the semantic segmentation of breast cancer histopathology images. On the one hand, an autoencoder architecture for unsupervised segmentation is proposed, and on the other hand, an improvement U-Net architecture for supervised segmentation is proposed. We evaluate these models on a public dataset of histological images of breast cancer. In addition, the performance of our segmentation methods is measured using several evaluation metrics such as accuracy, recall, precision and F1 score. The results are competitive with those of other modern methods.

A Novel Heteromorphous Convolutional Neural Network for Automated Assessment of Tumors in Colon and Lung Histopathology Images

The automated assessment of tumors in medical image analysis encounters challenges due to the resemblance of colon and lung tumors to non-mitotic nuclei and their heteromorphic characteristics. An accurate assessment of tumor nuclei presence is crucial for determining tumor aggressiveness and grading. This paper proposes a new method called ColonNet, a heteromorphous convolutional neural network (CNN) with a feature grafting methodology categorically configured for analyzing mitotic nuclei in colon and lung histopathology images. The ColonNet model consists of two stages: first, identifying potential mitotic patches within the histopathological imaging areas, and second, categorizing these patches into squamous cell carcinomas, adenocarcinomas (lung), benign (lung), benign (colon), and adenocarcinomas (colon) based on the model's guidelines. We develop and employ our deep CNNs, each capturing distinct structural, textural, and morphological properties of tumor nuclei, to construct the heteromorphous deep CNN. The execution of the proposed ColonNet model is analyzed by its comparison with state-of-the-art CNNs. The results demonstrate that our model surpasses others on the test set, achieving an impressive F1 score of 0.96, sensitivity and specificity of 0.95, and an area under the accuracy curve of 0.95. These outcomes underscore our hybrid model's superior performance, excellent generalization, and accuracy, highlighting its potential as a valuable tool to support pathologists in diagnostic activities.

Optimized detection and segmentation of nuclei in gastric cancer images using stain normalization and blurred artifact removal

Histological analysis with microscopy is the gold standard to diagnose and stage cancer, where slides or whole slide images are analyzed for cell morphological and spatial features by pathologists. The nuclei of cancerous cells are characterized by nonuniform chromatin distribution, irregular shapes, and varying size. As nucleus area and shape alone carry prognostic value, detection and segmentation of nuclei are among the most important steps in disease grading. However, evaluation of nuclei is a laborious, time-consuming, and subjective process with large variation among pathologists. Recent advances in digital pathology have allowed significant applications in nuclei detection, segmentation, and classification, but automated image analysis is greatly affected by staining factors, scanner variability, and imaging artifacts, requiring robust image preprocessing, normalization, and segmentation methods for clinically satisfactory results. In this paper, we aimed to evaluate and compare the digital image analysis techniques used in clinical pathology and research in the setting of gastric cancer. A literature review was conducted to evaluate potential methods of improving nuclei detection. Digitized images of 35 patients from a retrospective cohort of gastric adenocarcinoma at Oulu University Hospital in 1987-2016 were annotated for nuclei (n = 9085) by expert pathologists and 14 images of different cancer types from public TCGA dataset with annotated nuclei (n = 7000) were used as a comparison to evaluate applicability in other cancer types. The detection and segmentation accuracy with the selected color normalization and stain separation techniques were compared between the methods. The extracted information can be supplemented by patient's medical data and fed to the existing statistical clinical tools or subjected to subsequent AI-assisted classification and prediction models. The performance of each method is evaluated by several metrics against the annotations done by expert pathologists. The F1-measure of 0.854 ± 0.068 is achieved with color normalization for the gastric cancer dataset, and 0.907 ± 0.044 with color deconvolution for the public dataset, showing comparable results to the earlier state-of-the-art works. The developed techniques serve as a basis for further research on application and interpretability of AI-assisted tools for gastric cancer diagnosis.

Clopidogrel Administration Impairs Post-Stroke Learning and Memory Recovery in Mice

Clopidogrel, which is one of the most prescribed antiplatelet medications in the world, is given to stroke survivors for the prevention of secondary cardiovascular events. Clopidogrel exerts its antiplatelet activity via antagonism of the P2Y12 receptor (P2RY12). Although not widely known or considered during the initial clinical trials for clopidogrel, P2RY12 is also expressed on microglia, which are the brain's immune cells, where the receptor facilitates chemotactic migration toward sites of cellular damage. If microglial P2RY12 is blocked, microglia lose the ability to migrate to damaged sites and carry out essential repair processes. We aimed to investigate whether administering clopidogrel to mice post-stroke was associated with (i) impaired motor skills and cognitive recovery; (ii) physiological changes, such as survival rate and body weight; (iii) changes in the neurovascular unit, including blood vessels, microglia, and neurons; and (iv) changes in immune cells. Photothrombotic stroke (or sham surgery) was induced in adult male mice. From 24 h post-stroke, mice were treated daily for 14 days with either clopidogrel or a control. Cognitive performance (memory and learning) was assessed using a mouse touchscreen platform (paired associated learning task), while motor impairment was assessed using the cylinder task for paw asymmetry. On day 15, the mice were euthanized and their brains were collected for immunohistochemistry analysis. Clopidogrel administration significantly impaired learning and memory recovery, reduced mouse survival rates, and reduced body weight post-stroke. Furthermore, clopidogrel significantly increased vascular leakage, significantly increased the number and appearance of microglia, and significantly reduced the number of T cells within the peri-infarct region post-stroke. These data suggest that clopidogrel hampers cognitive performance post-stroke. This effect is potentially mediated by an increase in vascular permeability post-stroke, providing a pathway for clopidogrel to access the central nervous system, and thus, interfere in repair and recovery processes.

Grading of lung adenocarcinomas with simultaneous segmentation by artificial intelligence (GLASS-AI)

Preclinical genetically engineered mouse models (GEMMs) of lung adenocarcinoma are invaluable for investigating molecular drivers of tumor formation, progression, and therapeutic resistance. However, histological analysis of these GEMMs requires significant time and training to ensure accuracy and consistency. To achieve a more objective and standardized analysis, we used machine learning to create GLASS-AI, a histological image analysis tool that the broader cancer research community can utilize to grade, segment, and analyze tumors in preclinical models of lung adenocarcinoma. GLASS-AI demonstrates strong agreement with expert human raters while uncovering a significant degree of unreported intratumor heterogeneity. Integrating immunohistochemical staining with high-resolution grade analysis by GLASS-AI identified dysregulation of Mapk/Erk signaling in high-grade lung adenocarcinomas and locally advanced tumor regions. Our work demonstrates the benefit of employing GLASS-AI in preclinical lung adenocarcinoma models and the power of integrating machine learning and molecular biology techniques for studying the molecular pathways that underlie cancer progression.

A survey on recent trends in deep learning for nucleus segmentation from histopathology images

Nucleus segmentation is an imperative step in the qualitative study of imaging datasets, considered as an intricate task in histopathology image analysis. Segmenting a nucleus is an important part of diagnosing, staging, and grading cancer, but overlapping regions make it hard to separate and tell apart independent nuclei. Deep Learning is swiftly paving its way in the arena of nucleus segmentation, attracting quite a few researchers with its numerous published research articles indicating its efficacy in the field. This paper presents a systematic survey on nucleus segmentation using deep learning in the last five years (2017-2021), highlighting various segmentation models (U-Net, SCPP-Net, Sharp U-Net, and LiverNet) and exploring their similarities, strengths, datasets utilized, and unfolding research areas.

Explainable synthetic image generation to improve risk assessment of rare pediatric heart transplant rejection

Expert microscopic analysis of cells obtained from frequent heart biopsies is vital for early detection of pediatric heart transplant rejection to prevent heart failure. Detection of this rare condition is prone to low levels of expert agreement due to the difficulty of identifying subtle rejection signs within biopsy samples. The rarity of pediatric heart transplant rejection also means that very few gold-standard images are available for developing machine learning models. To solve this urgent clinical challenge, we developed a deep learning model to automatically quantify rejection risk within digital images of biopsied tissue using an explainable synthetic data augmentation approach. We developed this explainable AI framework to illustrate how our progressive and inspirational generative adversarial network models distinguish between normal tissue images and those containing cellular rejection signs. To quantify biopsy-level rejection risk, we first detect local rejection features using a binary image classifier trained with expert-annotated and synthetic examples. We converted these local predictions into a biopsy-wide rejection score via an interpretable histogram-based approach. Our model significantly improves upon prior works with the same dataset with an area under the receiver operating curve (AUROC) of 98.84% for the local rejection detection task and 95.56% for the biopsy-rejection prediction task. A biopsy-level sensitivity of 83.33% makes our approach suitable for early screening of biopsies to prioritize expert analysis. Our framework provides a solution to rare medical imaging challenges currently limited by small datasets.

Cell2Grid: an efficient, spatial, and convolutional neural network-ready representation of cell segmentation data

Purpose

Cell segmentation algorithms are commonly used to analyze large histologic images as they facilitate interpretation, but on the other hand they complicate hypothesis-free spatial analysis. Therefore, many applications train convolutional neural networks (CNNs) on high-resolution images that resolve individual cells instead, but their practical application is severely limited by computational resources. In this work, we propose and investigate an alternative spatial data representation based on cell segmentation data for direct training of CNNs.

Approach

We introduce and analyze the properties of Cell2Grid, an algorithm that generates compact images from cell segmentation data by placing individual cells into a low-resolution grid and resolves possible cell conflicts. For evaluation, we present a case study on colorectal cancer relapse prediction using fluorescent multiplex immunohistochemistry images.

Results

We could generate Cell2Grid images at 5 - μ m resolution that were 100 times smaller than the original ones. Cell features, such as phenotype counts and nearest-neighbor cell distances, remain similar to those of original cell segmentation tables ( p < 0.0001 ). These images could be directly fed to a CNN for predicting colon cancer relapse. Our experiments showed that test set error rate was reduced by 25% compared with CNNs trained on images rescaled to 5 μ m with bilinear interpolation. Compared with images at 1 - μ m resolution (bilinear rescaling), our method reduced CNN training time by 85%.

Conclusions

Cell2Grid is an efficient spatial data representation algorithm that enables the use of conventional CNNs on cell segmentation data. Its cell-based representation additionally opens a door for simplified model interpretation and synthetic image generation.

Platelet Endothelial Cell Adhesion Molecule (PECAM/CD31) Blockade Modulates Neutrophil Recruitment Patterns and Reduces Infarct Size in Experimental Ischemic Stroke

The infiltration of polymorphonuclear leukocytes (PMNs) in ischemia-reperfusion injury (I/RI) has been implicated as a critical component of inflammatory damage following ischemic stroke. However, successful blockade of PMN transendothelial migration (TEM) in preclinical studies has not translated to meaningful clinical outcomes. To investigate this further, leukocyte infiltration patterns were quantified, and these patterns were modulated by blocking platelet endothelial cell adhesion molecule-1 (PECAM), a key regulator of TEM. LysM-eGFP mice and microscopy were used to visualize all myeloid leukocyte recruitment following ischemia/reperfusion. Visual examination showed heterogeneous leukocyte distribution across the infarct at both 24 and 72 hours after I/RI. A semiautomated process was designed to precisely map PMN position across brain sections. Treatment with PECAM function-blocking antibodies did not significantly affect total leukocyte recruitment but did alter their distribution, with more observed at the cortex at both early and later time points (24 hours: 89% PECAM blocked vs. 72% control; 72 hours: 69% PECAM blocked vs. 51% control). This correlated with a decrease in infarct volume. These findings suggest that TEM, in the setting of I/RI in the cerebrovasculature, occurs primarily at the cortical surface. The reduction of stroke size with PECAM blockade suggests that infiltrating PMNs may exacerbate I/RI and indicate the potential therapeutic benefit of regulating the timing and pattern of leukocyte infiltration after stroke.

Pathological prognosis classification of patients with neuroblastoma using computational pathology analysis

Neuroblastoma is the most common extracranial solid tumor in early childhood. International Neuroblastoma Pathology Classification (INPC) is a commonly used classification system that provides clinicians with a reference for treatment stratification. However, given the complex and subjective assessment of the INPC, there will be inconsistencies in the analysis of the same patient by multiple pathologists. An automated, comprehensive and objective classification method is needed to identify different prognostic groups in patients with neuroblastoma. In this study, we collected 563 hematoxylin and eosin-stained histopathology whole-slide images from 107 patients with neuroblastoma who underwent surgical resection. We proposed a novel processing pipeline for nuclear segmentation, cell-level image feature extraction, and patient-level feature aggregation. Logistic regression model was built to classify patients with favorable histology (FH) and patients with unfavorable histology (UH). On the training/test dataset, patient-level of nucleus morphological/intensity features and age could correctly classify patients with a mean area under the receiver operating characteristic curve (AUC) of 0.946, a mean accuracy of 0.856, and a mean Matthews Correlation Coefficient (MCC) of 0.703,respectively. On the independent validation dataset, the classification model achieved a mean AUC of 0.938, a mean accuracy of 0.865 and a mean MCC of 0.630, showing good generalizability. Our results suggested that automatically derived image features could identify the differences in nuclear morphological and intensity between different prognostic groups, which could provide a reference to pathologists and facilitate the evaluation of the pathological prognosis in patients with neuroblastoma.

Tumor grading model employing geometric analysis of histopathological images with characteristic nuclei dictionary

Histopathological study has been shown to improve diagnosis of various disease classifications effectively as any disease condition is correlated to characteristic set of changes in the tissue structure. This study aims at developing an automated neural network system for grading brain tumors (Glioblastoma Multiforme) from histopathological images within the Whole Slide Images (WSI) of hematoxylin and eosin (H&E) stains with significant accuracy. Hematoxylin channels are extracted from the histopathological image patches using color de-convolution. Cell nuclei are precisely segmented using three level Otsu thresholding. From each segmented image, nuclei boundaries are extracted to extract nucleus level features based on their shape and size. Geometric features including ellipse eccentricities, nucleus perimeter, area, and polygon edge counts are extracted using geometric algorithms to define the nuclei boundaries of the segmented image. These features are collected for a large number of nuclei and the nuclei are clustered using the K-Means algorithm in order to create a dictionary. One of the major contributions involves the creation of dictionary of a fixed number of representative cell nuclei to speed up patch level classification. This optimal dictionary is used for clustering extracted cell nuclei and a fixed length histogram of counts on different types of nuclei is obtained. The proposed system has been tested with a total of 239600 TCGA patches of GBM and 206000 patches of LGG collected from GDC data portal and it showed good diagnosis performance with auto-classification accuracy of 97.2% compared to other state-of-art methods. Our results on segmentation and classification are encouraging, with better attainment with regard to precision and accuracy in contrast with previous models. The auto grading proposed system will act as a potential guide for pathologists to make more accurate decisions.

A disector-based framework for the automatic optical fractionator

Stereology-based methods provide the current state-of-the-art approaches for accurate quantification of numbers and other morphometric parameters of biological objects in stained tissue sections. The advent of artificial intelligence (AI)-based deep learning (DL) offers the possibility of improving throughput by automating the collection of stereology data. We have recently shown that DL can effectively achieve comparable accuracy to manual stereology but with higher repeatability, improved throughput, and less variation due to human factors by quantifying the total number of immunostained cells at their maximal profile of focus in extended depth of field (EDF) images. In the first of two novel contributions in this work, we propose a semi-automatic approach using a handcrafted Adaptive Segmentation Algorithm (ASA) to automatically generate ground truth on EDF images for training our deep learning (DL) models to automatically count cells using unbiased stereology methods. This update increases the amount of training data, thereby improving the accuracy and efficiency of automatic cell counting methods, without a requirement for extra expert time. The second contribution of this work is a Multi-channel Input and Multi-channel Output (MIMO) method using a U-Net deep learning architecture for automatic cell counting in a stack of z-axis images (also known as disector stacks). This DL-based digital automation of the ordinary optical fractionator ensures accurate counts through spatial separation of stained cells in the z-plane, thereby avoiding false negatives from overlapping cells in EDF images without the shortcomings of 3D and recurrent DL models. The contribution overcomes the issue of under-counting errors with EDF images due to overlapping cells in the z-plane (masking). We demonstrate the practical applications of these advances with automatic disector-based estimates of the total number of NeuN-immunostained neurons in a mouse neocortex. In summary, this work provides the first demonstration of automatic estimation of a total cell number in tissue sections using a combination of deep learning and the disector-based optical fractionator method.

Comparing machine learning and deep learning regression frameworks for accurate prediction of dielectrophoretic force

An intelligent sensing framework using Machine Learning (ML) and Deep Learning (DL) architectures to precisely quantify dielectrophoretic force invoked on microparticles in a textile electrode-based DEP sensing device is reported. The prediction accuracy and generalization ability of the framework was validated using experimental results. Images of pearl chain alignment at varying input voltages were used to build deep regression models using modified ML and CNN architectures that can correlate pearl chain alignment patterns of Saccharomyces cerevisiae(yeast) cells and polystyrene microbeads to DEP force. Various ML models such as K-Nearest Neighbor, Support Vector Machine, Random Forest, Neural Networks, and Linear Regression along with DL models such as Convolutional Neural Network (CNN) architectures of AlexNet, ResNet-50, MobileNetV2, and GoogLeNet have been analyzed in order to build an effective regression framework to estimate the force induced on yeast cells and microbeads. The efficiencies of the models were evaluated using Mean Absolute Error, Mean Absolute Relative, Mean Squared Error, R-squared, and Root Mean Square Error (RMSE) as evaluation metrics. ResNet-50 with RMSPROP gave the best performance, with a validation RMSE of 0.0918 on yeast cells while AlexNet with ADAM optimizer gave the best performance, with a validation RMSE of 0.1745 on microbeads. This provides a baseline for further studies in the application of deep learning in DEP aided Lab-on-Chip devices.

DenseRes-Unet: Segmentation of overlapped/clustered nuclei from multi organ histopathology images

Cancer is the second deadliest disease globally that can affect any human body organ. Early detection of cancer can increase the chances of survival in humans. Morphometric appearances of histopathology images make it difficult to segment nuclei effectively. We proposed a model to segment overlapped nuclei from H&E stained images. U-Net model achieved state-of-the-art performance in many medical image segmentation tasks; however, we modified the U-Net to learn a distinct set of consistent features. In this paper, we proposed the DenseRes-Unet model by integrating dense blocks in the last layers of the encoder block of U-Net, focused on relevant features from previous layers of the model. Moreover, we take advantage of residual connections with Atrous blocks instead of conventional skip connections, which helps to reduce the semantic gap between encoder and decoder paths. The distance map and binary threshold techniques intensify the nuclei interior and contour information in the images, respectively. The distance map is used to detect the center point of nuclei; moreover, it differentiates among nuclei interior boundary and core area. The distance map lacks a contour problem, which is resolved by using a binary threshold. Binary threshold helps to enhance the pixels around nuclei. Afterward, we fed images into the proposed DenseRes-Unet model, a deep, fully convolutional network to segment nuclei in the images. We have evaluated our model on four publicly available datasets for Nuclei segmentation to validate the model's performance. Our proposed model achieves 89.77% accuracy 90.36% F1-score, and 78.61% Aggregated Jaccard Index (AJI) on Multi organ Nucleus Segmentation (MoNuSeg).

Rheumatoid Arthritis Synovial Inflammation Quantification Using Computer Vision

OBJECTIVE

We quantified inflammatory burden in rheumatoid arthritis (RA) synovial tissue by using computer vision to automate the process of counting individual nuclei in hematoxylin and eosin images.

METHODS

We adapted and applied computer vision algorithms to quantify nuclei density (count of nuclei per unit area of tissue) on synovial tissue from arthroplasty samples. A pathologist validated algorithm results by labeling nuclei in synovial images that were mislabeled or missed by the algorithm. Nuclei density was compared with other measures of RA inflammation such as semiquantitative histology scores, gene-expression data, and clinical measures of disease activity.

RESULTS

The algorithm detected a median of 112,657 (range 8,160-821,717) nuclei per synovial sample. Based on pathologist-validated results, the sensitivity and specificity of the algorithm was 97% and 100%, respectively. The mean nuclei density calculated by the algorithm was significantly higher (P < 0.05) in synovium with increased histology scores for lymphocytic inflammation, plasma cells, and lining hyperplasia. Analysis of RNA sequencing identified 915 significantly differentially expressed genes in correlation with nuclei density (false discovery rate is less than 0.05). Mean nuclei density was significantly higher (P < 0.05) in patients with elevated levels of C-reactive protein, erythrocyte sedimentation rate, rheumatoid factor, and cyclized citrullinated protein antibody.

CONCLUSION

Nuclei density is a robust measurement of inflammatory burden in RA and correlates with multiple orthogonal measurements of inflammation.

Analysis of localized cAMP perturbations within a tissue reveal the effects of a local, dynamic gap junction state on ERK signaling

Beyond natural stimuli such as growth factors and stresses, the ability to experimentally modulate at will the levels or activity of specific intracellular signaling molecule(s) in specified cells within a tissue can be a powerful tool for uncovering new regulation and tissue behaviors. Here we perturb the levels of cAMP within specific cells of an epithelial monolayer to probe the time-dynamic behavior of cell-cell communication protocols implemented by the cAMP/PKA pathway and its coupling to the ERK pathway. The time-dependent ERK responses we observe in the perturbed cells for spatially uniform cAMP perturbations (all cells) can be very different from those due to spatially localized perturbations (a few cells). Through a combination of pharmacological and genetic perturbations, signal analysis, and computational modeling, we infer how intracellular regulation and regulated cell-cell coupling each impact the intracellular ERK response in single cells. Our approach reveals how a dynamic gap junction state helps sculpt the intracellular ERK response over time in locally perturbed cells.

An integrative approach of digital image analysis and transcriptome profiling to explore potential predictive biomarkers for TGFβ blockade therapy

Increasing evidence suggests that the presence and spatial localization and distribution pattern of tumor infiltrating lymphocytes (TILs) is associate with response to immunotherapies. Recent studies have identified TGFβ activity and signaling as a determinant of T cell exclusion in the tumor microenvironment and poor response to PD-1/PD-L1 blockade. Here we coupled the artificial intelligence (AI)-powered digital image analysis and gene expression profiling as an integrative approach to quantify distribution of TILs and characterize the associated TGFβ pathway activity. Analysis of T cell spatial distribution in the solid tumor biopsies revealed substantial differences in the distribution patterns. The digital image analysis approach achieves 74% concordance with the pathologist assessment for tumor-immune phenotypes. The transcriptomic profiling suggests that the TIL score was negatively correlated with TGFβ pathway activation, together with elevated TGFβ signaling activity observed in excluded and desert tumor phenotypes. The present results demonstrate that the automated digital pathology algorithm for quantitative analysis of CD8 immunohistochemistry image can successfully assign the tumor into one of three infiltration phenotypes: immune desert, immune excluded or immune inflamed. The association between “cold” tumor-immune phenotypes and TGFβ signature further demonstrates their potential as predictive biomarkers to identify appropriate patients that may benefit from TGFβ blockade.

Improved Quantification of Cell Density in the Arterial Wall-A Novel Nucleus Splitting Approach Applied to 3D Two-Photon Laser-Scanning Microscopy

Accurate information on vascular smooth muscle cell (VSMC) content, orientation, and distribution in blood vessels is indispensable to increase understanding of arterial remodeling and to improve modeling of vascular biomechanics. We have previously proposed an analysis method to automatically characterize VSMC orientation and transmural distribution in murine carotid arteries under well-controlled biomechanical conditions. However, coincident nuclei, erroneously detected as one large nucleus, were excluded from the analysis, hampering accurate VSMC content characterization and distorting transmural distributions. In the present study, therefore, we aim to (1) improve the previous method by adding a "nucleus splitting" procedure to split coinciding nuclei, (2) evaluate the accuracy of this novel method, and (3) test this method in a mouse model of VSMC apoptosis. After euthanasia, carotid arteries from SM22α-hDTR Apoe -/- and control Apoe -/- mice were bluntly dissected, excised, mounted in a biaxial biomechanical tester and brought to in vivo axial stretch and a pressure of 100 mmHg. Nuclei and elastin fibers were then stained using Syto-41 and Eosin-Y, respectively, and imaged using 3D two-photon laser scanning microscopy. Nuclei were segmented from images and coincident nuclei were split. The nucleus splitting procedure determines the likelihood that voxel pairs within coincident nuclei belong to the same nucleus and utilizes these likelihoods to identify individual nuclei using spectral clustering. Manual nucleus counts were used as a reference to assess the performance of our splitting procedure. Before and after splitting, automatic nucleus counts differed -26.6 ± 9.90% (p < 0.001) and -1.44 ± 7.05% (p = 0.467) from the manual reference, respectively. Whereas the slope of the relative difference between the manual and automated counts as a function of the manual count was significantly negative before splitting (p = 0.008), this slope became insignificant after splitting (p = 0.653). Smooth muscle apoptosis led to a 33.7% decrease in VSMC density (p = 0.008). Nucleus splitting improves the accuracy of automated cell content quantification in murine carotid arteries and overcomes the progressively worsening problem of coincident nuclei with increasing cell content in vessels. The presented image analysis framework provides a robust tool to quantify cell content, orientation, shape, and distribution in vessels to inform experimental and advanced computational studies on vascular structure and function.

Impaired mRNA splicing and proteostasis in preadipocytes in obesity-related metabolic disease

Preadipocytes are crucial for healthy adipose tissue expansion. Preadipocyte differentiation is altered in obese individuals, which has been proposed to contribute to obesity-associated metabolic disturbances. Here, we aimed at identifying the pathogenic processes underlying impaired adipocyte differentiation in obese individuals with insulin resistance (IR)/type 2 diabetes (T2D). We report that down-regulation of a key member of the major spliceosome, PRFP8/PRP8, as observed in IR/T2D preadipocytes from subcutaneous (SC) fat, prevented adipogenesis by altering both the expression and splicing patterns of adipogenic transcription factors and lipid droplet-related proteins, while adipocyte differentiation was restored upon recovery of PRFP8/PRP8 normal levels. Adipocyte differentiation was also compromised under conditions of endoplasmic reticulum (ER)-associated protein degradation (ERAD) hyperactivation, as occurs in SC and omental (OM) preadipocytes in IR/T2D obesity. Thus, targeting mRNA splicing and ER proteostasis in preadipocytes could improve adipose tissue function and thus contribute to metabolic health in obese individuals.

Evaluation of Deep Learning Architectures for Complex Immunofluorescence Nuclear Image Segmentation

Separating and labeling each nuclear instance (instance-aware segmentation) is the key challenge in nuclear image segmentation. Deep Convolutional Neural Networks have been demonstrated to solve nuclear image segmentation tasks across different imaging modalities, but a systematic comparison on complex immunofluorescence images has not been performed. Deep learning based segmentation requires annotated datasets for training, but annotated fluorescence nuclear image datasets are rare and of limited size and complexity. In this work, we evaluate and compare the segmentation effectiveness of multiple deep learning architectures (U-Net, U-Net ResNet, Cellpose, Mask R-CNN, KG instance segmentation) and two conventional algorithms (Iterative h-min based watershed, Attributed relational graphs) on complex fluorescence nuclear images of various types. We propose and evaluate a novel strategy to create artificial images to extend the training set. Results show that instance-aware segmentation architectures and Cellpose outperform the U-Net architectures and conventional methods on complex images in terms of F1 scores, while the U-Net architectures achieve overall higher mean Dice scores. Training with additional artificially generated images improves recall and F1 scores for complex images, thereby leading to top F1 scores for three out of five sample preparation types. Mask R-CNN trained on artificial images achieves the overall highest F1 score on complex images of similar conditions to the training set images while Cellpose achieves the overall highest F1 score on complex images of new imaging conditions. We provide quantitative results demonstrating that images annotated by under-graduates are sufficient for training instance-aware segmentation architectures to efficiently segment complex fluorescence nuclear images.

Deep learning provides high accuracy in automated chondrocyte viability assessment in articular cartilage using nonlinear optical microscopy

Chondrocyte viability is a crucial factor in evaluating cartilage health. Most cell viability assays rely on dyes and are not applicable for in vivo or longitudinal studies. We previously demonstrated that two-photon excited autofluorescence and second harmonic generation microscopy provided high-resolution images of cells and collagen structure; those images allowed us to distinguish live from dead chondrocytes by visual assessment or by the normalized autofluorescence ratio. However, both methods require human involvement and have low throughputs. Methods for automated cell-based image processing can improve throughput. Conventional image processing algorithms do not perform well on autofluorescence images acquired by nonlinear microscopes due to low image contrast. In this study, we compared conventional, machine learning, and deep learning methods in chondrocyte segmentation and classification. We demonstrated that deep learning significantly improved the outcome of the chondrocyte segmentation and classification. With appropriate training, the deep learning method can achieve 90% accuracy in chondrocyte viability measurement. The significance of this work is that automated imaging analysis is possible and should not become a major hurdle for the use of nonlinear optical imaging methods in biological or clinical studies.

Intratumoral injection of hydrogel-embedded nanoparticles enhances retention in glioblastoma

Intratumoral drug delivery is a promising approach for the treatment of glioblastoma multiforme (GBM). However, drug washout remains a major challenge in GBM therapy. Our strategy, aimed at reducing drug clearance and enhancing site-specific residence time, involves the local administration of a multi-component system comprised of nanoparticles (NPs) embedded within a thermosensitive hydrogel (HG). Herein, our objective was to examine the distribution of NPs and their cargo following intratumoral administration of this system in GBM. We hypothesized that the HG matrix, which undergoes rapid gelation upon increases in temperature, would contribute towards heightened site-specific retention and permanence of NPs in tumors. BODIPY-containing, infrared dye-labeled polymeric NPs embedded in a thermosensitive HG (HG-NPs) were fabricated and characterized. Retention and distribution dynamics were subsequently examined over time in orthotopic GBM-bearing mice. Results demonstrate that the HG-NPs system significantly improved site-specific, long-term retention of both NPs and BODIPY, with co-localization analyses showing that HG-NPs covered larger areas of the tumor and the peri-tumor region at later time points. Moreover, NPs released from the HG were shown to undergo uptake by surrounding GBM cells. Findings suggest that intratumoral delivery with HG-NPs has immense potential for GBM treatment, as well as other strategies where site-specific, long-term retention of therapeutic agents is warranted.

3-D Inorganic Crystal Structure Generation and Property Prediction via Representation Learning

Generative models have been successfully used to synthesize completely novel images, text, music, and speech. As such, they present an exciting opportunity for the design of new materials for functional applications. So far, generative deep-learning methods applied to molecular and drug discovery have yet to produce stable and novel 3-D crystal structures across multiple material classes. To that end, we, herein, present an autoencoder-based generative deep-representation learning pipeline for geometrically optimized 3-D crystal structures that simultaneously predicts the values of eight target properties. The system is highly general, as demonstrated through creation of novel materials from three separate material classes: binary alloys, ternary perovskites, and Heusler compounds. Comparison of these generated structures to those optimized via electronic-structure calculations shows that our generated materials are valid and geometrically optimized.

Generative Adversarial Networks in Digital Pathology: A Survey on Trends and Future Potential

Image analysis in the field of digital pathology has recently gained increased popularity. The use of high-quality whole-slide scanners enables the fast acquisition of large amounts of image data, showing extensive context and microscopic detail at the same time. Simultaneously, novel machine-learning algorithms have boosted the performance of image analysis approaches. In this paper, we focus on a particularly powerful class of architectures, the so-called generative adversarial networks (GANs) applied to histological image data. Besides improving performance, GANs also enable previously intractable application scenarios in this field. However, GANs could exhibit a potential for introducing bias. Hereby, we summarize the recent state-of-the-art developments in a generalizing notation, present the main applications of GANs, and give an outlook of some chosen promising approaches and their possible future applications. In addition, we identify currently unavailable methods with potential for future applications.

NuSeT: A deep learning tool for reliably separating and analyzing crowded cells

Segmenting cell nuclei within microscopy images is a ubiquitous task in biological research and clinical applications. Unfortunately, segmenting low-contrast overlapping objects that may be tightly packed is a major bottleneck in standard deep learning-based models. We report a Nuclear Segmentation Tool (NuSeT) based on deep learning that accurately segments nuclei across multiple types of fluorescence imaging data. Using a hybrid network consisting of U-Net and Region Proposal Networks (RPN), followed by a watershed step, we have achieved superior performance in detecting and delineating nuclear boundaries in 2D and 3D images of varying complexities. By using foreground normalization and additional training on synthetic images containing non-cellular artifacts, NuSeT improves nuclear detection and reduces false positives. NuSeT addresses common challenges in nuclear segmentation such as variability in nuclear signal and shape, limited training sample size, and sample preparation artifacts. Compared to other segmentation models, NuSeT consistently fares better in generating accurate segmentation masks and assigning boundaries for touching nuclei.