Deep learning provides high accuracy in automated chondrocyte viability assessment in articular cartilage using nonlinear optical microscopy

Mask R-CNN provides efficient and accurate measurement of chondrocyte viability in the label-free assessment of articular cartilage

Objective

Chondrocyte viability (CV) can be measured with the label-free method using second harmonic generation (SHG) and two-photon excitation autofluorescence (TPAF) imaging. To automate the image processing for the label-free CV measurement, we previously demonstrated a two-step deep-learning method: Step 1 used a U-Net to segment the lacuna area on SHG images; Step 2 used dual CNN networks to count live cells and the total number of cells in extracted cell clusters from TPAF images. This study aims to develop one-step deep learning methods to improve the efficiency of CV measurement.

Method

TPAF/SHG images were acquired simultaneously on cartilage samples from rats and pigs using two-photon microscopes and were merged to form RGB color images with red, green, and blue channels assigned to emission bands of oxidized flavoproteins, reduced forms of nicotinamide adenine dinucleotide, and SHG signals, respectively. Based on the Mask R-CNN, we designed a deep learning network and its denoising version using Wiener deconvolution for CV measurement.

Results

Using training and test datasets from rat and porcine cartilage, we have demonstrated that Mask R-CNN-based networks can segment and classify individual cells with a single-step processing flow. The absolute error (difference between the measured and the ground-truth CV) of the CV measurement using the Mask R-CNN with or without Wiener deconvolution denoising reaches 0.01 or 0.08, respectively; the error of the previous CV networks is 0.18, significantly larger than that of the Mask R-CNN methods.

Conclusions

Mask R-CNN-based deep-learning networks improve efficiency and accuracy of the label-free CV measurement.

Measuring chondrocyte viability of articular cartilage based on label-free two-photon microscopy and deep learning image analysis

Objective

Chondrocyte viability (CV) is an important indicator of articular cartilage health. Two-photon excitation autofluorescence (TPAF) and second harmonic generation (SHG) microscopy provide a label-free method for imaging chondrocytes. In this study, we propose an automated assessment of CV using deep learning cell segmentation and counting based on acquired TPAF/SHG images.

Design

Label-free TPAF/SHG images of cartilage samples from rats and porcine were acquired using both commercial and home-built two-photon microscopes, respectively. TPAF/SHG images were merged to form RGB color images with red, green, and blue channels assigned to TPAF (two channels) and SHG signals, respectively. To make the training datasets for the deep learning networks, individual chondrocyte areas on the RGB color images were manually circled and live or dead chondrocytes were validated by using Calcein-AM and Ethidium homodimer-1 dye labeling. We first built a chondrocyte viability network (MCV-Net) using the Mask R-CNN architecture, which could provide individual segmented cellular areas with live or dead status. Wiener deconvolution preprocessing was added before the input of MCV-Net to improve the accuracy of the CV analysis, forming the Wiener deconvolution CV network (wMCV-Net).

Results

Training (300 images) and test (120 images) datasets were built for rats and porcine cartilage respectively. Wiener deconvolution could improve the Peak Signal-to-Noise Ratio (PSNR) for 30-40%. We demonstrated that both MCV-Net and wMCV-Net significantly improved the accuracy of the CV measurement.

Conclusion

A custom desktop TPAF/SHG microscope was used in collaboration with deep learning algorithm wMCV-Net based label-free method to assess the CV and get 95% accuracy with both rats and porcine samples.

Longitudinal drug synergy assessment using convolutional neural network image-decoding of glioblastoma single-spheroid cultures

Background

In recent years, drug combinations have become increasingly popular to improve therapeutic outcomes in various diseases, including difficult to cure cancers such as the brain cancer glioblastoma. Assessing the interaction between drugs over time is critical for predicting drug combination effectiveness and minimizing the risk of therapy resistance. However, as viability readouts of drug combination experiments are commonly performed as an endpoint where cells are lysed, longitudinal drug-interaction monitoring is currently only possible through combined endpoint assays.

Methods

We provide a method for massive parallel monitoring of drug interactions for 16 drug combinations in 3 glioblastoma models over a time frame of 18 days. In our assay, viabilities of single neurospheres are to be estimated based on image information taken at different time points. Neurosphere images taken on the final day (day 18) were matched to the respective viability measured by CellTiter-Glo 3D on the same day. This allowed to use of machine learning to decode image information to viability values on day 18 as well as for the earlier time points (on days 8, 11, and 15).

Results

Our study shows that neurosphere images allow us to predict cell viability from extrapolated viabilities. This enables to assess of the drug interactions in a time window of 18 days. Our results show a clear and persistent synergistic interaction for several drug combinations over time.

Conclusions

Our method facilitates longitudinal drug-interaction assessment, providing new insights into the temporal-dynamic effects of drug combinations in 3D neurospheres which can help to identify more effective therapies against glioblastoma.

Deep learning-based high-speed, large-field, and high-resolution multiphoton imaging

Multiphoton microscopy is a formidable tool for the pathological analysis of tumors. The physical limitations of imaging systems and the low efficiencies inherent in nonlinear processes have prevented the simultaneous achievement of high imaging speed and high resolution. We demonstrate a self-alignment dual-attention-guided residual-in-residual generative adversarial network trained with various multiphoton images. The network enhances image contrast and spatial resolution, suppresses noise, and scanning fringe artifacts, and eliminates the mutual exclusion between field of view, image quality, and imaging speed. The network may be integrated into commercial microscopes for large-scale, high-resolution, and low photobleaching studies of tumor environments.

Machine learning in electron microscopy for advanced nanocharacterization: current developments, available tools and future outlook

In the last few years, electron microscopy has experienced a new methodological paradigm aimed to fix the bottlenecks and overcome the challenges of its analytical workflow. Machine learning and artificial intelligence are answering this call providing powerful resources towards automation, exploration, and development. In this review, we evaluate the state-of-the-art of machine learning applied to electron microscopy (and obliquely, to materials and nano-sciences). We start from the traditional imaging techniques to reach the newest higher-dimensionality ones, also covering the recent advances in spectroscopy and tomography. Additionally, the present review provides a practical guide for microscopists, and in general for material scientists, but not necessarily advanced machine learning practitioners, to straightforwardly apply the offered set of tools to their own research. To conclude, we explore the state-of-the-art of other disciplines with a broader experience in applying artificial intelligence methods to their research (e.g., high-energy physics, astronomy, Earth sciences, and even robotics, videogames, or marketing and finances), in order to narrow down the incoming future of electron microscopy, its challenges and outlook.

Pixel-level multimodal fusion deep networks for predicting subcellular organelle localization from label-free live-cell imaging

Complex intracellular organizations are commonly represented by dividing the metabolic process of cells into different organelles. Therefore, identifying sub-cellular organelle architecture is significant for understanding intracellular structural properties, specific functions, and biological processes in cells. However, the discrimination of these structures in the natural organizational environment and their functional consequences are not clear. In this article, we propose a new pixel-level multimodal fusion (PLMF) deep network which can be used to predict the location of cellular organelle using label-free cell optical microscopy images followed by deep-learning-based automated image denoising. It provides valuable insights that can be of tremendous help in improving the specificity of label-free cell optical microscopy by using the Transformer-Unet network to predict the ground truth imaging which corresponds to different sub-cellular organelle architectures. The new prediction method proposed in this article combines the advantages of a transformer's global prediction and CNN's local detail analytic ability of background features for label-free cell optical microscopy images, so as to improve the prediction accuracy. Our experimental results showed that the PLMF network can achieve over 0.91 Pearson's correlation coefficient (PCC) correlation between estimated and true fractions on lung cancer cell-imaging datasets. In addition, we applied the PLMF network method on the cell images for label-free prediction of several different subcellular components simultaneously, rather than using several fluorescent labels. These results open up a new way for the time-resolved study of subcellular components in different cells, especially for cancer cells.

Multiple Parallel Fusion Network for Predicting Protein Subcellular Localization from Stimulated Raman Scattering (SRS) Microscopy Images in Living Cells

Stimulated Raman Scattering Microscopy (SRS) is a powerful tool for label-free detailed recognition and investigation of the cellular and subcellular structures of living cells. Determining subcellular protein localization from the cell level of SRS images is one of the basic goals of cell biology, which can not only provide useful clues for their functions and biological processes but also help to determine the priority and select the appropriate target for drug development. However, the bottleneck in predicting subcellular protein locations of SRS cell imaging lies in modeling complicated relationships concealed beneath the original cell imaging data owing to the spectral overlap information from different protein molecules. In this work, a multiple parallel fusion network, MPFnetwork, is proposed to study the subcellular locations from SRS images. This model used a multiple parallel fusion model to construct feature representations and combined multiple nonlinear decomposing algorithms as the automated subcellular detection method. Our experimental results showed that the MPFnetwork could achieve over 0.93 dice correlation between estimated and true fractions on SRS lung cancer cell datasets. In addition, we applied the MPFnetwork method to cell images for label-free prediction of several different subcellular components simultaneously, rather than using several fluorescent labels. These results open up a new method for the time-resolved study of subcellular components in different cells, especially cancer cells.

Rapid histological imaging of bone without microtome sectioning using nonlinear microscopy

Tissue preparation for histologic evaluation of bone is particularly lengthy, limiting its use in intraoperative or intraprocedural histological evaluation. Nonlinear microscopy (NLM) is an optical sectioning microscopy method that can visualize pathology in freshly excised tissue without requiring physical microtome sectioning. This study describes a rapid protocol for NLM imaging of bone and associated cartilage. NLM imaging was performed on 71 specimens of normal bone as well as arthritic, malignant and inflammatory bone tissue from 40 patients who underwent joint replacement, amputation, bone marrow biopsy or autopsy. Specimens ranged in size from core needle biopsies to transections of entire femoral heads. Specimens were stained with acridine orange and sulforhodamine 101, nuclear and cytoplasmic/stromal fluorescent dyes, for 5 min, then rinsed for 30 s. NLM fluorescent images were displayed using colors analogous to hematoxylin and eosin (H&E) to facilitate interpretation. Pathologists examined NLM images of the specimens in real time by rapidly translating the specimen to areas of interest, similar to a standard transmission light microscope. By adjusting the NLM focus depth, images from a few-μm-thick layer could be obtained down to ~100 μm below the tissue surface, analogous to serial sectioning. Following real-time NLM imaging, the tissue was processed for conventional paraffin histology, and H&E slides were compared to recorded NLM images. Similarities and differences between NLM and paraffin H&E were assessed. NLM enabled visualization of normal bone architecture, including the lamellar matrix and osteocytes of trabecular bone, articular cartilage, as well as pathological bone features such osteoarthritis, osteomyelitis, and malignancy with an appearance resembling the paraffin H&E. Differences such as changes in cell border sharpness, cellular and nucleolar size, and color patterns were noted, suggesting that training is required for accurate evaluation of bone pathology with NLM. Irregular surface contours and debris generated by gross tissue preparation of bone can make some regions difficult to evaluate with NLM, but the ability to perform rapid three-dimensional translation and sub-surface imaging reduced these problems. NLM is a promising technique for rapid evaluation of bone pathology. Further studies assessing diagnostic performance are warranted.