Mask R-CNN provides efficient and accurate measurement of chondrocyte viability in the label-free assessment of articular cartilage

Objective

Chondrocyte viability (CV) can be measured with the label-free method using second harmonic generation (SHG) and two-photon excitation autofluorescence (TPAF) imaging. To automate the image processing for the label-free CV measurement, we previously demonstrated a two-step deep-learning method: Step 1 used a U-Net to segment the lacuna area on SHG images; Step 2 used dual CNN networks to count live cells and the total number of cells in extracted cell clusters from TPAF images. This study aims to develop one-step deep learning methods to improve the efficiency of CV measurement.

Method

TPAF/SHG images were acquired simultaneously on cartilage samples from rats and pigs using two-photon microscopes and were merged to form RGB color images with red, green, and blue channels assigned to emission bands of oxidized flavoproteins, reduced forms of nicotinamide adenine dinucleotide, and SHG signals, respectively. Based on the Mask R-CNN, we designed a deep learning network and its denoising version using Wiener deconvolution for CV measurement.

Results

Using training and test datasets from rat and porcine cartilage, we have demonstrated that Mask R-CNN-based networks can segment and classify individual cells with a single-step processing flow. The absolute error (difference between the measured and the ground-truth CV) of the CV measurement using the Mask R-CNN with or without Wiener deconvolution denoising reaches 0.01 or 0.08, respectively; the error of the previous CV networks is 0.18, significantly larger than that of the Mask R-CNN methods.

Conclusions

Mask R-CNN-based deep-learning networks improve efficiency and accuracy of the label-free CV measurement.

Correlation of systemic involvement and presence of pathological skin calcification assessed by ex vivo nonlinear microscopy in Pseudoxanthoma elasticum

Pseudoxanthoma elasticum (PXE (OMIM 264800)) is an autosomal recessive connective tissue disorder mainly caused by mutations in the ABCC6 gene. PXE results in ectopic calcification primarily in the skin, eye and blood vessels that can lead to blindness, peripheral arterial disease and stroke. Previous studies found correlation between macroscopic skin involvement and severe ophthalmological and cardiovascular complications. This study aimed to investigate correlation between skin calcification and systemic involvement in PXE. Ex vivo nonlinear microscopy (NLM) imaging was performed on formalin fixed, deparaffinized, unstained skin sections to assess the extent of skin calcification. The area affected by calcification (CA) in the dermis and density of calcification (CD) was calculated. From CA and CD, calcification score (CS) was determined. The number of affected typical and nontypical skin sites were counted. Phenodex + scores were determined. The relationship between the ophthalmological, cerebro- and cardiovascular and other systemic complications and CA, CD and CS, respectively, and skin involvement were analyzed. Regression models were built for adjustment to age and sex. We found significant correlation of CA with the number of affected typical skin sites (r = 0.48), the Phenodex + score (r = 0.435), extent of vessel involvement (V-score) (r = 0.434) and disease duration (r = 0.48). CD correlated significantly with V-score (r = 0.539). CA was significantly higher in patients with more severe eye (p = 0.04) and vascular (p = 0.005) complications. We found significantly higher CD in patients with higher V-score (p = 0.018), and with internal carotid artery hypoplasia (p = 0.045). Significant correlation was found between higher CA and the presence of macula atrophy (β = - 0.44, p = 0.032) and acneiform skin changes (β = 0.40, p = 0.047). Based on our results, the assessment of skin calcification pattern with nonlinear microscopy in PXE may be useful for clinicians to identify PXE patients who develop severe systemic complications.

Nonlinear optical imaging by detection with optical parametric amplification (invited paper)

Nonlinear optical imaging is a versatile tool that has been proven to be exceptionally useful in various research fields. However, due to the use of photomultiplier tubes (PMTs), the wide application of nonlinear optical imaging is limited by the incapability of imaging under ambient light. In this paper, we propose and demonstrate a new optical imaging detection method based on optical parametric amplification (OPA). As a nonlinear optical process, OPA intrinsically rejects ambient light photons by coherence gating. Periodical poled lithium niobate (PPLN) crystals are used in this study as the media for OPA. Compared to bulk nonlinear optical crystals, PPLN crystals support the generation of OPA signal with lower pump power. Therefore, this characteristic of PPLN crystals is particularly beneficial when using high-repetition-rate lasers, which facilitate high-speed optical signal detection, such as in spectroscopy and imaging. A PPLN-based OPA system was built to amplify the emitted imaging signal from second harmonic generation (SHG) and coherent anti-Stokes Raman scattering (CARS) microscopy imaging, and the amplified optical signal was strong enough to be detected by a biased photodiode under ordinary room light conditions. With OPA detection, ambient-light-on SHG and CARS imaging becomes possible, and achieves a similar result as PMT detection under strictly dark environments. These results demonstrate that OPA can be used as a substitute for PMTs in nonlinear optical imaging to adapt it to various applications with complex lighting conditions.

Raman Imaging as a powerful tool to elucidate chemical processes in a matrix: Medicated chewing gums with nicotine

Extruded medicated chewing gum is a convenient but complex drug delivery system. Description of gum ingredient distribution and interactions in literature is sparse, but fundamental in product characterization and stability prediction. Although Raman spectroscopy has been used for such characterization of numerous dosage forms, its applicability to medicated chewing gum has not been studied until now. The objective was to investigate the applicability of confocal Raman imaging on chewing gum for identification and distribution of excipients and the model drug nicotine, including changes occurring during shelf life. A sample preparation protocol was composed to present an even surface of a gum cross section without altering the gum matrix texture. High-resolution Raman maps were obtained by Non Negative Least Squares (NNLS) analysis for a reference gum and gums stored for 6 months at mild (25 °C/60% RH) and accelerated (40 °C/75% RH) conditions. Additional Empty Modelling™ analysis confirmed the results of NNLS. The NNLS analysis located nicotine and the following excipients: gum base, calcium carbonate, sorbitol, xylitol, sodium carbonate, sodium bicarbonate and talc in distinct domains of the reference sample. Changes of the sample stored at accelerated conditions was discovered as sodium carbonate was not observed in this sample. Additionally, stereo light microscopy showed changes in product appearance and high-performance liquid chromatography confirmed formation of the oxidation product nicotine-1'-N-oxide in this sample. The gum formulation and its ingredients displayed characteristic Raman spectra, proving Raman imaging as a useful method for characterizing medicated chewing gums, including changes occurring during stability testing.

Multiphoton Excitation Fluorescence Microscopy and Spectroscopic Multianalytical Approach for Characterization of Historical Glass Grisailles

The preservation of the integrity of artworks and cultural heritage items during characterization and conservation operations is of high priority, therefore, the application of non-invasive techniques is commonly suggested and recommended. Nonlinear optical microscopies (NLOM), based on the use of tightly focused pulsed femtosecond lasers, are emerging techniques for structural and chemical analysis of heritage objects with micrometric lateral and axial resolution. The results obtained with a set of optical and spectroscopic techniques for the chemical and physical characterization of grisaille paint layers on historical stained glasses, from different chronologies and provenance in Spain, are presented in this work. Optical behaviour and chemical composition were investigated by NLOM, using a laboratory set-up in the modality of Multi-Photon Excitation Fluorescence (MPEF), and by a multi-analytical combination of Field Emission Scanning Electron Microscopy-Energy Dispersive X-ray Spectrometry (FESEM-EDS), Laser Induced Breakdown Spectrosocopy (LIBS) and Laser Induced Fluorescence (LIF). Thicknesses values of the historical grisaille paint layers measured with MPEF were compared with those retrieved through FESEM, showing significant consistency and agreement. Under proper conditions, analysis via MPEF microscopy avoids the photochemical and physical damage to the examined materials, thus ensuring their preservation. This approach paves the way for future in-situ, non-invasive stratigraphic investigations on cultural heritage objects.

Nonlinear optical microscopy is a novel tool for the analysis of cutaneous alterations in pseudoxanthoma elasticum

Pseudoxanthoma elasticum (PXE, OMIM 264800) is a rare autosomal recessive disorder with ectopic mineralization and fragmentation of elastin fibers. It is caused by mutations of the ABCC6 gene that leads to decreased serum levels of inorganic pyrophosphate (PPi) anti-mineralization factor. The occurrence of severe complications among PXE patients highlights the importance of early diagnosis so that prompt multidisciplinary care can be provided to patients. We aimed to examine dermal connective tissue with nonlinear optical (NLO) techniques, as collagen emits second-harmonic generation (SHG) signal, while elastin can be excited by two-photon excitation fluorescence (TPF). We performed molecular genetic analysis, ophthalmological and cardiovascular assessment, plasma PPi measurement, conventional histopathological examination, and ex vivo SHG and TPF imaging in five patients with PXE and five age- and gender-matched healthy controls. Pathological mutations including one new variant were found in the ABCC6 gene in all PXE patients and their plasma PPi level was significantly lower compared with controls. Degradation and mineralization of elastin fibers and extensive calcium deposition in the mid-dermis was visualized and quantified together with the alterations of the collagen structure in PXE. Our data suggests that NLO provides high-resolution imaging of the specific histopathological features of PXE-affected skin. In vivo NLO may be a promising tool in the assessment of PXE, promoting early diagnosis and follow-up.

Nonlinear optical characterization of porous carbon materials by CARS, SHG and TPEF

The Coherent anti-Stokes Raman scattering (CARS) microscopy is employed to characterize porous carbon materials. The slices of two dimensional (2D) CARS images are obtained, using Z-scan technology. Furthermore, the three dimensional (3D) image of CARS is reconstructed with the Z-scanned 2D CARS images. The visualizations of rotation for 3D CARS are also shown. The 2D and 3D CARS images with rotations can clearly characterize the structures and properties of porous carbon materials, including the surface and inner parts. Nonlinear optical second harmonic generation demonstrates the inversion symmetry breaking of porous carbon materials, which is not centrosymmetric material. The two-photon excitation fluorescence image reveals that porous carbon material is of weak auto-fluorescence. The clearly observed signal of two photon-excitation fluorescence provides direct evidence that photon momentum transfer to the electron system occurs. Experimental results demonstrate that the nonlinear optical microscopy is a great optical analysis method for the study of porous carbon materials, and can be potentially used for the optical characterization on the materials at micro-nano-scale.

Multimodal nonlinear microscope based on a compact fiber-format laser source

We present a multimodal non-linear optical (NLO) laser-scanning microscope, based on a compact fiber-format excitation laser and integrating coherent anti-Stokes Raman scattering (CARS), stimulated Raman scattering (SRS) and two-photon-excitation fluorescence (TPEF) on a single platform. We demonstrate its capabilities in simultaneously acquiring CARS and SRS images of a blend of 6-μm poly(methyl methacrylate) beads and 3-μm polystyrene beads. We then apply it to visualize cell walls and chloroplast of an unprocessed fresh leaf of Elodea aquatic plant via SRS and TPEF modalities, respectively. The presented NLO microscope, developed in house using off-the-shelf components, offers full accessibility to the optical path and ensures its easy re-configurability and flexibility.

Redefining 3Dimensional placental membrane microarchitecture using multiphoton microscopy and optical clearing

INTRODUCTION

Remodeling of human placental membranes (amniochorionic or fetalmembrane) throughout gestation, a necessity to accommodate increasing uterine volume, involves continuous alterations (replacement of cells and remodeling of extracellular matrix). Methodologic limitations have obscured microscopic determination of cellular and layer-level alterations. This study used a combination of advanced imaging by multiphoton autofluorescence microscopy (MPAM) and second harmonic generation (SHG) microscopy along with tissue optical clearing to characterize the 3Dimensional multilayer organization of placental membranes.

METHODS

Placental membranes biopsies (6 mm) collected from term, not-in-labor cesarean deliveries (n = 7) were fixed in 10% formalin (native) or treated with 2,2'-thiodiethanol to render them transparent for deeper imaging. Native and cleared tissues were imaged using MPAM (cellular autofluorescence) and SHG (fibrillar collagen). Depth z-stacks captured the amnion epithelium, underlying matrix layers, and in the cleared biopsies, the decidua layer.

RESULTS

MPAM and SHG revealed fetal membrane epithelial topography and collagen organization in multiple matrix layers. Term amnion layers showed epithelial shedding and gaps. Optical clearing provided full-depth imaging with improved visualization of collagen structure, mesenchymal cells in extracellular matrix layers, and decidua morphology. Layer thicknesses measured by imaging corroborated with histology. Mosaic tiling of MPAM/SHG image stacks allowed large area visualization of entire biopsies.

CONCLUSION

MPAM-SHG microscopy allowed for study of this multi-layered tissue and revealed shedding, gap formation, and other structural changes. This approach could be used to study structural changes associated with membranes as well as other uterine tissues to better understand events in normal and abnormal parturition.

Label-free identification of the microstructure of rat spinal cords based on nonlinear optical microscopy

The spinal cord is a vital link between the brain and the body and mainly comprises neurons, glial cells and nerve fibres. In this work, nonlinear optical (NLO) microscopy based on intrinsic tissue properties was employed to label-freely analyze the cells and matrix in spinal cords at a molecular level. The high-resolution and high-contrast NLO images of unstained spinal cords demonstrate that NLO microscopy has the ability to show the microstructure of white and grey matter including ventral horn, intermediate area, dorsal horns, ventral column, lateral column and dorsal column. Neurons with various sizes were identified in grey matter by dark spots of nonfluorescent nuclei encircled by cytoplasm-emitting two-photon excited fluorescence signals. Nerve fibres and neuroglias were observed in white matter. Besides, the spinal arteries were clearly presented by NLO microscopy. Using spectral and morphological information, this technique was proved to be an effective tool for label-freely imaging spinal cord tissues, based on endogenous signals in biological tissue. With future development, we foresee promising applications of the NLO technique for in vivo, real-time assessment of spinal cord diseases or injures.

Nonlinear optical microscopy for label-free detection of gastrointestinal neuroendocrine tumors

Neuroendocrine tumors (NETs), which are rare and slow-growing neoplasms, pose a diagnostic challenge as they are clinically silent at the time of presentation. Here, gastrointestinal neuroendocrine tumors were researched by nonlinear microscopy, and results demonstrate that this technique has the capability to identify neuroendocrine tumors in the absence of labels and can, in particular, detect rare neuroendocrine tumor cells, vascular invasion, desmoplastic reaction, and fibroelastosis induced by neuroendocrine tumors. These conclusions highlight the possibility of nonlinear optical microscopy as a diagnostic tool for label-freely differentiating neuroendocrine tumors by these histopathologic features.

Nonlinear optical microscopy for immunoimaging: a custom optimized system of high-speed, large-area, multicolor imaging

BACKGROUND

The nonlinear optical microscopy has become the current state-of-the-art for intravital imaging. Due to its advantages of high resolution, superior tissue penetration, lower photodamage and photobleaching, as well as intrinsic z-sectioning ability, this technology has been widely applied in immunoimaging for a decade. However, in terms of monitoring immune events in native physiological environment, the conventional nonlinear optical microscope system has to be optimized for live animal imaging. Generally speaking, three crucial capabilities are desired, including high-speed, large-area and multicolor imaging. Among numerous high-speed scanning mechanisms used in nonlinear optical imaging, polygon scanning is not only linearly but also dispersion-freely with high stability and tunable rotation speed, which can overcome disadvantages of multifocal scanning, resonant scanner and acousto-optical deflector (AOD). However, low frame rate, lacking large-area or multicolor imaging ability make current polygonbased nonlinear optical microscopes unable to meet the requirements of immune event monitoring.

METHODS

We built up a polygon-based nonlinear optical microscope system which was custom optimized for immunoimaging with high-speed, large-are and multicolor imaging abilities.

RESULTS

Firstly, we validated the imaging performance of the system by standard methods. Then, to demonstrate the ability to monitor immune events, migration of immunocytes observed by the system based on typical immunological models such as lymph node, footpad and dorsal skinfold chamber are shown. Finally, we take an outlook for the possible advance of related technologies such as sample stabilization and optical clearing for more stable and deeper intravital immunoimaging.

CONCLUSIONS

This study will be helpful for optimizing nonlinear optical microscope to obtain more comprehensive and accurate information of immune events.

Nonlinear optical microscopy in decoding arterial diseases

Pathological understanding of arterial diseases is mainly attributable to histological observations based on conventional tissue staining protocols. The emerging development of nonlinear optical microscopy (NLOM), particularly in second-harmonic generation, two-photon excited fluorescence and coherent Raman scattering, provides a new venue to visualize pathological changes in the extracellular matrix caused by atherosclerosis progression. These techniques in general require minimal tissue preparation and offer rapid three-dimensional imaging. The capability of label-free microscopic imaging enables disease impact to be studied directly on the bulk artery tissue, thus minimally perturbing the sample. In this review, we look at recent progress in applications related to arterial disease imaging using various forms of NLOM.