Towards automated segmentation of cells and cell nuclei in nonlinear optical microscopy

Diagnosis of dengue virus infection using spectroscopic images and deep learning

Dengue virus (DENV) infection is one of the major health issues and a substantial epidemic infectious human disease. More than two billion humans are living in dengue susceptible regions with annual infection mortality rate is about 5%-20%. At initial stages, it is difficult to differentiate dengue virus symptoms with other similar diseases. The main objective of this research is to diagnose dengue virus infection in human blood sera for better treatment and rehabilitation process. A novel and robust approach is proposed based on Raman spectroscopy and deep learning. In this regard, the ResNet101 deep learning model is modified by exploiting transfer learning (TL) concept on Raman spectroscopic data of human blood sera. Sample size was selected using standard statistical tests. The proposed model is evaluated on 2,000 Raman spectra images in which 1,200 are DENV-infected of human blood sera samples, and 800 are healthy ones. It offers 96.0% accuracy on testing data for DENV infection diagnosis. Moreover, the developed approach demonstrated minimum improvement of 6.0% and 7.0% in terms of AUC and Kappa index respectively over the other state-of-the-art techniques. The developed model offers superior performance to capture minute Raman spectral variations due to the better residual learning capability and generalization ability compared to others deep learning models. The developed model revealed that it might be applied for diagnosis of DENV infection to save precious human lives.

Label-free multiphoton imaging allows brain tumor recognition based on texture analysis-a study of 382 tumor patients

Background

Label-free multiphoton microscopy has been suggested for intraoperative recognition and delineation of brain tumors. For any future clinical application, appropriate approaches for image acquisition and analysis have to be developed. Moreover, an evaluation of the reliability of the approach, taking into account inter- and intrapatient variability, is needed.

Methods

Coherent anti-Stokes Raman scattering (CARS), two-photon excited fluorescence (TPEF), and second-harmonic generation were acquired on cryosections of brain tumors of 382 patients and 28 human nontumor brain samples. Texture parameters of those images were calculated and used as input for linear discriminant analysis.

Results

The combined analysis of texture parameters of the CARS and TPEF signal proved to be most suited for the discrimination of nontumor brain versus brain tumors (low- and high-grade astrocytoma, oligodendroglioma, glioblastoma, recurrent glioblastoma, brain metastases of lung, colon, renal, and breast cancer and of malignant melanoma) leading to a correct rate of 96% (sensitivity: 96%, specificity: 100%). To approximate the clinical setting, the results were validated on 42 fresh, unfixed tumor biopsies. 82% of the tumors and, most important, all of the nontumor samples were correctly recognized. An image resolution of 1 µm was sufficient to distinguish brain tumors and nontumor brain. Moreover, the vast majority of single fields of view of each patient’s sample were correctly classified with high probabilities, which is important for clinical translation.

Conclusion

Label-free multiphoton imaging might allow fast and accurate intraoperative delineation of primary and secondary brain tumors in combination with endoscopic systems.

Identification of distinctive features in human intracranial tumors by label-free nonlinear multimodal microscopy

Nonlinear multimodal microscopy offers a series of label-free techniques with potential for intraoperative identification of tumor borders in situ using novel endoscopic devices. Here, we combined coherent anti-Stokes Raman scattering, two-photon excited fluorescence (TPEF) and second harmonic generation imaging to analyze biopsies of different human brain tumors, with the aim to understand whether the morphological information carried by single field of view images, similar to what delivered by present endoscopic systems, is sufficient for tumor recognition. We imaged 40 human biopsies of high and low grade glioma, meningioma, as well as brain metastases of melanoma, breast, lung and renal carcinoma, in comparison with normal brain parenchyma. Furthermore, five biopsies of schwannoma were analyzed and compared with nonpathological nerve tissue. Besides the high cellularity, the typical features of tumor, which were identified and quantified, are intracellular and extracellular lipid droplets, aberrant vessels, extracellular matrix collagen and diffuse TPEF. Each tumor type displayed a particular morphochemistry characterized by specific patterns of the above-mentioned features. Nonlinear multimodal microscopy performed on fresh unprocessed biopsies confirmed that the technique has the ability to visualize tumor structures and discern normal from neoplastic tissue likewise in conditions close to in situ.

Convolutional neural network initialized active contour model with adaptive ellipse fitting for nuclear segmentation on breast histopathological images

Automated detection and segmentation of nuclei from high-resolution histopathological images is a challenging problem owing to the size and complexity of digitized histopathologic images. In the context of breast cancer, the modified Bloom-Richardson Grading system is highly correlated with the morphological and topological nuclear features are highly correlated with Modified Bloom-Richardson grading. Therefore, to develop a computer-aided prognosis system, automated detection and segmentation of nuclei are critical prerequisite steps. We present a method for automated detection and segmentation of breast cancer nuclei named a convolutional neural network initialized active contour model with adaptive ellipse fitting (CoNNACaeF). The CoNNACaeF model is able to detect and segment nuclei simultaneously, which consist of three different modules: convolutional neural network (CNN) for accurate nuclei detection, (2) region-based active contour (RAC) model for subsequent nuclear segmentation based on the initial CNN-based detection of nuclear patches, and (3) adaptive ellipse fitting for overlapping solution of clumped nuclear regions. The performance of the CoNNACaeF model is evaluated on three different breast histological data sets, comprising a total of 257 H&E-stained images. The model is shown to have improved detection accuracy of F-measure 80.18%, 85.71%, and 80.36% and average area under precision-recall curves (AveP) 77%, 82%, and 74% on a total of 3 million nuclei from 204 whole slide images from three different datasets. Additionally, CoNNACaeF yielded an F-measure at 74.01% and 85.36%, respectively, for two different breast cancer datasets. The CoNNACaeF model also outperformed the three other state-of-the-art nuclear detection and segmentation approaches, which are blue ratio initialized local region active contour, iterative radial voting initialized local region active contour, and maximally stable extremal region initialized local region active contour models.

Quantitative imaging of lipid droplets in single cells

The combination of next generation sequencing (NGS) and automated liquid handling platforms has led to a revolution in single-cell genomic studies. However, many molecules that are critical to understanding the functional roles of cells in a complex tissue or organs, are not directly encoded in the genome, and therefore cannot be profiled with NGS. Lipids, for example, play a critical role in many metabolic processes but cannot be detected by sequencing. Recent developments in quantitative imaging, particularly coherent Raman scattering (CRS) techniques, have produced a suite of tools for studying lipid content in single cells. This article reviews CRS imaging and computational image processing techniques for non-destructive profiling of dynamic changes in lipid composition and spatial distribution at the single-cell level. As quantitative CRS imaging progresses synergistically with microfluidic and microscopic platforms for single-cell genomic analysis, we anticipate that these techniques will bring researchers closer towards combined lipidomic and genomic analysis.

Improved label-free diagnostics and pathological assessment of atherosclerotic plaques through nonlinear microscopy

Coronary heart disease is the most common type of heart disease caused by atherosclerosis. In fact, an arterial wall lesion centered on the accumulation of cholesterol-rich lipids and the accompanying inflammatory response generates a plaque, whose rupture may result in a thrombus with fatal consequences. Plaque characterization for assessing the severity of atherosclerosis is generally performed through standard histopathological examination based on hematoxylin/eosin staining, which is operator-dependent and requires relatively long procedures. In this framework, nonlinear optical microscopy is a valid, label-free alternative to standard diagnostic methods. We combined second-harmonic generation (SHG), two-photon excited fluorescence (TPEF) and fluorescence lifetime imaging microscopy in a multimodal scheme for obtaining morphological and molecular information on human carotid ex vivo specimens affected by atherosclerosis. In this study, discrimination between different tissues within the atherosclerotic plaque was achieved based on both lifetime, TPEF-to-SHG ratio, and image pattern analysis. The presented methodology aims to be a starting point for future fully automated and fast characterization of atherosclerotic biopsies; moreover, it could be extended to the study of other tissues and pathologies. Combined TPEF/SHG mapping of a carotid specimen affected by atherosclerosis.

A novel multiphoton microscopy images segmentation method based on superpixel and watershed

Multiphoton microscopy (MPM) imaging technique based on two-photon excited fluorescence (TPEF) and second harmonic generation (SHG) shows fantastic performance for biological imaging. The automatic segmentation of cellular architectural properties for biomedical diagnosis based on MPM images is still a challenging issue. A novel multiphoton microscopy images segmentation method based on superpixels and watershed (MSW) is presented here to provide good segmentation results for MPM images. The proposed method uses SLIC superpixels instead of pixels to analyze MPM images for the first time. The superpixels segmentation based on a new distance metric combined with spatial, CIE Lab color space and phase congruency features, divides the images into patches which keep the details of the cell boundaries. Then the superpixels are used to reconstruct new images by defining an average value of superpixels as image pixels intensity level. Finally, the marker-controlled watershed is utilized to segment the cell boundaries from the reconstructed images. Experimental results show that cellular boundaries can be extracted from MPM images by MSW with higher accuracy and robustness.

Raman spectroscopy for medical diagnostics--From in-vitro biofluid assays to in-vivo cancer detection

Raman spectroscopy is an optical technique based on inelastic scattering of light by vibrating molecules and can provide chemical fingerprints of cells, tissues or biofluids. The high chemical specificity, minimal or lack of sample preparation and the ability to use advanced optical technologies in the visible or near-infrared spectral range (lasers, microscopes, fibre-optics) have recently led to an increase in medical diagnostic applications of Raman spectroscopy. The key hypothesis underpinning this field is that molecular changes in cells, tissues or biofluids, that are either the cause or the effect of diseases, can be detected and quantified by Raman spectroscopy. Furthermore, multivariate calibration and classification models based on Raman spectra can be developed on large "training" datasets and used subsequently on samples from new patients to obtain quantitative and objective diagnosis. Historically, spontaneous Raman spectroscopy has been known as a low signal technique requiring relatively long acquisition times. Nevertheless, new strategies have been developed recently to overcome these issues: non-linear optical effects and metallic nanoparticles can be used to enhance the Raman signals, optimised fibre-optic Raman probes can be used for real-time in-vivo single-point measurements, while multimodal integration with other optical techniques can guide the Raman measurements to increase the acquisition speed and spatial accuracy of diagnosis. These recent efforts have advanced Raman spectroscopy to the point where the diagnostic accuracy and speed are compatible with clinical use. This paper reviews the main Raman spectroscopy techniques used in medical diagnostics and provides an overview of various applications.

Texture analysis and classification in coherent anti-Stokes Raman scattering (CARS) microscopy images for automated detection of skin cancer

Coherent anti-Stokes Raman scattering (CARS) microscopy is a powerful tool for fast label-free tissue imaging, which is promising for early medical diagnostics. To facilitate the diagnostic process, automatic image analysis algorithms, which are capable of extracting relevant features from the image content, are needed. In this contribution we perform an automated classification of healthy and tumor areas in CARS images of basal cell carcinoma (BCC) skin samples. The classification is based on extraction of texture features from image regions and subsequent classification of these regions into healthy and cancerous with a perceptron algorithm. The developed approach is capable of an accurate classification of texture types with high sensitivity and specificity, which is an important step towards an automated tumor detection procedure.

Coherent anti-Stokes Raman scattering and two photon excited fluorescence for neurosurgery

OBJECTIVE

There is no established method for in vivo imaging during biopsy and surgery of the brain, which is capable to generate competitive images in terms of resolution and contrast comparable with histopathological staining.

METHODS

Coherent anti-Stokes Raman scattering (CARS) and two photon excited fluorescence (TPEF) microscopy are non-invasive all optical imaging techniques that are capable of high resolution, label-free, real-time, nondestructive examination of living cells and tissues. They provide image contrast based on the molecular composition of the specimen which allows the study of large tissue areas of frozen tissue sections ex vivo.

RESULTS

Here, preliminary data on 55 lesions of the central nervous system are presented. The generated images very nicely demonstrate cytological and architectural features required for pathological tumor typing and grading. Furthermore, information on the molecular content of a probe is provided. The tool will be implemented into a biopsy needle or endoscope in the near future for in vivo studies.

CONCLUSION

With this promising multimodal imaging approach the neurosurgeon might directly see blood vessels to minimize the risk for biopsy associated hemorrhages. The attending neuropathologist might directly identify the tumor and guide the selection of representative specimens for further studies. Thus, collection of non-representative material could be avoided and the risk to injure eloquent brain tissue minimized.

Automated method for the segmentation and morphometry of nerve fibers in large-scale CARS images of spinal cord tissue

A fully automated method for large-scale segmentation of nerve fibers from coherent anti-Stokes Raman scattering (CARS) microscopy images is presented. The method is specifically designed for CARS images of transverse cross sections of nervous tissue but is also suitable for use with standard light microscopy images. After a detailed description of the two-part segmentation algorithm, its accuracy is quantified by comparing the resulting binary images to manually segmented images. We then demonstrate the ability of our method to retrieve morphological data from CARS images of nerve tissue. Finally, we present the segmentation of a large mosaic of CARS images covering more than half the area of a mouse spinal cord cross section and show evidence of clusters of neurons with similar g-ratios throughout the spinal cord.

Accumulating advantages, reducing limitations: multimodal nonlinear imaging in biomedical sciences - the synergy of multiple contrast mechanisms

Multimodal nonlinear microscopy has matured during the past decades to one of the key imaging modalities in life science and biomedicine due to its unique capabilities of label-free visualization of tissue structure and chemical composition, high depth penetration, intrinsic 3D sectioning, diffraction limited resolution and low phototoxicity. This review briefly summarizes first recent advances in the field regarding the methodology, e.g., contrast mechanisms and signal characteristics used for contrast generation as well as novel image processing approaches. The second part deals with technologic developments emphasizing improvements in penetration depth, imaging speed, spatial resolution and nonlinear labeling strategies. The third part focuses on recent applications in life science fundamental research and biomedical diagnostics as well as future clinical applications.

Molecular pathology via IR and Raman spectral imaging

During the last 15 years, vibrational spectroscopic methods have been developed that can be viewed as molecular pathology methods that depend on sampling the entire genome, proteome and metabolome of cells and tissues, rather than probing for the presence of selected markers. First, this review introduces the background and fundamentals of the spectroscopies underlying the new methodologies, namely infrared and Raman spectroscopy. Then, results are presented in the context of spectral histopathology of tissues for detection of metastases in lymph nodes, squamous cell carcinoma, adenocarcinomas, brain tumors and brain metastases. Results from spectral cytopathology of cells are discussed for screening of oral and cervical mucosa, and circulating tumor cells. It is concluded that infrared and Raman spectroscopy can complement histopathology and reveal information that is available in classical methods only by costly and time-consuming steps such as immunohistochemistry, polymerase chain reaction or gene arrays. Due to the inherent sensitivity toward changes in the bio-molecular composition of different cell and tissue types, vibrational spectroscopy can even provide information that is in some cases superior to that of any one of the conventional techniques.

Automated seeding-based nuclei segmentation in nonlinear optical microscopy

Nonlinear optical (NLO) microscopy based, e.g., on coherent anti-Stokes Raman scattering (CARS) or two-photon-excited fluorescence (TPEF) is a fast label-free imaging technique, with a great potential for biomedical applications. However, NLO microscopy as a diagnostic tool is still in its infancy; there is a lack of robust and durable nuclei segmentation methods capable of accurate image processing in cases of variable image contrast, nuclear density, and type of investigated tissue. Nonetheless, such algorithms specifically adapted to NLO microscopy present one prerequisite for the technology to be routinely used, e.g., in pathology or intraoperatively for surgical guidance. In this paper, we compare the applicability of different seeding and boundary detection methods to NLO microscopic images in order to develop an optimal seeding-based approach capable of accurate segmentation of both TPEF and CARS images. Among different methods, the Laplacian of Gaussian filter showed the best accuracy for the seeding of the image, while a modified seeded watershed segmentation was the most accurate in the task of boundary detection. The resulting combination of these methods followed by the verification of the detected nuclei performs high average sensitivity and specificity when applied to various types of NLO microscopy images.

Label-free evaluation of hepatic microvesicular steatosis with multimodal coherent anti-Stokes Raman scattering microscopy

Hepatic microvesicular steatosis is a hallmark of drug-induced hepatotoxicity and early-stage fatty liver disease. Current histopathology techniques are inadequate for the clinical evaluation of hepatic microvesicular steatosis. In this paper, we explore the use of multimodal coherent anti-Stokes Raman scattering (CARS) microscopy for the detection and characterization of hepatic microvesicular steatosis. We show that CARS microscopy is more sensitive than Oil Red O histology for the detection of microvesicular steatosis. Computer-assisted analysis of liver lipid level based on CARS signal intensity is consistent with triglyceride measurement using a standard biochemical assay. Most importantly, in a single measurement procedure on unprocessed and unstained liver tissues, multimodal CARS imaging provides a wealth of critical information including the detection of microvesicular steatosis and quantitation of liver lipid content, number and size of lipid droplets, and lipid unsaturation and packing order of lipid droplets. Such information can only be assessed by multiple different methods on processed and stained liver tissues or tissue extracts using current standard analytical techniques. Multimodal CARS microscopy also permits label-free identification of lipid-rich non-parenchymal cells. In addition, label-free and non-perturbative CARS imaging allow rapid screening of mitochondrial toxins-induced microvesicular steatosis in primary hepatocyte cultures. With its sensitivity and versatility, multimodal CARS microscopy should be a powerful tool for the clinical evaluation of hepatic microvesicular steatosis.