Dynamic Sparse Sampling for Confocal Raman Microscopy

On-the-fly Raman microscopy guaranteeing the accuracy of discrimination

Accelerating the measurement for discrimination of samples, such as classification of cell phenotype, is crucial when faced with significant time and cost constraints. Spontaneous Raman microscopy offers label-free, rich chemical information but suffers from long acquisition time due to extremely small scattering cross-sections. One possible approach to accelerate the measurement is by measuring necessary parts with a suitable number of illumination points. However, how to design these points during measurement remains a challenge. To address this, we developed an imaging technique based on a reinforcement learning in machine learning (ML). This ML approach adaptively feeds back "optimal" illumination pattern during the measurement to detect the existence of specific characteristics of interest, allowing faster measurements while guaranteeing discrimination accuracy. Using a set of Raman images of human follicular thyroid and follicular thyroid carcinoma cells, we showed that our technique requires 3,333 to 31,683 times smaller number of illuminations for discriminating the phenotypes than raster scanning. To quantitatively evaluate the number of illuminations depending on the requisite discrimination accuracy, we prepared a set of polymer bead mixture samples to model anomalous and normal tissues. We then applied a home-built programmable-illumination microscope equipped with our algorithm, and confirmed that the system can discriminate the sample conditions with 104 to 4,350 times smaller number of illuminations compared to standard point illumination Raman microscopy. The proposed algorithm can be applied to other types of microscopy that can control measurement condition on the fly, offering an approach for the acceleration of accurate measurements in various applications including medical diagnosis.

Fast Raman imaging through the combination of context-aware matrix completion and low spectral resolution

Raman hyperspectral imaging is an effective method for label-free imaging with chemical specificity, yet the weak signals and correspondingly long integration times have hindered its wide adoption as a routine analytical method. Recently, low resolution Raman imaging has been proposed to improve the spectral signal-to-noise ratio, which significantly improves the speed of Raman imaging. In this paper, low resolution Raman spectroscopy is combined with "context-aware" matrix completion, where regions of the sample that are not of interest are skipped, and the regions that are measured are under-sampled, then reconstructed with a low-rank constraint. Both simulations and experiment show that low-resolution Raman boosts the speed and image quality of the computationally-reconstructed Raman images, allowing deeper sub-sampling, reduced exposure time, and an overall >10-fold improvement in imaging speed, without sacrificing chemical specificity or spatial image quality. As the method utilizes traditional point-scan imaging, it retains full confocality and is "backwards-compatible" with pre-existing traditional Raman instruments, broadening the potential scope of Raman imaging applications.

Rapid Intracellular Detection and Analysis of Lipid Droplets' Morpho-Chemical Composition by Phase-Guided Raman Sampling

Intracellular lipid droplets (LDs) are dynamic, complex organelles involved in nearly all aspects of cellular metabolism. In situ characterization methods are primarily limited to fluorescence imaging, which yields limited chemical information, or Raman spectroscopy, which provides excellent chemical profiling but very low throughput. Here, we propose a new paradigm where locations of both large and small droplets are obtained automatically from high-resolution phase images and fed into a galvomirror-controlled Raman sampling arm to obtain the full spectrum of each LD efficiently. Using this phase-guided Raman sampling, we can characterize hundreds of LDs within a single cell in minutes and easily acquire more than 40,000 high-quality spectra. The data set revealed strong, cell line-dependent, cell-dependent, and individual droplet-dependent composition changes to various culture conditions. In particular, we revealed a strong competitive relationship between mono- and polyunsaturated fatty acids, where supplementation with one led to a relative decrease in the other.

Correction for Extrinsic Background in Raman Hyperspectral Images

Raman hyperspectral microscopy is a valuable tool in biological and biomedical imaging. Because Raman scattering is often weak in comparison to other phenomena, prevalent spectral fluctuations and contaminations have brought advancements in analytical and chemometric methods for Raman spectra. These chemometric advances have been key contributors to the applicability of Raman imaging to biological systems. As studies increase in scale, spectral contamination from extrinsic background, intensity from sources such as the optical components that are extrinsic to the sample of interest, has become an emerging issue. Although existing baseline correction schemes often reduce intrinsic background such as autofluorescence originating from the sample of interest, extrinsic background is not explicitly considered, and these methods often fail to reduce its effects. Here, we show that extrinsic background can significantly affect a classification model using Raman images, yielding misleadingly high accuracies in the distinction of benign and malignant samples of follicular thyroid cell lines. To mitigate its effects, we develop extrinsic background correction (EBC) and demonstrate its use in combination with existing methods on Raman hyperspectral images. EBC isolates regions containing the smallest amounts of sample materials that retain extrinsic contributions that are specific to the device or environment. We perform classification both with and without the use of EBC, and we find that EBC retains biological characteristics in the spectra while significantly reducing extrinsic background. As the methodology used in EBC is not specific to Raman spectra, correction of extrinsic effects in other types of hyperspectral and grayscale images is also possible.

Spectral classification by generative adversarial linear discriminant analysis

Generative adversarial linear discriminant analysis (GALDA) is formulated as a broadly applicable tool for increasing classification accuracy and reducing overfitting in spectrochemical analysis. Although inspired by the successes of generative adversarial neural networks (GANs) for minimizing overfitting artifacts in artificial neural networks, GALDA was built around an independent linear algebra framework distinct from those in GANs. In contrast to feature extraction and data reduction approaches for minimizing overfitting, GALDA performs data augmentation by identifying and adversarially excluding the regions in spectral space in which genuine data do not reside. Relative to non-adversarial analogs, loading plots for dimension reduction showed significant smoothing and more prominent features aligned with spectral peaks following generative adversarial optimization. Classification accuracy was evaluated for GALDA together with other commonly available supervised and unsupervised methods for dimension reduction in simulated spectra generated using an open-source Raman database (Romanian Database of Raman Spectroscopy, RDRS). Spectral analysis was then performed for microscopy measurements of microsphereroids of the blood thinner clopidogrel bisulfate and in THz Raman imaging of common constituents in aspirin tablets. From these collective results, the potential scope of use for GALDA is critically evaluated relative to alternative established spectral dimension reduction and classification methods.

Deep Learning Approach for Dynamic Sampling for Multichannel Mass Spectrometry Imaging

Mass Spectrometry Imaging (MSI), using traditional rectilinear scanning, takes hours to days for high spatial resolution acquisitions. Given that most pixels within a sample's field of view are often neither relevant to underlying biological structures nor chemically informative, MSI presents as a prime candidate for integration with sparse and dynamic sampling algorithms. During a scan, stochastic models determine which locations probabilistically contain information critical to the generation of low-error reconstructions. Decreasing the number of required physical measurements thereby minimizes overall acquisition times. A Deep Learning Approach for Dynamic Sampling (DLADS), utilizing a Convolutional Neural Network (CNN) and encapsulating molecular mass intensity distributions within a third dimension, demonstrates a simulated 70% throughput improvement for Nanospray Desorption Electrospray Ionization (nano-DESI) MSI tissues. Evaluations are conducted between DLADS, a Supervised Learning Approach for Dynamic Sampling, with Least-Squares regression (SLADS-LS), and a Multi-Layer Perceptron (MLP) network (SLADS-Net). When compared with SLADS-LS, limited to a single m / z channel, as well as multichannel SLADS-LS and SLADS-Net, DLADS respectively improves regression performance by 36.7%, 7.0%, and 6.2%, resulting in gains to reconstruction quality of 6.0%, 2.1%, and 3.4% for acquisition of targeted m / z .

High-Throughput Mass Spectrometry Imaging with Dynamic Sparse Sampling

Mass spectrometry imaging (MSI) enables label-free mapping of hundreds of molecules in biological samples with high sensitivity and unprecedented specificity. Conventional MSI experiments are relatively slow, limiting their utility for applications requiring rapid data acquisition, such as intraoperative tissue analysis or 3D imaging. Recent advances in MSI technology focus on improving the spatial resolution and molecular coverage, further increasing the acquisition time. Herein, a deep learning approach for dynamic sampling (DLADS) was employed to reduce the number of required measurements, thereby improving the throughput of MSI experiments in comparison with conventional methods. DLADS trains a deep learning model to dynamically predict molecularly informative tissue locations for active mass spectra sampling and reconstructs high-fidelity molecular images using only the sparsely sampled information. Experimental hardware and software integration of DLADS with nanospray desorption electrospray ionization (nano-DESI) MSI is reported for the first time, which demonstrates a 2.3-fold improvement in throughput for a linewise acquisition mode. Meanwhile, simulations indicate that a 5-10-fold throughput improvement may be achieved using the pointwise acquisition mode.

A Review of Pharmaceutical Robot based on Hyperspectral Technology

The quality and safety of medicinal products are related to patients’ lives and health. Therefore, quality inspection takes a key role in the pharmaceutical industry. Most of the previous solutions are based on machine vision, however, their performance is limited by the RGB sensor. The pharmaceutical visual inspection robot combined with hyperspectral imaging technology is becoming a new trend in the high-end medical quality inspection process since the hyperspectral data can provide spectral information with spatial knowledge. Yet, there is no comprehensive review about hyperspectral imaging-based medicinal products inspection. This paper focuses on the pivotal pharmaceutical applications, including counterfeit drugs detection, active component analysis of tables, and quality testing of herbal medicines and other medical materials. We discuss the technology and hardware of Raman spectroscopy and hyperspectral imaging, firstly. Furthermore, we review these technologies in pharmaceutical scenarios. Finally, the development tendency and prospect of hyperspectral imaging technology-based robots in the field of pharmaceutical quality inspection is summarized.

Deep Learning Approach for Dynamic Sparse Sampling for High-Throughput Mass Spectrometry Imaging

A Supervised Learning Approach for Dynamic Sampling (SLADS) addresses traditional issues with the incorporation of stochastic processes into a compressed sensing method. Statistical features, extracted from a sample reconstruction, estimate entropy reduction with regression models, in order to dynamically determine optimal sampling locations. This work introduces an enhanced SLADS method, in the form of a Deep Learning Approach for Dynamic Sampling (DLADS), showing reductions in sample acquisition times for high-fidelity reconstructions between ~ 70-80% over traditional rectilinear scanning. These improvements are demonstrated for dimensionally asymmetric, high-resolution molecular images of mouse uterine and kidney tissues, as obtained using Nanospray Desorption ElectroSpray Ionization (nano-DESI) Mass Spectrometry Imaging (MSI). The methodology for training set creation is adjusted to mitigate stretching artifacts generated when using prior SLADS approaches. Transitioning to DLADS removes the need for feature extraction, further advanced with the employment of convolutional layers to leverage inter-pixel spatial relationships. Additionally, DLADS demonstrates effective generalization, despite dissimilar training and testing data. Overall, DLADS is shown to maximize potential experimental throughput for nano-DESI MSI.

Translational biophotonics with Raman imaging: clinical applications and beyond

Clinical medicine continues to seek novel rapid non-invasive tools capable of providing greater insight into disease progression and management. Raman scattering based technologies constitute a set of tools under continuing development to address outstanding challenges spanning diagnostic medicine, surgical guidance, therapeutic monitoring, and histopathology. Here we review the mechanisms and clinical applications of Raman scattering, specifically focusing on high-speed imaging methods that can provide spatial context for translational biomedical applications.

Selective-sampling Raman imaging techniques for ex vivo assessment of surgical margins in cancer surgery

One of the main challenges in cancer surgery is to ensure the complete excision of the tumour while sparing as much healthy tissue as possible. Histopathology, the gold-standard technique used to assess the surgical margins on the excised tissue, is often impractical for intra-operative use because of the time-consuming tissue cryo-sectioning and staining, and availability of histopathologists to assess stained tissue sections. Raman micro-spectroscopy is a powerful technique that can detect microscopic residual tumours on ex vivo tissue samples with accuracy, based entirely on intrinsic chemical differences. However, raster-scanning Raman micro-spectroscopy is a slow imaging technique that typically requires long data acquisition times wich are impractical for intra-operative use. Selective-sampling Raman imaging overcomes these limitations by using information regarding the spatial properties of the tissue to reduce the number of Raman spectra. This paper reviews the latest advances in selective-sampling Raman techniques and applications, mainly based on multimodal optical imaging. We also highlight the latest results of clinical integration of a prototype device for non-melanoma skin cancer. These promising results indicate the potential impact of Raman spectroscopy for providing fast and objective assessment of surgical margins, helping surgeons ensure the complete removal of tumour cells while sparing as much healthy tissue as possible.

Fast confocal Raman imaging via context-aware compressive sensing

Raman hyperspectral imaging is a powerful method to obtain detailed chemical information about a wide variety of organic and inorganic samples noninvasively and without labels. However, due to the weak, nonresonant nature of spontaneous Raman scattering, acquiring a Raman imaging dataset is time-consuming and inefficient. In this paper we utilize a compressive imaging strategy coupled with a context-aware image prior to improve Raman imaging speed by 5- to 10-fold compared to classic point-scanning Raman imaging, while maintaining the traditional benefits of point scanning imaging, such as isotropic resolution and confocality. With faster data acquisition, large datasets can be acquired in reasonable timescales, leading to more reliable downstream analysis. On standard samples, context-aware Raman compressive imaging (CARCI) was able to reduce the number of measurements by ∼85% while maintaining high image quality (SSIM >0.85). Using CARCI, we obtained a large dataset of chemical images of fission yeast cells, showing that by collecting 5-fold more cells in a given experiment time, we were able to get more accurate chemical images, identification of rare cells, and improved biochemical modeling. For example, applying VCA to nearly 100 cells' data together, cellular organelles were resolved that were not faithfully reconstructed by a single cell's dataset.

Data mining in Raman imaging in a cellular biological system

•

Working flow of data mining in Raman imaging of cell system described.

•

Pre-processing, pattern recognition and validation discussed.

•

Machine learning methods applied at each step discussed.

•

Single-cell visualization, cell type classification and quantification applications.

The distribution and dynamics of biomolecules in the cell is of critical interest in biological research. Raman imaging techniques have expanded our knowledge of cellular biological systems significantly. The technological developments that have led to the optimization of Raman instrumentation have helped to improve the speed of the measurement and the sensitivity. As well as instrumental developments, data mining plays a significant role in revealing the complicated chemical information contained within the spectral data. A number of data mining methods have been applied to extract the spectral information and translate them into biological information. Single-cell visualization, cell classification and biomolecular/drug quantification have all been achieved by the application of data mining to Raman imaging data. Herein we summarize the framework for Raman imaging data analysis, which involves preprocessing, pattern recognition and validation. There are multiple methods developed for each stage of analysis. The characteristics of these methods are described in relation to their application in Raman imaging of the cell. Furthermore, we summarize the software that can facilitate the implementation of these methods. Through its careful selection and application, data mining can act as an essential tool in the exploration of information-rich Raman spectral data.

Low resolution Raman: the impact of spectral resolution on limit of detection and imaging speed in hyperspectral imaging

The majority of problems in analytical Raman spectroscopy are mathematically over-determined, where many more spectral variables are measured than analytic outputs (such as chemical concentrations) are calculated. Thus, to improve spectral throughput and simplify system design, some researchers have explored the use of low resolution Raman systems for cell or tissue classification, achieving accuracy independent of spectral resolution. However, the tradeoffs inherent in this approach have not been systematically studied. Here, we theoretically and experimentally explore the relationship between spectral resolution and analytical error. We show that decreased spectral resolution leads to spectral signal-to-noise ratio and therefore more reliable results and lower limits of detection for equivalent integration times in blind unmixing of hyperspectral images. Our theoretical analysis demonstrates that the primary benefit of low resolution Raman spectroscopy is in overcoming detector noise (such as thermal or electronic noise). Therefore, the benefits are most pronounced when utilizing lower-grade, uncooled detectors. Therefore, using a low-cost CMOS camera we experimentally demonstrate the ability of low resolution Raman spectroscopy to achieve substantially improved imaging performance compared to fully-resolved Raman spectral imaging, paving the way for cost-effective, pervasive Raman spectroscopy.