Deep learning as phase retrieval tool for CARS spectra

Non-resonant background removal in broadband CARS microscopy using deep-learning algorithms

Broadband Coherent anti-Stokes Raman (BCARS) microscopy is an imaging technique that can acquire full Raman spectra (400-3200 cm-1) of biological samples within a few milliseconds. However, the CARS signal suffers from an undesired non-resonant background (NRB), deriving from four-wave-mixing processes, which distorts the peak line shapes and reduces the chemical contrast. Traditionally, the NRB is removed using numerical algorithms that require expert users and knowledge of the NRB spectral profile. Recently, deep-learning models proved to be powerful tools for unsupervised automation and acceleration of NRB removal. Here, we thoroughly review the existing NRB removal deep-learning models (SpecNet, VECTOR, LSTM, Bi-LSTM) and present two novel architectures. The first one combines convolutional layers with Gated Recurrent Units (CNN + GRU); the second one is a Generative Adversarial Network (GAN) that trains an encoder-decoder network and an adversarial convolutional neural network. We also introduce an improved training dataset, generalized on different BCARS experimental configurations. We compare the performances of all these networks on test and experimental data, using them in the pipeline for spectral unmixing of BCARS images. Our analyses show that CNN + GRU and VECTOR are the networks giving the highest accuracy, GAN is the one that predicts the highest number of true positive peaks in experimental data, whereas GAN and VECTOR are the most suitable ones for real-time processing of BCARS images.

Estimation of biological variance in coherent Raman microscopy data of two cell lines using chemometrics

Broadband Coherent Anti-Stokes Raman Scattering (BCARS) is a valuable spectroscopic imaging tool for visualizing cellular structures and lipid distributions in biomedical applications. However, the inevitable biological changes in the samples (cells/tissues/lipids) introduce spectral variations in BCARS data and make analysis challenging. In this work, we conducted a systematic study to estimate the biological variance in BCARS data of two commonly used cell lines (HEK293 and HepG2) in biomedical research. The BCARS data were acquired from two different experimental setups (Leibniz Institute of Photonics Technology (IPHT) in Jena and Politecnico di Milano (POLIMI) in Milano) to evaluate the reproducibility of results. Also, spontaneous Raman data were independently acquired at POLIMI to validate those results. First, Kramers-Kronig (KK) algorithm was utilized to retrieve Raman-like signals from the BCARS data, and a pre-processing pipeline was subsequently used to standardize the data. Principal component analysis - Linear discriminant analysis (PCA-LDA) was performed using two cross-validation (CV) methods: batch-out CV and 10-fold CV. Additionally, the analysis was repeated, considering different spectral regions of the data as input to the PCA-LDA. Finally, the classification accuracies of the two BCARS datasets were compared with the results of spontaneous Raman data. The results demonstrated that the CH band region (2770-3070 cm-1) and spectral data in the 1500-1800 cm-1 region have significantly contributed to the classification. A maximum of 100% balanced accuracies were obtained for the 10-fold CV for both BCARS setups. However, in the case of batch-out CV, it is 92.4% for the IPHT dataset and 98.8% for the POLIMI dataset. This study offers a comprehensive overview for estimating biological variance in biomedical applications. The insights gained from this analysis hold promise for improving the reliability of BCARS measurements in biomedical applications, paving the way for more accurate and meaningful spectroscopic analyses in the study of biological systems.

PhAI: A deep-learning approach to solve the crystallographic phase problem

X-ray crystallography provides a distinctive view on the three-dimensional structure of crystals. To reconstruct the electron density map, the complex structure factors [Formula: see text] of a sufficiently large number of diffracted reflections must be known. In a conventional experiment, only the amplitudes [Formula: see text] are obtained, and the phases ϕ are lost. This is the crystallographic phase problem. In this work, we show that a neural network, trained on millions of artificial structure data, can solve the phase problem at a resolution of only 2 angstroms, using only 10 to 20% of the data needed for direct methods. The network works in common space groups and for modest unit-cell dimensions and suggests that neural networks could be used to solve the phase problem in the general case for weakly scattering crystals.

Coherent anti-Stokes Raman scattering spectroscopy system for observation of water molecules in anion exchange membrane

Anion exchange membrane fuel cells (AEMFCs) provide one of the most feasible remedies to fuel cells' dependency on the dwindling Pt group catalysts. Nevertheless, AEMFCs still suffer reduced durability, which requires an in-depth understanding of their membranes. The low thermal endurance of the anion exchange membranes (AEMs) usually limits the direct application of powerful techniques, such as Raman spectroscopy. We sought to establish a system for coherent anti-Stokes Raman scattering (CARS) spectroscopy capable of taking measurements inside an AEM rapidly and accurately without photodamage. A 785 nm CARS system was newly developed to study the water species in an AEM (QPAF-4) located vertically in a fuel cell. From the results of water measurement in a QPAF-4 membrane, the OH-related region was deconvoluted into nine Gaussian peaks: Five H-bonded OH peaks, non-H-bonded OH, OH-, and two CH peaks. The H-bonded species increased with increasing relative humidity, but the other species remained constant. These results open unlimited possibilities for studying and comparing different AEMFCs, enabling more rapid technology optimization.

Log-Gaussian gamma processes for training Bayesian neural networks in Raman and CARS spectroscopies

We propose an approach utilizing gamma-distributed random variables, coupled with log-Gaussian modeling, to generate synthetic datasets suitable for training neural networks. This addresses the challenge of limited real observations in various applications. We apply this methodology to both Raman and coherent anti-Stokes Raman scattering (CARS) spectra, using experimental spectra to estimate gamma process parameters. Parameter estimation is performed using Markov chain Monte Carlo methods, yielding a full Bayesian posterior distribution for the model which can be sampled for synthetic data generation. Additionally, we model the additive and multiplicative background functions for Raman and CARS with Gaussian processes. We train two Bayesian neural networks to estimate parameters of the gamma process which can then be used to estimate the underlying Raman spectrum and simultaneously provide uncertainty through the estimation of parameters of a probability distribution. We apply the trained Bayesian neural networks to experimental Raman spectra of phthalocyanine blue, aniline black, naphthol red, and red 264 pigments and also to experimental CARS spectra of adenosine phosphate, fructose, glucose, and sucrose. The results agree with deterministic point estimates for the underlying Raman and CARS spectral signatures.

Comparing transmission- and epi-BCARS: a round robin on solid-state materials

Broadband coherent anti-Stokes Raman scattering (BCARS) is a powerful spectroscopy method combining high signal intensity with spectral sensitivity, enabling rapid imaging of heterogeneous samples in biomedical research and, more recently, in crystalline materials. However, BCARS encounters spectral distortion due to a setup-dependent non-resonant background (NRB). This study assesses BCARS reproducibility through a round robin experiment using two distinct BCARS setups and crystalline materials with varying structural complexity, including diamond, 6H-SiC, KDP, and KTP. The analysis compares setup-specific NRB correction procedures, detected and NRB-removed spectra, and mode assignment. We determine the influence of BCARS setup parameters like pump wavelength, pulse width, and detection geometry and provide a practical guide for optimizing BCARS setups for solid-state applications.

Removing non-resonant background from broadband CARS using a physics-informed neural network

Broadband coherent anti-Stokes Raman scattering (BCARS) is capable of producing high-quality Raman spectra spanning broad bandwidths, 400-4000 cm-1, with millisecond acquisition times. Raw BCARS spectra, however, are a coherent combination of vibrationally resonant (Raman) and non-resonant (electronic) components that may challenge or degrade chemical analyses. Recently, we demonstrated a deep convolutional autoencoder network, trained on pairs of simulated BCARS-Raman datasets, which could retrieve the Raman signal with high quality under ideal conditions. In this work, we present a new computational system that incorporates experimental measurements of the laser system spectral and temporal properties, combined with simulated susceptibilities. Thus, the neural network learns the mapping between the susceptibility and the measured response for a specific BCARS system. The network is tested on simulated and measured experimental results taken with our BCARS system.

Weakly Supervised Identification and Localization of Drug Fingerprints Based on Label-Free Hyperspectral CARS Microscopy

Understanding drug fingerprints in complex biological samples is essential for the development of a drug. Hyperspectral coherent anti-Stokes Raman scattering (HS-CARS) microscopy, a label-free nondestructive chemical imaging technique, can profile biological samples based on their endogenous vibrational contrast. Here, we propose a deep learning-assisted HS-CARS imaging approach for the investigation of drug fingerprints and their localization at single-cell resolution. To identify and localize drug fingerprints in complex biological systems, an attention-based deep neural network, hyperspectral attention net (HAN), was developed. By formulating the task to a multiple instance learning problem, HAN highlights informative regions through the attention mechanism when being trained on whole-image labels. Using the proposed technique, we investigated the drug fingerprints of a hepatitis B virus therapy in murine liver tissues. With the increase in drug dosage, higher classification accuracy was observed, with an average area under the curve (AUC) of 0.942 for the high-dose group. Besides, highly informative tissue structures predicted by HAN demonstrated a high degree of similarity with the drug localization shown by the in situ hybridization staining results. These results demonstrate the potential of the proposed deep learning-assisted optical imaging technique for the label-free profiling, identification, and localization of drug fingerprints in biological samples, which can be extended to nonperturbative investigations of complex biological systems under various biological conditions.

Effect of non-resonant background on the extraction of Raman signals from CARS spectra using deep neural networks

We report the retrieval of the Raman signal from coherent anti-Stokes Raman scattering (CARS) spectra using a convolutional neural network (CNN) model. Three different types of non-resonant backgrounds (NRBs) were explored to simulate the CARS spectra viz (1) product of two sigmoids following the original SpecNet model, (2) Single Sigmoid, and (3) fourth-order polynomial function. Later, 50 000 CARS spectra were separately synthesized using each NRB type to train the CNN model and, after training, we tested its performance on 300 simulated test spectra. The results have shown that imaginary part extraction capability is superior for the model trained with Polynomial NRB, and the extracted line shapes are in good agreement with the ground truth. Moreover, correlation analysis was carried out to compare the retrieved Raman signals to real ones, and a higher correlation coefficient was obtained for the model trained with the Polynomial NRB (on average, ∼0.95 for 300 test spectra), whereas it was ∼0.89 for the other NRBs. Finally, the predictive capability is evaluated on three complex experimental CARS spectra (DMPC, ADP, and yeast), where the Polynomial NRB model performance is found to stand out from the rest. This approach has a strong potential to simplify the analysis of complex CARS spectroscopy and can be helpful in real-time microscopy imaging applications.

Coherent anti-Stokes Raman scattering cell imaging and segmentation with unsupervised data analysis

Coherent Raman imaging has been extensively applied to live-cell imaging in the last 2 decades, allowing to probe the intracellular lipid, protein, nucleic acid, and water content with a high-acquisition rate and sensitivity. In this context, multiplex coherent anti-Stokes Raman scattering (MCARS) microspectroscopy using sub-nanosecond laser pulses is now recognized as a mature and straightforward technology for label-free bioimaging, offering the high spectral resolution of conventional Raman spectroscopy with reduced acquisition time. Here, we introduce the combination of the MCARS imaging technique with unsupervised data analysis based on multivariate curve resolution (MCR). The MCR process is implemented under the classical signal non-negativity constraint and, even more originally, under a new spatial constraint based on cell segmentation. We thus introduce a new methodology for hyperspectral cell imaging and segmentation, based on a simple, unsupervised workflow without any spectrum-to-spectrum phase retrieval computation. We first assess the robustness of our approach by considering cells of different types, namely, from the human HEK293 and murine C2C12 lines. To evaluate its applicability over a broader range, we then study HEK293 cells in different physiological states and experimental situations. Specifically, we compare an interphasic cell with a mitotic (prophase) one. We also present a comparison between a fixed cell and a living cell, in order to visualize the potential changes induced by the fixation protocol in cellular architecture. Next, with the aim of assessing more precisely the sensitivity of our approach, we study HEK293 living cells overexpressing tropomyosin-related kinase B (TrkB), a cancer-related membrane receptor, depending on the presence of its ligand, brain-derived neurotrophic factor (BDNF). Finally, the segmentation capability of the approach is evaluated in the case of a single cell and also by considering cell clusters of various sizes.

Fingerprint multiplex CARS at high speed based on supercontinuum generation in bulk media and deep learning spectral denoising

We introduce a broadband coherent anti-Stokes Raman scattering (CARS) microscope based on a 2-MHz repetition rate ytterbium laser generating 1035-nm high-energy (≈µJ level) femtosecond pulses. These features of the driving laser allow producing broadband red-shifted Stokes pulses, covering the whole fingerprint region (400-1800 cm^-1), employing supercontinuum generation in a bulk crystal. Our system reaches state-of-the-art acquisition speed (<1 ms/pixel) and unprecedented sensitivity of ≈14.1 mmol/L when detecting dimethyl sulfoxide in water. To further improve the performance of the system and to enhance the signal-to-noise ratio of the CARS spectra, we designed a convolutional neural network for spectral denoising, coupled with a post-processing pipeline to distinguish different chemical species of biological tissues.

Deep Learning for Raman Spectroscopy: A Review

Raman spectroscopy (RS) is a spectroscopic method which indirectly measures the vibrational states within samples. This information on vibrational states can be utilized as spectroscopic fingerprints of the sample, which, subsequently, can be used in a wide range of application scenarios to determine the chemical composition of the sample without altering it, or to predict a sample property, such as the disease state of patients. These two examples are only a small portion of the application scenarios, which range from biomedical diagnostics to material science questions. However, the Raman signal is weak and due to the label-free character of RS, the Raman data is untargeted. Therefore, the analysis of Raman spectra is challenging and machine learning based chemometric models are needed. As a subset of representation learning algorithms, deep learning (DL) has had great success in data science for the analysis of Raman spectra and photonic data in general. In this review, recent developments of DL algorithms for Raman spectroscopy and the current challenges in the application of these algorithms will be discussed.

Raman signal extraction from CARS spectra using a learned-matrix representation of the discrete Hilbert transform

Removing distortions in coherent anti-Stokes Raman scattering (CARS) spectra due to interference with the nonresonant background (NRB) is vital for quantitative analysis. Popular computational approaches, the Kramers-Kronig relation and the maximum entropy method, have demonstrated success but may generate significant errors due to peaks that extend in any part beyond the recording window. In this work, we present a learned matrix approach to the discrete Hilbert transform that is easy to implement, fast, and dramatically improves accuracy of Raman retrieval using the Kramers-Kronig approach.

Development of deep learning algorithms to discriminate giant cell tumors of bone from adjacent normal tissues by confocal Raman spectroscopy

Raman spectroscopy is a non-destructive analysis technique that provides detailed information about the chemical structure of tumors. Raman spectra of 52 giant cell tumors of bone (GCTB) and 21 adjacent normal tissues of formalin-fixed paraffin embedded (FFPE) and frozen specimens were obtained using a confocal Raman spectrometer and analyzed with machine learning and deep learning algorithms. We discovered characteristic Raman shifts in the GCTB specimens. They were assigned to phenylalanine and tyrosine. Based on the spectroscopic data, classification algorithms including support vector machine, k-nearest neighbors and long short-term memory (LSTM) were successfully applied to discriminate GCTB from adjacent normal tissues of both the FFPE and frozen specimens, with the accuracy ranging from 82.8% to 94.5%. Importantly, our LSTM algorithm showed the best performance in the discrimination of the frozen specimens, with a sensitivity and specificity of 93.9% and 95.1% respectively, and the AUC was 0.97. The results of our study suggest that confocal Raman spectroscopy accomplished by the LSTM network could non-destructively evaluate a tumor margin by its inherent biochemical specificity which may allow intraoperative assessment of the adequacy of tumor clearance.

Hyperspectral CARS microscopy and quantitative unsupervised analysis of deuterated and non-deuterated fatty acid storage in human cells

Coherent anti-Stokes Raman scattering (CARS) implemented as a vibrational micro-spectroscopy modality eradicates the need for potentially perturbative fluorescent labeling while still providing high-resolution, chemically specific images of biological samples. Isotopic substitution of hydrogen atoms with deuterium introduces minimal change to molecular structures and can be coupled with CARS microscopy to increase chemical contrast. Here, we investigate HeLa cells incubated with non-deuterated or deuterium-labeled fatty acids, using an in-house-developed hyperspectral CARS microscope coupled with an unsupervised quantitative data analysis algorithm, to retrieve Raman susceptibility spectra and concentration maps of chemical components in physically meaningful units. We demonstrate that our unsupervised analysis retrieves the susceptibility spectra of the specific fatty acids, both deuterated and non-deuterated, in good agreement with reference Raman spectra measured in pure lipids. Our analysis, using the cell-silent spectral region, achieved excellent chemical specificity despite having no prior knowledge and considering the complex intracellular environment inside cells. The quantitative capabilities of the analysis allowed us to measure the concentration of deuterated and non-deuterated fatty acids stored within cytosolic lipid droplets over a 24 h period. Finally, we explored the potential use of deuterium-labeled lipid droplets for non-invasive cell tracking, demonstrating an effective application of the technique for distinguishing between cells in a mixed population over a 16 h period. These results further demonstrate the chemically specific capabilities of hyperspectral CARS microscopy to characterize and distinguish specific lipid types inside cells using an unbiased quantitative data analysis methodology.

Coherent Raman scattering imaging with a near-infrared achromatic metalens

Miniature handheld imaging devices and endoscopes based on coherent Raman scattering are promising for label-free in vivo optical diagnosis. Toward the development of these small-scale systems, a challenge arises from the design and fabrication of achromatic and high-end miniature optical components for both pump and Stokes laser wavelengths. Here, we report a metasurface converting a low-cost plano-convex lens into a water-immersion, nearly diffraction-limited and achromatic lens. The metasurface comprising amorphous silicon nanopillars is designed in a way that all incident rays arrive at the focus with the same phase and group delay, leading to corrections of monochromatic and chromatic aberrations of the refractive lens, respectively. Compared to the case without the metasurface, the hybrid metasurface-refractive lens has higher Strehl ratios than the plano-convex lens and a tighter depth of focus. The hybrid metasurface-refractive lens is utilized in spectroscopic stimulated Raman scattering and coherent anti-Stokes Raman scattering imaging for the differentiation of two different polymer microbeads. Subsequently, the hybrid metalens is harnessed for volumetric coherent Raman scattering imaging of bead and tissue samples. Finally, we discuss possible approaches to integrate such hybrid metalens in a miniature scanning system for label-free coherent Raman scattering endoscopes.