Deep learning a boon for biophotonics?

Combined Mutual Learning Net for Raman Spectral Microbial Strain Identification

Infectious diseases pose a significant threat to global health, yet traditional microbiological identification methods suffer from drawbacks, such as high costs and long processing times. Raman spectroscopy, a label-free and noninvasive technique, provides rich chemical information and has tremendous potential in fast microbial diagnoses. Here, we propose a novel Combined Mutual Learning Net that precisely identifies microbial subspecies. It demonstrated an average identification accuracy of 87.96% in an open-access data set with thirty microbial strains, representing a 5.76% improvement. 50% of the microbial subspecies accuracies were elevated by 1% to 46%, especially for E. coli 2 improved from 31% to 77%. Furthermore, it achieved a remarkable subspecies accuracy of 92.4% in the custom-built fiber-optical tweezers Raman spectroscopy system, which collects Raman spectra at a single-cell level. This advancement demonstrates the effectiveness of this method in microbial subspecies identification, offering a promising solution for microbiology diagnosis.

Efficacy of tyrosine kinase inhibitors examined by a combination of Raman micro-spectroscopy and a deep wavelet scattering-based multivariate analysis framework

HER2 is a crucial therapeutic target in breast cancer, and the survival rate of breast cancer patients has increased because of this receptor's inhibition. However, tumors have shown resistance to this therapeutic strategy due to oncogenic mutations that decrease the binding of several HER2-targeted drugs, including lapatinib, and confer resistance to this drug. Neratinib can overcome this drug resistance and effectively inhibit HER2 signaling and tumor growth. In the present study, we examined the efficacy of lapatinib and neratinib using breast cancer cells by Raman microscopy combined with a deep wavelet scattering-based multivariate analysis framework. This approach discriminated between control cells and drug-treated cells with high accuracy, compared to classical principal component analysis. Both lapatinib and neratinib induced changes in the cellular biochemical composition. Furthermore, the Raman results were compared with the results of several in vitro assays. For instance, drug-treated cells exhibited (i) inhibition of ERK and AKT phosphorylation, (ii) inhibition of cellular proliferation, (iii) cell-cycle arrest, and (iv) apoptosis as indicated by western blotting, real-time cell analysis (RTCA), cell-cycle analysis, and apoptosis assays. Thus, the observed Raman spectral changes are attributed to cell-cycle arrest and apoptosis. The results also indicated that neratinib is more potent than lapatinib. Moreover, the uptake and distribution of lapatinib in cells were visualized through its label-free marker bands in the fingerprint region using Raman spectral imaging. These results show the prospects of Raman microscopy in drug evaluation and presumably in drug discovery.

Siamese Networks for Clinically Relevant Bacteria Classification Based on Raman Spectroscopy

Identifying bacterial strains is essential in microbiology for various practical applications, such as disease diagnosis and quality monitoring of food and water. Classical machine learning algorithms have been utilized to identify bacteria based on their Raman spectra. However, convolutional neural networks (CNNs) offer higher classification accuracy, but they require extensive training sets and retraining of previous untrained class targets can be costly and time-consuming. Siamese networks have emerged as a promising solution. They are composed of two CNNs with the same structure and a final network that acts as a distance metric, converting the classification problem into a similarity problem. Classical machine learning approaches, shallow and deep CNNs, and two Siamese network variants were tailored and tested on Raman spectral datasets of bacteria. The methods were evaluated based on mean sensitivity, training time, prediction time, and the number of parameters. In this comparison, Siamese-model2 achieved the highest mean sensitivity of 83.61 ± 4.73 and demonstrated remarkable performance in handling unbalanced and limited data scenarios, achieving a prediction accuracy of 73%. Therefore, the choice of model depends on the specific trade-off between accuracy, (prediction/training) time, and resources for the particular application. Classical machine learning models and shallow CNN models may be more suitable if time and computational resources are a concern. Siamese networks are a good choice for small datasets and CNN for extensive data.

Label-free SERS detection of prostate cancer based on multi-layer perceptron surrogate model method

Prior surface-enhanced Raman spectroscopy (SERS) research has shown that pre-processing is necessary before analysis. Pre-processing also typically serves the dual purposes of removing the auto-fluorescence background and minimizing data volatility. This method allows for a more accurate comparison of spectral traits and relative SERS peak strength. However, because there are so many different kinds of samples, it can take a long time, and there is no assurance that the approach chosen will work well with a particular kind of sample. Therefore, this study employed a deep learning technique called multi-layer perceptron (MLP) to simplify the pre-processing of blood plasma SERS samples in patients with prostate cancer (PC), as well as to enhance the sensitivity and specificity of diagnosis using SERS technology. First of all, significant variations in peak intensity can be observed in the difference spectra, facilitating differentiation between PC and normal groups. Second, the data analysis was carried out in three different stages (raw data, defluorescenced data, and normalized data) using principal component analysis and linear discriminant analysis (PCA-LDA), as well as PCA-multi-layer perceptron (PCA-MLP). Finally, when SERS data was analyzed using PCA-LDA, there were significant differences in classification accuracy across each stage (The classification accuracy of three different stages were 76.90%, 85.60%, 95.20%, respectively). However, when PCA-MLP was utilized for SERS data analysis, the classification accuracy remained consistently high and stable (The classification accuracy of three different stages were 92.00%, 92.40%, 96.70%, respectively). The experimental results of PCA-MLP for classifying specific SERS data indicate that analyzing raw data directly can simplify the experimental process and enhance the efficacy of SERS analysis.

A review of recent progress in the application of Raman spectroscopy and SERS detection of microplastics and derivatives

The global environmental concern surrounding microplastic (MP) pollution has raised alarms due to its potential health risks to animals, plants, and humans. Because of the complex structure and composition of microplastics (MPs), the detection methods are limited, resulting in restricted detection accuracy. Surface enhancement of Raman spectroscopy (SERS), a spectral technique, offers several advantages, such as high resolution and low detection limit. It has the potential to be extensively employed for sensitive detection and high-resolution imaging of microplastics. We have summarized the research conducted in recent years on the detection of microplastics using Raman and SERS. Here, we have reviewed qualitative and quantitative analyses of microplastics and their derivatives, as well as the latest progress, challenges, and potential applications.

Dimensionality reduction for deep learning in infrared microscopy: a comparative computational survey

While infrared microscopy provides molecular information at spatial resolution in a label-free manner, exploiting both spatial and molecular information for classifying the disease status of tissue samples constitutes a major challenge. One strategy to mitigate this problem is to embed high-dimensional pixel spectra in lower dimensions, aiming to preserve molecular information in a more compact manner, which reduces the amount of data and promises to make subsequent disease classification more accessible for machine learning procedures. In this study, we compare several dimensionality reduction approaches and their effect on identifying cancer in the context of a colon carcinoma study. We observe surprisingly small differences between convolutional neural networks trained on dimensionality reduced spectra compared to utilizing full spectra, indicating a clear tendency of the convolutional networks to focus on spatial rather than spectral information for classifying disease status.

Computational approaches in rheumatic diseases - Deciphering complex spatio-temporal cell interactions

Inflammatory arthritis, including rheumatoid (RA), and psoriatic (PsA) arthritis, are clinically and immunologically heterogeneous diseases with no identified cure. Chronic inflammation of the synovial tissue ushers loss of function of the joint that severely impacts the patient's quality of life, eventually leading to disability and life-threatening comorbidities. The pathogenesis of synovial inflammation is the consequence of compounded immune and stromal cell interactions influenced by genetic and environmental factors. Deciphering the complexity of the synovial cellular landscape has accelerated primarily due to the utilisation of bulk and single cell RNA sequencing. Particularly the capacity to generate cell-cell interaction networks could reveal evidence of previously unappreciated processes leading to disease. However, there is currently a lack of universal nomenclature as a result of varied experimental and technological approaches that discombobulates the study of synovial inflammation. While spatial transcriptomic analysis that combines anatomical information with transcriptomic data of synovial tissue biopsies promises to provide more insights into disease pathogenesis, in vitro functional assays with single-cell resolution will be required to validate current bioinformatic applications. In order to provide a comprehensive approach and translate experimental data to clinical practice, a combination of clinical and molecular data with machine learning has the potential to enhance patient stratification and identify individuals at risk of arthritis that would benefit from early therapeutic intervention. This review aims to provide a comprehensive understanding of the effect of computational approaches in deciphering synovial inflammation pathogenesis and discuss the impact that further experimental and novel computational tools may have on therapeutic target identification and drug development.

Automatic classification of Candida species using Raman spectroscopy and machine learning

One of the problems that most affect hospitals is infections by pathogenic microorganisms. Rapid identification and adequate, timely treatment can avoid fatal consequences and the development of antibiotic resistance, so it is crucial to use fast, reliable, and not too laborious techniques to obtain quick results. Raman spectroscopy has proven to be a powerful tool for molecular analysis, meeting these requirements better than traditional techniques. In this work, we have used Raman spectroscopy combined with machine learning algorithms to explore the automatic identification of eleven species of the genus Candida, the most common cause of fungal infections worldwide. The Raman spectra were obtained from more than 220 different measurements of dried drops from pure cultures of each Candida species using a Raman Confocal Microscope with a 532 nm laser excitation source. After developing a spectral preprocessing methodology, a study of the quality and variability of the measured spectra at the isolate and species level, and the spectral features contributing to inter-class variations, showed the potential to discriminate between those pathogenic yeasts. Several machine learning and deep learning algorithms were trained using hyperparameter optimization techniques to find the best possible classifier for this spectral data, in terms of accuracy and lowest possible overfitting. We found that a one-dimensional Convolutional Neural Network (1-D CNN) could achieve above 80 % overall accuracy for the eleven classes spectral dataset, with good generalization capabilities.

A one-dimensional convolutional neural network based deep learning for high accuracy classification of transformation stages in esophageal squamous cell carcinoma tissue using micro-FTIR

Among the most frequently diagnosed cancers in developing countries, esophageal squamous cell carcinoma (ESCC) ranks among the top six causes of death. It would be beneficial if a rapid, accurate, and automatic ESCC diagnostic method could be developed to reduce the workload of pathologists and improve the effectiveness of cancer treatments. Using micro-FTIR spectroscopy, this study classified the transformation stages of ESCC tissues. Based on 6,352 raw micro-FTIR spectra, a one-dimensional convolutional neural network (1D-CNN) model was constructed to classify-five stages. Based on the established model, more than 93% accuracy was achieved at each stage, and the accuracy of identifying proliferation, low grade neoplasia, and ESCC cancer groups was achieved 99% for the test dataset. In this proof-of-concept study, the developed method can be applied to other diseases in order to promote the use of FTIR spectroscopy in cancer pathology.

Recent Trends in SERS-Based Plasmonic Sensors for Disease Diagnostics, Biomolecules Detection, and Machine Learning Techniques

Surface-enhanced Raman spectroscopy/scattering (SERS) has evolved into a popular tool for applications in biology and medicine owing to its ease-of-use, non-destructive, and label-free approach. Advances in plasmonics and instrumentation have enabled the realization of SERS's full potential for the trace detection of biomolecules, disease diagnostics, and monitoring. We provide a brief review on the recent developments in the SERS technique for biosensing applications, with a particular focus on machine learning techniques used for the same. Initially, the article discusses the need for plasmonic sensors in biology and the advantage of SERS over existing techniques. In the later sections, the applications are organized as SERS-based biosensing for disease diagnosis focusing on cancer identification and respiratory diseases, including the recent SARS-CoV-2 detection. We then discuss progress in sensing microorganisms, such as bacteria, with a particular focus on plasmonic sensors for detecting biohazardous materials in view of homeland security. At the end of the article, we focus on machine learning techniques for the (a) identification, (b) classification, and (c) quantification in SERS for biology applications. The review covers the work from 2010 onwards, and the language is simplified to suit the needs of the interdisciplinary audience.

Culture-Independent Raman Spectroscopic Identification of Bacterial Pathogens from Clinical Samples Using Deep Transfer Learning

The rapid identification of bacterial pathogens in clinical samples like blood, urine, pus, and sputum is the need of the hour. Conventional bacterial identification methods like culturing and nucleic acid-based amplification have limitations like poor sensitivity, high cost, slow turnaround time, etc. Raman spectroscopy, a label-free and noninvasive technique, has overcome these drawbacks by providing rapid biochemical signatures from a single bacterium. Raman spectroscopy combined with chemometric methods has been used effectively to identify pathogens. However, a robust approach is needed to utilize Raman features for accurate classification while dealing with complex data sets such as spectra obtained from clinical isolates, showing high sample-to-sample heterogeneity. In this study, we have used Raman spectroscopy-based identification of pathogens from clinical isolates using a deep transfer learning approach at the single-cell level resolution. We have used the data-augmentation method to increase the volume of spectra needed for deep-learning analysis. Our ResNet model could specifically extract the spectral features of eight different pathogenic bacterial species with a 99.99% classification accuracy. The robustness of our model was validated on a set of blinded data sets, a mix of cultured and noncultured bacterial isolates of various origins and types. Our proposed ResNet model efficiently identified the pathogens from the blinded data set with high accuracy, providing a robust and rapid bacterial identification platform for clinical microbiology.

Treatment of chronic liver disease using cell‑engineered constructs: morphofunctional characteristics

Objective: to study the effectiveness of correcting the morphofunctional characteristics of the liver in an experimental model of chronic liver disease (CLD), using implanted cell-engineered constructs (CECs).

Materials and methods. Experiments were carried out on male Wistar rats (n = 80) aged 6–8 months with an initial weight of 230–250 g. CLD was modeled by inoculating the rats with 60% CCl4 oil solution for 42 days based on a modified scheme. Microgel based on recombinant spidroin rS1/9 was used as a matrix for CECs fabrication. Allogeneic liver cells (LCs) and multipotent bone marrow-derived mesenchymal stem cells (BM-MSCs) from a healthy donor were used as the cellular component of the CECs. The effectiveness of the corrective effect of the implanted CECs was assessed in an experimental CLD model (n = 60) in two groups of rats: Group 1 (control, n = 20, 1 mL of saline solution was injected into the damaged liver parenchyma) and Group 2 (experimental, n = 40, CECs containing allogenic LCs and BM-MSCs in a 5 : 1 ratio in a volume of 1 mL were implanted into the damaged liver parenchyma). For long-term monitoring of the CEC state, the CECs were labeled by additional inclusion in Cytodex-3. The effectiveness of the regulatory effect of CECs on regenerative processes in the liver was evaluated using biochemical, morphological and morphometric techniques, as well as by flow cytometry at 90 days after implantation.

Results. In the control group, the mortality rate in CLD was 25%. There was no death in the experimental group with CLD after CEC implantation. The CECs were found to have a corrective effect on the biochemical and morphological parameters of the liver in CLD during 90 days of follow-up, with concomitant preservation of structural cellular homeostasis in the implanted CECs. Conclusion. Implantation of CECs in the liver facilitates effective correction of CLD by activating regenerative processes in the damaged liver, which is due to long-term preservation of structural cellular homeostasis in the CECs.

Deep Learning for Raman Spectroscopy: A Review

Raman spectroscopy (RS) is a spectroscopic method which indirectly measures the vibrational states within samples. This information on vibrational states can be utilized as spectroscopic fingerprints of the sample, which, subsequently, can be used in a wide range of application scenarios to determine the chemical composition of the sample without altering it, or to predict a sample property, such as the disease state of patients. These two examples are only a small portion of the application scenarios, which range from biomedical diagnostics to material science questions. However, the Raman signal is weak and due to the label-free character of RS, the Raman data is untargeted. Therefore, the analysis of Raman spectra is challenging and machine learning based chemometric models are needed. As a subset of representation learning algorithms, deep learning (DL) has had great success in data science for the analysis of Raman spectra and photonic data in general. In this review, recent developments of DL algorithms for Raman spectroscopy and the current challenges in the application of these algorithms will be discussed.

Fast intraoperative histology-based diagnosis of gliomas with third harmonic generation microscopy and deep learning

Management of gliomas requires an invasive treatment strategy, including extensive surgical resection. The objective of the neurosurgeon is to maximize tumor removal while preserving healthy brain tissue. However, the lack of a clear tumor boundary hampers the neurosurgeon's ability to accurately detect and resect infiltrating tumor tissue. Nonlinear multiphoton microscopy, in particular higher harmonic generation, enables label-free imaging of excised brain tissue, revealing histological hallmarks within seconds. Here, we demonstrate a real-time deep learning-based pipeline for automated glioma image analysis, matching video-rate image acquisition. We used a custom noise detection scheme, and a fully-convolutional classification network, to achieve on average 79% binary accuracy, 0.77 AUC and 0.83 mean average precision compared to the consensus of three pathologists, on a preliminary dataset. We conclude that the combination of real-time imaging and image analysis shows great potential for intraoperative assessment of brain tissue during tumor surgery.

Machine Learning of Raman Spectroscopy Data for Classifying Cancers: A Review of the Recent Literature

Raman Spectroscopy has long been anticipated to augment clinical decision making, such as classifying oncological samples. Unfortunately, the complexity of Raman data has thus far inhibited their routine use in clinical settings. Traditional machine learning models have been used to help exploit this information, but recent advances in deep learning have the potential to improve the field. However, there are a number of potential pitfalls with both traditional and deep learning models. We conduct a literature review to ascertain the recent machine learning methods used to classify cancers using Raman spectral data. We find that while deep learning models are popular, and ostensibly outperform traditional learning models, there are many methodological considerations which may be leading to an over-estimation of performance; primarily, small sample sizes which compound sub-optimal choices regarding sampling and validation strategies. Amongst several recommendations is a call to collate large benchmark Raman datasets, similar to those that have helped transform digital pathology, which researchers can use to develop and refine deep learning models.

Noises investigations and image denoising in femtosecond stimulated Raman scattering microscopy

In the literature of SRS microscopy, the hardware characterization usually remains separate from the image processing. In this article, we consider both these aspects and statistical properties analysis of image noise, which plays the vital role of joining links between them. Firstly, we perform hardware characterization by systematic measurements of noise sources, demonstrating that our in-house built microscope is shot noise limited. Secondly, we analyze the statistical properties of the overall image noise, and we prove that the noise distribution can be dependent on image direction, whose origin is the use of a lock-in time constant longer than pixel dwell time. Finally, we compare the performances of two widespread general algorithms, that is, singular value decomposition and discrete wavelet transform, with a method, that is, singular spectrum analysis (SSA), which has been adapted for stimulated Raman scattering images. In order to validate our algorithms, in our investigations lipids droplets have been used and we demonstrate that the adapted SSA method provides an improvement in image denoising.

High frame rate (∼3 Hz) circular photoacoustic tomography using single-element ultrasound transducer aided with deep learning

SIGNIFICANCE

In circular scanning photoacoustic tomography (PAT), it takes several minutes to generate an image of acceptable quality, especially with a single-element ultrasound transducer (UST). The imaging speed can be enhanced by faster scanning (with high repetition rate light sources) and using multiple-USTs. However, artifacts arising from the sparse signal acquisition and low signal-to-noise ratio at higher scanning speeds limit the imaging speed. Thus, there is a need to improve the imaging speed of the PAT systems without hampering the quality of the PAT image.

AIM

To improve the frame rate (or imaging speed) of the PAT system by using deep learning (DL).

APPROACH

For improving the frame rate (or imaging speed) of the PAT system, we propose a novel U-Net-based DL framework to reconstruct PAT images from fast scanning data.

RESULTS

The efficiency of the network was evaluated on both single- and multiple-UST-based PAT systems. Both phantom and in vivo imaging demonstrate that the network can improve the imaging frame rate by approximately sixfold in single-UST-based PAT systems and by approximately twofold in multi-UST-based PAT systems.

CONCLUSIONS

We proposed an innovative method to improve the frame rate (or imaging speed) by using DL and with this method, the fastest frame rate of ∼ 3    Hz imaging is achieved without hampering the quality of the reconstructed image.

Towards Raman imaging of centimeter scale tissue areas for real-time opto-molecular visualization of tissue boundaries for clinical applications

Raman spectroscopy combined with augmented reality and mixed reality to reconstruct molecular information of tissue surface.

Benchmarking Spectroscopic Techniques Combined with Machine Learning to Study Oak Barrels for Wine Ageing

Due to its physical, chemical, and structural properties, oakwood is widely used in the production of barrels for wine ageing. When in contact with the wine, oak continuously releases aromatic compounds such as lignin, tannin, and cellulose to the liquid. Due to the release process, oak loses its characteristic aromatic compounds in time; hence, the flavour that it gives to the enclosed wine decreases for repeated wine refills and a barrel replacement is required. Currently, the estimation of the maximum number of refills is empirical and its underestimation or overestimation can impose unnecessary costs and impair the quality of the wine. Therefore, there is a clear need to quantify the presence of the aforementioned aromatic compounds in an oak barrel prior to a refill. This work constitutes a study to examine noninvasive optical biosensing techniques for the characterization of an oak barrel used in wine ageing, towards the development of a model to unveil its lifespan without inducing structural damage. Spectroscopic diagnostic techniques, such as reflectance, fluorescence, and Raman scattering measurements are employed to assess the change in the chemical composition of the oakwood barrel (tannin and lignin presence) and its dependence on repeated refills. To our knowledge, this is the first time that we present a benchmarking study of oak barrel ageing characteristics through spectroscopic methods for the wine industry. The spectroscopic data are processed using standard chemometric techniques, such as Linear Discriminant Analysis and Partial Least Squares Discriminant Analysis. Results of a study of fresh, one-time-used, and two-times-used oak barrel samples demonstrate that reflectance spectroscopy can be a valuable tool for the characterization of oak barrels. Moreover, reflectance spectroscopy has demonstrated the most accurate classification performance. The highest accuracy has been obtained by a Partial Least Squares Discriminant Analysis model that has been able to classify all the oakwood samples from the barrels with >99% accuracy. These preliminary results pave a way for the application of cost-effective and non-invasive biosensing techniques based on reflectance spectroscopy for oak barrels assessment.

Deep learning-based image processing in optical microscopy

Optical microscopy has emerged as a key driver of fundamental research since it provides the ability to probe into imperceptible structures in the biomedical world. For the detailed investigation of samples, a high-resolution image with enhanced contrast and minimal damage is preferred. To achieve this, an automated image analysis method is preferable over manual analysis in terms of both speed of acquisition and reduced error accumulation. In this regard, deep learning (DL)-based image processing can be highly beneficial. The review summarises and critiques the use of DL in image processing for the data collected using various optical microscopic techniques. In tandem with optical microscopy, DL has already found applications in various problems related to image classification and segmentation. It has also performed well in enhancing image resolution in smartphone-based microscopy, which in turn enablse crucial medical assistance in remote places.

Graphical abstract

Attenuated Total Reflection (ATR) Micro-Fourier Transform Infrared (Micro-FT-IR) Spectroscopy to Enhance Repeatability and Reproducibility of Spectra Derived from Single Specimen Organic-Walled Dinoflagellate Cysts

The chemical composition of recent and fossil organic-walled dinoflagellate cyst walls and its diversity is poorly understood and analyses on single microscopic specimens are rare. A series of infrared spectroscopic experiments resulted in the proposition of a standardized attenuated total reflection micro-Fourier transform infrared-based method that allows the collection of robust data sets consisting of spectra from individual dinocysts. These data sets are largely devoid of nonchemical artifacts inherent to other infrared spectrochemical methods, which have typically been used to study similar specimens in the past. The influence of sample preparation, specimen morphology and size and spectral data processing steps is also assessed within this methodological framework. As a result, several guidelines are proposed which facilitate the collection and qualitative interpretation of highly reproducible and repeatable spectrochemical data. These, in turn, pave the way for a systematic exploration of dinocyst chemistry and its assessment as a chemotaxonomical tool or proxy.

RamanNet: a lightweight convolutional neural network for bacterial identification based on Raman spectra

Raman spectroscopy combined convolutional neural network (CNN) enables rapid and accurate identification of the species of bacteria. However, the existing CNN requires a complex hyperparameters model design. Herein, we propose a new simple network architecture with less hyperparameter design and low computation cost, RamanNet, for rapid and accurate identifying of bacteria at the species level based on its Raman spectra. We verified that compared with the previous CNN methods, the RamanNet reached comparable results on the Bacteria-ID Raman spectral dataset and PKU-bacterial Raman spectral datasets, but using only about 1/45 and 1/297 network parameters, respectively. RamanNet achieved an average isolate-level accuracy of 84.7 ± 0.3%, antibiotic treatment identification accuracy of 97.1 ± 0.3%, and distinguished accuracy of 81.6 ± 0.9% for methicillin-resistant and -susceptible Staphylococcus aureus (MRSA and MSSA) on the Bacteria-ID dataset, respectively. Moreover, it achieved an average accuracy of 96.04% on the PKU-bacterial dataset. The RamanNet model benefited from fewer model parameters that can be quickly trained even using CPU. Therefore, our method has the potential to rapidly and accurately identify bacterial species based on their Raman spectra and can be easily extended to other classification tasks based on Raman spectra.

We propose a novel CNN model named RamanNet for rapid and accurate identification of bacteria at the species-level based on Raman spectra. Compared to previous CNN methods, the RamanNet reached comparable results on the Bacteria-ID Raman spectral dataset.

Automated segmentation of metal stent and bioresorbable vascular scaffold in intravascular optical coherence tomography images using deep learning architectures

Percutaneous coronary intervention (PCI) with stent placement is a treatment effective for coronary artery diseases. Intravascular optical coherence tomography (OCT) with high resolution is used clinically to visualize stent deployment and restenosis, facilitating PCI operation and for complication inspection. Automated stent struts segmentation in OCT images is necessary as each pullback of OCT images could contain thousands of stent struts. In this paper, a deep learning framework is proposed and demonstrated for the automated segmentation of two major clinical stent types: metal stents and bioresorbable vascular scaffolds (BVS). U-Net, the current most prominent deep learning network in biomedical segmentation, was implemented for segmentation with cropped input. The architectures of MobileNetV2 and DenseNet121 were also adapted into U-Net for improvement in speed and accuracy. The results suggested that the proposed automated algorithm's segmentation performance approaches the level of independent human obsevers and is feasible for both types of stents despite their distinct appearance. U-Net with DenseNet121 encoder (U-Dense) performed best with Dice's coefficient of 0.86 for BVS segmentation, and precision/recall of 0.92/0.92 for metal stent segmentation under optimal crop window size of 256.

High-Throughput Molecular Imaging via Deep-Learning-Enabled Raman Spectroscopy

Raman spectroscopy enables nondestructive, label-free imaging with unprecedented molecular contrast, but is limited by slow data acquisition, largely preventing high-throughput imaging applications. Here, we present a comprehensive framework for higher-throughput molecular imaging via deep-learning-enabled Raman spectroscopy, termed DeepeR, trained on a large data set of hyperspectral Raman images, with over 1.5 million spectra (400 h of acquisition) in total. We first perform denoising and reconstruction of low signal-to-noise ratio Raman molecular signatures via deep learning, with a 10× improvement in the mean-squared error over common Raman filtering methods. Next, we develop a neural network for robust 2–4× spatial super-resolution of hyperspectral Raman images that preserve molecular cellular information. Combining these approaches, we achieve Raman imaging speed-ups of up to 40–90×, enabling good-quality cellular imaging with a high-resolution, high signal-to-noise ratio in under 1 min. We further demonstrate Raman imaging speed-up of 160×, useful for lower resolution imaging applications such as the rapid screening of large areas or for spectral pathology. Finally, transfer learning is applied to extend DeepeR from cell to tissue-scale imaging. DeepeR provides a foundation that will enable a host of higher-throughput Raman spectroscopy and molecular imaging applications across biomedicine.

Infrared nanoscopy and tomography of intracellular structures

Although techniques such as fluorescence-based super-resolution imaging or confocal microscopy simultaneously gather both morphological and chemical data, these techniques often rely on the use of localized and chemically specific markers. To eliminate this flaw, we have developed a method of examining cellular cross sections using the imaging power of scattering-type scanning near-field optical microscopy and Fourier-transform infrared spectroscopy at a spatial resolution far beyond the diffraction limit. Herewith, nanoscale surface and volumetric chemical imaging is performed using the intrinsic contrast generated by the characteristic absorption of mid-infrared radiation by the covalent bonds. We employ infrared nanoscopy to study the subcellular structures of eukaryotic (Chlamydomonas reinhardtii) and prokaryotic (Escherichia coli) species, revealing chemically distinct regions within each cell such as the microtubular structure of the flagellum. Serial 100 nm-thick cellular cross-sections were compiled into a tomogram yielding a three-dimensional infrared image of subcellular structure distribution at 20 nm resolution. The presented methodology is able to image biological samples complementing current fluorescence nanoscopy but at less interference due to the low energy of infrared radiation and the absence of labeling.

Deep learning-based classification of blue light cystoscopy imaging during transurethral resection of bladder tumors

Bladder cancer is one of the top 10 frequently occurring cancers and leads to most cancer deaths worldwide. Recently, blue light (BL) cystoscopy-based photodynamic diagnosis was introduced as a unique technology to enhance the detection of bladder cancer, particularly for the detection of flat and small lesions. Here, we aim to demonstrate a BL image-based artificial intelligence (AI) diagnostic platform using 216 BL images, that were acquired in four different urological departments and pathologically identified with respect to cancer malignancy, invasiveness, and grading. Thereafter, four pre-trained convolution neural networks were utilized to predict image malignancy, invasiveness, and grading. The results indicated that the classification sensitivity and specificity of malignant lesions are 95.77% and 87.84%, while the mean sensitivity and mean specificity of tumor invasiveness are 88% and 96.56%, respectively. This small multicenter clinical study clearly shows the potential of AI based classification of BL images allowing for better treatment decisions and potentially higher detection rates.

Sounding out the hidden data: A concise review of deep learning in photoacoustic imaging

The rapidly evolving field of photoacoustic tomography utilizes endogenous chromophores to extract both functional and structural information from deep within tissues. It is this power to perform precise quantitative measurements in vivo-with endogenous or exogenous contrast-that makes photoacoustic tomography highly promising for clinical translation in functional brain imaging, early cancer detection, real-time surgical guidance, and the visualization of dynamic drug responses. Considering photoacoustic tomography has benefited from numerous engineering innovations, it is of no surprise that many of photoacoustic tomography's current cutting-edge developments incorporate advances from the equally novel field of artificial intelligence. More specifically, alongside the growth and prevalence of graphical processing unit capabilities within recent years has emerged an offshoot of artificial intelligence known as deep learning. Rooted in the solid foundation of signal processing, deep learning typically utilizes a method of optimization known as gradient descent to minimize a loss function and update model parameters. There are already a number of innovative efforts in photoacoustic tomography utilizing deep learning techniques for a variety of purposes, including resolution enhancement, reconstruction artifact removal, undersampling correction, and improved quantification. Most of these efforts have proven to be highly promising in addressing long-standing technical obstacles where traditional solutions either completely fail or make only incremental progress. This concise review focuses on the history of applied artificial intelligence in photoacoustic tomography, presents recent advances at this multifaceted intersection of fields, and outlines the most exciting advances that will likely propagate into promising future innovations.

Perspective on fast-evolving photoacoustic tomography

SIGNIFICANCE

Acoustically detecting the rich optical absorption contrast in biological tissues, photoacoustic tomography (PAT) seamlessly bridges the functional and molecular sensitivity of optical excitation with the deep penetration and high scalability of ultrasound detection. As a result of continuous technological innovations and commercial development, PAT has been playing an increasingly important role in life sciences and patient care, including functional brain imaging, smart drug delivery, early cancer diagnosis, and interventional therapy guidance.

AIM

Built on our 2016 tutorial article that focused on the principles and implementations of PAT, this perspective aims to provide an update on the exciting technical advances in PAT.

APPROACH

This perspective focuses on the recent PAT innovations in volumetric deep-tissue imaging, high-speed wide-field microscopic imaging, high-sensitivity optical ultrasound detection, and machine-learning enhanced image reconstruction and data processing. Representative applications are introduced to demonstrate these enabling technical breakthroughs in biomedical research.

CONCLUSIONS

We conclude the perspective by discussing the future development of PAT technologies.

Diffuse reflectance and machine learning techniques to differentiate colorectal cancer ex vivo

In this study, we used machine learning techniques to reconstruct the wavelength dependence of the absorption coefficient of human normal and pathological colorectal mucosa tissues. Using only diffuse reflectance spectra from the ex vivo mucosa tissues as input to algorithms, several approaches were tried before obtaining good matching between the generated absorption coefficients and the ones previously calculated for the mucosa tissues from invasive experimental spectral measurements. Considering the optimized match for the results generated with the multilayer perceptron regression method, we were able to identify differentiated accumulation of lipofuscin in the absorption coefficient spectra of both mucosa tissues as we have done before with the corresponding results calculated directly from invasive measurements. Considering the random forest regressor algorithm, the estimated absorption coefficient spectra almost matched the ones previously calculated. By subtracting the absorption of lipofuscin from these spectra, we obtained similar hemoglobin ratios at 410/550 nm: 18.9-fold/9.3-fold for the healthy mucosa and 46.6-fold/24.2-fold for the pathological mucosa, while from direct calculations, those ratios were 19.7-fold/10.1-fold for the healthy mucosa and 33.1-fold/17.3-fold for the pathological mucosa. The higher values obtained in this study indicate a higher blood content in the pathological samples used to measure the diffuse reflectance spectra. In light of such accuracy and sensibility to the presence of hidden absorbers, with a different accumulation between healthy and pathological tissues, good perspectives become available to develop minimally invasive spectroscopy methods for in vivo early detection and monitoring of colorectal cancer.

Computational tissue staining of non-linear multimodal imaging using supervised and unsupervised deep learning

Hematoxylin and Eosin (H&E) staining is the 'gold-standard' method in histopathology. However, standard H&E staining of high-quality tissue sections requires long sample preparation times including sample embedding, which restricts its application for 'real-time' disease diagnosis. Due to this reason, a label-free alternative technique like non-linear multimodal (NLM) imaging, which is the combination of three non-linear optical modalities including coherent anti-Stokes Raman scattering, two-photon excitation fluorescence and second-harmonic generation, is proposed in this work. To correlate the information of the NLM images with H&E images, this work proposes computational staining of NLM images using deep learning models in a supervised and an unsupervised approach. In the supervised and the unsupervised approach, conditional generative adversarial networks (CGANs) and cycle conditional generative adversarial networks (cycle CGANs) are used, respectively. Both CGAN and cycle CGAN models generate pseudo H&E images, which are quantitatively analyzed based on mean squared error, structure similarity index and color shading similarity index. The mean of the three metrics calculated for the computationally generated H&E images indicate significant performance. Thus, utilizing CGAN and cycle CGAN models for computational staining is beneficial for diagnostic applications without performing a laboratory-based staining procedure. To the author's best knowledge, it is the first time that NLM images are computationally stained to H&E images using GANs in an unsupervised manner.

Disease-Relevant Single Cell Photonic Signatures Identify S100β Stem Cells and their Myogenic Progeny in Vascular Lesions

A hallmark of subclinical atherosclerosis is the accumulation of vascular smooth muscle cell (SMC)-like cells leading to intimal thickening and lesion formation. While medial SMCs contribute to vascular lesions, the involvement of resident vascular stem cells (vSCs) remains unclear. We evaluated single cell photonics as a discriminator of cell phenotype in vitro before the presence of vSC within vascular lesions was assessed ex vivo using supervised machine learning and further validated using lineage tracing analysis. Using a novel lab-on-a-Disk(Load) platform, label-free single cell photonic emissions from normal and injured vessels ex vivo were interrogated and compared to freshly isolated aortic SMCs, cultured Movas SMCs, macrophages, B-cells, S100β⁺ mVSc, bone marrow derived mesenchymal stem cells (MSC) and their respective myogenic progeny across five broadband light wavelengths (λ465 - λ670 ± 20 nm). We found that profiles were of sufficient coverage, specificity, and quality to clearly distinguish medial SMCs from different vascular beds (carotid vs aorta), discriminate normal carotid medial SMCs from lesional SMC-like cells ex vivo following flow restriction, and identify SMC differentiation of a series of multipotent stem cells following treatment with transforming growth factor beta 1 (TGF- β1), the Notch ligand Jagged1, and Sonic Hedgehog using multivariate analysis, in part, due to photonic emissions from enhanced collagen III and elastin expression. Supervised machine learning supported genetic lineage tracing analysis of S100β⁺ vSCs and identified the presence of S100β⁺vSC-derived myogenic progeny within vascular lesions. We conclude disease-relevant photonic signatures may have predictive value for vascular disease.

A representation learning approach for recovering scatter-corrected spectra from Fourier-transform infrared spectra of tissue samples

Infrared spectra obtained from cell or tissue specimen have commonly been observed to involve a significant degree of scattering effects, often Mie scattering, which probably overshadows biochemically relevant spectral information by a nonlinear, nonadditive spectral component in Fourier transform infrared (FTIR) spectroscopic measurements. Correspondingly, many successful machine learning approaches for FTIR spectra have relied on preprocessing procedures that computationally remove the scattering components from an infrared spectrum. We propose an approach to approximate this complex preprocessing function using deep neural networks. As we demonstrate, the resulting model is not just several orders of magnitudes faster, which is important for real-time clinical applications, but also generalizes strongly across different tissue types. Using Bayesian machine learning approaches, our approach unveils model uncertainty that coincides with a band shift in the amide I region that occurs when scattering is removed computationally based on an established physical model. Furthermore, our proposed method overcomes the trade-off between computation time and the corrected spectrum being biased towards an artificial reference spectrum.

Deep Learning for Biospectroscopy and Biospectral Imaging: State-of-the-Art and Perspectives

With the advances in instrumentation and sampling techniques, there is an explosive growth of data from molecular and cellular samples. The call to extract more information from the large data sets has greatly challenged the conventional chemometrics method. Deep learning, which utilizes very large data sets for finding hidden features therein and for making accurate predictions for a wide range of applications, has been applied in an unbelievable pace in biospectroscopy and biospectral imaging in the recent 3 years. In this Feature, we first introduce the background and basic knowledge of deep learning. We then focus on the emerging applications of deep learning in the data preprocessing, feature detection, and modeling of the biological samples for spectral analysis and spectroscopic imaging. Finally, we highlight the challenges and limitations in deep learning and the outlook for future directions.

LV-GAN: A deep learning approach for limited-view optoacoustic imaging based on hybrid datasets

The optoacoustic imaging (OAI) methods are rapidly evolving for resolving optical contrast in medical imaging applications. In practice, measurement strategies are commonly implemented under limited-view conditions due to oversized image objectives or system design limitations. Data acquired by limited-view detection may impart artifacts and distortions in reconstructed optoacoustic (OA) images. We propose a hybrid data-driven deep learning approach based on generative adversarial network (GAN), termed as LV-GAN, to efficiently recover high quality images from limited-view OA images. Trained on both simulation and experiment data, LV-GAN is found capable of achieving high recovery accuracy even under limited detection angles less than 60^° . The feasibility of LV-GAN for artifact removal in biological applications was validated by ex vivo experiments based on two different OAI systems, suggesting high potential of a ubiquitous use of LV-GAN to optimize image quality or system design for different scanners and application scenarios.

Toward rapid infectious disease diagnosis with advances in surface-enhanced Raman spectroscopy

In a pandemic era, rapid infectious disease diagnosis is essential. Surface-enhanced Raman spectroscopy (SERS) promises sensitive and specific diagnosis including rapid point-of-care detection and drug susceptibility testing. SERS utilizes inelastic light scattering arising from the interaction of incident photons with molecular vibrations, enhanced by orders of magnitude with resonant metallic or dielectric nanostructures. While SERS provides a spectral fingerprint of the sample, clinical translation is lagged due to challenges in consistency of spectral enhancement, complexity in spectral interpretation, insufficient specificity and sensitivity, and inefficient workflow from patient sample collection to spectral acquisition. Here, we highlight the recent, complementary advances that address these shortcomings, including (1) design of label-free SERS substrates and data processing algorithms that improve spectral signal and interpretability, essential for broad pathogen screening assays; (2) development of new capture and affinity agents, such as aptamers and polymers, critical for determining the presence or absence of particular pathogens; and (3) microfluidic and bioprinting platforms for efficient clinical sample processing. We also describe the development of low-cost, point-of-care, optical SERS hardware. Our paper focuses on SERS for viral and bacterial detection, in hopes of accelerating infectious disease diagnosis, monitoring, and vaccine development. With advances in SERS substrates, machine learning, and microfluidics and bioprinting, the specificity, sensitivity, and speed of SERS can be readily translated from laboratory bench to patient bedside, accelerating point-of-care diagnosis, personalized medicine, and precision health.

Another decade of photoacoustic imaging

Photoacoustic imaging - a hybrid biomedical imaging modality finding its way to clinical practices. Although the photoacoustic phenomenon was known more than a century back, only in the last two decades it has been widely researched and used for biomedical imaging applications. In this review we focus on the development and progress of the technology in the last decade (2010-2020). From becoming more and more user friendly, cheaper in cost, portable in size, photoacoustic imaging promises a wide range of applications, if translated to clinic. The growth of photoacoustic community is steady, and with several new directions researchers are exploring, it is inevitable that photoacoustic imaging will one day establish itself as a regular imaging system in the clinical practices.

SSNOMBACTER: A collection of scattering-type scanning near-field optical microscopy and atomic force microscopy images of bacterial cells

BACKGROUND

In recent years, a variety of imaging techniques operating at nanoscale resolution have been reported. These techniques have the potential to enrich our understanding of bacterial species relevant to human health, such as antibiotic-resistant pathogens. However, owing to the novelty of these techniques, their use is still confined to addressing very particular applications, and their availability is limited owing to associated costs and required expertise. Among these, scattering-type scanning near field optical microscopy (s-SNOM) has been demonstrated as a powerful tool for exploring important optical properties at nanoscale resolution, depending only on the size of a sharp tip. Despite its huge potential to resolve aspects that cannot be tackled otherwise, the penetration of s-SNOM into the life sciences is still proceeding at a slow pace for the aforementioned reasons.

RESULTS

In this work we introduce SSNOMBACTER, a set of s-SNOM images collected on 15 bacterial species. These come accompanied by registered Atomic Force Microscopy images, which are useful for placing nanoscale optical information in a relevant topographic context.

CONCLUSIONS

The proposed dataset aims to augment the popularity of s-SNOM and for accelerating its penetration in life sciences. Furthermore, we consider this dataset to be useful for the development and benchmarking of image analysis tools dedicated to s-SNOM imaging, which are scarce, despite the high need. In this latter context we discuss a series of image processing and analysis applications where SSNOMBACTER could be of help.