A versatile deep learning architecture for classification and label-free prediction of hyperspectral images

Viewing 3D spatial biology with highly-multiplexed Raman imaging: from spectroscopy to biotechnology

Understansding complex biological systems requires the simultaneous characterization of a large number of interacting components in their native 3D environment with high spatial resolution. Highly-multiplexed Raman imaging is an emerging general strategy for detecting biomarkers with scalable multiplexity and ultra-sensitivity based on a series of stimulated Raman scattering (SRS) techniques. Here we review recent advances in highly-multiplexed Raman imaging and how they contribute to the technological revolution in 3D spatial biology, focusing on the developmental pathway from spectroscopy study to biotechnology invention. We envision highly-multiplexed Raman imaging is taking off, which will greatly facilitate our understanding in biological and medical research fields.

Hyperspectral imaging and its applications: A review

Hyperspectral imaging has emerged as an effective powerful tool in plentiful military, environmental, and civil applications over the last three decades. The modern remote sensing approaches are adequate for covering huge earth surfaces with phenomenal temporal, spectral, and spatial resolutions. These features make HSI more effective in various applications of remote sensing depending upon the physical estimation of identical material identification and manifold composite surfaces having accomplished spectral resolutions. Recently, HSI has attained immense significance in the research on safety and quality assessment of food, medical analysis, and agriculture applications. This review focuses on HSI fundamentals and its applications like safety and quality assessment of food, medical analysis, agriculture, water resources, plant stress identification, weed & crop discrimination, and flood management. Various investigators have promising solutions for automatic systems depending upon HSI. Future research may use this review as a baseline and future advancement analysis.

Fluorescence excitation-scanning hyperspectral imaging with scalable 2D-3D deep learning framework for colorectal cancer detection

Colorectal cancer is one of the top contributors to cancer-related deaths in the United States, with over 100,000 estimated cases in 2020 and over 50,000 deaths. The most common screening technique is minimally invasive colonoscopy using either reflected white light endoscopy or narrow-band imaging. However, current imaging modalities have only moderate sensitivity and specificity for lesion detection. We have developed a novel fluorescence excitation-scanning hyperspectral imaging (HSI) approach to sample image and spectroscopic data simultaneously on microscope and endoscope platforms for enhanced diagnostic potential. Unfortunately, fluorescence excitation-scanning HSI datasets pose major challenges for data processing, interpretability, and classification due to their high dimensionality. Here, we present an end-to-end scalable Artificial Intelligence (AI) framework built for classification of excitation-scanning HSI microscopy data that provides accurate image classification and interpretability of the AI decision-making process. The developed AI framework is able to perform real-time HSI classification with different speed/classification performance trade-offs by tailoring the dimensionality of the dataset, supporting different dimensions of deep learning models, and varying the architecture of deep learning models. We have also incorporated tools to visualize the exact location of the lesion detected by the AI decision-making process and to provide heatmap-based pixel-by-pixel interpretability. In addition, our deep learning framework provides wavelength-dependent impact as a heatmap, which allows visualization of the contributions of HSI wavelength bands during the AI decision-making process. This framework is well-suited for HSI microscope and endoscope platforms, where real-time analysis and visualization of classification results are required by clinicians.

Recent Developments in Machine Learning for Mass Spectrometry

Statistical analysis and modeling of mass spectrometry (MS) data have a long and rich history with several modern MS-based applications using statistical and chemometric methods. Recently, machine learning (ML) has experienced a renaissance due to advents in computational hardware and the development of new algorithms for artificial neural networks (ANN) and deep learning architectures. Moreover, recent successes of new ANN and deep learning architectures in several areas of science, engineering, and society have further strengthened the ML field. Importantly, modern ML methods and architectures have enabled new approaches for tasks related to MS that are now widely adopted in several popular MS-based subdisciplines, such as mass spectrometry imaging and proteomics. Herein, we aim to provide an introductory summary of the practical aspects of ML methodology relevant to MS. Additionally, we seek to provide an up-to-date review of the most recent developments in ML integration with MS-based techniques while also providing critical insights into the future direction of the field.

Discriminating Spectral-Spatial Feature Extraction for Hyperspectral Image Classification: A Review

Hyperspectral images (HSIs) contain subtle spectral details and rich spatial contextures of land cover that benefit from developments in spectral imaging and space technology. The classification of HSIs, which aims to allocate an optimal label for each pixel, has broad prospects in the field of remote sensing. However, due to the redundancy between bands and complex spatial structures, the effectiveness of the shallow spectral-spatial features extracted by traditional machine-learning-based methods tends to be unsatisfying. Over recent decades, various methods based on deep learning in the field of computer vision have been proposed to allow for the discrimination of spectral-spatial representations for classification. In this article, the crucial factors to discriminate spectral-spatial features are systematically summarized from the perspectives of feature extraction and feature optimization. For feature extraction, techniques to ensure the discrimination of spectral features, spatial features, and spectral-spatial features are illustrated based on the characteristics of hyperspectral data and the architecture of models. For feature optimization, techniques to adjust the feature distances between classes in the classification space are introduced in detail. Finally, the characteristics and limitations of these techniques and future challenges in facilitating the discrimination of features for HSI classification are also discussed further.

Improving prediction of maternal health risks using PCA features and TreeNet model

Maternal healthcare is a critical aspect of public health that focuses on the well-being of pregnant women before, during, and after childbirth. It encompasses a range of services aimed at ensuring the optimal health of both the mother and the developing fetus. During pregnancy and in the postpartum period, the mother's health is susceptible to several complications and risks, and timely detection of such risks can play a vital role in women's safety. This study proposes an approach to predict risks associated with maternal health. The first step of the approach involves utilizing principal component analysis (PCA) to extract significant features from the dataset. Following that, this study employs a stacked ensemble voting classifier which combines one machine learning and one deep learning model to achieve high performance. The performance of the proposed approach is compared to six machine learning algorithms and one deep learning algorithm. Two scenarios are considered for the experiments: one utilizing all features and the other using PCA features. By utilizing PCA-based features, the proposed model achieves an accuracy of 98.25%, precision of 99.17%, recall of 99.16%, and an F1 score of 99.16%. The effectiveness of the proposed model is further confirmed by comparing it to existing state of-the-art approaches.

Inversion of heavy metal content in soil using hyperspectral characteristic bands-based machine learning method

The global concern regarding the adverse effects of heavy metal pollution in soil has grown significantly. Accurate prediction of heavy metal content in soil is crucial for environmental protection. This study proposes an inversion analysis method for heavy metals (As, Cd, Cr, Cu, Ni, Pb) in soil based on hyperspectral and machine learning algorithms for 21 soil reference materials from multiple provinces in China. On this basis, an integrated learning model called Stacked RF (the base model is XGBoost, LightGBM, CatBoost, and the meta-model is RF) was established to perform soil heavy metal inversion. Specifically, three popular algorithms were initially employed to preprocess the spectral data, then Random Forest (RF) was used to select the best feature bands to reduce the impact of noise, finally Stacking and four basic machine learning algorithms were used to establish comparisons and analysis of inversion model. Compared with traditional machine learning methods, the stacking model showcases enhanced stability and superior accuracy. Research results indicate that machine learning algorithms, especially ensemble learning models, have better inversion effects on heavy metals in soil. Overall, the MF-RF-Stacking model performed best in the inversion of the six heavy metals. The research results will provide a new perspective on the ensemble learning model method for soil heavy metal content inversion using data of hyperspectral characteristic bands collected from soil reference materials.

Noise learning of instruments for high-contrast, high-resolution and fast hyperspectral microscopy and nanoscopy

The low scattering efficiency of Raman scattering makes it challenging to simultaneously achieve good signal-to-noise ratio (SNR), high imaging speed, and adequate spatial and spectral resolutions. Here, we report a noise learning (NL) approach that estimates the intrinsic noise distribution of each instrument by statistically learning the noise in the pixel-spatial frequency domain. The estimated noise is then removed from the noisy spectra. This enhances the SNR by ca. 10 folds, and suppresses the mean-square error by almost 150 folds. NL allows us to improve the positioning accuracy and spatial resolution and largely eliminates the impact of thermal drift on tip-enhanced Raman spectroscopic nanoimaging. NL is also applicable to enhance SNR in fluorescence and photoluminescence imaging. Our method manages the ground truth spectra and the instrumental noise simultaneously within the training dataset, which bypasses the tedious labelling of huge dataset required in conventional deep learning, potentially shifting deep learning from sample-dependent to instrument-dependent.

Rapid single-particle chemical imaging of nanoplastics by SRS microscopy

Plastics are now omnipresent in our daily lives. The existence of microplastics (1 µm to 5 mm in length) and possibly even nanoplastics (<1 μm) has recently raised health concerns. In particular, nanoplastics are believed to be more toxic since their smaller size renders them much more amenable, compared to microplastics, to enter the human body. However, detecting nanoplastics imposes tremendous analytical challenges on both the nano-level sensitivity and the plastic-identifying specificity, leading to a knowledge gap in this mysterious nanoworld surrounding us. To address these challenges, we developed a hyperspectral stimulated Raman scattering (SRS) imaging platform with an automated plastic identification algorithm that allows micro-nano plastic analysis at the single-particle level with high chemical specificity and throughput. We first validated the sensitivity enhancement of the narrow band of SRS to enable high-speed single nanoplastic detection below 100 nm. We then devised a data-driven spectral matching algorithm to address spectral identification challenges imposed by sensitive narrow-band hyperspectral imaging and achieve robust determination of common plastic polymers. With the established technique, we studied the micro-nano plastics from bottled water as a model system. We successfully detected and identified nanoplastics from major plastic types. Micro-nano plastics concentrations were estimated to be about 2.4 ± 1.3 × 105 particles per liter of bottled water, about 90% of which are nanoplastics. This is orders of magnitude more than the microplastic abundance reported previously in bottled water. High-throughput single-particle counting revealed extraordinary particle heterogeneity and nonorthogonality between plastic composition and morphologies; the resulting multidimensional profiling sheds light on the science of nanoplastics.

Deep learning assisted classification of spectral photoacoustic imaging of carotid plaques

Spectral photoacoustic imaging (sPAI) is an emerging modality that allows real-time, non-invasive, and radiation-free assessment of tissue, benefiting from their optical contrast. sPAI is ideal for morphology assessment in arterial plaques, where plaque composition provides relevant information on plaque progression and its vulnerability. However, since sPAI is affected by spectral coloring, general spectroscopy unmixing techniques cannot provide reliable identification of such complicated sample composition. In this study, we employ a convolutional neural network (CNN) for the classification of plaque composition using sPAI. For this study, nine carotid endarterectomy plaques were imaged and were then annotated and validated using multiple histological staining. Our results show that a CNN can effectively differentiate constituent regions within plaques without requiring fluence or spectra correction, with the potential to eventually support vulnerability assessment in plaques.

AutoUnmix: an autoencoder-based spectral unmixing method for multi-color fluorescence microscopy imaging

Multiplexed fluorescence microscopy imaging is widely used in biomedical applications. However, simultaneous imaging of multiple fluorophores can result in spectral leaks and overlapping, which greatly degrades image quality and subsequent analysis. Existing popular spectral unmixing methods are mainly based on computational intensive linear models, and the performance is heavily dependent on the reference spectra, which may greatly preclude its further applications. In this paper, we propose a deep learning-based blindly spectral unmixing method, termed AutoUnmix, to imitate the physical spectral mixing process. A transfer learning framework is further devised to allow our AutoUnmix to adapt to a variety of imaging systems without retraining the network. Our proposed method has demonstrated real-time unmixing capabilities, surpassing existing methods by up to 100-fold in terms of unmixing speed. We further validate the reconstruction performance on both synthetic datasets and biological samples. The unmixing results of AutoUnmix achieve the highest SSIM of 0.99 in both three- and four-color imaging, with nearly up to 20% higher than other popular unmixing methods. For experiments where spectral profiles and morphology are akin to simulated data, our method realizes the quantitative performance demonstrated above. Due to the desirable property of data independency and superior blind unmixing performance, we believe AutoUnmix is a powerful tool for studying the interaction process of different organelles labeled by multiple fluorophores.

Weakly Supervised Identification and Localization of Drug Fingerprints Based on Label-Free Hyperspectral CARS Microscopy

Understanding drug fingerprints in complex biological samples is essential for the development of a drug. Hyperspectral coherent anti-Stokes Raman scattering (HS-CARS) microscopy, a label-free nondestructive chemical imaging technique, can profile biological samples based on their endogenous vibrational contrast. Here, we propose a deep learning-assisted HS-CARS imaging approach for the investigation of drug fingerprints and their localization at single-cell resolution. To identify and localize drug fingerprints in complex biological systems, an attention-based deep neural network, hyperspectral attention net (HAN), was developed. By formulating the task to a multiple instance learning problem, HAN highlights informative regions through the attention mechanism when being trained on whole-image labels. Using the proposed technique, we investigated the drug fingerprints of a hepatitis B virus therapy in murine liver tissues. With the increase in drug dosage, higher classification accuracy was observed, with an average area under the curve (AUC) of 0.942 for the high-dose group. Besides, highly informative tissue structures predicted by HAN demonstrated a high degree of similarity with the drug localization shown by the in situ hybridization staining results. These results demonstrate the potential of the proposed deep learning-assisted optical imaging technique for the label-free profiling, identification, and localization of drug fingerprints in biological samples, which can be extended to nonperturbative investigations of complex biological systems under various biological conditions.

Metasurface-Enhanced Mid-Infrared Spectrochemical Imaging of Tissues

Label-free and nondestructive mid-infrared vibrational hyperspectral imaging is an essential tissue analysis tool, providing spatially resolved biochemical information critical to understanding physiological and pathological processes. However, the chemically complex and spatially heterogeneous composition of tissue specimens and the inherently weak interaction of infrared light with biomolecules limit the analytical performance of infrared absorption spectroscopy. Here, an advanced mid-infrared spectrochemical tissue imaging modality is introduced using metasurfaces that support strong surface-localized electromagnetic fields to capture quantitative molecular maps of large-area murine brain tissue sections. The approach leverages polarization-multiplexed multi-resonance plasmonic metasurfaces to simultaneously detect various functional biomolecules. The surface-enhanced mid-infrared spectral imaging method eliminates the non-specific effects of bulk tissue morphology on quantitative spectral analysis and improves chemical selectivity. This study shows that metasurface enhancement increases the retrieval of amide I and II bands associated with protein secondary structures. Moreover, it is demonstrated that plasmonic metasurfaces enhance the chemical contrast in infrared images and enable detection of ultrathin tissue regions that are not otherwise visible to conventional mid-infrared spectral imaging. While this work uses murine brain tissue sections, the chemical imaging method is well-suited for other tissue types, which broadens its potential impact for translational research and clinical histopathology.

Off-flavor profiling of cultured salmonids using hyperspectral imaging combined with machine learning

Off-flavors can have significant impacts on the quality of salmonid products. This study investigated the possibility of comprehensive off-flavor profiling considering both olfactory and taste sensory perspectives by combining near-infrared hyperspectral imaging (NIR-HSI) and machine/deep learning. Four feature extraction algorithms were employed for the extraction and interpretation of spectral fingerprint information regarding off-flavor-related compounds. Classification models, including the partial least squares discriminant analysis, least-squares support vector machine, extreme learning machine, and one-dimensional convolutional neural network (1DCNN) were constructed using the full wavelengths and selected spectral features for the identification of off-flavor salmonids. The 1DCNN achieved the highest discrimination accuracy with full and selected wavelengths (i.e., 91.11 and 86.39 %, respectively). Furthermore, the prediction and visualization of off-flavor-related compounds were achieved with acceptable performances (R² > 0.6) for practical applications. These results indicate the potential of NIR-HSI for the off-flavor profiling of salmonid muscle samples for producers and researchers.

Label-Free Cytometric Evaluation of Mitosis via Stimulated Raman Scattering Microscopy and Spectral Phasor Analysis

Hyperspectral stimulated Raman scattering (SRS) microscopy is a robust imaging tool for the analysis of biological systems. Here, we present a unique perspective, a label-free spatiotemporal map of mitosis, by integrating hyperspectral SRS microscopy with advanced chemometrics to assess the intrinsic biomolecular properties of an essential process of mammalian life. The application of spectral phasor analysis to multiwavelength SRS images in the high-wavenumber (HWN) region of the Raman spectrum enabled the segmentation of subcellular organelles based on innate SRS spectra. Traditional imaging of DNA is primarily reliant on using fluorescent probes or stains which can affect the biophysical properties of the cell. Here, we demonstrate the label-free visualization of nuclear dynamics during mitosis coupled with an evaluation of its spectral profile in a rapid and reproducible manner. These results provide a snapshot of the cell division cycle and chemical variability between intracellular compartments in single-cell models, which is central to understanding the molecular foundations of these fundamental biological processes. The evaluation of HWN images by phasor analysis also facilitated the differentiation between cells in separate phases of the cell cycle based solely on their nuclear SRS spectral signal, which offers an interesting label-free approach in combination with flow cytometry. Therefore, this study demonstrates that SRS microscopy combined with spectral phasor analysis is a valuable method for detailed optical fingerprinting at the subcellular level.

Identification of geographical origins of Radix Paeoniae Alba using hyperspectral imaging with deep learning-based fusion approaches

The Radix Paeoniae Alba (Baishao) is a traditional Chinese medicine (TCM) with numerous clinical and nutritional benefits. Rapid and accurate identification of the geographical origins of Baishao is crucial for planters, traders and consumers. Hyperspectral imaging (HSI) was used in this study to acquire spectral images of Baishao samples from its two sides. Convolutional neural network (CNN) and attention mechanism was used to distinguish the origins of Baishao using spectra extracted from one side. The data-level and feature-level deep fusion models were proposed using information from both sides of the samples. CNN models outperformed the conventional machine learning methods in classifying Baishao origins. The generalized Gradient-weighted Class Activation Mapping (Grad-CAM++) was utilized to visualize and identify important wavelengths that significantly contribute to model performance. The overall results illustrated that HSI combined with deep learning strategies was effective in identifying the geographical origins of Baishao, having good prospects of real-world applications.

mHealth hyperspectral learning for instantaneous spatiospectral imaging of hemodynamics

Hyperspectral imaging acquires data in both the spatial and frequency domains to offer abundant physical or biological information. However, conventional hyperspectral imaging has intrinsic limitations of bulky instruments, slow data acquisition rate, and spatiospectral trade-off. Here we introduce hyperspectral learning for snapshot hyperspectral imaging in which sampled hyperspectral data in a small subarea are incorporated into a learning algorithm to recover the hypercube. Hyperspectral learning exploits the idea that a photograph is more than merely a picture and contains detailed spectral information. A small sampling of hyperspectral data enables spectrally informed learning to recover a hypercube from a red-green-blue (RGB) image without complete hyperspectral measurements. Hyperspectral learning is capable of recovering full spectroscopic resolution in the hypercube, comparable to high spectral resolutions of scientific spectrometers. Hyperspectral learning also enables ultrafast dynamic imaging, leveraging ultraslow video recording in an off-the-shelf smartphone, given that a video comprises a time series of multiple RGB images. To demonstrate its versatility, an experimental model of vascular development is used to extract hemodynamic parameters via statistical and deep learning approaches. Subsequently, the hemodynamics of peripheral microcirculation is assessed at an ultrafast temporal resolution up to a millisecond, using a conventional smartphone camera. This spectrally informed learning method is analogous to compressed sensing; however, it further allows for reliable hypercube recovery and key feature extractions with a transparent learning algorithm. This learning-powered snapshot hyperspectral imaging method yields high spectral and temporal resolutions and eliminates the spatiospectral trade-off, offering simple hardware requirements and potential applications of various machine learning techniques.

Small molecule metabolites: discovery of biomarkers and therapeutic targets

Metabolic abnormalities lead to the dysfunction of metabolic pathways and metabolite accumulation or deficiency which is well-recognized hallmarks of diseases. Metabolite signatures that have close proximity to subject's phenotypic informative dimension, are useful for predicting diagnosis and prognosis of diseases as well as monitoring treatments. The lack of early biomarkers could lead to poor diagnosis and serious outcomes. Therefore, noninvasive diagnosis and monitoring methods with high specificity and selectivity are desperately needed. Small molecule metabolites-based metabolomics has become a specialized tool for metabolic biomarker and pathway analysis, for revealing possible mechanisms of human various diseases and deciphering therapeutic potentials. It could help identify functional biomarkers related to phenotypic variation and delineate biochemical pathways changes as early indicators of pathological dysfunction and damage prior to disease development. Recently, scientists have established a large number of metabolic profiles to reveal the underlying mechanisms and metabolic networks for therapeutic target exploration in biomedicine. This review summarized the metabolic analysis on the potential value of small-molecule candidate metabolites as biomarkers with clinical events, which may lead to better diagnosis, prognosis, drug screening and treatment. We also discuss challenges that need to be addressed to fuel the next wave of breakthroughs.

Simultaneous Dual-Band Hyperspectral Stimulated Raman Scattering Microscopy with Femtosecond Optical Parametric Oscillators

Stimulated Raman scattering (SRS) microscopy is a label-free quantitative optical technique for imaging molecular distributions in cells and tissues by probing their intrinsic vibrational frequencies. Despite its usefulness, existing SRS imaging techniques have limited spectral coverage due to either a wavelength tuning constraint or narrow spectral bandwidth. High-wavenumber SRS imaging is commonly used to map lipid and protein distribution in biological cells and visualize cell morphology. However, to detect small molecules or Raman tags, imaging in the fingerprint region or "silent" region, respectively, is often required. For many applications, it is desirable to collect SRS images in two Raman spectral regions simultaneously for visualizing the distribution of specific molecules in cellular compartments or providing accurate ratiometric analysis. In this work, we present an SRS microscopy system using three beams generated by a femtosecond oscillator to acquire hyperspectral SRS image stacks in two arbitrary vibrational frequency bands, between 650-3280 cm-1, simultaneously. We demonstrate potential biomedical applications of the system in investigating fatty acid metabolism, cellular drug uptake and accumulation, and lipid unsaturation level in tissues. We also show that the dual-band hyperspectral SRS imaging system can be adapted for the broadband fingerprint region hyperspectral imaging (1100-1800 cm-1) by simply adding a modulator.

Perovskite single-detector visible-light spectrometer

We demonstrate a perovskite single-phototransistor visible-light spectrometer based on a deep-learning method. The size of the spectrometer is set to the scale of the phototransistor. A photoresponsivity matrix for the deep-learning system is learned from the characteristic parameters of the visible-light wavelength, gate voltage, and power densities of a commercial standard blackbody source. Unknown spectra are reconstructed using the corresponding photocurrent vectors. As a confirmatory experiment, a 532-nm laser and multipeak broadband spectrum are successfully reconstructed using our perovskite single-phototransistor spectrometer. The resolution is improved to 1 nm by increasing the number of sampling points from 80 to 400. In addition, a way to further improve the resolution is provided by increasing the number of sampling points, characteristic parameters, and training datasets. Furthermore, artificial intelligence technology may open pathways for on-chip visible-light spectroscopy.

Divide and Conquer: A Flexible Deep Learning Strategy for Exploring Metabolic Heterogeneity from Mass Spectrometry Imaging Data

Research on metabolic heterogeneity provides an important basis for the study of the molecular mechanism of a disease and personalized treatment. The screening of metabolism-related sub-regions that affect disease development is essential for the more focused exploration on disease progress aberrant phenotypes, even carcinogenesis and metastasis. The mass spectrometry imaging (MSI) technique has distinct advantages to reveal the heterogeneity of an organism based on in situ molecular profiles. The challenge of heterogeneous analysis has been to perform an objective identification among biological tissues with different characteristics. By introducing the divide-and-conquer strategy to architecture design and application, we establish here a flexible unsupervised deep learning model, called divide-and-conquer (dc)-DeepMSI, for metabolic heterogeneity analysis from MSI data without prior knowledge of histology. dc-DeepMSI can be used to identify either spatially contiguous regions of interest (ROIs) or spatially sporadic ROIs by designing two specific modes, spat-contig and spat-spor. Comparison results on fetus mouse data demonstrate that the dc-DeepMSI outperforms state-of-the-art MSI segmentation methods. We demonstrate that the novel learning strategy successfully obtained sub-regions that are statistically linked to the invasion status and molecular phenotypes of breast cancer as well as organizing principles during developmental phase.

Deep Learning in Medical Hyperspectral Images: A Review

With the continuous progress of development, deep learning has made good progress in the analysis and recognition of images, which has also triggered some researchers to explore the area of combining deep learning with hyperspectral medical images and achieve some progress. This paper introduces the principles and techniques of hyperspectral imaging systems, summarizes the common medical hyperspectral imaging systems, and summarizes the progress of some emerging spectral imaging systems through analyzing the literature. In particular, this article introduces the more frequently used medical hyperspectral images and the pre-processing techniques of the spectra, and in other sections, it discusses the main developments of medical hyperspectral combined with deep learning for disease diagnosis. On the basis of the previous review, tne limited factors in the study on the application of deep learning to hyperspectral medical images are outlined, promising research directions are summarized, and the future research prospects are provided for subsequent scholars.

Unsupervised segmentation of biomedical hyperspectral image data: tackling high dimensionality with convolutional autoencoders

Information about the structure and composition of biopsy specimens can assist in disease monitoring and diagnosis. In principle, this can be acquired from Raman and infrared (IR) hyperspectral images (HSIs) that encode information about how a sample's constituent molecules are arranged in space. Each tissue section/component is defined by a unique combination of spatial and spectral features, but given the high dimensionality of HSI datasets, extracting and utilising them to segment images is non-trivial. Here, we show how networks based on deep convolutional autoencoders (CAEs) can perform this task in an end-to-end fashion by first detecting and compressing relevant features from patches of the HSI into low-dimensional latent vectors, and then performing a clustering step that groups patches containing similar spatio-spectral features together. We showcase the advantages of using this end-to-end spatio-spectral segmentation approach compared to i) the same spatio-spectral technique not trained in an end-to-end manner, and ii) a method that only utilises spectral features (spectral k-means) using simulated HSIs of porcine tissue as test examples. Secondly, we describe the potential advantages/limitations of using three different CAE architectures: a generic 2D CAE, a generic 3D CAE, and a 2D convolutional encoder-decoder architecture inspired by the recently proposed UwU-net that is specialised for extracting features from HSI data. We assess their performance on IR HSIs of real colon samples. We find that all architectures are capable of producing segmentations that show good correspondence with HE stained adjacent tissue slices used as approximate ground truths, indicating the robustness of the CAE-driven spatio-spectral clustering approach for segmenting biomedical HSI data. Additionally, we stress the need for more accurate ground truth information to enable a precise comparison of the advantages offered by each architecture.

Ensemble learning-based feature engineering to analyze maternal health during pregnancy and health risk prediction

Maternal health is an important aspect of women’s health during pregnancy, childbirth, and the postpartum period. Specifically, during pregnancy, different health factors like age, blood disorders, heart rate, etc. can lead to pregnancy complications. Detecting such health factors can alleviate the risk of pregnancy-related complications. This study aims to develop an artificial neural network-based system for predicting maternal health risks using health data records. A novel deep neural network architecture, DT-BiLTCN is proposed that uses decision trees, a bidirectional long short-term memory network, and a temporal convolutional network. Experiments involve using a dataset of 1218 samples collected from maternal health care, hospitals, and community clinics using the IoT-based risk monitoring system. Class imbalance is resolved using the synthetic minority oversampling technique. DT-BiLTCN provides a feature set to obtain high accuracy results which in this case are provided by the support vector machine with a 98% accuracy. Maternal health exploratory data analysis reveals that the health conditions which are the strongest indications of health risk during pregnancy are diastolic and systolic blood pressure, heart rate, and age of pregnant women. Using the proposed model, timely prediction of health risks associated with pregnant women can be made thus mitigating the risk of health complications which helps to save lives.

Visualization of a Limonene Synthesis Metabolon Inside Living Bacteria by Hyperspectral SRS Microscopy

Monitoring biosynthesis activity at single-cell level is key to metabolic engineering but is still difficult to achieve in a label-free manner. Using hyperspectral stimulated Raman scattering imaging in the 670-900 cm^-1 region, localized limonene synthesis are visualized inside engineered Escherichia coli. The colocalization of limonene and GFP-fused limonene synthase is confirmed by co-registered stimulated Raman scattering and two-photon fluorescence images. The finding suggests a limonene synthesis metabolon with a polar distribution inside the cells. This finding expands the knowledge of de novo limonene biosynthesis in engineered bacteria and highlights the potential of SRS chemical imaging in metabolic engineering research.

Pixel-level multimodal fusion deep networks for predicting subcellular organelle localization from label-free live-cell imaging

Complex intracellular organizations are commonly represented by dividing the metabolic process of cells into different organelles. Therefore, identifying sub-cellular organelle architecture is significant for understanding intracellular structural properties, specific functions, and biological processes in cells. However, the discrimination of these structures in the natural organizational environment and their functional consequences are not clear. In this article, we propose a new pixel-level multimodal fusion (PLMF) deep network which can be used to predict the location of cellular organelle using label-free cell optical microscopy images followed by deep-learning-based automated image denoising. It provides valuable insights that can be of tremendous help in improving the specificity of label-free cell optical microscopy by using the Transformer-Unet network to predict the ground truth imaging which corresponds to different sub-cellular organelle architectures. The new prediction method proposed in this article combines the advantages of a transformer's global prediction and CNN's local detail analytic ability of background features for label-free cell optical microscopy images, so as to improve the prediction accuracy. Our experimental results showed that the PLMF network can achieve over 0.91 Pearson's correlation coefficient (PCC) correlation between estimated and true fractions on lung cancer cell-imaging datasets. In addition, we applied the PLMF network method on the cell images for label-free prediction of several different subcellular components simultaneously, rather than using several fluorescent labels. These results open up a new way for the time-resolved study of subcellular components in different cells, especially for cancer cells.

Multiple Parallel Fusion Network for Predicting Protein Subcellular Localization from Stimulated Raman Scattering (SRS) Microscopy Images in Living Cells

Stimulated Raman Scattering Microscopy (SRS) is a powerful tool for label-free detailed recognition and investigation of the cellular and subcellular structures of living cells. Determining subcellular protein localization from the cell level of SRS images is one of the basic goals of cell biology, which can not only provide useful clues for their functions and biological processes but also help to determine the priority and select the appropriate target for drug development. However, the bottleneck in predicting subcellular protein locations of SRS cell imaging lies in modeling complicated relationships concealed beneath the original cell imaging data owing to the spectral overlap information from different protein molecules. In this work, a multiple parallel fusion network, MPFnetwork, is proposed to study the subcellular locations from SRS images. This model used a multiple parallel fusion model to construct feature representations and combined multiple nonlinear decomposing algorithms as the automated subcellular detection method. Our experimental results showed that the MPFnetwork could achieve over 0.93 dice correlation between estimated and true fractions on SRS lung cancer cell datasets. In addition, we applied the MPFnetwork method to cell images for label-free prediction of several different subcellular components simultaneously, rather than using several fluorescent labels. These results open up a new method for the time-resolved study of subcellular components in different cells, especially cancer cells.

Deep neural networks with controlled variable selection for the identification of putative causal genetic variants

Deep neural networks (DNNs) have been successfully utilized in many scientific problems for their high prediction accuracy, but their application to genetic studies remains challenging due to their poor interpretability. Here we consider the problem of scalable, robust variable selection in DNNs for the identification of putative causal genetic variants in genome sequencing studies. We identified a pronounced randomness in feature selection in DNNs due to its stochastic nature, which may hinder interpretability and give rise to misleading results. We propose an interpretable neural network model, stabilized using ensembling, with controlled variable selection for genetic studies. The merit of the proposed method includes: flexible modelling of the nonlinear effect of genetic variants to improve statistical power; multiple knockoffs in the input layer to rigorously control the false discovery rate; hierarchical layers to substantially reduce the number of weight parameters and activations, and improve computational efficiency; and stabilized feature selection to reduce the randomness in identified signals. We evaluate the proposed method in extensive simulation studies and apply it to the analysis of Alzheimer's disease genetics. We show that the proposed method, when compared with conventional linear and nonlinear methods, can lead to substantially more discoveries.

Bringing Vibrational Imaging to Chemical Biology with Molecular Probes

As an emerging optical imaging modality, stimulated Raman scattering (SRS) microscopy provides invaluable opportunities for chemical biology studies using its rich chemical information. Through rapid progress over the past decade, the development of Raman probes harnessing the chemical biology toolbox has proven to play a key role in advancing SRS microscopy and expanding biological applications. In this perspective, we first discuss the development of biorthogonal SRS imaging using small tagging of triple bonds or isotopes and highlight their unique advantages for metabolic pathway analysis and microbiology investigations. Potential opportunities for chemical biology studies integrating small tagging with SRS imaging are also proposed. We next summarize the current designs of highly sensitive and super-multiplexed SRS probes, as well as provide future directions and considerations for next-generation functional probe design. These rationally designed SRS probes are envisioned to bridge the gap between SRS microscopy and chemical biology research and should benefit their mutual development.

Instant diagnosis of gastroscopic biopsy via deep-learned single-shot femtosecond stimulated Raman histology

Gastroscopic biopsy provides the only effective method for gastric cancer diagnosis, but the gold standard histopathology is time-consuming and incompatible with gastroscopy. Conventional stimulated Raman scattering (SRS) microscopy has shown promise in label-free diagnosis on human tissues, yet it requires the tuning of picosecond lasers to achieve chemical specificity at the cost of time and complexity. Here, we demonstrate that single-shot femtosecond SRS (femto-SRS) reaches the maximum speed and sensitivity with preserved chemical resolution by integrating with U-Net. Fresh gastroscopic biopsy is imaged in <60 s, revealing essential histoarchitectural hallmarks perfectly agreed with standard histopathology. Moreover, a diagnostic neural network (CNN) is constructed based on images from 279 patients that predicts gastric cancer with accuracy >96%. We further demonstrate semantic segmentation of intratumor heterogeneity and evaluation of resection margins of endoscopic submucosal dissection (ESD) tissues to simulate rapid and automated intraoperative diagnosis. Our method holds potential for synchronizing gastroscopy and histopathological diagnosis.

Diagnosis of gastric cancer currently requires gastroscopic biopsy, which requires time and expertize to perform. Here, the authors demonstrate a femto-SRS imaging method which showed high accuracy in diagnosing gastric cancer without the need for pathologistbased diagnosis.

Quantitative Stimulated Raman Scattering Microscopy: Promises and Pitfalls

Since its first demonstration, stimulated Raman scattering (SRS) microscopy has become a powerful chemical imaging tool that shows promise in numerous biological and biomedical applications. The spectroscopic capability of SRS enables identification and tracking of specific molecules or classes of molecules, often without labeling. SRS microscopy also has the hallmark advantage of signal strength that is directly proportional to molecular concentration, allowing for in situ quantitative analysis of chemical composition of heterogeneous samples with submicron spatial resolution and subminute temporal resolution. However, it is important to recognize that quantification through SRS microscopy requires assumptions regarding both system and sample. Such assumptions are often taken axiomatically, which may lead to erroneous conclusions without proper validation. In this review, we focus on the tacitly accepted, yet complex, quantitative aspect of SRS microscopy. We discuss the various approaches to quantitative analysis, examples of such approaches, challenges in different systems, and potential solutions. Through our examination of published literature, we conclude that a scrupulous approach to experimental design can further expand the powerful and incisive quantitative capabilities of SRS microscopy.

Live-cell fluorescence spectral imaging as a data science challenge

Live-cell fluorescence spectral imaging is an evolving modality of microscopy that uses specific properties of fluorophores, such as excitation or emission spectra, to detect multiple molecules and structures in intact cells. The main challenge of analyzing live-cell fluorescence spectral imaging data is the precise quantification of fluorescent molecules despite the weak signals and high noise found when imaging living cells under non-phototoxic conditions. Beyond the optimization of fluorophores and microscopy setups, quantifying multiple fluorophores requires algorithms that separate or unmix the contributions of the numerous fluorescent signals recorded at the single pixel level. This review aims to provide both the experimental scientist and the data analyst with a straightforward description of the evolution of spectral unmixing algorithms for fluorescence live-cell imaging. We show how the initial systems of linear equations used to determine the concentration of fluorophores in a pixel progressively evolved into matrix factorization, clustering, and deep learning approaches. We outline potential future trends on combining fluorescence spectral imaging with label-free detection methods, fluorescence lifetime imaging, and deep learning image analysis.

In vivo simultaneous nonlinear absorption Raman and fluorescence (SNARF) imaging of mouse brain cortical structures

Label-free multiphoton microscopy is a powerful platform for biomedical imaging. Recent advancements have demonstrated the capabilities of transient absorption microscopy (TAM) for label-free quantification of hemoglobin and stimulated Raman scattering (SRS) microscopy for pathological assessment of label-free virtual histochemical staining. We propose the combination of TAM and SRS with two-photon excited fluorescence (TPEF) to characterize, quantify, and compare hemodynamics, vessel structure, cell density, and cell identity in vivo between age groups. In this study, we construct a simultaneous nonlinear absorption, Raman, and fluorescence (SNARF) microscope with the highest reported in vivo imaging depth for SRS and TAM at 250–280 μm to enable these multimodal measurements. Using machine learning, we predict capillary-lining cell identities with 90% accuracy based on nuclear morphology and capillary relationship. The microscope and methodology outlined herein provides an exciting route to study several research topics, including neurovascular coupling, blood-brain barrier, and neurodegenerative diseases.

In this study a microscope is constructed that carries out simultaneous nonlinear absorption, Raman, and fluorescence (SNARF). Machine learning is then used to predict capillary-lining cell identities with 90% accuracy based on nuclear morphology and capillary relationship, which in combination with the developed microscope, can provide a means to study several fields such as neurovascular coupling.

Quantitative Chemical Imaging of Bone Tissue for Intraoperative and Diagnostic Applications

Bone is difficult to image using traditional histopathological methods, leading to challenges in intraoperative pathological evaluation that is critical in guiding surgical treatment, particularly in orthopedic oncology. In this study, we demonstrate that a multimodal quantitative imaging approach that combines stimulated Raman scattering (SRS) microscopy, two-photon fluorescence (TPF) microscopy, and second-harmonic generation (SHG) microscopy can provide useful diagnostic information regarding intact bone tissue fragments from surgical excision or biopsy specimens. We imaged bone samples from 17 patient cases and performed quantitative chemical and morphological analyses of both mineral and organic components of bone. Our main findings show that carbonate content combined with morphometric analysis of bone organic matrix can separate several major classes of bone cancer-associated diagnostic categories with an average accuracy of 92%. This proof-of-principle study demonstrates that quantitative multimodal imaging and machine learning-based analysis of bony tissue can provide crucial diagnostic information for guiding clinical decisions in orthopedic oncology. Moreover, the general methodology of morphological and chemical imaging combined with machine learning can be readily extended to other tissue types for tissue diagnosis in intraoperative and other clinical settings.

Positive-unlabeled learning in bioinformatics and computational biology: a brief review

Conventional supervised binary classification algorithms have been widely applied to address significant research questions using biological and biomedical data. This classification scheme requires two fully labeled classes of data (e.g. positive and negative samples) to train a classification model. However, in many bioinformatics applications, labeling data is laborious, and the negative samples might be potentially mislabeled due to the limited sensitivity of the experimental equipment. The positive unlabeled (PU) learning scheme was therefore proposed to enable the classifier to learn directly from limited positive samples and a large number of unlabeled samples (i.e. a mixture of positive or negative samples). To date, several PU learning algorithms have been developed to address various biological questions, such as sequence identification, functional site characterization and interaction prediction. In this paper, we revisit a collection of 29 state-of-the-art PU learning bioinformatic applications to address various biological questions. Various important aspects are extensively discussed, including PU learning methodology, biological application, classifier design and evaluation strategy. We also comment on the existing issues of PU learning and offer our perspectives for the future development of PU learning applications. We anticipate that our work serves as an instrumental guideline for a better understanding of the PU learning framework in bioinformatics and further developing next-generation PU learning frameworks for critical biological applications.

Unsupervised hyperspectral stimulated Raman microscopy image enhancement: denoising and segmentation via one-shot deep learning

Hyperspectral stimulated Raman scattering (SRS) microscopy is a label-free technique for biomedical and mineralogical imaging which can suffer from low signal-to-noise ratios. Here we demonstrate the use of an unsupervised deep learning neural network for rapid and automatic denoising of SRS images: UHRED (Unsupervised Hyperspectral Resolution Enhancement and Denoising). UHRED is capable of "one-shot" learning; only one hyperspectral image is needed, with no requirements for training on previously labelled datasets or images. Furthermore, by applying a k-means clustering algorithm to the processed data, we demonstrate automatic, unsupervised image segmentation, yielding, without prior knowledge of the sample, intuitive chemical species maps, as shown here for a lithium ore sample.

Fast denoising and lossless spectrum extraction in stimulated Raman scattering microscopy

Stimulated Raman scattering (SRS) microscopy is a nonlinear optical imaging method for visualizing chemical content based on molecular vibrational bonds. However, the imaging speed and sensitivity are currently limited by the noise of the light beam probing the Raman process. In this paper, we present a fast non-average denoising and high-precision Raman shift extraction method, based on a self-reinforcing signal-to-noise ratio (SNR) enhancement algorithm, for SRS spectroscopy and microscopy. We compare the results of this method with the filtering methods and the reported experimental methods to demonstrate its high efficiency and high precision in spectral denoising, Raman peak extraction and image quality improvement. We demonstrate a maximum SNR enhancement of 10.3 dB in fixed tissue imaging and 11.9 dB in vivo imaging. This method reduces the cost and complexity of the SRS system and allows for high-quality SRS imaging without use of special laser, complicated system design and Raman tags.