Optical hyperspectral imaging in microscopy and spectroscopy - a review of data acquisition

Transition from Light-Induced Phase Reconstruction to Halide Segregation in CsPbBr3-xIx Nanocrystal Thin Films

Inorganic metal-halide perovskite materials pave the way for many applications ranging from optoelectronics to quantum information due to their low cost, high photoluminescence and energy conversion efficiencies. However, light-induced bandgap instability due to ion migration in mixed-halide perovskites remains a significant challenge to the efficiency of optoelectronic devices. Thus, we combined hyperspectral fluorescence microspectroscopy and computational methods to understand the underlying transition mechanism between phase reconstruction and segregation in CsPbBr3-xIx (0 < x < 3) nanocrystal thin films. Our outcomes have shown that samples with x = 1.0 and x = 1.5 exhibit halide migration, favoring Br enrichment locally. In this case, an interplay between photo and thermal activation promotes the expulsion of I- from the perovskite lattice and generates a reconstruction of Br-rich domains, forming the CsPbBr3 phase. Thus, thermodynamic parameters such as the halide activation energy and phase reconstruction diffusibility were obtained by combining the kinetic parameters from linear unmixing data and Fick's second law. Moreover, we observed that the Br-I interdiffusion followed an Arrhenius-like behavior over laser-induced temperature increase. On the other hand, for samples with x = 2.0, phase segregation occurred due to the larger CsPbBrI2 nanocrystal size, iodine content and the high laser intensity employed. These three combined effects modify transport and recombination due to the reduction of charge carrier diffusion length (LD = 10.2 nm) and bandgap. Thus, iodide ions diffuse from the nanocrystal surface to the core forming a "type-II heterostructure", promoting a red shift in the fluorescence spectrum, which is characteristic of phase segregation. Furthermore, real-time dark recovery of light-induced halide segregation is reported for CsPbBrI2 nanocrystal thin films. Finally, the possible halide migration mechanism and physical origins of the transition between these phenomena are pointed out.

Snapshot computed tomographic microscopic imaging spectrometer and its video-level tracking of poisonous Microcystis aeruginosa cells in mixed algae

Conventional microscopic spectral imaging suffers from extended scanning times across wavelength or spatial dimension. To improve capabilities of dynamic microscopic spectral imaging, we developed a snapshot computed tomographic microscopic imaging spectrometer (CTMIS) based on the zeroth and first orders dispersive diffraction of a two-dimensional grating. Utilizing the CTMIS-UNET reconstruction algorithm, we can reconstruct a spectral cube (541x541x26) for each frame of micro spectral imaging video. Experimental results demonstrate a sub-4 μm spatial resolution achievable through a 20x objective lens and a spectral resolution better than 10 nm among 450-700 nm, while maintaining spectral cosine similarities exceeding 0.9989 when comparing reconstructed spectra with ground truth data. Spectral imaging videos of four species of algae and mixed algae were captured under 10 ms exposure time using the CTMIS system. Leveraging the self-developed UNET-SI26 algae recognition network, precise identification and tracking of four types of algae and poisonous microcysts aeruginosa in mixed algae were conducted. The pixel-level recognition accuracy exceeds 95 %, while the accuracy for counting the numbers of different types of cells surpasses 85 %, offering an efficient and accurate spectral imaging method for real-time monitoring and early warning of harmful algae at the cellular level.

Hyperspectral imaging in neurosurgery: a review of systems, computational methods, and clinical applications

Significance

Accurate identification between pathologic (e.g., tumors) and healthy brain tissue is a critical need in neurosurgery. However, conventional surgical adjuncts have significant limitations toward achieving this goal (e.g., image guidance based on pre-operative imaging becomes inaccurate up to 3 cm as surgery proceeds). Hyperspectral imaging (HSI) has emerged as a potential powerful surgical adjunct to enable surgeons to accurately distinguish pathologic from normal tissues.

Aim

We review HSI techniques in neurosurgery; categorize, explain, and summarize their technical and clinical details; and present some promising directions for future work.

Approach

We performed a literature search on HSI methods in neurosurgery focusing on their hardware and implementation details; classification, estimation, and band selection methods; publicly available labeled and unlabeled data; image processing and augmented reality visualization systems; and clinical study conclusions.

Results

We present a detailed review of HSI results in neurosurgery with a discussion of over 25 imaging systems, 45 clinical studies, and 60 computational methods. We first provide a short overview of HSI and the main branches of neurosurgery. Then, we describe in detail the imaging systems, computational methods, and clinical results for HSI using reflectance or fluorescence. Clinical implementations of HSI yield promising results in estimating perfusion and mapping brain function, classifying tumors and healthy tissues (e.g., in fluorescence-guided tumor surgery, detecting infiltrating margins not visible with conventional systems), and detecting epileptogenic regions. Finally, we discuss the advantages and disadvantages of HSI approaches and interesting research directions as a means to encourage future development.

Conclusions

We describe a number of HSI applications across every major branch of neurosurgery. We believe these results demonstrate the potential of HSI as a powerful neurosurgical adjunct as more work continues to enable rapid acquisition with smaller footprints, greater spectral and spatial resolutions, and improved detection.

Hyperspectral imaging of acoustically trapped plastics

Hyperspectral (HS) imaging bridges conventional imaging into spectroscopy and generates a spatial map of spectral variations. On the one hand, in HS imaging, the effect of the background on the final spectra has to be removed or managed. On the other hand, there are important classes of materials that need to be immobilized for investigation. In this Letter, we introduce acoustic trapping (AT) for contactless, nondestructive, and easy-to-implement immobilization of particles of up to several mm sizes, subjected to HS imaging experiments. We apply and validate the combined HS-AT apparatus for the identification of plastics, which these days have caused a multifaceted environmental pollution threat. We show that fluorescent HS imaging provides distinguishable fluorescent spectral signatures for an array of different acoustically trapped plastics. Moreover, the HS-AT enables tomographic spatio-spectral information of multi-component plastic samples by means of their acoustically controlled rotation. The integrated HS-AT has the potential to serve as a benchtop identification device.

Novel Deposition Technique for Fabricating Films with Customized Thickness Profiles

This study introduces a novel deposition technique capable of depositing thin films with any arbitrary thickness profile. The apparatus consists of a fixed shadow mask and a rotating sample carrier plate. The shadow mask features a specifically designed opening curve that corresponds to the particular thickness profile of the deposited film. We successfully designed two shadow masks and used them to deposit films with linear thickness gradients of 49.3 and 86.8 Å/mm and films with sinusoidal thickness profiles with a period of 40 mm. Furthermore, a linear variable filter was designed on the basis of a quarter-wavelength stack of Si3N4 and SiO2, combined with a TiO2 cavity layer with a linearly varying thickness. By coaxially rotating the sample carrier plate relative to the shadow mask, films with the desired thickness profiles could be fabricated in a single deposition step without the need for additional rotational or translational devices inside the deposition chamber. By rotating the carrier plate, the chips attached at different circumferential positions can achieve consistent thickness profiles, making this method well-suited for mass production.

Light sheet illumination in single-molecule localization microscopy for imaging of cellular architectures and molecular dynamics

Single-molecule localization microscopy has revealed cellular architectures and molecular dynamics beyond the diffraction limit of light. However, imaging thick samples presents challenges from increased fluorescence background. Light sheet illumination, which utilizes a plane of light for optical sectioning, is effective in reducing fluorescence background, photobleaching, and photodamage. Here, we present the principles of single-molecule localization microscopy and light sheet illumination, followed by an introduction to light sheet microscopy geometries and their imaging applications. Finally, we discuss light sheet illumination approaches for high- and super-resolution imaging of biological structures and dynamics.

Reflectance mapping with microsphere-assisted white light interference nanoscopy

The characterisation of novel materials presents a challenge that requires new and original developments. To face some of these demands for making measurements at the nanoscale, a new microsphere-assisted white light interference nanoscope performing local reflectance mapping is presented. This technique presents the advantages of being non-destructive, full-field and label-free. A 145 μm diameter microsphere, glued to the end of an optical fiber, is inserted inside the white light interference microscope to improve the lateral resolution from 940 nm to 520 nm. The acquisition and the Fourier transform processing of a stack of interference images superimposed on the virtual image produced by the microsphere allows the extraction of the local reflectance over a wavelength range of 460 nm to 900 nm and a field of view of 8 μm in diameter. The enhancement in the lateral resolution of the reflectance is demonstrated through the spectral distinction of neighboring ripples on a laser-textured colored stainless-steel sample that cannot be resolved without the microsphere, on regions with a surface of 279 × 279 nm2 horizontally spaced 279 nm apart. Future improvements could potentially lead to a lateral resolution of reflectance measurement over a 100 nm diameter area in air, paving the way to sub-diffraction reflectance mapping.

Hyperspectral retinal imaging in Alzheimer's disease and age-related macular degeneration: a review

While Alzheimer's disease and other neurodegenerative diseases have traditionally been viewed as brain disorders, there is growing evidence indicating their manifestation in the eyes as well. The retina, being a developmental extension of the brain, represents the only part of the central nervous system that can be noninvasively imaged at a high spatial resolution. The discovery of the specific pathological hallmarks of Alzheimer's disease in the retina of patients holds great promise for disease diagnosis and monitoring, particularly in the early stages where disease progression can potentially be slowed. Among various retinal imaging methods, hyperspectral imaging has garnered significant attention in this field. It offers a label-free approach to detect disease biomarkers, making it especially valuable for large-scale population screening efforts. In this review, we discuss recent advances in the field and outline the current bottlenecks and enabling technologies that could propel this field toward clinical translation.

Label-Free Assessment of Key Biological Autofluorophores: Material Characteristics and Opportunities for Clinical Applications

Autofluorophores are endogenous fluorescent compounds that naturally occur in the intra and extracellular spaces of all tissues and organs. Most have vital biological functions - like the metabolic cofactors NAD(P)H and FAD+, as well as the structural protein collagen. Others are considered to be waste products - like lipofuscin and advanced glycation end products - which accumulate with age and are associated with cellular dysfunction. Due to their natural fluorescence, these materials have great utility for enabling non-invasive, label-free assays with direct ties to biological function. Numerous technologies, with different advantages and drawbacks, are applied to their assessment, including fluorescence lifetime imaging microscopy, hyperspectral microscopy, and flow cytometry. Here, the applications of label-free autofluorophore assessment are reviewed for clinical and health-research applications, with specific attention to biomaterials, disease detection, surgical guidance, treatment monitoring, and tissue assessment - fields that greatly benefit from non-invasive methodologies capable of continuous, in vivo characterization.

Hyperspectral enhanced imaging analysis of nanoparticles using machine learning methods

Nanoparticle (NP)-based technologies have gained significant attention in targeted drug delivery, encompassing chemotherapies, photodynamic therapy, and immunotherapy. Hyperspectral imaging (HSI) emerges as a label-free, minimally invasive, and high-throughput technique for quantitative NP analysis. Despite its growing importance, the application of HSI to nanoparticle analysis, especially for label-free characterization and classification, remains limited. Here, we propose a novel method integrating hyperspectral imaging with a spectral noise reduction method and machine learning (ML) for robust nanoparticle classification. There are many challenges to extracting information from noisy and overlapping particles in HSI data. To surmount these challenges, we propose a spectral angle matching (SAM) algorithm to effectively denoise hyperspectral datasets. Complementing this, we employ a support vector machine (SVM) algorithm for classification, leveraging preprocessed HSI data to extract unique spectral signatures. Our hyperspectral imaging classification of multiple nanoparticle types reveals distinct spectral characteristics inherent to each class. The classification accuracy reaches 99.9% for single nanoparticle types, highlighting the efficiency of our method. In the case of classifying multiple particle types, the overall accuracy also reaches 99.9%. Visualization of the NP classification map further demonstrates the efficacy of our model. The application of the SAM-SVM algorithm in hyperspectral analysis outperforms traditional SVM methods in classifying multiple samples, highlighting the potential of our nanoparticle analysis. Our findings not only address the challenges posed by noisy and overlapping particles but also demonstrate the potential of hyperspectral imaging in advancing real-time and label-free detection systems for diverse biomedical applications.

SSTU: Swin-Spectral Transformer U-Net for hyperspectral whole slide image reconstruction

Whole Slide Imaging and Hyperspectral Microscopic Imaging provide great quality data with high spatial and spectral resolution for histopathology. Existing Hyperspectral Whole Slide Imaging systems combine the advantages of the techniques above, thus providing rich information for pathological diagnosis. However, it cannot avoid the problems of slow acquisition speed and mass data storage demand. Inspired by the spectral reconstruction task in computer vision and remote sensing, the Swin-Spectral Transformer U-Net (SSTU) has been developed to reconstruct Hyperspectral Whole Slide images (HWSis) from multiple Hyperspectral Microscopic images (HMis) of small Field of View and Whole Slide images (WSis). The Swin-Spectral Transformer (SST) module in SSTU takes full advantage of Transformer in extracting global attention. Firstly, Swin Transformer is exploited in space domain, which overcomes the high computation cost in Vision Transformer structures, while it maintains the spatial features extracted from WSis. Furthermore, Spectral Transformer is exploited to collect the long-range spectral features in HMis. Combined with the multi-scale encoder-bottleneck-decoder structure of U-Net, SSTU network is formed by sequential and symmetric residual connections of SSTs, which reconstructs a selected area of HWSi from coarse to fine. Qualitative and quantitative experiments prove the performance of SSTU in HWSi reconstruction task superior to other state-of-the-art spectral reconstruction methods.

Single-Molecule Spectroscopy and Super-Resolution Mapping of Physicochemical Parameters in Living Cells

By superlocalizing the positions of millions of single molecules over many camera frames, a class of super-resolution fluorescence microscopy methods known as single-molecule localization microscopy (SMLM) has revolutionized how we understand subcellular structures over the past decade. In this review, we highlight emerging studies that transcend the outstanding structural (shape) information offered by SMLM to extract and map physicochemical parameters in living mammalian cells at single-molecule and super-resolution levels. By encoding/decoding high-dimensional information-such as emission and excitation spectra, motion, polarization, fluorescence lifetime, and beyond-for every molecule, and mass accumulating these measurements for millions of molecules, such multidimensional and multifunctional super-resolution approaches open new windows into intracellular architectures and dynamics, as well as their underlying biophysical rules, far beyond the diffraction limit.

Development of a large volume line scanning, high spectral range and resolution 3D hyperspectral photoluminescence imaging microscope for diamond and other high refractive index materials

Hyperspectral photoluminescence (PL) imaging is a powerful technique that can be used to understand the spatial distribution of emitting species in many materials. Volumetric hyperspectral imaging of weakly emitting color centers often necessitates considerable data collection times when using commercial systems. We report the development of a line-scanning hyperspectral imaging microscope capable of measuring the luminescence emission spectra for diamond volumes up to 2.20 × 30.00 × 6.30 mm with a high lateral spatial resolution of 1-3 µm. In an single X-λ measurement, spectra covering a 711 nm range, in a band from 400-1100 nm, with a spectral resolution up to 0.25 nm can be acquired. Data sets can be acquired with 723 (X) × 643 (Y) × 1172 (λ) pixels at a rate of 6 minutes/planar image slice, allowing for volumetric hyperspectral imaging with high sampling. This instrument demonstrates the ability to detect emission from several different color centers in diamond both at the surface and internally, providing a non-destructive method to probe their 3D spatial distribution, and is currently not achievable with any other commonly used system or technique.

Handheld hyperspectral imaging as a tool for the post-mortem interval estimation of human skeletal remains

In forensic medicine, estimating human skeletal remains' post-mortem interval (PMI) can be challenging. Following death, bones undergo a series of chemical and physical transformations due to their interactions with the surrounding environment. Post-mortem changes have been assessed using various methods, but estimating the PMI of skeletal remains could still be improved. We propose a new methodology with handheld hyperspectral imaging (HSI) system based on the first results from 104 human skeletal remains with PMIs ranging between 1 day and 2000 years. To differentiate between forensic and archaeological bone material, the Convolutional Neural Network analyzed 65.000 distinct diagnostic spectra: the classification accuracy was 0.58, 0.62, 0.73, 0.81, and 0.98 for PMIs of 0 week-2 weeks, 2 weeks-6 months, 6 months-1 year, 1 year-10 years, and >100 years, respectively. In conclusion, HSI can be used in forensic medicine to distinguish bone materials >100 years old from those <10 years old with an accuracy of 98%. The model has adequate predictive performance, and handheld HSI could serve as a novel approach to objectively and accurately determine the PMI of human skeletal remains.

Wide-field mid-infrared hyperspectral imaging beyond video rate

Mid-infrared hyperspectral imaging has become an indispensable tool to spatially resolve chemical information in a wide variety of samples. However, acquiring three-dimensional data cubes is typically time-consuming due to the limited speed of raster scanning or wavelength tuning, which impedes real-time visualization with high spatial definition across broad spectral bands. Here, we devise and implement a high-speed, wide-field mid-infrared hyperspectral imaging system relying on broadband parametric upconversion of high-brightness supercontinuum illumination at the Fourier plane. The upconverted replica is spectrally decomposed by a rapid acousto-optic tunable filter, which records high-definition monochromatic images at a frame rate of 10 kHz based on a megapixel silicon camera. Consequently, the hyperspectral imager allows us to acquire 100 spectral bands over 2600-4085 cm-1 in 10 ms, corresponding to a refreshing rate of 100 Hz. Moreover, the angular dependence of phase matching in the image upconversion is leveraged to realize snapshot operation with spatial multiplexing for multiple spectral channels, which may further boost the spectral imaging rate. The high acquisition rate, wide-field operation, and broadband spectral coverage could open new possibilities for high-throughput characterization of transient processes in material and life sciences.

BINGO: a blind unmixing algorithm for ultra-multiplexing fluorescence images

MOTIVATION

Spectral imaging is often used to observe different objects with multiple fluorescent labels to reveal the development of the biological event. As the number of observed objects increases, the spectral overlap between fluorophores becomes more serious, and obtaining a "pure" picture of each fluorophore becomes a major challenge. Here, we propose a blind spectral unmixing algorithm called BINGO (Blind unmixing via SVD-based Initialization Nmf with project Gradient descent and spare cOnstrain), which can extract all kinds of fluorophores more accurately from highly overlapping multichannel data, even if the spectra of the fluorophores are extremely similar or their fluorescence intensity varies greatly.

RESULTS

BINGO can isolate up to 10 fluorophores from spectral imaging data for a single excitation. nine-color living HeLa cells were visualized distinctly with BINGO. It provides an important algorithmic tool for multiplex imaging studies, especially in intravital imaging. BINGO shows great potential in multicolor imaging for biomedical sciences.

AVAILABILITY AND IMPLEMENTATION

The source code used for this paper is available with the test data at https://github.com/Xinyuan555/BINGO_unmixing.

Fast data fitting scheme for compressive multispectral fluorescence lifetime imaging

A single-pixel camera combined with compressive sensing techniques is a promising fluorescence microscope scheme for acquiring a multidimensional dataset (space, spectrum, and lifetime) and for reducing the measurement time with respect to conventional microscope schemes. However, upon completing the acquisition, a computational step is necessary for image reconstruction and data analysis, which can be time-consuming, potentially canceling out the beneficial effect of compressive sensing. In this work, we propose and experimentally validate a fast-fit workflow based on global analysis and multiple linear fits, which significantly reduces the computation time from tens of minutes to less than 1 s. Moreover, as the method is interlaced with the measurement flow, it can be applied in parallel with the acquisitions.

Differentiation of Salivary Gland and Salivary Gland Tumor Tissue via Raman Imaging Combined with Multivariate Data Analysis

Salivary gland tumors (SGTs) are a relevant, highly diverse subgroup of head and neck tumors whose entity determination can be difficult. Confocal Raman imaging in combination with multivariate data analysis may possibly support their correct classification. For the analysis of the translational potential of Raman imaging in SGT determination, a multi-stage evaluation process is necessary. By measuring a sample set of Warthin tumor, pleomorphic adenoma and non-tumor salivary gland tissue, Raman data were obtained and a thorough Raman band analysis was performed. This evaluation revealed highly overlapping Raman patterns with only minor spectral differences. Consequently, a principal component analysis (PCA) was calculated and further combined with a discriminant analysis (DA) to enable the best possible distinction. The PCA-DA model was characterized by accuracy, sensitivity, selectivity and precision values above 90% and validated by predicting model-unknown Raman spectra, of which 93% were classified correctly. Thus, we state our PCA-DA to be suitable for parotid tumor and non-salivary salivary gland tissue discrimination and prediction. For evaluation of the translational potential, further validation steps are necessary.

High-spatial density snapshot imaging spectrometer enabled by 2-photon fabricated custom fiber bundles

We report on a proof-of-concept snapshot imaging spectrometer developed using an array of optical fibers fabricated with 2-photon polymerization (2PP). The dense input array maps to an output array with engineered void spaces for spectral information. Previously, the development and fabrication of custom fiber arrays for imaging spectrometers have been a complex, time-consuming, and costly process, requiring a semi-manual assembly of commercial components. This work applies an automatic development process based on 2PP additive manufacturing with the Nanoscribe GmbH Quantum X system. The technique allows printing of arbitrary optical quality structures with submicron resolution with less than 5 nm roughness, enabling small core fibers/integrated arrays. Specifically, we developed an array prototype of 40 × 80 with 6-micron pitch at the input and 80-micron pitch at the output. The air-clad fibers had a core diameter of 5 µm. Fabricated optical fiber arrays were incorporated into a prism-based imaging spectrometer system with 48 spectral channels to demonstrate multi-spectral imaging. Imaging of a USAF target and color printed letter C as well as spectral comparisons to a commercial spectrometer were used to validate the performance of the system. These results clearly demonstrate the functionality and potential applications of the 3D-printed fiber-based snapshot imaging spectrometer.

Data Augmentation Techniques for Machine Learning Applied to Optical Spectroscopy Datasets in Agrifood Applications: A Comprehensive Review

Machine learning (ML) and deep learning (DL) have achieved great success in different tasks. These include computer vision, image segmentation, natural language processing, predicting classification, evaluating time series, and predicting values based on a series of variables. As artificial intelligence progresses, new techniques are being applied to areas like optical spectroscopy and its uses in specific fields, such as the agrifood industry. The performance of ML and DL techniques generally improves with the amount of data available. However, it is not always possible to obtain all the necessary data for creating a robust dataset. In the particular case of agrifood applications, dataset collection is generally constrained to specific periods. Weather conditions can also reduce the possibility to cover the entire range of classifications with the consequent generation of imbalanced datasets. To address this issue, data augmentation (DA) techniques are employed to expand the dataset by adding slightly modified copies of existing data. This leads to a dataset that includes values from laboratory tests, as well as a collection of synthetic data based on the real data. This review work will present the application of DA techniques to optical spectroscopy datasets obtained from real agrifood industry applications. The reviewed methods will describe the use of simple DA techniques, such as duplicating samples with slight changes, as well as the utilization of more complex algorithms based on deep learning generative adversarial networks (GANs), and semi-supervised generative adversarial networks (SGANs).

Hyperspectral Imaging in Brain Tumor Surgery-Evidence of Machine Learning-Based Performance

BACKGROUND

Hyperspectral imaging (HSI) has the potential to enhance surgical tissue detection and diagnostics. Definite utilization of intraoperative HSI guidance demands validated machine learning and public datasets that currently do not exist. Moreover, current imaging conventions are dispersed, and evidence-based paradigms for neurosurgical HSI have not been declared.

METHODS

We presented the rationale and a detailed clinical paradigm for establishing microneurosurgical HSI guidance. In addition, a systematic literature review was conducted to summarize the current indications and performance of neurosurgical HSI systems, with an emphasis on machine learning-based methods.

RESULTS

The published data comprised a few case series or case reports aiming to classify tissues during glioma operations. For a multitissue classification problem, the highest overall accuracy of 80% was obtained using deep learning. Our HSI system was capable of intraoperative data acquisition and visualization with minimal disturbance to glioma surgery.

CONCLUSIONS

In a limited number of publications, neurosurgical HSI has demonstrated unique capabilities in contrast to the established imaging techniques. Multidisciplinary work is required to establish communicable HSI standards and clinical impact. Our HSI paradigm endorses systematic intraoperative HSI data collection, which aims to facilitate the related standards, medical device regulations, and value-based medical imaging systems.

Spectrally-shaped illumination for improved optical inspection of lateral III-V-semiconductor oxidation

We report an hyperspectral imaging microscopy system based on a spectrally-shaped illumination and its use to offer an enhanced in-situ inspection of a technological process that is critical in Vertical-Cavity Surface-Emitting Laser (VCSEL) manufacturing, the lateral III-V-semiconductor oxidation (AlOx). The implemented illumination source exploits a digital micromirror device (DMD) to arbitrarily tailor its emission spectrum. When combined to an imager, this source is shown to provide an additional ability to detect minute surface reflectance contrasts on any VCSEL or AlOx-based photonic structure and, in turn, to offer improved in-situ inspection of the oxide aperture shapes and dimensions down to the best-achievable optical resolution. The demonstrated technique is very versatile and could be readily extended to the real-time monitoring of oxidation or other semiconductor technological processes as soon as they rely on a real-time yet accurate measurement of spatio-spectral (reflectance) maps.

Discrimination between normal and necrotic small intestinal tissue using hyperspectral imaging and unsupervised classification

Objective and automatic clinical discrimination of normal and necrotic sites of small intestinal tissue remains challenging. In this study, hyperspectral imaging (HSI) and unsupervised classification techniques were used to distinguish normal and necrotic sites of small intestinal tissues. Small intestinal tissue hyperspectral images of eight Japanese large-eared white rabbits were acquired using a visible near-infrared hyperspectral camera, and K-means and density peaks (DP) clustering algorithms were used to differentiate between normal and necrotic tissue. The three cases in this study showed that the average clustering purity of the DP clustering algorithm reached 92.07% when the two band combinations of 500-622 and 700-858 nm were selected. The results of this study suggest that HSI and DP clustering can assist physicians in distinguishing between normal and necrotic sites in the small intestine in vivo.

Cascaded compressed-sensing single-pixel camera for high-dimensional optical imaging

Single-pixel detectors are popular devices in optical sciences because of their fast temporal response, high sensitivity, and low cost. However, when being used for imaging, they face a fundamental challenge in acquiring high-dimensional information of an optical field because they are essentially zero-dimensional sensors and measure only the light intensity. To address this problem, we developed a cascaded compressed-sensing single-pixel camera, which decomposes the measurement into multiple stages, sequentially reducing the dimensionality of the data from a high-dimensional space to zero dimension. This measurement scheme allows us to exploit the compressibility of a natural scene in multiple domains, leading to highly efficient data acquisition. We demonstrated our method in several demanding applications, including enabling tunable single-pixel full-waveform hyperspectral light detection and ranging (LIDAR) for the first time.

Tunable image projection spectrometry

We present tunable image projection spectrometry (TIPS), a Fourier-domain line-scan spectral imager with a tunable compression ratio. Compared to state-of-the-art spatial-domain pushbroom hyperspectral cameras, TIPS requires much fewer measurements and provides a higher light throughput. Using a rotating Dove prism and a cylindrical field lens, TIPS scans an input scene in the Fourier domain and captures a subset of multi-angled one-dimensional (1D) en face projections of the input scene, allowing a tailored data compression ratio for a given scene. We demonstrate the spectral imaging capability of TIPS with a hematoxylin and eosin (H&E) stained pathology slide. Moreover, we showed the spectral information obtained can be further converted to depths when combining TIPS with a low-coherence full-field spectral-domain interferometer.

A Novel Low Rank Smooth Flat-Field Correction Algorithm for Hyperspectral Microscopy Imaging

A flat-field correction method is proposed for multiple measured hyperspectral microscopy imaging in this paper. As the most crucial preprocessing process in quantitative microscopic analysis, flat-field correction solves the uneven illumination caused by vignetting in microscopic images, and guarantees the precision of spatial and spectral information in hyperspectral microscopic imaging. In order to carry out flat-field correction and extract uneven illumination among groups of hyperspectral microscopic data containing hundreds of bands simultaneously, two properties of vignetting have been exploited: i) low-rank property is reflected by little information contained in vignetting; ii) local smoothness can be observed as a gradual change in brightness of vignetting, which is typically equivalent to the sparseness in spatial frequency domain. Combining the two properties above, a novel Low Rank Smooth Flat-field Correction (LRSFC) model modified from common orthogonal basis extraction is proposed, while an optimization is solved based on alternating direction multiplier method (ADMM), obtaining a unique flat-field term with low-rank and smooth properties. Qualitative and quantitative experimental assessments indicate that LRSFC does not add extra cell texture to the extracted flat-field term, whose performance appears prior to other state-of-the-art flat-field correction methods.

Hyperspectral Microscopy Technology to Detect Syrups Adulteration of Endemic Guindo Santo and Quillay Honey Using Machine-Learning Tools

Honey adulteration is a common practice that affects food quality and sale prices, and certifying the origin of the honey using non-destructive methods is critical. Guindo Santo and Quillay are fundamental for the honey production of Biobío and the Ñuble region in Chile. Furthermore, Guindo Santo only exists in this area of the world. Therefore, certifying honey of this species is crucial for beekeeper communities-mostly natives-to give them advantages and competitiveness in the global market. To solve this necessity, we present a system for detecting adulterated endemic honey that combines different artificial intelligence networks with a confocal optical microscope and a tunable optical filter for hyperspectral data acquisition. Honey samples artificially adulterated with syrups at concentrations undetectable to the naked eye were used for validating different artificial intelligence models. Comparing Linear discriminant analysis (LDA), Support vector machine (SVM), and Neural Network (NN), we reach the best average accuracy value with SVM of 93% for all classes in both kinds of honey. We hope these results will be the starting point of a method for honey certification in Chile in an automated way and with high precision.

Super-multiplexing excitation spectral microscopy with multiple fluorescence bands

Fluorescence microscopy, with high molecular specificity and selectivity, is a valuable tool for studying complex biological systems and processes. However, the ability to distinguish a large number of distinct subcellular structures in a single sample is impeded by the broad spectra of molecular fluorescence. We have recently shown that excitation spectral microscopy provides a powerful means to unmix up to six fluorophores in a single fluorescence band. Here, by working with multiple fluorescence bands, we extend this approach to the simultaneous imaging of up to ten targets, with the potential for further expansions. By covering the excitation/emission bandwidth across the full visible range, an ultra-broad 24-wavelength excitation scheme is established through frame-synchronized scanning of the excitation wavelength from a white lamp via an acousto-optic tunable filter (AOTF), so that full-frame excitation-spectral images are obtained every 24 camera frames, offering superior spectral information and multiplexing capability. With numerical simulations, we validate the concurrent imaging of 10 fluorophores spanning the visible range to achieve exceptionally low (∼0.5%) crosstalks. For cell imaging experiments, we demonstrate unambiguous identification of up to eight different intracellular structures labeled by common fluorophores of substantial spectral overlap with minimal color crosstalks. We thus showcase an easy-to-implement, cost-effective microscopy system for visualizing complex cellular components with more colors and lower crosstalks.

Types of spectroscopy and microscopy techniques for cancer diagnosis: a review

Cancer is a life-threatening disease that has claimed the lives of many people worldwide. With the current diagnostic methods, it is hard to determine cancer at an early stage, due to its versatile nature and lack of genomic biomarkers. The rapid development of biophotonics has emerged as a potential tool in cancer detection and diagnosis. Using the fluorescence, scattering, and absorption characteristics of cells and tissues, it is possible to detect cancer at an early stage. The diagnostic techniques addressed in this review are highly sensitive to the chemical and morphological changes in the cell and tissue during disease progression. These changes alter the fluorescence signal of the cell/tissue and are detected using spectroscopy and microscopy techniques including confocal and two-photon fluorescence (TPF). Further, second harmonic generation (SHG) microscopy reveals the morphological changes that occurred in non-centrosymmetric structures in the tissue, such as collagen. Again, Raman spectroscopy is a non-destructive method that provides a fingerprinting technique to differentiate benign and malignant tissue based on Raman signal. Photoacoustic microscopy and spectroscopy of tissue allow molecule-specific detection with high spatial resolution and penetration depth. In addition, terahertz spectroscopic studies reveal the variation of tissue water content during disease progression. In this review, we address the applications of spectroscopic and microscopic techniques for cancer detection based on the optical properties of the tissue. The discussed state-of-the-art techniques successfully determines malignancy to its rapid diagnosis.

Review on the Application of Hyperspectral Imaging Technology of the Exposed Cortex in Cerebral Surgery

The study of brain science is vital to human health. The application of hyperspectral imaging in biomedical fields has grown dramatically in recent years due to their unique optical imaging method and multidimensional information acquisition. Hyperspectral imaging technology can acquire two-dimensional spatial information and one-dimensional spectral information of biological samples simultaneously, covering the ultraviolet, visible and infrared spectral ranges with high spectral resolution, which can provide diagnostic information about the physiological, morphological and biochemical components of tissues and organs. This technology also presents finer spectral features for brain imaging studies, and further provides more auxiliary information for cerebral disease research. This paper reviews the recent advance of hyperspectral imaging in cerebral diagnosis. Firstly, the experimental setup, image acquisition and pre-processing, and analysis methods of hyperspectral technology were introduced. Secondly, the latest research progress and applications of hyperspectral imaging in brain tissue metabolism, hemodynamics, and brain cancer diagnosis in recent years were summarized briefly. Finally, the limitations of the application of hyperspectral imaging in cerebral disease diagnosis field were analyzed, and the future development direction was proposed.

Leaf microscopy applications in photosynthesis research: identifying the gaps

Leaf imaging via microscopy has provided critical insights into research on photosynthesis at multiple junctures, from the early understanding of the role of stomata, through elucidating C4 photosynthesis via Kranz anatomy and chloroplast arrangement in single cells, to detailed explorations of diffusion pathways and light utilization gradients within leaves. In recent decades, the original two-dimensional (2D) explorations have begun to be visualized in three-dimensional (3D) space, revising our understanding of structure-function relationships between internal leaf anatomy and photosynthesis. In particular, advancing new technologies and analyses are providing fresh insight into the relationship between leaf cellular components and improving the ability to model net carbon fixation, water use efficiency, and metabolite turnover rate in leaves. While ground-breaking developments in imaging tools and techniques have expanded our knowledge of leaf 3D structure via high-resolution 3D and time-series images, there is a growing need for more in vivo imaging as well as metabolite imaging. However, these advances necessitate further improvement in microscopy sciences to overcome the unique challenges a green leaf poses. In this review, we discuss the available tools, techniques, challenges, and gaps for efficient in vivo leaf 3D imaging, as well as innovations to overcome these difficulties.

On Spectral-Spatial Classification of Hyperspectral Images Using Image Denoising and Enhancement Techniques, Wavelet Transforms and Controlled Data Set Partitioning

Obtaining relevant classification results for hyperspectral images depends on the quality of the data and the proposed selection of the samples and descriptors for the training and testing phases. We propose a hyperspectral image classification machine learning framework based on image processing techniques for denoising and enhancement and a parallel approach for the feature extraction step. This parallel approach is designed to extract the features by employing the wavelet transform in the spectral domain, and by using Local Binary Patterns to capture the texture-like information linked to the geometry of the scene in the spatial domain. The spectral and spatial features are concatenated for a Support Vector Machine-based supervised classifier. For the experimental validation, we propose a controlled sampling approach that ensures the independence of the selected samples for the training data set, respectively the testing data set, offering unbiased performance results. We argue that a random selection applied on the hyperspectral dataset to separate the samples for the learning and testing phases can cause overlapping between the two datasets, leading to biased classification results. The proposed approach, with the controlled sampling strategy, tested on three public datasets, Indian Pines, Salinas and Pavia University, provides good performance results.

When light meets biology - how the specimen affects quantitative microscopy

Fluorescence microscopy images should not be treated as perfect representations of biology. Many factors within the biospecimen itself can drastically affect quantitative microscopy data. Whereas some sample-specific considerations, such as photobleaching and autofluorescence, are more commonly discussed, a holistic discussion of sample-related issues (which includes less-routine topics such as quenching, scattering and biological anisotropy) is required to appropriately guide life scientists through the subtleties inherent to bioimaging. Here, we consider how the interplay between light and a sample can cause common experimental pitfalls and unanticipated errors when drawing biological conclusions. Although some of these discrepancies can be minimized or controlled for, others require more pragmatic considerations when interpreting image data. Ultimately, the power lies in the hands of the experimenter. The goal of this Review is therefore to survey how biological samples can skew quantification and interpretation of microscopy data. Furthermore, we offer a perspective on how to manage many of these potential pitfalls.

Polarization-controlled single-particle scattering imaging spectroscopy using waveguide excitation

An imaging spectroscopic system that enables spatially-resolved detection of single-particle scattering with polarization-controlled waveguide excitation scheme is presented. The detected microscopic images of inhomogeneous nanostructures are recorded in a time sequence into a data cube based on a Michelson interferometer. The interferograms on selected pixels are Fourier-transformed into multiple spectra. The waveguide excitation scheme is presented for both transmission and reflection measurements while the dark-field excitation scheme is presented in transmission measurements for comparison. Gold nanoparticles, nanorods, and particles on film are utilized in the detection of polarization-dependent spectra. Measurement results are verified with the finite-difference time-domain (FDTD) simulations. The polarization-controlled coupling conditions in nanorods and particle-on-film systems are discussed with simulated field distributions around the nanostructures.

Compressed sensing in fluorescence microscopy

Compressed sensing (CS) is a signal processing approach that solves ill-posed inverse problems, from under-sampled data with respect to the Nyquist criterium. CS exploits sparsity constraints based on the knowledge of prior information, relative to the structure of the object in the spatial or other domains. It is commonly used in image and video compression as well as in scientific and medical applications, including computed tomography and magnetic resonance imaging. In the field of fluorescence microscopy, it has been demonstrated to be valuable for fast and high-resolution imaging, from single-molecule localization, super-resolution to light-sheet microscopy. Furthermore, CS has found remarkable applications in the field of mesoscopic imaging, facilitating the study of small animals' organs and entire organisms. This review article illustrates the working principles of CS, its implementations in optical imaging and discusses several relevant uses of CS in the field of fluorescence imaging from super-resolution microscopy to mesoscopy.

Retinal Organoids Long-Term Functional Characterization Using Two-Photon Fluorescence Lifetime and Hyperspectral Microscopy

Pluripotent stem cell-derived organoid technologies have opened avenues to preclinical basic science research, drug discovery, and transplantation therapy in organ systems. Stem cell-derived organoids follow a time course similar to species-specific organ gestation in vivo. However, heterogeneous tissue yields, and subjective tissue selection reduce the repeatability of organoid-based scientific experiments and clinical studies. To improve the quality control of organoids, we introduced a live imaging technique based on two-photon microscopy to non-invasively monitor and characterize retinal organoids’ (RtOgs’) long-term development. Fluorescence lifetime imaging microscopy (FLIM) was used to monitor the metabolic trajectory, and hyperspectral imaging was applied to characterize structural and molecular changes. We further validated the live imaging experimental results with endpoint biological tests, including quantitative polymerase chain reaction (qPCR), single-cell RNA sequencing, and immunohistochemistry. With FLIM results, we analyzed the free/bound nicotinamide adenine dinucleotide (f/b NADH) ratio of the imaged regions and found that there was a metabolic shift from glycolysis to oxidative phosphorylation. This shift occurred between the second and third months of differentiation. The total metabolic activity shifted slightly back toward glycolysis between the third and fourth months and stayed relatively stable between the fourth and sixth months. Consistency in organoid development among cell lines and production lots was examined. Molecular analysis showed that retinal progenitor genes were expressed in all groups between days 51 and 159. Photoreceptor gene expression emerged around the second month of differentiation, which corresponded to the shift in the f/b NADH ratio. RtOgs between 3 and 6 months of differentiation exhibited photoreceptor gene expression levels that were between the native human fetal and adult retina gene expression levels. The occurrence of cone opsin expression (OPN1 SW and OPN1 LW) indicated the maturation of photoreceptors in the fourth month of differentiation, which was consistent with the stabilized level of f/b NADH ratio starting from 4 months. Endpoint single-cell RNA and immunohistology data showed that the cellular compositions and lamination of RtOgs at different developmental stages followed those in vivo.

Computerized fluorescence microscopy of microbial cells

The upgrading of fluorescence microscopy by the introduction of computer technologies has led to the creation of a new methodology, computerized fluorescence microscopy (CFM). CFM improves subjective visualization and combines it with objective quantitative analysis of the microscopic data. CFM has opened up two fundamentally new opportunities for studying microorganisms. The first is the quantitative measurement of the fluorescence parameters of the targeted fluorophores in association with certain structures of individual cells. The second is the expansion of the boundaries of visualization/resolution of intracellular components beyond the "diffraction limit" of light microscopy into the nanometer range. This enables to obtain unique information about the localization and dynamics of intracellular processes at the molecular level. The purpose of this review is to demonstrate the potential of CFM in the study of fundamental aspects of the structural and functional organization of microbial cells. The basics of computer processing and analysis of digital images are briefly described. The fluorescent molecules used in CFM with an emphasis on fluorescent proteins are characterized. The main methods of super-resolution microscopy (nanoscopy) are presented. The capabilities of various CFM methods for exploring microbial cells at the subcellular level are illustrated by the examples of various studies on yeast and bacteria.

Radiometric and design model for the tunable light-guide image processing snapshot spectrometer (TuLIPSS)

The tunable light-guide image processing snapshot spectrometer (TuLIPSS) is a novel remote sensing instrument that can capture a spectral image cube in a single snapshot. The optical modelling application for the absolute signal intensity on a single pixel of the sensor in TuLIPSS has been developed through a numerical simulation of the integral performance of each optical element in the TuLIPSS system. The absolute spectral intensity of TuLIPSS can be determined either from the absolute irradiance of the observed surface or from the tabulated spectral reflectance of various land covers and by the application of a global irradiance approach. The model is validated through direct comparison of the simulated results with observations. Based on tabulated spectral reflectance, the deviation between the simulated results and the measured observations is less than 5% of the spectral light flux across most of the detection bandwidth for a Lambertian-like surface such as concrete. Additionally, the deviation between the simulated results and the measured observations using global irradiance information is less than 10% of the spectral light flux across most of the detection bandwidth for all surfaces tested. This optical modelling application of TuLIPSS can be used to assist the optimal design of the instrument and explore potential applications. The influence of the optical components on the light throughput is discussed with the optimal design being a compromise among the light throughput, spectral resolution, and cube size required by the specific application under consideration. The TuLIPSS modelling predicts that, for the current optimal low-cost configuration, the signal to noise ratio can exceed 10 at 10 ms exposure time, even for land covers with weak reflectance such as asphalt and water. Overall, this paper describes the process by which the optimal design is achieved for particular applications and directly connects the parameters of the optical components to the TuLIPSS performance.

Hyperspectral imaging and artificial intelligence to detect oral malignancy - part 1 - automated tissue classification of oral muscle, fat and mucosa using a light-weight 6-layer deep neural network

Background

Hyperspectral imaging (HSI) is a promising non-contact approach to tissue diagnostics, generating large amounts of raw data for whose processing computer vision (i.e. deep learning) is particularly suitable. Aim of this proof of principle study was the classification of hyperspectral (HS)-reflectance values into the human-oral tissue types fat, muscle and mucosa using deep learning methods. Furthermore, the tissue-specific hyperspectral signatures collected will serve as a representative reference for the future assessment of oral pathological changes in the sense of a HS-library.

Methods

A total of about 316 samples of healthy human-oral fat, muscle and oral mucosa was collected from 174 different patients and imaged using a HS-camera, covering the wavelength range from 500 nm to 1000 nm. HS-raw data were further labelled and processed for tissue classification using a light-weight 6-layer deep neural network (DNN).

Results

The reflectance values differed significantly (p < .001) for fat, muscle and oral mucosa at almost all wavelengths, with the signature of muscle differing the most. The deep neural network distinguished tissue types with an accuracy of > 80% each.

Conclusion

Oral fat, muscle and mucosa can be classified sufficiently and automatically by their specific HS-signature using a deep learning approach. Early detection of premalignant-mucosal-lesions using hyperspectral imaging and deep learning is so far represented rarely in in medical and computer vision research domain but has a high potential and is part of subsequent studies.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13005-021-00292-0.

Out-of-Phase Imaging after Optical Modulation (OPIOM) for Multiplexed Fluorescence Imaging Under Adverse Optical Conditions

Fluorescence imaging has become a powerful tool for observations in biology. Yet it has also encountered limitations to overcome optical interferences of ambient light, autofluorescence, and spectrally interfering fluorophores. In this account, we first examine the current approaches which address these limitations. Then we more specifically report on Out-of-Phase Imaging after Optical Modulation (OPIOM), which has proved attractive for highly selective multiplexed fluorescence imaging even under adverse optical conditions. After exposing the OPIOM principle, we detail the protocols for successful OPIOM implementation.

Two-photon peak molecular brightness spectra reveal long-wavelength enhancements of multiplexed imaging depth and photostability

The broad use of two-photon microscopy has been enabled in part by Ti:Sapphire femtosecond lasers, which offer a wavelength-tunable source of pulsed excitation. Action spectra have thus been primarily reported for the tunable range of Ti:Sapphire lasers (∼700-1000 nm). However, longer wavelengths offer deeper imaging in tissue via reduced scattering and spectral dips in water absorption, and new generations of pulsed lasers offer wider tunable ranges. We present the peak molecular brightness spectra for eight Alexa Fluor dyes between 700-1300 nm as a first-order surrogate for action spectra measured with an unmodified commercial microscope, which reveal overlapping long-wavelength excitation peaks with potential for multiplexed excitation. We demonstrate simultaneous single-wavelength excitation of six spectrally overlapping fluorophores using either short (∼790 nm) or long (∼1090 nm) wavelengths, and that the newly characterized excitation peaks measured past 1000 nm offer improved photostability and enhanced fidelity of linear spectral unmixing at depth compared to shorter wavelengths.

Single Molecules Are Your Quanta: A Bottom-Up Approach toward Multidimensional Super-resolution Microscopy

The rise of single-molecule localization microscopy (SMLM) and related super-resolution methods over the past 15 years has revolutionized how we study biological and materials systems. In this Perspective, we reflect on the underlying philosophy of how diffraction-unlimited pictures containing rich spatial and functional information may gradually emerge through the local accumulation of single-molecule measurements. Starting with the basic concepts, we analyze the uniqueness of and opportunities in building up the final picture one molecule at a time. After brief introductions to the more established multicolor and three-dimensional measurements, we highlight emerging efforts to extend SMLM to new dimensions and functionalities as fluorescence polarization, emission spectra, and molecular motions, and discuss rising opportunities and future directions. With single molecules as our quanta, the bottom-up accumulation approach provides a powerful conduit for multidimensional microscopy at the nanoscale.

Excitation spectral microscopy for highly multiplexed fluorescence imaging and quantitative biosensing

The multiplexing capability of fluorescence microscopy is severely limited by the broad fluorescence spectral width. Spectral imaging offers potential solutions, yet typical approaches to disperse the local emission spectra notably impede the attainable throughput. Here we show that using a single, fixed fluorescence emission detection band, through frame-synchronized fast scanning of the excitation wavelength from a white lamp via an acousto-optic tunable filter, up to six subcellular targets, labeled by common fluorophores of substantial spectral overlap, can be simultaneously imaged in live cells with low (~1%) crosstalks and high temporal resolutions (down to ~10 ms). The demonstrated capability to quantify the abundances of different fluorophores in the same sample through unmixing the excitation spectra next enables us to devise novel, quantitative imaging schemes for both bi-state and Förster resonance energy transfer fluorescent biosensors in live cells. We thus achieve high sensitivities and spatiotemporal resolutions in quantifying the mitochondrial matrix pH and intracellular macromolecular crowding, and further demonstrate, for the first time, the multiplexing of absolute pH imaging with three additional target organelles/proteins to elucidate the complex, Parkin-mediated mitophagy pathway. Together, excitation spectral microscopy provides exceptional opportunities for highly multiplexed fluorescence imaging. The prospect of acquiring fast spectral images without the need for fluorescence dispersion or care for the spectral response of the detector offers tremendous potential.

Spectroscopic Estimation of N Concentration in Wheat Organs for Assessing N Remobilization Under Different Irrigation Regimes

Nitrogen (N) remobilization is a critical process that provides substantial N to winter wheat grains for improving yield productivity. Here, the remobilization of N from anthesis to maturity in two wheat cultivars under three irrigation regimes was measured and its relationship to organ N concentration was examined. Based on spectral data of organ powder samples, partial least squares regression (PLSR) models were calibrated to estimate N concentration (N _mass) and validated against laboratory-based measurements. Although spectral reflectance could accurately estimate N _mass, the PLSR-based N _mass-spectra predictive model was found to be organ-specific, organs at the top canopy (chaff and top three leaves) received the best predictions (R ² > 0.88). In addition, N remobilization efficiency (NRE) in the top two leaves and top third internode was highly correlated with its corresponding N concentration change (ΔN _mass) with an R ² of 0.90. ΔN _mass of the top first internode (TIN1) explained 78% variation of the whole-plant NRE. This study provides a proof of concept for estimating N concentration and assessing N remobilization using hyperspectral data of individual organs, which offers a non-chemical and low-cost approach to screen germplasms for an optimal NRE in drought-resistance breeding.

Snapshot Imaging Spectrometer Based on Pixel-Level Filter Array (PFA)

Snapshot spectral imaging technology plays an important role in many fields. However, most existing snapshot imaging spectrometers have the shortcomings of a large volume or heavy computational burden. In this paper, we present a novel snapshot imaging spectrometer based on the pixel-level filter array (PFA), which can simultaneously obtain both spectral and spatial information. The system is composed of a fore-optics, a PFA, a relay lens, and a monochromatic sensor. The incoming light first forms an intermediate image on the PFA through the fore-optics. Then, the relay lens reimages the spectral images on the PFA onto the monochromatic sensor. Through the use of the PFA, we can capture a three-dimensional (spatial coordinates and wavelength) datacube in a single exposure. Compared with existing technologies, our system possesses the advantages of a simple implementation, low cost, compact structure, and high energy efficiency by removing stacked dispersive or interferometric elements. Moreover, the characteristic of the direct imaging mode ensures the low computational burden of the system, thus shortening the imaging time. The principle and design of the system are described in detail. An experimental prototype is built and field experiments are carried out to verify the feasibility of the proposed scheme.

Multispectral compressive fluorescence lifetime imaging microscopy with a SPAD array detector

Multispectral/hyperspectral fluorescence lifetime imaging microscopy (λFLIM) is a promising tool for studying functional and structural biological processes. The rich information content provided by a multidimensional dataset is often in contrast with the acquisition speed. In this work, we develop and experimentally demonstrate a wide-field λFLIM setup, based on a novel time-resolved 18×1 single-photon avalanche diode array detector working in a single-pixel camera scheme, which parallelizes the spectral detection, reducing measurement time. The proposed system, which implements a single-pixel camera with a compressive sensing scheme, represents an optimal microscopy framework towards the design of λFLIM setups.

Multi-instrumental techniques for evaluating butterfly structural colors: A case study on Polyommatus bellargus (Rottemburg, 1775) (Lepidoptera: Lycaenidae: Polyommatinae)

Color is an important communication channel for day-flying butterflies. Chemical (pigmentary) coloration is often supplemented by physical color generated by photonic nanostructures. These nanoarchitectures - which are characteristic for a given species - exhibit wavelength ranges in which light propagation is forbidden. The photonic nanoarchitectures are located in the lumen of the wing scales and are developed individually by each scale during metamorphosis. This self-assembly process is governed by the genes in the nucleus of the scale producing cell. It is crucial to establish well-defined measurement methods for the unambiguous characterization and comparison of colors generated in such a complex manner. Owing to the intricate architecture ordered at multiple levels (from centimeters to tens of nanometers), the precise quantitative determination of butterfly wing coloration is not trivial. In this paper, we present an overview of several optical spectroscopy measurement methods and illustrate techniques for processing the obtained data, using the species Polyommatus bellargus as a test case, the males of which exhibit a variation in their blue structural color that is easily recognizable to the naked eye. The benefits and drawbacks of these optical methods are discussed and compared. Furthermore, the origin of the color differences is explained in relation to differences in the wing scale nanomorphology revealed by electron microscopy. This in turn is tentatively associated with the unusually large genetic drift reported for this species in the literature.

Integration of transcriptomes analysis with spectral signature of total RNA for generation of affordable remote sensing of Hepatocellular carcinoma in serum clinical specimens

Hepatocellular carcinoma (HCC) is a major global health problem with about 841,000 new cases and 782,000 deaths annually, due to lacking early biomarker/s, and centralized diagnosis. Transcriptomes research despite its infancy has proved excellence in its implementation in identifying a coherent specific cancer RNAs differential expression. However, results are sometimes overlapped by other cancer types which negatively affecting specificity, plus the high cost of the equipment used. Hyperspectral imaging (HSI) is an advanced tool with unique, spectroscopic features, is an emerging tool that has widely been used in cancer detection. Herein, a pilot study has been performed for HCC diagnosis, by exploiting HIS properties and the analysis of the transcriptome for the development of non-invasive remote HCC sensing. HSI data cube images of the sera extracted total RNA have been analyzed in HCC, normal subject, liver benign tumor, and chronic HCV with cirrhotic/non-cirrhotic liver groups. Data analyses have revealed a specific spectral signature for all groups and can be easily discriminated; at the computed optimum wavelength. Moreover, we have developed a simple setup based on a commercial laser pointer for sample illumination and a Smartphone CCD camera, with HSI consistent data output. We hypothesized that RNA differential expression and its spatial organization/folding are the key players in the obtained spectral signatures. To the best of our knowledge, we are the first to use HSI for sensing cancer based on total RNA in serum, using a Smartphone CCD camera/laser pointer. The proposed biosensor is simple, rapid (2 min), and affordable with specificity and sensitivity of more than 98% and high accuracy.

DMD-based hyperspectral microscopy with flexible multiline parallel scanning

As one of the most common hyperspectral microscopy (HSM) techniques, line-scanning HSM is currently utilized in many fields. However, its scanning efficiency is still considered to be inadequate since many biological and chemical processes occur too rapidly to be captured. Accordingly, in this work, a digital micromirror device (DMD) based on microelectromechanical systems (MEMS) is utilized to demonstrate a flexible multiline scanning HSM system. To the best of our knowledge, this is the first line-scanning HSM system in which the number of scanning lines N can be tuned by simply changing the DMD's parallel scanning units according to diverse applications. This brilliant strategy of effortless adjustability relies only on on-chip scanning methods and totally exploits the benefits of parallelization, aiming to achieve nearly an N-time improvement in the detection efficiency and an N-time decrease in the scanning time and data volume compared with the single-line method under the same operating conditions. To validate this, we selected a few samples of different spectral wavebands to perform reflection imaging, transmission imaging, and fluorescence imaging with varying numbers of scanning lines. The results show the great potential of our DMD-based HSM system for the rapid development of cellular biology, material analysis, and so on. In addition, its on-chip scanning process eliminates the inherent microscopic architecture, making the whole system compact, lightweight, portable, and not subject to site constraints.

Comprehensive review of surgical microscopes: technology development and medical applications

SIGNIFICANCE

Surgical microscopes provide adjustable magnification, bright illumination, and clear visualization of the surgical field and have been increasingly used in operating rooms. State-of-the-art surgical microscopes are integrated with various imaging modalities, such as optical coherence tomography (OCT), fluorescence imaging, and augmented reality (AR) for image-guided surgery.

AIM

This comprehensive review is based on the literature of over 500 papers that cover the technology development and applications of surgical microscopy over the past century. The aim of this review is threefold: (i) providing a comprehensive technical overview of surgical microscopes, (ii) providing critical references for microscope selection and system development, and (iii) providing an overview of various medical applications.

APPROACH

More than 500 references were collected and reviewed. A timeline of important milestones during the evolution of surgical microscope is provided in this study. An in-depth technical overview of the optical system, mechanical system, illumination, visualization, and integration with advanced imaging modalities is provided. Various medical applications of surgical microscopes in neurosurgery and spine surgery, ophthalmic surgery, ear-nose-throat (ENT) surgery, endodontics, and plastic and reconstructive surgery are described.

RESULTS

Surgical microscopy has been significantly advanced in the technical aspects of high-end optics, bright and shadow-free illumination, stable and flexible mechanical design, and versatile visualization. New imaging modalities, such as hyperspectral imaging, OCT, fluorescence imaging, photoacoustic microscopy, and laser speckle contrast imaging, are being integrated with surgical microscopes. Advanced visualization and AR are being added to surgical microscopes as new features that are changing clinical practices in the operating room.

CONCLUSIONS

The combination of new imaging technologies and surgical microscopy will enable surgeons to perform challenging procedures and improve surgical outcomes. With advanced visualization and improved ergonomics, the surgical microscope has become a powerful tool in neurosurgery, spinal, ENT, ophthalmic, plastic and reconstructive surgeries.