Deep learning enabled real time speckle recognition and hyperspectral imaging using a multimode fiber array

High-fidelity and high-speed wavefront shaping by leveraging complex media

High-precision light manipulation is crucial for delivering information through complex media. However, existing spatial light modulation devices face a fundamental speed-fidelity tradeoff. Digital micromirror devices have emerged as a promising candidate for high-speed wavefront shaping but at the cost of compromised fidelity due to the limited control degrees of freedom. Here, we leverage the sparse-to-random transformation through complex media to overcome the dimensionality limitation of spatial light modulation devices. We demonstrate that pattern compression by sparsity-constrained wavefront optimization allows sparse and robust wavefront representations in complex media, improving the projection fidelity without sacrificing frame rate, hardware complexity, or optimization time. Our method is generalizable to different pattern types and complex media, supporting consistent performance with up to 89% and 126% improvements in projection accuracy and speckle suppression, respectively. The proposed optimization framework could enable high-fidelity high-speed wavefront shaping through different scattering media and platforms without changes to the existing holographic setups, facilitating a wide range of physics and real-world applications.

Wide-field color imaging through multimode fiber with single wavelength illumination: plug-and-play approach

Multimode fiber (MMF) is extensively studied for its ability to transmit light modes in parallel, potentially minimizing optical fiber size in imaging. However, current research predominantly focuses on grayscale imaging, with limited attention to color studies. Existing colorization methods often involve costly white light lasers or multiple light sources, increasing optical system expenses and space. To achieve wide-field color images with typical monochromatic illumination MMF imaging system, we proposed a data-driven "colorization" approach and a neural network called SpeckleColorNet, merging U-Net and conditional GAN (cGAN) architectures, trained by a combined loss function. This approach, demonstrated on a 2-meter MMF system with single-wavelength illumination and the Peripheral Blood Cell (PBC) dataset, outperforms grayscale imaging and alternative colorization methods in readability, definition, detail, and accuracy. Our method aims to integrate MMF into clinical medicine and industrial monitoring, offering cost-effective high-fidelity color imaging. It serves as a plug-and-play replacement for conventional grayscale algorithms in MMF systems, eliminating the need for additional hardware.

Optical fiber speckle spectrometer based on reversed-lens smartphone microscope

Smartphones are a potentially powerful platform for scientific instruments. Here, we demonstrate speckle spectroscopy with smartphone-level hardware. This technique promises greater performance thresholds than traditional diffraction gratings. Light is injected into an optical fiber and the emergent speckle patterns are imaged by a reversed-lens smartphone camera. The smartphone then uses an algorithm, running on a mobile computing app, to determine, in less than one second, the (hitherto unknown) input spectrum. We reconstruct a variety of visible-wavelength (470-670 nm) single and multi-peaked spectra using a tunable source. The latter also include a metameric pair, i.e., two spectra that are different, yet represent colors that are indistinguishable to the human eye.

Compact wavemeter incorporating femtosecond laser-induced surface nanostructures enabled by deep learning

Miniature spectrometers have the advantage of high portability and integration, making them quick and easy to use in various working environments. The speckle patterns produced by light scattering through a disordered medium are highly sensitive to wavelength changes and can be used to design high-precision wavemeters and spectrometers. In this study, we used a self-organized, femtosecond laser-prepared nanostructure with a characteristic size of approximately 30-50 nm on a sapphire surface as a scattering medium to effectively induce spectral dispersion. By leveraging this random scattering structure, we successfully designed a compact scattering wavelength meter with efficient scattering properties. The collected speckle patterns were identified and classified using a neural network, and the variation of speckle patterns with wavelength was accurately extracted, achieving a measurement accuracy of 10 pm in multiple wavelength ranges. The system can effectively suppress instrument and environmental noise with high robustness. This work paves the way for the development of compact high-precision wavemeters.

High-quality color image restoration from a disturbed graded-index imaging system by deep neural networks

Imaging transmission plays an important role in endoscopic clinical diagnosis involved in modern medical treatment. However, image distortion due to various reasons has been a major obstacle to state-of-art endoscopic development. Here, as a preliminary study we demonstrate ultra-efficient recovery of exemplary 2D color images transmitted by a disturbed graded-index (GRIN) imaging system through the deep neural networks (DNNs). Indeed, the GRIN imaging system can preserve analog images through the GRIN waveguides with high quality, while the DNNs serve as an efficient tool for imaging distortion correction. Combining GRIN imaging systems and DNNs can greatly reduce the training process and achieve ideal imaging transmission. We consider imaging distortion under different realistic conditions and use both pix2pix and U-net type DNNs to restore the images, indicating the suitable network in each condition. This method can automatically cleanse the distorted images with superior robustness and accuracy, which can potentially be used in minimally invasive medical applications.

Central wavelength estimation in spectral imaging behind a diffuser via deep learning

Multispectral imaging through scattering media is an important practical issue in the field of sensing. The light from a scattering medium is expected to carry information about the spectral properties of the medium, as well as geometrical information. Because spatial and spectral information of the object is encoded in speckle images, the information about the structure and spectrum of the object behind the scattering medium can be estimated from those images. Here we propose a deep learning-based strategy that can estimate the central wavelength from speckle images captured with a monochrome camera. When objects behind scattering media are illuminated with narrowband light having different spectra with different spectral peaks, deep learning of speckle images acquired at different central wavelengths can extend the spectral region to reconstruct images and estimate the central wavelengths of the illumination light. The proposed method achieves central wavelength estimation in 1 nm steps for objects whose central wavelength varies in a range of 100 nm. Because our method can achieve image reconstruction and central wavelength estimation in a single shot using a monochrome camera, this technique will pave the way for multispectral imaging through scattering media.

Unsupervised full-color cellular image reconstruction through disordered optical fiber

Recent years have witnessed the tremendous development of fusing fiber-optic imaging with supervised deep learning to enable high-quality imaging of hard-to-reach areas. Nevertheless, the supervised deep learning method imposes strict constraints on fiber-optic imaging systems, where the input objects and the fiber outputs have to be collected in pairs. To unleash the full potential of fiber-optic imaging, unsupervised image reconstruction is in demand. Unfortunately, neither optical fiber bundles nor multimode fibers can achieve a point-to-point transmission of the object with a high sampling density, as is a prerequisite for unsupervised image reconstruction. The recently proposed disordered fibers offer a new solution based on the transverse Anderson localization. Here, we demonstrate unsupervised full-color imaging with a cellular resolution through a meter-long disordered fiber in both transmission and reflection modes. The unsupervised image reconstruction consists of two stages. In the first stage, we perform a pixel-wise standardization on the fiber outputs using the statistics of the objects. In the second stage, we recover the fine details of the reconstructions through a generative adversarial network. Unsupervised image reconstruction does not need paired images, enabling a much more flexible calibration under various conditions. Our new solution achieves full-color high-fidelity cell imaging within a working distance of at least 4 mm by only collecting the fiber outputs after an initial calibration. High imaging robustness is also demonstrated when the disordered fiber is bent with a central angle of 60°. Moreover, the cross-domain generality on unseen objects is shown to be enhanced with a diversified object set.

Data transmission with up to 100 orbital angular momentum modes via commercial multi-mode fiber and parallel neural networks

This work presents an artificial intelligence enhanced orbital angular momentum (OAM) data transmission system. This system enables encoded data retrieval from speckle patterns generated by an incident beam carrying different topological charges (TCs) at the distal end of a multi-mode fiber. An appropriately trained network is shown to support up to 100 different fractional TCs in parallel with TC intervals as small as 0.01, thus overcoming the problems with previous methods that only supported a few modes and could not use small TC intervals. Additionally, an approach using multiple parallel neural networks is proposed that can increase the system's channel capacity without increasing individual network complexity. When compared with a single network, multiple parallel networks can achieve the better performance with reduced training data requirements, which is beneficial in saving computational capacity while also expanding the network bandwidth. Finally, we demonstrate high-fidelity image transmission using a 16-bit system and four parallel 14-bit systems via OAM mode multiplexing through a 1-km-long commercial multi-mode fiber (MMF).

Machine learning for sensing with a multimode exposed core fiber specklegram sensor

Fiber specklegram sensors (FSSs) traditionally use statistical methods to analyze specklegrams obtained from fibers for sensing purposes, but can suffer from limitations such as vulnerability to noise and lack of dynamic range. In this paper we demonstrate that deep learning improves the analysis of specklegrams for sensing, which we show here for both air temperature and water immersion length measurements. Two deep neural networks (DNNs); a convolutional neural network and a multi-layer perceptron network, are used and compared to a traditional correlation technique on data obtained from a multimode fiber exposed-core fiber. The ability for the DNNs to be trained against a random noise source such as specklegram translations is also demonstrated.

Machine learning classification of origins and varieties of Tetrastigma hemsleyanum using a dual-mode microscopic hyperspectral imager

A dual-mode microscopic hyperspectral imager (DMHI) combined with a machine learning algorithm for the purpose of classifying origins and varieties of Tetrastigma hemsleyanum (T. hemsleyanum) was developed. By switching the illumination source, the DMHI can operate in reflection imaging and fluorescence detection modes. The DMHI system has excellent performance with spatial and spectral resolutions of 27.8 μm and 3 nm, respectively. To verify the capability of the DMHI system, a series of classification experiments of T. hemsleyanum were conducted. Captured hyperspectral datasets were analyzed using principal component analysis (PCA) for dimensional reduction, and a support vector machine (SVM) model was used for classification. In reflection microscopic hyperspectral imaging (RMHI) mode, the classification accuracies of T. hemsleyanum origins and varieties were 96.3% and 97.3%, respectively, while in fluorescence microscopic hyperspectral imaging (FMHI) mode, the classification accuracies were 97.3% and 100%, respectively. Combining datasets in dual mode, excellent predictions of origin and variety were realized by the trained model, both with a 97.5% accuracy on a newly measured test set. The results show that the DMHI system is capable of T. hemsleyanum origin and variety classification, and has the potential for non-invasive detection and rapid quality assessment of various kinds of medicinal herbs.

Water monitoring by means of digital microscopy identification and classification of microalgae

Marine and freshwater microalgae belong to taxonomically and morphologically diverse groups of organisms spanning many phyla with thousands of species. These organisms play an important role as indicators of water ecosystem conditions since they react quickly and predictably to a broad range of environmental stressors, thus providing early signals of dangerous changes. Traditionally, microscopic analysis has been used to identify and enumerate different types of organisms present within a given environment at a given point in time. However, this approach is both time-consuming and labor intensive, as it relies on manual processing and classification of planktonic organisms present within collected water samples. Furthermore, it requires highly skilled specialists trained to recognize and distinguish one taxa from another on the basis of often subtle morphological differences. Given these restrictions, a considerable amount of effort has been recently funneled into automating different steps of both the sampling and classification processes, making it possible to generate previously unprecedented volumes of plankton image data and obtain an essential database to analyze the composition of plankton assemblages. In this review we report state-of-the-art methods used for automated plankton classification by means of digital microscopy. The computer-microscope system hardware and the image processing techniques used for recognition and classification of planktonic organisms (segmentation, shape feature extraction, pigment signature determination and neural network grouping) will be described. An introduction and overview of the topic, its current state and indications of future directions the field is expected to take will be provided, organizing the review for both experts and researchers new to the field.

Classification, identification, and growth stage estimation of microalgae based on transmission hyperspectral microscopic imaging and machine learning

A transmission hyperspectral microscopic imager (THMI) that utilizes machine learning algorithms for hyperspectral detection of microalgae is presented. The THMI system has excellent performance with spatial and spectral resolutions of 4 µm and 3 nm, respectively. We performed hyperspectral imaging (HSI) of three species of microalgae to verify their absorption characteristics. Transmission spectra were analyzed using principal component analysis (PCA) and peak ratio algorithms for dimensionality reduction and feature extraction, and a support vector machine (SVM) model was used for classification. The average accuracy, sensitivity and specificity to distinguish one species from the other two species were found to be 94.4%, 94.4% and 97.2%, respectively. A species identification experiment for a group of mixed microalgae in solution demonstrates the usability of the classification method. Using a random forest (RF) model, the growth stage in a phaeocystis growth cycle cultivated under laboratory conditions was predicted with an accuracy of 98.1%, indicating the feasibility to evaluate the growth state of microalgae through their transmission spectra. Experimental results show that the THMI system has the capability for classification, identification and growth stage estimation of microalgae, with strong potential for in-situ marine environmental monitoring and early warning detection applications.

Neural-network-assisted in situ processing monitoring by speckle pattern observation

We propose a method to monitor the progress of laser processing using laser speckle patterns. Laser grooving and percussion drilling were performed using femtosecond laser pulses. The speckle patterns from a processing point were monitored with a high-speed camera and analyzed with a deep neural network. The deep neural network enabled us to extract multiple information from the speckle pattern without a need for analytical formulation. The trained neural network was able to predict the ablation depth with an uncertainty of 2 μm, as well as the material under processing, which will be useful for composite material processing.

Deep Learning-Based Image Classification through a Multimode Fiber in the Presence of Wavelength Drift

Deep neural networks (DNNs) are employed to recover information after its propagation through a multimode fiber (MMF) in the presence of wavelength drift. The intensity distribution of the speckle patterns generated at the output of an MMF when an input wavefront propagates along its length is highly sensitive to wavelength changes. We use a tunable laser to implement a wavelength drift with a controlled bandwidth, aiming to estimate the DNN’s performance in different cases and identify the limitations. We find that when the DNNs are trained with a dataset which includes the noise induced by wavelength changes, successful classification of a speckle pattern can be performed even for a large wavelength bandwidth drift. A single training step is found to be sufficient for high classification accuracy, removing the need for time-consuming recalibration at each wavelength.

Femtometer-resolved simultaneous measurement of multiple laser wavelengths in a speckle wavemeter

Many areas of optical science require an accurate measurement of optical spectra. Devices based on laser speckle promise compact wavelength measurement, with attometer-level sensitivity demonstrated for single wavelength laser fields. The measurement of multimode spectra using this approach would be attractive, yet this is currently limited to picometer resolution. Here, we present a method to improve the resolution and precision of speckle-based multi-wavelength measurements. We measure multiple wavelengths simultaneously, in a device comprising a single 1-m-long step-index multimode fiber and a fast camera. Independent wavelengths separated by as little as 1 fm are retrieved with 0.2 fm precision using principal component analysis. The method offers a viable way to measure sparse spectra containing multiple individual lines and may find application in the tracking of multiple lasers in fields such as quantum technologies and optical telecommunications.

Artificial compound eye-tipped optical fiber for wide field illumination

In this Letter, we present a novel, to the best of our knowledge, component with beam delivering and wide field beam homogenizing functions by grafting an artificial compound eye (ACE) micro-structure onto the polymer optical fiber (POF) end face. The 3D ACE mold is fabricated by femtosecond laser-assisted micro machining, and the ACE micro-structure is transferred onto the end face through high accuracy nano-imprinting. The resultant POF end face integrates over 400 spherical micro-lenses, enabling a 40% enhancement in both the acceptance angle and the effective numerical aperture. Meanwhile, the integrated ommatidia array serves as an outstanding beam homogenizer, shaping the output beam into quasi flat-top distribution, which demonstrates promise in wide field homogeneous illumination, by reflection and transmission imaging experiments in both visible and near infrared bands.