Image reconstruction through a multimode fiber with a simple neural network architecture

Seeing through multimode fibers using real-valued intensity transmission matrix with deep learning

Multimode fibers (MMFs) are emerging as a highly attractive technology for applications in biomedical endoscopy and telecommunications, thanks to their ability to transmit images and data through a large number of transverse optical modes. However, light transmission through MMFs suffers from distortions caused by mode dispersion and coupling. While recent deep learning advancements have shown potential for improving image transmission through MMFs, these methods typically require an extensive training dataset and often exhibit limited generalization capability. In this work, we propose a hybrid approach that combines a real-valued intensity transmission matrix (RVITM) with deep learning for enhanced image retrieval through MMFs. Our approach first characterizes the MMF and retrieves images using a RVITM algorithm, followed by refinement with a hierarchical, parallel multi-scale (HPM)-attention U-Net to improve image quality. Experimental results demonstrated that our approach achieved high-quality reconstructions, with structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR) values of up to 0.9524 and 33.244 dB, respectively. This approach also offers strong generalization capabilities, requires fewer training samples and converges more quickly compared to purely deep learning-based methods reported in the literature. These results highlight the potential of our method for ultrathin endoscopy applications and spatial-mode multiplexing in telecommunications using MMFs.

Unwrapping non-locality in the image transmission through turbid media

Achieving high-fidelity image transmission through turbid media is a significant challenge facing both the AI and photonic/optical communities. While this capability holds promise for a variety of applications, including data transfer, neural endoscopy, and multi-mode optical fiber-based imaging, conventional deep learning methods struggle to capture the nuances of light propagation, leading to weak generalization and limited reconstruction performance. To address this limitation, we investigated the non-locality present in the reconstructed images and discovered that conventional deep learning methods rely on specific features extracted from the training dataset rather than meticulously reconstructing each pixel. This suggests that they fail to effectively capture long-range dependencies between pixels, which are crucial for accurate image reconstruction. Inspired by the physics of light propagation in turbid media, we developed a global attention mechanism to approach this problem from a broader perspective. Our network harnesses information redundancy generated by peculiar non-local features across the input and output fiber facets. This mechanism enables a two-order-of-magnitude performance boost and high fidelity to the data context, ensuring an accurate representation of intricate details in a pixel-to-pixel reconstruction rather than mere loss minimization.

Exploiting holographically encoded variance to transmit labelled images through a multimode optical fiber

Artificial intelligence has emerged as promising tool to decode an image transmitted through a multimode fiber (MMF) by applying deep learning techniques. By transmitting thousands of images through the MMF, deep neural networks (DNNs) are able to decipher the seemingly random output speckle patterns and unveil the intrinsic input-output relationship. High fidelity reconstruction is obtained for datasets with a large degree of homogeneity, which underutilizes the capacity of the combined MMF-DNN system. Here, we show that holographic modulation can encode an additional layer of variance on the output speckle pattern, improving the overall transmissive capabilities of the system. Operatively, we have implemented this by adding a holographic label to the original dataset and injecting the resulting phase image into the fiber facet through a Fourier transform lens. The resulting speckle pattern dataset can be clustered primarily by holographic label, and can be reconstructed without loss of fidelity. As an application, we describe how color images may be segmented into RGB components and each color component may then be labelled by distinct hologram. A ResUNet architecture was then used to decode each class of speckle patterns and reconstruct the color image without the need for temporal synchronization between sender and receiver.

Adaptive noise-resilient deep learning for image reconstruction in multimode fiber scattering

This research offers a comprehensive exploration of three pivotal aspects within the realm of fiber optics and piezoelectric materials. The study delves into the influence of voltage variation on piezoelectric displacement, examines the effects of bending multimode fiber (MMF) on data transmission, and scrutinizes the performance of an autoencoder in MMF image reconstruction with and without additional noise. To assess the impact of voltage variation on piezoelectric displacement, experiments were conducted by applying varying voltages to a piezoelectric material, meticulously measuring its radial displacement. The results revealed a notable increase in displacement with higher voltage, presenting implications for fiber stability and overall performance. Additionally, the investigation into the effects of bending MMF on data transmission highlighted that the bending process causes the fiber to become leaky and radiate power radially, potentially affecting data transmission. This crucial insight emphasizes the necessity for further research to optimize data transmission in practical fiber systems. Furthermore, the performance of an autoencoder model was evaluated using a dataset of MMF images, in diverse scenarios. The autoencoder exhibited impressive accuracy in reconstructing MMF images with high fidelity. The results underscore the significance of ongoing research in these domains, propelling advancements in fiber optic technology.

Hollow core optical fiber enabled by epsilon-near-zero material

Hollow core optical fibers of numerous guiding mechanisms have been studied in the past decades for their advantages on guiding light in air core. This work demonstrates a new hollow core optical fiber based on a different guiding mechanism, which confines light with a cladding made of epsilon-near-zero (ENZ) material through total internal reflection. We show that the addition of a layer of ENZ material coating (e.g. indium tin oxide layer) significantly reduces the loss of the waveguide compared to the structure without the ENZ layer. We also show that the propagation loss of the ENZ hollow core fiber can be further improved by integrating ENZ materials with lower loss. This study presents a novel type of hollow core fiber, and can find advanced in-fiber photonic applications such as laser surgery/spectroscopy, novel gas-filled/discharge laser, in-fiber molecular/gas sensing, and low-latency optical fiber communication.

Spatial pilot-aided fast-adapted framework for stable image transmission over long multi-mode fiber

Multi-mode fiber (MMF) has emerged as a promising platform for spatial information transmission attributed to its high capacity. However, the scattering characteristic and time-varying nature of MMF pose challenges for long-term stable transmission. In this study, we propose a spatial pilot-aided learning framework for MMF image transmission, which effectively addresses these challenges and maintains accurate performance in practical applications. By inserting a few reference image frames into the transmitting image sequence and leveraging a fast-adapt network training scheme, our framework adaptively accommodates to the physical channel variations and enables online model update for continuous transmission. Experimented on 100 m length unstable MMFs, we demonstrate transmission accuracy exceeding 92% over hours, with pilot frame overhead around 2%. Our fast-adapt learning scheme requires training of less than 2% of network parameters and reduces the computation time by 70% compared to conventional tuning approaches. Additionally, we propose two pilot-insertion strategies and elaborately compare their applicability to a wide range of scenarios including continuous transmission, burst transmission and transmission after fiber re-plugging. The proposed spatial pilot-aided fast-adapt framework opens up the possibility for MMF spatial transmission in practical complicated applications.

Projecting colorful images through scattering media via deep learning

The existence of scatterers in the optical path has been the major obstacle that prohibits one from projecting images through solid walls, turbid water, clouds, and fog. Recent developments in wavefront shaping and neural networks demonstrate effective compensation for scattering effects, showing the promise to project clear images against strong scattering. However, previous studies were mainly restricted to projecting greyscale images using monochromatic light, mainly due to the increased complexity of simultaneously controlling multiple wavelengths. In this work, we fill this blank by developing a projector network, which enables the projection of colorful images through scattering media with three primary colors. To validate the performance of the projector network, we experimentally demonstrated projecting colorful images obtained from the MINST dataset through two stacked diffusers. Quantitatively, the averaged intensity Pearson's correlation coefficient for 1,000 test colorful images reaches about 90.6%, indicating the superiority of the developed network. We anticipate that the projector network can be beneficial to a variety of display applications in scattering environments.

High-quality color image restoration from a disturbed graded-index imaging system by deep neural networks

Imaging transmission plays an important role in endoscopic clinical diagnosis involved in modern medical treatment. However, image distortion due to various reasons has been a major obstacle to state-of-art endoscopic development. Here, as a preliminary study we demonstrate ultra-efficient recovery of exemplary 2D color images transmitted by a disturbed graded-index (GRIN) imaging system through the deep neural networks (DNNs). Indeed, the GRIN imaging system can preserve analog images through the GRIN waveguides with high quality, while the DNNs serve as an efficient tool for imaging distortion correction. Combining GRIN imaging systems and DNNs can greatly reduce the training process and achieve ideal imaging transmission. We consider imaging distortion under different realistic conditions and use both pix2pix and U-net type DNNs to restore the images, indicating the suitable network in each condition. This method can automatically cleanse the distorted images with superior robustness and accuracy, which can potentially be used in minimally invasive medical applications.

High-definition image transmission through dynamically perturbed multimode fiber by a self-attention based neural network

We implement faithful multimode fiber (MMF) image transmission by a self-attention-based neural network. Compared with a real-valued artificial neural network (ANN) based on a convolutional neural network (CNN), our method utilizes a self-attention mechanism to achieve a higher image quality. The enhancement measure (EME) and structural similarity (SSIM) of the dataset collected in the experiment improved by 0.79 and 0.04; the total number of parameters can be reduced by up to 25%. To enhance the robustness of the neural network to MMF bending in image transmission, we use a simulation dataset to prove that the hybrid training method is helpful in MMF transmission of a high-definition image. Our findings may pave the way for simpler and more robust single-MMF image transmission schemes with hybrid training; SSIM on datasets under different disturbances improve by 0.18. This system has the potential to be applied to various high-demand image transmission tasks, such as endoscopy.

Multimode fiber-based greyscale image projector enabled by neural networks with high generalization ability

Multimode fibers (MMFs) are emerging as promising transmission media for delivering images. However, strong mode coupling inherent in MMFs induces difficulties in directly projecting two-dimensional images through MMFs. By training two subnetworks named Actor-net and Model-net synergetically, [Nature Machine Intelligence2, 403 (2020)10.1038/s42256-020-0199-9] alleviated this issue and demonstrated projecting images through MMFs with high fidelity. In this work, we make a step further by improving the generalization ability to greyscale images. The modified projector network contains three subnetworks, namely forward-net, backward-net, and holography-net, accounting for forward propagation, backward propagation, and the phase-retrieval process. As a proof of concept, we experimentally trained the projector network using randomly generated phase maps and their corresponding resultant speckle images output from a 1-meter-long MMF. With the network being trained, we successfully demonstrated projecting binary images from MNIST and EMNIST and greyscale images from Fashion-MNIST, exhibiting averaged Pearson's correlation coefficients of 0.91, 0.92, and 0.87, respectively. Since all these projected images have never been seen by the projector network before, a strong generalization ability in projecting greyscale images is confirmed.

Unsupervised Data-Driven Classification of Topological Gapped Systems with Symmetries

A remarkable breakthrough in topological phase classification is the establishment of the topological periodic table, which is mainly based on the classifying space analysis or K theory, but not based on concrete Hamiltonians that possess finite bands or arise in a lattice. As a result, it is still difficult to identify the topological phase of an arbitrary Hamiltonian; the common practice is, instead, to check the incomplete and still growing list of topological invariants one by one, very often by trial and error. Here, we develop unsupervised classifications of topological gapped systems with symmetries, and demonstrate the data-driven construction of the topological periodic table without a priori knowledge of topological invariants. This unsupervised data-driven strategy can take into account spatial symmetries, and further classify phases that were previously classified as trivial in the past. Our Letter introduces machine learning into topological phase classification and paves the way for intelligent explorations of new phases of topological matter.

Dark soliton detection using persistent homology

Classifying images often requires manual identification of qualitative features. Machine learning approaches including convolutional neural networks can achieve accuracy comparable to human classifiers but require extensive data and computational resources to train. We show how a topological data analysis technique, persistent homology, can be used to rapidly and reliably identify qualitative features in experimental image data. The identified features can be used as inputs to simple supervised machine learning models, such as logistic regression models, which are easier to train. As an example, we consider the identification of dark solitons using a dataset of 6257 labeled atomic Bose-Einstein condensate density images.

All-fiber high-speed image detection enabled by deep learning

Ultra-high-speed imaging serves as a foundation for modern science. While in biomedicine, optical-fiber-based endoscopy is often required for in vivo applications, the combination of high speed with the fiber endoscopy, which is vital for exploring transient biomedical phenomena, still confronts some challenges. We propose all-fiber imaging at high speeds, which is achieved based on the transformation of two-dimensional spatial information into one-dimensional temporal pulsed streams by leveraging high intermodal dispersion in a multimode fiber. Neural networks are trained to reconstruct images from the temporal waveforms. It can not only detect content-aware images with high quality, but also detect images of different kinds from the training images with slightly reduced quality. The fiber probe can detect micron-scale objects with a high frame rate (15.4 Mfps) and large frame depth (10,000). This scheme combines high speeds with high mechanical flexibility and integration and may stimulate future research exploring various phenomena in vivo.

Surveillance of few-mode fiber-communication channels with a single hidden layer neural network

Multi- and few-mode fibers (FMFs) promise to enhance the capacity of optical communication networks by orders of magnitude. The key for this evolution was the strong advancement of computational approaches that allowed inherent complex light transmission to be surpassed, learned, or controlled, reined in by modal crosstalk and mode-dependent losses. However, complex light transmission through FMFs can be learned by a single hidden layer neural network (NN). The emerging developments in NNs additionally allow the implementation of novel concepts for security enhancements in optical communication. Once the transmission characteristics of FMFs are learned, it is possible to survey the incoming and outgoing light fields via monitoring channels during data transmission. If an eavesdropper tries to gain unauthorized access to the FMF, its transmission properties are impaired through sensitive modal crosstalk. This process is registered by the NN and thus the eavesdropper is revealed. With our solution, the security of optical communication can be improved.

Learning to image and compute with multimode optical fibers

Multimode fibers (MMF) were initially developed to transmit digital information encoded in the time domain. There were few attempts in the late 60s and 70s to transmit analog images through MMF. With the availability of digital spatial modulators, practical image transfer through MMFs has the potential to revolutionize medical endoscopy. Because of the fiber's ability to transmit multiple spatial modes of light simultaneously, MMFs could, in principle, replace the millimeters-thick bundles of fibers currently used in endoscopes with a single fiber, only a few hundred microns thick. That, in turn, could potentially open up new, less invasive forms of endoscopy to perform high-resolution imaging of tissues out of reach of current conventional endoscopes. Taking endoscopy by its general meaning as looking into, we review in this paper novel ways of imaging and transmitting images using a machine learning approach. Additionally, we review recent work on using MMF to perform machine learning tasks. The advantages and disadvantages of using machine learning instead of conventional methods is also discussed. Methods of imaging in scattering media and particularly MMFs involves measuring the phase and amplitude of the electromagnetic wave, coming out of the MMF and using these measurements to infer the relationship between the input and the output of the MMF. Most notable techniques include analog phase conjugation [A. Yariv, "On transmission and recovery of three-dimensional image information in optical waveguides," J. Opt. Soc. Am., vol. 66, no. 4, pp. 301-306, 1976; A. Gover, C. Lee, and A. Yariv, "Direct transmission of pictorial information in multimode optical fibers," J. Opt. Soc. Am., vol. 66, no. 4, pp. 306-311, 1976; G. J. Dunning and R. Lind, "Demonstration of image transmission through fibers by optical phase conjugation," Opt. Lett., vol. 7, no. 11, pp. 558-560, 1982; A. Friesem, U. Levy, and Y. Silberberg, "Parallel transmission of images through single optical fibers," Proc. IEEE, vol. 71, no. 2, pp. 208-221, 1983], digital phase conjugation [I. N. Papadopoulos, S. Farahi, C. Moser, and D. Psaltis, "Focusing and scanning light through a multimode optical fiber using digital phase conjugation," Opt. Express, vol. 20, no. 10, pp. 10583-10590, 2012; I. N. Papadopoulos, S. Farahi, C. Moser, and D. Psaltis, "High-resolution, lensless endoscope based on digital scanning through a multimode optical fiber," Biomed. Opt. Express, vol. 4, no. 2, pp. 260-270, 2013] or the full-wave holographic transmission matrix method. The latter technique, which is the current gold standard, measures both the amplitude and phase of the output patterns corresponding to multiple input patterns to construct a matrix of complex numbers relaying the input to the output [Y. Choi, et al., "Scanner-free and wide-field endoscopic imaging by using a single multimode optical fiber," Phys. Rev. Lett., vol. 109, no. 20, p. 203901, 2012; A. M. Caravaca-Aguirre, E. Niv, D. B. Conkey, and R. Piestun, "Real-time resilient focusing through a bending multimode fiber," Opt. Express, vol. 21, no. 10, pp. 12881-12887; R. Y. Gu, R. N. Mahalati, and J. M. Kahn, "Design of flexible multi-mode fiber endoscope," Opt. Express, vol. 23, no. 21, pp. 26905-26918, 2015; D. Loterie, S. Farahi, I. Papadopoulos, A. Goy, D. Psaltis, and C. Moser, "Digital confocal microscopy through a multimode fiber," Opt. Express, vol. 23, no. 18, pp. 23845-23858, 2015]. This matrix is then used for imaging of the inputs or projection of desired patterns. Other techniques rely on iteratively optimizing the pixel value of the input image to perform a particular task (such as focusing or displaying an image) [R. Di Leonardo and S. Bianchi, "Hologram transmission through multi-mode optical fibers," Opt. Express, vol. 19, no. 1, pp. 247-254, 2011; T. Čižmár and K. Dholakia, "Shaping the light transmission through a multimode optical fibre: complex transformation analysis and applications in biophotonics," Opt. Express, vol. 19, no. 20, pp. 18871-18884, 2011; T. Čižmár and K. Dholakia, "Exploiting multimode waveguides for pure fibre-based imaging," Nat. Commun., vol. 3, no. 1, pp. 1-9, 2012; S. Bianchi and R. Di Leonardo, "A multi-mode fiber probe for holographic micromanipulation and microscopy," Lab Chip, vol. 12, no. 3, pp. 635-639, 2012; E. R. Andresen, G. Bouwmans, S. Monneret, and H. Rigneault, "Toward endoscopes with no distal optics: video-rate scanning microscopy through a fiber bundle," Opt. Lett., vol. 38, no. 5, pp. 609-611, 2013].

Deeply Subwavelength Localization with Reverberation-Coded Aperture

Accessing subwavelength information about a scene from the far-field without invasive near-field manipulations is a fundamental challenge in wave engineering. Yet it is well understood that the dwell time of waves in complex media sets the scale for the waves' sensitivity to perturbations. Modern coded-aperture imagers leverage the degrees of freedom (d.o.f.) offered by complex media as natural multiplexor but do not recognize and reap the fundamental difference between placing the object of interest outside or within the complex medium. Here, we show that the precision of localizing a subwavelength object can be improved by several orders of magnitude simply by enclosing it in its far field with a reverberant passive chaotic cavity. We identify deep learning as a suitable noise-robust tool to extract subwavelength localization information encoded in multiplexed measurements, achieving resolutions well beyond those available in the training data. We demonstrate our finding in the microwave domain: harnessing the configurational d.o.f. of a simple programmable metasurface, we localize a subwavelength object along a curved trajectory inside a chaotic cavity with a resolution of λ/76 using intensity-only single-frequency single-pixel measurements. Our results may have important applications in photoacoustic imaging as well as human-machine interaction based on reverberating elastic waves, sound, or microwaves.