Transmission of natural scene images through a multimode fibre

Roadmap on computational methods in optical imaging and holography [invited]

Computational methods have been established as cornerstones in optical imaging and holography in recent years. Every year, the dependence of optical imaging and holography on computational methods is increasing significantly to the extent that optical methods and components are being completely and efficiently replaced with computational methods at low cost. This roadmap reviews the current scenario in four major areas namely incoherent digital holography, quantitative phase imaging, imaging through scattering layers, and super-resolution imaging. In addition to registering the perspectives of the modern-day architects of the above research areas, the roadmap also reports some of the latest studies on the topic. Computational codes and pseudocodes are presented for computational methods in a plug-and-play fashion for readers to not only read and understand but also practice the latest algorithms with their data. We believe that this roadmap will be a valuable tool for analyzing the current trends in computational methods to predict and prepare the future of computational methods in optical imaging and holography.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00340-024-08280-3.

Deep Learning-Based Simultaneous Temperature- and Curvature-Sensitive Scatterplot Recognition

Since light propagation in a multimode fiber (MMF) exhibits visually random and complex scattering patterns due to external interference, this study numerically models temperature and curvature through the finite element method in order to understand the complex interactions between the inputs and outputs of an optical fiber under conditions of temperature and curvature interference. The systematic analysis of the fiber's refractive index and bending loss characteristics determined its critical bending radius to be 15 mm. The temperature speckle atlas is plotted to reflect varying bending radii. An optimal end-to-end residual neural network model capable of automatically extracting highly similar scattering features is proposed and validated for the purpose of identifying temperature and curvature scattering maps of MMFs. The viability of the proposed scheme is tested through numerical simulations and experiments, the results of which demonstrate the effectiveness and robustness of the optimized network model.

Memory-less scattering imaging with ultrafast convolutional optical neural networks

The optical memory effect in complex scattering media including turbid tissue and speckle layers has been a critical foundation for macroscopic and microscopic imaging methods. However, image reconstruction from strong scattering media without the optical memory effect has not been achieved. Here, we demonstrate image reconstruction through scattering layers where no optical memory effect exists, by developing a multistage convolutional optical neural network (ONN) integrated with multiple parallel kernels operating at the speed of light. Training this Fourier optics-based, parallel, one-step convolutional ONN with the strong scattering process for direct feature extraction, we achieve memory-less image reconstruction with a field of view enlarged by a factor up to 271. This device is dynamically reconfigurable for ultrafast multitask image reconstruction with a computational power of 1.57 peta-operations per second (POPS). Our achievement establishes an ultrafast and high energy-efficient optical machine learning platform for graphic processing.

Recovery of angular scattering profiles through a flexible multimode fiber

Endoscopic angle-resolved light scattering methods have been developed for early cancer detection but they typically require multi-element coherent fiber optic bundles to recover scattering distributions from tissues. Recent work has focused on using a single multimode fiber (MMF) to measure angle resolved scattering but this approach has practical limitations to overcome before clinical translation. Here we address these limitations by proposing an MMF-based endoscope capable of measuring angular scattering patterns suitable for determining structure. Significantly, this approach implements a spectrally resolved detection scheme to reduce speckle and leverages the azimuthal symmetry of the angular scattering patterns to enable measurements that are robust to fiber bending. This results in a unique method that does not require matrix inversion or machine learning to measure a transmitted scattering distribution. The MMF utilized here is 1000 mm in length with a 200 µm core and is demonstrated to recover angular scattering distributions even with bending displacements of up to 30 cm. This advance has a significant impact on the clinical translation of biomedical endoscopic diagnostic techniques that use angular scattering to determine the size of cell nuclei to detect early cancer.

Exploiting holographically encoded variance to transmit labelled images through a multimode optical fiber

Artificial intelligence has emerged as promising tool to decode an image transmitted through a multimode fiber (MMF) by applying deep learning techniques. By transmitting thousands of images through the MMF, deep neural networks (DNNs) are able to decipher the seemingly random output speckle patterns and unveil the intrinsic input-output relationship. High fidelity reconstruction is obtained for datasets with a large degree of homogeneity, which underutilizes the capacity of the combined MMF-DNN system. Here, we show that holographic modulation can encode an additional layer of variance on the output speckle pattern, improving the overall transmissive capabilities of the system. Operatively, we have implemented this by adding a holographic label to the original dataset and injecting the resulting phase image into the fiber facet through a Fourier transform lens. The resulting speckle pattern dataset can be clustered primarily by holographic label, and can be reconstructed without loss of fidelity. As an application, we describe how color images may be segmented into RGB components and each color component may then be labelled by distinct hologram. A ResUNet architecture was then used to decode each class of speckle patterns and reconstruct the color image without the need for temporal synchronization between sender and receiver.

Deep learning sheds new light on non-orthogonal optical multiplexing

A deep neural network for non-orthogonal input channel encoding is proposed to recover speckle images through a multimode fiber. This novel approach could shed new light on the non-orthogonal optical multiplexing over a scattering medium.

Adaptive noise-resilient deep learning for image reconstruction in multimode fiber scattering

This research offers a comprehensive exploration of three pivotal aspects within the realm of fiber optics and piezoelectric materials. The study delves into the influence of voltage variation on piezoelectric displacement, examines the effects of bending multimode fiber (MMF) on data transmission, and scrutinizes the performance of an autoencoder in MMF image reconstruction with and without additional noise. To assess the impact of voltage variation on piezoelectric displacement, experiments were conducted by applying varying voltages to a piezoelectric material, meticulously measuring its radial displacement. The results revealed a notable increase in displacement with higher voltage, presenting implications for fiber stability and overall performance. Additionally, the investigation into the effects of bending MMF on data transmission highlighted that the bending process causes the fiber to become leaky and radiate power radially, potentially affecting data transmission. This crucial insight emphasizes the necessity for further research to optimize data transmission in practical fiber systems. Furthermore, the performance of an autoencoder model was evaluated using a dataset of MMF images, in diverse scenarios. The autoencoder exhibited impressive accuracy in reconstructing MMF images with high fidelity. The results underscore the significance of ongoing research in these domains, propelling advancements in fiber optic technology.

High-security learning-based optical encryption assisted by disordered metasurface

Artificial intelligence has gained significant attention for exploiting optical scattering for optical encryption. Conventional scattering media are inevitably influenced by instability or perturbations, and hence unsuitable for long-term scenarios. Additionally, the plaintext can be easily compromised due to the single channel within the medium and one-to-one mapping between input and output. To mitigate these issues, a stable spin-multiplexing disordered metasurface (DM) with numerous polarized transmission channels serves as the scattering medium, and a double-secure procedure with superposition of plaintext and security key achieves two-to-one mapping between input and output. In attack analysis, when the ciphertext, security key, and incident polarization are all correct, the plaintext can be decrypted. This system demonstrates excellent decryption efficiency over extended periods in noisy environments. The DM, functioning as an ultra-stable and active speckle generator, coupled with the double-secure approach, creates a highly secure speckle-based cryptosystem with immense potentials for practical applications.

Structured light enhanced machine learning for fiber bend sensing

The intricate optical distortions that occur when light interacts with complex media, such as few- or multi-mode optical fiber, often appear random in origin and are a fundamental source of error for communication and sensing systems. We propose the use of orbital angular momentum (OAM) feature extraction to mitigate phase-noise and allow for the use of intermodal-coupling as an effective tool for fiber sensing. OAM feature extraction is achieved by passive all-optical OAM demultiplexing, and we demonstrate fiber bend tracking with 94.1% accuracy. Conversely, an accuracy of only 14% was achieved for determining the same bend positions when using a convolutional-neural-network trained with intensity measurements of the output of the fiber. Further, OAM feature extraction used 120 times less information for training compared to intensity image based measurements. This work indicates that structured light enhanced machine learning could be used in a wide range of future sensing technologies.

Non-orthogonal optical multiplexing empowered by deep learning

Orthogonality among channels is a canonical basis for optical multiplexing featured with division multiplexing, which substantially reduce the complexity of signal post-processing in demultiplexing. However, it inevitably imposes an upper limit of capacity for multiplexing. Herein, we report on non-orthogonal optical multiplexing over a multimode fiber (MMF) leveraged by a deep neural network, termed speckle light field retrieval network (SLRnet), where it can learn the complicated mapping relation between multiple non-orthogonal input light field encoded with information and their corresponding single intensity output. As a proof-of-principle experimental demonstration, it is shown that the SLRnet can effectively solve the ill-posed problem of non-orthogonal optical multiplexing over an MMF, where multiple non-orthogonal input signals mediated by the same polarization, wavelength and spatial position can be explicitly retrieved utilizing a single-shot speckle output with fidelity as high as ~ 98%. Our results resemble an important step for harnessing non-orthogonal channels for high capacity optical multiplexing.

Self-supervised dynamic learning for long-term high-fidelity image transmission through unstabilized diffusive media

Multimode fiber (MMF) which supports parallel transmission of spatially distributed information is a promising platform for remote imaging and capacity-enhanced optical communication. However, the variability of the scattering MMF channel poses a challenge for achieving long-term accurate transmission over long distances, of which static optical propagation modeling with calibrated transmission matrix or data-driven learning will inevitably degenerate. In this paper, we present a self-supervised dynamic learning approach that achieves long-term, high-fidelity transmission of arbitrary optical fields through unstabilized MMFs. Multiple networks carrying both long- and short-term memory of the propagation model variations are adaptively updated and ensembled to achieve robust image recovery. We demonstrate >99.9% accuracy in the transmission of 1024 spatial degree-of-freedom over 1 km length MMFs lasting over 1000 seconds. The long-term high-fidelity capability enables compressive encoded transfer of high-resolution video with orders of throughput enhancement, offering insights for artificial intelligence promoted diffusive spatial transmission in practical applications.

Feature decoupled knowledge distillation enabled lightweight image transmission through multimode fibers

Multimode fibers (MMF) show tremendous potential in transmitting high-capacity spatial information. However, the quality of multimode transmission is quite sensitive to inherent scattering characteristics of MMF and almost inevitable external perturbations. Previous research has shown that deep learning may break through this limitation, while deep neural networks are intricately designed with huge computational complexity. In this study, we propose a novel feature decoupled knowledge distillation (KD) framework for lightweight image transmission through MMF. In this framework, the frequency-principle-inspired feature decoupled module significantly improves image transmission quality and the lightweight student model can reach the performance of the sophisticated teacher model through KD. This work represents the first effort, to the best of our knowledge, that successfully applies a KD-based framework for image transmission through scattering media. Experimental results demonstrate that even with up to 93.4% reduction in model computational complexity, we can still achieve averaged Structure Similarity Index Measure (SSIM) of 0.76, 0.85, and 0.90 in Fashion-MNIST, EMNIST, and MNIST images respectively, which are very close to the performance of cumbersome teacher models. This work dramatically reduces the complexity of high-fidelity image transmission through MMF and holds broad prospects for applications in resource-constrained environments and hardware implementations.

High-fidelity multi-channel optical information transmission through scattering media

High-fidelity optical information transmission through strongly scattering media is challenging, but is crucial for the applications such as the free-space optical communication in a haze or fog. Binarizing optical information can somehow suppress the disruptions caused by light scattering. However, this method gives a compromised communication throughput. Here, we propose high-fidelity multiplexing anti-scattering transmission (MAST). MAST encodes multiple bits into a complex-valued pattern, loads the complex-valued pattern to an optical field through modulation, and finally employs a scattering matrix-assisted retrieval technique to reconstruct the original information from the speckle patterns. In our demonstration, we multiplexed three channels and MAST achieved a high-fidelity transmission of 3072 (= 1024× 3) bits data per transmission and average transmission error as small as 0.06%.

Wavefront shaping improves the transparency of the scattering media: a review

Significance

Wavefront shaping (WFS) can compensate for distortions by optimizing the wavefront of the input light or reversing the transmission matrix of the media. It is a promising field of research. A thorough understanding of principles and developments of WFS is important for optical research.

Aim

To provide insight into WFS for researchers who deal with scattering in biomedicine, imaging, and optical communication, our study summarizes the basic principles and methods of WFS and reviews recent progress.

Approach

The basic principles, methods of WFS, and the latest applications of WFS in focusing, imaging, and multimode fiber (MMF) endoscopy are described. The practical challenges and prospects of future development are also discussed.

Results

Data-driven learning-based methods are opening up new possibilities for WFS. High-resolution imaging through MMFs can support small-diameter endoscopy in the future.

Conclusion

The rapid development of WFS over the past decade has shown that the best solution is not to avoid scattering but to find ways to correct it or even use it. WFS with faster speed, more optical modes, and more modulation degrees of freedom will continue to drive exciting developments in various fields.

Metafiber transforming arbitrarily structured light

Structured light has proven useful for numerous photonic applications. However, the current use of structured light in optical fiber science and technology is severely limited by mode mixing or by the lack of optical elements that can be integrated onto fiber end-faces for wavefront engineering, and hence generation of structured light is still handled outside the fiber via bulky optics in free space. We report a metafiber platform capable of creating arbitrarily structured light on the hybrid-order Poincaré sphere. Polymeric metasurfaces, with unleashed height degree of freedom and a greatly expanded 3D meta-atom library, were 3D laser nanoprinted and interfaced with polarization-maintaining single-mode fibers. Multiple metasurfaces were interfaced on the fiber end-faces, transforming the fiber output into different structured-light fields, including cylindrical vector beams, circularly polarized vortex beams, and arbitrary vector field. Our work provides a paradigm for advancing optical fiber science and technology towards fiber-integrated light shaping, which may find important applications in fiber communications, fiber lasers and sensors, endoscopic imaging, fiber lithography, and lab-on-fiber technology.

Spatial pilot-aided fast-adapted framework for stable image transmission over long multi-mode fiber

Multi-mode fiber (MMF) has emerged as a promising platform for spatial information transmission attributed to its high capacity. However, the scattering characteristic and time-varying nature of MMF pose challenges for long-term stable transmission. In this study, we propose a spatial pilot-aided learning framework for MMF image transmission, which effectively addresses these challenges and maintains accurate performance in practical applications. By inserting a few reference image frames into the transmitting image sequence and leveraging a fast-adapt network training scheme, our framework adaptively accommodates to the physical channel variations and enables online model update for continuous transmission. Experimented on 100 m length unstable MMFs, we demonstrate transmission accuracy exceeding 92% over hours, with pilot frame overhead around 2%. Our fast-adapt learning scheme requires training of less than 2% of network parameters and reduces the computation time by 70% compared to conventional tuning approaches. Additionally, we propose two pilot-insertion strategies and elaborately compare their applicability to a wide range of scenarios including continuous transmission, burst transmission and transmission after fiber re-plugging. The proposed spatial pilot-aided fast-adapt framework opens up the possibility for MMF spatial transmission in practical complicated applications.

Multi-channel data transmission through a multimode fiber based on OAM phase encoding

Data transmission based on the transmission matrix method has realized the multiplexing of a large number of orbital angular momentum (OAM) modes under scattering, which encodes the data by modulating the amplitude of the OAM modes. However, this amplitude modulation (amplitude encoding) method has obvious cross talk when the number of output modes is small, resulting in a non-negligible bit error rate. Here, a multi-channel data transmission method based on OAM phase modulation (phase encoding) under scattering is proposed. This method can resist the multiple-scattering effect of multimode fibers and realize accurate data transmission with very few rows of camera pixels for output mode measurement, which is suitable for high-speed data transmission under scattering. Experimentally, we have achieved a bit error rate of less than 0.005% in the data transmission of a color image through a 60 m multimode fiber with only 2 rows of camera pixels for output mode measurement. Experiments also showed that the proposed method has a higher stability than amplitude encoding when the proportion of "1" or "0" in the code changes.

Referenceless characterization of complex media using physics-informed neural networks

In this work, we present a method to characterize the transmission matrices of complex scattering media using a physics-informed, multi-plane neural network (MPNN) without the requirement of a known optical reference field. We use this method to accurately measure the transmission matrix of a commercial multi-mode fiber without the problems of output-phase ambiguity and dark spots, leading to up to 58% improvement in focusing efficiency compared with phase-stepping holography. We demonstrate how our method is significantly more noise-robust than phase-stepping holography and show how it can be generalized to characterize a cascade of transmission matrices, allowing one to control the propagation of light between independent scattering media. This work presents an essential tool for accurate light control through complex media, with applications ranging from classical optical networks, biomedical imaging, to quantum information processing.

Robust real-time imaging through flexible multimode fibers

Conventional endoscopes comprise a bundle of optical fibers, associating one fiber for each pixel in the image. In principle, this can be reduced to a single multimode optical fiber (MMF), the width of a human hair, with one fiber spatial-mode per image pixel. However, images transmitted through a MMF emerge as unrecognizable speckle patterns due to dispersion and coupling between the spatial modes of the fiber. Furthermore, speckle patterns change as the fiber undergoes bending, making the use of MMFs in flexible imaging applications even more complicated. In this paper, we propose a real-time imaging system using flexible MMFs, but which is robust to bending. Our approach does not require access or feedback signal from the distal end of the fiber during imaging. We leverage a variational autoencoder to reconstruct and classify images from the speckles and show that these images can still be recovered when the bend configuration of the fiber is changed to one that was not part of the training set. We utilize a MMF 300 mm long with a 62.5 μm core for imaging [Formula: see text] cm objects placed approximately at 20 cm from the fiber and the system can deal with a change in fiber bend of 50[Formula: see text] and range of movement of 8 cm.

Cylindrical Multimode Waveguides as Focusing Interferometric Systems

Delivery and focusing of radiation requires a variety of optical elements such as waveguides and mirrors or lenses. Heretofore, they were used separately, the former for radiation delivery, the latter for focusing. Here, we show that cylindrical multimode waveguides can both deliver and simultaneously focus radiation, without any external lenses or parabolic mirrors. We develop an analytical, ray-optical model to describe radiation propagation within and after the end of cylindrical multimode waveguides and demonstrate the focusing effect theoretically and experimentally at terahertz frequencies. In the focused spot, located at a distance of several millimeters to a few centimeters away from the waveguide end, typical for focal lengths in optical setups, we achieve a more than 8.4× higher intensity than the cross-sectional average intensity and compress the half-maximum spot area of the incident beam by a factor of >15. Our results represent the first practical realization of a focusing system consisting of only a single cylindrical multimode waveguide, that delivers radiation from one focused spot into another focused spot in free space, with focal distances that are much larger than both the radiation wavelength and the waveguide radius. The results enable design and optimization of cylindrical waveguide-containing systems and demonstrate a precise optical characterization method for cylindrical structures and objects.

High-definition image transmission through dynamically perturbed multimode fiber by a self-attention based neural network

We implement faithful multimode fiber (MMF) image transmission by a self-attention-based neural network. Compared with a real-valued artificial neural network (ANN) based on a convolutional neural network (CNN), our method utilizes a self-attention mechanism to achieve a higher image quality. The enhancement measure (EME) and structural similarity (SSIM) of the dataset collected in the experiment improved by 0.79 and 0.04; the total number of parameters can be reduced by up to 25%. To enhance the robustness of the neural network to MMF bending in image transmission, we use a simulation dataset to prove that the hybrid training method is helpful in MMF transmission of a high-definition image. Our findings may pave the way for simpler and more robust single-MMF image transmission schemes with hybrid training; SSIM on datasets under different disturbances improve by 0.18. This system has the potential to be applied to various high-demand image transmission tasks, such as endoscopy.

Adaptive inverse mapping: a model-free semi-supervised learning approach towards robust imaging through dynamic scattering media

Imaging through scattering media is a useful and yet demanding task since it involves solving for an inverse mapping from speckle images to object images. It becomes even more challenging when the scattering medium undergoes dynamic changes. Various approaches have been proposed in recent years. However, none of them are able to preserve high image quality without either assuming a finite number of sources for dynamic changes, assuming a thin scattering medium, or requiring access to both ends of the medium. In this paper, we propose an adaptive inverse mapping (AIP) method, which requires no prior knowledge of the dynamic change and only needs output speckle images after initialization. We show that the inverse mapping can be corrected through unsupervised learning if the output speckle images are followed closely. We test the AIP method on two numerical simulations: a dynamic scattering system formulated as an evolving transmission matrix and a telescope with a changing random phase mask at a defocused plane. Then we experimentally apply the AIP method to a multimode-fiber-based imaging system with a changing fiber configuration. Increased robustness in imaging is observed in all three cases. AIP method's high imaging performance demonstrates great potential in imaging through dynamic scattering media.

Imaging through scattering media using differential intensity transmission matrices with different Hadamard orderings

A transmission matrix (TM) is a powerful tool for light focusing and imaging through scattering media. For measuring it, the normal way requires establishing a multiple-step phase-shifting interferometer, which makes the TM measurement not only complex and sensitive but also time-consuming. Imaging through scattering media using an intensity TM method can make the setup for TM measurement without the phase-shifting interferometer, thus it is much simple, more stable, and several times faster. Here, based upon a differential intensity TM method, we demonstrated it to do imaging through scattering media using different Hadamard orderings. To accelerate the TM measuring speed while degrading as little as possible of the imaging quality, a relatively reasonable strategy to plan Hadamard orderings for the TM measurement is designed since it can suggest us to preferentially measure the components in TM that are more important to the imaging quality. Thanks to the different Hadamard orderings, their influences on the imaging quality at different measuring ratios are investigated, thus an optimal measuring ordering for accelerating the TM measurement can be obtained, while only sacrificing as little as possible of the image fidelity. Simulations and experiments verify the effectiveness of the proposed method.

Speckle classification of a multimode fiber based on Inception V3

Multimode optical fiber plays an important role in endoscope miniaturization. With the development of deep learning and machine learning, neural networks can be used to identify and classify speckle patterns obtained at the fiber output. Based on the speckle pattern of a HERLEV dataset cell image transmitted by a multimode fiber, this paper studies the recognition accuracy of various types of speckle by a support vector machine, K-nearest neighbor, and convolutional neural network (Inception V3). Meanwhile, we propose an image classification optimization algorithm based on improved Inception V3. The experimental results show that the improved algorithm model is better than the traditional machine learning method; the accuracy rate is 97.92%, which effectively improves the performance of the pathological cell diagnosis deep learning model and lays a theoretical and practical foundation for further clinical application.

Speckle-Based Optical Cryptosystem and its Application for Human Face Recognition via Deep Learning

Face recognition has become ubiquitous for authentication or security purposes. Meanwhile, there are increasing concerns about the privacy of face images, which are sensitive biometric data and should be protected. Software-based cryptosystems are widely adopted to encrypt face images, but the security level is limited by insufficient digital secret key length or computing power. Hardware-based optical cryptosystems can generate enormously longer secret keys and enable encryption at light speed, but most reported optical methods, such as double random phase encryption, are less compatible with other systems due to system complexity. In this study, a plain yet highly efficient speckle-based optical cryptosystem is proposed and implemented. A scattering ground glass is exploited to generate physical secret keys of 17.2 gigabit length and encrypt face images via seemingly random optical speckles at light speed. Face images can then be decrypted from random speckles by a well-trained decryption neural network, such that face recognition can be realized with up to 98% accuracy. Furthermore, attack analyses are carried out to show the cryptosystem's security. Due to its high security, fast speed, and low cost, the speckle-based optical cryptosystem is suitable for practical applications and can inspire other high-security cryptosystems.

Physics-based neural network for non-invasive control of coherent light in scattering media

Optical imaging through complex media, such as biological tissues or fog, is challenging due to light scattering. In the multiple scattering regime, wavefront shaping provides an effective method to retrieve information; it relies on measuring how the propagation of different optical wavefronts are impacted by scattering. Based on this principle, several wavefront shaping techniques were successfully developed, but most of them are highly invasive and limited to proof-of-principle experiments. Here, we propose to use a neural network approach to non-invasively characterize and control light scattering inside the medium and also to retrieve information of hidden objects buried within it. Unlike most of the recently-proposed approaches, the architecture of our neural network with its layers, connected nodes and activation functions has a true physical meaning as it mimics the propagation of light in our optical system. It is trained with an experimentally-measured input/output dataset built from a series of incident light patterns and corresponding camera snapshots. We apply our physics-based neural network to a fluorescence microscope in epi-configuration and demonstrate its performance through numerical simulations and experiments. This flexible method can include physical priors and we show that it can be applied to other systems as, for example, non-linear or coherent contrast mechanisms.

Interplay of Thermalization and Strong Disorder: Wave Turbulence Theory, Numerical Simulations, and Experiments in Multimode Optical Fibers

We address the problem of thermalization in the presence of a time-dependent disorder in the framework of the nonlinear Schrödinger (or Gross-Pitaevskii) equation with a random potential. The thermalization to the Rayleigh-Jeans distribution is driven by the nonlinearity. On the other hand, the structural disorder is responsible for a relaxation toward the homogeneous equilibrium distribution (particle equipartition), which thus inhibits thermalization (energy equipartition). On the basis of the wave turbulence theory, we derive a kinetic equation that accounts for the presence of strong disorder. The theory unveils the interplay of disorder and nonlinearity. It unexpectedly reveals that a nonequilibrium process of condensation and thermalization can take place in the regime where disorder effects dominate over nonlinear effects. We validate the theory by numerical simulations of the nonlinear Schrödinger equation and the derived kinetic equation, which are found in quantitative agreement without using any adjustable parameter. Experiments realized in multimode optical fibers with an applied external stress evidence the process of thermalization in the presence of strong disorder.

Learning to sense three-dimensional shape deformation of a single multimode fiber

Optical fiber bending, deformation or shape sensing are important measurement technologies and have been widely deployed in various applications including healthcare, structural monitoring and robotics. However, existing optical fiber bending sensors require complex sensor structures and interrogation systems. Here, inspired by the recent renewed interest in information-rich multimode optical fibers, we show that the multimode fiber (MMF) output speckles contain the three-dimensional (3D) geometric shape information of the MMF itself. We demonstrate proof-of-concept 3D multi-point deformation sensing via a single multimode fiber by using k-nearest neighbor (KNN) machine learning algorithm, and achieve a classification accuracy close to 100%. Our results show that a single MMF based deformation sensor is excellent in terms of system simplicity, resolution and sensitivity, and can be a promising candidate in deformation monitoring or shape-sensing applications.

A Review of Optical Neural Networks

With the continuous miniaturization of conventional integrated circuits, obstacles such as excessive cost, increased resistance to electronic motion, and increased energy consumption are gradually slowing down the development of electrical computing and constraining the application of deep learning. Optical neuromorphic computing presents various opportunities and challenges compared with the realm of electronics. Algorithms running on optical hardware have the potential to meet the growing computational demands of deep learning and artificial intelligence. Here, we review the development of optical neural networks and compare various research proposals. We focus on fiber-based neural networks. Finally, we describe some new research directions and challenges.

Simultaneous illumination and imaging based on a single multimode fiber

Due to the small core diameter, a single-core multimode fiber (MMF) has been extensively investigated for endoscopic imaging. However, an extra light path is always utilized for illumination in MMF imaging system, which takes more space and is inapplicable in practical endoscopy imaging. In order to make the imaging system more practical and compact, we proposed a dual-function MMF imaging system, which can simultaneously transmit the illumination light and the images through the same imaging fiber. Meanwhile, a new deep learning-based encoder-decoder network with full-connected (FC) layers was designed for image reconstruction. We conducted an experiment of transmitting images via a 1.6 m long MMF to verify the effectiveness of the dual-function MMF imaging system. The experimental results show that the proposed network achieves the best reconstruction performance compared with the other four networks on different datasets. Besides, it is worth mentioning that the cropped speckle patterns can still be used to reconstruct the original images, which helps to reduce the computing complexity significantly. We also demonstrated the ability of cross-domain generalization of the proposed network. The proposed system shows the potential for more compact endoscopic imaging without external illumination.

Complex amplitude field reconstruction in atmospheric turbulence based on deep learning

In this paper, we use deep neural networks (DNNs) to simultaneously reconstruct the amplitude and phase information of the complex light field transmitted in atmospheric turbulence based on deep learning. The results of amplitude and phase reconstruction by four different training methods are compared comprehensively. The obtained results indicate that the training method that can more accurately reconstruct the complex amplitude field is to input the amplitude and phase pattern pairs into the neural network as two channels to train the model.

Machine learning for sensing with a multimode exposed core fiber specklegram sensor

Fiber specklegram sensors (FSSs) traditionally use statistical methods to analyze specklegrams obtained from fibers for sensing purposes, but can suffer from limitations such as vulnerability to noise and lack of dynamic range. In this paper we demonstrate that deep learning improves the analysis of specklegrams for sensing, which we show here for both air temperature and water immersion length measurements. Two deep neural networks (DNNs); a convolutional neural network and a multi-layer perceptron network, are used and compared to a traditional correlation technique on data obtained from a multimode fiber exposed-core fiber. The ability for the DNNs to be trained against a random noise source such as specklegram translations is also demonstrated.

All-fiber high-speed image detection enabled by deep learning

Ultra-high-speed imaging serves as a foundation for modern science. While in biomedicine, optical-fiber-based endoscopy is often required for in vivo applications, the combination of high speed with the fiber endoscopy, which is vital for exploring transient biomedical phenomena, still confronts some challenges. We propose all-fiber imaging at high speeds, which is achieved based on the transformation of two-dimensional spatial information into one-dimensional temporal pulsed streams by leveraging high intermodal dispersion in a multimode fiber. Neural networks are trained to reconstruct images from the temporal waveforms. It can not only detect content-aware images with high quality, but also detect images of different kinds from the training images with slightly reduced quality. The fiber probe can detect micron-scale objects with a high frame rate (15.4 Mfps) and large frame depth (10,000). This scheme combines high speeds with high mechanical flexibility and integration and may stimulate future research exploring various phenomena in vivo.

Deep learning in optical metrology: a review

With the advances in scientific foundations and technological implementations, optical metrology has become versatile problem-solving backbones in manufacturing, fundamental research, and engineering applications, such as quality control, nondestructive testing, experimental mechanics, and biomedicine. In recent years, deep learning, a subfield of machine learning, is emerging as a powerful tool to address problems by learning from data, largely driven by the availability of massive datasets, enhanced computational power, fast data storage, and novel training algorithms for the deep neural network. It is currently promoting increased interests and gaining extensive attention for its utilization in the field of optical metrology. Unlike the traditional "physics-based" approach, deep-learning-enabled optical metrology is a kind of "data-driven" approach, which has already provided numerous alternative solutions to many challenging problems in this field with better performances. In this review, we present an overview of the current status and the latest progress of deep-learning technologies in the field of optical metrology. We first briefly introduce both traditional image-processing algorithms in optical metrology and the basic concepts of deep learning, followed by a comprehensive review of its applications in various optical metrology tasks, such as fringe denoising, phase retrieval, phase unwrapping, subset correlation, and error compensation. The open challenges faced by the current deep-learning approach in optical metrology are then discussed. Finally, the directions for future research are outlined.

Confocal 3D reflectance imaging through multimode fiber without wavefront shaping

Imaging through optical multimode fiber (MMF) has the potential to enable hair-thin endoscopes that reduce the invasiveness of imaging deep inside tissues and organs. Active wavefront shaping and fluorescent labeling have recently been exploited to overcome modal scrambling and enable MMF imaging. Here, we present a computational approach that circumvents the need for active wavefront control and exogenous fluorophores. We demonstrate the reconstruction of depth-gated confocal images through MMF using a raster-scanned, focused input illumination at the fiber proximal end. We show the compatibility of this approach with quantitative phase, dark-field, and polarimetric imaging. Computational imaging through MMF opens a new pathway for minimally invasive imaging in medical diagnosis and biological investigations.

Biomimetic apposition compound eye fabricated using microfluidic-assisted 3D printing

After half a billion years of evolution, arthropods have developed sophisticated compound eyes with extraordinary visual capabilities that have inspired the development of artificial compound eyes. However, the limited 2D nature of most traditional fabrication techniques makes it challenging to directly replicate these natural systems. Here, we present a biomimetic apposition compound eye fabricated using a microfluidic-assisted 3D-printing technique. Each microlens is connected to the bottom planar surface of the eye via intracorporal, zero-crosstalk refractive-index-matched waveguides to mimic the rhabdoms of a natural eye. Full-colour wide-angle panoramic views and position tracking of a point source are realized by placing the fabricated eye directly on top of a commercial imaging sensor. As a biomimetic analogue to naturally occurring compound eyes, the eye's full-colour 3D to 2D mapping capability has the potential to enable a wide variety of applications from improving endoscopic imaging to enhancing machine vision for facilitating human-robot interactions.

Compressively sampling the optical transmission matrix of a multimode fibre

The measurement of the optical transmission matrix (TM) of an opaque material is an advanced form of space-variant aberration correction. Beyond imaging, TM-based methods are emerging in a range of fields, including optical communications, micro-manipulation, and computing. In many cases, the TM is very sensitive to perturbations in the configuration of the scattering medium it represents. Therefore, applications often require an up-to-the-minute characterisation of the fragile TM, typically entailing hundreds to thousands of probe measurements. Here, we explore how these measurement requirements can be relaxed using the framework of compressive sensing, in which the incorporation of prior information enables accurate estimation from fewer measurements than the dimensionality of the TM we aim to reconstruct. Examples of such priors include knowledge of a memory effect linking the input and output fields, an approximate model of the optical system, or a recent but degraded TM measurement. We demonstrate this concept by reconstructing the full-size TM of a multimode fibre supporting 754 modes at compression ratios down to ∼5% with good fidelity. We show that in this case, imaging is still possible using TMs reconstructed at compression ratios down to ∼1% (eight probe measurements). This compressive TM sampling strategy is quite general and may be applied to a variety of other scattering samples, including diffusers, thin layers of tissue, fibre optics of any refractive profile, and reflections from opaque walls. These approaches offer a route towards the measurement of high-dimensional TMs either quickly or with access to limited numbers of measurements.

Image reconstruction through a multimode fiber with a simple neural network architecture

Multimode fibers (MMFs) have the potential to carry complex images for endoscopy and related applications, but decoding the complex speckle patterns produced by mode-mixing and modal dispersion in MMFs is a serious challenge. Several groups have recently shown that convolutional neural networks (CNNs) can be trained to perform high-fidelity MMF image reconstruction. We find that a considerably simpler neural network architecture, the single hidden layer dense neural network, performs at least as well as previously-used CNNs in terms of image reconstruction fidelity, and is superior in terms of training time and computing resources required. The trained networks can accurately reconstruct MMF images collected over a week after the cessation of the training set, with the dense network performing as well as the CNN over the entire period.

Inference in artificial intelligence with deep optics and photonics

Artificial intelligence tasks across numerous applications require accelerators for fast and low-power execution. Optical computing systems may be able to meet these domain-specific needs but, despite half a century of research, general-purpose optical computing systems have yet to mature into a practical technology. Artificial intelligence inference, however, especially for visual computing applications, may offer opportunities for inference based on optical and photonic systems. In this Perspective, we review recent work on optical computing for artificial intelligence applications and discuss its promise and challenges.

Reciprocity-induced symmetry in the round-trip transmission through complex systems

Reciprocity is a fundamental principle of wave physics and directly relates to the symmetry in the transmission through a system when interchanging the input and output. The coherent transmission matrix (TM) is a convenient method to characterize wave transmission through general media. Here, we demonstrate the optical reciprocal nature of complex media by exploring their TM properties. We measured phase-corrected TMs of forward and round-trip propagation in a single polarization state through a looped 1 m-long step-index optical multimode fiber (MMF) to experimentally verify a transpose relationship between the forward and backward transmission. This symmetry impedes straightforward MMF calibration from proximal measurements of the round-trip TM. Furthermore, we show how focusing through the MMF with digital optical phase conjugation is compromised by system loss since time reversibility relies on power conservation. These insights may inform the development of new imaging techniques through complex media and coherent control of waves in photonic systems.

Harnessing a multi-dimensional fibre laser using genetic wavefront shaping

The multi-dimensional laser is a fascinating platform not only for the discovery and understanding of new higher-dimensional coherent lightwaves but also for the frontier study of the complex three-dimensional (3D) nonlinear dynamics and solitary waves widely involved in physics, chemistry, biology and materials science. Systemically controlling coherent lightwave oscillation in multi-dimensional lasers, however, is challenging and has largely been unexplored; yet, it is crucial for both designing 3D coherent light fields and unveiling any underlying nonlinear complexities. Here, for the first time, we genetically harness a multi-dimensional fibre laser using intracavity wavefront shaping technology such that versatile lasing characteristics can be manipulated. We demonstrate that the output power, mode profile, optical spectrum and mode-locking operation can be genetically optimized by appropriately designing the objective function of the genetic algorithm. It is anticipated that this genetic and systematic intracavity control technology for multi-dimensional lasers will be an important step for obtaining high-performance 3D lasing and presents many possibilities for exploring multi-dimensional nonlinear dynamics and solitary waves that may enable new applications.

Seeing through multimode fibers with real-valued intensity transmission matrices

Image transmission through multimode optical fibers has been an area of immense interests driven by the demand for miniature endoscopes in biomedicine and higher speed and capacity in telecommunications. Conventionally, a complex-valued transmission matrix is obtained experimentally to link the input and output light fields of a multimode fiber for image retrieval, which complicates the experimental setup and increases the computational complexity. Here, we report a simple and high-speed method for image retrieval based on our demonstration of a pseudo-linearity between the input and output light intensity distributions of multimode fibers. We studied the impact of several key parameters to image retrieval, including image pixel count, fiber core diameter and numerical aperture. We further demonstrated with experiments and numerical simulations that a wide variety of input binary and gray scale images could be faithfully retrieved from the corresponding output speckle patterns. Thus, it promises to be useful for highly miniaturized endoscopy in biomedicine and spatial-mode-division multiplexing in telecommunications.

One-dimensional sensor learns to sense three-dimensional space

A sensor system with ultra-high sensitivity, high resolution, rapid response time, and a high signal-to-noise ratio can produce raw data that is exceedingly rich in information, including signals that have the appearances of "noise". The "noise" feature directly correlates to measurands in orthogonal dimensions, and are simply manifestations of the off-diagonal elements of 2^nd-order tensors that describe the spatial anisotropy of matter in physical structures and spaces. The use of machine learning techniques to extract useful meanings from the rich information afforded by ultra-sensitive one-dimensional sensors may offer the potential for probing mundane events for novel embedded phenomena. Inspired by our very recent invention of ultra-sensitive optical-based inclinometers, this work aims to answer a transformative question for the first time: can a single-dimension point sensor with ultra-high sensitivity, fidelity, and signal-to-noise ratio identify an arbitrary mechanical impact event in three-dimensional space? This work is expected to inspire researchers in the fields of sensing and measurement to promote the development of a new generation of powerful sensors or sensor networks with expanded functionalities and enhanced intelligence, which may provide rich n-dimensional information, and subsequently, data-driven insights into significant problems.

Speckle-Correlation Scattering Matrix Approaches for Imaging and Sensing through Turbidity

The development of optical and computational techniques has enabled imaging without the need for traditional optical imaging systems. Modern lensless imaging techniques overcome several restrictions imposed by lenses, while preserving or even surpassing the capability of lens-based imaging. However, existing lensless methods often rely on a priori information about objects or imaging conditions. Thus, they are not ideal for general imaging purposes. The recent development of the speckle-correlation scattering matrix (SSM) techniques facilitates new opportunities for lensless imaging and sensing. In this review, we present the fundamentals of SSM methods and highlight recent implementations for holographic imaging, microscopy, optical mode demultiplexing, and quantification of the degree of the coherence of light. We conclude with a discussion of the potential of SSM and future research directions.

Route to Intelligent Imaging Reconstruction via Terahertz Nonlinear Ghost Imaging

Terahertz (THz) imaging is a rapidly emerging field, thanks to many potential applications in diagnostics, manufacturing, medicine and material characterisation. However, the relatively coarse resolution stemming from the large wavelength limits the deployment of THz imaging in micro- and nano-technologies, keeping its potential benefits out-of-reach in many practical scenarios and devices. In this context, single-pixel techniques are a promising alternative to imaging arrays, in particular when targeting subwavelength resolutions. In this work, we discuss the key advantages and practical challenges in the implementation of time-resolved nonlinear ghost imaging (TIMING), an imaging technique combining nonlinear THz generation with time-resolved time-domain spectroscopy detection. We numerically demonstrate the high-resolution reconstruction of semi-transparent samples, and we show how the Walsh-Hadamard reconstruction scheme can be optimised to significantly reduce the reconstruction time. We also discuss how, in sharp contrast with traditional intensity-based ghost imaging, the field detection at the heart of TIMING enables high-fidelity image reconstruction via low numerical-aperture detection. Even more striking-and to the best of our knowledge, an issue never tackled before-the general concept of "resolution" of the imaging system as the "smallest feature discernible" appears to be not well suited to describing the fidelity limits of nonlinear ghost-imaging systems. Our results suggest that the drop in reconstruction accuracy stemming from non-ideal detection conditions is complex and not driven by the attenuation of high-frequency spatial components (i.e., blurring) as in standard imaging. On the technological side, we further show how achieving efficient optical-to-terahertz conversion in extremely short propagation lengths is crucial regarding imaging performance, and we propose low-bandgap semiconductors as a practical framework to obtain THz emission from quasi-2D structures, i.e., structure in which the interaction occurs on a deeply subwavelength scale. Our results establish a comprehensive theoretical and experimental framework for the development of a new generation of terahertz hyperspectral imaging devices.

Averaging speckle patterns to improve the robustness of compressive multimode fiber imaging against fiber bend

Fiber bend is a major challenge of multimode fiber (MMF) imaging. More robustness against fiber bend is demonstrated in compressive MMF imaging using mean speckle patterns captured at multiple potential bending configurations beforehand, rather than sticking to single patterns at initial configuration. Experiments demonstrate an overall quality improvement on recovered images than previous work, which is important for robust endoscopic application.

Photon-limited imaging through scattering medium based on deep learning

Imaging under ultra-weak light conditions is affected by Poisson noise heavily. The problem becomes worse if a scattering media is present in the optical path. Speckle patterns detected under ultra-weak light condition carry very little information which makes it difficult to reconstruct the image. Off-the-shelf methods are no longer available in this condition. In this paper, we experimentally demonstrate the use of a deep learning network to reconstruct images through scattering media under ultra-weak light illumination. The weak light limitation of this method is analyzed. The random Poisson detection under weak light condition obtains partial information of the object. Based on this property, we demonstrated better performance of our method by enlarging the training dataset with multiple detections of the speckle patterns. Our results demonstrate that our approach can reconstruct images through scattering media from close to 1 detected signal photon per pixel (PPP) per image.