Memory effect assisted imaging through multimode optical fibres

"There's plenty of room at the bottom": deep brain imaging with holographic endo-microscopy

Significance

Over more than 300 years, microscopic imaging keeps providing fundamental insights into the mechanisms of living organisms. Seeing microscopic structures beyond the reach of free-space light-based microscopy, however, requires dissection of the tissue-an intervention seriously disturbing its physiological functions. The hunt for low-invasiveness tools has led a growing community of physicists and engineers into the realm of complex media photonics. One of its activities represents exploiting multimode optical fibers (MMFs) as ultra-thin endoscopic probes. Employing wavefront shaping, these tools only recently facilitated the first peeks at cells and their sub-cellular compartments at the bottom of the mouse brain with the impact of micro-scale tissue damage.

Aim

Here, we aim to highlight advances in MMF-based holographic endo-microscopy facilitating microscopic imaging throughout the whole depth of the mouse brain.

Approach

We summarize the important technical and methodological prerequisites for stabile high-resolution imaging in vivo.

Results

We showcase images of the microscopic building blocks of brain tissue, including neurons, neuronal processes, vessels, intracellular calcium signaling, and red blood cell velocity in individual vessels.

Conclusions

This perspective article helps to understand the complexity behind the technology of holographic endo-microscopy, summarizes its recent advances and challenges, and stimulates the mind of the reader for further exploitation of this tool in the neuroscience research.

Roadmap on computational methods in optical imaging and holography [invited]

Computational methods have been established as cornerstones in optical imaging and holography in recent years. Every year, the dependence of optical imaging and holography on computational methods is increasing significantly to the extent that optical methods and components are being completely and efficiently replaced with computational methods at low cost. This roadmap reviews the current scenario in four major areas namely incoherent digital holography, quantitative phase imaging, imaging through scattering layers, and super-resolution imaging. In addition to registering the perspectives of the modern-day architects of the above research areas, the roadmap also reports some of the latest studies on the topic. Computational codes and pseudocodes are presented for computational methods in a plug-and-play fashion for readers to not only read and understand but also practice the latest algorithms with their data. We believe that this roadmap will be a valuable tool for analyzing the current trends in computational methods to predict and prepare the future of computational methods in optical imaging and holography.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00340-024-08280-3.

Demixing fluorescence time traces transmitted by multimode fibers

Optical methods based on thin multimode fibers (MMFs) are promising tools for measuring neuronal activity in deep brain regions of freely moving mice thanks to their small diameter. However, current methods are limited: while fiber photometry provides only ensemble activity, imaging techniques using of long multimode fibers are very sensitive to bending and have not been applied to unrestrained rodents yet. Here, we demonstrate the fundamentals of a new approach using a short MMF coupled to a miniscope. In proof-of-principle in vitro experiments, we disentangled spatio-temporal fluorescence signals from multiple fluorescent sources transmitted by a thin (200 µm) and short (8 mm) MMF, using a general unconstrained non-negative matrix factorization algorithm directly on the raw video data. Furthermore, we show that low-cost open-source miniscopes have sufficient sensitivity to image the same fluorescence patterns seen in our proof-of-principle experiment, suggesting a new avenue for novel minimally invasive deep brain studies using multimode fibers in freely behaving mice.

Perspectives on endoscopic functional photoacoustic microscopy

Endoscopy, enabling high-resolution imaging of deep tissues and internal organs, plays an important role in basic research and clinical practice. Recent advances in photoacoustic microscopy (PAM), demonstrating excellent capabilities in high-resolution functional imaging, have sparked significant interest in its integration into the field of endoscopy. However, there are challenges in achieving functional PAM in the endoscopic setting. This Perspective article discusses current progress in the development of endoscopic PAM and the challenges related to functional measurements. Then, it points out potential directions to advance endoscopic PAM for functional imaging by leveraging fiber optics, microfabrication, optical engineering, and computational approaches. Finally, it highlights emerging opportunities for functional endoscopic PAM in basic and translational biomedicine.

High-fidelity and high-speed wavefront shaping by leveraging complex media

High-precision light manipulation is crucial for delivering information through complex media. However, existing spatial light modulation devices face a fundamental speed-fidelity tradeoff. Digital micromirror devices have emerged as a promising candidate for high-speed wavefront shaping but at the cost of compromised fidelity due to the limited control degrees of freedom. Here, we leverage the sparse-to-random transformation through complex media to overcome the dimensionality limitation of spatial light modulation devices. We demonstrate that pattern compression by sparsity-constrained wavefront optimization allows sparse and robust wavefront representations in complex media, improving the projection fidelity without sacrificing frame rate, hardware complexity, or optimization time. Our method is generalizable to different pattern types and complex media, supporting consistent performance with up to 89% and 126% improvements in projection accuracy and speckle suppression, respectively. The proposed optimization framework could enable high-fidelity high-speed wavefront shaping through different scattering media and platforms without changes to the existing holographic setups, facilitating a wide range of physics and real-world applications.

Mode-locking and wavelength-tuning of a NPR fiber laser based on optical speckle

Passively mode-locked fiber lasers based on nonlinear polarization rotation (NPR) have been widely used due to their ability to produce short pulses with high peak power. Nevertheless, environmental perturbations can influence the mode-locked state, making it a challenge for the practical implementation. Therefore, researchers are searching for assessment criteria to quickly assist and maintain mode-locking of NPR fiber lasers. Speckle patterns containing spectral information can be generated when the laser transmits through a scattering medium, which can serve as indicators of the mode-locked state. The mode-locked regions are confined to the area close to the minimum texture contrast of speckle patterns. Based on these characteristics, we manually simulate the automatic mode-locking (AML). In addition, we utilize convolutional neural networks (CNNs) to recognize speckle patterns of wavelength tunable lasers and determine the center wavelength.

Memory-less scattering imaging with ultrafast convolutional optical neural networks

The optical memory effect in complex scattering media including turbid tissue and speckle layers has been a critical foundation for macroscopic and microscopic imaging methods. However, image reconstruction from strong scattering media without the optical memory effect has not been achieved. Here, we demonstrate image reconstruction through scattering layers where no optical memory effect exists, by developing a multistage convolutional optical neural network (ONN) integrated with multiple parallel kernels operating at the speed of light. Training this Fourier optics-based, parallel, one-step convolutional ONN with the strong scattering process for direct feature extraction, we achieve memory-less image reconstruction with a field of view enlarged by a factor up to 271. This device is dynamically reconfigurable for ultrafast multitask image reconstruction with a computational power of 1.57 peta-operations per second (POPS). Our achievement establishes an ultrafast and high energy-efficient optical machine learning platform for graphic processing.

Ptychographic lensless coherent endomicroscopy through a flexible fiber bundle

Conventional fiber-bundle-based endoscopes allow minimally invasive imaging through flexible multi-core fiber (MCF) bundles by placing a miniature lens at the distal tip and using each core as an imaging pixel. In recent years, lensless imaging through MCFs was made possible by correcting the core-to-core phase distortions pre-measured in a calibration procedure. However, temporally varying wavefront distortions, for instance, due to dynamic fiber bending, pose a challenge for such approaches. Here, we demonstrate a coherent lensless imaging technique based on intensity-only measurements insensitive to core-to-core phase distortions. We leverage a ptychographic reconstruction algorithm to retrieve the phase and amplitude profiles of reflective objects placed at a distance from the fiber tip, using as input a set of diffracted intensity patterns reflected from the object when the illumination is scanned over the MCF cores. Our approach thus utilizes an acquisition process equivalent to confocal microendoscopy, only replacing the single detector with a camera.

Exploiting holographically encoded variance to transmit labelled images through a multimode optical fiber

Artificial intelligence has emerged as promising tool to decode an image transmitted through a multimode fiber (MMF) by applying deep learning techniques. By transmitting thousands of images through the MMF, deep neural networks (DNNs) are able to decipher the seemingly random output speckle patterns and unveil the intrinsic input-output relationship. High fidelity reconstruction is obtained for datasets with a large degree of homogeneity, which underutilizes the capacity of the combined MMF-DNN system. Here, we show that holographic modulation can encode an additional layer of variance on the output speckle pattern, improving the overall transmissive capabilities of the system. Operatively, we have implemented this by adding a holographic label to the original dataset and injecting the resulting phase image into the fiber facet through a Fourier transform lens. The resulting speckle pattern dataset can be clustered primarily by holographic label, and can be reconstructed without loss of fidelity. As an application, we describe how color images may be segmented into RGB components and each color component may then be labelled by distinct hologram. A ResUNet architecture was then used to decode each class of speckle patterns and reconstruct the color image without the need for temporal synchronization between sender and receiver.

Wide-field color imaging through multimode fiber with single wavelength illumination: plug-and-play approach

Multimode fiber (MMF) is extensively studied for its ability to transmit light modes in parallel, potentially minimizing optical fiber size in imaging. However, current research predominantly focuses on grayscale imaging, with limited attention to color studies. Existing colorization methods often involve costly white light lasers or multiple light sources, increasing optical system expenses and space. To achieve wide-field color images with typical monochromatic illumination MMF imaging system, we proposed a data-driven "colorization" approach and a neural network called SpeckleColorNet, merging U-Net and conditional GAN (cGAN) architectures, trained by a combined loss function. This approach, demonstrated on a 2-meter MMF system with single-wavelength illumination and the Peripheral Blood Cell (PBC) dataset, outperforms grayscale imaging and alternative colorization methods in readability, definition, detail, and accuracy. Our method aims to integrate MMF into clinical medicine and industrial monitoring, offering cost-effective high-fidelity color imaging. It serves as a plug-and-play replacement for conventional grayscale algorithms in MMF systems, eliminating the need for additional hardware.

Calibration-free quantitative phase imaging in multi-core fiber endoscopes using end-to-end deep learning

Quantitative phase imaging (QPI) through multi-core fibers (MCFs) has been an emerging in vivo label-free endoscopic imaging modality with minimal invasiveness. However, the computational demands of conventional iterative phase retrieval algorithms have limited their real-time imaging potential. We demonstrate a learning-based MCF phase imaging method that significantly reduced the phase reconstruction time to 5.5 ms, enabling video-rate imaging at 181 fps. Moreover, we introduce an innovative optical system that automatically generated the first, to the best of our knowledge, open-source dataset tailored for MCF phase imaging, comprising 50,176 paired speckles and phase images. Our trained deep neural network (DNN) demonstrates a robust phase reconstruction performance in experiments with a mean fidelity of up to 99.8%. Such an efficient fiber phase imaging approach can broaden the applications of QPI in hard-to-reach areas.

Wavefront shaping improves the transparency of the scattering media: a review

Significance

Wavefront shaping (WFS) can compensate for distortions by optimizing the wavefront of the input light or reversing the transmission matrix of the media. It is a promising field of research. A thorough understanding of principles and developments of WFS is important for optical research.

Aim

To provide insight into WFS for researchers who deal with scattering in biomedicine, imaging, and optical communication, our study summarizes the basic principles and methods of WFS and reviews recent progress.

Approach

The basic principles, methods of WFS, and the latest applications of WFS in focusing, imaging, and multimode fiber (MMF) endoscopy are described. The practical challenges and prospects of future development are also discussed.

Results

Data-driven learning-based methods are opening up new possibilities for WFS. High-resolution imaging through MMFs can support small-diameter endoscopy in the future.

Conclusion

The rapid development of WFS over the past decade has shown that the best solution is not to avoid scattering but to find ways to correct it or even use it. WFS with faster speed, more optical modes, and more modulation degrees of freedom will continue to drive exciting developments in various fields.

Spatially resolved readout of a Fabry-Perot ultrasound sensor interrogated through a multimode optical fiber using wavefront shaping

The spatially resolved interrogation of a Fabry-Perot ultrasound sensor using a laser beam focused through a multimode fiber is demonstrated. To scan the beam across the sensor as required to read it out, optical wavefront shaping was employed to compensate for the scrambling of light in the fiber. By providing a means to map ultrasound through inexpensive, lightweight fibers, this could lead to new ultrasonic and photoacoustic imaging systems, such as endoscopes and flexible handheld probes.

Imaging through a square multimode fiber by scanning focused spots with the memory effect

The existence of a shift-shift memory effect in square waveguides, whereby any translation of the input field induces translations in the output field in four symmetrical directions, has been previously observed by correlation measurements. Here we demonstrate that this memory effect is also observed in real space and can be put to use for imaging purposes. First, a focus is created at the output of a square-core multimode fiber, by wavefront shaping based on feedback from a guide-star. Then, because of the memory effect, four symmetrical spots can be scanned at the fiber output by shifting the wavefront at the fiber input. We demonstrate that this property can be exploited to perform fluorescence imaging through the multimode fiber, without requiring the measurement of a transmission matrix.

Robust real-time imaging through flexible multimode fibers

Conventional endoscopes comprise a bundle of optical fibers, associating one fiber for each pixel in the image. In principle, this can be reduced to a single multimode optical fiber (MMF), the width of a human hair, with one fiber spatial-mode per image pixel. However, images transmitted through a MMF emerge as unrecognizable speckle patterns due to dispersion and coupling between the spatial modes of the fiber. Furthermore, speckle patterns change as the fiber undergoes bending, making the use of MMFs in flexible imaging applications even more complicated. In this paper, we propose a real-time imaging system using flexible MMFs, but which is robust to bending. Our approach does not require access or feedback signal from the distal end of the fiber during imaging. We leverage a variational autoencoder to reconstruct and classify images from the speckles and show that these images can still be recovered when the bend configuration of the fiber is changed to one that was not part of the training set. We utilize a MMF 300 mm long with a 62.5 μm core for imaging [Formula: see text] cm objects placed approximately at 20 cm from the fiber and the system can deal with a change in fiber bend of 50[Formula: see text] and range of movement of 8 cm.

Speckle-correlation imaging through a kaleidoscopic multimode fiber

Speckle-correlation imaging techniques are widely used for noninvasive imaging through complex scattering media. While light propagation through multimode fibers and scattering media share many analogies, reconstructing images through multimode fibers from speckle correlations remains an unsolved challenge. Here, we exploit a kaleidoscopic memory effect emerging in square-core multimode fibers and demonstrate fluorescence imaging with no prior knowledge on the fiber. Experimentally, our approach simply requires to translate random speckle patterns at the input of a square-core fiber and to measure the resulting fluorescence intensity with a bucket detector. The image of the fluorescent object is then reconstructed from the autocorrelation of the measured signal by solving an inverse problem. This strategy does not require the knowledge of the fragile deterministic relation between input and output fields, which makes it promising for the development of flexible minimally invasive endoscopes.

110 μm thin endo-microscope for deep-brain in vivo observations of neuronal connectivity, activity and blood flow dynamics

Light-based in-vivo brain imaging relies on light transport over large distances of highly scattering tissues. Scattering gradually reduces imaging contrast and resolution, making it difficult to reach structures at greater depths even with the use of multiphoton techniques. To reach deeper, minimally invasive endo-microscopy techniques have been established. These most commonly exploit graded-index rod lenses and enable a variety of modalities in head-fixed and freely moving animals. A recently proposed alternative is the use of holographic control of light transport through multimode optical fibres promising much less traumatic application and superior imaging performance. We present a 110 μm thin laser-scanning endo-microscope based on this prospect, enabling in-vivo volumetric imaging throughout the whole depth of the mouse brain. The instrument is equipped with multi-wavelength detection and three-dimensional random access options, and it performs at lateral resolution below 1 μm. We showcase various modes of its application through the observations of fluorescently labelled neurones, their processes and blood vessels. Finally, we demonstrate how to exploit the instrument to monitor calcium signalling of neurones and to measure blood flow velocity in individual vessels at high speeds.

High-gain and high-speed wavefront shaping through scattering media

Wavefront shaping (WFS) is emerging as a promising tool for controlling and focusing light in complex scattering media. The shaping system's speed, the energy gain of the corrected wavefronts, and the control degrees of freedom (DOF) are the most important metrics for WFS, especially for highly scattering and dynamic samples. Despite recent advances, current methods suffer from trade-offs that limit satisfactory performance to only one or two of these metrics. Here, we report a WFS technique that simultaneously achieves high speed, high energy gain, and high control DOF. By combining photorefractive crystal-based analog optical phase conjugation (AOPC) and stimulated emission light amplification, our technique achieves an energy gain approaching unity, more than three orders of magnitude larger than conventional AOPC. The response time of ~10 μs with about 106 control modes corresponds to an average mode time of about 0.01 ns/mode, which is more than 50 times lower than some of the fastest WFS systems to date. We anticipate that this technique will be instrumental in overcoming the optical diffusion limit in photonics and translate WFS techniques to real-world applications.

Sub-diffraction computational imaging via a flexible multicore-multimode fiber

An ultra-thin multimode fiber is an ideal platform for minimally invasive microscopy with the advantages of a high density of modes, high spatial resolution, and a compact size. In practical applications, the probe needs to be long and flexible, which unfortunately destroys the imaging capabilities of a multimode fiber. In this work, we propose and experimentally demonstrate sub-diffraction imaging through a flexible probe based on a unique multicore-multimode fiber. A multicore part consists of 120 Fermat's spiral distributed single-mode cores. Each of the cores offers stable light delivery to the multimode part, which provides optimal structured light illumination for sub-diffraction imaging. As a result, perturbation-resilient fast sub-diffraction fiber imaging by computational compressive sensing is demonstrated.

Wavefront shaping: A versatile tool to conquer multiple scattering in multidisciplinary fields

Optical techniques offer a wide variety of applications as light-matter interactions provide extremely sensitive mechanisms to probe or treat target media. Most of these implementations rely on the usage of ballistic or quasi-ballistic photons to achieve high spatial resolution. However, the inherent scattering nature of light in biological tissues or tissue-like scattering media constitutes a critical obstacle that has restricted the penetration depth of non-scattered photons and hence limited the implementation of most optical techniques for wider applications. In addition, the components of an optical system are usually designed and manufactured for a fixed function or performance. Recent advances in wavefront shaping have demonstrated that scattering- or component-induced phase distortions can be compensated by optimizing the wavefront of the input light pattern through iteration or by conjugating the transmission matrix of the scattering medium. This offers unprecedented opportunities in many applications to achieve controllable optical delivery or detection at depths or dynamically configurable functionalities by using scattering media to substitute conventional optical components. In this article, the recent progress of wavefront shaping in multidisciplinary fields is reviewed, from optical focusing and imaging with scattering media, functionalized devices, modulation of mode coupling, and nonlinearity in multimode fiber to multimode fiber-based applications. Apart from insights into the underlying principles and recent advances in wavefront shaping implementations, practical limitations and roadmap for future development are discussed in depth. Looking back and looking forward, it is believed that wavefront shaping holds a bright future that will open new avenues for noninvasive or minimally invasive optical interactions and arbitrary control inside deep tissues. The high degree of freedom with multiple scattering will also provide unprecedented opportunities to develop novel optical devices based on a single scattering medium (generic or customized) that can outperform traditional optical components.

Holographic Manipulation of Nanostructured Fiber Optics Enables Spatially-Resolved, Reconfigurable Optical Control of Plasmonic Local Field Enhancement and SERS

Integration of plasmonic structures on step-index optical fibers is attracting interest for both applications and fundamental studies. However, the possibility to dynamically control the coupling between the guided light fields and the plasmonic resonances is hindered by the turbidity of light propagation in multimode fibers (MMFs). This pivotal point strongly limits the range of studies that can benefit from nanostructured fiber optics. Fortunately, harnessing the interaction between plasmonic modes on the fiber tip and the full set of guided modes can bring this technology to a next generation progress. Here, the intrinsic wealth of information of guided modes is exploited to spatiotemporally control the plasmonic resonances of the coupled system. This concept is shown by employing dynamic phase modulation to structure both the response of plasmonic MMFs on the plasmonic facet and their response in the corresponding Fourier plane, achieving spatial selective field enhancement and direct control of the probe's work point in the dispersion diagram. Such a conceptual leap would transform the biomedical applications of holographic endoscopic imaging by integrating new sensing and manipulation capabilities.

High-speed label-free multimode-fiber-based compressive imaging beyond the diffraction limit

Glass fibers are miniature optical components that serve as ultra-narrow endoscopy probes. Ideally, one would want to perform imaging through a fiber at the highest achievable resolution and speed. State-of-the-art super-resolution techniques have shattered the diffraction limit, but more than twofold improvement requires fluorescent labeling and a long acquisition time. Moreover, it is challenging to implement super-resolution microscopy in a fiber format. Here we present fiber-based label-free video-rate imaging at more than 2-fold higher resolution than the diffraction limit. Our work paves the way to rapid, sub-wavelength endo-microscopy in unlabeled live specimens.

Confocal 3D reflectance imaging through multimode fiber without wavefront shaping

Imaging through optical multimode fiber (MMF) has the potential to enable hair-thin endoscopes that reduce the invasiveness of imaging deep inside tissues and organs. Active wavefront shaping and fluorescent labeling have recently been exploited to overcome modal scrambling and enable MMF imaging. Here, we present a computational approach that circumvents the need for active wavefront control and exogenous fluorophores. We demonstrate the reconstruction of depth-gated confocal images through MMF using a raster-scanned, focused input illumination at the fiber proximal end. We show the compatibility of this approach with quantitative phase, dark-field, and polarimetric imaging. Computational imaging through MMF opens a new pathway for minimally invasive imaging in medical diagnosis and biological investigations.

Scan-less machine-learning-enabled incoherent microscopy for minimally-invasive deep-brain imaging

Deep-brain microscopy is strongly limited by the size of the imaging probe, both in terms of achievable resolution and potential trauma due to surgery. Here, we show that a segment of an ultra-thin multi-mode fiber (cannula) can replace the bulky microscope objective inside the brain. By creating a self-consistent deep neural network that is trained to reconstruct anthropocentric images from the raw signal transported by the cannula, we demonstrate a single-cell resolution (< 10μm), depth sectioning resolution of 40 μm, and field of view of 200 μm, all with green-fluorescent-protein labelled neurons imaged at depths as large as 1.4 mm from the brain surface. Since ground-truth images at these depths are challenging to obtain in vivo, we propose a novel ensemble method that averages the reconstructed images from disparate deep-neural-network architectures. Finally, we demonstrate dynamic imaging of moving GCaMp-labelled C. elegans worms. Our approach dramatically simplifies deep-brain microscopy.

Time-of-flight 3D imaging through multimode optical fibers

[Figure: see text].

Compressively sampling the optical transmission matrix of a multimode fibre

The measurement of the optical transmission matrix (TM) of an opaque material is an advanced form of space-variant aberration correction. Beyond imaging, TM-based methods are emerging in a range of fields, including optical communications, micro-manipulation, and computing. In many cases, the TM is very sensitive to perturbations in the configuration of the scattering medium it represents. Therefore, applications often require an up-to-the-minute characterisation of the fragile TM, typically entailing hundreds to thousands of probe measurements. Here, we explore how these measurement requirements can be relaxed using the framework of compressive sensing, in which the incorporation of prior information enables accurate estimation from fewer measurements than the dimensionality of the TM we aim to reconstruct. Examples of such priors include knowledge of a memory effect linking the input and output fields, an approximate model of the optical system, or a recent but degraded TM measurement. We demonstrate this concept by reconstructing the full-size TM of a multimode fibre supporting 754 modes at compression ratios down to ∼5% with good fidelity. We show that in this case, imaging is still possible using TMs reconstructed at compression ratios down to ∼1% (eight probe measurements). This compressive TM sampling strategy is quite general and may be applied to a variety of other scattering samples, including diffusers, thin layers of tissue, fibre optics of any refractive profile, and reflections from opaque walls. These approaches offer a route towards the measurement of high-dimensional TMs either quickly or with access to limited numbers of measurements.