Calibration-free imaging through a multicore fiber using speckle scanning microscopy

Ptychographic lensless coherent endomicroscopy through a flexible fiber bundle

Conventional fiber-bundle-based endoscopes allow minimally invasive imaging through flexible multi-core fiber (MCF) bundles by placing a miniature lens at the distal tip and using each core as an imaging pixel. In recent years, lensless imaging through MCFs was made possible by correcting the core-to-core phase distortions pre-measured in a calibration procedure. However, temporally varying wavefront distortions, for instance, due to dynamic fiber bending, pose a challenge for such approaches. Here, we demonstrate a coherent lensless imaging technique based on intensity-only measurements insensitive to core-to-core phase distortions. We leverage a ptychographic reconstruction algorithm to retrieve the phase and amplitude profiles of reflective objects placed at a distance from the fiber tip, using as input a set of diffracted intensity patterns reflected from the object when the illumination is scanned over the MCF cores. Our approach thus utilizes an acquisition process equivalent to confocal microendoscopy, only replacing the single detector with a camera.

Image scanning lensless fiber-bundle endomicroscopy

Fiber-based confocal endomicroscopy has shown great promise for minimally-invasive deep-tissue imaging. Despite its advantages, confocal fiber-bundle endoscopy inherently suffers from undersampling due to the spacing between fiber cores, and low collection efficiency when the target is not in proximity to the distal fiber facet. Here, we demonstrate an adaptation of image-scanning microscopy (ISM) to lensless fiber bundle endoscopy, doubling the spatial sampling frequency and significantly improving collection efficiency. Our approach only requires replacing the confocal detector with a camera. It improves the spatial resolution for targets placed at a distance from the fiber tip, and addresses the fundamental challenge of aliasing/pixelization artifacts.

Imaging operator in indirect imaging correlography

Indirect imaging correlography (IIC) is a coherent imaging technique that provides access to the autocorrelation of the albedo of objects obscured from line-of-sight. This technique is used to recover sub-mm resolution images of obscured objects at large standoffs in non-line-of-sight (NLOS) imaging. However, predicting the exact resolving power of IIC in any given NLOS scene is complicated by the interplay between several factors, including object position and pose. This work puts forth a mathematical model for the imaging operator in IIC to accurately predict the images of objects in NLOS imaging scenes. Using the imaging operator, expressions for the spatial resolution as a function of scene parameters such as object position and pose are derived and validated experimentally. In addition, a self-supervised deep neural network framework to reconstruct images of objects from their autocorrelation is proposed. Using this framework, objects with ≈ 250 μ m features, located at 1 mt standoffs in an NLOS scene, are successfully reconstructed.

Fourier holographic endoscopy for imaging continuously moving objects

Coherent fiber bundles are widely used for endoscopy, but conventional approaches require distal optics to form an object image and acquire pixelated information owing to the geometry of the fiber cores. Recently, holographic recording of a reflection matrix enables a bare fiber bundle to perform pixelation-free microscopic imaging as well as allows a flexible mode operation, because the random core-to-core phase retardations due to any fiber bending and twisting could be removed in situ from the recorded matrix. Despite its flexibility, the method is not suitable for a moving object because the fiber probe should remain stationary during the matrix recording to avoid the alteration of the phase retardations. Here, we acquire a reflection matrix of a Fourier holographic endoscope equipped with a fiber bundle and explore the effect of fiber bending on the recorded matrix. By removing the motion effect, we develop a method that can resolve the perturbation of the reflection matrix caused by a continuously moving fiber bundle. Thus, we demonstrate high-resolution endoscopic imaging through a fiber bundle, even when the fiber probe changes its shape along with the moving objects. The proposed method can be used for minimally invasive monitoring of behaving animals.

Real-time holographic lensless micro-endoscopy through flexible fibers via fiber bundle distal holography

Fiber-based micro-endoscopes are a critically important tool for minimally-invasive deep-tissue imaging. However, current micro-endoscopes cannot perform three-dimensional imaging through dynamically-bent fibers without the use of bulky optical elements such as lenses and scanners at the distal end, increasing the footprint and tissue-damage. Great efforts have been invested in developing approaches that avoid distal bulky optical elements. However, the fundamental barrier of dynamic optical wavefront-distortions in propagation through flexible fibers limits current approaches to nearly-static or non-flexible fibers. Here, we present an approach that allows holographic, bend-insensitive, coherence-gated, micro-endoscopic imaging using commercially available multi-core fibers (MCFs). We achieve this by adding a partially-reflecting mirror to the distal fiber-tip, allowing to perform low-coherence full-field phase-shifting holography. We demonstrate widefield diffraction-limited reflection imaging of amplitude and phase targets through dynamically bent fibers at video-rate. Our approach holds potential for label-free investigations of dynamic samples.

Flexible-type ultrathin holographic endoscope for microscopic imaging of unstained biological tissues

Ultrathin lensless fibre endoscopes offer minimally invasive investigation, but they mostly operate as a rigid type due to the need for prior calibration of a fibre probe. Furthermore, most implementations work in fluorescence mode rather than label-free imaging mode, making them unsuitable for general medical diagnosis. Herein, we report a fully flexible ultrathin fibre endoscope taking 3D holographic images of unstained tissues with 0.85-μm spatial resolution. Using a bare fibre bundle as thin as 200-μm diameter, we design a lensless Fourier holographic imaging configuration to selectively detect weak reflections from biological tissues, a critical step for label-free endoscopic reflectance imaging. A unique algorithm is developed for calibration-free holographic image reconstruction, allowing us to image through a narrow and curved passage regardless of fibre bending. We demonstrate endoscopic reflectance imaging of unstained rat intestine tissues that are completely invisible to conventional endoscopes. The proposed endoscope will expedite a more accurate and earlier diagnosis than before with minimal complications.

Wavefront shaping: A versatile tool to conquer multiple scattering in multidisciplinary fields

Optical techniques offer a wide variety of applications as light-matter interactions provide extremely sensitive mechanisms to probe or treat target media. Most of these implementations rely on the usage of ballistic or quasi-ballistic photons to achieve high spatial resolution. However, the inherent scattering nature of light in biological tissues or tissue-like scattering media constitutes a critical obstacle that has restricted the penetration depth of non-scattered photons and hence limited the implementation of most optical techniques for wider applications. In addition, the components of an optical system are usually designed and manufactured for a fixed function or performance. Recent advances in wavefront shaping have demonstrated that scattering- or component-induced phase distortions can be compensated by optimizing the wavefront of the input light pattern through iteration or by conjugating the transmission matrix of the scattering medium. This offers unprecedented opportunities in many applications to achieve controllable optical delivery or detection at depths or dynamically configurable functionalities by using scattering media to substitute conventional optical components. In this article, the recent progress of wavefront shaping in multidisciplinary fields is reviewed, from optical focusing and imaging with scattering media, functionalized devices, modulation of mode coupling, and nonlinearity in multimode fiber to multimode fiber-based applications. Apart from insights into the underlying principles and recent advances in wavefront shaping implementations, practical limitations and roadmap for future development are discussed in depth. Looking back and looking forward, it is believed that wavefront shaping holds a bright future that will open new avenues for noninvasive or minimally invasive optical interactions and arbitrary control inside deep tissues. The high degree of freedom with multiple scattering will also provide unprecedented opportunities to develop novel optical devices based on a single scattering medium (generic or customized) that can outperform traditional optical components.

Real-time complex light field generation through a multi-core fiber with deep learning

The generation of tailored complex light fields with multi-core fiber (MCF) lensless microendoscopes is widely used in biomedicine. However, the computer-generated holograms (CGHs) used for such applications are typically generated by iterative algorithms, which demand high computation effort, limiting advanced applications like fiber-optic cell manipulation. The random and discrete distribution of the fiber cores in an MCF induces strong spatial aliasing to the CGHs, hence, an approach that can rapidly generate tailored CGHs for MCFs is highly demanded. We demonstrate a novel deep neural network-CoreNet, providing accurate tailored CGHs generation for MCFs at a near video rate. The CoreNet is trained by unsupervised learning and speeds up the computation time by two magnitudes with high fidelity light field generation compared to the previously reported CGH algorithms for MCFs. Real-time generated tailored CGHs are on-the-fly loaded to the phase-only spatial light modulator (SLM) for near video-rate complex light fields generation through the MCF microendoscope. This paves the avenue for real-time cell rotation and several further applications that require real-time high-fidelity light delivery in biomedicine.

De-noising imaging through diffusers with autocorrelation

Recovering targets through diffusers is an important topic as well as a general problem in optical imaging. The difficulty of recovering is increased due to the noise interference caused by an imperfect imaging environment. Existing approaches generally require a high-signal-to-noise-ratio (SNR) speckle pattern to recover the target, but still have limitations in de-noising or generalizability. Here, featuring information of high-SNR autocorrelation as a physical constraint, we propose a two-stage (de-noising and reconstructing) method to improve robustness based on data driving. Specifically, a two-stage convolutional neural network (CNN) called autocorrelation reconstruction (ACR) CNN is designed to de-noise and reconstruct targets from low-SNR speckle patterns. We experimentally demonstrate the robustness through various diffusers with different levels of noise, from simulative Gaussian noise to the detector and photon noise captured by the actual optical system. The de-noising stage improves the peak SNR from 20 to 38 dB in the system data, and the reconstructing stage, compared with the unconstrained method, successfully recovers targets hidden in unknown diffusers with the detector and photon noise. With the help of the physical constraint to optimize the learning process, our two-stage method is realized to improve generalizability and has potential in various fields such as imaging in low illumination.

Recurrent neural network reveals transparent objects through scattering media

Scattering generally worsens the condition of inverse problems, with the severity depending on the statistics of the refractive index gradient and contrast. Removing scattering artifacts from images has attracted much work in the literature, including recently the use of static neural networks. S. Li et al. [Optica5(7), 803 (2018)10.1364/OPTICA.5.000803] trained a convolutional neural network to reveal amplitude objects hidden by a specific diffuser; whereas Y. Li et al. [Optica5(10), 1181 (2018)10.1364/OPTICA.5.001181] were able to deal with arbitrary diffusers, as long as certain statistical criteria were met. Here, we propose a novel dynamical machine learning approach for the case of imaging phase objects through arbitrary diffusers. The motivation is to strengthen the correlation among the patterns during the training and to reveal phase objects through scattering media. We utilize the on-axis rotation of a diffuser to impart dynamics and utilize multiple speckle measurements from different angles to form a sequence of images for training. Recurrent neural networks (RNN) embedded with the dynamics filter out useful information and discard the redundancies, thus quantitative phase information in presence of strong scattering. In other words, the RNN effectively averages out the effect of the dynamic random scattering media and learns more about the static pattern. The dynamical approach reveals transparent images behind the scattering media out of speckle correlation among adjacent measurements in a sequence. This method is also applicable to other imaging applications that involve any other spatiotemporal dynamics.

Medical Imaging of Microrobots: Toward In Vivo Applications

Medical microrobots (MRs) have been demonstrated for a variety of non-invasive biomedical applications, such as tissue engineering, drug delivery, and assisted fertilization, among others. However, most of these demonstrations have been carried out in in vitro settings and under optical microscopy, being significantly different from the clinical practice. Thus, medical imaging techniques are required for localizing and tracking such tiny therapeutic machines when used in medical-relevant applications. This review aims at analyzing the state of the art of microrobots imaging by critically discussing the potentialities and limitations of the techniques employed in this field. Moreover, the physics and the working principle behind each analyzed imaging strategy, the spatiotemporal resolution, and the penetration depth are thoroughly discussed. The paper deals with the suitability of each imaging technique for tracking single or swarms of MRs and discusses the scenarios where contrast or imaging agent's inclusion is required, either to absorb, emit, or reflect a determined physical signal detected by an external system. Finally, the review highlights the existing challenges and perspective solutions which could be promising for future in vivo applications.

Using fiber-bending-generated speckles for improved working distance and background rejection in lensless micro-endoscopy

Lensless flexible fiber-bundle-based endoscopes allow imaging at depths beyond the reach of conventional microscopes with a minimal footprint. These multicore fibers provide a simple solution for wide-field fluorescent imaging when the target is adjacent to the fiber facet. However, they suffer from a very limited working distance and out-of-focus background. Here, we carefully study the dynamic speckle illumination patterns generated by bending a commercial fiber bundle and show that they can be exploited to allow extended working distance and background rejection, using a super-resolution fluctuations imaging analysis of multiple frames, without the addition of any optical elements.

Holographic lensless fiber endoscope with needle size using self-calibration

Endoscopes enable optical keyhole access in many applications for instance in biomedicine. In general, coherent fiber bundles (CFB) are used in conjunction with rigid lens systems which determine a fixed image plane. However, the lens system limits the minimum diameter of the endoscope typically to several millimeters. Additionally, only pixelated two-dimensional amplitude patterns can be transferred due to phase scrambling between adjacent cores. These limitations can be overcome by digital optical elements. Thus, in principle thinner, lensless, holographic endoscopes with a three-dimensional adjustable focus for imaging and illumination can be realized. So far, several techniques based on single mode CFB and multi mode fibers (MMF) have been presented. However, these techniques require access to both sides of the fiber, in order to calibrate the bending and temperature sensitive phase distortion, which is not possible in a real application. We present the feasibility of an in-situ calibration and compensation of a CFB with single sided access. A lensless endoscope with a diameter of only 500 µm, a spatial resolution around 1 µm and video rate capability is realized.

Two-photon lensless micro-endoscopy with in-situ wavefront correction

Multi-core fiber-bundle endoscopes provide a minimally-invasive solution for deep tissue imaging and opto-genetic stimulation, at depths beyond the reach of conventional microscopes. Recently, wavefront-shaping has enabled lensless bundle-based micro-endoscopy by correcting the wavefront distortions induced by core-to-core inhomogeneities. However, current wavefront-shaping solutions require access to the fiber distal end for determining the bend-sensitive wavefront-correction. Here, we show that it is possible to determine the wavefront correction in-situ, without any distal access. Exploiting the nonlinearity of two-photon excited fluorescence, we adaptively determine the wavefront correction in-situ using only proximal detection of epi-detected fluorescence. We experimentally demonstrate diffraction-limited, three-dimensional, two-photon lensless microendoscopy with commercially-available ordered- and disordered multi-core fiber bundles.

Complete polarization control in multimode fibers with polarization and mode coupling

Multimode optical fibers have seen increasing applications in communication, imaging, high-power lasers, and amplifiers. However, inherent imperfections and environmental perturbations cause random polarization and mode mixing, causing the output polarization states to be different from the input polarization states. This difference poses a serious issue for employing polarization-sensitive techniques to control light-matter interactions or nonlinear optical processes at the distal end of a fiber probe. Here, we demonstrate complete control of polarization states for all output channels by only manipulating the spatial wavefront of a laser beam into the fiber. Arbitrary polarization states for individual output channels are generated by wavefront shaping without constraining the input polarization. The strong coupling between the spatial and polarization degrees of freedom in a multimode fiber enables full polarization control with the spatial degrees of freedom alone; thus, wavefront shaping can transform a multimode fiber into a highly efficient reconfigurable matrix of waveplates for imaging and communication applications.

Self-calibration of lensless holographic endoscope using programmable guide stars

Coherent fiber bundle (CFB)-based endoscopes enable optical keyhole access in applications such as biophotonics. In conjunction with objective lenses, CFBs allow imaging of intensity patterns. In contrast, digital optical phase conjugation enables lensless holographic endoscopes for the generation of pixelation-free arbitrary light patterns. For real-world applications, however, this requires a non-invasive in situ calibration of the complex optical transfer function of the CFB with only single-sided access. We show that after an initial calibration in a forward direction, a differential phase measurement of the back-reflected light allows for tracking and compensating of bending-induced phase distortions. Furthermore, we present a novel in situ calibration procedure based on a programmable guide star, which requires access to only one side of the fiber.

Wavefront sensing with a thin diffuser

We propose and implement a broadband, compact, and low-cost wavefront sensing scheme by simply placing a thin diffuser in the close vicinity of a camera. The local wavefront gradient is determined from the local translation of the speckle pattern. The translation vector map is computed thanks to a fast diffeomorphic image registration algorithm and integrated to reconstruct the wavefront profile. The simple translation of speckle grains under local wavefront tip/tilt is ensured by the so-called "memory effect" of the diffuser. Quantitative wavefront measurements are experimentally demonstrated, both for the few first Zernike polynomials and for phase-imaging applications requiring high resolution. We finally provided a theoretical description of the resolution limit that is supported experimentally.

Bending-induced inter-core group delays in multicore fibers

We examine the impact of fiber bends on ultrashort pulse propagation in a 169-core multicore fiber (MCF) by numerical simulations and experimental measurements. We show that an L-shaped bend (where only one end of the MCF is fixed) induces significant changes in group delays that are a function of core position but linear along the bending axis with a slope directly proportional to the bending angle. For U- and S-shaped bends (where both ends of the MCF are fixed) the induced refractive index and group delay changes are much smaller than the residual, intrinsic inter-core group delay differences of the unbent MCF. We further show that when used for point-scanning lensless endoscopy with ultrashort pulse excitation, bend-induced group delays in the MCF degrade the point-spread function due to spatiotemporal coupling. Our results show that bend-induced effects in MCFs can be parametrized with only two parameters: the angle of the bend axis and the amplitude of the bend. This remains valid for bend amplitudes up to at least 200 degrees.

Scatter-plate microscope for lensless microscopy with diffraction limited resolution

Scattering media have always been looked upon as an obstacle in imaging. Various methods, ranging from holography to phase compensation as well as to correlation techniques, have been proposed to cope with this obstacle. We, on the other hand, have a different understanding about the role of the diffusing media. In this paper we propose and demonstrate a ‘scatter-plate microscope’ that utilizes the diffusing property of the random medium for imaging micro structures with diffraction-limited resolution. The ubiquitous property of the speckle patterns permits to exploit the scattering medium as an ultra-thin lensless microscope objective with a variable focal length and a large working distance. The method provides a light, flexible and cost effective imaging device as an alternative to conventional microscope objectives. In principle, the technique is also applicable to lensless imaging in UV and X-ray microscopy. Experiments were performed with visible light to demonstrate the microscopic imaging of USAF resolution test target and a biological sample with varying numerical aperture (NA) and magnifications.

Removal of back-reflection noise at ultrathin imaging probes by the single-core illumination and wide-field detection

Thin waveguides such as graded-index lenses and fiber bundles are often used as imaging probes for high-resolution endomicroscopes. However, strong back-reflection from the end surfaces of the probes makes it difficult for them to resolve weak contrast objects, especially in the reflectance-mode imaging. Here we propose a method to spatially isolate illumination pathways from detection channels, and demonstrate wide-field reflectance imaging free from back-reflection noise. In the image fiber bundle, we send illumination light through individual core fibers and detect signals from target objects through the other fibers. The transmission matrix of the fiber bundle is measured and used to reconstruct a pixelation-free image. We demonstrated that the proposed imaging method improved 3.2 times on the signal to noise ratio produced by the conventional illumination-detection scheme.

Phase retrieval in multicore fiber bundles

Multicore fiber bundles are widely used in endoscopy due to their miniature size and their direct imaging capabilities. They have recently been used, in combination with spatial light modulators, in various realizations of endoscopy with little or no optics at the distal end. These schemes require characterization of the relative phase offsets between the different cores, typically done using off-axis holography, thus requiring both an interferometric setup and, typically, access to the distal tip. Here we explore the possibility of employing phase retrieval to extract the necessary phase information. We show that in the noise-free case, disordered fiber bundles are superior for phase retrieval over their periodic counterparts, and demonstrate experimentally accurate retrieval of phase information for up to 10 simultaneously illuminated cores. Thus, phase retrieval is presented as a viable alternative for real-time monitoring of phase distortions in multicore fiber bundles.

Compressive fluorescence imaging using a multi-core fiber and spatially dependent scattering

We demonstrate imaging using a multi-core fiber with a scattering distal tip and compressed sensing signal acquisition. We illuminate objects with randomly structured speckle patterns generated by a coherent light source separately coupled through each fiber core to a ground glass diffuser at the distal end. Using the characterized speckle patterns and the total light collected from the object, we computationally recover pixelation-free object images with up to a seven times higher space-bandwidth product than the number of cores. The proposed imaging system is insensitive to bending of the fiber and extremely compact, making it suitable for minimally invasive endomicroscopy.

Ultrathin endoscopes based on multicore fibers and adaptive optics: a status review and perspectives

We take stock of the progress that has been made into developing ultrathin endoscopes assisted by wave front shaping. We focus our review on multicore fiber-based lensless endoscopes intended for multiphoton imaging applications. We put the work into perspective by comparing with alternative approaches and by outlining the challenges that lie ahead.