Endo-microscopy beyond the Abbe and Nyquist limits

Multimode fiber endoscopes for computational brain imaging

Advances in imaging tools have always been a pivotal driver for new discoveries in neuroscience. An ability to visualize neurons and subcellular structures deep within the brain of a freely behaving animal is integral to our understanding of the relationship between neural activity and higher cognitive functions. However, fast high-resolution imaging is limited to sub-surface brain regions and generally requires head fixation of the animal under the microscope. Developing new approaches to address these challenges is critical. The last decades have seen rapid progress in minimally invasive endo-microscopy techniques based on bare optical fibers. A single multimode fiber can be used to penetrate deep into the brain without causing significant damage to the overlying structures and provide high-resolution imaging. Here, we discuss how the full potential of high-speed super-resolution fiber endoscopy can be realized by a holistic approach that combines fiber optics, light shaping, and advanced computational algorithms. The recent progress opens up new avenues for minimally invasive deep brain studies in freely behaving mice.

High-fidelity and high-speed wavefront shaping by leveraging complex media

High-precision light manipulation is crucial for delivering information through complex media. However, existing spatial light modulation devices face a fundamental speed-fidelity tradeoff. Digital micromirror devices have emerged as a promising candidate for high-speed wavefront shaping but at the cost of compromised fidelity due to the limited control degrees of freedom. Here, we leverage the sparse-to-random transformation through complex media to overcome the dimensionality limitation of spatial light modulation devices. We demonstrate that pattern compression by sparsity-constrained wavefront optimization allows sparse and robust wavefront representations in complex media, improving the projection fidelity without sacrificing frame rate, hardware complexity, or optimization time. Our method is generalizable to different pattern types and complex media, supporting consistent performance with up to 89% and 126% improvements in projection accuracy and speckle suppression, respectively. The proposed optimization framework could enable high-fidelity high-speed wavefront shaping through different scattering media and platforms without changes to the existing holographic setups, facilitating a wide range of physics and real-world applications.

Exploiting holographically encoded variance to transmit labelled images through a multimode optical fiber

Artificial intelligence has emerged as promising tool to decode an image transmitted through a multimode fiber (MMF) by applying deep learning techniques. By transmitting thousands of images through the MMF, deep neural networks (DNNs) are able to decipher the seemingly random output speckle patterns and unveil the intrinsic input-output relationship. High fidelity reconstruction is obtained for datasets with a large degree of homogeneity, which underutilizes the capacity of the combined MMF-DNN system. Here, we show that holographic modulation can encode an additional layer of variance on the output speckle pattern, improving the overall transmissive capabilities of the system. Operatively, we have implemented this by adding a holographic label to the original dataset and injecting the resulting phase image into the fiber facet through a Fourier transform lens. The resulting speckle pattern dataset can be clustered primarily by holographic label, and can be reconstructed without loss of fidelity. As an application, we describe how color images may be segmented into RGB components and each color component may then be labelled by distinct hologram. A ResUNet architecture was then used to decode each class of speckle patterns and reconstruct the color image without the need for temporal synchronization between sender and receiver.

Optical fiber bundle differential compressive imaging

We present a differential compressive imaging method for an optical fiber bundle (OFB), which provides a solution for an ultrathin bend-resistant endoscope with high resolution. This method uses an OFB and a diffuser to generate speckle illumination patterns. Differential operation is additionally applied to the speckle patterns to produce sensing matrices, by which the correlation between the matrices is greatly reduced from 0.875 to 0.0275, which ensures the high quality of image reconstruction. Pixilation artifacts from the fiber core arrangement are also effectively eliminated with this configuration. We demonstrate high-resolution reconstruction of images of 132 × 132 pixels with a compression rate of 12% using 77 fiber cores, the total diameter of which is only about 91 µm. An experimental verification proves that this method is tolerant to a limited degree of fiber bending, which provides a potential approach for robust high-resolution fiber endoscopy.

Lightweight super-resolution multimode fiber imaging with regularized linear regression

Super-resolution multimode fiber imaging provides the means to image samples quickly with compact and flexible setups finding many applications from biology and medicine to material science and nanolithography. Typically, fiber-based imaging systems suffer from low spatial resolution and long measurement times. State-of-the-art computational approaches can achieve fast super-resolution imaging through a multimode fiber probe but currently rely on either per-sample optimised priors or large data sets with subsequent long training and image reconstruction times. This unfortunately hinders any real-time imaging applications. Here we present an ultimately fast non-iterative algorithm for compressive image reconstruction through a multimode fiber. The proposed approach helps to avoid many constraints by determining the prior of the target distribution from a simulated set and solving the under-determined inverse matrix problem with a mathematical closed-form solution. We have demonstrated theoretical and experimental evidence for enhanced image quality and sub-diffraction spatial resolution of the multimode fiber optical system.

Super-resolution multimode fiber imaging with an untrained neural network

Multimode fiber endoscopes provide extreme miniaturization of imaging components for minimally invasive deep tissue imaging. Typically, such fiber systems suffer from low spatial resolution and long measurement time. Fast super-resolution imaging through a multimode fiber has been achieved by using computational optimization algorithms with hand-picked priors. However, machine learning reconstruction approaches offer the promise of better priors, but require large training datasets and therefore long and unpractical pre-calibration time. Here we report a method of multimode fiber imaging based on unsupervised learning with untrained neural networks. The proposed approach solves the ill-posed inverse problem by not relying on any pre-training process. We have demonstrated both theoretically and experimentally that untrained neural networks enhance the imaging quality and provide sub-diffraction spatial resolution of the multimode fiber imaging system.

Speckle-correlation imaging through a kaleidoscopic multimode fiber

Speckle-correlation imaging techniques are widely used for noninvasive imaging through complex scattering media. While light propagation through multimode fibers and scattering media share many analogies, reconstructing images through multimode fibers from speckle correlations remains an unsolved challenge. Here, we exploit a kaleidoscopic memory effect emerging in square-core multimode fibers and demonstrate fluorescence imaging with no prior knowledge on the fiber. Experimentally, our approach simply requires to translate random speckle patterns at the input of a square-core fiber and to measure the resulting fluorescence intensity with a bucket detector. The image of the fluorescent object is then reconstructed from the autocorrelation of the measured signal by solving an inverse problem. This strategy does not require the knowledge of the fragile deterministic relation between input and output fields, which makes it promising for the development of flexible minimally invasive endoscopes.

Swept-source multimode fiber imaging

High-resolution compressive imaging via a flexible multimode fiber is demonstrated using a swept-laser source and wavelength dependent speckle illumination. An in-house built swept-source allowing for independent control of bandwidth and scanning range is used to explore and demonstrate a mechanically scan-free approach for high-resolution imaging through an ultrathin and flexible fiber probe. The computational image reconstruction is shown by utilizing a narrow sweeping bandwidth of [Formula: see text] nm while acquisition time is decreased by 95% compared to conventional raster scanning endoscopy. Demonstrated narrow-band illumination in the visible spectrum is vital for the detection of fluorescence biomarkers in neuroimaging applications. The proposed approach yields device simplicity and flexibility for minimally invasive endoscopy.

Sub-diffraction computational imaging via a flexible multicore-multimode fiber

An ultra-thin multimode fiber is an ideal platform for minimally invasive microscopy with the advantages of a high density of modes, high spatial resolution, and a compact size. In practical applications, the probe needs to be long and flexible, which unfortunately destroys the imaging capabilities of a multimode fiber. In this work, we propose and experimentally demonstrate sub-diffraction imaging through a flexible probe based on a unique multicore-multimode fiber. A multicore part consists of 120 Fermat's spiral distributed single-mode cores. Each of the cores offers stable light delivery to the multimode part, which provides optimal structured light illumination for sub-diffraction imaging. As a result, perturbation-resilient fast sub-diffraction fiber imaging by computational compressive sensing is demonstrated.

Computational imaging with spectral coding increases the spatial resolution of fiber optic bundles

Fiber optic bundles are used in narrow-diameter medical and industrial instruments for acquiring images from confined locations. Images transmitted through these bundles contain only one pixel of information per fiber core and fail to capture information from the cladding region between cores. Both factors limit the spatial resolution attainable with fiber bundles. We show here that computational imaging (CI) can be combined with spectral coding to overcome these two fundamental limitations and improve spatial resolution in fiber bundle imaging. By acquiring multiple images of a scene with a high-resolution mask pattern imposed, up to 17 pixels of information can be recovered from each fiber core. A dispersive element at the distal end of the bundle imparts a wavelength-dependent lateral shift on light from the object. This enables light that would otherwise be lost at the inter-fiber cladding to be transmitted through adjacent fiber cores. We experimentally demonstrate this approach using synthetic and real objects. Using CI with spectral coding, object features 5× smaller than individual fiber cores were resolved, whereas conventional imaging could only resolve features at least 1.5× larger than each core. In summary, CI combined with spectral coding provides an approach for overcoming the two fundamental limitations of fiber optic bundle imaging.

Speckle classification of a multimode fiber based on Inception V3

Multimode optical fiber plays an important role in endoscope miniaturization. With the development of deep learning and machine learning, neural networks can be used to identify and classify speckle patterns obtained at the fiber output. Based on the speckle pattern of a HERLEV dataset cell image transmitted by a multimode fiber, this paper studies the recognition accuracy of various types of speckle by a support vector machine, K-nearest neighbor, and convolutional neural network (Inception V3). Meanwhile, we propose an image classification optimization algorithm based on improved Inception V3. The experimental results show that the improved algorithm model is better than the traditional machine learning method; the accuracy rate is 97.92%, which effectively improves the performance of the pathological cell diagnosis deep learning model and lays a theoretical and practical foundation for further clinical application.

Design of optoelectrodes for the remote imaging of cells and in situ electrochemical detection of neurosecretory events

Optical fibers have opened avenues for remote imaging, bioanalyses and recently optogenetics. Besides, miniaturized electrochemical sensors have offered new opportunities in sensing directly redox neurotransmitters. The combination of both optical and electrochemical approaches was usually performed on the platform of microscopes or within microsystems. In this work, we developed optoelectrodes which features merge the advantages of both optical fibers and microelectrodes. Optical fiber bundles were modified at one of their extremity by a transparent ITO deposit. The electrochemical responses of these ITO-modified bundles were characterized for the detection of dopamine, epinephrine and norepinephrine. The analytical performances of the optoelectrodes were equivalent to the ones reported for carbon microelectrodes. The remote imaging of model neurosecretory PC12 cells by optoelectrodes was performed upon cell-staining with common fluorescent dyes: acridine orange and calcein-AM. An optoelectrode placed by micromanipulation at a few micrometers-distance from the cells offered remote images with single cell resolution. Finally, in situ electrochemical sensing was demonstrated by additions of K⁺-secretagogue solutions near PC12 cells under observation, leading to exocytotic events detected as amperometric spikes at the ITO surface. Such dual sensors should pave the way for in vivo remote imaging, optogenetic stimulation, and simultaneous detection of neurosecretory activities.

Edge detection in single multimode fiber imaging based on deep learning

We propose a new edge detection scheme based on deep learning in single multimode fiber imaging. In this scheme, we creatively design a novel neural network, whose input is a one-dimensional light intensity sequence, and the output is the edge detection result of the target. Different from the traditional scheme, we can directly obtain the edge information of unknown objects by using this neural network without rebuilding the image. Simulation and experimental results show that, compared with the traditional method, this method can get better edge details, especially in the case of low sampling rates. It can increase the structural similarity index of edge detection imaging from 0.38 to 0.62 at the sampling rate of 0.6%. At the same time, the robustness of the method to fiber bending is also proved. This scheme improves the edge detection performance of endoscopic images and provides a promising way for the practical application of multimode fiber endoscopy.

Compact fiber-integrated scattering device based on mixed-phase TiO2 for speckle spectrometer

A universal, repeatable, and controllable integration of single-mode optical fiber and mixed-phase TiO₂ is used to manufacture a compact fiber-integrated scattering device. Based on the device, we achieve a high-performance and compact fiber-based speckle spectrometer, which has a resolution of 20 pm over a bandwidth of 15 nm, in the 1550 nm range. We test the capability of our proposed spectrometer to reconstruct narrow linewidth and broadband optical spectrums, and compare the performance with that of a traditional optical spectrum analyzer.

High-speed label-free multimode-fiber-based compressive imaging beyond the diffraction limit

Glass fibers are miniature optical components that serve as ultra-narrow endoscopy probes. Ideally, one would want to perform imaging through a fiber at the highest achievable resolution and speed. State-of-the-art super-resolution techniques have shattered the diffraction limit, but more than twofold improvement requires fluorescent labeling and a long acquisition time. Moreover, it is challenging to implement super-resolution microscopy in a fiber format. Here we present fiber-based label-free video-rate imaging at more than 2-fold higher resolution than the diffraction limit. Our work paves the way to rapid, sub-wavelength endo-microscopy in unlabeled live specimens.

Far-field super-resolution ghost imaging with a deep neural network constraint

Ghost imaging (GI) facilitates image acquisition under low-light conditions by single-pixel measurements and thus has great potential in applications in various fields ranging from biomedical imaging to remote sensing. However, GI usually requires a large amount of single-pixel samplings in order to reconstruct a high-resolution image, imposing a practical limit for its applications. Here we propose a far-field super-resolution GI technique that incorporates the physical model for GI image formation into a deep neural network. The resulting hybrid neural network does not need to pre-train on any dataset, and allows the reconstruction of a far-field image with the resolution beyond the diffraction limit. Furthermore, the physical model imposes a constraint to the network output, making it effectively interpretable. We experimentally demonstrate the proposed GI technique by imaging a flying drone, and show that it outperforms some other widespread GI techniques in terms of both spatial resolution and sampling ratio. We believe that this study provides a new framework for GI, and paves a way for its practical applications.

Recent Development in X-Ray Imaging Technology: Future and Challenges

X-ray imaging is a low-cost, powerful technology that has been extensively used in medical diagnosis and industrial nondestructive inspection. The ability of X-rays to penetrate through the body presents great advances for noninvasive imaging of its internal structure. In particular, the technological importance of X-ray imaging has led to the rapid development of high-performance X-ray detectors and the associated imaging applications. Here, we present an overview of the recent development of X-ray imaging-related technologies since the discovery of X-rays in the 1890s and discuss the fundamental mechanism of diverse X-ray imaging instruments, as well as their advantages and disadvantages on X-ray imaging performance. We also highlight various applications of advanced X-ray imaging in a diversity of fields. We further discuss future research directions and challenges in developing advanced next-generation materials that are crucial to the fabrication of flexible, low-dose, high-resolution X-ray imaging detectors.

Computational image enhancement of multimode fibre-based holographic endo-microscopy: harnessing the muddy modes

In imaging geometries, which employ wavefront-shaping to control the light transport through a multi-mode optical fibre (MMF), this terminal hair-thin optical component acts as a minimally invasive objective lens, enabling high resolution laser-scanning fluorescence microscopy inside living tissues at depths hardly accessible by any other light-based technique. Even in the most advanced systems, the diffraction-limited foci scanning the object across the focal plane are contaminated by a stray optical signal carrying typically few tens of % of the total optical power. The stray illumination takes the shape of a randomised but reproducible speckle, and is unique for each position of the focus. We experimentally demonstrate that the performance of imaging a fluorescent object can be significantly improved, when resulting images are computationally post-processed, utilising records of intensities of all speckle-contaminated foci used in the imaging procedure. We present two algorithms based on a regularised iterative inversion and regularised direct pseudo-inversion respectively which lead to enhancement of the image contrast and resolution.

Phonon imaging in 3D with a fibre probe

We show for the first time that a single ultrasonic imaging fibre is capable of simultaneously accessing 3D spatial information and mechanical properties from microscopic objects. The novel measurement system consists of two ultrafast lasers that excite and detect high-frequency ultrasound from a nano-transducer that was fabricated onto the tip of a single-mode optical fibre. A signal processing technique was also developed to extract nanometric in-depth spatial measurements from GHz frequency acoustic waves, while still allowing Brillouin spectroscopy in the frequency domain. Label-free and non-contact imaging performance was demonstrated on various polymer microstructures. This singular device is equipped with optical lateral resolution, 2.5 μm, and a depth-profiling precision of 45 nm provided by acoustics. The endoscopic potential for this device is exhibited by extrapolating the single fibre to tens of thousands of fibres in an imaging bundle. Such a device catalyses future phonon endomicroscopy technology that brings the prospect of label-free in vivo histology within reach.

Micro two-dimensional slit-array for super resolution beyond pixel Nyquist limits in grating spectrometers

This paper presents a new micro 2-D slit-array device for spectral resolution enhancement in grating spectrometers. The 2-D slit-array is encoded in Hadamard matrix and the device is fabricated based on the micro-electromechanical system (MEMS) technology. By just using this 2-D slit-array to replace the single slit in the conventional grating spectrometer, real-time super spectral resolution detection beyond the pixel Nyquist limit, which is determined by the size of the detector pixel, can be realized. Furthermore, no other configuration of the spectrometer is changed, no movable parts are used, and the spectral range and instrument size remain almost unchanged while the resolution is improved. A series of experimental verifications for the feasibility of this design are included in this work.

Comparison of round- and square-core fibers for sensing, imaging, and spectroscopy

Multimode fibers (MMFs) show great promise as miniature probes for sensing, imaging, and spectroscopy applications. Different parameters of the fibers, such as numerical aperture, refractive index profile and length, have been already optimized for better performance. Here we investigate the role of the core shape, in particular for wavefront shaping applications where a focus is formed at the output of the MMF. We demonstrate that in contrast to a conventional round-core MMF, a square-core design does not suffer from focus aberrations. Moreover, we find that how the interference pattern behind a square-core fiber decorrelates with the input frequency is largely independent of the input light coupling. Finally, we demonstrate that a square core shape provides an on-average uniform distribution of the output intensity, free from the input-output correlations seen in round fibers, showing great promise for imaging and spectroscopy applications.

Ultimate resolution limits of speckle-based compressive imaging

Compressive imaging using sparsity constraints is a very promising field of microscopy that provides a dramatic enhancement of the spatial resolution beyond the Abbe diffraction limit. Moreover, it simultaneously overcomes the Nyquist limit by reconstructing an N-pixel image from less than N single-point measurements. Here we present fundamental resolution limits of noiseless compressive imaging via sparsity constraints, speckle illumination and single-pixel detection. We addressed the experimental setup that uses randomly generated speckle patterns (in a scattering media or a multimode fiber). The optimal number of measurements, the ultimate spatial resolution limit and the surprisingly important role of discretization are demonstrated by the theoretical analysis and numerical simulations. We show that, in contrast to conventional microscopy, oversampling may decrease the resolution and reconstruction quality of compressive imaging.

Spatiotemporal focusing through a multimode fiber via time-domain wavefront shaping

We shape fs optical pulses and deliver them in a single spatial mode to the input of a multimode fiber. The pulse is shaped in time such that at the output of the multimode fiber an ultrashort pulse appears at a predefined focus. Our result shows how to raster scan an ultrashort pulse at the output of a stiff piece of square-core step-index multimode fiber and in this way show the potential for making a nonlinear fluorescent image of the scene behind the fiber, while the connection to the multimode fiber can be established via a thin and flexible single-mode fiber. The experimental results match our numerical simulation well.