Depixelation of coherent fiber bundle endoscopy based on learning patterns of image prior

Accelerated fixed-point iterative reconstruction for fiber borescope imaging

Computational imaging systems with embedded processing have potential advantages in power consumption, computing speed, and cost. However, common processors in embedded vision systems have limited computing capacity and low level of parallelism. The widely used iterative algorithms for image reconstruction rely on floating-point processors to ensure calculation precision, which require more computing resources than fixed-point processors. Here we present a regularized Landweber fixed-point iterative solver for image reconstruction, implemented on a field programmable gated array (FPGA). Compared with floating-point embedded uniprocessors, iterative solvers implemented on the fixed-point FPGA gain 1 to 2 orders of magnitude acceleration, while achieving the same reconstruction accuracy in comparable number of effective iterations. Specifically, we have demonstrated the proposed fixed-point iterative solver in fiber borescope image reconstruction, successfully correcting the artifacts introduced by the lenses and fiber bundle.

Depixelation of coherent fiber bundle imaging by fiber-core-targeted scanning

A novel fast proximal scanning method, to the best of our knowledge, termed fiber-core-targeted scanning (FCTS), is proposed for illuminating individual fiber cores sequentially to remove the pixelation effect in fiber bundle (FB) imaging. FCTS is based on a galvanometer scanning system. Through a dynamic control of the scan trajectory and speed using the prior knowledge of fiber core positions, FCTS experimentally verifies a precise sequential delivery of laser pulses into fiber cores at a maximal speed of 45,000 cores per second. By applying FCTS on a FB-based photoacoustic forward-imaging probe, the results demonstrate that FCTS eliminates the pixelation effect and improves the imaging quality.

Image computing for fibre-bundle endomicroscopy: A review

Endomicroscopy is an emerging imaging modality, that facilitates the acquisition of in vivo, in situ optical biopsies, assisting diagnostic and potentially therapeutic interventions. While there is a diverse and constantly expanding range of commercial and experimental optical biopsy platforms available, fibre-bundle endomicroscopy is currently the most widely used platform and is approved for clinical use in a range of clinical indications. Miniaturised, flexible fibre-bundles, guided through the working channel of endoscopes, needles and catheters, enable high-resolution imaging across a variety of organ systems. Yet, the nature of image acquisition though a fibre-bundle gives rise to several inherent characteristics and limitations necessitating novel and effective image pre- and post-processing algorithms, ranging from image formation, enhancement and mosaicing to pathology detection and quantification. This paper introduces the underlying technology and most prevalent clinical applications of fibre-bundle endomicroscopy, and provides a comprehensive, up-to-date, review of relevant image reconstruction, analysis and understanding/inference methodologies. Furthermore, current limitations as well as future challenges and opportunities in fibre-bundle endomicroscopy computing are identified and discussed.

Fiber bundle image restoration using deep learning

We propose a deep learning-based restoration method to remove honeycomb patterns and improve resolution for fiber bundle (FB) images. By building and calibrating a dual-sensor imaging system, we capture FB images and corresponding ground truth data to train the network. Images without fiber bundle fixed patterns are restored from raw FB images as direct inputs, and spatial resolution is significantly enhanced for the trained sample type. We also construct the brightness mapping between the two image types for the effective use of all data, providing the ability to output images of the expected brightness. We evaluate our framework with data obtained from lens tissues and human histological specimens using both objective and subjective measures.

Handheld line-scanned dual-axis confocal microscope with pistoned MEMS actuation for flat-field fluorescence imaging

A handheld line-scanned dual-axis confocal (LS-DAC) microscope has been developed for high-speed (16 frames/s) fluorescence imaging of tissues with sub-nuclear resolution. This is the first miniature fluorescence LS-DAC system that has been fully packaged for handheld clinical use on patients. A novel micro-electro-mechanical system scanning mechanism, with synchronized tilting and pistoning, is used to achieve flat-field en face imaging. We show that this facilitates video mosaicking to generate images that sample an extended lateral field of view.

Compact high-resolution endomicroscopy based on fiber bundles and image stitching

In this Letter, we report a compact endomicroscope (ϕ=2.8  mm) based on a fiber bundle and a two-axis piezoelectric tube scanner, achieving a resolution of ∼1  μm and an imaging speed of 30-120 fps. Compared with distal end scanning systems, typical fiber-bundle-based endomicroscopes achieve a more compact envelope (ϕ∼1.5  mm) at the expense of compromised imaging quality. The resolution of fiber-bundle-based systems is largely limited by the diameter of the constituent fibers (ϕ∼5.0  μm), where each fiber serves as a single pixel, i.e., a sampling point, in the imaging system. To retrieve the lost information, we integrate a piezo tube scanner at the tip of the fiber bundle. Next, we rapidly scan the fiber tip over a range of ±2.5  μm and combine the signals obtained at different inter-fiber locations. Direct alignment and feature-based registration methods are applied to register the raw images. Imaging experiments are performed on a resolution target and biological samples to demonstrate the performance enhancement.

Resolution enhancement for fiber bundle imaging using maximum a posteriori estimation

We propose a new framework to jointly improve spatial resolution and remove fixed structural patterns for coherent fiber bundle imaging systems, based on inverting a principled forward model. The forward model maps a high-resolution representation to multiple images modeling random probe motions. We then apply a point spread function to simulate low-resolution figure bundle image capture. Our forward model also uses a smoothing prior. We compute a maximum a posteriori (MAP) estimate of the high-resolution image from one or more low-resolution images using conjugate gradient descent. Unique aspects of our approach include (1) supporting a variety of possible applicable transformations; (2) applying principled forward modeling and MAP estimation to this domain. We test our method on data synthesized from the USAF target, data captured from a transmissive USAF target, and data from lens tissue. In the case of the USAF target and 16 low-resolution captures, spatial resolution is enhanced by a factor of 2.8.

Iterative l(1)-min algorithm for fixed pattern noise removal in fiber-bundle-based endoscopic imaging

In this study, we developed a signal processing method for fixed pattern noise removal in fiber-bundle-based endoscopic imaging. We physically acquired the fixed pattern of the fiber bundle and used it as a prior image in an l₁ norm minimization (l₁-min) algorithm. We chose an iterative shrinkage thresholding algorithm for l₁ norm minimization. In addition to fixed pattern noise removal, this method also improved image contrast while preserving spatial resolution. The effectiveness of this method was demonstrated on images obtained from a dark-field illuminated reflectance fiber-optic microscope (DRFM). The iterative l₁-min algorithm presented in this paper, in combination with the DRFM system that we previously developed, enables high-resolution, high-sensitivity, intrinsic-contrast, and in situ cellular imaging which has great potential in clinical diagnosis and biomedical research.

Image processing for cameras with fiber bundle image relay

Some high-performance imaging systems generate a curved focal surface and so are incompatible with focal plane arrays fabricated by conventional silicon processing. One example is a monocentric lens, which forms a wide field-of-view high-resolution spherical image with a radius equal to the focal length. Optical fiber bundles have been used to couple between this focal surface and planar image sensors. However, such fiber-coupled imaging systems suffer from artifacts due to image sampling and incoherent light transfer by the fiber bundle as well as resampling by the focal plane, resulting in a fixed obscuration pattern. Here, we describe digital image processing techniques to improve image quality in a compact 126° field-of-view, 30 megapixel panoramic imager, where a 12 mm focal length F/1.35 lens made of concentric glass surfaces forms a spherical image surface, which is fiber-coupled to six discrete CMOS focal planes. We characterize the locally space-variant system impulse response at various stages: monocentric lens image formation onto the 2.5 μm pitch fiber bundle, image transfer by the fiber bundle, and sensing by a 1.75 μm pitch backside illuminated color focal plane. We demonstrate methods to mitigate moiré artifacts and local obscuration, correct for sphere to plane mapping distortion and vignetting, and stitch together the image data from discrete sensors into a single panorama. We compare processed images from the prototype to those taken with a 10× larger commercial camera with comparable field-of-view and light collection.