Depixelation and enhancement of fiber bundle images by bundle rotation

Optical fiber bundle differential compressive imaging

We present a differential compressive imaging method for an optical fiber bundle (OFB), which provides a solution for an ultrathin bend-resistant endoscope with high resolution. This method uses an OFB and a diffuser to generate speckle illumination patterns. Differential operation is additionally applied to the speckle patterns to produce sensing matrices, by which the correlation between the matrices is greatly reduced from 0.875 to 0.0275, which ensures the high quality of image reconstruction. Pixilation artifacts from the fiber core arrangement are also effectively eliminated with this configuration. We demonstrate high-resolution reconstruction of images of 132 × 132 pixels with a compression rate of 12% using 77 fiber cores, the total diameter of which is only about 91 µm. An experimental verification proves that this method is tolerant to a limited degree of fiber bending, which provides a potential approach for robust high-resolution fiber endoscopy.

Fiberscopic pattern removal for optimal coverage in 3D bladder reconstructions of fiberscope cystoscopy videos

Purpose

In the current clinical standard of care, cystoscopic video is not routinely saved because it is cumbersome to review. Instead, clinicians rely on brief procedure notes and still frames to manage bladder pathology. Preserving discarded data via 3D reconstructions, which are convenient to review, has the potential to improve patient care. However, many clinical videos are collected by fiberscopes, which are lower cost but induce a pattern on frames that inhibit 3D reconstruction. The aim of our study is to remove the honeycomb-like pattern present in fiberscope-based cystoscopy videos to improve the quality of 3D bladder reconstructions.

Approach

Our study introduces an algorithm that applies a notch filtering mask in the Fourier domain to remove the honeycomb-like pattern from clinical cystoscopy videos collected by fiberscope as a preprocessing step to 3D reconstruction. We produce 3D reconstructions with the video before and after removing the pattern, which we compare with a metric termed the area of reconstruction coverage (A RC ), defined as the surface area (in pixels) of the reconstructed bladder. All statistical analyses use paired t -tests.

Results

Preprocessing using our method for pattern removal enabled reconstruction for all (n = 5 ) cystoscopy videos included in the study and produced a statistically significant increase in bladder coverage (p = 0.018 ).

Conclusions

This algorithm for pattern removal increases bladder coverage in 3D reconstructions and automates mask generation and application, which could aid implementation in time-starved clinical environments. The creation and use of 3D reconstructions can improve documentation of cystoscopic findings for future surgical navigation, thus improving patient treatment and outcomes.

Real-timing processing of fiber bundle endomicroscopy images in Python using PyFibreBundle

Fiber imaging bundles allow the transfer of optical images from place-to-place along narrow and flexible conduits. Traditionally used extensively in medical endoscopy, bundles are now finding new applications in endoscopic microscopy and other emerging techniques. PyFibreBundle is an open-source Python package for fast processing of images acquired through imaging bundles. This includes detection and removal of the fiber core pattern by filtering or interpolation, and application of background and flat-field corrections. It also allows images to be stitched together to create mosaics and resolution to be improved by combining multiple shifted images. This paper describes the technical implementation of PyFibreBundle and provides example results from three endomicroscopy imaging systems: color transmission, monochrome transmission, and confocal fluorescence. This allows various processing options to be compared quantitatively and qualitatively, and benchmarking demonstrates that PyFibreBundle can achieve state-of-the-art performance in an open-source package. The paper demonstrates core removal by interpolation and mosaicing at over 100 fps, real-time multi-frame resolution enhancement and the first demonstration of real-time endomicroscopy image processing, including core removal, on a Raspberry Pi single board computer. This demonstrates that PyFibreBundle is potentially a valuable tool for the development of low-cost, high-performance fiber bundle imaging systems.

Contrastive Adversarial Learning for Endomicroscopy Imaging Super-Resolution

Endomicroscopy is an emerging imaging modality for real-time optical biopsy. One limitation of existing endomicroscopy based on coherent fibre bundles is that the image resolution is intrinsically limited by the number of fibres that can be practically integrated within the small imaging probe. To improve the image resolution, Super-Resolution (SR) techniques combined with image priors can enhance the clinical utility of endomicroscopy whereas existing SR algorithms suffer from the lack of explicit guidance from ground truth high-resolution (HR) images. In this article, we propose an unsupervised SR pipeline to allow stable offline and kernel-generic learning. Our method takes advantage of both internal statistics and external cross-modality priors. To improve the joint learning process, we present a Sharpness-aware Contrastive Generative Adversarial Network (SCGAN) with two dedicated modules, a sharpness-aware generator and a contrastive-learning discriminator. In the generator, an auxiliary task of sharpness discrimination is formulated to facilitate internal learning by considering the rankings of training instances in various sharpness levels. In the discriminator, we design a contrastive-learning module to mitigate the ill-posed nature of SR tasks via constraints from both positive and negative images. Experiments on multiple datasets demonstrate that SCGAN reduces the performance gap between previous unsupervised approaches and the upper bounds defined in supervised settings by more than 50%, delivering a new state-of-the-art performance score for endomicroscopy super-resolution. Further application on a realistic Voronoi-based pCLE downsampling kernel proves that SCGAN attains PSNR of 35.851 dB, improving 5.23 dB compared with the traditional Delaunay interpolation.

Block-based compressed sensing for fast optic fiber bundle imaging with high spatial resolution

The resolution of traditional fiber bundle imaging is usually limited by the density and the diameter of the fiber cores. To improve the resolution, compression sensing was introduced to resolve multiple pixels from a single fiber core, but current methods have the drawbacks of excessive sampling and long reconstruction time. In this paper, we present, what we believe to be, a novel block-based compressed sensing scheme for fast realization of high-resolution optic fiber bundle imaging. In this method, the target image is segmented into multiple small blocks, each of which covers the projection area of one fiber core. All block images are independently and simultaneously sampled and the intensities are recorded by a two-dimensional detector after they are collected and transmitted through corresponding fiber cores. Because the size of sampling patterns and the sampling numbers are greatly reduced, the reconstruction complexity and reconstruction time are also decreased. According to the simulation analysis, our method is 23 times faster than the current compressed sensing optical fiber imaging for reconstructing a fiber image of 128 × 128 pixels, while the sampling number is only 0.39%. Experiment results demonstrate that the method is also effective for reconstructing large target images and the number of sampling does not increase with the size of the image. Our finding may provide a new idea for high-resolution real-time imaging of fiber bundle endoscope.

Computational imaging with spectral coding increases the spatial resolution of fiber optic bundles

Fiber optic bundles are used in narrow-diameter medical and industrial instruments for acquiring images from confined locations. Images transmitted through these bundles contain only one pixel of information per fiber core and fail to capture information from the cladding region between cores. Both factors limit the spatial resolution attainable with fiber bundles. We show here that computational imaging (CI) can be combined with spectral coding to overcome these two fundamental limitations and improve spatial resolution in fiber bundle imaging. By acquiring multiple images of a scene with a high-resolution mask pattern imposed, up to 17 pixels of information can be recovered from each fiber core. A dispersive element at the distal end of the bundle imparts a wavelength-dependent lateral shift on light from the object. This enables light that would otherwise be lost at the inter-fiber cladding to be transmitted through adjacent fiber cores. We experimentally demonstrate this approach using synthetic and real objects. Using CI with spectral coding, object features 5× smaller than individual fiber cores were resolved, whereas conventional imaging could only resolve features at least 1.5× larger than each core. In summary, CI combined with spectral coding provides an approach for overcoming the two fundamental limitations of fiber optic bundle imaging.

Fiber Bundle Image Reconstruction Using Convolutional Neural Networks and Bundle Rotation in Endomicroscopy

Fiber-bundle endomicroscopy has several recognized drawbacks, the most prominent being the honeycomb effect. We developed a multi-frame super-resolution algorithm exploiting bundle rotation to extract features and reconstruct underlying tissue. Simulated data was used with rotated fiber-bundle masks to create multi-frame stacks to train the model. Super-resolved images are numerically analyzed, which demonstrates that the algorithm can restore images with high quality. The mean structural similarity index measurement (SSIM) improved by a factor of 1.97 compared with linear interpolation. The model was trained using images taken from a single prostate slide, 1343 images were used for training, 336 for validation, and 420 for testing. The model had no prior information about the test images, adding to the robustness of the system. Image reconstruction was completed in 0.03 s for 256 × 256 images indicating future real-time performance is within reach. The combination of fiber bundle rotation and multi-frame image enhancement through machine learning has not been utilized before in an experimental setting but could provide a much-needed improvement to image resolution in practice.

Honeycomb Artifact Removal Using Convolutional Neural Network for Fiber Bundle Imaging

We present a new deep learning framework for removing honeycomb artifacts yielded by optical path blocking of cladding layers in fiber bundle imaging. The proposed framework, HAR-CNN, provides an end-to-end mapping from a raw fiber bundle image to an artifact-free image via a convolution neural network (CNN). The synthesis of honeycomb patterns on ordinary images allows conveniently learning and validating the network without the enormous ground truth collection by extra hardware setups. As a result, HAR-CNN shows significant performance improvement in honeycomb pattern removal and also detailed preservation for the 1961 USAF chart sample, compared with other conventional methods. Finally, HAR-CNN is GPU-accelerated for real-time processing and enhanced image mosaicking performance.

Learned end-to-end high-resolution lensless fiber imaging towards real-time cancer diagnosis

Recent advances in label-free histology promise a new era for real-time diagnosis in neurosurgery. Deep learning using autofluorescence is promising for tumor classification without histochemical staining process. The high image resolution and minimally invasive diagnostics with negligible tissue damage is of great importance. The state of the art is raster scanning endoscopes, but the distal lens optics limits the size. Lensless fiber bundle endoscopy offers both small diameters of a few 100 microns and the suitability as single-use probes, which is beneficial in sterilization. The problem is the inherent honeycomb artifacts of coherent fiber bundles (CFB). For the first time, we demonstrate an end-to-end lensless fiber imaging with exploiting the near-field. The framework includes resolution enhancement and classification networks that use single-shot CFB images to provide both high-resolution imaging and tumor diagnosis. The well-trained resolution enhancement network not only recovers high-resolution features beyond the physical limitations of CFB, but also helps improving tumor recognition rate. Especially for glioblastoma, the resolution enhancement network helps increasing the classification accuracy from 90.8 to 95.6%. The novel technique enables histological real-time imaging with lensless fiber endoscopy and is promising for a quick and minimally invasive intraoperative treatment and cancer diagnosis in neurosurgery.

Depixelation of coherent fiber bundle imaging by fiber-core-targeted scanning

A novel fast proximal scanning method, to the best of our knowledge, termed fiber-core-targeted scanning (FCTS), is proposed for illuminating individual fiber cores sequentially to remove the pixelation effect in fiber bundle (FB) imaging. FCTS is based on a galvanometer scanning system. Through a dynamic control of the scan trajectory and speed using the prior knowledge of fiber core positions, FCTS experimentally verifies a precise sequential delivery of laser pulses into fiber cores at a maximal speed of 45,000 cores per second. By applying FCTS on a FB-based photoacoustic forward-imaging probe, the results demonstrate that FCTS eliminates the pixelation effect and improves the imaging quality.