Image processing for cameras with fiber bundle image relay

Real-time placental vessel segmentation in fetoscopic laser surgery for Twin-to-Twin Transfusion Syndrome

Twin-to-Twin Transfusion Syndrome (TTTS) is a rare condition that affects about 15% of monochorionic pregnancies, in which identical twins share a single placenta. Fetoscopic laser photocoagulation (FLP) is the standard treatment for TTTS, which significantly improves the survival of fetuses. The aim of FLP is to identify abnormal connections between blood vessels and to laser ablate them in order to equalize blood supply to both fetuses. However, performing fetoscopic surgery is challenging due to limited visibility, a narrow field of view, and significant variability among patients and domains. In order to enhance the visualization of placental vessels during surgery, we propose TTTSNet, a network architecture designed for real-time and accurate placental vessel segmentation. Our network architecture incorporates a novel channel attention module and multi-scale feature fusion module to precisely segment tiny placental vessels. To address the challenges posed by FLP-specific fiberscope and amniotic sac-based artifacts, we employed novel data augmentation techniques. These techniques simulate various artifacts, including laser pointer, amniotic sac particles, and structural and optical fiber artifacts. By incorporating these simulated artifacts during training, our network architecture demonstrated robust generalizability. We trained TTTSNet on a publicly available dataset of 2060 video frames from 18 independent fetoscopic procedures and evaluated it on a multi-center external dataset of 24 in-vivo procedures with a total of 2348 video frames. Our method achieved significant performance improvements compared to state-of-the-art methods, with a mean Intersection over Union of 78.26% for all placental vessels and 73.35% for a subset of tiny placental vessels. Moreover, our method achieved 172 and 152 frames per second on an A100 GPU, and Clara AGX, respectively. This potentially opens the door to real-time application during surgical procedures. The code is publicly available at https://github.com/SanoScience/TTTSNet.

Self-Supervised Joint Learning for pCLE Image Denoising

Probe-based confocal laser endoscopy (pCLE) has emerged as a powerful tool for disease diagnosis, yet it faces challenges such as the formation of hexagonal patterns in images due to the inherent characteristics of fiber bundles. Recent advancements in deep learning offer promise in image denoising, but the acquisition of clean-noisy image pairs for training networks across all potential scenarios can be prohibitively costly. Few studies have explored training denoising networks on such pairs. Here, we propose an innovative self-supervised denoising method. Our approach integrates noise prediction networks, image quality assessment networks, and denoising networks in a collaborative, jointly trained manner. Compared to prior self-supervised denoising methods, our approach yields superior results on pCLE images and fluorescence microscopy images. In summary, our novel self-supervised denoising technique enhances image quality in pCLE diagnosis by leveraging the synergy of noise prediction, image quality assessment, and denoising networks, surpassing previous methods on both pCLE and fluorescence microscopy images.

Defect detection and response non-uniformity correction of a monocentric camera based on fiber optic relay imaging

The monocentric camera based on fiber relay imaging offers benefits of light weight, compact size envelope, vast field of view, and high resolution, which can fully fulfill the index requirements of space-based surveillance systems. However, the fiber optic plate's (FOP) defects will result in the loss of imaging data, and the FOP's discrete structural features will exacerbate the imaging's non-uniformity. A global defect detection approach based on manual threshold segmentation of saturated frames is suggested to detect FOP defect features. The suggested method's efficacy and accuracy are confirmed when compared to the classical Otsu algorithm. Additionally, through tests, the relative imaging response coefficients of each pixel are identified, the response non-uniformity of the pixels is corrected, and the whole image non-uniformity drops from 10.01% to 0.78%. The study in this paper expedites the use of fiber relay imaging-based monocentric cameras in the field of space-based surveillance, and the technique described in this paper is also appropriate for large-array optical fiber coupled relay image transmission systems.

Bio-inspired Compact, High-resolution Snapshot Hyperspectral Imaging System with 3D Printed Glass Lightguide Array

To address the major challenges to obtain high spatial resolution in snapshot hyperspectral imaging, 3D printed glass lightguide array has been developed to sample the intermediate image in high spatial resolution and redistribute the pixels in the output end to achieve high spectral resolution. Curved 3D printed lightguide array can significantly simplify the snapshot hyperspectral imaging system, achieve better imaging performance, and reduce the system complexity and cost. We have developed two-photon polymerization process to print glass lightguide array, and demonstrated the system performance with biological samples. This new snapshot technology will catalyze new hyperspectral imaging system development and open doors for new applications from UV to IR.

Research on the Coupled Modulation Transfer Function of the Discrete Sampling System with Hexagonal Fiber-Optic Imaging Bundles

In this study, we developed a numerical model of the coupled modulation transfer function (coupled-MTF) based on the discrete sampling system from the perspective of optical system imaging quality evaluation for coupled two-dimensional discrete sampling characteristics of the hexagonally aligned fiber-optic imaging bundles and CCD image elements. The results show that when the spatial frequency of the input target signal deviates from the Nyquist frequency by 1%, an increase in the number of fibers leads to a faster convergence of the oscillation of the coupled-MTF, and the coupled-MTF converges to a stable value when the number of fibers reaches 1000 × 1000. The deviation of the spatial frequency of the input target signal from the Nyquist frequency is within 1%, and the oscillatory convergence of the coupled-MTF accelerates with increasing deviation. The coupled-MTF oscillates with the deviation period of the wave peak of the input target signal from the initial position of the fiber center, and the theoretical oscillation spatial period is twice the fiber diameter. This study produces important guidelines for the selection of the number of fibers, input spatial frequency, and initial position deviation of the hexagonally arranged fiber imaging bundles.

Broadband high-spatial-resolution high-spectral-resolution flexible imaging spectrometer design study

A broadband high-spatial-resolution high-spectral-resolution flexible imaging spectrometer (B-2HSR-FIS) is presented, which includes two microlens arrays (MLAs), multiple fiber bundles, a scanning Fabry-Perot interferometer (FPI), a reflection grating, a cylindrical lens, and an area-array detector. The first MLA is arranged in a circular arc to obtain a field angle between 8° and 60° in the horizontal plane. The second MLA is arranged in a straight line. Each fiber bundle containing seven optical fibers is coupled to a separate microlens of the first MLA, subdividing the field angle of each microlens into seven smaller field angles to improve spatial resolution. The combination of a scanning FPI and a reflection grating enables the B-2HSR-FIS to obtain both high spectral resolution and broadband spectral range in the ultraviolet to near-infrared spectral region. Compared with all existing imaging spectrometers, the B-2HSR-FIS is the first to simultaneously obtain high spatial resolution, high spectral resolution, broadband spectral range, and moderate field angle, to the best of our knowledge. The B-2HSR-FIS has great potential for vision intelligence (e.g., as an eye of a robot).

Volumetric imaging efficiency: the fundamental limit to compactness of imaging systems

A new metric for imaging systems, the volumetric imaging efficiency (VIE), is introduced. It compares the compactness and capacity of an imager against fundamental limits imposed by diffraction. Two models are proposed for this fundamental limit based on an idealized thin-lens and the optical volume required to form diffraction-limited images. The VIE is computed for 2,871 lens designs and plotted as a function of FOV; this quantifies the challenge of creating compact, wide FOV lenses. We identify an empirical limit to the VIE given by VIE < 0.920 × 10^-0.582×FOV when using conventional bulk optics imaging onto a flat sensor. We evaluate VIE for lenses employing curved image surfaces and planar, monochromatic metasurfaces to show that these new optical technologies can surpass the limit of conventional lenses and yield >100x increase in VIE.

Design study of a compact ultra-wide-angle high-spatial-resolution high-spectral-resolution snapshot imaging spectrometer

An ultra-wide-angle high-spatial-resolution high-spectral-resolution snapshot imaging spectrometer (UWA-2HSR-SIS) is presented, which comprises a microlens array (MLA), multiple fiber bundles, a micro-cylindrical-lens array (MCLA), a cylindrical lens, a static grating interferometer (SGI), and an area-array detector. The MLA is arranged in a circular arc of 120° or more. The MCLA is arranged in a straight line. The SGI includes a fixed reflection grating in Littrow configuration, a beam splitter, and a fixed plane mirror. Each fiber bundle containing multiple optical fibers is coupled to a separate microlens of the MLA, subdividing the field angle of each microlens into multiple smaller field angles. The light passing through each subdivided smaller field angle of each microlens of the MLA is received by a separate part of the detector. The UWA-2HSR-SIS is a new concept that not only obtains both high spatial resolution and high spectral resolution based on a single sensor for the first time, but also has an ultra-wide field angle in the horizontal plane, can obtain spectral information covering the full spectral range of interest in real time, and is very stable against various disturbances. The UWA-2HSR-SIS has great potential for remote sensing electro-optical reconnaissance sensors in the visible and near-infrared region.

Analysis and characterization of high-resolution and high-aspect-ratio imaging fiber bundles

High-contrast imaging fiber bundles (FBs) are characterized and modeled for wide-angle and high-resolution imaging applications. Scanning electron microscope images of FB cross sections are taken to measure physical parameters and verify the variations of irregular fibers due to the fabrication process. Modal analysis tools are developed that include irregularities in the fiber core shapes and provide results in agreement with experimental measurements. The modeling demonstrates that the irregular fibers significantly outperform a perfectly regular "ideal" array. Using this method, FBs are designed that can provide high contrast with core pitches of only a few wavelengths of the guided light. Structural modifications of the commercially available FB can reduce the core pitch by 60% for higher resolution image relay.

Quantitative analysis and temperature-induced variations of moiré pattern in fiber-coupled imaging sensors

Imaging fiber bundles can map the curved image surface formed by some high-performance lenses onto flat focal plane detectors. The relative alignment between the focal plane array pixels and the quasi-periodic fiber-bundle cores can impose an undesirable space variant moiré pattern, but this effect may be greatly reduced by flat-field calibration, provided that the local responsivity is known. Here we demonstrate a stable metric for spatial analysis of the moiré pattern strength, and use it to quantify the effect of relative sensor and fiber-bundle pitch, and that of the Bayer color filter. We measure the thermal dependence of the moiré pattern, and the achievable improvement by flat-field calibration at different operating temperatures. We show that a flat-field calibration image at a desired operating temperature can be generated using linear interpolation between white images at several fixed temperatures, comparing the final image quality with an experimentally acquired image at the same temperature.