Learning-based keypoint registration for fetoscopic mosaicking

PURPOSE

In twin-to-twin transfusion syndrome (TTTS), abnormal vascular anastomoses in the monochorionic placenta can produce uneven blood flow between the two fetuses. In the current practice, TTTS is treated surgically by closing abnormal anastomoses using laser ablation. This surgery is minimally invasive and relies on fetoscopy. Limited field of view makes anastomosis identification a challenging task for the surgeon.

METHODS

To tackle this challenge, we propose a learning-based framework for in vivo fetoscopy frame registration for field-of-view expansion. The novelties of this framework rely on a learning-based keypoint proposal network and an encoding strategy to filter (i) irrelevant keypoints based on fetoscopic semantic image segmentation and (ii) inconsistent homographies.

RESULTS

We validate our framework on a dataset of six intraoperative sequences from six TTTS surgeries from six different women against the most recent state-of-the-art algorithm, which relies on the segmentation of placenta vessels.

CONCLUSION

The proposed framework achieves higher performance compared to the state of the art, paving the way for robust mosaicking to provide surgeons with context awareness during TTTS surgery.

Toward a navigation framework for fetoscopy

PURPOSE

Fetoscopic laser photocoagulation of placental anastomoses is the most effective treatment for twin-to-twin transfusion syndrome (TTTS). A robust mosaic of placenta and its vascular network could support surgeons' exploration of the placenta by enlarging the fetoscope field-of-view. In this work, we propose a learning-based framework for field-of-view expansion from intra-operative video frames.

METHODS

While current state of the art for fetoscopic mosaicking builds upon the registration of anatomical landmarks which may not always be visible, our framework relies on learning-based features and keypoints, as well as robust transformer-based image-feature matching, without requiring any anatomical priors. We further address the problem of occlusion recovery and frame relocalization, relying on the computed features and their descriptors.

RESULTS

Experiments were conducted on 10 in-vivo TTTS videos from two different fetal surgery centers. The proposed framework was compared with several state-of-the-art approaches, achieving higher [Formula: see text] on 7 out of 10 videos and a success rate of [Formula: see text] in occlusion recovery.

CONCLUSION

This work introduces a learning-based framework for placental mosaicking with occlusion recovery from intra-operative videos using a keypoint-based strategy and features. The proposed framework can compute the placental panorama and recover even in case of camera tracking loss where other methods fail. The results suggest that the proposed framework has large potential to pave the way to creating a surgical navigation system for TTTS by providing robust field-of-view expansion.

Miniature Flexible Instrument with Fibre Bragg Grating-Based Triaxial Force Sensing for Intraoperative Gastric Endomicroscopy

Optical biopsy methods, such as probe-based endomicroscopy, can be used to identify early-stage gastric cancer in vivo. However, it is difficult to scan a large area of the gastric mucosa for mosaicking during endoscopy. In this work, we propose a miniaturised flexible instrument based on contact-aided compliant mechanisms and fibre Bragg grating (FBG) sensing for intraoperative gastric endomicroscopy. The instrument has a compact design with an outer diameter of 2.7 mm, incorporating a central channel with a diameter of 1.9 mm for the endomicroscopic probe to pass through. Experimental results show that the instrument can achieve raster trajectory scanning over a large tissue surface with a positioning accuracy of 0.5 mm. The tip force sensor provides a 4.6 mN resolution for the axial force and 2.8 mN for transverse forces. Validation with random samples shows that the force sensor can provide consistent and accurate three-axis force detection. Endomicroscopic imaging experiments were conducted, and the flexible instrument performed no gap scanning (mosaicking area more than 3 mm²) and contact force monitoring during scanning, demonstrating the potential of the system in clinical applications.

A fully-annotated imagery dataset of sublittoral benthic species in Svalbard, Arctic

Underwater imagery is widely used for a variety of applications in marine biology and environmental sciences, such as classification and mapping of seabed habitats, marine environment monitoring and impact assessment, biogeographic reconstructions in the context of climate change, etc. This approach is relatively simple and cost-effective, allowing the rapid collection of large amounts of data. However, due to the laborious and time-consuming manual analysis procedure, only a small part of the information stored in the archives of underwater images is retrieved. Emerging novel deep learning methods open up the opportunity for more effective, accurate and rapid analysis of seabed images than ever before. We present annotated images of the bottom macrofauna obtained from underwater video recorded in Spitsbergen island's European Arctic waters, Svalbard Archipelago. Our videos were filmed in both the photic and aphotic zones of polar waters, often influenced by melting glaciers. We used artificial lighting and shot close to the seabed (<1 m) to preserve natural colours and avoid the distorting effect of muddy water. The underwater video footage was captured using a remotely operated vehicle (ROV) and a drop-down camera. The footage was converted to 2D mosaic images of the seabed. 2D mosaics were manually annotated by several experts using the Labelbox tool and co-annotations were refined using the SurveyJS platform. A set of carefully annotated underwater images associated with the original videos can be used by marine biologists as a biological atlas, as well as practitioners in the fields of machine vision, pattern recognition, and deep learning as training materials for the development of various tools for automatic analysis of underwater imagery.

Effective deep learning training for single-image super-resolution in endomicroscopy exploiting video-registration-based reconstruction

Purpose

Probe-based confocal laser endomicroscopy (pCLE) is a recent imaging modality that allows performing in vivo optical biopsies. The design of pCLE hardware, and its reliance on an optical fibre bundle, fundamentally limits the image quality with a few tens of thousands fibres, each acting as the equivalent of a single-pixel detector, assembled into a single fibre bundle. Video registration techniques can be used to estimate high-resolution (HR) images by exploiting the temporal information contained in a sequence of low-resolution (LR) images. However, the alignment of LR frames, required for the fusion, is computationally demanding and prone to artefacts.

Methods

In this work, we propose a novel synthetic data generation approach to train exemplar-based Deep Neural Networks (DNNs). HR pCLE images with enhanced quality are recovered by the models trained on pairs of estimated HR images (generated by the video registration algorithm) and realistic synthetic LR images. Performance of three different state-of-the-art DNNs techniques were analysed on a Smart Atlas database of 8806 images from 238 pCLE video sequences. The results were validated through an extensive image quality assessment that takes into account different quality scores, including a Mean Opinion Score (MOS).

Results

Results indicate that the proposed solution produces an effective improvement in the quality of the obtained reconstructed image.

Conclusion

The proposed training strategy and associated DNNs allows us to perform convincing super-resolution of pCLE images.

Electronic supplementary material

The online version of this article (10.1007/s11548-018-1764-0) contains supplementary material, which is available to authorized users.

Retinal slit lamp video mosaicking

PURPOSE

To this day, the slit lamp remains the first tool used by an ophthalmologist to examine patient eyes. Imaging of the retina poses, however, a variety of problems, namely a shallow depth of focus, reflections from the optical system, a small field of view and non-uniform illumination. For ophthalmologists, the use of slit lamp images for documentation and analysis purposes, however, remains extremely challenging due to large image artifacts. For this reason, we propose an automatic retinal slit lamp video mosaicking, which enlarges the field of view and reduces amount of noise and reflections, thus enhancing image quality.

METHODS

Our method is composed of three parts: (i) viable content segmentation, (ii) global registration and (iii) image blending. Frame content is segmented using gradient boosting with custom pixel-wise features. Speeded-up robust features are used for finding pair-wise translations between frames with robust random sample consensus estimation and graph-based simultaneous localization and mapping for global bundle adjustment. Foreground-aware blending based on feathering merges video frames into comprehensive mosaics.

RESULTS

Foreground is segmented successfully with an area under the curve of the receiver operating characteristic curve of 0.9557. Mosaicking results and state-of-the-art methods were compared and rated by ophthalmologists showing a strong preference for a large field of view provided by our method.

CONCLUSIONS

The proposed method for global registration of retinal slit lamp images of the retina into comprehensive mosaics improves over state-of-the-art methods and is preferred qualitatively.