Endoscopic Vision Augmentation Using Multiscale Bilateral-Weighted Retinex for Robotic Surgery

LSKANet: Long Strip Kernel Attention Network for Robotic Surgical Scene Segmentation

Surgical scene segmentation is a critical task in Robotic-assisted surgery. However, the complexity of the surgical scene, which mainly includes local feature similarity (e.g., between different anatomical tissues), intraoperative complex artifacts, and indistinguishable boundaries, poses significant challenges to accurate segmentation. To tackle these problems, we propose the Long Strip Kernel Attention network (LSKANet), including two well-designed modules named Dual-block Large Kernel Attention module (DLKA) and Multiscale Affinity Feature Fusion module (MAFF), which can implement precise segmentation of surgical images. Specifically, by introducing strip convolutions with different topologies (cascaded and parallel) in two blocks and a large kernel design, DLKA can make full use of region- and strip-like surgical features and extract both visual and structural information to reduce the false segmentation caused by local feature similarity. In MAFF, affinity matrices calculated from multiscale feature maps are applied as feature fusion weights, which helps to address the interference of artifacts by suppressing the activations of irrelevant regions. Besides, the hybrid loss with Boundary Guided Head (BGH) is proposed to help the network segment indistinguishable boundaries effectively. We evaluate the proposed LSKANet on three datasets with different surgical scenes. The experimental results show that our method achieves new state-of-the-art results on all three datasets with improvements of 2.6%, 1.4%, and 3.4% mIoU, respectively. Furthermore, our method is compatible with different backbones and can significantly increase their segmentation accuracy. Code is available at https://github.com/YubinHan73/LSKANet.

Deep polyp image enhancement using region of interest with paired supervision

Endoscopic medical imaging in complex curved intestinal structures are prone to uneven illumination, low contrast and lack of texture information. These problems may lead to diagnostic challenges. This paper described the first supervised deep learning based image fusion framework to enable the polyp region highlight through a global image enhancement and a local region of interest (ROI) with paired supervision. Firstly, we conducted a dual attention based network in global image enhancement. The Detail Attention Maps was used to preserve more image details and the Luminance Attention Maps was used to adjust the global illumination of the image. Secondly, we adopted the advanced polyp segmentation network ACSNet to obtain the accurate mask image of lesion region in local ROI acquisition. Finally, a new image fusion strategy was proposed to realize the local enhancement effect of polyp image. Experimental results show that our method can highlight the local details of the lesion area better and reach the optimal comprehensive performance with comparing with 16 traditional and state-of-the-art enhancement algorithms. And 8 doctors and 12 medical students were asked to evaluate our method for assisting clinical diagnosis and treatment effectively. Furthermore, the first paired image dataset LHI was constructed, which will be made available as an open source to research communities.

A Method for Local Contrast Enhancement of Endoscopic Images Based on Color Tensor Transformation into a Matrix of Color Vectors’ Modules Using a Sliding Window

A new method aimed at endoscopic color images’ local contrast enhancement is proposed, based on local sliding histogram equalization with adaptive threshold limitation, color distortions correction, and image brightness preservation. For this, the original RGB image, represented as a tensor of size M × N × 3, is transformed into a matrix of size M × N, composed by the color vectors’ modules. As a result of local contrast enhancement, the obtained color vectors are symmetrical in respect of the input ones, because they satisfy the requirement for invariance after rotation. To enhance the local contrast, recursive local histogram equalization with adaptive thresholding is applied to each matrix element. This threshold divides the histogram into two regions of equal areas. A new metric for local contrast enhancement evaluation based on the mean square difference entropy is proposed. The presented new method is characterized by low computational complexity, due to the lack of direct and inverse color conversion and the possibility for adaptive local contrast enhancement, which is essential for accurate medical diagnosis based on endoscopic images analysis. In addition, the presented method performs both the correction of color distortions and the brightness preservation of each pixel.

Endoscopic image luminance enhancement based on the inverse square law for illuminance and retinex

BACKGROUND

In a single-port robotic system where the 3D endoscope possesses two bending segments, only point light sources can be integrated at the tip due to space limitations. However, point light sources usually provide non-uniform illumination, causing the endoscopic images to appear bright in the centre and dark near the corners.

METHODS

Based on the inverse square law for illuminance, an initial luminance weighting is first proposed to increase the image luminance uniformity. Then, a saturation-based model is proposed to finalise the luminance weighting to avoid overexposure and colour discrepancy, while the single-scale retinex (SSR) scheme is employed for noise control.

RESULTS

Via qualitative and quantitative comparisons, the proposed method performs effectively in enhancing the luminance and uniformity of endoscopic images, in terms of both visual perception and objective assessment.

CONCLUSIONS

The proposed method can effectively reduce the image degradation caused by point light sources.

Computer-Aided Image Enhanced Endoscopy Automated System to Boost Polyp and Adenoma Detection Accuracy

Colonoscopy is the gold standard to detect colon polyps prematurely. Early detection, characterization and resection of polyps decrease colon cancer incidence. Colon polyp missing rate remains high despite novel methods development. Narrowed-band imaging (NBI) is one of the image enhance techniques used to boost polyp detection and characterization, which uses special filters to enhance the contrast of the mucosa surface and vascular pattern of the polyp. However, the single-button-activated system is not convenient for a full-time colonoscopy operation. We selected three methods to simulate the NBI system: Color Transfer with Mean Shift (CTMS), Multi-scale Retinex with Color Restoration (MSRCR), and Gamma and Sigmoid Conversions (GSC). The results show that the classification accuracy using the original images is the lowest. All color transfer methods outperform the original images approach. Our results verified that the color transfer has a positive impact on the polyp identification and classification task. Combined analysis results of the mAP and the accuracy show an excellent performance of the MSRCR method.

TXI: Texture and Color Enhancement Imaging for Endoscopic Image Enhancement

Recognition of lesions with subtle morphological and/or color changes during white light imaging (WLI) endoscopy remains a challenge. Often the endoscopic image suffers from nonuniform illumination across the image due to curvature in the lumen and the direction of the illumination light of the endoscope. We propose an image enhancement technology to resolve the drawbacks above called texture and color enhancement imaging (TXI). TXI is designed to enhance three image factors in WLI (texture, brightness, and color) in order to clearly define subtle tissue differences. In our proposed method, retinex-based enhancement is employed in the chain of endoscopic image processing. Retinex-based enhancement is combined with color enhancement to greatly accentuate color tone differences of mucosal surfaces. We apply TXI to animal endoscopic images and evaluate the performance of TXI compared with conventional endoscopic enhancement technologies, conventionally used techniques for real-world image processing, and newly proposed techniques for surgical endoscopic image augmentation. Our experimental results show that TXI can enhance brightness selectively in dark areas of an endoscopic image and can enhance subtle tissue differences such as slight morphological or color changes while simultaneously preventing over-enhancement. These experimental results demonstrate the potential of the proposed TXI algorithm as a future clinical tool for detecting gastrointestinal lesions having difficult-to-recognize tissue differences.

An Image Contrast Measure Based on Retinex Principles

The image contrast is a feature capturing the variation of the image signal across the space. Such a feature is very useful to describe the local image structure at different scales and thus it is relevant to many computer vision applications, like image/texture retrieval and object recognition. In this work, we present MiRCo, a novel measure of image contrast derived from the Retinex theory. MiRCo is robust against in-plane rotations and light changes at multiple scales. Thanks to these properties, MiRCo enables an accurate and robust description of the local image structure. Here we describe and discuss the mathematical insights of MiRCo also in comparison with other popular contrast measures.