Intraoperative Robotic-Assisted Large-Area High-Speed Microscopic Imaging and Intervention

An FPGA-Implemented Antinoise Fuzzy Recurrent Neural Network for Motion Planning of Redundant Robot Manipulators

When a robot completes end-effector tasks, internal error noises always exist. To resist internal error noises of robots, a novel fuzzy recurrent neural network (FRNN) is proposed, designed, and implemented on field-programmable gated array (FPGA). The implementation is pipeline-based, which guarantees the order of overall operations. The data processing is based on across-clock domain, which is beneficial for computing units' acceleration. Compared with traditional gradient-based neural networks (NNs) and zeroing neural networks (ZNNs), the proposed FRNN has faster convergence rate and higher correctness. Practical experiments on a 3 degree-of-freedom (DOs) planar robot manipulator show that the proposed fuzzy RNN coprocessor needs 496 lookup table random access memories (LUTRAMs), 205.5 block random access memories (BRAMs), 41384 lookup tables (LUTs), and 16743 flip-flops (FFs) of the Xilinx XCZU9EG chip.

Real-timing processing of fiber bundle endomicroscopy images in Python using PyFibreBundle

Fiber imaging bundles allow the transfer of optical images from place-to-place along narrow and flexible conduits. Traditionally used extensively in medical endoscopy, bundles are now finding new applications in endoscopic microscopy and other emerging techniques. PyFibreBundle is an open-source Python package for fast processing of images acquired through imaging bundles. This includes detection and removal of the fiber core pattern by filtering or interpolation, and application of background and flat-field corrections. It also allows images to be stitched together to create mosaics and resolution to be improved by combining multiple shifted images. This paper describes the technical implementation of PyFibreBundle and provides example results from three endomicroscopy imaging systems: color transmission, monochrome transmission, and confocal fluorescence. This allows various processing options to be compared quantitatively and qualitatively, and benchmarking demonstrates that PyFibreBundle can achieve state-of-the-art performance in an open-source package. The paper demonstrates core removal by interpolation and mosaicing at over 100 fps, real-time multi-frame resolution enhancement and the first demonstration of real-time endomicroscopy image processing, including core removal, on a Raspberry Pi single board computer. This demonstrates that PyFibreBundle is potentially a valuable tool for the development of low-cost, high-performance fiber bundle imaging systems.

Miniature Flexible Instrument with Fibre Bragg Grating-Based Triaxial Force Sensing for Intraoperative Gastric Endomicroscopy

Optical biopsy methods, such as probe-based endomicroscopy, can be used to identify early-stage gastric cancer in vivo. However, it is difficult to scan a large area of the gastric mucosa for mosaicking during endoscopy. In this work, we propose a miniaturised flexible instrument based on contact-aided compliant mechanisms and fibre Bragg grating (FBG) sensing for intraoperative gastric endomicroscopy. The instrument has a compact design with an outer diameter of 2.7 mm, incorporating a central channel with a diameter of 1.9 mm for the endomicroscopic probe to pass through. Experimental results show that the instrument can achieve raster trajectory scanning over a large tissue surface with a positioning accuracy of 0.5 mm. The tip force sensor provides a 4.6 mN resolution for the axial force and 2.8 mN for transverse forces. Validation with random samples shows that the force sensor can provide consistent and accurate three-axis force detection. Endomicroscopic imaging experiments were conducted, and the flexible instrument performed no gap scanning (mosaicking area more than 3 mm²) and contact force monitoring during scanning, demonstrating the potential of the system in clinical applications.

Laparoscopic surgical device with modular end tools for real-time endomicroscopy and therapy

In recent years, laparoscopic surgery attracts considerable attention for its distinct advantages owing to minimized invasiveness. However, traditional surgical instruments have a single function and lack the ability to provide high-resolution images for real-time tissue assessment. Thus, it remains a challenge to perform complex procedures with the traditional surgical instruments. In this paper, a novel laparoscopic surgical device for real-time endomicroscopy and therapy is proposed. The device is integrated with a visualization unit to provide endoscopic images in a constrained cavity for real-time observation. Moreover, modular end tools can be assembled and switched with the multi-DOF manipulator. The experimental results demonstrate that the surface-scanning tool can perform smooth scanning and obtain a large area mosaic of 6.99 mm² in 40.2 s for tissue assessment. The spring-loaded forceps can generate an output force of 1.47 N to manipulate tissue. These experimental results demonstrate the potential value of the device for intraoperative endomicroscopy-guided surgery and robot-assisted surgery.

Robust Mosaicing of Endomicroscopic Videos via Context-Weighted Correlation Ratio

Probe-based confocal laser endomicroscopy (pCLE) is a promising imaging tool that provides in situ and in vivo optical imaging to perform real-time pathological assessments. However, due to limited field of view, it is difficult for clinicians to get a full understanding of the scanned tissues. In this paper, we develop a novel mosaicing framework to assemble all frame sequences into a full view image. First, a hybrid rigid registration that combines feature matching and template matching is presented to achieve a global alignment of all frames. Then, the parametric free-form deformation (FFD) model with a multiresolution architecture is implemented to accommodate non-rigid tissue distortions. More importantly, we devise a robust similarity metric called context-weighted correlation ratio (CWCR) to promote registration accuracy, where spatial and geometric contexts are incorporated into the estimation of functional intensity dependence. Experiments on both robotic setup and manual manipulation have demonstrated that the proposed scheme significantly precedes some state-of-the-art mosaicing schemes in the presence of intensity fluctuations, insufficient overlap and tissue distortions. Moreover, the comparisons of the proposed CWCR metric and two other metrics have validated the effectiveness of the context-weighted strategy in quantifying the differences between two frames. Benefiting from more rational and delicate mosaics, the proposed scheme is more suitable to instruct diagnosis and treatment during optical biopsies.

Ex vivo confocal microscopy performs real-time assessment of renal biopsy in non-neoplastic diseases

BACKGROUND

Ex vivo confocal microscopy is a technique for tissue examination, which generates images of fresh samples with an optical resolution comparable to those obtained by conventional pathology. The objective of this study was to evaluate the feasibility of using ex vivo confocal microscopy in fusion mode (reflectance and fluorescence) and the H&E-like digital staining that is obtained for the analysis of non-neoplastic kidney biopsies.

METHODS

Twenty-four renal samples acquired from autopsies were scanned in a 4th generation ex vivo confocal microscopy device. The imaging process was completed in an average of three minutes.

RESULTS

Confocal images correlated very well to the corresponding conventional histological sections, both in normal tissue and in chronic lesions (glomerulosclerosis, fibrosis and tubular atrophy). The ex vivo confocal microscopy protocol did not add artifacts to the sample for the ulterior study with light microscopy, nor to the histochemical or immunohistochemical studies.

CONCLUSION

The ease and speed of grayscale and fluorescence image acquisition, together with the quality of the H&E-like digitally stained images obtained with this approach, suggest that this technique shows promise for use in clinical nephrology and renal transplantation.

Hand-Held Instrument with Integrated Parallel Mechanism for Active Tremor Compensation During Microsurgery

Physiological hand tremor seriously influences the surgical instrument's tip positioning accuracy during microsurgery. To solve this problem, hand-held active tremor compensation instruments are developed to improve tip positioning accuracy during microsurgery. This paper presents the design and performance of a new hand-held instrument that aims to stabilize hand tremors and increase accuracy in microsurgery. The key components are a three degrees of freedom (DOF) integrated parallel manipulator and a high-performance inertial measurement unit (IMU). The IMU was developed to sense the 3-DOF motion of the instrument tip. A customized filter was applied to extract specific hand tremor motion. Then, the instrument was employed to generate the reverse motion simultaneously to reduce tremor motion. Experimental results show that the tremor compensation mechanism is effective. The average RMS reduction ratio of bench test is 56.5% that is a significant tremor reduction ratio. For hand-held test, it has an average RMS reduction ratio of 41.0%. Hence, it could reduce hand tremor magnitudes by 31.7% RMS in 2-DOF.

En-face optical coherence tomography/fluorescence endomicroscopy for minimally invasive imaging using a robotic scanner

We report a compact rigid instrument capable of delivering en-face optical coherence tomography (OCT) images alongside (epi)-fluorescence endomicroscopy (FEM) images by means of a robotic scanning device. Two working imaging channels are included: one for a one-dimensional scanning, forward-viewing OCT probe and another for a fiber bundle used for the FEM system. The robotic scanning system provides the second axis of scanning for the OCT channel while allowing the field of view (FoV) of the FEM channel to be increased by mosaicking. The OCT channel has resolutions of 25  /  60  μm (axial/lateral) and can provide en-face images with an FoV of 1.6  ×  2.7  mm2. The FEM channel has a lateral resolution of better than 8  μm and can generate an FoV of 0.53  ×  3.25  mm2 through mosaicking. The reproducibility of the scanning was determined using phantoms to be better than the lateral resolution of the OCT channel. Combined OCT and FEM imaging were validated with ex-vivo ovine and porcine tissues, with the instrument mounted on an arm to ensure constant contact of the probe with the tissue. The OCT imaging system alone was validated for in-vivo human dermal imaging with the handheld instrument. In both cases, the instrument was capable of resolving fine features such as the sweat glands in human dermal tissue and the alveoli in porcine lung tissue.

Using Brain Activation to Evaluate Arrangements Aiding Hand-Eye Coordination in Surgical Robot Systems

GOAL

To realize intuitive, minimally invasive surgery, surgical robots are often controlled using master-slave systems. However, the surgical robot's structure often differs from that of the human body, so the arrangement between the monitor and master must reflect this physical difference. In this study, we validate the feasibility of an embodiment evaluation method that determines the arrangement between the monitor and master. In our constructed cognitive model, the brain's intraparietal sulcus activates significantly when somatic and visual feedback match. Using this model, we validate a cognitively appropriate arrangement between the monitor and master.

METHODS

In experiments, we measure participants' brain activation using an imaging device as they control the virtual surgical simulator. Two experiments are carried out that vary the monitor and hand positions.

CONCLUSION

There are two common arrangements of the monitor and master at the brain activation's peak: One is placing the monitor behind the master, so the user feels that the system is an extension of his arms into the monitor; the other arranges the monitor in front of the master, so the user feels the correspondence between his own arm and the virtual arm in the monitor.

SIGNIFICANCE

From these results, we conclude that the arrangement between the monitor and master impacts embodiment, enabling the participant to feel apparent posture matches in master-slave surgical robot systems.

Modular Robotic Scanning Device for Real-Time Gastric Endomicroscopy

Optical biopsy methods, such as probe-based endomicroscopy, can be used for the identification of early mucosal dysplasia in various gastrointestinal conditions and have potential applications in the screening of early-stage gastric cancer in vivo. However, it is difficult to scan a large area of the gastric mucosa for mosaicing during standard endoscopy. This paper proposes a novel 'snap-on' robotic scanning device that can integrate distally with a commercial endoscope. A customized low-cost endomicroscopy system is used for obtaining micro imaging. The developed device could scan a large area of gastric tissue during standard endoscopy. The device achieves positioning accuracy that is less than 0.23 mm. Experimental results showed that the device could achieve large area mosaicing (15.8-18.6 mm²) and demonstrated the potential clinical value of the device for real-time gastric tissue identification and margin assessment. This approach presents an important alternative to current histology techniques for gastrointestinal tract diagnosis.