Shape Acquisition and Registration for 3D Endoscope Based on Grid Pattern Projection

Fast Multichannel Inverse Design through Augmented Partial Factorization

Computer-automated design and discovery have led to high-performance nanophotonic devices with diverse functionalities. However, massively multichannel systems such as metasurfaces controlling many incident angles and photonic-circuit components coupling many waveguide modes still present a challenge. Conventional methods require Min forward simulations and Min adjoint simulations-2Min simulations in total-to compute the objective function and its gradient for a design involving the response to Min input channels. Here, we develop a formalism that uses the recently proposed augmented partial factorization method to obtain both the objective function and its gradient for a massively multichannel system in a single or a few simulations, achieving over 2 orders of magnitude speedup and reduced memory usage. We use this method to inverse design a metasurface beam splitter that separates the incident light to the target diffraction orders for all incident angles of interest, a key component of the dot projector for 3D sensing. This formalism enables efficient inverse design for a wide range of multichannel optical systems.

Monocular endoscope 6-DoF tracking with constrained evolutionary stochastic filtering

Monocular endoscopic 6-DoF camera tracking plays a vital role in surgical navigation that involves multimodal images to build augmented or virtual reality surgery. Such a 6-DoF camera tracking generally can be formulated as a nonlinear optimization problem. To resolve this nonlinear problem, this work proposes a new pipeline of constrained evolutionary stochastic filtering that originally introduces spatial constraints and evolutionary stochastic diffusion to deal with particle degeneracy and impoverishment in current stochastic filtering methods. With its application to endoscope 6-DoF tracking and validation on clinical data including more than 59,000 endoscopic video frames acquired from various surgical procedures, the experimental results demonstrate the effectiveness of the new pipeline that works much better than state-of-the-art tracking methods. In particular, it can significantly improve the accuracy of current monocular endoscope tracking approaches from (4.83 mm, 10.2∘) to (2.78 mm, 7.44∘).

Fully Auto-calibrated Active-stereo-based 3D Endoscopic System using Correspondence Estimation with Graph Convolutional Network

We have developed a series of 3D endoscopic systems where a micro-sized pattern projector is inserted through the instrument channel of the endoscope and shapes are reconstructed by a structured light technique using captured images of the endoscopic camera. One problem of the previous works is that the accuracy of shape reconstruction is low, because the projector cannot be fixed to the endoscope, and thus, the pose of the pattern projector w.r.t. the camera cannot be pre-calibrated. In this paper, we propose a method to auto-calibrate the pose of the projector without using any special devices nor manual process. Since the technique is one-shot, multiple shapes can be reconstructed from an image sequence and a large 3D scene can be recovered by merging them. Experiments are conducted using the real system.

3D Reconstruction of Whole Stomach from Endoscope Video Using Structure-from-Motion

Gastric endoscopy is a common clinical practice that enables medical doctors to diagnose the stomach inside a body. In order to identify a gastric lesion's location such as early gastric cancer within the stomach, this work addressed to reconstruct the 3D shape of a whole stomach with color texture information generated from a standard monocular endoscope video. Previous works have tried to reconstruct the 3D structures of various organs from endoscope images. However, they are mainly focused on a partial surface. In this work, we investigated how to enable structure-from-motion (SfM) to reconstruct the whole shape of a stomach from a standard endoscope video. We specifically investigated the combined effect of chromo-endoscopy and color channel selection on SfM. Our study found that 3D reconstruction of the whole stomach can be achieved by using red channel images captured under chromo-endoscopy by spreading indigo carmine (IC) dye on the stomach surface.