Distributed visual positioning for surgical instrument tracking

In clinical operations, it is crucial for surgeons to know the location of the surgical instrument. Traditional positioning systems have difficulty dealing with camera occlusion, marker occlusion, and environmental interference. To address these issues, we propose a distributed visual positioning system for surgical instrument tracking in surgery. First, we design the marker pattern with a black and white triangular grid and dot that can be adapted to various instrument surfaces and improve the marker location accuracy of the feature. The cross-points in the marker are the features that each feature has a unique ID. Furthermore, we proposed detection and identification for the position-sensing marker to realize the accurate location and identification of features. Second, we introduce multi Perspective-n-Point (mPnP) method, which fuses feature coordinates from all cameras to deduce the final result directly by the intrinsic and extrinsic parameters. This method provides a reliable initial value for the Bundle Adjustment algorithms. During instrument tracking, we assess the motion state of the instrument and select either dynamic or static Kalman filtering to mitigate any jitter in the instrument's movement. The core algorithms comparison experiment indicates our positioning algorithm has a lower reprojection error comparison to the mainstream algorithms. A series of quantitative experiments showed that the proposed system positioning error is below 0.207 mm, and the run time is below 118.842 ms. The results demonstrate the tremendous clinical application potential of our system providing accurate positioning of instruments promoting the efficiency and safety of clinical surgery.

Image-guided navigation system for minimally invasive total hip arthroplasty (MITHA) using an improved position-sensing marker

PURPOSE

Minimally invasive total hip arthroplasty (MITHA) is a treatment for hip arthritis, and it causes less tissue trauma, blood loss, and recovery time. However, the limited incision makes it difficult for surgeons to perceive the instruments' location and orientation. Computer-assisted navigation systems can help improve the medical outcome of MITHA. Directly applying existing navigation systems for MITHA, however, suffers from problems of bulky fiducial marker, severe feature-loss, multiple instruments tracking confusion, and radiation exposure. To tackle these problems, we propose an image-guided navigation system for MITHA using a novel position-sensing marker.

METHODS

A position-sensing marker is proposed to serve as the fiducial marker with high-density and multi-fold ID tags. It results in less feature span and enables the use of ID for each feature, overcoming the problem of bulky fiducial markers and multiple instruments tracking confusion. And the marker can be recognized even when a large part of locating features is obscured. As for the elimination of intraoperative radiation exposure, we propose a point-based method to achieve patient-image registration based on anatomical landmarks.

RESULTS

Quantitative experiments are conducted to evaluate the feasibility of our system. The accuracy of instrument positioning is achieved at 0.33 ± 0.18 mm, and that of patient-image registration is achieved at 0.79 ± 0.15 mm. And qualitative experiments are also performed, verifying that our system can be used in compact surgical spatial volume and can address severe feature-loss and tracking confusion problems. In addition, our system does not require any intraoperative medical scans.

CONCLUSION

Experimental results indicate that our proposed system can assist surgeons without larger space occupations, radiation exposure, and extra incision, showing its potential application value in MITHA.

HydraMarker: Efficient, Flexible, and Multifold Marker Field Generation

An n-order marker field is a special binary matrix whose n×n subregions are all distinct from each other in four orientations. It is commonly used to guide the composing process of position-sensing markers, which can be detected and identified in a camera image with very limited scope or severe visibility problems. Despite the advantages, position-sensing markers are rare and overlooked because generating marker fields is difficult. In this article, we broaden the definition of marker field, making it more powerful and flexible. Then, we propose bWFC (binary wave function collapse) and its high-speed version, fast-bWFC, to solve the generation problem. The methods are packaged into an open-sourced toolkit named HydraMarker, with which, users not only can generate marker fields on laptops within a short period of time, but also can highly customize them: preset values; fields and subregions in any shape; multifold local uniqueness. Comparative results indicate that the proposed method has superior efficiency, quality, and capability. It makes marker field generation accessible to common marker designers, opening up more possibilities for fiducial markers.

Global optimization point-set registration based on translation/rotation decoupling for image-guided surgery applications

PURPOSE

In image-guided surgery systems, image-to-patient spatial registration is to get the spatial transformation between the image space and the actual operating space. Although the image-to-patient spatial registration methods using paired point or surface matching are used in some image-guided neurosurgery systems, the key problem is that the global optimization registration result cannot be achieved. Therefore, this paper proposes a new rotation invariant feature for decoupling rotation and translation space, based on which global optimization point set registration method is proposed.

METHODS

The new rotation invariant features, constructed based on the edges and the angles, are the rotation invariant, which has high feature resolution. Some of them are not only the rotation invariant, but also the translation invariant. To obtain the global optimal solution, branch and bound search strategy is used to search the parameter space of the translation and the computational cost is reduced simultaneously. The registration accuracy of the spatial registration method is analyzed by comparing the difference between the estimated transform and the standard transform to calculate the registration error.

RESULTS

To validate the performance of the spatial registration method proposed, the registration performance was analyzed by comparing the experimental results with the results of the two mainstream registration methods (the iterative closest point [ICP] registration method and the coherent point drift method). In the experiments, the comparison was based on the registration accuracy and the execution time. We show our registration method can obtain higher accuracy in a shorter time in most cases. At the same time, when using ICP to further refine our results, the ICP method can converge in a very short time, which also shows that our method provides a good initial pose for the ICP method and can help the ICP converge to the global optimal solution faster. Our method can achieve an average rotation error of 0.124 degrees and an average translation error of 0.38 mm on 10 clinical data.

CONCLUSIONS

The results reveal that the surface registration method based on translation rotation decoupling can achieve superior performance regarding both the registration accuracy and the time efficiency in the image-to-patient spatial registration.

A Hybrid 3D-2D Image Registration Framework for Pedicle Screw Trajectory Registration between Intraoperative X-ray Image and Preoperative CT Image

Pedicle screw insertion is considered a complex surgery among Orthopaedics surgeons. Exclusively to prevent postoperative complications associated with pedicle screw insertion, various types of image intensity registration-based navigation systems have been developed. These systems are computation-intensive, have a small capture range and have local maxima issues. On the other hand, deep learning-based techniques lack registration generalizability and have data dependency. To overcome these limitations, a patient-specific hybrid 3D-2D registration principled framework was designed to map a pedicle screw trajectory between intraoperative X-ray image and preoperative CT image. An anatomical landmark-based 3D-2D Iterative Control Point (ICP) registration was performed to register a pedicular marker pose between the X-ray images and axial preoperative CT images. The registration framework was clinically validated by generating projection images possessing an optimal match with intraoperative X-ray images at the corresponding control point registration. The effectiveness of the registered trajectory was evaluated in terms of displacement and directional errors after reprojecting its position on 2D radiographic planes. The mean Euclidean distances for the Head and Tail end of the reprojected trajectory from the actual trajectory in the AP and lateral planes were shown to be 0.6–0.8 mm and 0.5–1.6 mm, respectively. Similarly, the corresponding mean directional errors were found to be 4.90 and 20. The mean trajectory length difference between the actual and registered trajectory was shown to be 2.67 mm. The approximate time required in the intraoperative environment to axially map the marker position for a single vertebra was found to be 3 min. Utilizing the markerless registration techniques, the designed framework functions like a screw navigation tool, and assures the quality of surgery being performed by limiting the need of postoperative CT.

Oriented Localization of Surgical Tools by Location Encoding

Surgical tool localization is the foundation to a series of advanced surgical functions e.g. image guided surgical navigation. For precise scenarios like surgical tool localization, sophisticated tools and sensitive tissues can be quite close. This requires a higher localization accuracy than general object localization. And it is also meaningful to know the orientation of tools. To achieve these, this paper proposes a Compressive Sensing based Location Encoding scheme, which formulates the task of surgical tool localization in pixel space into a task of vector regression in encoding space. Furthermore with this scheme, the method is able to capture orientation of surgical tools rather than simply outputting horizontal bounding boxes. To prevent gradient vanishing, a novel back-propagation rule for sparse reconstruction is derived. The back-propagation rule is applicable to different implementations of sparse reconstruction and renders the entire network end-to-end trainable. Finally, the proposed approach gives more accurate bounding boxes as well as capturing the orientation of tools, and achieves state-of-the-art performance compared with 9 competitive both oriented and non-oriented localization methods (RRD, RefineDet, etc) on a mainstream surgical image dataset: m2cai16-tool-locations. A range of experiments support our claim that regression in CSLE space performs better than traditionally detecting bounding boxes in pixel space.

ROBUST SURFACE-MATCHING REGISTRATION BASED ON THE STRUCTURE INFORMATION FOR IMAGE-GUIDED NEUROSURGERY SYSTEM

Image-to-patient space registration is to make the accurate alignment between the actual operating space and the image space. Although the image-to-patient space registration using paired-point is used in some image-guided neurosurgery systems, the current paired-point registration method has some drawbacks and usually cannot achieve the best registration result. Therefore, surface-matching registration is proposed to solve this problem. This paper proposes a surface-matching method that accomplishes image-to-patient space registration automatically. We represent the surface point clouds by the Gaussian Mixture Model (GMM), which can smoothly approximate the probability density distribution of an arbitrary point set. We also use mutual information as the similarity measure between the point clouds and take into account the structure information of the points. To analyze the registration error, we introduce a method for the estimation of Target Registration Error (TRE) by generating simulated data. In the experiments, we used the point sets of the cranium surface and the model of the human head determined by a CT and laser scanner. The TRE was less than 2[Formula: see text]mm, and the TRE had better accuracy in the front and the posterior region. Compared to the Iterative Closest Point algorithm, the surface registration based on GMM and the structure information of the points proved superior in registration robustness and accurate implementation of image-to-patient registration.

Roadmap on 3D integral imaging: sensing, processing, and display

This Roadmap article on three-dimensional integral imaging provides an overview of some of the research activities in the field of integral imaging. The article discusses various aspects of the field including sensing of 3D scenes, processing of captured information, and 3D display and visualization of information. The paper consists of a series of 15 sections from the experts presenting various aspects of the field on sensing, processing, displays, augmented reality, microscopy, object recognition, and other applications. Each section represents the vision of its author to describe the progress, potential, vision, and challenging issues in this field.

3-D spatial floating display using multi-wavelength integral photography

Three-dimensional (3-D) autostereoscopic display with dedicated multiple spatial information under corresponding illumination is critical, especially for anti-counterfeiting, entertainment, etc. In this paper, we propose a 3-D spatial floating display using multi-wavelength integral photography (IP). Using dedicated inkjet printer and refraction-based IP algorithm, a complex two-dimensional (2-D) elemental image array (EIA) can be printed for both fluorescent and normal 3-D autostereoscopic display. With a micro-convex lens array (MLA) and a medium attached on the EIA, normal 3-D images are reconstructed under visible light, while fluorescent 3-D images can be reconstructed under ultraviolet (UV) light. Moreover, to provide comfortable 3-D images with multiple information in space, a feasible 3-D spatial floating display system is also proposed considering the spatial position of the observer with less UV radiation. The proposed method takes the wavelength of 3-D display into consideration to provide spatial multi-information, and can be applied for media, entertainment, etc. Experimental results verified the availability of the proposed method.