Development of a surgical navigation system based on 3D Slicer for intraoperative implant placement surgery

Open-source navigation system for tracking dissociated parts with multi-registration

PURPOSE

During reconstructive surgery, knee and hip replacements, and orthognathic surgery, small misalignments in the pose of prosthesis and bones can lead to severe complications. Hence, the translational and angular accuracies are critical. However, traditional image-based surgical navigation lacks orientation data between structures, and imageless systems are unsuitable for cases of deformed anatomy. We introduce an open-source navigation system using a multiple registration approach that can track instruments, implants, and bones to precisely guide the surgeon in emulating a preoperative plan.

METHODS

We derived the analytical error of our method and designed a set of phantom experiments to measure its precision and accuracy. Additionally, we trained two classification models to predict the system reliability from fiducial points and surface matching registration data. Finally, to demonstrate the procedure feasibility, we conducted a complete workflow for a real clinical case of a patient with fibrous dysplasia and anatomical misalignment of the right femur using plastic bones.

RESULTS

The system is able to track the dissociated fragments of the clinical case and average alignment errors in the anatomical phantoms of [Formula: see text]  mm and [Formula: see text]. While the fiducial-points registration showed satisfactory results given enough points and covered volume, we acknowledge that the surface refinement step is mandatory when attempting surface matching registrations.

CONCLUSION

We believe that our device could bring significant advantages for the personalized treatment of complex surgical cases and that its multi-registration attribute is convenient for intraoperative registration loosening cases.

A review of advances in image-guided orthopedic surgery

Orthopedic surgery remains technically demanding due to the complex anatomical structures and cumbersome surgical procedures. The introduction of image-guided orthopedic surgery (IGOS) has significantly decreased the surgical risk and improved the operation results. This review focuses on the application of recent advances in artificial intelligence (AI), deep learning (DL), augmented reality (AR) and robotics in image-guided spine surgery, joint arthroplasty, fracture reduction and bone tumor resection. For the pre-operative stage, key technologies of AI and DL based medical image segmentation, 3D visualization and surgical planning procedures are systematically reviewed. For the intra-operative stage, the development of novel image registration, surgical tool calibration and real-time navigation are reviewed. Furthermore, the combination of the surgical navigation system with AR and robotic technology is also discussed. Finally, the current issues and prospects of the IGOS system are discussed, with the goal of establishing a reference and providing guidance for surgeons, engineers, and researchers involved in the research and development of this area.

An automatic measurement system of distal femur morphological parameters using 3D slicer software

In the field of joint surgery, the computer-aided design of knee prostheses suitable for the Chinese population requires a large quantity of anatomical knee data. In this study, we propose a new method that uses 3D Slicer software to automatically measure the morphological parameters of the distal femur. First, 141 femur samples were segmented from CT data to establish the femoral shape library. Next, balanced iterative reducing and clustering using hierarchies (BIRCH) combined with iterative closest point (ICP) and generalised procrustes analysis (GPA) were used to achieve fast registration of the femur samples. The statistical model was automatically calculated from the registered femur samples, and an orthopaedic surgeon marked the points on the statistical model. Finally, we developed an automatic measurement system using 3D Slicer software, and a deformable model matching method was applied to establish the point correspondence between the statistical model and the other samples. By matching points on the statistical model to corresponding points in other samples, we measured all other samples. We marked six points and measured eight parameters. We evaluated the performance of automatic matching by comparing the points marked manually with those matched automatically and verified the accuracy of the system by comparing the manual and automatic measurement results. The results indicated that the average error of the automatic matching points was 1.03 mm, and the average length error and average angle error measured automatically by the system were 0.37 mm and 0.63°, respectively. These errors were smaller than the intra-rater and inter-rater errors measured manually by two different surgeons, which showed that the accuracy of our automatic method was high. Taken together, this study established an accurate and automatic measurement system for the distal femur based on the secondary development of 3D Slicer software to assist orthopaedic surgeons in completing the measurements of big data and further promote the improved design of Chinese-specific knee prostheses.

3D-Slicer Software-Assisted Neuroendoscopic Surgery in the Treatment of Hypertensive Cerebral Hemorrhage

Objective. To explore the 3D-slicer software-assisted endoscopic treatment for patients with hypertensive cerebral hemorrhage. Methods. A total of 120 patients with hypertensive cerebral hemorrhage were selected and randomly divided into control group and 3D-slicer group with 60 cases each. Patients in the control group underwent traditional imaging positioning craniotomy, and patients in the 3D-slicer group underwent 3D-slicer followed by precision puncture treatment. In this paper, we evaluate the hematoma clearance rate, nerve function, ability of daily living, complication rate, and prognosis. Results. The 3D-slicer group is better than the control group in various indicators. Compared with the control group, the 3D-slicer group has lower complications, slightly higher hematoma clearance rate, and better recovery of nerve function and daily living ability before and after surgery. The incidence of poor prognosis is low. Conclusion. The 3D-slicer software-assisted endoscopic treatment for patients with hypertensive intracerebral hemorrhage has a better hematoma clearance effect, which is beneficial to the patient’s early recovery and reduces the damage to the brain nerve of the patient.

Development and Application of Digital Maxillofacial Surgery System Based on Mixed Reality Technology

Objective

To realize the three-dimensional visual output of surgical navigation information by studying the cross-linking of mixed reality display devices and high-precision optical navigators.

Methods

Applying quaternion-based point alignment algorithms to realize the positioning configuration of mixed reality display devices, high-precision optical navigators, real-time patient tracking and calibration technology; based on open source SDK and development tools, developing mixed reality surgery based on visual positioning and tracking system. In this study, four patients were selected for mixed reality-assisted tumor resection and reconstruction and re-examined 1 month after the operation. We reconstructed postoperative CT and use 3DMeshMetric to form the error distribution map, and completed the error analysis and quality control.

Results

Realized the cross-linking of mixed reality display equipment and high-precision optical navigator, developed a digital maxillofacial surgery system based on mixed reality technology and successfully implemented mixed reality-assisted tumor resection and reconstruction in 4 cases.

Conclusions

The maxillofacial digital surgery system based on mixed reality technology can superimpose and display three-dimensional navigation information in the surgeon's field of vision. Moreover, it solves the problem of visual conversion and space conversion of the existing navigation system. It improves the work efficiency of digitally assisted surgery, effectively reduces the surgeon's dependence on spatial experience and imagination, and protects important anatomical structures during surgery. It is a significant clinical application value and potential.

MedQ: Lossless ultra-low-bit neural network quantization for medical image segmentation

Implementing deep convolutional neural networks (CNNs) with boolean arithmetic is ideal for eliminating the notoriously high computational expense of deep learning models. However, although lossless model compression via weight-only quantization has been achieved in previous works, it is still an open problem about how to reduce the computation precision of CNNs without losing performance, especially for medical image segmentation tasks where data dimension is high and annotation is scarce. This paper presents a novel CNN quantization framework that can squeeze a deep model (both parameters and activation) to extremely low bitwidth, e.g., 1∼2 bits, while maintaining its high performance. In the new method, we first design a strong baseline quantizer with an optimizable quantization range. Then, to relieve the back-propagation difficulty caused by the discontinuous quantization function, we design a radical residual connection scheme that allows gradients to flow through every quantized layer freely. Moreover, a tanh-based derivative function is used to further boost gradient flow and a distributional loss is employed to regularize the model output. Extensive experiments and ablation studies are conducted on two well-established public 3D segmentation datasets, i.e., BRATS2020 and LiTS. Experimental results evidence that our framework not only outperforms state-of-the-art quantization approaches significantly, but also achieves lossless performance on both datasets with ternary (2-bit) quantization.

A hybrid feature-based patient-to-image registration method for robot-assisted long bone osteotomy

PURPOSE

The purpose of this study is to provide a simple, feasible and effective patient-to-image registration method for robot-assisted long bone osteotomy, which has rarely been systematically reported. The practical requirement is to meet the accuracy of 1 mm or even higher without bone-implanted markers.

METHODS

A hybrid feature-based registration method termed CR-RAMSICP is proposed. Point-based coarse registration (CR) is accomplished relying on the optical retro-reflective markers attached to the tracked rigid body fixed out of the bone. In surface-based fine registration, an improved iterative closest point (ICP) algorithm based on the range-adaptive matching strategy (termed RAMSICP) is presented to cope with the robust precise matching between the asymmetric patient and image point clouds, which avoids converging to a local minimum.

RESULTS

A series of registration experiments based on the isolated porcine iliums are carried out. The results illustrate that CR-RAMSICP not only significantly outperforms CR and CR-ICP in the accuracy and reproducibility, but also exhibits better robustness to the CR errors and less sensitiveness to the distribution and number of fiducial points located in the patient point cloud than CR-ICP.

CONCLUSION

The proposed registration method CR-RAMSICP can stably satisfy the desired registration accuracy without the use of bone-implanted markers like fiducial screws. Besides, the RAMSICP algorithm used in fine registration is convenient for programming because any complex metrics or models are not involved.

An electromagnetic tracking implantation navigation system in dentistry with virtual calibration

BACKGROUND

Dental implant placement navigation systems based on optical tracking have been widely used in clinics. However, electromagnetic (EM) navigation method that does not suffer from problems of hidden line-of-light has not yet been described.

METHODS

This work proposes an EM-guided navigation method named TianShu-ESNS with virtual calibration. Model (12 implants) and animal experiments (pig head: six implants) were conducted to evaluate its performance and stability.

RESULT

The mean virtual calibration error was 0.83 ± 0.20 mm. The mean deviations at the entry point, end point and angle in the phantom experiment of TianShu-ESNS were 1.23 ± 0.17 mm, 1.59 ± 0.20 mm and 1.83 ± 0.27°, respectively. In the animal experiment, the same deviations were 1.25 ± 0.07 mm, 1.57 ± 0.35 mm and 1.90 ± 0.60°, respectively.

CONCLUSIONS

The experimental results show that TianShu-ESNS with the virtual calibration method could serve as a promising tool to eliminate the line-of-light hidden problem and simplify operation procedure in dental implant placement.

The development of non-contact user interface of a surgical navigation system based on multi-LSTM and a phantom experiment for zygomatic implant placement

PURPOSE

Image-guided surgical navigation system (SNS) has proved to be an increasingly important assistance tool for mini-invasive surgery. However, using standard devices such as keyboard and mouse as human-computer interaction (HCI) is a latent vector of infectious medium, causing risks to patients and surgeons. To solve the human-computer interaction problem, we proposed an optimized structure of LSTM based on a depth camera to recognize gestures and applied it to an in-house oral and maxillofacial surgical navigation system (Qin et al. in Int J Comput Assist Radiol Surg 14(2):281-289, 2019).

METHODS

The proposed optimized structure of LSTM named multi-LSTM allows multiple input layers and takes into account the relationships between inputs. To combine the gesture recognition with the SNS, four left-hand signs waving along four directions were designed to correspond to four operations of the mouse, and the motion of right hand was used to control the movement of the cursor. Finally, a phantom study for zygomatic implant placement was conducted to evaluate the feasibility of multi-LSTM as HCI.
 RESULTS: 3D hand trajectories of both wrist and elbow from 10 participants were collected to train the recognition network. Then tenfold cross-validation was performed for judging signs, and the mean accuracy was 96% ± 3%. In the phantom study, four implants were successfully placed, and the average deviations of planned-placed implants were 1.22 mm and 1.70 mm for the entry and end points, respectively, while the angular deviation ranged from 0.4° to 2.9°.

CONCLUSION

The results showed that this non-contact user interface based on multi-LSTM could be used as a promising tool to eliminate the disinfection problem in operation room and alleviate manipulation complexity of surgical navigation system.

Towards More Structure: Comparing TNM Staging Completeness and Processing Time of Text-Based Reports versus Fully Segmented and Annotated PET/CT Data of Non-Small-Cell Lung Cancer

Results of PET/CT examinations are communicated as text-based reports which are frequently not fully structured. Incomplete or missing staging information can be a significant source of staging and treatment errors. We compared standard text-based reports to a manual full 3D-segmentation-based approach with respect to TNM completeness and processing time. TNM information was extracted retrospectively from 395 reports. Moreover, the RIS time stamps of these reports were analyzed. 2995 lesions using a set of 41 classification labels (TNM features + location) were manually segmented on the corresponding image data. Information content and processing time of reports and segmentations were compared using descriptive statistics and modelling. The TNM/UICC stage was mentioned explicitly in only 6% (n=22) of the text-based reports. In 22% (n=86), information was incomplete, most frequently affecting T stage (19%, n=74), followed by N stage (6%, n=22) and M stage (2%, n=9). Full NSCLC-lesion segmentation required a median time of 13.3 min, while the median of the shortest estimator of the text-based reporting time (R1) was 18.1 min (p=0.01). Tumor stage (UICC I/II: 5.2 min, UICC III/IV: 20.3 min, p < 0.001), lesion size (p < 0.001), and lesion count (n=1: 4.4 min, n=12: 37.2 min, p < 0.001) correlated significantly with the segmentation time, but not with the estimators of text-based reporting time. Numerous text-based reports are lacking staging information. A segmentation-based reporting approach tailored to the staging task improves report quality with manageable processing time and helps to avoid erroneous therapy decisions based on incomplete reports. Furthermore, segmented data may be used for multimedia enhancement and automatization.

Novel Thoracoscopic Navigation System With Augmented Real-Time Image Guidance for Chest Wall Tumors

BACKGROUND

We developed a thoracoscopic surgical navigation system with real-time augmented image guidance to assess the potential benefits for minimally invasive resection of chest wall tumors. The accuracy of localization of tumor and resection margin and the effect on task workload and confidence were evaluated in a chest wall tumor phantom.

METHODS

After scanning a realistic tumor phantom by cone-beam computed tomography and registering the data into the system, three-dimensional contoured tumor and resection margin was displayed. Fifteen surgeons were asked to localize the tumor margin and surgical margins with the thoracoscope alone. The same procedure was performed with the surgical navigation system activated, and results were compared between each attempt. A questionnaire and National Aeronautics and Space Administration Task Load Index were completed after.

RESULTS

The surgical navigation system significantly reduced localization error for the medial (p = 0.002) and superior tumor margin (p < 0.001), which was difficult to visualize by thoracoscopy alone. All surgical resection margins were improved circumferentially, including margins that were readily visible by thoracoscopy. National Aeronautics and Space Administration Task Load Index response scores showed a statistically significant reduction in workload in all subscales. There was a more than 50% mean reduction in workload for performance (10.1 vs 4.4, p = 0.001) and frustration (13.0 vs 5.4, p = 0.001).

CONCLUSIONS

This study showed that the thoracoscopic surgical navigation system providing augmented image guidance decreased tumor localization error for regions difficult to visualize thoracoscopically and also reduced surgical margin error circumferentially, regardless of thoracoscopic visibility. This system also reduced workload and increased surgeon's confidence in localizing challenging chest wall tumors.

Continuous dynamic gesture spotting algorithm based on Dempster-Shafer Theory in the augmented reality human computer interaction

BACKGROUND

Human-computer interaction (HCI) is an important feature of augmented reality (AR) technology. The naturalness is the inevitable trend of HCI. Gesture is the most natural and frequently used body auxiliary interaction mode in daily interactions except for language. However, there are often meaningless, subconscious gesture intervals between the two adjacent dynamic gestures. So, continuous dynamic gesture spotting is the premise and basis of dynamic gesture recognition, but there is no mature and unified algorithm to solve this problem.

AIMS

In order to realize the natural HCI based on gesture recognition entirely, a general AR application development platform is presented in this paper.

METHODS

According to the position and pose tracking data of the user's hand, the dynamic gesture spotting algorithm based on evidence theory is proposed. Firstly, Through analysis of the speed change of hand motion during the dynamic gestures, three knowledge rules are summed up. Then, accurate dynamic gesture spotting is realized with the application of evidence reasoning. Moreover, this algorithm first detects the starting point of gesture in the rising trend of hand motion speed, eliminates the delay between spotting and recognition, and thus ensures real-time performance. Finally, the algorithm is verified in several AR applications developed on the platform.

RESULTS

There are two main experimental results. First, there are six users participating in the dynamic gesture spotting experiment, and the gesture spotting accuracy can meet the demand. Second, The accuracy of recognition after spotting is higher than that of the simultaneous recognition and spotting.

CONCLUSION

So, It can be concluded that the proposed continuous dynamic gesture spotting algorithm based on Dempster-Shafer theory can extract almost all the effective dynamic gestures in the HCI of our AR platform, and on this basis, it can effectively improve the accuracy of the subsequent dynamic gesture recognition.