Mixed Reality Improves 3D Visualization and Spatial Awareness of Bone Tumors for Surgical Planning in Orthopaedic Oncology: A Proof of Concept Study

Mixed reality infrastructure based on deep learning medical image segmentation and 3D visualization for bone tumors using DCU-Net

Objective

Segmenting and reconstructing 3D models of bone tumors from 2D image data is of great significance for assisting disease diagnosis and treatment. However, due to the low distinguishability of tumors and surrounding tissues in images, existing methods lack accuracy and stability. This study proposes a U-Net model based on double dimensionality reduction and channel attention gating mechanism, namely the DCU-Net model for oncological image segmentation. After realizing automatic segmentation and 3D reconstruction of osteosarcoma by optimizing feature extraction and improving target space clustering capabilities, we built a mixed reality (MR) infrastructure and explored the application prospects of the infrastructure combining deep learning-based medical image segmentation and mixed reality in the diagnosis and treatment of bone tumors.

Methods

We conducted experiments using a hospital dataset for bone tumor segmentation, used the optimized DCU-Net and 3D reconstruction technology to generate bone tumor models, and used set similarity (DSC), recall (R), precision (P), and 3D vertex distance error (VDE) to evaluate segmentation performance and 3D reconstruction effects. Then, two surgeons conducted clinical examination experiments on patients using two different methods, viewing 2D images and virtual reality infrastructure, and used the Likert scale (LS) to compare the effectiveness of surgical plans of the two methods.

Results

The DSC, R and P values of the model introduced in this paper all exceed 90%, which has significant advantages compared with methods such as U-Net and Attention-Uet. Furthermore, LS showed that clinicians in the DCU-Net-based MR group had better spatial awareness of tumor preoperative planning.

Conclusion

The deep learning DCU-Net algorithm model can improve the performance of tumor CT image segmentation, and the reconstructed fine model can better reflect the actual situation of individual tumors; the MR system constructed based on this model enhances clinicians' understanding of tumor morphology and spatial relationships. The MR system based on deep learning and three-dimensional visualization technology has great potential in the diagnosis and treatment of bone tumors, and is expected to promote clinical practice and improve efficacy.

Artificial intelligence for optimizing otologic surgical video: effects of video inpainting and stabilization on microscopic view

BACKGROUND

Optimizing the educational experience of trainees in the operating room is important; however, ear anatomy and otologic surgery are challenging for trainees to grasp. Viewing otologic surgeries often involves limitations related to video quality, such as visual disturbances and instability.

OBJECTIVES

We aimed to (1) improve the quality of surgical videos (tympanomastoidectomy [TM]) by using artificial intelligence (AI) techniques and (2) evaluate the effectiveness of processed videos through a questionnaire-based assessment from trainees.

MATERIALS AND METHODS

We conducted prospective study using video inpainting and stabilization techniques processed by AI. In each study set, we enrolled 21 trainees and asked them to watch processed videos and complete a questionnaire.

RESULTS

Surgical videos with the video inpainting technique using the implicit neural representation (INR) model were found to be the most helpful for medical students (0.79 ± 0.58) in identifying bleeding focus. Videos with the stabilization technique via point feature matching were more helpful for low-grade residents (0.91 ± 0.12) and medical students (0.78 ± 0.35) in enhancing overall visibility and understanding surgical procedures.

CONCLUSIONS AND SIGNIFICANCE

Surgical videos using video inpainting and stabilization techniques with AI were beneficial for educating trainees, especially participants with less anatomical knowledge and surgical experience.

Case Report: Custom made 3D implants for glenoid tumor reconstruction should be designed as reverse total shoulder arthroplasty

Background and objectives

Isolated bone tumors of the glenoid are exceedingly rare occurrence and pose a substantial surgical challenge. 3D printing technology has been proved to be a reliable tool to reconstruct complex anatomical part of the skeleton. We initially used this technology to reconstruct the glenoid component of the shoulder in a hemiarthroplasty configuration. We subsequently changed to a reverse shoulder arthroplasty.

Methods

Two patients were reconstructed with a hemiarthroplasty and 2 with a reverse configuration. Patients files were reviewed for radiographic analysis, pain and function scores.

Results

Mean follow-up was 36.44 ± 16.27 months. All patients are alive and disease free. The two patients who benefitted from a hemiarthroplasty demonstrated a rapid deterioration of the proximal humeral articular surface. Given their pain and function scores, they subsequently required revision towards a total shoulder arthroplasty. Following this conversion, one patient presented a shoulder dislocation requiring surgical reintervention. We did not observe any loosening or infection in this short series.

Conclusions

Custom made glenoid reconstruction should be designed as a reverse shoulder arthroplasty given the mechanical constrains on the proximal humerus and the extent of the surgery invariably damaging the suprascapular neurovascular bundle.

The role of imaging in extremity sarcoma surgery

The surgical management of extremity bone and soft tissue sarcomas has evolved significantly over the last 50 years. The introduction and refinement of high-resolution cross-sectional imaging has allowed accurate assessment of anatomy and tumor extent, and in the current era more than 90% of patients can successfully undergo limb-salvage surgery. Advances in imaging have also revolutionized the clinician's ability to assess treatment response, detect metastatic disease, and perform intraoperative surgical navigation. This review summarizes the broad and essential role radiology plays in caring for sarcoma patients from diagnosis to post-treatment surveillance. Present evidence-based imaging paradigms are highlighted along with key future directions.

Exploration of the application of augmented reality technology for teaching spinal tumor's anatomy and surgical techniques

Background

Augmented reality (AR) technology is gradually being applied in surgical teaching as an innovative teaching method. Developing innovative teaching methods to replicate clinical theory and practical teaching scenarios, simulate preoperative planning and training for bone tumor surgery, and offer enhanced training opportunities for young physicians to acquire and apply clinical knowledge is a crucial concern that impacts the advancement of the discipline and the educational standards for young orthopedic physicians.

Objective

This study explores the application effect of augmented reality technology in anatomy teaching and surgical clinical teaching for spinal tumor.

Methods

The method utilizes virtual reality and augmented reality technology to present a spinal tumor model and the surgical process of percutaneous vertebroplasty. We conducted a random selection of 12 students forming into the augmented reality teaching group and 13 students forming into the traditional teaching group among the 8-year medical students from Peking Union Medical College and Tsinghua University, ensuring that the age and learning stage of the students in both groups were similar. Two groups of students were taught using traditional teaching methods and augmented reality technology-assisted teaching methods, respectively. A questionnaire survey was conducted after class to assess the quality of course instruction, student motivation in learning, their proficiency in anatomical structures, their comprehension of spinal tumor growth and metastasis, and their understanding and proficiency in percutaneous vertebroplasty.

Results

This study was the first to apply augmented reality technology in teaching, using spinal tumors and percutaneous vertebroplasty as examples, a head-mounted augmented reality device was used to create learning scenarios, presenting the complex three-dimensional spatial structure intuitively. The two groups of students differ significantly in their rating of teaching quality, enthusiasm for learning, knowledge of anatomical features, understanding of spinal trabecular structure, and understanding of steps in percutaneous vertebroplasty. The augmented reality technology-assisted teaching system demonstrates outstanding advantages.

Conclusion

Augmented reality technology has great potential and broad prospects in teaching bone tumors, which can help improve the visualization, interactivity, and three-dimensional spatial sense of medical teaching in spinal tumor. The application and development prospects of using augmented reality technology for anatomy instruction, surgical teaching, and simulation training are extensive.

The application of extended reality technology-assisted intraoperative navigation in orthopedic surgery

Extended reality (XR) technology refers to any situation where real-world objects are enhanced with computer technology, including virtual reality, augmented reality, and mixed reality. Augmented reality and mixed reality technologies have been widely applied in orthopedic clinical practice, including in teaching, preoperative planning, intraoperative navigation, and surgical outcome evaluation. The primary goal of this narrative review is to summarize the effectiveness and superiority of XR-technology-assisted intraoperative navigation in the fields of trauma, joint, spine, and bone tumor surgery, as well as to discuss the current shortcomings in intraoperative navigation applications. We reviewed titles of more than 200 studies obtained from PubMed with the following search terms: extended reality, mixed reality, augmented reality, virtual reality, intraoperative navigation, and orthopedic surgery; of those 200 studies, 69 related papers were selected for abstract review. Finally, the full text of 55 studies was analyzed and reviewed. They were classified into four groups-trauma, joint, spine, and bone tumor surgery-according to their content. Most of studies that we reviewed showed that XR-technology-assisted intraoperative navigation can effectively improve the accuracy of implant placement, such as that of screws and prostheses, reduce postoperative complications caused by inaccurate implantation, facilitate the achievement of tumor-free surgical margins, shorten the surgical duration, reduce radiation exposure for patients and surgeons, minimize further damage caused by the need for visual exposure during surgery, and provide richer and more efficient intraoperative communication, thereby facilitating academic exchange, medical assistance, and the implementation of remote healthcare.