Fully automatic brain tumor segmentation for 3D evaluation in augmented reality

Narrative review of patient-specific 3D visualization and reality technologies in skull base neurosurgery: enhancements in surgical training, planning, and navigation

Recent advances in medical imaging, computer vision, 3-dimensional (3D) modeling, and artificial intelligence (AI) integrated technologies paved the way for generating patient-specific, realistic 3D visualization of pathological anatomy in neurosurgical conditions. Immersive surgical simulations through augmented reality (AR), virtual reality (VR), mixed reality (MxR), extended reality (XR), and 3D printing applications further increased their utilization in current surgical practice and training. This narrative review investigates state-of-the-art studies, the limitations of these technologies, and future directions for them in the field of skull base surgery. We begin with a methodology summary to create accurate 3D models customized for each patient by combining several imaging modalities. Then, we explore how these models are employed in surgical planning simulations and real-time navigation systems in surgical procedures involving the anterior, middle, and posterior cranial skull bases, including endoscopic and open microsurgical operations. We also evaluate their influence on surgical decision-making, performance, and education. Accumulating evidence demonstrates that these technologies can enhance the visibility of the neuroanatomical structures situated at the cranial base and assist surgeons in preoperative planning and intraoperative navigation, thus showing great potential to improve surgical results and reduce complications. Maximum effectiveness can be achieved in approach selection, patient positioning, craniotomy placement, anti-target avoidance, and comprehension of spatial interrelationships of neurovascular structures. Finally, we present the obstacles and possible future paths for the broader implementation of these groundbreaking methods in neurosurgery, highlighting the importance of ongoing technological advancements and interdisciplinary collaboration to improve the accuracy and usefulness of 3D visualization and reality technologies in skull base surgeries.

Deep learning-aided 3D proxy-bridged region-growing framework for multi-organ segmentation

Accurate multi-organ segmentation in 3D CT images is imperative for enhancing computer-aided diagnosis and radiotherapy planning. However, current deep learning-based methods for 3D multi-organ segmentation face challenges such as the need for labor-intensive manual pixel-level annotations and high hardware resource demands, especially regarding GPU resources. To address these issues, we propose a 3D proxy-bridged region-growing framework specifically designed for the segmentation of the liver and spleen. Specifically, a key slice is selected from each 3D volume according to the corresponding intensity histogram. Subsequently, a deep learning model is employed to pinpoint the semantic central patch on this key slice, to calculate the growing seed. To counteract the impact of noise, segmentation of the liver and spleen is conducted on superpixel images created through proxy-bridging strategy. The segmentation process is then extended to adjacent slices by applying the same methodology iteratively, culminating in the comprehensive segmentation results. Experimental results demonstrate that the proposed framework accomplishes segmentation of the liver and spleen with an average Dice Similarity Coefficient of approximately 0.93 and a Jaccard Similarity Coefficient of around 0.88. These outcomes substantiate the framework's capability to achieve performance on par with that of deep learning methods, albeit requiring less guidance information and lower GPU resources.

Evolution of the Transforaminal Lumbar Interbody Fusion (TLIF): From Open to Percutaneous to Patient-Specific

The transforaminal lumbar interbody fusion (TLIF) has seen significant evolution since its early inception, reflecting advancements in surgical techniques, patient safety, and outcomes. Originally described as an improvement over the posterior lumbar interbody fusion (PLIF), the TLIF began as an open surgical procedure, that notably reduced the need for the extensive neural retractation that hindered the PLIF. In line with the broader practice of surgery, trending toward minimally invasive access, the TLIF was followed by the development of the minimally invasive TLIF (MIS-TLIF), a technique that further decreased tissue trauma and postoperative complications. Subsequent advancements, including Trans-Kambin's Triangle TLIF (percLIF) and transfacet LIF, have continued to refine surgical access, minimize surgical footprint, and reduce the risk of injury to the patient. The latest evolution, as we will describe it, the patient-specific TLIF, is a culmination of the aforementioned adaptations and incorporates advanced imaging and segmentation technologies into perioperative planning, allowing surgeons to tailor approaches based on individual patient anatomy and pathology. These developments signify a shift towards more precise methods in spine surgery. The ongoing evolution of the TLIF technique illustrates the dynamic nature of surgery and emphasizes the need for continued adaptation and refinement.

Not without Context-A Multiple Methods Study on Evaluation and Correction of Automated Brain Tumor Segmentations by Experts

RATIONALE AND OBJECTIVES

Brain tumor segmentations are integral to the clinical management of patients with glioblastoma, the deadliest primary brain tumor in adults. The manual delineation of tumors is time-consuming and highly provider-dependent. These two problems must be addressed by introducing automated, deep-learning-based segmentation tools. This study aimed to identify criteria experts use to evaluate the quality of automatically generated segmentations and their thought processes as they correct them.

MATERIALS AND METHODS

Multiple methods were used to develop a detailed understanding of the complex factors that shape experts' perception of segmentation quality and their thought processes in correcting proposed segmentations. Data from a questionnaire and semistructured interview with neuro-oncologists and neuroradiologists were collected between August and December 2021 and analyzed using a combined deductive and inductive approach.

RESULTS

Brain tumors are highly complex and ambiguous segmentation targets. Therefore, physicians rely heavily on the given context related to the patient and clinical context in evaluating the quality and need to correct brain tumor segmentation. Most importantly, the intended clinical application determines the segmentation quality criteria and editing decisions. Physicians' personal beliefs and preferences about the capabilities of AI algorithms and whether questionable areas should not be included are additional criteria influencing the perception of segmentation quality and appearance of an edited segmentation.

CONCLUSION

Our findings on experts' perceptions of segmentation quality will allow the design of improved frameworks for expert-centered evaluation of brain tumor segmentation models. In particular, the knowledge presented here can inspire the development of brain tumor-specific metrics for segmentation model training and evaluation.

NnU-Net versus mesh growing algorithm as a tool for the robust and timely segmentation of neurosurgical 3D images in contrast-enhanced T1 MRI scans

PURPOSE

This study evaluates the nnU-Net for segmenting brain, skin, tumors, and ventricles in contrast-enhanced T1 (T1CE) images, benchmarking it against an established mesh growing algorithm (MGA).

METHODS

We used 67 retrospectively collected annotated single-center T1CE brain scans for training models for brain, skin, tumor, and ventricle segmentation. An additional 32 scans from two centers were used test performance compared to that of the MGA. The performance was measured using the Dice-Sørensen coefficient (DSC), intersection over union (IoU), 95th percentile Hausdorff distance (HD95), and average symmetric surface distance (ASSD) metrics, with time to segment also compared.

RESULTS

The nnU-Net models significantly outperformed the MGA (p < 0.0125) with a median brain segmentation DSC of 0.971 [95CI: 0.945-0.979], skin: 0.997 [95CI: 0.984-0.999], tumor: 0.926 [95CI: 0.508-0.968], and ventricles: 0.910 [95CI: 0.812-0.968]. Compared to the MGA's median DSC for brain: 0.936 [95CI: 0.890, 0.958], skin: 0.991 [95CI: 0.964, 0.996], tumor: 0.723 [95CI: 0.000-0.926], and ventricles: 0.856 [95CI: 0.216-0.916]. NnU-Net performance between centers did not significantly differ except for the skin segmentations Additionally, the nnU-Net models were faster (mean: 1139 s [95CI: 685.0-1616]) than the MGA (mean: 2851 s [95CI: 1482-6246]).

CONCLUSIONS

The nnU-Net is a fast, reliable tool for creating automatic deep learning-based segmentation pipelines, reducing the need for extensive manual tuning and iteration. The models are able to achieve this performance despite a modestly sized training set. The ability to create high-quality segmentations in a short timespan can prove invaluable in neurosurgical settings.

The Feasibility and Accuracy of Holographic Navigation with Laser Crosshair Simulator Registration on a Mixed-Reality Display

Addressing conventional neurosurgical navigation systems' high costs and complexity, this study explores the feasibility and accuracy of a simplified, cost-effective mixed reality navigation (MRN) system based on a laser crosshair simulator (LCS). A new automatic registration method was developed, featuring coplanar laser emitters and a recognizable target pattern. The workflow was integrated into Microsoft's HoloLens-2 for practical application. The study assessed the system's precision by utilizing life-sized 3D-printed head phantoms based on computed tomography (CT) or magnetic resonance imaging (MRI) data from 19 patients (female/male: 7/12, average age: 54.4 ± 18.5 years) with intracranial lesions. Six to seven CT/MRI-visible scalp markers were used as reference points per case. The LCS-MRN's accuracy was evaluated through landmark-based and lesion-based analyses, using metrics such as target registration error (TRE) and Dice similarity coefficient (DSC). The system demonstrated immersive capabilities for observing intracranial structures across all cases. Analysis of 124 landmarks showed a TRE of 3.0 ± 0.5 mm, consistent across various surgical positions. The DSC of 0.83 ± 0.12 correlated significantly with lesion volume (Spearman rho = 0.813, p < 0.001). Therefore, the LCS-MRN system is a viable tool for neurosurgical planning, highlighting its low user dependency, cost-efficiency, and accuracy, with prospects for future clinical application enhancements.

Mixed Reality for Cranial Neurosurgical Planning: A Single-Center Applicability Study With the First 107 Subsequent Holograms

BACKGROUND AND OBJECTIVES

Mixed reality (MxR) benefits neurosurgery by improving anatomic visualization, surgical planning and training. We aim to validate the usability of a dedicated certified system for this purpose.

METHODS

All cases prepared with MxR in our center in 2022 were prospectively collected. Holographic rendering was achieved using an incorporated fully automatic algorithm in the MxR application, combined with contrast-based semiautomatic rendering and/or manual segmentation where necessary. Hologram segmentation times were documented. Visualization during surgical preparation (defined as the interval between finalized anesthesiological induction and sterile draping) was performed using MxR glasses and direct streaming to a side screen. Surgical preparation times were compared with a matched historical cohort of 2021. Modifications of the surgical approach after 3-dimensional (3D) visualization were noted. Usability was assessed by evaluating 7 neurosurgeons with more than 3 months of experience with the system using a Usefulness, Satisfaction and Ease of use (USE) questionnaire.

RESULTS

One hundred-seven neurosurgical cases prepared with a 3D hologram were collected. Surgical indications were oncologic (63/107, 59%), cerebrovascular (27/107, 25%), and carotid endarterectomy (17/107, 16%). Mean hologram segmentation time was 39.4 ± 20.4 minutes. Average surgical preparation time was 48.0 ± 17.3 minutes for MxR cases vs 52 ± 17 minutes in the matched 2021 cohort without MxR (mean difference 4, 95% CI 1.7527-9.7527). Based on the 3D hologram, the surgical approach was modified in 3 cases. Good usability was found by 57% of the users.

CONCLUSION

The perioperative use of 3D holograms improved direct anatomic visualization while not significantly increasing intraoperative surgical preparation time. Usability of the system was adequate. Further technological development is necessary to improve the automatic algorithms and reduce the preparation time by circumventing manual and semiautomatic segmentation. Future studies should focus on quantifying the potential benefits in teaching, training, and the impact on surgical and functional outcomes.

Effect of Intraoperative Mixed-Reality use on Nonsurgical Team Members in the Neurosurgical Operating Room: An Explorative Study

OBJECTIVES

Mixed-reality (MxR) implementation in the neurosurgical operating room (OR) is emerging, but the impact of this technology on the nonsurgical OR staff has not been investigated yet. The purpose of this study is to evaluate the nonsurgical OR team's perception of the impact of intraoperative MxR use.

METHODS

The evaluation occurred in a neurosurgical institution implementing MxR perioperatively on a daily basis for 15 months. The questionnaire measuring the impact of MxR consisted of 5 binary questions and 4 subquestions measured on a 7-point Likert scale.

RESULTS

Fifteen nonsurgical staff members of a neurosurgical OR team were interviewed. Most (85%) of the cohort stated that MxR changed their perception of the surgery, improving understanding of the spatial orientation (median 6, interquartile range 5-7) and of the pathology dimensions (6, 5.5-7). One participant (7%) was of the opinion that MxR disrupted the OR workflow. The majority (60%) stated that use of the holograms improved efficiency in the OR. The participants were neutral about the potential role of MxR to improve communication among different OR team members (4, 2-5) and overall teamwork (4, 2-5).

CONCLUSIONS

The use of 3-Dimensional interactive holograms of neurosurgical cases in the nonsterile intraoperative phase was not perceived as distractive, and neither was OR flow disruption by members of the nonsurgical OR staff. MxR was considered an adjunct to improve OR efficiency. A thorough understanding of the impact of MxR's implementation on the nonsurgical staff could lead to targeted improvement of the MxR use and, potentially, to increasing the quality of the teamwork.

MRI-based automatic identification and segmentation of extrahepatic cholangiocarcinoma using deep learning network

BACKGROUND

Accurate identification of extrahepatic cholangiocarcinoma (ECC) from an image is challenging because of the small size and complex background structure. Therefore, considering the limitation of manual delineation, it's necessary to develop automated identification and segmentation methods for ECC. The aim of this study was to develop a deep learning approach for automatic identification and segmentation of ECC using MRI.

METHODS

We recruited 137 ECC patients from our hospital as the main dataset (C1) and an additional 40 patients from other hospitals as the external validation set (C2). All patients underwent axial T1-weighted imaging (T1WI), T2-weighted imaging (T2WI), and diffusion-weighted imaging (DWI). Manual delineations were performed and served as the ground truth. Next, we used 3D VB-Net to establish single-mode automatic identification and segmentation models based on T1WI (model 1), T2WI (model 2), and DWI (model 3) in the training cohort (80% of C1), and compared them with the combined model (model 4). Subsequently, the generalization capability of the best models was evaluated using the testing set (20% of C1) and the external validation set (C2). Finally, the performance of the developed models was further evaluated.

RESULTS

Model 3 showed the best identification performance in the training, testing, and external validation cohorts with success rates of 0.980, 0.786, and 0.725, respectively. Furthermore, model 3 yielded an average Dice similarity coefficient (DSC) of 0.922, 0.495, and 0.466 to segment ECC automatically in the training, testing, and external validation cohorts, respectively.

CONCLUSION

The DWI-based model performed better in automatically identifying and segmenting ECC compared to T1WI and T2WI, which may guide clinical decisions and help determine prognosis.

Augmented reality for intracranial meningioma resection: a mini-review

Augmented reality (AR) integrates computer-generated content and real-world scenarios. Artificial intelligence's continuous development has allowed AR to be integrated into medicine. Neurosurgery has progressively introduced image-guided technologies. Integration of AR into the operating room has permitted a new perception of neurosurgical diseases, not only for neurosurgical planning, patient positioning, and incision design but also for intraoperative maneuvering and identification of critical neurovascular structures and tumor boundaries. Implementing AR, virtual reality, and mixed reality has introduced neurosurgeons into a new era of artificial interfaces. Meningiomas are the most frequent primary benign tumors commonly related to paramount neurovascular structures and bone landmarks. Integration of preoperative 3D reconstructions used for surgical planning into AR can now be inserted into the microsurgical field, injecting information into head-up displays and microscopes with integrated head-up displays, aiming to guide neurosurgeons intraoperatively to prevent potential injuries. This manuscript aims to provide a mini-review of the usage of AR for intracranial meningioma resection.

Simulation of the occipital transtentorial approach incorporating visualization of the cerebellar tentorium using three-dimensional computed tomography angiography and gadolinium-enhanced T1-weighted magnetic resonance imaging: technical note

The occipital transtentorial approach (OTA) is one of the useful approaches to the lesions of the pineal region, dorsal brainstem, and supracerebellar region. However, a wide operative field is sometimes difficult to obtain due to the tentorial sinus and bridging veins. This study evaluated the usefulness of preoperative simulation of OTA, specifically including the cerebellar tentorium in 9 patients. All patients underwent computed tomography angiography and venography and gadolinium-enhanced three-dimensional T1-weighted magnetic resonance images (Gd-3D-T1WI). The images were fused, and the cerebellar tentorium, vessels, and tumor were manually extracted from Gd-3D-T1WI to obtain the simulation images. Visualization of the cerebellar tentorium could discriminate between bridging veins from the occipital lobe and cerebellum, and recognize the site of bridging to the tentorial sinus and variants which may interfere with the tentorial incision. Simulation of the tentorial incision was also possible based on the relationships between the tumor, tentorial sinus, bridging vein, and cerebellar tentorium. The simulation suggested that safe tentorial incision was difficult in two sides because of the crossed tentorial sinus draining the left basal vein and draining veins from the glioblastoma. The OTA was performed in eight cases, and no difficulty was experienced in the tentorial incision in all cases. The simulation findings of the bridging vein and tentorial sinus were consistent with the intraoperative findings. Preoperative simulation including the cerebellar tentorium is useful for determining the optimum and safe side and required extent of the tentorial incision necessary for tumor resection with the OTA.

Case report: Impact of mixed reality on anatomical understanding and surgical planning in a complex fourth ventricular tumor extending to the lamina quadrigemina

Background and importance

Tumors of the fourth ventricle account for 1%-5% of all intracranial neoplastic lesions and present with different configurations and anatomical challenges. Microsurgery represents the primary therapeutic strategy for the majority of fourth ventricular tumors, and adequate anatomical understanding and visualization are paramount to surgical planning and success. The authors present the case of a young patient with a complex fourth ventricular tumor, whose surgery was successfully planned using a novel mixed reality (MxR) system.

Case description

We present a case of a 31-year-old woman with a lesion extending from the fourth ventricle to the lamina quadrigemina and causing symptomatic hydrocephalus occlusus. Through the combined use of routine 2D images and an interactive 3D anatomical model, an interhemispheric transtentorial approach was used to remove 98% of the lesion with successful functional outcomes.

Conclusions

The application of advanced 3D visualization with a novel MxR system to the surgical planning of a complex fourth ventricular lesion proved relevant in designing the best surgical approach and trajectory to better identify potential intraoperative challenges and rehearse the patient-specific anatomy. The present case report endorses the implementation of advanced 3D visualization in routine perioperative practice.

Feasibility study of adaptive radiotherapy for esophageal cancer using artificial intelligence autosegmentation based on MR-Linac

Objective

We proposed a scheme for automatic patient-specific segmentation in Magnetic Resonance (MR)-guided online adaptive radiotherapy based on daily updated, small-sample deep learning models to address the time-consuming delineation of the region of interest (ROI) in the adapt-to-shape (ATS) workflow. Additionally, we verified its feasibility in adaptive radiation therapy for esophageal cancer (EC).

Methods

Nine patients with EC who were treated with an MR-Linac were prospectively enrolled. The actual adapt-to-position (ATP) workflow and simulated ATS workflow were performed, the latter of which was embedded with a deep learning autosegmentation (AS) model. The first three treatment fractions of the manual delineations were used as input data to predict the next fraction segmentation, which was modified and then used as training data to update the model daily, forming a cyclic training process. Then, the system was validated in terms of delineation accuracy, time, and dosimetric benefit. Additionally, the air cavity in the esophagus and sternum were added to the ATS workflow (producing ATS+), and the dosimetric variations were assessed.

Results

The mean AS time was 1.40 [1.10-1.78 min]. The Dice similarity coefficient (DSC) of the AS model gradually approached 1; after four training sessions, the DSCs of all ROIs reached a mean value of 0.9 or more. Furthermore, the planning target volume (PTV) of the ATS plan showed a smaller heterogeneity index than that of the ATP plan. Additionally, V5 and V10 in the lungs and heart were greater in the ATS+ group than in the ATS group.

Conclusion

The accuracy and speed of artificial intelligence-based AS in the ATS workflow met the clinical radiation therapy needs of EC. This allowed the ATS workflow to achieve a similar speed to the ATP workflow while maintaining its dosimetric advantage. Fast and precise online ATS treatment ensured an adequate dose to the PTV while reducing the dose to the heart and lungs.

A Scoping Review of Deep Learning in Cancer Nursing Combined with Augmented Reality: the Era of Intelligent Nursing is Coming

Artificial intelligence has been developing greatly in the field of medicine. As a new research hotspot of artificial intelligence, deep learning (DL) has been widely applied in the fields of cancer risk assessment, symptom recognition, and cancer detection. Therefore, nursing care issues in terms of consuming time and energy, lower accuracy, and lower efficiency can be solved with applying DL in caring cancer patients. In addition, augmented reality (AR) has great navigation potential through combining computer-generated virtual elements with the real world. Thus, DL ​+ ​AR may facilitate patients with cancer to possess a brand-new model of nursing care that is more intelligent, mobile, and adapted to the information age, compared to traditional nursing. With the advent of the era of intelligent nursing, future nursing models can not only learn from the DL ​+ ​AR model to meet the needs of patients with cancer but also reduce nursing workload, save healthcare resources, and improve work efficiency, the quality of nursing care, as well as the quality of life for cancer patients.

Usage of image registration and three-dimensional visualization tools on serial computed tomography for the analysis of patients with traumatic intraparenchymal hemorrhages

The aim of this study was to apply registration and three-dimensional (3D) display tools to assess the evolution of intraparenchymal hemorrhage (IPH) in patients with traumatic brain injury (TBI). We identified 109 TBI patients who had two computed tomography (CT) scans within 4 days retrospectively. The IPH was manually outlined. The registration was performed in 39 lesions from 29 patients with lesion volume < 1.5 cm on both baseline and follow-up CT. The center of mass (COM) of each lesion was calculated, and the distance between baseline and follow-up CT was used to evaluate the registration effect. The mean distances of COM before registration in the XYZ, XY, and YZ coordinates were 20.5 ± 10.2 mm, 17.8 ± 9.4 mm, and 15.9 ± 9.4 mm, respectively, which decreased significantly (p < 0.001) to 7.9 ± 4.9, 7.8 ± 5.0, and 6.1 ± 4.1 mm after registration. A 3D short video displaying the rendering view of all lesions in 34 randomly selected patients from baseline and follow-up scans were presented side-by-side for comparison. The detection rate of new IPH lesions increased in 3D videos (100%) as compared with axial CT slices (78.6-92.9%). A very high interrater agreement (k = 0.856) on perceiving IPH lesion progression upon viewing 3D video was noted, and the absolute volume increase was significantly higher (p < 0.001) for progressive lesions (median 7.36 cc) over non-progressive lesions (median 0.01 cc). Compared to patients with spontaneous hemorrhagic stroke, evaluation of multiple small traumatic hemorrhages in TBI is more challenging. The applied image analysis and visualization methods may provide helpful tools for comparing changes between serial CT scans.