respiTrack: Patient-specific real-time respiratory tumor motion prediction using magnetic tracking

Toward Intraoperative Visual Intelligence: Real-Time Surgical Instrument Segmentation for Enhanced Surgical Monitoring

BACKGROUND

Open surgery relies heavily on the surgeon's visual acuity and spatial awareness to track instruments within a dynamic and often cluttered surgical field.

METHODS

This system utilizes a head-mounted depth camera to monitor surgical scenes, providing both image data and depth information. The video captured from this camera is scaled down, compressed using MPEG, and transmitted to a high-performance workstation via the RTSP (Real-Time Streaming Protocol), a reliable protocol designed for real-time media transmission. To segment surgical instruments, we utilize the enhanced U-Net with GridMask (EUGNet) for its proven effectiveness in surgical tool segmentation.

RESULTS

For rigorous validation, the system's performance reliability and accuracy are evaluated using prerecorded RGB-D surgical videos. This work demonstrates the potential of this system to improve situational awareness, surgical efficiency, and generate data-driven insights within the operating room. In a simulated surgical environment, the system achieves a high accuracy of 85.5% in identifying and segmenting surgical instruments. Furthermore, the wireless video transmission proves reliable with a latency of 200 ms, suitable for real-time processing.

CONCLUSIONS

These findings represent a promising step towards the development of assistive technologies with the potential to significantly enhance surgical practice.

Extracting lung contour deformation features with deep learning for internal target motion tracking: a preliminary study

Objective. To propose lung contour deformation features (LCDFs) as a surrogate to estimate the thoracic internal target motion, and to report their performance by correlating with the changing body using a cascade ensemble model (CEM). LCDFs, correlated to the respiration driver, are employed without patient-specific motion data sampling and additional training before treatment.Approach. LCDFs are extracted by matching lung contours via an encoder-decoder deep learning model. CEM estimates LCDFs from the currently captured body, and then uses the estimated LCDFs to track internal target motion. The accuracy of the proposed LCDFs and CEM were evaluated using 48 targets' motion data, and compared with other published methods.Main results. LCDFs estimated the internal targets with a localization error of 2.6 ± 1.0 mm (average ± standard deviation). CEM reached a localization error of 4.7 ± 0.9 mm and a real-time performance of 256.9 ± 6.0 ms. With no internal anatomy knowledge, they achieved a small accuracy difference (of 0.34∼1.10 mm for LCDFs and of 0.43∼1.75 mm for CEM at the 95% confidence level) with a patient-specific lung biomechanical model and the deformable image registration models.Significance. The results demonstrated the effectiveness of LCDFs and CEM on tracking target motion. LCDFs and CEM are non-invasive, and require no patient-specific training before treatment. They show potential for broad applications.

Development of AI-driven prediction models to realize real-time tumor tracking during radiotherapy

BACKGROUND

In infrared reflective (IR) marker-based hybrid real-time tumor tracking (RTTT), the internal target position is predicted with the positions of IR markers attached on the patient's body surface using a prediction model. In this work, we developed two artificial intelligence (AI)-driven prediction models to improve RTTT radiotherapy, namely, a convolutional neural network (CNN) and an adaptive neuro-fuzzy inference system (ANFIS) model. The models aim to improve the accuracy in predicting three-dimensional tumor motion.

METHODS

From patients whose respiration-induced motion of the tumor, indicated by the fiducial markers, exceeded 8 mm, 1079 logfiles of IR marker-based hybrid RTTT (IR Tracking) with the gimbal-head radiotherapy system were acquired and randomly divided into two datasets. All the included patients were breathing freely with more than four external IR markers. The historical dataset for the CNN model contained 1003 logfiles, while the remaining 76 logfiles complemented the evaluation dataset. The logfiles recorded the external IR marker positions at a frequency of 60 Hz and fiducial markers as surrogates for the detected target positions every 80-640 ms for 20-40 s. For each logfile in the evaluation dataset, the prediction models were trained based on the data in the first three quarters of the recording period. In the last quarter, the performance of the patient-specific prediction models was tested and evaluated. The overall performance of the AI-driven prediction models was ranked by the percentage of predicted target position within 2 mm of the detected target position. Moreover, the performance of the AI-driven models was compared to a regression prediction model currently implemented in gimbal-head radiotherapy systems.

RESULTS

The percentage of the predicted target position within 2 mm of the detected target position was 95.1%, 92.6% and 85.6% for the CNN, ANFIS, and regression model, respectively. In the evaluation dataset, the CNN, ANFIS, and regression model performed best in 43, 28 and 5 logfiles, respectively.

CONCLUSIONS

The proposed AI-driven prediction models outperformed the regression prediction model, and the overall performance of the CNN model was slightly better than that of the ANFIS model on the evaluation dataset.

Automated fiducial marker detection and localization in volumetric computed tomography images: a three-step hybrid approach with deep learning

Purpose: Automating fiducial detection and localization in the patient’s pre-operative images can lead to better registration accuracy, reduced human errors, and shorter intervention time. Most current approaches are optimized for a single marker type, mainly spherical adhesive markers. A fully automated algorithm is proposed and evaluated for screw and spherical titanium fiducials, typically used in high-accurate frameless surgical navigation.

Approach: The algorithm builds on previous approaches with morphological functions and pose estimation algorithms. A 3D convolutional neural network (CNN) is proposed for the fiducial classification task and evaluated for both traditional closed-set and emerging open-set classifiers. A proposed digital ground-truth experiment, with cone-beam computed tomography (CBCT) imaging software, is performed to determine the localization accuracy of the algorithm. The localized fiducial positions in the CBCT images by the presented algorithm were compared to the actual known positions in the virtual phantom models. The difference represents the fiducial localization error (FLE).

Results: A total of 241 screws, 151 spherical fiducials, and 1550 other structures are identified with the best true positive rate 95.9% for screw and 99.3% for spherical fiducials at 8.7% and 3.4% false positive rate, respectively. The best achieved FLE mean and its standard deviation for a screw and spherical marker are 58 (14) and 14 (6)  μm, respectively.

Conclusions: Accurate marker detection and localization were achieved, with spherical fiducials being superior to screws. Large marker volume and smaller voxel size yield significantly smaller FLEs. Attenuating noise by mesh smoothing has a minor effect on FLE. Future work will focus on expanding the CNN for image segmentation.