Joint scene and object tracking for cost-Effective augmented reality guided patient positioning in radiation therapy

Innovative integration of augmented reality and optical surface imaging: A coarse-to-precise system for radiotherapy positioning

BACKGROUND

Traditional methods of radiotherapy positioning have shortcomings such as fragile skin-markers, additional doses, and lack of information integration. Emerging technologies may provide alternatives for the relevant clinical practice.

PURPOSE

To propose a noninvasive radiotherapy positioning system integrating augmented reality (AR) and optical surface, and to evaluate its feasibility in clinical workflow.

METHODS

AR and structured light-based surface were integrated to implement the coarse-to-precise positioning through two coherent steps, the AR-based coarse guidance and the optical surface-based precise verification. To implement quality assurance, recognition of face and pattern was used for patient authentication, case association, and accessory validation in AR scenes. The holographic images reconstructed from simulation computed tomography (CT) images, guided the initial posture correction by virtual-real alignment. The point clouds of body surface were fused, with the calibration and pose estimation of structured light cameras, and segmented according to the preset regions of interest (ROIs). The global-to-local registration for cross-source point clouds was achieved to calculate couch shifts in six degrees-of-freedom (DoF), which were ultimately transmitted to AR scenes. The evaluation based on phantom and human-body (4 volunteers) included, (i) quality assurance workflow, (ii) errors of both steps and correlation analysis, (iii) receiver operating characteristic (ROC), (iv) distance characteristics of accuracy, and (v) clinical positioning efficiency.

RESULTS

The maximum errors in phantom evaluation were 3.4 ± 2.5 mm in Vrt and 1.4 ± 1.0° in Pitch for the coarse guidance step, while 1.6 ± 0.9 mm in Vrt and 0.6 ± 0.4° in Pitch for the precise verification step. The Pearson correlation coefficients between precise verification and cone beam CT (CBCT) results were distributed in the interval [0.81, 0.85]. In ROC analysis, the areas under the curve (AUC) were 0.87 and 0.89 for translation and rotation, respectively. In human body-based evaluation, the errors of thorax and abdomen (T&A) were significantly greater than those of head and neck (H&N) in Vrt (2.6 ± 1.1 vs. 1.7 ± 0.8, p < 0.01), Lng (2.3 ± 1.1 vs. 1.4 ± 0.9, p < 0.01), and Rtn (0.8 ± 0.4 vs. 0.6 ± 0.3, p = 0.01) while relatively similar in Lat (1.8 ± 0.9 vs. 1.7 ± 0.8, p = 0.07). The translation displacement range, after coarse guidance step, required for high accuracy of the optical surface component of the integrated system was 0-42 mm, and the average positioning duration of the integrated system was significantly less than that of conventional workflow (355.7 ± 21.7 vs. 387.7 ± 26.6 s, p < 0.01).

CONCLUSIONS

The combination of AR and optical surface has utility and feasibility for patient positioning, in terms of both safety and accuracy.

A Novel Method for Fast Generation of 3D Objects from Multiple Depth Sensors

Scanning real 3D objects face many technical challenges. Stationary solutions allow for accurate scanning. However, they usually require special and expensive equipment. Competitive mobile solutions (handheld scanners, LiDARs on vehicles, etc.) do not allow for an accurate and fast mapping of the surface of the scanned object. The article proposes an end-to-end automated solution that enables the use of widely available mobile and stationary scanners. The related system generates a full 3D model of the object based on multiple depth sensors. For this purpose, the scanned object is marked with markers. Markers type and positions are automatically detected and mapped to a template mesh. The reference template is automatically selected for the scanned object, which is then transformed according to the data from the scanners with non-rigid transformation. The solution allows for the fast scanning of complex and varied size objects, constituting a set of training data for segmentation and classification systems of 3D scenes. The main advantage of the proposed solution is its efficiency, which enables real-time scanning and the ability to generate a mesh with a regular structure. It is critical for training data for machine learning algorithms. The source code is available at https://github.com/SATOffice/improved_scanner3D.

Patient posture correction and alignment using mixed reality visualization and the HoloLens 2

PURPOSE

The purpose of this study was to develop and preliminarily test a radiotherapy system for patient posture correction and alignment using mixed reality (MixR) visualization. The write-up of this work also provides an opportunity to introduce the concepts and technology of MixR for a medical physics audience who may be unfamiliar with the topic.

METHODS

A MixR application was developed for on optical-see-through head-mounted display (HoloLens 2) allowing a user to simultaneously and directly view a patient and a reference hologram derived from their simulation CT scan. The hologram provides a visual reference for the exact posture needed during treatment and is initialized in relation to the origin of a radiotherapy device using marker-based tracking. The system further provides marker-less tracking that allows the user tofreely navigate the room as they view and align the patient from various angles. The system was preliminarily tested using both a rigid (pelvis) and nonrigid (female mannequin) anthropomorphic phantom. Each phantom was aligned via hologram and accuracy quantified using CBCT and CT.

RESULTS

A fully realized system was developed. Rigid registration accuracy was on the order of 3.0 ± 1.5 mm based on the performance of three users repeating alignment five times each. The lateral direction showed the most variability among users and was associated with the largest off-sets (approximately 2.0 mm). For nonrigid alignment, the MixR setup outperformed a setup based on three-point alignment and setup photos, the latter of which showed a difference in arm position of 2 cm and a torso roll of 6-7°.

CONCLUSIONS

MixR visualization is a rapidly emerging domain that has the potential to significantly impact the field of medicine. The current application is an illustration of this and highlights the advantages of MixR for patient setup in radiation oncology. The key feature of the system is the way in which it transforms nonrigid registration into rigid registration by providing an efficient, portable, and cost-effective mechanism for reproducing patient posture without the use of ionizing radiation. Preliminary estimates of registration accuracy indicate clinical viability and form the foundation for further development and clinical testing.