Use of Receiver Operating Curve Analysis and Machine Learning With an Independent Dose Calculation System Reduces the Number of Physical Dose Measurements Required for Patient-Specific Quality Assurance

Is it still necessary to perform measured based pre-treatment patient-specific QA for SRS HyperArc treatments?

PURPOSE

The Mobius3D system was validated as a modern secondary check dosimetry system. In particular, our objective has been to assess the suitability of the M3D as pre-treatment patient-specific Quality Assurance (QA) tool for Stereotactic Radiosurgery (SRS) HyperArc (HA) treatments. We aimed to determine whether Mobius3D could safely replace the measurements-based patient-specific QA for this type of treatment.

METHODS

30 SRS HA treatment plans for brain were selected. The dose distributions, calculated by Mobius and our routinely used algorithm (AcurosXB v.15.6), were compared using gamma analysis index and DVH parameters based on the patient's CT dataset. All 30 plans were then delivered across the ionization chamber in a homogeneous phantom and the measured dose was compared with both M3D and TPS calculated one. The plans were delivered and verified in terms of PSQA using the electronic portal imaging device (EPID) with Portal Dosimetry (PD) and myQA SRS (IBA Dosimetry) detector. Plans that achieved a global gamma passing rate (GPR) ≥ 97% based on 2%/2 mm criteria, with both Mobius3D and the conventional methods were evaluated acceptable. Finally, we assessed the capability of the M3D system to detect errors related to the position of the Multi-Leaf Collimator (MLC) in comparison to the analyzed measurement-based systems.

RESULTS

No relevant differences were observed in the comparison between the dose calculated on the CT-dataset by M3D and the TPS. Observed discrepancies are imputable to different used algorithms, but no discrepancies related to goodness of plans have been found. Average differences between calculated (M3D and TPS) vs measured dose with ionization chamber were 2.5% (from 0.41% to 3.2%) and 1.81% (from 0.66% to 2.65%), for M3D and TPS, respectively. All plans passed with a gamma passing rate > 97% using conventional PSQA methods with a gamma criterion of 2% dose difference and 2 mm distance-to-agreement. The average gamma passing rate for the M3D system was determined to be 99.4% (from 97.3% to 100%). Results from this study also demonstrated Mobius has better error detectability than conventional measurement-based systems.

CONCLUSION

Our study shows Mobius3D could be a suitable alternative to conventional measured based QA methods for SRS HyperArc treatments.

Virtual pretreatment patient-specific quality assurance of volumetric modulated arc therapy using deep learning

BACKGROUND

Automatic patient-specific quality assurance (PSQA) is recently explored using artificial intelligence approaches, and several studies reported the development of machine learning models for predicting the gamma pass rate (GPR) index only.

PURPOSE

To develop a novel deep learning approach using a generative adversarial network (GAN) to predict the synthetic measured fluence.

METHODS AND MATERIALS

A novel training method called "dual training," which involves the training of the encoder and decoder separately, was proposed and evaluated for cycle GAN (cycle-GAN) and conditional GAN (c-GAN). A total of 164 VMAT treatment plans, including 344 arcs (training data: 262, validation data: 30, and testing data: 52) from various treatment sites, were selected for prediction model development. For each patient, portal-dose-image-prediction fluence from TPS was used as input, and measured fluence from EPID was used as output/response for model training. Predicted GPR was derived by comparing the TPS fluence with the synthetic measured fluence generated by the DL models using gamma evaluation of criteria 2%/2 mm. The performance of dual training was compared against the traditional single-training approach. In addition, we also developed a separate classification model specifically designed to detect automatically three types of errors (rotational, translational, and MU-scale) in the synthetic EPID-measured fluence.

RESULTS

Overall, the dual training improved the prediction accuracy of both cycle-GAN and c-GAN. Predicted GPR results of single training were within 3% for 71.2% and 78.8% of test cases for cycle-GAN and c-GAN, respectively. Moreover, similar results for dual training were 82.7% and 88.5% for cycle-GAN and c-GAN, respectively. The error detection model showed high classification accuracy (>98%) for detecting errors related to rotational and translational errors. However, it struggled to differentiate the fluences with "MU scale error" from "error-free" fluences.

CONCLUSION

We developed a method to automatically generate the synthetic measured fluence and identify errors within them. The proposed dual training improved the PSQA prediction accuracy of both the GAN models, with c-GAN demonstrating superior performance over the cycle-GAN. Our results indicate that the c-GAN with dual training approach combined with error detection model, can accurately generate the synthetic measured fluence for VMAT PSQA and identify the errors. This approach has the potential to pave the way for virtual patient-specific QA of VMAT treatments.

Computation of the distribution of model accuracy statistics in machine learning: Comparison between analytically derived distributions and simulation-based methods

Background and Aims

All fields have seen an increase in machine-learning techniques. To accurately evaluate the efficacy of novel modeling methods, it is necessary to conduct a critical evaluation of the utilized model metrics, such as sensitivity, specificity, and area under the receiver operator characteristic curve (AUROC). For commonly used model metrics, we proposed the use of analytically derived distributions (ADDs) and compared it with simulation-based approaches.

Methods

A retrospective cohort study was conducted using the England National Health Services Heart Disease Prediction Cohort. Four machine learning models (XGBoost, Random Forest, Artificial Neural Network, and Adaptive Boost) were used. The distribution of the model metrics and covariate gain statistics were empirically derived using boot-strap simulation (N = 10,000). The ADDs were created from analytic formulas from the covariates to describe the distribution of the model metrics and compared with those of bootstrap simulation.

Results

XGBoost had the most optimal model having the highest AUROC and the highest aggregate score considering six other model metrics. Based on the Anderson-Darling test, the distribution of the model metrics created from bootstrap did not significantly deviate from a normal distribution. The variance created from the ADD led to smaller SDs than those derived from bootstrap simulation, whereas the rest of the distribution remained not statistically significantly different.

Conclusions

ADD allows for cross study comparison of model metrics, which is usually done with bootstrapping that rely on simulations, which cannot be replicated by the reader.

Treatment plan prescreening for patient-specific quality assurance measurements using independent Monte Carlo dose calculations

PURPOSE

This study proposes a method to identify plans that failed patient-specific quality assurance (QA) and attempts to establish a criterion to prescreen treatment plans for patient-specific QA measurements with independent Monte Carlo dose calculations.

MATERIALS AND METHODS

Patient-specific QA results measured with an ArcCHECK diode array of 207 patients (head and neck: 25; thorax: 61; abdomen: 121) were retrospectively analyzed. All patients were treated with the volumetric modulated arc therapy (VMAT) technique and plans were optimized with a Pinnacle v16.2 treatment planning system using an analytical algorithm-based dose engine. Afterwards, phantom verification plans were designed and recalculated by an independent GPU-accelerated Monte Carlo (MC) dose engine, ArcherQA. Moreover, sensitivity and specificity analyzes of gamma passing rates between measurements and MC calculations were carried out to show the ability of MC to monitor failing plans (ArcCHECK 3%/3 mm,<90%), and attempt to determine the appropriate threshold and gamma passing rate criterion utilized by ArcherQA to prescreen treatment plans for ArcCHECK measurements. The receiver operator characteristic (ROC) curve was also utilized to characterize the performance of different gamma passing rate criterion used by ArcherQA.

RESULTS

The thresholds for 100% sensitivity to detect plans that failed patient-specific QA by independent calculation were 97.0%, 95.4%, and 91.0% for criterion 3%/3 mm, 3%/2 mm, and 2%/2 mm, respectively, which corresponded to specificities of 0.720, 0.528, and 0.585, respectively. It was shown that the 3%/3 mm criterion with 97% threshold for ArcherQA demonstrated perfect sensitivity and the highest specificity compared with other criteria, which may be suitable for prescreening treatment plans treated with the investigated machine to implement measurement-based patient-specific QA of patient plans. In addition, the area under the curve (AUC) calculated from ROC analysis for criterion 3%/3 mm, 3%/2 mm, and 2%/2 mm used by ArcherQA were 0.948, 0.924, and 0.929, respectively.

CONCLUSIONS

Independent dose calculation with the MC-based program ArcherQA has potential as a prescreen treatment for measurement-based patient-specific QA. AUC values (>0.9) showed excellent classification accuracy for monitoring failing plans with independent MC calculations.

Prospective clinical validation of virtual patient-specific quality assurance of VMAT radiation therapy plans

PURPOSE

Performing measurement-based patient-specific quality assurance (PSQA) is recognized as a resource-intensive and time inefficient task in the radiotherapy treatment workflow. Paired with technological refinements in modern radiotherapy, research towards measurement-free PSQA has seen increased interest over the last five years. However, these efforts have not been clinically implemented or prospectively validated in the U.S. We propose a virtual QA (VQA) system and workflow to assess the safety and workload reduction of measurement-free PSQA.

METHODS

An XGBoost machine learning model was designed to predict PSQA outcomes of VMAT plans, represented as percent differences between the measured ion chamber point dose in a phantom and the corresponding planned dose. The final model was deployed within a web application to predict PSQA outcomes of clinical plans within an existing clinical workflow. The application also displays relevant feature importance and plan-specific distribution analyses relative to database plans for documentation and to aid physicist interpretation and evaluation. VQA predictions were prospectively validated over three months of measurements at our clinic to assess safety and efficiency gains.

RESULTS

Over three months, VQA predictions for 445 VMAT plans were prospectively validated at our institution. VQA predictions for these plans had a mean absolute error of 1.08 +/- 0.77%, with a maximum absolute error of 2.98%. Employing a 1% prediction threshold (i.e. plans predicted to have an absolute error of less than 1% would not require a measurement) would yield a 69.2% reduction in QA workload - saving 32.5 hours per month on average - with 81.5%/72.4%/0.81 sensitivity/specificity/AUC at a 3% clinical threshold and 100%/70%/0.93 sensitivity/specificity/AUC at a 4% clinical threshold.

CONCLUSION

This is the first prospective clinical implementation and validation of VQA in the U.S., which we observed to be efficient. Using a conservative threshold, VQA can substantially reduce the number of required measurements for PSQA, leading to more effective allocation of clinical resources.