Statistical process control to monitor use of a web-based autoplanning tool

Application and Challenges of Statistical Process Control in Radiation Therapy Quality Assurance

Quality assurance (QA) is important for ensuring precision in radiation therapy. The complexity and resource-intensive nature of QA has increased with the continual evolution of equipment and techniques. An effective approach is to improve the process control technology and resource optimization. Statistical process control is an economical and efficient tool that has been widely used to monitor, control, and improve quality management processes and is now being increasingly used for radiation therapy QA. This article reviews the development and methodology of statistical process control technology, evaluates its suitability in radiation therapy QA practices, and assesses its importance and challenges in optimizing radiation therapy QA processes.

Monitoring Variations in the Use of Automated Contouring Software

PURPOSE

Our purpose was to identify variations in the clinical use of automatically generated contours that could be attributed to software error, off-label use, or automation bias.

METHODS AND MATERIALS

For 500 head and neck patients who were contoured by an in-house automated contouring system, Dice similarity coefficient and added path length were calculated between the contours generated by the automated system and the final contours after editing for clinical use. Statistical process control was used and control charts were generated with control limits at 3 standard deviations. Contours that exceeded the thresholds were investigated to determine the cause. Moving mean control plots were then generated to identify dosimetrists who were editing less over time, which could be indicative of automation bias.

RESULTS

Major contouring edits were flagged for: 1.0% brain, 3.1% brain stem, 3.5% left cochlea, 2.9% right cochlea, 4.8% esophagus, 4.1% left eye, 4.0% right eye, 2.2% left lens, 4.9% right lens, 2.5% mandible, 11% left optic nerve, 6.1% right optic nerve, 3.8% left parotid, 5.9% right parotid, and 3.0% of spinal cord contours. Identified causes of editing included unexpected patient positioning, deviation from standard clinical practice, and disagreement between dosimetrist preference and automated contouring style. A statistically significant (P < .05) difference was identified between the contour editing practice of dosimetrists, with 1 dosimetrist editing more across all organs at risk. Eighteen percent (27/150) of moving mean control plots created for 5 dosimetrists indicated the amount of contour editing was decreasing over time, possibly corresponding to automation bias.

CONCLUSIONS

The developed system was used to detect statistically significant edits caused by software error, unexpected clinical use, and automation bias. The increased ability to detect systematic errors that occur when editing automatically generated contours will improve the safety of the automatic treatment planning workflow.

Automated contouring and statistical process control for plan quality in a breast clinical trial

Background and purpose

Automatic review of breast plan quality for clinical trials is time-consuming and has some unique challenges due to the lack of target contours for some planning techniques. We propose using an auto-contouring model and statistical process control to independently assess planning consistency in retrospective data from a breast radiotherapy clinical trial.

Materials and methods

A deep learning auto-contouring model was created and tested quantitatively and qualitatively on 104 post-lumpectomy patients' computed tomography images (nnUNet; train/test: 80/20). The auto-contouring model was then applied to 127 patients enrolled in a clinical trial. Statistical process control was used to assess the consistency of the mean dose to auto-contours between plans and treatment modalities by setting control limits within three standard deviations of the data's mean. Two physicians reviewed plans outside the limits for possible planning inconsistencies.

Results

Mean Dice similarity coefficients comparing manual and auto-contours was above 0.7 for breast clinical target volume, supraclavicular and internal mammary nodes. Two radiation oncologists scored 95% of contours as clinically acceptable. The mean dose in the clinical trial plans was more variable for lymph node auto-contours than for breast, with a narrower distribution for volumetric modulated arc therapy than for 3D conformal treatment, requiring distinct control limits. Five plans (5%) were flagged and reviewed by physicians: one required editing, two had clinically acceptable variations in planning, and two had poor auto-contouring.

Conclusions

An automated contouring model in a statistical process control framework was appropriate for assessing planning consistency in a breast radiotherapy clinical trial.