Computerized System for Safety Verification of External Beam Radiation Therapy Planning

An Automated, Dynamic Radiation Oncology Prescription Checking System

PURPOSE

Despite serving as a critical communication tool, radiation oncology prescriptions are entered manually and prone to error. An automated prescription checking system was developed and implemented to help address this problem.

METHODS AND MATERIALS

Rules defining clinically appropriate prescriptions were generated, examining specific types of errors: (1) unapproved dose per fraction for a given disease site; (2) dose per fraction too large for nonstereotactic treatment technique; and (3) dose per fraction too low. With a goal of catching errors as upstream as possible to minimize their propagation, a report was created and ran every 30 minutes to check all newly written or approved prescriptions against the 3 rules. When a prescription violated these rules, an automated email was immediately sent to the prescriber alerting them of the potential error. System performance was continuously monitored and the criteria triggering an alert adjusted to balance error detection against false positives. Alerts leading to prescription amendment were considered true errors.

RESULTS

From June 2021 to November 2022, the system checked 24,047 prescriptions. A total of 241 email alerts were triggered, for an average alert rate of 1%. Of the 241 alerts, 198 (82.2%) were unapproved doses per fraction for the disease site, 14 (5.8%) were doses per fraction that were too low, and 29 (12%) were doses too large for nonstereotactic treatment technique. Thirty-one percent of alerts led to a change of prescription, suggesting they were true errors. The baseline rate of erroneous prescription entry was 0.3%. A regression model showed that trainee prescription entry and dose per fraction <150 cGy were significantly associated with true errors.

CONCLUSIONS

Given the significant consequences of erroneous prescription entry, which ranged from wasted resources and treatment delays to potentially serious misadministration, there is significant value in implementing automated prescription checking systems in radiation oncology clinics.

Implementation of a knowledge-based decision support system for treatment plan auditing through automation

BACKGROUND

Independent auditing is a necessary component of a comprehensive quality assurance (QA) program and can also be utilized for continuous quality improvement (QI) in various radiotherapy processes. Two senior physicists at our institution have been performing a time intensive manual audit of cross-campus treatment plans annually, with the aim of further standardizing our planning procedures, updating policies and guidelines, and providing training opportunities of all staff members.

PURPOSE

A knowledge-based automated anomaly-detection algorithm to provide decision support and strengthen our manual retrospective plan auditing process was developed. This standardized and improved the efficiency of the assessment of our external beam radiotherapy (EBRT) treatment planning across all eight campuses of our institution.

METHODS

A total of 843 external beam radiotherapy plans for 721 lung patients from January 2020 to March 2021 were automatically acquired from our clinical treatment planning and management systems. From each plan, 44 parameters were automatically extracted and pre-processed. A knowledge-based anomaly detection algorithm, namely, "isolation forest" (iForest), was then applied to the plan dataset. An anomaly score was determined for each plan using recursive partitioning mechanism. Top 20 plans ranked with the highest anomaly scores for each treatment technique (2D/3D/IMRT/VMAT/SBRT) including auto-populated parameters were used to guide the manual auditing process and validated by two plan auditors.

RESULTS

The two auditors verified that 75.6% plans with the highest iForest anomaly scores have similar concerning qualities that may lead to actionable recommendations for our planning procedures and staff training materials. The time to audit a chart was approximately 20.8 min on average when done manually and 14.0 min when done with the iForest guidance. Approximately 6.8 min were saved per chart with the iForest method. For our typical internal audit review of 250 charts annually, the total time savings are approximately 30 hr per year.

CONCLUSION

iForest effectively detects anomalous plans and strengthens our cross-campus manual plan auditing procedure by adding decision support and further improve standardization. Due to the use of automation, this method was efficient and will be used to establish a standard plan auditing procedure, which could occur more frequently.

Automatic quality assurance of radiotherapy treatment plans using Bayesian networks: A multi-institutional study

Purpose

Artificial intelligence applications in radiation oncology have been the focus of study in the last decade. The introduction of automated and intelligent solutions for routine clinical tasks, such as treatment planning and quality assurance, has the potential to increase safety and efficiency of radiotherapy. In this work, we present a multi-institutional study across three different institutions internationally on a Bayesian network (BN)-based initial plan review assistive tool that alerts radiotherapy professionals for potential erroneous or suboptimal treatment plans.

Methods

Clinical data were collected from the oncology information systems in three institutes in Europe (Maastro clinic - 8753 patients treated between 2012 and 2020) and the United States of America (University of Vermont Medical Center [UVMMC] - 2733 patients, University of Washington [UW] - 6180 patients, treated between 2018 and 2021). We trained the BN model to detect potential errors in radiotherapy treatment plans using different combinations of institutional data and performed single-site and cross-site validation with simulated plans with embedded errors. The simulated errors consisted of three different categories: i) patient setup, ii) treatment planning and iii) prescription. We also compared the strategy of using only diagnostic parameters or all variables as evidence for the BN. We evaluated the model performance utilizing the area under the receiver-operating characteristic curve (AUC).

Results

The best network performance was observed when the BN model is trained and validated using the dataset in the same center. In particular, the testing and validation using UVMMC data has achieved an AUC of 0.92 with all parameters used as evidence. In cross-validation studies, we observed that the BN model performed better when it was trained and validated in institutes with similar technology and treatment protocols (for instance, when testing on UVMMC data, the model trained on UW data achieved an AUC of 0.84, compared with an AUC of 0.64 for the model trained on Maastro data). Also, combining training data from larger clinics (UW and Maastro clinic) and using it on smaller clinics (UVMMC) leads to satisfactory performance with an AUC of 0.85. Lastly, we found that in general the BN model performed better when all variables are considered as evidence.

Conclusion

We have developed and validated a Bayesian network model to assist initial treatment plan review using multi-institutional data with different technology and clinical practices. The model has shown good performance even when trained on data from clinics with divergent profiles, suggesting that the model is able to adapt to different data distributions.

Dosimetry audit for megavoltage photon beams applied in non-reference conditions

PURPOSE

We have conducted for the first time a Malaysian postal dosimetry audit of external beam under non-reference conditions by evaluating the output performance while screening for systematic errors within the dosimetry chain. The potential use from the choice of detector were investigated along with the search for other sources of discrepancies.

METHODS

Ten radiotherapy centres were audited, encompassing 16 megavoltage photon beam arrangements, adopting the IAEA postal dosimetry protocol for non-reference conditions, with a holder modified to accommodate three TLD types: Ge-doped cylindrical silica fibres (CF), Ge-doped flat silica fibres (FF), and TLD-100 powder.

RESULTS

Eight of the centres operated within ± 5% of stated dose, one other exceeding tolerance for all measured points, and one did not return any dosimeters for analysis after failing the initial irradiations. Post remedial measures, the mean relative response for CF, FF, and TLD-100 was 1.00, 0.99, and 0.98 respectively, with associated coefficients of variation 6.87%, 6.45%, and 5.06%.

CONCLUSION

High quality radiotherapy clinical practice postal dosimetry audits that are based on sensitive TLDs are seen to be particularly effective in identifying and resolving dose delivery discrepancies.

Barriers and facilitators to clinical implementation of radiotherapy treatment planning automation: A survey study of medical dosimetrists

PURPOSE

Little is known about the scale of clinical implementation of automated treatment planning techniques in the United States. In this work, we examine the barriers and facilitators to adoption of commercially available automated planning tools into the clinical workflow using a survey of medical dosimetrists.

METHODS/MATERIALS

Survey questions were developed based on a literature review of automation research and cognitive interviews of medical dosimetrists at our institution. Treatment planning automation was defined to include auto-contouring and automated treatment planning. Survey questions probed frequency of use, positive and negative perceptions, potential implementation changes, and demographic and institutional descriptive statistics. The survey sample was identified using both a LinkedIn search and referral requests sent to physics directors and senior physicists at 34 radiotherapy clinics in our state. The survey was active from August 2020 to April 2021.

RESULTS

Thirty-four responses were collected out of 59 surveys sent. Three categories of barriers to use of automation were identified. The first related to perceptions of limited accuracy and usability of the algorithms. Eighty-eight percent of respondents reported that auto-contouring inaccuracy limited its use, and 62% thought it was difficult to modify an automated plan, thus limiting its usefulness. The second barrier relates to the perception that automation increases the probability of an error reaching the patient. Third, respondents were concerned that automation will make their jobs less satisfying and less secure. Large majorities reported that they enjoyed plan optimization, would not want to lose that part of their job, and expressed explicit job security fears.

CONCLUSION

To our knowledge this is the first systematic investigation into the views of automation by medical dosimetrists. Potential barriers and facilitators to use were explicitly identified. This investigation highlights several concrete approaches that could potentially increase the translation of automation into the clinic, along with areas of needed research.

Improving the Quality of Care in Radiation Oncology using Artificial Intelligence

Radiation therapy is a complex process involving multiple professionals and steps from simulation to treatment planning to delivery, and these procedures are prone to error. Additionally, the imaging and treatment delivery equipment in radiotherapy is highly complex and interconnected and represents another risk point in the quality of care. Numerous quality assurance tasks are carried out to ensure quality and to detect and prevent potential errors in the process of care. Recent developments in artificial intelligence provide potential tools to the radiation oncology community to improve the efficiency and performance of quality assurance efforts. Targets for artificial intelligence enhancement include the quality assurance of treatment plans, target and tissue structure delineation used in the plans, delivery of the plans and the radiotherapy delivery equipment itself. Here we review recent developments of artificial intelligence applications that aim to improve quality assurance processes in radiation therapy and discuss some of the challenges and limitations that require further development work to realise the potential of artificial intelligence for quality assurance.

Saudi Radiology Personnel's Perceptions of Artificial Intelligence Implementation: A Cross-Sectional Study

PURPOSE

Artificial intelligence (AI) in radiology has been a subject of heated debate. The external perception is that algorithms and machines cannot offer better diagnosis than radiologists. Reluctance to implement AI maybe due to the opacity in how AI applications work and the challenging and lengthy validation process. In this study, Saudi radiology personnel's familiarity with AI applications and its usefulness in clinical practice were investigated.

METHODS

A cross-sectional study was conducted in Saudi Arabia among radiology personnel from March to April 2021. Radiology personnel nationwide were surveyed electronically using Google form. The questionnaire included 12-questions related to AI usefulness in clinical practice and participants' knowledge about AI and their acceptance level to learn and implement this technology into clinical practice. Participants' trust level was also measured; Kruskal-Wallis test was used to examine differences between groups.

RESULTS

A total of 224 respondents from various radiology-related occupations participated in the survey. The lowest trust level in AI applications was shown by radiologists (p = 0.033). Eighty-two percent of participants (n = 184) had never used AI in their departments. Most respondents (n = 160, 71.4%) reported lack of formal education regarding AI-based applications. Most participants (n = 214, 95.5%) showed strong interest in AI education and are willing to incorporate it into the clinical practice of radiology. Almost half of radiography students (22/46, 47.8%) believe that their job might be at risk due to AI application (p = 0.038).

CONCLUSION

Radiology personnel's knowledge of AI has a significant impact on their willingness to learn, use and adapt this technology in clinical practice. Participants demonstrated a positive attitude towards AI, showed a reasonable understanding and are highly motivated to learn and incorporate it into clinical practice. Some participants felt that their jobs were threatened by AI adaptation, but this belief might change with good training and education programmes.

Artificial intelligence: The opinions of radiographers and radiation therapists in Ireland

INTRODUCTION

Implementation of Artificial Intelligence (AI) into medical imaging is much debated. Diagnostic Radiographers (DRs) and Radiation Therapists (RTTs) are at the forefront of this technological leap, thus an understanding of their views, in particular changes to their current roles, is key to safe, optimal implementation.

METHODS

An online survey was designed, including themes: role changes, clinical priorities for AI, patient benefits, and education. It was distributed nationally in the Republic of Ireland via the national professional body, clinical management, and social media.

RESULTS

318 DRs and 77 RTTs participated. Priority areas for development included quality assurance, clinical audit, radiation dose optimisation, and improved workflow for DRs and treatment planning algorithm optimisation, clinical audit, and post processing for RTTs. There was resistance regarding AI use for patient facing roles and final image interpretation. 27.6% of DRs and 40.3% of RTTs currently use AI clinically and 46.1% of DRs and 41.2% of RTTs anticipate reduced staffing levels with AI. 64.9% of DRs and 70.6% of RTTs felt AI will be positive for patients, with the majority promoting AI regulation through national legislation. 86.1% of DRs and 94.0% of RTTs were favourable to AI implementation.

CONCLUSION

This research identifies priority AI development and implementation areas for DRs and RTTs. It thus highlights that DRs and RTTs should be involved in development of AI tools that would best support practice, and that clearly defined pathways for AI implementation into these key professions requires discussion so that optimum use and patient safety can ensue.

IMPLICATIONS FOR PRACTICE

Understanding opinions of AI has significant implications for practice, for ensuring optimal product development, implementation, and training, together with planning for potential DR and RTT role changes.

Toward automation of initial chart check for photon/electron EBRT: the clinical implementation of new AAPM task group reports and automation techniques

Purpose

The recently published AAPM TG‐275 and the public review version of TG‐315 list new recommendations for comprehensive and minimum physics initial chart checks, respectively. This article addresses the potential development and benefit of initial chart check automation when these recommendations are implemented for clinical photon/electron EBRT.

Methods

Eight board‐certified physicists with 2–20 years of clinical experience performed initial chart checks using checklists from TG‐275 and TG‐315. Manual check times were estimated for three types of plans (IMRT/VMAT, 3D, and 2D) and for prostate, whole pelvis, lung, breast, head and neck, and brain cancers. An expert development team of three physicists re‐evaluated the automation feasibility of TG‐275 checklist based on their experience of developing and implementing the in‐house and the commercial automation tools in our institution. Three levels of initial chart check automation were simulated: (1) Auto_UMMS_tool (which consists of in‐house program and commercially available software); (2) Auto_TG275 (with full and partial automation as indicated in TG‐275); and (3) Auto_UMMS_exp (with full and partial automation as determined by our experts’ re‐evaluation).

Results

With no automation of initial chart checks, the ranges of manual check times were 29–56 min (full TG‐315 list) and 102–163 min (full TG‐275 list), which varied significantly with physicists but varied little at different tumor sites. The 69 of 71 checks which were considered as “not fully automated” in TG‐275 were re‐evaluated with more automation feasibility. Compared to no automation, the higher levels of automation yielded a great reduction in both manual check times (by 44%–98%) and potentially residual detectable errors (by 15–85%).

Conclusion

The initial chart check automation greatly improves the practicality and efficiency of implementing the new TG recommendations. Revisiting the TG reports with new technology/practice updates may help develop and utilize more automation clinically.

Graph-based risk assessment and error detection in radiation therapy

PURPOSE

The objective of this study was to formalize and automate quality assurance (QA) in radiation oncology. Quality assurance in radiation oncology entails a multistep verification of complex, personalized radiation plans to treat cancer involving an interdisciplinary team and high technology, multivendor software and hardware systems. We addressed the pretreatment physics chart review (TPCR) using methods from graph theory and constraint programming to study the effect of dependencies between variables and automatically identify logical inconsistencies and how they propagate.

MATERIALS AND METHODS

We used a modular approach to decompose the TPCR process into tractable units comprising subprocesses, modules and variables. Modules represented the main software entities comprised in the radiation treatment planning workflow and subprocesses grouped the checks to be performed by functionality. Module-associated variables served as inputs to the subprocesses. Relationships between variables were modeled by means of a directed graph. The detection of errors, in the form of inconsistencies, was formalized as a constraint satisfaction problem whereby checks were encoded as logical formulae. The sequence in which subprocesses were visited was described in an activity diagram.

RESULTS

The comprehensive model for the TPCR process comprised 5 modules, 19 subprocesses and 346 variables, 225 of which were distinct. Modules included "Treatment Planning System" and "Record and Verify System." Subprocesses included "Dose Prescription," "Documents," "CT Integrity," "Anatomical Contours," "Beam Configuration," "Dose Calculation," "3D Dose Distribution Quality," and "Treatment Approval." Variable inconsistencies, and their source and propagation were determined by checking for constraint violation and through graph traversal. Impact scores, obtained through graph traversal, combined with severity scores associated with an inconsistency, allowed risk assessment.

CONCLUSIONS

Directed graphs combined with constraint programming hold promise for formalizing complex QA processes in radiation oncology, performing risk assessment and automating the TPCR process. Though complex, the process is tractable.

Cautiously optimistic: A survey of radiation oncology professionals' perceptions of automation in radiotherapy planning

INTRODUCTION

While there is evidence to show the positive effects of automation, the impact on radiation oncology professionals has been poorly considered. This study examined radiation oncology professionals' perceptions of automation in radiotherapy planning.

METHOD

An online survey link was sent to the chief radiation therapists (RT) of all Australian radiotherapy centres to be forwarded to RTs, medical physicists (MP) and radiation oncologists (RO) within their institution. The survey was open from May-July 2019.

RESULTS

Participants were 204 RTs, 84 MPs and 37 ROs (response rates ∼10% of the overall radiation oncology workforce). Respondents felt automation resulted in improvement in consistency in planning (90%), productivity (88%), quality of planning (57%), and staff focus on patient care (49%). When asked about perceived impact of automation, the responses were; will change the primary tasks of certain jobs (66%), will allow staff to do the remaining components of their job more effectively (51%), will eliminate jobs (20%), and will not have an impact on jobs (6%). 27% of respondents believe automation will reduce job satisfaction. 71% of respondents strongly agree/agree that automation will cause a loss of skills, while only 25% strongly agree/agree that the training and education tools in their department are sufficient.

CONCLUSION

Although the effect of automation is perceived positively, there are some concerns on loss of skillsets and the lack of training to maintain this. These results highlight the need for continued education to ensure that skills and knowledge are not lost with automation.

Optimizing efficiency and safety in external beam radiotherapy using automated plan check (APC) tool and six sigma methodology

PURPOSE

To develop and implement an automated plan check (APC) tool using a Six Sigma methodology with the aim of improving safety and efficiency in external beam radiotherapy.

METHODS

The Six Sigma define-measure-analyze-improve-control (DMAIC) framework was used by measuring defects stemming from treatment planning that were reported to the departmental incidence learning system (ILS). The common error pathways observed in the reported data were combined with our departmental physics plan check list, and AAPM TG-275 identified items. Prioritized by risk priority number (RPN) and severity values, the check items were added to the APC tool developed using Varian Eclipse Scripting Application Programming Interface (ESAPI). At 9 months post-APC implementation, the tool encompassed 89 check items, and its effectiveness was evaluated by comparing RPN values and rates of reported errors. To test the efficiency gains, physics plan check time and reported error rate were prospectively compared for 20 treatment plans.

RESULTS

The APC tool was successfully implemented for external beam plan checking. FMEA RPN ranking re-evaluation at 9 months post-APC demonstrated a statistically significant average decrease in RPN values from 129.2 to 83.7 (P < .05). After the introduction of APC, the average frequency of reported treatment-planning errors was reduced from 16.1% to 4.1%. For high-severity errors, the reduction was 82.7% for prescription/plan mismatches and 84.4% for incorrect shift note. The process shifted from 4σ to 5σ quality for isocenter-shift errors. The efficiency study showed a statistically significant decrease in plan check time (10.1 ± 7.3 min, P = .005) and decrease in errors propagating to physics plan check (80%).

CONCLUSIONS

Incorporation of APC tool has significantly reduced the error rate. The DMAIC framework can provide an iterative and robust workflow to improve the efficiency and quality of treatment planning procedure enabling a safer radiotherapy process.

Early detection of potential errors during patient treatment planning

Purpose

Data errors caught late in treatment planning require time to correct, resulting in delays up to 1 week. In this work, we identify causes of data errors in treatment planning and develop a software tool that detects them early in the planning workflow.

Methods

Two categories of errors were studied: data transfer errors and TPS errors. Using root cause analysis, the causes of these errors were determined. This information was incorporated into a software tool which uses ODBC‐SQL service to access TPS's Postgres and Mosaiq MSSQL databases for our clinic. The tool then uses a read‐only FTP service to scan the TPS unix file system for errors. Detected errors are reviewed by a physicist. Once confirmed, clinicians are notified to correct the error and educated to prevent errors in the future. Time‐cost analysis was performed to estimate the time savings of implementing this software clinically.

Results

The main errors identified were incorrect patient entry, missing image slice, and incorrect DICOM tag for data transfer errors and incorrect CT‐density table application, incorrect image as reference CT, and secondary image imported to incorrect patient for TPS errors. The software has been running automatically since 2015. In 2016, 84 errors were detected with the most frequent errors being incorrect patient entry (35), incorrect CT‐density table (17), and missing image slice (16). After clinical interventions to our planning workflow, the number of errors in 2017 decreased to 44. Time savings in 2016 with the software is estimated to be 795 h. This is attributed to catching errors early and eliminating the need to replan cases.

Conclusions

New QA software detects errors during planning, improving the accuracy and efficiency of the planning process. This important QA tool focused our efforts on the data communication processes in our planning workflow that need the most improvement.

Risk factors for near-miss events and safety incidents in pediatric radiation therapy

BACKGROUND AND PURPOSE

Factors contributing to safety- or quality-related incidents (e.g. variances) in children are unknown. We identified clinical and RT treatment variables associated with risk for variances in a pediatric cohort.

MATERIALS AND METHODS

Using our institution's incident learning system, 81 patients age ≤21 years old who experienced variances were compared to 191 pediatric patients without variances. Clinical and RT treatment variables were evaluated as potential predictors for variances using univariate and multivariate analyses.

RESULTS

Variances were primarily documentation errors (n = 46, 57%) and were most commonly detected during treatment planning (n = 14, 21%). Treatment planning errors constituted the majority (n = 16 out of 29, 55%) of near-misses and safety incidents (NMSI), which excludes workflow incidents. Therapists reported the majority of variances (n = 50, 62%). Physician cross-coverage (OR = 2.1, 95% CI = 1.04-4.38) and 3D conformal RT (OR = 2.3, 95% CI = 1.11-4.69) increased variance risk. Conversely, age >14 years (OR = 0.5, 95% CI = 0.28-0.88) and diagnosis of abdominal tumor (OR = 0.2, 95% CI = 0.04-0.59) decreased variance risk.

CONCLUSIONS

Variances in children occurred in early treatment phases, but were detected at later workflow stages. Quality measures should be implemented during early treatment phases with a focus on younger children and those cared for by cross-covering physicians.