1
|
Pashtan IM, Kosak T, Shin KY, Molodowitch C, Killoran JH, Hancox C, Czerminska M, Bredfeldt JS, Cail DW, Kearney M, Tishler RB, Mak RH. An Automated, Dynamic Radiation Oncology Prescription Checking System. Pract Radiat Oncol 2024; 14:343-352. [PMID: 38151183 DOI: 10.1016/j.prro.2023.12.002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/19/2023] [Revised: 10/29/2023] [Accepted: 12/01/2023] [Indexed: 12/29/2023]
Abstract
PURPOSE Despite serving as a critical communication tool, radiation oncology prescriptions are entered manually and prone to error. An automated prescription checking system was developed and implemented to help address this problem. METHODS AND MATERIALS Rules defining clinically appropriate prescriptions were generated, examining specific types of errors: (1) unapproved dose per fraction for a given disease site; (2) dose per fraction too large for nonstereotactic treatment technique; and (3) dose per fraction too low. With a goal of catching errors as upstream as possible to minimize their propagation, a report was created and ran every 30 minutes to check all newly written or approved prescriptions against the 3 rules. When a prescription violated these rules, an automated email was immediately sent to the prescriber alerting them of the potential error. System performance was continuously monitored and the criteria triggering an alert adjusted to balance error detection against false positives. Alerts leading to prescription amendment were considered true errors. RESULTS From June 2021 to November 2022, the system checked 24,047 prescriptions. A total of 241 email alerts were triggered, for an average alert rate of 1%. Of the 241 alerts, 198 (82.2%) were unapproved doses per fraction for the disease site, 14 (5.8%) were doses per fraction that were too low, and 29 (12%) were doses too large for nonstereotactic treatment technique. Thirty-one percent of alerts led to a change of prescription, suggesting they were true errors. The baseline rate of erroneous prescription entry was 0.3%. A regression model showed that trainee prescription entry and dose per fraction <150 cGy were significantly associated with true errors. CONCLUSIONS Given the significant consequences of erroneous prescription entry, which ranged from wasted resources and treatment delays to potentially serious misadministration, there is significant value in implementing automated prescription checking systems in radiation oncology clinics.
Collapse
Affiliation(s)
- I M Pashtan
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, Massachusetts.
| | - T Kosak
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, Massachusetts
| | - K-Y Shin
- Department of Data Science, Dana-Farber Cancer Institute, Boston, Massachusetts
| | - C Molodowitch
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, Massachusetts
| | - J H Killoran
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, Massachusetts
| | - C Hancox
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, Massachusetts
| | - M Czerminska
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, Massachusetts
| | - J S Bredfeldt
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, Massachusetts
| | - D W Cail
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, Massachusetts
| | - M Kearney
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, Massachusetts
| | - R B Tishler
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, Massachusetts
| | - R H Mak
- Department of Radiation Oncology, Brigham and Women's Hospital/Dana-Farber Cancer Institute, Boston, Massachusetts
| |
Collapse
|
2
|
Eguale T, Bastardot F, Song W, Motta-Calderon D, Elsobky Y, Rui A, Marceau M, Davis C, Ganesan S, Alsubai A, Matthews M, Volk LA, Bates DW, Rozenblum R. A Machine Learning Application to Classify Patients at Differing Levels of Risk of Opioid Use Disorder: Clinician-Based Validation Study. JMIR Med Inform 2024; 12:e53625. [PMID: 38842167 PMCID: PMC11185289 DOI: 10.2196/53625] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/20/2023] [Revised: 03/15/2024] [Accepted: 04/20/2024] [Indexed: 06/07/2024] Open
Abstract
Background Despite restrictive opioid management guidelines, opioid use disorder (OUD) remains a major public health concern. Machine learning (ML) offers a promising avenue for identifying and alerting clinicians about OUD, thus supporting better clinical decision-making regarding treatment. Objective This study aimed to assess the clinical validity of an ML application designed to identify and alert clinicians of different levels of OUD risk by comparing it to a structured review of medical records by clinicians. Methods The ML application generated OUD risk alerts on outpatient data for 649,504 patients from 2 medical centers between 2010 and 2013. A random sample of 60 patients was selected from 3 OUD risk level categories (n=180). An OUD risk classification scheme and standardized data extraction tool were developed to evaluate the validity of the alerts. Clinicians independently conducted a systematic and structured review of medical records and reached a consensus on a patient's OUD risk level, which was then compared to the ML application's risk assignments. Results A total of 78,587 patients without cancer with at least 1 opioid prescription were identified as follows: not high risk (n=50,405, 64.1%), high risk (n=16,636, 21.2%), and suspected OUD or OUD (n=11,546, 14.7%). The sample of 180 patients was representative of the total population in terms of age, sex, and race. The interrater reliability between the ML application and clinicians had a weighted kappa coefficient of 0.62 (95% CI 0.53-0.71), indicating good agreement. Combining the high risk and suspected OUD or OUD categories and using the review of medical records as a gold standard, the ML application had a corrected sensitivity of 56.6% (95% CI 48.7%-64.5%) and a corrected specificity of 94.2% (95% CI 90.3%-98.1%). The positive and negative predictive values were 93.3% (95% CI 88.2%-96.3%) and 60.0% (95% CI 50.4%-68.9%), respectively. Key themes for disagreements between the ML application and clinician reviews were identified. Conclusions A systematic comparison was conducted between an ML application and clinicians for identifying OUD risk. The ML application generated clinically valid and useful alerts about patients' different OUD risk levels. ML applications hold promise for identifying patients at differing levels of OUD risk and will likely complement traditional rule-based approaches to generating alerts about opioid safety issues.
Collapse
Affiliation(s)
- Tewodros Eguale
- School of Pharmacy, Massachusetts College of Pharmacy and Health Sciences, Boston, MA, United States
- Division of General Internal Medicine, Brigham and Women's Hospital, Boston, MA, United States
| | - François Bastardot
- Innovation and Clinical Research Directorate, Lausanne University Hospital (CHUV), Lausanne, Switzerland
- Medical Directorate, Lausanne University Hospital (CHUV), Lausanne, Switzerland
| | - Wenyu Song
- Division of General Internal Medicine, Brigham and Women's Hospital, Boston, MA, United States
- Harvard Medical School, Boston, MA, United States
| | | | - Yasmin Elsobky
- Division of General Internal Medicine, Brigham and Women's Hospital, Boston, MA, United States
- Alexandria University, Alexandria, Egypt
| | - Angela Rui
- Division of General Internal Medicine, Brigham and Women's Hospital, Boston, MA, United States
| | - Marlika Marceau
- Clinical Quality and IS Analysis, Mass General Brigham, Somerville, MA, United States
| | - Clark Davis
- Division of General Internal Medicine, Brigham and Women's Hospital, Boston, MA, United States
| | - Sandya Ganesan
- Division of General Internal Medicine, Brigham and Women's Hospital, Boston, MA, United States
| | - Ava Alsubai
- Division of General Internal Medicine, Brigham and Women's Hospital, Boston, MA, United States
| | - Michele Matthews
- School of Pharmacy, Massachusetts College of Pharmacy and Health Sciences, Boston, MA, United States
- Department of Pharmacy, Brigham and Women's Hospital, Boston, MA, United States
| | - Lynn A Volk
- Clinical Quality and IS Analysis, Mass General Brigham, Somerville, MA, United States
| | - David W Bates
- Division of General Internal Medicine, Brigham and Women's Hospital, Boston, MA, United States
- Harvard Medical School, Boston, MA, United States
- Clinical Quality and IS Analysis, Mass General Brigham, Somerville, MA, United States
- Harvard TH Chan School of Public Health, Boston, MA, United States
| | - Ronen Rozenblum
- Division of General Internal Medicine, Brigham and Women's Hospital, Boston, MA, United States
- Harvard Medical School, Boston, MA, United States
| |
Collapse
|
3
|
Benavente S, Giraldo A, Seoane A, Ramos M, Vergés R. Clinical effects of re-evaluating a lung SBRT failure mode and effects analysis in a radiotherapy department. Clin Transl Oncol 2024:10.1007/s12094-024-03539-9. [PMID: 38831192 DOI: 10.1007/s12094-024-03539-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2024] [Accepted: 05/24/2024] [Indexed: 06/05/2024]
Abstract
PURPOSE The increasing complexity of radiation treatments can hinder its clinical success. This study aimed to better understand evolving risks by re-evaluating a Failure Mode and Effects Analysis (FMEA) in lung SBRT. METHODS An experienced multidisciplinary team conducted an FMEA and made a reassessment 3 years later. A process map was developed with potential failure modes (FMs) identified. High-risk FMs and their possible causes and corrective actions were determined. The initial FMEA analysis was compared to gain a deeper perspective. RESULTS We identified 232 FMs. The high-risk processes were plan approval, target contouring, and patient evaluation. The corrective measures were based on stricter standardization of plan approval, pre-planning peer review, and a supporting pretreatment checklist, which substantially reduced the risk priority number in the revised FMEA. In the FMEA reassessment, we observed that the increased complexity and number of patients receiving lung SBRT conditioned a more substantial presence of human factors and communication errors as causal conditions and a potential wrong dose as a final effect. CONCLUSIONS Conducting a lung SBRT FMEA analysis has identified high-risk conditions that have been effectively mitigated in an FMEA reanalysis. Plan approval has shown to be a weak link in the process. The increasing complexity of treatments and patient numbers have shifted causal factors toward human failure and communication errors. The potential of a wrong dose as a final effect augments in this scenario. We propose that digital and artificial intelligence options are needed to mitigate potential errors in high-complexity and high-risk RT scenarios.
Collapse
Affiliation(s)
- Sergi Benavente
- Department of Radiation Oncology, Vall d'Hebron University Hospital Campus, Barcelona, Spain.
| | - Alexandra Giraldo
- Department of Radiation Oncology, Vall d'Hebron University Hospital Campus, Barcelona, Spain
| | - Alejandro Seoane
- Department of Medical Physics and Radiation Protection, Vall d'Hebron University Hospital Campus, Barcelona, Spain
| | - Mónica Ramos
- Department of Radiation Oncology, Vall d'Hebron University Hospital Campus, Barcelona, Spain
| | - Ramona Vergés
- Department of Radiation Oncology, Vall d'Hebron University Hospital Campus, Barcelona, Spain
| |
Collapse
|
4
|
Liu S, Chapman KL, Berry SL, Bertini J, Ma R, Fu Y, Yang D, Moran JM, Della-Biancia C. Implementation of a knowledge-based decision support system for treatment plan auditing through automation. Med Phys 2023; 50:6978-6989. [PMID: 37211898 DOI: 10.1002/mp.16472] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2023] [Revised: 04/19/2023] [Accepted: 04/27/2023] [Indexed: 05/23/2023] Open
Abstract
BACKGROUND Independent auditing is a necessary component of a comprehensive quality assurance (QA) program and can also be utilized for continuous quality improvement (QI) in various radiotherapy processes. Two senior physicists at our institution have been performing a time intensive manual audit of cross-campus treatment plans annually, with the aim of further standardizing our planning procedures, updating policies and guidelines, and providing training opportunities of all staff members. PURPOSE A knowledge-based automated anomaly-detection algorithm to provide decision support and strengthen our manual retrospective plan auditing process was developed. This standardized and improved the efficiency of the assessment of our external beam radiotherapy (EBRT) treatment planning across all eight campuses of our institution. METHODS A total of 843 external beam radiotherapy plans for 721 lung patients from January 2020 to March 2021 were automatically acquired from our clinical treatment planning and management systems. From each plan, 44 parameters were automatically extracted and pre-processed. A knowledge-based anomaly detection algorithm, namely, "isolation forest" (iForest), was then applied to the plan dataset. An anomaly score was determined for each plan using recursive partitioning mechanism. Top 20 plans ranked with the highest anomaly scores for each treatment technique (2D/3D/IMRT/VMAT/SBRT) including auto-populated parameters were used to guide the manual auditing process and validated by two plan auditors. RESULTS The two auditors verified that 75.6% plans with the highest iForest anomaly scores have similar concerning qualities that may lead to actionable recommendations for our planning procedures and staff training materials. The time to audit a chart was approximately 20.8 min on average when done manually and 14.0 min when done with the iForest guidance. Approximately 6.8 min were saved per chart with the iForest method. For our typical internal audit review of 250 charts annually, the total time savings are approximately 30 hr per year. CONCLUSION iForest effectively detects anomalous plans and strengthens our cross-campus manual plan auditing procedure by adding decision support and further improve standardization. Due to the use of automation, this method was efficient and will be used to establish a standard plan auditing procedure, which could occur more frequently.
Collapse
Affiliation(s)
- Shi Liu
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, New York, USA
| | - Katherine L Chapman
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, New York, USA
| | - Sean L Berry
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, New York, USA
| | - Julian Bertini
- Committee on Medical Physics, Biological Science Division, University of Chicago, Chicago, Illinois, USA
| | - Rongtao Ma
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, New York, USA
| | - Yabo Fu
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, New York, USA
| | - Deshan Yang
- Department of Radiation Oncology, Duke Cancer Institute, Durham, North Carolina, USA
| | - Jean M Moran
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, New York, USA
| | - Cesar Della-Biancia
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, New York, USA
| |
Collapse
|
5
|
Tools for large-scale data analytics of an international multi-center study in radiation oncology for cervical cancer. Radiother Oncol 2023; 182:109524. [PMID: 36764459 DOI: 10.1016/j.radonc.2023.109524] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/18/2022] [Revised: 02/01/2023] [Accepted: 02/02/2023] [Indexed: 02/11/2023]
Abstract
PURPOSE To develop and implement a software that enables centers, treating patients with state-of-the-art radiation oncology, to compare their patient, treatment, and outcome data to a reference cohort, and to assess the quality of their treatment approach. MATERIALS AND METHODS A comprehensive data dashboard was designed, which al- lowed holistic assessment of institutional treatment approaches. The software was tested in the ongoing EMBRACE-II study for locally advanced cervical cancer. The tool created individualized dashboards and automatic analysis scripts, verified pro- tocol compliance and checked data for inconsistencies. Identified quality assurance (QA) events were analysed. A survey among users was conducted to assess usability. RESULTS The survey indicated favourable feedback to the prototype and highlighted its value for internal monitoring. Overall, 2302 QA events were identified (0.4% of all collected data). 54% were due to missing or incomplete data, and 46% originated from other causes. At least one QA event was found in 519/1001 (52%) of patients. QA events related to primary study endpoints were found in 16% of patients. Sta- tistical methods demonstrated good performance in detecting anomalies, with precisions ranging from 71% to 100%. Most frequent QA event categories were Treatment Technique (27%), Patient Characteristics (22%), Dose Reporting (17%), Outcome 156 (15%), Outliers (12%), and RT Structures (8%). CONCLUSION A software tool was developed and tested within a clinical trial in radia- tion oncology. It enabled the quantitative and qualitative comparison of institutional patient and treatment parameters with a large multi-center reference cohort. We demonstrated the value of using statistical methods to automatically detect implau- sible data points and highlighted common pitfalls and uncertainties in radiotherapy for cervical cancer.
Collapse
|