Establishment of practice standards in nomenclature and prescription to enable construction of software and databases for knowledge-based practice review

Towards a safe and efficient clinical implementation of machine learning in radiation oncology by exploring model interpretability, explainability and data-model dependency

The interest in machine learning (ML) has grown tremendously in recent years, partly due to the performance leap that occurred with new techniques of deep learning, convolutional neural networks for images, increased computational power, and wider availability of large datasets. Most fields of medicine follow that popular trend and, notably, radiation oncology is one of those that are at the forefront, with already a long tradition in using digital images and fully computerized workflows. ML models are driven by data, and in contrast with many statistical or physical models, they can be very large and complex, with countless generic parameters. This inevitably raises two questions, namely, the tight dependence between the models and the datasets that feed them, and the interpretability of the models, which scales with its complexity. Any problems in the data used to train the model will be later reflected in their performance. This, together with the low interpretability of ML models, makes their implementation into the clinical workflow particularly difficult. Building tools for risk assessment and quality assurance of ML models must involve then two main points: interpretability and data-model dependency. After a joint introduction of both radiation oncology and ML, this paper reviews the main risks and current solutions when applying the latter to workflows in the former. Risks associated with data and models, as well as their interaction, are detailed. Next, the core concepts of interpretability, explainability, and data-model dependency are formally defined and illustrated with examples. Afterwards, a broad discussion goes through key applications of ML in workflows of radiation oncology as well as vendors' perspectives for the clinical implementation of ML.

Development and Validation of a Comprehensive Multivariate Dosimetric Model for Predicting Late Genitourinary Toxicity Following Prostate Cancer Stereotactic Body Radiotherapy

Purpose: Dosimetric predictors of toxicity after Stereotactic Body Radiation Therapy (SBRT) are not well-established. We sought to develop a multivariate model that predicts Common Terminology Criteria for Adverse Events (CTCAE) late grade 2 or greater genitourinary (GU) toxicity by interrogating the entire dose-volume histogram (DVH) from a large cohort of prostate cancer patients treated with SBRT on prospective trials.

Methods: Three hundred and thirty-nine patients with late CTCAE toxicity data treated with prostate SBRT were identified and analyzed. All patients received 40 Gy in five fractions, every other day, using volumetric modulated arc therapy. For each patient, we examined 910 candidate dosimetric features including maximum dose, volumes of each organ [CTV, organs at risk (OARs)], V100%, and other granular volumetric/dosimetric indices at varying volumetric/dosimetric values from the entire DVH as well as ADT use to model and predict toxicity from SBRT. Training and validation subsets were generated with 90 and 10% of the patients in our cohort, respectively. Predictive accuracy was assessed by calculating the area under the receiver operating curve (AROC). Univariate analysis with student t-test was first performed on each candidate DVH feature. We subsequently performed advanced machine-learning multivariate analyses including classification and regression tree (CART), random forest, boosted tree, and multilayer neural network.

Results: Median follow-up time was 32.3 months (range 3–98.9 months). Late grade ≥2 GU toxicity occurred in 20.1% of patients in our series. No single dosimetric parameter had an AROC for predicting late grade ≥2 GU toxicity on univariate analysis that exceeded 0.599. Optimized CART modestly improved prediction accuracy, with an AROC of 0.601, whereas other machine learning approaches did not improve upon univariate analyses.

Conclusions: CART-based machine learning multivariate analyses drawing from 910 dosimetric features and ADT use modestly improves upon clinical prediction of late GU toxicity alone, yielding an AROC of 0.601. Biologic predictors may enhance predictive models for identifying patients at risk for late toxicity after SBRT.

Interinstitutional Plan Quality Assessment of 2 Linac-Based, Single-Isocenter, Multiple Metastasis Radiosurgery Techniques

Purpose

Interest and application of stereotactic radiosurgery for multiple brain metastases continue to increase. Various planning systems are available for linear accelerator (linac)–based single-isocenter multiple metastasis radiosurgery. Two of the most advanced systems are BrainLAB Multiple Metastases Elements (MME), a dynamic conformal arc (DCA) approach, and Varian RapidArc (RA), a volumetric modulated arc therapy (VMAT) approach. In this work, we systematically compared plan quality between the 2 techniques.

Methods and Materials

Thirty patients with 4 to 10 metastases (217 total; median 7.5; V_min = 0.014 cm³; V_max = 17.73 cm³) were planned with both Varian RA and MME at 2 different institutions with extensive experience in each respective technique. All plans had a single isocenter and used Varian linac equipped with high-definition multileaf collimator. RA plans used 2 to 4 noncoplanar VMAT arcs with 10 MV flattening filter-free beam. MME plans used 4 to 9 noncoplanar DCAs and 6 MV flattening filter-free beam, (minimum planning target volume [PTV_min] = 0.49 cm³; PTV_max = 27.32 cm³; PTV_median = 7.05 cm³). Prescriptions were 14 to 24 Gy in a single fraction. Target coverage goal was 99% of volume receiving prescription dose (D99% ≥ 100%). Plans were evaluated by Radiation Therapy Oncology Group/Paddick conformity index (CI) score, 12 Gy volume (V_12Gy), V_8Gy, V_5Gy, mean brain dose (D_mean), and beam-on time.

Results

Conformity was favorable among RA plans (median: MME CI^RTOG = 1.38; RA CI^RTOG = 1.21; P < .0001). V_12Gy and V_8Gy were lower for RA plans (median: MME V12 = 23.7 cm³; RA V12 = 19.2 cm³; P = .0001; median: MME V_8Gy = 53.6 cm³; RA V_8Gy = 44.1 cm³; P = .024). V_5Gy was lower for MME plans (median: MME V_5Gy = 141.4 cm³; RA V_5Gy = 142.8 cm³; P = .009). Mean brain was lower for MME plans (median: MME D_mean = 2.57 Gy; RA D_mean = 2.76 Gy; P < .0001).

Conclusions

For linac-based multiple metastasis stereotactic radiosurgery, RapidArc VMAT facilitates favorable conformity and V_12Gy/V_8Gy volume compared with the MME DCA plan. MME planning facilitates reduced dose spill at levels ≤V_5Gy.

Retrospective analysis of breast radiotherapy treatment plans: Curating the 'non-curated'

INTRODUCTION

This paper provides a demonstration of how non-curated data can be retrospectively cleaned, so that existing repositories of radiotherapy treatment planning data can be used to complete bulk retrospective analyses of dosimetric trends and other plan characteristics.

METHODS

A non curated archive of 1137 radiotherapy treatment plans accumulated over a 12-month period, from five radiotherapy centres operated by one institution, was used to investigate and demonstrate a process of clinical data cleansing, by: identifying and translating inconsistent structure names; correcting inconsistent lung contouring; excluding plans for treatments other than breast tangents and plans without identifiable PTV, lung and heart structures; and identifying but not excluding plans that deviated from the local planning protocol. PTV, heart and lung dose-volume metrics were evaluated, in addition to a sample of personnel and linac load indicators.

RESULTS

Data cleansing reduced the number of treatment plans in the sample by 35.7%. Inconsistent structure names were successfully identified and translated (e.g. 35 different names for lung). Automatically separating whole lung structures into left and right lung structures allowed the effect of contralateral and ipsilateral lung dose to be evaluated, while introducing some small uncertainties, compared to manual contouring. PTV doses were indicative of prescription doses. Breast treatment work was unevenly distributed between oncologists and between metropolitan and regional centres.

CONCLUSION

This paper exemplifies the data cleansing and data analysis steps that may be completed using existing treatment planning data, to provide individual radiation oncology departments with access to information on their own patient populations. Clearly, the well-planned and systematic recording of new, high quality data is the preferred solution, but the retrospective curation of non-curated data may be a useful interim solution, for radiation oncology departments where the systems for recording of new data have yet to be designed and agreed.

Big Data Readiness in Radiation Oncology: An Efficient Approach for Relabeling Radiation Therapy Structures With Their TG-263 Standard Name in Real-World Data Sets

Purpose

To prepare for big data analyses on radiation therapy data, we developed Stature, a tool-supported approach for standardization of structure names in existing radiation therapy plans. We applied the widely endorsed nomenclature standard TG-263 as the mapping target and quantified the structure name inconsistency in 2 real-world data sets.

Methods and Materials

The clinically relevant structures in the radiation therapy plans were identified by reference to randomized controlled trials. The Stature approach was used by clinicians to identify the synonyms for each relevant structure, which was then mapped to the corresponding TG-263 name. We applied Stature to standardize the structure names for 654 patients with prostate cancer (PCa) and 224 patients with head and neck squamous cell carcinoma (HNSCC) who received curative radiation therapy at our institution between 2007 and 2017. The accuracy of the Stature process was manually validated in a random sample from each cohort. For the HNSCC cohort we measured the resource requirements for Stature, and for the PCa cohort we demonstrated its impact on an example clinical analytics scenario.

Results

All but 1 synonym group (“Hydrogel”) was mapped to the corresponding TG-263 name, resulting in a TG-263 relabel rate of 99% (8837 of 8925 structures). For the PCa cohort, Stature matched a total of 5969 structures. Of these, 5682 structures were exact matches (ie, following local naming convention), 284 were matched via a synonym, and 3 required manual matching. This original radiation therapy structure names therefore had a naming inconsistency rate of 4.81%. For the HNSCC cohort, Stature mapped a total of 2956 structures (2638 exact, 304 synonym, 14 manual; 10.76% inconsistency rate) and required 7.5 clinician hours. The clinician hours required were one-fifth of those that would be required for manual relabeling. The accuracy of Stature was 99.97% (PCa) and 99.61% (HNSCC).

Conclusions

The Stature approach was highly accurate and had significant resource efficiencies compared with manual curation.

Treatment data and technical process challenges for practical big data efforts in radiation oncology

The term Big Data has come to encompass a number of concepts and uses within medicine. This paper lays out the relevance and application of large collections of data in the radiation oncology community. We describe the potential importance and uses in clinical practice. The important concepts are then described and how they have been or could be implemented are discussed. Impediments to progress in the collection and use of sufficient quantities of data are also described. Finally, recommendations for how the community can move forward to achieve the potential of big data in radiation oncology are provided.

Machine Learning Applications in Head and Neck Radiation Oncology: Lessons From Open-Source Radiomics Challenges

Radiomics leverages existing image datasets to provide non-visible data extraction via image post-processing, with the aim of identifying prognostic, and predictive imaging features at a sub-region of interest level. However, the application of radiomics is hampered by several challenges such as lack of image acquisition/analysis method standardization, impeding generalizability. As of yet, radiomics remains intriguing, but not clinically validated. We aimed to test the feasibility of a non-custom-constructed platform for disseminating existing large, standardized databases across institutions for promoting radiomics studies. Hence, University of Texas MD Anderson Cancer Center organized two public radiomics challenges in head and neck radiation oncology domain. This was done in conjunction with MICCAI 2016 satellite symposium using Kaggle-in-Class, a machine-learning and predictive analytics platform. We drew on clinical data matched to radiomics data derived from diagnostic contrast-enhanced computed tomography (CECT) images in a dataset of 315 patients with oropharyngeal cancer. Contestants were tasked to develop models for (i) classifying patients according to their human papillomavirus status, or (ii) predicting local tumor recurrence, following radiotherapy. Data were split into training, and test sets. Seventeen teams from various professional domains participated in one or both of the challenges. This review paper was based on the contestants' feedback; provided by 8 contestants only (47%). Six contestants (75%) incorporated extracted radiomics features into their predictive model building, either alone (n = 5; 62.5%), as was the case with the winner of the "HPV" challenge, or in conjunction with matched clinical attributes (n = 2; 25%). Only 23% of contestants, notably, including the winner of the "local recurrence" challenge, built their model relying solely on clinical data. In addition to the value of the integration of machine learning into clinical decision-making, our experience sheds light on challenges in sharing and directing existing datasets toward clinical applications of radiomics, including hyper-dimensionality of the clinical/imaging data attributes. Our experience may help guide researchers to create a framework for sharing and reuse of already published data that we believe will ultimately accelerate the pace of clinical applications of radiomics; both in challenge or clinical settings.

Setting the Benchmark for the Ground and Air Medical Quality in Transport International Quality Improvement Collaborative

OBJECTIVE

Critical care transport (CCT) supports regionalization of medical care. Focus on the quality of CCT care prompted the development of the Ground and Air Medical qUality in Transport (GAMUT) Quality Improvement collaborative database which tracks consensus quality metrics. The Institute of Medicine recommends benchmarking of comparative data to accelerate improvement. Herein, we report the strategies and rationale for GAMUT QI Collaborative benchmarking.

METHODS

The GAMUT database includes >350 programs internationally with >200,000 annual patient contacts. Evidence-based literature review performed in May 2016 and October 2017 identified benchmarking strategies were evaluated and summarized, specific to the GAMUT metrics. Statistical analyses include simple statistics and weighted expectation calculations for benchmark examples (Pearson chi-square with Bonferroni adjusted post-hoc z tests).

RESULTS

Evidence-based literature search yielded 70 articles, and 31 were selected for inclusion in our evidence table. 5 evidence-based benchmark strategies were considered: average (mean), average (median), adjusted benchmark (based on expected outcome), Achievable Benchmark of Care (ABC), and Delphi. ABC threshold establishes a higher target (90th percentile) forcing more programs to achieve higher performance.

CONCLUSION

Benchmarking is not well-suited for a single strategy and requires customized consideration based on each metric, though adjusted benchmark and ABC generally set higher performance benchmarks.

Data collection of patient outcomes: one institution's experience

Patient- and provider-reported outcomes are recognized as important in evaluating quality of care, guiding health care policy, comparative effectiveness research, and decision-making in radiation oncology. Combining patient and provider outcome data with a detailed description of disease and therapy is the basis for these analyses. We report on the combination of technical solutions and clinical process changes at our institution that were used in the collection and dissemination of this data. This initiative has resulted in the collection of treatment data for 23 541 patients, 20 465 patients with provider-based adverse event records, and patient-reported outcome surveys submitted by 5622 patients. All of the data is made accessible using a self-service web-based tool.

Evaluation of multiple factors affecting normal brain dose in single-isocenter multiple target radiosurgery

We investigated the effects of multiple planning factors on normal brain dose for single-isocenter VMAT stereotactic radiosurgery (SRS). Ten patients were retrospectively planned using a standardized objective function and all 16 combinations of 2 versus 4 arcs, collimator angle 45° versus selected per beam to minimize area of normal brain exposed in the beams-eye-view, fixed jaw versus following the trailing MLC leaf, and a 2 Gy mean dose objective for healthy brain versus no low dose objective. Limiting the normal brain mean dose in the optimization objective function significantly reduced the low dose spill into the normal brain without changing target coverage. Jaw tracking and appropriate selection of collimator also reduced the low dose volume, but to a lesser extent. To reduce low dose spill into normal brain for single isocenter VMAT radiosurgery of multiple targets, it is important to incorporate a limit on low dose spill into the objective function. This study has implications beyond single-isocenter VMAT radiosurgery. When comparing different inverse-planned treatment techniques, metrics that are important for evaluation of plan quality must be included the objective function.

American Association of Physicists in Medicine Task Group 263: Standardizing Nomenclatures in Radiation Oncology

A substantial barrier to the single- and multi-institutional aggregation of data to supporting clinical trials, practice quality improvement efforts, and development of big data analytics resource systems is the lack of standardized nomenclatures for expressing dosimetric data. To address this issue, the American Association of Physicists in Medicine (AAPM) Task Group 263 was charged with providing nomenclature guidelines and values in radiation oncology for use in clinical trials, data-pooling initiatives, population-based studies, and routine clinical care by standardizing: (1) structure names across image processing and treatment planning system platforms; (2) nomenclature for dosimetric data (eg, dose-volume histogram [DVH]-based metrics); (3) templates for clinical trial groups and users of an initial subset of software platforms to facilitate adoption of the standards; (4) formalism for nomenclature schema, which can accommodate the addition of other structures defined in the future. A multisociety, multidisciplinary, multinational group of 57 members representing stake holders ranging from large academic centers to community clinics and vendors was assembled, including physicists, physicians, dosimetrists, and vendors. The stakeholder groups represented in the membership included the AAPM, American Society for Radiation Oncology (ASTRO), NRG Oncology, European Society for Radiation Oncology (ESTRO), Radiation Therapy Oncology Group (RTOG), Children's Oncology Group (COG), Integrating Healthcare Enterprise in Radiation Oncology (IHE-RO), and Digital Imaging and Communications in Medicine working group (DICOM WG); A nomenclature system for target and organ at risk volumes and DVH nomenclature was developed and piloted to demonstrate viability across a range of clinics and within the framework of clinical trials. The final report was approved by AAPM in October 2017. The approval process included review by 8 AAPM committees, with additional review by ASTRO, European Society for Radiation Oncology (ESTRO), and American Association of Medical Dosimetrists (AAMD). This Executive Summary of the report highlights the key recommendations for clinical practice, research, and trials.

Incorporating big data into treatment plan evaluation: Development of statistical DVH metrics and visualization dashboards

Purpose

To develop statistical dose-volume histogram (DVH)–based metrics and a visualization method to quantify the comparison of treatment plans with historical experience and among different institutions.

Methods and materials

The descriptive statistical summary (ie, median, first and third quartiles, and 95% confidence intervals) of volume-normalized DVH curve sets of past experiences was visualized through the creation of statistical DVH plots. Detailed distribution parameters were calculated and stored in JavaScript Object Notation files to facilitate management, including transfer and potential multi-institutional comparisons. In the treatment plan evaluation, structure DVH curves were scored against computed statistical DVHs and weighted experience scores (WESs). Individual, clinically used, DVH-based metrics were integrated into a generalized evaluation metric (GEM) as a priority-weighted sum of normalized incomplete gamma functions. Historical treatment plans for 351 patients with head and neck cancer, 104 with prostate cancer who were treated with conventional fractionation, and 94 with liver cancer who were treated with stereotactic body radiation therapy were analyzed to demonstrate the usage of statistical DVH, WES, and GEM in a plan evaluation. A shareable dashboard plugin was created to display statistical DVHs and integrate GEM and WES scores into a clinical plan evaluation within the treatment planning system. Benchmarking with normal tissue complication probability scores was carried out to compare the behavior of GEM and WES scores.

Results

DVH curves from historical treatment plans were characterized and presented, with difficult-to-spare structures (ie, frequently compromised organs at risk) identified. Quantitative evaluations by GEM and/or WES compared favorably with the normal tissue complication probability Lyman-Kutcher-Burman model, transforming a set of discrete threshold-priority limits into a continuous model reflecting physician objectives and historical experience.

Conclusions

Statistical DVH offers an easy-to-read, detailed, and comprehensive way to visualize the quantitative comparison with historical experiences and among institutions. WES and GEM metrics offer a flexible means of incorporating discrete threshold-prioritizations and historic context into a set of standardized scoring metrics. Together, they provide a practical approach for incorporating big data into clinical practice for treatment plan evaluations.

Dosimetric analysis of liver toxicity after liver metastasis stereotactic body radiation therapy

PURPOSE

The aim of this study is to describe the incidence and type of liver toxicity seen following liver metastases stereotactic body radiation therapy (SBRT) and the corresponding clinical and dosimetric factors associated with toxicity.

METHODS AND MATERIALS

Between 2002 and 2009, 81 evaluable patients with liver metastases were treated on 2 prospective studies assessing SBRT, with prescription doses based on the effective liver volume irradiated evaluated. Toxicity was defined as grade ≥2 classic or nonclassic radiation induced liver disease (RILD). Specific toxicity endpoints evaluated were worsening transaminases and albumin levels within 3 months of SBRT.

RESULTS

Seventy percent of patients had colorectal carcinoma, 55% had extrahepatic disease, 1 patient had hepatitis B, and 54% had received prior chemotherapy. Baseline transaminases were elevated at Common Terminology Criteria for Adverse Effects, V4.0, grade 1, 2, and 3 levels in 33 (41%), 2 (2%), and 0 (0%) patients. The mean prescription dose was 43 Gy (27.7-60 Gy) in 6 fractions. The mean liver (minus gross tumor volume) dose (MLD) was 16 Gy (3-25.6 Gy) in 6 fractions. No classic or nonclassical ≥grade 2 RILD was observed. Within 3 months of SBRT, 49 (61%) patients had worsening of grade of transaminase and 23 (28%) patients had a reduction in albumin, all transient (majority grade ≤2 toxicity) without subsequent clinical toxicity. Seventeen patients exceeded Quantitative Analysis of Normal Tissue Effects in the Clinic MLD guidelines (≤20 Gy), 13 (76%) of whom had worsening of transaminase grade. On multivariate analysis, worsening of liver enzymes was more likely in patients with higher doses to the spared 700 mL of liver (P = .026), and reduction of albumin was more likely with higher effective liver volume (odds ratio, 1.53 [range, 1.08-2.16]) P = .016).

CONCLUSIONS

Liver metastases SBRT is safe with a low risk of transient biochemical liver toxicity, more likely in patients with a higher effective liver volume and higher doses to the spared uninvolved liver volume.

The big data effort in radiation oncology: Data mining or data farming?

Although large volumes of information are entered into our electronic health care records, radiation oncology information systems and treatment planning systems on a daily basis, the goal of extracting and using this big data has been slow to emerge. Development of strategies to meet this goal is aided by examining issues with a data farming instead of a data mining conceptualization. Using this model, a vision of key data elements, clinical process changes, technology issues and solutions, and role for professional societies is presented. With a better view of technology, process and standardization factors, definition and prioritization of efforts can be more effectively directed.

Standardizing dose prescriptions: An ASTRO white paper

This white paper recommends the standardization (content and presentation order) of several "key components" of the radiation therapy prescription to facilitate accurate communication between radiation therapy care providers. The rationale, other similar efforts, and detailed considerations are described. In brief, the Task Force recommends that the prescription's "elements" include: treatment site, method of delivery, dose per fraction, total number of fractions, total dose (eg, right breast, tangent photons, 267 cGy * 16 = 4272 cGy). A similar formalism is recommended for brachytherapy (eg, cervix, Ir-192 brachytherapy, 600cGy * 5 = 3000 cGy) and other modalities. The white paper also considers future directions for other items such as the simulation order, treatment planning objectives, prescription point or volume, treatment schedule, localization imaging, laboratory monitoring, concurrent chemotherapy, patient instructions for treatment, etc. The intent of this white paper is to facilitate accurate communication among providers to support safe practice as well as to guide vendors in product development that is consistent with this standard prescription.