Survey of Resident Education in Intensity-Modulated Radiation Therapy

Prospective assessment of an atlas-based intervention combined with real-time software feedback in contouring lymph node levels and organs-at-risk in the head and neck: Quantitative assessment of conformance to expert delineation

PURPOSE

A number of studies have previously assessed the role of teaching interventions to improve organ-at-risk (OAR) delineation. We present a preliminary study demonstrating the benefit of a combined atlas and real time software-based feedback intervention to aid in contouring of OARs in the head and neck.

METHODS AND MATERIALS

The study consisted of a baseline evaluation, a real-time feedback intervention, atlas presentation, and a follow-up evaluation. At baseline evaluation, 8 resident observers contoured 26 OARs on a computed tomography scan without intervention or aid. They then received feedback comparing their contours both statistically and graphically to a set of atlas-based expert contours. Additionally, they received access to an atlas to contour these structures. The resident observers were then asked to contour the same 26 OARs on a separate computed tomography scan with atlas access. In addition, 6 experts (5 radiation oncologists specializing in the head and neck, and 1 neuroradiologist) contoured the 26 OARs on both scans. A simultaneous truth and performance level estimation (STAPLE) composite of the expert contours was used as a gold-standard set for analysis of OAR contouring.

RESULTS

Of the 8 resident observers who initially participated in the study, 7 completed both phases of the study. Dice similarity coefficients were calculated for each user-drawn structure relative to the expert STAPLE composite for each structure. Mean dice similarity coefficients across all structures increased between phase 1 and phase 2 for each resident observer, demonstrating a statistically significant improvement in overall OAR-contouring ability (P < .01). Additionally, intervention improved contouring in 16/26 delineated organs-at-risk across resident observers at a statistically significant level (P ≤ .05) including all otic structures and suprahyoid lymph node levels of the head and neck.

CONCLUSIONS

Our data suggest that a combined atlas and real-time feedback-based educational intervention detectably improves contouring of OARs in the head and neck.

Assessment and improvement of radiation oncology trainee contouring ability utilizing consensus-based penalty metrics

INTRODUCTION

The objective of this study was to develop and assess the feasibility of utilizing consensus-based penalty metrics for the purpose of critical structure and organ at risk (OAR) contouring quality assurance and improvement.

METHODS

A Delphi study was conducted to obtain consensus on contouring penalty metrics to assess trainee-generated OAR contours. Voxel-based penalty metric equations were used to score regions of discordance between trainee and expert contour sets. The utility of these penalty metric scores for objective feedback on contouring quality was assessed by using cases prepared for weekly radiation oncology radiation oncology trainee treatment planning rounds.

RESULTS

In two Delphi rounds, six radiation oncology specialists reached agreement on clinical importance/impact and organ radiosensitivity as the two primary criteria for the creation of the Critical Structure Inter-comparison of Segmentation (CriSIS) penalty functions. Linear/quadratic penalty scoring functions (for over- and under-contouring) with one of four levels of severity (none, low, moderate and high) were assigned for each of 20 OARs in order to generate a CriSIS score when new OAR contours are compared with reference/expert standards. Six cases (central nervous system, head and neck, gastrointestinal, genitourinary, gynaecological and thoracic) then were used to validate 18 OAR metrics through comparison of trainee and expert contour sets using the consensus derived CriSIS functions. For 14 OARs, there was an improvement in CriSIS score post-educational intervention.

CONCLUSIONS

The use of consensus-based contouring penalty metrics to provide quantitative information for contouring improvement is feasible.

Results and resident evaluation of the 2007 American College of Radiology in-training examination in radiation oncology

PURPOSE

To report the results, demographic data, and resident evaluation of the 2007 ACR radiation oncology in-training (TXIT) examination.

METHODS

The 2007 TXIT examination consisted of 360 multiple-choice questions covering 13 different subject areas. It included 9 demographic questions and 7 evaluation items. Five hundred seventy-two residents from 85 institutions took the 2007 TXIT examination.

RESULTS

The median raw score was 218.3 +/- 29 (range, 140-295). The mean item difficulty was 60.7%, with mean item discrimination of 0.19. The reliability coefficient was 0.92. Of the respondents to the demographic questions, 550 (96.2%), 493 (86.2%), and 402 (70.3%) residents had formal physics, biology, and clinical oncology instruction, respectively. Clinical experience was highest in breast, genitourinary, and lung cancers and lowest in soft-tissue sarcomas, pediatric tumors, and orbital neoplasms. Formal instruction in cancer biology, intensity-modulated radiation therapy, and statistics was reported by 416 (72.7%), 383 (67.0%), and 245 (42.8%) residents, respectively. On the basis of residents' responses to the evaluation questions, 31 (5.4%) did not know that the examination book was available to them after the test, 47 (8.2%) did not know that an answer key was available, and 79 (13.8%) residents did not know that rationales were available for future study.

CONCLUSIONS

The reliability estimate for the ACR TXIT examination was high, typical of carefully constructed in-training examinations. The availability of the examination book, answer key, and rationales was not known by approximately 5% to 15% of residents taking the TXIT examination; therefore, a better method of informing the residents of these educational tools is needed.