Evaluation of an audit with feedback continuing medical education program for radiation oncologists

Costs of Physician Continuous Professional Development: A Systematic Review

PURPOSE

An essential yet oft-neglected step in cost evaluations is the selection of resources (ingredients) to include in cost estimates. The ingredients that most influence the cost of physician continuous professional development (CPD) are unknown, as are the relative costs of instructional modalities. This study's purpose was to estimate the costs of cost ingredients and instructional modalities in physician CPD.

METHOD

The authors conducted a systematic review in April 2020, searching MEDLINE, Embase, PsycInfo, and the Cochrane Library for comparative cost evaluations of CPD for practicing physicians. Two reviewers, working independently, screened articles for inclusion and extracted information on costs (converted to 2021 U.S. dollars) for each intervention overall, each ingredient, and each modality.

RESULTS

Of 3,338 eligible studies, 62 were included, enumerating costs for 86 discrete training interventions or instructional modalities. The most frequently reported ingredients were faculty time (25 of 86 interventions), materials (24), administrator/staff time (23), and travel (20). Ingredient costs varied widely, ranging from a per-physician median of $4 for postage (10 interventions) to $525 for learner time (13); equipment (9) and faculty time were also relatively expensive (median > $170). Among instructional modalities (≤ 11 interventions per modality), audit and feedback performed by physician learners, computer-based modules, computer-based virtual patients, in-person lectures, and experiences with real patients were relatively expensive (median > $1,000 per physician). Mailed paper materials, video clips, and audit and feedback performed by others were relatively inexpensive (median ≤ $62 per physician). Details regarding ingredient selection (10 of 62 studies), quantitation (10), and pricing (26) were reported infrequently.

CONCLUSIONS

Some ingredients, including time, are more important (i.e., contribute more to total costs) than others and should be prioritized in cost evaluations. Data on the relative costs of instructional modalities are insightful but limited. The methods and reporting of cost valuations merit improvement.

Why quality assurance is necessary in gynecologic radiation oncology

Quality assurance (QA) in radiation oncology involves all checks and processes that ensure that radiotherapy is delivered in an optimal and intended manner. QA is essential for the accurate delivery of brachytherapy and external beam radiotherapy in patients diagnosed with gynecologic malignancies. Inadequate QA can adversely impact clinical outcomes and reduce the reliability of clinical trials. This review highlights the importance of QA in gynecologic radiation oncology and explores the pertinent issues related to its implementation.

Piloting online self-audit of methadone treatment in Irish general practice: results, reflections and educational outcomes

BACKGROUND

Work based learning underpins the training and CPD of medical practitioners. Medical audit operates on two levels; individual self-assessment and professional/practice development. In Ireland, annual practice improvement audit is an essential requirement for the successful completion of continuous professional development (CPD) as determined by the regulatory body, the Irish Medical Council. All general practice (GP) doctors providing methadone maintenance treatment (MMT) in Ireland have a contractual obligation to partake in a yearly methadone practice audit. The Irish College of General Practitioners (ICGP) as national training provider is tasked to facilitate this annual audit process. The purpose of this audit is to assess the quality of care provided to patients against an agreed set of national standards, enhance learning, and promote practice improvement and reflective practice. The aim was to present an online MTP self-audit and evaluate results from a 12-month pilot among GPs providing MMT in Ireland.

METHODS

A mixed method study describing three phases (design and development, pilot/implementation and evaluation) of a new online self -audit tool was conducted. Descriptive and thematic analysis of audit and evaluation data was conducted.

RESULTS

Survey Monkey is a suitable software package for the development and hosting of an easy to use online audit for MMT providing doctors. Analysis of the audit results found that the majority of GPs scored 80% or over for the 25 identified essential criteria for MMT provision. The evaluation of the GP audit experience underscores the positive outcomes of the online self-audit in terms of improving practice systems, encouraging reflective practice, enhanced patient care and doctor commitment to continued provision of MMT in addiction clinics and in primary care.

CONCLUSIONS

Results from this audit demonstrate a high level of compliance with best practise MMT guidelines by Irish GPs providing MMT. The online self-audit process was well received and encouraged reflective practice. The audit process hinged on the individual GP's ability to review and critically analyse their professional practice, and manage change. This model of audit could be adapted and used to monitor the management of other chronic illnesses in general practice.

Assessing the Quality of a Radiation Oncology Case-Based, Peer-Review Program in an Integrated Academic and Community Cancer Center Network

Purpose:

Academic centers increasingly find a need to define a comprehensive peer-review program that can translate high-quality radiation therapy (RT) to community network sites. In this study, we describe the initial results of a quarterly quality audit program that aims to improve RT peer-review and provider educational processes across community sites.

Materials and Methods:

An electronic tool was used by community-based certified member (CM) sites to enter clinical treatment information about patients undergoing peer review. At least 10% of the patient load for each CM physician was selected for audit on a quarterly basis by expert academic faculty. Quality metrics included the review of the management plan, technical plan, and other indicators. RT was scored as being concordant or nonconcordant with institutional guidelines, national standards, or expert judgment.

Results:

A total of 719 patients were entered into the peer-review database by the first four CM sites. Of 14% of patients audited, 17% (18 of 104) were deemed nonconcordant. Nonconcordance rates were lowest in prevalent disease sites, such as breast (16%), colorectal (14%), and lung (12%), whereas rates were highest in lymphoma (50%), brain (44%), and gynecology (27%). Deficiencies included incomplete staging work-up, incorrect target and normal tissue delineation, and nonadherence to accepted dose-volume constraints.

Conclusion:

Given the high rate of nonconcordance, we recommend prospective, pre-RT peer review of all patients, and, in particular, expert review of patients that are from low-volume or complex disease sites. An integrated approach to peer review holds a promise of improving the quality, safety, and value of cancer therapy in the community setting.

A pan-Canadian survey of peer review practices in radiation oncology

PURPOSE

Peer review (PR) of treatment plans has been recognized internationally as a key component of quality care in radiation oncology programs (ROPs). We conducted a survey of Canadian ROPs to describe current PR practices and identify barriers/facilitators to PR optimization.

METHODS AND MATERIALS

A 42-item e-survey was sent to all Canadian ROPs (n = 44). Survey development was guided by expert consensus, literature review, and existing guidelines. One multidisciplinary response per ROP was requested.

RESULTS

Response rate was 100.0% (44/44). All ROPs (100.0%) reported conducting some PR and rated its importance as 7/10 or higher (10 = extremely important). One-half of ROPs (52.3%) peer-reviewed >80% of curative treatment plans. ROPs reported performing PR "always/almost always" pretreatment (38.6%) or before 25% of radiation therapy delivery (52.3%). The majority of ROPs reported recommending major plan changes in <5% of plans (88.6%) and documenting findings in the medical record (58.1%). Barriers to PR were radiation oncologist availability (34.1%) and time constraints (27.3%). Facilitators included development of PR standards (97.7%) and education/support (90.9%).

CONCLUSIONS

The ROPs perceive PR as highly important, but substantial variation in the extent, timing, and documentation of PR exists. The understanding of current PR activities, barriers, and facilitators will inform the development of initiatives to optimize PR in radiation oncology.

Impact of the introduction of weekly radiotherapy quality assurance meetings at one UK cancer centre

OBJECTIVE

The complexity of radiotherapy planning is increasing rapidly. Delivery and planning is subject to detailed quality assurance (QA) checks. The weakest link is often the oncologists' delineation of the clinical target volume (CTV). Weekly departmental meetings for radiotherapy QA (RTQA) were introduced into the Royal Wolverhampton Hospital, Wolverhampton, UK, in October 2011. This article describes the impact of this on patient care.

METHODS

CTVs for megavoltage photon radiotherapy courses for all radical, adjuvant and palliative treatments longer than five fractions (with the exception of two field tangential breast treatments not enrolled into clinical trials) were reviewed in the RTQA meeting. Audits were carried out in January 2012 (baseline) and September 2013, each over a 4-week period. Adherence to departmental contouring protocols was assessed and the number of major and minor alterations following peer review were determined.

RESULTS

There was no statistically significant difference for major alterations between the two study groups; 8 alterations in 80 patients (10%) for the baseline audit vs 3 alterations from 72 patients (4.2%) in the second audit (p = 0.17). A trend towards a reduction in alterations following peer review was observed. There has, however, been a change in practice resulting in a reduction in variation in CTV definition within our centre and greater adherence to protocols. There is increasing confidence in the quality and constancy of care delivered.

CONCLUSION

Introduction of a weekly QA meeting for target volume definition has facilitated consensus and adoption of departmental clinical guidelines within the unit.

ADVANCES IN KNOWLEDGE

The weakest areas in radiotherapy are patient selection and definition of the CTV. Engagement in high-quality RTQA is paramount. This article describes the impact of this in one UK cancer centre.

A survey of radiation treatment planning peer-review activities in a provincial radiation oncology programme: current practice and future directions

OBJECTIVES

To describe current patterns of practice of radiation oncology peer review within a provincial cancer system, identifying barriers and facilitators to its use with the ultimate aim of process improvement.

DESIGN

A survey of radiation oncology programmes at provincial cancer centres.

SETTING

All cancer centres within the province of Ontario, Canada (n=14). These are community-based outpatient facilities overseen by Cancer Care Ontario, the provincial cancer agency.

PARTICIPANTS

A delegate from each radiation oncology programme filled out a single survey based on input from their multidisciplinary team.

OUTCOME MEASURES

Rated importance of peer review; current utilisation; format of the peer-review process; organisation and timing; case attributes; outcomes of the peer-review process and perceived barriers and facilitators to expanding peer-review processes.

RESULTS

14 (100%) centres responded. All rated the importance of peer review as at least 8/10 (10=extremely important). Detection of medical error and improvement of planning processes were the highest rated perceived benefits of peer review (each median 9/10). Six centres (43%) reviewed at least 50% of curative cases; four of these centres (29%) conducted peer review in more than 80% of cases treated with curative intent. Fewer than 20% of cases treated with palliative intent were reviewed in most centres. Five centres (36%) reported usually conducting peer review prior to the initiation of treatment. Five centres (36%) recorded the outcomes of peer review on the medical record. Thirteen centres (93%) planned to expand peer-review activities; a critical mass of radiation oncologists was the most important limiting factor (median 6/10).

CONCLUSIONS

Radiation oncology peer-review practices can vary even within a cancer system with provincial oversight. The application of guidelines and standards for peer-review processes, and monitoring of implementation and outcomes, will require effective knowledge translation activities.

Ten-year results of quality assurance in radiotherapy chart round

Background

The Royal Australian and New Zealand College of Radiologists (RANZCR) initiated a unique instrument to audit the quality of patient notes and radiotherapy prescriptions. We present our experience collected over ten years from the use of the RANZCR audit instrument.

Methods

In this study, the results of data collected prospectively from January 1999 to June 2009 through the audit instrument were assessed. Radiotherapy chart rounds were held weekly in the uro-oncology tumour stream and real time feedback was provided. Electronic medical records were retrospectively assessed in September 2009 to see if any omissions were subsequently corrected.

Results

In total 2597 patients were audited. One hundred and thirty seven (5%) patients had one hundred and ninety nine omissions in documentation or radiotherapy prescription. In 79% of chart rounds no omissions were found at all, in 12% of chart rounds one omission was found and in 9% of chart rounds two or more omissions were found. Out of 199 omissions, 95% were of record keeping and 2% were omissions in the treatment prescription. Of omissions, 152 (76%) were unfiled investigation results of which 77 (51%) were subsequently corrected.

Conclusions

Real-time audit with feedback is an effective tool in assessing the standards of radiotherapy documentation in our department, and also probably contributed to the high level of attentiveness. A large proportion of omissions were investigation results, which highlights the need for an improved system of retrieval of investigation results in the radiation oncology department.

Enhancing the role of case-oriented peer review to improve quality and safety in radiation oncology: Executive summary

This report is part of a series of white papers commissioned for the American Society for Radiation Oncology (ASTRO) Board of Directors as part of ASTRO's Target Safely Campaign, focusing on the role of peer review as an important component of a broad safety/quality assurance (QA) program. Peer review is one of the most effective means for assuring the quality of qualitative, and potentially controversial, patient-specific decisions in radiation oncology. This report summarizes many of the areas throughout radiation therapy that may benefit from the application of peer review. Each radiation oncology facility should evaluate the issues raised and develop improved ways to apply the concept of peer review to its individual process and workflow. This might consist of a daily multidisciplinary (eg, physicians, dosimetrists, physicists, therapists) meeting to review patients being considered for, or undergoing planning for, radiation therapy (eg, intention to treat and target delineation), as well as meetings to review patients already under treatment (eg, adequacy of image guidance). This report is intended to clarify and broaden the understanding of radiation oncology professionals regarding the meaning, roles, benefits, and targets for peer review as a routine quality assurance tool. It is hoped that this work will be a catalyst for further investigation, development, and study of the efficacy of peer review techniques and how these efforts can help improve the safety and quality of our treatments.

Impact of a real-time peer review audit on patient management in a radiation oncology department

In September 2006, the Royal Australian and New Zealand College of Radiologists (RANZCR) endorsed the modified Peer Review Audit Tool (PRAT). We aimed to assess the feasibility of using this tool in a busy radiation oncology department using an electronic medical record (EMR) system, identify areas of compliance and assess the impact of the audit process on patient management. Fortnightly random clinical audit was undertaken by using the revised RANZCR PRAT in the departments of radiation oncology at Liverpool and Macarthur Cancer Therapy Centres (LCTC and MCTC). Following audit of the EMR, treatment plans were audited by peer review. Data were collected prospectively from June 2007 to June 2008. Audits were carried out on 208 patients. Behaviour criteria were well documented in the EMR, but scanning of histology and medical imaging reports did not occur in up to a third of cases. With electronic prescriptions, treatment prescription errors were rare. In total, 8 (3.8%) out of 208 patients had a change to management recommended. Variability in interpretation of PRAT 'protocol/study' criteria was identified. We found that real-time audit is feasible and effective in detecting both issues with documentation in the EMR, and a small number of patients in whom a change to management is recommended. Recommendations have been made in order to continue to improve the audit process including documentation of any changes recommended and whether the recommended change occurred.

Is rural radiation oncology practice quality as good as the big smoke? Results of the Australian radiotherapy single machine unit trial

Radiotherapy utilization rates in rural Australia are suboptimal, with one solution being the building of single machine units (SMUs). One concern raised with such an approach is the quality of care delivered in SMUs. The Australian and Victorian governments have established two SMUs in the state of Victoria, with each SMU operated as a satellite service of a major 'hub' site. We report on the planned evaluation of practice quality. Radiation oncologist (RO) clinical practice was externally audited using the Royal Australian and New Zealand College of Radiologists Peer Review Audit instrument. This tool splits RO clinical practice into documentation/quality assurance (QA) criteria and decision-making criteria. Over the four sites, 130 patients were randomly selected for audit. At hub sites, 79.6% of all criteria audited were adequate, compared with 84.4% of criteria audited at SMUs (P = 0.0002). This difference was largely because of better adherence to documentation/QA criteria at the SMU sites. RO decision-making and protocol adherence were routinely very high and consistent with other clinical practice audits. There were no significant differences between hubs and SMUs for adherence to decision-making criteria; however, the few potential deficiencies in patient care identified occurred only at the hub sites. In at least one of these cases, potential suboptimal management was as a direct result of inadequate documentation. This audit found that SMUs provide as high a standard of radiotherapeutic care as larger hub departments. The findings also emphasize the need for all departments to target clinical documentation.

Efficacy of an integrated continuing medical education (CME) and quality improvement (QI) program on radiation oncologist (RO) clinical practice

PURPOSE

There has been little radiation oncologist (RO)-specific research in continuing medical education (CME) or quality improvement (QI) program efficacy. Our aim was to evaluate a CME/QI program for changes in RO behavior, performance, and adherence to department protocols/studies over the first 12 months of the program.

METHODS AND MATERIALS

The CME/QI program combined chart audit with feedback (C-AWF), simulation review AWF (SR-AWF), reminder checklists, and targeted CME tutorials. Between April 2003 and March 2004, management of 75 patients was evaluated by chart audit with feedback (C-AWF) and 178 patients via simulation review audit (SR-AWF) using a validated instrument. Scores were presented, and case management was discussed with individualized educational feedback. RO behavior and performance was compared over the first year of the program.

RESULTS

Comparing the first and second 6 months, there was a significant improvement in mean behavior (12.7-13.6 of 14, p = 0.0005) and RO performance (7.6-7.9 of 8, p = 0.018) scores. Protocol/study adherence significantly improved from 90.3% to 96.6% (p = 0.005). A total of 50 actions were generated, including the identification of learning needs to direct CME tutorials, the systematic change of suboptimal RO practice, and the alteration of deficient management of 3% of patients audited during the program.

CONCLUSION

An integrated CME/QI program combining C-AWF, SR-AWF, QI reminders, and targeted CME tutorials effectively improved targeted RO behavior and performance over a 12-month period. There was a corresponding increase in departmental protocol and study adherence.

External audit of clinical practice and medical decision making in a new Asian oncology center: Results and implications for both developing and developed nations

PURPOSE

The external audit of oncologist clinical practice is increasingly important because of the incorporation of audits into national maintenance of certification (MOC) programs. However, there are few reports of external audits of oncology practice or decision making. Our institution (The Cancer Institute, Singapore) was asked to externally audit an oncology department in a developing Asian nation, providing a unique opportunity to explore the feasibility of such a process.

METHODS AND MATERIALS

We audited 100 randomly selected patients simulated for radiotherapy in 2003, using a previously reported audit instrument assessing clinical documentation/quality assurance and medical decision making.

RESULTS

Clinical documentation/quality assurance, decision making, and overall performance criteria were adequate 74.4%, 88.3%, and 80.2% of the time, respectively. Overall 52.0% of cases received suboptimal management. Multivariate analysis revealed palliative intent was associated with improved documentation/clinical quality assurance (p = 0.07), decision making (p = 0.007), overall performance (p = 0.003), and optimal treatment rates (p = 0.07); non-small-cell lung cancer or central nervous system primary sites were associated with better decision making (p = 0.001), overall performance (p = 0.03), and optimal treatment rates (p = 0.002).

CONCLUSIONS

Despite the poor results, the external audit had several benefits. It identified learning needs for future targeting, and the auditor provided facilitating feedback to address systematic errors identified. Our experience was also helpful in refining our national revalidation audit instrument. The feasibility of the external audit supports the consideration of including audit in national MOC programs.