Domains-based outcomes assessment of continuing medical education: the VA's model

Improving Outcomes

OBJECTIVES

To determine the strength of study design and outcomes in literature describing pathology education for medical students.

METHODS

A search was conducted for articles related to pathology education published over 45 years describing an educational intervention. The primary data collected included phase of education, domain of objectives, number of learners and institutions, type of intervention, use of a comparison/control group, randomization, and strength of statistical analysis.

RESULTS

Of 655 articles, 54 (8%) met inclusion criteria. The majority (65%) reported experiences of 100 learners or fewer, and only one was multi-institutional. Only 46% used a comparison/control group. Statistical significance of results was not reported in 39%. None examined outcomes at a point distant from the educational intervention.

CONCLUSIONS

Most studies describing pathology educational interventions are not of strong experimental design. Consumers of educational research should be cognizant of these potential weaknesses in educational studies.

The Quality Of Evidence In Preclinical Medical Education Literature: A Systematic Review

INTRODUCTION

To practice effective evidence-based teaching, the need for well-designed studies that describe outcomes related to educational interventions is critical. The quality of the literate in basic science disciplines is unknown. The study objective was to conduct a systematic review of the literature to assess study design in articles describing innovations in preclinical medical education.

METHOD

The authors searched PubMed for all articles published in English between 2000 and 2017 describing interventions in preclinical medical education related to anatomy, physiology, and biochemistry. Articles were scored using a modification of the Medical Education Research Study Quality Instrument.

RESULTS

Of the 817 articles identified, 177 met final inclusion criteria (75 anatomy, 86 physiology, and 16 biochemistry). Laboratory, student-driven, and online activities were the most frequently reported. The average score for all papers was 15.7 (27 points possible). More than 80% reported experiences with one cohort of students and >97% involved only one institution. Only 25-49% of reports utilized a comparison (control) group. Proper statistical models for analysis of results were used in only 44-62% of papers.

CONCLUSION

Manuscripts had a strong tendency toward single institutional studies that involved one cohort of students. The use of a control/comparison group when assessing effectiveness was seen in <50% and nearly all reported outcomes solely in the form of student satisfaction or factual recall/skill performance.

A novel housestaff educational model for quaternary-care patients at an academic health center

PURPOSE

Work hour restrictions, external pressure to reduce hospitalization costs, and rising inpatient acuity have prompted a variety of changes in the ward-based educational models traditionally employed in residency training. The impact of these changes remains largely unstudied.

METHOD

The authors collected data retrospectively on hospital patients with advanced liver disease before (July 1, 2003, to May 31, 2004) and after (July 1, 2004, to May 31, 2005) implementation of a novel specialized housestaff service (SHS) model for those patients, supervised by a multidisciplinary hepatology team led by an attending hepatologist. The authors also assessed the satisfaction of the 118 internal medicine residents who had rotated through both the SHS model and a traditional housestaff service (THS) in cardiology.

RESULTS

In univariate analysis, there was a trend toward a reduction in mean length of stay (LOS) after implementation of the liver service that did not meet statistical significance (P = .1). After adjustment for patient acuity and Model-for-End-Stage-Liver-Disease score, implementation of the liver service was associated with a statistically significant reduction in LOS (P = .05). In contrast, during the study period, there was an increase in LOS on a comparable, nonspecialized medicine housestaff ward even after adjustment for patient acuity (P < .01). With respect to housestaff satisfaction, 90% of housestaff reported being satisfied overall with their experience caring for patients under the SHS model, with a mean score of 4.1 on a 5-point scale.

CONCLUSION

The implementation of the SHS model of patient care is associated with a decreased LOS and increased trainee satisfaction when compared with the THS model of patient care.

Improving depression care for ethnic and racial minorities: a concept for an intervention that integrates CME planning with improvement strategies

Depression is one of the most common reasons that individuals seek treatment in the primary care setting. Research in the past 15 years has shown that dramatic improvement in the management of patients with depression is possible. Advances in pharmacotherapy and delivery of depression care have been reported, but few currently benefit members of ethnic and racial minorities. Educating physicians and other health professionals has been suggested as one approach to address the issues related to disparities in depression care. There is little evidence, however, that education alone is effective. The authors of this article believe that incorporating physician learning activities that are planned using approaches that have been shown to be effective in interventions currently demonstrating some success in improving depression care provided to ethnic and racial minorities will enhance the impact and sustainability of these interventions. This article--the conclusion of this supplement--will describe an intervention concept that integrates a quality improvement model (the Institute for Health Improvement's Breakthrough Series Collaborative model) with an evidence-based approach to planning CME and supports the integration by using action inquiry technologies and community-based participatory research methods. Relevant approaches from implementation research are discussed, and suggestions for testing the intervention concept are provided.

A conceptual model for using action inquiry technologies to address disparities in depression

Disparities in depression care remain an important problem in the United States. Action inquiry technologies may assist individuals and communities in their attempts to reduce or eliminate these disparities--and the multiple factors contributing to them--through a recurring cycle of planning, action, evaluation, and new actions based on reflections about what occurred during previous actions. This article will briefly review different action inquiry methods--specifically, participatory action research (PAR) in communities and action research in physicians' practices and offices of continuing medical education (CME). The authors develop a conceptual model in which those involved in providing, receiving, and improving depression care can use action inquiry strategies that are coordinated using the domains-based outcomes assessment model. This conceptual model may help CME planners collaborate with others to address disparities in depression care.

A call for outcomes research in medical education

The primary goal of medical education is to produce physicians who deliver high-quality health care. Recent calls for greater accountability in medical education and the development of outcomes research methodologies should encourage a new research effort to examine the effects of medical training upon clinical outcomes. The authors offer a research agenda that links medical education and quality of health care and give specific examples of potential research projects that would begin to examine that relationship. A proposed model of patient outcomes research in medical education recognizes the contributory effects of health care system-level factors as well as the continuum of medical education, process measures, and individual training and preparedness to deliver high-quality care. There exists an opportunity to create a research agenda in medical education outcomes research that is multidisciplinary, broad based, and focused on patient-centered outcomes.

Using Baldrige criteria to meet or exceed Accreditation Council for Continuing Medical Education Standards

Continuing medical education providers accredited by the Accreditation Council for Continuing Medical Education (ACCME) may apply organizational assessment strategies beyond the ACCME Essential Areas, Elements, and Criteria. The Malcolm Baldrige National Quality Program offers an organizational assessment strategy commonly used in business, health care, and education settings. An analysis of both standards pointed out useful associations between the ACCME Essential Areas and the Baldrige National Quality Program Education Criteria (2003). Including leadership, governance, and social responsibility, the Baldrige Education Criteria provide a more comprehensive organizational assessment and stronger emphasis on a wider variety of results. The present analysis suggests that a continuing medical education provider could meet, and possibly exceed, the ACCME standards by applying the Baldrige Education Criteria in a "self-study" process to define, measure, monitor, and document fundamental organizational responsibilities and performance.

Lessons learned at sea--ocean sailing as a metaphor for surgical training

Surgical education is in the process of tumultuous change. Mastering this change will require a new set of competencies and a new understanding of the medical education process. While accreditation agencies are rapidly working to define the new criteria and benchmarks, training programs are quickly pulling together curricula, objectives, and evaluation tools. Yet much has already been learned in other complex, high-risk activities. Blue water sailing, ocean racing, and trans-Atlantic crossing are all activities that require a renewed form of leadership and an understanding of how knowledge, skill, and behavior come together to define the competent sailor. Ideas learned in such endeavors may assist the surgical educator in defining the horizons and the hazards of this uncharted voyage.

Continuing medical education activity and American Board of Surgery examination performance

BACKGROUND

Surgical knowledge is the basis of successful clinical problem solving, so is thought to be an important component of overall clinical ability. Continuing medical education (CME) reinforces basic knowledge and provides exposure to new knowledge within a field. Specialty board examination performance measures this knowledge but few studies have investigated a link between such performance and CME activity. This study assessed that link on the American Board of Surgery Recertification Examination.

STUDY DESIGN

The study sample comprised 278 randomly chosen applicants for the 2000 examination. Study variables included practice type, career activity, age, gender, other Board certifications, examination attempts, community size, geographic region, nationality, and ethnicity.

RESULTS

The study sample was remarkably similar to the total candidate cohort with regard to study variables. Of the 245 sample Diplomates who took the Recertification Examination, 10.2% failed. The Pass group reported 53% more total CME hours and 38% more Category I CME hours than the Fail group. The vast majority of Category I activities were surgical, clinical. Analyzed by quartiles of total CME hours, the failure rate was only 3.4% for the highest quartile but 25.8% for the lowest quartile. For Category I hours, respective failure rates were 4.8% and 19.4%. When further stratified by practice type, the failure rate of those in solo practice was 6% for those in the highest quartile of total CME hours and 37% for those in the lowest quartile. For Category I hours, the respective failure rates were 0% and 31%.

CONCLUSIONS

There is a strong relationship between CME activity and performance on the American Board of Surgery Recertification Examination. Low CME activity and practice type appear to be independent risk factors for examination failure. The relationship of these findings to patient care outcomes has important implications.