The certification status of generalist physicians and the mortality of their patients after acute myocardial infarction

Medical specialty certification exams studied according to the Ottawa Quality Criteria: a systematic review

BACKGROUND

Medical specialty certification exams are high-stakes summative assessments used to determine which doctors have the necessary skills, knowledge, and attitudes to treat patients independently. Such exams are crucial for patient safety, candidates' career progression and accountability to the public, yet vary significantly among medical specialties and countries. It is therefore of paramount importance that the quality of specialty certification exams is studied in the scientific literature.

METHODS

In this systematic literature review we used the PICOS framework and searched for papers concerning medical specialty certification exams published in English between 2000 and 2020 in seven databases using a diverse set of search term variations. Papers were screened by two researchers independently and scored regarding their methodological quality and relevance to this review. Finally, they were categorized by country, medical specialty and the following seven Ottawa Criteria of good assessment: validity, reliability, equivalence, feasibility, acceptability, catalytic and educational effect.

RESULTS

After removal of duplicates, 2852 papers were screened for inclusion, of which 66 met all relevant criteria. Over 43 different exams and more than 28 different specialties from 18 jurisdictions were studied. Around 77% of all eligible papers were based in English-speaking countries, with 55% of publications centered on just the UK and USA. General Practice was the most frequently studied specialty among certification exams with the UK General Practice exam having been particularly broadly analyzed. Papers received an average of 4.2/6 points on the quality score. Eligible studies analyzed 2.1/7 Ottawa Criteria on average, with the most frequently studied criteria being reliability, validity, and acceptability.

CONCLUSIONS

The present systematic review shows a growing number of studies analyzing medical specialty certification exams over time, encompassing a wider range of medical specialties, countries, and Ottawa Criteria. Due to their reliance on multiple assessment methods and data-points, aspects of programmatic assessment suggest a promising way forward in the development of medical specialty certification exams which fulfill all seven Ottawa Criteria. Further research is needed to confirm these results, particularly analyses of examinations held outside the Anglosphere as well as studies analyzing entire certification exams or comparing multiple examination methods.

Costs and Benefits of Initial Certification for Emergency Medicine Residency Graduates

BACKGROUND

Graduates of emergency medicine residency programs can seek certification from the American Board of Emergency Medicine (ABEM), yet the costs and perceived value by residents is not clear.

OBJECTIVE

This report sought to better define the value of board certification by asking physicians taking the ABEM Oral Certification Examination (OCE) to describe its costs (eg, time, money) and perceived benefits.

METHODS

A descriptive, cross-sectional, voluntary, anonymous survey was administered to physicians taking the 2018 spring and fall ABEM OCEs. Response frequencies were used to report response rates.

RESULTS

There were 2016 physicians who participated in the 2018 OCEs, of whom 1565 (78%) completed a survey. With respect to preparation, 38% (599 of 1565 responses) spent more than 30 hours preparing for the examination. Regarding the expense of preparing for the examination, 21% (328) spent nothing, 50% (776) spent less than $1,000, and 2% (38) spent more than $3,000. Most physicians (80%, 1254) reported a learning benefit to preparing for and taking the OCE. There were 49% (765) of respondents who reported that preparing for the examination reinforced their knowledge of emergency medicine; 20% (311) reported no learning benefit. Most physicians (92%, 1442) reported that ABEM certification provided a career benefit, the most common of which was more career opportunities (69%, 1076).

CONCLUSIONS

Initial certification requires a considerable investment of time and money. Physicians seeking initial ABEM certification found both learning and professional benefits, with the most frequently reported being reinforcement of medical knowledge and more career opportunities.

Associations Between National Board Exam Performance and Residency Program Emphasis on Patient Safety and Interprofessional Teamwork

OBJECTIVE

Several aspects of medical training may contribute to the ultimate goal of producing excellent physicians whose patients will have the best possible outcomes. However, the relative importance of education, evaluation and feedback, duty hours, practice structure, and program culture in achieving this goal is unclear. This study assessed associations among in-training exam performance, Accreditation Council for Graduate Medical Education (ACGME) Resident Survey responses, and American Board of Medical Specialties (ABMS) national board exam performance.

METHODS

Residency training programs at a university teaching hospital were classified as having 5-year first-time ABMS pass rates above (n=12) or below (n=3) the national average for their specialty. These groups were compared by ACGME Resident Survey data and in-training exam performance.

RESULTS

Surveys were collected from 484/543 eligible residents (89%), including 177 surveys from programs with below-average board pass rates and 307 surveys from programs with aboveaverage board pass rates. In-training exam performance was similar between groups. Aboveaverage programs had stronger agreement with statements that their culture reinforced patient safety (4.72 vs. 4.30, p=0.006) and that information was not lost during transitions of care (4.14 vs. 3.63, p=0.001). Although the occurrence of interprofessional teamwork was similar between groups, above-average programs had stronger agreement with the statement that interprofessional teamwork was effective (4.60 vs. 4.17, p=0.003).

CONCLUSION

Residency programs emphasizing patient safety and effective interprofessional teamwork had above-average first-time national board pass rates.

Relationships Between the ACGME Resident and Faculty Surveys and Program Pass Rates on the ABIM Internal Medicine Certification Examination

PURPOSE

The Accreditation Council for Graduate Medical Education (ACGME) has surveyed residents since 2003, and faculty since 2012. Surveys are designed to assess program functioning and specify areas for improvement. The purpose of this study was to assess the association of the ACGME's resident and faculty surveys with residency-program-specific performance on the American Board of Internal Medicine (ABIM) certification exam.

METHOD

Data were available from residents and faculty in 375 U.S. ACGME-accredited internal medicine programs from the 2012-2013, 2013-2014, and 2014-2015 academic years. Analysis of variance and correlations were used to examine the relationship between noncompliance with ACGME program requirements as assessed by the resident and faculty surveys, and ABIM program pass rates.

RESULTS

Noncompliance reported on the resident and faculty surveys was highest for programs not meeting the ACGME program requirement of an 80% pass rate on the ABIM certification examination. This relationship was significant for overall noncompliance, both within the resident (P < .001) and faculty (P < .05) surveys, for many areas within the two surveys (correlations ranged between -.07 and -.25, and P values ranged between .20 and < .001), and for the highest levels of noncompliance across areas of the resident (P < .001) and faculty (P < .04) surveys.

CONCLUSIONS

ACGME resident and faculty surveys were significantly associated with ABIM program pass rates, supporting the importance of these surveys within the ACGME's Next Accreditation System.

The ABMS MOC Part III Examination: Value, Concerns, and Alternative Formats

This article describes the presentations and discussions at a conference co-convened by the Council on Medical Education of the American Medical Association (AMA) and by the American Board of Medical Specialties (ABMS). The conference focused on the ABMS Maintenance of Certification (MOC) Part III Examination. This article, reflecting the conference agenda, covers the value of and evidence supporting the examination, as well as concerns about the cost of the examination, and-given the current format-its relevance. In addition, the article outlines alternative formats for the examination that four ABMS member boards are currently developing or implementing. Lastly, the article presents contrasting views on the approach to professional self-regulation. One view operationalizes MOC as a high-stakes, pass-fail process while the other perspective holds MOC as an organized approach to support continuing professional development and improvement. The authors hope to begin a conversation among the AMA, the ABMS, and other professional stakeholders about how knowledge assessment in MOC might align with the MOC program's educational and quality improvement elements and best meet the future needs of both the public and the physician community.

Association of Physician Certification in Interventional Cardiology With In-Hospital Outcomes of Percutaneous Coronary Intervention

BACKGROUND

The value of American Board of Internal Medicine certification has been questioned. We evaluated the Association of Interventional Cardiology certification with in-hospital outcomes of patients undergoing percutaneous coronary intervention (PCI) in 2010.

METHODS AND RESULTS

We identified physicians who performed ≥10 PCIs in 2010 in the CathPCI Registry and determined interventional cardiology (ICARD) certification status using American Board of Internal Medicine data. We compared in-hospital outcomes of patients treated by certified and noncertified physicians using hierarchical multivariable models adjusted for differences in patient characteristics and PCI volume. Primary end points were all-cause in-hospital mortality and bleeding complications. Secondary end points included emergency coronary artery bypass grafting, vascular complications, and a composite of any adverse outcome. With 510,708 PCI procedures performed by 5175 physicians, case mix and unadjusted outcomes were similar among certified and noncertified physicians. The adjusted risks of in-hospital mortality (odds ratio, 1.10; 95% confidence interval, 1.02-1.19) and emergency coronary artery bypass grafting (odds ratio, 1.32; 95% confidence interval, 1.12-1.56) were higher in the non-ICARD-certified group, but the risks of bleeding and vascular complications and the composite end point were not statistically significantly different between groups.

CONCLUSIONS

We did not observe a consistent association between ICARD certification and the outcomes of PCI procedures. Although there was a significantly higher risk of mortality and emergency coronary artery bypass grafting in patients treated by non-ICARD-certified physicians, the risks of vascular complications and bleeding were similar. Our findings suggest that ICARD certification status alone is not a strong predictor of patient outcomes and indicate a need to enhance the value of subspecialty certification.

Simulation for Maintenance of Certification

Maintenance of certification (MOC) is a process through which practitioners are able to show continuing competence in their areas of expertise. Simulation plays an increasingly important role in the assessment of students and residents, as well as in the initial practice certification for health care professionals. The use of simulation as an assessment tool in MOC has been sluggish to be universally accepted. This article discusses the role of simulation in health care education, how simulation might be effectively applied in the MOC process, and the future role of simulation in the MOC process.

Is anyone listening? Variation in PSA screening among providers for men 75+ before and after United States Preventive Services Task Force recommendations against it: a retrospective cohort study

Background

In 2008, the United States Preventive Services Task Force recommended against prostate specific antigen (PSA) testing for cancer screening in men age 75+.

Purpose

To assess PSA screening by primary care physicians (PCPs) before and after recommendations.

Methods

In 2013, this retrospective cohort study analyzed PCPs in Texas with 20+ male patients aged 75+ in both 2007 and 2010, with Parts A and B Medicare. The main outcome was percent of PCP’s male patients 75+ who received PSA testing ordered by the PCP in 2007 and 2010, with no recent symptoms suggestive of prostate cancer.

Results

In both 2007 and 2010, 1,083 PCPs cared for at least 20 men aged 75 or older. The rate of PSA screening ordered by PCPs was 33.2% in 2007 and 30.6% in 2010. In multilevel analyses controlling for patient characteristics, the variation in PSA screening attributable to the PCP (intraclass correlation coefficient) increased from 23% in 2007 to 26% in 2010, p<0.001. Men with PCPs older than age 60 had 9% lower odds (95% CI, 1–17%) in 2010 compared to 2007 of receiving a PSA test, vs. a 4% increase (95% CI, 4% decrease to 12% increase) in men with PCPs aged 50 or younger. Patients with Board Certified PCPs had a 12% lower odds (95% CI, 8% to 16%) from 2007 to 2010, vs. 2% increase (95% CI 11% decrease to 18% increase) in men with PCPs without board certification.

Conclusions

The USPSTF recommendation did not increase consensus among PCPs regarding PSA screening of older men.

Specialty board certification in the United States: issues and evidence

BACKGROUND

The American Board of Medical Specialties (ABMS) certification and maintenance of certification (MOC) programs strive to provide the public with guidance about a physician's competence. This study summarizes the literature on the effectiveness of these programs.

METHOD

A literature search was conducted for studies published between 1986 and April 2013 and limited to ABMS certification. A modified version of Kirkpatrick's 4 levels of program evaluation included the reaction of stakeholders to certification, the extent to which physicians are encouraged to improve, the relationship between performance in the programs and nonclinical external measures of physician competence, and the relationship of performance in the programs with clinical quality measures.

RESULTS

Patients' and hospitals' value of board certification and physician participation in MOC are high. Physicians are conflicted as to whether the effort involved is worth its value. Self-reported evidence shows improvement in knowledge, practice infrastructure, communication with patients and peers, and clinical care. Certification performance is generally related to nonclinical external measures such as types of training, practice characteristics, demographics, and disciplinary actions. In general, physicians who are board certified provide better patient care, albeit the results have modest effect sizes and are not unequivocal.

CONCLUSIONS

Certification boards should continuously try to improve their programs in response to feedback from stakeholders, changes in the way physicians practice, as well as the growth in the fields of measurement and technology. Keeping pace with these changes in a responsible and evidence-based way is important.

Do early career indicators of clinical skill predict subsequent career outcomes and practice characteristics for general internists?

OBJECTIVE

To study relationships between clinical skill measures assessed at the beginning of general internists' careers and their career outcomes and practice characteristics.

DATA SOURCES

General Internist Community Tracking Study Physician Survey respondents (2000-2001, 2004-2005) linked with residency program evaluations and American Board of Internal Medicine board certification examination score records; n = 2,331.

STUDY DESIGN

Cross-sectional regressions of career outcome and practice characteristic measures on board examination scores/success, residency evaluations interacted with residency type, and potential confounding variables.

PRINCIPAL FINDINGS

Failure to achieve board certification was associated with $27,206 (18 percent, p < .05) less income and 14.9 percent more minority patients relative to physicians scoring in the bottom quartile on their initial examination who eventually became certified (p < .01). Other skill measures were not associated with income. Scoring in the top rather than bottom quartile on the board certification examination was associated with 9 percent increased likelihood of reporting high career satisfaction (p < .05). Among physicians trained in community hospital residency programs, lower evaluations were associated with 14.5 percent higher share of minority patients (p < .05). Both skill measures were associated with practice type.

CONCLUSIONS

There are associations between early career skill measures and career outcomes. In addition, minority patients are more likely to be treated by physicians with lower early career clinical skills measures than nonminority patients.

Factors associated with American Board of Medical Specialties member board certification among US medical school graduates

CONTEXT

Certification by an American Board of Medical Specialties (ABMS) member board is emerging as a measure of physician quality.

OBJECTIVE

To identify demographic and educational factors associated with ABMS member board certification of US medical school graduates.

DESIGN, SETTING, AND PARTICIPANTS

Retrospective study of a national cohort of 1997-2000 US medical school graduates, grouped by specialty choice at graduation and followed up through March 2, 2009. In separate multivariable logistic regression models for each specialty category, factors associated with ABMS member board certification were identified.

MAIN OUTCOME MEASURE

ABMS member board certification.

RESULTS

Of 42,440 graduates in the study sample, 37,054 (87.3%) were board certified. Graduates in all specialty categories with first-attempt passing scores in the highest tertile (vs first-attempt failing scores) on US Medical Licensing Examination Step 2 Clinical Knowledge were more likely to be board certified; adjusted odds ratios (AORs) varied by specialty category, with the lowest odds for emergency medicine (87.4% vs 73.6%; AOR, 1.82; 95% CI, 1.03-3.20) and highest odds for radiology (98.1% vs 74.9%; AOR, 13.19; 95% CI, 5.55-31.32). In each specialty category except family medicine, graduates self-identified as underrepresented racial/ethnic minorities (vs white) were less likely to be board certified, ranging from 83.5% vs 95.6% in the pediatrics category (AOR, 0.44; 95% CI, 0.33-0.58) to 71.5% vs 83.7% in the other nongeneralist specialties category (AOR, 0.79; 95% CI, 0.64-0.96). With each $50,000 unit increase in debt (vs no debt), graduates choosing obstetrics/gynecology were less likely to be board certified (AOR, 0.89; 95% CI, 0.83-0.96), and graduates choosing family medicine were more likely to be board certified (AOR, 1.13; 95% CI, 1.01-1.26).

CONCLUSION

Demographic and educational factors were associated with board certification among US medical school graduates in every specialty category examined; findings varied among specialty categories.

Measuring physicians' performance in clinical practice: reliability, classification accuracy, and validity

Much research has been devoted to addressing challenges in achieving reliable assessments of physicians' clinical performance but less work has focused on whether valid and accurate classification decisions are feasible. This study used 957 physicians certified in internal medicine (IM) or a subspecialty, who completed the American Board of Internal Medicine (ABIM) Diabetes Practice Improvement Module (PIM). Ten clinical and two patient-experience measures were aggregated into a composite measure. The composite measure score was highly reliable (r = .91) and classification accuracy was high across the entire score scale (>0.90), which indicated that it is possible to differentiate high-performing and low-performing physicians. Physicians certified in endocrinology and those who scored higher on their IM certification examination had higher composite scores, providing some validity evidence. In summary, it is feasible to create a psychometrically robust composite measure of physicians' clinical performance, specifically for the quality of care they provide to patients with diabetes.