Is the public being protected? Prevention of suboptimal medical practice through training programs and credentialing examinations

Medical Council of Canada Qualifying Examinations and performance in future practice

The purpose of medical licensing examinations is to protect the public from practitioners who do not have adequate knowledge, skills, and abilities to provide acceptable patient care, and therefore evaluating the validity of these examinations is a matter of accountability. Our objective was to discuss the Medical Council of Canada's Qualifying Examinations (MCCQEs) Part I (QE1) and Part II (QE2) in terms of how well they reflect future performance in practice. We examined the supposition that satisfactory performance on the MCCQEs are important determinants of practice performance and, ultimately, patient outcomes. We examined the literature before the implementation of the QE2 (pre-1992), post QE2 but prior to the implementation of the new Blueprint (1992-2018), and post Blueprint (2018-present). The literature suggests that MCCQE performance is predictive of future physician behaviours, that the relationship between examination performance and outcomes did not attenuate with practice experience, and that associations between examination performance and outcomes made sense clinically. While the evidence suggests the MCC qualifying examinations measure the intended constructs and are predictive of future performance, the validity argument is never complete. As new competency requirements emerge, we will need to develop valid and reliable mechanisms for determining practice readiness in these areas.

An international stakeholder survey of the role of chiropractic qualifying examinations: A qualitative analysis

OBJECTIVE

Many countries require examinations as a gateway to chiropractic licensure; however, the relevance of these exams to the profession has not been explored. The purposes of this study were to analyze perceptions of international stakeholders about chiropractic qualifying examinations (CQEs), observe if their beliefs were in alignment with those that society expects of professions, and suggest how this information may be used when making future decisions about CQEs.

METHODS

We designed an electronic survey that included open-ended questions related to CQEs. In August 2019, the survey was distributed to 234 international stakeholders representing academic institutions, qualifying boards, students, practitioners, association officers, and others. Written comments were extracted, and concepts were categorized and collapsed into 4 categories (benefits, myths, concerns, solutions). Qualitative analysis was used to identify themes.

RESULTS

The response rate was 56.4% representing 43 countries and yielding 775 comments. Perceived benefits included that CQEs certify a minimum standard of knowledge and competency and are part of the professionalization of chiropractic. Myths included that CQEs are able to screen for future quality of care or ethical practices. Concerns included a lack of standardization between jurisdictions and uncertainty about the cost/value of CQEs and what they measure. Solutions included suggestions to standardize exams across jurisdictions and focus on competencies.

CONCLUSION

International stakeholders identified concepts about CQEs that may facilitate or hinder collaboration and efforts toward portability. Stakeholder beliefs were aligned with those expected of learned professions. This qualitative analysis identified 9 major themes that may be used when making future decisions about CQEs.

BEME systematic review: predictive values of measurements obtained in medical schools and future performance in medical practice

BACKGROUND

Effectiveness of medical education programs is most meaningfully measured as performance of its graduates.

OBJECTIVES

To assess the value of measurements obtained in medical schools in predicting future performance in medical practice.

METHODS

SEARCH STRATEGY

The English literature from 1955 to 2004 was searched using MEDLINE, Embase, Cochrane's EPOC (Effective Practice and Organization of Care Group), Controlled Trial databases, ERIC, British Education Index, Psych Info, Timelit, Web of Science and hand searching of medical education journals.

INCLUSION & EXCLUSIONS

Selected studies included students assessed or followed up to internship, residency and/or practice after postgraduate training. Assessment systems and instruments studied (Predictors) were the National Board Medical Examinations (NBME) I and II, preclinical and clerkship grade-point average, Observed Standardized Clinical Examination scores and Undergraduate Dean's rankings and honors society. Outcome measures were residency supervisor ratings, NBME III, residency in-training examinations, American Specialty Board examination scores, and on-the-job practice performance.

DATA EXTRACTION

Data were extracted by using a modification of the BEME data extraction form study objectives, design, sample variables, statistical analysis and results. All included studies are summarized in a tabular form. DATA ANALYSIS AND SYNTHESIS: Quantitative meta-analysis and qualitative approaches were used for data analysis and synthesis including the methodological quality of the studies included.

RESULTS

Of 569 studies retrieved with our search strategy, 175 full text studies were reviewed. A total of 38 studies met our inclusion criteria and 19 had sufficient data to be included in a meta-analysis of correlation coefficients. The highest correlation between predictor and outcome was NBME Part II and NBME Part III, r = 0.72, 95% CI 0.30-0.49 and the lowest between NBME I and supervisor rating during residency, r = 0.22, 95% CI 0.13-0.30. The approach to studying the predictive value of assessment tools varied widely between studies and no consistent approach could be identified. Overall, undergraduate grades and rankings were moderately correlated with internship and residency performance. Performance on similar instruments was more closely correlated. Studies assessing practice performance beyond postgraduate training programs were few.

CONCLUSIONS

There is a need for a more consistent and systematic approach to studies of the effectiveness of undergraduate assessment systems and tools and their predictive value. Although existing tools do appear to have low to moderate correlation with postgraduate training performance, little is known about their relationship to longer-term practice patterns and outcomes.

Effect of a community oriented problem based learning curriculum on quality of primary care delivered by graduates: historical cohort comparison study

OBJECTIVE

To assess whether the transition from a traditional curriculum to a community oriented problem based learning curriculum at Sherbrooke University is associated with the expected improvements in preventive care and continuity of care without a decline in diagnosis and management of disease.

DESIGN

Historical cohort comparison study.

SETTING

Sherbrooke University and three traditional medical schools in Quebec, Canada.

PARTICIPANTS

751 doctors from four graduation cohorts (1988-91); three before the transition to community based problem based learning (n = 600) and one after the transition (n = 151).

OUTCOME MEASURES

Annual performance in preventive care (mammography screening rate), continuity of care, diagnosis (difference in prescribing rates for specific diseases and relief of symptoms), and management (prescribing rate for contraindicated drugs) assessed using provincial health databases for the first 4-7 years of practice.

RESULTS

After transition to a community oriented problem based learning curriculum, graduates of Sherbrooke University showed a statistically significant improvement in mammography screening rates (55 more women screened per 1000, 95% confidence interval 10.6 to 99.3) and continuity of care (3.3% more visits coordinated by the doctor, 0.9% to 5.8%) compared with graduates of a traditional medical curriculum. Indicators of diagnostic and management performance did not show the hypothesised decline. Sherbrooke graduates showed a significant fourfold increase in disease specific prescribing rates compared with prescribing for symptom relief after the transition.

CONCLUSION

Transition to a community oriented problem based learning curriculum was associated with significant improvements in preventive care and continuity of care and an improvement in indicators of diagnostic performance.

Specialty board certification and clinical outcomes: the missing link

PURPOSE

Specialty board certification status is often used as a standard of excellence, but no systematic review has examined the link between certification and clinical outcomes. The authors evaluated published studies tracking clinical outcomes and certification status.

METHOD

Data sources consisted of studies cited between 1966 and July 1999 in OVID-Medline, psychological abstracts (PsycLit), and the Educational Research Information Clearinghouse (ERIC). Screening criteria included: only U.S. patients and physicians used as subjects; verified specialty board certification status by an American Board of Medical Specialties' (ABMS') member board using the ABMS database or derivative sources; described selection criteria for patients and physicians; selected nationally recognized standards of care for outcomes; and nested patient data by individual physician. The computerized searches that were conducted in 1999 identified 1,204 papers; one author and a research assistant selected 237 papers based on subject relevance, and reduced the list to 56 based on study quality. The authors independently applied inclusion and exclusion criteria to identify 13 of the 56 papers containing 33 separable relevant findings.

RESULTS

Of the 33 findings, 16 demonstrated a significant positive association between certification status and positive clinical outcomes, three revealed worse outcomes for certified physicians, and 14 showed no association. Three negative findings and one finding of no association were identified in two papers with insufficient case-mix adjustments in the analyses. Meta-analytic statistics were not feasible due to variability in outcome measures across studies.

CONCLUSIONS

Few published studies (5%) used research methods appropriate for the research question, and among the screened studies more than half support an association between board certification status and positive clinical outcomes.

Professionalism, regulation, and the market: impact on accountability for quality of care

This paper examines the interplay of professionalism, regulation, and the market in shaping accountability on the part of hospitals, physicians, and health plans. We pay particular attention to the role of accreditation. We review the development of accountability and examine its recent evolution in the context of changing information technology, consumer demands, the decline of the staff- and group-model HMO, and the reemergence of health care cost inflation. The market is emerging as the dominant influence on accountability; this development will require changes in the roles and structure of regulation, professionalism, and accreditation in assuring accountability.

General surgery in evolution: technology and competence

This paper addresses the current status of General Surgery, the direction in which it is moving, and how to prepare for future changes. The paper points out that, since the predicted effects of new technology on the practice of General Surgery are more technical complexity, decreasing invasiveness, and fewer traditional operations, the consequences for the training of surgeons, the division of responsibilities among specialists, and the organization of surgical care will be substantial. In light of the current difficulty in dealing with the stresses within General Surgery, the paper discusses how changes in residency training and an increased use of Fellowships will enable the specialty to handle such disruptive changes in the future.

Validation of professional licensure examinations. Professions theory, test design, and construct validity

Although the unitary view of test validity has gained support recently, it has real limitations where professional licensing examinations are concerned. A strategy for validation of professional licensure tests requires modifying conventional approaches in three ways. First, a theory of professions must be incorporated into the test development process so as to acknowledge the social character of professions. Second, the importance of test design in the validation of licensing tests should be enhanced. Third, the concept of construct validation must be expanded to accommodate the special features of content that inhere to professional licensure testing. Methods for accomplishing these three things are described and the implications of these and other views discussed.

Validating interpretive arguments for licensure and certification examinations

To validate the interpretation assigned to test scores is to support the rationale for this interpretation. The interpretations for licensure and certification tests involve a sequence of inferences, or an argument, leading from the test score to decisions about licensure or certification. This article examines several possible interpretive arguments for licensure and certification test scores and analyzes the evidence required to support each type of argument. Particular attention is given to a competency-based argument that involves a sequence of several inferences leading from test scores to statements about competence, and then to conclusions about expected performance in practice. Decisions about licensure or certification are based on expected performance in practice.

A research agenda for licensing and certification testing validation studies

The set of articles appearing in this issue of Evaluation & the Health Professions provides an in-depth examination of the validation process for licensing and certification test score decisions. The winter 1990 issue of Educational Measurement: Issues and Practice also contains an excellent set of articles about licensing and certification tests. These special editions are much needed because these kinds of testing programs are increasing in numbers and aim to ensure public safety, promote growth of professions, and positively affect the careers of persons dedicated to serving in these professions. This article will comment on issues raised by the authors of these articles. But before doing this, a context is described that affects these issues.

Research on standards for professional licensure and certification examinations

The purpose of this article is to outline an agenda for research on standards for professional licensure and certification examinations. To avoid confusion, scores are defined as content-based decisions about the correctness of responses, and standards refer to the educational/social decisions about how many questions need to be answered correctly to pass. Standards are the focus of this article. The two major types of standards, relative and absolute, are described and the major standard-setting methods for each are presented. Some of the published results in four different areas are reviewed: (a) basis of the judgements, (b) efficiency of the process, (c) group effects, and (d) content and expertise effects. Topics for future research in all four areas are identified.