Evaluating the Construct Validity of Competencies: A Retrospective Analysis

Background

A competency-based framework focuses on alignment between professional standards and assessment design. This alignment implies improved measurement validity, yet it has not been established that competence in one context predicts performance in another context. High-stakes competence assessments offer insights into the relationship between assessment design and competencies.

Methods/Analyses

The internationally educated nurses competency assessment program (IENCAP) was developed at Touchstone Institute in collaboration with the College of Nurses of Ontario (CNO) and includes a 12-station OSCE. Each station evaluated the same 10 competencies. We submitted competency scores to a multi-trait multi-method matrix analysis to evaluate the convergent and discriminant validity of competencies.

Results/Observations

All correlations were significant and positive; however, we did not find evidence of convergent or discriminant validity. Correlations were higher between different competencies evaluated within the same station (mean correlation = 0.60) compared to identical competencies evaluated across different stations (mean correlation = 0.19).

Discussion

The results do not provide evidence of construct validity for competencies. While competency-based approaches emphasize various generalized knowledge, skills, and attitudes, these findings indicate that the clinical context is a major determinant of performance.

Conclusion

The context-dependent nature of competencies requires multiple assessments in varied contexts. Performance on a single competency cannot be determined in a single occasion.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40670-023-01794-z.

The impact of case specificity and generalisable skills on clinical performance: a correlated traits-correlated methods approach

CONTEXT

The finding of case or content specificity in medical problem solving moved the focus of research away from generalisable skills towards the importance of content knowledge. However, controversy about the content dependency of clinical performance and the generalisability of skills remains.

OBJECTIVES

This study aimed to explore the relative impact of both perspectives (case specificity and generalisable skills) on different components (history taking, physical examination, communication) of clinical performance within and across cases.

METHODS

Data from a clinical performance examination (CPX) taken by 350 Year 3 students were used in a correlated traits-correlated methods (CTCM) approach using confirmatory factor analysis, whereby 'traits' refers to generalisable skills and 'methods' to individual cases. The baseline CTCM model was analysed and compared with four nested models using structural equation modelling techniques. The CPX consisted of three skills components and five cases.

RESULTS

Comparison of the four different models with the least-restricted baseline CTCM model revealed that a model with uncorrelated generalisable skills factors and correlated case-specific knowledge factors represented the data best. The generalisable processes found in history taking, physical examination and communication were responsible for half the explained variance, in comparison with the variance related to case specificity. Conclusions Pure knowledge-based and pure skill-based perspectives on clinical performance both seem too one-dimensional and new evidence supports the idea that a substantial amount of variance contributes to both aspects of performance. It could be concluded that generalisable skills and specialised knowledge go hand in hand: both are essential aspects of clinical performance.

Clinical competence: general ability or case-specific?

BACKGROUND

Before the 1970s, research into the development of clinical competence was mainly focused on general problem-solving abilities. The scope of research changed when Elstein and colleagues discovered that individual ability to solve clinical problems varies considerably across cases. It was concluded that problem solving abilities are highly dependent on domain-specific knowledge rather than on general problem solving skills. Elstein called this phenomenon "case specificity."

PURPOSE

The finding of content specificity will be contrasted with the existence of a general clinical problem solving ability, and the relationship between preclinical knowledge and a problem solving ability will be investigated.

METHODS

A correlation matrix was calculated with clerkship final scores from 10 disciplines to examine the magnitude of the interrelations. A confirmatory factor analysis was applied to the corresponding covariances using structural equation modeling to investigate whether scores on finals shared any common variance across clerkships. Finally, two additional models were tested to examine the nature of the relationship between preclinical knowledge and problem solving.

RESULTS

Low to moderate correlations across clerkship disciplines were found, supporting the original findings of content specificity. Further investigation showed that in addition to specific knowledge, a general, content-independent ability is needed to perform on these examinations.

CONCLUSIONS

Clinical competence, as measured in this study, is based on a combination of specific preclinical knowledge and a problem-solving ability. Case specificity fits perfectly well in this interactional perspective on clinical problem solving but does not explain it. The phenomenon "case specificity" is therefore not solely a result of content knowledge, but of level of experience and level of case difficulty.

The Department of Medical Education at the University of Michigan Medical School: a case study in medical education research productivity

The Department of Medical Education (DME) at the University of Michigan Medical School has a strong and sustained history of contributing to medical education research. The author identifies several dimensions that contribute to this productivity: (1) the quality of department faculty and the complementary areas of expertise they possess; (2) a critical mass of educational scholars, both within and outside the DME; (3) extensive collaborations of DME faculty with colleagues in other departments on educational innovation; (4) the departmental status of the DME; (5) the separation of the DME from the administration of the curriculum; (6) the need to balance the missions of research and educational support of the medical school; (7) the research-intensive nature of the larger University of Michigan environment; (8) the complex challenge of funding the educational scholarship mission; and (9) the importance of maintaining visibility within the institution. Factors that will affect the health of future educational scholarship include (1) the response to the Accreditation Council for Graduate Medical Education's definition of resident competencies and similar initiatives; (2) the growth of opportunities for advanced training in educational scholarship and the corresponding expansion of medical school faculty with greater interest and skills in educational research; (3) an emerging emphasis on the importance of behavioral science in medical care; (4) demands on the clinical productivity of collaborating faculty; and (5) the paucity of funding for medical education research.