Competence by Design: The Role of High-Stakes Examinations in a Competence Based Medical Education System

Competency based medical education is developed utilizing a program of assessment that ideally supports learners to reflect on their knowledge and skills, allows them to exercise a growth mindset that prepares them for coaching and eventual lifelong learning, and can support important progression and certification decisions. Examinations can serve as an important anchor to that program of assessment, particularly when considering their strength as an independent, third-party assessment with evidence that they can predict future physician performance and patient outcomes. This paper describes the aims of the Royal College of Physicians and Surgeons of Canada's ("the Royal College") certification examinations, their future role, and how they relate to the Competence by Design model, particularly as the culture of workplace assessment and the evidence for validity evolves. For example, high-stakes examinations are stressful to candidates and focus learners on exam preparation rather than clinical learning opportunities, particularly when they should be developing greater autonomy. In response, the Royal College moved the written examination earlier in training and created an exam quality review, by a specialist uninvolved in development, to review the exam for clarity and relevance. While learners are likely to continue to focus on the examination as an important hurdle to overcome, they will be preparing earlier in training, allowing them the opportunity to be more present and refine their knowledge when discussing clinical cases with supervisors in the Transition to Practice phase. The quality review process better aligns the exam to clinical practice and can improve the educational impact of the examination preparation process.

Modification of an OSCE format to enhance patient continuity in a high-stakes assessment of clinical performance

BACKGROUND

Traditional Objective Structured Clinical Examinations (OSCEs) are psychometrically sound but have the limitation of fragmenting complex clinical cases into brief stations. We describe a pilot study of a modified OSCE that attempts to balance a typical OSCE format with a semblance of a continuous, complex, patient case.

METHODS

Two OSCE scenarios were developed. Each scenario involved a single standardized patient and was subdivided into three sequential 10 minute sections that assessed separate content areas and competencies. Twenty Canadian PGY-4 internal medicine trainees were assessed by trained examiner pairs during each OSCE scenario. Paired examiners rated participant performance independent of each other, on each section of each scenario using a validated global rating scale. Inter-rater reliabilities and Pearson correlations between ratings of the 3 sections of each scenario were calculated. A generalizability study was conducted. Participant and examiner satisfaction was surveyed.

RESULTS

There was no main effect of section or scenario. Inter-rater reliability was acceptable. The g-coefficient was 0.68; four scenarios would achieve 0.80. Moderate correlations between sections of a scenario suggest a possible halo effect. The majority of examiners and participants felt that the modified OSCE provided a sense of patient continuity.

CONCLUSIONS

The modified OSCE provides another approach to the assessment of clinical performance. It attempts to balance the advantages of a traditional OSCE with a sense of patient continuity.

Assessing cardiac physical examination skills using simulation technology and real patients: a comparison study

OBJECTIVE

High-stakes assessments of doctors' physical examination skills often employ standardised patients (SPs) who lack physical abnormalities. Simulation technology provides additional opportunities to assess these skills by mimicking physical abnormalities. The current study examined the relationship between internists' cardiac physical examination competence as assessed with simulation technology compared with that assessed with real patients (RPs).

METHODS

The cardiac physical examination skills and bedside diagnostic accuracy of 28 internists were assessed during an objective structured clinical examination (OSCE). The OSCE included 3 modalities of cardiac patients: RPs with cardiac abnormalities; SPs combined with computer-based, audio-video simulations of auscultatory abnormalities, and a cardiac patient simulator (CPS) manikin. Four cardiac diagnoses and their associated cardiac findings were matched across modalities. At each station, 2 examiners independently rated a participant's physical examination technique and global clinical competence. Two investigators separately scored diagnostic accuracy.

RESULTS

Inter-rater reliability between examiners for global ratings (GRs) ranged from 0.75-0.78 for the different modalities. Although there was no significant difference between participants' mean GRs for each modality, the correlations between participants' performances on each modality were low to modest: RP versus SP, r = 0.19; RP versus CPS, r = 0.22; SP versus CPS, r = 0.57 (P < 0.01).

CONCLUSIONS

Methodological limitations included variability between modalities in the components contributing to examiners' GRs, a paucity of objective outcome measures and restricted case sampling. No modality provided a clear 'gold standard' for the assessment of cardiac physical examination competence. These limitations need to be addressed before determining the optimal patient modality for high-stakes assessment purposes.

Assessing the relationship between cardiac physical examination technique and accurate bedside diagnosis during an objective structured clinical examination (OSCE)

BACKGROUND

Many standardized patient (SP) encounters employ SPs without physical findings and, thus, assess physical examination technique. The relationship between technique, accurate bedside diagnosis, and global competence in physical examination remains unclear.

METHOD

Twenty-eight internists undertook a cardiac physical examination objective structured clinical examination, using three modalities: real cardiac patients (RP), "normal" SPs combined with related cardiac audio-video simulations, and a cardiology patient simulator (CPS). Two examiners assessed physical examination technique and global bedside competence. Accuracy of cardiac diagnosis was scored separately.

RESULTS

The correlation coefficients between participants' physical examination technique and diagnostic accuracy were 0.39 for RP (P < .05), 0.29 for SP, and 0.30 for CPS. Patient modality impacted the relative weighting of technique and diagnostic accuracy in the determination of global competence.

CONCLUSIONS

Assessments of physical examination competence should evaluate both technique and diagnostic accuracy. Patient modality affects the relative contributions of each outcome towards a global rating.

Does physical examination competence correlate with bedside diagnostic acumen? An observational study

AIM

To examine the relationship between a physician's ability to examine a standardized patient (SP) and their ability to correctly identify related clinical findings created with simulation technology.

METHOD

The authors conducted an observational study of 347 candidates during a Canadian national specialty examination at the end of post-graduate internal medicine training. Stations were created that combined physical examination of an SP with evaluation of a related audio-video simulation of a patient abnormality, in the domains of cardiology and neurology. Examiners evaluated a candidate's competence at performing a physical examination of an SP and their accuracy in diagnosing a related audio-video simulation.

RESULTS

For the cardiology stations, the correlation between the physical examination scores and recognition of simulation abnormalities was 0.31 (p < 0.01). For the neurology stations, the correlation was 0.27 (p < 0.01). Addition of the simulations identified 18% of 197 passing candidates on the cardiology stations and 17% of 240 passing candidates on the neurology stations who were competent in their physical examination technique but did not achieve the passing score for diagnostic skills.

CONCLUSIONS

Assessments incorporating SPs without physical findings may need to include other methodologies to assess bedside diagnostic acumen.

A randomized crossover trial of Quick Medical Reference (QMR) as a teaching tool for medical interns

OBJECTIVE

To determine whether the addition of Quick Medical Reference (QMR) to usual educational tools improves an intern's performance in managing clinical cases that test diagnostic and investigative skills.

DESIGN

Randomized, controlled, single-blinded crossover study.

SETTING

Tertiary care teaching hospital.

SUBJECTS

Comprehensive medical interns who are proficient in the use of QMR.

INTERVENTION

A total of 16 interns trained in the use of QMR were randomized to work up a total of six diagnostically challenging cases, three using medical textbooks plus access to QMR and three using textbooks and no access to QMR.

MEASUREMENTS

The interns provided their solutions to the cases, which consisted of the differential diagnosis and investigations. They were scored by comparing their answers with the consensus answers provided by subspecialty consultants. For each intern, the difference in mean total scores with and without QMR was calculated.

RESULTS

The mean difference in total scores was 7.2% benefit (p < 0.05, 95% CI = 0.05 to 14.4) using QMR. The beneficial difference was driven mainly by improvement in diagnostic scores, with the mean difference being 11.6% (p = 0.01, 95% CI = 2.4 to 20.8). Regression analysis showed that the more difficult the case, the greater the benefit of QMR.

CONCLUSIONS

Quick Medical Reference may be a useful adjunct to interns in formulating diagnostic strategies for difficult clinical cases. However, since optimal conditions were chosen for QMR benefit in this study design, the small benefit in test scores must be weighed against the time required to teach QMR to interns.

Mitral valve replacement for mitral regurgitation with and without preservation of chordae tendineae

Left ventricular function was compared in 15 patients who had conventional mitral valve replacement (Group 1) and 12 patients who had replacement with preservation of chordae tendineae and papillary muscles (Group 2) for isolated chronic mitral regurgitation. Function was assessed by radionuclide ventriculography. The two groups were similar in terms of age, preoperative functional class, valve pathology, and preoperative left ventricular function. There was uniform intraoperative and postoperative management by a single surgeon. All patients received porcine bioprostheses. Left ventricular function was assessed 1 day before and 3 to 6 months after the operation. Left ventricular ejection fraction decreased from 0.55 +/- 0.09 to 0.48 +/- 0.14 in Group 1 (p less than 0.01) and did not change significantly in Group 2 (from 0.53 +/- 0.14 to 0.52 +/- 0.16). Left ventricular end-diastolic volume, end-systolic volume, and stroke volume indices decreased significantly in both groups. During maximal postoperative bicycle exercise with the patient supine, left ventricular ejection fraction and stroke volume index increased significantly in Group 2 only. Cardiac output increased in both groups during exercise, but this increase was due to increased heart rate in Group 1 and to increased heart rate and stroke volume in Group 2. These findings suggest that preservation of chordae tendineae and papillary muscles during mitral valve replacement for chronic mitral regurgitation has a beneficial effect on postoperative left ventricular function.