Appraisal of clinical competence during clerkships: how knowledgeable in curriculum and assessment development should a physician-examiner be?

Construct Validity and Generalizability of Pediatrics Clerkship Evaluation at a Problem-Based Medical School, Bahrain

The aim of this study is to identify the sources of construct validity evidence and determine generalizability of the scores given to medical students on pediatrics clerkship evaluation at the College of Medicine and Medical Sciences, Arabian Gulf University (CMMS-AGU). The CMMS-AGU is a problem-based medical school in Bahrain. The sample was composed of 140 students. Validity evidence and generalizability theory were sought. Results indicated that pediatrics clerkship evaluation at CMMS-AGU has content validity. Some limitations may have occurred in response process, but the students performance was observed directly by the tutors. There was evidence of range restriction and ``halo'' effect. Interrater reliability was 0.74, and the generalizability coefficient for three raters was 0.61. Students' contribution to the variance component of the global clerkship evaluation scores was 24%. Decision study indicated that seven tutors are required to achieve a generalizability coefficient of about 0.80. In conclusion, this study indicated that the pediatrics clerkship evaluation at the CMMS-AGU has, overall, some acceptable content and internal structure evidence of validity and that the raters are the major source of error. To improve the construct validity, there is a need to increase the number of raters and to train raters about the proper process of clerkship evaluation.

Effects of item and rater characteristics on checklist recording: what should we look for?

OBJECTIVE

Examinations based on using standardised patients (SPs) commonly use checklist recordings to evaluate students' clinical performance. This paper examines whether and to what extent item and rater characteristics affect the reliability of history checklist recording in an SP-based assessment.

METHODS

Checklist items were reviewed for the presence or absence of 5 item characteristics and a 2-point versus 3-point scoring scale. Agreement between checklist recordings obtained from SPs and clinician-examiners (CEs) were compared by item characteristics, scoring scale and CEs' level of involvement in the assessment.

RESULTS

Based on 3179 pairs of recordings, the overall percentage of agreement between SPs and CEs was 83% (kappa = 0.64). Agreement was significantly higher for items scored on a 2-point than on a 3-point scale, and when the CE was also the author and the trainer of the station. After controlling for other factors, item characteristics were only marginally associated with level of interrater agreement.

CONCLUSIONS

This study suggests that attention should be paid to specific aspects of checklist development and checklist recording training when an SP or CE is used as recorder.