Effects of item and rater characteristics on checklist recording: what should we look for?

Validated checklist for assessing communication skills in undergraduate medical students: bridging the gap for effective doctor-patient interactions

Communication skills are fundamental in healthcare, but assessing them among medical students presents challenges. In the Indian context, the lack of a specific assessment tool further compounds the issue. Thus this study aimed to develop and validate an observation-based communication skills checklist tailored to Phase I Bachelor of Medicine and Bachelor of Surgery (MBBS) students. The checklist was developed using both inductive and deductive approaches and underwent rigorous testing to ensure its reliability and validity. After piloting, the finalized version was administered to 84 Phase I MBBS students. Results indicated a critical content validity ratio of 0.78, face validity of 0.80, and an impressive Cronbach's alpha of 0.91, indicating good internal consistency and reliability of the checklist. The students scored over 80% in all checklist domains, except for empathy (73%) and support (74%), highlighting areas for potential improvement. Nonetheless, about 75% of students expressed satisfaction with the checklist's communication skills assessment. The feedback from faculty members was positive, as they found the checklist was easy to use, quick, and effective for evaluating communication skills. Consequently, the checklist's introduction has been well received by both students and faculty. In conclusion, the developed checklist proves to be an effective and valid instrument for assessing communication skills in Phase I MBBS students. By integrating this tool into observed station clinical examinations, medical educators can comprehensively evaluate students' communication behaviors. Moreover, the checklist serves as a valuable resource for bridging the gap between theoretical knowledge and practical application, enabling future physicians to excel in doctor-patient interactions, a crucial aspect of patient-centered care.NEW & NOTEWORTHY This article presents a highly unique and novel approach by introducing a structured checklist for communication skill assessment in medical students. Its rigorous validation process ensures reliability and effectiveness, while its adaptation to sociocultural norms highlights its relevance and applicability. The user-friendly design with a single-page layout and dichotomous scale further enhances its practicality in objective structured clinical examinations. The developed checklist equips educators with a valuable resource to assess and improve students' communication abilities.

Relationship Between Faculty and Standardized Patient Assessment Scores of Podiatric Medical Students During a Standardized Performance Assessment Laboratory

BACKGROUND

Direct assessment of health professional student performance of clinical skills can be accurately performed in the standardized performance assessment laboratory (SPAL), typically by health professional faculty. However, owing to time and economic considerations, nonmedical individuals have been specially trained to perform the same function (standardized patients [SPs]). This study compared the assessment scores of the history and physical examination components of a SPAL designed for second-year podiatric medical students at Des Moines University (DMU) by a podiatry medical faculty member and SPs.

METHODS

A total of 101 students from the classes of 2015 and 2016 were evaluated in 2013 and 2014 by 11 to 13 SPs from the DMU SPAL program. The video recordings of these 101 students were then evaluated by one faculty member from the College of Podiatric Medicine and Surgery at DMU.

RESULTS

The Pearson correlation coefficient for each class showed a strong linear relationship between SP and faculty assessment scores. The associations between SP and faculty assessment scores in the history, physical examination, and combined history and physical examination components for the 2016 class (0.706, 0.925, and 0.911, respectively) were found to be stronger than those for the 2015 class (0.697, 0.791, and 0.791, respectively).

CONCLUSIONS

This study indicated that there are strong associations between the assessment scores of trained SPs and faculty for the history, physical examination, and combined history and physical examination components of second-year SPAL activity for podiatric medical students.

Twelve tips for designing and implementing a structured rater training in OSCEs

BACKGROUND

Observation, rating, and grading are parts of the routine of the faculty when in contact with medical students. Most interestingly, there is a dearth of proper instructions on rater training for objective structured clinical examinations (OSCEs) in medical education literature.

AIMS

We assembled 12 tips for implementing a structured rater training program. To highlight these tips, we provided examples of our implementation of rater training for an OSCE in 2009.

METHODS

We developed a tailored rater training program by compiling information from medical education and other related research disciplines.

RESULTS

The structured rater training at hand proved feasible, and the acceptance rate of this training by examiners was high.

Checking the checklist: a content analysis of expert- and evidence-based case-specific checklist items

OBJECTIVES

Research on objective structured clinical examinations (OSCEs) is extensive. However, relatively little has been written on the development of case-specific checklists on history taking and physical examination. Background information on the development of these checklists is a key element of the assessment of their content validity. Usually, expert panels are involved in the development of checklists. The objective of this study is to compare expert-based items on OSCE checklists with evidence-based items identified in the literature.

METHODS

Evidence-based items covering both history taking and physical examination for specific clinical problems and diseases were identified in the literature. Items on nine expert-based checklists for OSCE examination stations were evaluated by comparing them with items identified in the literature. The data were grouped into three categories: (i) expert-based items; (ii) evidence-based items, and (iii) evidence-based items with a specific measure of their relevance.

RESULTS

Out of 227 expert-based items, 58 (26%) were not found in the literature. Of 388 evidence-based items found in the literature, 219 (56%) were not included in the expert-based checklists. Of these 219 items, 82 (37%) had a specific measure of importance, such as an odds ratio for a diagnosis, making that diagnosis more or less probable.

CONCLUSIONS

Expert-based, case-specific checklist items developed for OSCE stations do not coincide with evidence-based items identified in the literature. Further research is needed to ascertain what this inconsistency means for test validity.

Approaches to medical student remediation after a comprehensive clinical skills examination

OBJECTIVE

Most US medical schools conduct comprehensive clinical skills assessments during Years 3 and 4. This study explores strategies used to identify and remediate students who perform poorly on these assessments.

METHODS

In the academic year 2005-06, we conducted 33 semi-structured interviews with individuals responsible for standard setting in and remediation after their schools' comprehensive clinical skills assessments. We coded interviews to identify major themes.

RESULTS

Prior to remediation, some schools employed a 'verification' step to ensure the accuracy of the failing score or need for remediation. Participants described a remediation process that included some or all of 3 steps. Firstly, students' specific learning deficits were diagnosed. Next, students participated in remedial activities such as performance review sessions or practice with standardised or actual patients. Lastly, students were re-tested, usually with a shorter, more formative examination. All participants reported using a diagnostic step, most offered or required remedial activities and many re-tested, although schools varied in the emphasis placed on each step. Many participants cited the individualised attention students received from remediation faculty staff as a strength of their approach, although they raised concerns about the substantial time demands placed on remediation faculty. Most respondents reported some dissatisfaction with their school's remediation process, particularly uncertainty about efficacy or rigour.

CONCLUSIONS

Schools vary in the intensity and scope of remediation offered to students who perform poorly on clinical skills assessments. Although many schools invest significant resources in remediation, the effect of these efforts on students' subsequent clinical performance is unknown.