Trained lay observers can reliably assess medical students' communication skills

Implementation of Pediatric Allergic Rhinitis Module as a Part of AETCOM among First-Year Medical Undergraduates: Mixed Methods Evaluation

Background

Children suffering from allergic rhinitis (AR) in their earlier days of life, not receiving proper treatment, subsequently develop asthma. To sensitize the first-year medical undergraduates about AR by implementing pediatric allergic rhinitis (PAR) module as a part of their attitude, ethics, and communication (AETCOM) curriculum.

Materials and Methods

Triangulation type of mixed method study was conducted from January 2021 to June 2021 among 125 first-year medical undergraduate students. The PAR module communication checklist was developed and validated by an interprofessional (IP) team. Twenty multiple-choice questions (MCQs) were framed for both pretest and posttest cognitive assessment of the students. The pretest assessment was done (first 15 min) followed by the teaching of the PAR module (30 min), and lastly the posttest assessment along with open-ended feedback (last 15 min). Objective Structured Clinical Examination (OSCE) communication checklist along with the guidelines was given to the observer during the student-patient encounter to score the learner and to assess the communication skill. Apart from descriptive analysis, paired t-test and content analysis were done.

Results

A statistically significant difference in the mean scores before and after the PAR module and communication checklist (P < 0.001). Majority (78/81, 96%) of the students favored this module, while (28/81) 34.6% suggested modifications. Most of the parent's feedback was good about the student's communication skill in terms of empathy (118), behavior (107), and greet (125); however, 33 parents were about the opinion of difficulties in closing the session, 17 parents commented about student's language problem and 27 about feedback.

Conclusion

The PAR module should be taught in the current medical curriculum as a part of AETCOM in the foundation course as early clinical exposure with some modifications in the existing module.

Students' perception and scores in Paediatrics end-of-clerkship and final professional Objective Structured Clinical Examination (OSCE): A comparative study

Objectives:

This study aims to understand students’ perception of the usefulness of end-of-clerkship (EOC) as compared to professional exam Objective Structured Clinical Examinations (OSCE) and to compare student performance in both examinations.

Methods:

We conducted a cross-sectional study of medical students who were studying in the final year at Shifa College of Medicine, Shifa Tameer-e-Millat University, and appeared in both end-of-clerkship and final professional exams. The study was conducted in October 2019. The data was collected through a self-constructed questionnaire. The scores of all participants were also recorded. The data obtained was analyzed on IBM’s statistical package for the social sciences (SPSS) version 23 (IBM, Armonk, NY). Descriptive statistics were used for qualitative variables. Mean and standard deviation was calculated for quantitative variables. Chi-square test was also applied to assess for significant differences.

Results:

Out of 115 participants, 57 (49.6%) were males and 58 (50.4%) were females. Most of the participants (75.7%) agreed that both OSCEs were well-structured. Participants found that both the exams give a good cross-section of paediatrics and allowed them to learn something new (p=0.00). 45% of participants perceived that the end-of-clerkship OSCE exam helped in orienting them for the final professional exam by identifying their weaknesses.

Conclusion:

Although the EOC OSCE served as a good preparatory exercise, it did not alleviate the stress levels. However, the results in both OSCEs were comparable and students were satisfied with the current weightage of marks distribution.

Assessing Communication Skills of Medical Students in Objective Structured Clinical Examinations (OSCE)--A Systematic Review of Rating Scales

Background

Teaching and assessment of communication skills have become essential in medical education. The Objective Structured Clinical Examination (OSCE) has been found as an appropriate means to assess communication skills within medical education. Studies have demonstrated the importance of a valid assessment of medical students’ communication skills. Yet, the validity of the performance scores depends fundamentally on the quality of the rating scales used in an OSCE. Thus, this systematic review aimed at providing an overview of existing rating scales, describing their underlying definition of communication skills, determining the methodological quality of psychometric studies and the quality of psychometric properties of the identified rating scales.

Methods

We conducted a systematic review to identify psychometrically tested rating scales, which have been applied in OSCE settings to assess communication skills of medical students. Our search strategy comprised three databases (EMBASE, PsycINFO, and PubMed), reference tracking and consultation of experts. We included studies that reported psychometric properties of communication skills assessment rating scales used in OSCEs by examiners only. The methodological quality of included studies was assessed using the COnsensus based Standards for the selection of health status Measurement INstruments (COSMIN) checklist. The quality of psychometric properties was evaluated using the quality criteria of Terwee and colleagues.

Results

Data of twelve studies reporting on eight rating scales on communication skills assessment in OSCEs were included. Five of eight rating scales were explicitly developed based on a specific definition of communication skills. The methodological quality of studies was mainly poor. The psychometric quality of the eight rating scales was mainly intermediate.

Discussion

Our results reveal that future psychometric evaluation studies focusing on improving the methodological quality are needed in order to yield psychometrically sound results of the OSCEs assessing communication skills. This is especially important given that most OSCE rating scales are used for summative assessment, and thus have an impact on medical students’ academic success.

Examining rater and occasion influences in observational assessments obtained from within the clinical environment

BACKGROUND

When ratings of student performance within the clerkship consist of a variable number of ratings per clinical teacher (rater), an important measurement question arises regarding how to combine such ratings to accurately summarize performance. As previous G studies have not estimated the independent influence of occasion and rater facets in observational ratings within the clinic, this study was designed to provide estimates of these two sources of error.

METHOD

During 2 years of an emergency medicine clerkship at a large midwestern university, 592 students were evaluated an average of 15.9 times. Ratings were performed at the end of clinical shifts, and students often received multiple ratings from the same rater. A completely nested G study model (occasion: rater: person) was used to analyze sampled rating data.

RESULTS

The variance component (VC) related to occasion was small relative to the VC associated with rater. The D study clearly demonstrates that having a preceptor rate a student on multiple occasions does not substantially enhance the reliability of a clerkship performance summary score.

CONCLUSIONS

Although further research is needed, it is clear that case-specific factors do not explain the low correlation between ratings and that having one or two raters repeatedly rate a student on different occasions/cases is unlikely to yield a reliable mean score. This research suggests that it may be more efficient to have a preceptor rate a student just once. However, when multiple ratings from a single preceptor are available for a student, it is recommended that a mean of the preceptor's ratings be used to calculate the student's overall mean performance score.

Reliability and validity of OSCE checklists used to assess the communication skills of undergraduate medical students: A systematic review

OBJECTIVES

To explore inter-rater agreement between reviewers comparing reliability and validity of checklist forms that claim to assess the communication skills of undergraduate medical students in Objective Structured Clinical Examinations (OSCEs).

METHODS

Papers explaining rubrics of OSCE checklist forms were identified from Pubmed, Embase, PsycINFO, and the ProQuest Education Databases up to 2013. Included were those studies that report empirical validity or reliability values for the communication skills assessment checklists used. Excluded were those papers that did not report reliability or validity.

RESULTS

Papers focusing on generic communication skills, history taking, physician-patient communication, interviewing, negotiating treatment, information giving, empathy and 18 other domains (ICC -0.12-1) were identified. Regarding the validity and reliability of the communication skills checklists, agreement between reviewers was 0.45.

CONCLUSIONS

Heterogeneity in the rubrics used in the assessment of communication skills and a lack of agreement between reviewers makes comparison of student competences within and across institutions difficult.

PRACTICE IMPLICATIONS

Consideration should be afforded to the adoption of a standardized measurement instrument to assess communication skills in undergraduate medical education. Future research will focus upon evaluating the potential impact of adoption of a standardized measurement instrument.

Contextualizing SEGUE: Evaluating Residents' Communication Skills Within the Framework of a Structured Medical Interview

BACKGROUND

The SEGUE (Set the stage, Elicit information, Give information, Understand the patient's perspective, and End the encounter) Framework is a checklist-style rating scale to facilitate the teaching and assessment of communication skills in medical learners. It has been used for over 15 years, and it is recommended in the Accreditation Council for Graduate Medical Education toolbox of assessment methods for resident training. When it was developed, its ability to provide objective scoring was a substantial improvement over global ratings.

METHODS

In this article we describe the strengths and weaknesses of the SEGUE Framework. We highlight one residency program's experience with using the SEGUE Framework to evaluate residents' communication skills. Specifically, we cite previous studies and describe our own analysis of resident interviewing performance that demonstrates how the SEGUE Framework did not distinguish between different levels of interviewing skill level in our sample.

RESULTS

Two case examples illustrate how the SEGUE Framework is not an ideal instrument to measure either the quality or the process of medical interviews.

CONCLUSION

Therefore, we propose a new method of contextualized assessment that builds on the SEGUE Framework. Our system evaluates discrete interviewing behaviors within the context of an ambulatory medical interview. We describe our interview structure, as well as a new instrument (the Wy-Mii, pronounced "why me"), to assess both communication and interpersonal skills. We expect that our new method of contextualized assessment will better differentiate between beginning and advanced levels of medical interviewing skills for residents.