Assessment of clinical competence of medical students using the objective structured clinical examination: first 2 years' experience in Taipei Veterans General Hospital

Medical school grades may predict future clinical competence

BACKGROUND

In real-world medical education, there is a lack of reliable predictors of future clinical competencies. Hence, we aim to identify the factors associated with clinical competencies and construct a prediction model to identify "improvement required" trainees.

METHODS

We analyzed data from medical students who graduated from National Yang-Ming University with clerkship training and participated in the postgraduate year (PGY) interview at Taipei Veterans General Hospital. Clinical competencies were evaluated using grades of national objective structured clinical examination (OSCEs). This study used data from medical students who graduated in July 2018 as the derivation cohort (N = 50) and those who graduated in July 2020 (n = 56) for validation.

RESULTS

Medical school grades were associated with the performance of national OSCEs (Pearson r = 0.34, p = 0.017), but the grades of the structured PGY interviews were marginally associated with the national OSCE (Pearson r = 0.268, p = 0.06). A prediction model was constructed to identify "improvement required" trainees, defined: trainees with the lowest 25% of scores in the national OSCEs. According to this model, trainees with the lowest 25% medical school grades predicted a higher risk of the "improvement required" clinical performance (Q1-Q3 vs Q4 = 15% vs 60%, odds ratio = 8.5 [95% confidence interval = 1.8-39.4], p = 0.029). In the validation cohort, our prediction model could accurately classify 76.7% "improvement required" and "nonimprovement required" students.

CONCLUSION

Our study suggests that interventions for students with unsatisfactory medical school grades are warranted to improve their clinical competencies.

Comparison of OSCE performance between 6- and 7-year medical school curricula in Taiwan

BACKGROUND

The year 2013 marks a watershed in the history of medical education in Taiwan. Following Taiwan's Taskforce of Medical School Curriculum Reform recommendations, the medical school curriculum was reduced from 7 to 6 years. This study aimed to analyze the impact of medical school curriculum reform on medical students' performance in objective structured clinical examinations (OSCEs).

METHODS

We retrospectively analyzed the OSCE records at Taipei Veterans General Hospital (Taipei VGH), one of Taiwan's largest tertiary medical centers, between November 2016 and July 2020. The eligibility criteria were medical students receiving a full one-year clinical sub-internship training at Taipei VGH and in their last year of medical school. All medical students received a mock OSCE-1 at the beginning of their sub-internship, a mock OSCE-2 after six months of training, and a national OSCE at the end of their sub-internship. The parameters for performance in OSCEs included "percentage of scores above the qualification standard" and "percentage of qualified stations."

RESULTS

Between November 2016 and July 2020, 361 undergraduates underwent clinical sub-internship training at Taipei VGH. Among them, 218 were taught under the 7-year curriculum, and 143 were instructed under the 6-year curriculum. Based on baseline-adjusted ANCOVA results, medical students under the 7-year curriculum had a higher percentage of scores above the qualification standard than those under the 6-year curriculum at the mock OSCE-1 (7-year curriculum vs. 6-year curriculum: 33.8% [95% CI 32.0-35.7] vs. 28.2% [95% CI 25.9-30.4], p < 0.001), and mock OSCE-2 (7-year curriculum vs. 6-year curriculum: 89.4% [95% CI 87.4-91.4] vs. 84.0% [95% CI 81.5-86.4], p = 0.001). Moreover, medical students in the 7-year curriculum had a higher percentage of qualified stations in mock OSCE-1 (7-year curriculum vs. 6-year curriculum: 89.4% [95% CI 87.4-91.4] vs. 84.0% [95% CI 81.5-86.4], p = 0.001) and mock OSCE-2 (7-year curriculum vs. 6-year curriculum: 91.9% [95% CI 90.1-93.8] vs. 86.1% [95% CI 83.8-88.3], p = 0.001). After clinical sub-internship training, there were no differences in the percentage of scores above the qualification standard (7-year curriculum vs. 6-year curriculum: 33.5% [95% CI 32.2-34.9] vs. 34.6 [95% CI 32.9-36.3], p = 0.328) and percentage of qualified stations (7-year curriculum vs. 6-year curriculum: 89.4% [95% CI 88.1-90.7] vs. 90.2% [95% CI 88.6-91.8], p = 0.492).

CONCLUSIONS

At the beginning of the sub-internship, medical students under the 7-year curriculum had better OSCE performance than those under the 6-year curriculum. After the clinical sub-internship training in Taipei VGH, there was no difference in the national OSCE score between the 6- and 7-year curricula. Our study suggests that clinical sub-internship is crucial for the development of clinical skills and performance in the national OSCE.

The Impact of COVID-19 Pandemic on the Learning Outcomes of Medical Students in Taiwan: A Two-Year Prospective Cohort Study of OSCE Performance

Background/Aims: To avoid the negative impacts of the COVID-19 pandemic on clinical clerkship, supplemental teachings such as digital materials in the scenario-based distal simulations were implemented. This study utilized the OSCE (objective-structured clinical examination) to evaluate the impact of COVID-19 pandemic on the learning outcome of medical students from the regular group (class of 2020) and pandemic-impacted group (class of 2021). Methods: All medical students serially took, firstly, the mock-OSCE, secondly, the mock-OSCE, and the national OSCE. Then, the serial OSCE scores were compared between groups. Results: Although with similar scores in the first mock OSCE, the regular group (n = 78) had a higher average score in the national OSCE than the pandemic-impacted group (n = 80) (872.18 vs. 834.96, p = 0.003). In terms of improvement, the performances of the regular group were also better than the pandemic-impacted group between the second mock OSCE and the national OSCE (79.10 vs. 38.14, p = 0.014), and between the second mock OSCE and the national OSCE (125.11 vs. 77.52, p = 0.003). While separating distinct genres, the regular group had more of a score increment in standardized patient-based stations between the second mock OSCE and the national OSCE (regular vs. pandemic-impacted: 57.03 vs. 18.95, p = 0.003), as well as between the first mock OSCE and the national OSCE (75.97 vs. 26.36, p < 0.001), but there was no significant difference among the skill-based stations. In particular, the scores of the emergency medicine associated station in the national OSCE of the pandemic-impacted group was lower. Conclusions: Our study implies that the pandemic significantly hampered the learning outcomes of final year medical students in their clinical participation. Especially facing the COVID-19 pandemic, more supplemental teachings are needed to compensate the decreasing emergency medicine exposure.

Self-perception of competences in clinical practice among recently graduated physicians from Lima, Peru

Objective

To describe the self-perception of basic competencies in clinical practice and evaluate their associated factors, among recently graduated physicians from Lima, Peru.

Methods

Cross-sectional study. We evaluated the self-perception of the competencies in recently graduated physicians of four dimensions of the Tuning Project. Each item had six possible responses on a Likert scale: "non-existent" (1 point), "insufficient" (2 points), "sufficient" (3 points), "good" (4 points), "very good" (5 points) and "excellent" (6 points). To evaluate associated factors of the average scores for each dimension, we used linear regressions with the bootstrap method.

Results

We analyzed data from 425 (54.9% were between 22 and 25 years old), which represent 31.1% of all physicians who graduated in 2016 from all medical schools located in Lima. The average self-perception score of the assessed dimensions was, in descending order: 4.49 for carrying out a patient consultation with a patient; 4.13 for carrying out practical procedures; 4.12 for providing immediate care of medical emergencies; and 4.04. for applying the principles, skills, and knowledge of evidence-based medicine (EBM). Regarding the factors associated with the average score per dimension, physicians from one university had higher average scores in all dimensions, and having done an externship and done an internship at social security hospitals was associated with a higer score with self-perception in two dimensions.

Conclusion

Self-perception of competence was greater for the patient consultation dimension, and lower for the EBM. Only physicians from one university had higher average scores in all dimensions.

Evaluation of Communication Skills Among Physicians: A Systematic Review of Existing Assessment Tools

OBJECTIVE

The importance of physician training in communication skills for motivating patients to adopt a healthy life-style and optimize clinical outcomes is increasingly recognized. This study inventoried and systematically reviewed the psychometric properties of, and the skills assessed by, existing assessment tools used to evaluate communication skills among physicians.

METHODS

This review was conducted in accordance with the PRISMA guidelines (PROSPERO: CRD42018091932). Four databases (PUBMED, EMBASE, PsychINFO, and SCOPUS) were searched up to December 2018, generating 3902 unique articles, which were screened by two authors. A total of 57 articles met the inclusion criteria and underwent full data extraction.

RESULTS

Forty-five different assessment tools were identified. Only 47% of the studies mentioned underlying theories or models for designing the tool. Fifteen communication skills were assessed across the tools, the five most prevalent were information giving (46%) or gathering (40%), eliciting patients' perspectives (44%), planning/goal setting (37%), and closing the session (32%). Most tools (93%) assessed communication skills using in-person role play exercises with standardized (61%) or real (32%) patients, but only 54% described the expertise of the raters who performed the evaluations. Overall, reporting of the psychometric properties of the assessment tools was poor-moderate (4.5 ± 1.3 out of 9).

CONCLUSIONS

Despite identifying several existing physician communication assessment tools, a high degree of heterogeneity between these tools, in terms of skills assessed and study quality, was observed, and most have been poorly validated. Research is needed to rigorously develop and validate accessible, convenient, "user-friendly," and easy to administer and score communication assessment tools.

Reliability and validity of OSCE checklists used to assess the communication skills of undergraduate medical students: A systematic review

OBJECTIVES

To explore inter-rater agreement between reviewers comparing reliability and validity of checklist forms that claim to assess the communication skills of undergraduate medical students in Objective Structured Clinical Examinations (OSCEs).

METHODS

Papers explaining rubrics of OSCE checklist forms were identified from Pubmed, Embase, PsycINFO, and the ProQuest Education Databases up to 2013. Included were those studies that report empirical validity or reliability values for the communication skills assessment checklists used. Excluded were those papers that did not report reliability or validity.

RESULTS

Papers focusing on generic communication skills, history taking, physician-patient communication, interviewing, negotiating treatment, information giving, empathy and 18 other domains (ICC -0.12-1) were identified. Regarding the validity and reliability of the communication skills checklists, agreement between reviewers was 0.45.

CONCLUSIONS

Heterogeneity in the rubrics used in the assessment of communication skills and a lack of agreement between reviewers makes comparison of student competences within and across institutions difficult.

PRACTICE IMPLICATIONS

Consideration should be afforded to the adoption of a standardized measurement instrument to assess communication skills in undergraduate medical education. Future research will focus upon evaluating the potential impact of adoption of a standardized measurement instrument.

The assessment of undergraduate medical students' satisfaction levels with the objective structured clinical examination

Background:

The objective structured clinical examination (OSCE) has been introduced as an efficient method for the assessment of medical students.

Objectives:

The aim of the present study was to determine the satisfaction level of undergraduate medical students of internal medicine department with the OSCE.

Materials and Methods:

This was a descriptive cross-sectional study, performed on all available undergraduate students at the end of their internal medicine training period in Mashhad University of Medical Sciences. The students responded to 15 multiple-choice questions with confirmed validity and reliability.

Results:

The majority of the students (94.5%) had a positive attitude toward the OSCE and mentioned that the OSCE format was a more appropriate type of exam than other methods of testing; however, 79.1% thought that the OSCE format was stressful. In addition, the participants’ sex had no effect on their level of satisfaction with the examination. Likewise, there was no significant correlation between their level of satisfaction and their age, marital status, or lack of previous experience with this type of exam.

Conclusions:

If the exam standards are met and a uniform dispersion of the scientific content is maintained, the OSCE method of assessment can be recommended as an efficient and applicable method for assessing medical students.

Assessment of first-year post-graduate residents: usefulness of multiple tools

BACKGROUND

Objective Structural Clinical Examination (OSCE) usually needs a large number of stations with long test time, which usually exceeds the resources available in a medical center. We aimed to determine the reliability of a combination of Direct Observation of Procedural Skills (DOPS), Internal Medicine in-Training Examination (IM-ITE(®)) and OSCE, and to verify the correlation between the small-scale OSCE+DOPS+IM-ITE(®)-composited scores and 360-degree evaluation scores of first year post-graduate (PGY(1)) residents.

METHODS

Between 2007 January to 2010 January, two hundred and nine internal medicine PGY1 residents completed DOPS, IM-ITE(®) and small-scale OSCE at our hospital. Faculty members completed 12-item 360-degree evaluation for each of the PGY(1) residents regularly.

RESULTS

The small-scale OSCE scores correlated well with the 360-degree evaluation scores (r = 0.37, p < 0.021). Interestingly, the addition of DOPS scores to small-scale OSCE scores [small-scale OSCE+DOPS-composited scores] increased it's correlation with 360-degree evaluation scores of PGY(1) residents (r = 0.72, p < 0.036). Further, combination of IM-ITE(®) score with small-scale OSCE+DOPS scores [small-scale OSCE+DOPS+IM-ITE(®)-composited scores] markedly enhanced their correlation with 360-degree evaluation scores (r = 0.85, p < 0.016).

CONCLUSION

The strong correlations between 360-degree evaluation and small-scale OSCE+DOPS+IM-ITE(®)-composited scores suggested that both methods were measuring the same quality. Our results showed that the small-scale OSCE, when associated with both the DOPS and IM-ITE(®), could be an important assessment method for PGY(1) residents.