Assessment of clinical skills with standardized patients: state of the art revisited

Longitudinal variation of correlations between different components of assessment within a medical school

BACKGROUND

An assessment program should be inclusive and ensure that the various components of medical knowledge, clinical skills, and professionalism are assessed. The level and the variation over time in the strength of the correlation between these components of assessment is still a matter of study. Based on the meaningful learning theory and the integrated learning theory, we hypothesize that these components increase their connections during the medical school course.

METHODS

This is a retrospective cohort study that analyzed data collected for a 10-year period in one medical school. We included students from the 3rd to 6th year of medical school from 2011 to 2021. Three assessment components were addressed: Knowledge, Clinical Skills, and Professionalism. For data analysis, Pearson correlation coefficients (R) and R2 were calculated to study the correlation between variables and a z-test on Fisher's r-to-z was used to determine the differences between correlation coefficients.

RESULTS

949 medical students were included in the study. The correlation between Clinical Skills and Professionalism showed a medium to strong association (Pearson's R ranging from 0.485 to 0.734), while the correlation between Knowledge and Professionalism was weaker but exhibited a steady evolution with Pearson's R fluctuating between 0.075 and 0.218. The Knowledge and Clinical Skills correlation became statistically significant from 2013 onwards and peaking at Pearson's R of 0.440 for the cohort spanning 2016-2019. We also revealed a strengthening of correlations between Professionalism and Clinical Skills from the beginning to the end of clinical training, but not with the knowledge component.

CONCLUSIONS

This analysis contributes to our understanding of the dynamics of correlations of different assessment components within an institution and provides a framework for how they interact and influence each other.

TRIAL REGISTRATION

This study was not a clinical trial, but a retrospective observational study, without health care interventions. Nevertheless, we provide herein the number of the study as submitted to the Ethics committee - CEICVS 146/2021.

OSABSS: An authentic examination for assessing basic surgical skills in surgical residents

Objectives

This study aimed to develop and validate the OSABSS (Objective Structured Assessment of Basic Surgical Skills), a modified Objective Structured Clinical Examination (OSCE), to assess basic surgical skills in residents.

Design

A developmental study conducted in two phases. Basic skills were identified through literature review and gap analysis. The OSABSS was then designed as a modified OSCE.

Setting

This study took place at Alborz University of Medical Sciences in Iran.

Interventions

The OSABSS was created using Harden's OSCE (Objective Structured Clinical Examination) methodology. Scenarios, checklists, and station configurations were developed through expert panels. The exam was piloted and implemented with residents as participants and faculty as evaluators.

Participants

32 surgical residents in gynecology, general surgery, orthopedics, and neurosurgery participated. 22 faculty members were evaluators.

Primary and secondary outcome measures

The primary outcome was OSABSS exam scores. Secondary outcomes were written exam scores, and national residency entrance ranks.

Main results

The mean OSABSS score was 16.59 ± 0.19 across all stations. Criterion validity was demonstrated through correlations between OSABSS scores, written scores and entrance ranks. Reliability was high, with a Cronbach's alpha of 0.87. No significant inter-rater score differences were found.

Conclusions

The rigorous OSABSS development process produced an exam demonstrating strong validity and reliability for assessing basic surgical skills. The comprehensive station variety evaluates diverse technical and non-technical competencies. Further research should expand participant samples across surgical disciplines.

Intercultural effectiveness in GPs' communication and clinical assessment: An experimental study

OBJECTIVE

This study aimed to investigate potential disparities in general practitioners' overall communication and clinical assessments based on patient ethnicity, while examining the influence of intercultural effectiveness.

METHODS

Employing a 2 × 2 experimental study design, online video recorded consultations with simulated patients were conducted and analyzed using OSCEs. Each GP (N = 100) completed a consultation with both an ethnic majority and an ethnic minority patient. Additionally, a follow-up survey was administered to gather supplementary data. Paired sample t-tests explored ethnic disparities, correlation and regression analyses determined associations with intercultural attitudes, traits and capabilities.

RESULTS

No statistically significant differences in GPs' communication or clinical assessment were found based on patients' ethnic background. Positive associations were observed between all aspects of intercultural effectiveness and GPs' consultation behavior. Intercultural traits emerged as a strong and robust predictor of clinical assessment of ethnic minority patients.

CONCLUSION

Intercultural traits, such as ethnocultural empathy, may play a critical role in GPs' clinical assessment skills during intercultural consultations.

PRACTICE IMPLICATIONS

Findings provide valuable insights into the determinants of intercultural effectiveness in healthcare, fostering promising targets for interventions and training programs aiming to ensure higher-quality and more equitable care delivery.

A realist evaluation of how, why and when objective structured clinical exams (OSCEs) are experienced as an authentic assessment of clinical preparedness

INTRODUCTION

Whilst rarely researched, the authenticity with which Objective Structured Clinical Exams (OSCEs) simulate practice is arguably critical to making valid judgements about candidates' preparedness to progress in their training. We studied how and why an OSCE gave rise to different experiences of authenticity for different participants under different circumstances.

METHODS

We used Realist evaluation, collecting data through interviews/focus groups from participants across four UK medical schools who participated in an OSCE which aimed to enhance authenticity.

RESULTS

Several features of OSCE stations (realistic, complex, complete cases, sufficient time, autonomy, props, guidelines, limited examiner interaction etc) combined to enable students to project into their future roles, judge and integrate information, consider their actions and act naturally. When this occurred, their performances felt like an authentic representation of their clinical practice. This didn't work all the time: focusing on unavoidable differences with practice, incongruous features, anxiety and preoccupation with examiners' expectations sometimes disrupted immersion, producing inauthenticity.

CONCLUSIONS

The perception of authenticity in OSCEs appears to originate from an interaction of station design with individual preferences and contextual expectations. Whilst tentatively suggesting ways to promote authenticity, more understanding is needed of candidates' interaction with simulation and scenario immersion in summative assessment.

The 7 Elements Communication Rating Form for Teaching and Assessing Medical Communication Competency

INTRODUCTION

Medical communication skills are a critical component of clinical medicine and patient satisfaction. Communication skills are difficult to teach and evaluate, necessitating tools that are effective and efficient. This study presents and validates the 7 Elements Communication Rating Form (7E-CRF), a streamlined, dual-purpose, evidence-based medical communication checklist that functions as a teaching and assessment tool.

METHOD

A 14-item teaching and assessment tool is described and validated using face, concurrent, and predictive validity indices. The study was conducted with 661 medical students from the West Virginia School of Osteopathic Medicine (WVSOM). Student performance was assessed in year 1 labs, year 2 labs, and year 2 and year 3 objective structured clinical examination (OSCE). These internal indices were compared with student performance on the Humanistic Domain of the Comprehensive Osteopathic Medical Licensing Examination (COMLEX) Level 2-Performance Evaluation (PE), a licensure exam previously taken in years 3 or 4 of osteopathic medical schools.

RESULTS

The evidence of interrater reliability and predictive validity is strong. Data from the 7E-CRF is compared to performance on the COMLEX Level 2-PE, Humanistic Domain. The 7E-CRF can identify students who are at a 10-fold increased risk of failure on the COMLEX Level 2-PE Humanistic Domain.  Conclusions: The 7E-CRF integrates instruction and assessment, based on a national and international model. The simplicity, foundation in professional consensus, ease of use, and predictive efficacy make the 7E-CRF a highly valuable instrument for medical schools in teaching and evaluating competency in medical communication skills.

Learning curve of ultrasound-guided surgeon-administered transversus abdominis plane (UGSA-TAP) block on a porcine model

Background

Surgeons commonly perform ultrasound-guided Transversus Abdominis Plane blocks to manage acute pain following abdominal surgeries. There is no consensus on whether surgeons should undergo basic hands-on training to perform TAP blocks or if video-based learning is sufficient. We theorized that simulation-based learning is superior to video-based learning. In the present study, we present the analysis of technical skills of UGSA-TAP block performance on a live porcine model by general surgery trainees after undergoing video or simulation-based learning.

Methods

We performed a prospective, double-blinded, randomized study. Ten surgery residents and two surgery critical-care fellows (n = 12) without prior experience in performing the TAP block were recruited. The participants were randomized either into a video-based or simulation-based training group. After that, all participants performed a TAP block on a live anesthetized pig, which was recorded and scored by three blinded anesthesiologists. All participants completed a post-performance survey to assess their confidence in gaining competency in the UGSA-TAP block. Statistical analyses were performed to assess the differences between the two groups. P < 0.05 was considered statistically significant.

Results

All simulation-based learning participants successfully performed a survey scan, identified the three muscular layers of the abdominal wall, and identified the transversus abdominis plane compared to 50 %, 50 %, and 33 % video-based learning group participants for the respective parameters (p < 0.05). While some performance metrics showed no statistically significant differences between the groups, substantial effect sizes (Cohen's ℎ up to 1.07) highlighted notable differences in participants' performance. Both groups exhibited confidence in core competencies, with varied rates of satisfactory skill execution. Performance assessed using a global rating scale revealed a higher passing rate for the simulation group (83 % vs. 33 %). Participant feedback via the Likert scale reflected confidence post-training. Inter-rater reliability (0.83-1) confirmed the robustness of study evaluations.

Conclusion

The UGSA-TAP block curriculum should be introduced into the surgical residency programs with an emphasis on simulation-based learning to enhance the procedural skills of the trainees before transitioning to surgical patients.

Quality in screening and measuring blood pressure in China's primary health care: a national cross-sectional study using unannounced standardized patients

Background

This study aims to evaluate primary care providers' adherence to the standard of measuring blood pressure for people aged 35 or above during their initial visit, as per Chinese guidelines, and to identify factors affecting their practices.

Methods

We developed 11 standardized patients (SP) cases as tracer conditions to evaluate primary care, and deployed trained SPs for unannounced visits to randomly selected providers in seven provinces of China. The SPs used a checklist based on guidelines to record whether and how blood pressure was measured. Data were analyzed descriptively and regression analysis was performed to examine the association between outcomes and factors such as provider, patient, facility, and clinical case characteristics.

Findings

The SPs conducted 1201 visits and found that less than one-third of USPs ≥35 had their blood pressure measured. Only 26.9% of migraine and 15.4% of diabetes cases received blood pressure measurements. Additionally, these measurements did not follow the proper guidelines and recommended steps. On average, 55.6% of the steps were followed with few providers considering influencing factors before measurement and only 6.0% of patients received both-arm measurements. The use of wrist sphygmomanometers was associated with poor blood pressure measurement.

Interpretation

In China, primary care hypertension screening practices fall short of guidelines, with infrequent initiation of blood pressure measurements and inadequate adherence to proper measurement steps. To address this, priority should be placed on adopting, implementing, and upholding guidelines for hypertension screening and measurement.

Funding

National Natural Science Foundation of China, Swiss Agency for Development and Cooperation, Doctoral Fund Project of Inner Mongolia Medical University, China Postdoctoral Science Foundation.

Structures, processes and outcomes of objective structured clinical examinations in dental education during the COVID-19 pandemic: A scoping review

INTRODUCTION

Objective structured clinical examinations (OSCEs) are an essential examination tool within undergraduate dental education. Fear of spread of the COVID-19 virus led to dental institutions exploring alternative means of conducting OSCEs. The aim of this scoping review was to investigate what structures, processes and outcomes of dental OSCEs were reported during the COVID-19 pandemic.

MATERIALS AND METHODS

This scoping review was conducted and reported adhering to the Preferred Reporting Items for Systematic Reviews and Meta-analyses extension for scoping review guidelines (PRISMA-ScR). Published literature was identified through a systematic search of PubMed, Embase, Cumulative Index to Nursing and Allied Health Literature (CINAHL), Education Resources Information Center (Eric), ProQuest and Google Scholar. Identified articles were independently reviewed by two authors (KS, AD), followed by synthesis in terms of the reported structures, processes and outcomes. Articles reporting cancellation or rescheduling were also included, extracting data on reasons and any suggestions/recommendations.

RESULTS

The search yielded a total of 290 studies of which 239 sources were excluded after removal of duplicates, leaving 51 studies for title and abstract evaluation. Thirty-four articles were excluded as they did not report on the topic of interest, leaving 17 for full-text evaluation, of which nine were analysed according to the pre-set themes. All dental OSCEs taking place (n = 6) were conducted online whilst the remaining (n = 3) were either cancelled or rescheduled. Data on structures reported specific online videoconferencing software used and provision of staff and student training. Processes on the execution of online OSCEs varied significantly from one study to the other, providing rich data on how dental institutions may carry out such assessments tailored to their need. Information regarding outcomes was sparse, as little attention was paid to the results of the students compared to pre-pandemic, lacking investigation into reliability and validity of online dental OSCEs.

CONCLUSION

Dental OSCEs could be conducted online implementing well-planned structures and processes; however, further evidence is needed to prove its reliability and validity based on outcomes. Dental institutions may need to consider alternative methods to assess practical competencies if online OSCEs are to take place.

Diagnosing virtual patients: the interplay between knowledge and diagnostic activities

Clinical reasoning theories agree that knowledge and the diagnostic process are associated with diagnostic success. However, the exact contributions of these components of clinical reasoning to diagnostic success remain unclear. This is particularly the case when operationalizing the diagnostic process with diagnostic activities (i.e., teachable practices that generate knowledge). Therefore, we conducted a study investigating to what extent knowledge and diagnostic activities uniquely explain variance in diagnostic success with virtual patients among medical students. The sample consisted of N = 106 medical students in their third to fifth year of university studies in Germany (6-years curriculum). Participants completed professional knowledge tests before diagnosing virtual patients. Diagnostic success with the virtual patients was assessed with diagnostic accuracy as well as a comprehensive diagnostic score to answer the call for more extensive measurement of clinical reasoning outcomes. The three diagnostic activities hypothesis generation, evidence generation, and evidence evaluation were tracked. Professional knowledge predicted performance in terms of the comprehensive diagnostic score and displayed a small association with diagnostic accuracy. Diagnostic activities predicted comprehensive diagnostic score and diagnostic accuracy. Hierarchical regressions showed that the diagnostic activities made a unique contribution to diagnostic success, even when knowledge was taken into account. Our results support the argument that the diagnostic process is more than an embodiment of knowledge and explains variance in diagnostic success over and above knowledge. We discuss possible mechanisms explaining this finding.

Enhancing Nonverbal Communication Through Virtual Human Technology: Protocol for a Mixed Methods Study

BACKGROUND

Communication is a critical component of the patient-provider relationship; however, limited research exists on the role of nonverbal communication. Virtual human training is an informatics-based educational strategy that offers various benefits in communication skill training directed at providers. Recent informatics-based interventions aimed at improving communication have mainly focused on verbal communication, yet research is needed to better understand how virtual humans can improve verbal and nonverbal communication and further elucidate the patient-provider dyad.

OBJECTIVE

The purpose of this study is to enhance a conceptual model that incorporates technology to examine verbal and nonverbal components of communication and develop a nonverbal assessment that will be included in the virtual simulation for further testing.

METHODS

This study will consist of a multistage mixed methods design, including convergent and exploratory sequential components. A convergent mixed methods study will be conducted to examine the mediating effects of nonverbal communication. Quantitative (eg, MPathic game scores, Kinect nonverbal data, objective structured clinical examination communication score, and Roter Interaction Analysis System and Facial Action Coding System coding of video) and qualitative data (eg, video recordings of MPathic-virtual reality [VR] interventions and student reflections) will be collected simultaneously. Data will be merged to determine the most crucial components of nonverbal behavior in human-computer interaction. An exploratory sequential design will proceed, consisting of a grounded theory qualitative phase. Using theoretical, purposeful sampling, interviews will be conducted with oncology providers probing intentional nonverbal behaviors. The qualitative findings will aid the development of a nonverbal communication model that will be included in a virtual human. The subsequent quantitative strand will incorporate and validate a new automated nonverbal communication behavior assessment into the virtual human simulation, MPathic-VR, by assessing interrater reliability, code interactions, and dyadic data analysis by comparing Kinect responses (system recorded) to manually scored records for specific nonverbal behaviors. Data will be integrated using building integration to develop the automated nonverbal communication behavior assessment and conduct a quality check of these nonverbal features.

RESULTS

Secondary data from the MPathic-VR randomized controlled trial data set (210 medical students and 840 video recordings of interactions) were analyzed in the first part of this study. Results showed differential experiences by performance in the intervention group. Following the analysis of the convergent design, participants consisting of medical providers (n=30) will be recruited for the qualitative phase of the subsequent exploratory sequential design. We plan to complete data collection by July 2023 to analyze and integrate these findings.

CONCLUSIONS

The results from this study contribute to the improvement of patient-provider communication, both verbal and nonverbal, including the dissemination of health information and health outcomes for patients. Further, this research aims to transfer to various topical areas, including medication safety, informed consent processes, patient instructions, and treatment adherence between patients and providers.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/46601.

Measuring antenatal counseling skill with a milestone-based assessment tool: a validation study

BACKGROUND

Antenatal counseling for parents in the setting of expected preterm delivery is an important component of pediatric training. However, healthcare professionals receive a variable amount and quality of formal training. This study evaluated and discussed validity of a practical tool to assess antenatal counseling skills and provide evaluative feedback: the Antenatal Counseling Milestones Scale (ACoMS).

METHODS

Experts in antenatal counseling developed an anchored milestone-based tool to evaluate observable skills. Study participants with a range of antenatal counseling skills were recruited to participate in simulation of counseling sessions in person or via video with standardized patient actors presenting with preterm labor at 23 weeks' gestation. Two faculty observers scored each session independently using the ACoMS. Participants completed an ACoMS self-assessment, demographic, and feedback survey. Validity was measured with weighted kappas for inter-rater agreement, Kruskal-Wallis and Dunn's tests for milestone levels between degrees of expertise in counseling, and cronbach's alpha for item consistency.

RESULTS

Forty-two participants completed observed counseling sessions. Of the 17 items included in the tool, 15 items were statistically significant with scores scaling with level of training. A majority of elements had fair-moderate agreement between raters, and there was high internal consistency amongst all items.

CONCLUSION

This study demonstrates that the internal structure of the ACoMS rubric has greater than fair inter-rater reliability and high internal consistency amongst items. Content validity is supported by the scale's ability to discern level of training. Application of the ACoMS to clinical encounters is needed to determine utility in clinical practice.

Multidimensional Evaluation of Screening Brief Intervention and Referral to Treatment Training for Medical Students

OBJECTIVE

Screening, Brief Intervention, and Referral to Treatment (SBIRT) is an evidence-based approach to identifying and addressing alcohol use in non-specialty settings. Many medical schools teach SBIRT, but most published evaluations of these efforts exclude rigorous skill assessments and teaching methods.

METHODS

During the 2017-2018 academic year, 146 third-year medical students received classroom-based learning on SBIRT and motivational interviewing (MI) and at least two SBIRT practices with feedback as part of a 4-week psychiatry clerkship. The objective of this curriculum was to improve SBIRT knowledge, attitudes, and confidence and enable learners to skillfully deliver SBIRT. Outcomes evaluated included satisfaction, knowledge, attitudes and confidence, and clinical skill in delivering SBIRT to a standardized patient (rated by the actor, as well as an expert).

RESULTS

Results indicated acceptable satisfaction at post-curriculum and significant improvements in attitudes and knowledge from pre- to post-curriculum. On the clinical skills exam, all students were rated as having mastered at least 80% of SBIRT elements by standardized patients and 91.8% were rated at this level by a faculty expert. Student attitudes and knowledge were unrelated to expert ratings, and standardized patient ratings had limited associations with expert ratings.

CONCLUSIONS

These results suggest curriculum objectives were achieved and provide unique contributions to the SBIRT curricular outcome research for healthcare trainees. Other findings included that trainee knowledge and confidence may not relate to skill, and standardized patient feedback provides different information on SBIRT and MI skill than expert ratings.

Objective structured clinical examination versus traditional written examinations: a prospective observational study

BACKGROUND

Recently, Objective Structured Clinical Examinations (OSCE) became an official evaluation modality for 6-year medical students in France. Before, standard examination modalities were: written progressive clinical cases (PCC), written critical reading of scientific articles (CRA), and internship evaluation (IE). The aim of this study was to assess the performances of 6-year medical students in their final faculty tests by comparing OSCE-exams with standard examination modalities.

METHODS

This was a prospective observational study. We included all 6-year medical students in our university from 2020 to 2021. The endpoints were the scores obtained at the following final faculty tests during the 6^th year of medical studies: OSCE-training, OSCE-exams, written PCC, written CRA, and IE. All scores were compared in a paired-analysis.

RESULTS

A total of 400 students were included in the study. No student was excluded in the final analysis. The mean scores obtained at the OSCE-exams were significantly different from those obtained at OSCE-training, PCC, CRA, and IE (12.6 ± 1.7, 11.7 ± 1.7, 13.4 ± 1.4, 13.2 ± 1.5, 14.7 ± 0.9, respectively; p < 0.001). OSCE-exams scores were moderately and significantly correlated with OSCE-training and PCC (Spearman rho coefficient = 0.4, p < 0.001); OSCE examination scores were lowly but significantly correlated with CRA and IE (Spearman rho coefficient = 0.3, p < 0.001). OSCE-scores significantly increased after an OSCE training session.

CONCLUSION

In our faculty, 6-year medical students obtained lower scores at OSCE exams compared to other standard evaluation modalities. The correlation was weak to moderate but significant. These results suggest that OSCE are not redundant with the other evaluation modalities. Interestingly, a single OSCE training session led to an improvement in OSCE scores underlining the importance of a specific training.

OSCE Standard Setting: Three Borderline Group Methods

High-stakes assessments must discriminate between examinees who are sufficiently competent to practice in the health professions and examinees who are not. In these settings, criterion-referenced standard-setting methods are strongly preferred over norm referenced methods. While there are many criterion-referenced options, few are feasible or cost effective for objective structured clinical examinations (OSCEs). The human and financial resources required to organize OSCEs alone are often significant, leaving little in an institution's budget for additional resource-intensive standard-setting methods. The modified borderline group method introduced by Dauphinee et al. for a large-scale, multi-site OSCE is a very feasible option but is not as defensible for smaller scale OSCEs. This study compared the modified borderline group method to two adaptations that address its limitations for smaller scale OSCEs while retaining its benefits, namely feasibility. We evaluated decision accuracy and consistency of calculated cut scores derived from (1) modified, (2) regression-based, and (3) 4-facet Rasch model borderline group methods. Data were from a 12-station OSCE that assessed 112 nurses for entry to practice in a Canadian context. The three cut scores (64-65%) all met acceptable standards of accuracy and consistency; however, the modified borderline group method was the most influenced by lower scores within the borderline group, leading to the lowest cut score. The two adaptations may be more defensible than modified BGM in the context of a smaller (n < 100-150) OSCE.

Memory, credibility and insight: How video-based feedback promotes deeper reflection and learning in objective structured clinical exams

INTRODUCTION

Providing high-quality feedback from Objective Structured Clinical Exams (OSCEs) is important but challenging. Whilst prior research suggests that video-based feedback (VbF), where students review their own performances alongside usual examiner feedback, may usefully enhance verbal or written feedback, little is known about how students experience or interact with VbF or what mechanisms may underly any such benefits.

METHODS

We used social constructive grounded theory to explore students' interaction with VbF. Within semi-structured interviews, students reviewed their verbal feedback from examiners before watching a video of the same performance, reflecting with the interviewer before and after the video. Transcribed interviews were analysed using grounded theory analysis methods.

RESULT

Videos greatly enhanced students' memories of their performance, which increased their receptivity to and the credibility of examiners' feedback. Reflecting on video performances produced novel insights for students beyond the points described by examiners. Students triangulated these novel insights with their own self-assessment and experiences from practice to reflect deeply on their performance which led to the generation of additional, often patient-orientated, learning objectives.

CONCLUSIONS

The array of beneficial mechanisms evoked by VbF suggests it may be a powerful means to richly support students' learning in both formative and summative contexts.

A video-based situational judgement test of medical students' communication competence in patient encounters: Development and first evaluation

OBJECTIVE

We developed and evaluated the Video-Based Assessment of Medical Communication Competence (VA-MeCo), a construct-driven situational judgement test measuring medical students' communication competence in patient encounters.

METHODS

In the construction phase, we conducted two expert studies (n_panel1 = 6, n_panel2 = 13) to ensure curricular and content validity and sufficient expert agreement on the answer key. In the evaluation phase, we conducted a cognitive pre-test (n = 12) and a pilot study (n = 117) with medical students to evaluate test usability and acceptance, item statistics and test reliability depending on the applied scoring method (raw consensus vs. pairwise comparison scoring).

RESULTS

The results of the expert interviews indicated good curricular and content validity. Expert agreement on the answer key was high (ICCs> .86). The pilot study showed favourable usability and acceptance by students. Irrespective of the scoring method, reliability for the complete test (Cronbach's α >.93) and its subscales (α >.83) was high.

CONCLUSION

There is promising evidence that medical communication competence can be validly and reliably measured using a construct-driven and video-based situational judgement test.

PRACTICE IMPLICATIONS

Video-based SJTs allow efficient online assessment of medical communication competence and are well accepted by students and educators.

Patient-Tailored 3D-Printing Models in the Subspecialty Training of Spinal Tumors: A Comparative Study and Questionnaire Survey

BACKGROUND

Training in the subspecialty of spinal tumors is challenging and less researched. The anatomic variations and complex relationship with paraspinal structures tend to be the main obstacle for the trainees in this field. Three-dimensional (3D)-printing technique has the advantage of individual customization and high fidelity, and can produce case-tailored models as auxiliary tools in medical training.

METHODS

The main parts of the study included case-based lectures with tailored 3D-printing models, evaluating their performances in a controlled examination and anonymous questionnaire survey regarding the trainees' opinion towards the tailored models. The examination was designed as case-based clinical analysis. All trainees were randomly allocated to the study group and control group, and the former group was additively provided a case-tailored model.

RESULTS

Thirty-six participants were recruited in this study, including 16 residents and 20 fellows. In the section of examination, there was significant difference in the aspects of describing the involvement of paraspinal structures and discriminating the relationship between the tumor and large vessels (P < 0.05), but similar in the aspects of surgical planning and relevant complications (P > 0.05). In the survey, most participants gave favorable responses to 3D-printing models in the aspects of understanding anatomic structures and relationship, inter-trainee communication, surgical planning, and enhancement of interest and confidence (50.0% to 94.4%, respectively).

CONCLUSIONS

The 3D-printing model is a valuable tool in the training of new residents and fellows in the subspecialty of spinal tumors. It can facilitate the trainees' understanding of tumor anatomy, surgical readiness, and confidence as well.

Determining influence, interaction and causality of contrast and sequence effects in objective structured clinical exams

INTRODUCTION

Differential rater function over time (DRIFT) and contrast effects (examiners' scores biased away from the standard of preceding performances) both challenge the fairness of scoring in objective structured clinical exams (OSCEs). This is important as, under some circumstances, these effects could alter whether some candidates pass or fail assessments. Benefitting from experimental control, this study investigated the causality, operation and interaction of both effects simultaneously for the first time in an OSCE setting.

METHODS

We used secondary analysis of data from an OSCE in which examiners scored embedded videos of student performances interspersed between live students. Embedded video position varied between examiners (early vs. late) whilst the standard of preceding performances naturally varied (previous high or low). We examined linear relationships suggestive of DRIFT and contrast effects in all within-OSCE data before comparing the influence and interaction of 'early' versus 'late' and 'previous high' versus 'previous low' conditions on embedded video scores.

RESULTS

Linear relationships data did not support the presence of DRIFT or contrast effects. Embedded videos were scored higher early (19.9 [19.4-20.5]) versus late (18.6 [18.1-19.1], p < 0.001), but scores did not differ between previous high and previous low conditions. The interaction term was non-significant.

CONCLUSIONS

In this instance, the small DRIFT effect we observed on embedded videos can be causally attributed to examiner behaviour. Contrast effects appear less ubiquitous than some prior research suggests. Possible mediators of these finding include the following: OSCE context, detail of task specification, examiners' cognitive load and the distribution of learners' ability. As the operation of these effects appears to vary across contexts, further research is needed to determine the prevalence and mechanisms of contrast and DRIFT effects, so that assessments may be designed in ways that are likely to avoid their occurrence. Quality assurance should monitor for these contextually variable effects in order to ensure OSCE equivalence.

Determining the influence of different linking patterns on the stability of students' score adjustments produced using Video-based Examiner Score Comparison and Adjustment (VESCA)

BACKGROUND

Ensuring equivalence of examiners' judgements across different groups of examiners is a priority for large scale performance assessments in clinical education, both to enhance fairness and reassure the public. This study extends insight into an innovation called Video-based Examiner Score Comparison and Adjustment (VESCA) which uses video scoring to link otherwise unlinked groups of examiners. This linkage enables comparison of the influence of different examiner-groups within a common frame of reference and provision of adjusted "fair" scores to students. Whilst this innovation promises substantial benefit to quality assurance of distributed Objective Structured Clinical Exams (OSCEs), questions remain about how the resulting score adjustments might be influenced by the specific parameters used to operationalise VESCA. Research questions, How similar are estimates of students' score adjustments when the model is run with either: fewer comparison videos per participating examiner?; reduced numbers of participating examiners?

METHODS

Using secondary analysis of recent research which used VESCA to compare scoring tendencies of different examiner groups, we made numerous copies of the original data then selectively deleted video scores to reduce the number of 1/ linking videos per examiner (4 versus several permutations of 3,2,or 1 videos) or 2/examiner participation rates (all participating examiners (76%) versus several permutations of 70%, 60% or 50% participation). After analysing all resulting datasets with Many Facet Rasch Modelling (MFRM) we calculated students' score adjustments for each dataset and compared these with score adjustments in the original data using Spearman's correlations.

RESULTS

Students' score adjustments derived form 3 videos per examiner correlated highly with score adjustments derived from 4 linking videos (median Rho = 0.93,IQR0.90-0.95,p < 0.001), with 2 (median Rho 0.85,IQR0.81-0.87,p < 0.001) and 1 linking videos (median Rho = 0.52(IQR0.46-0.64,p < 0.001) producing progressively smaller correlations. Score adjustments were similar for 76% participating examiners and 70% (median Rho = 0.97,IQR0.95-0.98,p < 0.001), and 60% (median Rho = 0.95,IQR0.94-0.98,p < 0.001) participation, but were lower and more variable for 50% examiner participation (median Rho = 0.78,IQR0.65-0.83, some ns).

CONCLUSIONS

Whilst VESCA showed some sensitivity to the examined parameters, modest reductions in examiner participation rates or video numbers produced highly similar results. Employing VESCA in distributed or national exams could enhance quality assurance or exam fairness.

The utility of a formative one-station objective structured clinical examination for Substance use disorders in a dental curriculum

Substance use disorders (SUD) are chronic relapsing medical conditions characterised by compulsive substance seeking and use. They constitute a substantial disease burden globally. Labelling of persons with SUD has created barriers to treatment but there are effective management strategies. The dental profession has embraced reforms designed to address the SUD epidemic by promoting continuing education for practitioners and initiating curriculum changes in dental schools. Screening, Brief Intervention and Referral to Treatment (SBIRT) is an evidence-based model for managing patients with SUD. The use of a formative 1-station Objective Structured Clinical Examination (OSCE) for learning and assessment in SBIRT, operationalised with the MD3 rating scale is presented in this study. In 3 years of implementation, the SBIRT OSCE successfully integrated into the curriculum of the College of Dental Medicine, Columbia University. Mean score of total adherent behaviours was 11.80 (SD =4.23) (range: 2 - 24) and Cronbach's coefficient alpha for across-items reliability in adherent behaviours was 0.66. Adherent behaviours correlated with the global ratings (r = 0.66). Mean of global rating scores were 2.90 (SD =1.01) for collaboration and 2.97 (SD =1.00) for empathy and the global rating scores correlated with each other (r = 0.85). Histograms of global rating scores resembled normal distribution. The 1-station OSCE is a good model for learning about SBIRT. Psychometric analysis was useful in understanding the underlying construct of the MD3 rating scale and supported its reliability, validity and utility in dental education.

The Past, Present, and Future of the United States Medical Licensing Examination Step 2 Clinical Skills Examination

In January 2021, the United States Medical Licensing Examination (USMLE) announced the permanent suspension of their Step 2 Clinical Skills (CS) examination. Launched in 2004, the Step 2 CS examination was intended to ensure that physicians entering graduate medical education possess the necessary information gathering, clinical reasoning, and communication skills necessary to provide patient care. Although the requirement that doctors pass a clinical skills examination as a condition of licensure likely improved some elements of medical education and physician practice, the Step 2 CS examination was deeply unpopular among many medical students since its inception. The demise of USMLE Step 2 CS provides an opportunity to re-examine the test’s value and incorporate improvements in future iterations. However, doing so requires a clear understanding of why the test was so vigorously challenged. Here, we review the history of clinical skills examinations used for medical licensure in the United States and highlight the persistent concerns regarding Step 2 CS’s cost, value, validity, and lack of examinee feedback before proposing future improvements to address each concern.

Design, Implementation, and Evaluation of a Distance Learning Framework to Adapt to the Changing Landscape of Anatomy Instruction in Medical Education During COVID-19 Pandemic: A Proof-of-Concept Study

This study presents the design of a DL-framework to deliver anatomy teaching that provides a microfiche of the onsite anatomy learning experience during the mandated COVID-19 lockdown. First, using nominal-group technique, we identified the DL learning theories to be employed in blueprinting the DL-framework. Effectiveness of the designed DL-framework in anatomy teaching was demonstrated using the exemplar of the Head and Neck (H&N) course during COVID-19 lockdown, in the pre-clerkship curriculum at our medical school. The dissemination of the DL-framework in the anatomy course was informed by the Analyse, Design, Develop, Implement, and Evaluate (ADDIE) model. The efficiency of the DL-framework was evaluated using the first two levels of Kirkpatrick's model. Versatility of the DL-framework was demonstrated by aligning its precepts with individual domains of key learning outcomes framework. The framework's blueprint was designed amalgamating principles of: Garrison's community inquiry, Siemens' connectivism and Harasim's online-collaborative-learning; and improved using Anderson's DL-model. Following the implementation of the DL-framework in the H&N course informed by ADDIE, the framework's efficiency was evaluated. In total, 70% students responded to the survey assessing perception toward DL (Kirkpatrick's Level: 1). Descriptive analysis of the survey results showed that the DL-framework was positively received by students and attested that students had an enriched learning experience, which promoted collaborative-learning and student-autonomy. For, Kirkpatrick's Level: 2 i.e., cognitive development, we compared the summative assessment performance in the H&N course across three cohort of students. The results show that the scores of the cohort, which experienced the course entirely through DL modality was statistically higher (P < 0.01) than both the other cohorts, indicating that shift to DL did not have an adverse effect on students' learning. Using Bourdieu's Theory of Practice, we showed that the DL-framework is an efficient pedagogical approach, pertinent for medical schools to adopt; and is versatile as it attests to the key domains of students' learning outcomes in the different learning outcomes framework. To our knowledge this is the first-study of its kind where a rationale and theory-guided approach has been availed not only to blueprint a DL framework, but also to implement it in the MBBS curriculum.

Measuring the Effect of Examiner Variability in a Multiple-Circuit Objective Structured Clinical Examination (OSCE)

PURPOSE

Ensuring that examiners in different parallel circuits of objective structured clinical examinations (OSCEs) judge to the same standard is critical to the chain of validity. Recent work suggests examiner-cohort (i.e., the particular group of examiners) could significantly alter outcomes for some candidates. Despite this, examiner-cohort effects are rarely examined since fully nested data (i.e., no crossover between the students judged by different examiner groups) limit comparisons. In this study, the authors aim to replicate and further develop a novel method called Video-based Examiner Score Comparison and Adjustment (VESCA), so it can be used to enhance quality assurance of distributed or national OSCEs.

METHOD

In 2019, 6 volunteer students were filmed on 12 stations in a summative OSCE. In addition to examining live student performances, examiners from 8 separate examiner-cohorts scored the pool of video performances. Examiners scored videos specific to their station. Video scores linked otherwise fully nested data, enabling comparisons by Many Facet Rasch Modeling. Authors compared and adjusted for examiner-cohort effects. They also compared examiners' scores when videos were embedded (interspersed between live students during the OSCE) or judged later via the Internet.

RESULTS

Having accounted for differences in students' ability, different examiner-cohort scores for the same ability of student ranged from 18.57 of 27 (68.8%) to 20.49 (75.9%), Cohen's d = 1.3. Score adjustment changed the pass/fail classification for up to 16% of students depending on the modeled cut score. Internet and embedded video scoring showed no difference in mean scores or variability. Examiners' accuracy did not deteriorate over the 3-week Internet scoring period.

CONCLUSIONS

Examiner-cohorts produced a replicable, significant influence on OSCE scores that was unaccounted for by typical assessment psychometrics. VESCA offers a promising means to enhance validity and fairness in distributed OSCEs or national exams. Internet-based scoring may enhance VESCA's feasibility.

The Use of Simulation in Teaching

Simulation-based medical education (SBME) provides experiential learning for medical trainees without any risk of harm to patients. Simulation is now included in most medical school and residency curricula. In psychiatric education, simulation programs are rapidly expanding and innovating. Major applications of SBME in psychiatry include achieving close observation of trainees with patients, preparing trainees for unstable patient scenarios, and exposing trainees to a broader range of psychopathology. This review article covers the history of SBME, simulation modalities, current use of SBME in psychiatry, a case study from one institution, and recommendations for incorporating simulation in psychiatry education.

Moving beyond Cronbach's Alpha and Inter-Rater Reliability: A Primer on Generalizability Theory for Pharmacy Education

BACKGROUND

When available, empirical evidence should help guide decision-making. Following each administration of a learning assessment, data becomes available for analysis. For learning assessments, Kane's Framework for Validation can helpfully categorize evidence by inference (i.e., scoring, generalization, extrapolation, implications). Especially for test-scores used within a high-stakes setting, generalization evidence is critical. While reporting Cronbach's alpha, inter-rater reliability, and other reliability coefficients for a single measurement error are somewhat common in pharmacy education, dealing with multiple concurrent sources of measurement error within complex learning assessments is not. Performance-based assessments (e.g., OSCEs) that use raters, are inherently complex learning assessments.

PRIMER

Generalizability Theory (G-Theory) can account for multiple sources of measurement error. G-Theory is a powerful tool that can provide a composite reliability (i.e., generalization evidence) for more complex learning assessments, including performance-based assessments. It can also help educators explore ways to make a learning assessment more rigorous if needed, as well as suggest ways to better allocate resources (e.g., staffing, space, fiscal). A brief review of G-Theory is discussed herein focused on pharmacy education.

MOVING FORWARD

G-Theory has been common and useful in medical education, though has been used rarely in pharmacy education. Given the similarities in assessment methods among health-professions, G-Theory should prove helpful in pharmacy education as well. Within this Journal and accompanying this Idea Paper, there are multiple reports that demonstrate use of G-Theory in pharmacy education.

Performance assessment: Consensus statement and recommendations from the 2020 Ottawa Conference

INTRODUCTION

In 2011 the Consensus Statement on Performance Assessment was published in Medical Teacher. That paper was commissioned by AMEE (Association for Medical Education in Europe) as part of the series of Consensus Statements following the 2010 Ottawa Conference. In 2019, it was recommended that a working group be reconvened to review and consider developments in performance assessment since the 2011 publication.

METHODS

Following review of the original recommendations in the 2011 paper and shifts in the field across the past 10 years, the group identified areas of consensus and yet to be resolved issues for performance assessment.

RESULTS AND DISCUSSION

This paper addresses developments in performance assessment since 2011, reiterates relevant aspects of the 2011 paper, and summarises contemporary best practice recommendations for OSCEs and WBAs, fit-for-purpose methods for performance assessment in the health professions.

Translation and further validation of a global rating scale for the assessment of clinical competence in prehospital emergency care

Global rating scales are useful to assess clinical competence at a general level based on specific word dimensions. The aim of this study was to translate and culturally adapt the Paramedic Global Rating Scale, and to contribute validity evidence and instrument usefulness in training results and clinical competence assessments of students undergoing training to become ambulance nurses and paramedics at Swedish and Finnish universities. The study included translation, expert review and inter-rater reliability (IRR) tests. The scale was translated and culturally adapted to clinical and educational settings in both countries. A content validity index (CVI) was calculated using eight experts. IRR tests were performed with five registered nurses working as university lecturers, and with six clinicians working as ambulance nurses. They individually rated the same simulated ambulance assignment. Based on the ratings IRR was calculated with intra-class correlation (ICC). The scale showed excellent CVI for items and scale. The ICC indicated substantial agreement in the group of lecturers and a high degree of agreement in the group of clinicians. This study provides validity evidence for a Swedish version of the scale, supporting its use in measuring clinical competence among students undergoing training to become ambulance nurses and paramedics.

Cross-Cultural Adaptation of the Communication Assessment Tool for Use in a Simulated Clinical Setting

Construct: The Communication Assessment Tool (CAT) is a 14-item instrument developed in English to assess medical trainees' interpersonal communication skills from the patient's perspective in clinical settings. Background: Using validated instruments and simulated patients constitutes good practice in assessing doctor-patient communication. The CAT was designed for use in real practice, but has not yet been applied to assessing OB-GYN residents' delivery of bad news in Objective Structured Clinical Examination (OSCE) stations. This study aims to provide validity evidence for using the CAT to assess residents' interpersonal communication skills under difficult circumstances in a simulated clinical setting in Brazil. Approach: Cross-cultural adaptation comprised translation into Portuguese, synthesis of translations, and back-translation. Next, a committee of 10 external and independent experts rated the items for linguistic equivalence and relevance to the overall scale. Researchers used the expert ratings to produce a preliminary Brazilian-Portuguese version. This version was applied by four simulated patients to assess 28 OB-GYN residents completing two, 10-minute OSCE stations focused on delivering bad news. Item and scale content validity indices and internal-consistency reliability were calculated. Simulated patients were interviewed to clarify any doubt regarding the content and usability of the tool and their response process. Findings: Thirteen of the 14 items in the Brazilian-Portuguese version were considered "equivalent" by at least 70% of the experts. All items were considered relevant by 100% of the experts. The Item Content Validity Index ranged from .9 to 1, and the Scale Content Validity Index was .99. The instrument showed good reliability for both scenarios (Cronbach's alpha > .90). Simulated patients considered the CAT easy to understand and complete. Conclusions: This study provides validity evidence for using the Brazilian-Portuguese CAT in a simulated clinical environment to assess OB-GYN residents' delivery of bad news. Based on this study's findings, the OB-GYN Department organized an annual formative assessment for residents to improve their interpersonal communication skills. This version of the CAT may also be applicable to other specialties.

Objective assessment of obstetrics residents' surgical skills in caesarean: Development and evaluation of a specific rating scale

OBJECTIVE

To develop a modified version of Objective Structured Assessment of Technical Skill (OSATS) rating scale for evaluation of surgical skills specific to caesarean and to assess its relevance in documenting the residents' learning curve during their training. Secondarily, to verify the scale's stability to caesarean's level of difficulty and comparing self-assessment to hetero-assessment in order to propose a practical application of this rating scale during residency.

STUDY DESIGN

We conducted a multicentre observational prospective study, from May 2018 to November 2018. All residents at that time could participate and fill in the rating scale after caesarean. Senior surgeons had to fill in the same rating scale. We analysed correlation between self-assessments and hetero-assessments and sensitivity to change of the rating scale. Analysis of feature's relevance was performed by principal component analysis, factor analysis and reliability analysis.

RESULTS

In total, 234 rating scales were completed evaluating 18 residents. Our study demonstrated that our rating scale could be used to evaluate surgical skills of residents during caesarean and distinguish their year of residency (p < 0.001) with a high correlation between self and hetero-assessment (Intraclass Correlation coefficient for global score: 0.78; 95% CI 0.68-0.86). The principal component analysis revealed two dimensions corresponding to the two parts of the rating scale and the factorial analysis allowed us to confirm distribution of features according to these two dimensions. Cronbach's alpha allowed us to highlight the percentage of representation of the scale's features in relation to all potential theoretical features (0.93, 95% CI 0.82-0.95).

CONCLUSION

Our rating scale could be used for self-assessment during residency and as a hetero-assessment tool for validating defined stages of the internship.

Reliability of Observational Assessment Methods for Outcome-based Assessment of Surgical Skill: Systematic Review and Meta-analyses

BACKGROUND

Reliable performance assessment is a necessary prerequisite for outcome-based assessment of surgical technical skill. Numerous observational instruments for technical skill assessment have been developed in recent years. However, methodological shortcomings of reported studies might negatively impinge on the interpretation of inter-rater reliability.

OBJECTIVE

To synthesize the evidence about the inter-rater reliability of observational instruments for technical skill assessment for high-stakes decisions.

DESIGN

A systematic review and meta-analysis were performed. We searched Scopus (including MEDLINE) and Pubmed, and key publications through December, 2016. This included original studies that evaluated reliability of instruments for the observational assessment of technical skills. Two reviewers independently extracted information on the primary outcome (the reliability statistic), secondary outcomes, and general information. We calculated pooled estimates using multilevel random effects meta-analyses where appropriate.

RESULTS

A total of 247 documents met our inclusion criteria and provided 491 inter-rater reliability estimates. Inappropriate inter-rater reliability indices were reported for 40% of the checklists estimates, 50% of the rating scales estimates and 41% of the other types of assessment instruments estimates. Only 14 documents provided sufficient information to be included in the meta-analyses. The pooled Cohen's kappa was .78 (95% CI 0.69-0.89, p < 0.001) and pooled proportion agreement was 0.84 (95% CI 0.71-0.96, p < 0.001). A moderator analysis was performed to explore the influence of type of assessment instrument as a possible source of heterogeneity.

CONCLUSIONS AND RELEVANCE

For high-stakes decisions, there was often insufficient information available on which to base conclusions. The use of suboptimal statistical methods and incomplete reporting of reliability estimates does not support the use of observational assessment instruments for technical skill for high-stakes decisions. Interpretations of inter-rater reliability should consider the reliability index and assessment instrument used. Reporting of inter-rater reliability needs to be improved by detailed descriptions of the assessment process.

The role of standardized patient assessment forms in medical communication skills education

Communication skills training is a routine practice in medical education designed to instruct and evaluate future physicians in matters of patient-provider interaction. Based on the United States Medical Licensing Examination Step 2 Clinical Skills (CS), medical schools across the United States hire and train standardized patients (SPs) to act as patients in and evaluators of simulated interactions with medical students (MSs). Using discourse analysis, I examine how a computerized assessment form creates a particularized version of communication skills with implications for future practice. The 39-item checklist is completed by SPs following a simulated interaction designed to prepare third-year MSs for the Step 2 CS. Specifically, I analyze how the form is structured to make recognizable specific communication skills tasks, who should complete said tasks, and what varying degrees of communication skills competency are within the realm of task completion. By analyzing the form, I consider the agency of texts in medical education, the implications of technologizing communication as an institutional skill, and the limitations of enlisting SPs to evaluate communication skills competency under the guise of a patient perspective.

A Tale of Two Inquiries (or, Doing Being Competent in a Clinical Skills Exam)

Phenomenon: In high-stakes evaluations of communicative competency, data-gathering skills are commonly assessed through the use of standardized patient encounters. This article seeks to document inquiry practices in 2 such encounters in a setting designed to emulate a consequential, clinical skills examination. Approach: Drawing on the methods and findings of Conversation Analysis, we examine selected fragments seeking to understand how, in the ways in which they are organized, they produce quite different outcomes. Findings: In the first encounter, the topic of the patient's history of depression arises naturally in the course of the interview. It happens to be a checklist item for the case and the examinee receives credit for having elicited it. In the second encounter, though the examinee was the more clinically experienced, the topic does not come up and the examinee fails to receive credit. Insights: When we examine how the two inquiry sequences develop on a turn-by-turn basis, it becomes clear that the differences between inquiry practices that carefully constrain patient responses and those that leave space for patient elaboration are subtle but evident. Both types of practice, however, are presumably a part of competent clinical performance. We argue that looking carefully at how specific interactional practices operate within clinical interviews can enable us to become more articulate as to what might count as communicative competence in the clinic.

Developing a video-based method to compare and adjust examiner effects in fully nested OSCEs

BACKGROUND

Although averaging across multiple examiners' judgements reduces unwanted overall score variability in objective structured clinical examinations (OSCE), designs involving several parallel circuits of the OSCE require that different examiner cohorts collectively judge performances to the same standard in order to avoid bias. Prior research suggests the potential for important examiner-cohort effects in distributed or national examinations that could compromise fairness or patient safety, but despite their importance, these effects are rarely investigated because fully nested assessment designs make them very difficult to study. We describe initial use of a new method to measure and adjust for examiner-cohort effects on students' scores.

METHODS

We developed video-based examiner score comparison and adjustment (VESCA): volunteer students were filmed 'live' on 10 out of 12 OSCE stations. Following the examination, examiners additionally scored station-specific common-comparator videos, producing partial crossing between examiner cohorts. Many-facet Rasch modelling and linear mixed modelling were used to estimate and adjust for examiner-cohort effects on students' scores.

RESULTS

After accounting for students' ability, examiner cohorts differed substantially in their stringency or leniency (maximal global score difference of 0.47 out of 7.0 [Cohen's d = 0.96]; maximal total percentage score difference of 5.7% [Cohen's d = 1.06] for the same student ability by different examiner cohorts). Corresponding adjustment of students' global and total percentage scores altered the theoretical classification of 6.0% of students for both measures (either pass to fail or fail to pass), whereas 8.6-9.5% students' scores were altered by at least 0.5 standard deviations of student ability.

CONCLUSIONS

Despite typical reliability, the examiner cohort that students encountered had a potentially important influence on their score, emphasising the need for adequate sampling and examiner training. Development and validation of VESCA may offer a means to measure and adjust for potential systematic differences in scoring patterns that could exist between locations in distributed or national OSCE examinations, thereby ensuring equivalence and fairness.

Twelve tips for developing an OSCE that measures what you want

The Objective Structured Clinical Examination (OSCE) is used globally for both high and low stakes assessment. Despite its extensive use, very few published articles provide a set of best practices for developing an OSCE, and of those that do, none apply a modern understanding of validity. This article provides 12 tips for developing an OSCE guided by Kane's validity framework to ensure the OSCE is assessing what it purports to measure. The 12 tips are presented in the order they would be operationalized during OSCE development.

[Validity and reproducibility of two direct observation assessment forms for evaluation of internal medicine residents' clinical skills]

INTRODUCTION

The revision of the French medical studies' third cycle ought to be competency-based. In internal medicine, theoretical and practical knowledge will be assessed online with e-learning and e-portfolio. In parallel, a reflection about clinical skills assessment forms is currently ongoing. In this context, our aim was to assess the reproducibility and validity of two assessment forms based on direct clinical observation.

METHOD

A prospective and multicentric study has been conducted from November 2015 to October 2016 aiming at evaluating the French translations of the MINI-Clinical Examination Exercice (MINI-CEX) and the Standardized Patient Satisfaction Questionnaire (SPSQ). Included residents have been assessed 2 times over a period of 6 months by the same binoma of judges.

RESULTS

Nineteen residents have been included. The inter-judge reproducibility was satisfactory for the MINI-CEX: intraclass coefficients (ICC) between 0.4 and 0.8 and moderate for the SPSQ: ICC between 0.2 and 0.7 with a good internal coherence for both questionnaires (Cronbach between 0.92 and 0.94). Significant differences between the distributions of the scores given by the judges and a significant inter-center variability have been found.

CONCLUSION

If the absolute value of the scores should not be taken into account in the evaluation process given its high variability, it could be of interest for the follow-up of the progression in the competencies. These forms could support the residents' debriefing based on the general trends given by the scores.

Video-based feedback as a method for training rural healthcare workers to manage medical emergencies: a pilot study

BACKGROUND

Video-based feedback has been shown to aid knowledge retention, skills learning and improve team functionality. We explored the use of video-based feedback and low fidelity simulation for training rural healthcare workers along the Thailand-Myanmar border and Papua New Guinea (PNG) to manage medical emergencies effectively.

METHODS

Twenty-four study participants were recruited from three Shoklo Malaria Research Unit clinics along the Thailand-Myanmar border and eight participants from Kudjip Nazarene Hospital, PNG. The teams were recorded on video managing a simulated medical emergency scenario and the video was used to aid feedback and assess performance using Observed Structured Clinical Examination (OSCE) scoring and Team Emergency Assessment Measure (TEAM) questionnaire. The process was repeated post-feedback at both sites and at 6 weeks at the Thailand-Myanmar border site. Thailand-Myanmar border participants' individual confidence levels and baseline knowledge (using OSCE scoring) were assessed before team assessment and feedback at week 1 and repeated post-feedback and at 6 weeks. Focus group discussions (FGD) were held at each Thailand-Myanmar border clinic at week 1 (8 participants at each clinic).

RESULTS

Individual paired tests of OSCE scores showed significant improvement post-feedback at week 1 (p < 0.001) and week 6 (p < 0.001) compared to baseline OSCE scores. There was a trend for increased team OSCE scores compared to baseline at week 1 (p = 0.068) and week 6 (p = 0.109) although not significant. Thailand-Myanmar border TEAM scores demonstrated improvement post-feedback mainly in leadership, teamwork and task management which was sustained up to week 6. PNG showed an improvement mainly in teamwork and task management. The global rating of the teams' non-technical performance at both sites improved post feedback and at week 6 on the Thailand-Myanmar border site. Self-rated confidence scores by Thailand-Myanmar border participants increased significantly from baseline following training at week 1 (p = 0.020), and while higher at 6 weeks follow up than at baseline, this was not significant (p = 0.471). The FGD revealed majority of participants felt that watching the video recording of their performance and the video-based feedback contributed most to their learning.

CONCLUSION

Video-assisted feedback resulted in an improvement in clinical knowledge, confidence and quality of teamwork for managing medical emergencies in two low resource medical facilities in South East Asia and the South Pacific.

Preparing Residents Effectively in Emergency Skills Training With a Serious Game

Supplemental digital content is available in the text.

Introduction

Training emergency care skills is critical for patient safety but cost intensive. Serious games have been proposed as an engaging self-directed learning tool for complex skills. The objective of this study was to compare the cognitive skills and motivation of medical residents who only used a course manual as preparation for classroom training on emergency care with residents who used an additional serious game.

Methods

This was a quasi-experimental study with residents preparing for a rotation in the emergency department. The “reading” group received a course manual before classroom training; the “reading and game” group received this manual plus the game as preparation for the same training. Emergency skills were assessed before training (with residents who agreed to participate in an extra pretraining assessment), using validated competency scales and a global performance scale. We also measured motivation.

Results

All groups had comparable important characteristics (eg, experience with acute care). Before training, the reading and game group felt motivated to play the game and spent more self-study time (+2.5 hours) than the reading group. Game-playing residents showed higher scores on objectively measured and self-assessed clinical competencies but equal scores on the global performance scale and were equally motivated for training, compared with the reading group. After the 2-week training, no differences between groups existed.

Conclusions

After preparing training with an additional serious game, residents showed improved clinical competencies, compared with residents who only studied course material. After a 2-week training, this advantage disappeared. Future research should study the retention of game effects in blended designs.

Communication skills in candidates for accreditation in rheumatology are correlated with candidate's performance in the objective structured clinical examination

BACKGROUND

The Mexican Accreditation Council for Rheumatology annually certifies trainees in Rheumatology using a multiple-choice test and an objective structured clinical examination (OSCE). Since 2015, candidate's communication skills (CS) have been rated by both patients and by physician examiners and correlated with results on the OSCE. This study compared the CS from candidates to annual accreditation in Rheumatology as rated by patients and by physician examiners, and assessed whether these correlated with candidate's performance in the OSCE.

MATERIAL AND METHODS

From 2015 to 2017, 8areas of CS were evaluated using a Likert scale, in each OSCE station that involved a patient. Both patient and physician evaluators were trained annually and their evaluations were performed blindly. The associations were calculated using the Pearson correlation coefficient.

RESULTS

In general, candidates were given high CS scores; the scores from patients of the candidate's CS were better than those of physician examiners; within the majority of the stations, both scores were found to correlate moderately. In addition, the scoring of CS correlated with trainee performance at the corresponding OSCE station. Interestingly, better correlations were found when the skills were rated by the patients compared to physician scores. The average CS score was correlated with the overall OSCE performance for each trainee, but not with the multiple-choice test, except in the 2017 accreditation process, when a weak correlation was found.

CONCLUSIONS

CS assessed during a national accreditation process correlated with the candidate's performance at the station level and with the overall OSCE.

Physical examination skills training: Faculty staff vs. patient instructor feedback-A controlled trial

BACKGROUND

Standardized patients are widely used in training of medical students, both in teaching and assessment. They also frequently lead complete training sessions delivering physical examination skills without the aid of faculty teaching staff-acting as "patient instructors" (PIs). An important part of this training is their ability to provide detailed structured feedback to students which has a strong impact on their learning success. Yet, to date no study has assessed the quality of physical examination related feedback by PIs. Therefore, we conducted a randomized controlled study comparing feedback of PIs and faculty staff following a physical examination assessed by students and video assessors.

METHODS

14 PIs and 14 different faculty staff physicians both delivered feedback to 40 medical students that had performed a physical examination on the respective PI while the physicians observed the performance. The physical examination was rated by two independent video assessors to provide an objective performance standard (gold standard). Feedback of PI and physicians was content analyzed by two different independent video assessors based on a provided checklist and compared to the performance standard. Feedback of PIs and physicians was also rated by medical students and video assessors using a questionnaire consisting of 12 items.

RESULTS

There was no statistical significant difference concerning overall matching of physician or PI feedback with gold standard ratings by video assessment (p = .219). There was also no statistical difference when focusing only on items that were classified as major key steps (p = .802), mistakes or parts that were left out during physical examination (p = .219) or mistakes in communication items (p = .517). The feedback of physicians was significantly better rated than PI feedback both by students (p = .043) as well as by video assessors (p = .034).

CONCLUSIONS

In summary, our study demonstrates that trained PIs are able to provide feedback of equal quantitative value to that of faculty staff physicians with regard to a physical examination performed on them. However, both the students and the video raters judged the quality of the feedback given by the physicians to be significantly better than that of the PIs.

The effect on reliability and sensitivity to level of training of combining analytic and holistic rating scales for assessing communication skills in an internal medicine resident OSCE

OBJECTIVE

Although previous research has compared checklists to rating scales for assessing communication, the purpose of this study was to compare the effect on reliability and sensitivity to level of training of an analytic, a holistic, and a combined analytic-holistic rating scale in assessing communication skills.

METHODS

The University of Alberta Internal Medicine Residency runs OSCEs for postgraduate year (PGY) 1 and 2 residents and another for PGY-4 residents. Communication stations were scored with an analytic scale (empathy, non-verbal skills, verbal skills, and coherence subscales) and a holistic scale. Authors analyzed reliability of individual and combined scales using generalizability theory and evaluated each scale's sensitivity to level of training.

RESULTS

For analytic, holistic, and combined scales, 12, 12, and 11 stations respectively yielded a Phi of 0.8 for the PGY-1,2 cohort, and 16, 16, and 14 stations yielded a Phi of 0.8 for the PGY-4 cohort. PGY-4 residents scored higher on the combined scale, the analytic rating scale, and the non-verbal and coherence subscales.

CONCLUSION

A combined analytic-holistic rating scale increased score reliability and was sensitive to level of training.

PRACTICE IMPLICATIONS

Given increased validity evidence, OSCE developers should consider combining analytic and holistic scales when assessing communication skills.

[Development of an objective structured clinical examination (OSCE) for evaluating clinical competence in vascular medicine]

OBJECTIVES

Vascular medicine is now a clinical specialty in France. During their studies, students will acquire clinical reasoning in addition to technical skills. An Objective Structured Clinical Examination (OSCE) is considered as the gold standard for evaluating clinical competence. Our main objective was to evaluate the feasibility and acceptability of OSCE for the evaluation of students, secondarily their performance.

METHODS

Three representative clinical cases of the specialty were developed. The OSCE consisted of a sequence of clinical situations presented in three stations of 7minutes each. The role of the simulated patient was played by medical students. At the end of the OSCE, observers and students completed the evaluation form. We compared the performances between junior and senior vascular medicine students. Written questionnaires were used to measure OSCE satisfaction.

RESULTS

We were able to develop and organize this examination without difficulties. Fifteen students were evaluated. All participants agreed that the clinical situations were representative of vascular medicine practice, the cases were realistic and standardized patients were convincing. The performance of senior students was statistically higher than junior students in one case.

DISCUSSION

Our study demonstrates the feasibility and acceptability of the OSCE in students in vascular medicine. The small number of stations and candidates requires further studies on a larger scale to evaluate their performance.

Training and Validation of Standardized Patients for Assessing Communication and Counseling Skills of Pharmacy Students: A Pilot Study

OBJECTIVE

The objective of this study is to describe the process of training valid simulated patients (SPs) for assessing communication and counseling skills of pharmacy students' performance.

METHODS

This is a cross-sectional and correlational study. Psychometric properties of checklist and SPs' portrayals and their filling of the checklist regarding assessing pharmacy students were assessed. Five SPs who were working in the simulated patient's pool were volunteered to take part in the project, which one of the SPs failed. Three scenarios, along with corresponding checklists, were developed based on the usual medications of different diseases consisting of asthma, respiratory infections, and osteoporosis. The SPs' role-play performance was video-recorded and rated independently by two experts according to an observational rating scale to assess validity. The role-play was repeated after 1 week with the same scenario and the same doctor, to assess test-retest reliability. The inter-rater agreement between SPs and experts was determined by calculating the intraclass correlation coefficient and kappa coefficient.

FINDINGS

The four eligible SPs were all women, with an average age of 37 years. The correlation between mean scores of raters and mean scores of SPs was 0.91 and 0.85, respectively. The Pearson's correlation between mean scores of raters with SPs was 0.75. The checklists' reliability, Cronbach's alpha, was calculated to be 0.72. The measured weighted Cohen's kappa for the ratings of by each SP, and the gold standard was between 0.53 and 0.57, indicating a moderate agreement. The inter-rater reliability kappa coefficient between raters was 0.75 (P = 0.01).

CONCLUSION

The authors have demonstrated the technique of using standardized patients to evaluate communication and counseling skills of pharmacy students. The findings indicated that trained SPs can be used as an effective tool to assess pharmacy students' communication and counseling skills.

Do Standardized Patients Have Concerns About Students Not Captured by Traditional Assessment Forms?

Construct: Traditionally, standardized patients (SPs) assess students' clinical skills principally through numerical rating forms-an approach that may not fully capture SPs' concerns. SPs are students' closest approximation to real patients. To maximally benefit students' clinical training and evaluation it is important to find ways to give voice to the totality of SPs' perspectives.

BACKGROUND

SPs have been shown to be a reliable and valid means to assess medical students' clinical skills in clinical skills examinations. We noticed, however, that SPs often express "off the record" concerns about students, which they do not include on traditional assessment forms.

APPROACH

To explore these "off the record" concerns, we designed a Concerns item and added it to the traditional assessment form for an end-of-3rd-year clinical skills examination shared by three medical schools. We asked SPs to use this Concerns item to identify students about whom they had any "gut-level" concerns and provided them with a narrative opportunity to explain why. SPs were informed that the purpose of the item was to help students with difficulties and was not part of the student's grade.

RESULTS

We analyzed the concerns data using quantitative and qualitative methods. Of 551 students at three schools, 223 (∼40%) had concerns recorded. Seventy students received two or more concerns. Qualitative analysis of SPs' comments revealed 3 major categories of concern: communication and interpersonal skills, history taking, and physical exam. Grouped under each were several subcategories. More than half of the written comments from the SPs related to the communication/interpersonal skills category and included subcategories commonly addressed in communications courses: lack of empathy, good listening skills, and lack of connection to the patient. They also included subcategories that in our experience are less commonly addressed: odd or off-putting mannerisms, lack of confidence, unprofessional behavior, domineering behavior, and biased behavior. Another 47% of concerns identified deficiencies in history taking and physical examination. Of the students with concerns noted by two or more SPs, SPs' narrative comments on 84%, 42%, and 48% of the students in the domains of communications, history, and physical exam respectively indicated potential problems not identified by scores on the traditional assessment form.

CONCLUSION

The Concerns item is a narrative assessment method that may add value to traditional quantitative scoring by identifying and characterizing problematic student performance not captured by the traditional assessment form. It may thus contribute to giving fuller voice to the totality of SPs' perspective.

[Validation of a questionnaire for standardized-patient assessment of clinical skills]

BACKGROUND

This study was designed in the context of a new educational program based on standardized-patients (SPs). The study objectives were (1) to evaluate the reliability of the assessment form used by SPs and (2) to compare global ratings to checklists reproducibility.

METHOD

History taking, physical examination and communication skills were assessed by SPs at the end of clinical encounters with year 3 medical students, using generic global rating scales and case-specific checklists. The validation process followed four steps: (1) correlation analysis between global rating and their relative checklist of specific items, (2) internal consistency estimation, (3) validation of the questionnaire dimensions, (4) estimation of the reliability of SPs' ratings compared to medical teachers' ratings.

RESULTS

A total of 3322 consultations have been performed by 444 year 3 medical students. Statistical analysis showed a good internal reliability (Cronbach α was greater than 0.7) and an acceptable inter-judge reproducibility except for communication skills. Case-specific checklists did not prove to be more reliable than global ratings. Reproducibility was lower with SPs' than with medical teachers' assessment.

CONCLUSION

Global rating-based assessment should be preferred to checklists because they are faster and easier to use (shorter duration of SPs training). As SPs proved to be acceptable examiners, no third person seems to be required as external observer.

An experimental study on the effects of a simulation game on students' clinical cognitive skills and motivation

Simulation games are becoming increasingly popular in education, but more insight in their critical design features is needed. This study investigated the effects of fidelity of open patient cases in adjunct to an instructional e-module on students' cognitive skills and motivation. We set up a three-group randomized post-test-only design: a control group working on an e-module; a cases group, combining the e-module with low-fidelity text-based patient cases, and a game group, combining the e-module with a high-fidelity simulation game with the same cases. Participants completed questionnaires on cognitive load and motivation. After a 4-week study period, blinded assessors rated students' cognitive emergency care skills in two mannequin-based scenarios. In total 61 students participated and were assessed; 16 control group students, 20 cases students and 25 game students. Learning time was 2 h longer for the cases and game groups than for the control group. Acquired cognitive skills did not differ between groups. The game group experienced higher intrinsic and germane cognitive load than the cases group (p = 0.03 and 0.01) and felt more engaged (p < 0.001). Students did not profit from working on open cases (in adjunct to an e-module), which nonetheless challenged them to study longer. The e-module appeared to be very effective, while the high-fidelity game, although engaging, probably distracted students and impeded learning. Medical educators designing motivating and effective skills training for novices should align case complexity and fidelity with students' proficiency level. The relation between case-fidelity, motivation and skills development is an important field for further study.

Relationship Between Faculty and Standardized Patient Assessment Scores of Podiatric Medical Students During a Standardized Performance Assessment Laboratory

BACKGROUND

Direct assessment of health professional student performance of clinical skills can be accurately performed in the standardized performance assessment laboratory (SPAL), typically by health professional faculty. However, owing to time and economic considerations, nonmedical individuals have been specially trained to perform the same function (standardized patients [SPs]). This study compared the assessment scores of the history and physical examination components of a SPAL designed for second-year podiatric medical students at Des Moines University (DMU) by a podiatry medical faculty member and SPs.

METHODS

A total of 101 students from the classes of 2015 and 2016 were evaluated in 2013 and 2014 by 11 to 13 SPs from the DMU SPAL program. The video recordings of these 101 students were then evaluated by one faculty member from the College of Podiatric Medicine and Surgery at DMU.

RESULTS

The Pearson correlation coefficient for each class showed a strong linear relationship between SP and faculty assessment scores. The associations between SP and faculty assessment scores in the history, physical examination, and combined history and physical examination components for the 2016 class (0.706, 0.925, and 0.911, respectively) were found to be stronger than those for the 2015 class (0.697, 0.791, and 0.791, respectively).

CONCLUSIONS

This study indicated that there are strong associations between the assessment scores of trained SPs and faculty for the history, physical examination, and combined history and physical examination components of second-year SPAL activity for podiatric medical students.

Simulated consultations: a sociolinguistic perspective

BACKGROUND

Assessment of consulting skills using simulated patients is widespread in medical education. Most research into such assessment is sited in a statistical paradigm that focuses on psychometric properties or replicability of such tests. Equally important, but less researched, is the question of how far consultations with simulated patients reflect real clinical encounters--for which sociolinguistics, defined as the study of language in its socio-cultural context, provides a helpful analytic lens.

DISCUSSION

In this debate article, we draw on a detailed empirical study of assessed role-plays, involving sociolinguistic analysis of talk in OSCE interactions. We consider critically the evidence for the simulated consultation (a) as a proxy for the real; (b) as performance; (c) as a context for assessing talk; and (d) as potentially disadvantaging candidates trained overseas. Talk is always a performance in context, especially in professional situations (such as the consultation) and institutional ones (the assessment of professional skills and competence). Candidates who can handle the social and linguistic complexities of the artificial context of assessed role-plays score highly--yet what is being assessed is not real professional communication, but the ability to voice a credible appearance of such communication. Fidelity may not be the primary objective of simulation for medical training, where it enables the practising of skills. However the linguistic problems and differences that arise from interacting in artificial settings are of considerable importance in assessment, where we must be sure that the exam construct adequately embodies the skills expected for real-life practice. The reproducibility of assessed simulations should not be confused with their validity. Sociolinguistic analysis of simulations in various professional contexts has identified evidence for the gap between real interactions and assessed role-plays. The contextual conditions of the simulated consultation both expect and reward a particular interactional style. Whilst simulation undoubtedly has a place in formative learning for professional communication, the simulated consultation may distort assessment of professional communication These sociolinguistic findings contribute to the on-going critique of simulations in high-stakes assessments and indicate that further research, which steps outside psychometric approaches, is necessary.

Trends in national licensing examinations in medicine

CONTEXT

As a contribution to this special issue commemorating the journal's 50th volume, this paper seeks to explore directions for national licensing examinations (NLEs) in medicine. Increases in the numbers of new medical schools and the mobility of doctors across national borders mean that NLEs are becoming even more important to ensuring physician competence.

OBJECTIVES

The purpose of this paper is to explore the use of NLEs in the future in the context of global changes in medical education and health care delivery.

METHODS

Because the literature related to NLEs is so large, we have not attempted a comprehensive review, but have focused instead on a small number of topics on which we think we have something useful to say. The paper is organised around five predicted trends for NLEs.

DISCUSSION

The first section discusses reasons why we think the use of NLEs will increase in the coming years. The second section discusses the ongoing problem of content specificity and its implications for the design of NLEs. The third section examines the evolution of large-scale, standardised cognitive assessments in NLEs and suggests some future directions. Reflecting the fact that NLEs are, increasingly, attempting to assess more than just knowledge, the fourth section addresses the future of large-scale clinical skills assessments in NLEs, predicting both increases in their use and some shifts in the nature of the stations used. The fifth section discusses workplace-based assessments, predicting increases in their use for formative assessment and identifying some limitations in their direct application in NLEs. The concluding section discusses the cost of NLEs and indulges in some further speculations about their evolution.

Constructing a validity argument for the Objective Structured Assessment of Technical Skills (OSATS): a systematic review of validity evidence

In order to construct and evaluate the validity argument for the Objective Structured Assessment of Technical Skills (OSATS), based on Kane's framework, we conducted a systematic review. We searched MEDLINE, EMBASE, CINAHL, PsycINFO, ERIC, Web of Science, Scopus, and selected reference lists through February 2013. Working in duplicate, we selected original research articles in any language evaluating the OSATS as an assessment tool for any health professional. We iteratively and collaboratively extracted validity evidence from included articles to construct and evaluate the validity argument for varied uses of the OSATS. Twenty-nine articles met the inclusion criteria, all focussed on surgical technical skills assessment. We identified three intended uses for the OSATS, namely formative feedback, high-stakes assessment and program evaluation. Following Kane's framework, four inferences in the validity argument were examined (scoring, generalization, extrapolation, decision). For formative feedback and high-stakes assessment, there was reasonable evidence for scoring and extrapolation. However, for high-stakes assessment there was a dearth of evidence for generalization aside from inter-rater reliability data and an absence of evidence linking multi-station OSATS scores to performance in real clinical settings. For program evaluation, the OSATS validity argument was supported by reasonable generalization and extrapolation evidence. There was a complete lack of evidence regarding implications and decisions based on OSATS scores. In general, validity evidence supported the use of the OSATS for formative feedback. Research to provide support for decisions based on OSATS scores is required if the OSATS is to be used for higher-stakes decisions and program evaluation.