What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment

A study of internal structure validity for the American board of surgery in training examination

BACKGROUND

As principles of competency-based medical education are implemented into graduate medical education [GME] programs, it is imperative that the assessments employed are reliable and valid. Internal structure is a core component of validity evidence that has been under studied. In this study, we examined elements of the internal structure for the American Board of Surgery In-Training Examination [ABSITE].

METHODS

This national retrospective cohort study utilized performance data for general surgery residents from 2018 to 2023 to determine aspects of the internal structure for the ABSITE. The item difficulty, item discrimination, internal consistency, and dimensionality of the exam were calculated.

RESULTS

55,986 unique test results were obtained in the study. The ABSITE exhibits strong reliability, (Cronbach's alpha >0.9), and appropriate item difficulty (0.66) and item discrimination index (>0.2) for a formative examination. Results of the exploratory factor analysis reveals that the ABSITE is unidimensional.

CONCLUSIONS

The ABSITE is a unidimensional examination with strong reliability. Furthermore, the item difficulty and item discrimination levels of the exam are appropriate for a formative test. Future studies utilizing consensus methods could be employed to determine what specific construct is measured by the ABSITE.

Surgical training simulation modalities in minimally invasive surgery: How to achieve evidence-based curricula by translational research

BACKGROUND

Surgery has evolved from a hands-on discipline where skills were acquired via the "learning by doing" principle to a surgical science with attention to patient safety, health care effectiveness and evidence-based research. A variety of simulation modalities have been developed to meet the need for effective resident training. So far, research regarding surgical training for minimally invasive surgery has been extensive but also heterogenous in grade of evidence.

METHODS

A literature search was conducted to summarize current knowledge about simulation training and to guide research towards evidence-based curricula with translational effects. This was conducted using a variety of terms in PubMed for English articles up to October 2024. Results are presented in a structured narrative review.

RESULTS

For virtual reality simulators, there is sound evidence for effective training outcomes. The required instruments for the development of minimally invasive surgery curricula combining different simulation modalities to create a clinical benefit are known and published.

CONCLUSION

Surgeons are the main creators for minimally invasive surgery training curricula and often follow a hands-on oriented approach that leaves out equally important aspects of assessment, evaluation, and feedback. Further high-quality research that includes available evidence in this field promises to improve patient safety in surgical disciplines.

Correlation of Performance on the ENTRUST Assessment Platform With Other Variables in Competency-Based Surgical Education

OBJECTIVE

With the implementation of American Board of Surgery (ABS) Entrustable Professional Activities (EPAs), there is continued need for objective, evidence-based assessment tools to augment existing microassessments and inform readiness for entrustment. The ENTRUST Assessment Platform is an online virtual-patient simulation platform to assess trainees' surgical decision-making competence across preoperative, intraoperative, and postoperative phases of care. This study collects additional validity evidence for the ENTRUST platform in its relationship to other established variables in competency-based surgical education.

DESIGN

This is a prospective analysis of surgical resident performance on the ENTRUST Right Lower Quadrant (RLQ) pain/Appendicitis EPA Assessment. ENTRUST scores were analyzed by PGY-level and correlations with Accreditation Council for Graduate Medical Education (ACGME) Case Logs, ACGME Surgery Milestones, and ABS In-Service Training Examination (ABSITE) scores were evaluated. Bivariate analyses were performed using Spearman rank correlations.

SETTING

This study was conducted at a tertiary academic center (Stanford University, Palo Alto, CA) in a proctored exam setting.

PARTICIPANTS

Thirty-two PGY-1 though PGY-5 general surgery residents completed the ENTRUST RLQ Pain/Appendicitis EPA Assessment containing four case scenarios which were iteratively developed and scored by expert consensus and aligned with ABS EPA definitions.

RESULTS

ENTRUST grand total score was positively correlated with PGY-level (rho = 0.57, p = 0.001), ACGME appendectomy case log volume (rho = 0.55, p = 0.002), and ABSITE raw score (rho = 0.66, p = 0.0004). ENTRUST performance was significantly correlated with all eighteen ACGME Surgery Milestones (rho = 0.43 to rho = 0.54, all p≤0.01), with the strongest correlation seen for PC1 (Patient Evaluation and Decision Making) (rho = 0.54, p = 0.006).

CONCLUSIONS

Performance on ENTRUST was significantly correlated with established variables in surgical training, including ACGME Appendectomy Case Logs, ABSITE, and ACGME Surgery Milestones. This study strengthens existing validity evidence for the ENTRUST Assessment Platform as an objective assessment of clinical decision-making. ENTRUST is an assessment tool which can augment microassessments and support competency-based medical education.

The pattern of reporting and presenting validity evidence of extended matching questions (EMQs) in health professions education: a systematic review

The Extended matching Questions (EMQs), or R-type questions, are format of selected-response. The validity evidence for this format is crucial, but there have been reports of misunderstandings about validity. It is unclear what kinds of evidence should be presented and how to present them to support their educational impact. This review explores the pattern and quality of reporting the sources of validity evidence of EMQs in health professions education, encompassing content, response process, internal structure, relationship to other variables, and consequences. A systematic search in the electronic databases including MEDLINE via PubMed, Scopus, Web of Science, CINAHL, and ERIC was conducted to extract studies that utilize EMQs. The framework for a unitary concept of validity was applied to extract data. A total of 218 titles were initially selected, the final number of titles was 19. The most reported pieces of evidence were the reliability coefficient, followed by the relationship to another variable. Additionally, the adopted definition of validity is mostly the old tripartite concept. This study found that reporting and presenting validity evidence appeared to be deficient. The available evidence can hardly provide a strong validity argument that supports the educational impact of EMQs. This review calls for more work on developing a tool to measure the reporting and presenting validity evidence.

Beyond reliability: assessing rater competence when using a behavioural marker system

BACKGROUND

Behavioural marker systems are used across several healthcare disciplines to assess behavioural (non-technical) skills, but rater training is variable, and inter-rater reliability is generally poor. Inter-rater reliability provides data about the tool, but not the competence of individual raters. This study aimed to test the inter-rater reliability of a new behavioural marker system (PhaBS - pharmacists' behavioural skills) with clinically experienced faculty raters and near-peer raters. It also aimed to assess rater competence when using PhaBS after brief familiarisation, by assessing completeness, agreement with an expert rater, ability to rank performance, stringency or leniency, and avoidance of the halo effect.

METHODS

Clinically experienced faculty raters and near-peer raters attended a 30-min PhaBS familiarisation session. This was immediately followed by a marking session in which they rated a trainee pharmacist's behavioural skills in three scripted immersive acute care simulated scenarios, demonstrating good, mediocre, and poor performances respectively. Inter-rater reliability in each group was calculated using the two-way random, absolute agreement single-measures intra-class correlation co-efficient (ICC). Differences in individual rater competence in each domain were compared using Pearson's chi-squared test.

RESULTS

The ICC for experienced faculty raters was good at 0.60 (0.48-0.72) and for near-peer raters was poor at 0.38 (0.27-0.54). Of experienced faculty raters, 5/9 were competent in all domains versus 2/13 near-peer raters (difference not statistically significant). There was no statistically significant difference between the abilities of clinically experienced versus near-peer raters in agreement with an expert rater, ability to rank performance, stringency or leniency, or avoidance of the halo effect. The only statistically significant difference between groups was ability to compete the assessment (9/9 experienced faculty raters versus 6/13 near-peer raters, p = 0.0077).

CONCLUSIONS

Experienced faculty have acceptable inter-rater reliability when using PhaBS, consistent with other behaviour marker systems; however, not all raters are competent. Competence measures for other assessments can be helpfully applied to behavioural marker systems. When using behavioural marker systems for assessment, educators must start using such rater competence frameworks. This is important to ensure fair and accurate assessments for learners, to provide educators with information about rater training programmes, and to provide individual raters with meaningful feedback.

Legitimation Without Argumentation: An Empirical Discourse Analysis of 'Validity as an Argument' in Assessment

Introduction

Validity is frequently conceptualized in health professions education (HPE) assessment as an argument that supports the interpretation and uses of data. However, previous work has shown that many validity scholars believe argument and argumentation are relatively lacking in HPE. To better understand HPE's discourse around argument and argumentation with regard to assessment validity, the authors explored the discourses present in published HPE manuscripts.

Methods

The authors used a bricolage of critical discourse analysis approaches to understand how the language in influential peer reviewed manuscripts has shaped HPE's understanding of validity arguments and argumentation. The authors used multiple search strategies to develop a final corpus of 39 manuscripts that were seen as influential in how validity arguments are conceptualized within HPE. An analytic framework drawing on prior research on Argumentation Theory was used to code manuscripts before developing themes relevant to the research question.

Results

The authors found that the elaboration of argument and argumentation within HPE's validity discourse is scant, with few components of Argumentation Theory (such as intended audience) existing within the discourse. The validity as an argument discourse was legitimized via authorization (reference to authority), rationalization (reference to institutionalized action), and mythopoesis (narrative building). This legitimation has cemented the validity as an argument discourse in HPE despite minimal exploration of what argument and argumentation are.

Discussion

This study corroborates previous work showing the dearth of argument and argumentation present within HPE's validity discourse. An opportunity exists to use Argumentation Theory in HPE to better develop validation practices that support use of argument.

Systematic Review of Procedural Skill Simulation in Health Care in Low- and Middle-Income Countries

SUMMARY STATEMENT

Low- and middle-income countries (LMICs) have adopted procedural skill simulation, with researchers increasingly investigating simulation efforts in resource-strained settings. We aim to summarize the current state of procedural skill simulation research in LMICs focusing on methodology, clinical area, types of outcomes and cost, cost-effectiveness, and overall sustainability. We performed a comprehensive literature review of original articles that assessed procedural skill simulation from database inception until April 2022.From 5371 screened articles, 262 were included in this review. All included studies were in English. Most studies were observational cohort studies (72.9%) and focused on obstetrics and neonatal medicine (32.4%). Most measured outcome was the process of task performance (56.5%). Several studies mentioned cost (38.9%) or sustainability (29.8%). However, few articles included actual monetary cost information (11.1%); only 1 article assessed cost-effectiveness. Based on our review, future research of procedural skill simulation in LMICS should focus on more rigorous research, cost assessments, and on less studied areas.

Validity in the Next Era of Assessment: Consequences, Social Impact, and Equity

Validity has long held a venerated place in education, leading some authors to refer to it as the "sine qua non" or "cardinal virtue" of assessment. And yet, validity has not held a fixed meaning; rather it has shifted in its definition and scope over time. In this Eye Opener, the authors explore if and how current conceptualizations of validity fit a next era of assessment that prioritizes patient care and learner equity. They posit that health profession education's conceptualization of validity will change in three related but distinct ways. First, consequences of assessment decisions will play a central role in validity arguments. Second, validity evidence regarding impacts of assessment on patients and society will be prioritized. Third, equity will be seen as part of validity rather than an unrelated concept. The authors argue that health professions education has the agency to change its ideology around validity, and to align with values that predominate the next era of assessment such as high-quality care and equity for learners and patients.

TEAMs go VR-validating the TEAM in a virtual reality (VR) medical team training

BACKGROUND

Inadequate collaboration in healthcare can lead to medical errors, highlighting the importance of interdisciplinary teamwork training. Virtual reality (VR) simulation-based training presents a promising, cost-effective approach. This study evaluates the effectiveness of the Team Emergency Assessment Measure (TEAM) for assessing healthcare student teams in VR environments to improve training methodologies.

METHODS

Forty-two medical and nursing students participated in a VR-based neurological emergency scenario as part of an interprofessional team training program. Their performances were assessed using a modified TEAM tool by two trained coders. Reliability, internal consistency, and concurrent validity of the tool were evaluated using intraclass correlation coefficients (ICC) and Cronbach's alpha.

RESULTS

Rater agreement on TEAM's leadership, teamwork, and task management domains was high, with ICC values between 0.75 and 0.90. Leadership demonstrated strong internal consistency (Cronbach's alpha = 0.90), while teamwork and task management showed moderate to acceptable consistency (alpha = 0.78 and 0.72, respectively). Overall, the TEAM tool exhibited high internal consistency (alpha = 0.89) and strong concurrent validity with significant correlations to global performance ratings.

CONCLUSION

The TEAM tool proved to be a reliable and valid instrument for evaluating team dynamics in VR-based training scenarios. This study highlights VR's potential in enhancing medical education, especially in remote or distanced learning contexts. It demonstrates a dependable approach for team performance assessment, adding value to VR-based medical training. These findings pave the way for more effective, accessible interdisciplinary team assessments, contributing significantly to the advancement of medical education.

Validation of the Learner Engagement Instrument for Continuing Professional Development

PURPOSE

Learner engagement is the energy learners exert to remain focused and motivated to learn. The Learner Engagement Instrument (LEI) was developed to measure learner engagement in a short continuing professional development (CPD) activity. The authors validated LEI scores using validity evidence of internal structure and relationships with other variables.

METHOD

Participants attended 1 of 4 CPD courses (1 in-person, 2 online livestreamed, and 1 either in-person or livestreamed) in 2018, 2020, 2021, and 2022. Confirmatory factor analysis was used to examine model fit for several alternative structural models, separately for each course. The authors also conducted a generalizability study to estimate score reliability. Associations were evaluated between LEI scores and Continuing Medical Education Teaching Effectiveness (CMETE) scores and participant demographics. Statistical methods accounted for repeated measures by participants.

RESULTS

Four hundred fifteen unique participants attended 203 different CPD presentations and completed the LEI 11,567 times. The originally hypothesized 4-domain model of learner engagement (domains: emotional, behavioral, cognitive in-class, cognitive out-of-class) demonstrated best model fit in all 4 courses, with comparative fit index ≥ 0.99, standardized root mean square residual ≤ 0.031, and root mean square error of approximation ≤ 0.047. The reliability for overall scores and domain scores were all acceptable (50-rater G-coefficient ≥ 0.74) except for the cognitive in-class domain (50-rater G-coefficient of 0.55 to 0.66). Findings were similar for both in-person and online delivery modalities. Correlation of LEI scores with teaching effectiveness was confirmed (rho=0.58), and a small correlation was found with participant age (rho=0.19); other associations were small and not statistically significant. Using these findings, we generated a shortened 4-item instrument, the LEI Short Form.

CONCLUSIONS

This study confirms a 4-domain model of learner engagement and provides validity evidence that supports using LEI scores to measure learner engagement in both in-person and livestreamed CPD activities.

Variance due to the examination conditions and factors associated with success in objective structured clinical examinations (OSCEs): first experiences at Paris-Saclay medical school

BACKGROUND

We aimed to measure the variance due to examination conditions during the first sessions of objective structured clinical examinations (OSCEs) performed at a French medical school and identify factors associated with student success.

METHODS

We conducted a retrospective, observational study using data from the first three OSCEs sessions performed at Paris-Saclay medical school in 2021 and 2022. For all sessions (each organized in 5 parallel circuits), we tested a circuit effect using a linear mixed-effects model adjusted for sex and the average academic level of students (according to written tests). Then, we studied the factors associated with student success at one station using a multivariate linear mixed-effects model, including the characteristics of students, assessors, and standardized patients.

RESULTS

The study included three OSCEs sessions, with 122, 175, and 197 students and a mean (± SD) session score of 13.7(± 1.5)/20, 12.7(± 1.7)/20 and 12.7(± 1.9)/20, respectively. The percentage of variance due to the circuit was 6.5%, 18.2% (statistically significant), and 3.8%, respectively. For all sessions, the student's average level and station scenario were significantly associated with the score obtained in a station. Still, specific characteristics of assessors or standardized patients were only associated with the student's score in April 2021 (first session).

CONCLUSION

The percentage of the variance of students' performance due to the examination conditions was significant in one out of three of the first OSCE sessions performed at Paris-Saclay medical school. This result seems more related to individual behaviors rather than specific characteristics of assessors or standardized patients, highlighting the need to continue training teaching teams.

NATIONAL CLINICAL TRIAL NUMBER

Not applicable.

Multi-institutional beta testing of a novel cervical esophagogastric anastomosis simulator

Objective

A novel simulator developed to offer hands-on practice for the stapled side-to-side cervical esophagogastric anastomosis was tested previously in a single-center study that supported its value in surgical education. This multi-institutional trial was undertaken to evaluate validity evidence from 6 independent thoracic surgery residency programs.

Methods

After a virtual session for simulation leaders, learners viewed a narrated video of the procedure and then alternated as surgeon or first assistant. Using an online survey, perceived value was measured across fidelity domains: physical attributes, realism of materials, realism of experience, value, and relevance. Objective assessment included time, number of sutures tearing, bubble test, and direct inspection. Comparison across programs was performed using the Kruskal-Wallis test.

Results

Surveys were completed by 63 participants as surgeons (17 junior and 20 senior residents, 18 fellows, and 8 faculty). For 3 of 5 tasks, mean ratings of 4.35 to 4.44 correlated with "somewhat easy" to "very easy" to perform. The interrupted outer layer of the anastomosis rated lowest, suggesting this task was the most difficult. The simulator was rated as a highly valuable training tool. For the objective measurements of performance, "direct inspection" rated highest followed by "time." A total of 90.5% of participants rated the simulator as ready for use with only minor improvements.

Conclusions

Results from this multi-institutional study suggest the cervical esophagogastric anastomosis simulator is a useful adjunct for training and assessment. Further research is needed to determine its value in assessing competence for independent operating and associations between improved measured performance and clinical outcomes.

Third Year Veterinary Student Academic Encumbrances and Tenacity: Navigating Clinical Skills Curricula and Assessment

This study is a qualitative exploration of the student experience of stress at the University of Calgary Faculty of Veterinary Medicine (UCVM). Ten third-year students from the class of 2019 participated in a semi-structured interview designed to explore the student experience surrounding stress, resiliency, and their perception of mental health awareness initiatives. Transcripts were de-identified and analyzed using thematic analysis. Two central themes of academic encumbrances and academic tenacity were identified, and each theme was further delineated into three main sub-themes associated with the Objective Structured Clinical Examination assessment. Students described the clinical skills course OSCEs as a primary source of stress and fear. Academic encumbrances were delineated through the sub-themes of academic load, fear of failure, and low academic self-efficacy. Students navigated these encumbrances through their academic tenacity, rooted in their sense of belonging among peers, family, and friends, a growth mindset, and self-care practices. Study results exemplify the importance of attending to student needs beyond acquiring clinical knowledge and skills. As we continue to redefine veterinary curricula and assessment, our work highlights the need for curricular streamlining, attending to student well-being, and fostering an academically balanced lifestyle. Lastly, the findings will help inform student services about the mental health needs of the students specific to academic achievement.

The impact of prior performance information on subsequent assessment: is there evidence of retaliation in an anonymous multisource assessment system?

Few studies have engaged in data-driven investigations of the presence, or frequency, of what could be considered retaliatory assessor behaviour in Multi-source Feedback (MSF) systems. In this study, authors explored how assessors scored others if, before assessing others, they received their own assessment score. The authors examined assessments from an established MSF system in which all clinical team members - medical students, interns, residents, fellows, and supervisors - anonymously assessed each other. The authors identified assessments in which an assessor (i.e., any team member providing a score to another) gave an aberrant score to another individual. An aberrant score was defined as one that was more than two standard deviations from the assessment receiver's average score. Assessors who gave aberrant scores were categorized according to whether their behaviour was preceded by: (1) receiving a score or not from another individual in the MSF system (2) whether the score they received was aberrant or not. The authors used a multivariable logistic regression model to investigate the association between the type of score received and the type of score given by that same individual. In total, 367 unique assessors provided 6091 scores on the performance of 484 unique individuals. Aberrant scores were identified in 250 forms (4.1%). The chances of giving an aberrant score were 2.3 times higher for those who had received a score, compared to those who had not (odds ratio 2.30, 95% CI:1.54-3.44, P < 0.001). Individuals who had received an aberrant score were also 2.17 times more likely to give an aberrant score to others compared to those who had received a non-aberrant score (2.17, 95% CI:1.39-3.39, P < 0.005) after adjusting for all other variables. This study documents an association between receiving scores within an anonymous multi-source feedback (MSF) system and providing aberrant scores to team members. These findings suggest care must be given to designing MSF systems to protect against potential downstream consequences of providing and receiving anonymous feedback.

Developing the TeamOBS-vacuum-assisted delivery checklist to assess clinical performance in a vacuum-assisted delivery: a Delphi study with initial validation

Introduction

In Northern Europe, vacuum-assisted delivery (VAD) accounts for 6-15% of all deliveries; VAD is considered safe when conducted by adequately trained personnel. However, failed vacuum extraction can be harmful to both the mother and child. Therefore, the clinical performance in VAD must be assessed to guide learning, determine a performance benchmark, and evaluate the quality to achieve an overall high performance. We were unable to identify a pre-existing tool for evaluating the clinical performance in real-life vacuum-assisted births.

Objective

We aimed to develop and validate a checklist for assessing the clinical performance in VAD.

Methods

We conducted a Delphi process, described as an interactive process where experts answer questions until answers converge toward a "joint opinion" (consensus). We invited international experts as Delphi panelists and reached a consensus after four Delphi rounds, described as follows: (1) the panelists were asked to add, remove, or suggest corrections to the preliminary list of items essential for evaluating clinical performance in VAD; (2) the panelists applied weights of clinical importance on a Likert scale of 1-5 for each item; (3) each panelist revised their original scores after reviewing a summary of the other panelists' scores and arguments; and (4) the TeamOBS-VAD was tested using videos of real-life VADs, and the Delphi panel made final adjustments and approved the checklist.

Results

Twelve Delphi panelists from the UK (n = 3), Norway (n = 2), Sweden (n = 3), Denmark (n = 3), and Iceland (n = 1) were included. After four Delphi rounds, the Delphi panel reached a consensus on the checklist items and scores. The TeamOBS-VAD checklist was tested using 60 videos of real-life vacuum extractions. The inter-rater agreement had an intraclass correlation coefficient (ICC) of 0.73; 95% confidence interval (95% CI) of [0.58, 0.83], and that for the average of two raters was ICC 0.84 95% CI [0.73, 0.91]. The TeamOBS-VAD score was not associated with difficulties in delivery, such as the number of contractions during vacuum extraction delivery, cephalic level, rotation, and position. Failed vacuum extraction occurred in 6% of the video deliveries, but none were associated with the teams with low clinical performance scores.

Conclusion

The TeamOBS-VAD checklist provides a valid and reliable evaluation of the clinical performance of vaginal-assisted vacuum extraction.

Surgical Artificial Intelligence in Urology: Educational Applications

Surgical education has seen immense change recently. Increased demand for iterative evaluation of trainees from medical school to independent practice has led to the generation of an overwhelming amount of data related to an individual's competency. Artificial intelligence has been proposed as a solution to automate and standardize the ability of stakeholders to assess the technical and nontechnical abilities of a surgical trainee. In both the simulation and clinical environments, evidence supports the use of machine learning algorithms to both evaluate trainee skill and provide real-time and automated feedback, enabling a shortened learning curve for many key procedural skills and ensuring patient safety.

Development of a Tool to Assess Surgical Resident Competence On-Call: The Western University Call Assessment Tool (WUCAT)

BACKGROUND

A central tenet of competency-based medical education is the formative assessment of trainees. There are currently no assessments designed to examine resident competence on-call, despite the on-call period being a significant component of residency, characterized by less direct supervision compared to daytime. The purpose of this study was to design a formative on-call assessment tool and collect valid evidence on its application.

METHODS

Nominal group technique was used to identify critical elements of surgical resident competence on-call to inform tool development. The tool was piloted over six months in the Division of Plastic & Reconstructive Surgery at our institution. Quantitative and qualitative evidence was collected to examine tool validity.

RESULTS

A ten-item tool was developed based on the consensus group results. Sixty-three assessments were completed by seven staff members on ten residents during the pilot. The tool had a reliability coefficient of 0.67 based on a generalizability study and internal item consistency was 0.92. Scores were significantly associated with years of training. We found the tool improved the quantity and structure of feedback given and that the tool was considered feasible and acceptable by both residents and staff members.

CONCLUSIONS

The Western University Call Assessment Tool (WUCAT) has multiple sources of evidence supporting its use in assessing resident competence on-call.

"Rater training" re-imagined for work-based assessment in medical education

In this perspective, the authors critically examine "rater training" as it has been conceptualized and used in medical education. By "rater training," they mean the educational events intended to improve rater performance and contributions during assessment events. Historically, rater training programs have focused on modifying faculty behaviours to achieve psychometric ideals (e.g., reliability, inter-rater reliability, accuracy). The authors argue these ideals may now be poorly aligned with contemporary research informing work-based assessment, introducing a compatibility threat, with no clear direction on how to proceed. To address this issue, the authors provide a brief historical review of "rater training" and provide an analysis of the literature examining the effectiveness of rater training programs. They focus mainly on what has served to define effectiveness or improvements. They then draw on philosophical and conceptual shifts in assessment to demonstrate why the function, effectiveness aims, and structure of rater training requires reimagining. These include shifting competencies for assessors, viewing assessment as a complex cognitive task enacted in a social context, evolving views on biases, and reprioritizing which validity evidence should be most sought in medical education. The authors aim to advance the discussion on rater training by challenging implicit incompatibility issues and stimulating ways to overcome them. They propose that "rater training" (a moniker they suggest be reserved for strong psychometric aims) be augmented with "assessor readiness" programs that link to contemporary assessment science and enact the principle of compatibility between that science and ways of engaging with advances in real-world faculty-learner contexts.

Validation of a culturally adapted Swedish-language version of the Death Literacy Index

The death literacy index (DLI) was developed in Australia to measure death literacy, a set of experience-based knowledge needed to understand and act on end-of-life (EOL) care options but has not yet been validated outside its original context. The aim of this study was to develop a culturally adapted Swedish-language version of the DLI, the DLI-S, and assess sources of evidence for its validity in a Swedish context. The study involved a multi-step process of translation and cultural adaptation and two validation phases: examining first content and response process validity through expert review (n = 10) and cognitive interviews (n = 10); and second, internal structure validity of DLI-S data collected from an online cross-sectional survey (n = 503). The psychometric evaluation involved analysis of descriptive statistics on item and scale-level, internal consistency and test-retest reliability, and confirmatory factor analysis. During translation and adaptation, changes were made to adjust items to the Swedish context. Additional adjustments were made following findings from the expert review and cognitive interviews. The content validity index exceeded recommended thresholds (S-CVIAve = 0.926). The psychometric evaluation provided support for DLI-S' validity. The hypothesized six-factor model showed good fit (χ2 = 1107.631 p<0.001, CFI = 0.993, TLI = 0.993, RMSEA = 0.064, SRMR = 0.054). High internal consistency reliability was demonstrated for the overall scale (Cronbach's α = 0.94) and each sub-scale (α 0.81-0.92). Test-retest reliability was acceptable, ICC ranging between 0.66-0.85. Through a comprehensive assessment of several sources of evidence, we show that the DLI-S demonstrates satisfactory validity and acceptability to measure death literacy in the Swedish context. There are, however, indications that the sub-scales measuring community capacity perform worse in comparison to other sca and may function differently in Sweden than in the original context. The DLI-S has potential to contribute to research on community-based EOL interventions.

Difficulties in using simulation to assess abdominal palpation skills

OBJECTIVES

Abdominal palpation is an essential examination to diagnose various digestive system diseases. This study aimed to develop an objective and standardized test based on abdominal palpation simulators, and establish a credible pass/fail standard of basic competency.

METHODS

Two tests were designed using the newly developed Jucheng abdominal palpation simulator (test 1) and the AbSim simulator (test 2), respectively. Validity evidence for both tests was gathered according to Messick's contemporary framework by using experts to define test content and then administering the tests in a highly standardized way to participants of different experience. Different simulator setups modified by the built-in software were selected from hepatomegaly, splenomegaly, positive McBurney's sign plus rebound tenderness, gallbladder tenderness (Murphy's sign), pancreas tenderness, and a normal setup without pathologies, with six sets used in test 1 and five sets used in test 2. Different novices and experienced were included in the tests, and test 1 was also administered to an intermediate group. Scores and test time were collected and analyzed statistically.

RESULTS

The internal consistency reliability of test 1 and test 2 showed low Cronbach's alphas of 0.35 and -0.41, respectively. Cronbach's alpha for palpation time across cases were 0.65 for test 1 and 0.76 for test 2. There was no statistical difference in total time spent and total scores among the three groups in test 1 (P-values (ANOVA) were 0.53 and 0.35 respectively), nor between novices and experienced groups in test 2 (P-values (t-test) were 0.13 and 1.0 respectively). It was not relevant to try to establish pass/fail standards due to the low reliability and lack of discriminatory ability of the tests.

CONCLUSIONS

It was not possible to measure abdominal palpation skills in a valid way using either of the two standardized, simulation-based tests in our study. Assessment of the patient's abdomen using palpation is a challenging clinical skill that is difficult to simulate as it highly relies on tactile sensations and adequate responsiveness from the patients.

Development and validation of a new measurement instrument to assess internship experience of medical doctors in low-income and middle-income countries

Routine surveys are used to understand the training quality and experiences of junior doctors but there are lack of tools designed to evaluate the training experiences of interns in low-income and middle-income countries (LMICs) where working conditions and resource constraints are challenging. We describe our process developing and validating a 'medical internship experience scale' to address this gap, work involving nine LMICs that varied in geographical locations, income-level and internship training models. We used a scoping review of existing tools, content validity discussions with target populations and an expert panel, back-and-forth translations into four language versions and cognitive interviews to develop and test the tool. Using data collected from 1646 interns and junior medical doctors, we assessed factor structure and assessed its reliability and validity. Fifty items about experiences of medical internship were retained from an initial pool of 102 items. These 50 items represent 6 major factors (constructs): (1) clinical learning and supervision, (2) patient safety, (3) job satisfaction, (4) stress and burnout, (5) mental well-being, and (6) fairness and discrimination. We reflect on the process of multicountry scale development and highlight some considerations for others who may use our scale, using preliminary analyses of the 1646 responses to illustrate that the tool may produce useful data to identify priorities for action. We suggest this tool could enable LMICs to assess key metrics regarding intern straining and initial work experiences and possibly allow comparison across countries and over time, to inform better internship planning and management.

Maximizing Standardization While Ensuring Equity: Exploring the Role of Applicant Experiences, Attributes, and Metrics on Performance of a Surgery-Specific Situational Judgment Test

BACKGROUND

Situational judgment tests (SJT) are hypothetical but realistic scenario-based assessments that allow residency programs to measure judgment and decision-making among future trainees. A surgery-specific SJT was created to identify highly valued competencies among residency applicants. We aim to demonstrate a stepwise process for validation of this assessment for applicant screening through exploration of two often-overlooked sources of validity evidence - relations with other variables and consequences.

METHODS

This was a prospective multi-institutional study involving 7 general surgery residency programs. All applicants completed the SurgSJT, a 32-item test aimed to measure 10 core competencies: adaptability, attention to detail, communication, dependability, feedback receptivity, integrity, professionalism, resilience, self-directed learning, and team orientation. Performance on the SJT was compared to application data, including race, ethnicity, gender, medical school, and USMLE scores. Medical school rankings were determined based on the 2022 U.S. News & World Report rankings.

RESULTS

In total, 1491 applicants across seven residency programs were invited to complete the SJT. Of these, 1454 (97.5%) candidates completed the assessment. Applicants were predominantly White (57.5%), Asian (21.6%), Hispanic (9.7%), Black (7.3%), and 52% female. A total of 208 medical schools were represented, majority were allopathic (87.1%) and located in United States (98.7%). Less than a quarter of applicants (22.8%; N=337) were from a top 25 school based on U.S. News & World Report rankings for primary care, surgery, or research. Average USMLE Step 1 score was 235 (SD 37) and Step 2 score was 250 (SD 29). Sex, race, ethnicity, and medical school ranking did not significantly impact performance on the SJT. There was no relationship between SJT score and USMLE scores and medical school rankings.

CONCLUSIONS

We demonstrate the process of validity testing and importance of two specific sources of evidence-consequences and relations with other variables, in implementing future educational assessments.

Current Status of Simulation in Thoracic Surgical Training

BACKGROUND

Simulation is playing an increasingly important role in surgical training but is not yet a mandatory part of most surgical curricula. A simulator must undergo rigorous validation to verify it as a reliable tool. The aim of this study was to review the literature to identify simulators that are currently available to augment thoracic surgical training and to analyze any evidence supporting or validating them.

METHODS

A literature search of the MEDLINE (1946 to November 2022) and Embase (1947 to November 2022) databases was performed to identify simulators for basic skills and procedures in thoracic surgery. A selection of keywords were used to perform the literature search. After identification of appropriate articles, data were extracted and analyzed.

RESULTS

Thirty-three simulators were found in 31 articles. Simulators for basic skills (n = 13) and thoracic lobectomy (n = 13) were most commonly described, followed by miscellaneous (n = 7). Most models were of a hybrid modality (n = 18). Evidence of validity was established in 48.5% (n = 16) of simulators. In total, 15.2% (n = 5) of simulators had 3 or more elements of validity demonstrated, and only 3.0% (n = 1) accomplished full validation.

CONCLUSIONS

Numerous simulators of varying modality and fidelity exist for a variety of thoracic surgical skills and procedures, although validation evidence is frequently inadequate. Simulation models may be able to provide training in basic surgical and procedural skills; however, further assessment of validity needs to be undertaken before consideration of their integration into training programs.

Gathering validity evidence for a 3D-printed simulator for training of myringotomy and ventilation tube insertion

Objectives

This study aimed to gather validity evidence according to Messick's framework for a novel 3D-printed simulator for myringotomy with ventilation tube insertion for use in technical skills training of otorhinolaryngology (ORL) residents.

Methods

The study included 15 junior ORL residents (trainees) and 13 experienced teaching otolaryngologists (experts). Experts and trainees first received an identically structured introduction to the procedure, simulator, and simulation setup. Five procedures performed by each participant were video-recorded and ordered randomly for blinded rating by two independent raters. The rating tools used were a global rating scale (GBRS) and a task-specific checklist. Validity evidence was collected according to Messick's framework. Differences in time consumption and performance scores were analyzed. Finally, a pass/fail standard was established using the contrasting groups' method.

Results

Trainees used significantly more time per procedure (109 s, 95% CI: 99-120) than experts (82 s, 95% CI: 71-93; p < .001). Adjusted for repetition and rater leniency, experts achieved an average GBRS score of 18.8 (95% CI: 18.3-19.2) out of 20 points, whereas trainees achieved an average of 17.1 points (95% CI: 16.6-17.5; p < .001). In contrast to the task-specific checklist, the GBRS score discriminated between repetition number and participant experience. The pass/fail standard for the GBRS was established at 18.4 points.

Conclusion

We established educational validity evidence for a novel 3D-printed model for simulation-based training of ventilation tube insertion and established a reliable pass/fail standard.

Level of Evidence

1b.

3-D-Printed Models for Temporal Bone Training: A Validity Study

OBJECTIVE

3-D printing offers convenient and low-cost mastoidectomy training; nonetheless, training benefits using 3-D-printed temporal bones remain largely unexplored. In this study, we have collected validity evidence for a low-cost, 3-D-printed temporal bone for mastoidectomy training and established a credible pass/fail score for performance on the model.

STUDY DESIGN

A prospective educational study gathering validity evidence using Messick's validity framework.

SETTING

Seven Danish otorhinolaryngology training institutions.

PARTICIPANTS

Eighteen otorhinolaryngology residents (novices) and 11 experienced otosurgeons (experts).

INTERVENTION

Residents and experienced otosurgeons each performed two to three anatomical mastoidectomies on a low-cost, 3-D-printed temporal bone model produced in-house. After drilling, mastoidectomy performances were rated by three blinded experts using a 25-item modified Welling scale (WS).

MAIN OUTCOME MEASURE

Validity evidence using Messick's framework including reliability assessment applying both classical test theory and Generalizability theory.

RESULTS

Novices achieved a mean score of 13.9 points; experienced otosurgeons achieved 23.2 points. Using the contrasting groups method, we established a 21/25-point pass/fail level. The Generalizability coefficient was 0.91, and 75% of the score variance was attributable to participant performance, indicating a high level of assessment reliability. Subsequent D studies revealed that two raters rating one performance or one rater rating two performances were sufficiently reliable for high-stakes assessment.

CONCLUSION

Validity evidence supports using a low-cost, 3-D-printed model for mastoidectomy training. The model can be printed in-house using consumer-grade 3-D printers and serves as an additional training tool in the temporal bone curriculum. For competency-based training, we established a cut-off score of 21 of 25 WS points using the contrasting groups method.

Assessment Practices in Continuing Professional Development Activities in Health Professions: A Scoping Review

INTRODUCTION

In continuing professional development (CPD), educators face the need to develop and implement innovative assessment strategies to adhere to accreditation standards and support lifelong learning. However, little is known about the development and validation of these assessment practices. We aimed to document the breadth and depth of what is known about the development and implementation of assessment practices within CPD activities.

METHODS

We conducted a scoping review using the framework proposed by Arksey and O'Malley (2005) and updated in 2020. We examined five databases and identified 1733 abstracts. Two team members screened titles and abstracts for inclusion/exclusion. After data extraction, we conducted a descriptive analysis of quantitative data and a thematic analysis of qualitative data.

RESULTS

A total of 130 studies were retained for the full review. Most reported assessments are written assessments (n = 100), such as multiple-choice items (n = 79). In 99 studies, authors developed an assessment for research purpose rather than for the CPD activity itself. The assessment validation process was detailed in 105 articles. In most cases, the authors examined the content with experts (n = 57) or pilot-tested the assessment (n = 50). We identified three themes: 1-satisfaction with assessment choices; 2-difficulties experienced during the administration of the assessment; and 3-complexity of the validation process.

DISCUSSION

Building on the adage "assessment drives learning," it is imperative that the CPD practices contribute to the intended learning and limit the unintended negative consequences of assessment. Our results suggest that validation processes must be considered and adapted within CPD contexts.

Correlation of Performance on ENTRUST and Traditional Oral Objective Structured Clinical Examination for High-Stakes Assessment in the College of Surgeons of East, Central, and Southern Africa

BACKGROUND

To address the global need for accessible evidence-based tools for competency-based education, we developed ENTRUST, an innovative online virtual patient simulation platform to author and securely deploy case scenarios to assess surgical decision-making competence.

STUDY DESIGN

In partnership with the College of Surgeons of East, Central, and Southern Africa, ENTRUST was piloted during the Membership of the College of Surgeons (MCS) 2021 examination. Examinees (n = 110) completed the traditional 11-station oral objective structured clinical examinations (OSCEs), followed by 3 ENTRUST cases, authored to query similar clinical content of 3 corresponding OSCE cases. ENTRUST scores were analyzed for associations with MCS Examination outcome using independent sample t tests. Correlation of ENTRUST scores to MCS Examination Percentage and OSCE station scores was calculated with Pearson correlations. Bivariate and multivariate analyses were performed to evaluate predictors of performance.

RESULTS

ENTRUST performance was significantly higher in examinees who passed the MCS examination compared with those who failed (p < 0.001). The ENTRUST score was positively correlated with MCS Examination Percentage (p < 0.001) and combined OSCE station scores (p < 0.001). On multivariate analysis, there was a strong association between MCS Examination Percentage and ENTRUST Grand Total Score (p < 0.001), Simulation Total Score (p = 0.018), and Question Total Score (p < 0.001). Age was a negative predictor for ENTRUST Grand Total and Simulation Total Score, but not for Question Total Score. Sex, native language status, and intended specialty were not associated with performance on ENTRUST.

CONCLUSIONS

This study demonstrates feasibility and initial validity evidence for the use of ENTRUST in a high-stakes examination context for assessment of surgical decision-making. ENTRUST holds potential as an accessible learning and assessment platform for surgical trainees worldwide.

A study of validity and usability evidence for non-technical skills assessment tools in simulated adult resuscitation scenarios

BACKGROUND

Non-technical skills (NTS) assessment tools are widely used to provide formative and summative assessment for healthcare professionals and there are now many of them. This study has examined three different tools designed for similar settings and gathered evidence to test their validity and usability.

METHODS

Three NTS assessment tools designed for use in the UK were used by three experienced faculty to review standardized videos of simulated cardiac arrest scenarios: ANTS (Anesthetists' Non-Technical Skills), Oxford NOTECHS (Oxford NOn-TECHnical Skills) and OSCAR (Observational Skill based Clinical Assessment tool for Resuscitation). Internal consistency, interrater reliability and quantitative and qualitative analysis of usability were analyzed for each tool.

RESULTS

Internal consistency and interrater reliability (IRR) varied considerably for the three tools across NTS categories and elements. Intraclass correlation scores of three expert raters ranged from poor (task management in ANTS [0.26] and situation awareness (SA) in Oxford NOTECHS [0.34]) to very good (problem solving in Oxford NOTECHS [0.81] and cooperation [0.84] and SA [0.87] in OSCAR). Furthermore, different statistical tests of IRR produced different results for each tool. Quantitative and qualitative examination of usability also revealed challenges in using each tool.

CONCLUSIONS

The lack of standardization of NTS assessment tools and training in their use is unhelpful for healthcare educators and students. Educators require ongoing support in the use of NTS assessment tools for the evaluation of individual healthcare professionals or healthcare teams. Summative or high-stakes examinations using NTS assessment tools should be undertaken with at least two assessors to provide consensus scoring. In light of the renewed focus on simulation as an educational tool to support and enhance training recovery in the aftermath of COVID-19, it is even more important that assessment of these vital skills is standardized, simplified and supported with adequate training.

Serious gaming and virtual reality in the multimodal training of laparoscopic inguinal hernia repair: a randomized crossover study

BACKGROUND

The aim of this study was to assess the transferability of surgical skills for the laparoscopic hernia module between the serious game Touch Surgery™ (TS) and the virtual reality (VR) trainer Lap Mentor™. Furthermore, this study aimed to collect validity evidence and to discuss "sources of validity evidence" for the findings using the laparoscopic inguinal hernia module on TS.

METHODS

In a randomized crossover study, medical students (n = 40) in their clinical years performed laparoscopic inguinal hernia modules on TS and the VR trainer. TS group started with "Laparoscopic Inguinal Hernia Module" on TS (phase 1: Preparation, phase 2: Port Placement and Hernia Repair), performed the module first in training, then in test mode until proficiency was reached. VR group started with "Inguinal Hernia Module" on the VR trainer (task 1: Anatomy Identification, task 2: Incision and Dissection) and also performed the module until proficiency. Once proficiency reached in the first modality, the groups performed the other training modality until reaching proficiency. Primary endpoint was the number of attempts needed to achieve proficiency for each group for each task/phase.

RESULTS

Students starting with TS needed significantly less attempts to reach proficiency for task 1 on the VR trainer than students who started with the VR trainer (TS = 2.7 ± 0.6 vs. VR = 3.2 ± 0.7; p = 0.028). No significant differences for task 2 were observed between groups (TS = 2.3 ± 1.1 vs. VR = 2.1 ± 0.8; p = 0.524). For both phases on TS, no significant skill transfer from the VR trainer to TS was observed. Aspects of validity evidence for the module on TS were collected.

CONCLUSION

The results show that TS brought additional benefit to improve performances on the VR trainer for task 1 but not for task 2. Skill transfer from the VR trainer to TS could not be shown. VR and TS should thus be used in combination with TS first in multimodal training to ensure optimal training conditions.

Comparing Simulator Metrics and Rater Assessment of Laparoscopic Suturing Skills

BACKGROUND

Laparoscopic intracorporeal suturing is important to master and competence should be ensured using an optimal method in a simulated environment before proceeding to real operations. The objectives of this study were to gather validity evidence for two tools for assessing laparoscopic intracorporeal knot tying and compare the rater-based assessment of laparoscopic intracorporeal suturing with the assessment based on simulator metrics.

METHODS

Twenty-eight novices and 19 experienced surgeons performed four laparoscopic sutures on a Simball Box simulator twice. Two surgeons used the Intracorporeal Suturing Assessment Tool (ISAT) for blinded video rating.

RESULTS

Composite Simulator Score (CSS) had higher test-retest reliability than the ISAT. The correlation between the number performed procedures including suturing and ISAT score was 0.51, p<0.001, and 0.59 p<0.001 for CSS. We found an inter-rater reliability (0.72, p<0.001 for test 1 and 0.53 p<0.001 for test 2). The pass/fail rates for ISAT and CSS were similar.

CONCLUSION

CSS and ISAT provide similar results for assessing laparoscopic suturing but assess different aspects of performance. Using simulator metrics and raters' assessments in combination should be considered for a more comprehensive evaluation of laparoscopic knot-tying competency.

Simulation-based summative assessment in healthcare: an overview of key principles for practice

BACKGROUND

Healthcare curricula need summative assessments relevant to and representative of clinical situations to best select and train learners. Simulation provides multiple benefits with a growing literature base proving its utility for training in a formative context. Advancing to the next step, "the use of simulation for summative assessment" requires rigorous and evidence-based development because any summative assessment is high stakes for participants, trainers, and programs. The first step of this process is to identify the baseline from which we can start.

METHODS

First, using a modified nominal group technique, a task force of 34 panelists defined topics to clarify the why, how, what, when, and who for using simulation-based summative assessment (SBSA). Second, each topic was explored by a group of panelists based on state-of-the-art literature reviews technique with a snowball method to identify further references. Our goal was to identify current knowledge and potential recommendations for future directions. Results were cross-checked among groups and reviewed by an independent expert committee.

RESULTS

Seven topics were selected by the task force: "What can be assessed in simulation?", "Assessment tools for SBSA", "Consequences of undergoing the SBSA process", "Scenarios for SBSA", "Debriefing, video, and research for SBSA", "Trainers for SBSA", and "Implementation of SBSA in healthcare". Together, these seven explorations provide an overview of what is known and can be done with relative certainty, and what is unknown and probably needs further investigation. Based on this work, we highlighted the trustworthiness of different summative assessment-related conclusions, the remaining important problems and questions, and their consequences for participants and institutions of how SBSA is conducted.

CONCLUSION

Our results identified among the seven topics one area with robust evidence in the literature ("What can be assessed in simulation?"), three areas with evidence that require guidance by expert opinion ("Assessment tools for SBSA", "Scenarios for SBSA", "Implementation of SBSA in healthcare"), and three areas with weak or emerging evidence ("Consequences of undergoing the SBSA process", "Debriefing for SBSA", "Trainers for SBSA"). Using SBSA holds much promise, with increasing demand for this application. Due to the important stakes involved, it must be rigorously conducted and supervised. Guidelines for good practice should be formalized to help with conduct and implementation. We believe this baseline can direct future investigation and the development of guidelines.

Can web-based implementation interventions improve physician early diagnosis of cerebral palsy? Protocol for a 3-arm parallel superiority randomised controlled trial and cost-consequence analysis comparing adaptive and non-adaptive virtual patient instructional designs with control to evaluate effectiveness on physician behaviour, diagnostic skills and patient outcomes

INTRODUCTION

Cerebral palsy (CP) is the most common childhood physical disability. Accurate diagnosis before 6 months is possible using predictive tools and decision-making skills. Yet diagnosis is typically made at 12-24 months of age, hindering access to early interventions that improve functional outcomes. Change in practice is required for physicians in key diagnostic behaviours. This study aims to close the identified research-practice gap and increase accurate CP diagnosis before 6 months of age through tailored web-based implementation interventions. This trial will determine whether adaptive e-learning using virtual patients, targeting CP diagnostic behaviours and clinical decision-making skills, effectively changes physician behaviour and practice compared with non-adaptive e-learning instructional design or control.

METHODS AND ANALYSIS

This study is a 3-arm parallel superiority randomised controlled trial of two tailored e-learning interventions developed to expedite physician CP diagnosis. The trial will compare adaptive (arm 1) and non-adaptive (arm 2) instructional designs with waitlist control (arm 3) to evaluate change in physician behaviour, skills and diagnostic practice. A sample size of 275 paediatric physicians enables detection of small magnitude effects (0.2) of primary outcomes between intervention comparators with 90% power (α=0.05), allowing for 30% attrition. Barrier analysis, Delphi survey, Behaviour Change Wheel and learning theory frameworks guided the intervention designs. Adaptive and non-adaptive video and navigation sequences utilising virtual patients and clinical practice guideline content were developed, integrating formative key features assessment targeting clinical decision-making skills relative to CP diagnosis.Physician outcomes will be evaluated based on postintervention key feature examination scores plus preintervention/postintervention behavioural intentions and practice measures. Associations with CP population registers will evaluate real-world diagnostic patient outcomes. Intervention costs will be reported in a cost-consequence analysis from funders' and societal perspectives.

ETHICS AND DISSEMINATION

Ethics approved from The University of Sydney (Project number 2021/386). Results will be disseminated through peer-reviewed journals and scientific conferences.

TRIAL REGISTRATION NUMBER

Australian New Zealand Clinical Trials Registry: ACTRN 12622000184774.

How argumentation theory can inform assessment validity: A critical review

INTRODUCTION

Many health professions education (HPE) scholars frame assessment validity as a form of argumentation in which interpretations and uses of assessment scores must be supported by evidence. However, what are purported to be validity arguments are often merely clusters of evidence without a guiding framework to evaluate, prioritise, or debate their merits. Argumentation theory is a field of study dedicated to understanding the production, analysis, and evaluation of arguments (spoken or written). The aim of this study is to describe argumentation theory, articulating the unique insights it can offer to HPE assessment, and presenting how different argumentation orientations can help reconceptualize the nature of validity in generative ways.

METHODS

The authors followed a five-step critical review process consisting of iterative cycles of focusing, searching, appraising, sampling, and analysing the argumentation theory literature. The authors generated and synthesised a corpus of manuscripts on argumentation orientations deemed to be most applicable to HPE.

RESULTS

We selected two argumentation orientations that we considered particularly constructive for informing HPE assessment validity: New rhetoric and informal logic. In new rhetoric, the goal of argumentation is to persuade, with a focus on an audience's values and standards. Informal logic centres on identifying, structuring, and evaluating arguments in real-world settings, with a variety of normative standards used to evaluate argument validity.

DISCUSSION

Both new rhetoric and informal logic provide philosophical, theoretical, or practical groundings that can advance HPE validity argumentation. New rhetoric's foregrounding of audience aligns with HPE's social imperative to be accountable to specific stakeholders such as the public and learners. Informal logic provides tools for identifying and structuring validity arguments for analysis and evaluation.

Validity Evidence for ENTRUST as an Assessment of Surgical Decision-Making for the Inguinal Hernia Entrustable Professional Activity (EPA)

OBJECTIVE

As the American Board of Surgery (ABS) moves toward implementation of Entrustable Professional Activities (EPAs), there is a growing need for objective evaluation of readiness for entrustment of residents. This requires not only assessment of technical skills and knowledge, but also surgical decision-making in preoperative, intraoperative, and postoperative settings. We developed and piloted an Inguinal Hernia EPA Assessment on ENTRUST, a serious game-based online virtual patient simulation platform to assess trainees' decision-making competence.

DESIGN

This is a prospective analysis of resident performance on the ENTRUST Inguinal Hernia EPA Assessment using bivariate analyses.

SETTING

This study was conducted at an academic institution in a proctored exam setting.

PARTICIPANTS

Forty-three surgical residents completed the ENTRUST Inguinal Hernia EPA Assessment.

RESULTS

Four case scenarios for the Inguinal Hernia EPA and corresponding scoring algorithms were iteratively developed by expert consensus aligned with ABS EPA descriptions and functions. ENTRUST Inguinal Hernia Grand Total Score was positively correlated with PGY-level (p < 0.0001). Preoperative, Intraoperative, and Postoperative Total Scores were also positively correlated with PGY-level (p = 0.001, p = 0.006, and p = 0.038, respectively). Total Case Scores were positively correlated with PGY-level for cases representing elective unilateral inguinal hernia (p = 0.0004), strangulated inguinal hernia (p < 0.0001), and elective bilateral inguinal hernia (p = 0.0003). Preoperative Sub-Scores were positively correlated with PGY-level for all cases (p < 0.01). Intraoperative Sub-Scores were positively correlated with PGY-level for strangulated inguinal hernia and bilateral inguinal hernia (p = 0.0007 and p = 0.0002, respectively). Grand Total Score and Intraoperative Sub-Score were correlated with prior operative experience (p < 0.0001). Prior video game experience did not correlate with performance on ENTRUST (p = 0.56).

CONCLUSIONS

Performance on the ENTRUST Inguinal Hernia EPA Assessment was positively correlated to PGY-level and prior inguinal hernia operative performance, providing initial validity evidence for its use as an objective assessment for surgical decision-making. The ENTRUST platform holds potential as tool for assessment of ABS EPAs in surgical residency programs.

Development and validation of a composed canine simulator for advanced veterinary laparoscopic training

The development of innovative simulation models for veterinary laparoscopic surgery training is a priority today. This study aimed to describe a didactic simulation tool for the training of total laparoscopic gastropexy (TLG) with intracorporeal sutures in dogs. CALMA Veterinary Lap-trainer composite simulator (CLVTS) was developed from a plaster cast of 2 Great Dane canines mimicking the space and the correct position to carry out a TLG. After video instruction, 16 veterinarians with different degrees of experience in minimally invasive surgery (Experts, n = 6 and intermediates, n = 10) evaluated four sequential simulating TLG with intracorporeal suturing in the CLVTS. Subsequently, they completed an anonymous questionnaire analyzing the realism, usefulness, and educational quality of the simulator. The CLVTS showed a good preliminary acceptance (4.7/5) in terms of the usefulness and adequacy of the exercises that, in the participants' opinion, are appropriate and are related to the difficulty of the TLG. In addition, both experienced and intermediate surgeons gave high marks (4.5/5) to the feeling of realism, design, and practicality. There were no significant differences between the responses of the two groups. The results suggest that the CVLTS has both face and content validity. Where it can be practiced in a structured environment for the development of a total laparoscopic gastropexy with intracorporeal suture and without compromising patient safety, but still has some limitations of the scope of the study. Further studies are needed to establish the ability to assess or measure technical skills, including the degree of transferability to the actual surgical environment.

Using the DiCoT framework for integrated multimodal analysis in mixed-reality training environments

Simulation-based training (SBT) programs are commonly employed by organizations to train individuals and teams for effective workplace cognitive and psychomotor skills in a broad range of applications. Distributed cognition has become a popular cognitive framework for the design and evaluation of these SBT environments, with structured methodologies such as Distributed Cognition for Teamwork (DiCoT) used for analysis. However, the analysis and evaluations generated by such distributed cognition frameworks require extensive domain-knowledge and manual coding and interpretation, and the analysis is primarily qualitative. In this work, we propose and develop the application of multimodal learning analysis techniques to SBT scenarios. Using these analysis methods, we can use the rich multimodal data collected in SBT environments to generate more automated interpretations of trainee performance that supplement and extend traditional DiCoT analysis. To demonstrate the use of these methods, we present a case study of nurses training in a mixed-reality manikin-based (MRMB) training environment. We show how the combined analysis of the video, speech, and eye-tracking data collected as the nurses train in the MRMB environment supports and enhances traditional qualitative DiCoT analysis. By applying such quantitative data-driven analysis methods, we can better analyze trainee activities online in SBT and MRMB environments. With continued development, these analysis methods could be used to provide targeted feedback to learners, a detailed review of training performance to the instructors, and data-driven evidence for improving the environment to simulation designers.

Validity as a social imperative: users' and leaders' perceptions

INTRODUCTION

Recently, validity as a social imperative was proposed as an emerging conceptualization of validity in the assessment literature in health professions education (HPE). To further develop our understanding, we explored the perceived acceptability and anticipated feasibility of validity as a social imperative with users and leaders engaged with assessment in HPE in Canada.

METHODS

We conducted a qualitative interpretive description study. Purposeful and snowball sampling were used to recruit participants for semi-structured individual interviews and focus groups. Each transcript was analyzed by two team members and discussed with the team until consensus was reached.

RESULTS

We conducted five focus group and eleven interviews with two different stakeholder groups (users and leaders). Our findings suggest that the participants perceived the concept of validity as a social imperative as acceptable. Regardless of group, participants shared similar considerations regarding: the limits of traditional validity models, the concept's timeliness and relevance, the need to clarify some terms used to characterize the concept, the similarities with modern theories of validity, and the anticipated challenges in applying the concept in practice. In addition, participants discussed some limits with current approaches to validity in the context of workplace-based and programmatic assessment.

CONCLUSION

Validity as a social imperative can be interwoven throughout existing theories of validity and may represent how HPE is adapting traditional models of validity in order to respond to the complexity of assessment in HPE; however, challenges likely remain in operationalizing the concept prior to its implementation.

Validity Evidence for a Novel, Comprehensive Bag-Mask Ventilation Assessment Tool

OBJECTIVE

To develop a comprehensive competency assessment tool for pediatric bag-mask ventilation (pBMV) and demonstrate multidimensional validity evidence for this tool.

STUDY DESIGN

A novel pBMV assessment tool was developed consisting of 3 components: a 22-item-based checklist (trichotomized response), global rating scale (GRS, 5-point), and entrustment assessment (4-point). Participants' performance in a realistic simulation scenario was video-recorded and assessed by blinded raters. Multidimensional validity evidence for procedural assessment, including evidence for content, response-process, internal structure, and relation to other variables, was assessed. The scores of each scale were compared with training level. Item-based checklist scores also were correlated with GRS and entrustment scores.

RESULTS

Fifty-eight participants (9 medical students, 10 pediatric residents, 18 critical care/neonatology fellows, 21 critical care/neonatology attendings) were evaluated. The pBMV tool was supported by high internal consistency (Cronbach α = 0.867). Inter-rater reliability for the item-based checklist component was acceptable (r = 0.65, P < .0001). The item-based checklist scores differentiated between medical students and other providers (P < .0001), but not by other trainee level. GRS and entrustment scores significantly differentiated between training levels (P < .001). Correlation between skill item-based checklist and GRS was r = 0.489 (P = .0001) and between item-based checklist and entrustment score was r = 0.52 (P < .001). This moderate correlation suggested each component measures pBMV skills differently. The GRS and entrustment scores demonstrated moderate inter-rater reliability (0.42 and 0.46).

CONCLUSIONS

We established evidence of multidimensional validity for a novel entrustment-based pBMV competence assessment tool, incorporating global and entrustment-based assessments. This comprehensive tool can provide learner feedback and aid in entrustment decisions as learners progress through training.

Non-technical Skills for Medical Students: Validating the Tools of the Trade

The Medical Students’ Non-Technical Skills (Medi-StuNTS) is a behavioural marker scheme (BMS) designed to assess non-technical skills (NTS) in medical students in emergency simulations. This study aimed to assess the evidence for validity and usability of Medi-StuNTS by naive, near-peer educators. Nine doctors assessed four students in simulations of common medical emergencies. The scores were used to assess inter-rater reliability, inter-class correlation, and observability. Students and assessors completed questionnaires that assessed the tool’s usability and consequence.

Inter-rater agreement across all skill elements was “high” with rWG scores >0.8. An inter-class correlation was “good” with ICC3K kappa scores of 0.86 and 0.89 overall, when measured per simulation and per skills element respectively. Overall skill observability was high (>80%) except for coping with stress. Assessors found the tool “difficult to use” but “useful for feeding back in a constructive way”. Students appreciated the comprehensiveness of the feedback as well as knowing what to expect during debriefs.

This study has shown that the Medi-StuNTS BMS has good usability and evidence of validity in naive assessors and near-peer educators. It shows the particularly good internal structure and overall beneficial consequences. Further study will be necessary to understand how best to deploy it in formative and summative contexts.

Using Immersive Virtual Reality Simulation to Ensure Competence in Contrast-Enhanced Ultrasound

Contrast-enhanced ultrasound (CEUS) is used in various medical specialties as a diagnostic imaging tool and for procedural guidance. Experience in the procedure is currently attained via supervised clinical practice that is challenged by patient availability and risks. Prior simulation-based training and subsequent assessment could improve and ensure competence before performance on patients, but no simulator currently exists. Immersive virtual reality (IVR) is a new promising simulation tool that can replicate complex interactions and environments that are unfeasible to achieve by traditional simulators. This study was aimed at developing an IVR simulation-based test for core CEUS competencies and gathering validity evidence for the test in accordance with Messick's framework. The test was developed by IVR software specialists and clinical experts in CEUS and medical education and imitated a CEUS examination of a patient with a focal liver lesion with emphasis on the pre-contrast preparations. Twenty-five medical doctors with varying CEUS experience were recruited as test participants, and their results were used to analyze test quality and to establish a pass/fail standard. The final test of 23 test items had good internal reliability (Cronbach's α = 0.85) and discriminatory abilities. The risks of false positives and negatives (9.1% and 23.6%, respectively) were acceptable for the test to be used as a certification tool prior to supervised clinical training in CEUS.

Coagulation test understanding and ordering by medical trainees: Novel teaching approach

Background

Coagulation testing provides a prime opportunity to make an impact on the reduction of unnecessary laboratory test ordering, as there are clear indications for testing. Despite the prothrombin time/international normalized ratio and activated partial thromboplastin time being validated for specific clinical indications, they are frequently ordered as screening tests and often ordered together, suggesting a gap in understanding of coagulation.

Methods

Based on a needs assessment, we developed an online educational module on coagulation for trainees, incorporating education on testing cost, specificity, and sensitivity. Fifty participating resident physicians and medical students completed a validated premodule quiz, postmodule quiz after completion of the module, and a latent quiz 3 to 6 months after to assess longer-term knowledge retention. Trainees provided responses regarding their subjective laboratory test-ordering practices before and after module completion.

Results

The median premodule quiz score was 67% (n = 50; range, 24%-86%) with an increase of 24% to a median postmodule quiz score of 91% (n = 50; range, 64%-100%). There was evidence of sustained knowledge acquisition with a latent quiz median score of 89% (n = 40; range, 67%-100%). Trainees were more likely to consider the sensitivity, specificity, and cost of laboratory investigations before ordering them following completion of the educational module.

Conclusions

Using the expertise of medical educators and incorporating trainee feedback, we employed a novel approach to the teaching of coagulation to maximize its approachability and clinical relevance. We found sustained knowledge retention regarding coagulation and appropriate coagulation test ordering, and a subjective change to trainee ordering habits following participation in our educational intervention.

Physician Training for Electrocardiogram Interpretation: A Systematic Review and Meta-Analysis

PURPOSE

Using electrocardiogram (ECG) interpretation as an example of a widely taught diagnostic skill, the authors conducted a systematic review and meta-analysis to demonstrate how research evidence on instruction in diagnosis can be synthesized to facilitate improvement of educational activities (instructional modalities, instructional methods, and interpretation approaches), guide the content and specificity of such activities, and provide direction for research.

METHOD

The authors searched PubMed/MEDLINE, Embase, Cochrane CENTRAL, PsycInfo, CINAHL, ERIC, and Web of Science databases through February 21, 2020, for empirical investigations of ECG interpretation training enrolling medical students, residents, or practicing physicians. They appraised study quality with the Medical Education Research Study Quality Instrument and pooled standardized mean differences (SMDs) using random effects meta-analysis.

RESULTS

Of 1,002 articles identified, 59 were included (enrolling 17,251 participants). Among 10 studies comparing instructional modalities, 8 compared computer-assisted and face-to-face instruction, with pooled SMD 0.23 (95% CI, 0.09, 0.36) indicating a small, statistically significant difference favoring computer-assisted instruction. Among 19 studies comparing instructional methods, 5 evaluated individual versus group training (pooled SMD -0.35 favoring group study [95% CI, -0.06, -0.63]), 4 evaluated peer-led versus faculty-led instruction (pooled SMD 0.38 favoring peer instruction [95% CI, 0.01, 0.74]), and 4 evaluated contrasting ECG features (e.g., QRS width) from 2 or more diagnostic categories versus routine examination of features within a single ECG or diagnosis (pooled SMD 0.23 not significantly favoring contrasting features [95% CI, -0.30, 0.76]). Eight studies compared ECG interpretation approaches, with pooled SMD 0.92 (95% CI, 0.48, 1.37) indicating a large, statistically significant effect favoring more systematic interpretation approaches.

CONCLUSIONS

Some instructional interventions appear to improve learning in ECG interpretation; however, many evidence-based instructional strategies are insufficiently investigated. The findings may have implications for future research and design of training to improve skills in ECG interpretation and other types of visual diagnosis.

Assessments of Physicians' Electrocardiogram Interpretation Skill: A Systematic Review

PURPOSE

To identify features of instruments, test procedures, study design, and validity evidence in published studies of electrocardiogram (ECG) skill assessments.

METHOD

The authors conducted a systematic review, searching MEDLINE, Embase, Cochrane CENTRAL, PsycINFO, CINAHL, ERIC, and Web of Science databases in February 2020 for studies that assessed the ECG interpretation skill of physicians or medical students. Two authors independently screened articles for inclusion and extracted information on test features, study design, risk of bias, and validity evidence.

RESULTS

The authors found 85 eligible studies. Participants included medical students (42 studies), postgraduate physicians (48 studies), and practicing physicians (13 studies). ECG selection criteria were infrequently reported: 25 studies (29%) selected single-diagnosis or straightforward ECGs; 5 (6%) selected complex cases. ECGs were selected by generalists (15 studies [18%]), cardiologists (10 studies [12%]), or unspecified experts (4 studies [5%]). The median number of ECGs per test was 10. The scoring rubric was defined by 2 or more experts in 32 studies (38%), by 1 expert in 5 (6%), and using clinical data in 5 (6%). Scoring was performed by a human rater in 34 studies (40%) and by computer in 7 (8%). Study methods were appraised as low risk of selection bias in 16 studies (19%), participant flow bias in 59 (69%), instrument conduct and scoring bias in 20 (24%), and applicability problems in 56 (66%). Evidence of test score validity was reported infrequently, namely evidence of content (39 studies [46%]), internal structure (11 [13%]), relations with other variables (10 [12%]), response process (2 [2%]), and consequences (3 [4%]).

CONCLUSIONS

ECG interpretation skill assessments consist of idiosyncratic instruments that are too short, composed of items of obscure provenance, with incompletely specified answers, graded by individuals with underreported credentials, yielding scores with limited interpretability. The authors suggest several best practices.

Validity of a cardiology fellow performance assessment: reliability and associations with standardized examinations and awards

BACKGROUND

Most work on the validity of clinical assessments for measuring learner performance in graduate medical education has occurred at the residency level. Minimal research exists on the validity of clinical assessments for measuring learner performance in advanced subspecialties. We sought to determine validity characteristics of cardiology fellows' assessment scores during subspecialty training, which represents the largest subspecialty of internal medicine. Validity evidence included item content, internal consistency reliability, and associations between faculty-of-fellow clinical assessments and other pertinent variables.

METHODS

This was a retrospective validation study exploring the domains of content, internal structure, and relations to other variables validity evidence for scores on faculty-of-fellow clinical assessments that include the 10-item Mayo Cardiology Fellows Assessment (MCFA-10). Participants included 7 cardiology fellowship classes. The MCFA-10 item content included questions previously validated in the assessment of internal medicine residents. Internal structure evidence was assessed through Cronbach's α. The outcome for relations to other variables evidence was overall mean of faculty-of-fellow assessment score (scale 1-5). Independent variables included common measures of fellow performance.

FINDINGS

Participants included 65 cardiology fellows. The overall mean ± standard deviation faculty-of-fellow assessment score was 4.07 ± 0.18. Content evidence for the MCFA-10 scores was based on published literature and core competencies. Cronbach's α was 0.98, suggesting high internal consistency reliability and offering evidence for internal structure validity. In multivariable analysis to provide relations to other variables evidence, mean assessment scores were independently associated with in-training examination scores (beta = 0.088 per 10-point increase; p = 0.05) and receiving a departmental or institutional award (beta = 0.152; p = 0.001). Assessment scores were not associated with educational conference attendance, compliance with completion of required evaluations, faculty appointment upon completion of training, or performance on the board certification exam. R² for the multivariable model was 0.25.

CONCLUSIONS

These findings provide sound validity evidence establishing item content, internal consistency reliability, and associations with other variables for faculty-of-fellow clinical assessment scores that include MCFA-10 items during cardiology fellowship. Relations to other variables evidence included associations of assessment scores with performance on the in-training examination and receipt of competitive awards. These data support the utility of the MCFA-10 as a measure of performance during cardiology training and could serve as the foundation for future research on the assessment of subspecialty learners.

Ensuring competence in ultrasound-guided procedures-a validity study of a newly developed assessment tool

OBJECTIVES

To investigate the validity of the Interventional Ultrasound Skills Evaluation (IUSE) tool for assessment of procedural competence in ultrasound-guided procedures in a clinical environment, including a pass/fail score.

METHODS

Novices and experienced radiologists were recruited from four hospitals and were observed and assessed while performing ultrasound-guided procedures. Performances were assessed using the IUSE tool by two independent raters. Validity evidence was gathered in accordance with Messick's framework: response process was ensured by standardisation of written rater instructions. Internal structure was explored using Cronbach's alpha for internal consistency reliability; inter-rater reliability was calculated as Pearson's r independently across all ratings, and test-retest reliability was reported using Cronbach's alpha. Relationship to other variables was investigated by comparing performances of the participants in each group. Consequences evidence was explored by calculating a pass/fail standard using the contrasting groups method.

RESULTS

Six novices and twelve experienced radiologists were enrolled. The IUSE tool had high internal consistency (Cronbach's alpha = 0.96, high inter-rater reliability (Pearson's r = 0.95), and high test-retest reliability (Cronbach's alpha = 0.98), and the mean score was 33.28 for novices and 59.25 for experienced with a highly significant difference (p value < 0.001). The pass/fail score was set at 55 resulting in no false positives or false negatives.

CONCLUSIONS

Validity evidence from multiple sources supports the use of the IUSE tool for assessment of competence in ultrasound-guided procedures in a clinical environment and its use in high-stakes assessment such as certification. A credible pass/fail criterion was established to inform decision-making.

KEY POINTS

• A multi-site validity investigation established that the Interventional Ultrasound Skills Evaluation (IUSE) tool can be used to assess procedural competence in ultrasound-guided procedures. • Validity evidence was gathered according to Messick's framework validity from the following sources: response process, internal structure, relationship to other variables, and consequences evidence. • The IUSE tool can be used for both formative and summative assessment, and a credible pass/fail score was established to help inform decision-making such as certification.

Validity evidence for the Anesthesia Clinical Encounter Assessment (ACEA) tool to support competency-based medical education

BACKGROUND

Workplace-based assessment (WBA) is key to a competency-based assessment strategy. Concomitantly with our programme's launch of competency-based medical education, we developed an entrustment-based WBA, the Anesthesia Clinical Encounter Assessment (ACEA), to assess readiness for independent practice of competencies essential to perioperative patient care. This study aimed to examine validity evidence of the ACEA during postgraduate anaesthesiology training.

METHODS

The ACEA comprises an eight-item global rating scale (GRS), an overall independence rating, an eight-item checklist, and case details. ACEA data were extracted for University of Toronto anaesthesia residents from July 2017 to January 2020 from the programme's online assessment portal. Validity evidence was generated following Messick's validity framework, including response process, internal structure, relations with other variables, and consequences.

RESULTS

We analysed 8664 assessments for 137 residents completed by 342 assessors. From generalisability analysis, 10 independent observations (two assessments each from five assessors) were sufficient to achieve a reliability threshold of ≥0.70 for in-training assessments. A composite GRS score of 3.65/5 provided optimal sensitivity (93.6%) and specificity (90.8%) for determining entrustment on receiver operator characteristic curve analysis. Test-retest reliability was high (intraclass correlation coefficient [ICC_2,1]=0.81) for matched assessments within 14 days of each other. Composite GRS scores differed significantly between residents based on their training level (P<0.0001) and correlated highly with overall independence (0.91, P<0.001). The internal consistency of the GRS (α=0.96) was excellent.

CONCLUSIONS

This study supports the validity of the ACEA for assessing the competence of residents performing perioperative care and supports its use in competency-based anaesthesiology training.

Does a measure of Medical Professional Identity Formation predict communication skills performance?

OBJECTIVE

To validate an approach to measuring professional identity formation (PIF), we explore if the Professional Identity Essay (PIE), a stage score measure of medical professional identity (PI), predicts clinical communication skills.

METHODS

Students completed the PIE during medical school orientation and a 3-case Objective Structured Clinical Exam (OSCE) where standardized patients reliably assessed communication skills in 5 domains. Using mediation analyses, relationships between PIE stage scores and communication skills were explored.

RESULTS

For the 351 (89%) consenting students, controlling for individual characteristics, there were increases in patient counseling (6.5%, p<0.01), information gathering (4.3%, p = 0.01), organization and management (4.1%, p = 0.02), patient assessment (3.6%, p = 0.04), and relationship development (3.5%, p = 0.03) skills for every half stage increase in PIE score. The communication skills of lower socio-economic status (SES) students are indirectly impacted by their slightly higher PIE stage scores.

CONCLUSION

Higher PIE stage scores are associated with higher communication skills and lower SES.

PRACTICE IMPLICATIONS

PIE predicts critical clinical skills and identifies how SES and other characteristics indirectly impact future clinical performance, providing validity evidence for using PIE as a tool in longitudinal formative academic coaching, program and curriculum evaluation, and research.