The National Clinical Assessment Tool for Medical Students in the Emergency Department (NCAT-EM)

Nurse perspectives on the assessment of emergency medicine residents: A qualitative study

Background

Multisource and 360-degree feedback are emerging methods in the medical education assessment literature. Nursing feedback in particular has been identified as a point of emphasis regarding what medical education leadership would most value for incorporating into their evaluations. There is no currently accepted tool for multisource feedback in the emergency department (ED). To answer this call, we conducted a qualitative study to elucidate nurse perspectives on evaluation of emergency medicine residents.

Methods

We conducted individual semistructured interviews of ED nurses with experience working alongside residents via volunteer recruitment. Transcripts were coded independently by each author and then reviewed collaboratively to resolve differences. Codes were created by inductive thematic analysis and subsequently underwent constant comparison and classical content analysis. Recruitment continued until analysis of transcripts showed thematic saturation.

Results

We organized the interview transcript into chunks of information, arranging 407 quotes from the 10 interview transcripts into 17 codes and six themes regarding observable resident behaviors. Representative quotes and exemplar cases added detailed description to the meaning of these codes. The identified themes for observable resident behaviors included "general communication style," "medical knowledge," "efficiency," "patient communication," "nurse communication," and "professionalism." An additional 13 codes addressed the themes of barriers and catalysts for feedback.

Conclusions

The identified themes on observable resident behaviors aligned well with prior literature and with the Accreditation Council for Graduate Medical Education (ACGME) Milestones. The interviews also highlighted nurse perspectives on barriers and catalysts for feedback that would be useful to incorporate into the design of an assessment tool.

Assessing Individual Competency Differences Between Third- and Fourth-Year Medical Students Using the National Clinical Assessment Tool for Medical Students in the Emergency Department

Medical students rotating through emergency departments as part of their clinical education are typically evaluated using an on-shift evaluation tool. The National Clinical Assessment Tool for Medical Students in the Emergency Department (NCAT-EM) is the current standard of evaluation for medical students in the emergency department, regardless of level of training. This study aims to evaluate whether the NCAT-EM can detect differences in skill levels between third-year medical students (MS3s) and fourth-year medical students (MS4s) rotating at a level 1 trauma center and teaching institution. These authors hypothesized that MS4s should outperform MS3s across all assessment domains given their additional training. A total of 930 performance evaluations were gathered for MS3 and MS4 rotating between May 2022 and June 2023. There were 321 evaluations of MS3s and 609 evaluations of MS4s. Across the six assessment domains, MS4s had statistically significant higher performances in two domains - namely emergency recognition and management (fully entrustable: 37.4% vs. 23.8% (p = 0.03)) and communication (fully entrustable: 46.2% vs. 33.6% (p = 0.03)). These findings indicate that the use of the NCAT-EM at this institution reliably differentiated between MS3s and MS4s in these two assessment domains. There were trends suggesting MS4s outperform MS3s in the other four domains, which did not rise to the level of statistical significance, but are consistent with prior validation studies of the NCAT.

Validity Evidence for the Emergency Medicine Standardized Letter of Evaluation

BACKGROUND

The standardized letter of evaluation (SLOE) is the application component that program directors value most when evaluating candidates to interview and rank for emergency medicine (EM) residency. Given its successful implementation, other specialties, including otolaryngology, dermatology, and orthopedics, have adopted similar SLOEs of their own, and more specialties are considering creating one. Unfortunately, for such a significant assessment tool, no study to date has comprehensively examined the validity evidence for the EM SLOE.

OBJECTIVE

We summarized the published evidence for validity for the EM SLOE using Messick's framework for validity evidence.

METHODS

A scoping review of the validity evidence of the EM SLOE was performed in 2020. A scoping review was chosen to identify gaps and future directions, and because the heterogeneity of the literature makes a systematic review difficult. Included articles were assigned to an aspect of Messick's framework and determined to provide evidence for or against validity.

RESULTS

There have been 22 articles published relating to validity evidence for the EM SLOE. There is evidence for content validity; however, there is a lack of evidence for internal structure, relation to other variables, and consequences. Additionally, the literature regarding response process demonstrates evidence against validity.

CONCLUSIONS

Overall, there is little published evidence in support of validity for the EM SLOE. Stakeholders need to consider changing the ranking system, improving standardization of clerkships, and further studying relation to other variables to improve validity. This will be important across GME as more specialties adopt a standardized letter.

Assessment of Professionalism During the Emergency Medicine Clerkship Using the National Clinical Assessment Tool for Medical Students in Emergency Medicine

OBJECTIVES

In 2016, a national consensus conference created the National Clinical Assessment Tool for Medical Students in Emergency Medicine (NCAT-EM), a standardized end-of-shift assessment tool. We report the first large-scale analysis of professionalism concerns collected from May 2017 through December 2018 by a multisite consortium using the NCAT-EM. Our primary objective was to characterize the nature and frequency of professionalism concerns. Our secondary objective was to identify characteristics associated with giving or receiving a professionalism flag.

METHODS

The consortium database includes assessments for all students on EM clerkships at participating sites. This report presents descriptive statistics about the frequency of different flags, the distribution of flags among different student categories, assessor and student characteristics, and distribution of global assessment scores on assessments citing concerns. We used Fisher's exact test to look for associations between the frequency of professionalism flags and the sex of the students and assessors and across student categories. We used logistic regression to look for relationships between professionalism concerns and global assessment scores as well as intent to apply in EM.

RESULTS

We screened 6,768 assessments of 784 students by 719 assessors from 13 sites. After excluding assessments without flags and assessments with apparent data entry errors, we analyzed 57 (0.8%) assessments containing 79 flags. The most frequent flags were punctuality (25/79, 31.6%) and initiative (20/79, 25.3%). Few students received flags (42/784, 5.4%). Few assessors flagged concerns (41/719, 5.7%). We detected no correlation between the frequency of flags and whether a student was applying in EM or between the sex of students and assessors. Global scores of lower one-third appeared more often in assessments with a flag (30/57, 52.6% vs. 233/6,711, 3.5%).

CONCLUSIONS

Only 5.4% of students received flags. Punctuality and initiative accounted for a majority of citations. Professionalism flags correlated strongly with lower global assessment scores.

Multi-institutional Implementation of the National Clinical Assessment Tool in Emergency Medicine: Data From the First Year of Use

OBJECTIVES

Uniformly training physicians to provide safe, high-quality care requires reliable assessment tools to ensure learner competency. The consensus-derived National Clinical Assessment Tool in Emergency Medicine (NCAT-EM) has been adopted by clerkships across the country. Analysis of large-scale deidentified data from a consortium of users is reported.

METHODS

Thirteen sites entered data into a Web-based platform resulting in over 6,400 discrete NCAT-EM assessments from 748 students and 704 assessors. Reliability, internal consistency analysis, and factorial analysis of variance for hypothesis generation were performed.

RESULTS

All categories on the NCAT-EM rating scales and professionalism subdomains were used. Clinical rating scale and global assessment scores were positively skewed, similar to other assessments commonly used in emergency medicine (EM). Professionalism lapses were noted in <1% of assessments. Cronbach's alpha was >0.8 for each site; however, interinstitutional variability was significant. M4 students scored higher than M3 students, and EM-bound students scored higher than non-EM-bound students. There were site-specific differences based on number of prior EM rotations, but no overall association. There were differences in scores based on assessor faculty rank and resident training year, but not by years in practice. There were site-specific differences based on student sex, but overall no difference.

CONCLUSIONS

To our knowledge, this is the first large-scale multi-institutional implementation of a single clinical assessment tool. This study demonstrates the feasibility of a unified approach to clinical assessment across multiple diverse sites. Challenges remain in determining appropriate score distributions and improving consistency in scoring between sites.

Medical student perceptions of assessment systems, subjectivity, and variability on introductory dermatology clerkships

Background

Elective introductory clerkships in dermatology serve a critical function in providing formative experiences to medical students interested in the field. Although dermatology clerkships play a pivotal role in students’ career choices and residency preparation, the assessment systems used to evaluate students on these clerkships are widely different and likely affect student experiences.

Objective

This study aimed to explore the relationship between dermatology clerkship assessment systems and student experiences through interviews with students about their clerkship reflections and perceptions of assessment.

Methods

The authors contacted clerkship directors via the Association of Professors of Dermatology mailing list and invited them to provide a description of the assessment system at their institution. The authors, via contacted clerkship directors, then invited students who had completed an introductory dermatology clerkship in between 2018 and 2019 to provide a description of the assessment system at their institution and to participate in a qualitative interview about their experiences with assessment systems. The authors then iteratively synthesized interview transcripts using phenomenological analysis, in which a templated approach was used to achieve comprehensive thematic categorization.

Results

Prior to clerkship onset, students expressed a limited understanding of their clinical role and the assessment system. During the clerkship, students endorsed variable expectations across preceptors, limited feedback experiences, and pressures to perform for evaluators. After their clerkship, students continued to perceive assessment systems as nontransparent, subjective, and preordained.

Conclusion

Medical students perceived assessment systems on introductory dermatology clerkships to be unclear and arbitrary. Encouragingly, students also viewed these challenges in assessment as malleable, identifying several opportunities for educational reform in dermatology clerkships.

Development and Validation of a Lecture Assessment Tool for Emergency Medicine Residents

BACKGROUND

Delivering quality lectures is a critical skill for residents seeking careers in academia yet no validated tools for assessing resident lecture skills exist.

OBJECTIVES

The authors sought to develop and validate a lecture assessment tool.

METHODS

Using a nominal group technique, the authors derived a behaviorally anchored assessment tool. Baseline characteristics of resident lecturers including prior lecturing experience and perceived comfort with lecturing were collected. Faculty and senior residents used the tool to assess lecturer performance at weekly conference. A postintervention survey assessed the usability of the form and the quantity and quality of the feedback. Analysis of variance was used to identify relationships in performance within individual domains to baseline data. Generalizability coefficients and scatterplots with jitter were used to assess inter-rater reliability.

RESULTS

Of 64 residents assessed, most (68.8%) had previous lecturing experience and 6.3% had experience as a regional/national speaker. There was a significant difference in performance within the domains of Content Expertise (p < 0.001), Presentation Design/Structure (p = 0.014), and Lecture Presence (p = 0.001) for first-year versus fourth-year residents. Residents who had higher perceived comfort with lecturing performed better in the domains of Content Expertise (p = 0.035), Presentation Design/Structure (p = 0.037), and Lecture Presence (p < 0.001). We found fair agreement between raters in all domains except Goals and Objectives. Both lecturers and evaluators perceived the feedback delivered as specific and of adequate quantity and quality. Evaluators described the form as highly useable.

CONCLUSIONS

The derived behaviorally anchored assessment tool is a sufficiently valid instrument for the assessment of resident-delivered lectures.