A Pilot Study to Assess Feasibility of Lay Representation in Dental School Admissions Interviews

The McMaster Narrative Comment Rating Tool: Development and Initial Validity Evidence

CONSTRUCT

The McMaster Narrative Comment Rating Tool aims to capture critical features reflecting the quality of written narrative comments provided in the medical education context: valence/tone of language, degree of correction versus reinforcement, specificity, actionability, and overall usefulness.

BACKGROUND

Despite their role in competency-based medical education, not all narrative comments contribute meaningfully to the development of learners' competence. To develop solutions to mitigate this problem, robust measures of narrative comment quality are needed. While some tools exist, most were created in specialty-specific contexts, have focused on one or two features of feedback, or have focused on faculty perceptions of feedback, excluding learners from the validation process. In this study, we aimed to develop a detailed, broadly applicable narrative comment quality assessment tool that drew upon features of high-quality assessment and feedback and could be used by a variety of raters to inform future research, including applications related to automated analysis of narrative comment quality.

APPROACH

In Phase 1, we used the literature to identify five critical features of feedback. We then developed rating scales for each of the features, and collected 670 competency-based assessments completed by first-year surgical residents in the first six-weeks of training. Residents were from nine different programs at a Canadian institution. In Phase 2, we randomly selected 50 assessments with written feedback from the dataset. Two education researchers used the scale to independently score the written comments and refine the rating tool. In Phase 3, 10 raters, including two medical education researchers, two medical students, two residents, two clinical faculty members, and two laypersons from the community, used the tool to independently and blindly rate written comments from another 50 randomly selected assessments from the dataset. We compared scores between and across rater pairs to assess reliability.

FINDINGS

Single and average measures intraclass correlation (ICC) scores ranged from moderate to excellent (ICCs = .51-.83 and .91-.98) across all categories and rater pairs. All tool domains were significantly correlated (p's <.05), apart from valence, which was only significantly correlated with degree of correction versus reinforcement.

CONCLUSION

Our findings suggest that the McMaster Narrative Comment Rating Tool can reliably be used by multiple raters, across a variety of rater types, and in different surgical contexts. As such, it has the potential to support faculty development initiatives on assessment and feedback, and may be used as a tool to conduct research on different assessment strategies, including automated analysis of narrative comments.

Postbaccalaureate and graduate applicants to dental school: An analysis of holistic review

Although they are valued for their perceived maturity, resiliency, and diverse insight into dentistry, postbaccalaureate and graduate (PBGR) applicants face significant challenges in the admissions process. This study looks at how PBGR applicants are evaluated during the selection process at a US dental school. An analysis of metrics associated with PBGR applicants was performed, focusing on the demographic makeup, academic performance, and total experience hours compared to traditional applicants. Our results suggest that PBGR applicants who are successful in their postbaccalaureate/graduate course of study are also more likely to be admitted if they have a history of strong undergraduate performance. Our results also suggest that PBGR applicants with high self-disclosed employment hours are strongly considered. Taken together, these results suggest that holistic review has helped PBGR applicants in the admissions process but that further adjustments need to be implemented.