Psychometric Properties of Entrustable Professional Activity-Based Objective Structured Clinical Examinations During Transition From Undergraduate to Graduate Medical Education: A Generalizability Study

PURPOSE

The objective structured clinical examination (OSCE) assesses clinical competence in health sciences education. There is little research regarding the reliability and validity of using an OSCE during the transition from undergraduate to graduate medical education. The goal of this study was to measure the reliability of a unique 2-rater entrustable professional activity (EPA)-based OSCE format for transition to internship using generalizability theory for estimating reliability.

METHOD

During the 2018 to 2022 academic years, 5 cohorts of interns (n = 230) at the University of Iowa Hospital and Clinics participated in a 6-station OSCE assessment delivered during orientation. A univariate and multivariate generalizability study (G study) was conducted on the scores generated by the 3 cases in the orientation OSCE that shared the 2-rater format. This analysis was supplemented with an associated decision study (D study).

RESULTS

The univariate G study for the cases that used a simulated patient and a faculty rater demonstrated that this OSCE generated a moderately reliable score with 3 cases. The D study showed that increasing the OCSE to 12 cases yielded a mean score reliable enough (G = 0.76) for making high-stakes normative decisions regarding remediation and readiness to practice. The universe score correlation between 2 types of raters was 0.398. The faculty ratings displayed a larger proportion of universe (true) score variance and yielded a more reliable (G = 0.433) score compared with the standardized patient ratings (G = 0.337).

CONCLUSIONS

This study provides insight into the development of an EPA-based OSCE. The univariate G study demonstrated that when using the 2 rater types, this assessment could generate a moderately reliable score with 3 cases. The multivariate G study showed that the 2 types of raters assessed different aspects of clinical skills and faculty raters were more reliable.

ACR Appropriateness Criteria® Staging and Post-Therapy Assessment of Head and Neck Cancer

Imaging of head and neck cancer at initial staging and as part of post-treatment surveillance is a key component of patient care as it guides treatment strategy and aids determination of prognosis. Head and neck cancer includes a heterogenous group of malignancies encompassing several anatomic sites and histologies, with squamous cell carcinoma the most common. Together this comprises the seventh most common cancer worldwide. At initial staging comprehensive imaging delineating the anatomic extent of the primary site, while also assessing the nodal involvement of the neck is necessary. The treatment of head and neck cancer often includes a combination of surgery, radiation, and chemotherapy. Post-treatment imaging is tailored for the evaluation of treatment response and early detection of local, locoregional, and distant recurrent tumor. Cross-sectional imaging with CT or MRI is recommended for the detailed anatomic delineation of the primary site. PET/CT provides complementary metabolic information and can map systemic involvement. The American College of Radiology Appropriateness Criteria are evidence-based guidelines for specific clinical conditions that are reviewed annually by a multidisciplinary expert panel. The guideline development and revision include an extensive analysis of current medical literature from peer reviewed journals and the application of well-established methodologies (RAND/UCLA Appropriateness Method and Grading of Recommendations Assessment, Development, and Evaluation or GRADE) to rate the appropriateness of imaging and treatment procedures for specific clinical scenarios. In those instances where evidence is lacking or equivocal, expert opinion may supplement the available evidence to recommend imaging or treatment.