The Ottawa resident observation form for nurses (O-RON): evaluation of an assessment tool's psychometric properties in different specialties

BACKGROUND

Workplace-based assessment (WBA) used in post-graduate medical education relies on physician supervisors' feedback. However, in a training environment where supervisors are unavailable to assess certain aspects of a resident's performance, nurses are well-positioned to do so. The Ottawa Resident Observation Form for Nurses (O-RON) was developed to capture nurses' assessment of trainee performance and results have demonstrated strong evidence for validity in Orthopedic Surgery. However, different clinical settings may impact a tool's performance. This project studied the use of the O-RON in three different specialties at the University of Ottawa.

METHODS

O-RON forms were distributed on Internal Medicine, General Surgery, and Obstetrical wards at the University of Ottawa over nine months. Validity evidence related to quantitative data was collected. Exit interviews with nurse managers were performed and content was thematically analyzed.

RESULTS

179 O-RONs were completed on 30 residents. With four forms per resident, the ORON's reliability was 0.82. Global judgement response and frequency of concerns was correlated (r = 0.627, P < 0.001).

CONCLUSIONS

Consistent with the original study, the findings demonstrated strong evidence for validity. However, the number of forms collected was less than expected. Exit interviews identified factors impacting form completion, which included clinical workloads and interprofessional dynamics.

Competence By Design: a transformational national model of time-variable competency-based postgraduate medical education

Postgraduate medical education is an essential societal enterprise that prepares highly skilled physicians for the health workforce. In recent years, PGME systems have been criticized worldwide for problems with variable graduate abilities, concerns about patient safety, and issues with teaching and assessment methods. In response, competency based medical education approaches, with an emphasis on graduate outcomes, have been proposed as the direction for 21st century health profession education. However, there are few published models of large-scale implementation of these approaches. We describe the rationale and design for a national, time-variable competency-based multi-specialty system for postgraduate medical education called Competence by Design. Fourteen innovations were bundled to create this new system, using the Van Melle Core Components of competency based medical education as the basis for the transformation. The successful execution of this transformational training system shows competency based medical education can be implemented at scale. The lessons learned in the early implementation of Competence by Design can inform competency based medical education innovation efforts across professions worldwide.

Design and Implementation of a National Program of Assessment Model - Integrating Entrustable Professional Activity Assessments in Canadian Specialist Postgraduate Medical Education

Traditional approaches to assessment in health professions education systems, which have generally focused on the summative function of assessment through the development and episodic use of individual high-stakes examinations, may no longer be appropriate in an era of competency based medical education. Contemporary assessment programs should not only ensure collection of high-quality performance data to support robust decision-making on learners' achievement and competence development but also facilitate the provision of meaningful feedback to learners to support reflective practice and performance improvement. Programmatic assessment is a specific approach to designing assessment systems through the intentional selection and combination of a variety of assessment methods and activities embedded within an educational framework to simultaneously optimize the decision-making and learning function of assessment. It is a core component of competency based medical education and is aligned with the goals of promoting assessment for learning and coaching learners to achieve predefined levels of competence. In Canada, postgraduate specialist medical education has undergone a transformative change to a competency based model centred around entrustable professional activities (EPAs). In this paper, we describe and reflect on the large scale, national implementation of a program of assessment model designed to guide learning and ensure that robust data is collected to support defensible decisions about EPA achievement and progress through training. Reflecting on the design and implications of this assessment system may help others who want to incorporate a competency based approach in their own country.

Abdominal MRI on a Commercial 0.55T System: Initial Evaluation and Comparison to Higher Field Strengths

RATIONALE AND OBJECTIVES

This study aims to assess the quality of abdominal MR images acquired on a commercial 0.55T scanner and compare these images with those acquired on conventional 1.5T/3T scanners in both healthy subjects and patients.

MATERIALS AND METHODS

Fifteen healthy subjects and 52 patients underwent abdominal Magnetic Resonance Imaging at 0.55T. Images were also collected in healthy subjects at 1.5T, and comparison 1.5/3T images identified for 28 of the 52 patients. Image quality was rated by two radiologists on a 4-point Likert scale. Readers were asked whether they could answer the clinical question for patient studies. Wilcoxon signed-rank test was used to test for significant differences in image ratings and acquisition times, and inter-reader reliability was computed.

RESULTS

The overall image quality of all sequences at 0.55T were rated as acceptable in healthy subjects. Sequences were modified to improve signal-to-noise ratio and reduce artifacts and deployed for clinical use; 52 patients were enrolled in this study. Radiologists were able to answer the clinical question in 52 (reader 1) and 46 (reader 2) of the patient cases. Average image quality was considered to be diagnostic (>3) for all sequences except arterial phase FS 3D T1w gradient echo (GRE) and 3D magnetic resonance cholangiopancreatography for one reader. In comparison to higher field images, significantly lower scores were given to 0.55T IP 2D GRE and arterial phase FS 3D T1w GRE, and significantly higher scores to diffusion-weighted echo planar imaging at 0.55T; other sequences were equivalent. The average scan time at 0.55T was 54 ± 10 minutes vs 36 ± 11 minutes at higher field strengths (P < .001).

CONCLUSION

Diagnostic-quality abdominal MR images can be obtained on a commercial 0.55T scanner at a longer overall acquisition time compared to higher field systems, although some sequences may benefit from additional optimization.

Exploring gender influences in the quality of workplace-based assessments

INTRODUCTION

Workplace-based assessments are an important tool for trainee feedback and as a means of reporting expert judgments of trainee competence in the workplace. However, the literature has demonstrated that gender bias can exist within these assessments. We aimed to determine whether gender differences in the quality of workplace-based assessment data exist in our residency training program.

METHODS

This study was conducted at the University of Ottawa in the Department of Emergency Medicine. Four end-of-shift workplace-based assessments completed by men faculty and four completed by women faculty were randomly selected for each resident during the 2018-2019 academic year. Two blinded raters scored each workplace-based assessment using the Completed Clinical Evaluation Report Rating (CCERR), a published nine-item quantitative measure of workplace-based assessment quality. A 2 × 2 mixed measures analysis of variance (ANOVA) of resident gender and faculty gender was conducted, with mean CCERR score as the dependent variable. The ANOVA was repeated with mean workplace-based assessment rating as the dependent variable.

RESULTS

A total of 363 workplace-based assessments were analyzed for 46 residents. There were no significant effects of faculty or resident gender on the quality of workplace-based assessments (p = 0.30). There was no difference in mean workplace-based assessment ratings between women and men residents (p = 0.92), and no interaction between resident and faculty gender (p = 0.62). Mean CCERR score was 25.8, SD = 4.2, indicating average quality assessments.

CONCLUSIONS

We did not find faculty or resident gender differences in the quality of workplace-based assessments completed in our training program. While the literature has previously demonstrated gender bias in trainee assessments, our results are not surprising as assessment culture varies by institution and program. Our study cautions against generalizing gender bias across contexts, and offers an approach that educators can use to evaluate whether gender bias in the quality of trainee assessments exists within their program.

Cross slope gait biomechanics for individuals with and without a unilateral transtibial amputation

BACKGROUND

This research was conducted to better understand compensatory strategies during cross-slope walking for adults with and without a unilateral transtibial amputation.

METHODS

Fourteen individuals with unilateral transtibial amputation and 14 individuals with no lower limb amputation participated in this study. Motion and force data were captured while participants walked on a treadmill in a virtual reality environment for level and ± 5° cross slopes. Temporal-spatial parameters, kinematics (ankle, knee, hip, pelvis, trunk), and ground reaction forces were examined.

FINDINGS

Compared to level, participants had similar step width but slightly longer steps for top-cross-slope and slightly shorter steps for bottom-cross-slope. Top-cross-slope required a more flexed limb with ankle eversion, and bottom-cross-slope required a more extended limb with ankle inversion. Participants had similar lateral pelvis and trunk motion for all walking conditions, but slightly more anterior trunk lean for top cross-slope with more anterior trunk lean observed for individuals with a lower limb amputation than without lower limb amputation. Participants with a lower limb amputation compensated for limited prosthetic ankle-foot dorsiflexion on the top-cross-slope by increasing prosthetic side hip flexion, reducing intact ankle/knee flexion, and increasing intact push-off force.

INTERPRETATION

Gait adaptations during cross-slope walking were primarily in the lower extremities and were largely similar for those with and without a transtibial amputation. The information presented in this paper provides a better understanding of gait strategies adopted during cross-slope walking and can guide researchers and industry in prosthetic development.

Navigating the uncertainty of health advocacy teaching and evaluation from the trainee's perspective

BACKGROUND

There may be no competency more shrouded in uncertainty than health advocacy (HA), raising questions about the robustness of advocacy training in postgraduate medical education. By understanding how programs currently train HA, we can identify whether trainees' learning needs are being met.

METHODS

From 2017 to 2019, we reviewed curricular documents across nine direct-entry specialties at all Ontario medical schools, comparing content for the HA and communicator roles to delineate role-specific challenges. We then conducted semi-structured interviews with trainees (n = 9) and faculty (n = 6) to review findings and discuss their impact. Data were analyzed using thematic content analysis.

RESULTS

Curricular documents revealed vague objectives and ill-defined modes of assessment for both intrinsic roles. This uncertainty was perceived as more problematic for HA, in part because HA seemed both undervalued in, and disconnected from, clinical learning. Trainees felt that the onus was on them to figure out how to develop and demonstrate HA competence, causing many to turn their learning attention elsewhere.

DISCUSSION

Lack of curricular focus seems to create the perception that advocacy isn't valuable, deterring trainees-even those keen to become competent advocates-from developing HA skills. Such ambivalence may have troubling downstream effects for both patient care and trainees' professional development.

The Ottawa Resident Observation Form for Nurses (O-RON): Assessment of Resident Performance through the Eyes of the Nurses

OBJECTIVE

Most work-place based assessment relies on physician supervisors making observations of residents. Many areas of performance are not directly observed by physicians but rather by other healthcare professionals, most often nurses. Assessment of resident performance by nurses is captured with multi-source feedback tools. However, these tools combine the assessments of nurses with other healthcare professionals and so their perspective can be lost. A novel tool was developed and implemented to assess resident performance on a hospital ward from the perspective of the nurses.

DESIGN

Through a nominal group technique, nurses identified dimensions of performance that are reflective of high-quality physician performance on a hospital ward. These were included as items in the Ottawa Resident Observation Form for Nurses (O-RON). The O-RON was voluntarily completed during an 11-month period. Validity evidence related to quantitative and qualitative data was collected.

SETTING

The Orthopedic Surgery Residency Program at the University of Ottawa.

PARTICIPANTS

49 nurses on the Orthopedic Surgery wards at The Ottawa Hospital (tertiary care).

RESULTS

The O-RON has 15 items rated on a 3-point frequency scale, one global judgment yes/no question regarding whether they would want the resident on their team and a space for comments. 1079 O-RONs were completed on 38 residents. There was an association between the response to the global judgment question and the frequency of concerns (p < 0.01). With 8 forms per resident, the reliability of the O-RON was 0.80. Open-ended responses referred to aspects of interpersonal skills, responsiveness, dependability, communication skills, and knowledge.

CONCLUSIONS

The O-RON demonstrates promise as a work-place based assessment tool to provide residents and training programs with feedback on aspects of their performance on a hospital ward through the eyes of the nurses. It appears to be easy to use, has solid evidence for validity and can provide reliable data with a small number of completed forms.

Exploring the use of rating scales with entrustment anchors in workplace-based assessment

PURPOSE

Competency-based medical education (CBME) has prompted widespread implementation of workplace-based assessment (WBA) tools using entrustment anchors. This study aimed to identify factors that influence faculty's rating choices immediately following assessment and explore their experiences using WBAs with entrustment anchors, specifically the Ottawa Surgical Competency Operating Room Evaluation scale.

METHOD

A convenience sample of 50 semi-structured interviews with Emergency Medicine (EM) physicians from a single Canadian hospital were conducted between July and August 2019. All interviews occurred within two hours of faculty completing a WBA of a trainee. Faculty were asked what they considered when rating the trainee's performance and whether they considered an alternate rating. Two team members independently analysed interview transcripts using conventional content analysis with line-by-line coding to identify themes.

RESULTS

Interviews captured interactions between 70% (26/37) of full-time EM faculty and 86% (19/22) of EM trainees. Faculty most commonly identified the amount of guidance the trainee required as influencing their rating. Other variables such as clinical context, trainee experience, past experiences with the trainee, perceived competence and confidence were also identified. While most faculty did not struggle to assign ratings, some had difficulty interpreting the language of entrustment anchors, being unsure whether their assessment should be retrospective or prospective in nature, and if/how the assessment should change whether they were 'in the room' or not.

CONCLUSIONS

By going to the frontline during WBA encounters, this study captured authentic and honest reflections from physicians immediately engaged in assessment using entrustment anchors. While many of the factors identified are consistent with previous retrospective work, we highlight how some faculty consider factors outside the prescribed approach and struggle with the language of entrustment anchors. These results further our understanding of 'in-the-moment' assessments using entrustment anchors and may facilitate effective faculty development regarding WBA in CBME.

Improving MR Image Quality in Patients with Metallic Implants

The number of implanted devices such as orthopedic hardware and cardiac implantable devices continues to increase with an increase in the age of the patient population, as well as an increase in the number of indications for specific devices. Many patients with these devices have or will develop clinical conditions that are best depicted at MRI. However, implanted devices containing paramagnetic or ferromagnetic substances can cause significant artifact, which could limit the diagnostic capability of this modality. Performing imaging with MRI when an implant is present may be challenging, and there are numerous techniques the radiologist and technologist can use to help minimize artifacts related to implants. First, knowledge of the presence of an implant before patient arrival is critical to ensure safety of the patient when the device is subjected to a strong magnetic field. Once safety is ensured, the examination should be performed with the MRI system that is expected to provide the best image quality. The selection of the MRI system includes multiple considerations such as the effects of field strength and availability of specific sequences, which can reduce metal artifact. Appropriate patient positioning, attention to MRI parameters (including bandwidth, voxel size, and echo), and appropriate selection of sequences (those with less metal artifact and advanced metal reduction sequences) are critical to improve image quality. Patients with implants can be successfully imaged with MRI with appropriate planning and understanding of how to minimize artifacts. This improves image quality and the diagnostic confidence of the radiologist. ^©RSNA, 2021.

Attaining full professor: Women's and men's experiences in medical education

INTRODUCTION

The underrepresentation of women among senior faculty members in medical education is a longstanding problem. The purpose of this international qualitative investigation was to explore women and men's experiences of attaining full professorship and to investigate why women remain underrepresented among the senior faculty ranks.

METHODS

Conducted within a social constructionist orientation, our qualitative study employed narrative analysis. Two female and two male participants working in medical education were recruited from five nations: Australia, Canada, the Netherlands, United Kingdom and United States. All participants held an MD or PhD. During telephone interviews, participants narrated the story of their careers. The five faculty members on the research team were also interviewed. Their narratives were included in analysis, rendering their experiences equal to those of the participants.

RESULTS

A total of 24 full professors working in medical education were interviewed (n = 15 females and n = 9 males). While some aspects were present across all narratives (ie personal events, career milestones and facilitating and/or impeding factors), participants' experience of those aspects differed by gender. Men did not narrate fatherhood as a role navigated professionally, but women narrated motherhood as intimately connected to their professional roles. Both men and women narrated career success in terms of hard work and overcoming obstacles; however, male participants described promotion as inevitable, whereas women narrated promotion as a tenuous navigation of social structures towards uncertain outcomes. Female and male participants encountered facilitators and inhibitors throughout their careers but described acting on those experiences differently within the cultural contexts they faced.

DISCUSSION

Our data suggest that female and male participants had different experiences of the work involved in achieving full professor status. Understanding these gendered experiences and their impact on career progression is an important advancement for better understanding what leads to the underrepresentation of women among senior faculty members in medical education.

eConsult Specialist Quality of Response (eSQUARE): A novel tool to measure specialist correspondence via electronic consultation

High-quality correspondence between healthcare providers is critical for effective patient care. We developed an assessment tool to measure the quality of specialist correspondence to primary care providers (PCPs) via electronic consultation (eConsult), where specialists provide advice without specialist-patient interactions. We incorporated fourteen previously described features of high-quality eConsult correspondence into an assessment tool named the eConsult Specialist Quality of Response (eSQUARE). Six PCPs and two specialists applied the 10-item eSQUARE tool to 30 eConsults of varying quality as informed by PCP survey data. Content, response process, and internal structure validity evidence was gathered. Psychometric properties were calculated using descriptive statistics and generalizability analyses. Mean total score for low-quality eConsults (M = 24 ± 5.6) was significantly lower than moderate-quality eConsults (M = 38 ± 4.7; p<0.001) which was significantly lower than high-quality eConsults (M = 46 ± 3.0; p = 0.002). Reliability measures were high, including generalizability coefficient (0.96), inter-item (≥0.55) and item-total correlations (≥0.68). A decision study demonstrated that a single rater was adequate to achieve a reliability measure of ≥0.70. This study demonstrates initial validity evidence including multiple reliability measures for the eSQUARE. A single rater is adequate to achieve reliability measures for formative feedback. Future studies can apply the eSQUARE when planning educational initiatives aiming to improve specialist-to-PCP correspondence via eConsult.

The Ottawa Emergency Department Shift Observation Tool (O-EDShOT): A New Tool for Assessing Resident Competence in the Emergency Department

OBJECTIVES

The outcome of emergency medicine (EM) training is to produce physicians who can competently run an emergency department (ED) shift. However, there are few tools with supporting validity evidence specifically designed to assess multiple key competencies across an entire shift. The investigators developed and gathered validity evidence for a novel entrustment-based tool to assess a resident's ability to safely run an ED shift.

METHODS

Through a nominal group technique, local and national stakeholders identified dimensions of performance that are reflective of a competent ED physician and are required to safely manage an ED shift. These were included as items in the Ottawa Emergency Department Shift Observation Tool (O-EDShOT), and each item was scored using an entrustment-based rating scale. The tool was implemented in 2018 at the University of Ottawa Department of Emergency Medicine, and quantitative data and qualitative feedback were collected over 6 months.

RESULTS

A total of 1,141 forms were completed by 78 physicians for 45 residents. An analysis of variance demonstrated an effect of training level with statistically significant increases in mean O-EDShOT scores with each subsequent postgraduate year (p < 0.001). Scores did not vary by ED treatment area. Residents rated as able to safely run the shift had significantly higher mean ± SD scores (4.8 ± 0.3) than those rated as not able (3.8 ± 0.6; p < 0.001). Faculty and residents reported that the tool was feasible to use and facilitated actionable feedback aimed at progression toward independent practice.

CONCLUSIONS

The O-EDShOT successfully discriminated between trainees of different levels regardless of ED treatment area. Multiple sources of validity evidence support the O-EDShOT as a tool to assess a resident's ability to safely run an ED shift. It can serve as a stimulus for daily observation and feedback making it practical to use within an EM residency program.

Validity of the Comprehensive High-Level Activity Mobility Predictor in a heterogeneous population with lower extremity amputations

BACKGROUND AND OBJECTIVE

Lower extremity amputee outcome measures assess basic mobility. The Comprehensive High-Level Activity Mobility Predictor was developed to assess high-level mobility. Validity evidence was collected in military men with traumatic lower extremity amputations. This study examines its validity in a broader population.

STUDY DESIGN

Cross-sectional.

METHODS

Forty-five lower extremity amputees (Medicare Functional Classification Level K3 or K4) completed the 2-min walk test, Amputee Mobility Predictor with Prosthesis, and Comprehensive High-Level Activity Mobility Predictor.

RESULTS

The Comprehensive High-Level Activity Mobility Predictor correlated with the Amputee Mobility Predictor with Prosthesis (r = 0.77, p < 0.01) and the 2-min walk test (r = 0.65, p < 0.01). The Comprehensive High-Level Activity Mobility Predictor differentiated between K-levels, age groups, etiology of amputation, and amputation level (p < 0.005). No ceiling effect was observed (range: 2.5-29/40).

CONCLUSION

This study provides convergent and discriminative validity evidence for Comprehensive High-Level Activity Mobility Predictor use in a more heterogeneous population than previously published, suggesting that clinicians should feel confident to use it as an outcome measure for individuals with amputations who are capable of more than level-ground walking.

CLINICAL RELEVANCE

Clinically, the Comprehensive High-Level Activity Mobility Predictor has validity evidence for use in a more heterogeneous population than originally demonstrated, including civilians, women, people over age 40 years, and non-traumatic etiologies. The Comprehensive High-Level Activity Mobility Predictor may be more useful than standard outcome measures for high-level mobility.

Orthosis Comfort Score: Establishing initial evidence of reliability and validity in ankle foot orthosis users

BACKGROUND

Comfort of an orthosis is an important characteristic that is likely to dictate use of and satisfaction with a device. However, instruments to assess only orthosis user comfort do not exist. The Prosthetic Socket Fit Comfort Score, developed previously for prosthesis users, may be adapted to serve this purpose.

OBJECTIVES

This study's purpose was to assess the validity and reliability of the Orthosis Comfort Score, a self-report instrument adapted from the Prosthetic Socket Fit Comfort Score.

STUDY DESIGN

This is a prospective, observational study designed to establish initial evidence of validity and reliability for an outcome measure that assesses comfort.

METHODS

Ankle foot orthosis users completed the Orthosis Comfort Score and two validated patient satisfaction questionnaires. An orthotist documented an assessment of fit. Post-visit Orthosis Comfort Scores were documented after the appointment and 2-4 weeks later. Orthosis Comfort Scores were compared to the patient satisfaction questionnaires, assessment of fit and orthosis use (hours per week).

RESULTS

There were 46 study participants. Orthosis Comfort Scores had a moderate positive correlation with their orthotist's assessment of fit, very strong positive correlations with patient satisfaction questionnaires and fair positive correlation with orthosis use (all correlations p < 0.05).

CONCLUSION

This study demonstrates initial evidence for the validity and reliability of the Orthosis Comfort Score in ankle foot orthosis users.

CLINICAL RELEVANCE

The Orthosis Comfort Score is a simple patient-reported outcome measure that can be readily incorporated into clinical practice or research study to obtain a rapid assessment of comfort. It can be used to facilitate communication about device fit, evaluate comfort over time and/or assess changes in comfort with a new device.

Faculty and Resident Perspectives on Using Entrustment Anchors for Workplace-Based Assessment

BACKGROUND

Research suggests that workplace-based assessment (WBA) tools using entrustment anchors provide more reliable assessments than those using traditional anchors. There is a lack of evidence describing how and why entrustment anchors work.

OBJECTIVE

The purpose of this study is to better understand the experience of residents and faculty with respect to traditional and entrustment anchors.

METHODS

We used constructivist grounded theory to guide data collection and analysis (March-December 2017) and semistructured interviews to gather reflections on anchors. Phase 1 involved residents and faculty (n = 12) who had only used assessment tools with traditional anchors. Phase 2 involved participants who had used tools with entrustment anchors (n = 10). Data were analyzed iteratively.

RESULTS

Participants expressed that the pragmatic language of entrustment anchors made WBA (1) concrete and justifiable; (2) transparent as they explicitly link clinical assessment and learning progress; and (3) align with training outcomes, enabling better feedback. Participants with no prior experience using entrustment anchors outlined contextual concerns regarding their use. Participants with experience described how they addressed these concerns. Participants expressed that entrustment anchors leave a gap in assessment information because they do not provide normative data.

CONCLUSIONS

Insights from this analysis contribute to a theoretical framework of benefits and challenges related to the adoption of entrustment anchors. This richer understanding of faculty and resident perspectives of entrustment anchors may assist WBA developers in creating more acceptable tools and inform the necessary faculty development initiatives that must accompany the use of these new WBA tools.  .

Comparison of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) to a Single-Item Performance Score

Construct: We compared a single-item performance score with the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) for their ability in assessing surgical competency.

BACKGROUND

Surgical programs are adopting competency-based frameworks. The adoption of these frameworks for assessment requires tools that produce accurate and valid assessments of knowledge and technical performance. An assessment tool that is quick to complete could improve feasibility, reduce delays, and result in a higher volume of assessments of learners. Previous work demonstrated that the 9-item O-SCORE can produce valid results; the goal of this study was to determine if a single-item performance rating (Is candidate competent to independently complete procedure: yes or no) completed at a separate viewing would correlate to the O-SCORE, thus increasing feasibility of procedural competence assessment.

APPROACH

Nineteen residents and 2 staff orthopedic surgeons from the University of Ottawa volunteered for a 2-part OSCE-style station including a written questionnaire and videotaped simulated open reduction and internal fixation midshaft radius fracture. Each performance was rated independently by 3 orthopedic surgeons using a single-item performance score (Time 1). The performances were assessed again 6 weeks later by the 3 raters using the O-SCORE (Time 2). Correlation between the single-item performance score and the O-SCORE were evaluated.

RESULTS

Three orthopedic surgeons completed 21 ratings each resulting in 63 orthopedic ratings. There was a high level of correlation and agreement between the single-item performance score at Time 1 and Time 2 (κ correlation =0.72-1.00; p < .001; percentage agreement =90%-100%). The reliability of the O-SCORE at Time 2 with three raters was 0.83 and the internal consistency was 0.89. There was a tendency for each rater to assign more yes responses to the more senior trainees.

CONCLUSIONS

A single-item performance score correlated highly with the O-SCORE in an orthopedic setting. A single-item score could be used to supplement a multi-item score with similar results in orthopedics. There is still benefit in completing multi-item scores such as the O-SCORE evaluations to guide specific areas of improvement and direct feedback.

Cyberbiosecurity: A New Perspective on Protecting U.S. Food and Agricultural System

Our national data and infrastructure security issues affecting the "bioeconomy" are evolving rapidly. Simultaneously, the conversation about cyber security of the U.S. food and agricultural system (cyber biosecurity) is incomplete and disjointed. The food and agricultural production sectors influence over 20% of the nation's economy ($6.7T) and 15% of U.S. employment (43.3M jobs). The food and agricultural sectors are immensely diverse and they require advanced technologies and efficiencies that rely on computer technologies, big data, cloud-based data storage, and internet accessibility. There is a critical need to safeguard the cyber biosecurity of our bio economy, but currently protections are minimal and do not broadly exist across the food and agricultural system. Using the food safety management Hazard Analysis Critical Control Point system concept as an introductory point of reference, we identify important features in broad food and agricultural production and food systems: dairy, food animals, row crops, fruits and vegetables, and environmental resources (water). This analysis explores the relevant concepts of cyber biosecurity from food production to the end product user (such as the consumer) and considers the integration of diverse transportation, supplier, and retailer networks. We describe common challenges and unique barriers across these systems and recommend solutions to advance the role of cyber biosecurity in the food and agricultural sectors.

Faculty Development- Is Some Better Than None?

This article was migrated. The article was marked as recommended. Introduction: With the advent of competency-based medical education there is an emphasis on formative workplace based assessment. The quality of these assessments is a concern for medical educators and their trainees. Faculty development (FD) strategies to improve assessment quality have resulted in some success. However, few faculty participate, and those who do are likely more motivated to improve, making it difficult to demonstrate a conclusive benefit. To address these weaknesses, we designed a FD initiative to improve the quality of completed in-training evaluation reports (ITERs). All faculty within a division participated. We hypothesized that clinical supervisors would improve their ITER quality based on feedback, regardless of their own motivation to do so, with a simple, point-in-time intervention. Methods: In this three-phase study, two independent raters used the Completed Clinical Evaluation Report Rating (CCERR) to assess the quality of ITERs completed by all faculty in the Division of Orthopedic Surgery at the University of Ottawa. In phase one, ITERs from the previous nine months were evaluated. In phase two, the participants were aware that their ITERs were being evaluated, but they did not receive feedback. In phase three, participants received regular feedback on their performance in the form of their mean CCERR scores. Mean CCERR scores from the different phases of the study were compared. Results: CCERR scores were similar for all three phases (one: 17.56 ± 1.02, two: 17.65 ± 0.96, three: 17.54 ± 0.75, p=0.98). Discussion and Conclusions: There was no evidence in our study that participants' improved their ITER quality despite being aware that they were being evaluated and/or receiving feedback. Potentially, this was related to a lack of motivation. Alternatively, the intensity and/or frequency of the feedback may have been inadequate to create change. These results raise concerns that some faculty development may not necessarily be better than none.

Avoid reinventing the wheel: implementation of the Ottawa Clinic Assessment Tool (OCAT) in Internal Medicine

BACKGROUND

Workplace based assessment (WBA) is crucial to competency-based education. The majority of healthcare is delivered in the ambulatory setting making the ability to run an entire clinic a crucial core competency for Internal Medicine (IM) trainees. Current WBA tools used in IM do not allow a thorough assessment of this skill. Further, most tools are not aligned with the way clinical assessors conceptualize performances. To address this, many tools aligned with entrustment decisions have recently been published. The Ottawa Clinic Assessment Tool (OCAT) is an entrustment-aligned tool that allows for such an assessment but was developed in the surgical setting and it is not known if it can perform well in an entirely different context. The aim of this study was to implement the OCAT in an IM program and collect psychometric data in this different setting. Using one tool across multiple contexts may reduce the need for tool development and ensure that tools used have proper psychometric data to support them.

METHODS

Psychometrics characteristics were determined. Descriptive statistics and effect sizes were calculated. Scores were compared between levels of training (juniors (PGY1), seniors (PGY2s and PGY3s) & fellows (PGY4s and PGY5s)) using a one-way ANOVA. Safety for independent practice was analyzed with a dichotomous score. Variance components were generated and used to estimate the reliability of the OCAT.

RESULTS

Three hundred ninety OCATs were completed over 52 weeks by 86 physicians assessing 44 residents. The range of ratings varied from 2 (I had to talk them through) to 5 (I did not need to be there) for most items. Mean scores differed significantly by training level (p < .001) with juniors having lower ratings (M = 3.80 (out of 5), SD = 0.49) than seniors (M = 4.22, SD = - 0.47) who had lower ratings than fellows (4.70, SD = 0.36). Trainees deemed safe to run the clinic independently had significantly higher mean scores than those deemed not safe (p < .001). The generalizability coefficient that corresponds to internal consistency is 0.92.

CONCLUSIONS

This study's psychometric data demonstrates that we can reliably use the OCAT in IM. We support assessing existing tools within different contexts rather than continuous developing discipline-specific instruments.

Using In-Training Evaluation Report (ITER) Qualitative Comments to Assess Medical Students and Residents: A Systematic Review

PURPOSE

In-training evaluation reports (ITERs) constitute an integral component of medical student and postgraduate physician trainee (resident) assessment. ITER narrative comments have received less attention than the numeric scores. The authors sought both to determine what validity evidence informs the use of narrative comments from ITERs for assessing medical students and residents and to identify evidence gaps.

METHOD

Reviewers searched for relevant English-language studies in MEDLINE, EMBASE, Scopus, and ERIC (last search June 5, 2015), and in reference lists and author files. They included all original studies that evaluated ITERs for qualitative assessment of medical students and residents. Working in duplicate, they selected articles for inclusion, evaluated quality, and abstracted information on validity evidence using Kane's framework (inferences of scoring, generalization, extrapolation, and implications).

RESULTS

Of 777 potential articles, 22 met inclusion criteria. The scoring inference is supported by studies showing that rich narratives are possible, that changing the prompt can stimulate more robust narratives, and that comments vary by context. Generalization is supported by studies showing that narratives reach thematic saturation and that analysts make consistent judgments. Extrapolation is supported by favorable relationships between ITER narratives and numeric scores from ITERs and non-ITER performance measures, and by studies confirming that narratives reflect constructs deemed important in clinical work. Evidence supporting implications is scant.

CONCLUSIONS

The use of ITER narratives for trainee assessment is generally supported, except that evidence is lacking for implications and decisions. Future research should seek to confirm implicit assumptions and evaluate the impact of decisions.

The failure to fail underperforming trainees in health professions education: A BEME systematic review: BEME Guide No. 42

BACKGROUND

Many clinical educators feel unprepared and/or unwilling to report unsatisfactory trainee performance. This systematic review consolidates knowledge from medical, nursing, and dental literature on the experiences and perceptions of evaluators or assessors with this failure to fail phenomenon.

METHODS

We searched the English language literature in CINAHL, EMBASE, and MEDLINE from January 2005 to January 2015. Qualitative and quantitative studies were included. Following our review protocol, registered with BEME, reviewers worked in pairs to identify relevant articles. The investigators participated in thematic analysis of the qualitative data reported in these studies. Through several cycles of analysis, discussion and reflection, the team identified the barriers and enablers to failing a trainee.

RESULTS

From 5330 articles, we included 28 publications in the review. The barriers identified were (1) assessor's professional considerations, (2) assessor's personal considerations, (3) trainee related considerations, (4) unsatisfactory evaluator development and evaluation tools, (5) institutional culture and (6) consideration of available remediation for the trainee. The enablers identified were: (1) duty to patients, to society, and to the profession, (2) institutional support such as backing a failing evaluation, support from colleagues, evaluator development, and strong assessment systems, and (3) opportunities for students after failing.

DISCUSSION/CONCLUSIONS

The inhibiting and enabling factors to failing an underperforming trainee were common across the professions included in this study, across the 10 years of data, and across the educational continuum. We suggest that these results can inform efforts aimed at addressing the failure to fail problem.

Direct Observation of Clinical Skills Feedback Scale: Development and Validity Evidence

Construct: This article describes the development and validity evidence behind a new rating scale to assess feedback quality in the clinical workplace.

BACKGROUND

Competency-based medical education has mandated a shift to learner-centeredness, authentic observation, and frequent formative assessments with a focus on the delivery of effective feedback. Because feedback has been shown to be of variable quality and effectiveness, an assessment of feedback quality in the workplace is important to ensure we are providing trainees with optimal learning opportunities. The purposes of this project were to develop a rating scale for the quality of verbal feedback in the workplace (the Direct Observation of Clinical Skills Feedback Scale [DOCS-FBS]) and to gather validity evidence for its use.

APPROACH

Two panels of experts (local and national) took part in a nominal group technique to identify features of high-quality feedback. Through multiple iterations and review, 9 features were developed into the DOCS-FBS. Four rater types (residents n = 21, medical students n = 8, faculty n = 12, and educators n = 12) used the DOCS-FBS to rate videotaped feedback encounters of variable quality. The psychometric properties of the scale were determined using a generalizability analysis. Participants also completed a survey to gather data on a 5-point Likert scale to inform the ease of use, clarity, knowledge acquisition, and acceptability of the scale.

RESULTS

Mean video ratings ranged from 1.38 to 2.96 out of 3 and followed the intended pattern suggesting that the tool allowed raters to distinguish between examples of higher and lower quality feedback. There were no significant differences between rater type (range = 2.36-2.49), suggesting that all groups of raters used the tool in the same way. The generalizability coefficients for the scale ranged from 0.97 to 0.99. Item-total correlations were all above 0.80, suggesting some redundancy in items. Participants found the scale easy to use (M = 4.31/5) and clear (M = 4.23/5), and most would recommend its use (M = 4.15/5). Use of DOCS-FBS was acceptable to both trainees (M = 4.34/5) and supervisors (M = 4.22/5).

CONCLUSIONS

The DOCS-FBS can reliably differentiate between feedback encounters of higher and lower quality. The scale has been shown to have excellent internal consistency. We foresee the DOCS-FBS being used as a means to provide objective evidence that faculty development efforts aimed at improving feedback skills can yield results through formal assessment of feedback quality.

Daily Encounter Cards-Evaluating the Quality of Documented Assessments

BACKGROUND

Concerns over the quality of work-based assessment (WBA) completion has resulted in faculty development and rater training initiatives. Daily encounter cards (DECs) are a common form of WBA used in ambulatory care and shift work settings. A tool is needed to evaluate initiatives aimed at improving the quality of completion of this widely used form of WBA.

OBJECTIVE

The completed clinical evaluation report rating (CCERR) was designed to provide a measure of the quality of documented assessments on in-training evaluation reports. The purpose of this study was to provide validity evidence to support using the CCERR to assess the quality of DEC completion.

METHODS

Six experts in resident assessment grouped 60 DECs into 3 quality categories (high, average, and poor) based on how informative each DEC was for reporting judgments of the resident's performance. Eight supervisors (blinded to the expert groupings) scored the 10 most representative DECs in each group using the CCERR. Mean scores were compared to determine if the CCERR could discriminate based on DEC quality.

RESULTS

Statistically significant differences in CCERR scores were observed between all quality groups (P < .001). A generalizability analysis demonstrated the majority of score variation was due to differences in DECs. The reliability with a single rater was 0.95.

CONCLUSIONS

The CCERR is a reliable and valid tool to evaluate DEC quality. It can serve as an outcome measure for studying interventions targeted at improving the quality of assessments documented on DECs.

A New Instrument for Assessing Resident Competence in Surgical Clinic: The Ottawa Clinic Assessment Tool

BACKGROUND

The shift toward competency-based medical education has created a demand for feasible workplace-based assessment tools. Perhaps, more important than competence to assess an individual patient is the ability to successfully manage a surgical clinic. Trainee performance in clinic is a critical component of learning to manage a surgical practice, yet no assessment tool currently exists to assess daily performance in outpatient clinics for surgery residents. The development of a competency-based assessment tool, the Ottawa Clinic Assessment Tool (OCAT), is described here to address this gap.

STUDY DESIGN

A consensus group of experts was gathered to generate dimensions of performance reflective of a competent "generalist" surgeon in clinic. A 6-month pilot study of the OCAT was conducted in orthopedics, general surgery, and obstetrics and gynecology with quantitative and qualitative evidence of validity collected. In all, 2 subsequent feedback sessions and a survey for staff and residents evaluated the OCAT for clarity and utility.

RESULTS

The OCAT is a 9-item tool, with a global assessment item and 2 short-answer questions. Among the 2 divisions, 44 staff surgeons completed 132 OCAT assessments of 79 residents. Psychometric data was collected as evidence of validity. Analysis of feedback indicated that the entrustability rating scale was useful for surgeons and residents and that the items could be correlated with individual competencies.

CONCLUSIONS

Multiple sources of validity evidence collected in this study demonstrate that the OCAT can measure resident clinic competency in a valid and feasible manner.

Going places: Does the two-minute walk test predict the six-minute walk test in lower extremity amputees?

OBJECTIVE

Assessing a patient's ability to walk the distance required for community ambulation (at least 300 m) is important in amputee rehabilitation. During the 2-min walk test, most amputees cannot walk 300 m. Thus, the 6-min walk test may be preferred, but it has not been fully validated in this population. This study examined the convergent and discriminative validity of the 6-min walk test and assessed whether the 2-min test could predict the results of the 6-min test.

METHODS

A total of 86 patients with unilateral or bilateral amputations at the Syme, transtibial, knee disarticulation or transfemoral level completed the 6-min walk test, 2-min walk test, Timed Up and Go test, Locomotor Capabilities Index version 5, Houghton Scale of Prosthetic Use, and Activity-Specific Balance Confidence scale.

RESULTS

The 6-min walk test correlated with the other tests (R = 0.57-0.95), demonstrating convergent validity. It demonstrated discriminative validity with respect to age, aetiology of amputation, and K-level (p < 0.0001). The 2-min walk test was highly predictive of the 6-min walk test distance (R2 = 0.91).

CONCLUSION

The 6-min walk test is a valid measure of amputee ambulation. However, the results suggest that it may not be necessary, since the 2-min walk test strongly predicts the 6-min walk test. Clinicians could therefore save time by using the shorter test.

Gait adaptations of transfemoral prosthesis users across multiple walking tasks

BACKGROUND

For people with lower extremity amputations, the decreased confidence and suboptimal gait associated with dynamic instability can negatively affect mobility and quality of life. Quantifying dynamic instability could enhance clinical decision making related to lower extremity prosthetics and inform future prosthetic research.

OBJECTIVE

To quantitatively examine gait adaptations in transfemoral amputees across various walking conditions.

STUDY DESIGN

Cross-sectional study.

METHODS

Plantar-pressure data were collected from 11 individuals with unilateral transfemoral amputations using an in-shoe plantar-pressure measurement system while navigating rigid and soft ground, ramp, and stair conditions. Six parameters were examined: anterior-posterior and medial-lateral center-of-pressure direction changes, sensor cell loading frequency (cell triggering), maximum lateral force position, double support time, and stride time. Paired t-tests and analyses of variance were used to examine differences between limbs and walking conditions, respectively.

RESULTS

Values for medial-lateral center-of-pressure direction change, sensor cell loading frequency, and double support time were significantly greater on the intact limb than the prosthetic limb. Significant differences between conditions occurred only for anterior-posterior center-of-pressure direction change and double support time on the prosthetic limb.

CONCLUSION

Higher values on the intact limb suggest that it plays a key role in maintaining stability and optimizing body progression during different tasks. Differences between participants, limbs, and walking condition indicate parameter sensitivity to adaptive gait strategies.

CLINICAL RELEVANCE

This plantar-pressure-based approach is a viable option for point-of-care evaluation of locomotor performance, across common various mobility tasks and activities of daily living. The information obtained could be valuable for prosthetic prescription and optimization of prosthetic fit and alignment, potentially improving mobility for prosthetic users with dynamic stability deficits.

Entrustability Scales: Outlining Their Usefulness for Competency-Based Clinical Assessment

Meaningful residency education occurs at the bedside, along with opportunities for situated in-training assessment. A necessary component of workplace-based assessment (WBA) is the clinical supervisor, whose subjective judgments of residents' performance can yield rich and nuanced ratings but may also occasionally reflect bias. How to improve the validity of WBA instruments while simultaneously capturing meaningful subjective judgment is currently not clear. This Perspective outlines how "entrustability scales" may help bridge the gap between the assessment judgments of clinical supervisors and WBA instruments. Entrustment-based assessment evaluates trainees against what they will actually do when independent; thus, "entrustability scales"-defined as behaviorally anchored ordinal scales based on progression to competence-reflect a judgment that has clinical meaning for assessors. Rather than asking raters to assess trainees against abstract scales, entrustability scales provide raters with an assessment measure structured around the way evaluators already make day-to-day clinical entrustment decisions, which results in increased reliability. Entrustability scales help raters make assessments based on narrative descriptors that reflect real-world judgments, drawing attention to a trainee's readiness for independent practice rather than his/her deficiencies. These scales fit into milestone measurement both by allowing an individual resident to strive for independence in entrustable professional activities across the entire training period and by allowing residency directors to identify residents experiencing difficulty. Some WBA tools that have begun to use variations of entrustability scales show potential for allowing raters to produce valid judgments. This type of anchor scale should be brought into wider circulation.

Twelve tips for completing quality in-training evaluation reports

Assessing learners in the clinical setting is vital to determining their level of professional competence. Clinical performance assessments can be documented using In-training evaluation reports (ITERs). Previous research has suggested a need for faculty development in order to improve the quality of these reports. Previous work identified key features of high-quality completed ITERs which primarily involve the narrative comments. This aligns well with the recent discourse in the assessment literature focusing on the value of qualitative assessments. Evidence exists to demonstrate that faculty can be trained to complete higher quality ITERs. We present 12 key strategies to assist clinical supervisors in improving the quality of their completed ITERs. Higher quality completed ITERs will improve the documentation of the trainee's progress and be more defensible when questioned in an appeal or legal process.

Dynamic MRA with four-dimensional time-resolved angiography using keyhole at 3 tesla in head and neck vascular lesions

Conventional MRA provides inadequate visualization of the dynamic features of blood flow in vascular lesions of the head and neck. Four-dimensional time-resolved angiography using keyhole (4D-TRAK) is a new technique of performing contrast-enhanced MRA. By combining parallel imaging with sensitivity encoding (SENSE) with the keyhole imaging technique and a high field strength (3 T) magnet, we have been able to obtain detailed hemodynamic information similar to that obtained via catheter angiography with digital subtraction (DSA), but without the risks associated with ionizing radiation exposure, iodizing contrast agents, or catheterization itself.

Dramatic age-related changes in nuclear and genome copy number in the nematode Caenorhabditis elegans

The nematode Caenorhabditis elegans has become one of the most widely used model systems for the study of aging, yet very little is known about how C. elegans age. The development of the worm, from egg to young adult has been completely mapped at the cellular level, but such detailed studies have not been extended throughout the adult lifespan. Numerous single gene mutations, drug treatments and environmental manipulations have been found to extend worm lifespan. To interpret the mechanism of action of such aging interventions, studies to characterize normal worm aging, similar to those used to study worm development are necessary. We have used 4',6'-diamidino-2-phenylindole hydrochloride staining and quantitative polymerase chain reaction to investigate the integrity of nuclei and quantify the nuclear genome copy number of C. elegans with age. We report both systematic loss of nuclei or nuclear DNA, as well as dramatic age-related changes in nuclear genome copy number. These changes are delayed or attenuated in long-lived daf-2 mutants. We propose that these changes are important pathobiological characteristics of aging nematodes.