The reliability, validity, and feasibility of multisource feedback physician assessment: a systematic review

A 360-degree evaluation of the professionalism and communication skills of technologists working in the radiology department of a public sector tertiary care hospital in Karachi, Pakistan

BACKGROUND

Radiologic technologists (RTs) are at the forefront of radiology departments and require optimal skills to demonstrate professionalism and effective provider-patient communication. Hence, the objective of this study was to determine the reliability of 360-degree evaluation as a feedback tool to improve competencies in this crucial workforce.

METHODS

A planned 360-degree evaluation was conducted using a 10-item tool to evaluate communication skills and professionalism among radiological technologists working at Dow University Hospital, Karachi. In total, 311 evaluations of fifteen radiological technologists were conducted while performing various diagnostic radiology procedures at the radiology department of Dow University Hospital. The data were analysed using SPSS version 26 to compute the evaluation scores and the reliability of 360-degree evaluation.

RESULTS

A total of 311 RT-patient interactions were recorded over 48 days, with scores from all three raters. The evaluation tool was found to have good internal consistency for patients, faculty, and RT, with Cronbach's alpha values of 0.89, 087, and 0.74, respectively. The study found an intraclass correlation of 0.66 (95% CI; 0.58-0.72), showing moderate reliability of the 360-degree evaluation across different raters.

CONCLUSION

We conclude that 360-degree evaluation is a valid and reliable tool for determining the professionalism and communication skills of radiologic technologists and should be incorporated into training programs for formative and summative assessments. However, large-scale multicenter studies are crucial for generalizing these findings and incorporating 360 evaluations in radiological technologist training programs for formative and summative assessments.

Moral distress from professionalism dilemmas and its association with self-rated professionalism behaviors among Chinese residents

OBJECTIVES

Residents inevitably witness or participate in a diverse range of professionalism dilemmas. However, few studies have focused on residents' moral distress from professionalism dilemmas and its relationship with residents' professionalism. This study aimed to understand the moral distress that Chinese residents may face after exposure to professionalism dilemmas and to examine the associations between moral distress and residents' perceived fulfillment of professionalism behaviors.

METHODS

We conducted a cross-sectional survey of residents from four standardized residency training bases in Liaoning Province, China, using stratified cluster sampling. A checklist of professionalism dilemmas, the Moral Distress Scale, and the Behavior-based Medical Professionalism Inventory were used to assess residents' moral distress from professionalism dilemmas and their perceived fulfillment of professionalism behaviors. Descriptive statistics, non-parametric tests, multiple linear regressions, and binary logistic regressions were used to analyze the data.

RESULTS

A total of 647 (81.1%) residents effectively completed the survey. The proportion of residents suffering from moral distress ranged from 58.4 to 90.6% for different professionalism dilemmas. As the number of professionalism dilemmas associated with moral distress increased, residents reported lower fulfillment of professionalism behaviors (β < 0, p < 0.05). Compared with residents with no distress, residents suffering from distress reported lower fulfillment of professionalism behaviors (OR < 1, p < 0.05). Among residents suffering from distress, as the distress intensity increased, residents reported higher fulfillment of professionalism behaviors (OR > 1, p < 0.05).

CONCLUSIONS

Residents suffered a wide range of moral distress from professionalism dilemmas, and residents with moral distress reported lower fulfillment of professional behaviors. A responsive reporting system for residents and reflection on role modeling may help residents cope with the negative effects of moral distress and professionalism dilemmas.

Patient satisfaction with women vs men surgical interns and senior residents

BACKGROUND

Patient satisfaction is critical for referrals and reimbursement of surgical faculty but remains poorly characterized for residents. We investigated whether patient evaluations of surgical trainees vary by resident gender.

METHODS

Surgical inpatients evaluated surgical resident care postoperatively after positively identifying trainees. Evaluations (Consumer Assessment of Healthcare Providers and Systems Surgical Care Surveys (S-CAHPS)) were scored by the "top-box" method, stratified by training level, and compared between women and men residents.

RESULTS

Ninety-one percent of patients participated (n ​= ​324/357). Patients recognized women interns less than men (75.0 ​% vs 87.2 ​%, p ​= ​0.01). S-CAHPS scores for women vs men interns were equivalent except for spending sufficient time with patients (75.6 ​% vs 88.0 ​%, p ​= ​0.02). For senior residents, there was no difference in patient recognition of women vs men (83.9 ​% vs 85.2 ​%, p ​= ​0.91) or in any S-CAHPS scores (p ​> ​0.05).

CONCLUSIONS

Gendered differences in patient evaluations of surgical trainees exist for interns but resolve by senior years. Future work should explore how patient evaluations can support trainee development while ensuring patients recognize the role of surgical residents regardless of gender.

Nurse perspectives on the assessment of emergency medicine residents: A qualitative study

Background

Multisource and 360-degree feedback are emerging methods in the medical education assessment literature. Nursing feedback in particular has been identified as a point of emphasis regarding what medical education leadership would most value for incorporating into their evaluations. There is no currently accepted tool for multisource feedback in the emergency department (ED). To answer this call, we conducted a qualitative study to elucidate nurse perspectives on evaluation of emergency medicine residents.

Methods

We conducted individual semistructured interviews of ED nurses with experience working alongside residents via volunteer recruitment. Transcripts were coded independently by each author and then reviewed collaboratively to resolve differences. Codes were created by inductive thematic analysis and subsequently underwent constant comparison and classical content analysis. Recruitment continued until analysis of transcripts showed thematic saturation.

Results

We organized the interview transcript into chunks of information, arranging 407 quotes from the 10 interview transcripts into 17 codes and six themes regarding observable resident behaviors. Representative quotes and exemplar cases added detailed description to the meaning of these codes. The identified themes for observable resident behaviors included "general communication style," "medical knowledge," "efficiency," "patient communication," "nurse communication," and "professionalism." An additional 13 codes addressed the themes of barriers and catalysts for feedback.

Conclusions

The identified themes on observable resident behaviors aligned well with prior literature and with the Accreditation Council for Graduate Medical Education (ACGME) Milestones. The interviews also highlighted nurse perspectives on barriers and catalysts for feedback that would be useful to incorporate into the design of an assessment tool.

Extravert Surgical Resident Applicants Get Higher 360-Degree Evaluations From Coworkers

INTRODUCTION

360-degree evaluations are used as an assessment in order to identify strengths and weaknesses of, or as a continuous evaluation for, residents. The aim of this study was to investigate the relationship between personality and ratings on 360-degree evaluations among surgical residency applicants. A secondary aim was to describe the personality profile of applicants for a surgical residency position.

METHODS

Doctors interviewed for a residency or locum position in general, urology, or pediatric surgery were included. Participants rated their personality on the Neutralized Big Five Inventory. A 360-degree assessment was conducted. Scores from two laparoscopic simulators were used as a measure of technical ability. Univariate analyses were used to assess the results. Student's t-test was used to compare personality and Pearson correlations between 360-degree assessment and personality.

RESULTS

Fifty doctors participated: data were complete for 38. Personality profiles showed higher emotional stability, agreeableness, conscientiousness and openness ratings than the norm. Correlations revealed a significant relationship between extraversion and higher scores on the 360-degree assessments. Significant univariate correlations were found between extraversion and the 360-degree assessments, and two of the correlations held up after adjustment for multiple tests. No correlations with performance when using laparoscopic simulators were found.

CONCLUSIONS

Applicants for surgical residency rated significantly higher than the norm in four personality domains. Extraversion correlated with overall higher scores in 360-degree assessments by coworkers. Higher scores were not related to objective measures of technical skill, highlighting the importance of using objective measures for assessment.

Exploring Interprofessional Development of Entrustable Professional Activities For Pediatric Intensive Care Fellows: A Proof-of-Concept Study

Phenomenon: Entrustable professional activities (EPAs) delineate major professional activities that an individual in a given specialty must be "entrusted" to perform, ultimately without supervision, to provide quality patient care. Until now, most EPA frameworks have been developed by professionals within the same specialty. As safe, effective, and sustainable health care ultimately depends on interprofessional collaboration, we hypothesized that members of interprofessional teams might have clear and possibly additional insight into which activities are essential to the professional work of a medical specialist. Approach: We recently employed a national modified Delphi study to develop and validate a set of EPAs for Dutch pediatric intensive care fellows. In this proof-of-concept study, we explored what pediatric intensive care physicians' non-physician team members (physician assistants, nurse practitioners, and nurses) constitute as essential professional activities for PICU physicians and how they regarded the newly developed set of nine EPAs. We compared their judgments with the PICU physicians' opinions. Findings: This study shows that non-physician team members share a mental model with physicians about which EPAs are indispensable for pediatric intensive care physicians. Despite this agreement however, descriptions of EPAs are not always clear for non-physician team members who have to work with them on a daily basis. Insights: Ambiguity as to what an EPA entails when qualifying a trainee can have implications for patient safety and trainees themselves. Input from non-physician team members may add to the clarity of EPA descriptions. This finding supports the involvement of non-physician team members in the developmental process of EPAs for (sub)specialty training programs.

Family Medicine Resident Experience Toward Workplace-Based Assessment Form in Improving Clinical Teaching: An Exploratory Qualitative Study

Background

Workplace-Based Assessment (WPBA) has been widely utilized for assessing performance in training sites for both formative and summative purposes. Currently, with the recently updated duration of the family medicine (FM) training program in Saudi Arabia from four years to three years, the possible impact of such a change on assessment would need to be investigated. This objective was to explore the experiences of FM residents regarding the usage of WPBA as an assessment tool for improving clinical teaching at King Abdulaziz Hospital (KAH), Al Ahsa, Saudi Arabia.

Methods

The study involves an exploratory qualitative phenomenological approach targeting family medicine resident in KAH was used. Purposive sampling techniques were used. In this descriptive study, data was collected through the utilization of 1:1 semi-structured interviews guided by directive prompts. All recorded interviews were transcribed verbatim. An inductive analytical approach was applied for thematic analysis of transcripts.

Results

Fifteen participants were individually interviewed until data saturation was reached. The themes that emerged were organized into the categories of underlying principles of WPBA, the impact of the learning environment, associated opportunities and challenges, and making WPBA more effective. Participants expressed that the orientation provided by the program was insufficient, although the core principles were clear to them. They valued the senior peers' support and encouragement for the creation of a positive learning environment. However, time limit, workload, and a lack of optimum ideal implementation reduced the educational value and effectiveness of WPBA among senior residents.

Conclusion

The study examined residents' experiences with WPBA and concluded that low levels of satisfaction were attributed to implementation-related problems. Improvements should be made primarily in two areas: better use of available resources and more systematic prior planning. Revision and assignment of the selection process were suggested, in addition to the implementation of the new curriculum. The research will assist stakeholders in selecting and carrying out evaluation techniques that will enhance residents' abilities.

Interprofessional communication skills training to improve medical students' and nursing trainees' error communication - quasi-experimental pilot study

BACKGROUND

Interprofessional communication is of extraordinary importance for patient safety. To improve interprofessional communication, joint training of the different healthcare professions is required in order to achieve the goal of effective teamwork and interprofessional care. The aim of this pilot study was to develop and evaluate a joint training concept for nursing trainees and medical students in Germany to improve medication error communication.

METHODS

We used a mixed-methods, quasi-experimental study with a pre-post design and two study arms. This study compares medical students (3rd year) and nursing trainees (2nd year) who received an interprofessional communication skills training with simulation persons (intervention group, IG) with a control group (CG). Both cohorts completed identical pre- and post-training surveys using the German Interprofessional Attitudes Scale (G-IPAS) and a self-developed interprofessional error communication scale. Descriptive statistics, Mann-Whitney-U-test and Wilcoxon-test were performed to explore changes in interprofessional error communication.

RESULTS

A total of 154 were medical students, and 67 were nursing trainees (IG: 66 medical students, 28 nursing trainees / CG: 88 medical students, 39 nursing trainees). After training, there were significant improvements observed in the "interprofessional error communication" scale (p < .001) and the "teamwork, roles, and responsibilities" subscale (p = .012). Median scores of the subscale "patient-centeredness" were similar in both groups and remained unchanged after training (median = 4.0 in IG and CG).

CONCLUSIONS

Future studies are needed to find out whether the training sustainably improves interprofessional teamwork regarding error communication in acute care.

Development of a Tool to Assess Surgical Resident Competence On-Call: The Western University Call Assessment Tool (WUCAT)

BACKGROUND

A central tenet of competency-based medical education is the formative assessment of trainees. There are currently no assessments designed to examine resident competence on-call, despite the on-call period being a significant component of residency, characterized by less direct supervision compared to daytime. The purpose of this study was to design a formative on-call assessment tool and collect valid evidence on its application.

METHODS

Nominal group technique was used to identify critical elements of surgical resident competence on-call to inform tool development. The tool was piloted over six months in the Division of Plastic & Reconstructive Surgery at our institution. Quantitative and qualitative evidence was collected to examine tool validity.

RESULTS

A ten-item tool was developed based on the consensus group results. Sixty-three assessments were completed by seven staff members on ten residents during the pilot. The tool had a reliability coefficient of 0.67 based on a generalizability study and internal item consistency was 0.92. Scores were significantly associated with years of training. We found the tool improved the quantity and structure of feedback given and that the tool was considered feasible and acceptable by both residents and staff members.

CONCLUSIONS

The Western University Call Assessment Tool (WUCAT) has multiple sources of evidence supporting its use in assessing resident competence on-call.

Improving optometry student interpersonal skills through online patient, clinician and student evaluation and feedback

CLINICAL RELEVANCE

Interpersonal skills are crucial for successful clinician-patient interactions. To prepare future optometrists for clinical practice, pedagogical evaluation is important to support the implementation of new strategies for teaching and evaluating interpersonal skills.

BACKGROUND

Optometry students largely develop their interpersonal skills through in-person patient interactions. Telehealth is increasing, yet strategies to develop the interpersonal skills of students for teleconsulting have not been explored. This study aimed to assess the feasibility, effectiveness and perceived usefulness of an online, multisource (patients, clinicians and students) evaluation and feedback program for developing interpersonal skills.

METHODS

Via an online teleconferencing platform, optometry students (n = 40) interacted with a volunteer patient, observed by a teaching clinician. Patients and clinicians evaluated the interpersonal skills of the student in two ways: (1) qualitative written feedback, and (2) quantitative rating (Doctors' Interpersonal Skills Questionnaire). All students received written patient and clinician feedback after the session, but not their quantitative ratings. A subset of students (n = 19) completed two sessions, self-ratings, and were provided with their written feedback and an audiovisual recording from their first interaction before completing the second session. All participants were invited to complete an anonymous survey at program completion.

RESULTS

Patient and clinician overall interpersonal skills ratings were positively correlated (Spearman's r = 0.35, p = 0.03) and showed moderate agreement (Lin's concordance coefficient = 0.34). Student self-ratings did not match patient ratings (r = 0.01, p = 0.98), whereas there was moderate agreement between clinician and student ratings (Lin's concordance coefficient = 0.30). Ratings improved at the second visit (p = 0.01). Patient ratings were higher than clinicians (p = 0.01) and students (p = 0.03). All participants agreed that the program was feasible, useful and effective at fostering good interpersonal skills.

CONCLUSION

Multisource feedback about interpersonal skills contributes to improvement in student performance. Patients and clinicians can evaluate and provide useful feedback to optometry students about their interpersonal skills using online methods.

Comparison Between Surgical Trainee Self-Assessment With Supervisor Scores in Technical and Nontechnical Skills to Identify Underperforming Trainees

OBJECTIVE

This study aims to compare trainee self-assessment with supervisor assessment to identify differences in correlations of the demographic data and evaluate whether the instrument can be utilized to identify underperforming trainees.

DESIGN

A novel instrument was designed based off the Royal Australasian College of Surgeons original 9 competencies utilizing the JDocs framework and covers 48-items across all surgical competencies. A multiple regression model using age, gender, postgraduate year, IMG status, and level of training as the variables was performed with backwards elimination, and pairwise comparisons made to identify the degree and direction of influence each variable contributed to trainee and supervisor ratings.

SETTING

Surgical trainees employed in tertiary centers within the Australian Capital Territory and South-East New South Wales health network in Australia.

PARTICIPANTS

A total of 36 of 59 (61%) trainees responded. Two grossly incomplete responses were excluded from further analysis resulting in 34 completed self-assessments for analysis. There was a total of 68 supervisor assessments provided by 25 different nominated supervisors. Of these assessments, 67 were fully completed and one was partially complete.

RESULTS

We identified that for both self-ratings and supervisor ratings, the most significant correlation is with the postgraduate year of the trainee, with correlation identified in 7 out of 9 competencies, although in different competency domains. International Medical Graduate status is associated in 2 of 9 self-ratings, and in 3 of 9 supervisor ratings. Underperforming trainees were able to be identified through supervisor assessment but not self-assessment.

CONCLUSION

The supervisor assessment form was able to identify underperforming trainees. Our findings resonate with existing literature in other specialty fields, and surgical units that employ assessment forms should feel more confident in the interpretation of the data and provision of feedback to trainees.

180° view on general practitioners' leadership skills: practice-level comparisons of leader and staff assessments using data from the cluster-randomised controlled IMPROVEjob study

OBJECTIVES

Strong primary care leaders are needed to assure high quality services for patient populations. This study analysed general practitioners' (GP) leadership skills comparing practice-level self and staff assessments based on the full range of leadership model and the leader-member exchange (LMX).

SETTING

The questionnaire survey was conducted among German general practice leaders and their staff participating in the IMPROVEjob trial.

PARTICIPANTS

The study population comprised 60 German general practices with 366 participants: 84 GP practice leaders and 282 employees (28 physicians and 254 practice assistants).

PRIMARY AND SECONDARY OUTCOME MEASURES

Leadership skills of the practice leaders were measured using the Integrative Leadership Questionnaire (German Fragebogen für integrative Führung) and the LMX-7 questionnaire. Leaders rated themselves and practice staff rated their leaders. The data was analysed by paired mean comparisons on the practice level.

RESULTS

For most leadership dimensions, practice leaders rated themselves higher than their employees rated them. Differences were found for transformational leadership (p<0.001, d=0.41), especially for the dimensions 'innovation' (p<0.001, d=0.69) and 'individuality focus' (p<0.001, d=0.50). For transactional leadership, the dimension 'goal setting' differed significantly (p<0.01, d=0.30) but not the other dimensions. Scores for negative leadership were low and showed no differences between leaders and employees. Interestingly, employed physicians' rated their practice leaders higher on the two transformational ('performance development', 'providing a vision') and all transactional dimensions. The LMX-7 scale showed high quality relationships between leaders and employees.

CONCLUSIONS

This 180° analysis of GPs' leadership skills with self and employee ratings indicated good relationships. There is a potential to improve leadership regarding goal-setting, innovation and focusing on individual team members. These results allow for the development of targeted interventions.

TRIAL REGISTRATION NUMBER

German Clinical Trials Register, DRKS00012677. Registered 16 October 2019.

Catalytic effect of multisource feedback for trauma team captains: a mixed-methods prospective study

OBJECTIVES

To evaluate the impact and feasibility of multisource feedback compared with traditional feedback for trauma team captains (TTCs).

DESIGN

A mixed-methods, non-randomised prospective study.

SETTING

A level one trauma centre in Ontario, Canada.

PARTICIPANTS

Postgraduate medical residents in emergency medicine and general surgery participating as TTCs. Selection was based on a convenience sampling method.

INTERVENTION

Postgraduate medical residents participating as TTCs received either multisource feedback or standard feedback following trauma cases.

MAIN OUTCOME MEASURES

TTCs completed questionnaires designed to measure the self-reported intention to change practice (catalytic effect), immediately following a trauma case and 3 weeks later. Secondary outcomes included measures of perceived benefit, acceptability, and feasibility from TTCs and other trauma team members.

RESULTS

Data were collected following 24 trauma team activations: TTCs from 12 activations received multisource feedback and 12 received standard feedback. The self-reported intention for practice change was not significantly different between groups initially (4.0 vs 4.0, p=0.57) and at 3 weeks (4.0 vs 3.0, p=0.25). Multisource feedback was perceived to be helpful and superior to the existing feedback process. Feasibility was identified as a challenge.

CONCLUSIONS

The self-reported intention for practice change was no different for TTCs who received multisource feedback and those who received standard feedback. Multisource feedback was favourably received by trauma team members, and TTCs perceived multisource feedback as useful for their development.

Relationship between selection criteria and trainee performance in medical specialty training: A retrospective longitudinal study

BACKGROUND

Investigation of tools used for candidate selection as predictors of future performance in medical education has been primarily undertaken within the undergraduate setting, but little is known about the selection to medical specialist programs. This retrospective longitudinal study aims to explore correlations between selection tools and the performance of trainees enrolled in the Australasian College of Dermatologists (ACD), the accredited specialist medical college for training in dermatology in Australia.

METHODS

Data were collected from consecutive cohorts of ACD trainees commencing from 2007 to 2015. Predictive variables were trainee demographics and selection tools (prior academic qualification; research experience; clinical experience; curriculum vitae [CV] rating; interview scores). The outcome variables were exam performance. First Year ([Pharmacology and Clinical Sciences] and Fellowship Exam [Written, Clinical, and Overall Score]). Statistical analyses included: descriptive statistics summaries; bivariate correlation of selection criteria and demographic data with exam performance; and linear regression analysis to identify predictors of exam performance.

RESULTS

Demographic analysis (N = 172) showed that 64% of trainees were female, the average age was 30.5 years (± 3.47) and trainees living in high socioeconomic status (SES) areas were over-represented. Using Pearson correlation analysis, interview scores were significantly positively correlated with First Year Pharmacology results (p = 0.018), Fellowship Written results (p = 0.002), and Fellowship Overall Scores (p = 0.006). First Year Pharmacology Exam performance was most highly correlated with Fellowship Exam performance (p = 0.000). No association was identified between exam performance and gender, SES, prior academic qualification, research experience or CV rating. Linear regression analysis showed that interview score (p = 0.012), entry age (p = 0.026) and First Year Pharmacology score (p = 0.002) were predictors of Fellowship exam performance.

CONCLUSION

These results suggest that the comprehensive selection tools used by ACD are effective, with interviews being a highly valid tool linked to exam performance. These findings have important practical implications for assessing the selection process in specialist dermatology training.

Effective components of teachers' professionalism in viewpoints of various stakeholders

BACKGROUND

The teacher must conduct educational responsibilities in the best way by observing the ethical principles to meet students' educational needs. The aim of the study was to investigate the main factors of professionalism among faculty members from the viewpoints of various stakeholders including learners, peers, and educational officials, as well as the components were assessed from the viewpoints of the stakeholders.

MATERIALS AND METHODS

This was a descriptive analytical study conducted in two phases. In the first phase, a questionnaire was developed and validated to assess the professional behavior of faculty members. In the second phase, the evaluation of the teachers' professional behavior was done from the viewpoint of various stakeholders, including learners, middle and senior education managers and peers. The participants were faculty members from 10 schools enrolled in the study by census (n = 427). To extract the factors that constitute professional behavior in faculty members, exploratory factor analysis was used. Data were summarized using descriptive indices (mean, standard deviation, and percent). Exploratory factor analysis (EFA) was used to extract the factors that constitute professional behavior in faculty members. Data was analyzed by using SPSS software (version 23.0).

RESULTS

In the first phase, the questionnaire with 11 items was developed. The validity and reliability of the tool was confirmed. The mean (SD) of the faculty members' scores was 4.54(±0.34), with a minimum score of 2.33 and a maximum score of 5.00. The highest scores of faculty members were reported in the items of "altruism to colleagues and students" and the lowest scores were related to "feedback seeking and feedback acceptance". Professional behavior was categorized under two domains: accountability to professional duties and adherence to professional values. Fifty percent of the participants followed the adherence to professional values (P = 0.22), but less than 50% of the individuals followed the adherence to professional duties significantly (P = 0.002).

CONCLUSION

The results of the present study showed that adherence to professional duties was low in faculty members. Assessment of individuals' adherence in these two domains showed that faculty members' compliance in the domain of accountability to professional duties was significantly lower.

Competencies of hospital managers - A systematic scoping review

Hospital managers around the world work under constant pressure to adapt their organizations to new challenges and health policy goals. This requires a comprehensive set of competencies. The objective of this scoping review was to identify, map, and systematize the literature on hospital manager competencies. The review involved six steps: (1) defining research questions; (2) identifying relevant literature; (3) selecting publications; (4) data extraction; (5) data analysis and result reporting; and (6) consultations. A total of 57 full-text publications were included (46 empirical studies, six literature reviews, four expert opinions/guidelines, and one dissertation). Interest in this topic has grown in recent years, with most of the identified studies published since 2015. The empirical studies fall into three major groups: 34.8% (16/46) examined hospital managers' competencies in terms of their types or classifications; 30.4% (14/46) focused on their measurement; and 30.4% (14/46) examined both aspects. In majority of studies, both 'hard competencies,' such as specific technical knowledge or skills acquired through practical training, and 'soft competencies,' e.g., adaptability, leadership, communication, teamwork, are echoed for effective hospital management. These point out the importance of both 'external' formal education trainings as well as 'internal' peer-support and/or coaching as complementary competency improvement approaches. This scoping review helps build a knowledge base around the topic and provides implications for future research. The latter can involve: a targeted systematic review addressing the methods for measuring the level of competence of hospital managers or studies focused on identifying the need for new types of competencies.

The Impact of Multisource Feedback on Continuing Medical Education, Clinical Performance and Patient Experience: Innovation in a Child and Adolescent Mental Health Service

This paper reiterates the importance of the role of multisource feedback (MSF) in continuing medical education/continuing professional development (CME/CPD) and its impact on doctors' performance and patient experience globally. It summarises a unique initiative of robust utilisation of internationally recognised multisource feedback tools in an outpatient child and adolescent mental health service (CAMHS) in Qatar. The process involved the effective adoption and administering of the General Medical Council's (GMC) self-assessment questionnaire (SQ), patient questionnaire (PQ), and colleague questionnaire (CQ) followed by the successful incorporation of these tools in CME/CPD. The original version of the PQ questionnaire and the instructions to the patient document were translated into Arabic through the blind back-translation technique. This initiative of introducing gold-standard MSF tools and processes into clinical practice, among other quality-improvement projects, has contributed to the improvement of service standards and doctors' clinical practice. Patient satisfaction was measured through the annual patient experience analysis using the Experience of Service Questionnaire (ESQ) whereas changes in doctors' performance were evaluated by comparing annual appraisal scores before and after implementation of this initiative. We have demonstrated that when MSF is obtained impartially and transparently using recognised and valid tools, it can improve patient experience and enhance doctors' performance.

Assessing electrodiagnostic skills among residents and fellows: Relationships between workplace-based assessments using the Electromyography Direct Observation Tool and other measures of trainee performance

INTRODUCTION/AIMS

Graduate medical education programs must ensure residents and fellows acquire skills needed for independent practice. Workplace-based observational assessments are informative but can be time- and resource-intensive. In this study we sought to gather "relations-to-other-variables" validity evidence for scores generated by the Electromyography Direct Observation Tool (EMG-DOT) to inform its use as a measure of electrodiagnostic skill acquisition.

METHODS

Scores on multiple assessments were compiled by trainees during Clinical Neurophysiology and Electromyography rotations at a large US academic medical center. Relationships between workplace-based EMG-DOT scores (n = 298) and scores on a prerequisite simulated patient exercise, patient experience surveys (n = 199), end-of-rotation evaluations (n = 301), and an American Association of Neuromuscular & Electrodiagnostic Medicine (AANEM) self-assessment examination were assessed using Pearson correlations.

RESULTS

Among 23 trainees, EMG-DOT scores assigned by physician raters correlated positively with end-of-rotation evaluations (r = 0.63, P = .001), but EMG-DOT scores assigned by technician raters did not (r = 0.10, P = .663). When physician and technician ratings were combined, higher EMG-DOT scores correlated with better patient experience survey scores (r = 0.42, P = .047), but not with simulated patient or AANEM self-assessment examination scores.

DISCUSSION

End-of-rotation evaluations can provide valid assessments of trainee performance when completed by individuals with ample opportunities to directly observe trainees. Inclusion of observational assessments by technicians and patients provides a more comprehensive view of trainee performance. Workplace- and classroom-based assessments provide complementary information about trainee performance, reflecting underlying differences in types of skills measured.

Multisource feedback: an overview of its use and application as a formative assessment

Multisource feedback (MSF), often termed 360-degree feedback, is a formative performance assessment in which data about an individual's observable workplace behaviors are collected through questionnaires from those interacting with the individual; data are aggregated for anonymity and confidentiality; the aggregated data, along with self-assessment if available, are provided to the individual; and the recipient meets with a trusted individual to review the data and develop an action plan. It is used along the continuum of medical education. This article provides an overview of MSF's utility, its evidence base and cautions.

Patient involvement in assessment of postgraduate medical learners: A scoping review

CONTEXT

Competency-based assessment of learners may benefit from a more holistic, inclusive, approach for determining readiness for unsupervised practice. However, despite movements towards greater patient partnership in health care generally, inclusion of patients in postgraduate medical learners' assessment is largely absent.

METHODS

We conducted a scoping review to map the nature, extent and range of literature examining the inclusion (or exclusion) of patients within the assessment of postgraduate medical learners. Guided by Arskey and O'Malley's framework and informed by Levac et al. and Thomas et al., we searched two databases (MEDLINE® and Embase®) from inception until February 2021 using subheadings related to assessment, patients and postgraduate learners. Data analysis examined characteristics regarding the nature and factor influencing patient involvement in assessment.

RESULTS

We identified 41 papers spanning four decades. Some literature suggests patients are willing to be engaged in assessment, however choose not to engage when, for example, language barriers may exist. When stratified by specialty or clinical setting, the influence of factors such as gender, race, ethnicity or medical condition seems to remain consistent. Patients may participate in assessment as a stand-alone group or part of a multi-source feedback process. Patients generally provided high ratings but commented on the observed professional behaviours and communication skills in comparison with physicians who focused on medical expertise.

CONCLUSION

Factors that influence patient involvement in assessment are multifactorial including patients' willingness themselves, language and reading-comprehension challenges and available resources for training programmes to facilitate the integration of patient assessments. These barriers however are not insurmountable. While understudied, research examining patient involvement in assessment is increasing; however, our review suggests that the extent which the unique insights will be taken up in postgraduate medical education may be dependent on assessment systems readiness and, in particular, physician readiness to partner with patients in this way.

How does multisource feedback influence residency training? A qualitative case study

INTRODUCTION

Multisource feedback (MSF), also called 360-degree assessment, is one form of assessment used in postgraduate training. However, there is an ongoing discussion on its value, because the factors that influence the impact of MSF and the main impact of MSF are not fully understood. In this study, we investigated both the influencing factors and the impact of MSF on residency training.

METHODS

We conducted a qualitative case study within the boundaries of the residency training for paediatricians and paediatric surgeons at a University Hospital. We collected data from seven focus group interviews with stakeholders of MSF (residents, raters and supervisors). By performing a reflexive thematic analysis, we extracted the influencing factors and the impact of MSF.

RESULTS

We found seven influencing factors: MSF is facilitated by the announcement of a clear goal of MSF, the training of raters on the MSF instrument, a longitudinal approach of observation, timing not too early and not too late during the rotation, narrative comments as part of the ratings, the residents' self-assessment and a supervisor from the same department. We found three themes on the impact of MSF: MSF supports the professional development of residents, enhances interprofessional teamwork and increases the raters' commitment to the training of residents.

CONCLUSION

This study illuminates the influencing factors and impact of MSF on residency training. We offer novel recommendations on the continuity of observation, the timing during rotations and the role of the supervisor. Moreover, by discussing our results through the lens of identity formation theory, this work advances our conceptual understanding of MSF. We propose identity formation theory as a framework for future research on MSF to leverage the potential of MSF in residency training.

Development of a novel behaviourally anchored instrument for the assessment of surgical trainees

Background

The Royal Australasian College of Surgeons (RACS) created its competency framework in 2003 which initially consisted of nine competencies each regarded as equally important for a practising surgeon. The JDocs Framework is aligned to these competencies and provides guidance for junior doctors working towards the Surgical Education and Training program.

Methods

A novel assessment instrument was designed around the JDocs framework using 48 behaviourally anchored questions. The study was completed in 2020 across five public hospitals in the ACT and NSW. Participants were invited to complete the self‐assessment form online.

Results

Thirty‐six of 59 (61%) trainees participated in the study, with 67 of 68 (98.5%) supervisors having completed the assessment form. Trainee self‐rating scores were lower than that of supervisor ratings across all competencies except communication. The self‐rating scores were negatively correlated with the seniority of a trainee's level in all nine competencies. The years of post‐graduate experience was positively correlated with seven of the nine competencies. For gender and International Medical Graduate status, correlation was only identified for health advocacy and medical expertise. There was no correlation identified with a trainee's age.

Conclusion

This pilot study has provided an opportunity to explore a new assessment instrument for surgical trainees that is aligned to the RACS competency framework using behaviourally anchored questions. Looking ahead, a better understanding of this instrument will potentially be helpful in early identification of underperforming trainees in order to facilitate early intervention, or its use as a selection tool for formal training programs.

Multi-source feedback following simulated resuscitation scenarios: a qualitative study

BACKGROUND

The direct observation and assessment of learners' resuscitation skills by an attending physician is challenging due to the unpredictable and time-sensitive nature of these events. Multisource feedback (MSF) may address this challenge and improve the quality of assessments provided to learners. We aimed to describe the similarities and differences in the assessment rationale of attending physicians, registered nurses, and resident peers in the context of a simulation-based resuscitation curriculum.

METHODS

We conducted a qualitative content analysis of narrative MSF of medical residents in their first postgraduate year of training who were participating in a simulation-based resuscitation course at two Canadian institutions. Assessments included an entrustment score and narrative comments from attending physicians, registered nurses, and resident peers in addition to self-assessment. Narrative comments were transcribed and analyzed thematically using a constant comparative method.

RESULTS

All 87 residents (100%) participating in the 2017-2018 course provided consent. A total of 223 assessments were included in our analysis. Four themes emerged from the narrative data: 1) Communication, 2) Leadership, 3) Demeanor, and 4) Medical Expert. Relative to other assessor groups, feedback from nurses focused on patient-centred care and communication while attending physicians focused on the medical expert theme. Peer feedback was the most positive. Self-assessments included comments within each of the four themes.

CONCLUSIONS

In the context of a simulation-based resuscitation curriculum, MSF provided learners with different perspectives in their narrative assessment rationale and may offer a more holistic assessment of resuscitation skills within a competency-based medical education (CBME) program of assessment.

Junior doctors' experiences with interprofessional collaboration: Wandering the landscape

CONTEXT

The transition from medical student to junior doctor is challenging. Junior doctors need to become part of the physician community of practice (CoP), while dealing with new responsibilities, tasks and expectations. At the same time, they need to learn how to navigate the frontiers and intersections with the other communities of practice that form the Landscape of Practice (LoP). This study aims to understand how junior doctors experience interprofessional collaboration (IPC) and what elements shape these experiences considering their transition to clinical practice.

METHODS

In this multicentre qualitative study, 13 junior doctors individually drew two rich pictures of IPC experiences, one positive and one negative. A rich picture is a visual representation, a drawing of a particular situation intended to capture the complex and non-verbal elements of an experience. We used semi-structured interviews to deepen the understanding of junior doctors' depicted IPC experiences. We analysed both visual materials and interview transcripts iteratively, for which we adopted an inductive constructivist thematic analysis.

RESULTS

While transitioning into a doctor, junior doctors become foremost members of the physician CoP and shape their professional identity based on perceived values in their physician community. Interprofessional learning occurs implicitly, without input from the interprofessional team. As a result, junior doctors struggle to bridge the gap between themselves and the interprofessional team, preventing IPC learning from developing into an integrative process. This professional isolation leaves junior doctors wandering the landscape of practice without understanding roles, attitudes and expectations of others.

CONCLUSIONS

Learning IPC needs to become a collective endeavour and an explicit learning goal, based on multisource feedback to take advantage of the expertise already present in the LoP. Furthermore, junior doctors need a safe environment to embrace and reflect on the emotions aroused by interprofessional interactions, under the guidance of experienced facilitators.

How psychiatrists’ attitudes towards multi-source feedback including patient feedback influenced the educational value: a qualitative study

Background:  Multi-source feedback (MSF) is well-established in psychiatric training. However, evidence on the educational impact is not definitive and there is scanty evidence exploring its value for professional development of psychiatry trainees in the United Kingdom (UK). Evidence suggests the MSF tool currently used is not suitable for specialist trainees. This qualitative research project explored psychiatric doctors’ attitudes towards MSF with patient feedback, to determine how this influenced the feedback’s educational usefulness. Methods: A qualitative study using a phenomenological approach based on a constructivist approach. Purposive sampling identified trainee psychiatrists who completed a more extensive MSF, including patient feedback, than they currently use. They discussed their results in supervised sessions to plan how to use the feedback. Semi-structured interviews were conducted separately with trainees and their supervisors following completion of MSF. The data was analysed thematically. The study was completed in 2020. Results: Seven trainees and five supervisors participated. Four themes were identified. Most had positive opinions about the educational usefulness of MSF, including patient feedback, and made changes to their behaviour following the feedback. Interviewees valued patient feedback and identified it as important in psychiatry. Most valued their patient feedback over their colleague feedback. The complexities of patient feedback in psychiatry and how this may influence the educational usefulness of the feedback were discussed in detail. Conclusions: Findings suggest a need to review the current system of MSF in psychiatry in order to maximise educational benefits. In particular, this research points to the benefits of psychiatric trainees engaging with patient feedback.

Unobserved Observers: Nurses' Perspectives About Sharing Feedback on the Performance of Resident Physicians

PURPOSE

Postgraduate training programs are incorporating feedback from registered nurses (RNs) to facilitate holistic assessments of resident performance. RNs are a potentially rich source of feedback because they often observe trainees during clinical encounters when physician supervisors are not present. However, RN perspectives about sharing feedback have not been deeply explored. This study investigated RN perspectives about providing feedback and explored the facilitators and barriers influencing their engagement.

METHOD

Constructivist grounded theory methodology was used in interviewing 11 emergency medicine and 8 internal medicine RNs at 2 campuses of a tertiary care academic medical center in Ontario, Canada, between July 2019 and March 2020. Interviews explored RN experiences working with and observing residents in clinical practice. Data collection and analysis were conducted iteratively. Themes were identified using constant comparative analysis.

RESULTS

RNs felt they could observe authentic day-to-day behaviors of residents often unwitnessed by supervising physicians and offer unique feedback related to patient advocacy, communication, leadership, collaboration, and professionalism. Despite a strong desire to contribute to resident education, RNs were apprehensive about sharing feedback and reported barriers related to hierarchy, power differentials, and a fear of overstepping professional boundaries. Although infrequent, a key stimulus that enabled RNs to feel safe in sharing feedback was an invitation from the supervising physician to provide input.

CONCLUSIONS

Perceived hierarchy in academic medicine is a critical barrier to engaging RNs in feedback for residents. Accessing RN feedback on authentic resident behaviors requires dismantling the negative effects of hierarchy and fostering a collaborative interprofessional working environment. A critical step toward this goal may require supervising physicians to model feedback-seeking behavior by inviting RNs to share feedback. Until a workplace culture is established that validates nurses' input and creates safe opportunities for them to contribute to resident education, the voices of nurses will remain unheard.

Employability as a Guiding Outcome in Veterinary Education: Findings of the VetSet2Go Project

This paper presents a mini-review of employability as a guiding outcome in veterinary education-its conceptualisation, utility, core elements and dimensions, and pedagogical approaches-through a summary of the findings of a major international project with the same aims (the VetSet2Go project). Guided by a conception of the successful veterinary professional as one capable of navigating and sustainably balancing the (sometimes competing) needs and expectations of multiple stakeholders, the project integrated multiple sources of evidence to derive an employability framework representing the dimensions and capabilities most important to veterinary professional success. This framework provides a useful complement to those based in narrower views of competency and professionalism. One notable difference is its added emphasis on broad success outcomes of satisfaction and sustainability as well as task-oriented efficacy, thus inserting "the self" as a major stakeholder and bringing attention to resilience and sustainable well-being. The framework contains 18 key capabilities consistently identified as important to employability in the veterinary context, aligned to five broad, overlapping domains: veterinary capabilities (task-oriented work performance), effective relationships (approaches to others), professional commitment (approaches to work and the broader professional "mission"), psychological resources (approaches to self), plus a central process of reflective self-awareness and identity formation. A summary of evidence supporting these is presented, as well as recommendations for situating, developing, and accessing these as learning outcomes within veterinary curricula. Though developed within the specific context of veterinarian transition-to-practise, this framework would be readily adaptable to other professions, particularly in other health disciplines.

Competence of medical and obstetric registrars in the management of systemic lupus erythematosus in pregnancy

Introduction

Systemic lupus erythematosus has a predilection for women of childbearing age. Globally a shortage of rheumatologists exists resulting in general physicians and obstetricians treating systemic lupus erythematosus in pregnancy.

Methods

We conducted a survey amongst medical and obstetric registrars in South Africa to assess their subjective and objective competence in managing pregnant women with systemic lupus erythematosus.

Results

The pass rate for the objective section was 70.8% with no statistically significant difference in the pass rate between medical and obstetric registrars. Participants felt unprepared to manage pregnant women with systemic lupus erythematosus, with a mean overall score of 3.4 out of 7 for the subjective section, based on four Likert scale type questions.

Conclusion

Trainees are not able to accurately assess their own levels of competence in order to identify their learning needs. Due to the shortage of rheumatologists and lack of obstetric physicians in South Africa, general physicians and obstetricians must be equipped to provide adequate care to pregnant women with systemic lupus erythematosus.

Bachelor of Science in Nursing Students' Perceptions Regarding Educator Feedback

BACKGROUND

Feedback is imperative for academic success, learning, and professional growth. Faculty competence to provide constructive feedback is a cornerstone in the educational process, environment, and outcomes.

METHODS

This cross-sectional study evaluated the quantity, quality, timing, and utilization of feedback among 38 Bachelor of Science in Nursing students. An assessment experience questionnaire was used to collected data.

RESULTS

The majority of students (n = 27 [71%]) expressed that the timing of feedback is critical; 21 (55%) students reported feedback threatened their self-esteem, 19 (50%) students perceived feedback did not help in completing assignments, and 24 (63%) students reported feedback did not help in understanding assignments. Students reported the negative impact of feedback resulted in feeling discouraged regarding reflection (n = 20 [53%]) and self-regulation (n = 21 [55%]).

CONCLUSION

Although feedback is imperative for academic success and professional development, the quantity and quality affects students' utilization of feedback. [J Nurs Educ. 2021;60(10):577-581.].

Assessment of Emergency Medicine Resident Performance in a Pediatric In Situ Simulation Using Multi-Source Feedback

Introduction

Multi-source feedback (MSF) is an evaluation method mandated by the Accreditation Council for Graduate Medical Education (ACGME). The Queen’s Simulation Assessment Tool (QSAT) has been validated as being able to distinguish between resident performances in a simulation setting. The QSAT has also been demonstrated to have excellent MSF agreement when used in an adult simulation performed in a simulation lab. Using the QSAT, this study sought to determine the degree of agreement of MSF in a single pediatric (Peds) simulation case conducted in situ in a Peds emergency department (ED).

Methods

This Institutional Review Board-approved study was conducted in a four-year emergency medicine residency. A Peds resuscitation case was developed with specific behavioral anchors on the QSAT, which uses a 1-5 scale in each of five categories: Primary Assessment, Diagnostic Actions, Therapeutic Actions, Communication, and Overall Assessment. Data was gathered from six participants for each simulation. The lead resident self-evaluated and received MSF from a junior peer resident, a fixed Peds ED nurse, a random ED nurse, and two faculty (one fixed, the other from a dyad). The agreement was calculated with intraclass correlation coefficients (ICC).

Results

The simulation was performed on 35 separate days over two academic years. A total of 106 MSF participants were enrolled. Enrollees included three faculty members, 35 team leaders, 34 peers, 33 ED registered nurses (RN), and one Peds RN; 50% of the enrollees were female (n=53). Mean QSAT scores ranged from 20.7 to 23.4. A fair agreement was demonstrated via ICC; there was no statistically significant difference between sources of MSF. Removing self-evaluation led to the highest ICC. ICC for any single or grouped non-faculty source of MSF was poor.

Conclusion

Using the QSAT, the findings from this single-site cohort suggest that faculty must be included in MSF. Self-evaluation appears to be of limited value in MSF with the QSAT. The degree of MSF agreement as gathered by the QSAT was lower in this cohort than previously reported for adult simulation cases performed in the simulation lab. This may be due to either the pediatric nature of the case, the location of the simulation, or both.

The Ottawa Resident Observation Form for Nurses (O-RON): Assessment of Resident Performance through the Eyes of the Nurses

OBJECTIVE

Most work-place based assessment relies on physician supervisors making observations of residents. Many areas of performance are not directly observed by physicians but rather by other healthcare professionals, most often nurses. Assessment of resident performance by nurses is captured with multi-source feedback tools. However, these tools combine the assessments of nurses with other healthcare professionals and so their perspective can be lost. A novel tool was developed and implemented to assess resident performance on a hospital ward from the perspective of the nurses.

DESIGN

Through a nominal group technique, nurses identified dimensions of performance that are reflective of high-quality physician performance on a hospital ward. These were included as items in the Ottawa Resident Observation Form for Nurses (O-RON). The O-RON was voluntarily completed during an 11-month period. Validity evidence related to quantitative and qualitative data was collected.

SETTING

The Orthopedic Surgery Residency Program at the University of Ottawa.

PARTICIPANTS

49 nurses on the Orthopedic Surgery wards at The Ottawa Hospital (tertiary care).

RESULTS

The O-RON has 15 items rated on a 3-point frequency scale, one global judgment yes/no question regarding whether they would want the resident on their team and a space for comments. 1079 O-RONs were completed on 38 residents. There was an association between the response to the global judgment question and the frequency of concerns (p < 0.01). With 8 forms per resident, the reliability of the O-RON was 0.80. Open-ended responses referred to aspects of interpersonal skills, responsiveness, dependability, communication skills, and knowledge.

CONCLUSIONS

The O-RON demonstrates promise as a work-place based assessment tool to provide residents and training programs with feedback on aspects of their performance on a hospital ward through the eyes of the nurses. It appears to be easy to use, has solid evidence for validity and can provide reliable data with a small number of completed forms.

A Training Intervention through a 360° Multisource Feedback Model

Physicians and other health sciences professionals need continuous training, not only in technical aspects of their activity but also in nontechnical, transversal competencies with a cost-efficient impact on the proper functioning of healthcare. The objective of this paper is to analyze the behavioral change among health professionals at a large public hospital following a training intervention on a set of core nontechnical competencies: Teamwork, Adaptability-Flexibility, Commitment-Engagement, Results Orientation, and Leadership Skills for Supervisors. The 360° Multisource Feedback (MSF) model was applied using three sources of information: supervisors, co-workers, and the workers themselves (self-assessment). A quasi-experimental pretest–post-test single-group design with two points in time was utilized. The training intervention improved the scores of only one of the trained competencies—the “Results Orientation” competency—although the scores were slightly inflated. Moreover, significant discrepancies were detected between the three sources, with supervisors awarding the highest scores. The magnitude of behavioral change was related to certain sociodemographic and organizational variables. The study was not immune to the ceiling effect, despite control measures aimed at avoiding it. The empirical evidence suggests that the 360° MSF model must be maintained over time to enhance and reinforce an evaluation culture for better patient care.

Feasibility, quality and validity of narrative multisource feedback in postgraduate training: a mixed-method study

OBJECTIVES

To examine a narrative multisource feedback (MSF) instrument concerning feasibility, quality of narrative comments, perceptions of users (face validity), consequential validity, discriminating capacity and number of assessors needed.

DESIGN

Qualitative text analysis supplemented by quantitative descriptive analysis.

SETTING

Internal Medicine Departments in Zealand, Denmark.

PARTICIPANTS

48 postgraduate trainees in internal medicine specialties, 1 clinical supervisor for each trainee and 376 feedback givers (respondents).

INTERVENTION

This study examines the use of an electronic, purely narrative MSF instrument. After the MSF process, the trainee and the supervisor answered a postquestionnaire concerning their perception of the process. The authors coded the comments in the MSF reports for valence (positive or negative), specificity, relation to behaviour and whether the comment suggested a strategy for improvement. Four of the authors independently classified the MSF reports as either 'no reasons for concern' or 'possibly some concern', thereby examining discriminating capacity. Through iterative readings, the authors furthermore tried to identify how many respondents were needed in order to get a reliable impression of a trainee.

RESULTS

Out of all comments coded for valence (n=1935), 89% were positive and 11% negative. Out of all coded comments (n=4684), 3.8% were suggesting ways to improve. 92% of trainees and supervisors preferred a narrative MSF to a numerical MSF, and 82% of the trainees discovered performance in need of development, but only 53% had made a specific plan for development. Kappa coefficients for inter-rater correlations between four authors were 0.7-1. There was a significant association (p<0.001) between the number of negative comments and the qualitative judgement by the four authors. It was not possible to define a specific number of respondents needed.

CONCLUSIONS

A purely narrative MSF contributes with educational value and experienced supervisors can discriminate between trainees' performances based on the MSF reports.

"When in Doubt, Ask the Patient": A Quantitative, Patient-Oriented Approach to Formative Assessment of CanMEDS Roles

INTRODUCTION

Since the introduction of competency-based frameworks into postgraduate medical curricula, educators have struggled to implement robust assessment tools that document the progression of necessary skills. The global movement towards competency-based medical education demands validated assessment tools. Our objective was to provide validity evidence for the Ottawa CanMEDS Competency Assessment Tool (OCCAT), designed to assess clinical performance in the communicator, professional, and health advocate CanMEDS roles.

METHODS

We developed the OCCAT, a 29-item questionnaire informed by specialty-specific Entrustable Professional Activities and consultation with stakeholders, including patients. Our sample included nine neonatal-perinatal medicine and maternal fetal medicine fellows rotating through antenatal high-risk clinics at the Ottawa Hospital. Following 70 unique encounters, the OCCAT was completed by patients and learners. Generalizability theory was used to determine overall reliability of scores. Differences in self and patient ratings were assessed using analyses of variance.

RESULTS

Generalizability analysis demonstrated that both questionnaires produced reliable scores (G-coefficient > 0.9). Self-scores were significantly lower than patient scores across all competencies, F(1, 6) = 13.9, p = .007. Variability analysis demonstrated that trainee scores varied across all competencies, suggesting both groups were able to recognize competencies as distinct and discriminate favorable behaviors belonging to each.

DISCUSSION

Our findings lend support to the movement to integrate self-assessment and patient feedback in formal evaluations for the purpose of enriched learner experiences and improved patient outcomes. We anticipate that the OCCAT will facilitate bridging to competency-based medical education.

Educational value of surgical telementoring

Educating surgeons is a time‐consuming process. In addition to theoretical knowledge, the practical tasks of surgical procedures must be mastered. Translation of such knowledge from mentor to mentee may be efficiently done by surgical telementoring (ST). This is a review on surgical telementoring. Recent technological advances have made this tool in surgical education more available and applicable but future applications of ST have to be wisely guided by high‐quality trials.

Multisource Feedback in the Trauma Context: Priorities and Perspectives

OBJECTIVES

Trauma resuscitations require competence in both clinical and nonclinical skills but these can be difficult to observe and assess. Multisource feedback (MSF) is workplace-based, involves the direct observation of learners, and can provide feedback on nonclinical skills. We sought to compare and contrast the priorities of multidisciplinary trauma team members when assessing resident trauma team captain (TTC) performance. Additionally, we aimed to describe the nature of the assessment and perceived the utility of incorporating MSF into the trauma context.

METHODS

A convenience sample of 10 trauma team activations were observed. Following each activation, the attending physician trauma team leader (TTL), TTC, and a registered nurse (RN) participated in a semistructured interview. MSF was not provided to the TTC for the purpose of this study because MSF was not part of the assessment process of TTCs at the time of this study and maintaining anonymity may have encouraged more honest interview responses. Transcripts from each assessor group (TTL, TTC, RN) were coded and assigned to one of the five crisis resource management skills: leadership, communication, situational awareness, resource utilization, and problem-solving. Comments were also coded as positive, negative, or neutral as interpreted by the coder.

RESULTS

All assessor groups mentioned communication skills most frequently. After communication, the RN and TTC groups commented on situational awareness most frequently, comprising 15 and 29% of their total responses, respectively, whereas 31% of the TTL comments focused on leadership skills. The RN and TTL groups provided positive assessments, with 51 and 42% of their respective comments coded as positive. Forty-five percent of self-assessment comments in the TTC group were negative. All (100%) of the TTC and TTL respondents felt that incorporating MSF would add to the quality of feedback, only 66% of the RN group felt that way.

CONCLUSIONS

We found that each assessor group brings a unique focus and perspective to the assessment of resident TTC performance. The future inclusion of MSF in the trauma team context has the potential to enhance the learning environment in a clinical arena that is difficult to directly observe and assess.

Video consultations during the coronavirus disease 2019 pandemic are associated with high satisfaction for both doctors and patients

Background and Aim

Telehealth has become the standard of care during the COVID-19 outbreak. This study aimed to assess doctor and patient satisfaction of endoscopy-related telehealth clinics with video consultations.

Methods

A prospective observational study of patients consecutively booked to attend two endoscopy-related telehealth clinics at an ambulatory tertiary care setting was conducted from July to October 2020. Data collected from our previously published study using phone consultations (data collected in April-May 2020) were used as a control arm. The primary outcome (satisfaction) was assessed through the six-question score (6Q_score) as per previous research. Secondary outcomes included failure-to-attend (FTA) rate and perceived necessity of physical examination/in-person follow-up appointment.

Results

There were 962 endoscopy clinic appointments between July and October, of which 157 were conducted through video. Data on 127 doctor questionnaires and 94 patient questionnaires were analyzed. The median age (years) of patients reviewed via video [57, interquartile range (IQR) 48-66] was lower than those reviewed via phone (65, IQR 55-74, P < 0.01). Patient average 6Q_score was higher with video compared to phone (85.1% vs 78.4%, P = 0.01), as was doctors' 6Q_score (97.5% vs 91.9%, P = 0.02). FTA rates remained similar between the two assessments (6.4% in April/May and 4.4% between July/October, P = 0.12). The requirement for in-person follow-up/physical examination was identified in two video consultations (1.6%).

Conclusion

Video consultations during the COVID-19 outbreak demonstrated higher patient and doctor satisfaction compared to phone consultations. There was no significant difference in FTA rates and need for in-person follow-up consultations/physical examination between the telehealth two modalities.

Assessing anaesthesiology and intensive care specialty physicians: An Italian language multisource feedback system

BACKGROUND

Physician professionalism, including anaesthesiologists and intensive care doctors, should be continuously assessed during training and subsequent clinical practice. Multi-source feedback (MSF) is an assessment system in which healthcare professionals are assessed on several constructs (e.g., communication, professionalism, etc.) by multiple people (medical colleagues, coworkers, patients, self) in their sphere of influence. MSF has gained widespread acceptance for both formative and summative assessment of professionalism for reflecting on how to improve clinical practice.

METHODS

Instrument development and psychometric analysis (feasibility, reliability, construct validity via exploratory factor analysis) for MSF questionnaires in a postgraduate specialty training in Anaesthesiology and intensive care in Italy. Sixty-four residents at the Università del Piemonte Orientale (Italy) Anesthesiology Residency Program. Main outcomes assessed were: development and psychometric testing of 4 questionnaires: self, medical colleague, coworker and patient assessment.

RESULTS

Overall 605 medical colleague questionnaires (mean of 9.3 ±1.9) and 543 coworker surveys (mean 8.4 ±1.4) were collected providing high mean ratings for all items (> 4.0 /5.0). The self-assessment item mean score ranged from 3.1 to 4.3. Patient questionnaires (n = 308) were returned from 31 residents (40%; mean 9.9 ± 6.2). Three items had high percentages of "unable to assess" (> 15%) in coworker questionnaires. Factor analyses resulted in a two-factor solution: clinical management with leadership and accountability accounting for at least 75% of the total variance for the medical colleague and coworker's survey with high internal consistency reliability (Cronbach's α > 0.9). Patient's questionnaires had a low return rate, a limited exploratory analysis was performed.

CONCLUSIONS

We provide a feasible and reliable Italian language MSF instrument with evidence of construct validity for the self, coworkers and medical colleague. Patient feedback was difficult to collect in our setting.

Implementing online patient feedback in a 'special measures' acute hospital: A case study using Normalisation Process Theory

Objective

Online patient feedback is becoming increasingly prevalent on an international scale. However, limited research has explored how healthcare organisations implement such feedback. This research sought to explore how an acute hospital, recently placed into ‘special measures’ by a regulatory body implemented online feedback to support its improvement journey.

Methods

Semi-structured interviews were conducted with eleven key stakeholders involved in the implementation and/or use of online patient feedback. Data was analysed using deductive thematic analysis with Normalisation Process Theory used as the analytical framework. Research findings are translated into the Engage, Support and Promote (ESP) model, a model of rapid feedback adoption.

Results

Participants viewed the implementation of online feedback as an opportunity to learn, change and improve. Factors found to facilitate implementation were often linked to engagement, support and promotion. Although less frequently described, barriers to implementation included staff anxieties about time pressures, moderation processes and responding responsibilities. Such anxieties were often addressed by activities including the provision of evidence based responder training. Overall, staff were overwhelmingly positive about the value of online feedback with 24 impacts identified at an individual and organisational level, including the ability to boost staff morale, resilience and pride.

Conclusions

The rapid implementation of online patient feedback can be achieved in a ‘special measures’ organisation. However, the difficulties of implementing such feedback should not be underestimated. In order to embed online feedback, staff members need to be engaged and feel supported, with opportunities to provide, respond and invite patient feedback frequently promoted to both patients and staff members.

Feedback from health professionals in postgraduate medical education: Influence of interprofessional relationship, identity and power

INTRODUCTION

Capitalising on direct workplace observations of residents by interprofessional team members might be an effective strategy to promote formative feedback in postgraduate medical education. To better understand how interprofessional feedback is conceived, delivered, received and used, we explored both feedback provider and receiver perceptions of workplace feedback.

METHODS

We conducted 17 individual interviews with residents and eight focus groups with health professionals (HPs) (two nurses, two rehabilitation therapists, two pharmacists and two social workers), for a total of 61 participants. Using a constructivist grounded theory approach, data collection and analysis proceeded as an iterative process using constant comparison to identify and explore themes.

RESULTS

Conceptualisations and content of feedback were dependent on whether the resident was perceived as a learner or a peer within the interprofessional relationship. Residents relied on interprofessional role understanding to determine how physician competencies align with HP roles. The perceived alignment was unique to each profession and influenced feedback credibility judgements. Residents prioritised feedback from physicians or within the Medical Expertise domain-a role that HPs felt was over-valued. Despite ideal opportunities for direct observation, operational enactment of feedback was influenced by power differentials between the professions.

DISCUSSION

Our results illuminate HPs' conceptualisation of feedback for residents and the social constructs influencing how their feedback is disseminated. Professional identity and social categorisation added complexity to feedback acceptance and incorporation. To ensure that interprofessional feedback can achieve desired outcomes, education programmes should implement strategies to help mitigate intergroup bias and power imbalance.

Benefits of phone consultation for endoscopy-related clinics in the COVID-19 pandemic

BACKGROUND AND AIM

During COVID-19 outbreak, restrictions to in-person consultations were introduced with a rise in telehealth. An indirect benefit of telehealth could be better attendance. This study aimed to assess "failure-to-attend" (FTA) rate and satisfaction for two endoscopy-related compulsory telehealth clinics during the COVID-19 outbreak.

METHODS

Consecutive patients booked for endoscopy-related telehealth clinics at a tertiary hospital were prospectively assessed. In-person clinic control data were assessed retrospectively. Sample size was calculated to detect an anticipated increase in attendance of 8%. Secondary outcomes included FTA differences between clinics and evaluation of patients and doctors satisfaction. Satisfaction was assessed based on six Likert scale questions used in previous telehealth research and asked to both patients and doctors (6Q_score). This study was exempt from IRB review after institutional IRB review.

RESULTS

There were 691 patients booked for appointments in our endoscopy clinics during the study periods (373 in 2020). FTA rates were lowered by half during the compulsory telehealth clinics (12.6% to 6.4%, P < 0.01). The patient 6Q_score was higher for the advanced endoscopy clinic (84.6% vs 73.8%, P < 0.01), while the doctor 6Q_score was similar between both advanced clinics and post endoscopy clinics (91.1% vs 92.5% respectively, P = 0.80). An in-person follow-up consultation was suggested for 3.5% of the appointments, while the necessity of physical examination was flagged in 5.1%.

CONCLUSIONS

The use of phone consultations in endoscopy-related clinics during the COVID-19 outbreak has improved FTA rates while demonstrating high satisfaction rates. The need for in-person follow-up consultations and physical examination were low.

Assessing leadership in junior resident physicians: using a new multisource feedback tool to measure Learning by Evaluation from All-inclusive 360 Degree Engagement of Residents (LEADER)

Background

The multifaceted nature of leadership as a construct has implications for measuring leadership as a competency in junior residents in healthcare settings. In Canada, the Royal College of Physicians and Surgeons of Canada’s CanMEDS physician competency framework includes the Leader role calling for resident physicians to demonstrate collaborative leadership and management within the healthcare system. The purpose of this study was to explore the construct of leadership in junior resident physicians using a new multisource feedback tool.

Methods

To develop and test the Learning by Evaluation from All-Inclusive 360 Degree Engagement of Residents (LEADER) Questionnaire, we used both qualitative and quantitative research methods in a multiphase study. Multiple assessors including peer residents, attending physicians, nurses, patients/family members and allied healthcare providers as well as residents’ own self-assessments were gathered in healthcare settings across three residency programmes: internal medicine, general surgery and paediatrics. Data from the LEADER were analysed then triangulated using a convergent-parallel mixed-methods study design.

Results

There were 230 assessments completed for 27 residents. Based on key concepts of the Leader role, two subscales emerged: (1) Personal leadership skills subscale (Cronbach’s alpha=0.81) and (2) Physicians as active participant-architects within the healthcare system (abbreviated to active participant-architects subscale, Cronbach’s alpha=0.78). There were seven main themes elicited from the qualitative data which were analogous to the five remaining intrinsic CanMEDS roles. The remaining two themes were related to (1) personal attributes unique to the junior resident and (2) skills related to management and administration.

Conclusions

For healthcare organisations that aspire to be proactive rather than reactive, we make three recommendations to develop leadership competence in junior physicians: (1) teach and assess leadership early in training, (2) empower patients to lead and transform training and care by evaluating doctors, (3) activate frontline care providers to be leaders by embracing patient and team feedback.

A german-language competency-based multisource feedback instrument for residents: development and validity evidence

BACKGROUND

In medical settings, multisource feedback (MSF) is a recognised method of formative assessment. It collects feedback on a doctor's performance from several perspectives in the form of questionnaires. Yet, no validated MSF questionnaire has been publicly available in German. Thus, we aimed to develop a German MSF questionnaire based on the CanMEDS roles and to investigate the evidence of its validity.

METHODS

We developed a competency-based MSF questionnaire in German, informed by the literature and expert input. Four sources of validity evidence were investigated: (i) Content was examined based on MSF literature, blueprints of competency, and expert-team discussions. (ii) The response process was supported by analysis of a think-aloud study, narrative comments, "unable to comment" ratings and evaluation data. (iii) The internal structure was assessed by exploratory factor analysis, and inter-rater reliability by generalisability analysis. Data were collected during two runs of MSF, in which 47 residents were evaluated once (first run) or several times (second and third run) on 81 occasions of MSF. (iv) To investigate consequences, we analysed the residents' learning goals and the progress as reported via MSF.

RESULTS

Our resulting MSF questionnaire (MSF-RG) consists of 15 items and one global rating, which are each rated on a scale and accompanied by a field for narrative comments and cover a construct of a physician's competence. Additionally, there are five open questions for further suggestions. Investigation of validity evidence revealed that: (i) The expert group agreed that the content comprehensively addresses clinical competence; (ii) The response processes indicated that the questions are understood as intended and supported the acceptance and usability; (iii) For the second run, factor analysis showed a one-factor solution, a Cronbach's alpha of 0.951 and an inter-rater reliability of 0.797 with 12 raters; (iv) There are indications that residents benefitted, considering their individual learning goals and based on their ratings reported via MSF itself.

CONCLUSIONS

To support residency training with multisource feedback, we developed a German MSF questionnaire (MSF-RG), which is supported by four sources of validity evidence. This MSF questionnaire may be useful to implement MSF in residency training in German-speaking regions.

A randomised controlled trial of feedback to improve patient satisfaction and consultation skills in medical students

BACKGROUND

The use of feedback has been integral to medical student learning, but rigorous evidence to evaluate its education effect is limited, especially in the role of patient feedback in clinical teaching and practice improvement. The aim of the Patient Teaching Associate (PTA) Feedback Study was to evaluate whether additional written consumer feedback on patient satisfaction improved consultation skills among medical students and whether multisource feedback (MSF) improved student performance.

METHODS

In this single site, double-blinded randomised controlled trial, 71 eligible medical students from two universities in their first clinical year were allocated to intervention or control and followed up for one semester. They participated in five simulated student-led consultations in a teaching clinic with patient volunteers living with chronic illness. Students in the intervention group received additional written feedback on patient satisfaction combined with guided self-reflection. The control group received usual immediate formative multisource feedback from tutors, patients and peers. Student characteristics, baseline patient-rated satisfaction scores and tutor-rated consultation skills were measured.

RESULTS

Follow-up assessments were complete in 70 students attending the MSF program. At the final consultation episodes, both groups improved patient-rated rapport (P = 0.002), tutor-rated patient-centeredness and tutor-rated overall consultation skills (P = 0.01). The intervention group showed significantly better tutor-rated patient-centeredness (P = 0.003) comparing with the control group. Distress relief, communication comfort, rapport reported by patients and tutor-rated clinical skills did not differ significantly between the two groups.

CONCLUSIONS

The innovative multisource feedback program effectively improved consultation skills in medical students. Structured written consumer feedback combined with guided student reflection further improved patient-centred practice and effectively enhanced the benefit of an MSF model. This strategy might provide a valuable adjunct to communication skills education for medical students.

TRIAL REGISTRATION

Australian New Zealand Clinical Trials Registry Number ACTRN12613001055796 .

Survey of physician attitudes to using multisource feedback for competence assessment in Alberta

BACKGROUND

The use of multisource feedback (MSF) for assessing physician performance is widespread and rapidly growing. Findings from early very small research studies using highly selected participants suggest high levels of satisfaction and support. However, after nearly two decades of experience using MSF to evaluate all physicians in Alberta, we are sceptical of this.

OBJECTIVES

To determine physicians' actual opinions of MSF using the entire physician population of Alberta, Canada DESIGN: Online survey.

SETTING

Alberta, Canada.

PARTICIPANTS

All physicians with a full licence to practice in Alberta in 2015.

INTERVENTIONS

All participants were asked to grade how well they thought MSF was at assessing various aspects of physician performance using a 10-point Likert-type scale. There was also a text response field for written comments.

OUTCOMES

Mean responses to quantitative questions. Qualitative content and thematic analysis of open-ended text responses.We analysed the data using SPSS V.23 and NVivo V.11 and built a multivariate model highlighting the predictors of high and low opinions of MSF.

RESULTS

Survey response rate was high for physicians with 2215 responses (25%). The mean rating for how successful MSF was at assessing a variety of dimensions, varied from a low of 5.03/10 for medical knowledge to a high of 6.38/10 for professionalism and communication. Canadian-trained MDs rated MSF significantly lower on every dimension by approximately 20% compared with non-Canadian-trained MDs.

CONCLUSIONS

Alberta physicians have much lower opinions about the ability of MSF to measure any dimension of their performance than what has been suggested in the literature. Canadian-trained MDs have a particularly low opinion of MSF for reasons that remain unclear. The results of this survey offer a serious challenge to the effectiveness of a programme that is designed to promote self-reflection and performance improvement.

Understanding unwarranted variation in clinical practice: a focus on network effects, reflective medicine and learning health systems

In the past decades, extensive research has been performed on the phenomenon of unwarranted clinical variation in clinical practice. Many studies have been performed on signaling, describing and visualizing clinical variation. We argue that it is time for next steps in practice variation research. In addition to describing and signaling variation patterns, we argue that a better understanding of causes of variation should be gained. Moreover, target points for improving and decreasing clinical variation should be created. Key elements in this new focus should be research on the complex interaction of networks, reflective medicine, patient beliefs and objective criteria for treatment choices. By combining these different concepts, alternative research objectives and new targets for improving and reducing unwarranted variation may be defined. In this perspective, we reflect on these concepts and propose target points for future research.

Internal Medicine Resident Professionalism Assessments: Exploring the Association With Patients' Overall Satisfaction With Their Hospital Stay

PURPOSE

Successful training of internal medicine (IM) residents requires accurate assessments. Patients could assess IM residents in a hospital setting, but medical educators must understand how contextual factors may affect assessments. The objective was to investigate relationships between patient, resident, and hospital-encounter characteristics and the results of patient assessments of IM resident professionalism.

METHOD

The authors performed a prospective cohort study of postgraduate year 1 (PGY-1) IM residents and their patients at 4 general medicine inpatient teaching services at Mayo Clinic Hospital-Rochester, Saint Marys Campus in Rochester, Minnesota, from July 1, 2015, through June 30, 2016. Patient assessments of resident professionalism were adapted from validated instruments. Multivariable modeling with generalized estimating equations was used to determine associations between patient assessment scores and characteristics of residents, residents' clinical performance and evaluations (including professionalism assessments in other settings), patients, and hospital encounters and to account for repeated assessments of residents.

RESULTS

A total of 409 patients assessed 72 PGY-1 residents (mean [SD], 5.7 [3.0] patient assessments per resident). In the multivariable model, only the highest rating out of 5 levels for overall satisfaction with hospital stay was significantly associated with patient assessment scores of resident professionalism (β [SE], 0.80 [0.08]; P < .001). Hospitalized patients' assessment scores of resident professionalism were not significantly correlated with assessment scores of resident professionalism in other clinical settings.

CONCLUSIONS

Hospitalized patients' assessment scores of in-hospital resident professionalism were strongly correlated with overall patient satisfaction with hospital stay but were not correlated with resident professionalism in other settings. The limitations of patient evaluations should be considered before incorporating these evaluations into programs of assessment.