Assessing the quality of clinical teachers: a systematic review of content and quality of questionnaires for assessing clinical teachers

Medical Teachers' Roles from the Students' Perspective: A Moroccan National Survey-based Study

BACKGROUND

The present study aimed to explore medical students' expectations from their teachers in Moroccan medical schools, regarding their primary educational roles.

METHODS

An online questionnaire was distributed to medical students from the five main public medical schools in Morocco using a snowball sampling strategy.

RESULTS

Two thousand four hundred and eighteen complete answers were collected and analyzed. The respondents gave the highest scores to "information provider" and the lowest scores to "Planner." When asked about their most preferred role they would want to see in their teachers, 40.6% of the respondents chose "Facilitator."

CONCLUSION

These results highlight that Moroccan medical schools should become more invested in training their faculty members to meet the rapidly changing demands and requirements of modern medical education.

A scoping review of the questionnaires used for the assessment of the perception of undergraduate students of the learning environment in healthcare professions education programs

The learning environment (LE) includes social interactions, organizational culture, structures, and physical and virtual spaces that influence the learning experiences of students. Despite numerous studies exploring the perception of healthcare professional students (HCPS) of their LE, the validity evidence of the utilized questionnaires remains unclear. This scoping review aimed to identify questionnaires used to examine the perception of undergraduate HCPS of their LE and to assess their validity evidence. Five key concepts were used: (1) higher education; (2) questionnaire; (3) LE; (4) perception; and (5) health professions (HP). PubMed, ERIC, ProQuest, and Cochrane databases were searched for studies developing or adapting questionnaires to examine LE. This review employed the APERA standards of validity evidence and Beckman et al. (J Gen Intern Med 20:1159-1164, 2005) interpretation of these standards according to 5 categories: content, internal structure, response process, relation to other variables, and consequences. Out of 41 questionnaires included in this review, the analysis revealed a predominant emphasis on content and internal structure categories. However, less than 10% of the included questionnaires provided information in relation to other variables, consequences, and response process categories. Most of the identified questionnaires received extensive coverage in the fields of medicine and nursing, followed by dentistry. This review identified diverse questionnaires utilized for examining the perception of students of their LE across different HPs. Given the limited validity evidence for existing questionnaires, future research should prioritize the development and validation of psychometric measures. This will ultimately ensure sound and evidence-based quality improvement measures of the LE in HP education programs.

Creation and Initial Validation of the Mayo Outpatient Precepting Evaluation Tool

BACKGROUND AND OBJECTIVES

Preceptors in family medicine residencies need feedback to improve. When we found no validated, behavior-based tool to assess the outpatient precepting of family medicine residents, we sought to fill this gap by developing and initially validating the Mayo Outpatient Precepting Evaluation Tool (MOPET).

METHODS

To develop the MOPET, we applied the Stanford Faculty Development Program (SFDP) theoretical framework for education, more recent work on peer review of medical teaching, and expert review of items. The residency behavioral scientist and a volunteer physician independently completed the MOPET while co-observing a precepting physician during continuity clinic sessions (N=20). We assessed the tool's validity via interrater reliability and cross-validation with the SFDP-26.

RESULTS

The tool demonstrated high interrater reliability for the following effective teaching behaviors: (a) allowing the resident to present without interrupting, (b) encouraging the formulation of a goal, (c) checking in on the resident's goal, (d) using multimodal teaching aids, (e) asking to discuss the differential diagnosis, (f) asking to discuss alternative management, (g) encouraging the resident to pursue literature and/or other resources, and (h) reinforcing self-directed learning. The MOPET measures strongly correlated with most items from the SFDP-26, indicating good cross-validity.

CONCLUSIONS

The MOPET is a theoretically sound, behavior-based, reliable, and initially validated tool for peer review of outpatient family medicine resident teaching. This tool can support faculty development in outpatient clinical learning environments.

Better Together: Development and Implementation of Fellow Group Evaluations of Faculty

Background

High-quality trainee evaluations of faculty are essential for meaningful faculty development and for improving the clinical learning environment. However, concerns about anonymity can limit usefulness of trainee evaluations, particularly in smaller programs, such as subspecialty fellowships.

Objective

To develop and implement a fellow-driven group evaluation process to enhance trainee confidentiality and generate high-quality feedback for pulmonary and critical care medicine faculty.

Methods

A novel process was developed for faculty evaluation and feedback consisting of quarterly, structured, fellow-led group evaluation sessions focused on collecting confidential, behaviorally oriented, actionable feedback for faculty. Upper-year fellow moderators utilized a standard format to structure discussion, generating strengths and areas for growth for each faculty member while explicitly asking for input from fellows with divergent perspectives. Moderators compiled anonymized session notes for the program director, who delivered feedback to individual faculty. After the first six sessions, an electronic survey was distributed to assess fellow perceptions of the group evaluation model.

Results

Thirty-seven faculty members were evaluated in 11 group sessions over 42 months. Fellows rated group-generated feedback as more confidential, more specific, more accurate, more efficient, more actionable, and less biased when compared with individual written evaluations (P < 0.01 for all categories).

Conclusion

The authors successfully developed and implemented a process for fellow-led group evaluation of faculty, designed to facilitate fellow confidentiality and enrich the quality of feedback. Fellows preferred the group evaluation process and perceived group-generated feedback more favorably compared with individual written evaluations.

Peer observation of teaching: A feasible and effective method of physician faculty development

Physician faculty learn teaching skills informally while fighting competing professional obligations. One underutilized proven technique to improve teaching skills is peer observation with feedback. We aimed to understand benefits and challenges of a physician faculty development program based on peer observation of teaching and to develop best practice recommendations for future program development. The authors developed a peer observation-based physician faculty development program from 2015 to 2017. Two interviewers conducted and analyzed qualitative interviews with 13 faculty participants and four non-participants using content analysis to identify themes and subthemes in NVivo©. Participant-identified program benefits included conveyed institutional support for teaching, the opportunity for peer observation with direct and timely feedback, the opportunity for community building, and overall program feasibility. Program challenges included competing scheduling demands, variability in feedback quality, and difficulty maintaining engagement for the program duration. Potential areas for improvement included participation incentives, external faculty involvement, assistance with program logistics and administration, and improvement in the consistency of the feedback experience. While peer observation is a valued approach to physician faculty development of teaching skills, competing demands on physicians may still limit program effectiveness. Program sustainability depends on optimizing feedback quality, boosting motivation for participation, and providing administrative support.

Necessary but Insufficient and Possibly Counterproductive: The Complex Problem of Teaching Evaluations

The evaluation of clinical teachers' performance has long been a subject of research and debate, yet teaching evaluations (TEs) by students remain problematic. Despite their intuitive appeal, there is little evidence that TEs are associated with students' learning in the classroom or clinical setting. TEs are also subject to many forms of bias and are confounded by construct-irrelevant factors, such as the teacher's physical attractiveness or personality. Yet they are used almost exclusively as evaluations of and feedback to teachers. In this commentary, the authors review the literature on what TEs are meant to do, what they actually do in the real world, and their overall impact. The authors also consider productive ways forward. While TEs are certainly necessary to provide the crucial student voice, they are insufficient as the sole way to assess teachers. Further, they are often counterproductive. TEs carry so much weight for faculty that they can act as a disincentive for teachers to challenge learners and provide them with the critical feedback they often need, lest students give them poor ratings. To address these challenges, changes are needed, including embedding TEs in a programmatic assessment framework. For example, TEs might be used for formative feedback only, while other sources of data, such as peer assessments, learning outcomes, 360-degree feedback, and teacher reflections, could be collated into a portfolio to provide a more meaningful evaluation for teachers. Robust, transparent systems should be in place that dictate how TE data are used and to ensure they are not misused. Clinical teachers who do not "fail to fail" learners but instead take the time and effort to identify and support learners in difficulty should be recognized and rewarded. Learners need this support to succeed and the obligation to protect patients demands it.

"Influential" Intraoperative Educators and Variability of Teaching Styles

OBJECTIVES

Academic surgeons manage their role as intraoperative educators in a variety of ways. Such variability is neither idiosyncratic nor is there a single best approach. This study sought to explore the practices of surgeons deemed influential by their residents, allowing insight into a variety of potentially effective practices.

PARTICIPANTS

Constructivist grounded theory guided data collection and analysis. Data sources included surveys from senior surgical residents (PGY3-6) and recent graduates from an academic hospital in Canada (36% response rate), intraoperative observations of teaching interactions, and semi-structured interviews with observed surgeons. Rigour was supported by data triangulation, constant comparison, and collection to theoretical sufficiency.

DESIGN

We developed a framework grouping effective teaching into three overlapping approaches: exacting, empowering, and fostering. The approaches differ based on the level of independence granted and the degree of expectation placed on individual residents. Each demonstrates different strategies for balancing the multiple supervisory roles and patient care obligations faced by academic surgeons. We also identified strategies that could be used across approaches to enhance learning.

CONCLUSIONS

For surgical educators seeking to improve upon the quality of the intraoperative supervision they provide, frameworks such as this may serve as models of effective supervision. Enhancing surgeons' knowledge of proven strategies, combined with reflecting on how they teach and how they balance responsibilities to patients and trainees, may allow them to broaden their educational practice.

"What is the mechanism?": Cues, barriers, and opportunities to discuss foundational science during internal medicine rounds

BACKGROUND

Repeated application of foundational science (FS) during medical reasoning results in encapsulation of knowledge needed to develop clinical expertise. Despite proven benefit of educating learners using a FS framework to anchor clinical decision making, how FS is integrated on clinical rotations has not been well characterized. This study examines how and when FS discussion occurs on internal medicine teaching rounds.

MATERIAL AND METHODS

We performed a convergent mixed method study. Six internal medicine teams at a quaternary hospital were observed during rounds and team members interviewed. Transcripts were analyzed using thematic analysis. Descriptive statistics provided a summary of the observations.

RESULTS

Our study revealed that rounds used a teacher-centered model where FS knowledge was transmitted as pearls external to the clinical context. FS content arose primarily when the patient was complex. Barriers preventing FS discussion were lack of time and perceived lack of personal FS knowledge.

CONCLUSION

Our study describes scenarios that commonly elicit discussion of FS on inpatient medicine rounds highlighting a 'transmission' model of FS knowledge. We suggest a learner-centered model that engages students in the practice of integrating FS into clinical reasoning.

Characteristics of good clinical teachers in anesthesiology from medical students' perspective: a qualitative descriptive study

PURPOSE

Learning needs are influenced by the stage of learning and medical specialty. We sought to investigate the characteristics of a good clinical teacher in anesthesiology from the medical students' perspective.

METHODS

We conducted a qualitative descriptive study to analyze written comments of medical students about their clinical teachers' performances. Our analysis strategy was the inductive content analysis method. The results are reported as a descriptive summary with major themes as the final product.

RESULTS

Our study identified four themes. The first theme, teachers' individual characteristics, includes characteristics that are usually more related to students' subjective experiences and feelings. The second theme, teachers' characteristics that advance student learning, seems to be one of the most important contributions to learning because it increases the practice of procedural skills. The third theme, teachers' characteristics that prepare students for success, shows characteristics that facilitate students' learning by promoting a healthy and safe environment. Lastly, the fourth theme, characteristics related to teaching approaches, includes characteristics that can guide clinical teachers more objectively.

CONCLUSION

Our analysis of the written comments of medical students identified many characteristics of a good clinical teacher that were organized in four different themes. These themes contribute to expand on existing understandings of clinical teaching in the anesthesiology clerkship environment, and add new interpretations that can be reflected upon and explored by other clinical educators.

Evaluating Clinical Educators' Competence in an East Asian Context: Who Values What?

Background

How to evaluate clinical educators is an important question in faculty development. The issue of who are best placed to evaluate their performance is also critical. However, the whos and the hows of clinical educator evaluation may differ culturally. This study aims to understand what comprises suitable evaluation criteria, alongside who is best placed to undertake the evaluation of clinical educators in medicine within an East Asian culture: specifically Taiwan.

Methods

An 84-item web-based questionnaire was created based on a literature review and medical educational experts' opinions focusing on potential raters (i.e., who) and domains (i.e., what) for evaluating clinical educators. Using purposive sampling, we sent 500 questionnaires to clinical educators, residents, Post-Graduate Year Trainees (PGYs), Year-4~6/Year-7 medical students (M4~6/M7) and nurses.

Results

We received 258 respondents with 52% response rate. All groups, except nurses, chose “teaching ability” as the most important domain. This contrasts with research from Western contexts that highlights role modeling, leadership and enthusiasm. The clinical educators and nurses have the same choices of the top five items in the “personal qualities” domain, but different choices in “assessment ability” and “curriculum planning” domains. The best fit rater groups for evaluating clinical educators were educators themselves and PGYs.

Conclusions

There may well be specific suitable domains and populations for evaluating clinical educators' competence in East Asian culture contexts. Further research in these contexts is required to examine the reach of these findings.

Attributes of excellent clinician teachers and barriers to recognizing and rewarding clinician teachers' performances and achievements: a narrative review

BACKGROUND

Over the last 31 years, there have been several institutional efforts to better recognize and reward clinician teachers. However, the perception of inadequate recognition and rewards by clinician teachers for their clinical teaching performance and achievements remains. The objective of this narrative review is two-fold: deepen understanding of the attributes of excellent clinician teachers considered for recognition and reward decisions and identify the barriers clinician teachers face in receiving recognition and rewards.

METHODS

We searched OVID Medline, Embase, Education Source and Web of Science to identify relevant papers published between 1990 and 2020. After screening for eligibility, we conducted a content analysis of the findings from 43 relevant papers to identify key trends and issues in the literature.

RESULTS

We found the majority of relevant papers from the US context, a paucity of relevant papers from the Canadian context, and a declining international focus on the attributes of excellent clinician teachers and barriers to the recognition and rewarding of clinician teachers since 2010. 'Provides feedback', 'excellent communication skills', 'good supervision', and 'organizational skills' were common cognitive attributes considered for recognition and rewards. 'Stimulates', 'passionate and enthusiastic', and 'creates supportive environment', were common non-cognitive attributes considered for recognition and rewards. The devaluation of teaching, unclear criteria, and unreliable metrics were the main barriers to the recognition and rewarding of clinician teachers.

CONCLUSIONS

The findings of our narrative review highlight a need for local empirical research on recognition and reward issues to better inform local, context-specific reforms to policies and practices.

Psychometric properties of the Persian version of System for Evaluation of Teaching Qualities by students: A tool for assessing clinical tutors from students' viewpoint

BACKGROUND

Effective clinical teaching is crucially important for patient care in future. Therefore, proper clinical training is essential to make physicians capable of delivering high-quality health care.

MATERIALS AND METHODS

The present study was a cross-sectional research. After translating the questionnaire into Persian, it was distributed among medical students in the clinical years of medical education in teaching hospitals affiliated to Ahvaz Jundishapur University of Medical Sciences in 2018. The System for Evaluation of Teaching Qualities (SETQ) has 25 questions in a 6-scale Likert scale that evaluates clinical tutors in five dimensions of teaching and learning environment, professional attitude toward students, transferring of goals, evaluation of students, feedback, and promoting self-directed learning. Instrument reliability was assessed by calculating the Cronbach's alpha coefficient, whereas questionnaire content validity was evaluated by relative content validity ratio (CVR) and content validity index (CVI). To evaluate the structural validity, an exploratory factor analysis was conducted.

RESULTS

The SETQ was completed by 127 medical students. Cronbach's alpha coefficient of the total questionnaire was estimated as 0.908. The factor analysis showed that the questionnaire was composed of six factors, explaining 66.14% of the total variance. The CVI and CVR indices of the individual items were also acceptable.

CONCLUSION

The findings of our study showed that the Persian version of SETQ questionnaire had the acceptable reliability and validity to be used in assessing clinical tutors in different hospitals in Iran.

Perceptions of postgraduate family medicine supervision at decentralised training sites, South Africa

Background

Specialist training in family medicine (FM) is growing rapidly in sub-Saharan Africa. The strong emphasis on workplace-based learning for speciality training makes it vital to gain in-depth insights into registrar supervision. Previous studies have explored aspects of supervision at decentralised sites in high-income countries, however, little is known about the benefits and constraints of decentralised postgraduate supervision in low- to middle-income countries, especially in Africa.

Aim

This study aimed to explore family physicians’ and registrars’ perceptions of the strengths and challenges of clinical and educational supervision across decentralised training sites.

Setting

The study was conducted across two provinces at five decentralised training sites affiliated with the University of the Witwatersrand, Johannesburg.

Methods

This qualitative study involved semi-structured interviews with a purposive sample of 11 FPs and 11 registrars. The data were thematically analysed.

Results

Two of the four themes identified, ‘supervision is context-specific and supervisor-dependent’, and ‘the nature of engagement matters’, involved strengths and challenges. The other two, ‘supervision is not ideal’ and ‘the training environment is challenging’, focussed on challenges.

Conclusion

Supervisors and registrars described the postgraduate FM supervision as context-specific and supervisor-dependent. Supervisors displayed good clinical-teacher characteristics and supervisory relationships. However, several challenges, including registrars’ workload, resource shortages and a lack of standardisation across training sites, need to be addressed. Regular faculty development is essential for supervisors to be aware of relevant aspects of, and current trends in, postgraduate training.

Training and Preparedness of Clinical Coaches for Their Role in Training Student Veterinary Nurses in the United Kingdom: An Exploratory Inquiry

The experience that student veterinary nurses (SVNs) have in the clinical learning environment can be greatly influenced by the clinical coach (CC); the supervisory relationship will affect student retention and clinical competency. To support a positive student experience, the training and development of CCs must be critically reviewed and regularly updated. This research aimed to ascertain the current CC training undertaken and the preparedness of CCs for their role in training SVNs. We used a prospective cross-sectional study design. An online survey was distributed over 4 weeks to CCs across the United Kingdom representing a range of educational institutions, and it generated 80 responses. Prior to undertaking their initial CC training, CCs had been qualified practitioners for a median of 2.2 years (IQR = 4.16y). CCs stated they needed more course content during their training regarding student teaching and pastoral support, more support from associated institutions, and there was a call for a longer training period leading to a formal qualification. Providing CC training with broader course content and some level of evaluation should be considered to develop and assess the non-clinical skills that are vital to the role.

Diamond goals not graphite! A triangulation approach to clinical teachers' needs assessment

Background: Clinical faculty development plays a significant role in the professional empowerment of future physicians. Identification of educational needs is an important step in planning faculty development. This study identified the educational needs of medical faculties in the clinical setting.

Methods: This cross-sectional needs assessment study was conducted in Iranian medical universities during 2016-2018 using a triangulation paradigm. A total of 384 medical clinical faculties, 54 medical education specialists, and 194 faculty evaluation forms completed by medical residents participated in the study using a convenient randomized sampling method. The data were gleaned with a researcher-made questionnaire with 14 areas developed on the basis of clinical education goals and contexts and were analyzed with SPSS16 using descriptive statistic indices such as mean, standard deviation, and frequency percentile. Analytical tests including independent t-test, chi-square and Cramer's V were also applied (p<0.05). The content validity, face validity, and reliability were approved.

Results: The response rate was %59 (227) for clinical faculties, %77 (42) for medical education specialists, and %58 (110) for residents. Professionalism was the first priority of needs from the viewpoint of clinical faculties and faculty development planners. The clinical teachers' highest level skills, in their own perspective and also students’ perspective, were procedure training and grand round, whereas their lowest level skills were emotional intelligence and morning report. The greatest gap existed between the current skill and the need is management and leadership in the clinical setting. Cramer’s index ranged between 0.18 and 0.34 (p<0.05); hence, there was a correlation between the current status and the announced needs in all subjects.

Conclusion: Designers of faculty development programs ought to pay due attention to areas of professionalism, management, and leadership and carry out accurate and comprehensive planning to enable students to become competent future physicians in the roles of therapist, manager, teacher, supporter, and researcher.

A Weighted Evaluation Study of Clinical Teacher Performance at Five Hospitals in the UK

INTRODUCTION

Evaluation of individual teachers in undergraduate medical education helps clinical teaching fellows identify their own strengths and weaknesses. In addition, evaluation data can be used to guide career decisions. In order for evaluation results to adequately reflect true teaching performance, a range of parameters should be considered when designing data collection tools.

METHODS

Clinical teaching fellows at five London teaching hospitals were evaluated by third-year students they had supervised during a ten-week clinical attachment. The questionnaire addressed (a) general teaching skills and (b) student learning outcome measured via comparative self-assessments. Teachers were ranked using different algorithms with various weights assigned to these two factors.

RESULTS

A total of 133 students evaluated 14 teaching fellows. Overall, ratings on teaching skills were largely favourable, while the perceived increase in student performance was modest. Considerable variability across teachers was observed for both factors. Teacher rankings were strongly influenced by the weighting algorithm used. Depending on the algorithm, one teacher was assigned any rank between #2 and #10.

CONCLUSION

Both parts of the questionnaire address different outcomes and thus highlight specific strengths and weaknesses of individual teachers. Programme directors need to carefully consider the weight assigned to individual components of teacher evaluations in order to ensure a fair appraisal of teacher performance.

Psychometric Properties of Visual Indicators of Teaching and Learning Success "VITALS" Instrument for Evaluation of Clinical Teachers

PURPOSE

We have previously developed an instrument for students' evaluation of clinical teachers that we called Visual Indicators of Clinical Teaching and Learning Success (VITALS). This study measures the reliability of VITALS as an instrument for student evaluation of clinical tutors. Additionally, the study explores the minimum number of student raters necessary for an acceptable reliability, and provides evidence of construct validity of the evaluation scores.

MATERIALS AND METHODS

This retrospective study included 1825 evaluation forms completed by medical students evaluating clinical tutors using the VITALS instrument. Reliability was measured by applying generalizability theory (G-theory) analysis using a two-facet design (raters and items). A D-study was used to determine the minimum number of raters required to achieve a reliability ≥0.80. Face validity was tested by measuring tutors' degree of agreement with the items of the study instrument.

RESULTS

The overall G-coefficient was 0.89. The subject of measurement (clinical tutors' scores) represented 15.8% of the variance across all tutors and items. The variance due to the interaction between raters (students) and tutors contributed to 43.5%, while the variance due to items was negligible. The remaining 40% of the variance was due to unexplained sources of error. The D-study demonstrated that a minimum of 12 raters (students) are required to achieve a reliability of 0.80. Finally, most of the clinical tutors agreed that all items in the instrument were appropriate.

CONCLUSION

We demonstrate that VITALS exhibits good psychometric properties. There should be at least 12 students rating each clinical tutor to have an acceptable level of reliability for the study instrument. Face validity of the study instrument was evidenced by its high level of approval among clinical tutors.

The wolf you feed: Challenging intraprofessional workplace-based education norms

CONTEXT

The trajectory towards becoming a medical professional is strongly situated within the clinical workplace. Through participatory engagement, medical trainees learn to address complex health care issues through collaboration with the interprofessional health care team. To help explain learning and teaching dynamics within the clinical workplace, many scholars have relied on socio-cultural learning theories. In the field of medical education, this research has largely adopted a limited interpretation of a crucial dimension within socio-cultural learning theory: the expert who guides the trainee into the community is almost exclusively from the same profession. We contend that this narrow interpretation is not necessary. This limited focus is one we choose to maintain-be that choice intentional or implicit. In this cross-cutting edge paper, we argue that choosing an interprofessional orientation towards workplace learning and guidance may better prepare medical trainees for their future role in health care practice.

METHODS

By applying Communities of Practice and Landscapes of Practice , and supported by empirical examples, we demonstrate how medical trainees are not solely on a trajectory towards the Community of Physician Practice (CoPP) but also on a trajectory towards various Landscapes of Healthcare Practice (LoHCP). We discuss some of the barriers present within health care organisations and professions that have likely inhibited adoption of the broader LoHCP perspective. We suggest three perspectives that might help to deliberately and meaningfully incorporate the interprofessional learning and teaching dynamic within the medical education continuum.

CONCLUSION

Systematically incorporating Landscapes of Competence, Assessment, and Guidance in workplace-based education-in addition to our current intraprofessional approach-can better prepare medical trainees for their roles within the LoHCP. By advocating and researching this interprofessional perspective, we can embark on a journey towards fully harnessing and empowering the health care team within workplace-based education.

An online discussion between students and teachers: a way forward for meaningful teacher feedback?

BACKGROUND

Student evaluation is an essential component in feedback processes in faculty and learner development. Ease of use and low cost have made paper evaluation forms a popular method within teaching programmes, but they are often seen as a formality, offering variable value towards the improvement of teaching. Students report poor motivation to engage with existing feedback tools whilst teachers describe receiving vague, contradicting, or irrelevant information. We believe that feedback for teachers needs to be a two-way process, similar to feedback for students, for it to be effective. An online feedback tool has been implemented for third-year medical students from Imperial College London to promote open discussion between teachers and students. The feedback tool is accessible throughout students' clinical attachment with the option of maintaining anonymity. We aim to explore the benefits and challenges of this online feedback tool and assess its value as a method for teacher feedback.

METHODS

Qualitative data was obtained from both volunteer third-year medical students of Imperial College London and Clinical Teaching Fellows using three focus groups and a questionnaire. Data was analysed through iterative coding and thematic analysis to provide over-arching analytical themes.

RESULTS

Twenty-nine students trialled this feedback tool with 17 responding to the evaluative questionnaire. Four over-arching themes were identified: reasons for poor participation with traditional feedback tools; student motivators to engage with 'open feedback'; evaluative benefits from open feedback; concerns and barriers with open feedback.

CONCLUSION

This feedback tool provides a platform for two-way feedback by encouraging open, transparent discussion between teachers and learners. It gives a unique insight into both teachers and peers' perspectives. Students engage better when their responses are acknowledged by the teachers. We elaborate on the benefits and challenges of public open feedback and approaches to consider in addressing the self-censorship of critical comments.

A prediction-based method to estimate student learning outcome: Impact of response rate and gender differences on evaluation results

BACKGROUND

Low response rates threaten the reliability and validity of student evaluations of teaching. Previous research has shown that asking students to predict how satisfied their fellow students were with a course produces reliable results at lower response rates. The aim of this study was to investigate whether this prediction-based method can also be used to evaluate student learning outcome.

METHODS

Before and after a cardiorespiratory module, 128 fourth-year medical students provided self-assessments and predictions of performance on 27 specific learning objectives and took formative tests on the respective contents. Pre-post performance gain was compared across all three modalities.

RESULTS

Formative exam results indicated a performance gain of 63.0%. Self-assessed and prediction-based performance gains were identical (67.8%) but both slightly overestimated actual performance gain. Irrespective of the method used, a response rate of 20% was sufficient to produce reliable results. Compared to male students, females greatly overestimated their peers' performance which led to inflated performance gain values.

CONCLUSIONS

Student self-assessments and predictions are equally valid sources of learning outcome measures, and low response rates are sufficient to produce stable results. When using a prediction-based approach, a tendency to overestimate learning outcome in female students needs to be taken into account.

Purpose, Pleasure, Pace and Contrasting Perspectives: Teaching and Learning in the Emergency Department

OBJECTIVES

Teaching and learning in the clinical setting are vital for the training and development of emergency physicians. Increasing service provision and time pressures in the emergency department (ED) have led to junior trainees' perceptions of a lack of teaching and a lack of support during clinical shifts. We sought to explore the perceptions of learners and supervisors in our ED regarding teaching within this diverse and challenging context.

METHODS

Nine ED physicians and eight ED trainees were interviewed to explore perceptions of teaching in the ED. Clinical teaching was described as "on-the-floor" teaching during work shifts. We used a validated clinical teaching assessment instrument to help pilot and develop some of our interview questions, and data were analyzed using qualitative thematic analysis.

RESULTS

We identified three major themes in our study: 1) the strong sense of purpose and the pleasure gained through teaching and learning interactions, despite both groups being unsure of each other's engagement and enthusiasm; 2) contrasting perspectives of teaching with registrars holding a traditional knowledge transmission view, yet shared perspectives of teacher as being ED consultants; and 3) the effect of patient acuity and volume, which both facilitated learning until a critical point of busyness beyond which service provision pressures and staffing limitations were perceived to negatively impact learning.

CONCLUSIONS

The ED is a complex and fluid working and learning environment. We need to develop a shared understanding of teaching and learning opportunities in the ED, which helps all stakeholders move beyond learning as knowledge acquisition and sees the potential for learning from teachers of a multitude of professional backgrounds.

The characteristics of an effective clinical instructor from the perspective of nursing students: a qualitative descriptive study in Iran

Background

Clinical instructors have an important role in advancing nursing students to achieve the program objectives. Nursing student perceptions about the characteristics of an effective clinical instructors may help programs improve clinical instruction. As such, the purpose of this study was to understand the characteristics perceived by students to define an effective clinical instructor.

Methods

In this qualitative descriptive study, semi-structured interviews were conducted with undergraduate and postgraduate nursing students. The sample was purposefully selected. MAXQDA software was used for the content analysis. The codes were organized into subcategories and consolidated into categories.

Results

Data saturation was reached with twelve participants, including seven women and five men, between 21 and 36 years of age. General and special characteristics were the two main categories that emerged from the data. These categories were defined by nine subcategories including internal motivation, professional acceptability, clinical competency, teaching skill, clinical experience, values, being a faculty member, appropriate appearance, and communication skills.

Conclusions

Effective clinical instructors have a specific characteristics identified by student. The most important characteristics of effective clinical instructors were related to communication and teaching skills, internal motivation, and professional appearance. More research is necessary to determine the relationship between the characteristics, instructor competency, and student learning outcomes.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12912-021-00556-9.

A multi-institutional study assessing general surgery faculty teaching evaluations

BACKGROUND

Resident evaluation of faculty teaching is an important metric in general surgery training, however considerable variability in faculty teaching evaluation (FE) instruments exists.

STUDY DESIGN

Twenty-two general surgery programs provided their FE and program demographics. Three clinical education experts performed blinded assessment of FEs, assessing adherence 2018 ACGME common program standards and if the FE was meaningful.

RESULTS

Number of questions per FE ranged from 1 to 29. The expert assessments demonstrated that no evaluation addressed all 5 ACGME standards. There were significant differences in the FEs effectiveness of assessing the 5 ACGME standards (p < 0.001), with teaching abilities and professionalism rated the highest and scholarly activities the lowest.

CONCLUSION

There was wide variation between programs regarding FEs development and adhered to ACGME standards. Faculty evaluation tools consistently built around all suggested ACGME standards may allow for a more accurate and useful assessment of faculty teaching abilities to target professional development.

Initial survey on the use of animals in scientific research and teaching reveals divided opinion of the Brazilian population

Objective:

Specific legislation regulating the use of animals in research in Brazil was introduced in 2008. However, the viewpoint of the Brazilian population regarding the use of animals in research and teaching activities remains largely unknown. Investigation of the public viewpoint on and understanding of the topic is required given the current shifts in the animal ethics scenario in Brazil. The objective of this study was to provide the first insight into the Brazilian population viewpoint on the use of animals in scientific research and teaching activities.

Methods:

Data collected in a survey involving 2,115 individuals aged 16 years or older and residing in 130 municipalities distributed across the five Brazilian macroregions (North, Northeast, South, Southeast, and Midwest) were analyzed. The margin of error for entire sample was set at 2%, with a 95% confidence interval.

Results:

This survey revealed that most Brazilian citizens are in favor of the use animals in research, particularly for medical purposes. Different views depending on the nature of research were identified. Approximately 80% of respondents were also in favor of frequent oversight of laboratories and animal facilities.

Conclusion:

Survey findings indicate that the opinion of the Brazilian population is divided when it comes to the use of animals in scientific research and teaching. Divided opinions expose a limited understanding of the importance of basic sciences and emphasizes the need for improved communication between the scientific community and the general population. Further strategies aimed to promote animal welfare are discussed.

Clinical educator self-efficacy, self-evaluation and its relationship with student evaluations of clinical teaching

BACKGROUND

In a whole-of-system approach to evaluation of teaching across any degree, multiple sources of information can help develop an educators' understanding of their teaching quality. In the health professions, student evaluations of clinical teaching are commonplace. However, self-evaluation of teaching is less common, and exploration of clinical educators' self-efficacy even less so. The aim of the study was to evaluate how a clinical educator's self-evaluation of teaching intersects with their self-efficacy, to ascertain if that matches student evaluation of their teaching. This information may assist in facilitating targeted professional development to improve teaching quality.

METHODS

Clinical educators in the osteopathy program at Victoria University (VU) were invited to complete: a) self-evaluation version of the Osteopathy Clinical Teaching Questionnaire (OCTQ); and b) the Self-Efficacy in Clinical Teaching (SECT) questionnaire. Students in the VU program completed the OCTQ for each of the clinical educators they worked with during semester 2, 2017.

RESULTS

Completed OCTQ and SECT were received from 37 clinical educators. These were matched with 308 student evaluations (mean of 6 student ratings per educator). Three possible educator cohorts were identified: a) high clinical eductor self-OCTQ with low student evaluation; b) low clinical educator self-evaluation and high student evaluations; and, c) no difference between self- and student evaulations. Clinical educators in the first cohort demonstrated significantly higher SECT subscale scores (effect size > 0.42) than their colleagues. Age, gender, teaching qualification, and years practicing or years as a clinical educator were not associated with clinical educator OCTQ scores or the SECT subscales.

CONCLUSIONS

Targeted professional development directed towards fostering self-efficacy may provide an avenue for engaging those clinical educators whose self-efficacy is low and/or those who did not receive high student evaluations. Given there is no gold standard measure of clinical teaching quality, educators should engage with multiple sources of feedback to benchmark their current performance level, and identify opportunities to improve. Student and self-evaluations using the OCTQ and evaluation of self-efficacy using the SECT, are useful tools for inclusion in a whole-of-system approach to evaluation of the clinical learning environment.

Inter-Institutional Collaboration for the Development of a Local Peer Observation Process to Enhance Teaching

Local peer observation of teaching is considered an important mechanism for instructors to improve the quality and effectiveness of their teaching, but there is an absence of uniformity to establish a best practice for this process in veterinary curricula. The Regional Teaching Academy (RTA) of the Consortium of Western Colleges of Veterinary Medicine is comprised of educational advocates from five western veterinary colleges with a common goal of enhancing the quality and effectiveness of education in veterinary medical curricula. Members of the RTA recognized this deficit in best practices for local peer observation (LPO) and formed a working group called "Local Peer Observation of Teaching." The goal was to meet a critical need for the enhancement of individual teaching skills by using a scholarly approach to develop robust methods for peer observation of teaching. Two rubric-based instruments were developed: one for large-group/didactic settings, and the second for small-group/clinical settings. Each is accompanied by pre- and post-observation worksheets which are considered instrumental to success. Results of a qualitative survey of instrument users' experiences are shared. Both observers and observees view the experiential learning from faculty peer colleagues very positively and the meaningful feedback is appreciated and incorporated by observees. Suggestions for implementation of the peer observation process are discussed, considering strengths and challenges. The purpose of this article is to describe in depth, the development process and output of the efforts of the Local Peer Observation of Teaching working group as a potential best practice guideline for peer observation.

An Inter-Institutional External Peer-Review Process to Evaluate Educators at Schools of Veterinary Medicine

Despite its fundamental importance, the educational mission of most schools of veterinary medicine receives far less recognition and support than the missions of research and discovery. This disparity is evident in promotion and tenure processes. Despite the frequent assertion that education is every college's core mission, there is a broad consensus that faculty are promoted primarily on the basis of meeting expectations relative to publications and grant funding. This expectation is evident in the promotion packets faculty are expected to produce and the criteria by which those packets are reviewed. Among the outcomes is increasing difficulty in hiring and retaining faculty, including young clinicians and basic scientists who are drawn to academic institutions because of the opportunity to teach. The Regional Teaching Academy (RTA) of the West Region Consortium of Colleges of Veterinary Medicine initiated an inter-institutional collaboration to address the most important obstacles to recognizing and rewarding teaching in its five member colleges. Working from the medical education literature, the RTA developed an Educator's Promotion Dossier, workshops to train promotion applicants, and an external review process. Initial use has shown that the reviews are efficient and complete. Administrators have expressed strong support for the product, a letter of external review that is returned to a promotion applicant's home institution. The overall result is an evidence-based, structured process by which teaching-intensive faculty can more fully document their achievements in teaching and educational leadership and a more rigorous external review process by which member colleges can assess quality, impact, and scholarly approach.

Variability of residents' ratings of faculty's teaching performance measured by five- and seven-point response scales

BACKGROUND

Medical faculty's teaching performance is often measured using residents' feedback, collected by questionnaires. Researchers extensively studied the psychometric qualities of resulting ratings. However, these studies rarely consider the number of response categories and its consequences for residents' ratings of faculty's teaching performance. We compared the variability of residents' ratings measured by five- and seven-point response scales.

METHODS

This retrospective study used teaching performance data from Dutch anaesthesiology residency training programs. Questionnaires with five- and seven-point response scales from the extensively studied System for Evaluation of Teaching Qualities (SETQ) collected the ratings. We inspected ratings' variability by comparing standard deviations, interquartile ranges, and frequency (percentage) distributions. Relevant statistical tests were used to test differences in frequency distributions and teaching performance scores.

RESULTS

We examined 3379 residents' ratings and 480 aggregated faculty scores. Residents used the additional response categories provided by the seven-point scale - especially those differentiating between positive performances. Residents' ratings and aggregated faculty scores showed a more even distribution on the seven-point scale compared to the five-point scale. Also, the seven-point scale showed a smaller ceiling effect. After rescaling, the mean scores and (most) standard deviations of ratings from both scales were comparable.

CONCLUSIONS

Ratings from the seven-point scale were more evenly distributed and could potentially yield more nuanced, specific and user-friendly feedback. Still, both scales measured (almost) similar teaching performance outcomes. In teaching performance practice, residents and faculty members should discuss whether response scales fit their preferences and goals.

Faculty Feedback Program Evaluation in CIMS Multan, Pakistan

Faculty feedback program (FFP) at CMH Multan Institute of Medical Sciences (CIMS) was conducted for obtaining feedback for basic medical sciences faculty and evaluated to highlight its weaknesses for future improvement. The evaluation design was utilization-focused evaluation (UFE) keeping in mind its two essential elements. First element is the primary intended users (PIU) of the evaluation, namely the college faculty and students which were clearly identified and personally engaged to investigate intended use of the evaluation. Second element required the evaluator to ensure that the intended use of evaluation by PIU guide all other decisions made about the evaluation process. It was a mixed method study (qualitative and quantitative methods both) conducted from August 2018 to August 2019 in CIMS Multan with IRB approval following the steps of UFE. The whole program evaluation was conducted in two parts - first part constituted the 2018 manual FFP evaluation that provided suggestions for a FFP conducted in 2019 online. In step 2 the 2019 online FFP was evaluated again forming basis for future recommendations. Hence the PIUs response was recorded twice in the evaluation cycle - initially after the manual 2018 basic science FFP (response rate: 53%) - after which based on our findings a report was generated and recommendations suggested which were implemented in the 2019 online FFP and response observed again (response rate: 85.7%) to complete the evaluation cycle. Open-end questions were asked from faculty (qualitative analysis) with three themes emerging regarding FFP procedure, questionnaire and timing. An acknowledgement of shift of FFP procedure from manual (2018) to online system (2019) was observed in which faculty praised the ease (72.2%), confidentiality (66.6%), anonymity (50%) and transparency (33.3%) of the online system compared to manual FFP, which was reported to be a rather tense experience (83%). Regarding questionnaire, 38% faculty members reported feedback questions asked from students to be vague and 66.6% claimed that the timing was inappropriate and should have been end of academic year. When asked for suggestions for improvement in 2018 FFP, 72% faculty suggested training students on providing feedback and making the procedure user friendly (83%). Student response regarding both feedback was obtained online by a survey with closed ended questions (quantitative study). Fifty-three percent college students were satisfied with the online FFP giving an average rating of 3.2 to the software user interface and 85% affirmed that using the online software aided in providing anonymous responses helping them provide candid feedback. Seventy-five percent students agreed that online feedback system in 2019 had streamlined the feedback process and made it more efficient compared to the paper-based manual survey of 2018. After evaluating 2019 online FFP, few suggestions were recommended for future FFP including obtaining formative as well as summative faculty feedback, supplementing feedback with teacher’s self-assessment/pen picture and incorporating 360 multi-source feedback.

Students' Perspectives on Their Experience in Clinical Placements: Using a Modified Delphi Methodology to Engage Physiotherapy Stakeholders in Revising the National Form

Purpose: We developed an evidence-informed Student Evaluation of the Clinical Placement form. This form gives students the opportunity to share their feedback and perceptions of their clinical placement experiences and provides meaningful data to all stakeholders. Method: We used a modified Delphi process to engage a sample of national stakeholders: physiotherapy clinical education leads of academic departments, centre coordinators of clinical education, clinical instructors, and students. An expert consultant panel, in addition to the investigators, reviewed the responses from each round and helped develop the questionnaire for the subsequent round and finalize the evaluation form. Results: The response rate was 65.3% (47 of 72) for Round 1, 76.6% (36 of 47) for Round 2, and 100% (36 of 36) for Round 3. After three rounds of questionnaires, 89% of participants thought that the evaluation form met their needs. Conclusions: We developed a revised Student Evaluation of the Clinical Placement form that is informed by the literature and meaningful to all stakeholders. This form is being implemented in physiotherapy university programmes across Canada to enable students to share their experiences at clinical sites.

Assessing Preceptor Use of Cognitive Apprenticeship: Is the Maastricht Clinical Teaching Questionnaire (MCTQ) a Useful Approach?

Construct: The authors investigated the utility of the Maastricht Clinical Teaching Questionnaire (MCTQ) to assess preceptor use of cognitive apprenticeship teaching methods. Background: The MCTQ is an instrument first studied in medical and veterinary education; it was shown to produce valid and reliable data when evaluating how well preceptors engage in cognitive apprenticeship. Additional validity evidence, however, is needed to determine if the MCTQ is useful and applicable in other healthcare professions, such as pharmacy. Approach: Student pharmacists and pharmacy residents at a large academic medical center completed the MCTQ to evaluate pharmacist preceptors. A confirmatory factor analysis (CFA) was used to determine how consistent the data were with the 5-factor structure of the cognitive apprenticeship teaching methods (e.g., modeling, coaching, articulation, exploration, and safe learning environment). A structural equation model (SEM) was used to evaluate the relationship between the 5 factors. Preceptor and student perceptions were also collected to evaluate the response process and potential consequences of using the instrument. Results: Thirty-eight learners (2nd-year and 4th-year student pharmacists and pharmacy residents) submitted 157 evaluations. The CFA showed appreciable fit with the 5-factor structure of the cognitive apprenticeship framework after 2 modifications (removal of 1 item and moving 1 item to a different factor). The SEM had poor model fit compared to previous studies, and an exploratory analysis suggests that there is a complex relationship between the various teaching methods. Preceptors and students highly agreed that the instrument was valuable for providing feedback. Conclusion: The MCTQ has additional validity evidence that supports it as a viable instrument to assess preceptor engagement in teaching methods consistent with the cognitive apprenticeship framework. Additional research is warranted to explore the relationship between components in the cognitive apprenticeship framework to determine how preceptors can best focus their efforts to improve clinical teaching.

Exploring Validity Evidence Associated With Questionnaire-Based Tools for Assessing the Professional Performance of Physicians: A Systematic Review

PURPOSE

To collect and examine-using an argument-based validity approach-validity evidence of questionnaire-based tools used to assess physicians' clinical, teaching, and research performance.

METHOD

In October 2016, the authors conducted a systematic search of the literature seeking articles about questionnaire-based tools for assessing physicians' professional performance published from inception to October 2016. They included studies reporting on the validity evidence of tools used to assess physicians' clinical, teaching, and research performance. Using Kane's validity framework, they conducted data extraction based on four inferences in the validity argument: scoring, generalization, extrapolation, and implications.

RESULTS

They included 46 articles on 15 tools assessing clinical performance and 72 articles on 38 tools assessing teaching performance. They found no studies on research performance tools. Only 12 of the tools (23%) gathered evidence on all four components of Kane's validity argument. Validity evidence focused mostly on generalization and extrapolation inferences. Scoring evidence showed mixed results. Evidence on implications was generally missing.

CONCLUSIONS

Based on the argument-based approach to validity, not all questionnaire-based tools seem to support their intended use. Evidence concerning implications of questionnaire-based tools is mostly lacking, thus weakening the argument to use these tools for formative and, especially, for summative assessments of physicians' clinical and teaching performance. More research on implications is needed to strengthen the argument and to provide support for decisions based on these tools, particularly for high-stakes, summative decisions. To meaningfully assess academic physicians in their tripartite role as doctor, teacher, and researcher, additional assessment tools are needed.

Correlation between students' and trainers' evaluations while learning delegated surgical procedures: A prospective cohort study

BACKGROUND

The delegation of procedures within the medical competence to the nurses can increase the effectiveness of the healthcare provided. The objectives of the study are (1) to assess the quality of training courses for delegated surgical procedures through implementation for graduate scrub nursing ("students") (2) and to evaluate the correlation between the evaluation of this training carried out by students and the self-assessment conducted by the faculty ("trainers").

MATERIALS AND METHODS

We set up a 49-h training for five groups of 10 students from July 2016 to July 2017 in our tertiary academic hospital. The course consisted mostly in simulations based on the "Zwisch" model and focused on acquiring the control of the gesture as well as on the development of critical reasoning. An evaluation of the training by the students but also a self-assessment of trainers were prospectively collected using the SFDP26 questionnaire.

RESULTS

52 active scrub nursing students and 21 trainers were included. 96% of students and 86% of trainers evaluated the training from "good" to "very good". Progress was observed for 41 (79%) of the students and 18 (86%) of the trainers, and 98% of students felt able to put their new skills into clinical practice after training. There was no difference between the total scores of students and teachers (p = 0.153). A statistically significant difference between the evaluations produced by the students and the self-evaluations produced by the trainers was observed for 8 of the 26 items of assessment. In case of inadequacy, the trainers' scores were always lower than those of the students.

CONCLUSIONS

Training in performing delegated surgical procedures by mixed cognitive and motor gestures learning, based on the development of critical thinking and simulations seems to be effective, with a significant improvement in students' knowledge and skills. Expectations of students and trainers are well correlated.

Objective structured teaching examination (OSTE): an underused tool developed to assess clinical teaching skills. A narrative review of the literature

BACKGROUND

There are plenty of options for evaluating medical students and medical residents' clinical skills. Objective structured clinical evaluations (OSCEs) have emerged as a powerful and reliable tool for assessing multiple cognition domains of clinical expertise. In the same way as OSCEs have emerged to assess clinical skills, objective structured teaching evaluations (OSTEs) have come to light as promising and unbiased interventions for evaluating the act of clinical teaching.

DESIGN AND SETTING

Narrative review developed at Universidade Federal de Uberlândia, Brazil.

METHODS

We searched the literature regarding OSTEs using the MEDLINE (via PubMed) and LILACS (viaBiblioteca Virtual em Saude) databases. The SciELO library was also searched for Brazilian papers. Systematic reviews, reviews and randomized controlled trials specifically assessing how OSTEs performed in relation to development of academic staff and medical residents were then selected.

RESULTS

Our search retrieved 178 papers, of which 40 were considered eligible for intensive review. Most of the studies selected reported positive effects from OSTE activities. However, there was little quantitative data to gauge the impact of OSTEs on improvement of teaching skills.

CONCLUSIONS

Considering that OSCEs have become a widely used tool for assessing medical students' and residents' clinical skills, it is high time to incorporate OSTEs for evaluating teaching skills in Brazil. Encouraging data to support implementation of this assessment tool in this country is available from abroad. The net benefit from this would possibly encompass medical students, residents and academic staff, through bringing awareness about the importance of excelling in teaching skills.

Teaching methods used by internal medicine residents on rounds: what works?

BACKGROUND

The specific teaching methods used by internal medicine residents on walk rounds are unknown.

OBJECTIVES

1) To characterize in real time the specific teaching methods used by internal medicine residents on rounds and 2) to identify attributes of successful resident teaching on rounds.

MATERIALS AND METHODS

We conducted a prospective mixed-methods study on inpatient medical services at a single academic medical center from September 2016 to January 2017. Participants were internal medicine residents (postgraduate year [PGY]-1, PGY-2, and PGY-3) and attending physicians. Teachers were PGY-2 and PGY-3 residents, and learners were PGY-1 residents. Residents' teaching on rounds was observed and characterized according to resident demographics, specific teaching methods, and length of time. Participants completed a survey with Likert scale and free-text questions.

RESULTS AND CONCLUSION

Among 136 surveys across 28 separate teaching encounters, we noted that PGY-3 residents' teaching was rated significantly better than PGY-2 residents' teaching. Teaching lasting >1 minute was rated significantly better than teaching lasting <1 minute. Free-text responses emphasized the value of immediate clinical relevance, citing published evidence, conciseness, clarity, and pertinence to the patient. Our findings may help guide internal medicine residents aiming to teach better on rounds and inform further research into specific resident teaching methods.

Validating the modified System for Evaluation of Teaching Qualities: a teaching quality assessment instrument

CONSTRUCT

We assessed the validity of the modified System for Evaluation of Teaching Qualities (mSETQ) in evaluating clinical teachers in Bahrain.

BACKGROUND

Clinical teacher assessment tools are essential for improving teaching quality. The mSETQ is a teaching quality measurement tool, and demonstrating the validity of this tool could provide a stronger evidence base for the utilization of this questionnaire for assessing medical teachers in Bahrain.

APPROACH

This study assessed the construct validity of this questionnaire in medical schools across Bahrain using 400 medical students and 149 clinical teachers. Data were analyzed using confirmatory factor analysis (CFA). The goodness-of-fit index (GFI), comparative fit index (CFI), root mean square residual, and standardized root mean square error of approximation (RMSEA) indices were used to evaluate the model fit. The internal consistency reliability was assessed using Cronbach's alpha.

RESULTS

The results of the CFA revealed an acceptable fit. All criteria for a good model fit were met except for the RMSEA fit index and the standardized root mean square residual (SRMR) value, which was very close to an acceptable value. Good overall reliability was found in the study (α=0.94).

CONCLUSION

The overall findings of this study provided some evidence supporting the reliability and validity of the mSETQ instrument.

Monitoring preceptors' supportive role: A measuring instrument for increased accountability

PURPOSE

This article describes the development of a measuring instrument to monitor support offered by preceptors during their accompaniment of students in clinical facilities.

DESIGN

A quantitative methodological study design was used to develop the instrument.

METHODS

Data were collected by means of a self-completed questionnaire. Total sampling of 192 undergraduate nursing students was done. Descriptive data analysis was conducted regarding the biographical characteristics, Cronbach's alpha was computed to determine the reliability, and an exploratory factor analysis was done to describe the construct validity of the developed instrument.

RESULTS

The Cronbach's alpha of 0.98 indicates high reliability and high internal consistency. Three constructs regarding clinical support, namely cognitive-, emotional- and system support were identified by means of the exploratory factor analysis.

CONCLUSION

The new conceptualisation of support gives insight into the value of the preceptor's role. The instrument designed for this study could be used to assess and monitor the support offered by preceptors while they accompany students in clinical practice.

CLINICAL RELEVANCE

Considering the need to strengthen nursing and midwifery education systems, this instrument contributes to measuring and monitoring clinical accompaniment of students by preceptors.

Medical faculties' view about the importance of educational roles

BACKGROUND

Changes in medical education have drawn attention to student-based learning. It is necessary that teachers' educational roles be redefined to conform to these changes. While numerous educational activities are listed for medical teachers, it seems that they involve themselves in only some of these roles. We conducted this study to assess the importance of different educational roles in the view of medical faculties.

METHODS

A questionnaire that consisted of a total of 12 roles classified into six categories including information provider, role model, facilitator, examiner, planner, and resource developer was prepared. Faculty members were asked to score the importance of each role using a 1-10 scale.

RESULTS

Participants assigned the highest score to "on-the-job role model" (9.47) and the lowest score to "curriculum planner" (8.31) from their own point of view. They also assigned the highest score to "planning or participating in student exams" (8.10) and the lowest score to "learning facilitator" (6.51) from the perspective of the importance of roles in their school's programs.

CONCLUSION

Faculty members are generally familiar with different educational roles but they need to be informed about some of the roles which have gained lower scores in this study.

Exploring the measurement properties of the osteopathy clinical teaching questionnaire using Rasch analysis

Background

Clinical teaching evaluations are common in health profession education programs to ensure students are receiving a quality clinical education experience. Questionnaires students use to evaluate their clinical teachers have been developed in professions such as medicine and nursing. The development of a questionnaire that is specifically for the osteopathy on-campus, student-led clinic environment is warranted. Previous work developed the 30-item Osteopathy Clinical Teaching Questionnaire. The current study utilised Rasch analysis to investigate the construct validity of the Osteopathy Clinical Teaching Questionnaire and provide evidence for the validity argument through fit to the Rasch model.

Methods

Senior osteopathy students at four institutions in Australia, New Zealand and the United Kingdom rated their clinical teachers using the Osteopathy Clinical Teaching Questionnaire. Three hundred and ninety-nine valid responses were received and the data were evaluated for fit to the Rasch model. Reliability estimations (Cronbach's alpha and McDonald's omega) were also evaluated for the final model.

Results

The initial analysis demonstrated the data did not fit the Rasch model. Accordingly, modifications to the questionnaire were made including removing items, removing person responses, and rescoring one item. The final model contained 12 items and fit to the Rasch model was adequate. Support for unidimensionality was demonstrated through both the Principal Components Analysis/t-test, and the Cronbach's alpha and McDonald's omega reliability estimates. Analysis of the questionnaire using McDonald's omega hierarchical supported a general factor (quality of clinical teaching in osteopathy).

Conclusion

The evidence for unidimensionality and the presence of a general factor support the calculation of a total score for the questionnaire as a sufficient statistic. Further work is now required to investigate the reliability of the 12-item Osteopathy Clinical Teaching Questionnaire to provide evidence for the validity argument.

Broadening the Scope of Feedback to Promote Its Relevance to Workplace Learning

The common goal in medical education is to support the health care workforce, both present and future, in becoming and remaining competent professionals. Both during and after medical training, learning takes place in the clinical workplace. Yet, how feedback is defined in medical education and how it is practiced in clinical training situations, combined with a research focus on "what works," limits its potential for learning. This article explores the theoretical background of learning in interaction and current trends in medical education to broaden the scope of feedback and promote its relevance to workplace learning.A new, wider perspective is outlined in which feedback could be redefined as "performance-relevant information" (PRI). PRI can incorporate all information that is deemed relevant to the learner, drawn from interaction in workplace learning and one's interpretation of performance in the clinical workplace. This information can, for example, come from the evaluation of patient outcomes after treatment; observations of role models' performance; evaluations and assessments; exploring feelings of failure or success; and responses of colleagues and peers.PRI draws attention to learning opportunities that better fit the highly social learning of clinical workplaces and current trends in medical education. It supports the interpretation of individual or team performance in terms of relevance to learning. This allows for a comprehensive way of viewing and stimulating workplace learning and the performance of professionals, providing an opportunity to create lifelong learning strategies and potentially improving the care of patients.

Development and validation of the ExPRESS instrument for primary health care providers' evaluation of external supervision

Background: External supervision of primary health care facilities to monitor and improve services is common in low-income countries. Currently there are no tools to measure the quality of support in external supervision in these countries.

Aim: To develop a provider-reported instrument to assess the support delivered through external supervision in Rwanda and other countries.

Methods: “External supervision: Provider Evaluation of Supervisor Support” (ExPRESS) was developed in 18 steps, primarily in Rwanda. Content validity was optimised using systematic search for related instruments, interviews, translations, and relevance assessments by international supervision experts as well as local experts in Nigeria, Kenya, Uganda and Rwanda. Construct validity and reliability were examined in two separate field tests, the first using exploratory factor analysis and a test–retest design, the second for confirmatory factor analysis.

Results: We included 16 items in section A (‘The most recent experience with an external supervisor’), and 13 items in section B (‘The overall experience with external supervisors’). Item-content validity index was acceptable. In field test I, test–retest had acceptable kappa values and exploratory factor analysis suggested relevant factors in sections A and B used for model hypotheses. In field test II, models were tested by confirmatory factor analysis fitting a 4-factor model for section A, and a 3-factor model for section B.

Conclusions: ExPRESS is a promising tool for evaluation of the quality of support of primary health care providers in external supervision of primary health care facilities in resource-constrained settings. ExPRESS may be used as specific feedback to external supervisors to help identify and address gaps in the supervision they provide. Further studies should determine optimal interpretation of scores and the number of respondents needed per supervisor to obtain precise results, as well as test the functionality of section B.

What should we teach the teachers? Identifying the learning priorities of clinical supervisors

Clinicians who teach are essential for the health workforce but require faculty development to improve their educational skills. Curricula for faculty development programs are often based on expert frameworks without consideration of the learning priorities as defined by clinical supervisors themselves. We sought to inform these curricula by highlighting clinical supervisors own requirements through answering the research question: what do clinical supervisors identify as relative strengths and areas for improvement in their teaching practice? This mixed methods study employed a modified version of the Maastricht Clinical Teaching Questionnaire (mMCTQ) which included free-text reflections. Descriptive statistics were calculated and content analysis was conducted on textual comments. 481 (49%) of 978 clinical supervisors submitted their mMCTQs and associated reflections for the research study. Clinical supervisors self-identified relatively strong capability with interpersonal skills or attributes and indicated least capability with assisting learners to explore strengths, weaknesses and learning goals. The qualitative category 'establishing relationships' was the most reported strength with 224 responses. The qualitative category 'feedback' was the most reported area for improvement, with 151 responses. Key areas for curricular focus include: improving feedback practices; stimulating reflective and agentic learning; and managing the logistics of a clinical education environment. Clinical supervisors' self-identified needs provide a foundation for designing engaging and relevant faculty development programs.

Evaluation of clinical teaching quality in competency-based residency training in Lithuania

BACKGROUND AND AIM

In 2013, all residency programs at the Lithuanian University of Health Sciences were renewed into the competency-based medical education curriculum (CBME). In 2015, we implemented the validated EFFECT questionnaire together with the EFFECT-System for quality assessment of clinical teaching in residency training. The aim of this study was to investigate the influence of characteristics of the resident (year of training) and clinical teacher (gender, age, and type of academic position) on teaching quality, as well as to assess areas for teaching quality improvement.

MATERIALS AND METHODS

Residents from 7 different residency study programs filled out 333 EFFECT questionnaires evaluating 146 clinical teachers. We received 143 self-evaluations of clinical teachers using the same questionnaire. Items were scored on a 6-point Likert scale. Main outcome measures were residents' mean overall (MOS), mean subdomain (MSS) and clinical teachers' self-evaluation scores. The overall comparisons of MOS and MSS across study groups and subgroups were done using Student's t test and ANOVA for trend. The intraclass correlation coefficient (ICC) was calculated in order to see how residents' evaluations match with self-evaluations for every particular teacher. To indicate areas for quality improvement items were analyzed subtracting their mean score from the respective (sub)domain score.

RESULTS

MOS for domains of "role modeling", "task allocation", "feedback", "teaching methodology" and "assessment" valued by residents were significantly higher than those valued by teachers (P<0.01). Teachers who filled out self-evaluation questionnaires were rated significantly higher by residents in role modeling subdomains (P<0.05). Male teachers in (sub)domains "role modeling: CanMEDS roles and reflection", "task allocation", "planning" and "personal support" were rated significantly higher than the female teachers (P<0.05). Teachers aged 40 years or younger were rated higher (P<0.01). Residents ratings by type of teachers' academic position almost in all (sub)domains differed significantly (P<0.05). No correlation observed between MOS of a particular teacher and MOS as rated by residents (ICC=0.055, P=0.399). The main areas for improvement were "feedback" and "assessment".

CONCLUSIONS

Resident evaluations of clinical teachers are influenced by teachers' age, gender, year of residency training, type of teachers' academic position and whether or not a clinical teacher performed self-evaluation. Development of CBME should be focused on the continuous evaluation of quality, clinical teachers educational support and the implementation of e-portfolio.

Evaluation in undergraduate medical education: Conceptualizing and validating a novel questionnaire for assessing the quality of bedside teaching

BACKGROUND

Evaluation is an integral part of curriculum development in medical education. Given the peculiarities of bedside teaching, specific evaluation tools for this instructional format are needed. Development of these tools should be informed by appropriate frameworks. The purpose of this study was to develop a specific evaluation tool for bedside teaching based on the Stanford Faculty Development Program's clinical teaching framework.

METHODS

Based on a literature review yielding 47 evaluation items, an 18-item questionnaire was compiled and subsequently completed by undergraduate medical students at two German universities. Reliability and validity were assessed in an exploratory full information item factor analysis (study one) and a confirmatory factor analysis as well as a measurement invariance analysis (study two).

RESULTS

The exploratory analysis involving 824 students revealed a three-factor structure. Reliability estimates of the subscales were satisfactory (α = 0.71-0.84). The model yielded satisfactory fit indices in the confirmatory factor analysis involving 1043 students.

DISCUSSION

The new questionnaire is short and yet based on a widely-used framework for clinical teaching. The analyses presented here indicate good reliability and validity of the instrument. Future research needs to investigate whether feedback generated from this tool helps to improve teaching quality and student learning outcome.

Development of culture-sensitive clinical teacher evaluation sheet in the Japanese context

AIM

Many instruments for evaluating clinical teaching have been developed, albeit most in Western countries. This study aims to develop a validated cultural and local context sensitive instrument for clinical teachers in an East Asian setting (Japan), Japanese Clinical Teacher Evaluation Sheet (JaCTES).

METHODS

A multicenter, cross-sectional evaluation study was conducted. We collected a total of 1368 questionnaires on 304 clinical teachers, completed by residents in 16 teaching hospitals. The construct validity was examined by conducting a factor analysis and using structural equation modeling (SEM). We also assessed the reliability using generalizability analysis and decision study.

RESULTS

Exploratory factor analysis resulted in three-factor (role model, teaching activities, and accessibility) model including 18 items. Confirmatory factor analysis was performed, using SEM. The comparative fit index was 0.931 and the root mean square error of approximation was 0.087, meaning an acceptable goodness of fit for this model. To obtain a reliable dependability-coefficient of at least 0.70 or higher, 5-8 resident responses are necessary.

DISCUSSION AND CONCLUSION

JaCTES is the first reported instrument with validity evidence of content and internal structure and high feasibility in Japan, an East Asian setting. Medical educators should be aware of the local context and cultural aspects in evaluating clinical teachers.

Validation of the EFFECT questionnaire for competence-based clinical teaching in residency training in Lithuania

BACKGROUND AND AIM

In 2013, all residency programs at the Lithuanian University of Health Sciences were renewed into a competency-based medical education curriculum. To assess the quality of clinical teaching in residency training, we chose the EFFECT (evaluation and feedback for effective clinical teaching) questionnaire designed and validated at the Radboud University Medical Centre in the Netherlands. The aim of this study was to validate the EFFECT questionnaire for quality assessment of clinical teaching in residency training.

MATERIALS AND METHODS

The research was conducted as an online survey using the questionnaire containing 58 items in 7 domains. The questionnaire was double-translated into Lithuanian. It was sent to 182 residents of 7 residency programs (anesthesiology reanimathology, cardiology, dermatovenerology, emergency medicine, neurology, obstetrics and gynecology, physical medicine and rehabilitation). Overall, 333 questionnaires about 146 clinical teachers were filled in. To determine the item characteristics and internal consistency (Cronbach's α), the item and reliability analyses were performed. Furthermore, confirmatory factor analysis (CFI) was performed using a model for maximum-likelihood estimation.

RESULTS

Cronbach's α within different domains ranged between 0.91 and 0.97 and was comparable with the original version of the questionnaire. Confirmatory factor analysis demonstrated satisfactory model-fit with CFI of 0.841 and absolute model-fit RMSEA of 0.098.

CONCLUSIONS

The results suggest that the Lithuanian version of the EFFECT maintains its original validity and may serve as a valid instrument for quality assessment of clinical teaching in competency-based residency training in Lithuania.

Analysis of psychometric properties of the modified SETQ tool in undergraduate medical education

BACKGROUND

Effective clinical teaching is crucially important for the future of patient care. Robust clinical training therefore is essential to produce physicians capable of delivering high quality health care. Tools used to evaluate medical faculty teaching qualities should be reliable and valid. This study investigates the psychometric properties of modification of the System for Evaluation of Teaching Qualities (SETQ) instrument in the clinical years of undergraduate medical education.

METHODS

This cross-sectional multicenter study was conducted in four teaching hospitals in the Kingdom of Bahrain. Two-hundred ninety-eight medical students were invited to evaluate 105 clinical teachers using the SETQ instrument between January 2015 and March 2015. Questionnaire feasibility was analyzed using average time required to complete the form and the number of raters required to produce reliable results. Instrument reliability (stability) was assessed by calculating the Cronbach's alpha coefficient for the total scale and for each sub-scale (factor). To provide evidence of construct validity, an exploratory factor analysis was conducted to identify which items on the survey belonged together, which were then grouped as factors.

RESULTS

One-hundred twenty-five medical students completed 1161 evaluations of 105 clinical teachers. The response rates were 42% for student evaluations and 57% for clinical teacher self-evaluations. The factor analysis showed that the questionnaire was composed of six factors, explaining 76.7% of the total variance. Cronbach's alpha was 0.94 or higher for the six factors in the student survey; for the clinical teacher survey, Cronbach's alpha was 0.88. In both instruments, the item-total correlation was above 0.40 for all items within their respective scales.

CONCLUSION

Our modified SETQ questionnaire was found to be both reliable and valid, and was implemented successfully across various departments and specialties in different hospitals in the Kingdom of Bahrain.