Measuring teaching effectiveness in a pre-clinical multi-instructor course: a case study in the development and application of a brief instructor rating scale

Rapid Feedback: Assessing Pre-clinical Teaching in the Era of Online Learning

INTRODUCTION

Medical schools vary in their approach to providing feedback to faculty. The purpose of this study was to test the effects of rapid student feedback in a course utilizing novel virtual learning methods.

METHODS

Second-year medical students were supplied with an optional, short questionnaire at the end of each class session and asked to provide feedback within 48 h. At the close of each survey, results were emailed to faculty. After the course, students and faculty were asked to rate the effectiveness of this method. This study did not affect administration of the usual end-of-course summative evaluations.

RESULTS

Ninety-one percent of students who participated noted increased engagement in the feedback process, but only 18% on average chose to participate. Faculty rated rapid feedback as more actionable than summative feedback (67%), 50% rated it as more specific, and 42% rated it as more helpful. Some wrote that comments were too granular, and others noted a negative personal emotional response.

CONCLUSION

Rapid feedback engaged students, provided actionable feedback, and increased communication between students and instructors, suggesting that this approach added value. Care must be taken to reduce the student burden and support relational aspects of the process.

Student evaluations of teaching and the development of a comprehensive measure of teaching effectiveness for medical schools

The evaluation of courses and faculty is of vital importance in all higher education institutions including medical schools. Student Evaluations of Teaching (SETs) commonly take the form of completion of anonymous questionnaires and even though they were originally developed to evaluate courses and programmes, throughout the years they have also been used to measure teaching effectiveness and subsequently to guide important decisions related to the faculty's career progression. Nevertheless, certain factors and biases may influence SET rates and may not measure teaching effectiveness objectively. Although the literature on course and faculty evaluations is well-researched in general higher education, there are concerns with regards to the use of the same tools for evaluation of courses and teachers in medical programmes. Specifically, the SETs in general higher education cannot be directly applied to the structure of courses and delivery of curriculum in medical schools. This review provides an overview of how SETs can be improved at the levels of instrumentation, administration and interpretation. In addition, the paper supports that through the collection and triangulation of data from multiple sources, including students, peers, program administrators and self-awareness via the use of different methods such as peer reviews, focus groups and self-evaluations, it will be possible to develop a comprehensive evaluation system that will present an effective measure of teaching effectiveness, will support the professional development of medical teachers and will improve the quality of teaching in medical education.

Individual class evaluation and effective teaching characteristics in integrated curricula

BACKGROUND

In an integrated curriculum, multiple instructors take part in a course in the form of team teaching. Accordingly, medical schools strive to manage each course run by numerous instructors. As part of the curriculum management, course evaluation is conducted, but a single, retrospective course evaluation does not comprehensively capture student perception of classes by different instructors. This study aimed to demonstrate the need for individual class evaluation, and further to identify teaching characteristics that instructors need to keep in mind when preparing classes.

METHODS

From 2014 to 2015, students at one medical school left comments on evaluation forms after each class. Courses were also assessed after each course. Their comments were categorized by connotation (positive or negative) and by subject. Within each subject category, test scores were compared between positively and negatively mentioned classes. The Mann-Whitney U test was performed to test group differences in scores. The same method was applied to the course evaluation data.

RESULTS

Test results for course evaluation showed group difference only in the practice/participation category. However, test results for individual class evaluation showed group differences in six categories: difficulty, main points, attitude, media/contents, interest, and materials. That is, the test scores of classes positively mentioned in six domains were significantly higher than those of negatively mentioned classes.

CONCLUSIONS

It was proved that individual class evaluation is needed to manage multi-instructor courses in integrated curricula of medical schools. Based on the students' extensive feedback, we identified teaching characteristics statistically related to academic achievement. School authorities can utilize these findings to encourage instructors to develop effective teaching characteristics in class preparation.

Peer review of teaching at the Budapest University of Technology and Economics Faculty of Economic and Social Sciences

Purpose

This paper aims to describe an internal quality enhancement system based on peer reviewing and summarizes the first results of application at the Budapest University of Technology and Economics Faculty of Economic and Social Sciences.

Design/methodology/approach

A peer review framework has been developed to evaluate and further develop the teaching programs and practices. The questionnaire-based peer review program included 22 courses and involved almost 100 lecturers. Peer review outcomes are completed by end-of-semester student course evaluations.

Findings

The results allow us to map differences between lecturers and courses and to identify correlations between the assessment criteria applied for peer reviewing.

Practical implications

The implemented framework implies individual, faculty and organizational development to enhance a deeper understanding of how to create quality in teaching programs and processes. Secondly, the peer review program contributes to the establishment of a learning community with a growing common understanding of what is considered good quality in business education.

Originality/value

The paper is valuable as a guide to faculty management wishing to implement a peer review framework within their own institution. The novelty of the presented approach is that it focuses on a semester-long teaching performance including classroom performance, course outlines, teaching materials, course requirements and processes and means of student performance assessments.

Evaluation of large-group lectures in medicine - development of the SETMED-L (Student Evaluation of Teaching in MEDical Lectures) questionnaire

BACKGROUND

The seven categories of the Stanford Faculty Development Program (SFDP) represent a framework for planning and assessing medical teaching. Nevertheless, so far there is no specific evaluation tool for large-group lectures that is based on these categories. This paper reports the development and psychometric validation of a short German evaluation tool for large-group lectures in medical education (SETMED-L: 'Student Evaluation of Teaching in MEDical Lectures') based on the SFDP-categories.

METHODS

Data were collected at two German medical schools. In Study 1, a full information factor analysis of the new 14-item questionnaire was performed. In Study 2, following cognitive debriefings and adjustments, a confirmatory factor analysis was performed. The model was tested for invariance across medical schools and student gender. Convergent validity was assessed by comparison with results of the FEVOR questionnaire.

RESULTS

Study 1 (n = 922) yielded a three-factor solution with one major (10 items) and two minor factors (2 items each). In Study 2 (n = 2740), this factor structure was confirmed. Scale reliability ranged between α = 0.71 and α = 0.88. Measurement invariance was given across student gender but not across medical schools. Convergent validity in the subsample tested (n = 246) yielded acceptable results.

CONCLUSION

The SETMED-L showed satisfactory to very good psychometric characteristics. The main advantages are its short yet comprehensive form, the integration of SFDP-categories and its focus on medical education.

Potential bias factors that affect the course evaluation of students in preclinical courses

PURPOSE

We aim to identify what potential bias factors affected students' overall course evaluation, and to observe what factors should be considered in the curriculum evaluation system of medical schools.

METHODS

This study analyzed students' ratings of preclinical instructions at the Ajou University School of Medicine. The ratings of instructions involved 41 first-year and 45 second-year medical students.

RESULTS

There was a statistically significant difference between years of study and ratings' scoring. Learning difficulty, learning amount, student assessment, and teacher preparation from second-year students were significantly higher than first-year students (p<0.05). The analysis results revealed that student assessment was the predictor of ratings from first-year students, while teacher preparation was the predictor of ratings from second-year students.

CONCLUSION

We found significant interactions between year of study and the students' rating results. We were able to confirm that satisfaction of instructions factors perceived by medical students were different for the characteristics of courses. Our results may be an important resource for evaluating preclinical curriculums.

How Many Responses Do We Need? Using Generalizability Analysis to Estimate Minimum Necessary Response Rates for Online Student Evaluations

CONSTRUCT: The study compares paper and online ratings of instructional units and analyses, with the G-study using the symmetry principle, the response rates needed to ensure acceptable precision of the measure when compliance is low.

BACKGROUND

Students' ratings of teaching contribute to the quality of medical training programs. To date, many schools have replaced pen-and-paper questionnaires with electronic forms, despite the lower response rates consistently reported with the latter. Few available studies have examined the effects of low response rates on the reliability and precision of the evaluation measure. Moreover, the minimum number of raters to target when response rates are low remains unclear.

APPROACH

Descriptive data were derived from 799 students' paper and online ratings of 11 preclinical instructional units (PIUs). Reliability was assessed by Cronbach's alpha coefficients. The generalizability method applying the symmetry principle approach was used to analyze the precision of the measure with a reference standard error of mean (SEM) set at 0.10; optimization models were built to estimate minimum response rates.

RESULTS

Overall, response rates were 74% and 30% (p < .001) and PIUs ratings were 3.8 ± 0.5 and 3.6 ± 0.5 (p = .02), respectively in paper and online questionnaires. Higher SEM levels and significantly larger 95% confidence intervals of PIUs rating scores were observed with online evaluations. To keep the SEM within preset limits of precision, a minimum of 48% response rate was estimated for online formats.

CONCLUSIONS

The proposed generalizability analysis allowed estimating the minimum response needed to maintain acceptable precision in online evaluations. The effects of response rates on accuracy are discussed.

The development of an instrument to assess clinical teaching with linkage to CanMEDS roles: A psychometric analysis

BACKGROUND

Assessment of clinical teaching by learners is of value to teachers, department heads, and program directors, and must be comprehensive and feasible.

AIMS

To review published evaluation instruments with psychometric evaluations and to develop and psychometrically evaluate an instrument for assessing clinical teaching with linkages to the CanMEDS roles.

METHOD

We developed a 19-item questionnaire to reflect 10 domains relevant to teaching and the CanMEDS roles. A total of 317 medical learners assessed 170 instructors. Fourteen (4.4 %) clinical clerks, 229 (72.3%) residents, and 53 (16.7%) fellows assessed 170 instructors. Twenty-one (6.6%) did not specify their position.

RESULTS

A mean number of eight raters assessed each instructor. The internal consistency reliability of the 19-item instrument was Cronbach's α = 0.95. The generalizability coefficient (Ep(2)) analysis indicated that the raters achieved Ep(2) of 0.95. The factor analysis showed three factors that accounted for 67.97% of the total variance. The three factors together, with the variance accounted for and their internal consistency reliability, are teaching skills (variance = 53.25s%; Cronbach's α = 0.92), Patient interaction (variance = 8.56%; Cronbach's α = 0.91), and professionalism (variance = 6.16%; Cronbach's α = 0.86). The three factors are intercorrelated (correlations = 0.48, 0.58, 0.46; p < 0.01).

CONCLUSION

It is feasible to assess clinical teaching with the 19-item instrument that has demonstrated evidence of both validity and reliability.

Use of critical incidents to develop a rating form for resident evaluation of faculty teaching

CONTEXT

Monitoring the teaching effectiveness of attending physicians is important to enhancing the quality of graduate medical education.

METHODS

We used a critical incident technique with 35 residents representing a cross-section of programmes in a teaching hospital to develop a 23-item rating form. We obtained ratings of 11 attending physicians in internal medicine and general surgery from 54 residents. We performed linear and logistic regression analysis to relate the items on the form to the residents' overall ratings of the attending physicians and the programme directors' ratings of the attending physicians.

RESULTS

The residents rated the attending physicians highly in most areas, but lower in provision of feedback, clarity of written communication and cost-effectiveness in making clinical decisions. When we used the residents' overall ratings as the criterion, the most important aspects of attending physicians' teaching were clarity of written communication, cost-effectiveness, commitment of time and energy and whether the resident would refer a family member or friend to the physician. When we used the programme directors' ratings as the criterion, the additional important aspects of performance were concern for the residents' professional well-being, knowledge of the literature and the delivery of clear verbal and written communication.

CONCLUSIONS

The critical incident technique can be used to develop an instrument that demonstrates content and construct validity. We found that residents consider commitment of time to teaching and clinical effectiveness to be the most important dimensions of faculty teaching. Other important dimensions include written and verbal communication, cost-effectiveness and concern for residents' professional development.