Faculty and the observation of trainees' clinical skills: problems and opportunities

Use of SPRAT for peer review of paediatricians in training

OBJECTIVE

To determine whether a multisource feedback questionnaire, SPRAT (Sheffield peer review assessment tool), is a feasible and reliable assessment method to inform the record of in-training assessment for paediatric senior house officers and specialist registrars.

DESIGN

Trainees' clinical performance was evaluated using SPRAT sent to clinical colleagues of their choosing. Responses were analysed to determine variables that affected ratings and their measurement characteristics.

SETTING

Three tertiary hospitals and five secondary hospitals across a UK deanery.

PARTICIPANTS

112 paediatric senior house officers and middle grades.

MAIN OUTCOME MEASURES

95% confidence intervals for mean ratings; linear and hierarchical regression to explore potential biasing factors; time needed for the process per doctor.

RESULTS

20 middle grades and 92 senior house officers were assessed using SPRAT to inform their record of in-training assessment; 921/1120 (82%) of their proposed raters completed a SPRAT form. As a group, specialist registrars (mean 5.22, SD 0.34) scored significantly higher (t = - 4.765) than did senior house officers (mean 4.81, SD 0.35) (P < 0.001). The grade of the doctor accounted for 7.6% of the variation in the mean ratings. The hierarchical regression showed that only 3.4% of the variation in the means could be additionally attributed to three main factors (occupation of rater, length of working relationship, and environment in which the relationship took place) when the doctor's grade was controlled for (significant F change < 0.001). 93 (83%) of the doctors in this study would have needed only four raters to achieve a reliable score if the intent was to determine if they were satisfactory. The mean time taken to complete the questionnaire by a rater was six minutes. Just over an hour of administrative time is needed for each doctor.

CONCLUSIONS

SPRAT seems to be a valid way of assessing large numbers of doctors to support quality assurance procedures for training programmes. The feedback from SPRAT can also be used to inform personal development planning and focus quality improvements.

Structured continuous objective-based assessment of resident's performance at point of care (SCOPA)

The assessment of the clinical performance of physicians-in-training is an important task. The critical care rotation is a mandatory rotation for most residency training programs and is designed to ensure the graduation of trainees who are able to initiate lifesaving management during medical emergencies. Ensuring that each resident fulfills the objectives of the rotation is of paramount importance. Unfortunately, the current assessment methods are subjective and suffer from many threats to validity and reliability that make the assessment inaccurate. In this review, the current assessment method is analyzed, and causes for inaccuracy are identified. A new model for assessment that is continuous, structured, objective-based and at the point of care (SCOPA) is proposed based on the best available assessment methods. Such a model might be useful for the assessment of trainee's performance in critical care as well as non-critical care rotations.

Improving resident performance assessment data: numeric precision and narrative specificity

PURPOSE

To evaluate the use of a systems approach for diagnosing performance assessment problems in surgery residencies, and intervene to improve the numeric precision of global rating scores and the behavioral specificity of narrative comments.

METHOD

Faculty and residents at two surgery programs participated in parallel before- and-after trials. During the baseline year, quality assurance data were gathered and problems were identified. During two subsequent intervention years, an educational specialist at each program intervened with an organizational change strategy to improve information feedback loops. Three quality-assurance measures were analyzed: (1) percentage return rate of forms, (2) generalizability coefficients and 95% confidence intervals of scores, and (3) percentage of forms with behaviorally specific narrative comments.

RESULTS

Median return rates of forms increased significantly from baseline to intervention Year 1 at Site A (71% to 100%) and Site B (75% to 100%), and then remained stable during Year 2. Generalizability coefficients increased between baseline and intervention Year 1 at Site A (0.65 to 0.85) and Site B (0.58 to 0.79), and then remained stable. The 95% confidence interval around resident mean scores improved at Site A from baseline to intervention Year 1 (0.78 to 0.58) and then remained stable; at Site B, it remained constant throughout (0.55 to 0.56). The median percentage of forms with behaviorally specific narrative comments at Site A increased significantly from baseline to intervention Years 1 and 2 (50%, 57%, 82%); at Site B, the percentage increased significantly in intervention Year 1, and then remained constant (50%, 60%, 67%).

CONCLUSIONS

Diagnosing performance assessment system problems and improving information feedback loops improved the quality of resident performance assessment data at both programs.

Promoting fundamental clinical skills: a competency-based college approach at the University of Washington

The focus on fundamental clinical skills in undergraduate medical education has declined over the last several decades. Dramatic growth in the number of faculty involved in teaching and increasing clinical and research commitments have contributed to depersonalization and declining individual attention to students. In contrast to the close teaching and mentoring relationship between faculty and students 50 years ago, today's medical students may interact with hundreds of faculty members without the benefit of a focused program of teaching and evaluating clinical skills to form the core of their four-year curriculum. Bedside teaching has also declined, which may negatively affect clinical skills development. In response to these and other concerns, the University of Washington School of Medicine has created an integrated developmental curriculum that emphasizes bedside teaching and role modeling, focuses on enhancing fundamental clinical skills and professionalism, and implements these goals via a new administrative structure, the College system, which consists of a core of clinical teachers who spend substantial time teaching and mentoring medical students. Each medical student is assigned a faculty mentor within a College for the duration of his or her medical school career. Mentors continuously teach and reflect with students on clinical skills development and professionalism and, during the second year, work intensively with them at the bedside. They also provide an ongoing personal faculty contact. Competency domains and benchmarks define skill areas in which deepening, progressive attention is focused throughout medical school. This educational model places primary focus on the student.

Using standardized patient outcome to measure the effect of teaching asthma-related patient education and information-giving skills to medical students

BACKGROUND

Patient education and giving information is a core skill that improves patient adherence and medical outcomes.

PURPOSE

To evaluate the impact of a teaching intervention on 3rd-year students' competency in patient education and information giving about asthma medication delivery.

METHODS

Students (n=81) completed a 1-hr teaching intervention of didactics followed by role playing of asthma patient education scenarios. Using a standardized patient post intervention, patient education and information-giving skills about spacer/metered dose inhalers were scored overall and on a 12-item checklist and compared to a control group (n=70). Students' knowledge was evaluated using a short answer test.

RESULTS

The performance of intervention students on overall patient education, 10 of the 12 checklist items, and the test was significantly higher than controls but did not approach competency.

CONCLUSIONS

The 1-hr intervention improved clinical performance and knowledge, but students did not become competent. Future studies should investigate how competence in this and other core patient education skills can be successfully achieved.

Face-to-face clinical skills feedback: lessons from the analysis of standardized patient's work

BACKGROUND

Current evidence suggests that trainees are evaluated less stringently when feedback is given face-to-face, limiting its value.

PURPOSE

We expected that standardized patients (SPs) would also be more lenient in scoring when they gave immediate feedback to the students.

METHODS

Data from 6 clinical skills assessments of students in Years 1 to 4 of medical school were used. Comparisons in scoring were made for the cases on which students received feedback and those where they did not. Analysis was done using a hierarchical linear regression model to test for significant differences.

RESULTS

The results indicated no significant difference in scoring for the history and physical exam checklists. There was a small but significant difference on the measure of interpersonal and interviewing skills.

CONCLUSIONS

The SPs were trained to avoid stringency, leniency, and restricted range in scoring students. These training methods may be useful for training faculty to give face-to-face feedback.

Toward meaningful evaluation of clinical competence: the role of direct observation in clerkship ratings

PROBLEM STATEMENT AND PURPOSE

The lack of direct observation by faculty may affect meaningful judgments of clinical competence. The purpose of this study was to explore the influence of direct observation on reliability and validity evidence for family medicine clerkship ratings of clinical performance.

METHOD

Preceptors rating family medicine clerks (n = 172) on a 16-item evaluation instrument noted the data-source for each rating: note review, case discussion, and/or direct observation. Mean data-source scores were computed and categorized as low, medium or high, with the high-score group including the most direct observation. Analyses examined the influence of data-source on interrater agreement and associations between clerkship clinical scores (CCS) and scores from the National Board of Medical Examiners (NBME(R)) subject examination as well as a fourth-year standardized patient-based clinical competence examination (M4CCE).

RESULTS

Interrater reliability increased as a function of data-source; for the low, medium, and high groups, intraclass correlation coefficients were.29,.50, and.74, respectively. For the high-score group, there were significant positive correlations between CCS and NBME score (r =.311, p =.054); and between CCS and M4CCE (r =.423, p =.009).

CONCLUSION

Reliability and validity evidence for clinical competence is enhanced when more direct observation is included as a basis for clerkship ratings.

Medical Students' and Residents' preferred site characteristics and preceptor behaviours for learning in the ambulatory setting: a cross-sectional survey

BACKGROUND

Medical training is increasingly occurring in the ambulatory setting for final year medical students and residents. This study looks to identify if gender, school, level of training, or specialty affects learner's (final year medical students and residents) preferred site characteristics and preceptor behaviours for learning in the ambulatory setting.

METHODS

All final year medical students and residents at the five medical schools in Ontario (N = 3471) were surveyed about the site characteristics and preceptor behaviours most enhancing their learning in the ambulatory setting. Preferred site characteristics and preceptor behaviours were rank ordered. Factor analysis grouped the site characteristics and preceptor behaviours into themes which were then correlated with gender, school, level of training, and specialty.

RESULTS

Having an adequate number and variety of patients while being supervised by enthusiastic preceptors who give feedback and are willing to discuss their reasoning processes and delegate responsibility are site characteristics and preceptor behaviours valued by almost all learners. Some teaching strategies recently suggested to improve efficiency in the ambulatory teaching setting, such as structuring the interview for the student and teaching and reviewing the case in front of the patient, were found not to be valued by learners. There was a striking degree of similarity in what was valued by all learners but there were also some educationally significant differences, particularly between learners at different levels and in different specialties. Key findings between the different levels include preceptor interaction being most important for medical students as opposed to residents who most value issues pertaining to patient logistics. Learning resources are less valued early and late in training. Teaching and having the case reviewed in front of the patient becomes increasingly less valued as learners advance in their training. As one approaches the end of ones' training office management instruction becomes increasingly valued. Differences between specialties pertain most to the type of practice residents will ultimately end up in (ie: office based specialties particularly valuing instruction in office management and health care system interaction).

CONCLUSIONS

Preceptors need to be aware of, and make efforts to provide, teaching strategies such as feedback and discussing clinical reasoning, that learners have identified as being helpful for learning. If strategies identified as not being valued for learning, such as teaching in front of the patient, must continue it will be important to explore the barriers they present to learning. Although what all learners want from their preceptors and clinic settings to enhance their learning is remarkably similar, being aware of the educationally significant differences, particularly for learners at different levels and in different specialties, will enhance teaching in the ambulatory setting.