Initial use of a novel instrument to measure professionalism in surgical residents

BACKGROUND

No universally accepted method to measure professionalism exists. We developed an instrument to measure specific aspects of professionalism in surgical residents.

METHODS

Professionalism was deconstructed into 15 domains. Behavioral descriptors were determined for extreme and selected intermediate anchors. It became evident that residents could "go too far" in some professional behaviors. Therefore, although a 7-point continuous ordinal scale forms the framework, a score of 7 does not necessarily indicate the ideal. This characteristic minimizes the problem of inflated ratings.

RESULTS

The instrument was utilized by attending faculty to evaluate residents and also by residents as a self-evaluation. Calculated from ordinal values, mean (SD) of the ratings across domains for faculty evaluations of residents was 4.95 (0.38) while mean for self-evaluations was 4.95 (0.39). Reliability was high (coefficient alpha 0.85).

CONCLUSIONS

This instrument provides a means to measure professional behaviors during surgical residency. Repeated use will be required to thoroughly establish validity and reliability.

Ensuring global standards for medical graduates: a pilot study of international standard-setting

Increasing physician and patient mobility has led to a move toward internationalization of standards for physician competence. The Institute for International Medical Education proposed a set of outcome-based standards for student performance, which were then measured using three assessment tools in eight leading schools in China: a 150-item multiple-choice examination, a 15-station OSCE and a 16-item faculty observation form. The purpose of this study was to empanel a group of experts to determine whether international student-level performance standards could be set. The IIME convened an international panel of experts in student education with specialty and geographic diversity. The group was split into two, with each sub-group establishing standards independently. After a discussion of the borderline student, the sub-groups established minimally acceptable cut-off scores for performance on the multiple-choice examination (Angoff and Hofstee methods), the OSCE station and global rating performance (modified Angoff method and holistic criterion reference), and faculty observation domains (holistic criterion reference). Panelists within each group set very similar standards for performance. In addition, the two independent parallel panels generated nearly identical performance standards. Cut-off scores changed little before and after being shown pilot data but standard deviations diminished. International experts agreed on a minimum set of competences for medical student performance. In addition, they were able to set consistent performance standards with multiple examination types. This provides an initial basis against which to compare physician performance internationally.

The prediction of professional behaviour

OBJECTIVE

The purpose of this study was to establish outcome measures for professionalism in medical students and to identify predictors of these outcomes.

DESIGN

Retrospective cohort study.

SETTING

A US medical school.

PARTICIPANTS

All students entering in 1995 and graduating within 5 years.

MEASURES

Outcome measures included review board identification of professionalism problems and clerkship evaluations for items pertaining to professionalism. Pre-clinical predictor variables included material from the admissions application, completion of required course evaluations, students' self-reporting of immunisation compliance, students' performance on standardised patient (SP) exercises, and students' self-assessed performance on SP exercises.

RESULTS

The outcome measures of clerkship professionalism scores were found to be highly reliable (alpha 0.88-0.96). No data from the admissions material was found to be predictive of professional behaviour in the clinical years. Using multivariate regression, failing to complete required course evaluations (B = 0.23) and failing to report immunisation compliance (B = 0.29) were significant predictors of unprofessional behaviour found by the review board in subsequent years. Immunisation non-compliance predicted low overall clerkship professional evaluation scores (B = - 0.34). Student self-assessment accuracy (SP score minus self-assessed score) (B = 0.03) and immunisation non-compliance (B = 0.54) predicted the internal medicine clerkship professionalism score.

CONCLUSIONS

This study identifies a set of reliable, context-bound outcome measures in professionalism. Although we searched for predictors of behaviour in the admissions application and other domains commonly felt to be predictive of professionalism, we found significant predictors only in domains where students had had opportunities to demonstrate conscientious behaviour or humility in self-assessment.

Bedside interactions from the other side of the bedrail

OBJECTIVE

To assess the importance to patients of various aspects of bedside interactions with physician teams.

DESIGN

Cross-sectional survey.

SETTING

VA hospital.

PATIENTS

Ninety-seven medical inpatients.

INTERVENTION

Survey of 44 questions including short answer, multiple choice, and Likert-type questions.

MEASUREMENTS AND MAIN RESULTS

Data analysis included descriptive statistics. The sample was predominantly male, with a mean age of 62. Overall satisfaction with the hospital experience and with the team of doctors were both high (95% and 96% reported being very or mostly satisfied, respectively). Patients reported learning about several issues during their interactions with the teams; the 3 most highly rated areas were new problems, tests that will be done, and treatments that will be done. Most patients (76%) felt that their teams cared about them very much. Patients were made comfortable when the team showed that they cared, listened, and appeared relaxed (reported by 63%, 57%, and 54%, respectively). Patients were made uncomfortable by the team using language they did not understand (22%) and when several people examined them at once (13%). Many (58%) patients felt personally involved in teaching. The majority of patients liked having medical students and residents involved in their care (69% and 64%, respectively).

CONCLUSIONS

Patients have much to teach about what is important about interacting with physician teams. Although patients' reactions to team interactions are generally positive, patients are different with respect to what makes them comfortable and uncomfortable. Taking their preferences into account could improve the experience of being in a teaching hospital.

Trading Health Services Across Borders: GATS, Markets, And Caveats

The General Agreement on Trade in Services (GATS) is the result of an ongoing process of opening national markets to foreign health services within an international framework of trade agreements that prohibit discriminatory treatment of foreign suppliers. Developing markets are growing, as is demand for health care services, and most of this demand is being met by the private market. The globalization of health services requires the resources of the academic and corporate sectors of the developed world for equitable and sustainable growth. Health services trade should be seen as a tool for achieving these goals, rather than as an end in itself.

A content analysis of e-mail communication between patients and their providers: patients get the message

OBJECTIVE

E-mail use in the clinical setting has been slow to diffuse for several reasons, including providers' concerns about patients' inappropriate and inefficient use of the technology. This study examined the content of a random sample of patient-physician e-mail messages to determine the validity of those concerns.

DESIGN

A qualitative analysis of patient-physician e-mail messages was performed.

MEASUREMENTS

A total of 3,007 patient-physician e-mail messages were collected over 11 months as part of a randomized, controlled trial of a triage-based e-mail system in two primary care centers (including 98 physicians); 10% of messages were randomly selected for review. Messages were coded across such domains as message type, number of requests per e-mail, inclusion of sensitive content, necessity of a physician response, and message tone.

RESULTS

The majority (82.8%) of messages addressed a single issue. The most common message types included information updates to the physicians (41.4%), prescription renewals (24.2%), health questions (13.2%), questions about test results (10.9%), referrals (8.8%), "other" (including thank yous, apologies) (8.8%), appointments (5.4%), requests for non-health-related information (4.8%), and billing questions (0.3%). Overall, messages were concise, formal, and medically relevant. Very few (5.1%) included sensitive content, and none included urgent messages. Less than half (43.2%) required a physician response.

CONCLUSION

A triage-based e-mail system promoted e-mail exchanges appropriate for primary care. Most patients adhered to guidelines aimed at focusing content, limiting the number of requests per message, and avoiding urgent requests or highly sensitive content. Thus, physicians' concerns about the content of patients' e-mails may be unwarranted.

The physical examination of patients with abdominal pain: the long-term effect of adding standardized patients and small-group feedback to a lecture presentation

BACKGROUND

One of the most effective methods for teaching physical diagnosis may be standardized patient instructors.

PURPOSE

To determine if a lecture plus standardized patient instructors with small-group sessions is more effective than a lecture alone for teaching the evaluation of patients with abdominal pain.

METHODS

Control (class of 2001) and intervention (class of 2002) groups both attended a lecture on the abdominal examination. The intervention group then underwent an exercise with standardized patient instructors and a review session with surgical faculty. An evaluation 18 months later used standardized patient instructors to complete evaluations assessing history-taking and physical examination skills.

RESULTS

The intervention group performed significantly better than the control group on both the history and the physical examination subscales.

CONCLUSION

It is possible to have an important, measurable, and lasting effect on physical examination skills by adding standardized patient instructors and small-group discussion to a lecture presentation.

The assessment of Global Minimum Essential Requirements in medical education

Using an international network of experts in medical education, the Institute for International Medical Education (IIME) developed the Global Minimum Essential Requirements (GMER) as a set of competence-based outcomes for graduating students. To establish a set of tools to evaluate these competences, the IIME then convened a Task Force of international experts on assessment that reviewed the GMER. After screening 75 potential assessment tools, they identified three that could be used most effectively. Of the 60 competences envisaged in the GMER, 36 can be assessed using a 150-item multiple-choice question (MCQ) examination, 15 by using a 15-station objective structured clinical examination (OSCE), and 17 by using a 15-item faculty observation form. In cooperation with eight leading medical schools in China, the MCQ, OSCE and Faculty Observation Form were developed to be used in an assessment program that is scheduled to be given to all seven-year students in October 2003.

Expanding the boundaries of medical education: evidence for cross-cultural exchanges

PROBLEM STATEMENT AND BACKGROUND

Cross-cultural experiences are in increasing demand by both graduate and undergraduate medical students, yet the benefits of these experiences are not clearly established.

METHOD

A review of the literature was conducted to identify articles on the outcomes of cross-cultural experiences. Themes were identified and categorized into domains.

RESULTS

Forty-two studies were found; 27 articles used qualitative methods, nine used quantitative methods, and six used both. Most (24) were from the nursing literature, 18 were from the medical literature. All studies reported positive outcomes along four domains: students' professional development, students' personal development, medical school benefits, and host population benefits.

CONCLUSIONS

Studies reviewed were primarily case controlled or case series. Future research is needed that more clearly defines outcome measures and uses more rigorous methods. Although results suggest positive outcomes in all domains, additional research is needed before cross-cultural rotations can be supported based on evidence.

Effect of a triage-based E-mail system on clinic resource use and patient and physician satisfaction in primary care: a randomized controlled trial

OBJECTIVES

E-mail communication between patients and their providers has diffused slowly in clinical practice. To address concerns about the use of this technology, we performed a randomized controlled trial of a triage-based e-mail system in primary care. DESIGN AND PATIENTS/PARTICIPANTS: Physicians in 2 university-affiliated primary care centers were randomized to a triage-based e-mail system promoted to their patients. E-mails from patients of intervention physicians were routed to a central account and parsed to the appropriate staff for response. Control group physicians and their patients did not have access to the system. We collected information on patient e-mail use, phone calls, and visit distribution by physician over the 10 months and performed physician and patient surveys to examine attitudes about communication.

RESULTS

E-mail volume was greater for intervention versus control physicians (46 weekly e-mails per 100 scheduled visits vs 9 in the control group at the study midpoint; P <.01) but there were no between-group differences in phone volume (67 weekly phone calls per 100 scheduled visits vs 55 in the control group; P =.45) or rates of patient no-shows (5% in both groups; P =.77). Intervention physicians reported more favorable attitudes toward electronic communication than did control physicians but there were no differences in attitudes toward patient or staff communication in general. There were few between-group differences in patient attitudes toward electronic communication or communication in general.

CONCLUSIONS

E-mail generated through a triage-based system did not appear to substitute for phone communication or to reduce visit no-shows in a primary care setting. Physicians' attitudes toward electronic communication were improved, but physicians' and patients' attitudes toward general communication did not change. Growth of e-mail communication in primary care settings may not improve the efficiency of clinical care.

Bridging the electronic divide: patient and provider perspectives on e-mail communication in primary care

OBJECTIVE

To determine e-mail utilization patterns and attitudes toward e-mail use among primary care physicians and their ambulatory outpatient clinic patients.

STUDY DESIGN

Cross-sectional baseline survey.

PARTICIPANTS AND METHODS

Participants included 476 consecutive outpatient clinic patients, 126 general medical and family practice physicians, and 16 clinical and office staff from 2 large primary care centers within an academic teaching system. They completed a survey about e-mail usage patterns and their attitudes toward using e-mail for patient-provider communication.

RESULTS

More than half of patients (52.1%) were self-defined e-mail users, yet only 10.5% of those users had ever used e-mail to communicate with their doctors. Seventy percent of all patients said they would be willing to use e-mail to communicate with their doctors. Overall, patients were concerned about logistics, such as whether the message would get to the right person and how long it would take to get a response. Physicians and staff were more optimistic than patients about the potential for e-mail to improve the doctor-patient relationship. Patient e-mail users, patient e-mail nonusers, physicians, and staff reported low levels of concern about the security and privacy of e-mail.

CONCLUSIONS

Patient-provider e-mail may diffuse slowly into the primary care clinical practice setting because of patient concerns about efficiency and effectiveness and whether e-mail use will improve their relationship with providers. Managed care organizations that plan to build e-mail and Web-based patient portals will need to promote these technologies in a way that educates both patient and providers about their appropriate use.

Using a multimedia tool to improve cardiac auscultation knowledge and skills

OBJECTIVE

Today's medical school graduates have significant deficits in physical examination skills. Medical educators have been searching for methods to effectively teach and maintain these skills in students. The objective of this study was to determine if an auscultation curriculum centered on a portable multimedia CD-ROM was effective in producing and maintaining significant gains in cardiac auscultatory skills.

DESIGN

Controlled cohort study.

PARTICIPANTS

All 168 third-year medical students at 1 medical school in an academic medical center.

INTERVENTIONS

Students were tested before and after exposure to 1 or more elements of the auscultation curriculum: teaching on ward/clinic rotations, CD-ROM comprehensive cases with follow-up seminars, and a CD-ROM 20-case miniseries. The primary outcome measures were student performance on a 10-item test of auscultation skill (listening and identifying heart sound characteristics) and a 30-item test of auscultation knowledge (factual questions about auscultation). A subset of students was tested for attenuation effects 9 or 12 months after the intervention.

RESULTS

Compared with the control group (1 month clinical rotation alone), students who were also exposed to the CD-ROM 20-case miniseries had significant improvements in auscultation skills scores (P < .05), but not knowledge. Additional months of clerkship, comprehensive CD-ROM cases, and follow-up seminars increased auscultation knowledge beyond the miniseries alone (P < .05), but did not further improve auscultation skills. Students' auscultation knowledge diminished one year after the intervention, but auscultation skills did not.

CONCLUSION

In addition to the standard curriculum of ward and conference teaching, portable multimedia tools may help improve quality of physical examination skills.

Teaching the one-minute preceptor. A randomized controlled trial

OBJECTIVE

The One-Minute Preceptor (OMP) model of faculty development is used widely to improve teaching, but its effect on teaching behavior has not been assessed. We aim to evaluate the effect of this intervention on residents' teaching skills.

DESIGN

Randomized controlled trial.

SETTING

Inpatient teaching services at both a tertiary care hospital and a Veterans Administration Medical Center affiliated with a University Medical Center.

PARTICIPANTS

Participants included 57 second- and third-year internal medicine residents that were randomized to the intervention group (n = 28) or to the control group (n = 29).

INTERVENTION

The intervention was a 1-hour session incorporating lecture, group discussion, and role-play.

MEASUREMENTS AND MAIN RESULTS

Primary outcome measures were resident self-report and learner ratings of resident performance of the OMP teaching behaviors. Residents assigned to the intervention group reported statistically significant changes in all behaviors (P <.05). Eighty-seven percent of residents rated the intervention as "useful or very useful" on a 1-5 point scale with a mean of 4.28. Student ratings of teacher performance showed improvements in all skills except "Teaching General Rules." Learners of the residents in the intervention group reported increased motivation to do outside reading when compared to learners of the control residents. Ratings of overall teaching effectiveness were not significantly different between the 2 groups.

CONCLUSIONS

The OMP model is a brief and easy-to-administer intervention that provides modest improvements in residents' teaching skills.

Comparison of resident and medical student evaluation of faculty teaching

Recognizing and rewarding teaching faculty are increasingly important to medical schools and are often hampered by low perceived reliability and validity of measures of teaching ability. The purpose of this study was to cross-validate two independently generated measures of teaching from medical students and residents. A total of 2,318 medical student and 4,425 resident scores for single-item measures of teaching ability for 129 teaching faculty members of a department of internal medicine over a 6-year period were compared. Results showed that average teaching scores were higher for medical students than residents. Rank order of faculty were within 2 quintiles for the two groups for over 90% of faculty. Highly discordant evaluations were seen for only 8% of faculty. The authors conclude the general concordance of two independent measures of teaching ability adds evidence to the existing literature of the validity of single-item measures of teaching ability from two different types of learners.

Is there a relationship between attending physicians' and residents' teaching skills and students' examination scores?

PURPOSE

Faculty development programs and faculty incentive systems have heightened the need to validate a connection between the quality of teaching and students' learning. This study was designed to determine the association between attending physicians' and residents' teacher ratings and their students' examination scores.

METHOD

From a database of 362 students, 138 faculty, and 107 residents in internal medicine, student-faculty (n = 476) and student-resident (n = 474) pairs were identified. All students were in their third year, rotating on inpatient general medicine and cardiology services, July 1994 through June 1996, at a single institution. The outcome measure for students' knowledge was the NBME Subject Examination in internal medicine. To control for students' baseline knowledge, the predictors were scores on the USMLE Step 1 and a sequential examination (a clinically-based pre- and post-clerkship examination). Teaching abilities of faculty and residents were rated by a global item on the post-clerkship evaluation. Faculty's ratings used only scores from prior to the study period; residents' ratings included those scores students gave during the study period.

RESULTS

Multivariate analyses showed faculty's teaching ratings were a small but significant predictor of the increase in students' knowledge. Residents' teaching ratings did not predict an increase in students' knowledge.

CONCLUSION

Attending faculty's clinical teaching ability has a positive and significant effect on medical students' learning.