201
|
Abstract
OBJECTIVE To determine whether a multisource feedback questionnaire, SPRAT (Sheffield peer review assessment tool), is a feasible and reliable assessment method to inform the record of in-training assessment for paediatric senior house officers and specialist registrars. DESIGN Trainees' clinical performance was evaluated using SPRAT sent to clinical colleagues of their choosing. Responses were analysed to determine variables that affected ratings and their measurement characteristics. SETTING Three tertiary hospitals and five secondary hospitals across a UK deanery. PARTICIPANTS 112 paediatric senior house officers and middle grades. MAIN OUTCOME MEASURES 95% confidence intervals for mean ratings; linear and hierarchical regression to explore potential biasing factors; time needed for the process per doctor. RESULTS 20 middle grades and 92 senior house officers were assessed using SPRAT to inform their record of in-training assessment; 921/1120 (82%) of their proposed raters completed a SPRAT form. As a group, specialist registrars (mean 5.22, SD 0.34) scored significantly higher (t = - 4.765) than did senior house officers (mean 4.81, SD 0.35) (P < 0.001). The grade of the doctor accounted for 7.6% of the variation in the mean ratings. The hierarchical regression showed that only 3.4% of the variation in the means could be additionally attributed to three main factors (occupation of rater, length of working relationship, and environment in which the relationship took place) when the doctor's grade was controlled for (significant F change < 0.001). 93 (83%) of the doctors in this study would have needed only four raters to achieve a reliable score if the intent was to determine if they were satisfactory. The mean time taken to complete the questionnaire by a rater was six minutes. Just over an hour of administrative time is needed for each doctor. CONCLUSIONS SPRAT seems to be a valid way of assessing large numbers of doctors to support quality assurance procedures for training programmes. The feedback from SPRAT can also be used to inform personal development planning and focus quality improvements.
Collapse
Affiliation(s)
- Julian C Archer
- Academic Unit of Child Health, Sheffield Children's Hospital, Sheffield S10 2HT
| | | | | |
Collapse
|
202
|
Hijazi M. Structured continuous objective-based assessment of resident's performance at point of care (SCOPA). Ann Saudi Med 2005; 25:193-7. [PMID: 16119518 PMCID: PMC6147979 DOI: 10.5144/0256-4947.2005.193] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 11/01/2004] [Indexed: 11/26/2022] Open
Abstract
The assessment of the clinical performance of physicians-in-training is an important task. The critical care rotation is a mandatory rotation for most residency training programs and is designed to ensure the graduation of trainees who are able to initiate lifesaving management during medical emergencies. Ensuring that each resident fulfills the objectives of the rotation is of paramount importance. Unfortunately, the current assessment methods are subjective and suffer from many threats to validity and reliability that make the assessment inaccurate. In this review, the current assessment method is analyzed, and causes for inaccuracy are identified. A new model for assessment that is continuous, structured, objective-based and at the point of care (SCOPA) is proposed based on the best available assessment methods. Such a model might be useful for the assessment of trainee's performance in critical care as well as non-critical care rotations.
Collapse
Affiliation(s)
- Mohammed Hijazi
- Department of Medicine, King Faisal Specialist Hospital & Research Centre, Riyadh, Saudi Arabia.
| |
Collapse
|
203
|
Littlefield JH, Darosa DA, Paukert J, Williams RG, Klamen DL, Schoolfield JD. Improving resident performance assessment data: numeric precision and narrative specificity. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2005; 80:489-495. [PMID: 15851464 DOI: 10.1097/00001888-200505000-00018] [Citation(s) in RCA: 14] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/24/2023]
Abstract
PURPOSE To evaluate the use of a systems approach for diagnosing performance assessment problems in surgery residencies, and intervene to improve the numeric precision of global rating scores and the behavioral specificity of narrative comments. METHOD Faculty and residents at two surgery programs participated in parallel before- and-after trials. During the baseline year, quality assurance data were gathered and problems were identified. During two subsequent intervention years, an educational specialist at each program intervened with an organizational change strategy to improve information feedback loops. Three quality-assurance measures were analyzed: (1) percentage return rate of forms, (2) generalizability coefficients and 95% confidence intervals of scores, and (3) percentage of forms with behaviorally specific narrative comments. RESULTS Median return rates of forms increased significantly from baseline to intervention Year 1 at Site A (71% to 100%) and Site B (75% to 100%), and then remained stable during Year 2. Generalizability coefficients increased between baseline and intervention Year 1 at Site A (0.65 to 0.85) and Site B (0.58 to 0.79), and then remained stable. The 95% confidence interval around resident mean scores improved at Site A from baseline to intervention Year 1 (0.78 to 0.58) and then remained stable; at Site B, it remained constant throughout (0.55 to 0.56). The median percentage of forms with behaviorally specific narrative comments at Site A increased significantly from baseline to intervention Years 1 and 2 (50%, 57%, 82%); at Site B, the percentage increased significantly in intervention Year 1, and then remained constant (50%, 60%, 67%). CONCLUSIONS Diagnosing performance assessment system problems and improving information feedback loops improved the quality of resident performance assessment data at both programs.
Collapse
Affiliation(s)
- John H Littlefield
- University of Texas Health Science Center at San Antonio, Academic Informatics Services, 78229-3900, USA.
| | | | | | | | | | | |
Collapse
|
204
|
Goldstein EA, Maclaren CF, Smith S, Mengert TJ, Maestas RR, Foy HM, Wenrich MD, Ramsey PG. Promoting fundamental clinical skills: a competency-based college approach at the University of Washington. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2005; 80:423-33. [PMID: 15851451 DOI: 10.1097/00001888-200505000-00003] [Citation(s) in RCA: 27] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/10/2023]
Abstract
The focus on fundamental clinical skills in undergraduate medical education has declined over the last several decades. Dramatic growth in the number of faculty involved in teaching and increasing clinical and research commitments have contributed to depersonalization and declining individual attention to students. In contrast to the close teaching and mentoring relationship between faculty and students 50 years ago, today's medical students may interact with hundreds of faculty members without the benefit of a focused program of teaching and evaluating clinical skills to form the core of their four-year curriculum. Bedside teaching has also declined, which may negatively affect clinical skills development. In response to these and other concerns, the University of Washington School of Medicine has created an integrated developmental curriculum that emphasizes bedside teaching and role modeling, focuses on enhancing fundamental clinical skills and professionalism, and implements these goals via a new administrative structure, the College system, which consists of a core of clinical teachers who spend substantial time teaching and mentoring medical students. Each medical student is assigned a faculty mentor within a College for the duration of his or her medical school career. Mentors continuously teach and reflect with students on clinical skills development and professionalism and, during the second year, work intensively with them at the bedside. They also provide an ongoing personal faculty contact. Competency domains and benchmarks define skill areas in which deepening, progressive attention is focused throughout medical school. This educational model places primary focus on the student.
Collapse
Affiliation(s)
- Erika A Goldstein
- University of Washington School of Medicine (UWSOM) Colleges, Seattle, WA 98195-7430, USA.
| | | | | | | | | | | | | | | |
Collapse
|
205
|
Lo Sasso AA, Lane JL, Malloy RB. Using standardized patient outcome to measure the effect of teaching asthma-related patient education and information-giving skills to medical students. TEACHING AND LEARNING IN MEDICINE 2005; 17:228-32. [PMID: 16042517 DOI: 10.1207/s15328015tlm1703_5] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/03/2023]
Abstract
BACKGROUND Patient education and giving information is a core skill that improves patient adherence and medical outcomes. PURPOSE To evaluate the impact of a teaching intervention on 3rd-year students' competency in patient education and information giving about asthma medication delivery. METHODS Students (n=81) completed a 1-hr teaching intervention of didactics followed by role playing of asthma patient education scenarios. Using a standardized patient post intervention, patient education and information-giving skills about spacer/metered dose inhalers were scored overall and on a 12-item checklist and compared to a control group (n=70). Students' knowledge was evaluated using a short answer test. RESULTS The performance of intervention students on overall patient education, 10 of the 12 checklist items, and the test was significantly higher than controls but did not approach competency. CONCLUSIONS The 1-hr intervention improved clinical performance and knowledge, but students did not become competent. Future studies should investigate how competence in this and other core patient education skills can be successfully achieved.
Collapse
Affiliation(s)
- Alisa Alfonsi Lo Sasso
- Department of Pediatrics, Temple University Medical School, Philadelphia, PA 19083, USA.
| | | | | |
Collapse
|
206
|
Anderson CI, Jentz AB, Kareti LR, Harkema JM, Apelgren KN, Slomski CA. Assessing the competencies in general surgery residency training. ACTA ACUST UNITED AC 2005; 62:111-6. [PMID: 15708161 DOI: 10.1016/j.cursur.2004.07.016] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022]
Affiliation(s)
- Cheryl I Anderson
- Department of Surgery, Michigan State University, East Lansing, Michigan 48912, USA.
| | | | | | | | | | | |
Collapse
|
207
|
Pfeiffer CA, Kosowicz LY, Holmboe E, Wang Y. Face-to-face clinical skills feedback: lessons from the analysis of standardized patient's work. TEACHING AND LEARNING IN MEDICINE 2005; 17:254-6. [PMID: 16042521 DOI: 10.1207/s15328015tlm1703_9] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/03/2023]
Abstract
BACKGROUND Current evidence suggests that trainees are evaluated less stringently when feedback is given face-to-face, limiting its value. PURPOSE We expected that standardized patients (SPs) would also be more lenient in scoring when they gave immediate feedback to the students. METHODS Data from 6 clinical skills assessments of students in Years 1 to 4 of medical school were used. Comparisons in scoring were made for the cases on which students received feedback and those where they did not. Analysis was done using a hierarchical linear regression model to test for significant differences. RESULTS The results indicated no significant difference in scoring for the history and physical exam checklists. There was a small but significant difference on the measure of interpersonal and interviewing skills. CONCLUSIONS The SPs were trained to avoid stringency, leniency, and restricted range in scoring students. These training methods may be useful for training faculty to give face-to-face feedback.
Collapse
Affiliation(s)
- Carol A Pfeiffer
- Department of Medicine, University of Connecticut School of Medicine, Farmington, Connecticut 06030, USA.
| | | | | | | |
Collapse
|
208
|
Kernan WN, Fagan MJ, Michael Elnicki D. The Clerkship Directors in Internal Medicine's agenda for advancing medical student education in internal medicine. Am J Med 2004; 117:713-7. [PMID: 15501217 DOI: 10.1016/j.amjmed.2004.08.008] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 10/26/2022]
Affiliation(s)
- Walter N Kernan
- Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut, USA
| | | | | |
Collapse
|
209
|
Hasnain M, Connell KJ, Downing SM, Olthoff A, Yudkowsky R. Toward meaningful evaluation of clinical competence: the role of direct observation in clerkship ratings. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2004; 79:S21-4. [PMID: 15383380 DOI: 10.1097/00001888-200410001-00007] [Citation(s) in RCA: 16] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/14/2023]
Abstract
PROBLEM STATEMENT AND PURPOSE The lack of direct observation by faculty may affect meaningful judgments of clinical competence. The purpose of this study was to explore the influence of direct observation on reliability and validity evidence for family medicine clerkship ratings of clinical performance. METHOD Preceptors rating family medicine clerks (n = 172) on a 16-item evaluation instrument noted the data-source for each rating: note review, case discussion, and/or direct observation. Mean data-source scores were computed and categorized as low, medium or high, with the high-score group including the most direct observation. Analyses examined the influence of data-source on interrater agreement and associations between clerkship clinical scores (CCS) and scores from the National Board of Medical Examiners (NBME(R)) subject examination as well as a fourth-year standardized patient-based clinical competence examination (M4CCE). RESULTS Interrater reliability increased as a function of data-source; for the low, medium, and high groups, intraclass correlation coefficients were.29,.50, and.74, respectively. For the high-score group, there were significant positive correlations between CCS and NBME score (r =.311, p =.054); and between CCS and M4CCE (r =.423, p =.009). CONCLUSION Reliability and validity evidence for clinical competence is enhanced when more direct observation is included as a basis for clerkship ratings.
Collapse
Affiliation(s)
- Memoona Hasnain
- Department of Family Medicine, College of Medicine, University of Illinois at Chicago, 60612, USA.
| | | | | | | | | |
Collapse
|
210
|
Schultz KW, Kirby J, Delva D, Godwin M, Verma S, Birtwhistle R, Knapper C, Seguin R. Medical Students' and Residents' preferred site characteristics and preceptor behaviours for learning in the ambulatory setting: a cross-sectional survey. BMC MEDICAL EDUCATION 2004; 4:12. [PMID: 15298710 PMCID: PMC514563 DOI: 10.1186/1472-6920-4-12] [Citation(s) in RCA: 56] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/08/2003] [Accepted: 08/06/2004] [Indexed: 05/12/2023]
Abstract
BACKGROUND Medical training is increasingly occurring in the ambulatory setting for final year medical students and residents. This study looks to identify if gender, school, level of training, or specialty affects learner's (final year medical students and residents) preferred site characteristics and preceptor behaviours for learning in the ambulatory setting. METHODS All final year medical students and residents at the five medical schools in Ontario (N = 3471) were surveyed about the site characteristics and preceptor behaviours most enhancing their learning in the ambulatory setting. Preferred site characteristics and preceptor behaviours were rank ordered. Factor analysis grouped the site characteristics and preceptor behaviours into themes which were then correlated with gender, school, level of training, and specialty. RESULTS Having an adequate number and variety of patients while being supervised by enthusiastic preceptors who give feedback and are willing to discuss their reasoning processes and delegate responsibility are site characteristics and preceptor behaviours valued by almost all learners. Some teaching strategies recently suggested to improve efficiency in the ambulatory teaching setting, such as structuring the interview for the student and teaching and reviewing the case in front of the patient, were found not to be valued by learners. There was a striking degree of similarity in what was valued by all learners but there were also some educationally significant differences, particularly between learners at different levels and in different specialties. Key findings between the different levels include preceptor interaction being most important for medical students as opposed to residents who most value issues pertaining to patient logistics. Learning resources are less valued early and late in training. Teaching and having the case reviewed in front of the patient becomes increasingly less valued as learners advance in their training. As one approaches the end of ones' training office management instruction becomes increasingly valued. Differences between specialties pertain most to the type of practice residents will ultimately end up in (ie: office based specialties particularly valuing instruction in office management and health care system interaction). CONCLUSIONS Preceptors need to be aware of, and make efforts to provide, teaching strategies such as feedback and discussing clinical reasoning, that learners have identified as being helpful for learning. If strategies identified as not being valued for learning, such as teaching in front of the patient, must continue it will be important to explore the barriers they present to learning. Although what all learners want from their preceptors and clinic settings to enhance their learning is remarkably similar, being aware of the educationally significant differences, particularly for learners at different levels and in different specialties, will enhance teaching in the ambulatory setting.
Collapse
Affiliation(s)
- Karen W Schultz
- Department of Family Medicine, Queen's University, 220 Bagot St., PO Bag 8888, Kingston, ON, Canada, K7L 5E9
| | - John Kirby
- Faculty of Education, Queen's University, 511 Union St., Kingston, ON, Canada, K7M 5R4
| | - Dianne Delva
- Department of Family Medicine, Queen's University, 220 Bagot St., PO Bag 8888, Kingston, ON, Canada, K7L 5E9
| | - Marshall Godwin
- Department of Family Medicine, Queen's University, 220 Bagot St., PO Bag 8888, Kingston, ON, Canada, K7L 5E9
| | - Sarita Verma
- Department of Family Medicine, Queen's University, 220 Bagot St., PO Bag 8888, Kingston, ON, Canada, K7L 5E9
| | - Richard Birtwhistle
- Department of Family Medicine, Queen's University, 220 Bagot St., PO Bag 8888, Kingston, ON, Canada, K7L 5E9
| | - Chris Knapper
- Department of Psychology, Queen's University Kingston, ON, Canada, K7L 3N6
| | - Rachelle Seguin
- Department of Family Medicine, Queen's University, 220 Bagot St., PO Bag 8888, Kingston, ON, Canada, K7L 5E9
| |
Collapse
|
211
|
Whelan A, Appel J, Alper EJ, De Fer TM, Dickinson TA, Fazio SB, Friedman E, Kuzma MA, Reddy S. The future of medical student education in internal medicine. Am J Med 2004; 116:576-80. [PMID: 15063829 DOI: 10.1016/j.amjmed.2004.02.001] [Citation(s) in RCA: 8] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/22/2022]
|