Using patients' charts to assess medical trainees in the workplace: a systematic review

Education leads to increased adherence to the 'Australian Competency Standards for Occupational Therapy Driver Assessors'

BACKGROUND

Occupational therapy competency standards provide an evidence-base to inform clinical best practice, however it is not known whether education about competency standards will increase occupational therapists' adherence to their use.

AIMS/OBJECTIVES

To investigate if education about the 'Australian Competency Standards for Occupational Therapy Driver Assessors' leads to increased adherence to the competency standards in the clinical practice of occupational therapy driver assessors.

MATERIALS AND METHODS

A mixed methods multiple case study design was used to evaluate 5 occupational therapy driver assessors' adherence to the competency standards. An audit of 25 client files and interviews were conducted to evaluate practice against the competency standards prior to an education session, followed by an audit of a further 25 files, interview and feedback after education.

RESULTS

Qualitative and quantitative analyses suggest that education about the 'Australian Competency Standards for Occupational Therapy Driver Assessors' was associated with increased adherence to the competency standards in clinical practice. The results also support the use of competency standards in clinical practice.

CONCLUSIONS AND SIGNIFICANCE

Competency standards can be used to inform and guide clinical practice, and individualised education and feedback of practice against the competency standards can increase occupational therapists' adherence to these standards.

Can case-based discussions in a group setting be used to assess residents' clinical skills?

OBJECTIVES

The purpose of this study was to explore residents' and assessors' perception of a new group assessment concept.

METHODS

This qualitative study consists of observations of four group assessment sessions, followed by semi-structured interviews with six residents and four assessors (specialists in internal medicine), who all volunteered to be interviewed. All residents at a medical department (eleven to fifteen each time) and four assessors participated in four group assessments, where the residents' clinical skills were assessed through case-based discussions. An external consultant (an anthropologist) performed the observations and the interviews. Notes from the observations and the interviews were analyzed using an inductive approach.

RESULTS

Eight of the ten interviewed participants preferred group assessment to individual assessment. Results from the interviews suggested that the group assessments were more consistent and that the level of discussion was perceived to be higher in the group discussions compared to the one-to-one discussions. All residents indicated that they had acquired new knowledge during their assessment and reported having learned from listening to the assessment of their peers. Assessors similarly reported gaining new knowledge.

CONCLUSIONS

The residents and assessors expressed very favourable attitudes toward the new group assessment concept. The assessment process was perceived to be higher in quality and more consistent, contributing to learning for all participating doctors in the department. Group assessment is feasible and acceptable, and provides a promising tool for assessment of clinical skills in the future.

A Medical Record Peer-Review System to Evaluate Residents' Clinical Competence: Criterion Validity Analysis

In contrast to input evaluation (education delivered at school) and output evaluation (students' capability at graduation), the methods of outcome evaluation (performance after graduation) of medical education have not been sufficiently established. To establish a method to measure the quality of patient care and conduct outcome evaluation, we have been developing a peer review system of medical records. Here, we undertook a pilot study to evaluate the criterion validity of our system by using "evaluation by program directors (supervisors in the hospitals)" as a criterion standard. We selected 13 senior residents from three teaching hospitals. Five reviewers (general internists working in other hospitals) visited the hospitals independently and evaluated five patients' records for each resident based on the previously established sheet comprising 15 items. Independently, program directors of the senior residents evaluated their clinical performance using an evaluation sheet comprising ten items. Pearson's analysis revealed statistically significant correlation coefficients in three pairs of assessments including clinical reasoning (r = 0.5848, P = 0.0358). Bootstrap analysis revealed statistically significant correlation coefficients in additional 5 pairs including history taking (r = 0.509, 95% confidence interval: 0.034-0.847). In contrast, the correlation coefficients were low in some items: r = 0.132 (-0.393-0.639) for physical examination and r = 0.089 (-0.847-0.472) for attitude toward patients. To the best of our knowledge, this is the first study, albeit a pilot one, that investigates the criterion validity of medical record evaluations conducted by comparing the assessments of medical records with those by program directors.

Specialty Training in Family Medicine in Montenegro - an Evaluation of The Programme By the First Generation of Trainees

Introduction

One of the aims of health care reform in Montenegro is to strengthen primary care. An important step forward is the implementation of specialty training in family medicine (FM). The aim of this article is to evaluate the implementation of specialty training in family medicine in Montenegro, regarding the content, structure and methods, by the first generation of trainees and the coordinator of the training.

Methods

A questionnaire was sent by mail in July and August 2017 to all 26 eligible trainees who started specialty training in 2013. Twenty-two of the 26 trainees (84.6%) responded. The questionnaire consisted of closed and open-ended questions related to the evaluation of the training. A descriptive quantitative and qualitative analysis with predefined themes and a semi-structured interview with the coordinator were carried out.

Results

The process of training in FM was assessed positively by both trainees and the coordinator. The positive assessment included that the specialisation course offered modern design through modules and practice, and trainees both improved their existing knowledge and skills and acquired new ones necessary for everyday work. The coordinator emphasised the importance of the introduction of new teaching methods and formative assessment, the important role of mentors, and the involvement of Slovenian colleagues in the teaching process and supervision of the programme.

Conclusions

The implementation of speciality training in FM in Montenegro was successful. Several assessment methods were used that can be further developed in individual structured feedback, which could stimulate the continual improvement of trainees’ knowledge and competencies.

Assessment of SOAP note evaluation tools in colleges and schools of pharmacy

INTRODUCTION

To describe current methods used to assess SOAP notes in colleges and schools of pharmacy.

METHODS

Members of the American Association of Colleges of Pharmacy Laboratory Instructors Special Interest Group were invited to share assessment tools for SOAP notes. Content of submissions was evaluated to characterize overall qualities and how the tools assessed subjective, objective, assessment, and plan information.

RESULTS

Thirty-nine assessment tools from 25 schools were evaluated. Twenty-nine (74%) of the tools were rubrics and ten (26%) were checklists. All rubrics included analytic scoring elements, while two (7%) were mixed with holistic and analytic scoring elements. A majority of the rubrics (35%) used a four-item rating scale. Substantial variability existed in how tools evaluated subjective and objective sections. All tools included problem identification in the assessment section. Other assessment items included goals (82%) and rationale (69%). Seventy-seven percent assessed drug therapy; however, only 33% assessed non-drug therapy. Other plan items included education (59%) and follow-up (90%).

DISCUSSION AND CONCLUSIONS

There is a great deal of variation in the specific elements used to evaluate SOAP notes in colleges and schools of pharmacy. Improved consistency in assessment methods to evaluate SOAP notes may better prepare students to produce standardized documentation when entering practice.

ARCADO - Adding random case analysis to direct observation in workplace-based formative assessment of general practice registrars

BACKGROUND

Workplace-based formative assessments using consultation observation are currently conducted during the Australian general practice training program. Assessment reliability is improved by using multiple assessment methods. The aim of this study was to explore experiences of general practice medical educator assessors and registrars (trainees) when adding random case analysis to direct observation (ARCADO) during formative workplace-based assessments.

METHODS

A sample of general practice medical educators and matched registrars were recruited. Following the ARCADO workplace assessment, semi-structured qualitative interviews were conducted. The data was analysed thematically.

RESULTS

Ten registrars and eight medical educators participated. Four major themes emerged - formative versus summative assessment; strengths (acceptability, flexibility, time efficiency, complementarity and authenticity); weaknesses (reduced observation and integrity risks); and contextual factors (variation in assessment content, assessment timing, registrar-medical educator relationship, medical educator's approach and registrar ability).

CONCLUSION

ARCADO is a well-accepted workplace-based formative assessment perceived by registrars and assessors to be valid and flexible. The use of ARCADO enabled complementary insights that would not have been achieved with direct observation alone. Whilst there are some contextual factors to be considered in its implementation, ARCADO appears to have utility as formative assessment and, subject to further evaluation, high-stakes assessment.

When Professors Don't Return Essays

ISSUE

Students devote hundreds of hours to writing notes during medical school clerkships but receive very limited feedback on that work. Medical student notes are like college essays-both are persuasive compositions. But attending physicians rarely scrutinize student notes like college professors analyze essays. This is a missed opportunity to teach clinical reasoning.

EVIDENCE

A survey at our institution showed that only 16% of students received written feedback and 31% received oral feedback on their notes from more than 3 attending physicians during the first 8 months of 3rd-year clerkships. Many studies have reported a paucity of feedback across multiple domains and a sense among students that clinical reasoning is not being adequately taught during clerkships. Meanwhile, college professors teach written composition and reasoning through interactive methods that help students to develop structured, well-reasoned arguments. A recent study showed that 85% of Oxford undergraduates favored these demanding and time-intensive tutorials.

IMPLICATIONS

Attending physicians who adopt a tutorial-based approach toward their students' notes would have a forum to teach clinical reasoning and emphasize the importance of written composition in medical practice.