Formative assessment: a student perspective

Peers as OSCE assessors for junior medical students - a review of routine use: a mixed methods study

BACKGROUND

Peer-assisted learning is well established in medical education; however, peer tutors rarely act as assessors for the OSCE. In the compulsory, near-peer teaching programme covering basic medical skills at the University of Heidelberg, peer tutors serve as assessors on a formative OSCE. This study aimed to investigate the feasibility and acceptance of peer assessors and to survey the perceived advantages and disadvantages of their use.

METHODS

In 2016 and 2017 all OSCE peer assessors (third to sixth-year medical students) and all of the peer-assessed students in 2017 (second-year-medical students) were invited to participate in a survey. Both groups were asked to complete a tablet-based questionnaire immediately after the OSCE. Peer assessors were asked to rate eight statements and the peer-assessed students to rate seven statements on a five-point Likert scale. Both were asked to comment on the advantages and disadvantages of peer-assessors.

RESULTS

Overall, 74 of 76 peer assessors and 307 of 308 peer-assessed students participated in the study. 94% (67/74) of peer assessors and 90% (276/307) of the peer-assessed group thought that it is important to have peer tutors as assessors. Of the peer assessors, 92% (68/74) felt confident in giving structured feedback during the OSCE and 66% (49/74) felt they had improved their teaching skills. Of the peer-assessed students, 99% (306/307) were satisfied with their peers as OSCE assessors and 96% (292/307) considered the peer feedback during the OSCE as helpful. The participants mentioned structural benefits, such as lower costs, and suggested the quality of the OSCE was higher due to the use of peer assessors. The use of peer assessors was found to be beneficial for the learners in the form of high-quality feedback and an overall reduction in stress. Furthermore, the use of peer assessors was found to be beneficial for the peer assessors (improved teaching and clinical skills).

CONCLUSION

From a learner's perspective, the use of peer assessors for a formative OSCE that is part of a near-peer teaching program aimed at junior medical students is favourable for all. A broad implementation of peer assessment in the formative OSCE should be encouraged to investigate effects on quality and stress-reduction.

Using a Novel Student-centered Teaching Method to Improve Pharmacy Student Learning

Objective. To improve students' learning and develop their learning skills in pharmacy education. Methods. A novel teaching method composed of Self-study, Test, Question and Discussion (STQD) sessions uses self-, peer-, co-learning, active learning, inductive teaching, and formative assessment to promote student-centered teaching in pharmacy education. STQD has been implemented within courses focusing on instrumental analysis and analytical chemistry. In a four-year study, qualitative and quantitative approaches were used to evaluate the effectiveness of STQD in facilitating students' learning. Results. Students positively evaluated their experience in STQD class with an overall mean and standard deviation of 4.35 (0.64) in a 5-point Likert scale, where a score of 1 is "strongly disagree" and a score of 5 is "strongly agree." The final examination results showed students' average score in an STQD class was higher than students' average score in a traditional class. Most students indicated that the STQD improved their learning ability, communication ability and facilitated their learning. Conclusion. The use of STQD in an integrated basic science curriculum has the potential to develop students' learning skills and improve their learning in pharmacy education.

Students as stakeholders in assessment: how students perceive the value of an assessment

INTRODUCTION

For assessment to fill an educational role, students must see the results generated by assessment as valuable, and actively engage with this feedback in order to support learning. Few studies include examinees as stakeholders in validation beyond general notions of acceptability. Here, we explore students as stakeholders in the validation of a newly implemented assessment.

METHODS

A student-relevant validity framework based on the unified theory of validity was created and adapted to a survey format. Likert-style items were used to examine first- and second-year medical students' perceptions of a new cumulative assessment, with several open-ended items. Analysis included: mean ratings per subscale of validity evidence, thematic analysis of comments, and a correlation between questionnaire subscores and exam performance.

RESULTS

Seventy-seven students participated (20.5%). Student perceptions of the assessment were favourable, with significantly different ratings across validity evidence (Response Process (4.8 (SD = 0.7); scored/6), Content (4.6(0.9)), Consequential (4.4(0.8)), Internal Structure (4.2(0.9)), and Relationship to Other Variables (4.0(1.0))). Exam performance correlated with subscores for Relationship to Other Variables (r = 0.34, p < 0.005) and Response Process (r = 0.24, p < 0.05).

DISCUSSION

Students perceived the assessment as facilitating learning, providing 'checkpoints', and were disappointed when it did not meet their expectations regarding the purpose of assessment. If students perceive that results do not reflect their future performance in clinical environments, or do not align with their perceived purpose of assessment, the educational value of assessment may be limited. It is critical to understand when, and how students engage in interpreting and integrating assessment-generated feedback to ensure that assessment contributes positively to learning.

Integrating retinal simulation with a peer-assessed group OSCE format to teach direct ophthalmoscopy

OBJECTIVE

To describe the conception and evaluation of a novel educational intervention to teach direct ophthalmoscopy with retinal simulators using a peer-assessed group objective structured clinical examination (OSCE) format.

DESIGN

Prospective, single-centre educational trial at Oxford University Medical School, Oxford, U.K.

PARTICIPANTS

A total of 160 consecutive undergraduate fifth-year medical students participated in the study.

METHODS

Students identified prior experience, teaching, examination, and feedback relevant to direct ophthalmoscopy. Students self-evaluated their perceived confidence across 6 domains of direct ophthalmoscopy examination before and after the educational intervention using a Likert-type psychometric scale. Wilcoxon matched pair testing was used to determine statistical significance for each domain.

RESULTS

The group OSCE intervention increased confidence in direct ophthalmoscopy overall from 2.5% to 63.8% (p < 0.001). Confidence improved in all 6 domains, most significantly in controls of ophthalmoscope (p < 0.001) and sequence of examination (p < 0.001) but also in the identification and interpretation of retinal signs (p < 0.001). Students rated the tutorial as very effective or effective across all 6 domains, and 96.29% rated the tutorial as effective overall.

CONCLUSIONS

Retinal simulation, integrated with a peer-assessed group OSCE format, is effective in increasing confidence in all aspects of direct ophthalmoscopy. It may be insufficient alone for training in the identification and interpretation of posterior segment clinical signs. Diminishing ophthalmology clerkships worldwide require ophthalmologists to identify innovative teaching methods, using modern technology and pedagogy to deliver high-quality, yet high-throughput, training in direct ophthalmoscopy. This novel teaching strategy may be considered by ophthalmologists responsible for direct ophthalmoscopy training in a teaching hospital context.

Is the OSCE a feasible tool to assess competencies in undergraduate medical education?

BACKGROUND

The Objective Structured Clinical Examination (OSCE) was introduced by Harden et al. (1975) trying to answer the problems regarding the assessment of clinical competencies. Despite increasingly widespread use of OSCEs, debate continues with arguments as 'why using such a demanding format if other methods are available?'

AIM

To review and synthesize evidence on technical and economic feasibility of OSCE in undergraduate medical studies.

METHODS

Best Evidence Medical Education methodology was applied by two independent coders to 1083 studies identified by literature search from 1975 until the end of 2008.

KEY FINDINGS

The OSCE is a feasible approach to the assessment of clinical competence for use in different cultural and geographical contexts; to assess a wide range of learning outcomes; in different specialties and disciplines; for formative and summative purposes; to assess students a curriculum or an educational intervention; in the different phases of education including the early and later years of the undergraduate curriculum; and in different health care professions.

CONCLUSION

Despite being an expensive test format, evidence suggests that the use of OSCE produces reliable results. The study also suggests that one reason for the wide-scale adoption of the OSCE and the feasibility of its use in different contexts and situations is its inherent flexibility in terms of the number of students that can be assessed, the number of examiners included, the type of patients represented and the format of the examination itself, including the length of the examination, the number and duration of stations.

Teaching the trauma teachers: an international review of the Advanced Trauma Life Support Instructor Course

OBJECTIVES

To identify educational variations between Advanced Trauma Life Support (ATLS) Instructor Courses in the UK, USA and Australia, to determine the impact of any such variations, and to consider their contributory role in future course development.

METHODS

Observational, participatory qualitative study of ATLS Instructor Courses held in three representative teaching centres in 2006 and 2007.

RESULTS

Each international territory has developed individual approaches to the teaching of common course content. These approaches have a variable impact upon the educational efficacy of the course.

CONCLUSIONS

Observation of educational practice in international course centres allows for a richer understanding of the nature and extent of variations in teaching methods and course organisation. The amalgamation of these variations will facilitate a more inclusive international common platform for the 8th edition Instructor Course.

Differential effects of two types of formative assessment in predicting performance of first-year medical students

Formative assessments are systematically designed instructional interventions to assess and provide feedback on students' strengths and weaknesses in the course of teaching and learning. Despite their known benefits to student attitudes and learning, medical school curricula have been slow to integrate such assessments into the curriculum. This study investigates how performance on two different modes of formative assessment relate to each other and to performance on summative assessments in an integrated, medical-school environment. Two types of formative assessment were administered to 146 first-year medical students each week over 8 weeks: a timed, closed-book component to assess factual recall and image recognition, and an un-timed, open-book component to assess higher order reasoning including the ability to identify and access appropriate resources and to integrate and apply knowledge. Analogous summative assessments were administered in the ninth week. Models relating formative and summative assessment performance were tested using Structural Equation Modeling. Two latent variables underlying achievement on formative and summative assessments could be identified; a "formative-assessment factor" and a "summative-assessment factor," with the former predicting the latter. A latent variable underlying achievement on open-book formative assessments was highly predictive of achievement on both open- and closed-book summative assessments, whereas a latent variable underlying closed-book assessments only predicted performance on the closed-book summative assessment. Formative assessments can be used as effective predictive tools of summative performance in medical school. Open-book, un-timed assessments of higher order processes appeared to be better predictors of overall summative performance than closed-book, timed assessments of factual recall and image recognition.

OSCEs--seven years on the bandwagon: the progress of an objective structured clinical evaluation programme

The original purpose of Objective Structured Clinical Examination (OSCE), as first described in the medical literature [Harden, R.M., Stevenson, M., Downie, W.W., Wilson, G.M., 1975. Assessment of clinical competence using objective structured examination. British Medical Journal, 1, 447-451], provided a means of examining the skills acquisition of medical students. A review of the literature, since that time, provides the background to the development of OSCEs into pre-registration nursing curricula, with the OSCE programme at the University of Salford presented here as a case study. The original student sample was a mixture of 150-250 adult, child and mental health students in each of seven cohorts over a period of four years. Each student undertook a 30-min formative, simulated patient, holistic care OSCE in their second year of the programme. Later developments included one remote workstation connected to and as part of the holistic patient care encounter. In subsequent curricula, the larger cohorts of around 250-300 students were accommodated in a formative rotational three-workstation OSCE, based on clinical skills to be acquired prior to their first clinical placement. A summative patient-centred OSCE was undertaken in practice at a later date. The educational and practice value of OSCEs regarding their clinical content and context in nursing curricula now and in the future are explored, along with the practicalities of implementation.

The Advanced Trauma Life Support course: a history of its development and review of related literature

The origins, development, and success of the ATLS course are described with reference to the literature.

The impact of evidence-based education on a perinatal capacity-building initiative in Macedonia

CONTEXT

The perinatal mortality rate (PMR) in Macedonia is among the highest in Europe. The World Bank supported a consultant (HEJ) to collaborate with a Macedonian team to develop a national perinatal strategy with the goal of reducing the PMR. Education was given priority in the form of a hospital-based initiative to develop the capacity of health professionals to introduce evidence-based perinatal practice into 16 participating hospitals. A "train the teachers" approach was used, with trainees introduced to modern education and clinical practice in Sydney and subsequently supported to train their colleagues in Skopje.

OBJECTIVES

To describe the development, implementation and evaluation of the educational intervention.

METHODS

A curriculum, based on specific Macedonian needs, was developed in order to integrate teaching in the knowledge, skills and attitudinal domains of learning, using small group, interactive techniques. Twenty-five Macedonian doctors and nurses participated in 4-month (phase 1a) and 6-month (phase 1b) teaching programmes at a tertiary perinatal unit in Sydney. Australian staff conducted 4 2-week modules for 36 trainees in Macedonia (phase 2). The phase 1 trainees conducted 8 modules for 57 colleagues in Skopje (phase 3). The intervention was evaluated by trainee questionnaires, assessments of competence, changes in hospital practice and pre- (1997-99) and post-intervention (2000-01) comparisons of PMR.

RESULTS

A total of 115 doctors and nurses graduated from the programme. Positive responses to the education programme exceeded 80%. Evidence-based practice in 16 participating hospitals (covering 91% of all Macedonian births) was verified in 6 key areas of neonatology. The PMR fell significantly from 27.4 to 21.5 per 1000 births (RR 0.79, 95% CI 0.73, 0.85). The early neonatal death rate in babies weighing over 1000 g fell by 36%.

CONCLUSIONS

The intervention has increased the capacity of Macedonians to practise best-evidence perinatal medicine and improve outcomes. Sustainability is predicted by the "train the teachers" approach, with concurrent strengthening of the infrastructure and organisational framework.

Nursing students' and lecturers' perspectives of objective structured clinical examination incorporating simulation

Objective Structured Clinical Examination (OSCE) has been widely and increasingly used since it was developed. Research has shown that it is an effective evaluation tool to assess practical skills (Sloan et al. 1995). In many instances the OSCE process has been adapted to test trainees from different healthcare related disciplines. In nursing education, as presented in this paper, principles of OSCE can also be used in a formative way to enhance skill acquisition through simulation. The aim of this approach to teaching is safely to help students gain more confidence when confronted by technical instruments present in the hospital environment, and to encourage them to reflect on a range of skills and competences they need to acquire. The OSCE stations can be designed in the form of small scenarios where students have to set-up or interact with technical instruments, or communicate with patients. This type of simulation exercise can be varied as a whole and specifically within each station at the same time. The use of this hybrid formative OSCE is being assessed by nursing students and lecturers. The feedback received regarding this teaching method and the results of this study are useful and show that OSCE are favourably perceived.

A survey of medical students' views about the purposes and fairness of assessment

OBJECTIVE

To survey medical students' views about the purposes and fairness of assessment procedures.

METHOD

The survey used a 19-item questionnaire designed for self-completion. Respondents were invited to "strongly agree", "agree", "disagree" or "strongly disagree" with a series of statements about the purposes and fairness of assessment. There was space for free text comments relating to each statement.

RESULTS

A total of 312 students out of a sample of 381 completed questionnaires (82% response rate). Whilst the majority of students (> 95%) agreed that ensuring competence, providing feedback and guiding student learning were important purposes of assessment, only half (51%) felt that assessment should be used to predict performance as a doctor. A clear majority of students (81%) agreed that, on the whole, assessment at Newcastle Medical School was fair. Data interpretation papers (comprising a combination of multiple true/false, "one best answer" and short answers) were perceived to be the fairest assessment tool; the assessment of clinical rotations by supervisors was perceived as the least fair. Differences in perception about the fairness of several assessment methods emerged between junior and senior students. A large number of respondents expressed desire for the provision of more feedback on performance in order to guide future learning.

CONCLUSIONS

Whilst students' views about the fairness of specific assessment tools may sometimes be at variance with published research on assessment, their perceptions will influence the acceptability of assessment. Students would welcome the introduction of methods that provide meaningful assessment feedback.

The use of an objective structured clinical examination (OSCE) for formative and summative assessment in a general practice clinical attachment and its relationship to final medical school examination performance

OBJECTIVES

To report the use of OSCEs for both formative and summative purposes within a general practice undergraduate clinical attachment and to compare student performance in the departmental OSCEs with that of their final medical school examinations.

METHODS

Twenty-eight students rotated through the attachment and undertook pre- and post-attachment OSCEs of similar format but different content. Results were analysed to determine relationships between mean scores in the two OSCEs and student performance in their final medical school MBBS examinations.

RESULTS

There was a marked improvement in all OSCE station scores. Pre-attachment scores for those stations measuring physical examination and problem-solving skills were unrelated to prior clinical experience. Post-attachment OSCE mean scores were significantly correlated with final examination OSCE and total mean scores.

CONCLUSION

The general practice attachment appears to upgrade those clinical skills measured by the pre- and post-attachment OSCE, however, there was no control group of students. Problem-solving and focused physical examination skills need to be targeted by all undergraduate clinical departments. The department's post-attachment OSCE and total assessment results are predictors of final examination OSCE and total results. The use of pre- and post-attachment OSCEs facilitates both students' formative learning processes and the department's evaluation of its educational programme.

Competency-based learning in traumatology

BACKGROUND

A multidisciplinary, competency-based trauma teaching program was introduced for final year medical students (n = 67) at Royal Prince Alfred Hospital (RPAH) in 1994 to complement the surgical clerkship.

METHODS

The method involved small groups rotating through a series of teaching stations each structured to address a predetermined competency. Four 3-hour sessions were held on the subject areas of resuscitation, plastic, orthopedic, and neurotrauma. Performance in the trauma section of a summative Objective Structured Clinical Examination (OSCE), 6 months after the teaching, was compared with that of a control group (n = 127) from other campuses where trauma was taught by a series of discipline-based lectures. Three trauma OSCE stations were designed to test psychomotor skills while five addressed aspects of the cognitive domain. Checklists were used to ensure standardization of scoring in a range of questions asked for skills tested at each station.

RESULTS

The marks of the RPAH students (mean 78% +/- SD 9%) were significantly higher (P < 0.0005) than the controls (mean 70% +/- SD 9%) in the 8 trauma questions. There was no significant difference (P = 0.8) in marks obtained by the study group (mean 61% +/- SD 8%) and controls (mean 63% +/- SD 7%) in 22 questions sampling a wide spectrum of nontrauma subject areas. The study group performed significantly better in one of the three skills stations and three of the five problem-solving stations when compared with the control group.

CONCLUSION

The innovation has the potential to fulfill a need for an integrated trauma program in the undergraduate core curriculum.

Competency-based learning in neonatology

An educational model integrating structured teaching with clinical experience, or clerkship, has been designed to enable students to learn the core knowledge, skills and attitudes necessary to care for the newborn. The programme is run for fifth year medical students as part of four, 9-week periods in Obstetrics and Gynaecology each year. A SCORPIO teaching session is held in week 1 to introduce students to the core competencies in the subject areas of newborn examination, breastfeeding, resuscitation, respiratory distress and anthropometry. Groups of four students rotate through each topic, which is conducted by a neonatologist or registrar in training. Eight problem-based learning sessions are held during weeks 2-9. Several students assess a clinical problem, identify learning issues and meet colleagues and a facilitator to share their learning experiences and resolve the problem. The clinical experience, or clerkship, is based in the neonatal nursery where 2-3 students spend a week consolidating their clinical and procedural skills. A study group was assessed at the end of the programme by an Objective Structured Clinical Examination (OSCE) and a Multiple Choice Questionnaire (MCQ). A control group did the same assessment in week 1. All students were asked to rate the educational value of the three learning methods on a 5-point Likert scale. The study group (n = 20) achieved a mean composite mark of 66% (SD 10%). This was significantly higher (P < 0.001) than that of the control group (n = 18), mean 45% (SD 7%). All students (100%) rated the educational value of SCORPIO as high or very high, and the comparative rating for problem-based learning and clerkship was 65%, respectively. The programme was enthusiastically received by the students and resulted in mastery of a range of core competencies necessary for care of the newborn.