Application of best practice guidelines for OSCEs-An Australian evaluation of their feasibility and value

Outcome Evaluation of a Transnational Postgraduate Capacity-Building Program Using the Objective Structured Clinical Examination

Previous studies have applied interview-oriented self-reported or peer-centered evaluation methods, rather than an objective and quantitative method, to evaluate outcome of a postgraduate capacity-building program and have mainly focused on the cognitive level. To investigate the feasibility of the objective structured clinical examination (OSCE) in an international setting and report the results of the outcome evaluation for the behavioral aspect. A case-control study. Ninety examinees divided into 2 case-control groups: 17 program-experienced doctors and 17 control doctors in the first group, and 28 mentees of the program-experienced doctors and 28 control doctors in the second group. A six-station OSCE was implemented. The OSCE scores were measured to evaluate (1) the direct educational effect regarding learning in the first group and (2) the indirect educational effects regarding transfer in the second group. Written questionnaire and interview data were collected for qualitative analysis. The quantitative results of the overall or subcomponent OSCE scores indicated no significant differences in the comparisons of the first and second case-control groups. The qualitative data indicated that the program improved participants' medical knowledge, skills, and self-confidence, however, it also revealed limited learning environment provided by the program. This transnational study has demonstrated the process for introducing and successfully completing the testing of an OSCE in Laos. Discrepancy in the goals of the OSCE and the education program limited the usefulness of OSCE as an assessment tool, leading to the lack of significant differences in its results.

A comparison of the academic performance of graduate entry and undergraduate entry pharmacy students at the course exit level

INTRODUCTION

Graduate entry (GE) pharmacy students are trained in a shorter timeframe than undergraduate entry (UE) students. This study compares the academic performance of GE and UE pharmacy students at the course exit point.

METHODS

A retrospective analysis of final exam grades in written and objective structured clinical examination (OSCE) was performed between GE and UE students from three graduating cohorts. Final written examination contained clinical case study questions, whereas OSCE involved role play with simulated patients or doctors. Statistical analyses were performed by t-test and one-way analysis of variance at .05 significance level and Pearson's correlation coefficient.

RESULTS

No significant difference in academic performance was seen between GE and UE groups at course exit (P > .05). There was a trend for GE students performing marginally better in OSCE than UE students. Females showed better performances in verbal communication than males. GE males showed significantly lower empathy scores than all other groups. No significant difference was seen in problem-solving scores amongst all groups. Both UE and GE groups scored significantly better in written examinations compared with OSCE.

CONCLUSIONS

Graduate entry pharmacy students from accelerated learning pathway and UE students performed similarly at the course exit point, providing empirical support for non-traditional graduate entry pathway as a viable option.

Quality management of objective structured clinical examinations: An integrative literature review

BACKGROUND

Objective Structured Clinical Examinations are widely adopted as a clinical assessment method in nurse education. Quality is an integral aspect of the design and implementation of OSCEs, facilitating their credibility and rigour. However, there is a dearth of literature regarding the management of the quality of Objective Structured Clinical Examinations.

OBJECTIVE

To summarise existing literature regarding best practices on the management of the quality of Objective Structured Clinical Examinations in health science education.

DESIGN

An integrative literature review.

DATA SOURCES

EBSCOhost, including CINAHL, eBook Collection, E-journals, ERIC, Health Source-Consumer Edition, Health Source-Nursing/Academic Edition, Humanities International Complete and MEDLINE, together with Cochrane Online, PubMed, Taylor & Francis Online and ScienceDirect, were searched, followed by hand searching of references as well as a manual search for grey literature, using Google.

REVIEW METHODS

Using predetermined inclusion and exclusion criteria, research documents (experimental, non-experimental, descriptive and qualitative studies), non-research documents (editorials, opinion letters), including grey literature, published between January 2010 and March 2021.

RESULTS

A total of thirteen (n = 13) studies were included in the review. A total of 22 quality measures were identified, which should be applied in the preparation and planning, implementation and evaluation phases of Objective Structured Clinical Examinations. The preparation and planning phase is crucial for the quality of Objective Structured Clinical Examinations, as more than half (n = 12) of the 22 quality measures were identified in this phase.

CONCLUSIONS

There is limited available recorded evidence that defines and describes the management of the quality of Objective Structured Clinical Examinations. Future research should focus on the context-specific use of psychometric analysis to ensure generation of valid, objective and consistent assessment during Objective Structured Clinical Examinations. More rigorous large-scale studies (such as quantitative or randomised controlled trials) regarding management of variances amongst examiner scores, impact of moderation on Objective Structured Clinical Examinations, and virtual Objective Structured Clinical Examinations should be conducted.

Objective structured clinical examination vs traditional clinical examination to evaluate students' clinical competence: A systematic review of nursing faculty and students' perceptions and experiences

BACKGROUND

Assessment of clinical competence of nursing students is an essential requirement in professional nursing education. This article summarizes the current published evidence indicating the nursing faculty and students' perceptions and experiences on benefits of OSCE over TCE.

METHODS

A systematic review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) recommendations. Electronic databases including Scopus, Medline, Science Direct, CINAHL, EBSCO, PsychINFO, and Pubmed Central were used to identify relevant articles. The studies published between January 1, 2010 and December 31, 2020 and fourteen full-text articles that met all the inclusion criteria were included in the review.

RESULTS

The review identified five themes namely: a) Student's perception of OSCE; b) student's satisfaction regarding OSCE as an examination approach; c) students' perception of TCE as an examination approach; d) student's level of stress and anxiety towards OSCE vs TCE; and e) faculty member's perception and experience of OSCE.

CONCLUSION

We conclude that OSCE is a more credible assessment format to evaluate the clinical competence of undergraduate nursing students compared to the TCE method.

The use and effect of OSCES in post-registration nurses: An integrative review

BACKGROUND

Objective structured clinical examinations (OSCEs) have been used in the undergraduate medical setting since the 1970s, however little is known about its use and effectiveness in post-registration nurse education.

OBJECTIVES

The purpose of this review was to critically analyse studies using OSCEs in post-registration nurse education and to explore the use and effectiveness of OSCEs in this cohort.

DESIGN

This review was conducted using the Whittemore and Knafl's framework for integrated reviews.

DATA SOURCES

Using the search terms OSCE OR OSCA OR objective structured clinical examination AND nursing NOT undergraduate, a comprehensive review was conducted using CINHAL and MEDLINE.

METHODS

The initial search located 173 studies. After screening and checking eligibility 19 studies were critically appraised. The final number of studies included in this review was 16.

RESULTS

Three themes were generated: (i) Application of OSCEs in post-registration level nursing (ii) OSCEs more than an exam: Learning enhancements and (iii) Participant perceived impact of OSCEs.

CONCLUSION

OSCEs for post-registration nurses have evolved from the original OSCE design and have been adapted to a wide range of clinical settings and specialities. OSCEs were recognized as a valuable learning and assessment tool across the world for post-registration nurses. OSCEs offered strengths in terms of learner satisfaction and increased self-efficacy for this cohort. There was limited evidence supporting the effects of OSCEs within post-registration nursing education in comparison with other educational tools. Further research is needed to explore whether the knowledge gained in OSCEs is translated into clinical practice and whether simulation-based education is more effective in achieving enhanced knowledge compared to traditional-based education. Future research is required using RCT methods to compare the impact of OSCE to traditional-based education.

Introduction of the Objective Structured Clinical Examination in Speech and Language Therapy Education: Student Perspectives

BACKGROUND/AIMS

The Objective Structured Clinical Examination (OSCE) has an established history of assessing clinical competence for medical and allied healthcare professionals. No research has investigated the use of the OSCE within speech and language therapy undergraduate education. This study aimed to evaluate undergraduate speech and language therapy students' perceptions of the OSCE and to determine if perceptions differed depending on stage of undergraduate education.

METHODS

An online survey was distributed to second- and final-year students in a 4-year undergraduate speech and language therapy university programme after completing an OSCE. Quantitative survey data were analysed descriptively and statistically using Fisher's exact tests. Thematic analysis was used to analyse narrative survey comments.

RESULTS

The response rate was 80%. The students perceived the OSCE to be a fair (95%) and meaningful way (97%) of assessing clinical competencies, which provided them with greater confidence for practice (91%). The OSCE was deemed to be less stressful than a written exam (65%). No significant differences were observed between second- and fourth-year students' perceptions. Five major themes included clarity of expectations, time allocated, consistency between examiners, influence on student learning, and personal impact.

DISCUSSION

The survey findings suggest that undergraduate speech and language therapy students across different stages of education and different clinical contexts perceive the OSCE as a fair and meaningful assessment approach. This study provides initial evidence that the OSCE is suitable for assessing clinical competencies specific to the profession of speech and language therapist.

The predictive validity of Multiple Mini Interviews (MMIs) in nursing and midwifery programmes: Year three findings from a cross-discipline cohort study

BACKGROUND

Education literature worldwide is replete with studies evaluating the effectiveness of Multiple Mini Interviews (MMIs) in admissions to medicine but <1% of published studies have been conducted in selection to nursing and midwifery programmes.

OBJECTIVES

To examine the predictive validity of MMIs using end of programme clinical and academic performance indicators of pre-registration adult, child, and mental health nursing and midwifery students.

DESIGN AND SETTING

A cross-sectional cohort study at one university in the United Kingdom.

PARTICIPANTS

A non-probability consecutive sampling strategy whereby all applicants to the September 2015 pre-registration adult, child, mental health nursing and midwifery programmes were invited to participate. Of the 354 students who commenced year one, 225 (64%) completed their three-year programme and agreed to take part (adult 120, child 32, mental health nursing 30 and midwifery 43).

METHODS

All applicants were interviewed using MMIs with six and seven station, four-minute models deployed in nursing and midwifery student selection respectively. Associations between MMI scores and the cross-discipline programme performance indicators available for each student at this university at the end of year three: clinical practice (assessed by mentors) and academic attainment (dissertation mark) were explored using multiple linear regression adjusting for applicant age, academic entry level, discipline and number of MMI stations.

RESULTS

In the adjusted models, students with higher admissions MMI score (at six and seven stations) performed better in clinical practice (p < 0.001) but not in academic attainment (p = 0.122) at the end of their three-year programme.

CONCLUSION

These findings provide the first report of the predictive validity of MMIs for performance in clinical practice using six and seven station models in nursing and midwifery programmes. Further evidence is required from both clinical and academic perspectives from larger, multi-site evaluations.

Simulation videos presented in a blended learning platform to improve Australian nursing students' knowledge of family assessment

The provision of simulation to enhance learning is becoming common practice as clinical placement becomes harder to secure within Bachelor of Nursing programs. The use of simulation videos within a blended learning platform enables students to view best practice and provides relevant links between theory and practice. Four simulation videos depicting family assessment viewed by a cohort of Australian undergraduate nursing students were evaluated. These videos were professionally developed using actors and experienced family nurses. Surveys were used to explore the students' self-assessed knowledge, confidence and learning preferences before and after exposure to blended learning resources. Students' engagement with the simulated videos was captured via the Learning Management System. Time 1 survey was completed by 163 students and Time 2 by 91 students. There was a significant increase in students' perceived knowledge of family theory Item 1 from a mean 4.13 (SD = 1.04) at Time 1 to 4.74 (SD = 0.89) (Z = -4.54 p < 0.001) at Time 2; Item 2- Knowledge of family assessment improved from mean 3.91 (SD = 1.02) at Time 1 to 4.90 (SD = 0.67) (Z = -7.86 p < 0.001) at Time 2. Also a significant increase in their confidence undertaking family assessment Item 5 from a mean 3.55 (SD = 1.14) at Time 1 to 4.44 (SD = 0.85) (Z = -6.12 p < 0.001) at Time 2. The students watched the videos an average of 1.9 times. The simulated videos as a blended learning resource increases the students' understanding of family assessment and is worth incorporating into future development of courses.

An Innovative Evaluation in Fundamental Nursing Curriculum for Novice Nursing Students: An Observational Research

BACKGROUND

<Fundamental Nursing> is the first specialized curriculum for nursing students, laying a foundation for nursing education. <Fundamental Nursing> includes many important basic nursing skills, which suggests the traditional examination could not evaluate how well students learn. Since the OSCE (objective structured clinical examination) developed, it has been used among newly graduated nurses, postgraduate nursing students and achieved some success. OSCE might be a better way to evaluate and reflect students' fundamental nursing skills and comprehensive abilities, although few studies have put it into practice.

OBJECTIVES

To build an examination model relating to <Fundamental Nursing> based on OSCE, and put this examination model into practice, and then evaluate the reliability and validity of this examination model.

DESIGN

This study is to design an examination model based on literature and teaching purposes of <Fundamental Nursing>. A quantitative descriptive survey methodology was used to obtain the students' outcome and the views of the students on the process and outcome.

SETTINGS

This study was conducted in the nursing school of one comprehensive university in Beijing, China.

PARTICIPANTS

99 undergraduate nursing students in second year participated in this study.

METHODS

The OSCE consisted of 4 interactive simulated patient stations, 2 operating stations without standard patients, 2 theoretical stations. We designed 4 standardized process-based case scenarios and corresponding scripts pertinent to each station. 4 standard patients and 8 teachers of <Fundamental Nursing> were recruited, trained and evaluated before the implementation of objective structured clinical examination.

RESULTS

The average score of overall fundamental nursing skills was 90.51 ± 4.00. The Cronbach alpha was 0.665, demonstrating the OSCE is reliable. The content validity of each station varies from 0.90-1, and the criterion validity was promising (R = 0.661, p = 0.000). Also, most students were in favor of OSCE to evaluate in <Fundamental Nursing>.

CONCLUSIONS

The OSCE is a reliable and valid assessment tool in <Fundamental Nursing>. Meanwhile, it could inspire students' critical thinking and promote students' communication skills. Through successive application of OSCE in nursing education from the very beginning, it might also have some potential effect on the clinical performance.

Enhancing OSCE preparedness with video exemplars in undergraduate nursing students. A mixed method study

BACKGROUND

Objective structured clinical examinations (OSCEs) are designed to assess clinical skill performance and competency of students in preparation for 'real world' clinical responsibilities. OSCEs are commonly used in health professional education and are typically associated with high levels of student anxiety, which may present a significant barrier to performance. Students, including nursing students, have identified that flexible access to exemplar OSCEs might reduce their anxiety and enable them to better prepare for such examinations.

AIM

To implement and evaluate an innovative approach to preparing students for OSCEs in an undergraduate (registration) acute care nursing course.

METHOD

A set of digitized OSCE exemplars were prepared and embedded in the University-based course website as part of usual course learning activities. Use of the exemplars was monitored, pre and post OSCE surveys were conducted, and qualitative data were collected to evaluate the approach. OSCE grades were also examined.

FINDINGS

The online OSCE exemplars increased self-rated student confidence, knowledge, and capacity to prepare and provided clarity around assessment expectations. OSCE exemplars were accessed frequently and positively received; but did not impact on performance.

CONCLUSION

Video exemplars aid student preparation for OSCEs, providing a flexible, innovative and clear example of the assessment process. Video exemplars improved self-rated student confidence and understanding of performance expectations, leading to increased engagement and reduced anxiety when preparing for the OSCE, but not overall OSCE performance. Such OSCE exemplars could be used to increase staff capacity and improve the quality of the student learning experience.

Does quantity ensure quality? Standardized OSCE-stations for outcome-oriented evaluation of practical skills at different medical faculties

BACKGROUND

Practical skills are often assessed using Objective Structured Clinical Skill Exams (OSCE). Nevertheless, in Germany, interchange and agreement between different medical faculties or a general agreement on the minimum standard for passing is lacking.

METHODS

We developed standardized OSCE-stations for assessing structured clinical examination of knee and shoulder joint with identical checklists and evaluation standards. These were implemented into the OSCE-course at five different medical faculties. Learning objectives for passing the stations were agreed beforehand. At each faculty, one reference examiner scored independently of the local examiner. Outcome of the students at the standardized station was compared between faculties and correlated to their total outcome at the OSCE, to their results at the Part One of the National Medical Licensing Examination as a reference test during medical studies and to their previous amount of lessons in examining joints.

RESULTS

Comparing the results of the reference examiner, outcome at the station differed significantly between some of the participating medical faculties. Depending on the faculty, mean total results at the knee-examination-station differed from 64.4% to 77.9% and at the shoulder-examination-station from 62.6% to 79.2%. Differences were seen in knowledge-based items and also in competencies like communication and professional manner. There was a weak correlation between outcome at the joint-examination-OSCE-station and Part One of the National Medical Licensing Examination, and a modest correlation between outcome at the joint-examination-station and total OSCE-result. Correlation to the previous amount of lessons in examining joint was also weak.

CONCLUSION

Although addressing approved learning objectives, different outcomes were achieved when testing a clinical skill at different medical faculties with a standardized OSCE-station. Results can be used as a tool for evaluating lessons, training and curricula at the different sites. Nevertheless, this study shows the importance of information exchange and agreement upon certain benchmarks and evaluation standards when assessing practical skills.

Developing a Framework for Objective Structured Clinical Examinations Using the Nominal Group Technique

Objective. To use the nominal group technique to develop a framework to improve existing and develop new objective structured clinical examinations (OSCEs) within a four-year bachelor of pharmacy course. Design. Using the nominal group technique, a unique method of group interview that combines qualitative and quantitative data collection, focus groups were conducted with faculty members, practicing pharmacists, and undergraduate pharmacy students. Five draft OSCEs frameworks were suggested and participants were asked to generate new framework ideas. Assessment. Two focus groups (n=9 and n=7) generated nine extra frameworks. Two of these frameworks, one from each focus group, ranked highest (mean scores of 4.4 and 4.1 on a 5-point scale) and were similar in nature. The project team used these two frameworks to produce the final framework, which includes an OSCE in every year of the course, earlier implementation of teaching OSCEs, and the use of independent simulated patients who are not examiners. Conclusions. The new OSCE framework provides a consistent structure from course entry to exit and ensures graduates meet internship requirements.

OSCE best practice guidelines-applicability for nursing simulations

Background

Objective structured clinical examinations (OSCEs) have been used for many years within healthcare programmes as a measure of students' and clinicians' clinical performance. OSCEs are a form of simulation and are often summative but may be formative. This educational approach requires robust design based on sound pedagogy to assure practice and assessment of holistic nursing care. As part of a project testing seven OSCE best practice guidelines (BPGs) across three sites, the BPGs were applied to an existing simulation activity. The aim of this study was to determine the applicability and value of the OSCE BPGs in an existing formative simulation.

Methods

A mixed methods approach was used to address the research question: in what ways do OSCE BPGs align with simulations. The BPGs were aligned and compared with all aspects of an existing simulation activity offered to first-year nursing students at a large city-based university, prior to their first clinical placement in an Australian healthcare setting. Survey questions, comprised of Likert scales and free-text responses, used at other sites were slightly modified for reference to simulation. Students' opinions about the refined simulation activity were collected via electronic survey immediately following the simulation and from focus groups. Template analysis, using the BPGs as existing or a priori thematic codes, enabled interpretation and illumination of the data from both sources.

Results

Few changes were made to the existing simulation plan and format. Students' responses from surveys (n = 367) and four focus groups indicated that all seven BPGs were applicable for simulations in guiding their learning, particularly in the affective domain, and assisting their perceived needs in preparing for upcoming clinical practice.

Discussion

Similarities were found in the intent of simulation and OSCEs informed by the BPGs to enable feedback to students about holistic practice across affective, cognitive and psychomotor domains. The similarities in this study are consistent with findings from exploring the applicability of the BPGs for OSCEs in other nursing education settings, contexts, universities and jurisdictions. The BPGs also aligned with other frameworks and standards often used to develop and deliver simulations.

Conclusions

Findings from this study provide further evidence of the applicability of the seven OSCE BPGs to inform the development and delivery of, in this context, simulation activities for nurses. The manner in which simulation is offered to large cohorts requires further consideration to meet students' needs in rehearsing the registered nurse role.