Improving the content and face validity of OSCE assessment marking criteria on an undergraduate midwifery programme: A quality initiative

Longitudinal assessment of undergraduate dental students: Building evidence for validity

PURPOSE

To investigate the content and criterion validity, and reliability of longitudinal clinical assessment of undergraduate dental student clinical competence by determining patterns of clinical performance and comparing them with validated standalone undergraduate examinations.

METHODS

Group-based trajectory models tracking students' clinical performance over time were produced from LIFTUPP© data for three dental student cohorts (2017-19; n = 235) using threshold models based on the Bayesian information criterion. Content validity was investigated using LIFTUPP© performance indicator 4 as the threshold for competence. Criterion validity was investigated using performance indicator 5 to create distinct trajectories of performance before linking and cross-tabulating trajectory group memberships with a 'top 20%' performance in the final Bachelor of Dental Surgery (BDS) examinations. Reliability was calculated using Cronbach's alpha.

RESULTS

Threshold 4 models showed all students followed a single upward trajectory in all three cohorts, showing clear progression in competence over three clinical BDS years. A threshold 5 model produced two distinct trajectories, and in each cohort a 'better performing' trajectory was identified. Students allocated to the 'better performing' trajectories scored higher on average in the final examinations for cohort 2 (29% vs 18% (BDS4); 33% vs. 15% (BDS5)) and cohort 3 (19% vs. 16% (BDS4); 21% vs. 16% (BDS5)). Reliability for the undergraduate examinations was high for all three cohorts (≥0.8815) and did not change appreciably when longitudinal assessment was included.

CONCLUSIONS

There is some evidence to support that longitudinal data have a degree of content and criterion validity for assessing the development of clinical competence in undergraduate dental students, which should increase confidence in decisions based on these data. The findings also provide a good foundation for subsequent research.

Midwifery students' experiences of objective structured clinical examinations: A qualitative evidence synthesis

OBJECTIVES

To identify and synthesise the evidence of midwifery students' views and experiences of Objective Structured Clinical Examinations (OSCEs).

DESIGN

A qualitative evidence synthesis was undertaken using Thomas and Harden's synthesis methods and reported according to the ENTEREQ guidelines.

DATA SOURCES

PubMED, CINAHL Plus, Embase, ERIC, PsycINFO, Social Science Citation Index (via Web of Science) and Scopus were searched from inception to 19th February 2020 and an updated search was performed on 12th May 2021 to identify qualitative and mixed method studies, reporting qualitative data of undergraduate midwifery students' views and experiences of OSCEs.

REVIEW METHODS

Studies were independently screened by title and abstract according to a predetermined inclusion and exclusion criteria. Full text review was then conducted independently by two authors. Studies providing qualitative data on undergraduate midwifery students' views and experiences of OSCEs during their midwifery registration education programmes were included. Quality appraisal was performed using the Critical Appraisal Skills Programme (CASP) tool. Data synthesis was performed using Thomas and Harden's thematic synthesis method.

RESULTS

Eleven studies were identified and included. Thematic synthesis identified 13 descriptive themes which translated into four analytical themes that represented midwifery students' views and experiences of OSCEs: the preparation phase, the assessment phase, the feedback phase and the translation phase.

CONCLUSIONS

This synthesis offers insight into undergraduate midwifery students' views and experiences of OSCEs and how they are experienced across four phases. This is determined to be a useful structure to guide midwifery educators and researchers in their consideration of the student experience of OSCEs in the future. Midwifery students in this synthesis reported high levels of nervousness, anxiety, and stress during the OSCE process. The findings reveal that there are a number of controllable factors impacting on midwifery students' feelings of safety in the assessment environment, and that these should be addressed to maximise the advantages that the OSCE has to offer in terms of learning and preparation for clinical practice.

Training needs assessment for practicing pediatric critical care nurses in Malawi to inform the development of a specialized master's education pathway: a cohort study

Background

Significant improvements in under-five mortality in Malawi have been demonstrated over the past thirty years; however, Malawian healthcare remains with gaps in availability and access to quality pediatric critical care nursing training and education. To improve expertise of pediatric critical care nurses in Malawi, Kamuzu University of Health Sciences (KUHeS), Queen Elizabeth Central Hospital (QECH), and Mercy James Center (MJC) entered a partnership with Seed Global Health, a US non-governmental organization. A needs assessment was conducted to understand the training needs of nurses currently working in pediatric critical care and in preparation for the development of a specialized Master’s in Child Health pathway in Pediatric Critical Care (PCC) Nursing at KUHeS.

Methods

The needs assessment was completed using a survey questionnaire formatted using an ABCDE (Airway, Breathing, Circulation, Disability, and Exposure) framework. The questionnaire had Likert scale and yes/no questions. Data was manually entered into excel and was analyzed using descriptive statistics.

Results

One hundred and fifty-three nurses at QECH and MJC responded to the survey. Most nurses were between the ages of 25 and 35 years (N = 98, 64%), female (N = 105, 69%), and held either a Bachelors (N = 72, 47%) or diploma (N = 70, 46%) in nursing. Nurses had high rates of confidence in certain skills: airway management (N = 120, 99%), breathing assessment & management (N = 153, 100%). However, nurses demonstrated little to no confidence in areas such as: mechanical ventilation (N = 68, 44%), ECG evaluation (N = 74, 48%), and arterial blood gas collection & interpretation (N = 49, 32%).

Conclusion

It is important to identify priority areas for training and skills development to address in the PCC master’s within the child health pathway at KUHeS. Ideally this partnership will produce practice-ready PCC nurses and will establish a recognized PCC nursing workforce in Malawi.

A systematic review and meta-analysis of measurement properties of objective structured clinical examinations used in physical therapy licensure and a structured review of licensure practices in countries with well-developed regulation systems

BACKGROUND

The Objective Structured Clinical Examination (OSCE) is a commonly used tool internationally to assess clinical competency. Physical therapy (PT) licensure processes vary internationally. The OSCE is the tool used in Canada to assess clinical competency for PT graduates seeking licensure. Previous studies that examined the measurement properties of OSCEs present contradictory results.

OBJECTIVES

The first objective was to investigate the reliability and validity of OSCEs when administered to PTs during their education or as part of a licensure process. The second objective was to conduct a structured review to report PT educational and licensing components and policies in 17 countries with well-developed PT regulation systems.

METHODS

An electronic search was performed in four databases from inception to 31st March 2021 to identify relevant articles. Two reviewers performed the critical appraisal of the included studies using a validated quality assessment tool. We deployed a random effects meta-analysis on reliability and validity estimates of OSCEs and examined sources of heterogeneity with univariate meta-regressions. We searched websites of professional regulatory bodies and associations for data on educational and licencing components and policies. Educational and licensing components across countries were synthesized descriptively.

RESULTS

A pooled estimate of Cronbach's alpha of 0.55, (95% CI: 0.41, 0.67) was determined for OSCEs. The pooled estimate of Intraclass Correlation Coefficient (ICC) between assessors was 0.77 (95% CI: 0.70, 0.83). The pooled estimate of Pearson Correlation between multiple OSCE stations' scores was 0.27 (95% CI: 0.15, 0.39); and between each station score and the total score was 0.71 (95% CI: 0.61, 0.79). The pooled estimates for kappa Coefficients were 0.75 (95% CI: 0.58, 0.86) and 0.84, (95% CI: 0.72, 0.91) for intra-rater and inter-rater reliability of the standardised patient respectively. From the 17 included countries, Canada (excluding Quebec) was the only country that required both a clinical and written competency exam following graduation from an accredited PT program. Two countries (USA, UAE) required a written competency exam. The remaining 14 countries did not require an additional competency examination after completion of degree requirements from an accredited program.

CONCLUSIONS

We found weak evidence that OSCE examinations items are internally consistent when used to assess PTs. Canada (excluding Quebec) is the only country out of 17 implementing a national clinical competency examination for their PT graduates to achieve licensure after completing professional degree requirements.

Design of a nursing objective structured clinical examination of a first-year clinical practice program

OBJECTIVE

The aim of the present study was to design a content-valid nursing objective structured clinical examination attending a first-year clinical nursing practice program.

METHOD

The examination was designed following a procedure based on the consensus of experts which was comprised of three phases: selection of the activities in which students should be competent according to the learning outcomes of the course, clinical case design, and integration of the clinical cases designed into the stations of the test.

RESULTS

Of the 44 surveys submitted for the design of the stations, 37 were answered, of which 31 respondents met the inclusion criteria of the panel of experts. The activities on which the experts reached the highest degrees of consensus were: basic physical assessment and monitoring of vital signs, assessment of hygiene and skin status, ability to develop care plans, management of safety principles in administration of medication and administration of oral medication. Based on the selected activities, the experts developed 20 clinical cases, from which a four-station nursing objective structured clinical examination was designed.

CONCLUSION

The structured methodology based on the design of experts enabled the design of a content-valid objective structured clinical examination appropriate for the evaluation of the learning outcomes achieved by the students attending a clinical practice program.

Improving the assessment of communication competencies in a national licensing OSCE: lessons learned from an experts' symposium

BACKGROUND

As the communication competencies of physicians are crucial for providing optimal patient care, their assessment in the context of the high-stakes Objective Structured Clinical Examination (OSCE) is of paramount importance. Despite abundant literature on the topic, evidence-based recommendations for the assessment of communication competencies in high stakes OSCEs are scarce. As part of a national project to improve communication-competencies assessments in the Swiss licensing exam, we held a symposium with national and international experts to derive corresponding guidelines.

METHODS

Experts were invited on account of their recognized expertise either in teaching or assessing communication competencies, or in conducting national high-stakes OSCEs. They were asked to propose concrete solutions related to four potential areas for improvement: the station design, the rating tool, the raters' training, and the role of standardized patients. Data gene.rated in the symposium was available for analysis and consisted of video recordings of plenary sessions, of the written summaries of group work, and the cards with participants' personal take-home messages. Data were analyzed using a thematic analysis approach.

RESULTS

Nine major suggestions for improving communication-competencies assessments emerged from the analysis and were classified into four categories, namely, the roles of the OSCE scenarios, rating tool, raters' training, and simulated patients.

CONCLUSION

In the absence of established evidence-based guidelines, an experts' symposium facilitated the identification of nine practical suggestions for improving the assessment of communication competencies in the context of high-stakes OSCEs. Further research is needed to test effectiveness of the suggestions and how they contribute to improvements in the quality of high-stakes communication-competencies assessment.

Assessing student pharmacist communication skills: Development and implementation of a communication rubric

PROBLEM DESCRIPTION

Communication between healthcare providers and those they serve has become increasingly important as the need to prevent and manage chronic conditions has increased. To adequately learn these skills, future healthcare providers must be trained and assessed prior to entry into practice.

QUALITY IMPROVEMENT METHODS

With the implementation of an integrated curriculum, the Harrison School of Pharmacy developed a communication rubric to assess student pharmacists' patient communication skills. Developed over five stages, the communication rubric contained seven criteria with three standards or performance levels. Prior to finalizing the rubric, faculty tested the rubric with 56 standardized persons (SPs) to determine its readability, functionality, and grader consistency, and also collected student perceptions of the rubric for feedback purposes.

RESULTS OF CQI INQUIRY

At the conclusion of the 2017/18 academic year, 66.5% of second- and third-professional year students (P2s and P3s) somewhat agreed to strongly agreed that reviewing the rubric helped improve their overall patient communication; 65% of students felt that feedback was more substantial than the previous checklist. Of the P3s surveyed, 73% indicated that they preferred the new rubric.

INTERPRETATION AND DISCUSSION

The integration of the new communication rubric allowed for richer, and more meaningful feedback. The addition of multiple performance levels allowed students to focus on specific areas for improvement.

CONCLUSIONS

The communication rubric was successfully integrated into the curriculum, particularly with objective structured clinical examinations. With its integration into other curriculum areas, the communication skills of student pharmacists may be further enhanced.

Experiences of nursing students and examiners with the Objective Structured Clinical Examination method in physical assessment education: A mixed methods study

This study explores the experiences of undergraduate nursing students and examiners with the Objective Structured Clinical Examination (OSCE) as an evaluation of physical assessment skills. A mixed methods approach captures participants' perceptions and experiences with the OSCE. The sample consisted of 55 students enrolled in the physical assessment course and eight external examiners. Data were collected using a standardized questionnaire and focus group interviews, using semi-structured questions, recorded and transcribed for thematic analysis. Quantitative data were analyzed using SPSS software. Results indicated that participants regarded the OSCE as a valuable assessment tool that enhances in-depth learning and preparation of students for clinical practice. Participants also perceived the exam to be a stressful experience, citing the need for more time on examination stations. Recommendations for improving the OSCE process included extending preparation prior to the exam, training of simulated patients, adopting a blended learning model, and upgrading the setting of the exam. This paper supports evidence of the value of the OSCE as an appraisal of physical assessment skills in undergraduate nursing education.

How to maintain equity and objectivity in assessing the communication skills in a large group of student nurses during a long examination session, using the Objective Structured Clinical Examination (OSCE)

BACKGROUND

While development, testing, and innovation of the Objective Structured Clinical Examination (OSCE) are common in the international literature, studies from the United States of America (USA), Australia, and the United Kingdom (UK) predominate. There is little known about OSCE use in European countries, such as Italy, where other than cost analysis, there is little reporting of OSCE use or validation.

OBJECTIVES

This paper reports on one Italian initiative, which evaluated the equity and objectivity of the OSCE method of assessing communication skills.

DESIGN

An OSCE method was used to assess the communication skills of first-year students of the Degree Course in Nursing. A method of simulation was implemented through role-playing with standardized patients. An observational method was used to collect data.

PARTICIPANTS AND SETTINGS

Four hundred and twenty-one first-year undergraduate nursing students at one university site in Italy took part.

METHODS

Ten examination sessions were carried out. The students' performances were assessed by two examiners who used a structured observation grid and conducted their assessment separately. A situation simulated by four nurses with experience as actors was used as the topic for the students' examination.

RESULTS

Calculation of the daily rate of students who passed the examination revealed a random distribution over time. The nonparametric correlation indexes referring to the assessments and to the scores assigned by the two examiners proved statistically significant (P≤0.001).

CONCLUSIONS

The study confirmed the validity of the OSCE method in ensuring equity and objectivity of communication skills assessment in a large population of nursing students for the purpose of certification throughout the duration of the examination. This has important implications for nurse education and practice as the extent to which OSCE approaches, while deemed objective, are culturally sensitive or valid and reliable across cultures is not clear. This is something that requires further research and examination in this field.

Reliability and validity of OSCE checklists used to assess the communication skills of undergraduate medical students: A systematic review

OBJECTIVES

To explore inter-rater agreement between reviewers comparing reliability and validity of checklist forms that claim to assess the communication skills of undergraduate medical students in Objective Structured Clinical Examinations (OSCEs).

METHODS

Papers explaining rubrics of OSCE checklist forms were identified from Pubmed, Embase, PsycINFO, and the ProQuest Education Databases up to 2013. Included were those studies that report empirical validity or reliability values for the communication skills assessment checklists used. Excluded were those papers that did not report reliability or validity.

RESULTS

Papers focusing on generic communication skills, history taking, physician-patient communication, interviewing, negotiating treatment, information giving, empathy and 18 other domains (ICC -0.12-1) were identified. Regarding the validity and reliability of the communication skills checklists, agreement between reviewers was 0.45.

CONCLUSIONS

Heterogeneity in the rubrics used in the assessment of communication skills and a lack of agreement between reviewers makes comparison of student competences within and across institutions difficult.

PRACTICE IMPLICATIONS

Consideration should be afforded to the adoption of a standardized measurement instrument to assess communication skills in undergraduate medical education. Future research will focus upon evaluating the potential impact of adoption of a standardized measurement instrument.

Educational efficacy of objective structured clinical examination on clinical training of traditional Chinese medicine - A qualitative study

OBJECTIVE

We propose applying objective structured clinical examinations (OSCEs) and discuss the educational efficacy of such examinations regarding the clinical competence of traditional Chinese medicine (TCM) practitioners.

DESIGN

TCM OSCEs were implemented for evaluation and instruction from 2011 to 2013. Trainees received feedback from clinician-educators and standardized patients. Trainees' survey data were extracted from post-OSCE questionnaires and interviews to analyze TCM OSCEs.

RESULTS

Five TCM OSCEs were administered, and the educational backgrounds of the 37 participants were analyzed. According to analysis of the questionnaires, all trainees agreed OSCEs were beneficial. In interviews, trainees expressed appreciation for the direct, real-time feedback during the OSCE and felt it closely resembled actual clinical work. However, the simulation models of OSCEs must be upgraded.

CONCLUSIONS

OSCEs can be used in evaluating, teaching, and certifying TCM clinical competencies to improve the quality of TCM practices. The patient-centered training aspect of TCM OSCEs is particularly helpful for participants.

An innovative approach to enhance dermatology competencies for advanced practice nurses: service–learning with a migrant farm worker health clinic

The purpose of this article is to describe a novel service–learning opportunity for graduate nursing students that promotes competency in dermatology. A hybrid service–learning course with online didactic content is described, along with tools for evaluation of dermatology competencies. Student evaluation of the course is discussed, and selected research articles are reviewed. Advanced practice nursing and medical education frequently does not adequately prepare primary care providers to be competent in the assessment and management of dermatologic conditions. Embedding dermatology content in a service–learning program can optimize the provision of care, strengthen competencies in dermatology and inter-professional care, and allow students to gain a deeper understanding of the population with which they work. The innovative service–learning program presented is a model for advanced practice nursing education. Tools for evaluating clinical competency and courses often need validation.

'I found the OSCE very stressful': student midwives' attitudes towards an objective structured clinical examination (OSCE)

BACKGROUND

The Objective Structured Clinical Examination (OSCE) has become widely accepted as a strategy for assessing clinical competence in nursing and midwifery education and training. There is a dearth of information, however, on the OSCE procedure from the perspective of midwifery students. In particular, there is an absence of an objective quantification of midwifery students' attitudes towards the OSCE.

OBJECTIVES

The objective of this study is to report the conduct and findings of a survey of midwifery students' attitudes towards a Lactation and Infant Feeding OSCE and to consider these attitudes in the context of the international literature and the empirical evidence base.

METHODS

A descriptive survey design using an 18-item Likert (1 to 5 point) scale was used to capture the relevant data. Potential participants were 3rd year midwifery students who had undertaken a Lactation and Infant Feeding OSCE (n=35) in one School of Nursing & Midwifery in the Republic of Ireland. Survey responses were analysed using the Statistical Package for the Social Sciences Version 18.

RESULTS

Thirty-three students completed the survey providing a 94% response rate. Midwifery students' attitudes towards individual aspects of the OSCE varied. Overall, midwifery students were neutral/unsure of the OSCE as a strategy for assessing clinical competence (mean 3.3). Most agreed that the examiner made them feel at ease (mean 3.94). Contrastingly this does not appear to appease student nerves and stress as the majority agreed that the OSCE evokes nervousness (mean 4.27) and stress (mean 4.30). Midwifery students, overall, disagreed that the OSCE reflected real life clinical situations (mean 2.48). Midwifery students were neutral/unsure that the OSCE provided an opportunity to show their practical skills (mean 3.36).

CONCLUSION

The findings of this study identified that midwifery students were neutral/unsure of the OSCE as a strategy for assessing clinical competence. This has relevance for OSCE development at the authors' institution. The results suggest the need to explore further why students responded in this way. This will assist to develop this OSCE further to ensure that it becomes a positive assessment process for midwifery students and for student learning as they progress through their midwifery education and training.