Developing a community of practice for quality assurance within healthcare assessment

Strengthening the planning and design of Objective Structured Clinical Examinations

Background

Although Objective Structured Clinical Examinations (OSCEs) offer innovative, objective, and fair methods of clinical assessment, their quality is compromised by poor planning and design.

Aim

This study aimed to describe the development and present evidence-based recommendations on strengthening the planning and design of OSCEs for a South African public College of Nursing.

Setting

A South African public College of Nursing.

Methods

Recommendations were developed based on synthesising two sets of qualitative data. Set 1 included two main themes with each of the four sub-themes related to barriers and facilitators towards quality in OSCE designs from 14 nurse educator interviews and 15 external moderator reports. Set 2 included 12 quality measures to be adopted in the quality design of OSCEs derived from an integrative literature review. The draft recommendations were reviewed by eight experts to be finalised.

Results

Seven recommendations were developed for strengthening OSCEs' planning and design, related to: (1) policy framework, standard operating procedures and stakeholder code of conduct; (2) blueprinting and mapping of the OSCE content; (3) developing a bank of OSCE stations; (4) scoring rubric and standard-setting method selection; (5) examiners and standardised patients' recruitment and training; (6) venue selection; and (7) station piloting.

Conclusion

The seven developed recommendations can strengthen the quality of OSCEs in the South African public College of Nursing context.

Contribution

The developed recommendations can assist nurse educators in planning and designing to conduct quality OSCEs following piloting and implementation.

Experiences of a community of research practice: a service evaluation

BACKGROUND

A community of research practice (CRP) was established to increase research capacity and provide learning opportunities and networking for healthcare practitioners, working within a Children's Care Group. The CRP aimed to engage research-interested practitioners in research to develop their skills and confidence, encourage networking, and build research capacity.

AIM

To report the results of a service evaluation that was undertaken to review the CRP's value in practice.

DISCUSSION

Thematic analysis revealed four themes - 'positive environment', 'confidence', 'professional development' and 'networking' - highlighting benefits from the CRP, alongside the challenges encountered.

CONCLUSION

This article highlights the significant contribution of CRP for practitioners in the context of an innovative organisation with a supportive culture.

IMPLICATIONS FOR PRACTICE

A CRP empowers healthcare practitioners to engage with research while in clinical practice, enabling increased research confidence, and the development of research skill and knowledge and enhanced networking. CRP can potentially influence recruitment and retention.

Virtual OSCE Delivery and Quality Assurance During a Pandemic: Implications for the Future

Background

During 2020, the COVID-19 pandemic caused worldwide disruption to the delivery of clinical assessments, requiring medicals schools to rapidly adjust their design of established tools. Derived from the traditional face-to-face Objective Structured Clinical Examination (OSCE), the virtual OSCE (vOSCE) was delivered online, using a range of school-dependent designs. The quality of these new formats was evaluated remotely through virtual quality assurance (vQA). This study synthesizes the vOSCE and vQA experiences of stakeholders from participating Australian medical schools based on a Quality framework.

Methods

This study utilized a descriptive phenomenological qualitative design. Focus group discussions (FGD) were held with 23 stakeholders, including examiners, academics, simulated patients, professional staff, students and quality assurance examiners. The data was analyzed using a theory-driven conceptual Quality framework.

Results

The vOSCE was perceived as a relatively fit-for purpose assessment during pandemic physical distancing mandates. Additionally, the vOSCE was identified as being value-for-money and was noted to provide procedural benefits which lead to an enhanced experience for those involved. However, despite being largely delivered fault-free, the current designs are considered limited in the scope of skills they can assess, and thus do not meet the established quality of the traditional OSCE.

Conclusions

Whilst virtual clinical assessments are limited in their scope of assessing clinical competency when compared with the traditional OSCE, their integration into programs of assessment does, in fact, have significant potential. Scholarly review of stakeholder experiences has elucidated quality aspects that can inform iterative improvements to the design and implementation of future vOSCEs.

Changes to objective structured clinical examinations (OSCE) at Australian medical schools in response to the COVID-19 pandemic

INTRODUCTION

Objective Structured Clinical Examinations (OSCE) are used to assess clinical skills. We investigated how exit OSCEs changed in Australian medical schools in response to the COVID-19 pandemic.

MATERIALS AND METHODS

The lead ACCLAiM assessment academic from 12 eligible Australian medical school members of the Australian Collaboration for Clinical Assessment in Medicine (ACCLAiM) received a 45-item semi-structured online questionnaire.

RESULTS

All schools (12/12) responded. Exit OSCEs were not used by one school in 2019, and 3/11 schools in 2020. Of eight remaining schools, four reduced station numbers and testing time. The minimum OSCE testing time decreased from 64 min in 2019 to 54 min in 2020. Other modifications included: a completely online 'e-OSCE' (n = 1); hybrid delivery (n = 4); stations using: videos of patient encounters (n = 3), telephone calls (n = 2), skill completion without face-to-face patient encounters (n = 3). The proportion of stations involving physical examination reduced from 33% to 17%. Fewer examiners were required, and university faculty staff formed a higher proportion of examiners.

CONCLUSIONS

All schools changed their OSCEs in 2020 in response to COVID-19. Modifications varied from reducing station numbers and changing delivery methods to removing OSCE and complete assessment re-structuring. Several innovative methods of OSCE delivery were implemented to preserve OSCE validity and reliability whilst balancing feasibility.

Has the OSCE Met Its Final Demise? Rebalancing Clinical Assessment Approaches in the Peri-Pandemic World

The Objective Structured Clinical Examination (OSCE) has been traditionally viewed as a highly valued tool for assessing clinical competence in health professions education. However, as the OSCE typically consists of a large-scale, face-to-face assessment activity, it has been variably criticized over recent years due to the extensive resourcing and relative expense required for delivery. Importantly, due to COVID-pandemic conditions and necessary health guidelines in 2020 and 2021, logistical issues inherent with OSCE delivery were exacerbated for many institutions across the globe. As a result, alternative clinical assessment strategies were employed to gather assessment datapoints to guide decision-making regarding student progression. Now, as communities learn to "live with COVID", health professions educators have the opportunity to consider what weight should be placed on the OSCE as a tool for clinical assessment in the peri-pandemic world. In order to elucidate this timely clinical assessment issue, this qualitative study utilized focus group discussions to explore the perceptions of 23 clinical assessment stakeholders (examiners, students, simulated patients and administrators) in relation to the future role of the traditional OSCE. Thematic analysis of the FG transcripts revealed four major themes in relation to participants' views on the future of the OSCE vis-a-vis other clinical assessments in this peri-pandemic climate. The identified themes are (a) enduring value of the OSCE; (b) OSCE tensions; (c) educational impact; and (d) the importance of programs of assessment. It is clear that the OSCE continues to play a role in clinical assessments due to its perceived fairness, standardization and ability to yield robust results. However, recent experiences have resulted in a diminishing and refining of its role alongside workplace-based assessments in the new, peri-pandemic programs of assessment. Future programs of assessment should consider the strategic positioning of the OSCE within the context of utilizing a range of tools when determining students' clinical competence.

Healthcare professionals' longitudinal perceptions of group phenomena as determinants of self-assessed learning in organizational communities of practice

BACKGROUND

Given the importance of continuous learning as a response to the increasing complexity of health care practice, there is a need to better understand what makes communities of practice in health effective at fostering learning. Despite the conceptual stance that communities of practice facilitate individual learning, the scientific literature does not offer much evidence for this. Known factors associated with the effectiveness of communities of practice - such as collaboration, psychological safety within the community, and commitment to the community - have been studied in cross-sectional qualitative designs. However, no studies to date have used a quantitative predictive design. The objective of this study is to assess how members of a community of practice perceive interactions among themselves and determine the extent to which these interactions predict self-assessed learning over time.

METHODS

Data was collected using validated questionnaires from six communities of practice (N = 83) in four waves of measures over the course of 36 months and was analysed by means of General Estimating Equations. This allowed to build a longitudinal model of the associations between perceptions of collaboration, psychological safety within the community, commitment to the community and self-assessed learning over time.

RESULTS

Perception of collaboration in the community of practice, a personal sense of psychological safety and a commitment to the community of practice are predictors longitudinally associated with self-assessed learning.

CONCLUSIONS

In terms of theory, conceptual links can be made between intensity of collaboration and learning over time in the context of a community of practice. Recent work on psychological safety suggests that it is still unclear whether psychological safety acts as a direct enhancer of learning or as a remover of barriers to learning. This study's longitudinal results suggest that psychological safety may enhance how and to what extent professionals feel they learn over time. Commitment towards the community of practice is a strong predictor of learning over time, which hints at differential effects of affective, normative and continuance commitment. Communities of practice can therefore apply these findings by making collaboration, psychological safety, commitment and learning regular reflexive topics of discussion.