Self-assessment in CPD: lessons from the UK undergraduate and postgraduate education domains

Assessment Practices in Continuing Professional Development Activities in Health Professions: A Scoping Review

INTRODUCTION

In continuing professional development (CPD), educators face the need to develop and implement innovative assessment strategies to adhere to accreditation standards and support lifelong learning. However, little is known about the development and validation of these assessment practices. We aimed to document the breadth and depth of what is known about the development and implementation of assessment practices within CPD activities.

METHODS

We conducted a scoping review using the framework proposed by Arksey and O'Malley (2005) and updated in 2020. We examined five databases and identified 1733 abstracts. Two team members screened titles and abstracts for inclusion/exclusion. After data extraction, we conducted a descriptive analysis of quantitative data and a thematic analysis of qualitative data.

RESULTS

A total of 130 studies were retained for the full review. Most reported assessments are written assessments (n = 100), such as multiple-choice items (n = 79). In 99 studies, authors developed an assessment for research purpose rather than for the CPD activity itself. The assessment validation process was detailed in 105 articles. In most cases, the authors examined the content with experts (n = 57) or pilot-tested the assessment (n = 50). We identified three themes: 1-satisfaction with assessment choices; 2-difficulties experienced during the administration of the assessment; and 3-complexity of the validation process.

DISCUSSION

Building on the adage "assessment drives learning," it is imperative that the CPD practices contribute to the intended learning and limit the unintended negative consequences of assessment. Our results suggest that validation processes must be considered and adapted within CPD contexts.

Measuring medical students' reflection on their learning: modification and validation of the motivated strategies for learning questionnaire (MSLQ)

BACKGROUND

Reflection on learning is an essential component of effective learning. Deconstructing the components of reflection on learning using a self-regulated learning (SRL) framework, allows the assessment of students' ability to reflect on their learning. The aim of this study was to validate an instrument to measure medical students' reflection on their learning.

METHODS

A systematic search was conducted to identify the most suitable instrument to measure students' reflection on their learning based on the theoretical framework of SRL. The search identified the Motivated Strategies for Learning Questionnaire (MSLQ) which contained five subscales: internal goal orientation, self-efficacy, critical thinking, metacognitive/self-regulation, help seeking and peer learning. Using the original MSLQ as the foundation, we carried out three phases of a research program to develop a useful set of items: an expert panel's review of items, a substantial pilot study, and a factor analysis of ratings of a modified set of items by preclinical and final year medical students.

RESULTS

The factor analysis of the Modified MSLQ extracted four subscales with reasonable internal consistency: self-orientation, critical thinking, self-regulation and feedback-seeking. Each subscale correlates highly with the Modified MSLQ score, with modest inter-correlations between the subscales suggesting that they are measuring different components of the total score.

CONCLUSION

Medical students and their educators need to be able to monitor their learning in their complex academic and clinical environments. The Modified MSLQ provides a means of investigating and tracking individual medical students' reflections on their learning.

A mentor-based portfolio program to evaluate pharmacy students' self-assessment skills

Objective. To evaluate pharmacy students' self-assessment skills with an electronic portfolio program using mentor evaluators. Design. First-year (P1) and second-year (P2) pharmacy students used online portfolios that required self-assessments of specific graded class assignments. Using a rubric, faculty and alumni mentors evaluated students' self-assessments and provided feedback. Assessment. Eighty-four P1 students, 74 P2 students, and 59 mentors participated in the portfolio program during 2010-2011. Both student groups performed well overall, with only a small number of resubmissions required. P1 students showed significant improvements across semesters for 2 of the self-assessment questions; P2 students' scores did not differ significantly. The P1 scores were significantly higher than P2 scores for 3 questions during spring 2011. Mentors and students had similar levels of agreement with the extent to which students put forth their best effort on the self-assessments. Conclusion. An electronic portfolio using mentors based inside and outside the school provided students with many opportunities to practice their self-assessment skills. This system represents a useful method of incorporating self-assessments into the curriculum that allows for feedback to be provided to the students.

How do postgraduate GP trainees regulate their learning and what helps and hinders them? A qualitative study

BACKGROUND

Self-regulation is essential for professional development. It involves monitoring of performance, identifying domains for improvement, undertaking learning activities, applying newly learned knowledge and skills and self-assessing performance. Since self-assessment alone is ineffective in identifying weaknesses, learners should seek external feedback too. Externally regulated educational interventions, like reflection, learning portfolios, assessments and progress meetings, are increasingly used to scaffold self-regulation.The aim of this study is to explore how postgraduate trainees regulate their learning in the workplace, how external regulation promotes self-regulation and which elements facilitate or impede self-regulation and learning.

METHODS

In a qualitative study with a phenomenologic approach we interviewed first- and third-year GP trainees from two universities in the Netherlands. Twenty-one verbatim transcripts were coded. Through iterative discussion the researchers agreed on the interpretation of the data and saturation was reached.

RESULTS

Trainees used a short and a long self-regulation loop. The short loop took one week at most and was focused on problems that were easy to resolve and needed minor learning activities. The long loop was focused on complex or recurring problems needing multiple and planned longitudinal learning activities. External assessments and formal training affected the long but not the short loop. The supervisor had a facilitating role in both loops. Self-confidence was used to gauge competence.Elements influencing self-regulation were classified into three dimensions: personal (strong motivation to become a good doctor), interpersonal (stimulation from others) and contextual (organizational and educational features).

CONCLUSIONS

Trainees did purposefully self-regulate their learning. Learning in the short loop may not be visible to others. Trainees should be encouraged to actively seek and use external feedback in both loops. An important question for further research is which educational interventions might be used to scaffold learning in the short loop. Investing in supervisor quality remains important, since they are close to trainee learning in both loops.

Workplace learning

This critical review found Dutch research to be strong at the undergraduate and residency levels and more or less absent in continuing medical education. It confirms the importance of coaching medical students, giving constructive feedback, and ensuring practice environments are conducive to learning though it has proved hard to improve them. Residents learn primarily from experiences encountered in the course of clinical work but the fine balance between delivering clinical services and learning can easily be upset by work pressure. More intervention studies are needed. Qualitative research designs need to be more methodologically sophisticated and use a wider range of data sources including direct observation, audio-diaries, and text analysis. Areas for improvement are clear but achieving results will require persistence and patience.

The value of training in communication skills for continuing medical education

OBJECTIVE

Question arises as to what extent communication skills are considered in continuing medical education (CME).

METHODS

Analysis for CME-courses in communication skills in the area of the Chamber of Physicians North Rhine (ÄkNo), Germany. Supply Arm(A): CME events (n = 19,320) certified in 2007 were evaluated. Demand Arm(B): course participation of 850 family physicians in the period 2002-2007 was analyzed (n = 37,724). Tests were calculated to the level 0.05 using Mann-Whitney U-test.

RESULTS

(A) 388 (2.0%) events were concerned with the topic communications. 59.3% involved active cooperation of the participants. 0.5% events devoted more than 50% of their duration to the topic communication. Proportions in the subjects of internal medicine, general medicine and pediatrics amounted to 0.2%. (B) 803 (2.1%) events with a focus on communication were identified. Women took part in significantly more events than men (p<0.002) and selected more interactive courses.

CONCLUSION

Content on communication training was small. Increasing experience does not automatically improve communication skills but an extent of deliberate praxis seems to be necessary and must be sought and developed.

PRACTICE IMPLICATIONS

Communication skills are still insufficiently provided in CME-courses and should be more directed to focus as treatment strategies and scientifically investigated for outcome improvements.

Current use of and attitudes to e-learning in otolaryngology: questionnaire survey of UK otolaryngology trainees

Introduction:

The E-lefENT programme is a collaborative e-learning project developed by the Department of Health and ENTUK. Prior to its introduction, we aimed to assess current use of and attitudes towards e-learning amongst UK otolaryngology trainees.

Method:

A web-based questionnaire was distributed to UK otolaryngology trainees via regional programme directors.

Results:

A total of 117 responses was received (35 per cent response rate). Most trainees reported increasing exposure to e-learning throughout their training and rated their current computer literacy skills highly, although satisfaction with current e-learning resources was found to vary widely for a variety of reasons.

Conclusion:

Meeting the educational needs of otolaryngology trainees in the current era continues to be a challenge. Most trainees are already familiar with e-learning, although the quality and delivery of available websites is currently variable. The majority of trainees who responded seemed receptive to the implementation of a free, otolaryngology-specific e-learning resource.

Self-assessment in pharmacy and health science education and professional practice

Self-assessment is an important skill necessary for continued development of a health care professional from student pharmacist throughout their professional career. This paper reviews the literature on student and practitioner self-assessment and whether this skill can be improved upon. Although self-assessment appears to be a skill that can be improved, both students and professionals continue to have difficulty with accurate self-assessment. Experts' external assessment of students should remain the primary method of testing skills and knowledge until self-assessment strategies improve. While self-assessment is important to lifelong learning, external assessment is also important for practitioners' continuing professional development.

Bridging the Gap: Training Critical Care Clinician-Echocardiographers through a Collaborative Curriculum

In response to the 2008 position statement of the Collaborative Working Group of the British Society of Echocardiography and the Intensive Care Society, we describe a recently launched Fellowship in Critical Care Echocardiography at the John Radcliffe Hospitals Oxford. We discuss the educational principles underlying the design and ongoing evaluation of the curriculum.