Assessing the assessment of interventions: we're not there yet

Acceptability and feasibility of mini-clinical evaluation exercise as a formative assessment tool for workplace-based assessment for surgical postgraduate students

Background:

Despite an increasing emphasis on workplace-based assessment (WPBA) during medical training, the existing assessment system largely relies on summative assessment while formative assessment is less valued. Various tools have been described for WPBA, mini-clinical evaluation exercise (mini-CEX) being one of them. Mini-CEX is well accepted in Western countries, however, reports of its use in India are scarce. We conducted this study to assess acceptability and feasibility of mini-CEX as a formative assessment tool for WPBA of surgical postgraduate students in an Indian setting.

Methods:

Faculty members and 2^nd year surgical residents were sensitized toward mini-CEX and requisite numbers of exercises were conducted. The difficulties during conduction of these exercises were identified, recorded, and appropriate measures were taken to address them. At the conclusion, the opinion of residents and faculty members regarding their experience with mini-CEX was taken using a questionnaire. The results were analyzed using simple statistical tools.

Results:

Nine faculty members out of 11 approached participated in the study (81.8%). All 16 2^nd year postgraduate surgical residents participated (100%). Sixty mini-CEX were conducted over 7 months. Each resident underwent 3–5 encounters. The mean time taken by the assessor for observation was 12.3 min (8–30 min) while the mean feedback time was 4.2 min (3–10 min). The faculty reported good overall satisfaction with mini-CEX and found it acceptable as a formative assessment tool. Three faculty members (33.3%) reported mini-CEX as more time-consuming while 2 (22.2%) found it difficult to carry the exercises often. All residents accepted mini-CEX and most of them reported good to high satisfaction with the exercises conducted.

Conclusions:

Mini-CEX is well accepted by residents and faculty as a formative assessment tool. It is feasible to utilize mini-CEX for WPBA of postgraduate students of surgery.

Profiling postgraduate workplace-based assessment implementation in Ireland: a retrospective cohort study

In 2010, workplace-based assessment (WBA) was formally integrated as a method of formative trainee assessment into 29 basic and higher specialist medical training (BST/HST) programmes in six postgraduate training bodies in Ireland. The aim of this study is to explore how WBA is being implemented and to examine if WBA is being used formatively as originally intended. A retrospective cohort study was conducted and approved by the institution’s Research Ethics Committee. A profile of WBA requirements was obtained from 29 training programme curricula. A data extraction tool was developed to extract anonymous data, including written feedback and timing of assessments, from Year 1 and 2 trainee ePortfolios in 2012–2013. Data were independently quality assessed and compared to the reference standard number of assessments mandated annually where relevant. All 29 training programmes mandated the inclusion of at least one case-based discussion (max = 5; range 1–5). All except two non-clinical programmes (93 %) required at least two mini-Clinical Evaluation Exercise assessments per year and Direct Observation of Procedural Skills assessments were mandated in 27 training programmes over the course of the programme. WBA data were extracted from 50 % of randomly selected BST ePortfolios in four programmes (n = 142) and 70 % of HST ePortfolios (n = 115) in 21 programmes registered for 2012–2013. Four programmes did not have an eligible trainee for that academic year. In total, 1142 WBAs were analysed. A total of 164 trainees (63.8 %) had completed at least one WBA. The average number of WBAs completed by HST trainees was 7.75 (SD 5.8; 95 % CI 6.5–8.9; range 1–34). BST trainees completed an average of 6.1 assessments (SD 9.3; 95 % CI 4.01–8.19; range 1–76). Feedback—of varied length and quality—was provided on 44.9 % of assessments. The majority of WBAs were completed in the second half of the year. There is significant heterogeneity with respect to the frequency and quality of feedback provided during WBAs. The completion of WBAs later in the year may limit available time for feedback, performance improvement and re-evaluation. This study sets the scene for further work to explore the value of formative assessment in postgraduate medical education.