Robust, defensible, and fair: The AMEE guide to selection into medical school: AMEE Guide No. 153

Selection is the first assessment of medical education and training. Medical schools must select from a pool of academically successful applicants and ensure that the way in which they choose future clinicians is robust, defensible, fair to all who apply and cost-effective. However, there is no comprehensive and evidence-informed guide to help those tasked with setting up or rejuvenating their local selection process. To address this gap, our guide draws on the latest research, international case studies and consideration of common dilemmas to provide practical guidance for designing, implementing and evaluating an effective medical school selection system. We draw on a model from the field of instructional design to frame the many different activities involved in doing so: the ADDIE model. ADDIE provides a systematic framework of Analysis (of the outcomes to be achieved by the selection process, and the barriers and facilitators to achieving these), Design (what tools and content are needed so the goals of selection are achieved), Development (what materials and resources are needed and available), Implementation (plan [including piloting], do study and adjust) and Evaluation (quality assurance is embedded throughout but the last step involves extensive evaluation of the entire process and its outcomes).HIGHLIGHTSRobust, defensible and fair selection into medical school is essential. This guide systematically covers the processes required to achieve this, from needs analysis through design, development and implementation, to evaluation of the success of a selection process.

Assessing feasibility, construct validity, and reliability of a new aged care-specific preference-based quality of life instrument: evidence from older Australians in residential aged care

BACKGROUND

Quality of Life-Aged Care Consumers (QOL-ACC) is a new older-person-specific quality of life instrument designed for application in quality assessment and economic evaluation in aged care. The QOL-ACC was designed from its inception with older people receiving aged care services ensuring its strong content validity. Given that the QOL-ACC has already been validated in home care settings and a preference-weighted value set developed, we aimed to assess feasibility, construct validity and reliability of the QOL-ACC in residential aged care settings.  METHODS: Individuals living in residential aged care facilities participated in an interviewer-facilitated survey. The survey included the QOL-ACC, QCE-ACC (quality of aged care experience measure) and two other preference-based quality of life instruments (ASCOT and EQ-5D-5L). Feasibility was assessed using missing data and ceiling/floor effects. Construct validity was assessed by exploring the relationship between the QOL-ACC and other instruments (convergent validity) and the QOL-ACC's ability to discriminate varying levels of self-rated health and quality of life. Internal consistency reliability was assessed using Cronbach's alpha (α).

RESULTS

Of the 200 residents (mean age, 85 ± 7.7 years) who completed the survey, 60% were female and 69% were born in Australia. One in three participating residents self-rated their health as fair/poor. The QOL-ACC had no missing data but had small floor effects (0.5%) and acceptable ceiling effects (7.5%). It demonstrated moderate correlation with ASCOT (r = 0.51, p < 0.001) and EQ-5D-5L (r = 0.52, p < 0.001) and a stronger correlation with the QCE-ACC (r = 0.57, p < 0.001). Residents with poor self-rated health and quality of life had significantly lower scores on the QOL-ACC. The internal consistency reliability of the QOL-ACC and its dimensions was good (α = 0.70-0.77).

CONCLUSIONS

The QOL-ACC demonstrated good feasibility, construct validity and internal consistency reliability to assess aged care-related quality of life. Moderate correlations of the QOL-ACC and other instruments provide evidence of its construct validity and signifies that the QOL-ACC adds non-redundant and non-interchangeable information beyond the existing instruments. A stronger correlation with the QCE-ACC than other instruments may indicate that quality of life is more intimately connected with the care experience than either health- or social-related quality of life in residential aged care settings.

Does the Scottish pharmacist post-registration foundation programme change self-reported competence? A before and after questionnaire study

Introduction

The two-year Post-registration Foundation Programme for pharmacists in Scotland1, is based on an eight-element framework, and supports early career pharmacists to develop core competences relevant to all sectors of practice. Pharmacists have 6 monthly appraisals and are assessed by an online portfolio and a summative OSCE.

Aim

The aim of this paper is to describe the changes in pharmacists’ self-reported competence at the end-of-programme compared to baseline.

Methods

Eligible participants were pharmacists from any sector registering for the programme in September 2017/February 2018. They were invited by email to complete an online questionnaire at baseline and exit to assess their self-reported level of competence with 19 statements linked to the framework using a ten-point Likert scale (1 not at all confident ,10 very confident). Additional questions on expectations (baseline only) and experiences (end-of-programme only: resources used, general views on the programme and suggestions for improvement (open text) were included. No reminders were sent. Questionnaires were hosted on a Qualtrics platform. Data was collated in Excel then exported to an SPSS version 27 database. Analyses were based on the sub-sample who completed both questionnaires; frequencies and means/SD are reported and changes from baseline to end-of-programme compared using paired t test. Open-text responses were subjected to content analysis. Ethical approval was not required for this evaluation.

Results

Ninety-six pharmacists registered for the programme: 14 males; 11 Health Boards. To date 56 have completed the programme (5 community, 34 hospital, 15 hospital/primary-care and 2 primary-care). Baseline and end-of-programme questionnaires were completed by 73/96 pharmacists (76% response) and 54/87 (50 completers and 4 early exiters) (62.1% response) respectively. 37 pharmacists completed both questionnaires. At the end-of-programme there was improvement in self-reported competence across all statements compared to baseline. Based on a paired t-test changes were significant for all competencies at p&lt;0.001, apart from demonstrating a non-discriminatory attitude which was significant at p&lt;0.05 and was the competency at baseline with the highest self-reported competence (9.36, SD 1.16 cf 9.89, SD 0.32). Regular meetings with tutors/NES staff and support from colleagues and healthcare professionals were the resources rated most highly. Around 50% of participants rated the NES resource booklets, guidance documents and the assessment handbook essential/very useful. 15% found NES personnel input essential and 45% rated peer sessions essential/very useful. FPs commented that undertaking the training had; increased confidence, improved clinical skills, increased ability to reflect, furthered their career prospects and enabled progression to independent prescribing. It had also improved: knowledge and skills, patient care, ability to support other health care professionals. Suggestions for improvement related to the online portfolio, support, and the final assessment.

Discussion/Conclusion

Limitations include small numbers and programme delivery limited to Scotland. Overall, there was a marked improvement in all competency and practice statements at programme exit. Resources provided were generally rated useful and had been utilised. The training programme had also had a positive impact on confidence, reflective practice, and enabled progress to Independent Prescribing courses.

References

1. Post- registration Foundation Programme for Newly Qualified Pharmacists in Scotland https://nes.scot.nhs.uk/our-work/post-registration-foundation-programme-for-newly-qualified-pharmacists-in-scotland-autumn-2021-onwards/ Accessed October 12th, 2021

Foundation pharmacists’ experiences of personal development whilst undertaking the nes post-registration foundation programme: a qualitative study (Scotland)

Introduction

The NES post-registration Foundation Programme1 for pharmacists introduced in 1989, has recently been extended to primary-care and community sectors. The programme, approximately two years in duration, is based on a competency framework (six core elements & 2 sector specific). Foundation pharmacists (FPs) are assigned a tutor, have 6 monthly appraisals, and are assessed by an e-portfolio and OSCE.

Aim

The aim of this paper is to describe FPs’ experiences and personal development when undertaking the Programme. This study is part of larger mixed methods evaluation of the Programme.

Methods

Eligible participants were pharmacists registering with the programme in September 2017/February 2018. NES staff sent emails inviting them to take part in either a focus group or interview (face-to-face or virtual) at baseline, mid-point, and-programme completion, to explore their expectations and experiences of the programme. Proceedings were digitally recorded, transcribed, and managed using NVivo. Analysis was inductive and thematic2 and synthesised in relation to Bandura’s Social Cognitive theory3 (SCT) which posits that learning and knowledge enhancement are related to interaction between environmental, person/cognitive, and behavioural factors. Facilitators and barriers to undertaking the Programme were identified. IRAS advised ethical approval was not required; NHS Research & Development approval was given.

Results

Ninety-six pharmacists registered for the Programme. In total 22 participated in 7 baseline focus groups, 7 in 2 midway focus group/3 in interviews, and 4 in end-of-programme interviews. At baseline environmental factors predominated in motivation to undertake the Programme such as the structured approach and robust support network (tutors, peers & NES). These also contributed to the perceived facilitators for programme completion. Conversely environmental factors linked to the external working environment could also be barriers. Person/cognitive factors were other motivators to undertaking the Programme because of perceived improved employability, and benefits of increased confidence. Finally, the behavioural domain also contributed to motivation with perceptions of benefits in both transferable and clinical skills. At midway, pharmacists confirmed their improvements in both the personal/cognitive domain (confidence, resilience, reflective practice) and the behavioural domain of clinical and transferable skills. Facilitators (support & resources) and barriers (lack of support, time, and feedback) were almost exclusively environmental. Comments around professional identity also emerged inextricably linked to FPs’ relationships with and recognition by other members of the HCP team. At programme end the views from the midway point remain unchanged. Perceived skills had increased largely in the behavioural domain but also in the person/cognitive domain with understanding the value of reflective practice emerging as a key area of improvement and being reported as influencing professional identity. Facilitators were solely reported in terms of environmental factors relating to extensive support networks and barriers again related to environmental factors which included lack of support, resources, and external influences. Views of the programme overall were very positive

Discussion/Conclusion

All three SCT factors influenced the learning and development of the FPs, but environmental influences dominated the barriers. As a qualitative study these results cannot be generalised, but they highlight areas to explore in more details for future programme delivery.

References

1. Post-registration Foundation Programme for Newly Qualified Pharmacists in Scotland. Available from: https://nes.scot.nhs.uk/our-work/post-registration-foundation-programme-for-newly-qualified-pharmacists-in-scotland-autumn-2021-onwards/ Accessed July 25, 2022

2. Braun, V., Clarke, V. Using thematic analysis in psychology. Qualitative Research in Psychology, 2006;3(2):77-101. ISSN 1478-0887. Available from: http://eprints.uwe.ac.uk/11735 Accessed October 12, 2021

3. Bandura, A. Social learning theory. Englewood Cliffs, N.J.: Prentice Hall; 1977.

Cardiac resynchronization therapy in patients with a history of atrial fibrillation: insights from five major clinical trials

Funding Acknowledgements

Type of funding sources: Public Institution(s). Main funding source(s): National Heart, Lung, and Blood Institute

Background

Many patients with heart failure who are considered for cardiac resynchronization therapy (CRT) have a history of (h/o) atrial fibrillation (AF) but there are doubts about the efficacy of CRT in patients with AF.

Purpose

To investigate the association of CRT on morbidity and mortality among patients with and without a h/o AF.

Methods

Original, patient-level data from five clinical trials of CRT that permitted enrolment of patients with a h/o AF were included: COMPANION, MADIT-CRT, BLOCK HF, REVERSE, and MIRACLE trial. Patients with permanent or persistent AF were excluded from these trials, and therefore from this analysis. The outcomes of interest were the composite endpoint of time to heart failure hospitalization (HFH) or all-cause mortality or all-cause mortality alone. The association of CRT (versus no CRT) with outcomes for patients with and without a h/o AF was assessed using a Bayesian-Weibull survival regression model with random terms for the trial-specific treatment effects and the trial-specific baseline hazard functions including an interaction between history of paroxysmal AF and CRT. All results are presented as hazard ratios (HRs) with 95% posterior credible intervals (CIs) and posterior probabilities of no association, adjusting for baseline characteristics.

Results

A total of 4062 patients were included, 661 (16.3%) of whom had a h/o AF. Patients with a h/o AF were older (mean [SD] age 68 [10] years versus 64 [11] years) and had a higher proportion of ischemic cardiomyopathy (67% versus 53%, p&lt;0.001), a higher baseline serum creatinine (1.3 mg/dl versus 1.2 mg/dl, p&lt;0.001), and a lower left ventricular ejection fraction (25% versus 26%, p&lt;0.001). The HRs for all outcomes and the interaction term are shown in Table 1. For the overall population, CRT delayed the time to HFH or all-cause mortality (HR: 0.74, 95% CI: 0.62 – 0.87, p=0.005); for patients with a h/o AF, it did not (HR: 0.87, 95% CI: 0.64 to 1.19, p=0.37). In this patient-level meta-analysis, CRT was not associated with a reduction in mortality, overall or by h/o AF. Howevber, the interaction (estimate shown as a ratio of HRs) between those with or without a h/o AF and the effects of CRT was not significant for either outcome (Table 1).

Conclusion

In the largest post hoc analysis to date, we confirm the benefits of CRT in patients without a h/o AF in reducing HFH or mortality. There was no statistically significant interaction between CRT and h/o AF for any analysed outcome.

Is the post-registration foundation programme fit for purpose for community pharmacists? An exploration of pharmacist experiences

Introduction

The two-year, NHS Education for Scotland (NES) post-registration foundation programme supports early career pharmacists in patient-facing sectors of practice. The experiential programme, based on an eight-element competency framework, also includes webinars, online resources, and tutor support. Learners complete an online evidence portfolio and undertake a summative OSCE.

Aim

The aim of this paper is to report the experiences of the community-pharmacist participants, with a focus on the ‘fitness-for-purpose’ of the programme.

Methods

This was a longitudinal mixed-methods study theoretically underpinned by Miller’s triangle and social cognitive theory. Eligible participants were all pharmacists registering for the programme in Scotland in September 2017 and February 2018, all participating Welsh community-pharmacists, and all tutors. Invitation packs were emailed by NES/HEIW staff with names forwarded to researchers following signed consent. Focus groups/interviews (face-to-face or virtual according to participant preference) were undertaken at start, mid-point and exit of programme, to explore expectations (benefits, social gains, professional identify), experiences (challenges, facilitators, meeting of learners’ needs) and barriers. Proceedings were digitally recorded, transcribed verbatim and managed using NVivo. Thematic analysis (1) was based on social cognitive theory (transferable behavioural skills and professional attitudes). An inductive analysis additionally identified emergent themes. Participants in Scotland were invited to complete an on-line base-line questionnaire to describe their self-assessed competence against the NES Foundation framework (personal and professional practice, membership of healthcare team, communication, patient centred approach to practice). Data was analysed in SPSS using descriptive statistics. Themes from qualitative and quantitative data were integrated. IRAS ethical approval was not required; NHS Research & Development approval was given.

Results

96 pharmacists registered for the programme: 18 community-pharmacists in Scotland (11 health boards); 14 community pharmacists in Wales. In Scotland 15 community-pharmacists completed questionnaires: 9 expected an ‘increase in confidence’ and 11 to provide ‘better patient care’. Self-assessed competence against the framework was generally high. Across Scotland and Wales, 12 focus-groups (involving 19 community-pharmacists), 12 community-pharmacist interviews, 10 tutor focus-groups (8 community-pharmacist tutors) and 3 community-pharmacist tutor interviews were conducted.

At midpoint and exit pharmacists and tutors reported increased confidence, the ability to reflect and pride in their achievement. Barriers: included lack of protected time; workload; and lack of support (tutor and employer). There were also programme issues (practicalities of portfolio; workplace-based assessment, no access to medical records); and cultural issues in community-pharmacy (‘speed & safety’; lack of recognition). Reasons for dropping out of the programme included: moved geographical area; too experienced; workload pressures; no incentive; no employer support. Four community-pharmacists in Scotland and none in Wales completed the programme.

Conclusion

Study limitations include the small numbers, programme delivery limited to Scotland and Wales, and limited response rate to focus-groups/interviews, exacerbated by COVID19. Overall community pharmacist expectations were met, and they perceived the programme was fit-for-purpose and worthwhile. However, barriers particularly related to the community pharmacy context, may have led to the high drop-out rate. These findings should be considered as the new UK-wide RPS curriculum for foundation pharmacists (2) is implemented in Scotland, to optimise its successful delivery.

References

(1) Braun, V. and Clarke, V. Using thematic analysis in psychology. Qualitative Research in Psychology. [Internet]. 2006; 3 (2): 77-101. ISSN 1478-0887 Available from: http://eprints.uwe.ac.uk/11735

(2) Post- registration Foundation Programme for Newly Qualified Pharmacists in Scotland [Internet] https://nes.scot.nhs.uk/our-work/post-registration-foundation-programme-for-newly-qualified-pharmacists-in-scotland-autumn-2021-onwards Accessed October 12, 2021.

Quantifying changes in ultrasound tongue-shape pre- and post-intervention in speakers with submucous cleft palate: an illustrative case study

Ultrasound Tongue Imaging is increasingly used during assessment and treatment of speech sound disorders. Recent literature has shown that ultrasound is also useful for the quantitative analysis of a wide range of speech errors. So far, the compensatory articulations of speakers with cleft palate have only been analysed qualitatively. This study provides a pilot quantitative ultrasound analysis, drawing on longitudinal intervention data from a child with submucous cleft palate. Two key ultrasound metrics were used: 1. articulatory t-tests were used to compare tongue-shapes for perceptually collapsed phonemes on a radial measurement grid and 2. the Mean Radial Difference was reported to quantify the extent to which the two tongue shapes differ, overall. This articulatory analysis supplemented impressionistic phonetic transcriptions and identified covert contrasts. Articulatory errors identified in this study using ultrasound were in line with errors identified in the speech of children with cleft palate in previous literature. While compensatory error patterns commonly found in speakers with cleft palate have been argued to facilitate functional phonological development, the nature of our findings suggest that the compensatory articulations uncovered are articulatory in nature.

Inter operator variability of machine learning researchers predicting all-cause mortality in patients admitted to intensive care unit

Background

The number of publications using machine learning (ML) to predict cardiovascular outcomes and identify clusters of patients at greater risk has risen dramatically in recent years. However, research papers which use ML often fail to provide sufficient information about their algorithms to enable results to be replicated by others in the same or different datasets.

Aim

To test the reproducibility of results from ML algorithms given three different levels of information commonly found in publications: model type alone, a description of the model, and complete algorithm.

Methods

MIMIC-III is a healthcare dataset comprising detailed information from over 60,000 intensive care unit (ICU) admissions from the Beth Israel Deaconess Medical Centre between 2001 and 2012. Access is available to everyone pending approval and completion of a short training course.

Using this dataset, three models for predicting all-cause in-hospital mortality were created, two from a PhD student working in ML, and one from an existing research paper which used the same dataset and provided complete model information. A second researcher (a PhD student in ML and cardiology) was given the same dataset and was tasked with reproducing their results. Initially, this second researcher was told what type of model was created in each case, followed by a brief description of the algorithms. Finally, the complete algorithms from each participant were provided. In all three scenarios, recreated models were compared to original models using Area Under the Receiver Operating Characteristic Curve (AUC).

Results

After excluding those younger than 18 years and events with missing or invalid entries, 21,139 ICU admissions remained from 18,094 patients between 2001 and 2012, including 2,797 in-hospital deaths. Three models were produced: two Recurrent Neural Networks (RNNs) which differed significantly in internal weights and variables, and a Boosted Tree Classifier (BTC). The AUC of the first reproduced RNN matched that of the original RNN (Figure 1), however the second RNN and the BTC could not be reproduced given model type alone. As more information was provided about these algorithms, the results from the reproduced models matched the original results more closely.

Conclusions

In order to create clinically useful ML tools with results that are reproducible and consistent, it is vital that researchers share enough detail about their models. Model type alone is not enough to guarantee reproducibility. Although some models can be recreated with limited information, this is not always the case, and the best results are found when the complete algorithm is shared. These findings have huge relevance when trying to apply ML in clinical practice.

Funding Acknowledgement

Type of funding sources: None.

50 A Cross-Sectional Study Examining the Association Between MRCS Performance and Surgeons Receiving Fitness to Practice Sanctions

Aim

Fitness to practice (FtP) investigations by the General Medical Council (GMC) can be one of the most stressful experiences in a surgeon’s career. Demographic factors are known to alter the likelihood of GMC investigation. Despite this there are no studies assessing risk factors in surgeons for FtP sanctions. Thus, we used the newly created Intercollegiate Membership of Royal College of Surgeons (MRCS) examination database to identify risk factors for and the prevalence of GMC sanctions in early-career surgeons.

Method

FtP sanction data contained in the GMC list of registered medical practitioners (LRMP) database was linked at person-level to all UK graduates who had attempted MRCS Part A or Part B between September 2007 and January 2020. Data were anonymised by the Royal College of Surgeons of England prior to analysis.

Results

f 11,660 candidates who had attempted the MRCS within the study period only 31 (0.3%) candidates had GMC FtP sanctions within the last two years. Of these, 12 had active conditions on their registration, 7 had active undertakings and 14 had warnings. Candidate demographics were similar between cohorts and there were no significant differences between MRCS performance identified.

Conclusions

In this, the largest study of MRCS candidates to date, the prevalence of active FtP sanctions in early-career surgeons was 0.3%, significantly lower than the prevalence of sanctions across more experienced UK surgeons (0.9%). These data highlight early-career surgeons as a low-risk group for disciplinary action and should reassure patients and medical professionals of the rarity of FtP sanctions.

751 Improving Engagement with Home-Based Surgical Skills Simulation

Introduction

Surgical simulation has been repeatedly shown to facilitate technical skill acquisition. However, trainee engagement with self-directed practice remains variable, despite access to resources. Understanding the motivators and barriers to participation is crucial to develop modules which can effectively meet the learning needs of current, and future, surgical trainees. The aim of this qualitative study was to examine factors which influence trainee engagement with home-based surgical skills simulation.

Method

A series of one-to-one semi-structured interviews were conducted remotely with ST3 vascular trainees who had previously consented to take part in a national programme of home-based technical skills simulation. Interview data was transcribed and thematically analysed.

Results

12 trainees were interviewed during a 4-week period. Overall, trainees valued simulation but found it difficult to balance against clinical commitments and mandatory training requirements, particularly if there were limited opportunities for skill transfer to the real-world environment. Although simulation was acknowledged to be a safe environment for experiential learning, trainees alluded to an underlying culture of perfection which limited willingness to learn from mistakes, even within a simulated setting. In addition, traditional attitudes about the apprenticeship model of surgical training prevail, with simulation often viewed as inferior to learning “on the job” in theatre.

Conclusions

Trainee engagement with home-based surgical skills simulation may be influenced by a range of systemic factors. In future, formal certification of simulation modules, mandating simulated competencies and curricular integration may help improve participation, as well as supporting cultural shift towards recognition of simulation as a vital component of modern surgical training.

238 MRCS Performance Predicts Surgical Career Outcomes

Aim

Successful completion of the MRCS examination is mandatory for progression into higher surgical (registrar) training in the UK. National selection for training programmes is a highly competitive process. Despite this, the ranking of applicants does not currently include MRCS performance scores. This offers a unique opportunity to compare two independent assessments of surgical trainees to establish whether MRCS performance can predict future surgical career outcomes and competitiveness in training.

Method

All UK candidates who attempted MRCS between 2007-2020 were matched to career outcome data using the GMC list of registered medical practitioners and anonymised (n = 2910). Chi-squared tests determined associations with first attempt MRCS pass/fail outcomes. Multinomial regression models were developed to establish the predictive power of success at MRCS in determining surgical specialty and training deanery choices.

Results

There was statistically significant variability in MRCS Part A pass rates, ranging from 55.1%-76.7% between surgical specialties (P=0.001) but no significant variability in Part B pass rates (P=0.655). There was significant variability in Part A and Part B pass rates between training deaneries (P=0.010 and P=0.036 respectively). Pass rates ranged from 59.9%-77.7% for Part A and 70.1%-85.0% for Part B between training deaneries. Those in more competitive specialties and training deaneries performed better in MRCS in terms of first attempt pass rate, scores and number of attempts taken to pass.

Conclusions

MRCS performance is an early predictor of surgical career choice and competitiveness. Trainees who perform well at MRCS are more likely to enter competitive surgical specialties and training deaneries.

1253 Unlocking Growth Options in Surgical Education and Training During the Pandemic

Introduction

The COVID-19 pandemic brought widespread disruption to structured surgical education and training. The knee-jerk reaction is often pessimism about surgical training’s future, particularly in the Improved Surgical Training (IST) pilot’s context. However, Einstein famously once said, “In the midst of every crises lies great opportunity”. Unlocking growth during periods of high uncertainty is a premise of real options theory; one utilised by supply chain managers and decision scientists, but novel to medical education. This study explores the growth options that have resulted from new operational models during the pandemic.

Method

Using a qualitative case study approach, data were obtained from interviews with core surgical trainees across Scotland. Data coding and inductive thematic analysis were undertaken.

Results

Forty-six trainees participated. Analysis from trainees’ perspective revealed: unexpected fulfilment from redeployment to non-surgical specialties, benefits to personal development from the unintended broad-based training across surgical specialties, improved collaborative teamworking between specialties and allied healthcare professionals, and enhanced supervised learning opportunities. Institutional growth options reported by trainees included: rapid uptake of telemedicine and digital technology, implementation of single hospital episode encounters for minor conditions, streamlined processes in theatre and acute admissions, and changes in working culture towards rationalising and teamworking.

Conclusions

Growth options have been deliberately and unintentionally unlocked due to individual and institutional adaptions and innovations in response to the exogenous disruption. While some changes may be temporary, hopefully structured reflection on these changes and responders to them will drive surgical education and training into a new sustainable and resilient post-pandemic era.

51 The Impact of Disability on Performance in the Intercollegiate Membership of the Royal College of Surgeons examination (MRCS)

Aim

The number of clinicians with disabilities is increasing, however there is no data on the performance of doctors with registered disabilities in the postgraduate environment. If we are to ensure diversity and equality within the workforce, we must first identify whether attainment differences exist in markers of performance. To address this, we assessed the impact of disabilities on performance in the MRCS.

Method

All UK medical graduates who had attempted MRCS Part A (n = 9,597) and Part B (n = 4,562) between 2007-2017 with linked disability data in the UK Medical Education Database (https://www.ukmed.ac.uk) were included. Univariate analysis identified associations with MRCS performance and logistic regression models identified independent predictors of success.

Results

Candidates with registered disabilities (n = 635) had lower MRCS Part A pass rates (46.3% vs 59.8% (p &lt; 0.001)) but similar Part B pass rates (68.2% vs 70.9% (p = 0.339)). They were nearly twice as likely to fail Part A (odds ratio 0.55 [95% Confidence Interval 0.46-0.64]). When prior academic attainment (A-Levels and medical school performance) was accounted for, there was no statistically significant difference in the likelihood of MRCS success (p&gt;0.05). Pass rates were similar for candidates with specific learning difficulties and those with other registered disabilities (p&gt;0.05).

Conclusions

This is the first study to assess the impact of disability status and type on performance at a postgraduate medical examination. It appears that candidates with registered disabilities performed less well in formal, written examinations generally. Although our data indicate that current MRCS testing accommodations are fair, enabling performance unrestricted by disability.

Anomalous diffusion driven by the redistribution of internal stresses

This article explores the mathematical description of anomalous diffusion, driven not by thermal fluctuations but by internal stresses. A continuous time random walk framework is outlined in which the waiting times between displacements (jumps), generated by the dynamics of internal stresses, are described by the generalized Γ distribution. The associated generalized diffusion equation is then identified. The solution to this equation is obtained as an integral over an infinite series of Fox H functions. The probability density function is identified as initially non-Gaussian, while at longer timescales Gaussianity is recovered. Likewise, the second moment displays a transient nature, shifting between subdiffusive and diffusive character. The potential application of this mathematical description to the quaking observed in several soft-matter systems is discussed briefly.

The impact of disability on performance in a high-stakes postgraduate surgical examination: a retrospective cohort study

OBJECTIVE

Despite rising numbers of doctors in the workforce with disabilities, little is known about the impact of disabilities on postgraduate performance. To ensure all groups are treated fairly in surgical training, it is essential to know whether any attainment differences exist in markers of surgical performance. To address this gap, we assessed the impact of disabilities on performance on the Intercollegiate Membership of the Royal College of Surgeons examination (MRCS).

DESIGN

Retrospective cohort study.

SETTING

Secondary care.

PARTICIPANTS

All UK MRCS candidates attempting Part A (n = 9600) and Part B (n = 4560) between 2007 and 2017 with linked disability data in the UK Medical Education Database (https://www.ukmed.ac.uk) were included.

MAIN OUTCOME MEASURES

Chi-square tests and correlation coefficients established univariate associations with MRCS performance, while multiple logistic regressions identified independent predictors of success.

RESULTS

Though MRCS Part B pass rates were similar (p = 0.339), candidates with registered disabilities had significantly lower first-attempt Part A pass rates (46.3% vs. 59.8%, p < 0.001). Candidates with disabilities also performed less well in examinations taken throughout school and medical school, and after adjusting for prior academic performance and sociodemographic predictors of success, logistic regression found that candidates with disabilities were no less likely to pass MRCS than their peers (odds ratio 1.04, 95% confidence interval 0.66 to 1.62). No significant variation was found in MRCS performance between type of disability or degree of limitations caused by disability (p > 0.05).

CONCLUSION

Although candidates with registered disabilities performed less well in formal, written examinations, our data indicate that they are as likely to pass MRCS at first attempt as their peers who achieved similar grades at high school and medical school. In order to enable equity in career progression, further work is needed to investigate the causes of attainment differences in early career assessments.

A cross-sectional study examining the association between MRCS performance and surgeons receiving sanctions against their medical registration

BACKGROUND

Fitness to practice (FtP) investigations by the General Medical Council (GMC) safeguard patients and maintain the integrity of the medical profession. The likelihood of FtP sanctions is influenced by specialty and socio-demographic factors and can be predicted by performance at postgraduate examinations. This is the first study to characterise the prevalence of FtP sanctions in early-career surgeons and to examine the association with performance at the Membership of the Royal College of Surgeons (MRCS) examination.

METHODS

All UK graduates who attempted MRCS between September 2007-January 2020 were matched to the GMC list of registered medical practitioners. Clinicians who had active FtP sanctions between 28th August 2018 and 28th August 2020 were identified. Data were anonymised by RCS England prior to analysis.

RESULTS

Of 11,660 candidates who attempted MRCS within the study period, only 31 (0.3%) had FtP sanctions between 2018 and 2020. Of these, 12 had active conditions on registration, seven had undertakings and 14 had warnings. There was no statistically significant difference in MRCS performance in either Parts A or B of the examination for those with and those free from FtP sanctions (P > 0.05).

CONCLUSIONS

In this, the largest study of MRCS candidates to date, the prevalence of active FtP sanctions in early-career surgeons was 0.3%, significantly lower than the prevalence of sanctions across more experienced UK surgeons (0.9%). These data highlight early-career surgeons as a low-risk group for disciplinary action and should reassure patients and medical professionals of the rarity of FtP sanctions.

221 UK Medical School Selection Criteria Predicts Success in Part A of The Intercollegiate Membership of The Royal College of Surgeons (MRCS) Examination

Background

Selection into UK medical school typically involves a combination of three measures: prior academic attainment, selection tests (e.g., the University Clinical Aptitude Test (UCAT), Biomedical Admissions Test (BMAT), Graduate Medical School Admissions Test (GAMSAT)), and an interview. We investigated whether prior attainment and selection test scores can predict MRCS success.

Method

We used the UKMED database to analyse selection data for all UK graduates who attempted MRCS Part A (n = 9729) and Part B (n = 4644) between 2007-2017. Univariate analysis and Pearson correlation coefficients were used to examine the relationship between selection scores and first attempt MRCS success.

Results

Successful MRCS Part A candidates had better A-Levels and higher scores in UCAT, BMAT and GAMSAT examinations (p &lt; 0.001) than their unsuccessful peers. No statistically significant difference was observed for MRCS Part B. A moderate positive correlation was found between Part A, BMAT (r = 0.315, p &lt; 0.001) and GAMSAT scores (r = 0.346, p &lt; 0.001). A weak positive correlation was found between Part A, A-Level (r = 0.144, p &lt; 0.001) and UCAT scores (r = 0.246, p &lt; 0.001).

Conclusions

A-level results and medical school selection tests predict success in the knowledge-based (Part A) MRCS examination.

191 Performance at Medical School and Foundation Programme Selection Can Predict Success in The MRCS Examination

Background

Identifying factors that predict success in the Membership of the Royal College of Surgeons (MRCS) examination can aid trainees’ exam preparation and help deaneries identify candidates likely to require additional support. We assessed whether performance at Medical School and UK Foundation Programme selection (Educational Performance Measurement (EPM) and Situational Judgement Test (SJT)) predicted MRCS success.

Method

We analysed data from the UKMED Database for UK graduates who had attempted MRCS Part A (n = 1,975) and Part B (n = 630) between 2013-2017. Univariate analysis examined the relationship between performance and the likelihood of passing MRCS at first attempt. Logistic regression identified independent predictors of MRCS success.

Results

Each additional EPM decile increased the chances of passing MRCS by 52% for Part A (odds ratio 1.52 [95% confidence interval 1.46-1.60]) and 27% for Part B (1.27 [1.18-1.38]). Each EPM point awarded for degrees increased the likelihood of passing Part A by 29% (1.29 [1.12-1.48]). SJT score was not a statistically significant independent predictor of MRCS success (P&gt;0.05).

Conclusions

This is the first study to investigate the relationship between UK medical school performance and postgraduate surgical exam success. Higher ranked students are more likely to succeed in the MRCS compared to their lower ranked peers.

220 Choice of UK Medical School Predicts Success in The Intercollegiate Membership of The Royal College of Surgeons (MRCS) Examination

Background

UK medical schools vary in terms of factors such as mission, specific curricula and pedagogy. As relatively little is understood about the impact of these differences at a post-graduate level, we examined the relationship between medical school and MRCS success.

Method

Using the UKMED database we analysed data on UK medical graduates who attempted MRCS Part A (n = 9729) and MRCS Part B (n = 4644) between 2007-2017. Univariate analysis characterised the relationship between medical school and first attempt MRCS success. Logistic regression modelling identified independent predictors of MRCS success.

Results

MRCS pass rates differed significantly between medical schools (P &lt; 0.001). Trainees from standard-entry 5-year programmes were more likely to pass MRCS at first attempt compared to those from extended (Gateway) courses ((Part A (Odds Ratio (OR) 3.72 [95% Confidence Interval (CI) 2.69-5.15]); Part B (OR 1.67 [1.02-2.76])). Non-graduates were more likely to pass Part A (OR 1.40 [1.19-1.64]) and Part B (OR 1.66 [1.24-2.24]). Russell Group graduates were more likely to pass MRCS Part A (OR 1.79 [1.56-2.05]) and Part B (OR 1.24 [1.03-1.49])).

Conclusions

Medical programme and medical school are associated with MRCS success. Further research is needed to tease out the relationship between individual factors, medical school and MRCS performance.

Temporal trends in heart failure medication prescription in a population-based cohort study

OBJECTIVE

We examined temporal heart failure (HF) prescription patterns in a large representative sample of real-world patients in the UK, using electronic health records (EHR).

METHODS

From primary and secondary care EHR, we identified 85 732 patients with a HF diagnosis between 2002 and 2015. Almost 50% of patients with HF were women and the median age was 79.1 (IQR 70.2-85.7) years, with age at diagnosis increasing over time.

RESULTS

We found several trends in pharmacological HF management, including increased beta blocker prescriptions over time (29% in 2002-2005 and 54% in 2013-2015), which was not observed for mineralocorticoid receptor-antagonists (MR-antagonists) (18% in 2002-2005 and 18% in 2013-2015); higher prescription rates of loop diuretics in women and elderly patients together with lower prescription rates of angiotensin-converting enzyme inhibitors and/or angiotensin II receptor blockers, beta blockers or MR-antagonists in these patients; little change in medication prescription rates occurred after 6 months of HF diagnosis and, finally, patients hospitalised for HF who had no recorded follow-up in primary care had considerably lower prescription rates compared with patients with a HF diagnosis in primary care with or without HF hospitalisation.

CONCLUSION

In the general population, the use of MR-antagonists for HF remained low and did not change throughout 13 years of follow-up. For most patients, few changes were seen in pharmacological management of HF in the 6 months following diagnosis.

Protocol for an implementation study of an evidence-based home cardiac rehabilitation programme for people with heart failure and their caregivers in Scotland (SCOT:REACH-HF)

INTRODUCTION

Despite evidence that cardiac rehabilitation (CR) is an essential component of care for people with heart failure, uptake is low. A centre-based format is a known barrier, suggesting that home-based programmes might improve accessibility. The aim of SCOT: Rehabilitation EnAblement in CHronic Heart Failure (REACH-HF) is to assess the implementation of the REACH-HF home-based CR intervention in the context of the National Health Service (NHS) in Scotland.This paper presents the design and protocol for this observational implementation study. Specific objectives of SCOT:REACH-HF are to: (1) assess service-level facilitators and barriers to the implementation of REACH-HF; (2) compare real-world patient and caregiver outcomes to those seen in a prior clinical trial; and (3) estimate the economic (health and social) impact of implementing REACH-HF in Scotland.

METHODS AND ANALYSIS

The REACH-HF intervention will be delivered in partnership with four 'Beacon sites' across six NHS Scotland Health Boards, covering rural and urban areas. Health professionals from each site will be trained to facilitate delivery of the 12-week programme to 140 people with heart failure and their caregivers. Patient and caregiver outcomes will be assessed at baseline and 4-month follow-up. Assessments include the Minnesota Living with Heart Failure Questionnaire (MLHFQ), five-dimension EuroQol 5L, Hospital Anxiety and Depression Scale, and the Caregiver Burden Questionnaire. Qualitative interviews will be conducted with up to 20 health professionals involved in programme delivery (eg, cardiac nurses, physiotherapists). 65 facilitator-patient consultations will be audio recorded and assessed for fidelity. Integrative analysis will address key research questions on fidelity, context and CR participant-related outcomes. The SCOT:REACH-HF findings will inform the future potential roll-out of REACH-HF in Scotland.

ETHICS AND DISSEMINATION

The study has been given ethical approval by the West of Scotland Research Ethics Service (reference 20/WS/0038, approved 25 March 2020). Written informed consent will be obtained from all participants. The study is listed on the ISRCTN registry with study ID ISRCTN53784122. The research team will ensure that the study is conducted in accordance with both General Data Protection Regulations and the University of Glasgow's Research Governance Framework. Findings will be reported to the funder and shared with Beacon Sites, to facilitate service evaluation, planning and good practice. To broaden interest in, and understanding of REACH-HF, we will seek to publish in peer-reviewed scientific journals and present at stakeholder events, national and international conferences.

Exploring shared surgical decision-making from the patient's perspective: is the personality of the surgeon important?

AIM

The aim was to determine the importance of a colorectal surgeon's personality to patients and its influence on their decision-making.

METHODS

We present a two-part mixed methods study using the Guidance for Reporting Involvement of Patients and the Public (GRIPP-2) long form. Part 1 was an online survey (25 questions) and Part 2 a face-to-face patient and public involvement exercise. Part 1 included patient demographics, details of surgery, overall patient satisfaction (net promoter score) and patient views on surgeon personality (Gosling 10 Item Personality Index). The thematic analysis of free-text responses generated four themes that were taken forward to Part 2. These themes were used to structure focus group discussions on surgeon-patient interactions.

RESULTS

Part 1 yielded 296 responses: 72% women, 75.3% UK-based and 55.1% aged 40-59 years. Inflammatory bowel disease (45.3%) and cancer (40.2%) were the main indications. 84.1% of respondents reported satisfaction with their surgical experience (net promoter score). Four key themes were generated from Part 1 and validated in Part 2: (i) surgeon personality stereotypes (media differed from patients' perspective); (ii) favourable and unfavourable surgical personality traits (openness, conscientiousness, emotional stability preferred over risk-taking and narcissism); (iii) patient-surgeon interaction (mutual respect and rapport valued); (iv) impact of surgeon personality on decision-making (majority unaware of second opinion option; management of postoperative complications).

CONCLUSION

Patients believe surgeon personality influences shared decision-making. Low levels of emotional stability and conscientiousness are perceived by patients to increase the likelihood of postoperative adverse events. Further work is required to explore the potential influence of surgeon personality on shared decision-making and postoperative outcomes.

Technology-supported home monitoring in heart failure patients

Background

Digital health promises to enhance the prevailing episodic models of chronic heart failure (HF) care.

Purpose

We aimed to test the hypothesis that digital home monitoring with centralised specialist support for remote management of HF and major vascular comorbidities is more effective in optimising medical therapy and improving patients' quality of life than digital home monitoring alone.

Methods and results

In a two-armed partially blinded parallel randomised controlled trial, seven sites in the United Kingdom recruited a total of 202 adults with HF (71.3 years SD 11.1; mean left ventricular ejection fraction 32.9% SD 15.4). Participants were selected for being at high risk of adverse outcomes or high potential to benefit from remote management. Participants in both study arms were given an internet-enabled tablet computer, Bluetooth-enabled blood pressure monitor and weighing scales for health monitoring. After a run-in period, participants randomized to intervention received additional regular feedback to support self-management and their primary care doctors received instructions on blood investigations and pharmacological treatment. The primary outcome was the use of recommended medical therapy, for chronic HF and major comorbidities, measured as a composite opportunity score. Co-primary outcome was change in physical score of Minnesota Living with Heart failure questionnaire.

At the end of the trial, the weighted opportunity score was 0.54 (CI 95% 0.46, 0.62) in the control group and 0.61 (CI 95% 0.52, 0.70) in the intervention arm (p for mean difference=0.25). Physical well-being of participants did not differ significantly between the groups either (p=0.55).

Conclusions

Central provision of tailored specialist management in a multimorbid HF population was feasible. However, there was no strong evidence for improvement in use of evidence-based therapies nor health-related quality of life.

Figure 1

Funding Acknowledgement

Type of funding source: Public Institution(s). Main funding source(s): National Institute for Health Research (NIHR) Health Services Research and Delivery; NIHR Career Development Fellowship

A comparison of machine learning models for predicting rehospitalisation and death after a first hospitalisation with heart failure

Background

Many machine learning models exist, including Multilayer Perceptron (MLP), Random Forest algorithm (RF), Support Vector Machine (SVM), and Gradient Boosted Machine (GBM), but their value for predicting outcome in patients with heart failure has not been compared.

Aim

To predict rehospitalisation (all-cause) and death (all-cause) at 1-, 3- and 12 months after discharge from a first hospitalisation for heart failure using four machine learning models.

Methods

The National Health Service Greater Glasgow and Clyde Health Board serves a population of ∼1.1 million. We obtained de-identified administrative data, including investigations, diagnosis and prescriptions, linked to hospital admissions and deaths for anyone with a diagnosis of vascular disease or heart failure or prescribed loop diuretics, statins or neuro-endocrine antagonists at any time between 1st January 2010 and 1st June 2018. Patients who were under 18 or had no prior hospitalisation for heart failure were excluded. Four ML algorithms using 46 variables were applied.

Results

Of 360,000 people who met the above criteria between 2010–2018, 6,372 had a hospitalisation for heart failure prior to 1st January 2010 and 8,304 had a first hospitalisation for heart failure thereafter. Between 2010 and 2018 there were 3,086 re-hospitalisations over 24 hours and 3,706 patients died, with 5,070 patients experiencing the composite outcome.

GBM and RF consistently outperformed MLP and SVM when comparing AUC, sensitivity and specificity combined, with GBM performing best in all scenarios. Since GBM and RF are both tree-based models, and with SVM and MLP regularly reporting very poor sensitivity or specificity despite a similar AUC to the others, this suggests that SVM and MLP may be suffering from overfitting and might perform better in larger data-sets.

Both GBM and RF work by ordering variables, so the final model can be used to determine the most important prediction variables. Age, number of times a blood sample was taken out of hospital, length of stay, social deprivation index and haemoglobin concentration consistently ranked amongst the most important variables. Models predicted all 1-month events better than later events.

Conclusions

Some, but not all, ML models applied to this data-set predicted rehospitalisation and death with great accuracy for up to 3 months after a first hospitalisation for heart failure. The models identified several important prognostic variables that are currently seldom collected in clinical research registries but perhaps should be.

Funding Acknowledgement

Type of funding source: Public grant(s) – National budget only. Main funding source(s): Medical Research Council

Use of social media by health care professionals: is patients' privacy respected?

Background

The internet has a key role in sharing and expanding medical knowledge. Social networks offer health-care professionals the possibility to communicate, debate and learn from each other in real-time, thereby improving access to expertise and creating new knowledge. On Facebook, there are many groups that health-care professionals can join to discuss clinical cases. However, it is unknown if patient-privacy is respected and whether users are aware that they might be inadvertently involved in a criminal act for which they might be sued.

Methods

We identified the most popular Facebook group for cardiologists, “ECHO BOARD REVIEW forum” that has more than 47,000 users. We retrospectively evaluated all content posted by this group between February 6th and February 14th 2020. Information about the type of data posted, personal patient details, geographic location reported on images or videos, number of reactions and comments was collected. Privacy was considered fully violated when name and/or surname of the patient was identifiable. A comparison between the two cases (privacy preserved vs. privacy violated) was performed.

Results

Of 53 posts evaluated, 50 (94%) were echocardiograms; the remaining three cases were of an electrocardiogram, a cardiac magnetic resonance video and a coronary angiogram. The patient's identity was revealed in seven cases (13%). Compared to cases where identity was concealed, those revealing the patient's identify were more likely to report other important personal details including date of birth (n=0 (0%) vs 2 (29%)), age (n=14 (30%) vs 5 (71%)) and sex (n=15 (33%) vs 6 (86%)). The country (n=46, 87%), city (n=39, 74%), and hospital (n=34, 64%) where the cases were being evaluated were also frequently disclosed. Most cases were from Asia (n=23, 43%) or Africa (n=17, 32%). Of the 7 cases in which privacy was fully violated, 43% were from Africa, 27% were from South America and 14% were from Asia. In the majority of cases, the author of the post was seeking diagnostic help (n=33), less frequently a diagnosis was already made by the author (n=17). The median (interquartile range) number of comments/post was 12 (5–23), and of likes/post was 21 (12- 37), with no significant difference between cases in which privacy was violated or not.

Conclusion

Social media allows knowledge and expertise to be shared amongst health care professionals, but, alarmingly, violation of patient-confidentiality is common. In order to maintain patient-confidentiality and avoid breaking the law, strict rules should be applied to regulate the use of social media by health-care professionals.

Funding Acknowledgement

Type of funding source: None

Prediction of success at UK Specialty Board Examinations using the mandatory postgraduate UK surgical examination

Background

While performance in other mandatory examinations taken at the beginning of a doctor's career are predictive of final training outcomes, the influence early postgraduate surgical examinations might have on success at Specialty Board Exams in the UK is currently unknown. The aim was to investigate whether performance at the mandatory Membership of the Royal College of Surgeons (MRCS) examination, and other variables, are predictive of success at the Fellowship of the Royal College of Surgeons (FRCS) examination, thus potentially identifying those who may benefit from early academic intervention.

Methods

Pearson correlation coefficients examined the linear relationship between both examinations and logistic regression analysis identified potential independent predictors of FRCS success. All UK medical graduates who attempted either section of FRCS (Sections 1 and 2) between 2012 and 2018 were included.

Results

First attempt pass rates for Sections 1 and 2 FRCS were 87.4 per cent (n = 854) and 91.8 per cent (n = 797) respectively. In logistic regression analysis, sex (male: odds ratio (OR) 2.32, 95 per cent c.i 1.43 to 3.76), age (less than 29 years at graduation: OR 3.22, 1.88 to 5.51), Part B MRCS attempts (1 attempt: OR 1.77, 1.08 to 3.00), Part A score (OR 1.14, 1.09 to 1.89) and Part B score (OR 1.06, 1.03 to 1.09) were independent predictors of Section 1 FRCS success. Predictors of Section 2 FRCS success were age (less than 29 years at graduation: OR 3.55, 2.00 to 6.39), Part A score (OR 1.06, 1.02 to 1.11) and Section 1 FRCS score (OR 1.13, 1.07 to 1.18).

Conclusion

Part A and B MRCS performance were independent predictors of FRCS success, providing further evidence to support the predictive validity of this mandatory postgraduate exam. However, future research must explore the reasons between the attainment gaps observed for different groups of doctors.

AMEE Guide No. 123 - How to read studies of educational costs

Healthcare and health professions education share many of the same problems in decision making. In both cases, there is a finite amount of resources, and so choices need to be made between alternatives. To navigate the options available requires effective decision making. Choosing one option requires consideration of its opportunity cost - the benefit forgone of the other competing options. The purpose of this abridged AMEE guide is to introduce educational decision-makers to the economic concept of cost, and how to read studies about educational costs to inform effective cost-conscious decision-making. This guide leads with a brief review of study designs commonly utilized in this field of research, followed by an overview of how study findings are commonly presented. The tutorial will then offer a four-step model for appraising and considering the results of an economic evaluation. It asks the questions: (1) Can I trust the results? (2) What are the results telling me? (3) Could the results be transferred to my context? (4) Should I change my practice?

The effectiveness of superficial versus deep dry needling or acupuncture for reducing pain and disability in individuals with spine-related painful conditions: a systematic review with meta-analysis

Objective: The purpose of this systematic review was to evaluate the effects of deep versus superficial dry needling or acupuncture on pain and disability for spine-related painful conditions. A secondary purpose was to account for the differences of needling location in relation to the painful area. Methods: This PROSPERO (#CRD42018106237) registered review found 691 titles through a multi-database search. Following a comprehensive search, 12 manuscripts were included in the systematic review and 10 in the meta-analysis. Standardized mean differences (SMD) with 95% confidence intervals were calculated for pain and disability. Results: The included studies demonstrated an unclear to high risk of bias recommending a cautious interpretation of the results. A consistent effect supporting deep needling over superficial with an SMD of 0.585 [0.335, 0.835], p < 0.001 from 10 articles for pain but a non-significant effect of 0.197 [-0.066, 0.461], p = 0.14 from 2 studies for disability. A temporal examination was similar for effects on pain with an SMD of 0.450 [0.104, 0.796] immediately, 0.711 [0.375, 1.048] short-term (1 to 11 weeks), and 0.470 [0.135, 0.805] for time-points ≥12 weeks. Regionally, there was a greater effect needling the area of pain locally (SMD = 0.754) compared to remotely (SMD = 0.501). Discussion: Statistically significant between-group differences were observed favoring deep needling over superficial. Both superficial and deep needling resulted in clinically meaningful changes in pain scores over time. However, differences between groups may not be clinically meaningful. More high-quality trials are needed to better estimate the effect size of deep versus superficial needling while controlling for location and depth of the lesion. Level of evidence: 1a.

The effects of human factor related issues on assessors during the recruitment process for general and vascular surgery in the UK

INTRODUCTION

Healthcare professionals increasingly recognise how human factors (HF) can contribute to medical error. An understanding of HF is also important during other high stakes activities such as summative assessments or examinations. National organisations hosting such events need to be aware of potential fatigue, boredom and stress that can occur in interviewers during these often repetitive activities.

METHODS

A previously validated questionnaire based around the well known HF analysis and classification system (HFACS) was used to evaluate four factors (care and support, asking questions about the role as an interviewer, working within the rules and boundaries, and stress and pressure) at the 2018 UK general and vascular surgery trainee national selection process.

RESULTS

A total of 92 questionnaires were completed and analysed (48% response rate). After recoding for negative phrased questions, no significant differences were found between years of experience in the selection process and mean scores obtained for all four factor items. Interviewers had a positive experience during national selection with mean factor scores ranging from 3.84 to 3.98 (out of a maximum satisfaction score of 5).

CONCLUSIONS

Organisations need to plan carefully and recognise the human element to ensure that their interviewers are cared for during any high stakes assessment such as national selection. Our work suggests that a positive assessor experience will further help contribute to a reliable and fair recruitment process.

2018 Ottawa consensus statement: Selection and recruitment to the healthcare professions

Selection and recruitment into healthcare education and practice is a key area of interest for educators with significant developments in research, policy, and practice in recent years. This updated consensus statement, developed through a multi-stage process, examines future opportunities and challenges in selection and recruitment. There is both a gap in the literature around and a compelling case for further theoretical and empirical literature to underpin the development of overall selection philosophes and policies and their enactment. More consistent evidence has emerged regarding the quality of different selection methods. Approaches to selection are context-dependent, requiring the consideration of an institution's philosophy regarding what they are trying to achieve, the communities it purports to serve, along with the system within which they are used. Diversity and globalization issues continue to be critically important topics. Further research is required to explore differential attainment and explain why there are substantial differences in culturally acceptable ways of approaching diversity and widening access. More sophisticated evaluation approaches using multi-disciplinary theoretical frameworks are required to address the issues. Following a discussion of these areas, 10 recommendations are presented to guide future research and practice and to encourage debate between colleagues across the globe.

Does the Intercollegiate Membership of the Royal College of Surgeons (MRCS) examination predict 'on-the-job' performance during UK higher specialty surgical training?

INTRODUCTION

The Intercollegiate Membership of the Royal College of Surgeons (MRCS) is a mandatory examination to enter higher surgical specialty training in the UK. It is designed to help to ensure that successful candidates are competent to practice as higher surgical trainees. The annual review of competence progression (ARCP) assesses trainees' competence to progress to the next level of training and can be interpreted as a measure of 'on-the-job' performance. We investigated the relationship between MRCS performance and ARCP outcomes.

MATERIALS AND METHODS

All UK medical graduates who passed MRCS (Parts A and B) from 2007 to 2016 were included. MRCS scores, attempts and sociodemographics for each candidate were crosslinked with ARCP outcomes (satisfactory, unsatisfactory and insufficient evidence). Multinomial logistic regression was used to identify potential independent predictors of ARCP outcomes.

RESULTS

A total of 2570 trainees underwent 11,064 ARCPs; 1589 (61.8%) had only satisfactory outcomes recorded throughout training; 510 (19.9%) had at least one unsatisfactory outcome; and 471 (18.3%) supplied insufficient evidence. After adjusting for age, gender, first language and Part A performance, ethnicity (non-white vs white, OR 1.36, 95% CI 1.08 to 1.71), Part B passing score (OR 0.98, 95% CI 0.98 to 1.00) and number of attempts at Part B (two or more attempts vs one attempt, OR 1.50, 95% CI 1.16 to 1.94) were found to be independent predictors of an unsatisfactory ARCP outcome.

CONCLUSIONS

This is the first study to identify predictors of ARCP outcomes during higher surgical specialty training in the UK and provides further evidence of the predictive validity of the MRCS examination.

Paper versus electronic feedback in high stakes assessment

Tablet computers have emerged as increasingly useful tools in medical education, particularly for assessment. However, it is not fully established whether tablet computers influence the quality and/or quantity of feedback provided in high stakes assessments. It is also unclear how electronically-recorded feedback relates to student performance. Our primary aim was to determine whether differences existed in feedback depending on the tool used to record it.

METHODS

We compared quantitative and qualitative feedback between paper-scoring sheets versus iPads™ across two consecutive years of a final year MBChB (UK medical degree) Objective Structured Clinical Examination. Quality of comments (using a validated five-point rating scale), number of examiner comments and number of words were compared across both methods of recording assessment performance using chi-squared analysis and independent t-test. We also explored relationships between student performance (checklist and global scoring) and feedback.

RESULTS

Data from 190 students (2850 paper scored interactions) in 2015 and 193 (2895 iPad™ scored interactions) in 2016 were analysed. Overall, a greater number of comments were given with iPad™ compared to written (42% versus 20%; p < 0.001) but the quality of feedback did not differ significantly. For both written and electronic feedback, students with low global scores were more likely to receive comments (p < 0.001).

CONCLUSION

The use of iPads™ in high stakes assessment increases the quantity of feedback compared to traditional paper scoring sheets. The quantity and quality of feedback for poorer performing candidates (by global score) were also better with iPad™ feedback.

Factors predicting success in the Intercollegiate Membership of the Royal College of Surgeons (MRCS) examination: a summary for OMFS

The Intercollegiate Membership of the Royal College of Surgeons (MRCS, parts A and B) is a mandatory examination for entry into higher surgical training in the UK. We investigated which factors predict success in both the written (Part A) and clinical (Part B) parts of the examination, and provide a summary for oral and maxillofacial surgeons (OMFS). All UK graduates who attempted both parts between 2007 and 2016 were included. There was a positive correlation between the scores in parts A and B (r=0.41, p<0.01). For Part A, men (odds ratio (OR) 2.78; 95% CI 1.83 to 4.19), white candidates (OR 1.70; 95% CI 1.52 to 1.89), and younger graduates (under 29years of age), were more likely to pass (OR 2.60; 95% CI 1.81 to 3.63). Foundation year one (FY1) doctors had higher pass rates than all other grades (e.g. core surgical trainee 2 compared with FY1 OR 0.50; 95% CI 0.32 to 0.77). The number of attempts at Part A and the final score, as well as ethnicity and stage of training, were independent predictors of success in Part B. Candidates who did well in Part A were more likely to do well in Part B. Several independent predictors of success were identified, but only the stage of training and ethnicity were common predictors in both parts. Higher scores obtained by younger candidates might be relevant to OMFS trainees who take Part A after studying medicine as a second degree.

Predictors of success in the Intercollegiate Membership of the Royal College of Surgeons (MRCS) examination

Introduction The Intercollegiate Membership of the Royal College of Surgeons (MRCS, Parts A and B) is designed to safeguard patients and ensure high standards for trainees wishing to pursue a surgical career. We investigated which factors predict Part A and B success. Methods UK graduates who had attempted both parts of MRCS from September 2007 to February 2016 were included. Pearson correlation coefficients were used to examine the linear relationship between each part of MRCS and logistic regression analysis to identify potential independent predictors of MRCS pass/fail outcome. Results A positive correlation between Part A and B score was identified (r = 0.41, P < 0.01). In Part A, males had higher pass rates than females (odds ratio, OR, 2.78, 95% confidence interval, CI, 1.83-4.19), white candidates were more likely to pass than black minority and ethnic doctors (OR 1.70, 95% CI 1.52-1.89), foundation year 1 doctors had higher pass rates than all other grades (e.g. core surgical year 2 trainees vs. foundation year 1 doctors, OR 0.50, 95% CI 0.32-0.77) and younger graduates were more likely to pass than older graduates (OR 2.60, 95% CI 1.81-3.63). In addition to ethnicity and stage of training, Part A performance (number of attempts and score) was an independent predictor for Part B. Conclusions Part A performance is related to Part B score. Many independent predictors of MRCS success were identified, but only stage of training and ethnicity were found to be common predictors of both parts. This article summarises the findings of our most recent MRCS study which was published in The Surgeon (online) in October 2017. ¹.

Impact of performance in a mandatory postgraduate surgical examination on selection into specialty training

Background

The Intercollegiate Membership of the Royal College of Surgeons (MRCS) examination is undertaken by large numbers of trainees in the UK and internationally as a mandatory step within surgical training. Unlike some high-stakes medical examinations, the MRCS is yet to be validated. A quantitative study was undertaken to assess its predictive validity by investigating the relationship between MRCS (Parts A and B) and national selection interview scores for general and vascular surgery in the UK.

Methods

Pearson correlation coefficients were used to examine the linear relationship between each assessment, and linear regression analyses were employed to identify potential independent predictors of the national selection score. All UK medical graduates who attempted the interview in 2011-2015 were included.

Results

Some 84·4 per cent of the candidates (1231 of 1458) were matched with MRCS data. There was a significant positive correlation between the first attempt score at Part B of the MRCS examination and the national selection score (r = 0·38, P < 0·001). In multivariable analysis, 17 per cent of variance in the national selection first attempt score was explained by the Part B MRCS score and number of attempts (change in R ² value of 0·10 and 0·07 respectively; P < 0·001). Candidates who required more than two attempts at Part B were predicted to score 8·1 per cent less than equally matched candidates who passed at their first attempt.

Conclusion

This study supports validity of the MRCS examination, and indicates its predictive value regarding entry into specialist training.

Tablet computers in assessing performance in a high stakes exam: opinion matters

Background Tablet computers have emerged as a tool to capture, process and store data in examinations, yet evidence relating to their acceptability and usefulness in assessment is limited. Methods We performed an observational study to explore opinions and attitudes relating to tablet computer use in recording performance in a final year objective structured clinical examination at a single UK medical school. Examiners completed a short questionnaire encompassing background, forced-choice and open questions. Forced choice questions were analysed using descriptive statistics and open questions by framework analysis. Results Ninety-two (97% response rate) examiners completed the questionnaire of whom 85% had previous use of tablet computers. Ninety per cent felt checklist mark allocation was 'very/quite easy', while approximately half considered recording 'free-type' comments was 'easy/very easy'. Greater overall efficiency of marking and resource savings were considered the main advantages of tablet computers, while concerns relating to technological failure and ability to record free type comments were raised. Discussion In a context where examiners were familiar with tablet computers, they were preferred to paper checklists, although concerns were raised. This study adds to the limited literature underpinning the use of electronic devices as acceptable tools in objective structured clinical examinations.

Effectiveness of manual therapy versus surgery in pain processing due to carpal tunnel syndrome: A randomized clinical trial

BACKGROUND

People with carpal tunnel syndrome (CTS) exhibit widespread pressure pain and thermal pain hypersensitivity as a manifestation of central sensitization. The aim of our study was to compare the effectiveness of manual therapy versus surgery for improving pain and nociceptive gain processing in people with CTS.

METHODS

The trial was conducted at a local regional Hospital in Madrid, Spain from August 2014 to February 2015. In this randomized parallel-group, blinded, clinical trial, 100 women with CTS were randomly allocated to either manual therapy (n = 50), who received three sessions (once/week) of manual therapies including desensitization manoeuvres of the central nervous system, or surgical intervention (n = 50) group. Outcomes including pressure pain thresholds (PPT), thermal pain thresholds (HPT or CPT), and pain intensity which were assessed at baseline, and 3, 6, 9 and 12 months after the intervention by an assessor unaware of group assignment. Analysis was by intention to treat with mixed ANCOVAs adjusted for baseline scores.

RESULTS

At 12 months, 95 women completed the follow-up. Patients receiving manual therapy exhibited higher increases in PPT over the carpal tunnel at 3, 6 and 9 months (all, p < 0.01) and higher decrease of pain intensity at 3 month follow-up (p < 0.001) than those receiving surgery. No significant differences were observed between groups for the remaining outcomes.

CONCLUSIONS

Manual therapy and surgery have similar effects on decreasing widespread pressure pain sensitivity and pain intensity in women with CTS. Neither manual therapy nor surgery resulted in changes in thermal pain sensitivity.

SIGNIFICANCE

The current study found that manual therapy and surgery exhibited similar effects on decreasing widespread pressure pain sensitivity and pain intensity in women with carpal tunnel syndrome at medium- and long-term follow-ups investigating changes in nociceptive gain processing after treatment in carpal tunnel syndrome.

Management of COPD in Primary Care in North-East Scotland

Introduction

We wished to obtain a snapshot of current service provision and how this could best be developed approximately one year on from the introduction of the National Institute for Clinical Excellence (NICE) guidelines for the management of chronic obstructive pulmonary disease (COPD) and the inclusion of COPD care in the New GMS Contract Quality and Outcomes Framework (QOF).

Methodology

A questionnaire-based survey sent to every general practice (n = 84) in Grampian.

Results

Responses were received from 75 of 84 practices (89%). Questionnaires were returned by both general practitioners (GPs) and practice nurses in 45 practices (54%). All responding practices reported that they had COPD registers. 60/75 (80%) of practices reported having a dedicated COPD clinic; 70/75 (93%) had a spirometer. Areas identified for service development were: quality assuring training in COPD care and spirometry; expanding pulmonary rehabilitation provision (86%), delivering this service locally (54%) and in primary care (75%); standardising referral, assessment and communication about provision of home oxygen; training in pulse oximetry (71%).

Conclusion

This data has important implications for the validity of the quality indicators (QOF) under the new GMS contract. Our respondents identified areas where the new GMS contract QOF could be improved, as well as providing useful suggestions for service development. Respondents recognised that not all clinical services can be effectively delivered by general practice with data supporting the development of intermediate care services for people with COPD.