Assessing professional competence: a critical review of the Annual Review of Competence Progression

The Impact of the Covid-19 Pandemic on Annual Review of Competency Progression Outcomes Issued to General Surgical Trainees

BACKGROUND

The Covid-19 pandemic had a profound impact on surgical training. In this longitudinal cohort study, we quantify the effects of the pandemic on United Kingdom (UK) surgeons in higher specialty training by analyzing the Annual Review of Competency Progression (ARCP) Outcomes issued to them prior to, and during, the pandemic.

METHODS

Anonymized records were provided from the UK training management system- the Intercollegiate Surgical Curriculum Programme (ISCP)- on the ARCP Outcomes of higher specialty trainees in General Surgery over the period between January 2017 and December 2022. Demographic data including sex and age group on starting higher specialty training were considered, as were working pattern, phase of training during the height of the pandemic (2020 and 2021), and training region. The proportion of nonstandard outcomes, and the use of specific Covid-19 outcomes, were analyzed to assess the impact of these variables on ARCP outcome using univariate and multivariate logistic regression. Prepandemic outcomes in 2017 were used as a comparator.

RESULTS

A total of 7414 ARCP outcomes issued to 1874 General Surgery higher speciality trainees were analysed. The Adjusted Odds Ratio (AOR) for receiving a nonstandard outcome in 2020 (compared to 2017) was 3.07 (95% CI: 2.47-3.81, p < 0.001) not recovering to prepandemic levels by the end of 2022 (AOR 2.11 (95% CI: 1.69-2.64, p < 0.001)). Female sex (AOR 1.27 (95% CI: 1.13-1.43, p < 0.001) and being older on starting higher surgical training (AOR = 1.51 (95% CI: 1.34-1.70, p < 0.001) were both significantly associated with a higher chance of nonstandard outcome. Working pattern was linked to ARCP outcome on univariate analysis, but this relationship disappeared once corrected for other demographic factors (1.05, 95% CI: 0.88-1.24, p = 0.582). Being at a later stage of training during the pandemic was not linked to an increase in AOR of receiving a nonstandard outcome (1.09, 95% CI: 0.97-1.22, p = 0.134), but trainees receiving a nonstandard outcome in this group were more likely to have extra training time advised (15.49%, vs 4.27% in 2021). The highest AOR of receiving a Covid-19 outcome was in the Wessex Deanery at 2.85 (95% CI: 1.83-4.46, p < 0.001), whilst the lowest AOR were seen in Yorkshire and the Humber (0.32, 95% CI: 0.17-0.62, p < 0.001). Removing Covid-19 specific outcomes from the analysis shows a continued rise in the use of nonstandard outcomes in all years except 2020.

CONCLUSIONS

The Covid-19 Pandemic had a significant impact on the trajectory of training in General Surgery in the UK. Training extensions were more likely to be recommended later in training. There was considerable variation in the use of Covid-19 ARCP outcomes across the UK. There is ongoing evidence of differential attainment at ARCP in General Surgery, with female trainees and older graduates having greater chances of nonstandard outcomes. The underlying reasons for these associations need to be explored. Efforts to urgently address deficits in training post Covid-19 with an awareness of the intersectional nature of differential attainment are needed.

Designing a doctor evaluation index system for an online medical platform based on the information system success model in China

Objective

In the context of "internet + medical health" and emphasis on evaluation mechanism for medical and health talents in China, we design an evaluation index system for doctors on online medical platforms by synthesizing two patterns of existing online medical platforms, which is the first step to enhance the capabilities of doctors on online medical platforms.

Methods

Based on the doctor evaluation model integrating information systems success model (ISS-DE model) and grounded theory, the evaluation indicators were obtained through expert interviews, offline medical institutions investigation, online platforms investigation, and literature research, and were assigned weights using the analytic hierarchy process (AHP) method. A working group composed of 23 experts was set up to review and determine the competency standards of doctors on the online medical platforms.

Results

A new indicator framework covering 3 dimensions of system quality, service quality and information quality was constructed in this study. The index system included 3 first-level indicators, 8 s-level indicators and 60 third-level indicators, and each indicator was given different weightage.

Conclusion

The complete index system constructed by the Delphi method in this study is suitable for China's online medical platforms, which will help to improve the quality of platforms and the ability of doctors, thus promoting the process of internet medical integration.

Perceptions of the impact of annual review of competence progression (ARCP): a mixed methods case study

The annual review of competence progression (ARCP) is a high-stakes assessment which all UK postgraduate trainees undertake to ensure competence progression. Previous evaluations of the effectiveness of the ARCP as an assessment have reported deficiencies in both validity and reliability, however, there has been little focus on the educational impact of the ARCP.We conducted a mixed methods case study involving questionnaire, interviews and a focus group examining the impact of the ARCP on a respiratory higher specialist training programme. Participants included both trainers and trainees.Perceptions of impact were mixed. The ARCP was reported to promote broad curriculum coverage, enable educational planning, provide educational governance and facilitate relationships with supervisors. However, participants reported that activities promoted by the ARCP may detract from learning and that issues of reliability and validity undermined the process. In some cases, this was reported to lead to disillusionment and stress for trainees. Concerns were raised that the process promoted a reductionist approach to education.This research has resulted in several changes to local training, however, it has potential implications for the ARCP as a wider process. Trainers should be cognisant of the shortcomings of assessments and their impact on trainees, training and the future of the profession.

Multiprofessional leadership development: reflections from a doctor in training

Background

In July 2020, the National Health Service (NHS) People Plan was refreshed, giving further impetus to staff development and leadership training. Through a series of interwoven tales, I discuss my own journey of leadership development and offer an analysis of the value of dedicated courses and the importance of providing this to the wider workforce.

Story of self

I am a doctor in training and was among the first three cohorts placed onto the new Rosalind Franklin programme, organised by the NHS Leadership Academy. I share my key reflections of the impact of this course on my personal and professional development.

Story of us

My cohort contained professionals from a diverse range of backgrounds-their challenges, views and insights contrasted greatly with my own. Having the protected time to build trust, form teams and discuss issues that crossed organisational boundaries provided novel insights that helped all of us.

Story of now

As the COVID-19 pandemic has taken hold, we are in a state of extreme flux. As a result, I have become aware of how important it is to marry expertise with generalist skills and knowledge of the wider healthcare system. Enduring the initial surge of COVID-19 was about staff working together and blending specialism with generalist pragmatism. The ability to harness and sustain this type of working will represent a legacy from COVID-19 that is positive and one which galvanises our greatest asset-the talents and experiences of our diverse workforce-in order to meet future healthcare challenges.

'10% of your medical students will cause 90% of your problems': a prospective correlational study

OBJECTIVES

Our aim was to explore the relationship between medical student Conscientiousness Index scores and indicators of later clinical performance held in the UK Medical Education Database (UKMED). Objectives were to determine whether conscientiousness in first-year and second-year medical students predicts later performance in medical school and in early practice. Policy implications would permit targeted remediation where necessary or aid in selection.

DESIGN

A prospective correlational study.

SETTING

A single UK medical school and early years of practice, 2005-2018.

PARTICIPANTS

The data were obtained from the UKMED on 858 students. Full outcome data was available for variable numbers of participants, as described in the text.

MAIN OUTCOME MEASURES

Scores on the UK Foundation Programme Office's Situational Judgement Test (SJT) and Educational Performance Measure (EPM), the Prescribing Safety Assessment (PSA) and Annual Review of Competency Progression (ARCP) outcomes.

RESULTS

Linear regression analysis shows Conscientiousness Index scores significantly correlate with pregraduate and postgraduate performance variables: SJT scores (R=0.373, R²=0.139, B=0.066, p<0.001, n=539); PSA scores (R=0.249, R²=0.062, B=0.343, p<0.001, n=462); EPM decile scores for the first (lowest) decile are significantly lower than the remaining 90% (p=0.003, n=539), as are PSA scores (p<0.001, n=463), and ARCP year 2 scores (p=0.019, n=517). The OR that students in the first decile fail to achieve the optimum ARCP outcome is 1.6126 (CI: 1.1400 to 2.2809, p=0.0069, n=618).

CONCLUSIONS

Conscientiousness Index scores in years 1 and 2 of medical school have predictive value for later performance in knowledge, skills and clinical practice. This trait could be used either for selection or for targeted remediation to avoid potential problems in the future.

Improving the digital health of the workforce in the COVID-19 context: an opportunity to future-proof medical training

As well as its profound effects on healthcare and wider society, the COVID-19 pandemic will have far-reaching implications for the future training and professional development of healthcare workers and, in particular, doctors. While initial educational priorities focused on creating a more agile workforce with better cross-specialty skill-mix, attention must now shift to how our system can prepare a proportionate response that not only addresses the needs of the pandemic but also the underlying challenges of healthcare: multimorbidity, bridging the evidence-practice gap and delivering integrated, personalised medicine for all. It is our contention that meeting such challenges will require a rapid upskilling of the digital capabilities of the healthcare workforce. In short, optimising the health of the nation will depend, in part, on improving the digital health of the workforce. In this review, we examine how digital technology played its part in the COVID-19 response, and how fundamental changes to medical training are urgently needed in the context of a 'healthcare reset'. Familiarity with health informatics, data science and digital technology have to move to centre stage in order to future-proof our profession in the years to come. The people that deliver care are our systems' greatest asset, and at a time when change is accelerating, we cannot knowingly allow current and future colleagues to be ill-equipped to survive and thrive in the practice of medicine.

Construction of the matrix of individual nursing competences in surgical units

OBJECTIVE

to build and to describe an Individual Skills Matrix for nurses working in surgical units and their associated behaviors / attitudes.

METHOD

Exploratory, qualitative study. 43 nurses from five surgical units participated, and data collection was carried out between April and September 2017. The focus group technique was used and for data interpretation an inductive thematic analysis was performed. The competence matrix was built from the testimonies of the participants plus a search in the literature for concepts directed to each competence and description of the expected behaviors and / or attitudes.

RESULTS

For the Matrix, the following individual skills were identified: Planning; Communication, Relational Competence, Leadership, Decision Making and Ethics.

FINAL CONSIDERATIONS

The construction of a Matrix should assist managers in recognizing the professional profile and assessing their performance, strengthening the achievement of professional and organizational objectives, as well as contributing to the quality and effectiveness of the care provided by nurses in these places.