Implementation of a formal in-training assessment programme in anaesthesiology and preliminary results of acceptability

Re-examining single-moment-in-time high-stakes examinations in specialist training: A critical narrative review

In this critical narrative review, we challenge the belief that single-moment-in-time high-stakes examinations (SMITHSEx) are an essential component of contemporary specialist training. We explore the arguments both for and against SMITHSEx, examine potential alternatives, and discuss the barriers to change.SMITHSEx are viewed as the "gold standard" assessment of competence but focus excessively on knowledge assessment rather than capturing essential competencies required for safe and competent workplace performance. Contrary to popular belief, regulatory bodies do not mandate SMITHSEx in specialist training. Though acting as significant drivers of learning and professional identity formation, these attributes are not exclusive to SMITHSEx.Skills such as crisis management, procedural skills, professionalism, communication, collaboration, lifelong learning, reflection on practice, and judgement are often overlooked by SMITHSEx. Their inherent design raises questions about the validity and objectivity of SMITHSEx as a measure of workplace competence. They have a detrimental impact on trainee well-being, contributing to burnout and differential attainment.Alternatives to SMITHSEx include continuous low-stakes assessments throughout training, ongoing evaluation of competence in the workplace, and competency-based medical education (CBME) concepts. These aim to provide a more comprehensive and context-specific assessment of trainees' competence while also improving trainee welfare.Specialist training colleges should evolve from exam providers to holistic education sources. Assessments should emphasise essential practical knowledge over trivia, align with clinical practice, aid learning, and be part of a diverse toolkit. Eliminating SMITHSEx from specialist training will foster a competency-based approach, benefiting future medical professionals' well-being and success.

Development and validation of a postgraduate anaesthesiology core curriculum based on Entrustable Professional Activities: a Delphi study

Background: Postgraduate training curricula should not be based on time-spans or predefined numbers of performed procedures. One approach to link competencies to clinical tasks is the concept of Entrustable Professional Activities (EPA). The goal of this study was the definition, ranking and validation of EPAs for anaesthesiology postgraduate training and the creation of an anaesthesiologic core curriculum. Methods: Anaesthesiologists of different levels of training participated in the study (single-center, cross-sectional) . First, an expert group defined a preliminary list of EPAs. Then a first Delphi round (n= 47 participants) was applied to identify daily anaesthesiology tasks with the goal to define EPAs. From the first Delphi round a new set of EPAs was defined, using the template and mapping method. Through an alignment process, conducted by the expert group, the preliminary EPAs and the new set of EPAs from the first Delphi round were summarised into a new list of EPAs. This list was presented to the study participants in a second Delphi round (n=80 participants), with the goal to validate and rank each EPA and to define the year of entrustment. For this purpose, participants were asked in the second Delphi round if each EPA should be included into an anaesthesiology core curriculum and in which year of training entrustment should take place. Furthermore, they were asked to rank each EPA on a numeric scale, defining its importance. From this numeric scale, the content validity index (CVI) for each EPA was calculated. Consensus of the results from the second Delphi round was calculated, using the one-way random effects model to calculate Intra-Class-Correlations (ICC). Percentages of agreement among the whole set of EPAs of this study and a previously published set of EPAs were computed. Results: A core-curriculum comprising of 39 EPAs was developed. The EPAs were subdivided into superior/high and inferior/low scoring EPAs, reflecting their importance and were mapped to the year of entrustment. The results reached high consensus among the different participating anaesthesiologist groups (overall agreement was 0.96 for the CVI of each EPA and 0.83 for the year in which the EPAs should be entrusted). Agreement with the previously defined set of EPAs was 73.3%. Conclusion: This study provides a further step in transforming postgraduate anaesthesiology training into a more contemporary approach. Other studies are necessary to complete and amend the presented core curriculum of EPA based postgraduate anaesthesiology training.

Effects of implementing time-variable postgraduate training programmes on the organization of teaching hospital departments

INTRODUCTION

As competency-based education has gained currency in postgraduate medical education, it is acknowledged that trainees, having individual learning curves, acquire the desired competencies at different paces. To accommodate their different learning needs, time-variable curricula have been introduced making training no longer time-bound. This paradigm has many consequences and will, predictably, impact the organization of teaching hospitals. The purpose of this study was to determine the effects of time-variable postgraduate education on the organization of teaching hospital departments.

METHODS

We undertook exploratory case studies into the effects of time-variable training on teaching departments' organization. We held semi-structured interviews with clinical teachers and managers from various hospital departments.

RESULTS

The analysis yielded six effects: (1) time-variable training requires flexible and individual planning, (2) learners must be active and engaged, (3) accelerated learning sometimes comes at the expense of clinical expertise, (4) fast-track training for gifted learners jeopardizes the continuity of care, (5) time-variable training demands more of supervisors, and hence, they need protected time for supervision, and (6) hospital boards should support time-variable training.

CONCLUSIONS

Implementing time-variable education affects various levels within healthcare organizations, including stakeholders not directly involved in medical education. These effects must be considered when implementing time-variable curricula.

The effects of graduate competency-based education and mastery learning on patient care and return on investment: a narrative review of basic anesthetic procedures

BACKGROUND

Despite the widespread implementation of competency-based education, evidence of ensuing enhanced patient care and cost-benefit remains scarce. This narrative review uses the Kirkpatrick/Phillips model to investigate the patient-related and organizational effects of graduate competency-based medical education for five basic anesthetic procedures.

METHODS

The MEDLINE, ERIC, CINAHL, and Embase databases were searched for papers reporting results in Kirkpatrick/Phillips levels 3-5 from graduate competency-based education for five basic anesthetic procedures. A gray literature search was conducted by reference search in Google Scholar.

RESULTS

In all, 38 studies were included, predominantly concerning central venous catheterization. Three studies reported significant cost-effectiveness by reducing infection rates for central venous catheterization. Furthermore, the procedural competency, retention of skills and patient care as evaluated by fewer complications improved in 20 of the reported studies.

CONCLUSION

Evidence suggests that competency-based education with procedural central venous catheterization courses have positive effects on patient care and are both cost-effective. However, more rigorously controlled and reproducible studies are needed. Specifically, future studies could focus on organizational effects and the possibility of transferability to other medical specialties and the broader healthcare system.

In-training assessments: 'The difficulty is trying to balance reality and really tell the truth'

BACKGROUND

In-training assessments (ITA) aim to evaluate trainees' progress and give valuable feedback on their performance. Many factors can affect supervisors during their completion of assessments and these can influence the final results recorded.

METHODS

This is the second part of a study of supervisors of the Australasian College of Dermatologists (ACD) and presents the qualitative data on their opinions of the ACD ITA process and the influences on their ITA ratings.

RESULTS

Supervisors noted the benefits of this assessment tool, together with many limitations. Potential influences upon supervisor ratings included the relationship between the supervisor and trainee and the level of honesty in completing and delivery of the assessment.

CONCLUSIONS

Many factors influence supervisors in the completion of the ITA. These include the impact of interpersonal relationships and concerns about the consequences of delivering a negative assessment, which sometimes lead supervisors to modify the assessment they deliver to the trainee. Further research is needed into honesty in assessment judgements.

The Role of Generic Competencies in the Entrustment of Professional Activities: A Nationwide Competency-Based Curriculum Assessed

BACKGROUND

Entrustable professional activities (EPAs) seek to translate essential physician competencies into clinical practice. Until now, it is not known whether EPA-based curricula offer enhanced assessment and feedback to trainees.

OBJECTIVE

This study examined program directors' and senior residents' justifications for entrustment decisions and what role generic, cross-specialty competencies (such as communication skills, collaboration, and understanding health care systems) play in these decisions.

METHODS

Entrustment decisions for all Dutch obstetrics and gynecology residents between January 2010 and April 2014 were retrieved from their electronic portfolios. Justifications for entrustment were divided into 4 categories: the resident's experience, his or her technical performance, the presence of a generic competency, and training. Template analysis was used to analyze in depth the types of justifications, which play a role in entrustment decisions.

RESULTS

A total of 5139 entrustment decisions for 375 unique residents were extracted and analyzed. In 59% of all entrustment decisions, entrusting a professional task to a resident was justified by the experience of the resident. Generic competencies were mentioned in 0.5% of all entrustment decisions. Template analysis revealed that the amount of exposure and technical skills are leading factors, while the quality of the performance was not reported to be of any influence.

CONCLUSIONS

Entrustment decisions only rarely are based on generic competencies, despite the introduction of competency frameworks and EPAs. For program directors, a leading factor in entrustment decisions is a resident's exposure to an activity, and the quality of a resident's performance appears to play only a minor role.

Assessing Nurse Anaesthetists' Non-Technical Skills in the operating room

BACKGROUND

Incident reporting and fieldwork in operating rooms have shown that some of the errors that arise in anaesthesia relate to inadequate use of non-technical skills. To provide a tool for training and feedback on nurse anaesthetists' non-technical skills, this study aimed to adapt the Anaesthetists' Non-Technical Skills (ANTS) as a behavioural marker system for the formative assessment of nurse anaesthetists' non-technical skills in the operating room.

METHODS

A qualitative approach with focus group interviews was used to identify the non-technical skills of nurse anaesthetists in the operating room. The interview data were transcribed verbatim. Directed content analysis was used to code and sort data deductively into the ANTS categories: task management, team working, situation awareness and decision making. The prototype named Nurse Anaesthetists' Non-Technical Skills (N-ANTS) was presented and discussed in a group of subject matter experts to ensure face validity.

RESULTS

The N-ANTS system consists of the same four categories as ANTS and 15 underlying elements. Three to five good and poor behavioural markers for each element were identified. The headings and definitions of the categories and elements were adjusted to encompass the behavioural markers in N-ANTS. The differences that emerged mainly reflected statements regarding the establishment of role, competence, and task delegation.

CONCLUSION

A behavioural marker system, N-ANTS, for nurse anaesthetists was adapted from a behavioural marker system, ANTS, for anaesthesiologists.

A case for competency-based anaesthesiology training with entrustable professional activities: an agenda for development and research

Competency frameworks are based on what are considered to be the general essential qualities of a doctor. Competencies, being behavioural descriptors, need a strong link to clinical practice to allow trainers to observe and then use them in assessing trainees' performance. The emerging concept of entrustable professional activities (EPAs) may serve as such a link. An EPA is a description of an essential clinical task that frames competencies in the context of clinical practice. A full set of EPAs defines a specialty and constitutes the curriculum of specialty training. After observation of satisfactory performance on an EPA, the resident should be permitted to perform that activity without direct supervision. The terms of this should allow a trainer to provide justification for this decision. This makes graded assumption of responsibilities possible. We describe the potential benefits of working with EPAs in anaesthesiology training and set an agenda for curriculum development and research in this area.

Facilitators and barriers to a nationwide implementation of competency-based postgraduate medical curricula: a qualitative study

BACKGROUND

Postgraduate medical education (PGME) curricula are being redesigned across the western world.

AIM

This study examined the implementation process (what works where and why) of new competency-based PGME curricula and relevant factors influencing this process.

METHODS

In a nationwide project (2006-2010) in the Netherlands, competency-based PGME curricula were implemented for residents in Pediatrics and Obstetrics & Gynecology. The authors conducted 25 semi-structured interviews and used a multi-level theoretical framework to guide coding.

RESULTS

The implementation process proved to be highly dynamic, non-linear, and influenced by many factors. These could be divided into attributes of the innovations/adopters, the implementation process, and the organization. The context determined the speed, quality, and direction of the process and how a factor affected the process.

CONCLUSIONS

We identified specific features of PGME innovation: the challenge of implementing other competencies than that of the medical expert; the importance of regional implementation strategies and educational support; the balance between training and patient care; and the need for regional inter-organizational networks of hospitals. The authors recommend: design the curriculum with the needs of the users in mind; facilitate knowledge sharing; organize educational support; translate the national curriculum to the local workplace; and promote regional inter-organizational networks between hospitals.

Using participatory design to develop structured training in child and adolescent psychiatry

CONTEXT

Learning during residency in child and adolescent psychiatry (CAP) is primarily work-based and has traditionally been opportunistic. There are increasing demands from both postgraduate trainees and medical organisations for structured programmes with defined learning outcomes.

OBJECTIVES

The aim of this study was to partner with postgraduate trainees and consultants in psychiatry to identify key learning issues that should be considered during CAP residency and to use these in designing a structured programme to meet the learning outcome requirements of a competency framework.

METHODS

Participatory design was used to structure a learning and assessment programme in CAP. First, during working seminars, consultants and postgraduate trainees were interviewed about the characteristics of the learning and working in CAP. These interviews were audio taped, transcribed and analyzed for recurrent themes to identify key issues. Descriptive results were fed back to the participants for validation. In a subsequent iterative process the researchers and practitioners partnered to construct a learning and assessment programme.

RESULTS

The tasks within CAP were poorly described by study participants. Several other types of professionals within the healthcare team perform many of the tasks a CAP postgraduate trainee has to learn. Participants had difficulties describing how learning takes place and what postgraduate trainees need to learn in CAP. The partnership between researchers and practitioners identified three key issues to consider in CAP residencies: (1) Preparation for tasks postgraduate trainees are expected to fulfil, (2) Ensuring acquisition of physician-specific knowledge and skills, and (3) Clarifying roles and professional identity within the team. A structured training programme incorporating the key learning issues identified was created.

CONCLUSION

Participatory design was very helpful to structure a contextually suitable training programme in CAP. The researchers speculate that this approach will result in easier implementation of the new training programme.

Person-task-context: a model for designing curriculum and in-training assessment in postgraduate education

Structured curricula for senior house officers have often been lacking. The aim of this study was to trial a person-task-context model in designing a curriculum and in-training assessment (ITA) programme for SHOs in internal medicine. A working group designed the programme based on triangulation of information from interviews with trainees and programme directors, analysis of patient case mix and national quality assurance data. The interview data showed that the main difference currently between trainee levels was in expected degree of responsibility for patient management rather than in actual tasks. Key learning needs were how to take a structured approach to the tasks and get an overview of situations. SHOs expressed a need for explicit learning goals and standards of performance. SHOs requested formal teaching in non-medical aspects of competence such as communication, interpersonal skills and professionalism. This article points out how consideration of the type of trainees involved, the tasks they must do and learn, and the context in which they work are important in designing postgraduate curricula. The person-task-context model can be used to tailor curricula and ITA that support learning and may be especially beneficial in promoting learning in non-dominant areas of a specialty.

The effect of in-training assessment on clinical confidence in postgraduate education

INTRODUCTION

The literature on how in-training assessment (ITA) works in practice and what educational outcomes can actually be achieved is limited. One of the aims of introducing ITA is to increase trainees' clinical confidence; this relies on the assumption that assessment drives learning through its content, format and programming. The aim of this study was to investigate the effect of introducing a structured ITA programme on junior doctors' clinical confidence. The programme was aimed at first year trainees in anaesthesiology.

METHODS

The study involved a nationwide survey of junior doctors' self-confidence in clinical performance before (in 2001) and 2 years after (in 2003) the introduction of an ITA programme. Respondents indicated confidence on a 155-item questionnaire related to performance of clinical skills and tasks reflecting broad aspects of competence. A total of 23 of these items related to the ITA programme.

RESULTS

The response rate was 377/531 (71%) in 2001 and 344/521 (66%) in 2003. There were no statistically significant differences in mean levels of confidence before and 2 years after the introduction of the ITA programme - neither in aspects that were related to the programme nor in those that were unrelated to the programme.

DISCUSSION

This study demonstrates that the introduction of a structured ITA programme did not have any significant effect on trainees' mean level of confidence on a broad range of aspects of clinical competence. The importance of timeliness and rigorousness in the application of ITA is discussed.

Assessment in undergraduate dermatology

Assessment should be considered at an early stage in curriculum planning and must strive for the best balance between validity (testing what is important) and reliability (providing inferences about the student's competence). The primary purpose of summative assessment in clinical medicine is to determine whether the candidate is competent. Dermatologists should use a range of summative assessment techniques that are appropriate for testing the curricular outcomes. These will include tests of application of knowledge and of clinical skills. All students should also receive regular formative assessment to allow them to measure their progress, but this is not used for summative decisions. We review the key issues surrounding assessment and some specific techniques that might be appropriate for testing learning outcomes in undergraduate dermatology, either in the context of a clinical placement or in an integrated assessment involving a number of specialties or systems.

Residents lack knowledge and practical skills in handling the difficult airway

BACKGROUND

Full-scale simulators have been developed in anaesthesia. We aimed at assessing Danish anaesthesia residents' knowledge and management of a standardized difficult airway situation in an advanced airway simulator. The reliability of the assessment was also determined.

METHODS

Thirty-six residents answered a questionnaire testing their knowledge on difficult airway management and practice patterns. Secondly, all residents attempted management of a standardized 'cannot ventilate, cannot intubate' airway scenario. Three observers independently assessed a video recording of the performance.

RESULTS

Only 17% of the residents passed the written test. Ninety-seven per cent stated that they had difficulties recalling the American Society of Anesthesiologists'(ASA) difficult airway algorithm and 53% did not know how to oxygenate through the cricothyroid membrane. Seventeen per cent had previously been involved in emergency situations with severe complications due to insufficient airway management. In the scenario 75% of the residents established ventilation using a laryngeal mask airway. After establishing ventilation, 26% would continue with surgery without a definitive airway, despite the risk of aspiration. Analysis of the interobserver agreement showed good (Kappa = 0.63) to excellent (Kappa = 0.78) consistency for objective parameters in the scoring scheme. Inter-observer reliability was satisfactory (R = 0.43) for trained Crisis Resource Management evaluators.

CONCLUSIONS

Theoretical and practical education in difficult airway management needs to be improved among Danish anaesthesia residents. Simulator assessment should only be performed using well-defined objective parameters and trained assessors.

Educational impact of in-training assessment (ITA) in postgraduate medical education: a qualitative study of an ITA programme in actual practice

OBJECTIVES

To investigate the experiences and opinions of programme directors, clinical supervisors and trainees on an in-training assessment (ITA) programme on a broad spectrum of competence for first year training in anaesthesiology. How does the programme work in practice and what are the benefits and barriers? What are the users' experiences and thoughts about its effect on training, teaching and learning? What are their attitudes towards this concept of assessment?

METHODS

Semistructured interviews were conducted with programme directors, supervisors and trainees from 3 departments. Interviews were audiotaped and transcribed. The content of the interviews was analysed in a consensus process among the authors.

RESULTS

The programme was of benefit in making goals and objectives clear, in structuring training, teaching and learning, and in monitoring progress and managing problem trainees. There was a generally positive attitude towards assessment. Trainees especially appreciated the coupling of theory with practice and, in general, the programme inspired an academic dialogue. Issues of uncertainty regarding standards of performance and conflict with service declined over time and experience with the programme, and departments tended to resolve practical problems through structured planning.

DISCUSSION

Three interrelated factors appeared to influence the perceived value of assessment in postgraduate education: (1) the link between patient safety and individual practice when assessment is used as a licence to practise without supervision rather than as an end-of-training examination; (2) its benefits to educators and learners as an educational process rather than as merely a method of documenting competence, and (3) the attitude and rigour of assessment practice.