Transatlantic comparison of the competence of surgeons at the start of their professional career

Development and validation of a simulation-based assessment of operative competence for higher specialist trainees in general surgery

BACKGROUND

Simulation is increasingly being explored as an assessment modality. This study sought to develop and collate validity evidence for a novel simulation-based assessment of operative competence. We describe the approach to assessment design, development, pilot testing, and validity investigation.

METHODS

Eight procedural stations were generated using both virtual reality and bio-hybrid models. Content was identified from a previously conducted Delphi consensus study of trainers. Trainee performance was scored using an equally weighted Objective Structured Assessment of Technical Skills (OSATS) tool and a modified Procedure-Based Assessment (PBA) tool. Validity evidence was analyzed in accordance with Messick's validity framework. Both 'junior' (ST2-ST4) and 'senior' trainees (ST 5-ST8) were included to allow for comparative analysis.

RESULTS

Thirteen trainees were assessed by ten assessors across eight stations. Inter-station reliability was high (α = 0.81), and inter-rater reliability was acceptable (inter-class correlation coefficient 0.77). A significant difference in mean station score was observed between junior and senior trainees (44.82 vs 58.18, p = .004), while overall mean scores were moderately correlated with increasing training year (rs = .74, p = .004, Kendall's tau-b .57, p = 0.009). A pass-fail score generated using borderline regression methodology resulted in all 'senior' trainees passing and 4/6 of junior trainees failing the assessment.

CONCLUSION

This study reports validity evidence for a novel simulation-based assessment, designed to assess the operative competence of higher specialist trainees in general surgery.

Can I Leave? Perspectives on Parental Leave and Parenthood in Medical Training Among Program Directors and Trainees in Oncologic Specialties

Purpose: Peak fertility commonly occurs during medical training, and delaying parenthood can complicate pregnancies. Trainee parental leave policies are varied and lack transparency. Research on the impacts of parenthood on trainee education is limited. Methods: A Qualtrics-based survey was distributed via e-mail/social media to program directors (PDs) within oncologic specialties with a request to forward a parallel survey to trainees. Questions assessed awareness of parental leave policies, supportiveness of parenthood, and impacts on trainee education. Statistical analyses included descriptive frequencies and bivariable comparisons by key groups. Results: A total of 195 PDs and 286 trainees responded. Twelve percent and 29% of PDs were unsure of maternity/paternity leave options, respectively. PDs felt they were more supportive of trainee parenthood than trainees perceived they were. Thirty-nine percent of nonparent trainees (NPTs) would have children already if not in medicine, and >80% of women trainees were concerned about declining fertility. Perceived impacts of parenthood on trainee overall education and academic productivity were more negative for women trainees when rated by PDs and NPTs; however, men/women parents self-reported equal impacts. Leave burden was perceived as higher for women trainees. Conclusions: A significant portion of PDs lack awareness of parental leave policies, highlighting needs for increased transparency. Trainees' perception of PD support for parenthood is less than PD self-reported support. Alongside significant rates of delayed parenthood and fertility concerns, this poses a problem for trainees seeking to start a family, particularly women who are perceived more negatively. Further work is needed to create a supportive culture for trainee parenthood.

A scoping review of assessment tools for laparoscopic suturing

BACKGROUND

A needs assessment identified a gap in teaching and assessment of laparoscopic suturing (LS) skills. The purpose of this review is to identify assessment tools that were used to assess LS skills, to evaluate validity evidence available, and to provide guidance for selecting the right assessment tool for specific assessment conditions.

METHODS

Bibliographic databases were searched till April 2017. Full-text articles were included if they reported on assessment tools used in the operating room/simulation to (1) assess procedures that require LS or (2) specifically assess LS skills.

RESULTS

Forty-two tools were identified, of which 26 were used for assessing LS skills specifically and 26 for procedures that require LS. Tools had the most evidence in internal structure and relationship to other variables, and least in consequences.

CONCLUSION

Through identification and evaluation of assessment tools, the results of this review could be used as a guideline when implementing assessment tools into training programs.

Flexibility in Postgraduate Medical Training in the Netherlands

Postgraduate medical training in the Netherlands has become increasingly individualized. In this article, the authors describe current practices for three residency programs at the University Medical Center Utrecht: anesthesiology, pediatrics, and ophthalmology. These programs are diverse yet share characteristics allowing for individualized residency training. New residents enter each program throughout the year, avoiding a large simultaneous influx of inexperienced doctors. The usual duration of each is five years. However, the actual duration of rotations or of the program as a whole can be reduced because of residents' previous medical experience or demonstration of early mastery of relevant competencies. If necessary, the duration of training can also increase.Although working hours are already restricted by the European Working Time Directive, most residents choose to train on a part-time basis. The length of their program then is extended proportionally. The extension period added for those residents training part-time can be used to develop specific competencies, complete an elective rotation or research, or explore a focus area. If the resident meets all training objectives before the extension period is completed, the program director can choose to shorten the program length. Recently, entrustable professional activities have been introduced to strengthen workplace-based assessment. The effects on program duration have yet to be demonstrated.Flexible postgraduate training is feasible. Although improving work-life balance for residents is a necessity, attention must be paid to ensuring that they gain the necessary experience and competencies and maintain continuity of care to ensure that high-quality patient care is provided.

The History of Medical Education in Europe and the United States, With Respect to Time and Proficiency

In this article, the authors present a historic overview of the development of medical education in the United States and Europe (in particular the Netherlands), as it relates to the issues of time (duration of the course) and proficiency (performance requirements and examinations). This overview is necessarily limited and based largely on post hoc interpretation, as historic data on time frames are not well documented and the issue of competence has only recently been addressed.During times when there were few, if any, formal regulations, physicians were primarily "learned gentlemen" in command of few effective practical skills, and the duration of education and the competencies acquired by the end of a course simply did not appear to be issues of any interest to universities or state authorities. Though uniform criteria gradually developed for undergraduate medical education, postgraduate specialty training remained, before accreditation organizations set regulations, at the discretion of individual institutions and medical societies. This resulted in large variability in training time and acquired competencies between residency programs, which were often judged on the basis of opaque or questionable criteria. Considering the high costs of health care today and the increasing demand for patient safety and educational efficiency, continuing historic models of nonstandardized practices will no longer be feasible. Efforts to constrain, restructure, and individualize training time and licensing tracks to optimize training for safe care, both in the United States and Europe, are needed.

Comparison of Canadian and Swiss Surgical Training Curricula: Moving on Toward Competency-Based Surgical Education

OBJECTIVE

Quality of surgical training in the era of resident duty-hour restrictions (RDHR) is part of an ongoing debate. Most training elements are provided during surgical service. As exposure to surgical procedures is important but time-consuming, RDHR may affect quality of surgical training. Providing structured training elements may help to compensate for this shortcoming.

DESIGN

This binational anonymous questionnaire-based study evaluates frequency, time, and structure of surgical training programs at 2 typical academic teaching hospitals with different RDHR.

SETTING

Departments of Surgery of University of Basel (Basel, Switzerland) and the Queen's University (Kingston, Ontario, Canada).

PARTICIPANTS

Surgical consultants and residents of the Queen's University Hospital (Kingston, Ontario, Canada) and the University Hospital Basel (Basel, Switzerland) were eligible for this study.

RESULTS

Questionnaire response rate was 37% (105/284). Queen's residents work 80 hours per week, receiving 7 hours of formal training (8.8% of workweek). Basel residents work 60 hours per week, including 1 hour of formal training (1.7% of working time). Queen's faculty and residents rated their program as "structured" or "rather structured" in contrast to Basel faculty and residents who rated their programs as "neutral" in structure or "unstructured." Respondents identified specific structured training elements more frequently at Queen's than in Basel. Two-thirds of residents responded that they seek out additional surgical experiences through voluntary extra work. Basel participants articulated a stronger need for improvement of current surgical training. Although Basel residents and consultants in both institutions fear negative influence of RDHR on the training program, this was not the case in Queen's residents.

CONCLUSIONS

Providing more structured surgical training elements may be advantageous in providing optimal-quality surgical education in an era of work-hour restrictions.

Vertically integrated medical education and the readiness for practice of graduates

BACKGROUND

Medical curricula become more and more vertically integrated (VI) to prepare graduates better for clinical practice. VI curricula show early clinical education, integration of biomedical sciences and focus on increasing clinical responsibility levels for trainees. Results of earlier questionnaire-based studies indicate that the type of the curriculum can affect the perceived preparedness for work as perceived by students or supervisors. The aim of the present study is to determine difference in actual performance of graduates from VI and non-VI curricula.

METHODS

We developed and implemented an authentic performance assessment based on different facets of competence for medical near-graduates in the role of beginning residents on a very busy day. Fifty nine candidates participated: 30 VI (Utrecht, The Netherlands) and 29 non-VI (Hamburg, Germany). Two physicians, one nurse and five standardized patients independently assessed each candidate on different facets of competence. Afterwards, the physicians indicated how much supervision they estimated each candidate would require on nine so called "Entrustable Professional Activities (EPAs)" unrelated to the observed scenarios.

RESULTS

Graduates from a VI curriculum received significantly higher scores by the physicians for the facet of competence "active professional development", with features like 'reflection' and 'asking for feedback'. In addition, VI graduates scored better on the EPA "solving a management problem", while the non-VI graduates got higher scores for the EPA "breaking bad news".

CONCLUSIONS

This study gives an impression of the actual performance of medical graduates from VI and non-VI curricula. Even though not many differences were found, VI graduates got higher scores for features of professional development, which is important for postgraduate training and continuing education.

Validity, reliability and support for implementation of independence-scaled procedural assessment in laparoscopic surgery

Background

There is no widely used method to evaluate procedure-specific laparoscopic skills. The first aim of this study was to develop a procedure-based assessment method. The second aim was to compare its validity, reliability and feasibility with currently available global rating scales (GRSs).

Methods

An independence-scaled procedural assessment was created by linking the procedural key steps of the laparoscopic cholecystectomy to an independence scale. Subtitled and blinded videos of a novice, an intermediate and an almost competent trainee, were evaluated with GRSs (OSATS and GOALS) and the independence-scaled procedural assessment by seven surgeons, three senior trainees and six scrub nurses. Participants received a short introduction to the GRSs and independence-scaled procedural assessment before assessment. The validity was estimated with the Friedman and Wilcoxon test and the reliability with the intra-class correlation coefficient (ICC). A questionnaire was used to evaluate user opinion.

Results

Independence-scaled procedural assessment and GRS scores improved significantly with surgical experience (OSATS p = 0.001, GOALS p < 0.001, independence-scaled procedural assessment p < 0.001). The ICCs of the OSATS, GOALS and independence-scaled procedural assessment were 0.78, 0.74 and 0.84, respectively, among surgeons. The ICCs increased when the ratings of scrub nurses were added to those of the surgeons. The independence-scaled procedural assessment was not considered more of an administrative burden than the GRSs (p = 0.692).

Discussion/conclusion

A procedural assessment created by combining procedural key steps to an independence scale is a valid, reliable and acceptable assessment instrument in surgery. In contrast to the GRSs, the reliability of the independence-scaled procedural assessment exceeded the threshold of 0.8, indicating that it can also be used for summative assessment. It furthermore seems that scrub nurses can assess the operative competence of surgical trainees.

Review of selected national surgical curricula: quantity is not the sole marker of quality

BACKGROUND

Despite marked global variations in length and structure of surgical training programs, their common end product is a trained surgeon capable of independent practice. If variations exist, yet the end product is similar, modifications to curricula could potentially enhance the quality and efficiency of surgical training. This review evaluates global general surgery training programs and compares their curricula against the established standards for assessment of curricula.

METHODS

A convenience sampling method was employed during an online search for nationally recognized general surgery curricula. Curricula of Australia, Canada, Hong Kong, the United Kingdom, and the United States of America were individually reviewed and subsequently evaluated against the General Medical Council's "Standards for curricula and assessment systems."

RESULTS

Postgraduate surgical training is completed in 5 years in Canada and the United States, whereas this takes a minimum of 7, 7, and 10 years in Australia, Hong Kong, and the United Kingdom, respectively. However, when their general surgery curricula are objectively compared, they are remarkably similar. The principle disparities noted were in documentation and standardization of the structured in-training assessment system.

CONCLUSIONS

This review highlights variations in the structure of general surgery training programs globally. There is a need for an objective method to assess training quality, not reliant upon quantity alone. An evidence-based approach is the gold standard in patient care; it is essential to invest resources into developing an evidence-based curricular approach to ensure surgical training quality can be accurately evaluated to maintain and enhance the standards.

Multi-site study of surgical practice in neurosurgery based on surgical process models

Surgical Process Modelling (SPM) was introduced to improve understanding the different parameters that influence the performance of a Surgical Process (SP). Data acquired from SPM methodology is enormous and complex. Several analysis methods based on comparison or classification of Surgical Process Models (SPMs) have previously been proposed. Such methods compare a set of SPMs to highlight specific parameters explaining differences between populations of patients, surgeons or systems. In this study, procedures performed at three different international University hospitals were compared using SPM methodology based on a similarity metric focusing on the sequence of activities occurring during surgery. The proposed approach is based on Dynamic Time Warping (DTW) algorithm combined with a clustering algorithm. SPMs of 41 Anterior Cervical Discectomy (ACD) surgeries were acquired at three Neurosurgical departments; in France, Germany, and Canada. The proposed approach distinguished the different surgical behaviors according to the location where surgery was performed as well as between the categorized surgical experience of individual surgeons. We also propose the use of Multidimensional Scaling to induce a new space of representation of the sequences of activities. The approach was compared to a time-based approach (e.g. duration of surgeries) and has been shown to be more precise. We also discuss the integration of other criteria in order to better understand what influences the way the surgeries are performed. This first multi-site study represents an important step towards the creation of robust analysis tools for processing SPMs. It opens new perspectives for the assessment of surgical approaches, tools or systems as well as objective assessment and comparison of surgeon's expertise.

The transition to hospital consultant: Denmark and the Netherlands compared on preparedness for practice, perceived intensity and contextual factors

INTRODUCTION

Danish and Dutch new consultants' perceptions regarding the transition to consultant were compared to gain insight into this period, particularly the influence of contextual factors concerning the organisation of specialty training and health care therein. Preparation for medical and generic competencies, perceived intensity and burnout were compared. Additionally, effects of differences in working conditions and cultural dimensions were explored.

METHODS

All consultants registered in the Netherlands in 2007-2009 (n = 2643) and Denmark in 2007-2010 (n = 1336) received in June 2010 and April 2011, respectively, a survey about their preparation for medical and generic competencies, perceived intensity and burnout. Power analysis resulted in required sample sizes of 542. Descriptive statistics and independent t-tests were used for analysis.

RESULTS

Data were available of 792 new consultants in the Netherlands and 677 Danish new consultants. Compared to their Dutch counterparts, Danish consultants perceived specialty training and the transition less intensely, reported higher levels of preparation for generic competencies and scored lower on burnout.

CONCLUSIONS

The importance of contextual aspects in the transition is underscored and shows that Denmark appears to succeed better in aligning training with practice. Regulations regarding working hours and progressive independence of trainees appear to facilitate the transition.

A narrative review of surgical resident duty hour limits: where do we go from here?

BACKGROUND

Resident duty hour limits have been a point of debate among educators, administrators, and policymakers alike since the Libby Zion case in 1984. Advocates for duty hour limits in the surgical subspecialties cite improvements in patient safety, whereas opponents claim that limiting resident duty hours jeopardizes resident education and preparedness for independent surgical practice.

METHODS

Using orthopaedic surgery as an example, we describe the historical context of the implementation of the duty hour standards, provide a review of the literature presenting data that both supports and refutes continued restrictions, and outline suggestions for policy going forward that prioritize patient safety while maintaining an enhanced environment for resident education.

RESULTS

Although patient safety markers have improved in some studies since the implementation of duty hour limits, it is unclear whether this is due to changes in residency training or external factors. The literature is mixed regarding academic performance and trainee readiness during and after residency.

CONCLUSION

Although excessive duty hours and resident fatigue may have historically contributed to errors in the delivery of patient care, those are certainly not the only concerns. An overall "culture of safety," which includes pinpointing systematic improvements, identifying potential sources of error, raising performance standards and safety expectations, and implementing multiple layers of protection against medical errors, can continue to augment safety barriers and improve patient care. This can be achieved within a more flexible educational environment that protects resident education and ensures optimal training for the next generation of physicians and surgeons.

Changes in residents' opportunities for experiential learning over time

CONTEXT

Learning in the clinical environment is believed to be a crucial component of residency training. However, it remains unclear whether recent changes to postgraduate medical education, including the implementation of work hour limitations, have significantly impacted opportunities for experiential learning. Therefore, we sought to quantify opportunities to gain clinical experience within medical-surgical intensive care units (ICUs) over time.

METHODS

Data on the numbers of patients admitted and invasive procedures performed per day between 1 July 2001 and 30 June 2010 within three academic medical-surgical ICUs in Calgary, Alberta, Canada were obtained from electronic medical records. These data were matched to resident doctor on-call schedules and residents' opportunities to admit patients and participate in procedures were calculated and compared over time using Spearman's rho.

RESULTS

We found that over a 9-year period, the opportunities afforded to residents (n = 1156) to admit patients (n = 17 189) and perform procedures (n = 52 827) during ICU rotations decreased by 32% (p < 0.001) and 34% (p < 0.001), respectively.

CONCLUSIONS

Our results suggest that there has been a significant decrease in residents' clinical experiences in the ICU over time. Further investigations to better understand these changes and how they may impact on performance as residents become independent practising doctors are warranted.

Duty-hour limits and patient care and resident outcomes: can high-quality studies offer insight into complex relationships?

Long hours are an accepted component of resident education, yet data suggest they contribute to fatigue that may compromise patient safety. A systematic review confirms that limiting duty hours increases residents' hours of sleep and improves objective measures of alertness. Most studies of operative experience for surgical residents found no effect, and there is evidence of a limited positive effect on residents' mood. We find a mixed effect on patient safety, although problems with supervision, rather than the limits, may be responsible or contibute; evidence of reduced continuity of care and reduced continuity in residents' clinical education; and evidence that increased workload under the limits has a negative effect on patient and resident outcomes. We highlight specific areas for research and offer recommendations for national policy.

Virtual reality cataract surgery training: learning curves and concurrent validity

PURPOSE

To investigate initial learning curves on a virtual reality (VR) eye surgery simulator and whether achieved skills are transferable between tasks.

METHODS

Thirty-five medical students were randomized to complete ten iterations on either the VR Caspulorhexis module (group A) or the Cataract navigation training module (group B) and then two iterations on the other module. Learning curves were compared between groups. The second Capsulorhexis video was saved and evaluated with the performance rating tool Objective Structured Assessment of Cataract Surgical Skill (OSACSS). The students' stereoacuity was examined.

RESULTS

Both groups demonstrated significant improvements in performance over the 10 iterations: group A for all parameters analysed including score (p < 0.0001), time (p < 0.0001) and corneal damage (p = 0.0003), group B for time (p < 0.0001), corneal damage (p < 0.0001) but not for score (p = 0.752). Training on one module did not improve performance on the other. Capsulorhexis score correlated significantly with evaluation of the videos using the OSACSS performance rating tool. For stereoacuity < and ≥120 seconds of arc, sum of both modules' second iteration score was 73.5 and 41.0, respectively (p = 0.062).

CONCLUSION

An initial rapid improvement in performance on a simulator with repeated practice was shown. For capsulorhexis, 10 iterations with only simulator feedback are not enough to reach a plateau for overall score. Skills transfer between modules was not found suggesting benefits from training on both modules. Stereoacuity may be of importance in the recruitment and training of new cataract surgeons. Additional studies are needed to investigate this further. Concurrent validity was found for Capsulorhexis module.

Review article: closing the research gap at the interface of learning and clinical practice

PURPOSE

The dialogue at the interface of education and clinical practice highlights areas of critical importance to the development of new approaches for educating anesthesiologists. The purpose of this article is to examine the literature on education and acquisition of competence in three areas relevant to the interface of learning and clinical practice, with the aim to suggest a research agenda that adds to the evidence on preparing physicians for independent practice. The three areas are: 1) transitions across the continuum of education; 2) the effect of reductions in hours of clinical training on competence; and 3) efforts to incorporate the competencies and CanMEDS roles into teaching and evaluation.

PRINCIPAL FINDINGS

Fifty-six articles relevant to one or more of the themes were identified in the review, including 21 studies of transitions (in, during, and after residency education), 19 studies on the effects of duty hour limits on residents' acquisition of competence, and 16 articles that assessed competency-based teaching and assessment in anesthesiology. Overall, the findings suggested a relative paucity of scientific evidence and a need for research and the development of new scientific theory. Studies generally treated one of the themes in isolation, while in actuality they interact to produce optimal as well as suboptimal learning situations, while medical education research often is limited by small samples, brief follow-up, and threats to validity. This suggests a "research gap" where editorials and commentaries have moved ahead of an evidence base for education. Promising areas for research include preparation for care deemed important by society, work to apply knowledge about the development of expertise in other disciplines to medicine, and ways to embed the competencies in teaching and evaluation more effectively.

CONCLUSION

Closing the research gap in medical education will require clear direction for future work. The starting point, at an institution or nationally, is dialogue within the specialty to achieve consensus on some of the most pressing questions.

Implementation of simulation in surgical practice: minimally invasive surgery has taken the lead: the Dutch experience

Minimal invasive techniques are rapidly becoming standard surgical techniques for many surgical procedures. To develop the skills necessary to apply these techniques, box trainers and/or inanimate models may be used, but these trainers lack the possibility of inherent objective classification of results. In the past decade, virtual reality (VR) trainers were introduced for training minimal invasive techniques. Minimally invasive surgery (MIS) is, by nature, very suitable for this type of training. The specific psychomotor skills and eye-hand coordination needed for MIS can be mastered largely using VR simulation techniques. It is also possible to transfer skills learned on a simulator to real operations, resulting in error reduction and shortening of procedural operating time. The authors aim to enlighten the process of gaining acceptance in the Netherlands for novel training techniques. The Dutch Societies of Surgery, Obstetrics and Gynecology, and Urology each developed individual training curricula for MIS using simulation techniques, to be implemented in daily practice. The ultimate goal is to improve patient safety. The authors outline the opinions of actors involved, such as different simulators, surgical trainees, surgeons, surgical societies, hospital boards, government, and the public. The actual implementation of nationwide training curricula for MIS is, however, a challenging step.