"Staying in the Game": How Procedural Variation Shapes Competence Judgments in Surgical Education

Learning From Clinical Supervisor Practice Variability: Exploring Medical Resident and Fellow Experiences and Interpretations

PURPOSE

Clinical practice variability is characterized by 2 or more clinicians making different treatment decisions despite encountering a similar case. This study explores how medical residents and fellows experience and interpret intersupervisor clinical practice variability and how these variations influence learning.

METHOD

Seventeen senior residents or fellows in internal medicine, hematology, or thrombosis medicine (postgraduate year 3 or above) participated in semistructured interviews after a clinical rotation in thrombosis medicine from December 2019 to March 2021. Data collection and analysis occurred iteratively and concurrently in a manner consistent with constructivist grounded theory. Variation theory was used to guide the development of some interview questions. A central tenet of this theory is that learning occurs by experiencing 3 sequential patterns of variation: contrast, generalization, and fusion. Participants were recruited purposively with respect to specialty until theoretical sufficiency was reached.

RESULTS

Clinical practice variability was experienced by all participants. Residents and fellows attributed practice variability to intrinsic differences among supervisors; interinstitutional differences; selection and interpretation of evidence; patient preferences, priorities, and fears; and their own participation in the decision-making process. Clinical practice variability helped residents and fellows discern key features of cases that influenced decision-making (contrast), group similar cases so that the appropriate evidence could be applied (generalization), and develop attitudes consistent with providing individualized patient care (fusion). Observing practice variability was more helpful for fifth- and sixth-year residents and less helpful for third- and fourth-year residents.

CONCLUSIONS

Clinical practice variability helped residents and fellows discern critical aspects, group similar patients, and practice individualized medicine. Future research should characterize how clinical practice variability influences learning across the spectrum of training, how supervisors could encourage learning from practice variability, and how curricula could be modified to allow learners greater opportunity to reflect on and consolidate the practice differences they observe.

Scoping review: exploring residents' views of supervisor entrustment and its effect on learning and professional development

This scoping review aimed to systematically explore the complex and nuanced perceptions of residents entrusted with significant responsibilities by their supervisors in the clinical context. We conducted a comprehensive search strategy targeting six major electronic databases. Data were extracted using the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklist and a scoping review protocol. Resident perceptions regarding entrustment and its impact on learning and professional development were categorized. This review elucidated the role of supervisor entrustment in shaping residents' clinical performance, notably influencing autonomy, self-confidence, responsibility for patients, learning needs and goals, uncertainty management, and professional identity formation. Residents identified factors that promote and hinder performance and entrustment development resulting from supervisor entrustment and distinguished between appropriate supervision and over- or under-supervision. Our review suggests that residents' perceptions of supervisor entrustment, as bestowed by supervisors, serves as a catalyst for enhancing autonomy, thereby exerting a significant influence on clinical performance, and fostering the development of a professional identity. However, residents' views on appropriate supervision differ in some ways from supervisors' perspectives. Further research is needed to bridge the gap between residents' expectations of autonomy and what they are given, and to explore the intricate relationship between autonomy and entrustment.

Surgical Artificial Intelligence in Urology: Educational Applications

Surgical education has seen immense change recently. Increased demand for iterative evaluation of trainees from medical school to independent practice has led to the generation of an overwhelming amount of data related to an individual's competency. Artificial intelligence has been proposed as a solution to automate and standardize the ability of stakeholders to assess the technical and nontechnical abilities of a surgical trainee. In both the simulation and clinical environments, evidence supports the use of machine learning algorithms to both evaluate trainee skill and provide real-time and automated feedback, enabling a shortened learning curve for many key procedural skills and ensuring patient safety.

Medical competence as a multilayered construct

BACKGROUND

The conceptualisation of medical competence is central to its use in competency-based medical education. Calls for 'fixed standards' with 'flexible pathways', recommended in recent reports, require competence to be well defined. Making competence explicit and measurable has, however, been difficult, in part due to a tension between the need for standardisation and the acknowledgment that medical professionals must also be valued as unique individuals. To address these conflicting demands, a multilayered conceptualisation of competence is proposed, with implications for the definition of standards and approaches to assessment.

THE MODEL

Three layers are elaborated. This first is a core layer of canonical knowledge and skill, 'that, which every professional should possess', independent of the context of practice. The second layer is context-dependent knowledge, skill, and attitude, visible through practice in health care. The third layer of personalised competence includes personal skills, interests, habits and convictions, integrated with one's personality. This layer, discussed with reference to Vygotsky's concept of Perezhivanie, cognitive load theory, self-determination theory and Maslow's 'self-actualisation', may be regarded as the art of medicine. We propose that fully matured professional competence requires all three layers, but that the assessment of each layer is different.

IMPLICATIONS

The assessment of canonical knowledge and skills (Layer 1) can be approached with classical psychometric conditions, that is, similar tests, circumstances and criteria for all. Context-dependent medical competence (Layer 2) must be assessed differently, because conditions of assessment across candidates cannot be standardised. Here, multiple sources of information must be merged and intersubjective expert agreement should ground decisions about progression and level of clinical autonomy of trainees. Competence as the art of medicine (Layer 3) cannot be standardised and should not be assessed with the purpose of permission to practice. The pursuit of personal excellence in this level, however, can be recognised and rewarded.

The pursuit of fairness in assessment: Looking beyond the objective

Health professions education has undergone significant changes over the last few decades, including the rise of competency-based medical education, a shift to authentic workplace-based assessments, and increased emphasis on programmes of assessment. Despite these changes, there is still a commonly held assumption that objectivity always leads to and is the only way to achieve fairness in assessment. However, there are well-documented limitations to using objectivity as the 'gold standard' to which assessments are judged. Fairness, on the other hand, is a fundamental quality of assessment and a principle that almost no one contests. Taking a step back and changing perspectives to focus on fairness in assessment may help re-set a traditional objective approach and identify an equal role for subjective human judgement in assessment alongside objective methods. This paper explores fairness as a fundamental quality of assessments. This approach legitimises human judgement and shared subjectivity in assessment decisions alongside objective methods. Widening the answer to the question: 'What is fair assessment' to include not only objectivity but also expert human judgement and shared subjectivity can add significant value in ensuring learners are better equipped to be the health professionals required of the 21st century.

Adult Gastroenterology Trainees’ Experience of Receiving Feedback on Their Performance of Endoscopy in the Workplace

Background

Competency-based gastrointestinal endoscopy training is concerned with outcomes of the learning experience. Feedback allows for trainees to achieve the expected outcomes. However, little is known about trainees’ experience of receiving feedback. Gaining understanding of their experience could help improve feedback practices. The study was conducted to explore what it means for adult gastroenterology trainees to receive feedback on their performance of endoscopy in the workplace.

Methods

An interpretative phenomenological approach was used. Individual semi-structured interviews were conducted with six trainees from three Canadian adult gastroenterology residency programs. Interviews were audio-recorded and transcribed verbatim for analysis. Analysis was conducted to identify the phenomenological themes across participants’ accounts of lived experience to provide an insight into the meaning of experiencing the studied phenomenon.

Findings

Three phenomenological themes of experience were identified: taking pauses, negotiating understandings and accepting asymmetry. Taking pauses allowed for participants to receive feedback on their performance of endoscopy. Participants needed to negotiate attending gastroenterologists’ different understandings of gastrointestinal endoscopy while carrying their own whenever feedback was provided. They had to accept the asymmetry between the roles of care provider and learner as well.

Discussion

The study has captured the uniqueness and the complexity of the lived experience of receiving feedback on the performance of endoscopy in the workplace from the perspective of study participants. The gained understanding of this experience has enabled the authors to suggest how attending gastroenterologists’ feedback practices may be improved.

The embodiment of practice thresholds: from standardization to stabilization in surgical education

Surgeons practice their own variations on a procedure. Residents experience shifting thresholds between variations that one surgeon holds firmly as principle and another takes more lightly as preference. Such variability has implications for surgical education, but the impact is not well understood. This is a critical problem to investigate as programs seek to define procedures for competency-based medical education (CBME) and improve learning through deliberate practice. Our study analyzes the emergence of procedural variation in an early-adopter CBME program through a situational analysis of tonsillectomy, a foundation level procedure in this otolaryngology, head and neck surgical program. An earlier phase of the study identified frequent variations (n = 12) on tonsillectomy among co-located surgeons who routinely perform this procedure (n = 6). In the phase reported here we interviewed these surgeons (n = 4) and residents at different stages of training (n = 3) about their experiences of these variations to map the relations of contributing social and material actors. Our results show that even a basic procedure resists standardization. This study contributes a sociomaterial grounded theory of surgical practice as an embodied response to conditions materialized by intra-relations of human and more-than-human actors. Shifting root metaphors about practice in surgical education from standardization to stabilization can help residents achieve stable-for-now embodiments of performance as their practice thresholds continue to emerge.

Scut to Scholarship: Can Operative Notes be Educationally Useful?

OBJECTIVE

Efforts to implement competency-based medical education require new sources of workplace-based evidence of growth in learning. We used qualitative analysis of operative notes to explore procedural variation in a simple surgical procedure.

DESIGN

We used a grounded theory-based mixed methods approach to depict intersurgeon procedural variation. Our grounded theory approach to analysis included follow up interviews with surgeons and residents to probe their understandings of the reasons for variation in the dictated notes and the current and potential utility of operative notes as a reliable source of data for learning and assessment.

SETTING

Publicly funded tertiary care otolaryngology-head & neck surgery residency program in Ontario, Canada PARTICIPANTS: Using maximum variability sampling, all surgeons performing tonsillectomy in the department (n = 6) contributed operative notes from 65 tonsillectomies, 5 intraoperative observations, and 4 semi-structured interviews. An additional 3 residents from various levels of training contributed semistructured interviews.

RESULTS

Intersurgeon procedural variations persist even in simple surgical procedures such as tonsillectomy. Operative notes appear to capture procedural variations in a limited way. Surgeons and resident make informal educational use of the clerical work of writing and assessing operative notes, but optimization will be required to shift such hidden work into the formal educational domain.

CONCLUSIONS

The implementation of competency-based medical education requires surgical educators to both eliminate low-yield tasks for learning and to find new opportunities for multiple low-stakes assessment. Analysis of operative notes may become a high-yield strategy for learning and assessment if residents and surgeons are coached to use operative notes more reliably and efficiently.

How Do Thresholds of Principle and Preference Influence Surgeon Assessments of Learner Performance?

OBJECTIVE

The present study asks whether intraoperative principles are shared among faculty in a single residency program and explores how surgeons' individual thresholds between principles and preferences might influence assessment.

BACKGROUND

Surgical education continues to face significant challenges in the implementation of intraoperative assessment. Competency-based medical education assumes the possibility of a shared standard of competence, but intersurgeon variation is prevalent and, at times, valued in surgical education. Such procedural variation may pose problems for assessment.

METHODS

An entire surgical division (n = 11) was recruited to participate in video-guided interviews. Each surgeon assessed intraoperative performance in 8 video clips from a single laparoscopic radical left nephrectomy performed by a senior learner (>PGY5). Interviews were audio recorded, transcribed, and analyzed using the constant comparative method of grounded theory.

RESULTS

Surgeons' responses revealed 5 shared generic principles: choosing the right plane, knowing what comes next, recognizing normal and abnormal, making safe progress, and handling tools and tissues appropriately. The surgeons, however, disagreed both on whether a particular performance upheld a principle and on how the performance could improve. This variation subsequently shaped their reported assessment of the learner's performance.

CONCLUSIONS

The findings of the present study provide the first empirical evidence to suggest that surgeons' attitudes toward their own procedural variations may be an important influence on the subjectivity of intraoperative assessment in surgical education. Assessment based on intraoperative entrustment may harness such subjectivity for the purpose of implementing competency-based surgical education.

Supervised workplace learning in postgraduate training: a realist synthesis

Context

This paper presents a realist synthesis of the literature that began with the objective of developing a theory of workplace learning specific to postgraduate medical education (PME). As the review progressed, we focused on informal learning between trainee and senior doctor or supervisor, asking what mechanisms occur between trainee and senior doctor that lead to the outcomes of PME, and what contexts shape the operation of these mechanisms and the outcomes they produce?

Methods

We followed the procedures outlined in the RAMESES Publication Standards for Realist Synthesis. We searched the English‐language literature published between 1995 and 2017 for empirical papers related to informal workplace learning between supervisor and trainee, excluding formal interventions such as workplace‐based assessment. We made a pragmatic decision to exclude general practice training to keep the review within manageable limits.

Results

We reviewed 5197 papers and selected 90. Synthesis revealed three workplace learning processes occurring between supervisors and trainees, each underpinned by a pair of mechanisms: supervised participation in practice (entrustment and support seeking); mutual observation of practice (monitoring and modelling), and dialogue during practice (meaning making and feedback). These mechanisms result in outcomes of PME, including safe participation in practice, learning skills, attitudes and behaviours and professional identity development. Contexts shaping the outcomes of these mechanisms were identified at individual, interpersonal, local and systems levels.

Conclusions

Our realist theory of workplace learning between supervisors and trainees is informed by theory and empirical research. It highlights the two‐way nature of supervision, the importance of trainees’ agency in their own learning and the deleterious effect of fragmented working patterns on supervisor–trainee learning mechanisms. Further empirical research is required to test and refine this theory. In the meantime, it provides a useful framework for the design of supportive learning environments and for the preparation of supervisors and trainees for their roles in workplace learning.

The authors describe a realist theory of learning between supervisor and postgraduate trainee in the workplace, emphasising the role of individual, interpersonal, local and systems contexts in shaping key mechanisms and their outcomes.

Dealing with the tension: how residents seek autonomy and participation in the workplace

CONTEXT

The workplace can be a strenuous setting for residents: although it offers a wealth of learning opportunities, residents find themselves juggling their responsibilities. Even though supervisors regulate what is afforded to residents, the former find it difficult to strike the proper balance between residents' independence and support, which could create tensions. But what tensions do residents experience during clinical supervision and how do they cope with them to maximise their learning opportunities? Understanding how residents act on different affordances in the workplace is of paramount importance, as it influences their learning.

METHOD

Residents from different levels of training and disciplines participated in three focus groups (n = 19) and 10 semi-structured interviews (n = 10). The authors recruited these trainees using purposive and convenience sampling. Audio-recordings were transcribed verbatim and the ensuing scripts were analysed using a constructivist grounded theory methodology.

RESULTS

Residents reported that the autonomy and practice opportunities given by their supervisors were either excessive or too limited, and both were perceived as tensions. When in excess, trainees enlisted the help of their supervisor or peers, depending on how safe they recognised the learning environment to be. When practice opportunities were curtailed, trainees tried to negotiate more if they felt the learning environment was safe. When they did not, trainees became passive observers. Learning from each engagement was subject to the extent of intersubjectivity achieved between the actors involved.

CONCLUSIONS

Tensions arose when supervisors did not give trainees the desired degree of autonomy and opportunities to participate. Trainees responded in various ways to maximise their learning opportunities. For these different engagement-related responses to enhance workplace learning in specialty training, achieving intersubjectivity between trainee and supervisor seems foundational.

Scholarly Conversations in Medical Education

This supplement includes the eight research papers accepted by the 2016 Research in Medical Education Program Planning Committee. In this Commentary, the authors use "conversations in medical education" as a guiding metaphor to explore what these papers contribute to the current scholarly discourse in medical education. They organize their discussion around two domains: the topic of study and the methodological approach. The authors map the eight research papers to six "hot topics" in medical education: (1) curriculum reform, (2) duty hours restriction, (3) learner well-being, (4) innovations in teaching and assessment, (5) self-regulated learning, and (6) learning environment, and to three purposes commonly served by medical education research: (1) description, (2) justification, and (3) clarification. They discuss the range of methods employed in the papers. The authors end by encouraging educators to engage in these ongoing scholarly conversations.