Measuring progressive independence with the resident supervision index: empirical approach

Association Between Teaching Clinic Structure and the Readiness of Ophthalmology Residents to Enter Independent Practice

OBJECTIVE

Our objective is to determine if the structure of Graduate Medical Education teaching clinics is associated with how well ophthalmology residents are prepared to meet the workload demands of independent clinical practice.

DESIGN

Resident preparedness to enter independent practice was measured by the Readiness Index. Part of the Department of Veterans Affairs' new Workload-based Resident Academic Performance measures (WRAP), resident readiness is computed from electronic health records for residents by clinic and service-date. The index compares resident productivity net of supervision and adjusted for care quality to the average productivity of non-supervising ophthalmologists. Readiness comprises a Workload component (ratio of resident gross productivity to the average productivity of non-supervising ophthalmologists) and Supervision component (ratio of resident net of supervision to gross productivity). Teaching clinic factors include resident postgraduate-year level, resident-to-physician staff ratios, patient care complexity, and program size. Covariates include time into the academic year, facility quality ranking and complexity rating, and attending physician productivity rate.

SETTING

Study setting is 109 ophthalmology outpatient clinics from the United States Department of Veterans Affairs and its 1,300 annual ophthalmology resident positions rotating on 84,600 ophthalmology clinic-days during academic years from July 1, 2015, through June 30, 2019.

PARTICIPANTS

An average 2.6 residents at a second-year or higher saw 25.0 patients requiring 93.6 relative value units (RVUs) of workload.

RESULTS

Senior ophthalmology residents from clinics with higher resident-to-physician ratios had greater practice readiness than their counterparts primarily from having greater progressive autonomy from supervision. Residents from larger programs treating more complex patients had only slightly greater practice readiness than their counterparts primarily from having greater workload productivity.

CONCLUSIONS

The readiness of ophthalmology residents to enter independent practice is associated with their academic level and resident-to-physician staff ratios, and to a lesser extent care complexity and program size.

Patient Surgical Outcomes When Surgery Residents Are the Primary Surgeon by Intensity of Surgical Attending Supervision in Veterans Affairs Medical Centers

Objective

Using health records from the Department of Veterans Affairs (VA), the largest healthcare training platform in the United States, we estimated independent associations between the intensity of attending supervision of surgical residents and 30-day postoperation patient outcomes.

Background

Academic leaders do not agree on the level of autonomy from supervision to grant surgery residents to best prepare them to enter independent practice without risking patient outcomes.

Methods

Secondary data came from a national, systematic 1:8 sample of n = 862,425 teaching encounters where residents were listed as primary surgeon at 122 VA medical centers from July 1, 2004, through September 30, 2019. Independent associations between whether attendings had scrubbed or not scrubbed on patient 30-day all-cause mortality, complications, and 30-day readmission were estimated using generalized linear-mixed models. Estimates were tested for any residual confounding biases, robustness to different regression models, stability over time, and validated using moderator and secondary factors analyses.

Results

After accounting for potential confounding factors, residents supervised by scrubbed attendings in 733,997 nonemergency surgery encounters had fewer deaths within 30 days of the operation by 14.2% [0.3%, 29.9%], fewer case complications by 7.9% [2.0%, 14.0%], and fewer readmissions by 17.5% [11.2%, 24.2%] than had attendings not scrubbed. Over the 15 study years, scrubbed surgery attendings may have averted an estimated 13,700 deaths, 43,600 cases with complications, and 73,800 readmissions.

Conclusions

VA policies on attending surgeon supervision have protected patient safety while allowing residents in selected teaching encounters to have limited autonomy from supervision.

How Clinical Supervisors Conceptualize Procedural Entrustment: An Interview-Based Study of Entrustment Decision Making in Endoscopic Training

PURPOSE

Entrustment is central to assessment in competency-based medical education (CBME). To date, little research has addressed how clinical supervisors conceptualize entrustment, including factors they consider in making entrustment decisions. The aim of this study was to characterize supervisors' decision making related to procedural entrustment, using gastrointestinal endoscopy as a test case.

METHOD

Using methods from constructivist grounded theory, the authors interviewed 29 endoscopy supervisors in the United States and Canada across multiple specialties (adult and pediatric gastroenterology, surgery, and family medicine). Semistructured interviews, conducted between April and November 2019, focused on how supervisors conceptualize procedural entrustment, how they make entrustment decisions, and what factors they consider. Transcripts were analyzed using constant comparison to generate an explanatory framework and themes.

RESULTS

Three themes were identified from the analysis of interview transcripts: (1) entrustment occurs in varying degrees and fluctuates over time; (2) entrustment decisions can transfer within and across procedural and nonprocedural contexts; (3a) persistent static factors (e.g., supervisor competence, institutional culture, legal considerations) influence entrustment decisions, as do (3b) fluctuating, situated dynamic factors (e.g., trainee skills, patient acuity, time constraints), which tend to change from one training encounter to the next.

CONCLUSIONS

In the process of making procedural entrustment decisions, clinical supervisors appear to synthesize multiple dynamic factors against a background of static factors, culminating in a decision of whether to entrust. Entrustment decisions appear to fluctuate over time, and assessors may transfer decisions about specific trainees across settings. Understanding which factors supervisors perceive as influencing their decision making has the potential to inform faculty development, as well as competency committees seeking to aggregate faculty judgments about trainee unsupervised practice. Those leading CBME programs may wish to invest in optimizing the observed static factors, such that these foundational factors are tuned to facilitate trainee learning and achievement of entrustment.

The recommended description of an entrustable professional activity: AMEE Guide No. 140

Entrustable professional activities (EPAs) have received much attention in the literature since they were first proposed in 2005. Useful guidelines, workshops, courses, and conferences have supported faculty in developing programs and designing assessment procedures using EPAs and entrustment decision-making. Yet, the need for clarification remains, particularly as more programs make the step from design to implementation.Well-written EPAs provide a natural construct to establish the outcome of training. To be useful, EPAs require more than a suitable title. This AMEE Guide elaborates eight sections of a full EPA description, and provides explanations and justifications for each. These sections are: title; specification and limitations; risks in case of failure; most relevant competency domains; knowledge, skills, attitudes and experiences; information sources to assess progress and support summative entrustment; entrustment/supervision level expected at which stage of training; and time period to expiration if not practiced.Constructing fully elaborated EPAs creates a shared mental model amongst learners and programs, informs competency-based curriculum design, directs ad-hoc and formal entrustment decision-making, and provides standards for certifying bodies and boundaries for scope of practice. The framework intends to support curricular leaders looking to adopt new EPAs, or revise and define established EPAs for competency-based education.

Assessing Trainees and Making Entrustment Decisions: On the Nature and Use of Entrustment-Supervision Scales

Clinical teachers are continuously entrusting trainees with care responsibilities in health care settings. Entrustable professional activities employ entrustment decision making as an approach to assessment in the workplace.Various scales have been created to measure "entrustment," all basically expressing the level or type of supervision a trainee requires for safe and high-quality care. However, some of these scales are only weakly related to the purpose of making decisions about the autonomy trainees will be granted. The authors aim to increase understanding about the nature, purpose, and practice of supervision scales aimed at entrustment.After arguing for entrustment as a component of workplace-based assessment, the distinction between ad hoc entrustment decisions (daily decisions in health care settings) and summative entrustment decisions (with a certifying nature) is clarified. Next, the noncontinuous nature of entrustment-supervision (ES) scales, as opposed to most workplace-based assessment scales, is explained. ES scales have ordinal, rather than interval, properties and focus on discrete decisions. Finally, some scales are retrospective ("how much supervision was provided?"), and others are prospective ("how much supervision will be needed in the near future?"). Although retrospective scales reflect observed behavior, prospective scales truly focus on entrustment and ask for more holistic judgment, as they include a broader evaluation and a risk estimation to enable a decision about increase of autonomy.The analysis concludes with a discussion about entrustment for unsupervised practice and supervision of others, as well as the program, context, and specialty specificity of scales.

Residents' Ratings of Their Clinical Supervision and Their Self-Reported Medical Errors: Analysis of Data From 2009

BACKGROUND

Medical errors and patient safety are major concerns for the medical and medical education communities. Improving clinical supervision for residents is important in avoiding errors, yet little is known about how residents perceive the adequacy of their supervision and how this relates to medical errors and other education outcomes, such as learning and satisfaction.

METHODS

We analyzed data from a 2009 survey of residents in 4 large specialties regarding the adequacy and quality of supervision they receive as well as associations with self-reported data on medical errors and residents' perceptions of their learning environment.

RESULTS

Residents' reports of working without adequate supervision were lower than data from a 1999 survey for all 4 specialties, and residents were least likely to rate "lack of supervision" as a problem. While few residents reported that they received inadequate supervision, problems with supervision were negatively correlated with sufficient time for clinical activities, overall ratings of the residency experience, and attending physicians as a source of learning. Problems with supervision were positively correlated with resident reports that they had made a significant medical error, had been belittled or humiliated, or had observed others falsifying medical records.

CONCLUSIONS

Although working without supervision was not a pervasive problem in 2009, when it happened, it appeared to have negative consequences. The association between inadequate supervision and medical errors is of particular concern.

Supervision and autonomy of ophthalmology residents in the outpatient Clinic in the United States: a survey of ACGME-accredited programs

BACKGROUND

The development and demonstration of incremental trainee autonomy is required by the ACGME. However, there is scant published research concerning autonomy of ophthalmology residents in the outpatient clinic setting. This study explored the landscape of resident ophthalmology outpatient clinics in the United States.

METHODS

A link to an online survey using the QualtricsTM platform was emailed to the program directors of all 115 ACGME-accredited ophthalmology programs in the United States. Survey questions explored whether resident training programs hosted a continuity clinic where residents would see their own patients, and if so, the degree of faculty supervision provided therein. Metrics such as size of the resident program, number of faculty and clinic setting were also recorded. Correlations between the degree of faculty supervision and other metrics were explored.

RESULTS

The response rate was 94%; 69% of respondents indicated that their trainees hosted continuity clinics. Of those programs, 30% required a faculty member to see each patient treated by a resident, while 42% expected the faculty member to at least discuss (if not see) each patient. All programs expected some degree of faculty interaction based upon circumstances such as the level of training of the resident or complexity of the clinical situation. 67% of programs that tracked the contribution of the clinic to resident surgical caseloads reported that these clinics provided more than half of the resident surgical volumes. More ¾ of resident clinics were located in urban settings. The degree of faculty supervision did not correlate to any of the other metrics evaluated.

CONCLUSIONS

The majority of ophthalmology resident training programs in the United States host a continuity clinic located in an urban environment where residents follow their own patients. Furthermore, most of these clinics require supervising faculty to review both the patients seen and the medical documentation created by the resident encounters. The different degrees of faculty supervision outlined by this survey might provide a useful guide presuming they can be correlated with validated metrics of educational quality. Finally, this study could provide an adjunctive resource to current international efforts to standardize ophthalmic residency education.

How do attendings perceive housestaff autonomy? Attending experience, hospitalists, and trends over time

BACKGROUND

Graduated supervision is necessary for residents to progress to independence, but it is unclear what factors influence attendings' perception of housestaff autonomy.

OBJECTIVE

To determine if attending characteristics and secular trends are associated with variation in attendings' perception of housestaff autonomy.

DESIGN

Secondary data analysis of monthly survey data collected from 2001 to 2008.

SETTING/PARTICIPANTS

Attending hospitalists and nonhospitalists on teaching internal medicine services at an academic tertiary care center.

MEASUREMENTS

Attendings' perception of intern decision making and resident autonomy.

RESULTS

Response rate was 70% (514/738). Compared with early-career attendings, experienced attendings perceived more intern involvement in decision making (odds ratio [OR]: 2.16, 95% confidence interval [CI]: 1.17-3.97, P=0.013). Hospitalists perceived less intern involvement in decision making (OR: 0.19, 95% CI: 0.06-0.58, P=0.004) and resident autonomy (OR: 0.27, 95% CI: 0.11-0.66, P=0.004) compared with nonhospitalists. A significant interaction existed between hospitalists and experience; experienced hospitalists perceived more intern decision making (OR: 7.36, 95% CI: 1.86-29.1, P=0.004) and resident autonomy (OR: 5.85, 95% CI: 1.75-19.6, P=0.004) compared with early-career hospitalists. With respect to secular trends, spring season of the academic year was associated with greater perception of intern decision making compared with other seasons (OR: 1.94, 95% CI: 1.18-3.19, P=0.009). The 2003 resident duty-hours restrictions were associated with decreased perception of intern decision making (OR: 0.51, 95% CI: 0.29-0.87, P=0.014) and resident autonomy (OR: 0.49, 95% CI: 0.28-0.86, P=0.012).

CONCLUSIONS

Perception of housestaff autonomy varies with attending characteristics and time trends. Hospitalists perceive autonomy and clinical decision making differently, depending on their attending experience.

Supervision in psychiatry: terra incognita?

PURPOSE OF REVIEW

Supervision is accepted as a part of postgraduate psychiatric training programmes (at least in the western world). However, despite its ubiquity, it is little researched. The purpose of this review was to synthesize research on supervision in psychiatry in the last 3 years (2009-2011). Given the dearth of such research, the boundary was extended to include general medicine and other mental health professions.

RECENT FINDINGS

The lack of research into supervision in psychiatry was confirmed by a comprehensive search of literature. The few articles published specific to psychiatry did, however, reflect the position of psychiatry, bridging medicine and the psychotherapies. Thus, they span from the impact of workplace-based assessments and reframing the theories of learning applicable to psychiatric training, through to the learning of psychotherapy by psychiatric trainees.The literature on supervision in general medicine is dominated by the impact of competency-based training, with its associated methods of assessment, and issues around the decision to entrust trainees with various professional activities.The psychotherapy supervision literature has been notable for two disparate themes: the centenary of psychotherapeutic supervision in 2009 prompted a number of scholars to write reflective pieces capturing advances and areas for further research; whereas the development of evidence-based clinical supervision has given rise to a considerable body of work.

SUMMARY

Research into supervision in psychiatry remains largely a terra incognita. If supervision is to continue to occupy a role as a key component of psychiatric training, research to justify that is needed urgently.

The role of role: learning in longitudinal integrated and traditional block clerkships

CONTEXT

Traditional block clerkship (BC) structures may not optimally support medical student participation in the workplace, whereas longitudinal integrated clerkship (LIC) structures seem more conducive to students' active engagement in patient care over time. Understanding the ways in which these two clerkship models influence students' roles and responsibilities can inform clinical learning programme design.

METHODS

This was a multicentre qualitative study. We conducted semi-structured interviews with LIC and BC medical students at three institutions early and late in the core clinical year to explore their experiences with patients and the roles they served. Using the framework of 'workplace affordances', qualitative coding focused on students' roles and qualities of the learning environment that invited or inhibited student participation. We compared transcripts of early- and late-year interviews to assess students' changing roles and conducted discrepant case analysis to ensure that coding fit the data.

RESULTS

Fifty-four students participated in interviews. They described serving three major roles in clinical care that respectively involved: providing support to patients; sharing information about patients across health care settings, and functioning in a doctor-like role. Both LIC and BC students served in the providing support and transmitting information roles both early and late in the year. By contrast, LIC students commonly served in the doctor-like role in managing their patients' care, particularly late in the year, whereas BC students rarely served in this role. Continuity in settings and in supervisors, and preceptors' endorsement of students' legitimate role afforded opportunities for students to participate actively in patient care.

CONCLUSIONS

Although both LIC and BC students reported serving in important roles in supporting their patients and sharing information about their care, only LIC students consistently described opportunities to grow into a doctor role with patients. The high level of integration of LIC students into care systems and their deeper relationships with preceptors and patients enhanced their motivation and feelings of competence to provide patient-centred care.

Perspective: Competencies, outcomes, and controversy--linking professional activities to competencies to improve resident education and practice

Regulatory organizations have recently emphasized the importance of structuring graduate medical education around mastery of core competencies. The difficulty is that core competencies attempt to distill a range of professional behaviors into arguable abstractions. As such, competencies can be difficult to grasp for trainees and faculty, who see them as unrelated to the intricacies of daily patient care. In this article, the authors describe how two initiatives are converging in a way that should make competencies tangible and relevant. One initiative is based on the idea that competencies will be more meaningful if trainees understand specifically how they relate to important professional activities in their own specialty. The authors suggest that there is a dyadic relationship between competencies and major professional activities in pediatric medicine. They also suggest that these relationships should be discussed as part of the process by which trainees are entrusted to perform clinical activities without direct supervision. The other initiative proposes to construct narrative milestones that provide a picture of what progression toward mastery of core competencies might look like. Together, the authors argue, these two initiatives should illuminate the core competencies by providing relevant clinical context and valuable educational substance.

Measuring progressive independence with the resident supervision index: theoretical approach

BACKGROUND

Graduate medical education is based on an on-the-job training model in which residents provide clinical care under supervision. The traditional method is to offer residents graduated levels of responsibility that will prepare them for independent practice. However, if progressive independence from supervision exceeds residents' progressive professional development, patient outcomes may be at risk. Leaders in graduate medical education have called for "optimal" supervision, yet few studies have conceptually defined what optimal supervision means and whether optimal care is theoretically compatible with progressive independence, nor have they developed a test for progressive independence.

OBJECTIVE

This research develops theory and analytic models as part of the Resident Supervision Index to quantify the intensity of supervision.

METHODS

We introduce an explicit set of assumptions for an ideal patient-centered theory of optimal supervision of resident-provided care. A critical assumption is that informed attending staff will use available resources to optimize patient outcomes first and foremost, with residents gaining clinical competencies by contributing to optimal care. Next, we derive mathematically the consequences of these assumptions as theoretical results.

RESULTS

Under optimal supervision, (1) patient outcome is expected to be no worse than if residents were not involved, (2) supervisors will avoid undersupervising residents (when patients are at increased risk for poor outcomes) or oversupervising residents (when residents miss clinical opportunities to practice care), (3) optimal patient outcomes will be compatible with progressive independence, (4) progressive development can be inferred from progressive independence whenever residents contribute to patient care, and (5) analytic models that test for progressive independence will emphasize adjusting the association between length of graduate medical education training and supervision for case complexity and clinic workload, but not patient health outcomes.

CONCLUSION

An explicit theoretical framework is critical to measure scientifically progressive independence from supervision using graduate medical education data.

Measuring progressive independence with the resident supervision index: empirical approach

BACKGROUND

A Resident Supervision Index (RSI) developed by our research team quantifies the intensity of resident supervision in graduate medical education, with the goal of testing for progressive independence. The 4-part RSI method includes a survey instrument for staff and residents (RSI Inventory), a strategy to score survey responses, a theoretical framework (patient centered optimal supervision), and a statistical model that accounts for the presence or absence of supervision and the intensity of patient care.

METHODS

The RSI Inventory data came from 140 outpatient encounters involving 57 residents and 37 attending physicians during a 3-month period at a Department of Veterans Affairs outpatient clinic. Responses are scored to quantitatively measure the intensity of resident supervision across 10 levels of patient services (staff is absent, is present, participated, or provided care with or without a resident), case discussion (resident-staff interaction), and oversight (staff reviewed case, reviewed medical chart, consulted with staff, or assessed patient). Scores are analyzed by level and for patient care using a 2-part model (supervision initiated [yes or no] versus intensity once supervision was initiated).

RESULTS

All resident encounters had patient care supervision, resident oversight, or both. Consistent with the progressive independence hypothesis, residents were 1.72 (P  =  .019) times more likely to be fully responsible for patient care with each additional postgraduate year. Decreasing case complexity, increasing clinic workload, and advanced nonmedical degrees among attending staff were negatively associated with supervision intensity, although associations varied by supervision level.

CONCLUSIONS

These data are consistent with the progressive independence hypothesis in graduate medical education and offer empirical support for the 4-part RSI method to quantify the intensity of resident supervision for research, program evaluation, and resident assessment purposes. Before informing policy, however, more scientific research in actual teaching settings is needed to better understand the relationships among patient outcomes, clinic workload, case complexity, and graduate medical education experience in resident supervision and professional development.