Reliability of ranking anesthesiologists and nurse anesthetists using leniency-adjusted clinical supervision and work habits scores

Patient and Operational Factors Do Not Substantively Affect the Annual Departmental Quality of Anesthesiologists' Clinical Supervision and Nurse Anesthetists' Work Habits

INTRODUCTION

Although safety climate, teamwork, and other non-technical skills in operating rooms probably influence clinical outcomes, direct associations have not been shown, at least partially due to sample size considerations. We report data from a retrospective cohort of anesthesia evaluations that can simplify the design of prospective observational studies in this area. Associations between non-technical skills in anesthesia, specifically anesthesiologists' quality of clinical supervision and nurse anesthetists' work habits, and patient and operational factors were examined.

METHODS

Eight fiscal years of evaluations and surgical cases from one hospital were included. Clinical supervision by anesthesiologists was evaluated daily using a nine-item scale. Work habits of nurse anesthetists were evaluated daily using a six-item scale. The dependent variables for both groups of staff were binary, whether all items were given the maximum score or not. Associations were tested with patient and operational variables for the entire day.

RESULTS

There were 40,718 evaluations of faculty anesthesiologists by trainees, 53,772 evaluations of nurse anesthetists by anesthesiologists, and 296,449 cases that raters and ratees started together. Cohen's d values were small (≤0.10) for all independent variables, suggesting a lack of any clinically meaningful association between patient and operational factors and evaluations given the maximum scores. For supervision quality, the day's count of orthopedic cases was a significant predictor of scores (P = 0.0011). However, the resulting absolute marginal change in the percentage of supervision scores equal to the maximum was only 0.8% (99% confidence interval: 0.2% to 1.4%), i.e., too small to be of clinical or managerial importance. Neurosurgical cases may have been a significant predictor of work habits (P = 0.0054). However, the resulting marginal change in the percentage of work habits scores equal to the maximum, an increase of 0.8% (99% confidence interval: 0.1% to 1.6%), which was again too small to be important.

CONCLUSIONS

When evaluating the effect of assigning anesthesiologists and nurse anesthetists with different clinical performance quality on clinical outcomes, supervision quality and work habits scores may be included as independent variables without concern that their effects are confounded by association with the patient or case characteristics. Clinical supervision and work habits are measures of non-technical skills. Hence, these findings suggest that non-technical performance can be judged by observing the typical small sample size of cases. Then, associations can be tested with administrative data for a far greater number of patients because there is unlikely to be a confounding association between patient and case characteristics and the clinicians' non-technical performance.

Lack of Benefit of Adjusting Adaptively Daily Invitations for the Evaluation of the Quality of Anesthesiologists' Supervision and Nurse Anesthetists' Work Habits

Introduction Whenever a department implements the evaluation of professionals, a reasonable operational goal is to request as few evaluations as possible. In anesthesiology, evaluations of anesthesiologists (by trainees) and nurse anesthetists (by anesthesiologists) with valid and psychometrically reliable scales have been made by requesting daily evaluations of the ratee's performance on the immediately preceding day. However, some trainees or nurse anesthetists are paired with the same anesthesiologist for multiple days of the same week. Multiple evaluations from the same rater during a given week may contribute little incremental information versus one evaluation from that rater for the week. We address whether daily evaluation requests could be adjusted adaptively to be made once per week, hopefully substantively reducing the number of evaluation requests. Methods Every day since 1 July 2013 at the studied department, anesthesia residents and fellows have been requested by email to evaluate anesthesiologists' quality of supervision provided during the preceding day using the De Oliveira Filho supervision scale. Every day since 29 March 2015, the anesthesiologists have been requested by email to evaluate the work habits of the nurse anesthetists during the preceding day. Both types of evaluations were made for interactions throughout the workday together, not for individual cases. The criterion for an electronic request to be sent is that the pair worked together for at least one hour that day. The current study was performed using evaluations of anesthesiologists' supervision and nurse anesthetists' work habits through 30 June 2023. Results If every evaluation request were completed by trainees on the same day it was requested, trainees would have received 13.5% fewer requests to evaluate anesthesiologists (9367/69,420), the maximum possible reduction. If anesthesiologists were to do the same for their evaluations of nurse anesthetists, the maximum possible reduction would be 7.1% fewer requests (4794/67,274). However, because most evaluations were completed after the day of the request (71%, 96,451/136,694), there would be fewer requests only if the evaluation were completed before or on the day of the next pairing. Consequently, in actual practice, there would have been only 2.4% fewer evaluation requests to trainees and 1.5% fewer to anesthesiologists, both decreases being significantly less than 5% (both adjusted P <0.0001). Among the trainees' evaluations of faculty anesthesiologists, there were 1.4% with very low scores, specifically, a mean score of less than three out of four (708/41,778). Using Bernoulli cumulative sum (CUSUM) among successive evaluations, 72 flags were raised over the 10 years. Among those, there were 36% with more than one rater giving an exceptionally low score during the same week (26/72). There were 97% (70/72) with at least one rater contributing more than one score to the recent cumulative sum. Conclusion Conceptually, evaluation requests could be skipped if a rater has already evaluated the ratee that week during an earlier day working together. Our results show that the opportunity for reductions in evaluation requests is significantly less than 5%. There may also be impaired monitoring for the detection of sudden major decreases in ratee performance. Thus, the simpler strategy of requesting evaluations daily after working together is warranted.

Overall anesthesia department quality of clinical supervision of trainees over a year evaluated using mixed effects models

BACKGROUND

Earlier studies of supervision in anesthesiology focused on how to evaluate the quality of individual anesthesiologist's clinical supervision of trainees. What is unknown is how to evaluate clinical supervision collectively, as provided by the department's faculty anesthesiologists. This information can be a metric that departments report annually or use to evaluate the effect of programs on the quality of clinical supervision over time.

METHODS

This retrospective cohort study used all 48,788 evaluations of the 115 faculty anesthesiologists using the De Oliveira Filho supervision scale completed by 202 residents and fellows over nine academic years at one department.

RESULTS

The distributions of mean scores among raters had marked negative skewness and were inconsistent with normal distributions. Consequently, accurate confidence intervals were impracticably wide, and their interpretation suggested lack of validity. In contrast, the logits of the proportions of scores equaling the maximum possible value, calculated for each rater, followed distributions sufficiently close to normal for statistically reliable use in random effects modeling. Parameters and confidence intervals were estimated using the intercept only random effects models, and then inverses computed to convert results from the logit scale to proportions. That approach is analogous to random effect meta-analysis of proportional incidence (or prevalence). Departments that chose to use semi-annual or annual surveys of trainees regarding supervision quality, and report those raw counts, will have far lower estimates of supervision quality versus when calculated accurately using daily evaluations of individual anesthesiologists.

CONCLUSIONS

Random effects meta-analysis of percentage incidences of maximum scores is a suitable statistical approach to analyze the daily supervision scores of individual anesthesiologists to evaluate the overall quality of clinical supervision provided to the trainees by the department over a year.

Effect of Insufficient Interaction on the Evaluation of Anesthesiologists’ Quality of Clinical Supervision by Anesthesiology Residents and Fellows

Introduction

In this study, we tested whether raters’ (residents and fellows) decisions to evaluate (or not) critical care anesthesiologists were significantly associated with clinical interactions documented from electronic health record progress notes and whether that influenced the reliability of supervision scores. We used the de Oliveira Filho clinical supervision scale for the evaluation of faculty anesthesiologists. Email requests were sent to raters who worked one hour or longer with the anesthesiologist the preceding day in an operating room. In contrast, potential raters were requested to evaluate all critical care anesthesiologists scheduled in intensive care units during the preceding week.

Methods

Over 7.6 years, raters (N=172) received a total of 7764 requests to evaluate 21 critical care anesthesiologists. Each rater received a median/mode of three evaluation requests, one per anesthesiologist on service that week. In this retrospective cohort study, we related responses (2970 selections of “insufficient interaction” to evaluate the faculty, and 3127 completed supervision scores) to progress notes (N=25,469) electronically co-signed by the rater and anesthesiologist combination during that week.

Results

Raters with few jointly signed notes were more likely to select insufficient interaction for evaluation (P < 0.0001): 62% when no joint notes versus 1% with at least 20 joint notes during the week. Still, rater-anesthesiologist combinations with no co-authored notes accounted not only for most (78%) of the evaluation requests but also most (56%) of the completed evaluations (both P < 0.0001). Among rater and anesthesiologist combinations with each anesthesiologist receiving evaluations from multiple (at least nine) raters and each rater evaluating multiple anesthesiologists, most (72%) rater-anesthesiologist combinations were among raters who had no co-authored notes with the anesthesiologist (P < 0.0001).

Conclusions

Regular use of the supervision scale should be practiced with raters who were selected not only from their scheduled clinical site but also using electronic health record data verifying joint workload with the anesthesiologist.

Suggested Work Guidelines, Based on Operating Room Data, for Departments with a Breast Milk Pumping Supervising Anesthesiologist

Objective: Coordinating breast milk pumping sessions is challenging for lactating anesthesiologists who supervise multiple simultaneous anesthetics. We quantify the minimum percentages of adjacent operating rooms (ORs) for which there could reliably (≥95%) be at least 30 minutes during the surgical time when the anesthesiologist covering three anesthetics could have her rooms covered by another anesthesiologist. Methods: The historical cohort study was from a large U.S. teaching hospital. We calculated the 5% lower prediction bounds of surgical times from 3 years of historical data, and then applied them to surgical start times from adjacent ORs during the next 1 year. Results: For >2/3rd of cases, an anesthesiologist supervising three ORs would lack a reliable 30-minute period of overlapping surgical times, and an even smaller chance per case at the ambulatory surgery center, 10% (9-11%). For approximately 42% (41-43%) of sufficiently long individual cases, there was absence of a 30-minute period during which both of the two adjacent ORs' cases were suitable for the anesthesiologist to receive a break (p < 0.0001 compared with one-third). Conclusions: Even when making assumptions that were deliberately unrealistic (e.g., anesthesiologists' responsibilities are only for ongoing OR cases), there is no practical mechanism for an anesthesiologist supervising three ORs to start cases, be relieved for a breast milk pumping session, and then return in time for the end of the anesthetics (e.g., tracheal extubation). Departments with anesthesiologists who are breastfeeding should consider having options for temporary clinical assignments, commensurate with training and experience, that do not require supervising >2 ORs.

The anesthetizing sites supervised to anesthesiologist ratio is an invalid surrogate for group productivity in academic anesthesia departments when used without consideration of the corresponding managerial decisions

When the anesthesiologist does not individually perform the anesthesia care, then to make valid comparisons among US anesthesia departments, one must consider the staffing ratio (i.e., how many cases each anesthesiologist supervises when working with Certified Registered Nurse Anesthetists [CRNAs] or Certified Anesthesiologist Assistants [CAA]). The staffing ratio also must be considered when accurately measuring group productivity. In this narrative review, we consider anesthesia departments with non-physician anesthesia providers and anesthesiology residents. We investigate the validity of such departments assessing the overall ratio of anesthetizing sites supervised per anesthesiologist as a surrogate for group clinical productivity. The sites/anesthesiologist ratio can be estimated accurately using the arithmetic mean calculated by anesthesiologist, the harmonic mean calculated by case, or the harmonic mean calculated by CRNA or CAA, but not by the arithmetic mean ratio by case. However, there is lack of validity to benchmarking the percentage time that anesthesiologists are supervising the maximum possible number of CRNAs or CAAs when some of the anesthesiologists also are supervising resident physicians. Assignments can differ in the total number anesthesiologists needed while every anesthesiologist is supervising as many sites as possible. Similarly, there is lack of validity to limiting assessment to the anesthesiologists supervising only CRNAs or CAAs. There also is lack of validity to limiting assessment only to cases performed by supervised CRNAs or CAAs. When cases can be assigned to anesthesiology residents or CRNAs or CAAs, increasing sites/anesthesiologist while limiting consideration to the CRNAs or CAAs creates incentive for the CRNAs or CAAs to be assigned cases, even when lesser productivity is the outcome. Decisions also can increase sites/anesthesiologist without increasing productivity (e.g., when one anesthesiologist relieves another before the end of the regular workday). A suitable alternative approach to fallaciously treating the sites/anesthesiologist ratio as a surrogate for productivity is that, when a teaching hospital supplies financial support, a responsibility of the anesthesia department is to explain annually the principal factors affecting productivity at each facility it manages and to show annually that decisions were made that maximized productivity, subject to the facilities' constraints.

Association between leniency of anesthesiologists when evaluating certified registered nurse anesthetists and when evaluating didactic lectures

Daily evaluations of certified registered nurse anesthetists' (CRNAs') work habits by anesthesiologists should be adjusted for rater leniency. The current study tested the hypothesis that there is a pairwise association by rater between leniencies of evaluations of CRNAs' daily work habits and of didactic lectures. The historical cohorts were anesthesiologists' evaluations over 53 months of CRNAs' daily work habits and 65 months of didactic lectures by visiting professors and faculty. The binary endpoints were the Likert scale scores for all 6 and 10 items, respectively, equaling the maximums of 5 for all items, or not. Mixed effects logistic regression estimated the odds of each ratee performing above or below average adjusted for rater leniency. Bivariate errors in variables least squares linear regression estimated the association between the leniency of the anesthesiologists' evaluations of work habits and didactic lectures. There were 29/107 (27%) raters who were more severe in their evaluations of CRNAs' work habits than other anesthesiologists (two-sided P < 0.01); 34/107 (32%) raters were more lenient. When evaluating lectures, 3/81 (4%) raters were more severe and 8/81 (10%) more lenient. Among the 67 anesthesiologists rating both, leniency (or severity) for work habits was not associated with that for lectures (P = 0.90, unitless slope between logits 0.02, 95% confidence interval -0.34 to 0.30). Rater leniency is of large magnitude when making daily clinical evaluations, even when using a valid and psychometrically reliable instrument. Rater leniency was context dependent, not solely a reflection of raters' personality or rating style.

Reliability and Validity of Performance Evaluations of Pain Medicine Clinical Faculty by Residents and Fellows Using a Supervision Scale

BACKGROUND

Annual and/or semiannual evaluations of pain medicine clinical faculty are mandatory for multiple organizations in the United States. We evaluated the validity and psychometric reliability of a modified version of de Oliveira Filho et al clinical supervision scale for this purpose.

METHODS

Six years of weekly evaluations of pain medicine clinical faculty by resident physicians and pain medicine fellows were studied. A 1-4 rating (4 = "Always") was assigned to each of 9 items (eg, "The faculty discussed with me the management of patients before starting a procedure or new therapy and accepted my suggestions, when appropriate").

RESULTS

Cronbach α of the 9 items equaled .975 (95% confidence interval [CI], 0.974-0.976). A G coefficient of 0.90 would be expected with 18 raters; the N = 12 six-month periods had mean 18.8 ± 5.9 (standard deviation [SD]) unique raters in each period (median = 20).Concurrent validity was shown by Kendall τb = 0.45 (P < .0001) pairwise by combination of ratee and rater between the average supervision score and the average score on a 21-item evaluation completed by fellows in pain medicine. Concurrent validity also was shown by τb = 0.36 (P = .0002) pairwise by combination of ratee and rater between the average pain medicine supervision score and the average operating room supervision score completed by anesthesiology residents.Average supervision scores differed markedly among the 113 raters (η = 0.485; CI, 0.447-0.490). Pairings of ratee and rater were nonrandom (Cramér V = 0.349; CI, 0.252-0.446).Mixed effects logistic regression was performed with rater leniency as covariates and the dependent variable being an average score equaling the maximum 4 vs <4. There were 3 of 13 ratees with significantly more averages <4 than the other ratees, based on P < .01 criterion; that is, their supervision was reliably rated as below average. There were 3 of 13 different ratees who provided supervision reliably rated as above average.Raters did not report higher supervision scores when they had the opportunity to perform more interventional pain procedures.

CONCLUSIONS

Evaluations of pain medicine clinical faculty are required. As found when used for evaluating operating room anesthesiologists, a supervision scale has excellent internal consistency, achievable reliability using 1-year periods of data, concurrent validity with other ratings, and the ability to differentiate among ratees. However, to be reliable, routinely collected supervision scores must be adjusted for rater leniency.