Comparison of outlier identification methods in hospital surgical quality improvement programs

Urgent Inpatient Colectomy Carries a Higher Morbidity and Mortality than Elective Surgery

BACKGROUND

Emergency colorectal surgery confers a higher risk of adverse outcomes compared to elective surgery. Few studies have examined the outcomes after urgent colectomies, typically defined as those performed at the index admission, but not performed at admission in an emergency fashion. The aim of this study is to evaluate the risk of adverse outcomes following urgent inpatient colorectal surgery.

MATERIALS AND METHODS

All adult patients undergoing colectomy between 2013 and 2017 in the ACS NSQIP were included in the analysis. Patients were grouped into Elective, Urgent and Emergency groups. The Urgent group was further stratified by time from admission to surgery. Baseline characteristics and 30 day outcomes were compared between the Elective, Urgent and Emergency groups using univariable and multivariable analyses.

RESULTS

104,486 patients underwent elective colorectal resection. 23,179 underwent urgent while 22,241 had emergency resections. Patients undergoing urgent colectomy presented with increased comorbidities, and experienced higher mortality (2.5-4.1%, AOR 2.3 (1.9 - 2.8)) compared to elective surgery (0.4%). Urgent colectomy was an independent risk factor for the majority of short term complications documented in NSQIP. Moreover, patients undergoing urgent colectomy more than a week following admission had an increased risk of bleeding, deep venous thrombosis, pulmonary embolism, urinary tract infection, and prolonged hospitalization.

CONCLUSION

Urgent colectomies are associated with a greater risk of adverse outcomes compared to elective surgery. Urgent status is an independent risk factor for post operative mortality and morbidity. Further characterization of this patient population and their specific challenges may help ameliorate these adverse events.

Ordinal outcome analysis improves the detection of between-hospital differences in outcome

Background

There is a growing interest in assessment of the quality of hospital care, based on outcome measures. Many quality of care comparisons rely on binary outcomes, for example mortality rates. Due to low numbers, the observed differences in outcome are partly subject to chance. We aimed to quantify the gain in efficiency by ordinal instead of binary outcome analyses for hospital comparisons. We analyzed patients with traumatic brain injury (TBI) and stroke as examples.

Methods

We sampled patients from two trials. We simulated ordinal and dichotomous outcomes based on the modified Rankin Scale (stroke) and Glasgow Outcome Scale (TBI) in scenarios with and without true differences between hospitals in outcome. The potential efficiency gain of ordinal outcomes, analyzed with ordinal logistic regression, compared to dichotomous outcomes, analyzed with binary logistic regression was expressed as the possible reduction in sample size while keeping the same statistical power to detect outliers.

Results

In the IMPACT study (9578 patients in 265 hospitals, mean number of patients per hospital = 36), the analysis of the ordinal scale rather than the dichotomized scale (‘unfavorable outcome’), allowed for up to 32% less patients in the analysis without a loss of power. In the PRACTISE trial (1657 patients in 12 hospitals, mean number of patients per hospital = 138), ordinal analysis allowed for 13% less patients. Compared to mortality, ordinal outcome analyses allowed for up to 37 to 63% less patients.

Conclusions

Ordinal analyses provide the statistical power of substantially larger studies which have been analyzed with dichotomization of endpoints. We advise to exploit ordinal outcome measures for hospital comparisons, in order to increase efficiency in quality of care measurements.

Trial registration

We do not report the results of a health care intervention.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-020-01185-7.

A Comprehensive Estimation of the Costs of 30-Day Postoperative Complications Using Actual Costs from Multiple, Diverse Hospitals

BACKGROUND

The cost of surgical care is largely measured by charges or payments, both of which are inadequate. Actual cost data from the hospital's perspective are required to accurately quantify the financial return on investment of engaging in quality improvement. The objective of this study was to define the cost of individual, 30-day postoperative complications using robust cost data from a diverse group of hospitals.

METHODS

Using clinical data derived from the American College of Surgeons National Surgical Quality Improvement Program, this retrospective study assessed postoperative complications for patients who underwent surgery at one of four hospitals in 2016. Actual direct and indirect 30-day costs were obtained, and the adjusted cost per complication was determined.

RESULTS

From the 6,387 patients identified, the three complications associated with the highest independent adjusted cost per event were prolonged ventilation ($48,168; 95% confidence interval [CI]: $21,861-$74,476), unplanned intubation ($26,718; 95% CI: $15,374-$38,062), and renal failure ($18,528; CI: $17,076-$19,981). The three complications associated with the lowest independent adjusted cost per event were urinary tract infection (-$372; 95% CI: -$1,336-$592), superficial surgical site infection ($2,473; 95% CI: -$256-$5,201) and venous thromboembolism ($7,909; 95% CI: -$17,903-$33,721). After colectomy, the adjusted independent cost of anastomotic leak was $10,195 (95% CI: $5,941-$14,449), while the cost of postoperative ileus was $10,205 (95% CI: $6,259-$14,149).

CONCLUSION

The actual hospital costs of complications were estimated using cost data from four diverse hospitals. These data can be used by hospitals to estimate the financial benefit of reducing surgical complications.

Outlier classification performance of risk adjustment methods when profiling multiple providers

BACKGROUND

When profiling multiple health care providers, adjustment for case-mix is essential to accurately classify the quality of providers. Unfortunately, misclassification of provider performance is not uncommon and can have grave implications. Propensity score (PS) methods have been proposed as viable alternatives to conventional multivariable regression. The objective was to assess the outlier classification performance of risk adjustment methods when profiling multiple providers.

METHODS

In a simulation study based on empirical data, the classification performance of logistic regression (fixed and random effects), PS adjustment, and three PS weighting methods was evaluated when varying parameters such as the number of providers, the average incidence of the outcome, and the percentage of outliers. Traditional classification accuracy measures were considered, including sensitivity and specificity.

RESULTS

Fixed effects logistic regression consistently had the highest sensitivity and negative predictive value, yet a low specificity and positive predictive value. Of the random effects methods, PS adjustment and random effects logistic regression performed equally well or better than all the remaining PS methods for all classification accuracy measures across the studied scenarios.

CONCLUSIONS

Of the evaluated PS methods, only PS adjustment can be considered a viable alternative to random effects logistic regression when profiling multiple providers in different scenarios.

Risk Associated With Complications and Mortality After Urgent Surgery vs Elective and Emergency Surgery: Implications for Defining "Quality" and Reporting Outcomes for Urgent Surgery

Importance

Given the current climate of outcomes-driven quality reporting, it is critical to appropriately risk stratify patients using standardized metrics.

Objective

To elucidate the risk associated with urgent surgery on complications and mortality after general surgical procedures.

Design, Setting, and Participants

This retrospective review used the American College of Surgeons National Surgery Quality Improvement Program database to capture all general surgery cases performed at 435 hospitals nationwide between January 1, 2013, and December 31, 2013. Data analysis was performed from November 11, 2015, to February 16, 2017.

Exposures

Any operations coded as both nonelective and nonemergency were designated into a novel category titled urgent.

Main Outcomes and Measures

The primary outcome was 30-day mortality; secondary outcomes included 30-day rates of complications, reoperation, and readmission in urgent cases compared with both elective and emergency cases.

Results

Of 173 643 patients undergoing general surgery (101 632 females and 72 011 males), 130 235 (75.0%) were categorized as elective, 22 592 (13.0%) as emergency, and 20 816 (12.0%) as nonelective and nonemergency. When controlling for standard American College of Surgeons National Surgery Quality Improvement Program preoperative risk factors, with elective surgery as the reference value, the 3 groups had significantly distinct odds ratios (ORs) of experiencing any complication (urgent surgery: OR, 1.38; 95% CI, 1.30-1.45; P < .001; and emergency surgery: OR, 1.65; 95% CI, 1.55-1.76; P < .001) and of mortality (urgent surgery: OR, 2.32; 95% CI, 2.00-2.68; P < .001; and emergency surgery: OR, 2.91; 95% CI, 2.48-3.41; P < .001). Surgical procedures performed urgently had a 12.3% rate of morbidity (n = 2560) and a 2.3% rate of mortality (n = 471).

Conclusions and Relevance

This study highlights the need for improved risk stratification on the basis of urgency because operations performed urgently have distinct rates of morbidity and mortality compared with procedures performed either electively or emergently. Because we tie quality outcomes to reimbursement, such a category should improve predictive models and more accurately reflect the quality and value of care provided by surgeons who do not have traditional elective practices.

Widespread Implications of ACOSOG Z0011: Effect on Total Mastectomy Patients

The ACOSOG Z0011 trial demonstrated that axillary lymph node dissection (ALND) is not necessary for all breast cancer patients. Patients who underwent breast conservation surgery (BCS) or total mastectomy (TM) and met Z0011 criteria were identified and sorted into pre- and post-Z0011 cohorts. Four hundred ninety four patients had breast cancer surgery from July 2008 to February 2013. Of these, 255 were pre-Z0011 and 239 were post. Pre-Z0011, 14 patients met Z0011 inclusion criteria in BCS subgroup. ALND was performed in 10 of these patients (71%). Post-Z0011, 14 patients met inclusion criteria, only 1 received ALND (7%). The decrease in ALND rate was significant (p=0.001). Pre-Z0011, 28 patients in TM subgroup met extrapolated inclusion criteria, twenty-three of which received ALND (82%). Post-Z0011, 14 patients in TM subgroup met extrapolated criteria, five of which underwent ALND (36%). The decrease in rate of ALND was significant (p=0.005). This study demonstrates that the Z0011 trial significantly altered management of the axilla in all breast cancer patients with positive sentinel lymph nodes, not just those receiving BCS.

Optimizing ACS NSQIP modeling for evaluation of surgical quality and risk: patient risk adjustment, procedure mix adjustment, shrinkage adjustment, and surgical focus

The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) collects detailed clinical data from participating hospitals using standardized data definitions, analyzes these data, and provides participating hospitals with reports that permit risk-adjusted comparisons with a surgical quality standard. Since its inception, the ACS NSQIP has worked to refine surgical outcomes measurements and enhance statistical methods to improve the reliability and validity of this hospital profiling. From an original focus on controlling for between-hospital differences in patient risk factors with logistic regression, ACS NSQIP has added a variable to better adjust for the complexity and risk profile of surgical procedures (procedure mix adjustment) and stabilized estimates derived from small samples by using a hierarchical model with shrinkage adjustment. New models have been developed focusing on specific surgical procedures (eg, "Procedure Targeted" models), which provide opportunities to incorporate indication and other procedure-specific variables and outcomes to improve risk adjustment. In addition, comparative benchmark reports given to participating hospitals have been expanded considerably to allow more detailed evaluations of performance. Finally, procedures have been developed to estimate surgical risk for individual patients. This article describes the development of, and justification for, these new statistical methods and reporting strategies in ACS NSQIP.

Hazard regression models of early mortality in trauma centers

BACKGROUND

Factors affecting early hospital deaths after trauma can be different from factors affecting later hospital deaths, and the distribution of short and long prehospital times can vary among hospitals. Hazard regression (HR) models might therefore be more useful than logistic regression (LR) models for analysis of trauma mortality, especially when treatment effects at different time points are of interest.

STUDY DESIGN

We obtained data for trauma center patients from the 2008-2009 National Trauma Data Bank. Patients were included if they had complete data for prehospital times, hospital times, survival outcomes, age, vital signs, and severity scores. Patients were excluded if pulseless on admission, transferred in or out, or had an Injury Severity Score <9. Using covariates proposed for the Trauma Quality Improvement Program and an indicator for each hospital, we compared LR models predicting survival at 8 hours after injury with HR models with survival censored at 8 hours. Hazard regression models were then modified to allow time-varying hospital effects.

RESULTS

A total of 85,327 patients in 161 hospitals met inclusion criteria. Crude hazards peaked initially and then declined steadily. When hazard ratios were assumed constant in HR models, they were similar to odds ratios in LR models associating increased mortality with increased age, firearm mechanism, increased severity, more deranged physiology, and estimated hospital-specific effects. However, when hospital effects were allowed to vary by time, HR models demonstrated that hospital outliers were not the same at different times after injury.

CONCLUSIONS

Hazard regression models with time-varying hazard ratios reveal inconsistencies in treatment effects, data quality, and/or timing of early death among trauma centers. Hazard regression models are generally more flexible than LR models, can be adapted for censored data, and potentially offer a better tool for analysis of factors affecting early death after injury.

The effect and durability of a pregraduation boot cAMP on the confidence of senior medical student entering surgical residencies

OBJECTIVE

Medical school does not specifically prepare students for surgical internship. Preinternship courses are known to increase confidence in multiple key areas. We examined the immediate effect and durability of effect of a surgical pregraduation preparatory course or "boot camp" on provider confidence in technical and medical management skills.

DESIGN

A 5-day boot camp was offered to senior medical students (SMS) entering surgical programs. SMS were anonymously surveyed before, after, and 6 months following the course. The same survey was given 6 months into internship to a control group of surgical interns who graduated from the same medical school but did not participate in boot camp before graduation. Data were compared between the time intervals and across cases and controls using the Wilcoxon rank-sum and signed-rank tests and the Student t test.

SETTING

A joint effort between the University of Pennsylvania School of Medicine, the Department of Surgery at the Hospital of the University of Pennsylvania, and the Penn Medicine Simulation Center in Philadelphia, PA.

PARTICIPANTS

All senior medical students set to graduate from a single institution entering general surgery or surgery subspecialties were offered the course. Twenty-nine students participated in the course.

RESULTS

Post-boot camp confidence scores of SMS were significantly greater in all areas except placement of a peripheral intravenous catheter compared with pre-boot camp scores. Six months into internship, the SMS boot camp group felt more confident than controls in their ability to perform a cricothyroidotomy (median 2.5 vs 1.0, p = 0.04) and to insert a chest tube (median 3.3 vs 1.0, p = 0.05). Otherwise, there was no residual difference in confidence levels between the boot camp group and the controls.

CONCLUSIONS

Boot camps can improve self-confidence in young doctors in many areas of perioperative care before enrolling in surgical residency. The effect is most durable in high risk, infrequently performed technical tasks. Future studies are under design to examine the impact of boot camps on the "July Effect."