Brief tool to measure risk-adjusted surgical outcomes in resource-limited hospitals

External validation of the Codman score in colorectal surgery: a pragmatic tool to drive quality improvement

AIM

The simple six-variable Codman score is a tool designed to reduce the complexity of contemporary risk-adjusted postoperative mortality rate predictions. We sought to externally validate the Codman score in colorectal surgery.

METHODS

The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) participant user file and colectomy targeted dataset of 2020 were merged. A Codman score (composed of six variables: age, American Society of Anesthesiologists score, emergency status, degree of sepsis, functional status and preoperative blood transfusion) was assigned to every patient. The primary outcome was in-hospital mortality and secondary outcome was morbidity at 30 days. Logistic regression analyses were performed using the Codman score and the ACS NSQIP mortality and morbidity algorithms as independent variables for the primary and secondary outcomes. The predictive performance of discrimination area under receiver operating curve (AUC) and calibration of the Codman score and these algorithms were compared.

RESULTS

A total of 40 589 patients were included and a Codman score was generated for 40 557 (99.02%) patients. The median Codman score was 3 (interquartile range 1-4). To predict mortality, the Codman score had an AUC of 0.92 (95% CI 0.91-0.93) compared to the NSQIP mortality score 0.93 (95% CI 0.92-0.94). To predict morbidity, the Codman score had an AUC of 0.68 (95% CI 0.66-0.68) compared to the NSQIP morbidity score 0.72 (95% CI 0.71-0.73). When body mass index and surgical approach was added to the Codman score, the performance was no different to the NSQIP morbidity score. The calibration of observed versus expected predictions was almost perfect for both the morbidity and mortality NSQIP predictions, and only well fitted for Codman scores of less than 4 and greater than 7.

CONCLUSION

We propose that the six-variable Codman score is an efficient and actionable method for generating validated risk-adjusted outcome predictions and comparative benchmarks to drive quality improvement in colorectal surgery.

Determining the Minimum Dataset for Surgical Patients in Africa: A Delphi Study

BACKGROUND

It is often difficult for clinicians in African low- and middle-income countries middle-income countries to access useful aggregated data to identify areas for quality improvement. The aim of this Delphi study was to develop a standardised perioperative dataset for use in a registry.

METHODS

A Delphi method was followed to achieve consensus on the data points to include in a minimum perioperative dataset. The study consisted of two electronic surveys, followed by an online discussion and a final electronic survey (four Rounds).

RESULTS

Forty-one members of the African Perioperative Research Group participated in the process. Forty data points were deemed important and feasible to include in a minimum dataset for electronic capturing during the perioperative workflow by clinicians. A smaller dataset consisting of eight variables to define risk-adjusted perioperative mortality rate was also described.

CONCLUSIONS

The minimum perioperative dataset can be used in a collaborative effort to establish a resource accessible to African clinicians in improving quality of care.

Look in the Mirror, Not Out the Window

Over the past few decades, institutions have developed complex systems to compare themselves to others with the goal of improving healthcare quality. This process of comparison to others, called external benchmarking, has become the standard approach for quality improvement. However, external benchmarking is resource intensive, may not be flexible enough to focus on problems unique to individual institutions, and may lead to complacency for institutions ranking near the top of the quality bell curve for the measured metrics. Our singular focus on external benchmarking could also divert resources from other approaches. Here, we describe how the use of internal benchmarking, in which an institution focuses on improving their own processes over time, can offer unique advantages as well as offset the limitations of external benchmarking. We advocate for investment in both internal and external benchmarking as complimentary tools to improve healthcare quality.

Mini-abstract: External benchmarking, the most common approach to benchmarking today, compares outcomes across a group to improve performance. This article discusses internal benchmarking, a complementary approach with several strengths—including reduced data burden, selection of tailored metrics, motivation to improve for all performers, and incentive to look inward for solutions.

Creating an inexpensive hospital-wide surgical complication register for performance monitoring: a cohort study

Objectives

Basic tools that measure a hospital’s performance are required in order to benchmark or compare hospitals, but multispecialty institutional registries are rarely reported, and there is no consensus on their standard definitions and methodology. This study aimed to describe the setting up and first results of a hospital-wide surgical complication register that uses a minimal set of patient-related risk factors based on bedside data and produces outcomes data based on severity of complications.

Design

Cohort study.

Setting

Perioperative data related to all adult surgical procedures in a tertiary referral centre in Finland for 3 years (2016–2018) were included in the study. Complications were recorded according to a modified Clavien-Dindo classification, and the preoperative risk factors were compiled based on the literature and coded as numerical measures. The associations of preoperative risk factors with postoperative complications were analysed using the χ2 test or Fisher’s exact test.

Results

In total, 19 158 operations were performed between 2016 and 2018. Data on complications (Clavien 0–9) were recorded for 4529 surgical patients (23.6%), and 779 complications were reported (Clavien 1–9), leading to an overall complication rate of 17.2%. Of these, 4.6% were graded as major (Clavien 4–7). Patient-related risk factors with the strongest association with complications were growing American Society of Anesthesiologists Physical Status Classification System score (p<0.001), growing Charlson Index (p<0.001), poor nutritional status (Nutritional Risk Screening 2002), p=0.041) and urgency of surgery (p<0.001).

Conclusions

We describe an inexpensive hospital-wide surgical complication monitoring system that can produce valid numerical data for monitoring risk-adjusted surgical quality. The results showed that only a few patient-related risk factors were sufficient to account for the case mix.

Pediatric surgical quality improvement in low- and middle-income countries: What data to collect?

BACKGROUND

As surgical access expands in low- and middle-income countries, risk-adjusted outcomes data are needed to measure and improve surgical quality. Existing data collection tools in high-income countries are complex and may be burdensome to implement in low and middle income countries. This study determined the minimum dataset needed for adequate risk adjustment to predict perioperative mortality using data collected in a low- and middle-income countries.

METHODS

All patients admitted to the pediatric surgery ward at Mulago National Referral Hospital in Kampala, Uganda, from January 1, 2014 through December 31, 2018 were included. Studies were performed modelling the effects of reducing data granularity and reducing number of variables on the area under the receiver operating curve.

RESULTS

Of the 3,194 patients included, 1,941(61%) were male, 957(30%) were neonates, 1,714 (54%) had an operation, and the overall mortality rate was 14%. Granularity reduction analyses found that measuring age in ranges was equivalent to recording age in days (area under the receiver operating curve = 0.776; 95% confidence interval, 0.754%-0.798%, vs 0.815, 95% confidence interval, 0.794%-0.837%). Variable reduction analyses found that models with 3 predictor variables (diagnosis, procedure, and district) reached a maximum area under the receiver operating curve of 0.915 (95% confidence interval, 0.903%-0.928%), which was equivalent to the model using all available predictor variables (area under the receiver operating curve = 0.932; 95% confidence interval, 0.922%-0.943%). For all 3-variable models, the primary diagnosis contributed most to predictive ability (P < .001).

CONCLUSION

Effective risk adjustment for perioperative mortality can be performed in low and middle income countries using minimal, objective variables often already part of the patient's medical record. This approach can be used by clinicians, hospital administrators, and policymakers low- and middle-income countries looking to begin data collection to track and improve patient outcomes.

Development and Validation of Machine Learning-Based Race-Specific Models to Predict 10-Year Risk of Heart Failure: A Multicohort Analysis

BACKGROUND

Heart failure (HF) risk and the underlying risk factors vary by race. Traditional models for HF risk prediction treat race as a covariate in risk prediction and do not account for significant parameters such as cardiac biomarkers. Machine learning (ML) may offer advantages over traditional modeling techniques to develop race-specific HF risk prediction models and to elucidate important contributors of HF development across races.

METHODS

We performed a retrospective analysis of 4 large, community cohort studies (ARIC [Atherosclerosis Risk in Communities], DHS [Dallas Heart Study], JHS [Jackson Heart Study], and MESA [Multi-Ethnic Study of Atherosclerosis]) with adjudicated HF events. The study included participants who were >40 years of age and free of HF at baseline. Race-specific ML models for HF risk prediction were developed in the JHS cohort (for Black race-specific model) and White adults from ARIC (for White race-specific model). The models included 39 candidate variables across demographic, anthropometric, medical history, laboratory, and electrocardiographic domains. The ML models were externally validated and compared with prior established traditional and non-race-specific ML models in race-specific subgroups of the pooled MESA/DHS cohort and Black participants of ARIC. The Harrell C-index and Greenwood-Nam-D'Agostino χ2 tests were used to assess discrimination and calibration, respectively.

RESULTS

The ML models had excellent discrimination in the derivation cohorts for Black (n=4141 in JHS, C-index=0.88) and White (n=7858 in ARIC, C-index=0.89) participants. In the external validation cohorts, the race-specific ML model demonstrated adequate calibration and superior discrimination (Black individuals, C-index=0.80-0.83; White individuals, C-index=0.82) compared with established HF risk models or with non-race-specific ML models derived with race included as a covariate. Among the risk factors, natriuretic peptide levels were the most important predictor of HF risk across both races, followed by troponin levels in Black and ECG-based Cornell voltage in White individuals. Other key predictors of HF risk among Black individuals were glycemic parameters and socioeconomic factors. In contrast, prevalent cardiovascular disease and traditional cardiovascular risk factors were stronger predictors of HF risk in White adults.

CONCLUSIONS

Race-specific and ML-based HF risk models that integrate clinical, laboratory, and biomarker data demonstrated superior performance compared with traditional HF risk and non-race-specific ML models. This approach identifies distinct race-specific contributors of HF.

Epidemiology and mortality of pediatric surgical conditions: insights from a tertiary center in Uganda

INTRODUCTION/PURPOSE

The burden of pediatric surgical disease is largely unknown in low- and middle-income countries such as Uganda where access to care is limited.

METHODS

Implementation of a locally led database in January 2012 at a Ugandan tertiary referral hospital, and review of 3465 prospectively collected pediatric surgical admissions from January 2012 to August 2016.

RESULTS

2090 children (60.3%) underwent surgery during admission. 59% were male and 41% female. 28.6% of admissions were in neonates and 50.4% were in children less than 1 year old. Congenital anomalies including Hirschsprung's, anorectal malformations, intestinal atresias, omphalocele, and gastroschisis were the most common diagnoses (38.6%) followed by infections (15.0%) and tumors (8.6%). Mortality rates were substantially higher than those of high-income countries; for example, gastroschisis and intussusception had mortality rates of 90.1% and 19.7%, respectively. Post-operative mortality was highest in the congenital anomalies group (15.0%).

CONCLUSION

There is a high burden of infant congenital anomalies with higher mortality rates compared to high-income countries. The unit performs primarily specialized procedures appropriate for a tertiary center. We hope that these data will facilitate evaluation of ongoing quality improvement and capacity-building initiatives.

Local Research Catalyzes National Surgical Planning Comment on "Global Surgery - Informing National Strategies for Scaling Up Surgery in Sub-Saharan Africa"

In 2015 the Lancet Commission on Global Surgery (LCoGS) argued that surgical care is important to national health systems along with the economic viability of countries. Gajewski and colleagues outlined how the Commission’s blueprint has been implemented in sub-Saharan Africa, including two funded research projects that were integrated into national surgical plans. Here, we outline how the five processes proposed by Gajewski and colleagues are critical to integrate research, policy, and on-the-ground implementation. We also propose that, moving forward, the most pressing adjunct in many low- and middle-income countries (LMICs) may be a better characterization of rural surgical practices through rigorous research along with models that enable lessons to inform national policy.

Derivation, Validation and Application of a Pragmatic Risk Prediction Index for Benchmarking of Surgical Outcomes

BACKGROUND

Despite the existence of multiple validated risk assessment and quality benchmarking tools in surgery, their utility outside of high-income countries is limited. We sought to derive, validate and apply a scoring system that is both (1) feasible, and (2) reliably predicts mortality in a middle-income country (MIC) context.

METHODS

A 5-step methodology was used: (1) development of a de novo surgical outcomes database modeled around the American College of Surgeons' National Surgical Quality Improvement Program (ACS-NSQIP) in South Africa (SA dataset), (2) use of the resultant data to identify all predictors of in-hospital death with more than 90% capture indicating feasibility of collection, (3) use these predictors to derive and validate an integer-based score that reliably predicts in-hospital death in the 2012 ACS-NSQIP, (4) apply the score in the original SA dataset and demonstrate its performance, (5) identify threshold cutoffs of the score to prompt action and drive quality improvement.

RESULTS

Following step one-three above, the 13 point Codman's score was derived and validated on 211,737 and 109,079 patients, respectively, and includes: age 65 (1), partially or completely dependent functional status (1), preoperative transfusions ≥4 units (1), emergency operation (2), sepsis or septic shock (2) American Society of Anesthesia score ≥3 (3) and operative procedure (1-3). Application of the score to 373 patients in the SA dataset showed good discrimination and calibration to predict an in-hospital death. A Codman Score of 8 is an optimal cutoff point for defining expected and unexpected deaths.

CONCLUSION

We have designed a novel risk prediction score specific for a MIC context. The Codman Score can prove useful for both (1) preoperative decision-making and (2) benchmarking the quality of surgical care in MIC's.

Brief surgical procedure code lists for outcomes measurement and quality improvement in resource-limited settings

BACKGROUND

The lack of a classification system for surgical procedures in resource-limited settings hinders outcomes measurement and reporting. Existing procedure coding systems are prohibitively large and expensive to implement. We describe the creation and prospective validation of 3 brief procedure code lists applicable in low-resource settings, based on analysis of surgical procedures performed at Mbarara Regional Referral Hospital, Uganda's second largest public hospital.

METHODS

We reviewed operating room logbooks to identify all surgical operations performed at Mbarara Regional Referral Hospital during 2014. Based on the documented indication for surgery and procedure(s) performed, we assigned each operation up to 4 procedure codes from the International Classification of Diseases, 9th Revision, Clinical Modification. Coding of procedures was performed by 2 investigators, and a random 20% of procedures were coded by both investigators. These codes were aggregated to generate procedure code lists.

RESULTS

During 2014, 6,464 surgical procedures were performed at Mbarara Regional Referral Hospital, to which we assigned 435 unique procedure codes. Substantial inter-rater reliability was achieved (κ = 0.7037). The 111 most common procedure codes accounted for 90% of all codes assigned, 180 accounted for 95%, and 278 accounted for 98%. We considered these sets of codes as 3 procedure code lists. In a prospective validation, we found that these lists described 83.2%, 89.2%, and 92.6% of surgical procedures performed at Mbarara Regional Referral Hospital during August to September of 2015, respectively.

CONCLUSION

Empirically generated brief procedure code lists based on International Classification of Diseases, 9th Revision, Clinical Modification can be used to classify almost all surgical procedures performed at a Ugandan referral hospital. Such a standardized procedure coding system may enable better surgical data collection for administration, research, and quality improvement in resource-limited settings.

An efficient risk adjustment model to predict inpatient adverse events after surgery

BACKGROUND

Risk adjustment is an important component of surgical outcomes and quality analyses. Current models include numerous preoperative variables; however, the relative contribution of these variables may be limited. This research seeks to identify a model with the fewest number of variables necessary to perform an adequate risk adjustment to predict any inpatient adverse event for use in resource-limited settings.

METHODS

All patients from the National Surgical Quality Improvement Program (NSQIP) database from 2005 to 2010 were included. Outcomes were inpatient mortality or any surgical complication captured by NSQIP. Models were built by sequential addition of preoperative risk variables selected by their area under the receiver operator characteristic curve (AUC).

RESULTS

Among 863,349 patients, the single variable with the highest AUC was American Society of Anesthesiologists (ASA) classification (AUC = 0.7127). AUC values reached 0.7923 with five variables (ASA classification, wound classification, functional status prior to surgery, albumin, and age) and 0.7945 with six variables. The sixth variable was one of the following: alkaline phosphatase, weight loss, principal anesthesia technique, gender, or emergency status. The model with the highest discrimination that did not require laboratories included ASA classification, functional status prior to surgery, wound classification, and age (AUC = 0.7810). Including all 66 preoperative variables produced little additional gain (AUC = 0.8006).

CONCLUSIONS

Six variables are sufficient to develop a risk adjustment tool for inpatient surgical mortality and morbidity. This research has important implications for the field of surgical outcomes research by improving efficiency of data collection. This limited model can aid the expansion of risk-adjusted analyses to resource-limited settings worldwide.

Universities form research partnership to improve care in Mozambique

Mozambique, a country in sub-Saharan Africa, has experienced a severe physician shortage for many years, with patients having particularly limited access to surgical care. Medical education and surgical research partnerships formed in 2010 between Universidade Eduardo Mondlane (UEM) in Mozambique and the University of California, San Diego (UCSD), is developing potential solutions to the nation’s surgical care crisis. The findings that have emerged from this initiative, which are aimed at scaling up surgical services, will likely have important implications for improving surgical care in Mozambique and other underserved nations.