Dynamic trends in cardiac surgery: why the logistic EuroSCORE is no longer suitable for contemporary cardiac surgery and implications for future risk models

Cause of death coding in asthma

BACKGROUND

While clinical coding is intended to be an objective and standardized practice, it is important to recognize that it is not entirely the case. The clinical and bureaucratic practices from event of death to a case being entered into a research dataset are important context for analysing and interpreting this data. Variation in practices can influence the accuracy of the final coded record in two different stages: the reporting of the death certificate, and the International Classification of Diseases (Version 10; ICD-10) coding of that certificate.

METHODS

This study investigated 91,022 deaths recorded in the Scottish Asthma Learning Healthcare System dataset between 2000 and 2017. Asthma-related deaths were identified by the presence of any of ICD-10 codes J45 or J46, in any position. These codes were categorized either as relating to asthma attacks specifically (status asthmatic; J46) or generally to asthma diagnosis (J45).

RESULTS

We found that one in every 200 deaths in this were coded as being asthma related. Less than 1% of asthma-related mortality records used both J45 and J46 ICD-10 codes as causes. Infection (predominantly pneumonia) was more commonly reported as a contributing cause of death when J45 was the primary coded cause, compared to J46, which specifically denotes asthma attacks.

CONCLUSION

Further inspection of patient history can be essential to validate deaths recorded as caused by asthma, and to identify potentially mis-recorded non-asthma deaths, particularly in those with complex comorbidities.

External validation of a paediatric Smart triage model for use in resource limited facilities

Models for digital triage of sick children at emergency departments of hospitals in resource poor settings have been developed. However, prior to their adoption, external validation should be performed to ensure their generalizability. We externally validated a previously published nine-predictor paediatric triage model (Smart Triage) developed in Uganda using data from two hospitals in Kenya. Both discrimination and calibration were assessed, and recalibration was performed by optimizing the intercept for classifying patients into emergency, priority, or non-urgent categories based on low-risk and high-risk thresholds. A total of 2539 patients were eligible at Hospital 1 and 2464 at Hospital 2, and 5003 for both hospitals combined; admission rates were 8.9%, 4.5%, and 6.8%, respectively. The model showed good discrimination, with area under the receiver-operator curve (AUC) of 0.826, 0.784 and 0.821, respectively. The pre-calibrated model at a low-risk threshold of 8% achieved a sensitivity of 93% (95% confidence interval, (CI):89%-96%), 81% (CI:74%-88%), and 89% (CI:85%-92%), respectively, and at a high-risk threshold of 40%, the model achieved a specificity of 86% (CI:84%-87%), 96% (CI:95%-97%), and 91% (CI:90%-92%), respectively. Recalibration improved the graphical fit, but new risk thresholds were required to optimize sensitivity and specificity.The Smart Triage model showed good discrimination on external validation but required recalibration to improve the graphical fit of the calibration plot. There was no change in the order of prioritization of patients following recalibration in the respective triage categories. Recalibration required new site-specific risk thresholds that may not be needed if prioritization based on rank is all that is required. The Smart Triage model shows promise for wider application for use in triage for sick children in different settings.

Empirical data drift detection experiments on real-world medical imaging data

While it is common to monitor deployed clinical artificial intelligence (AI) models for performance degradation, it is less common for the input data to be monitored for data drift - systemic changes to input distributions. However, when real-time evaluation may not be practical (eg., labeling costs) or when gold-labels are automatically generated, we argue that tracking data drift becomes a vital addition for AI deployments. In this work, we perform empirical experiments on real-world medical imaging to evaluate three data drift detection methods' ability to detect data drift caused (a) naturally (emergence of COVID-19 in X-rays) and (b) synthetically. We find that monitoring performance alone is not a good proxy for detecting data drift and that drift-detection heavily depends on sample size and patient features. Our work discusses the need and utility of data drift detection in various scenarios and highlights gaps in knowledge for the practical application of existing methods.

Risk Prediction Models for Long-Term Survival after Cardiac Surgery: A Systematic Review

BACKGROUND

The reporting of alternative postoperative measures of quality after cardiac surgery is becoming increasingly important as in-hospital mortality rates continue to decline. This study aims to systematically review and assess risk models designed to predict long-term outcomes after cardiac surgery.

METHODS

The MEDLINE and Embase databases were searched for articles published between 1990 and 2020. Studies developing or validating risk prediction models for long-term outcomes after cardiac surgery were included. Data were extracted using checklists for critical appraisal and systematic review of prediction modeling studies.

RESULTS

Eleven studies were identified for inclusion in the review, of which nine studies described the development of long-term risk prediction models after cardiac surgery and two were external validation studies. A total of 70 predictors were included across the nine models. The most frequently used predictors were age (n = 9), peripheral vascular disease (n = 8), renal disease (n = 8), and pulmonary disease (n = 8). Despite all models demonstrating acceptable performance on internal validation, only two models underwent external validation, both of which performed poorly.

CONCLUSION

Nine risk prediction models predicting long-term mortality after cardiac surgery have been identified in this review. Statistical issues with model development, limited inclusion of outcomes beyond 5 years of follow-up, and a lack of external validation studies means that none of the models identified can be recommended for use in contemporary cardiac surgery. Further work is needed either to successfully externally validate existing models or to develop new models. Newly developed models should aim to use standardized long-term specific reproducible outcome measures.

Using temporal recalibration to improve the calibration of risk prediction models in competing risk settings when there are trends in survival over time

We have previously proposed temporal recalibration to account for trends in survival over time to improve the calibration of predictions from prognostic models for new patients. This involves first estimating the predictor effects using data from all individuals (full dataset) and then re-estimating the baseline using a subset of the most recent data whilst constraining the predictor effects to remain the same. In this article, we demonstrate how temporal recalibration can be applied in competing risk settings by recalibrating each cause-specific (or subdistribution) hazard model separately. We illustrate this using an example of colon cancer survival with data from the Surveillance Epidemiology and End Results (SEER) program. Data from patients diagnosed in 1995-2004 were used to fit two models for deaths due to colon cancer and other causes respectively. We discuss considerations that need to be made in order to apply temporal recalibration such as the choice of data used in the recalibration step. We also demonstrate how to assess the calibration of these models in new data for patients diagnosed subsequently in 2005. Comparison was made to a standard analysis (when improvements over time are not taken into account) and a period analysis which is similar to temporal recalibration but differs in the data used to estimate the predictor effects. The 10-year calibration plots demonstrated that using the standard approach over-estimated the risk of death due to colon cancer and the total risk of death and that calibration was improved using temporal recalibration or period analysis.

Redosing of long acting cardioplegic solutions in adult cardiac surgery: A comparative study

INTRODUCTION

Despite promising results regarding using long-acting cardioplegia in the adult population, little data exists specifically for operations requiring prolonged aortic cross-clamp needing additional doses. In this pilot study, we evaluated the outcomes of patients undergoing surgery with prolonged cross-clamp time based on four different redosing compositions.

METHODS

During the period from January 2019 until June 2022, 288 patients undergoing cardiac surgery with an expected cross-clamp time over 60 min were prospectively randomized regarding the type of the cardioplegia used: Group 1 (N = 150)- single-dose del Nido antegrade cardioplegia and Group 2 (N = 138)- single-dose Histidine-Tryptophane-Ketoglutarate (HTK) antegrade cardioplegia. In patients with ischemic time over 60 min, needing a redosing were further analyzed separately in four subgroups: (A) Cold whole blood (CWB) (4:1) (N = 95); (A1: DN-CWB; A2: HTK-CWB) and (B) St Thomas Solution (N = 92) (B1: DN-St Thomas; B2: HTK-St Thomas. Control groups were C1 (DN redosed by DN) and C2 (HTK by HTK).

RESULTS

Troponin levels in A1 and B1 groups were significantly lower than in DN-control. Respiratory support time and incidence of atrial fibrillation were significantly lower in Group A1 versus DN-control.

CONCLUSIONS

Long-acting cardioplegic techniques are becoming widely utilized in the adult population, with minimal data on redosing methods/compositions for prolonged cases. Due to the small patient population, further investigation is needed to delineate optimal redosing methods, but this report brings to attention the initial success of multiple strategies.

Effects of interventions targeting the systemic inflammatory response to cardiac surgery on clinical outcomes in adults

BACKGROUND

Organ injury is a common and severe complication of cardiac surgery that contributes to the majority of deaths. There are no effective treatment or prevention strategies. It has been suggested that innate immune system activation may have a causal role in organ injury. A wide range of organ protection interventions targeting the innate immune response have been evaluated in randomised controlled trials (RCTs) in adult cardiac surgery patients, with inconsistent results in terms of effectiveness.

OBJECTIVES

The aim of the review was to summarise the results of RCTs of organ protection interventions targeting the innate immune response in adult cardiac surgery. The review considered whether the interventions had a treatment effect on inflammation, important clinical outcomes, or both.

SEARCH METHODS

CENTRAL, MEDLINE, Embase, conference proceedings and two trial registers were searched on October 2022 together with reference checking to identify additional studies.

SELECTION CRITERIA

RCTs comparing organ protection interventions targeting the innate immune response versus placebo or no treatment in adult patients undergoing cardiac surgery where the treatment effect on innate immune activation and on clinical outcomes of interest were reported.

DATA COLLECTION AND ANALYSIS

Searches, study selection, quality assessment, and data extractions were performed independently by pairs of authors. The primary inflammation outcomes were peak IL-6 and IL-8 concentrations in blood post-surgery. The primary clinical outcome was in-hospital or 30-day mortality. Treatment effects were expressed as risk ratios (RR) and standardised mean difference (SMD) with 95% confidence intervals (CI). Meta-analyses were performed using random effects models, and heterogeneity was assessed using I2.

MAIN RESULTS

A total of 40,255 participants from 328 RCTs were included in the synthesis. The effects of treatments on IL-6 (SMD -0.77, 95% CI -0.97 to -0.58, I2 = 92%) and IL-8 (SMD -0.92, 95% CI -1.20 to -0.65, I2 = 91%) were unclear due to heterogeneity. Heterogeneity for inflammation outcomes persisted across multiple sensitivity and moderator analyses. The pooled treatment effect for in-hospital or 30-day mortality was RR 0.78, 95% CI 0.68 to 0.91, I2 = 0%, suggesting a significant clinical benefit. There was little or no treatment effect on mortality when analyses were restricted to studies at low risk of bias. Post hoc analyses failed to demonstrate consistent treatment effects on inflammation and clinical outcomes. Levels of certainty for pooled treatment effects on the primary outcomes were very low.

AUTHORS' CONCLUSIONS

A systematic review of RCTs of organ protection interventions targeting innate immune system activation did not resolve uncertainty as to the effectiveness of these treatments, or the role of innate immunity in organ injury following cardiac surgery.

Dynamics in cardiac surgery: trends in population characteristics and the performance of the EuroSCORE II over time

OBJECTIVES

The aim of this study was to investigate the performance of the EuroSCORE II over time and dynamics in values of predictors included in the model.

METHODS

A cohort study was performed using data from the Netherlands Heart Registration. All cardiothoracic surgical procedures performed between 1 January 2013 and 31 December 2019 were included for analysis. Performance of the EuroSCORE II was assessed across 3-month intervals in terms of calibration and discrimination. For subgroups of major surgical procedures, performance of the EuroSCORE II was assessed across 12-month time intervals. Changes in values of individual EuroSCORE II predictors over time were assessed graphically.

RESULTS

A total of 103 404 cardiothoracic surgical procedures were included. Observed mortality risk ranged between 1.9% [95% confidence interval (CI) 1.6-2.4] and 3.6% (95% CI 2.6-4.4) across 3-month intervals, while the mean predicted mortality risk ranged between 3.4% (95% CI 3.3-3.6) and 4.2% (95% CI 3.9-4.6). The corresponding observed:expected ratios ranged from 0.50 (95% CI 0.46-0.61) to 0.95 (95% CI 0.74-1.16). Discriminative performance in terms of the c-statistic ranged between 0.82 (95% CI 0.78-0.89) and 0.89 (95% CI 0.87-0.93). The EuroSCORE II consistently overestimated mortality compared to observed mortality. This finding was consistent across all major cardiothoracic surgical procedures. Distributions of values of individual predictors varied broadly across predictors over time. Most notable trends were a decrease in elective surgery from 75% to 54% and a rise in patients with no or New York Heart Association I class heart failure from 27% to 33%.

CONCLUSIONS

The EuroSCORE II shows good discriminative performance, but consistently overestimates mortality risks of all types of major cardiothoracic surgical procedures in the Netherlands.

Dynamic prediction modeling of postoperative mortality among patients undergoing surgical aortic valve replacement in a statewide cohort over a 12-year period

Objective

Clinical prediction models for surgical aortic valve replacement mortality, are valuable decision tools but are often limited in their ability to account for changes in medical practice, patient selection, and the risk of outcomes over time. Recent research has identified methods to update models as new data accrue, but their effect on model performance has not been rigorously tested.

Methods

The study population included 44,546 adults who underwent an isolated surgical aortic valve replacement from January 1, 1999, to December 31, 2018, statewide in Pennsylvania. After chronologically splitting the data into training and validation sets, we compared calibration, discrimination, and accuracy measures amongst a nonupdating model to 2 methods of model updating: calibration regression and the novel dynamic logistic state space model.

Results

The risk of mortality decreased significantly during the validation period (P < .01) and the nonupdating model demonstrated poor calibration and reduced accuracy over time. Both updating models maintained better calibration (Hosmer-Lemeshow χ2 statistic) than the nonupdating model: nonupdating (156.5), calibration regression (4.9), and dynamic logistic state space model (8.0). Overall accuracy (Brier score) was consistently better across both updating models: dynamic logistic state space model (0.0252), calibration regression (0.0253), and nonupdating (0.0256). Discrimination improved with the dynamic logistic state space model (area under the curve, 0.696) compared with the nonupdating model (area under the curve, 0.685) and calibration regression method (area under the curve, 0.687).

Conclusions

Dynamic model updating can improve model accuracy, discrimination, and calibration. The decision as to which method to use may depend on which measure is most important in each clinical context. Because competing therapies have emerged for valve replacement models, updating may guide clinical decision making.

There is no such thing as a validated prediction model

BACKGROUND

Clinical prediction models should be validated before implementation in clinical practice. But is favorable performance at internal validation or one external validation sufficient to claim that a prediction model works well in the intended clinical context?

MAIN BODY

We argue to the contrary because (1) patient populations vary, (2) measurement procedures vary, and (3) populations and measurements change over time. Hence, we have to expect heterogeneity in model performance between locations and settings, and across time. It follows that prediction models are never truly validated. This does not imply that validation is not important. Rather, the current focus on developing new models should shift to a focus on more extensive, well-conducted, and well-reported validation studies of promising models.

CONCLUSION

Principled validation strategies are needed to understand and quantify heterogeneity, monitor performance over time, and update prediction models when appropriate. Such strategies will help to ensure that prediction models stay up-to-date and safe to support clinical decision-making.

Reproducibility and validity of the Portuguese Edmonton Frail Scale version in cardiac surgery patients

INTRODUCTION

Frailty is a multidimensional syndrome characterized by the loss of functional reserve, associated with higher mortality and less functional survival in cardiac surgery patients. The Edmonton Frail Scale (EFS) is a comprehensive tool devised for brief frailty detection. To the best of our knowledge, there are no culturally adapted and validated frailty screening tools that enable the identification of vulnerability domains suited for use in the preoperative setting in Portugal. This was the motivation for this study.

OBJECTIVES

To assess the validity and reproducibility of the Portuguese version of the EFS.

METHODS

Prospective observational study, in a sample of elective cardiac surgery patients. The Edmonton Frail Scale (EFS) translation and backtranslation were performed. Demographic and clinical data were collected, and the translated EFS translated, Geriatric Depression Scale, and Mini Mental State Examination Portuguese versions, Katz and Clinical Frailty Scales were administered. To assess validity Mann-Whitney test, Spearman's correlation coefficient, marginal homogeneity test and Kappa coefficient were employed. Reproducibility was assessed estimating kappa coefficient for the frailty diagnosis and the 11 EFS items. Intra-class correlation coefficients and the corresponding 95% confidence interval were estimated using linear mixed effects model.

RESULTS

The EFS Portuguese version revealed construct validity for frailty identification, as well as criterion validity for cognition and mood domains. Reproducibility was demonstrated, with k=0.62 (95% confidence interval (CI) 0.42-0.82) and intraclass correlation (ICC)=0.94 (95% CI 0.89-0.97) in inter-observer test and k=0.48 (95% CI 0.26-0.70) and ICC=0.85 (95% CI 0.72-0.92) in intra-observer test.

CONCLUSIONS

The EFS Portuguese version is valid and reproducible for use, suiting pre-operative frailty screening in a cardiac surgery setting.

Cardiac surgery risk prediction using ensemble machine learning to incorporate legacy risk scores: A benchmarking study

Objective

The introduction of new clinical risk scores (e.g. European System for Cardiac Operative Risk Evaluation (EuroSCORE) II) superseding original scores (e.g. EuroSCORE I) with different variable sets typically result in disparate datasets due to high levels of missingness for new score variables prior to time of adoption. Little is known about the use of ensemble learning to incorporate disparate data from legacy scores. We tested the hypothesised that Homogenenous and Heterogeneous Machine Learning (ML) ensembles will have better performance than ensembles of Dynamic Model Averaging (DMA) for combining knowledge from EuroSCORE I legacy data with EuroSCORE II data to predict cardiac surgery risk.

Methods

Using the National Adult Cardiac Surgery Audit dataset, we trained 12 different base learner models, based on two different variable sets from either EuroSCORE I (LogES) or EuroScore II (ES II), partitioned by the time of score adoption (1996-2016 or 2012-2016) and evaluated on holdout set (2017-2019). These base learner models were ensembled using nine different combinations of six ML algorithms to produce homogeneous or heterogeneous ensembles. Performance was assessed using a consensus metric.

Results

Xgboost homogenous ensemble (HE) was the highest performing model (clinical effectiveness metric (CEM) 0.725) with area under the curve (AUC) (0.8327; 95% confidence interval (CI) 0.8323-0.8329) followed by Random Forest HE (CEM 0.723; AUC 0.8325; 95%CI 0.8320-0.8326). Across different heterogenous ensembles, significantly better performance was obtained by combining siloed datasets across time (CEM 0.720) than building ensembles of either 1996-2011 (t-test adjusted, p = 1.67×10-6) or 2012-2019 (t-test adjusted, p = 1.35×10-193) datasets alone.

Conclusions

Both homogenous and heterogenous ML ensembles performed significantly better than DMA ensemble of Bayesian Update models. Time-dependent ensemble combination of variables, having differing qualities according to time of score adoption, enabled previously siloed data to be combined, leading to increased power, clinical interpretability of variables and usage of data.

Targeted validation: validating clinical prediction models in their intended population and setting

Clinical prediction models must be appropriately validated before they can be used. While validation studies are sometimes carefully designed to match an intended population/setting of the model, it is common for validation studies to take place with arbitrary datasets, chosen for convenience rather than relevance. We call estimating how well a model performs within the intended population/setting "targeted validation". Use of this term sharpens the focus on the intended use of a model, which may increase the applicability of developed models, avoid misleading conclusions, and reduce research waste. It also exposes that external validation may not be required when the intended population for the model matches the population used to develop the model; here, a robust internal validation may be sufficient, especially if the development dataset was large.

Predicting incident heart failure among patients with type 2 diabetes mellitus: The DM-CURE risk score

AIM

Early identification and prediction of incident heart failure (HF) is important because of severe morbidity and mortality. This study aimed to predict onset of HF among patients with diabetes.

METHODS

A time-varying Cox model was derived from ACCORD clinical trial to predict the risk of incident HF, defined by hospitalization for HF (HHF). External validation was performed on patient-level data from the Harmony Outcome trial and Chronic Renal Insufficiency Cohort (CRIC) study. The model was transformed into an integer-based scoring algorithm for 10-year risk evaluation. A stepwise algorithm identified and selected predictors from demographic characteristics, physical examination, laboratory results, medical history, medication and health care utilization, to develop a risk prediction model. The main outcome was incident HF, defined by HHF. The C statistic and Brier score were used to assess model performance.

RESULTS

In total, 9649 patients with diabetes free of HF were used, with median follow-up of 4 years and 299 incident hospitalization of HF events. The model identified several predictors for the 10-year HF incidence risk score 'DM-CURE': socio-Demographic [education, age at type 2 diabetes (T2DM) diagnosis], Metabolic (glycated haemoglobin, systolic blood pressure, body mass index, high-density lipoproteins), diabetes-related Complications (myocardial infarction, revascularization, cardiovascular medications, neuropathy, hypertension duration, albuminuria, urine albumin-to-creatinine ratio, End Stage Kidney Disease), and health care Utilization (all-cause hospitalization, emergency room visits) for Risk Evaluation. Among them, the strongest impact factors for future HF were age at T2DM diagnosis, health care utilization and cardiovascular disease-related variables. The model showed good discrimination (C statistic: 0.838, 95% CI: 0.821-0.855) and calibration (Brier score: 0.006, 95% CI: 0.006-0.007) in the ACCORD data and good performance in the validation data (Harmony: C statistic: 0.881, 95% CI: 0.863-0.899; CRIC: C statistic: 0.813, 95% CI: 0.794-0.833). The 10-year risk of incident HF increased in a graded fashion, from ≤1% in quintile 1 (score ≤14), 1%-5% in quintile 2 (score 15-23), 5%-10% in quintile 3 (score 24-27), 10%-20% in quintile 4 (score 28-33) and ≥20% in quintile 5 (score >33).

CONCLUSIONS

The DM-CURE model and score were useful for population risk stratification of incident HHF among patients with T2DM and can be easily applied in clinical practice.

Quality improvement program at Latin America

BACKGROUND

The current challenge of cardiac surgery (CS) is to improve outcomes in adverse scenarios. The aim of this study was to assess the impact of a quality improvement program (QIP) on hospital mortality in the largest CS center in Latin America.

METHODS

Patients were divided into two groups: before (Jan 2013-Dec 2015, n = 3534) and after establishment of the QIP (Jan 2017-Dec 2019, n = 3544). The QIP consisted of the implementation of 10 central initiatives during 2016. The procedures evaluated were isolated coronary artery bypass grafting surgery (CABG), mitral valve surgery, aortic valve surgery, combined mitral and aortic valve surgery, and CABG associated with heart valve surgery. Propensity Score Matching (PSM) was used to adjust for inequality in patients' preoperative characteristics before and after the implementation of QIP. A multivariate logistic regression model was built to predict hospital mortality and validated using discrimination and calibration metrics.

RESULTS

The PMS paired two groups using 5 variables, obtaining 858 patients operated before (non-QIP) and 858 patients operated after the implementation of the QIP. When comparing the QIP versus Non-QIP group, there was a shorter length of stay in all phases of hospitalization. In addition, the patients evolved with less anemia (P = 0.001), use of intra-aortic balloon pump (P = 0.003), atrial fibrillation (P = 0.001), acute kidney injury (P < 0.001), cardiogenic shock (P = 0.011), sepsis (P = 0.046), and hospital mortality (P = 0.001). In the multiple model, among the predictors of hospital mortality, the lack of QIP increased the chances of mortality by 2.09 times.

CONCLUSION

The implementation of a first CS QIP in Latin America was associated with a reduction in length of hospital stay, complications and mortality after the cardiac surgeries analyzed.

Open questions and research gaps for monitoring and updating AI-enabled tools in clinical settings

As the implementation of artificial intelligence (AI)-enabled tools is realized across diverse clinical environments, there is a growing understanding of the need for ongoing monitoring and updating of prediction models. Dataset shift-temporal changes in clinical practice, patient populations, and information systems-is now well-documented as a source of deteriorating model accuracy and a challenge to the sustainability of AI-enabled tools in clinical care. While best practices are well-established for training and validating new models, there has been limited work developing best practices for prospective validation and model maintenance. In this paper, we highlight the need for updating clinical prediction models and discuss open questions regarding this critical aspect of the AI modeling lifecycle in three focus areas: model maintenance policies, performance monitoring perspectives, and model updating strategies. With the increasing adoption of AI-enabled tools, the need for such best practices must be addressed and incorporated into new and existing implementations. This commentary aims to encourage conversation and motivate additional research across clinical and data science stakeholders.

Maintaining a National Acute Kidney Injury Risk Prediction Model to Support Local Quality Benchmarking

BACKGROUND

The utility of quality dashboards to inform decision-making and improve clinical outcomes is tightly linked to the accuracy of the information they provide and, in turn, accuracy of underlying prediction models. Despite recognition of the need to update prediction models to maintain accuracy over time, there is limited guidance on updating strategies. We compare predefined and surveillance-based updating strategies applied to a model supporting quality evaluations among US veterans.

METHODS

We evaluated the performance of a US Department of Veterans Affairs-specific model for postcardiac catheterization acute kidney injury using routinely collected observational data over the 6 years following model development (n=90 295 procedures in 2013-2019). Predicted probabilities were generated from the original model, an annually retrained model, and a surveillance-based approach that monitored performance to inform the timing and method of updates. We evaluated how updating the national model impacted regional quality profiles. We compared observed-to-expected outcome ratios, where values above and below 1 indicated more and fewer adverse outcomes than expected, respectively.

RESULTS

The original model overpredicted risk at the national level (observed-to-expected outcome ratio, 0.75 [0.74-0.77]). Annual retraining updated the model 5×; surveillance-based updating retrained once and recalibrated twice. While both strategies improved performance, the surveillance-based approach provided superior calibration (observed-to-expected outcome ratio, 1.01 [0.99-1.03] versus 0.94 [0.92-0.96]). Overprediction by the original model led to optimistic quality assessments, incorrectly indicating most of the US Department of Veterans Affairs' 18 regions observed fewer acute kidney injury events than predicted. Both updating strategies revealed 16 regions performed as expected and 2 regions increasingly underperformed, having more acute kidney injury events than predicted.

CONCLUSIONS

Miscalibrated clinical prediction models provide inaccurate pictures of performance across clinical units, and degrading calibration further complicates our understanding of quality. Updating strategies tailored to health system needs and capacity should be incorporated into model implementation plans to promote the utility and longevity of quality reporting tools.

The Effect of Low Preoperative Ejection Fraction on Mortality After Cardiac Surgery in Indonesia

Background

Among cardiac surgery patients, low preoperative left ventricular ejection fraction (LVEF) is common and has been associated with poor outcomes. The objective of this study was to assess the association between LVEF and postoperative mortality in patients undergoing open-heart surgery in several hospitals in Indonesia.

Methods

We conducted a multicenter study with the retrospective design using data from patients undergoing open-heart surgery in 4 institutions in Indonesia. Data regarding LVEF and other potential risk factors were extracted from medical records and compiled in one datasheet. Statistical analyses were performed to assess if low LVEF was associated with postoperative mortality and identify other potential risk factors.

Results

A total of 4789 patients underwent cardiac surgery in participating centers during the study period. Of these, 189 subjects (3.9%) had poor preoperative LVEF. Poor LVEF was associated with postoperative mortality (adjusted OR 2.761, 95% CI 1.763–4.323, p < 0.001). Based on types of surgery, LVEF had a significant association with mortality only in CABG patients, while there was no such association in valve surgery and inconclusive in congenital surgery patients. Other significant independent predictors of in-hospital mortality included age more than 65 years old, non-elective surgery, the complexity of procedures, history of cardiac surgery, organ failure, CARE score ≥ 3, NYHA class ≥ III, and poor right ventricular function.

Conclusion

Patients with low preoperative LVEF undergoing open-heart surgery had a higher risk of postoperative mortality. Cardiac surgery can be performed with acceptable mortality rates. Accurate selection of patients, risk/benefit evaluation, and planning of surgical and anesthesiological management are mandatory to improve outcomes.

Galectin-3 as an independent prognostic factor after heart transplantation

BACKGROUND

Galectin-3 (GAL3) is linked to the prognosis of patients with heart failure and after heart transplantation (HTx). We assessed the prognostic role of GAL3 in a long-term follow-up after HTx.

METHODS

HTx patients (N = 121) were evaluated in a single-center, noninterventional, prospective, observational study. The median follow-up was 96 months (2,942 days, interquartile range (IQR) 2,408-3,264 days), and 40 patients died. GAL3 was measured before HTx, +10 days after HTx, and during the first posttransplant year. Survival analysis (all-cause mortality) was performed with adjustments for clinical and laboratory variables.

RESULTS

The median pretransplant GAL3 level was 18.0 μg/L (IQR 14.0-25.9), and higher values were associated with older age, worse kidney function, left ventricular assist device use before HTx, a higher IMPACT score and mortality. Increased pretransplant GAL3 predicted shorter survival time (HR 2.05, 95% CI 1.09-3.85, p<0.05). Similar prognostic power had GAL3 on the 10th posttransplant day (HR 2.03, 95% CI 1.08-3.82, p<0.05). GAL3 was an independent predictor of death after adjustment for clinical variables (age, infection, diabetes, smoking, IMPACT score, troponin).

CONCLUSIONS

GAL3 was significantly associated with all-cause mortality after adjusting for clinical and laboratory variables and may serve as an additional prognostic biomarker. This article is protected by copyright. All rights reserved.

Advanced cardiovascular risk prediction in the emergency department: updating a clinical prediction model - a large database study protocol

BACKGROUND

Patients presenting with chest pain represent a large proportion of attendances to emergency departments. In these patients clinicians often consider the diagnosis of acute myocardial infarction (AMI), the timely recognition and treatment of which is clinically important. Clinical prediction models (CPMs) have been used to enhance early diagnosis of AMI. The Troponin-only Manchester Acute Coronary Syndromes (T-MACS) decision aid is currently in clinical use across Greater Manchester. CPMs have been shown to deteriorate over time through calibration drift. We aim to assess potential calibration drift with T-MACS and compare methods for updating the model.

METHODS

We will use routinely collected electronic data from patients who were treated using TMACS at two large NHS hospitals. This is estimated to include approximately 14,000 patient episodes spanning June 2016 to October 2020. The primary outcome of acute myocardial infarction will be sourced from NHS Digital's admitted patient care dataset. We will assess the calibration drift of the existing model and the benefit of updating the CPM by model recalibration, model extension and dynamic updating. These models will be validated by bootstrapping and one step ahead prequential testing. We will evaluate predictive performance using calibrations plots and c-statistics. We will also examine the reclassification of predicted probability with the updated TMACS model.

DISCUSSION

CPMs are widely used in modern medicine, but are vulnerable to deteriorating calibration over time. Ongoing refinement using routinely collected electronic data will inevitably be more efficient than deriving and validating new models. In this analysis we will seek to exemplify methods for updating CPMs to protect the initial investment of time and effort. If successful, the updating methods could be used to continually refine the algorithm used within TMACS, maintaining or even improving predictive performance over time.

TRIAL REGISTRATION

ISRCTN number: ISRCTN41008456.

Activation of the innate immune response and organ injury after cardiac surgery: a systematic review and meta-analysis of randomised trials and analysis of individual patient data from randomised and non-randomised studies

BACKGROUND

It is unclear whether the innate immune response represents a therapeutic target for organ protection strategies in cardiac surgery.

METHODS

A systematic review of trials of interventions targeting the inflammatory response to cardiac surgery reporting treatment effects on both innate immune system cytokines and organ injury was performed. The protocol was registered at the International Prospective Register of Systematic Reviews: CRD42020187239. Searches of the Cochrane Central Register of Controlled Trials, MEDLINE, and Embase were performed. Random-effects meta-analyses were used for the primary analysis. A separate analysis of individual patient data from six studies (n=785) explored sources of heterogeneity for treatment effects on cytokine levels.

RESULTS

Searches to May 2020 identified 251 trials evaluating 24 interventions with 20 582 participants for inclusion. Most trials had important limitations. Methodological limitations of the included trials and heterogeneity of the treatment effects on cytokine levels between trials limited interpretation. The primary analysis demonstrated inconsistency in the direction of the treatment effects on innate immunity and organ failure or death between interventions. Analyses restricted to important subgroups or trials with fewer limitations showed similar results. Meta-regression, pooling available data from all trials, demonstrated no association between the direction of the treatment effects on inflammatory cytokines and organ injury or death. The analysis of individual patient data demonstrated heterogeneity in the association between the cytokine response and organ injury after cardiac surgery for people >75 yr old and those with some chronic diseases.

CONCLUSIONS

The certainty of the evidence for a causal relationship between innate immune system activation and organ injury after cardiac surgery is low.

Coronary artery bypass surgery in the UK, trends in activity and outcomes from a 15-year complete national series

OBJECTIVES

The aim of this study was to review the UK national trends in activity and outcome in coronary artery bypass graft (CABG) over a 15-year period (2002-2016).

METHODS

Validated data collected (2002-2016) and uploaded to National Institute for Cardiovascular Outcomes Research were used to generate summary data from the National Adult Cardiac Surgery Audit Database for the analysis. Logistic European System of Cardiac Operative Risk Evaluation was used for risk stratification with recalibration applied for governance. Data were analysed by financial year and presented as numerical, categorical, %, mean and standard deviation where appropriate. Mortality was recorded as death in hospital at any time after index CABG operation.

RESULTS

A total of 347 626 CABG procedures (282 883 isolated CABG, 61 109 CABG and valve and 4132 redo CABG) were recorded. Over this period annual activity reduced from 66.6% of workload to 41.7%. The mean age for isolated CABG was 65.7 years. The mean log European System of Cardiac Operative Risk Evaluation was 3.1, 5.9 and 23.2 for elective, urgent and emergency isolated CABG, respectively. There was a decline in the observed mortality for all procedures. Overall mortality for isolated CABG surgery is now 1.0% and only 0.6% for elective operations.

CONCLUSIONS

Quality of care and risk-adjusted mortality rates have consistently improved over the last 15 years despite the increasing risk profile of patients. There have been a consistent decline in overall case volumes and a three-fold increase in elderly cases.

A systematic review of risk prediction models for perioperative mortality after thoracic surgery

OBJECTIVES

Guidelines advocate that patients being considered for thoracic surgery should undergo a comprehensive preoperative risk assessment. Multiple risk prediction models to estimate the risk of mortality after thoracic surgery have been developed, but their quality and performance has not been reviewed in a systematic way. The objective was to systematically review these models and critically appraise their performance.

METHODS

The Cochrane Library and the MEDLINE database were searched for articles published between 1990 and 2019. Studies that developed or validated a model predicting perioperative mortality after thoracic surgery were included. Data were extracted based on the checklist for critical appraisal and data extraction for systematic reviews of prediction modelling studies.

RESULTS

A total of 31 studies describing 22 different risk prediction models were identified. There were 20 models developed specifically for thoracic surgery with two developed in other surgical specialties. A total of 57 different predictors were included across the identified models. Age, sex and pneumonectomy were the most frequently included predictors in 19, 13 and 11 models, respectively. Model performance based on either discrimination or calibration was inadequate for all externally validated models. The most recent data included in validation studies were from 2018. Risk of bias (assessed using Prediction model Risk Of Bias ASsessment Tool) was high for all except two models.

CONCLUSIONS

Despite multiple risk prediction models being developed to predict perioperative mortality after thoracic surgery, none could be described as appropriate for contemporary thoracic surgery. Contemporary validation of available models or new model development is required to ensure that appropriate estimates of operative risk are available for contemporary thoracic surgical practice.

Association between preoperative frailty and outcomes among adults undergoing cardiac surgery: a prospective cohort study

BACKGROUND

The identification of frailty before complex and invasive procedures may have relevance for prognostic and recovery purposes, to optimally inform patients, caregivers and clinicians about perioperative risk and postoperative care needs. The aim of this study was to estimate the prevalence of frailty and describe the associated clinical course and outcomes of patients referred for nonemergent cardiac surgery.

METHODS

A prospective cohort of patients aged 50 years and older referred for nonemergent cardiac surgery in Alberta, Canada, from November 2011 to March 2014 were screened preoperatively for frailty, defined as a Clinical Frailty Scale (CFS) score of 5 or greater. Postoperatively, patients were followed by telephone to assess CFS score, health services use and vital status. The primary outcome was all-cause hospital mortality. Secondary outcomes included health services use, hospital discharge disposition, 1-year health-related quality of life and all-cause 5-year mortality.

RESULTS

The cohort (n = 529) had a mean age of 67 (standard deviation [SD] 9) years; 25.9% were female, and the prevalence of frailty was 9.6% (n = 51; 95% confidence interval [CI] 7.3%-12.5%). Frail patients were older (median age 75, interquartile range [IQR] 65-80 v. 67, IQR 60-73, yr; p < 0.001), were more likely to be female (51.0% v. 23.2%; p < 0.001), had a higher mean EuroSCORE II (8, SD 3 v. 5, SD 3; p < 0.001) and received combined coronary artery bypass grafting and valve procedures more frequently (29.4% v. 15.9%; p = 0.02) than nonfrail patients. Postoperatively, frail patients had a longer median duration of stay in the cardiovascular intensive care unit (median difference 2.2, 95% CI 1.60-2.79) and hospital (median difference 9.3, 95% CI 8.2-10.3). Hospital mortality was 9.8% among frail patients and 1.0% among nonfrail patients (adjusted hazard ratio 3.84, 95% CI 0.90-16.34).

INTERPRETATION

Preoperative frailty was present in 10% of patients and was associated with a higher risk of morbidity and greater health services use. Preoperative frailty has important implications for the postoperative clinical course and resource utilization of patients undergoing cardiac surgery.

Developing more generalizable prediction models from pooled studies and large clustered data sets

Prediction models often yield inaccurate predictions for new individuals. Large data sets from pooled studies or electronic healthcare records may alleviate this with an increased sample size and variability in sample characteristics. However, existing strategies for prediction model development generally do not account for heterogeneity in predictor‐outcome associations between different settings and populations. This limits the generalizability of developed models (even from large, combined, clustered data sets) and necessitates local revisions. We aim to develop methodology for producing prediction models that require less tailoring to different settings and populations. We adopt internal‐external cross‐validation to assess and reduce heterogeneity in models' predictive performance during the development. We propose a predictor selection algorithm that optimizes the (weighted) average performance while minimizing its variability across the hold‐out clusters (or studies). Predictors are added iteratively until the estimated generalizability is optimized. We illustrate this by developing a model for predicting the risk of atrial fibrillation and updating an existing one for diagnosing deep vein thrombosis, using individual participant data from 20 cohorts (N = 10 873) and 11 diagnostic studies (N = 10 014), respectively. Meta‐analysis of calibration and discrimination performance in each hold‐out cluster shows that trade‐offs between average and heterogeneity of performance occurred. Our methodology enables the assessment of heterogeneity of prediction model performance during model development in multiple or clustered data sets, thereby informing researchers on predictor selection to improve the generalizability to different settings and populations, and reduce the need for model tailoring. Our methodology has been implemented in the R package metamisc.

Predicting breast cancer 5-year survival using machine learning: A systematic review

BACKGROUND

Accurately predicting the survival rate of breast cancer patients is a major issue for cancer researchers. Machine learning (ML) has attracted much attention with the hope that it could provide accurate results, but its modeling methods and prediction performance remain controversial. The aim of this systematic review is to identify and critically appraise current studies regarding the application of ML in predicting the 5-year survival rate of breast cancer.

METHODS

In accordance with the PRISMA guidelines, two researchers independently searched the PubMed (including MEDLINE), Embase, and Web of Science Core databases from inception to November 30, 2020. The search terms included breast neoplasms, survival, machine learning, and specific algorithm names. The included studies related to the use of ML to build a breast cancer survival prediction model and model performance that can be measured with the value of said verification results. The excluded studies in which the modeling process were not explained clearly and had incomplete information. The extracted information included literature information, database information, data preparation and modeling process information, model construction and performance evaluation information, and candidate predictor information.

RESULTS

Thirty-one studies that met the inclusion criteria were included, most of which were published after 2013. The most frequently used ML methods were decision trees (19 studies, 61.3%), artificial neural networks (18 studies, 58.1%), support vector machines (16 studies, 51.6%), and ensemble learning (10 studies, 32.3%). The median sample size was 37256 (range 200 to 659820) patients, and the median predictor was 16 (range 3 to 625). The accuracy of 29 studies ranged from 0.510 to 0.971. The sensitivity of 25 studies ranged from 0.037 to 1. The specificity of 24 studies ranged from 0.008 to 0.993. The AUC of 20 studies ranged from 0.500 to 0.972. The precision of 6 studies ranged from 0.549 to 1. All of the models were internally validated, and only one was externally validated.

CONCLUSIONS

Overall, compared with traditional statistical methods, the performance of ML models does not necessarily show any improvement, and this area of research still faces limitations related to a lack of data preprocessing steps, the excessive differences of sample feature selection, and issues related to validation. Further optimization of the performance of the proposed model is also needed in the future, which requires more standardization and subsequent validation.

Individual participant data meta-analysis for external validation, recalibration, and updating of a flexible parametric prognostic model

Individual participant data (IPD) from multiple sources allows external validation of a prognostic model across multiple populations. Often this reveals poor calibration, potentially causing poor predictive performance in some populations. However, rather than discarding the model outright, it may be possible to modify the model to improve performance using recalibration techniques. We use IPD meta-analysis to identify the simplest method to achieve good model performance. We examine four options for recalibrating an existing time-to-event model across multiple populations: (i) shifting the baseline hazard by a constant, (ii) re-estimating the shape of the baseline hazard, (iii) adjusting the prognostic index as a whole, and (iv) adjusting individual predictor effects. For each strategy, IPD meta-analysis examines (heterogeneity in) model performance across populations. Additionally, the probability of achieving good performance in a new population can be calculated allowing ranking of recalibration methods. In an applied example, IPD meta-analysis reveals that the existing model had poor calibration in some populations, and large heterogeneity across populations. However, re-estimation of the intercept substantially improved the expected calibration in new populations, and reduced between-population heterogeneity. Comparing recalibration strategies showed that re-estimating both the magnitude and shape of the baseline hazard gave the highest predicted probability of good performance in a new population. In conclusion, IPD meta-analysis allows a prognostic model to be externally validated in multiple settings, and enables recalibration strategies to be compared and ranked to decide on the least aggressive recalibration strategy to achieve acceptable external model performance without discarding existing model information.

Update and, internal and temporal-validation of the FRANCE-2 and ACC-TAVI early-mortality prediction models for Transcatheter Aortic Valve Implantation (TAVI) using data from the Netherlands heart registration (NHR)

Background

The predictive performance of the models FRANCE-2 and ACC-TAVI for early-mortality after Transcatheter Aortic Valve Implantation (TAVI) can decline over time and can be enhanced by updating them on new populations. We aim to update and internally and temporally validate these models using a recent TAVI-cohort from the Netherlands Heart Registration (NHR).

Methods

We used data of TAVI-patients treated in 2013-2017. For each original-model, the best update-method (model-intercept, model-recalibration, or model-revision) was selected by a closed-testing procedure. We internally validated both updated models with 1000 bootstrap samples. We also updated the models on the 2013-2016 dataset and temporally validated them on the 2017-dataset. Performance measures were the Area-Under ROC-curve (AU-ROC), Brier-score, and calibration graphs.

Results

We included 6177 TAVI-patients, with 4.5% observed early-mortality. The selected update-method for FRANCE-2 was model-intercept-update. Internal validation showed an AU-ROC of 0.63 (95%CI 0.62-0.66) and Brier-score of 0.04 (0.04-0.05). Calibration graphs show that it overestimates early-mortality. In temporal-validation, the AU-ROC was 0.61 (0.53-0.67).The selected update-method for ACC-TAVI was model-revision. In internal-validation, the AU-ROC was 0.63 (0.63-0.66) and Brier-score was 0.04 (0.04-0.05). The updated ACC-TAVI calibrates well up to a probability of 20%, and subsequently underestimates early-mortality. In temporal-validation the AU-ROC was 0.65 (0.58-0.72).

Conclusion

Internal-validation of the updated models FRANCE-2 and ACC-TAVI with data from the NHR demonstrated improved performance, which was better than in external-validation studies and comparable to the original studies. In temporal-validation, ACC-TAVI outperformed FRANCE-2 because it suffered less from changes over time.

Disparity in clinical outcomes after cardiac surgery between private and public (NHS) payers in England

BACKGROUND

There is little known about how payer status impacts clinical outcomes in a universal single-payer system such as the UK National Health Service (NHS). The aim of this study was to evaluate the relationship between payer status (private or public) and clinical outcomes following cardiac surgery from NHS providers in England.

METHODS

The National Adult Cardiac Surgery Audit (NACSA) registry was interrogated for patients who underwent adult cardiac surgery in England from 2009 to 2018. Information on socioeconomic status were provided by linkage with the Iteration of the English Indices of Deprivation (IoD). The primary outcome was in-hospital mortality. Secondary outcomes included incidence of in-hospital postoperative cerebrovascular accident (CVA), renal dialysis, sternal wound infection, and re-exploration. To assess whether payer status was an independent predictor of in-hospital mortality, binomial generalized linear mixed models (GLMM) were fitted along with 17 items forming the EuroSCORE and the IoD domains.

FINDINGS

The final sample consisted of 280,209 patients who underwent surgery in 31 NHS hospitals in England from 2009 to 2018. Of them, 5,967 (2.1%) and 274,242 (97.9%) were private and NHS payers respectively. Private payer status was associated with a lower risk of in-hospital mortality (OR 0.79; 95%CI 0.65 - 0.97;P = 0.026), CVA (OR 0.77; 95%CI 0.60 - 0.99; P = 0.039), need for re-exploration (OR 0.84; 95%CI 0.72 - 0.97; P = 0.017) and with non-significant lower risk of dialysis (OR 0.84; 95%CI 0.69 - 1.02; P = 0.074). Private payer status was found to be independently associated with lower risk of in-hospital mortality in the elective subgroup (OR 0.76; 95%CI 0.61 - 0.96; P = 0.020) but not in the non-elective subgroup (OR 1.01; 95%CI 0.64 - 1.58; P = 0.976).

INTERPRETATION

In conclusion, using a national database, we have found evidence of significant beneficial effect of payer status on hospital outcomes following cardiac surgery in favour of private payers regardless their socioeconomic factors.

Most deaths in low-risk cardiac surgery could be avoidable

It is observed that death rates in cardiac surgery has decreased, however, root causes that behave like triggers of potentially avoidable deaths (AD), especially in low-risk patients (less bias) are often unknown and underexplored, Phase of Care Mortality Analysis (POCMA) can be a valuable tool to identify seminal events (SE), providing valuable information where it is possible to make improvements in the quality and safety of future procedures. Our results show that in São Paul State, only one third of AD in low-risk cardiac surgery was related to specific surgical problems. After a revisited analysis, 75% of deaths could have been avoided, which in the pre-operative phase, the SE was related judgment, patient evaluation and preparation. In the intra-operative phase, most occurrences could have been avoided if other surgical technique had been used. Sepsis was responsible for 75% of AD in the intensive care unit. In the ward phase, the recognition/management of clinical decompensations and sepsis were the contributing factors. Logistic regression model identified age, previous coronary stent implantation, coronary artery bypass grafting + heart valve surgery, ≥ 2 combined heart valve surgery and hospital-acquired infection as independent predictors of AD.

Continual updating and monitoring of clinical prediction models: time for dynamic prediction systems?

Clinical prediction models (CPMs) have become fundamental for risk stratification across healthcare. The CPM pipeline (development, validation, deployment, and impact assessment) is commonly viewed as a one-time activity, with model updating rarely considered and done in a somewhat ad hoc manner. This fails to address the fact that the performance of a CPM worsens over time as natural changes in populations and care pathways occur. CPMs need constant surveillance to maintain adequate predictive performance. Rather than reactively updating a developed CPM once evidence of deteriorated performance accumulates, it is possible to proactively adapt CPMs whenever new data becomes available. Approaches for validation then need to be changed accordingly, making validation a continuous rather than a discrete effort. As such, "living" (dynamic) CPMs represent a paradigm shift, where the analytical methods dynamically generate updated versions of a model through time; one then needs to validate the system rather than each subsequent model revision.

The Impact of Initiation of an Intensivist-Led Patient Management Protocol on Outcomes After Cardiac Surgery

OBJECTIVES

Prolonged intensive care unit stay after cardiac surgery is associated with high mortality. The aim of this study was to evaluate the impact of the introduction of a quality improvement program under the supervision of an intensivist on the long-term mortality of high-risk patients with prolonged intensive care unit (ICU) stay after cardiac surgery.

DESIGN

Retrospective study of prospectively collected data.

SETTING

Cardiac surgery ICU.

PARTICIPANTS

A total of 7,549 patients after cardiac surgery.

INTERVENTIONS

Patients were divided into two periods: 2004 to 2007, before introducing the quality improvement program (3,315 patients), and 2009 to 2014, after introduction of the program (4,234 patients). In the period from 2004 to 2007, patients were divided into group I (ICU stay ≥ seven days), which included 242 patients, and group III (ICU stay < seven days), which included 3,073 patients. Also, in the period from 2009 to 2014 patients, were divided into group II (ICU stay ≥seven days), which included 326 patients, and group IV (ICU stay < seven days), which included 3,908 patients. Patient outcomes were compared. Follow-up was five years for each group.

MEASUREMENTS AND MAIN RESULTS

The European System for Cardiac Operative Risk Evaluation did not differ significantly among the groups. When comparing between group I and group II, 30-day mortality decreased significantly from 24.8% to 16.6%, six-month mortality from 27.3% to 19.3%, one-year mortality from 42.1% to 32.2%, 3-year mortality from 54.5% to 43.3%, and 5-year mortality from 61.2% to 51.8%. In comparing between group III and group IV, the authors did not observe a statistically significant decrease of mortality.

CONCLUSIONS

Intensivist-led patient management protocol is associated with decreased long-term mortality in high-risk patients with a prolonged ICU stay.

Detection of calibration drift in clinical prediction models to inform model updating

Model calibration, critical to the success and safety of clinical prediction models, deteriorates over time in response to the dynamic nature of clinical environments. To support informed, data-driven model updating strategies, we present and evaluate a calibration drift detection system. Methods are developed for maintaining dynamic calibration curves with optimized online stochastic gradient descent and for detecting increasing miscalibration with adaptive sliding windows. These methods are generalizable to support diverse prediction models developed using a variety of learning algorithms and customizable to address the unique needs of clinical use cases. In both simulation and case studies, our system accurately detected calibration drift. When drift is detected, our system further provides actionable alerts by including information on a window of recent data that may be appropriate for model updating. Simulations showed these windows were primarily composed of data accruing after drift onset, supporting the potential utility of the windows for model updating. By promoting model updating as calibration deteriorates rather than on pre-determined schedules, implementations of our drift detection system may minimize interim periods of insufficient model accuracy and focus analytic resources on those models most in need of attention.

External validation of the improving partial risk adjustment in surgery (PRAIS-2) model for 30-day mortality after paediatric cardiac surgery

OBJECTIVE

Independent temporal external validation of the improving partial risk adjustment in surgery model (PRAIS-2) to predict 30-day mortality in patients undergoing paediatric cardiac surgery.

DESIGN

Retrospective analysis of prospectively collected data.

SETTING

Paediatric cardiac surgery.

INTERVENTION

PRAIS-2 validation was carried out using a two temporally different single centre (Bristol, UK) cohorts: Cohort 1 surgery undertaken from April 2004 to March 2009 and Cohort 2 from April 2015 to July 2019. For each subject PRAIS-2 score was calculated according to the original formula.

PARTICIPANTS

A total of 1352 (2004-2009) and 1197 (2015-2019) paediatric cardiac surgical procedures were included in the Cohort 1 and Cohort 2, respectively (median age at the procedure 6.3 and 7.1 months).

PRIMARY AND SECONDARY OUTCOME MEASURES

PRAIS-2 performance was assessed in terms of discrimination by means of ROC (receiver operating characteristic) curve analysis and calibration by using the calibration belt method.

RESULTS

PRAIS-2 score showed excellent discrimination for both cohorts (AUC 0.72 (95%CI: 0.65 to 0.80) and 0.88 (95%CI: 0.82 to 0.93), respectively). While PRAIS-2 was only marginally calibrated in Cohort 1, with a tendency to underestimate risk in lowrisk and overestimate risk in high risk procedures (P-value = 0.033), validation in Cohort 2 showed good calibration with the 95% confidence belt containing the bisector for predicted mortality (P-value = 0.143). We also observed good prediction accuracy in the non-elective procedures (N = 483;AUC 0.78 (95%CI 0.68 to 0.87); Calibration belt containing the bisector (P-value=0.589).

CONCLUSIONS

In a single centre UK-based cohort, PRAIS-2 showed excellent discrimination and calibration in predicting 30-day mortality in paediatric cardiac surgery including in those undergoing non-elective procedures. Our results support a wider adoption of PRAIS-2 score in the clinical practice.

External validation of six existing multivariable clinical prediction models for short-term mortality in patients undergoing lung resection

 

OBJECTIVES

National guidelines advocate the use of clinical prediction models to estimate perioperative mortality for patients undergoing lung resection. Several models have been developed that may potentially be useful but contemporary external validation studies are lacking. The aim of this study was to validate existing models in a multicentre patient cohort.

METHODS

The Thoracoscore, Modified Thoracoscore, Eurolung, Modified Eurolung, European Society Objective Score and Brunelli models were validated using a database of 6600 patients who underwent lung resection between 2012 and 2018. Models were validated for in-hospital or 30-day mortality (depending on intended outcome of each model) and also for 90-day mortality. Model calibration (calibration intercept, calibration slope, observed to expected ratio and calibration plots) and discrimination (area under receiver operating characteristic curve) were assessed as measures of model performance.

RESULTS

Mean age was 66.8 years (±10.9 years) and 49.7% (n = 3281) of patients were male. In-hospital, 30-day, perioperative (in-hospital or 30-day) and 90-day mortality were 1.5% (n = 99), 1.4% (n = 93), 1.8% (n = 121) and 3.1% (n = 204), respectively. Model area under the receiver operating characteristic curves ranged from 0.67 to 0.73. Calibration was inadequate in five models and mortality was significantly overestimated in five models. No model was able to adequately predict 90-day mortality.

CONCLUSIONS

Five of the validated models were poorly calibrated and had inadequate discriminatory ability. The modified Eurolung model demonstrated adequate statistical performance but lacked clinical validity. Development of accurate models that can be used to estimate the contemporary risk of lung resection is required.

Validation and quality measurements for STS, EuroSCORE II and a regional risk model in Brazilian patients

OBJECTIVES

The objectives of this study were to describe a novel statewide registry for cardiac surgery in Brazil (REPLICCAR), to compare a regional risk model (SPScore) with EuroSCORE II and STS, and to understand where quality improvement and safety initiatives can be implemented.

METHODS

A total of 11 sites in the state of São Paulo, Brazil, formed an online registry platform to capture information on risk factors and outcomes after cardiac surgery procedures for all consecutive patients. EuroSCORE II and STS values were calculated for each patient. An SPScore model was designed and compared with EuroSCORE II and STS to predict 30-day outcomes: death, reoperation, readmission, and any morbidity.

RESULTS

A total of 5222 patients were enrolled in this study between November 2013 and December 2017. The observed 30-day mortality rate was 7.6%. Most patients were older, overweight, and classified as New York Heart Association (NYHA) functional class III; 14.5% of the patient population had a positive diagnosis of rheumatic heart disease, 10.9% had insulin-dependent diabetes, and 19 individuals had a positive diagnosis of Chagas disease. When evaluating the prediction performance, we found that SPScore outperformed EuroSCORE II and STS in the prediction of mortality (0.90 vs. 0.76 and 0.77), reoperation (0.84 vs. 0.60 and 0.56), readmission (0.84 vs. 0.55 and 0.51), and any morbidity (0.80 vs. 0.65 and 0.64), respectively (p<0.001).

CONCLUSIONS

The REPLICCAR registry might stimulate the creation of other cardiac surgery registries in developing countries, ultimately improving the regional quality of care provided to patients.

Risk modeling in transcatheter aortic valve replacement remains unsolved: an external validation study in 2946 German patients

Background

Surgical risk prediction models are routinely used to guide decision-making for transcatheter aortic valve replacement (TAVR). New and updated TAVR-specific models have been developed to improve risk stratification; however, the best option remains unknown.

Objective

To perform a comparative validation study of six risk models for the prediction of 30-day mortality in TAVR

Methods and results

A total of 2946 patients undergoing transfemoral (TF, n = 2625) or transapical (TA, n = 321) TAVR from 2008 to 2018 from the German Rhine Transregio Aortic Diseases cohort were included. Six surgical and TAVR-specific risk scoring models (LogES I, ES II, STS PROM, FRANCE-2, OBSERVANT, GAVS-II) were evaluated for the prediction of 30-day mortality. Observed 30-day mortality was 3.7% (TF 3.2%; TA 7.5%), mean 30-day mortality risk prediction varied from 5.8 ± 5.0% (OBSERVANT) to 23.4 ± 15.9% (LogES I). Discrimination performance (ROC analysis, c-indices) ranged from 0.60 (OBSERVANT) to 0.67 (STS PROM), without significant differences between models, between TF or TA approach or over time. STS PROM discriminated numerically best in TF TAVR (c-index 0.66; range of c-indices 0.60 to 0.66); performance was very similar in TA TAVR (LogES I, ES II, FRANCE-2 and GAVS-II all with c-index 0.67). Regarding calibration, all risk scoring models—especially LogES I—overestimated mortality risk, especially in high-risk patients.

Conclusions

Surgical as well as TAVR-specific risk scoring models showed mediocre performance in prediction of 30-day mortality risk for TAVR in the German Rhine Transregio Aortic Diseases cohort. Development of new or updated risk models is necessary to improve risk stratification.

Graphic abstract

Electronic supplementary material

The online version of this article (10.1007/s00392-020-01731-9) contains supplementary material, which is available to authorized users.

United States national trends in comorbidity and outcomes of adult cardiac surgery patients

BACKGROUND

Comorbidity profiles of cardiac surgery patients are known to have changed over time, but modern national trends in these comorbidities and outcomes are not described. This study describes comorbidity trends over time for common adult cardiac surgery procedures.

METHODS

A retrospective, cross-sectional analysis of the National Inpatient Sample was conducted for years 2005-2014. Hospitalizations with coronary artery bypass grafting (CABG), aortic valve replacement (AVR), and mitral valve repair/replacement (MVRR), as well as combined CABG/valve operations, were identified by ICD-9 procedure codes. Comorbidities were defined based on ICD-9 codes to discriminate between comorbidities and complications. Surgical volume, patient age, in-hospital mortality, and length of stay trends over time were evaluated by linear regression.

RESULTS

Incidence increased for AVR, MVRR, and CABG + AVR and declined for CABG and CABG + MVRR (P < .001). The mean number of comorbidities across all surgeries increased from 1.4 to 1.9 (P < .001). Length of stay declined for AVR, CABG + AVR, and CABG + MVRR (P < .001) with an overall decline from 10.1 to 9.7 days (P = .003). In-hospital mortality decreased in all categories over time (P < .001). Overall, in-hospital mortality decreased from 2.9% to 2.3% (P < .001).

CONCLUSIONS

Despite increasing comorbidity in cardiac surgery, operations are being conducted with fewer in-hospital mortalities across all types of surgery and decreasing length of stay for most types of surgery, which should inform the frequency of risk model updates and raise questions of the applicability of earlier studies in cardiac surgery to the modern population.

A Novel Patient-Specific Model for Predicting Severe Oliguria; Development and Comparison With Kidney Disease: Improving Global Outcomes Acute Kidney Injury Classification

OBJECTIVES

The Kidney Disease: Improving Global Outcomes urine output criteria for acute kidney injury lack specificity for identifying patients at risk of adverse renal outcomes. The objective was to develop a model that analyses hourly urine output values in real time to identify those at risk of developing severe oliguria.

DESIGN

This was a retrospective cohort study utilizing prospectively collected data.

SETTING

A cardiac ICU in the United Kingdom.

PATIENTS

Patients undergoing cardiac surgery between January 2013 and November 2017.

INTERVENTIONS

None.

MEASUREMENT AND MAIN RESULTS

Patients were randomly assigned to development (n = 981) and validation (n = 2,389) datasets. A patient-specific, dynamic Bayesian model was developed to predict future urine output on an hourly basis. Model discrimination and calibration for predicting severe oliguria (< 0.3 mL/kg/hr for 6 hr) occurring within the next 12 hours were tested in the validation dataset at multiple time points. Patients with a high risk of severe oliguria (p > 0.8) were identified and their outcomes were compared with those for low-risk patients and for patients who met the Kidney Disease: Improving Global Outcomes urine output criterion for acute kidney injury. Model discrimination was excellent at all time points (area under the curve > 0.9 for all). Calibration of the model's predictions was also excellent. After adjustment using multivariable logistic regression, patients in the high-risk group were more likely to require renal replacement therapy (odds ratio, 10.4; 95% CI, 5.9-18.1), suffer prolonged hospital stay (odds ratio, 4.4; 95% CI, 3.0-6.4), and die in hospital (odds ratio, 6.4; 95% CI, 2.8-14.0) (p < 0.001 for all). Outcomes for those identified as high risk by the model were significantly worse than for patients who met the Kidney Disease: Improving Global Outcomes urine output criterion.

CONCLUSIONS

This novel, patient-specific model identifies patients at increased risk of severe oliguria. Classification according to model predictions outperformed the Kidney Disease: Improving Global Outcomes urine output criterion. As the new model identifies patients at risk before severe oliguria develops it could potentially facilitate intervention to improve patient outcomes.

Predicting 30-Day Hospital Readmission Risk in a National Cohort of Patients with Cirrhosis

BACKGROUND

Early hospital readmission for patients with cirrhosis continues to challenge the healthcare system. Risk stratification may help tailor resources, but existing models were designed using small, single-institution cohorts or had modest performance.

AIMS

We leveraged a large clinical database from the Department of Veterans Affairs (VA) to design a readmission risk model for patients hospitalized with cirrhosis. Additionally, we analyzed potentially modifiable or unexplored readmission risk factors.

METHODS

A national VA retrospective cohort of patients with a history of cirrhosis hospitalized for any reason from January 1, 2006, to November 30, 2013, was developed from 123 centers. Using 174 candidate variables within demographics, laboratory results, vital signs, medications, diagnoses and procedures, and healthcare utilization, we built a 47-variable penalized logistic regression model with the outcome of all-cause 30-day readmission. We excluded patients who left against medical advice, transferred to a non-VA facility, or if the hospital length of stay was greater than 30 days. We evaluated calibration and discrimination across variable volume and compared the performance to recalibrated preexisting risk models for readmission.

RESULTS

We analyzed 67,749 patients and 179,298 index hospitalizations. The 30-day readmission rate was 23%. Ascites was the most common cirrhosis-related cause of index hospitalization and readmission. The AUC of the model was 0.670 compared to existing models (0.649, 0.566, 0.577). The Brier score of 0.165 showed good calibration.

CONCLUSION

Our model achieved better discrimination and calibration compared to existing models, even after local recalibration. Assessment of calibration by variable parsimony revealed performance improvements for increasing variable inclusion well beyond those detectable for discrimination.