Validation of the Simplified Acute Physiology Score 3 scoring system in a Korean intensive care unit

Development and validation of novel simple prognostic model for predicting mortality in Korean intensive care units using national insurance claims data

BACKGROUND/AIMS

Intensive care unit (ICU) quality is largely determined by the mortality rate. Therefore, we aimed to develop and validate a novel prognostic model for predicting mortality in Korean ICUs, using national insurance claims data.

METHODS

Data were obtained from the health insurance claims database maintained by the Health Insurance Review and Assessment Service of South Korea. From patients who underwent the third ICU adequacy evaluation, 42,489 cases were enrolled and randomly divided into the derivation and validation cohorts. Using the models derived from the derivation cohort, we analyzed whether they accurately predicted death in the validation cohort. The models were verified using data from one general and two tertiary hospitals.

RESULTS

Two severity correction models were created from the derivation cohort data, by applying variables selected through statistical analysis, through clinical consensus, and from performing multiple logistic regression analysis. Model 1 included six categorical variables (age, sex, Charlson comorbidity index, ventilator use, hemodialysis or continuous renal replacement therapy, and vasopressor use). Model 2 additionally included presence/absence of ICU specialists and nursing grades. In external validation, the performance of models 1 and 2 for predicting in-hospital and ICU mortality was not inferior to that of pre-existing scoring systems.

CONCLUSION

The novel and simple models could predict in-hospital and ICU mortality and were not inferior compared to the pre-existing scoring systems.

Impact of age in critically Ill infected patients: a post-hoc analysis of the INFAUCI study

PURPOSE

Older patients are the fastest expanding subgroup of intensive care units (ICU) and are particularly susceptible to bacterial infections and sepsis. The aim of this study was to address the epidemiology and the main determinants of outcome of infection in old and very old patients admitted to ICU.

METHODS

We performed a post hoc analysis of all infected patients admitted to ICU enrolled in a 1-year prospective, observational, multipurpose study. Patients aged < 65, 65-74 and ≥ 75 years were compared.

RESULTS

Of the 1652 patients included, 50% were older than 65 years. There were no significant differences between young, old and very old patients in either ICU, hospital length of stay, or nosocomial infection. All-cause mortality was significantly higher in participants aged ≥ 75. Increased Gram-negative microorganisms' isolates occurred in > 65 years (25% versus 31%; p = 0.034). Multidrug-resistant (MDR) microorganisms were directly associated to inappropriate empiric antibiotic therapy (OR 4.73; 95% CI 2.99-7.47) and inversely associated with community-acquired infection (OR 0.39; 95% CI 0.19-0.83). Age (65-74 years: OR 1.10; 95% CI 0.64-1.90 and ≥ 75 years: OR 1.52; 95% CI 0.89-2.59) and sepsis severity (sepsis: OR 0.67; 95% CI 0.18-2.46; severe sepsis: OR 1.17; 95% CI 0.40-3.44; septic shock: OR 0.77; 95% CI 0.27-2.24) were not associated to MDR bacteria.

CONCLUSION

Patients > 65 years accounted for 50% of infected patients admitted to an ICU. ICU and hospital length of stay, and nosocomial infection did not increase with age. Age did predispose to increased risk for infection by Gram-negatives. These findings may optimize strategies for infection management in older patients.

Performance of Simplified Acute Physiology Score 3 In Predicting Hospital Mortality In Emergency Intensive Care Unit

Background:

Since the 1980s, severity of illness scoring systems has gained increasing popularity in Intensive Care Units (ICUs). Physicians used them for predicting mortality and assessing illness severity in clinical trials. The objective of this study was to assess the performance of Simplified Acute Physiology Score 3 (SAPS 3) and its customized equation for Australasia (Australasia SAPS 3, SAPS 3 [AUS]) in predicting clinical prognosis and hospital mortality in emergency ICU (EICU).

Methods:

A retrospective analysis of the EICU including 463 patients was conducted between January 2013 and December 2015 in the EICU of Peking University Third Hospital. The worst physiological data of enrolled patients were collected within 24 h after admission to calculate SAPS 3 score and predicted mortality by regression equation. Discrimination between survivals and deaths was assessed by the area under the receiver operator characteristic curve (AUC). Calibration was evaluated by Hosmer-Lemeshow goodness-of-fit test through calculating the ratio of observed-to-expected numbers of deaths which is known as the standardized mortality ratio (SMR).

Results:

A total of 463 patients were enrolled in the study, and the observed hospital mortality was 26.1% (121/463). The patients enrolled were divided into survivors and nonsurvivors. Age, SAPS 3 score, Acute Physiology and Chronic Health Evaluation Score II (APACHE II), and predicted mortality were significantly higher in nonsurvivors than survivors (P < 0.05 or P < 0.01). The AUC (95% confidence intervals [CIs]) for SAPS 3 score was 0.836 (0.796–0.876). The maximum of Youden's index, cutoff, sensitivity, and specificity of SAPS 3 score were 0.526%, 70.5 points, 66.9%, and 85.7%, respectively. The Hosmer-Lemeshow goodness-of-fit test for SAPS 3 demonstrated a Chi-square test score of 10.25, P = 0.33, SMR (95% CI) = 0.63 (0.52–0.76). The Hosmer-Lemeshow goodness-of-fit test for SAPS 3 (AUS) demonstrated a Chi-square test score of 9.55, P = 0.38, SMR (95% CI) = 0.68 (0.57–0.81). Univariate and multivariate analyses were conducted for biochemical variables that were probably correlated to prognosis. Eventually, blood urea nitrogen (BUN), albumin, lactate and free triiodothyronine (FT3) were selected as independent risk factors for predicting prognosis.

Conclusions:

The SAPS 3 score system exhibited satisfactory performance even superior to APACHE II in discrimination. In predicting hospital mortality, SAPS 3 did not exhibit good calibration and overestimated hospital mortality, which demonstrated that SAPS 3 needs improvement in the future.

Performance of three prognostic models in patients with cancer in need of intensive care in a medical center in China

Objective

The aim of this study was to evaluate the performance of Acute Physiology and Chronic Health Evaluation II (APACHE II), Simplified Acute Physiology Score 3 (SAPS 3), and Acute Physiology and Chronic Health Evaluation IV (APACHE IV) in patients with cancer admitted to intensive care unit (ICU) in a single medical center in China.

Materials and Methods

This is a retrospective observational cohort study including nine hundred and eighty one consecutive patients over a 2-year period.

Results

The hospital mortality rate was 4.5%. When all 981 patients were evaluated, the area under the receiver operating characteristic curve (AUROC, 95% Confidential Intervals) of the three models in predicting hospital mortality were 0.948 (0.914–0.982), 0.863 (0.804–0.923), and 0.873 (0.813–0.934) for SAPS 3, APACHE II and APACHE IV respectively. The p values of Hosmer-Lemeshow statistics for the models were 0.759, 0.900 and 0.878 for SAPS 3, APACHE II and APACHE IV respectively. However, SAPS 3 and APACHE IV underestimated the in-hospital mortality with standardized mortality ratio (SMR) of 1.5 and 1.17 respectively, while APACHE II overestimated the in-hospital mortality with SMR of 0.72. Further analysis showed that discrimination power was better with SAPS 3 than with APACHE II and APACHE IV whether for emergency surgical and medical patients (AUROC of 0.912 vs 0.866 and 0.857) or for scheduled surgical patients (AUROC of 0.945 vs 0.834 and 0.851). Calibration was good for all models (all p > 0.05) whether for scheduled surgical patients or emergency surgical and medical patients. However, in terms of SMR, SAPS 3 was both accurate in predicting the in-hospital mortality for emergency surgical and medical patients and for scheduled surgical patients, while APACHE IV and APACHE II were not.

Conclusion

In this cohort, we found that APACHE II, APACHE IV and SAPS 3 models had good discrimination and calibration ability in predicting in-hospital mortality of critically ill patients with cancer in need of intensive care. Of these three severity scores, SAPS 3 was superior to APACHE II and APACHE IV, whether in terms of discrimination and calibration power, or standardized mortality ratios.

Evaluation of Simplified Acute Physiology Score 3 performance: a systematic review of external validation studies

Introduction

Simplified Acute Physiology Score 3 (SAPS 3) was the first critical care prognostic model developed from worldwide data. We aimed to systematically review studies that assessed the prognostic performance of SAPS 3 general and customized models for predicting hospital mortality in adult patients admitted to the ICU.

Methods

Medline, Lilacs, Scielo and Google Scholar were searched to identify studies which assessed calibration and discrimination of general and customized SAPS 3 equations. Additionally, we decided to evaluate the correlation between trial size (number of included patients) and the Hosmer-Lemeshow (H-L) statistics value of the SAPS 3 models.

Results

A total of 28 studies were included. Of these, 11 studies (42.8%) did not find statistically significant mis-calibration for the SAPS 3 general equation. There was a positive correlation between number of included patients and higher H-L statistics, that is, a statistically significant mis-calibration of the model (r = 0.747, P <0.001). Customized equations for major geographic regions did not have statistically significant departures from perfect calibration in 9 of 19 studies. Five studies (17.9%) developed a regional customization and in all of them this new model was not statistically different from a perfect calibration for their populations. Discrimination was at least very good in 24 studies (85.7%).

Conclusions

Statistically significant departure from perfect calibration for the SAPS 3 general equation was common in validation studies and was correlated with larger studies, as should be expected, since H-L statistics (both C and H) are strongly dependent on sample size This finding was also present when major geographic customized equations were evaluated. Local customizations, on the other hand, improved SAPS 3 calibration. Discrimination was almost always very good or excellent, which gives excellent perspectives for local customization when a precise local estimate is needed.

Validation of SAPS3 admission score and its customization for use in Korean intensive care unit patients: a prospective multicentre study

BACKGROUND AND OBJECTIVE

To externally validate the simplified acute physiology score 3 (SAPS3) and to customize it for use in Korean intensive care unit (ICU) patients.

METHODS

This is a prospective multicentre cohort study involving 22 ICUs from 15 centres throughout Korea. The study population comprised patients who were consecutively admitted to participating ICUs from 1 July 2010 to 31 January 2011.

RESULTS

A total of 4617 patients were enrolled. ICU mortality was 14.3%, and hospital mortality was 20.6%. The patients were randomly assigned into one of two cohorts: a development (n = 2309) or validation (n = 2308) cohort. In the development cohort, the general SAPS3 had good discrimination (area under the receiver operating characteristics curve = 0.829), but poor calibration (Hosmer-Lemeshow goodness-of-fit test H = 123.06, P < 0.001, C = 118.45, P < 0.001). The Australasia SAPS3 did not improve calibration (H = 73.53, P < 0.001, C = 70.52, P < 0.001). Customization was achieved by altering the logit of the original SAPS3 equation. The new equation for Korean ICU patients was validated in the validation cohort, and demonstrated both good discrimination (area under the receiver operating characteristics curve = 0.835) and good calibration (H = 4.61, P = 0.799, C = 5.67, P = 0.684).

CONCLUSIONS

General and regional Australasia SAPS3 admission scores showed poor calibration for use in Korean ICU patients, but the prognostic power of the SAPS3 was significantly improved by customization. Prediction models should be customized before being used to predict mortality in different regions of the world.

Performance of the SAPS 3 admission score as a predictor of ICU mortality in a Philippine private tertiary medical center intensive care unit

Background

This study aimed to assess the performance of the Simplified Acute Physiology Score 3 (SAPS 3) as a predictor of ICU mortality in critically ill patients of different case mixes admitted to an intensive care unit.

Methods

This retrospective cohort study was performed from January 2011 to August 2013 in the intensive care unit of a private tertiary referral center in the Philippines. Predicted ICU mortality was calculated using the SAPS 3 global model. Observed versus predicted mortality rates were compared, and the standardized mortality ratio (SMR) was calculated. The discrimination and calibration characteristics of the SAPS 3 system to predict ICU mortality were assessed.

Results

A total of 2,426 patients were included. The observed ICU mortality was 277 (11.42%). The SAPS 3 global model had fair to good discrimination with an area under the receiver operating characteristic curve of 0.80 (CI 0.78–0.81). Good calibration was seen with the Hosmer-Lemeshow goodness of fit at Ĉ = 11.51 (p = 0.175). Standardized mortality ratio was 0.36 (0.26–0.81).

Conclusion

The global SAPS 3 prediction model showed fair to good discrimination and good calibration in predicting mortality in our intensive care unit. Different levels of discrimination and calibration across the different subgroups analyzed suggest that overall ICU performance seemed to be affected by case mix variations.

Electronic supplementary material

The online version of this article (doi:10.1186/2052-0492-2-29) contains supplementary material, which is available to authorized users.

Usefulness of N-terminal pro-B-type natriuretic peptide in patients admitted to the intensive care unit: a multicenter prospective observational study

Background

The role of N-terminal pro-B-type natriuretic peptide (NT-pro-BNP) as a prognostic factor in patients admitted to the intensive care unit (ICU) is not yet fully established. We aimed to determine whether NT-pro-BNP is predictive of ICU mortality in a multicenter cohort of critically ill patients.

Methods

A total of 1440 patients admitted to 22 ICUs (medical, 14; surgical, six; multidisciplinary, two) in 15 tertiary or university-affiliated hospitals between July 2010 and January 2011 were assessed. Patient data, including NT-pro-BNP levels and Simplified Acute Physiology Score (SAPS) 3 scores, were recorded prospectively in a web-based database.

Results

The median age was 64 years (range, 53–73 years), and 906 (62.9%) patients were male. The median NT-pro-BNP level was 341 pg/mL (104–1,637 pg/mL), and the median SAPS 3 score was 57 (range, 47–69). The ICU mortality rate was 18.9%, and hospital mortality was 24.5%. Hospital survivors showed significantly lower NT-pro-BNP values than nonsurvivors (245 pg/mL [range, 82–1,053 pg/mL] vs. 875 pg/mL [241–5,000 pg/mL], respectively; p < 0.001). In prediction of hospital mortality, the area under the curve (AUC) for NT-pro-BNP was 0.67 (95% confidence interval [CI], 0.64–0.70) and SAPS 3 score was 0.83 (95% CI, 0.81–0.85). AUC increment by adding NT-pro-BNP is minimal and likely no different to SAPS 3 alone.

Conclusions

The NT-pro-BNP level was more elevated in nonsurvivors in a multicenter cohort of critically ill patients. However, there was little additional prognostic power when adding NT-pro-BNP to SAPS 3 score.

External validation of the Acute Physiology and Chronic Health Evaluation II in Korean intensive care units

PURPOSE

This study was designed to validate the usefulness of the Acute Physiology and Chronic Health Evaluation (APACHE) II for predicting hospital mortality of critically ill Korean patients.

MATERIALS AND METHODS

We analyzed data on 826 patients who had been admitted to nine intensive care units and were included in the Fever and Antipyretics in Critical Illness Evaluation study cohort.

RESULTS

Among the patients enrolled, 62% (512/826) were medical and 38% (314/826) were surgical patients. The median APACHE II score was 17 (11 to 23 interquartile range), and the hospital mortality rate was 19.5%. Age, underlying diseases, medical patients, mechanical ventilation, and renal replacement therapy were independently associated with hospital mortality. The calibration of APACHE II was poor (H=57.54, p<0.0001; C=55.99, p<0.0001), and the discrimination was modest [area under the receiver operating characteristic (aROC)=0.729]. Calibration was poor for both medical and surgical patients (H=63.56, p<0.0001; C=73.83, p<0.0001, and H=33.92, p<0.0001; C=33.34, p=0.0001, respectively), while discrimination was poor for medical patients (aROC=0.651) and modest for surgical patients (aROC=0.704). At the predicted risk of 50%, APACHE II had a sensitivity of 36.6% and a specificity of 87.4% for hospital mortality.

CONCLUSION

For Koreans, the APACHE II exhibits poor calibration and modest discrimination for hospital mortality. Therefore, a new model is needed to accurately predict mortality in critically ill Korean patients.