Performance of standard severity scoring systems for outcome prediction in patients admitted to a respiratory intensive care unit in North India

AIMS65, Glasgow-Blatchford bleeding score and modified Glasgow-Blatchford bleeding score in predicting outcomes of upper gastrointestinal bleeding: An accuracy and calibration study

BACKGROUND

Albumin, international normalized ratio (INR), mental status, systolic blood pressure, age >65 years (AIMS65), Glasgow-Blatchford bleeding score (GBS) and modified GBS (mGBS) are three pre-endoscopy scoring systems used in the risk stratification of upper gastrointestinal bleeding (UGIB). The utility of such scoring systems in a population is estimated by their accuracy and calibration in the population. We aimed at validating and comparing the accuracy of the three scoring systems in predicting clinical outcomes including in-hospital mortality, need for blood transfusion, endoscopic treatment and rebleeding risk.

METHOD

We conducted a single-center, retrospective cohort study on patients with UGIB at a tertiary care center in India over 12 months. Clinical and laboratory data was collected from all patients admitted with UGIB. All patients were risk stratified using AIMS65, GBS and mGBS. The clinical outcome examined were: in-hospital mortality, requirement of blood transfusion, need for endoscopic treatment and rebleeding during hospital stay. The area under receiver-operating curve (AUROC) was calculated to assess the performance and calibration curves (Hosmer-Lemeshow goodness of fit curve) were plotted to examine how accurately the model describes the data of all three scoring systems.

RESULTS

Total 260 patients were included in the study, of which 236 (90.8%) were males. As many as 144 (55.4%) patients required blood transfusion and 64 (30.8%) required endoscopic treatment. While the incidence of rebleeding was 7.7%, in hospital mortality was 15.4%. Of 208 who underwent endoscopy, the most common causes identified were varices (49%) and gastritis (18.2%), followed by ulcer (11%), Mallory-Weiss tear (8.1%), portal hypertensive gastropathy (6.7%), malignancy (4.8%) and esophageal candidiasis (1.9%). The median AIMS65 score was 1, GBS 7 and mGBS 6. The area under curve (AUROC) for AIMS65, GBS and mGBS was (0.77, 0.73,0.70), (0.75, 0.82,0.83), (0.56, 0.58,0.83), (0.81, 0.94,0.53) for in-hospital mortality, blood transfusion requirement, endoscopic treatment and rebleeding prediction, respectively.

CONCLUSION

GBS and mGBS are superior to AIMS65 in predicting the requirement of blood transfusion and rebleeding risk, whereas in-hospital mortality was better predicted by AIMS 65. Both scores performed poorly in predicting the need of endoscopic treatment. An AIMS65 of 0,1 and a GBS of ≤ 1 are not associated with significant adverse events. A poor calibration of the scores in our population points to the lack of generalizability of these scoring systems.

Towards unstructured mortality prediction with free-text clinical notes

Healthcare data continues to flourish yet a relatively small portion, mostly structured, is being utilized effectively for predicting clinical outcomes. The rich subjective information available in unstructured clinical notes can possibly facilitate higher discrimination but tends to be under-utilized in mortality prediction. This work attempts to assess the gain in performance when multiple notes that have been minimally preprocessed are used as an input for prediction. A hierarchical architecture consisting of both convolutional and recurrent layers is used to concurrently model the different notes compiled in an individual hospital stay. This approach is evaluated on predicting in-hospital mortality on the MIMIC-III dataset. On comparison to approaches utilizing structured data, it achieved higher metrics despite requiring less cleaning and preprocessing. This demonstrates the potential of unstructured data in enhancing mortality prediction and signifies the need to incorporate more raw unstructured data into current clinical prediction methods.

Performance of critical care prognostic scoring systems in low and middle-income countries: a systematic review

Background

Prognostic models—used in critical care medicine for mortality predictions, for benchmarking and for illness stratification in clinical trials—have been validated predominantly in high-income countries. These results may not be reproducible in low or middle-income countries (LMICs), not only because of different case-mix characteristics but also because of missing predictor variables. The study objective was to systematically review literature on the use of critical care prognostic models in LMICs and assess their ability to discriminate between survivors and non-survivors at hospital discharge of those admitted to intensive care units (ICUs), their calibration, their accuracy, and the manner in which missing values were handled.

Methods

The PubMed database was searched in March 2017 to identify research articles reporting the use and performance of prognostic models in the evaluation of mortality in ICUs in LMICs. Studies carried out in ICUs in high-income countries or paediatric ICUs and studies that evaluated disease-specific scoring systems, were limited to a specific disease or single prognostic factor, were published only as abstracts, editorials, letters and systematic and narrative reviews or were not in English were excluded.

Results

Of the 2233 studies retrieved, 473 were searched and 50 articles reporting 119 models were included. Five articles described the development and evaluation of new models, whereas 114 articles externally validated Acute Physiology and Chronic Health Evaluation, the Simplified Acute Physiology Score and Mortality Probability Models or versions thereof. Missing values were only described in 34% of studies; exclusion and or imputation by normal values were used. Discrimination, calibration and accuracy were reported in 94.0%, 72.4% and 25% respectively. Good discrimination and calibration were reported in 88.9% and 58.3% respectively. However, only 10 evaluations that reported excellent discrimination also reported good calibration. Generalisability of the findings was limited by variability of inclusion and exclusion criteria, unavailability of post-ICU outcomes and missing value handling.

Conclusions

Robust interpretations regarding the applicability of prognostic models are currently hampered by poor adherence to reporting guidelines, especially when reporting missing value handling. Performance of mortality risk prediction models in LMIC ICUs is at best moderate, especially with limitations in calibration. This necessitates continued efforts to develop and validate LMIC models with readily available prognostic variables, perhaps aided by medical registries.

Electronic supplementary material

The online version of this article (doi:10.1186/s13054-017-1930-8) contains supplementary material, which is available to authorized users.

A Review of Pediatric Critical Care in Resource-Limited Settings: A Look at Past, Present, and Future Directions

Fifteen years ago, United Nations world leaders defined millenium development goal 4 (MDG 4): to reduce under-5-year mortality rates by two-thirds by the year 2015. Unfortunately, only 27 of 138 developing countries are expected to achieve MDG 4. The majority of childhood deaths in these settings result from reversible causes, and developing effective pediatric emergency and critical care services could substantially reduce this mortality. The Ebola outbreak highlighted the fragility of health care systems in resource-limited settings and emphasized the urgent need for a paradigm shift in the global approach to healthcare delivery related to critical illness. This review provides an overview of pediatric critical care in resource-limited settings and outlines strategies to address challenges specific to these areas. Implementation of these tools has the potential to move us toward delivery of an adequate standard of critical care for all children globally, and ultimately decrease global child mortality in resource-limited settings.

Global health care of the critically ill in low-resource settings

The care of the critically ill patient in low-resource settings is challenging because of many factors, including limitations in the existing infrastructure, lack of disposables, and low numbers of trained healthcare workers. Although cost constraints in low-resource settings have traditionally caused critical care to be relegated to a low priority, ethical issues and the potential for mitigation of the lethal effects of often reversible acute conditions, such as sepsis and traumatic hemorrhage, argue for prudent deployment of critical care resources. Given these challenges, issues that require prioritization include timely and reliable delivery of evidence-based or generally accepted interventions to acutely ill patients before the development of organ failure, context-specific adaptation and evaluation of clinical evidence, and sustained investments in quality improvement and health systems strengthening. Specific examples include fluid resuscitation algorithms for patients with sepsis and reliable, low-cost, high-flow oxygen concentrators for patients with pneumonia. The lessons from new research on clinical management and sustainable education and quality improvement approaches will likely improve the care of critically ill patients worldwide.

Performance of the SAPS 3 admission score as a predictor of ICU mortality in a Philippine private tertiary medical center intensive care unit

Background

This study aimed to assess the performance of the Simplified Acute Physiology Score 3 (SAPS 3) as a predictor of ICU mortality in critically ill patients of different case mixes admitted to an intensive care unit.

Methods

This retrospective cohort study was performed from January 2011 to August 2013 in the intensive care unit of a private tertiary referral center in the Philippines. Predicted ICU mortality was calculated using the SAPS 3 global model. Observed versus predicted mortality rates were compared, and the standardized mortality ratio (SMR) was calculated. The discrimination and calibration characteristics of the SAPS 3 system to predict ICU mortality were assessed.

Results

A total of 2,426 patients were included. The observed ICU mortality was 277 (11.42%). The SAPS 3 global model had fair to good discrimination with an area under the receiver operating characteristic curve of 0.80 (CI 0.78–0.81). Good calibration was seen with the Hosmer-Lemeshow goodness of fit at Ĉ = 11.51 (p = 0.175). Standardized mortality ratio was 0.36 (0.26–0.81).

Conclusion

The global SAPS 3 prediction model showed fair to good discrimination and good calibration in predicting mortality in our intensive care unit. Different levels of discrimination and calibration across the different subgroups analyzed suggest that overall ICU performance seemed to be affected by case mix variations.

Electronic supplementary material

The online version of this article (doi:10.1186/2052-0492-2-29) contains supplementary material, which is available to authorized users.

Assessment of performance and utility of mortality prediction models in a single Indian mixed tertiary intensive care unit

OBJECTIVES

To assess the performance and utility of two mortality prediction models viz. Acute Physiology and Chronic Health Evaluation II (APACHE II) and Simplified Acute Physiology Score II (SAPS II) in a single Indian mixed tertiary intensive care unit (ICU). Secondary objectives were bench-marking and setting a base line for research.

MATERIALS AND METHODS

In this observational cohort, data needed for calculation of both scores were prospectively collected for all consecutive admissions to 28-bedded ICU in the year 2011. After excluding readmissions, discharges within 24 h and age <18 years, the records of 1543 patients were analyzed using appropriate statistical methods.

RESULTS

Both models overpredicted mortality in this cohort [standardized mortality ratio (SMR) 0.88 ± 0.05 and 0.95 ± 0.06 using APACHE II and SAPS II respectively]. Patterns of predicted mortality had strong association with true mortality (R (2) = 0.98 for APACHE II and R (2) = 0.99 for SAPS II). Both models performed poorly in formal Hosmer-Lemeshow goodness-of-fit testing (Chi-square = 12.8 (P = 0.03) for APACHE II, Chi-square = 26.6 (P = 0.001) for SAPS II) but showed good discrimination (area under receiver operating characteristic curve 0.86 ± 0.013 SE (P < 0.001) and 0.83 ± 0.013 SE (P < 0.001) for APACHE II and SAPS II, respectively). There were wide variations in SMRs calculated for subgroups based on International Classification of Disease, 10(th) edition (standard deviation ± 0.27 for APACHE II and 0.30 for SAPS II).

INTERPRETATION AND CONCLUSION

Lack of fit of data to the models and wide variation in SMRs in subgroups put a limitation on utility of these models as tools for assessing quality of care and comparing performances of different units without customization. Considering comparable performance and simplicity of use, efforts should be made to adapt SAPS II.

Prognostic performance of the Simplified Acute Physiology Score II in major Croatian hospitals: a prospective multicenter study

Aim

To perform an external validation of the original Simplified Acute Physiology Score II (SAPS II) system and to assess its performance in a selected group of patients in major Croatian hospitals.

Methods

A prospective, multicenter study was conducted in five university hospitals and one general hospital during a six-month period between November 1, 2007 and May 1, 2008. Standardized hospital mortality ratio (SMR) was calculated from the mean predicted mortality of all the 2756 patients and the actual mortality for the same group of patients. The validation of SAPS II was made using the area under receiver operating characteristic curve (AUC), 2 × 2 classification tables, and Hosmer-Lemeshow tests.

Results

The predicted mortality was as low as 14.6% due to a small proportion of medical patients and the SMR being 0.89 (95% confidence interval [CI], 0.78-0.98). The SAPS II system demonstrated a good discriminatory power as measured by the AUC (0.85; standard error [SE] = 0.012; 95% CI = 0.840-0.866; P < 0.001). This system significantly overestimated the actual mortality (Hosmer-Lemeshow goodness-of-fit H statistic: χ² = 584.4; P < 0.001 and C statistics: χ²₈ = 313.0; P < 0.001) in the group of patients included in the study.

Conclusion

The SAPS II had a good discrimination, but it significantly overestimated the observed mortality in comparison with the predicted mortality in this group of patients in Croatia. Therefore, caution is required when an evaluation is performed at the individual level.

Validation of the Simplified Acute Physiology Score 3 scoring system in a Korean intensive care unit

PURPOSE

The Simplified Acute Physiology Score (SAPS) 3 was recently proposed to reflect contemporary changes in intensive care practices. SAPS 3 features customized equations for the prediction of mortality in different geographic regions. However, the usefulness of SAPS 3 and its customized equation (Australasia SAPS 3) have never been externally validated in Korea. This study was designed to validate SAPS 3 and Australasia SAPS 3 for mortality prediction in Korea.

MATERIALS AND METHODS

A retrospective analysis of the prospective intensive care unit (ICU) registry was conducted in the medical ICU of Samsung Medical Center. Calibration and discrimination were determined by the Hosmer-Lemeshow test and area under the receiver operating characteristic (aROC) curve from 633 patients.

RESULTS

The mortalities (%) predicted by SAPS 3, Australasia SAPS 3, and SAPS II were 42 ± 28, 39 ± 27 and 37 ± 31, respectively. The calibration of SAPS II was poor (p = 0.003). SAPS 3 and Australasia SAPS 3 were appropriate (p > 0.05). The discriminative power of all models yielded aROC values less than 0.8.

CONCLUSION

In Korea, mortality rates predicted using general SAPS 3 and Australasia SAPS 3 exhibited good calibration and modest discrimination. However, Australasia SAPS 3 did not improve the mortality prediction. To better predict mortality in Korean ICUs, a new equation may be needed specifically for Korea.

Outcome prediction in critical care: the Mortality Probability Models

PURPOSE OF REVIEW

The comparison of morbidity, mortality, and length-of-stay outcomes in patients receiving critical care requires adjustment based on their presenting illness. These adjustments are made with severity-of-illness models. These models must be periodically updated to reflect current medical practices. This article will review the history of the Mortality Probability Model (MPM), discuss why and how it was recently updated, and outline examples of MPM use.

RECENT FINDINGS

All severity-of-illness models have limitations, especially if a unit's patient population becomes highly specialized. In these situations, customized models may provide better accuracy. The MPMs include those calculated at admission (MPM0) and additional models at 24, 48, and 72 h (MPM 24, MPM 48, and MPM 72). The model is now in its third iteration (MPM 0-III). Length of stay (LOS) and subgroup models have also been developed.

SUMMARY

Understanding appropriate application of models such as MPM is important as transparency in healthcare drives demand for severity-adjusted outcomes data.

Outcome prediction in critical care: the Simplified Acute Physiology Score models

PURPOSE OF REVIEW

Outcome prediction models measuring severity of illness of patients admitted to the intensive care unit should predict hospital mortality. This review describes the state-of-the-art of Simplified Acute Physiology Score models from the clinical and managerial perspectives. Methodological issues concerning the effects of differences between new samples and original databases in which the models were developed are considered.

RECENT FINDINGS

The progressive lack of fit of the Simplified Acute Physiology Score II in independent intensive care unit populations induced investigators to propose customizations and expansions as potential evolutions for Simplified Acute Physiology Score II. We do not know whether those solutions did solve the issue because there are no demonstrations of consistent good fit in new databases. The recently developed Simplified Acute Physiology Score 3 Admission Score with customization for geographical areas is discussed. The points shared by the Simplified Acute Physiology Score models and the pros and cons for each of them are introduced.

SUMMARY

Comparisons of intensive care unit performance should take into account not only the patient severity of illness, but also the effect of the 'intensive care unit variable', that is, differences in human resources, structure, equipment, management and organization of the intensive care unit. In the future, moving from patient and geographical area adjustment to resource use could allow the user to adjust for differences in healthcare provision.

Mortality predictions using current physiologic scoring systems in patients meeting criteria for early goal-directed therapy and the severe sepsis resuscitation bundle

Physiologic scoring systems are often used to prognosticate mortality in critically ill patients. This study examined the performance of Acute Physiology and Chronic Health Evaluation (APACHE) II, Simplified Acute Physiology Score (SAPS) II, Mortality in Emergency Department Sepsis (MEDS), and Mortality Probability Models (MPM) II0 in predicting in-hospital mortality of patients in the emergency department meeting criteria for early goal-directed therapy and the severe sepsis resuscitation bundle. The discrimination and calibration characteristics of APACHE II, SAPS II, MEDS, and MPM II0 were evaluated. Data are presented as median and quartiles (25th, 75th). Two-hundred forty-six patients aged 68 (52, 81) years were analyzed from a prospectively maintained sepsis registry, with 76.0% of patients in septic shock, 45.5% blood culture positive, and 35.0% in-hospital mortality. Acute Physiology and Chronic Health Evaluation II, SAPS II, and MEDS scores were 29 (21, 37), 54 (40, 70), and 13 (11, 16), with predicted mortalities of 64% (40%, 85%), 58% (25%, 84%), and 16% (9%, 39%), respectively. Mortality Probability Models II0 showed a predicted mortality of 60% (27%, 80%). The area under the receiver operating characteristic curves was 0.73 for APACHE II, 0.71 for SAPS II, 0.60 for MEDS, and 0.72 for MPM II0. The standardized mortality ratios were 0.59, 0.63, 1.68, and 0.64, respectively. Thus, APACHE II, SAPS II, MEDS, and MPM II0 have variable abilities to discriminate early and estimate in-hospital mortality of patients presenting to the emergency department requiring the severe sepsis resuscitation bundle. Adoption of these prognostication tools in this setting may influence therapy and resource use for these patients.

Nonpulmonary organ dysfunction and its impact on outcome in patients with acute respiratory failure

PURPOSES

This study aimed to define the prevalence, severity, and progression of nonpulmonary organ dysfunction, and its impact on outcome in patients with acute respiratory failure (ARF) at a respiratory ICU of a tertiary referral hospital in northern India.

METHODS

Daily patient data were collected on 711 adult patients with ARF to calculate component and total nonpulmonary sequential organ failure assessment (SOFA) scores. Hospital survival was the main outcome measure. Multiple logistic regression modeling was conducted to assess contribution of incremental dysfunction of various nonpulmonary organ systems to mortality. Kaplan-Meier curves were drawn to assess temporal trends in survival, and group comparisons were based on log-rank test. Cox proportional hazard modeling was performed to define hazards of earlier mortality. Discrimination was evaluated using receiver operating characteristic (ROC) curves.

RESULTS

Four hundred seventy-five patients (66.8%) had one or more nonpulmonary organ dysfunctions at hospital admission. The overall hospital mortality rate was 33.9%. Hospital survival rates and median survival declined steadily as the number of organs involved increased. Admission, maximum, and DeltaSOFA scores were significantly higher in nonsurvivors. Increasing baseline cardiovascular and neurologic SOFA scores, and corresponding DeltaSOFA scores, were associated with progressively higher odds of hospital mortality, as well as increasing hazard for earlier mortality after adjustment for etiology of respiratory failure. Maximum nonpulmonary SOFA score was the best discriminator in predicting mortality (area under ROC curve, 0.767).

CONCLUSION

Baseline and new-onset nonpulmonary organ dysfunction significantly influences hospital survival in patients with ARF.