The Case for Using Objective Scoring Systems to Predict Intensive Care Unit Outcome

[Validation of APACHE II and SOFA scores in 2 cohorts of patients with suspected infection and sepsis, not admitted to critical care units]

OBJECTIVE

To validate the APACHE II and SOFA scores in patients with suspected infection in clinical settings other than intensive care units.

MATERIALS AND METHODS

A secondary analysis was performed on 2,530 adult patients participating in 2 cohort studies, with suspected infection as admission diagnosis within the first 24 h of hospitalization. The performance of both scoring systems was studied in order to set calibration and discrimination, respectively, on the outcomes such as mortality, admission to Intensive Care Unit, development of septic shock, or multiple organ dysfunctions.

RESULTS

The AUC-ROC values for mortality at discharge and on day 28 in the first cohort were around 0.50 for the SOFA and APACHE II scores; whereas for the second cohort the discrimination value was around 0.70. Calibration of both scoring systems for primary outcomes, according to Hosmer-Lemeshow test, showed p>.05 in the first cohort; while in the second cohort calibration it only showed a p>.05 in the case of the SOFA for mortality at hospital discharge.

CONCLUSION

This validation study of SOFA and APACHE II scores in patients with suspected infection in-hospital units other than the Intensive Care Unit, showed no consistent performance for calibration and discrimination. Its application in emergency and in-hospital patients is limited.

Severity of illness scoring systems in the intensive care unit

OBJECTIVE

Adult intensive care unit prognostic models have been used for predicting patient outcome for three decades. The goal of this review is to describe the different versions of the main adult intensive care unit prognostic models and discuss their potential roles.

DATA SOURCE

PubMed search and review of the relevant medical literature.

SUMMARY

The main prognostic models for assessing the overall severity of illness in critically ill adults are Acute Physiology and Chronic Health Evaluation, Simplified Acute Physiology Score, and Mortality Probability Model. Simplified Acute Physiology Score and Mortality Probability Model have been updated to their third versions and Acute Physiology and Chronic Health Evaluation to its fourth version. The development of prognostic models is usually followed by internal and external validation and performance assessment. Performance is assessed by area under the receiver operating characteristic curve for discrimination and Hosmer-Lemeshow statistic for calibration. The areas under the receiver operating characteristic curve of Simplified Acute Physiology Score 3, Acute Physiology and Chronic Health Evaluation IV, and Mortality Probability Model0 III were 0.85, 0.88, and 0.82, respectively, and all these three fourth-generation models had good calibration. The models have been extensively used for case-mix adjustment in clinical research and epidemiology, but their role in benchmarking, performance improvement, resource use, and clinical decision support has been less well studied.

CONCLUSIONS

The fourth-generation Acute Physiology and Chronic Health Evaluation, Simplified Acute Physiology Score 3, Acute Physiology and Chronic Health Evaluation IV, and Mortality Probability Model0 III adult prognostic models, perform well in predicting mortality. Future studies are needed to determine their roles for benchmarking, performance improvement, resource use, and clinical decision support.

Decision support in multi-professional communication

This paper explores the role of clinical decision support systems (CDSS) in facilitating communication between physicians, nurses, patients and family members. Thirty-three critical care unit nurses and physicians were interviewed regarding the APACHE III CDSS. This qualitative, descriptive study suggests that registered nurses and physicians are primarily motivated to use CDSS when this technology allows them to forecast the potential outcomes of decisions prior to actually making those decisions. These forecasts are used to advocate for care decisions with other disciplines, patients and their family members. Implications for professional practice and recommendations for future research are described.

Critical care nurse practitioners and clinical nurse specialists interface patterns with computer-based decision support systems

PURPOSE

The purposes of this review are to examine the types of clinical decision support systems in use and to identify patterns of how critical care advanced practice nurses (APNs) have integrated these systems into their nursing care patient management practices. The decision-making process itself is analyzed with a focus on how automated systems attempt to capture and reflect human decisional processes in critical care nursing, including how systems actually organize and process information to create outcome estimations based on patient clinical indicators and prognosis logarithms. Characteristics of APN clinicians and implications of these characteristics on decision system use, based on the body of decision system user research, are introduced.

DATA SOURCES

A review of the Medline, Ovid, CINAHL, and PubMed literature databases was conducted using "clinical decision support systems,""computerized clinical decision making," and "APNs"; an examination of components of several major clinical decision systems was also undertaken.

CONCLUSIONS

Use patterns among APNs and other clinicians appear to vary; there is a need for original research to examine how APNs actually use these systems in their practices in critical care settings. Because APNs are increasingly responsible for admission to, and transfer from, critical care settings, more understanding is needed on how they interact with this technology and how they see automated decision systems impacting their practices.

IMPLICATIONS FOR PRACTICE

APNs who practice in critical care settings vary significantly in how they use the clinical decision systems that are in operation in their practice settings. These APNs must have an understanding of their use patterns with these systems and should critically assess whether their patient care decision making is affected by the technology.

Severity of Illness and Organ Failure Assessment in Adult Intensive Care Units

The critical care community has been using severity and organ failure assessment tools for over 2 decades. The major adult severity assessment models are Acute Physiology and Chronic Health Evaluation, Simplified Acute Physiology Score, and Mortality Probability Model. All three recent versions of these models perform well in predicting hospital mortality. Sequential Organ Failure Assessment score is the most used tool for assessment of multiple organ failure. These tools have been used extensively in clinical research involving critically ill patients and for benchmarking and the measurement of performance improvement. Their roles as clinical decision support tools at the bedside await future studies because of their unknown or poor performance at the individual patient level.

The acute physiology and chronic health evaluation III outcome prediction in patients admitted to the intensive care unit after pneumonectomy

PURPOSE

The Acute Physiology and Chronic Health Evaluation (APACHE) III prognostic system has not been previously validated in patients admitted to the intensive care unit (ICU) after pneumonectomy. The purpose of this study was to determine if the APACHE III predicts hospital mortality after pneumonectomy.

METHODS

A retrospective review of all adult patients admitted to a single thoracic surgical intensive care unit after pneumonectomy between October 1994 and December 2004. Patient demographics, ICU admission day APACHE III score, actual and predicted hospital mortality, and length of hospital and ICU stay data were collected. Data on preoperative pulmonary function tests and smoking habits were also collected. Univariate statistical methods and logistic regression were used. The performance of the APACHE III prognostic system was assessed by the Hosmer-Lemeshow statistic for calibration and area under receiver operating characteristic curve (AUC) for discrimination.

RESULTS

There were 417 pneumonectomies performed during the study period, of which 281 patients were admitted to the ICU. The mean age was 61.1 years, and 67.2% were men; 88.2% were smokers with a median of 40.0 (interquartile range, 18-62) pack-years of tobacco use. The mean APACHE III score on the day of ICU admission was 37.7 (+/- standard deviation 17.8), and the mean predicted hospital mortality rate was 6.4% (+/-10.4). The median (and interquartile range) lengths of ICU and hospital stay were 1.7 (0.9-3.1) and 9.0 (7.0-17.0) days, respectively. The observed ICU and hospital mortality rates were 4.6% (13/281 patients) and 8.2% (23/281), respectively. The standardized ICU and hospital mortality ratios with their 95% confidence intervals (CIs) were 1.55 (0.71-2.39) and 1.27 (0.75-1.78), respectively. There were significant differences in the mean APACHE III score (p < 0.001) and the predicted mortality rate (p < .001) between survivors and nonsurvivors. In predicting mortality, the AUC of APACHE III prediction was 0.801 (95% CI, 0.711-0.891), and the Hosmer-Lemeshow statistic was 9.898 with a p value of 0.272. Diffusion capacity of the lung for carbon monoxide (DLCO) and percentage predicted DLCO were higher in survivors, but the addition of either of these variables to a logistic regression model did not improve APACHE III mortality prediction.

CONCLUSIONS

In patients admitted to the ICU after pneumonectomy, the APACHE III discriminates moderately well between survivors and nonsurvivors. The calibration of the model appears to be good, although the low number of deaths limits the power of the calibration analysis. The use of APACHE III data in outcomes research involving patients who have undergone pneumonectomy is acceptable.

Prognosis in critical care

Prognostic risk prediction models have been employed in the intensive care unit (ICU) setting since the 1980s and provide health care providers with important information to help inform decisions related to treatment and prognosis, as well as to compare outcomes across institutions. Prognostic models for critical care are among the most widely utilized and tested predictive models in healthcare. In this article, we review and compare mortality prediction models, including the APACHE (1981), SAPS (1984), APACHE-II (1985), MPM (1987), APACHE-III (1991), SAPS-II (1993), and MPM-II (1993). We emphasize the importance of model calibration in this domain. In addition, we present a brief review of the statistical methodology, multiple logistic regression, which underlies most of the models currently used in critical care.

Prospective validation of the Rockall risk scoring system for upper GI hemorrhage in subgroups of patients with varices and peptic ulcers

OBJECTIVES

The Rockall risk assessment score was devised to allow prediction of the risk of rebleeding and death in patients with upper GI hemorrhage. The score was derived by multivariate analysis in a cohort of patients with upper GI hemorrhage and subsequently validated in a second cohort. Only 4.4% of patients included in the initial study had esophageal varices, and analysis was not performed according to the etiology of the bleeding. Our aim was to assess the validity of the Rockall risk scoring system in predicting rebleeding and mortality in patients with esophageal varices or peptic ulcers.

METHODS

Admissions (n = 358) over 32 months to a single specialist GI bleeding unit were scored prospectively. The distribution of episodes of rebleeding and mortality by Rockall score were statistically analyzed using Fisher's exact test with 99% CIs calculated using a Monte Carlo method. The Child-Pugh score was determined in patients with esophageal varices.

RESULTS

The Rockall score was predictive of both rebleeding and mortality in patients with variceal hemorrhage (both ps < 0.0005), as was the Child-Pugh score (p = 0.001 and p < 0.0005, respectively). The initial Rockall score was predictive of mortality in patients with peptic ulcers (p = 0.01), although the complete score was not (p > 0.05). The complete score did, however, predict rebleeding in these patients (p = 0.001).

CONCLUSION

This is the first study to validate the Rockall score in specific subgroups of patients with esophageal varices or peptic ulcers and suggests that it is particularly applicable to variceal hemorrhage.

Combining physician's subjective and physiology-based objective mortality risk predictions

OBJECTIVE

None of the currently available physiology-based mortality risk prediction models incorporate subjective judgements of healthcare professionals, a source of additional information that could improve predictor performance and make such systems more acceptable to healthcare professionals. This study compared the performance of subjective mortality estimates by physicians and nurses with a physiology-based method, the Pediatric Risk of Mortality (PRISM) III. Then, healthcare provider estimates were combined with PRISM III estimates using Bayesian statistics. The performance of the Bayesian model was then compared with the original two predictions.

DESIGN

Concurrent cohort study.

SETTING

A tertiary pediatric intensive care unit at a university affiliated children's hospital.

PATIENTS

Consecutive admissions to the pediatric intensive care unit.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

For each of the 642 consecutive eligible patients, an exact mortality estimate and the degree of certainty (continuous scale from 1 to 5) associated with the estimate was collected from the attending, fellow, resident, and nurse responsible for the patient's care. Bayesian statistics were used to combine the PRISM III and certainty weighted subjective predictions to create a third Bayesian estimate of mortality. PRISM III discriminated survivors from nonsurvivors very well (area under curve [AUC], 0.924) as did the physicians and nurses (AUCs attendings, 0.953; fellows, 0.870; residents, 0.923; nurses, 0.935). Although the AUCs of the healthcare providers were not significantly different from the AUCs of PRISM III, the Bayesian AUCs were higher than both the healthcare providers' AUCs (p < or = .09 for all) and PRISM III AUCs. Similarly, the calibration statistics for the Bayesian estimates were superior to the calibration statistics for both the healthcare providers and PRISM III models.

CONCLUSIONS

The results of this study demonstrated that healthcare providers' subjective mortality predictions and PRISM III mortality predictions perform equally well. The Bayesian model that combined provider and PRISM III mortality predictions was more accurate than either provider or PRISM III alone and may be more acceptable to physicians. A methodology using subjective outcome predictions could be more relevant to individual patient decision support.

An analysis of excess mortality not predicted to occur by APACHE III in an Australian level III intensive care unit

The APACHE III derived standardized mortality ratio has been suggested as a statistic to measure intensive care unit (ICU) effectiveness. From 1991 data collected on 519 consecutive admissions to the Royal Adelaide Hospital ICU a standardized mortality ratio of 1.25 was calculated. Of the 174 deaths only 95 had a prediction of death greater than 0.5. As part of a quality assurance study we undertook a retrospective case note audit to try to identify factors that were associated with the low mortality prediction (< 0.5) in hospital deaths. Firstly we analysed the patient population that died to determine the factors that were different between patients who had a mortality prediction of greater than 0.5 versus those who had a mortality prediction of less than 0.5. Next we analysed the patient population with a mortality prediction of less than 0.5 and compared actual survivors with patients who died in hospital. Amongst low mortality prediction patients admitted to the Royal Adelaide Hospital ICU we identified age, a history of acute myocardial infarction, presentation to ICU after a cardiac arrest or with an elevated creatinine and the development of acute renal failure and septicaemia during the ICU admission as being associated with in-hospital mortality. We also documented that late hospital deaths on the ward after ICU discharge occurred more frequently with low predicted hospital mortality ICU patients. Factors other than the APACHE III score may be associated with hospital deaths of ICU patients.

Intensive care unit prognostic scoring systems to predict death: a cost-effectiveness analysis

OBJECTIVE

To evaluate the cost-effectiveness, using the technique of decision analysis, of withdrawing care from patients in the intensive care unit (ICU) who are predicted to have a high probability of death (>90%) after 48 hrs using a mortality risk estimate based on daily Acute Physiology and Chronic Health Evaluation (APACHE) III scores.

MATERIALS AND METHODS

A decision tree model was constructed to compare the cost-effectiveness of two clinical strategies. In the first strategy, patients receive ICU care until they were discharged, died, or had care withdrawn based on subjective clinical criteria. In the second strategy, patients remained in the ICU until they were either discharged, died, or had life-sustaining care withdrawn based on subjective criteria or if they were predicted to have a >90% risk of mortality after 48 hrs by a prognostic scoring system. Transition probabilities were based on a retrospective data analysis of 4,106 noncardiac ICU patients admitted to a tertiary surgical ICU over a 9-yr period. Cost estimates were based on daily Therapeutic Intervention Scoring System (TISS) scores from our database and using published data on the estimated production cost for a TISS point. The sensitivity (16.6%) and specificity (99.6%) of the mortality risk estimate at 48 hrs (using the >90% decision point) based on daily APACHE III scores were derived from published data.

RESULTS

In the base case analysis, we assumed that the sensitivity and specificity of the prognostic risk estimate are unchanged when exported to a new environment. Not using a prognostic scoring system as the basis for withdrawing care resulted in a slightly higher survival rate (87.2% vs. 86.85%) at a cost-per-death prevented (CPDP) of $263,700. Since prognostic scoring systems have not been shown to retain the same predictive power when exported to new databases, we chose to explore the effect of varying the specificity of the scoring system on CPDP. Decreasing the specificity from .996 (baseline) to .98 causes the CPDP to drop to $53,300. Changing the specificity to .95 results in a CPDP prevented of $21,700. Using one-way sensitivity analysis, the CPDP is shown to be relatively insensitive to delaying the decision point from ICU day 3 to day 7. Sensitivity analysis also indicates that CPDP increases rapidly with hospital death rate. For a death rate of 30%, the CPDP increases to $768,600 (in the base case, the death rate is 12.8%); when the specificity is decreased to .95, the CPDP drops to $62,100.

CONCLUSION

Unless daily mortality risk estimates based on APACHE III can be shown to retain the same level of predictive power in ICUs outside the development database, it is unlikely that the incremental cost-effectiveness gained by using them as the basis to withdraw care is sufficient to justify their use in this manner.

[Value of the Hannover Intensive Score (HIS) in internal medicine intensive care]

BACKGROUND AND OBJECTIVES

Scoring systems are important tools for quality control and stratification of study populations in intensive care medicine. The study aims to systematically evaluate predictive ability and severity classification ability of the combined physiologic-therapeutic Hannover Intensiv Score (HIS). Such data are not existing regarding medical intensive care medicine.

METHODS

1060 consecutive patients (ICU stay > 4 hours) being admitted to a medical ICU were prospectively investigated. HIS was determined for all patients each day during ICU stay. The results were compared to the physiologically based APACHE II and to the therapeutically based TISS, which both were determined as well.

RESULTS

HIS provided sufficient discrimination between survival and nonsurvival [hospital mortality; area under the ROC curve (AUC) = 0.822] with no significant differences compared to APACHE II (AUC = 0.838) and TISS (AUC = 0.798), respectively. During longer course of ICU stay HIS offers better outcome prognostication compared to the unilateral systems with respect to specificity and total correct classification rate. There was a nearly linear increase of hospital mortality with an increase of day-1-HIS. The same was observed with APACHE II and TISS. Mean day-1-scores for survivors were significantly higher compared to non-survivors with all systems (p < 0.0001). Day-1-HIS moderately correlates with both other systems (APACE II: r = 0.766; TISS: r = 0.814).

CONCLUSIONS

The Hannover Intensiv Score as a model of a combined physiologic-therapeutic scoring system was successfully validated concerning hospital outcome prediction and severity of disease classification in a large medical ICU population. Thus, for these applications it can be used in similar German ICUs. A main argument for applying the system is the employment of a fairly small set of easily accessible parameters.

ICU scoring systems do not allow prediction of patient outcomes or comparison of ICU performance

Recent revisions of the major ICU scoring systems have broadened their database markedly and increased their statistical accuracy. For a specific patient, however, the systems cannot be accurate enough to direct management decisions. Significant questions remain about the reliability of these systems for comparing different ICUs and different patient populations, especially in surgical and trauma patients. Current scoring systems, therefore, cannot be used reliably in either the management of the individual patient or in the making of quality comparisons between ICUs.

Critical care medicine: opportunities and strategies for improvement

BACKGROUND

Like other areas of health care, critical care faces increasing pressure to improve the quality while reducing the cost of care. Strategies drawn from the literature and the authors' experiences are presented.

STRATEGIES AND OPPORTUNITIES FOR IMPROVEMENTS

Ten process- or structure-related areas are targeted as strategically important focuses of improvement: (1) restructuring administrative lines to better suit key processes; (2) physician leadership in critical care units; (3) management training for critical care managers; (4) triage; (5) multidisciplinary critical care; (6) standardization of care; (7) developing alternatives to critical care units; (8) timeliness of care delivery; (9) appropriate use of critical care resources; and (10) tracking quality improvement.

TIMELINESS OF CARE DELIVERY

Whatever the root cause(s) of unnecessary delays, the result is inefficient use of critical care resources-and ultimately either a need for more resources or longer wait times. Innovations designed to reduce wait times and waste, such as the establishment of a microchemistry stat laboratory, may prove valuable.

APPROPRIATE USE OF CRITICAL CARE RESOURCES

Possible strategies for the appropriate use of critical care resources include better selection of well-informed patients who undergo procedures. Reduction in variation among physicians and organizations in providing therapies will also likely lead to a reduction in some high-risk procedures offering little or no benefit, and therefore a reduction in need for critical care services. Better preparation of patients and families should also make end-of-life decisions easier when questions of "futility" arise. Better information on outcomes and cost-effectiveness and consensus on withdrawal of critical care treatments represent two additional strategies.

Critical care computing: outcomes, confidentiality, and appropriate use

This article reviews the current limitations of computerized outcome predictor models and severity scoring systems. A logical extension of predictor models, a "computational futility metric," is proposed with a discussion of potential uses and abuses. These types of electronic surveillance will not solve the problem of society's denial of death or resolve the allocation of medical resources. Issues related to the protection of patients and physicians under electronic epidemiologic surveillance are discussed.

Outcome evaluations as measures of quality in pediatric intensive care

In recent years, interest in assessing quality of care has blossomed. Quality care may be defined as providing the most appropriate treatment and providing it with great technical and managerial skill and proficiency in a manner that gains patient acceptance. For assessment purposes, variation in risk-adjusted outcomes between providers should be attributable to quality of care differences. Some methods for measuring outcomes and risk-adjustment for pediatric intensive care populations have been developed, but additional tools are needed for applications in outcomes management, continuous quality improvement, and outcomes research.