Customized probability models for early severe sepsis in adult intensive care patients. Intensive Care Unit Scoring Group

ICU scoring systems do not allow prediction of patient outcomes or comparison of ICU performance

Recent revisions of the major ICU scoring systems have broadened their database markedly and increased their statistical accuracy. For a specific patient, however, the systems cannot be accurate enough to direct management decisions. Significant questions remain about the reliability of these systems for comparing different ICUs and different patient populations, especially in surgical and trauma patients. Current scoring systems, therefore, cannot be used reliably in either the management of the individual patient or in the making of quality comparisons between ICUs.

ICU scoring systems allow prediction of patient outcomes and comparison of ICU performance

Too much time and effort are wasted in attempts to pass final judgment on whether systems for ICU prognostication are "good or bad" and whether they "do or do not" provide a simple answer to the complex and often unpredictable question of individual mortality in the ICU. A substantial amount of data supports the usefulness of general ICU prognostic systems in comparing ICU performance with respect to a wide variety of endpoints, including ICU and hospital mortality, duration of stay, and efficiency of resource use. Work in progress is analyzing both general resource use and specific therapeutic interventions. It also is time to fully acknowledge that statistics never can predict whether a patient will die with 100% accuracy. There always will be exceptions to the rule, and physicians frequently will have information that is not included in prognostic models. In addition, the values of both physicians and patients frequently lead to differences in how a probability in interpreted; for some, a 95% probability estimate means that death is near and, for others, this estimate represents a tangible 5% chance for survival. This means that physicians must learn how to integrate such estimates into their medical decisions. In doing so, it is our hope that prognostic systems are not viewed as oversimplifying or automating clinical decisions. Rather, such systems provide objective data on which physicians may ground a spectrum of decisions regarding either escalation or withdrawal of therapy in critically ill patients. These systems do not dehumanize our decision-making process but, rather, help eliminate physician reliance on emotional, heuristic, poorly calibrated, or overly pessimistic subjective estimates. No decision regarding patient care can be considered best if the facts upon which it is based on imprecise or biased. Future research will improve the accuracy of individual patient predictions but, even with the highest degree of precision, such predictions are useful only in support of, and not as a substitute for, good clinical judgment.

Factors affecting the performance of the models in the Mortality Probability Model II system and strategies of customization: a simulation study

OBJECTIVES

To examine the impact of hospital mortality and intensive care unit (ICU) size on the performance of the Mortality Probability Model II system for use in quality assessment, and to examine the ability of model customization to produce accurate estimates of hospital mortality to characterize patients by severity of illness for clinical trials.

DESIGN

Prospective evaluation of model performance, using retrospective data.

SETTING

Data for the simulation were assembled from six adult medical and surgical ICUs in Massachusetts and New York.

PATIENTS

Consecutive admissions (n = 4,224) to the Massachusetts and New York ICUs were studied. The mortality rate in the database was 18.7%.

INTERVENTIONS

A computer simulation of several different hospital mortality rates and ICU sample sizes, using a multicenter database of consecutive ICU admissions, was utilized. We simulated 20 different mortality rates by randomly changing the outcomes at hospital discharge from "survived" to "deceased" and from "deceased" to "survived". Four sample size simulations used 75%, 50%, 25%, and 10% of the database. Ten replications of each mortality rate and samples size were constructed, and model calibration and discrimination were assessed for each replication. Model coefficients were customized, using logistic regression.

MEASUREMENTS AND MAIN RESULTS

Vital status at hospital discharge was the outcome measure among the ICU patient population. Model performance was assessed using the Hosmer-Lemeshow C statistic for calibration, and the area under the receiver operating characteristic curve for discrimination. Goodness-of-fit tests and receiver operating characteristic curve areas demonstrated that the models were sensitive to differences in hospital mortality, indicating that they are useful quality assurance tools. Goodness-of-fit tests were more sensitive than the receiver operating characteristic curve areas. The further the hospital mortality rate diverged from the original rate, the worse the performance of the model. Sample size had an impact on these results. The smaller the sample size, the less likely the model was to perform poorly. Model coefficients were successfully customized to demonstrate that improved model performance can be achieved when necessary for clinical trial stratification.

CONCLUSION

Mortality Probability Model II models can be used to assess quality of care in ICUs, but the size of the sample should be considered when assessing calibration and discrimination.