Comparison of outcome from intensive care admission after adjustment for case mix by the APACHE III prognostic system

The effect of treatment and clinical course during Emergency Department stay on severity scoring and predicted mortality risk in Intensive Care patients

Background

Treatment and the clinical course during Emergency Department (ED) stay before Intensive Care Unit (ICU) admission may affect predicted mortality risk calculated by the Acute Physiology and Chronic Health Evaluation (APACHE)-IV, causing lead-time bias. As a result, comparing standardized mortality ratios (SMRs) among hospitals may be difficult if they differ in the location where initial stabilization takes place. The aim of this study was to assess to what extent predicted mortality risk would be affected if the APACHE-IV score was recalculated with the initial physiological variables from the ED. Secondly, to evaluate whether ED Length of Stay (LOS) was associated with a change (delta) in these APACHE-IV scores.

Methods

An observational multicenter cohort study including ICU patients admitted from the ED. Data from two Dutch quality registries were linked: the Netherlands Emergency department Evaluation Database (NEED) and the National Intensive Care Evaluation (NICE) registry. The ICU APACHE-IV, predicted mortality, and SMR based on data of the first 24 h of ICU admission were compared with an ED APACHE-IV model, using the most deviating physiological variables from the ED or ICU.

Results

A total of 1398 patients were included. The predicted mortality from the ICU APACHE-IV (median 0.10; IQR 0.03–0.30) was significantly lower compared to the ED APACHE-IV model (median 0.13; 0.04–0.36; p < 0.01). The SMR changed from 0.63 (95%CI 0.54–0.72) to 0.55 (95%CI 0.47–0.63) based on ED APACHE-IV. Predicted mortality risk changed more than 5% in 321 (23.2%) patients by using the ED APACHE-IV. ED LOS > 3.9 h was associated with a slight increase in delta APACHE-IV of 1.6 (95% CI 0.4–2.8) compared to ED LOS < 1.7 h.

Conclusion

Predicted mortality risks and SMRs calculated by the APACHE IV scores are not directly comparable in patients admitted from the ED if hospitals differ in their policy to stabilize patients in the ED before ICU admission. Future research should focus on developing models to adjust for these differences.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13054-022-03986-2.

Generalizability challenges of mortality risk prediction models: A retrospective analysis on a multi-center database

Modern predictive models require large amounts of data for training and evaluation, absence of which may result in models that are specific to certain locations, populations in them and clinical practices. Yet, best practices for clinical risk prediction models have not yet considered such challenges to generalizability. Here we ask whether population- and group-level performance of mortality prediction models vary significantly when applied to hospitals or geographies different from the ones in which they are developed. Further, what characteristics of the datasets explain the performance variation? In this multi-center cross-sectional study, we analyzed electronic health records from 179 hospitals across the US with 70,126 hospitalizations from 2014 to 2015. Generalization gap, defined as difference between model performance metrics across hospitals, is computed for area under the receiver operating characteristic curve (AUC) and calibration slope. To assess model performance by the race variable, we report differences in false negative rates across groups. Data were also analyzed using a causal discovery algorithm "Fast Causal Inference" that infers paths of causal influence while identifying potential influences associated with unmeasured variables. When transferring models across hospitals, AUC at the test hospital ranged from 0.777 to 0.832 (1st-3rd quartile or IQR; median 0.801); calibration slope from 0.725 to 0.983 (IQR; median 0.853); and disparity in false negative rates from 0.046 to 0.168 (IQR; median 0.092). Distribution of all variable types (demography, vitals, and labs) differed significantly across hospitals and regions. The race variable also mediated differences in the relationship between clinical variables and mortality, by hospital/region. In conclusion, group-level performance should be assessed during generalizability checks to identify potential harms to the groups. Moreover, for developing methods to improve model performance in new environments, a better understanding and documentation of provenance of data and health processes are needed to identify and mitigate sources of variation.

Towards unstructured mortality prediction with free-text clinical notes

Healthcare data continues to flourish yet a relatively small portion, mostly structured, is being utilized effectively for predicting clinical outcomes. The rich subjective information available in unstructured clinical notes can possibly facilitate higher discrimination but tends to be under-utilized in mortality prediction. This work attempts to assess the gain in performance when multiple notes that have been minimally preprocessed are used as an input for prediction. A hierarchical architecture consisting of both convolutional and recurrent layers is used to concurrently model the different notes compiled in an individual hospital stay. This approach is evaluated on predicting in-hospital mortality on the MIMIC-III dataset. On comparison to approaches utilizing structured data, it achieved higher metrics despite requiring less cleaning and preprocessing. This demonstrates the potential of unstructured data in enhancing mortality prediction and signifies the need to incorporate more raw unstructured data into current clinical prediction methods.

Risk of mortality and cardiopulmonary arrest in critical patients presenting to the emergency department using machine learning and natural language processing

Emergency department triage is the first point in time when a patient's acuity level is determined. The time to assign a priority at triage is short and it is vital to accurately stratify patients at this stage, since under-triage can lead to increased morbidity, mortality and costs. Our aim was to present a model that can assist healthcare professionals in triage decision making, namely in the stratification of patients through the risk prediction of a composite critical outcome-mortality and cardiopulmonary arrest. Our study cohort consisted of 235826 adult patients triaged at a Portuguese Emergency Department from 2012 to 2016. Patients were assigned to emergent, very urgent or urgent priorities of the Manchester Triage System (MTS). Demographics, clinical variables routinely collected at triage and the patients' chief complaint were used. Logistic regression, random forests and extreme gradient boosting were developed using all available variables. The term frequency-inverse document frequency (TF-IDF) natural language processing weighting factor was applied to vectorize the chief complaint. Stratified random sampling was used to split the data into train (70%) and test (30%) data sets. Ten-fold cross validation was performed in train to optimize model hyper-parameters. The performance obtained with the best model was compared against the reference model-a regularized logistic regression trained using only triage priorities. Extreme gradient boosting exhibited good calibration properties and yielded areas under the receiver operating characteristic and precision-recall curves of 0.96 (95% CI 0.95-0.97) and 0.31 (95% CI 0.26-0.36), respectively. The predictors ranked with higher importance by this model were the Glasgow coma score, the patients' age, pulse oximetry and arrival mode. Compared to the reference, the extreme gradient boosting model using clinical variables and the chief complaint presented higher recall for patients assigned MTS-3 and can identify those who are at risk of the composite outcome.

Developing well-calibrated illness severity scores for decision support in the critically ill

Illness severity scores are regularly employed for quality improvement and benchmarking in the intensive care unit, but poor generalization performance, particularly with respect to probability calibration, has limited their use for decision support. These models tend to perform worse in patients at a high risk for mortality. We hypothesized that a sequential modeling approach wherein an initial regression model assigns risk and all patients deemed high risk then have their risk quantified by a second, high-risk-specific, regression model would result in a model with superior calibration across the risk spectrum. We compared this approach to a logistic regression model and a sophisticated machine learning approach, the gradient boosting machine. The sequential approach did not have an effect on the receiver operating characteristic curve or the precision-recall curve but resulted in improved reliability curves. The gradient boosting machine achieved a small improvement in discrimination performance and was similarly calibrated to the sequential models.

Performance of APACHE IV in Medical Intensive Care Unit Patients: Comparisons with APACHE II, SAPS 3, and MPM0 III

Background

In this study, we analyze the performance of the Acute Physiology and Chronic Health Evaluation (APACHE) II, APACHE IV, Simplified Acute Physiology Score (SAPS) 3, and Mortality Probability Model (MPM)₀ III in order to determine which system best implements data related to the severity of medical intensive care unit (ICU) patients.

Methods

The present study was a retrospective investigation analyzing the discrimination and calibration of APACHE II, APACHE IV, SAPS 3, and MPM₀ III when used to evaluate medical ICU patients. Data were collected for 788 patients admitted to the ICU from January 1, 2015 to December 31, 2015. All patients were aged 18 years or older with ICU stays of at least 24 hours. The discrimination abilities of the three systems were evaluated using c-statistics, while calibration was evaluated by the Hosmer-Lemeshow test. A severity correction model was created using logistics regression analysis.

Results

For the APACHE IV, SAPS 3, MPM₀ III, and APACHE II systems, the area under the receiver operating characteristic curves was 0.745 for APACHE IV, resulting in the highest discrimination among all four scoring systems. The value was 0.729 for APACHE II, 0.700 for SAP 3, and 0.670 for MPM₀ III. All severity scoring systems showed good calibrations: APACHE II (chi-square, 12.540; P=0.129), APACHE IV (chi-square, 6.959; P=0.541), SAPS 3 (chi-square, 9.290; P=0.318), and MPM₀ III (chi-square, 11.128; P=0.133).

Conclusions

APACHE IV provided the best discrimination and calibration abilities and was useful for quality assessment and predicting mortality in medical ICU patients.

Comparison of the pediatric risk of mortality, pediatric index of mortality, and pediatric index of mortality 2 models in a pediatric intensive care unit in China: A validation study

This study was designed with the aim of comparing the performances of the pediatric risk of mortality (PRISM), pediatric index of mortality (PIM), and revised version pediatric index of mortality 2 (PIM2) models in a pediatric intensive care unit (PICU) in China.A total of 852 critically ill pediatric patients were recruited in the study between January 1 and December 31, 2014. The variables required to calculate PRISM, PIM, and PIM2 were collected. Mode l performance was evaluated by assessing the calibration and discrimination. Discrimination between death and survival was assessed by calculating the area under the receiver-operating characteristic curve (AUC). Calibration across deciles of risk was evaluated using the Hosmer-Lemeshow goodness-of-fit χ test.Of the 852 patients enrolled in this study, 745 patients survived until the end of the PICU stay (107 patients died, 12.56%). The AUCs (95% confidence intervals, CI) were 0.729 (0.670-0.788) for PRISM, 0.721 (0.667-0.776) for PIM, and 0.726 (0.671-0.781) for PIM2. The Hosmer-Lemeshow test revealed a chi-square of 7.26 (P = 0.51, v = 10) for PRISM, 26.28 (P = 0.0009, v = 10) for PIM, and 10.28 (P = 0.21, v = 10) for PIM2. The standardized mortality rate was 1.14 (95%CI: 0.93-1.36) for PRISM, 1.89 (95%CI: 1.55-2.27) for PIM, and 2.13 (95%CI: 1.75-2.55) for PIM2.The PRISM, PIM, and PIM2 scores demonstrated an acceptable discriminatory performance. With the exception of PIM, the PRISM and PIM2 models had good calibrations.

A comparison of Simplified Acute Physiology Score II, Acute Physiology and Chronic Health Evaluation II and Acute Physiology and Chronic Health Evaluation III scoring system in predicting mortality and length of stay at surgical intensive care unit

Background:

In critically ill patients, several scoring systems have been developed over the last three decades. The Acute Physiology and Chronic Health Evaluation (APACHE) and the Simplified Acute Physiology Score (SAPS) are the most widely used scoring systems in the intensive care unit (ICU). The aim of this study was to assess the prognostic accuracy of SAPS II and APACHE II and APACHE III scoring systems in predicting short-term hospital mortality of surgical ICU patients.

Materials and Methods:

Prospectively collected data from 202 patients admitted to Mashhad University Hospital postoperative ICU were analyzed. Calibration was estimated using the Hosmer-Lemeshow goodness-of-fit test. Discrimination was evaluated by using the receiver operating characteristic (ROC) curves and area under a ROC curve (AUC).

Result:

Two hundred and two patients admitted on post-surgical ICU were evaluated. The mean SAPS II, APACHE II, and APACHE III scores for survivors were found to be significantly lower than of non-survivors. The calibration was best for APACHE II score. Discrimination was excellent for APACHE II (AUC: 0.828) score and acceptable for APACHE III (AUC: 0.782) and SAPS II (AUC: 0.778) scores.

Conclusion:

APACHE II provided better discrimination than APACHE III and SAPS II calibration was good at APACHE II and poor at APACHE III and SAPS II. Use of APACHE II was excellent in this post-surgical ICU.

Prognostic categorization of intensive care septic patients

Sepsis is one of the leading worldwide causes of morbidity and mortality in critically-ill patients. Prediction of outcome in patients with sepsis requires repeated clinical interpretation of the patients’ conditions, clinical assessment of tissue hypoxia and the use of severity scoring systems, because the prognostic categorization accuracy of severity scoring indices alone, is relatively poor. Generally, such categorization depends on the severity of the septic state, ranging from systemic inflammatory response to septic shock. Now, there is no gold standard for the clinical assessment of tissue hypoxia which can be achieved by both global and regional oxygen extractabilities, added to prognostic pro-inflammatory mediators. Because the technology used to identify the genetic make-up of the human being is rapidly advancing, the structure of 30 000 genes which make-up the human DNA bank is now known. This would allow easy prognostic categorization of critically-ill patients including those suffering from sepsis. The present review spots lights on the main severity scoring systems used for outcome prediction in septic patients. For morbidity prediction, it discusses the Multiple Organ Dysfunction score, the sequential organ failure assessment score, and the logistic organ dysfunction score. For mortality/survival prediction, it discusses the Acute Physiology and Chronic Health Evaluation scores, the Therapeutic Intervention Scoring System, the Simplified acute physiology score and the Mortality Probability Models. An ideal severity scoring system for prognostic categorization of patients with systemic sepsis is far from being reached. Scoring systems should be used with repeated clinical interpretation of the patients’ conditions, and the assessment of tissue hypoxia in order to attain satisfactory discriminative performance and calibration power.

Caution when using prognostic models: a prospective comparison of 3 recent prognostic models

PURPOSE

Prognostic models have been developed to estimate mortality and to compare outcomes in different intensive care units. However, these models need to be validated before their use in different populations. In this study, we assessed the performance of 3 recently developed general prognostic models (Acute Physiologic and Chronic Health Evaluation [APACHE] IV, Simplified Acute Physiology Score [SAPS] 3 and Mortality Probability Model III [MPM(0)-III]) in a population admitted at 3 medical-surgical Brazilian intensive care units.

MATERIALS AND METHODS

All patients admitted from July 2008 to December 2009 were evaluated for inclusion in the study. Standardized mortality ratios were calculated for all models. Calibration was assessed by the Hosmer-Lemeshow goodness-of-fit test. Discrimination was evaluated using the area under the receiver operator curve.

RESULTS

A total of 5780 patients were included. Inhospital mortality was 9.1%. Discrimination was very good for all models (area under the receiver operator curve for APACHE IV, SAPS 3 and MPM(0)-III was 0.883, 0.855 and 0.840, respectively). APACHE IV showed better discrimination than SAPS 3 and MPM(0)-III (P < .001 for both comparisons). All models calibrated poorly and overestimated hospital mortality (Hosmer-Lemeshow statistic was 53.7, 134.2, 226.6 for APACHE IV, MPM(0)-III, and SAPS 3, respectively; P < .001 for all).

CONCLUSIONS

In this study, all models showed poor calibration, while discrimination was very good for all of them. As this has been a common finding in validation studies, caution is warranted when using prognostic models for benchmarking.

Using the E-PASS scoring system to estimate the risk of emergency abdominal surgery in patients with acute gastrointestinal disease

PURPOSE

The Estimation of Physiologic Ability and Surgical Stress (E-PASS) scoring system, which quantifies a patient's reserve and surgical stress, is used to predict morbidity and mortality in patients before elective gastrointestinal surgery. We conducted this study to clarify whether the E-PASS scoring system is useful for assessing the risks of emergency abdominal surgery.

METHODS

The subjects of this retrospective study were 51 patients who underwent emergency gastrointestinal surgery at a public general hospital. The main outcomes were the E-PASS scores and the postoperative course, defined by mortality and morbidity.

RESULTS

Postoperative complications developed in 15 of the 51 patients (29.4%). The E-PASS score was significantly higher in the patients with postoperative complications than in those without (0.61 ± 0.31 vs 0.20 ± 0.35, respectively; n = 36). The morbidity rates were significantly lower in the patients with a value less than 0.5 than in those with a value more than 0.5 (17.1% and 56.3%, respectively; P < 0.01). There were 7 operative deaths among the 16 patients with a high score, versus none among the 9 patients with a low score (P < 0.01). Three patients underwent laparoscopic-assisted bowel resection with a good postoperative course, with scores of less than 0.5.

CONCLUSIONS

The E-PASS scoring system is useful for surgical decision making and evaluating whether patients will tolerate emergency gastrointestinal surgery. Minimally invasive therapy would assist in lowering the risk of complications.

RGS4, a GTPase activator, improves renal function in ischemia-reperfusion injury

Acute kidney dysfunction after ischemia-reperfusion injury (IRI) may be a consequence of persistent intrarenal vasoconstriction. Regulators of G-protein signaling (RGSs) are GTPase activators of heterotrimeric G proteins that can regulate vascular tone. RGS4 is expressed in vascular smooth muscle cells in the kidney; however, its protein levels are low in many tissues due to N-end rule-mediated polyubiquitination and proteasomal degradation. Here, we define the role of RGS4 using a mouse model of IRI comparing wild-type (WT) with RGS4-knockout mice. These knockout mice were highly sensitized to the development of renal dysfunction following injury exhibiting reduced renal blood flow as measured by laser-Doppler flowmetry. The kidneys from knockout mice had increased renal vasoconstriction in response to endothelin-1 infusion ex vivo. The intrinsic renal activity of RGS4 was measured following syngeneic kidney transplantation, a model of cold renal IRI. The kidneys transplanted between knockout and WT mice had significantly reduced reperfusion blood flow and increased renal cell death. WT mice administered MG-132 (a proteasomal inhibitor of the N-end rule pathway) resulted in increased renal RGS4 protein and in an inhibition of renal dysfunction after IRI in WT but not in knockout mice. Thus, RGS4 antagonizes the development of renal dysfunction in response to IRI.

The variation in risk adjusted mortality of intensive care units

BACKGROUND

This study aimed to estimate risk adjusted mortality rate in the ICUs (Intensive care units) by APACHE (Acute Physiology And Chronic Health Evaluation) III for revealing the performance variation in ICUs.

METHODS

This study focused on 1,090 patients in the ICUs of 18 hospitals. For establishing risk adjusted mortality predictive model, logistic regression analysis was performed. APACHE III, surgery experience, admission route, and major disease categories were used as independent variables. The performance of each model was evaluated by c-statistic and goodness-of-fit test of Hosmer-Lemeshow. Using this predictive model, the performance of each ICU was tested as ratio of predictive mortality rate and observed mortality rate.

RESULTS

The average observed mortality rate was 24.1%. The model including APACHE III score, admission route, and major disease categories was signified as the fittest one. After risk adjustment, the ratio of predictive mortality rate and observed mortality rate was distributed from 0.49 to 1.55.

CONCLUSIONS

The variation in risk adjusted mortality among ICUs was wide. The effort to reduce this quality difference is needed.

[Medical emergency teams: current situation and perspectives of preventive in-hospital intensive care medicine]

Severe clinical incidents occur in up to 10% of all non-intensive care unit (ICU) patients, which have an estimated mortality of 5-8%. As in the prehospital setting, early clinical warning signs can be identified in the majority of cases. Studies suggest that introduction of an in-hospital medical emergency team (MET) which responds to objective criteria of physiological deterioration, may effectively reduce the incidence of in-hospital cardiac arrests as well as unanticipated or readmissions to the ICU. According to this concept, METs would evaluate and treat non-ICU patients at risk at an early stage before a potentially fatal deterioration of cardiorespiratory parameters occurs. This article reviews available data on preventive in-hospital intensive care medicine and reflects on the circumstances for an implementation of METs in Germany, Austria and Switzerland.

Comparison of the performance of SAPS II, SAPS 3, APACHE II, and their customized prognostic models in a surgical intensive care unit

BACKGROUND

The Simplified Acute Physiology Score (SAPS) 3 has recently been developed, but not yet validated in surgical intensive care unit (ICU) patients. We compared the performance of SAPS 3 with SAPS II and the Acute Physiology and Chronic Health Evaluation (APACHE) II score in surgical ICU patients.

METHODS

Prospectively collected data from all patients admitted to a German university hospital postoperative ICU between August 2004 and December 2005 were analysed. The probability of ICU mortality was calculated for SAPS II, APACHE II, adjusted APACHE II (adj-APACHE II), SAPS 3, and SAPS 3 customized for Europe [C-SAPS3 (Eu)] using standard formulas. To improve calibration of the prognostic models, a first-level customization was performed, using logistic regression on the original scores, and the corresponding probability of ICU death was calculated for the customized scores (C-SAPS II, C-SAPS 3, and C-APACHE II).

RESULTS

The study included 1851 patients. Hospital mortality was 9%. Hosmer and Lemeshow statistics showed poor calibration for SAPS II, APACHE II, adj-APACHE II, SAPS 3, and C-SAPS 3 (Eu), but good calibration for C-SAPS II, C-APACHE II, and C-SAPS 3. Discrimination was generally good for all models [area under the receiver operating characteristic curve ranged from 0.78 (C-APACHE II) to 0.89 (C-SAPS 3)]. The C-SAPS 3 score appeared to have the best calibration curve on visual inspection.

CONCLUSIONS

In this group of surgical ICU patients, the performance of SAPS 3 was similar to that of APACHE II and SAPS II. Customization improved the calibration of all prognostic models.

Transferring critically ill patients out of hospital improves the standardized mortality ratio: a simulation study

BACKGROUND

Transferring critically ill patients to other acute care hospitals may artificially impact benchmarking measures. We sought to quantify the effect of out-of-hospital transfers on the standardized mortality ratio (SMR), an outcome-based measure of ICU performance.

METHODS

We performed a cohort study and Monte Carlo simulation using data from 85 ICUs participating in the acute physiology and chronic health evaluation (APACHE) clinical information system from 2002 to 2003. The SMR (observed divided by expected hospital mortality) was calculated for each ICU using APACHE IV risk adjustment. A set number of patients was randomly assigned to be transferred out alive rather than experience their original outcome. The SMR was recalculated, and the mean simulated SMR was compared to the original.

RESULTS

The mean (+/- SD) baseline SMR was 1.06 +/- 0.19. In the simulation, increasing the number of transfers by 2% and 6% over baseline decreased the SMR by 0.10 +/- 0.03 and 0.14 +/- 0.03, respectively. At a 2% increase, 27 ICUs had a decrease in SMR of > 0.10, and two ICUs had a decrease in SMR of > 0.20. Transferring only one additional patient per month was enough to create a bias of > 0.1 in 27 ICUs.

CONCLUSIONS

Increasing the number of acute care transfers by a small amount can significantly bias the SMR, leading to incorrect inference about ICU quality. Sensitivity to the variation in hospital discharge practices greatly limits the use of the SMR as a quality measure.

Prognosis in critical care

Prognostic risk prediction models have been employed in the intensive care unit (ICU) setting since the 1980s and provide health care providers with important information to help inform decisions related to treatment and prognosis, as well as to compare outcomes across institutions. Prognostic models for critical care are among the most widely utilized and tested predictive models in healthcare. In this article, we review and compare mortality prediction models, including the APACHE (1981), SAPS (1984), APACHE-II (1985), MPM (1987), APACHE-III (1991), SAPS-II (1993), and MPM-II (1993). We emphasize the importance of model calibration in this domain. In addition, we present a brief review of the statistical methodology, multiple logistic regression, which underlies most of the models currently used in critical care.

The evolution of intensive care unit performance assessment

Intensive care units (ICUs) share the problems experienced by the health care system at large. Various approaches to define and manage the quality of care patients receive in the ICU have been proposed. Performance measurement involves the collection of data to evaluate an ICU's performance against itself (over time), other ICUs, or other appropriate benchmarks. Successful performance assessment requires the quantification of relevant indexes of performance. Although these indexes are increasingly being developed, it will be some time before widely recognized, validated systems are available.

Methods to assess performance of models estimating risk of death in intensive care patients: a review

Models that estimate the probability of death of intensive care unit patients can be used to stratify patients according to the severity of their condition and to control for casemix and severity of illness. These models have been used for risk adjustment in quality monitoring, administration, management and research and as an aid to clinical decision making. Models such as the Mortality Prediction Model family, SAPS II, APACHE II, APACHE III and the organ system failure models provide estimates of the probability of in-hospital death of ICU patients. This review examines methods to assess the performance of these models. The key attributes of a model are discrimination (the accuracy of the ranking in order of probability of death) and calibration (the extent to which the model's prediction of probability of death reflects the true risk of death). These attributes should be assessed in existing models that predict the probability of patient mortality, and in any subsequent model that is developed for the purposes of estimating these probabilities. The literature contains a range of approaches for assessment which are reviewed and a survey of the methodologies used in studies of intensive care mortality models is presented. The systematic approach used by Standards for Reporting Diagnostic Accuracy provides a framework to incorporate these theoretical considerations of model assessment and recommendations are made for evaluation and presentation of the performance of models that estimate the probability of death of intensive care patients.

Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today's critically ill patients

OBJECTIVE

To improve the accuracy of the Acute Physiology and Chronic Health Evaluation (APACHE) method for predicting hospital mortality among critically ill adults and to evaluate changes in the accuracy of earlier APACHE models.

DESIGN

: Observational cohort study.

SETTING

A total of 104 intensive care units (ICUs) in 45 U.S. hospitals.

PATIENTS

A total of 131,618 consecutive ICU admissions during 2002 and 2003, of which 110,558 met inclusion criteria and had complete data.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

We developed APACHE IV using ICU day 1 information and a multivariate logistic regression procedure to estimate the probability of hospital death for randomly selected patients who comprised 60% of the database. Predictor variables were similar to those in APACHE III, but new variables were added and different statistical modeling used. We assessed the accuracy of APACHE IV predictions by comparing observed and predicted hospital mortality for the excluded patients (validation set). We tested discrimination and used multiple tests of calibration in aggregate and for patient subgroups. APACHE IV had good discrimination (area under the receiver operating characteristic curve = 0.88) and calibration (Hosmer-Lemeshow C statistic = 16.9, p = .08). For 90% of 116 ICU admission diagnoses, the ratio of observed to predicted mortality was not significantly different from 1.0. We also used the validation data set to compare the accuracy of APACHE IV predictions to those using APACHE III versions developed 7 and 14 yrs previously. There was little change in discrimination, but aggregate mortality was systematically overestimated as model age increased. When examined across disease, predictive accuracy was maintained for some diagnoses but for others seemed to reflect changes in practice or therapy.

CONCLUSIONS

APACHE IV predictions of hospital mortality have good discrimination and calibration and should be useful for benchmarking performance in U.S. ICUs. The accuracy of predictive models is dynamic and should be periodically retested. When accuracy deteriorates they should be revised and updated.

Risk Scoring in Perioperative and Surgical Intensive Care Patients: A Review

PURPOSE

Assessing the risk and predicting the outcome of surgery, trauma, and surgical intensive care is an important aspect of perioperative practice. There have been attempts to devise and validate many scoring systems to predict the prognosis of patients having a similar severity of illness. This article reviews some of the commonly used systems with respect to their development, strengths, and limitations.

SOURCES

Published literature describing risk assessment scores and physiologic scoring systems for preoperative assessment, trauma, and surgical intensive care patients.

PRINCIPAL FINDINGS

Risk scores used in preoperative evaluation assist the clinician in optimizing the patient before, during, and after surgery. Scoring systems applied in intensive care units are useful as guidelines rather than accurate predictors of prognosis for individual patient. Many models are used for audit purposes, and some are used as performance measures and quality indicators of a unit; however, both utilities are controversial because of poor adjustment of these systems to case-mixtures.

CONCLUSIONS

Risk assessment scores may assist in the perioperative risk evaluation with respect to organ systems. Prognostication of critically ill patients belonging to a category of illness may be done using physiological scoring systems taking into account the difference in the case-mix of the particular unit.

Do we have reliable biochemical markers to predict the outcome of critical illness?

Current outcome prediction in critically ill patients relies on the art of clinical judgement and/or the science of prognostication using illness severity scores. The biochemical processes underlying critical illness have increasingly been unravelled. Several biochemical markers reflecting the process of inflammation, immune dysfunction, impaired tissue oxygenation and endocrine alterations have been evaluated for their predictive power in small subpopulations of critically ill patients. However, none of these parameters has been validated in large populations of unselected ICU patients as has been done for the illness severity and organ failure scores. A simple biochemical predictor of ICU mortality will probably remain elusive because the processes underlying critical illness are very complex and heterogeneous. Future prognostic models will need to be far more sophisticated.

Profit and loss analysis for an intensive care unit (ICU) in Japan: a tool for strategic management

Background

Accurate cost estimate and a profit and loss analysis are necessary for health care practice. We performed an actual financial analysis for an intensive care unit (ICU) of a university hospital in Japan, and tried to discuss the health care policy and resource allocation decisions that have an impact on critical intensive care.

Methods

The costs were estimated by a department level activity based costing method, and the profit and loss analysis was based on a break-even point analysis. The data used included the monthly number of patients, the revenue, and the direct and indirect costs of the ICU in 2003.

Results

The results of this analysis showed that the total costs of US$ 2,678,052 of the ICU were mainly incurred due to direct costs of 88.8%. On the other hand, the actual annual total patient days in the ICU were 1,549 which resulted in revenues of US$ 2,295,044. However, it was determined that the ICU required at least 1,986 patient days within one fiscal year based on a break-even point analysis. As a result, an annual deficit of US$ 383,008 has occurred in the ICU.

Conclusion

These methods are useful for determining the profits or losses for the ICU practice, and how to evaluate and to improve it. In this study, the results indicate that most ICUs in Japanese hospitals may not be profitable at the present time. As a result, in order to increase the income to make up for this deficit, an increase of 437 patient days in the ICU in one fiscal year is needed, and the number of patients admitted to the ICU should thus be increased without increasing the number of beds or staff members. Increasing the number of patients referred from cooperating hospitals and clinics therefore appears to be the best strategy for achieving these goals.

Statin therapy prior to ICU admission: protection against infection or a severity marker?

OBJECTIVE

Examine the impact of previous statin therapy on hospital mortality and whether it is due to a protective effect against ICU-acquired infections.

DESIGN AND SETTING

Cohort comparison study by retrospective chart-based analysis in a 26-bed, university-affiliated, medical-surgical ICU.

PATIENTS

We analyzed data from 438 patients at high risk of ICU-acquired infections, i.e., those receiving mechanical ventilation for more than 96 h, 38 (8.7%) of whom had been treated with statins prior to and during ICU admission.

MEASUREMENTS AND RESULTS

We recorded clinical characteristics, number and type of ICU-acquired infections, and ICU and hospital mortality. Statin-treated patients were older (71.7+/-8.3 vs. 61.5+/-18.3 years), but differences in predicted mortality risk by APACHE II (39.5+/-24.7 vs. 35.8+/-24.3%) did not reach statistical significance. The ICU-acquired infection rate in statin-treated patients was nonsignificantly lower (29% vs. 38%) and delayed (median 12 vs.10 days), without differences regarding the source of infections. Nevertheless, hospital mortality was significantly higher in statin-treated patients (61% vs. 42%), even after adjustment for APACHE II predicted risk (observed/expected ratio 1.53 vs. 1.17).

CONCLUSIONS

Statin therapy is associated with worse outcome, probably because underlying clinical conditions are insufficiently considered in mortality predictors. Its presumed protective effect against ICU infections remains unconfirmed.

The use of routine laboratory data to predict in-hospital death in medical admissions

The ability to predict clinical outcomes in the early phase of a patient's hospital admission could facilitate the optimal use of resources, might allow focused surveillance of high-risk patients and might permit early therapy. We investigated the hypothesis that the risk of in-hospital death of general medical patients can be modelled using a small number of commonly used laboratory and administrative items available within the first few hours of hospital admission. Matched administrative and laboratory data from 9497 adult hospital discharges, with a hospital discharge specialty of general medicine, were divided into two subsets. The dataset was split into a single development set, Q(1) (n=2257), and three validation sets, Q(2), Q(3) and Q(4) (n(1)=2335, n(2)=2361, n(3)=2544). Hospital outcome (survival/non-survival) was obtained for all discharges. An outcome model was constructed from binary logistic regression of the development set data. The goodness-of-fit of the model for the validation sets was tested using receiver-operating characteristics curves (c-index) and Hosmer-Lemeshow statistics. Application of the model to the validation sets produced c-indices of 0.779 (Q(2)), 0.764 (Q(3)) and 0.757 (Q(4)), respectively, indicating good discrimination. Hosmer-Lemeshow analysis gave chi(2)=9.43 (Q(2)), chi(2)=7.39 (Q(3)) and chi(2)=8.00 (Q(4)) (p-values of 0.307, 0.495 and 0.433) for 8 degrees of freedom, indicating good calibration. The finding that the risk of hospital death can be predicted with routinely available data very early on after hospital admission has several potential uses. It raises the possibility that the surveillance and treatment of patients might be categorised by risk assessment means. Such a system might also be used to assess clinical performance, to evaluate the benefits of introducing acute care interventions or to investigate differences between acute care systems.

Improving the ICU: part 1

ICUs are a vital component of modern health care. Improving ICU performance requires that we shift from a paradigm that concentrates on individual performance to a different paradigm that emphasizes the need to assess and improve ICU systems and processes. This is the first part of a two-part treatise. It discusses existing problems in ICU care, and the methods for defining and measuring ICU performance.

Studying outcomes of intensive care unit survivors: measuring exposures and outcomes

BACKGROUND

Measurement of long-term outcomes and the patient and intensive care unit (ICU) factors predicting them present investigators with unique challenges. There is little systematic guidance for measuring these outcomes and exposures within the ICU setting. As a result measurement methods are often variable and non-comparable across studies.

METHODS

We use examples from the critical care literature to describe measurement as it relates to three key elements of clinical studies: subjects, outcomes and exposures, and time. Using this framework we review the principles and challenges of measurement and make recommendations for long-term outcomes research in the field of critical care medicine.

DISCUSSION

Relevant challenges discussed include: (a) selection bias and heterogeneity of ICU research subjects, (b) appropriate selection and measurement of outcome and exposure variables, and (c) accounting for the effect of time in the exposure-outcome relationship, including measurement of baseline data and time-varying variables.

CONCLUSIONS

Addressing these methodological challenges will advance research aimed at improving the long-term outcomes of ICU survivors.

Adverse effect on a referral intensive care unitʼs performance of accepting patients transferred from another intensive care unit*

OBJECTIVE

To determine whether observed and predicted mortality for intensive care unit (ICU) transfer admissions is different from non-ICU transfer admissions and how that might affect ICU performance evaluation.

DESIGN, SETTING, AND PATIENTS

We retrospectively analyzed the charts of 3,416 patients admitted to our tertiary referral ICU from January 1995 to December 2001 and evaluated the effect on our performance (based on the Simplified Acute Physiology Score II risk model) of accepting patients transferred from another hospital's ICU.

MAIN RESULTS

During the study period, 597 patients (17%) had been transferred from a non-ICU setting in another hospital (hospital transfer) and 408 (12%) from another hospital's ICU (ICU transfer). ICU mortality and standardized mortality ratios were significantly higher for ICU-transfer patients than for hospital-transfer or directly admitted patients: 34% vs. 23% vs. 17% (p < .0001) and 0.95 (95% confidence interval, 0.83-1.08), 0.82 (95% confidence interval, 0.71-0.95), and 0.62 (95% confidence interval, 0.55-0.68), respectively. ICU-transfer patients had 3.6-fold longer mean ICU stays and 1.9-fold longer durations of mechanical ventilation than directly admitted patients. Hospital-transfer (odds ratio = 1.89) and ICU-transfer patients (odds ratio = 2.41) had significantly higher mortality rates, even after adjustment for case mix and disease severity. Consequently, a benchmarking program adjusting only for these latter variables, but not admission source, would penalize our ICU by 39 excess deaths per 1,000 admissions as compared with another ICU admitting no transfer patients. Finally, patients transferred from the ward of another hospital had significantly higher mortality rates (odds ratio = 1.56) as compared with patients directly admitted from the ward of our hospital, confirming the "transfer effect" for this homogeneous patients' subgroup.

CONCLUSIONS

Admission source remains a strong and independent predictor of ICU death, despite adjustment for case mix and disease severity at ICU admission. Specifically, accepting numerous ICU-transfer patients, for whom the probability of ICU death is the most underestimated by a system adjusting only for case mix and disease severity, can adversely affect the evaluation of referral centers' performance. Future benchmarking and profiling systems should evaluate and adequately account for the ICU-transfer factor to provide healthcare payers and consumers with more accurate and valid information on the true performance of referral centers.

Resource utilization and outcome in gravely ill intensive care unit patients with predicted in-hospital mortality rates of 95% or higher by APACHE III scores: the relationship with physician and family expectations

OBJECTIVE

To assess resource utilization and outcome in gravely ill patients admitted to an intensive care unit (ICU) and the potential association with health care workers' and family members' expectations.

PATIENTS AND METHODS

We retrospectively evaluated ICU patients with a predicted in-hospital mortality rate of 95% or higher (PM95) using the Acute Physiology and Chronic Health Evaluation III (APACHE III) on 2 consecutive days. All patients were admitted to a single institution between September 30, 1994, and August 9, 2001.

RESULTS

The APACHE III database contained data from 38,165 ICU patients during the study interval. Of these, 248 (0.65% of ICU admissions) achieved PM95 status and were included in the study. Between PM95 and hospital discharge, resource utilization (eg, blood transfusion, hemodialysis, surgery, and computed tomography or magnetic resonance imaging) was extensive. A total of 23% of patients survived to hospital discharge, yet all but 1 were moderately or severely disabled. One year after achieving PM95, 10% (95% confidence interval, 7%-15%) of patients were alive. For 229 patients, the medical records contained physician documentation that indicated a likely fatal outcome. Thirty-six of these medical records documented unrealistic family expectations of a good outcome. The latter finding correlated with increased resource utilization without significant improvement in 1-year survival. In contrast, absence of physician documentation of a likely fatal outcome In 19 patients correlated with an improved likelihood of hospital (74%) and 1-year (47%) survival.

CONCLUSION

Despite better-than-predicted survival outcomes, patient functionality and 1-year survival were poor. Unrealistic family expectations were associated with increased resource utilization without significant survival benefit, whereas absence of physician documentation of likely impending death (which correlated with improved survival) may denote the prognostication skills of experienced clinicians.

SAPS�II revisited

OBJECTIVE

To construct and validate an update of the Simplified Acute Physiology Score II (SAPS II) for the evaluation of clinical performance of Intensive Care Units (ICU).

DESIGN AND SETTING

Retrospective analysis of prospectively collected multicenter data in 32 ICUs located in the Paris area belonging to the Cub-Rea database and participating in a performance evaluation project.

PATIENTS

33,471 patients treated between 1999 and 2000.

MEASUREMENTS AND RESULTS

Two logistic regression models based on SAPS II were developed to estimate in-hospital mortality among ICU patients. The second model comprised reevaluation of original items of SAPS II and integration of the preadmission location and chronic comorbidity. Internal and external validation were performed. In the two validation samples the most complex model had better calibration than the original SAPS II for in-hospital mortality but its discrimination was not significantly higher (area under ROC curve 0.89 vs. 0.87 for SAPS II). Second-level customization and integration of new items improved uniformity of fit for various categories of patients except for diagnosis-related groups. The rank order of ICUs was modified according to the model used.

CONCLUSIONS

The overall performance of SAPS II derived models was good, even in the context of a community cohort and routinely gathered data. However, one-half the variation of outcome remains unexplained after controlling for admission characteristics, and uniformity of prediction across diagnostic subgroups was not achieved. Differences in case-mix still limit comparisons of quality of care.

The profit motive and spine surgery

STUDY DESIGN

The profit motive and market medicine have had a significant impact on clinical practice and research in the field of spine surgery. An overview of current concerns is presented.

OBJECTIVE

The objective of this study was to provide those involved in the study and treatment of spinal disorders with a critical overview of the effects of the profit motive on our practices.

SUMMARY OF BACKGROUND DATA

Historically, the profit motive has been viewed as eroding the standards of spine surgery, encouraging surgeons to operate aggressively and researchers to bias their results. Although there are legitimate concerns regarding the role played by such market forces, the profit motive exerts several quite positive effects on spine surgery as well.

METHODS

Negative and positive aspects of the profit motive in spine surgery are explored along with alternative approaches.

RESULTS

The profit motive in spine surgery can result in unnecessary surgery, as well as the push to market of unproven technologies. Yet, without a robust profit motive, it is unclear where sufficient funding could be found to support research and education, and to underwrite the advancement of new technologies.

CONCLUSIONS

The profit motive significantly influences the way we practice and conduct research in spine surgery. To minimize the negative aspects of the profit motive, spine surgeons and researchers must refrain from being used by companies to rush products to market and/or compromising patient care out of self-interest.

Severe sepsis epidemiology: sampling, selection, and society

Three new articles in Critical Care add to an expanding body of information on the epidemiology of severe sepsis. Although there have been a range of approaches to estimate the incidence of severe sepsis, most studies report severe sepsis in about 10 ± 4% of ICU patients with a population incidence of 1 ± 0.5 cases per 1000. Importantly, the availability of ICU services may well determine the number of treated cases of severe sepsis, and it seems clear that these studies are reporting the treated incidence, not the incidence, of severe sepsis. In the future, we must focus on whether all severe sepsis should be treated, and, consequently, what level of ICU services is optimal.

Can generic paediatric mortality scores calculated 4 hours after admission be used as inclusion criteria for clinical trials?

INTRODUCTION

Two generic paediatric mortality scoring systems have been validated in the paediatric intensive care unit (PICU). Paediatric RISk of Mortality (PRISM) requires an observation period of 24 hours, and PRISM III measures severity at two time points (at 12 hours and 24 hours) after admission, which represents a limitation for clinical trials that require earlier inclusion. The Paediatric Index of Mortality (PIM) is calculated 1 hour after admission but does not take into account the stabilization period following admission. To avoid these limitations, we chose to conduct assessments 4 hours after PICU admission. The aim of the present study was to validate PRISM, PRISM III and PIM at the time points for which they were developed, and to compare their accuracy in predicting mortality at those times with their accuracy at 4 hours.

METHODS

All children admitted from June 1998 to May 2000 in one tertiary PICU were prospectively included. Data were collected to generate scores and predictions using PRISM, PRISM III and PIM.

RESULTS

There were 802 consecutive admissions with 80 deaths. For the time points for which the scores were developed, observed and predicted mortality rates were significantly different for the three scores (P < 0.01) whereas all exhibited good discrimination (area under the receiver operating characteristic curve >or=0.83). At 4 hours after admission only the PIM had good calibration (P = 0.44), but all three scores exhibited good discrimination (area under the receiver operating characteristic curve >or=0.82).

CONCLUSIONS

Among the three scores calculated at 4 hours after admission, all had good discriminatory capacity but only the PIM score was well calibrated. Further studies are required before the PIM score at 4 hours can be used as an inclusion criterion in clinical trials.

Risk assessment for inpatient survival in the long-term acute care setting after prolonged critical illness

OBJECTIVE

The past decade has witnessed growth in the long-term acute care (LTAC) hospital industry. There are no reliable risk assessment models that can adjust outcomes across such facilities with different criteria for admitting patients. Variation in reported outcomes makes it difficult to determine whether a patient, or group of patients, may benefit from such care. This study sought to determine the extent to which survival in the LTAC setting is associated with age, race, residual organ system failures (OSFs), or APACHE (acute physiology and chronic health evaluation) III scores at the time of admission to LTAC.

DESIGN

Retrospective medical record review.

SETTING

Four freestanding facilities of a LTAC hospital.

PATIENTS

A sample of 300 hospital admissions weighted to represent the study hospital population.

MEASUREMENTS

Inpatient survival modeled as a function of age, APACHE III score calculated within 72 h prior to LTAC admission, and residual OSFs present on admission to LTAC.

RESULTS

Logistic regression analysis shows age and OSF were most predictive of inpatient survival (receiver operating characteristic curve area = 0.81). APACHE III score was not predictive of survival in the multivariate model.

CONCLUSIONS

Survival in LTAC is primarily associated with age and OSFs, which should be used to adjust for patient populations among LTAC settings when comparing outcomes. Our model identifies a group of patients with the poorest likelihood of survival in the LTAC setting, and may be used to facilitate dialogue with patients and family in cases where continued aggressive care is least effective.

External validation of the SAPS II, APACHE II and APACHE III prognostic models in South England: a multicentre study

OBJECTIVE

External validation of three prognostic models in adult intensive care patients in South England. DESIGN. Prospective cohort study.

SETTING

Seventeen intensive care units (ICU) in the South West Thames Region in South England.

PATIENTS AND PARTICIPANTS

Data of 16646 patients were analysed.

INTERVENTIONS

None.

MEASUREMENTS AND RESULTS

We compared directly the predictive accuracy of three prognostic models (SAPS II, APACHE II and III), using formal tests of calibration and discrimination. The external validation showed a similar pattern for all three models tested: good discrimination, but imperfect calibration. The areas under the receiver operating characteristics (ROC) curves, used to test discrimination, were 0.835 and 0.867 for APACHE II and III, and 0.852 for the SAPS II model. Model calibration was assessed by Lemeshow-Hosmer C-statistics and was Chi(2 )=232.1 for APACHE II, Chi(2 )=443.3 for APACHE III and Chi(2 )=287.5 for SAPS II.

CONCLUSIONS

Disparity in case mix, a higher prevalence of outcome events and important unmeasured patient mix factors are possible sources for the decay of the models' predictive accuracy in our population. The lack of generalisability of standard prognostic models requires their validation and re-calibration before they can be applied with confidence to new populations. Customisation of existing models may become an important strategy to obtain authentic information on disease severity, which is a prerequisite for reliably measuring and comparing the quality and cost of intensive care.

A comparison of management and short-term outcomes of acute myocardial infarction patients admitted to coronary care units and medical wards: the importance of case mix

BACKGROUND: Variation in case mix of patients can significantly influence outcome. In this study, the management and outcomes of patients with acute myocardial infarction who were admitted either to coronary care units or to internal medicine wards were examined. METHODS: A nationwide prospective study was performed during a 2-month period in all 26 coronary care units and in 82 of 96 internal medicine wards in Israel. All patients with a discharge diagnosis of acute myocardial infarction were included. Comorbidity was coded using the Index of Coexistent Diseases. RESULTS: A total of 1648 consecutive patients with acute myocardial infarction were identified. One thousand and eighty eight (66%) were admitted to coronary care units and 560 (34%) to internal medicine wards. The 30-day mortality for the coronary care unit group was 9.2% compared to 15.5% for patients in the internal medicine ward group. Using logistic regression, independent factors determining 30-day mortality were (odds ratio and 95% confidence interval): age (1.06 per year, 1.03-1.08), Killip score (2.09, 1.64-2.67), Q wave acute myocardial infarction (2.12, 1.31-3.43), and Index of Coexistent Diseases score (1.42, 1.12-1.80). After controlling for age, infarct type and severity, and coexisting medical conditions, no excess mortality was detected among patients admitted to internal medicine wards. CONCLUSIONS: Variance in the case mix has a great influence on the interpretation of mortality in studies of acute myocardial infarction.

Respiratory intensive care unit resource use and outcomes in Taiwan

Based on a revision of Donabedian's classic structure, process, and outcome conceptual framework, this study examined the relationship between resource use (length of stay) and outcome (transfer status) in two respiratory intensive care units (ICUs). Medical records of respiratory ICU patients (N = 194) from a medical center in northern Taiwan were reviewed. Data collection focused on patient demographic profile (age, gender, and medical diagnosis), Acute Physiology and Chronic Health Evaluation (APACHE) score, nursing diagnoses, ICU length of stay, and transfer status. The results indicate that patients with lower APACHE scores and more nursing diagnoses had a longer ICU length of stay. In addition, both the number of nursing diagnoses and APACHE scores significantly explained the variance in the length of stay. Nonetheless, a higher APACHE score was correlated with a poor transfer status. These findings indicate that, in addition to the traditional indicators, nursing diagnoses may be a vital variable in predicting ICU length of stay. The results also imply that patients with lower APACHE scores are in better physical condition and are therefore institutionalized longer in ICUs.

Identifying quality outliers in a large, multiple-institution database by using customized versions of the Simplified Acute Physiology Score II and the Mortality Probability Model II0

OBJECTIVE

To assess whether customized versions of the Simplified Acute Physiology Score (SAPS) II and the Mortality Probability Model (MPM) II0 agree on the identity of intensive care unit quality outliers within a multiple-center database.

DESIGN

Retrospective database analysis.

SETTING AND PATIENTS

Patient subset of the Project IMPACT database consisting of 39,617 adult patients admitted to surgical, medical, and mixed surgical-medical intensive care units at 54 hospitals between 1995 and 1999 who met inclusion criteria for SAPS II and MPM II0.

INTERVENTIONS

Customized versions of SAPS II and MPM II0 were obtained by fitting new logistic regressions to the data by using the risk score as the independent variable and outcome at hospital discharge as the dependent variable. The data set was divided randomly into a training set and a validation set. Each model was customized by using the training set; model performance was then assessed in the validation set by using the area under the receiver operating characteristic curve and the Hosmer-Lemeshow statistic. The final models were based on the entire data set. The level of agreement between the customized models on the identity of quality outliers was evaluated by using kappa analysis.

MEASUREMENTS AND MAIN RESULTS

Both customized models exhibited good discrimination and good calibration in this database. The area under the receiver operating characteristic curve was 0.83 for MPM II0 and 0.872 for SAPS II following model customization. The Hosmer-Lemeshow statistic was 12.3 ( >.14) for MPM II0, and 8.17 (p >.42) for SAPS II, after customization. Kappa analysis showed only fair agreement between the two customized models with regard to the identity of the quality outliers: kappa = 0.44 (95% confidence interval, 0.24, 0.65).

CONCLUSIONS

Customization of SAPS II and MPM II0 to the Project IMPACT database resulted in well-calibrated models. Despite this, the models exhibited only a moderate level of agreement in which hospitals were designated as quality outliers. Seventeen of the 54 hospitals were categorized differently depending on which of the two scoring systems was used. Therefore, the rating of quality of care appears, in part, to be a function of the prediction model used.

Rating the quality of intensive care units: is it a function of the intensive care unit scoring system?

OBJECTIVE

Intensive care units (ICUs) use severity-adjusted mortality measures such as the standardized mortality ratio to benchmark their performance. Prognostic scoring systems such as Acute Physiology and Chronic Health Evaluation (APACHE) II, Simplified Acute Physiology Score II, and Mortality Probability Model II0 permit performance-based comparisons of ICUs by adjusting for severity of disease and case mix. Whether different risk-adjustment methods agree on the identity of ICU quality outliers within a single database has not been previously investigated. The objective of this study was to determine whether the identity of ICU quality outliers depends on the ICU scoring system used to calculate the standardized mortality ratio.

DESIGN, SETTING, PATIENTS

Retrospective cohort study of 16,604 patients from 32 hospitals based on the outcomes database (Project IMPACT) created by the Society of Critical Care Medicine. The ICUs were a mixture of medical, surgical, and mixed medical-surgical ICUs in urban and nonurban settings. Standardized mortality ratios for each ICU were calculated using APACHE II, Simplified Acute Physiology Score II, and Mortality Probability Model II. ICU quality outliers were defined as ICUs whose standardized mortality ratio was statistically different from 1. Kappa analysis was used to determine the extent of agreement between the scoring systems on the identity of hospital quality outliers. The intraclass correlation coefficient was calculated to estimate the reliability of standardized mortality ratios obtained using the three risk-adjustment methods.

MEASUREMENTS AND MAIN RESULTS

Kappa analysis showed fair to moderate agreement among the three scoring systems in identifying ICU quality outliers; the intraclass correlation coefficient suggested moderate to substantial agreement between the scoring systems. The majority of ICUs were classified as high-performance ICUs by all three scoring systems. All three scoring systems exhibited good discrimination and poor calibration in this data set.

CONCLUSION

APACHE II, Simplified Acute Physiology Score II, and Mortality Probability Model II0 exhibit fair to moderate agreement in identifying quality outliers. However, the finding that most ICUs in this database were judged to be high-performing units limits the usefulness of these models in their present form for benchmarking.

The effects of two methods for customising the original SAPS II model for intensive care patients from South England

Model customisation is used to adjust prognostic models by re-calibrating them to obtain more reliable mortality estimates. We used two methods for customising the Simplified Acute Physiology Score II model for 15,511 intensive care patients by altering the logit and the coefficients of the original equation. Both methods significantly improved model calibration, but customising the coefficients was slightly more effective. The Hosmer-Lemeshow chi(2)-value improved from 306.0 (p< 0.001) before, to 14.5 (p < 0.07) and 23.3 (p < 0.06) after customisation of the coefficients and the logit, respectively. Discrimination was not affected. The standardised mortality ratio for the entire population declined from 1.16 (95% confidence interval: 1.13-1.20, p < 0.001) to 0.99 (95% confidence interval: 0.96-1.02, p < 0.22) after customisation of the coefficients. The uniformity-of-fit for patients grouped by operative status and comorbidities also improved, but remained imperfect for patients stratified by location before intensive care unit admission. Amalgamation of large, regional databases could provide the basis for the re-calibration of standard prognostic models, which could then be used as a national reference system to allow more reliable comparisons of the efficacy and quality of care based on severity adjusted outcome measures.

Prospective independent validation of APACHE III models in an Australian tertiary adult intensive care unit

Evaluation of the performance of the APACHE III (Acute Physiology and Chronic Health Evaluation) ICU (intensive care unit) and hospital mortality models at the Princess Alexandra Hospital, Brisbane is reported. Prospective collection of demographic, diagnostic, physiological, laboratory, admission and discharge data of 5681 consecutive eligible admissions (1 January 1995 to 1 January 2000) was conducted at the Princess Alexandra Hospital, a metropolitan Australian tertiary referral medical/surgical adult ICU ROC (receiver operating characteristic) curve areas for the APACHE III ICU mortality and hospital mortality models demonstrated excellent discrimination. Observed ICU mortality (9.1%) was significantly overestimated by the APACHE III model adjusted for hospital characteristics (10.1%), but did not significantly differ from the prediction of the generic APACHE III model (8.6%). In contrast, observed hospital mortality (14.8%) agreed well with the prediction of the APACHE III model adjusted for hospital characteristics (14.6%), but was significantly underestimated by the unadjusted APACHE III model (13.2%). Calibration curves and goodness-of-fit analysis using Hosmer-Lemeshow statistics, demonstrated that calibration was good with the unadjusted APACHE III ICU mortality model, and the APACHE III hospital mortality model adjusted for hospital characteristics. Post hoc analysis revealed a declining annual SMR (standardized mortality rate) during the study period. This trend was present in each of the non-surgical, emergency and elective surgical diagnostic groups, and the change was temporally related to increased specialist staffing levels. This study demonstrates that the APACHE III model performs well on independent assessment in an Australian hospital. Changes observed in annual SMR using such a validated model support an hypothesis of improved survival outcomes 1995-1999.

A cross-cultural comparison of critical care delivery: Japan and the United States

OBJECTIVE

To compare the utilization and outcomes of critical care services in a cohort of hospitals in the United States and Japan.

DESIGN

Prospective data collection on 5,107 patients and detailed organizational characteristics from each of the participating Japanese study hospitals between 1993 and 1995, with comparisons made to prospectively collected data on the 17,440 patients included in the US APACHE (acute physiology and chronic health evaluation) III database.

SETTING

Twenty-two Japanese and 40 US hospitals.

PATIENTS

Consecutive, unselected patients from medical, surgical, and mixed medical/surgical ICUs.

MEASUREMENTS

Severity of illness, predicted risk of in-hospital death, and ICU and hospital length of stay (LOS) were assessed using APACHE III. Japanese ICU directors completed a detailed survey describing their units.

MAIN RESULTS

US and Japanese ICUs have a similar array of modalities available for care. Only 1.0% (range, 0.56 to 2.7%) of beds in Japanese hospitals were designated as ICUs. The organization of the Japanese and US ICUs varied by hospital, but Japanese ICUs were more likely to be organized to care for heterogeneous diagnostic populations. Sample case-mix differences reflect different disease prevalence. ICU utilization for women is significantly lower (35.5% vs 44.8% of patients) and there were relatively fewer patients > or = 85 years old in the Japanese ICU cohort (1.2% vs 4.6%), despite a higher per capita rate of individuals > or = 85 years old in Japan. The utilization of ICUs for patients at low risk of death significantly less in Japan (10.2%) than in the United States (12.9%). The APACHE III score stratified patient risk. Overall mortality was similar in both national samples after accounting for differences in hospital LOS, utilizing a model that was highly discriminating (receiver operating characteristic, 0.87) when applied to the Japanese sample. The application of a US-based mortality model to a Japanese sample overestimated mortality across all but the highest (> 90%) deciles of risk. Significant variation in expected performance was noted between hospitals. Risk-adjusted ICU LOS was not significantly longer in Japan; however, total hospital stay was nearly twice that found in the US hospitals, reflecting differences in hospital utilization philosophies.

CONCLUSIONS

Similar high-technology critical care is available in both countries. Variations in ICU utilization reflect differences in case-mix and bed availability. Japanese ICU utilization by gender reflects differences in disease prevalence, whereas differences in utilization by age may reflect differences in cultural norms regarding the limits of care. Such differences provide context from which to assess the delivery of care across international borders. Miscalibration of predictive models applied to international data samples highlight the impact that differences in resource use and local practice cultures have on outcomes. Models may require modification in order to account for these differences. Nevertheless, with large databases, it is possible to assess critical care delivery systems between countries accounting for differences in case-mix, severity of illness, and cultural normative standards facilitating the design and management such systems.

The impact of low-risk intensive care unit admissions on mortality probabilities by SAPS II, APACHE II and APACHE III

A large proportion of intensive care unit patients are low-risk admissions. Mortality probabilities generated by predictive systems may not accurately reflect the mortality experienced by subpopulations of critically ill patients. We prospectively assessed the impact of low-risk admissions (mortality risk < 10%) on the mortality estimates generated by three prognostic models. We studied 1497 consecutive admissions to a general intensive care unit. The performance of the three models for subgroups and the whole population was analysed. The proportions of patients designated as low risk varied with the model and differences in model performance were most pronounced for these patients. The APACHE II mortality ratios (1.32 vs. 1.19) did not differ for low- and higher risk patients, but mortality ratios generated by APACHE III (2.38 vs. 1.23) and SAPS II (2.19 vs. 1.16) were nearly two-fold greater. Calibration for higher risk patients was similar for all three models but the APACHE III system calibrated worse than the other models for low-risk patients. This may have contributed to the poorer overall calibration of the APACHE III system (Hosmer-Lemeshow C-test: APACHE III chi(2) = 329; APACHE II chi(2) = 42; SAPS II chi(2) = 62). Imperfect characterisation of the large proportion of low-risk intensive care unit admissions may contribute to the deterioration of the models' predictive accuracies for the intensive care population as a whole.

Multicenter study of obstetric admissions to 14 intensive care units in southern England

OBJECTIVES

To identify pregnant and postpartum patients admitted to intensive care units (ICUs), the cause for their admission, and the proportion that might be appropriately managed in a high-dependency environment (HDU) by using an existing database. To estimate the goodness-of-fit for the Simplified Acute Physiology Score II, the Acute Physiology and Chronic Health Evaluation (APACHE) II, and the APACHE III scoring systems in the obstetrical population.

DESIGN

Retrospective analysis of demographic, diagnostic, treatment, and severity of illness data.

SETTING

Fourteen ICUs in Southern England.

PATIENTS

Pregnant or postpartum (<42 days) admissions between January 1, 1994, and December 31, 1996.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

We identified 210 patients, constituting 1.84% (210 of 11,385) of all ICU admissions and 0.17% (210 of 122,850) of all deliveries. Most admissions followed postpartum complications (hypertensive disease of pregnancy [39.5%] and major hemorrhage [33.3%]). Seven women were transferred to specialist ICUs. There was considerable variation between ICUs with respect to the number and type of interventions required by patients. Some 35.7% of patients stayed in ICU for <2 days and received no specific ICU interventions; these patients might have been safely managed in an HDU. There were seven maternal deaths (3.3%); fetal mortality rate was 20%. The area under the receiver operator characteristic curve and the standardized mortality ratio were 0.92 (confidence interval [CI], 0.85-0.99) and 0.43 for the Simplified Acute Physiology Score II, 0.94 (CI, 0.86-1.0) and 0.24 for APACHE II, and 0.98 (CI, 0.96-1.0) and 0.43 for APACHE III, respectively.

CONCLUSIONS

Existing databases can both identify critically ill obstetrical patients and provide important information about them. Obstetrical ICU admissions often require minimal intervention and are associated with low mortality rates. Many might be more appropriately managed in an HDU. The commonly used severity of illness scoring systems are good discriminators of outcome from intensive care admission in this group but may overestimate mortality rates. Severity of illness scoring systems may require modification in obstetrical patients to adjust for the normal physiologic responses to pregnancy.

Performance of APACHE III models in an Australian ICU

STUDY OBJECTIVE

Evaluation of the performance of the APACHE (acute physiology and chronic health evaluation) III ICU and hospital mortality models at an Australian tertiary adult ICU.

DESIGN

Noninterventional, observational study.

SETTING

Metropolitan, Australian, tertiary referral medical/surgical ICU.

PATIENTS

A total of 3,398 consecutive eligible admissions from January 1, 1995, to December 31, 1997.

MEASUREMENTS

Prospective collection of demographic, diagnostic, physiologic, laboratory, admission, and discharge data.

RESULTS

The patient sample was younger and more commonly male, with more comorbidities and a different operative and referral source mix, compared to the APACHE III development sample. Receiver operating characteristic curve areas for ICU (0.92) and hospital mortality (0.90) demonstrated excellent discrimination. Observed ICU mortality (9.9%) did not significantly differ from the prediction of the APACHE III model (8.9%) or the APACHE III model adjusted for hospital characteristics (10.5%). The hospital mortality (16.0%) was underestimated by the APACHE III model [13.6%; chi(2)(1) = 7.4; p = 0.01]. With proprietary adjustments for hospital characteristics (14.9%) or referenced to the US database (15.6%), agreement was closer. Good calibration was found with all models except the unadjusted hospital mortality model.

CONCLUSION

In contrast to other non-American studies, this Australian study demonstrates that the APACHE III can perform well on independent assessment. As perfect discrimination and calibration cannot coexist in a probabilistic model with dichotomous outcomes, performance of APACHE III models with proprietary adjustment for hospital characteristic provide a good compromise for use in quality surveillance.

Results of report cards for patients with congestive heart failure depend on the method used to adjust for severity

BACKGROUND

The validity of outcome report cards may depend on the ways in which they are adjusted for risk.

OBJECTIVES

To compare the predictive ability of generic and disease-specific survival prediction models appropriate for use in patients with heart failure, to simulate outcome report cards by comparing survival across hospitals and adjusting for severity of illness using these models, and to assess the ways in which the results of these comparisons depend on the adjustment method.

DESIGN

Analysis of data from a prospective cohort study.

SETTING

A university hospital, a Veterans Affairs (VA) medical center, and a community hospital.

PATIENTS

Sequential patients presenting in the emergency department with acute congestive heart failure.

MEASUREMENTS

Unadjusted 30-day and 1-year mortality across hospitals and 30-day and 1-year mortality adjusted by using disease-specific survival prediction models (two sickness-at-admission models, the Cleveland Health Quality Choice model, the Congestive Heart Failure Mortality Time-Independent Predictive Instrument) and generic models (Acute Physiology and Chronic Health Evaluation [APACHE] II, APACHE III, the mortality prediction model, and the Chadson comorbidity index).

RESULTS

The community hospital's unadjusted 30-day survival rate (85.0%) and the VA medical center's unadjusted 1-year survival rate (60.9%) were significantly lower than corresponding rates at the university hospital (92.7% and 67.5%, respectively). No severity model had excellent ability to discriminate patients by survival rates (all areas under the receiver-operating characteristic curve < 0.73). Whether the VA medical center, the community hospital, both, or neither had worse survival rates on simulated report cards than the university hospital depended on the prediction model used for adjustment.

CONCLUSIONS

Results of simulated outcome report cards for survival in patients with congestive heart failure depend on the method used to adjust for severity.