Evaluation of acute physiology and chronic health evaluation III predictions of hospital mortality in an independent database

Introduction: risk-adjustment issues in mental health services

State mental health authorities and other public and private entities are developing outcome measures and comparing results across providers, programs, and systems. To make comparisons equitable, outcomes must be risk adjusted. This article provides an introduction to mental health risk adjustment and outlines issues involved in the selection of outcome and risk variables, data collection protocols, and analytic methods. It stresses the importance of proper identification of risk-adjustment variables and models. The article concludes with the next steps necessary to develop a valid approach to the risk-adjustment methodology.

Comparing alternative risk-adjustment models

The use of mental health indicators to compare provider performance requires that comparisons be fair. Fair provider comparisons mean that scores are risk adjusted for client characteristics that influence scores and that are beyond provider control. Data for the study are collected from 336 outpatients receiving publicly funded mental health services in Washington State. The study compares alternative specifications of multiple regression-based risk-adjustment models to argue that the particular form of the model will lead to different conclusions about comparative treatment agency performance. In order to evaluate performance fairly it is necessary to not only incorporate risk adjustment, but also identify the most correct form that the risk-adjustment model should take. Future research is needed to specify; test, and validate the mental health risk-adjustment models best suited to particular treatment populations and performance indicators.

Who bounces back? Physiologic and other predictors of intensive care unit readmission

OBJECTIVE

To determine the influence of changes in acute physiology scores (APS) and other patient characteristics on predicting intensive care unit (ICU) readmission.

DESIGN

Secondary analysis of a prospective cohort study.

SETTING

Single large university medical intensive care unit.

PATIENTS

A total of 4,684 consecutive admissions from January 1, 1994, to April 1, 1998, to the medical ICU.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

The independent influence of patient characteristics, including daily APS, admission diagnosis, treatment status, and admission location, on ICU readmission was evaluated using logistic regression. After accounting for first ICU admission deaths, 3,310 patients were "at-risk" for ICU readmission and 317 were readmitted (9.6%). Hospital mortality was five times higher (43% vs. 8%; p < .0001), and length of stay was two times longer (16 +/- 16 vs. 32 +/- 28 days; p < .001) in readmitted patients. Mean discharge APS was significantly higher in the readmitted group compared with the not readmitted group (43 +/- 19 vs. 34 +/- 18; p > .01). Significant independent predictors of ICU readmission included discharge APS >40 (odds ratio [OR] 2.1; 95% confidence interval [CI] 1.6-2.7; p < .0001), admission to the ICU from a general medicine ward (Floor) (OR 1.9; 95% CI 1.4-2.6; p < .0001), and transfer to the ICU from other hospital (Transfer) (OR 1.7; 95% CI 1.3-2.3; p < .01). The overall model calibration and discrimination were (H-L chi2 = 3.8, df = 8; p = .85) and (receiver operating characteristic 0.67), respectively.

CONCLUSIONS

Patients readmitted to medical ICUs have significantly higher hospital lengths of stay and mortality. ICU readmissions may be more common among patients who respond poorly to treatment as measured by increased severity of illness at first ICU discharge and failure of prior therapy at another hospital or on a general medicine unit. Tertiary care ICUs may have higher than expected readmission rates and mortalities, even when accounting for severity of illness, if they care for significant numbers of transferred patients.

An evaluation of risk factors for mortality after burn trauma and the identification of gender-dependent differences in outcomes

BACKGROUND

The primary objective of this study was to determine an objective method for estimating the risk of mortality after burn trauma, and secondarily, to evaluate the relationship between gender and mortality, in the setting of a quantifiable inflammatory stimulus. Previously reported estimates of mortality risk after burn trauma may no longer be applicable, given the overall reduction in case-fatality rates after burn trauma. We expect that future advances in burn trauma research will require careful and ongoing quantification of mortality risk factors to measure the importance of newly identified factors and to determine the impact of new therapies. Conflicting clinical reports regarding the impact of gender on survival after sepsis and critical illness may in part, be from different study designs, patient samples, or failure to adequately control for additional factors contributing to the development ofsepsis and mortality.

STUDY DESIGN

Data from the prospectively maintained burn registry for patients admitted to the Parkland Memorial Hospital burn unit between January 1, 1989 and December 31, 1998 were analyzed. Logistic regression was used to generate estimates of the probability of death in half of the study sample, and this model was validated on the second half of the sample. Risk factors evaluated for their relationship with mortality were: age, inhalation injury, burn size, body mass (weight), preexisting medical conditions, nonburn injuries, and gender.

RESULTS

Of 4,927 patients, 5.3% died. The best model for estimating mortality included the percent of total body surface area burned; the percent of full-thickness burn size; the presence of an inhalation injury; age categories of: < 30 years, 30 to 59 years, > or = 60 years; and gender. The risk of death was approximately two-fold higher in women aged 30 to 59 years compared with men of the same age.

CONCLUSIONS

We have provided a detailed method for estimating the risk of mortality after burn trauma, based on a large, contemporary cohort of patients. These estimates were validated on a second sample and proved to predict mortality accurately. We have identified an increased mortality risk in women of 30 to 59 years of age.

Development of a program to manage costly outliers

The advanced practice nurse group devised a method to identify clinical variables of the elderly patients with multisystem failure requiring complex nursing care referred to as outlier. Outliers in this program were defined as patients whose hospital charges were greater than $50,000 with a length of stay greater than the primary diagnostic related group designated. Once criteria were identified, nursing strategies were developed to monitor the elderly patient, implement interventions, and evaluate patient outcomes. The goals of this program were to identify who the outliers might be prior to becoming outliers and to manage their nursing care early in their hospital course, attempting to match resource requirements with resource availability.

Performance of APACHE III models in an Australian ICU

STUDY OBJECTIVE

Evaluation of the performance of the APACHE (acute physiology and chronic health evaluation) III ICU and hospital mortality models at an Australian tertiary adult ICU.

DESIGN

Noninterventional, observational study.

SETTING

Metropolitan, Australian, tertiary referral medical/surgical ICU.

PATIENTS

A total of 3,398 consecutive eligible admissions from January 1, 1995, to December 31, 1997.

MEASUREMENTS

Prospective collection of demographic, diagnostic, physiologic, laboratory, admission, and discharge data.

RESULTS

The patient sample was younger and more commonly male, with more comorbidities and a different operative and referral source mix, compared to the APACHE III development sample. Receiver operating characteristic curve areas for ICU (0.92) and hospital mortality (0.90) demonstrated excellent discrimination. Observed ICU mortality (9.9%) did not significantly differ from the prediction of the APACHE III model (8.9%) or the APACHE III model adjusted for hospital characteristics (10.5%). The hospital mortality (16.0%) was underestimated by the APACHE III model [13.6%; chi(2)(1) = 7.4; p = 0.01]. With proprietary adjustments for hospital characteristics (14.9%) or referenced to the US database (15.6%), agreement was closer. Good calibration was found with all models except the unadjusted hospital mortality model.

CONCLUSION

In contrast to other non-American studies, this Australian study demonstrates that the APACHE III can perform well on independent assessment. As perfect discrimination and calibration cannot coexist in a probabilistic model with dichotomous outcomes, performance of APACHE III models with proprietary adjustment for hospital characteristic provide a good compromise for use in quality surveillance.

Intensive care unit length of stay: recent changes and future challenges

OBJECTIVE

To compare case-mix adjusted intensive care unit (ICU) length of stay for critically ill patients with a variety of medical and surgical diagnoses during a 5-yr interval.

DESIGN

Nonrandomized cohort study.

SETTING

A total of 42 ICUs at 40 US hospitals during 1988-1990 and 285 ICUs at 161 US hospitals during 1993-1996.

PATIENTS

A total of 17,105 consecutive ICU admissions during 1988-1990 and 38,888 consecutive ICU admissions during 1993-1996.

MEASUREMENTS AND MAIN RESULTS

We used patient demographic and clinical characteristics to compare observed and predicted ICU length of stay and hospital mortality. Outcomes for patients studied during 1993-1996 were predicted using multivariable models that were developed and cross-validated using the 1988-1990 database. The mean observed hospital length of stay decreased by 3 days (from 14.8 days during 1988-1990 to 11.8 days during 1993-1996), but the mean observed ICU length of stay remained similar (4.70 vs. 4.53 days). After adjusting for patient and institutional differences, the mean predicted 1993-1996 ICU stay was 4.64 days. Thus, the mean-adjusted ICU stay decreased by 0.11 days during this 5-yr interval (T-statistic, 4.35; p < .001). The adjusted mean ICU length of stay was not changed for patients with 49 (75%) of the 65 ICU admission diagnoses. In contrast, the mean observed hospital length of stay was significantly shorter for 47 (72%) of the 65 admission diagnoses, and no ICU admission diagnosis was associated with a longer hospital stay. Aggregate risk-adjusted hospital mortality during 1993-1996 (12.35%) was not significantly different during 1988-1990 (12.27%, p = .54).

CONCLUSIONS

For patients admitted to ICUs, the pressures associated with a decrease in hospital length of stay do not seem to have influenced the duration of ICU stay. Because of the high cost of intensive care, reduction in ICU stay may become a target for future cost-cutting efforts.

Automated computerized intensive care unit severity of illness measure in the Department of Veterans Affairs: preliminary results. SISVistA Investigators. Scrutiny of ICU Severity Veterans Health Sysyems Technology Architecture

OBJECTIVE

To evaluate the feasibility of an automated intensive care unit (ICU) risk adjustment tool (acronym: SISVistA) developed by selecting a subset of predictor variables from the Acute Physiology and Chronic Health Evaluation (APACHE) III available in the existing computerized database of the Department of Veterans Affairs (VA) healthcare system and modifying the APACHE diagnostic and comorbidity approach.

DESIGN

Retrospective cohort study.

SETTING

Six ICUs in three Ohio Veterans Affairs hospitals.

PATIENT SELECTION

The first ICU admission of all patients from February 1996 through July 1997.

OUTCOME MEASURE

Mortality at hospital discharge.

METHODS

The predictor variables, including age, comorbidity, diagnosis, admission source (direct or transfer), and laboratory results (from the +/- 24-hr period surrounding admission), were extracted from computerized VA databases, and APACHE III weights were applied using customized software. The weights of all laboratory variables were added and treated as a single variable in the model. A logistic regression model was fitted to predict the outcome and the model was validated using a boot-strapping technique (1,000 repetitions).

MAIN RESULTS

The analysis included all 4,651 eligible cases (442 deaths). The cohort was predominantly male (97.5%) and elderly (63.6 +/- 12.0 yrs). In multivariate analysis, significant predictors of hospital mortality included age (odds ratio [OR], 1.06; 95% confidence interval [CI], 1.04-1.09), comorbidity (OR, 1.11; 95% CI, 1.08-1.15), total laboratory score (OR, 1.07; 95% CI, 1.06-1.08), direct ICU admission (OR, 0.39; 95% CI, 0.31-0.49), and several broad ICU diagnostic categories. The SISVistA model had excellent discrimination and calibration (C statistic = 0.86, goodness-of-fit statistics; p > .20). The area under the receiver operating characteristic curve of the validated model was 0.86.

CONCLUSIONS

Using common data elements often found in hospital computer systems, SISVistA predicts hospital mortality among patients in Ohio VA ICUs. This preliminary study supports the development of an automated ICU risk prediction system on a more diverse population.

The clinical features of severe community-acquired pneumonia presenting as septic shock. Norasept II Study Investigators

PURPOSE

The purpose of this article was to determine the outcome, clinical and prognostic features, and microbiology of a large group of patients with community-acquired pneumonia (CAP) presenting in septic shock.

MATERIALS AND METHODS

The placebo limb of the Norasept II database was examined. Data were collected on patients in septic shock with a diagnosis of CAP who presented to a participating site from home.

RESULTS

One hundred and forty-eight patients met the study criteria. The 28-day survival was 53%. One hundred and four pathogens were isolated from 77 (52%) patients with 24 (16%) patients having polymicrobial infections. The most common pathogen was Streptococcus pneumoniae (19%), followed by Staphylococcus aureus (18%), Haemophilus influenzae (14%), Klebsiella pneumoniae (11%), and Pseudomonas aeruginosa (7%). Infection with P aeruginosa or Acinetobacter species carried a very high mortality (82%). The only clinical variables recorded in the database that could identify patients with pseudomonas or acinetobacter infection was a history of alcohol abuse. Comorbidities were present in 74% of patients, involving predominantly the cardiorespiratory system. Logistic regression analysis demonstrated APACHE II score and serum interleukin 6 (IL-6) concentration to be significant independent predictors of mortality. Patients with pseudomonas or acinetobacter infection had significantly higher IL-6 levels and significantly lower tumor necrosis factor alpha levels when compared with the rest of the cohort of patients.

CONCLUSION

A diverse spectrum of both gram-positive and gram-negative pathogens were implicated in patients with CAP presenting in septic shock, necessitating broad spectrum empiric antimicrobial coverage. This coverage should include antipseudomonal activity, particularly in alcoholic patients. Severity of illness (APACHE II score) and IL-6 levels were important prognostic factors. Infection with P aeruginosa and Acinetobacter species carried a very high mortality.

Patients readmitted to ICUs* : a systematic review of risk factors and outcomes

STUDY OBJECTIVE

To evaluate the causes, risk factors, and mortality rates associated with unexpected readmission to medical and surgical ICUs.

DATA SOURCES

MEDLINE citation review of primary articles focusing on ICU readmission or ICU outcomes from January 1966 to June 1999, and contact with authors of primary studies.

STUDY SELECTION

Eight primary studies of ICU readmission and eight multi-institutional ICU outcome studies that reported ICU readmission rates were included.

DATA EXTRACTION

We abstracted data on the methodology and design of the primary studies, overall rates, causes, predictors, outcomes, and measures of quality of care associated with ICU readmission.

DATA SYNTHESIS

The average ICU readmission rate of 7% (range, 4 to 14%) has remained relatively unchanged in both North America and Europe. Respiratory and cardiac conditions were the most common (30 to 70%) precipitating cause of ICU readmission. Patients readmitted to ICUs had average hospital stays at least twice as long as nonreadmitted patients. Hospital death rates were 2- to 10-times higher for readmitted patients than for those who survived an ICU admission and were never readmitted. Predictors of ICU readmission have been neither well studied nor reproducible. Unstable vital signs, especially respiratory and heart rate abnormalities, and the presence of poor pulmonary function at time of ICU discharge appear to be the most consistent predictors of ICU readmission. There were no consistent data supporting the use of readmission rates as a measure of quality of care.

CONCLUSIONS

ICU readmission is associated with dramatically higher hospital mortality. Unstable vital signs at the time of ICU discharge are the most consistent predictor of ICU readmission. Further studies focusing on processes of ICU and hospital care are needed to determine if ICU readmission rates are a measure of quality of care.

Performance of the Mortality Probability Models in assessing severity of illness during the first week in the intensive care unit

OBJECTIVE

To extend the Mortality Probability Models (MPM) II severity system to time periods between 4 and 7 days after admission to the intensive care unit (ICU).

DESIGN

Prospective inception cohort.

SETTING

Fifteen adult medical and surgical ICUs in Spain.

PATIENTS

A total of 1,441 patients aged > or =18 yrs consecutively admitted from April 1, 1995 through July 31, 1995.

INTERVENTIONS

Prospective data collection during the stay of the patient in the ICU. Data collected included demographic information, length-of-stay and vital status at both ICU and hospital discharge, as well as all variables necessary for computing the MPM II system at admission and during the first 7 days of stay in the ICU MEASUREMENTS AND MAIN RESULTS: Calibration and discrimination of the four existing MPM II models (MPM0, MPM24, MPM48, and MPM72) were assessed in the study database. The MPM II system overestimated the mortality of patients with probabilities of death > or =0.4. The MPM24 model was customized. Models for time periods between 48 hrs and 7 days (MPM48 to MPMd7) were obtained using the same strategy that was used to develop the original MPM48 and the MPM72 models. The variable coefficients of the MPM24 model were kept fixed and the constant terms of the MPM48 to MPMd7 models were estimated by logistic regression. The constant term stabilized after the fourth day of admission and it was similar to the constant term of the MPM72 model. The customized MPM72 performed very well for days 4 to 7 after admission to the ICU.

CONCLUSIONS

If the patient's condition stays the same day after day, the probability of dying in the hospital increases until 72 hrs, and then stabilizes. A severity measure that performs well at 72 hrs can be a useful tool for measuring severity at later time periods.

Review of the methodologies and applications of scoring systems in neonatal and pediatric intensive care

Scoring systems and risk prediction rules quantitate the severity of clinical conditions and stratify patients according to a specified outcome. In intensive care medicine, the complexity and number of clinical scoring systems is increasing as their utility in both health services research and clinical medicine broadens. We anticipate that with increasing healthcare costs and competition, the demand for risk adjusted outcomes and institutional benchmarking will increase. As academicians and clinicians, it is vital to be knowledgeable regarding the methodologies and applications of these scoring and risk prediction systems to ensure their quality and appropriate utilization.

The weaning continuum use of Acute Physiology and Chronic Health Evaluation III, Burns Wean Assessment Program, Therapeutic Intervention Scoring System, and Wean Index scores to establish stages of weaning

OBJECTIVE

To determine whether four stages of weaning (acute, prewean, wean, and outcome) could be identified by using clinical instruments designed to quantify severity of illness, patient stability, or weaning readiness. The instruments used were the Acute Physiology and Chronic Health Evaluation (APACHE III), the Therapeutic Intervention Scoring System (TISS), the Burns Wean Assessment Program (BWAP), and the Wean Index (WI). The stages were adapted from those proposed by the American Association of Critical Care Nurses Third National Study Group's Weaning Continuum Model.

DESIGN

Prospective, convenience cohort. This study was part of a larger study designed to test an outcomes managed approach to weaning by using an outcomes manager and a clinical pathway.

SETTING

University medical intensive care unit.

PATIENTS

Adult patients requiring mechanical ventilation >3 days admitted to the medical intensive care unit between November 1994 and May 1995.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

Scores for the APACHE III, TISS, BWAP, and WI were collected on 97 patients every other day until they weaned, were transferred, or died. Outcomes described for each stage of weaning were dated on the clinical pathway when achieved. Comments about patient stability and ventilator progress also were recorded along with a subjective determination of the stage of weaning. We used decision rules to identify time intervals for each stage of weaning and outcomes attained by stage. Finally, APACHE III, TISS, BWAP, and WI scores were placed in each stage by date for analysis. The APACHE III, TISS, and BWAP scores were able to differentiate the acute, prewean, and wean stages but not the outcome stage.

CONCLUSIONS

By identifying distinct scores for each stage, we may be able to better explore appropriate interventions for the stages as well as predict weaning outcomes. Indices that include physiologic and respiratory factors can differentiate weaning stages, but respiratory factors alone cannot.

Severity scoring systems and the prediction of outcome from intensive care

Severity scoring systems are tools that provide a predicted mortality for a group of intensive care unit patients on the basis of derangement of their physiology and some past medical history. This predicted mortality can then be compared with the actual mortality to give some indicator of the effectiveness of the package of care delivered by the intensive care unit, corrected for differences in case-mix. Thus, their primary use is in audit, and they are designed for use on large populations of patients and not on individuals. In spite of a large number of publications on the development, refinement and testing of scoring systems, papers describing their use in comparative audit are very rare. This may be partly due to limitations in their ability to predict mortality outside the population on which they were developed, and to the change in calibration of the system with time and advances in medical science. This review briefly addresses the limitations of severity scoring systems in light of recent publications.

Scoring systems for assessing organ dysfunction and survival

Sepsis is an ongoing disease process carrying a high risk of organ failure and death. Scoring systems to determine disease severity and risk of mortality may be useful in patient management and clinical trial enrollment, although the role of either type of score in the determination of admission or discharge criteria or in decisions relating to the continuation or withholding of treatment remains controversial. General scoring systems have been developed to quantify the severity of illness and the risk of mortality in ICU patients. Ideally, these should be customized before use in patients with septic shock, but in general noncustomized models are used, and this potential limitation should be acknowledged. Prognostic scores are remarkably reliable at predicting outcome in groups of patients and give an indication of severity of disease on admission, but they are unable to provide detail on how a patient is responding to treatment or on the disease progression. Organ function scores, however, can be assessed repeatedly and used to define a patient's progress. This approach can thus be used to evaluate individual patient care, to identify patients for enrollment in clinical trials or epidemiologic analyses, and to assess morbidity measures in clinical trials of new interventions. Organ dysfunction scores are just that, descriptors of organ dysfunction, and although high values correlate well with mortality, prognostication is not their prime aim; organ dysfunction scores and outcome prediction scores should rather be viewed as complementary systems in the description of ICU populations.

Prediction of outcome in intensive care unit trauma patients: a multicenter study of Acute Physiology and Chronic Health Evaluation (APACHE), Trauma and Injury Severity Score (TRISS), and a 24-hour intensive care unit (ICU) point system

OBJECTIVE

To conduct a multicenter study to validate the accuracy of the Acute Physiology and Chronic Health Evaluation (APACHE) II system, APACHE III system, Trauma and Injury Severity Score (TRISS) methodology, and a 24-hour intensive care unit (ICU) point system for prediction of mortality in ICU trauma patient admissions.

METHODS

The study population consisted of retrospectively identified, consecutive ICU trauma admissions (n = 2,414) from six Level I trauma centers. Probabilities of death were calculated by using logistic regression analysis. The predictive power of each system was evaluated by using decision matrix analysis to compare observed and predicted outcomes with a decision criterion of 0.50 for risk of hospital death. The Youden Index (YI) was used to compare the proportion of patients correctly classified by each system. Measures of model calibration were based on goodness-of-fit testing (Hosmer-Lemeshow statistic less than 15.5) and model discrimination were based on the area under the receiver operating characteristic curve (AUC).

RESULTS

Overall, APACHE II (sensitivity, 38%; specificity, 99%; YI, 37%; H-L statistic, 92.6; AUC, 0.87) and TRISS (sensitivity, 52%; specificity, 94%; YI, 46%; H-L statistic, 228.1; AUC, 0.82) were poor predictors of aggregate mortality, because they did not meet the acceptable thresholds for both model calibration and discrimination. APACHE III (sensitivity, 60%; specificity, 98%; YI, 58%; H-L statistic, 7.0; AUC, 0.89) was comparable to the 24-hour ICU point system (sensitivity, 51%; specificity, 98%; YI, 50%; H-L statistic, 14.7; AUC, 0.89) with both systems showing strong agreement between the observed and predicted outcomes based on acceptable thresholds for both model calibration and discrimination. The APACHE III system significantly improved upon APACHE II for estimating risk of death in ICU trauma patients (p < 0.001). Compared with the overall performance, for the subset of patients with nonoperative head trauma, the percentage correctly classified was decreased to 46% for APACHE II; increased to 71% for APACHE III (p < 0.001 vs. APACHE II); increased to 59% for TRISS; and increased to 62% for 24-hour ICU points. For operative head trauma, the percentage correctly classified was increased to 60% for APACHE II; increased to 61% for APACHE III; decreased to 43% for TRISS (p < 0.004 vs. APACHE III); and increased to 54% for 24-hour ICU points. For patients without head injuries, all of the systems were unreliable and considerably underestimated the risk of death. The percentage of nonoperative and operative patients without head trauma who were correctly classified was decreased, respectively, to 26% and 30% for APACHE II; 33% and 29% for APACHE III; 33% and 19% for TRISS; 20% and 23% for 24-hour ICU points.

CONCLUSION

For the overall estimation of aggregate ICU mortality, the APACHE III system was the most reliable; however, performance was most accurate for subsets of patients with head trauma. The 24-hour ICU point system also demonstrated acceptable overall performance with improved performance for patients with head trauma. Overall, APACHE II and TRISS did not meet acceptable thresholds of performance. When estimating ICU mortality for subsets of patients without head trauma, none of these systems had an acceptable level of performance. Further multicenter studies aimed at developing better outcome prediction models for patients without head injuries are warranted, which would allow trauma care providers to set uniform standards for judging institutional performance.

Severity scoring and outcome assessment. Computerized predictive models and scoring systems

Severity of illness scoring systems and standardized death ratios are being used with increasing frequency as markers of quality of care and to compare and contrast the performance of ICUs. However, numerous factors unrelated to the quality of care delivered may impact the severity of illness score and standardized death ratios. This article reviews the commonly used severity scoring systems and factors that affect their predictive performance.