Readiness for the scorecard era in cardiovascular medicine

National Veterans Health Administration inpatient risk stratification models for hospital-acquired acute kidney injury

OBJECTIVE

Hospital-acquired acute kidney injury (HA-AKI) is a potentially preventable cause of morbidity and mortality. Identifying high-risk patients prior to the onset of kidney injury is a key step towards AKI prevention.

MATERIALS AND METHODS

A national retrospective cohort of 1,620,898 patient hospitalizations from 116 Veterans Affairs hospitals was assembled from electronic health record (EHR) data collected from 2003 to 2012. HA-AKI was defined at stage 1+, stage 2+, and dialysis. EHR-based predictors were identified through logistic regression, least absolute shrinkage and selection operator (lasso) regression, and random forests, and pair-wise comparisons between each were made. Calibration and discrimination metrics were calculated using 50 bootstrap iterations. In the final models, we report odds ratios, 95% confidence intervals, and importance rankings for predictor variables to evaluate their significance.

RESULTS

The area under the receiver operating characteristic curve (AUC) for the different model outcomes ranged from 0.746 to 0.758 in stage 1+, 0.714 to 0.720 in stage 2+, and 0.823 to 0.825 in dialysis. Logistic regression had the best AUC in stage 1+ and dialysis. Random forests had the best AUC in stage 2+ but the least favorable calibration plots. Multiple risk factors were significant in our models, including some nonsteroidal anti-inflammatory drugs, blood pressure medications, antibiotics, and intravenous fluids given during the first 48 h of admission.

CONCLUSIONS

This study demonstrated that, although all the models tested had good discrimination, performance characteristics varied between methods, and the random forests models did not calibrate as well as the lasso or logistic regression models. In addition, novel modifiable risk factors were explored and found to be significant.

Development of inpatient risk stratification models of acute kidney injury for use in electronic health records

OBJECTIVE

Patients with hospital-acquired acute kidney injury (AKI) are at risk for increased mortality and further medical complications. Evaluating these patients with a prediction tool easily implemented within an electronic health record (EHR) would identify high-risk patients prior to the development of AKI and could prevent iatrogenically induced episodes of AKI and improve clinical management.

METHODS

The authors used structured clinical data acquired from an EHR to identify patients with normal kidney function for admissions from 1 August 1999 to 31 July 2003. Using administrative, computerized provider order entry and laboratory test data, they developed a 3-level risk stratification model to predict each of 2 severity levels of in-hospital AKI as defined by RIFLE criteria. The severity levels were defined as 150% or 200% of baseline serum creatinine. Model discrimination and calibration were evaluated using 10-fold cross-validation.

RESULTS

Cross-validation of the models resulted in area under the receiver operating characteristic (AUC) curves of 0.75 (150% elevation) and 0.78 (200% elevation). Both models were adequately calibrated as measured by the Hosmer-Lemeshow goodness-of-fit test chi-squared values of 9.7 (P = 0.29) and 12.7 (P = 0.12), respectively.

CONCLUSIONS

The authors generated risk prediction models for hospital-acquired AKI using only commonly available electronic data. The models identify patients at high risk for AKI who might benefit from early intervention or increased monitoring.

Effects of SVM parameter optimization on discrimination and calibration for post-procedural PCI mortality

Support vector machines (SVM) have become popular among machine learning researchers, but their applications in biomedicine have been somewhat limited. A number of methods, such as grid search and evolutionary algorithms, have been utilized to optimize model parameters of SVMs. The sensitivity of the results to changes in optimization methods has not been investigated in the context of medical applications. In this study, radial-basis kernel SVM and polynomial kernel SVM mortality prediction models for percutaneous coronary interventions were optimized using (a) mean-squared error, (b) mean cross-entropy error, (c) the area under the receiver operating characteristic, and (d) the Hosmer-Lemeshow goodness-of-fit test (HL chi(2)). A threefold cross-validation inner and outer loop method was used to select the best models using the training data, and evaluations were based on previously unseen test data. The results were compared to those produced by logistic regression models optimized using the same indices. The choice of optimization parameters had a significant impact on performance in both SVM kernel types.

Discrimination and calibration of mortality risk prediction models in interventional cardiology

OBJECTIVES

Using a local percutaneous coronary intervention (PCI) data repository, we sought to compare the performance of a number of local and well-known mortality models with respect to discrimination and calibration.

BACKGROUND

Accurate risk prediction is important for a number of reasons including physician decision support, quality of care assessment, and patient education. Current evidence on the value of applying PCI risk models to individual cases drawn from a different population is controversial.

METHODS

Data were collected from January 01, 2002 to September 30, 2004 on 5216 consecutive percutaneous coronary interventions at Brigham and Women's Hospital (Boston, MA). Logistic regression was used to create a local risk model for in-hospital mortality in these procedures, and a number of statistical methods were used to compare the discrimination and calibration of this new and old local risk models, as well as the Northern New England Cooperative Group, New York State (1992 and 1997), University of Michigan consortium, American College of Cardiology-National Cardiovascular Data Registry, and The Cleveland Clinic Foundation risk prediction models. Areas under the ROC (AUC) curves were used to evaluate discrimination, and the Hosmer-Lemeshow (HL) goodness-of-fit test and calibration curves assessed applicability of the models to individual cases.

RESULTS

Multivariate risk factors included in the newly constructed local model were: age, prior intervention, diabetes, unstable angina, salvage versus elective procedure, cardiogenic shock, acute myocardial infarction (AMI), and left anterior descending artery intervention. The area under the ROC curve (AUC) was 0.929 (SE=0.017), and the p value for the Hosmer-Lemeshow (HL) goodness-of-fit was 0.473. This indicates good discrimination and calibration. Bootstrap re-sampling indicated AUC stability. Evaluation of the external models showed an AUC range from 0.82 to 0.90 indicating good discrimination across all models, but poor calibration (HL p value < or = 0.0001).

CONCLUSIONS

Validation of AUC values across all models suggests that certain risk factors have remained important over the last decade. However, the lack of calibration suggests that small changes in patient populations and data collection methods quickly reduce the accuracy of patient level estimations over time. Possible solutions to this problem involve either recalibration of models using local data or development of new local models.

Critical pathways for management of patients with acute coronary syndromes: an assessment by the National Heart Attack Alert Program

BACKGROUND

The use of critical pathways for a variety of clinical conditions has grown rapidly in recent years, particularly pathways for patients with acute coronary syndromes (ACS). However, no systematic review exists regarding the value of critical pathways in this setting.

METHODS

The National Heart Attack Alert Program established a Working Group to review the utility of critical pathways on quality of care and outcomes for patients with ACS. A literature search of MEDLINE, cardiology textbooks, and cited references in any article identified was conducted regarding the use of critical pathways for patients with ACS.

RESULTS

Several areas for improving the care of patients with ACS through the application of critical pathways were identified: increasing the use of guideline-recommended medications, targeting use of cardiac procedures and other cardiac testing, and reducing the length of stay in hospitals and intensive care units. Initial studies have shown promising results in improving quality of care and reducing costs. No large studies designed to demonstrate an improvement in mortality or morbidity were identified in this literature review.

CONCLUSIONS

Critical pathways offer the potential to improve the care of patients with ACS while reducing the cost of care. Their use should improve the process and cost-effectiveness of care, but further research in this field is needed to determine whether these changes in the process of care will translate into improved clinical outcomes.

Operator volume and outcome of patients undergoing coronary stent placement

OBJECTIVES

The aim of this study was to assess the relation between operator experience in coronary stent placement procedures and the clinical outcome of patients.

BACKGROUND

The results of coronary balloon angioplasty are closely related to the experience of the operator performing the procedure. Data on the effect of operator experience on the results after coronary stent placement are missing.

METHODS

The study included 3,409 consecutive patients undergoing coronary stent placement for the management of coronary artery disease. A composite end point of cardiac death, myocardial infarction and aortocoronary bypass surgery during the first 30 days after the intervention, was the primary end point and the procedural failure was the secondary end point of the study.

RESULTS

Adverse clinical outcome occurred in 2.99% of the 3,409 patients undergoing coronary stent placement. Procedural failure was recorded in 2.08% of the patients. Operator volumes above 483 procedures were associated with a risk-adjusted adverse outcome rate of 1.70%+/-1.28%, which is significantly lower than the overall rate of 2.99%. Operator yearly volumes of under 90 procedures were associated with a risk-adjusted adverse outcome rate of 4.59%+/-1.17%, which is significantly higher than the overall rate of 2.99%. The operator experience was an independent predictor even after adjusting for the effect of other risk factors. The analysis demonstrated that an experience of at least 100 procedures is required to obtain better outcome even in patients with simple coronary lesions and that operators should perform at least 70 procedures annually to expect a better outcome in patients with both simple and complex coronary lesions.

CONCLUSIONS

Operator experience is a significant and independent predictor of the outcome of patients undergoing coronary stent placement. An experience of at least 100 procedures and an annual volume of at least 70 procedures are required to ensure a significantly better outcome after coronary stent implantation.

Relation of operator volume and experience to procedural outcome of percutaneous coronary revascularization at hospitals with high interventional volumes

BACKGROUND

Although an inverse relation between physician caseload and complications has been conclusively demonstrated for several surgical procedures, such data are lacking for percutaneous coronary intervention, and the ACC/AHA guidelines requiring > or = 75 cases per year for operator "competency" are considered by some physicians to be arbitrary.

METHODS AND RESULTS

From quality-controlled databases at five high-volume centers, models predictive of death and the composite outcome of death, Q-wave infarction, or emergency bypass surgery were developed from 12,985 consecutively treated patients during 1993 through 1994. Models had moderate to high discriminative capacity (area under ROC curves, 0.65 to 0.85), were well calibrated, and were not overfitted by standard tests. These models were used for risk adjustment, and the relations between both yearly caseload and years of interventional experience and the two adverse outcome measures were explored for all 38 physicians with > or = 30 cases per year. The average physician performed a mean +/- SD of 163 +/- 24 cases per year and had been practicing angioplasty for 8 +/- 5 years. Risk-adjusted measures of both death and the composite adverse outcome were inversely related to the number of cases each operator performed annually but bore no relation to total years of experience. Both adverse outcomes were more closely related to the logarithm of caseload (for death, r = .37, P = .01; for death, Q-wave infarction, or bypass surgery, r = .58, P < .001) than to linear caseload.

CONCLUSIONS

In this analysis, high-volume operators had a lower incidence of major complications than did lower-volume operators, but the difference was not consistent for all operators. If these data are validated, their implications for hospital, physician, and payer policy will require exploration.

Guidelines for internal peer review in the cardiac catheterization laboratory. Laboratory Performance Standards Committee, Society for Cardiac Angiography and Interventions

The Laboratory Performance Standards Committee of the Society for Cardiac Angiography and Interventions has proposed guidelines for establishing an internal peer review program in the cardiac catheterization laboratory. The first step is to establish a committee and a data base. This data base should include quality indicators that reflect: physician qualifications, outcomes of procedures, and processes of care. The outcomes must be risk-adjusted to account for the variable severity of illness. Data should be collected by catheterization laboratory personnel and entered into a laboratory-specific computerized data base. These data must be analyzed and organized into profiles that reflect the quality of care. Based on this information, the Committee would institute the following interventions to improve physician performance: education, clinical practice standardization, feedback and benchmarking, professional interaction, incentives, decision-support systems, and administrative interventions. The legal aspects of peer review are reviewed briefly.