Simplified risk score models accurately predict the risk of major in-hospital complications following percutaneous coronary intervention

Revisiting performance metrics for prediction with rare outcomes

Machine learning algorithms are increasingly used in the clinical literature, claiming advantages over logistic regression. However, they are generally designed to maximize the area under the receiver operating characteristic curve. While area under the receiver operating characteristic curve and other measures of accuracy are commonly reported for evaluating binary prediction problems, these metrics can be misleading. We aim to give clinical and machine learning researchers a realistic medical example of the dangers of relying on a single measure of discriminatory performance to evaluate binary prediction questions. Prediction of medical complications after surgery is a frequent but challenging task because many post-surgery outcomes are rare. We predicted post-surgery mortality among patients in a clinical registry who received at least one aortic valve replacement. Estimation incorporated multiple evaluation metrics and algorithms typically regarded as performing well with rare outcomes, as well as an ensemble and a new extension of the lasso for multiple unordered treatments. Results demonstrated high accuracy for all algorithms with moderate measures of cross-validated area under the receiver operating characteristic curve. False positive rates were <1%, however, true positive rates were <7%, even when paired with a 100% positive predictive value, and graphical representations of calibration were poor. Similar results were seen in simulations, with the addition of high area under the receiver operating characteristic curve (>90%) accompanying low true positive rates. Clinical studies should not primarily report only area under the receiver operating characteristic curve or accuracy.

Development and validation of a simple risk score to predict 30-day readmission after percutaneous coronary intervention in a cohort of medicare patients

OBJECTIVES

To develop a risk model that can be used to identify PCI patients at higher risk of readmission who may benefit from additional resources at the time of discharge.

BACKGROUND

A high proportion of patients undergoing PCI are readmitted within 30 days of discharge.

METHODS

The sample comprised patients aged ≥65 years who underwent PCI at a CathPCI Registry®-participating hospital and could be linked with 100% Medicare fee-for-service claims between 01/2007 and 12/2009. The sample (n = 388,078) was randomly divided into risk score development (n = 193,899) and validation (n = 194,179) cohorts. We did not count as readmissions those associated with staged revascularization procedures. Multivariable logistic regression models using stepwise selection models were estimated to identify variables independently associated with all-cause 30-day readmission.

RESULTS

The mean 30-day readmission rates for the development (11.36%) and validation (11.35%) cohorts were similar. In total, 19 variables were significantly associated with risk of 30-day readmission (P < 0.05), and model c-statistics were similar in the development (0.67) and validation (0.66) cohorts. The simple risk score based on 14 variables identified patients at high and low risk of readmission. Patients with a score of ≥13 (15.4% of sample) had more than an 18.5% risk of readmission, while patients with a score ≤6 (41.9% of sample) had less than an 8% risk of readmission.

CONCLUSION

Among PCI patients, risk of readmission can be estimated using clinical factors present at the time of the procedure. This risk score may guide clinical decision-making and resource allocation for PCI patients at the time of hospital discharge. © 2016 Wiley Periodicals, Inc.

Do Non-Clinical Factors Improve Prediction of Readmission Risk?: Results From the Tele-HF Study

OBJECTIVES

This study sought to determine whether a model that included self-reported socioeconomic, health status, and psychosocial characteristics obtained from patients recently discharged from hospitalizations for heart failure substantially improved 30-day readmission risk prediction compared with a model that incorporated only clinical and demographic factors.

BACKGROUND

Existing readmission risk models have poor discrimination and it is unknown whether they would be markedly improved by the inclusion of patient-reported information.

METHODS

As part of the Tele-HF (Telemonitoring to Improve Heart Failure Outcomes) trial, we conducted medical record abstraction and telephone interviews in a sample of 1,004 patients recently hospitalized for heart failure to obtain clinical, functional, and psychosocial information within 2 weeks of discharge. Candidate risk factors included 110 variables divided into 2 groups: demographic and clinical variables generally available from the medical record; and socioeconomic, health status, adherence, and psychosocial variables from patient interview.

RESULTS

The 30-day readmission rate was 17.1%. Using the 3-level risk score derived from the restricted medical record variables, patients with a score of 0 (no risk factors) had a readmission rate of 10.9% (95% confidence interval [CI]: 8.2% to 14.2%), and patients with a score of 2 (all risk factors) had a readmission rate of 32.1% (95% CI: 22.4% to 43.2%), a C-statistic of 0.62. Using the 5-level risk score derived from all variables, patients with a score of 0 (no risk factors) had a readmission rate of 9.6% (95% CI: 6.1% to 14.2%), and patients with a score of 4 (all risk factors) had a readmission rate of 55.0% (95% CI: 31.5% to 76.9%), a C-statistic of 0.65.

CONCLUSIONS

Self-reported socioeconomic, health status, adherence, and psychosocial variables are not dominant factors in predicting readmission risk for patients with heart failure. Patient-reported information improved model discrimination and extended the predicted ranges of readmission rates, but the model performance remained poor. (Telemonitoring to Improve Heart Failure Outcomes [Tele-HF]; NCT00303212).

Angiographic Lesion Complexity Score and In-Hospital Outcomes after Percutaneous Coronary Intervention

OBJECTIVE

We devised a percutaneous coronary intervention (PCI) scoring system based on angiographic lesion complexity and assessed its association with in-hospital complications.

BACKGROUND

Although PCI is finding increasing application in patients with coronary artery disease, lesion complexity can lead to in-hospital complications.

METHODS

Data from 3692 PCI patients were scored based on lesion complexity, defined by bifurcation, chronic total occlusion, type C, and left main lesion, along with acute thrombus in the presence of ST-segment elevation myocardial infarction (1 point assigned for each variable).

RESULTS

The patients' mean age was 67.5 +/- 10.8 years; 79.8% were male. About half of the patients (50.3%) presented with an acute coronary syndrome, and 2218 (60.1%) underwent PCI for at least one complex lesion. The patients in the higher-risk score groups were older (p < 0.001) and had present or previous heart failure (p = 0.02 and p = 0.01, respectively). Higher-risk score groups had significantly higher in-hospital event rates for death, heart failure, and cardiogenic shock (from 0 to 4 risk score; 1.7%, 4.5%, 6.3%, 7.1%, 40%, p < 0.001); bleeding with a hemoglobin decrease of >3.0 g/dL (3.1%, 11.0%, 13.1%, 10.3%, 28.6%, p < 0.001); and postoperative myocardial infarction (1.5%, 3.1%, 3.8%, 3.8%, 10%, p = 0.004), respectively. The association with adverse outcomes persisted after adjustment for known clinical predictors (odds ratio 1.72, p < 0.001).

CONCLUSION

The complexity score was cumulatively associated with in-hospital mortality and complication rate and could be used for event prediction in PCI patients.

Predictors of long term outcomes in 11,441 consecutive patients following percutaneous coronary interventions

Given the vicissitudes of percutaneous coronary intervention (PCI) technology, epidemiology, and mode of practice, the aim of this study was to define contemporary outcome predictors in a very large consecutive patient cohort. Data from 11,441 consecutive patients who underwent PCI at a tertiary medical center from April 2004 to September 2013 are presented. A comprehensive database was built using various data sources, with outcome end points defined as all-cause mortality and as a composite of death or nonfatal myocardial infarction during follow-up. Candidate variables to influence outcome were chosen a priori and were tested using multivariate time-dependent models to estimate each interaction. Mean follow-up was 5.5 years (range 3 months to 9.5 years). The cohort consisted of 75% men, 42% patients with diabetes, and 61% patients who underwent PCI in acute coronary syndrome settings and 7.8% for ST-elevation myocardial infarction. Drug-eluting stents were used in 43.4% of patients, bare-metal stents in 52%, and balloon angioplasty alone in 4.6%. In multivariate analysis, in addition to already well-recognized predictors of death or myocardial infarction such as advanced age (hazard ratio [HR] 1.031, p <0.001), female gender (HR 1.23, p <0.001), urgent setting (HR 1.23, p <0.001) and diabetes mellitus (HR 1.28, p <0.001), we particularly noted previous anemia (HR 1.55 p <0.001), previous chronic kidney injury (HR 1.93, p <0.001) and previous moderate to severe left ventricular dysfunction (HR 2.29, p <0.001). Drug-eluting stent placement was associated with better outcomes (HR 0.70, p <0.001). In conclusion, this analysis confirms the effect of some known predictors of PCI outcomes. However, the extent of their effect is modest, while other predictors may have a greater influence on outcomes. Risk stratification of PCI patients should take into account kidney injury, anemia, and left ventricular function. Drug-eluting stents provide sustained benefit.

Risk stratification for long-term mortality after percutaneous coronary intervention

BACKGROUND

A simple risk score to predict long-term mortality after percutaneous coronary intervention (PCI) using preprocedural risk factors is currently not available. In this study, we created one by simplifying the results of a Cox proportional hazards model.

METHODS AND RESULTS

A total of 11,897 patients who underwent PCI from October through December 2003 in New York State were randomly divided into derivation and validation samples. Patients' vital statuses were tracked using the National Death Index through the end of 2008. A Cox proportional hazards model was fit to predict death after PCI using the derivation sample, and a simplified risk score was created. The Cox model identified 12 separate risk factors for mortality including older age, extreme body mass indexes, multivessel disease, a lower ejection fraction, unstable hemodynamic state or shock, several comorbidities (cerebrovascular disease, peripheral vascular disease, congestive heart failure, chronic obstructive pulmonary disease, diabetes mellitus, and renal failure), and a history of coronary artery bypass graft surgery. The C statistics of this model when applied to the validation sample were 0.787, 0.785, and 0.773 for risks of death within 1, 3, and 5 years after PCI, respectively. In addition, the point-based risk score demonstrated good agreement between patients' observed and predicted risks of death.

CONCLUSIONS

A simple risk score created from a more complicated Cox proportional hazards model can be used to accurately predict a patient's risk of long-term mortality after PCI.

Performance of the REVEAL pulmonary arterial hypertension prediction model using non-invasive and routinely measured parameters

BACKGROUND

The REVEAL model for pulmonary arterial hypertension (PAH) uses 19 predictors to calculate a 1-year survival probability and can be repeated over time. It is currently unclear which of the 19 variables are the most essential for serial REVEAL score calculation. We aimed to identify high-yield predictors in the REVEAL score and hypothesized that the model could be simplified considerably without compromising performance.

METHODS

REVEAL scores were calculated in a cohort of 140 PAH patients (Full REVEAL Model). Scores were then recalculated excluding all right heart catheterization and pulmonary function test data (Simple Model) and again using only PAH type, New York Heart Association class, brain natriuretic peptide, renal function and right atrial pressure by echocardiogram (Clinical Model). The models were then tested for the ability to predict 1-year outcomes and the performance of the models was compared.

RESULTS

The c indices of the models to predict 1-year survival were not statistically different from one another (Full REVEAL Model: 0.765; Simple Model: 0.759; Clinical Model: 0.745; p = 0.92). For the composite outcome of survival or freedom from lung transplant at 1 year, the models were again not statistically different from one another (c indices: Full REVEAL Model: 0.805; Simple Model: 0.809; Clinical Model: 0.785; p = 0.73).

CONCLUSIONS

The original, Full REVEAL Model appeared to have comparable performance after selectively limiting the number of predictors. There is opportunity to re-evaluate large-registry PAH data to identify a limited number of high-yield variables and to develop a simplified, clinical model.

Comparison of six risk scores in patients with triple vessel coronary artery disease undergoing PCI: competing factors influence mortality, myocardial infarction, and target lesion revascularization

OBJECTIVES

To compare the discriminatory value of differing risk scores for predicting clinical outcomes following PCI in routine practice.

BACKGROUND

Various risk scores predict outcomes after PCI. However, these scores consider markedly different factors, from purely anatomical (SYNTAX risk score [SRS]) to purely clinical (ACEF, modified ACEF [ACEFmod], NCDR), while other scores combine both elements (Clinical SYNTAX score [CSS], NY State Risk Score [NYSRS]).

METHODS

Patients with triple vessel and/or LM disease with 12 month follow-up were studied from a single center PCI registry. Exclusion criteria included STEMI presentation, prior revascularization and shock. Clinical events at 12 months were compared to baseline risk scores, according to score tertiles and area under receiver-operating-characteristic curves (AUC).

RESULTS

We identified 584 eligible patients (69.8±12.3yrs, 405 males). All scores were predictive of mortality, with the SRS being least predictive (AUC=0.66). The most accurate scores for mortality were the CSS and ACEF (AUC=0.76 for both: P = 0.019 and 0.08 vs. SRS, respectively). For TLR, while the SRS trended toward being positively predictive (P = 0.075), several scores trended towards a negative association, which reached significance for the NCDR (P = 0.045). The SRS and CSS were the only scores predictive of MI (both P < 0.05). No score was particularly accurate for predicting MACE (death+MI+TLR), with AUCs ranging from 0.53 (NCDR) to 0.63 (SRS).

CONCLUSIONS

Competing factors influence mortality, MI and TLR after PCI. An increasing burden of comorbidities is associated with mortality, whereas anatomical complexity predicts MI. By combining these outcomes to predict MACE, all scores show reduced utility.

Grid Binary LOgistic REgression (GLORE): building shared models without sharing data

Objective

The classification of complex or rare patterns in clinical and genomic data requires the availability of a large, labeled patient set. While methods that operate on large, centralized data sources have been extensively used, little attention has been paid to understanding whether models such as binary logistic regression (LR) can be developed in a distributed manner, allowing researchers to share models without necessarily sharing patient data.

Material and methods

Instead of bringing data to a central repository for computation, we bring computation to the data. The Grid Binary LOgistic REgression (GLORE) model integrates decomposable partial elements or non-privacy sensitive prediction values to obtain model coefficients, the variance-covariance matrix, the goodness-of-fit test statistic, and the area under the receiver operating characteristic (ROC) curve.

Results

We conducted experiments on both simulated and clinically relevant data, and compared the computational costs of GLORE with those of a traditional LR model estimated using the combined data. We showed that our results are the same as those of LR to a 10⁻¹⁵ precision. In addition, GLORE is computationally efficient.

Limitation

In GLORE, the calculation of coefficient gradients must be synchronized at different sites, which involves some effort to ensure the integrity of communication. Ensuring that the predictors have the same format and meaning across the data sets is necessary.

Conclusion

The results suggest that GLORE performs as well as LR and allows data to remain protected at their original sites.

A patient-driven adaptive prediction technique to improve personalized risk estimation for clinical decision support

Objective

Competing tools are available online to assess the risk of developing certain conditions of interest, such as cardiovascular disease. While predictive models have been developed and validated on data from cohort studies, little attention has been paid to ensure the reliability of such predictions for individuals, which is critical for care decisions. The goal was to develop a patient-driven adaptive prediction technique to improve personalized risk estimation for clinical decision support.

Material and methods

A data-driven approach was proposed that utilizes individualized confidence intervals (CIs) to select the most ‘appropriate’ model from a pool of candidates to assess the individual patient's clinical condition. The method does not require access to the training dataset. This approach was compared with other strategies: the BEST model (the ideal model, which can only be achieved by access to data or knowledge of which population is most similar to the individual), CROSS model, and RANDOM model selection.

Results

When evaluated on clinical datasets, the approach significantly outperformed the CROSS model selection strategy in terms of discrimination (p<1e–14) and calibration (p<0.006). The method outperformed the RANDOM model selection strategy in terms of discrimination (p<1e–12), but the improvement did not achieve significance for calibration (p=0.1375).

Limitations

The CI may not always offer enough information to rank the reliability of predictions, and this evaluation was done using aggregation. If a particular individual is very different from those represented in a training set of existing models, the CI may be somewhat misleading.

Conclusion

This approach has the potential to offer more reliable predictions than those offered by other heuristics for disease risk estimation of individual patients.

Prediction of long-term mortality after percutaneous coronary intervention in older adults: results from the National Cardiovascular Data Registry

BACKGROUND

The purpose of this study was to develop a long-term model to predict mortality after percutaneous coronary intervention in both patients with ST-segment elevation myocardial infarction and those with more stable coronary disease.

METHODS AND RESULTS

The American College of Cardiology Foundation CathPCI Registry data were linked to the Centers for Medicare and Medicaid Services 100% denominator file by probabilistic matching. Preprocedure demographic and clinical variables from the CathPCI Registry were used to predict the probability of death over 3 years as recorded in the Centers for Medicare and Medicaid Services database. Between 2004 and 2007, 343 466 patients (66%) of 518 195 patients aged ≥65 years undergoing first percutaneous coronary intervention in the CathPCI Registry were successfully linked to Centers for Medicare and Medicaid Services data. This study population was randomly divided into 60% derivation and 40% validation cohorts. Median follow-up was 15 months, with mortality of 3.0% at 30 days and 8.7%, 13.4%, and 18.7% at 1, 2, and 3 years, respectively. Twenty-four characteristics related to demographics, clinical comorbidity, prior history of disease, and indices of disease severity and acuity were identified as being associated with mortality. The C indices in the validation cohorts for patients with and without ST-segment elevation myocardial infarction were 0.79 and 0.78. The model calibrated well across a wide range of predicted probabilities.

CONCLUSIONS

On the basis of the large and nationally representative CathPCI Registry, we have developed a model that has excellent discrimination, calibration, and validation to predict survival up to 3 years after percutaneous coronary intervention.

A novel percutaneous coronary intervention risk score to predict one-year mortality

Clinical and angiographic risk factors associated with adverse outcomes after percutaneous coronary intervention (PCI) have been included in previous validated risk scores. Complications after PCI are known to increase mortality and morbidity but have not been included in any model. Records of 6,932 consecutive patients who underwent PCI from 2000 to 2005 were reviewed. Patients presenting with cardiogenic shock were excluded. Logistic regression and bootstrap methods were used to build an integer risk score for estimating risk of death at 1 year after PCI using baseline, angiographic, and procedural characteristics and postprocedural complications. This risk score was validated in a set of consecutive patients who underwent PCI from 2006 to 2007. The following 8 variables were significantly correlated with outcome: older age, history of diabetes mellitus, chronic renal failure, heart failure, left main coronary artery disease, lower baseline hematocrit, greater hematocrit decrease after PCI, and Thrombolysis In Myocardial Infarction grade <3 flow after PCI. In the validation population (n = 973), average receiver operating characteristic curve area was 0.836. In conclusion, we developed and validated a simple integer risk score, including postprocedural variables that closely predict long-term mortality after PCI. This model emphasizes the significant impact of complications occurring after PCI on long-term outcomes.

Contemporary mortality risk prediction for percutaneous coronary intervention: results from 588,398 procedures in the National Cardiovascular Data Registry

OBJECTIVES

We sought to create contemporary models for predicting mortality risk following percutaneous coronary intervention (PCI).

BACKGROUND

There is a need to identify PCI risk factors and accurately quantify procedural risks to facilitate comparative effectiveness research, provider comparisons, and informed patient decision making.

METHODS

Data from 181,775 procedures performed from January 2004 to March 2006 were used to develop risk models based on pre-procedural and/or angiographic factors using logistic regression. These models were independently evaluated in 2 validation cohorts: contemporary (n = 121,183, January 2004 to March 2006) and prospective (n = 285,440, March 2006 to March 2007).

RESULTS

Overall, PCI in-hospital mortality was 1.27%, ranging from 0.65% in elective PCI to 4.81% in ST-segment elevation myocardial infarction patients. Multiple pre-procedural clinical factors were significantly associated with in-hospital mortality. Angiographic variables provided only modest incremental information to pre-procedural risk assessments. The overall National Cardiovascular Data Registry (NCDR) model, as well as a simplified NCDR risk score (based on 8 key pre-procedure factors), had excellent discrimination (c-index: 0.93 and 0.91, respectively). Discrimination and calibration of both risk tools were retained among specific patient subgroups, in the validation samples, and when used to estimate 30-day mortality rates among Medicare patients.

CONCLUSIONS

Risks for early mortality following PCI can be accurately predicted in contemporary practice. Incorporation of such risk tools should facilitate research, clinical decisions, and policy applications.

In-hospital mortality and complications after coronary angioplasty, City of Rio de Janeiro, Southeastern Brazil

OBJECTIVE

To estimate in-hospital mortality and prevalence of complications of percutaneous transluminal coronary angioplasty (PTCA) in public hospitals.

METHODS

Data for 2,913 PTCA were obtained from the Brazilian National Health System (SUS) Hospital Authorization Database in the city of Rio de Janeiro, Southeastern Brazil, between 1999 and 2003. After simple random sampling and data weighting, 529 medical records of patients undergoing PTCA, including all deaths, in four public hospitals (federal and state university, and federal and state reference hospitals) were studied. Comparison tests of mortality according to patient characteristics, comorbidities, complications, types of PTCA procedures, and indications for PTCA were performed using Poisson's regression models.

RESULTS

The overall in-hospital mortality was 1.6% (range: 0.9-6.8%). The age distribution of mortality was as follows: 0.2% in patients younger than 50; 1.6% in those 50-69; and 2.7% in those older than 69. High mortality was seen in primary and rescue PTCAs: 17.4% and 13.1%, respectively; and mortality in elective PTCA was 0.8%. The main complications during PTCA were dissection (5%; mortality: 11.5%) and artery occlusion (2.6%; mortality: 21.8%). Bleeding was seen in 5.9% of the patients (mortality: 5.6%) and 3.0% required blood transfusion (mortality: 12.0%). The complication of acute myocardial infarction was seen in 1.1% of patients (mortality: 38%) and stroke was associated with a mortality of 17.5%.

CONCLUSIONS

The cardiac in-hospital mortality was high when PTCA was performed for a patient with ST elevation acute myocardial infarction. Elective PTCA had mortality and complications levels above the expected in four public hospitals in the main city of Rio de Janeiro.

Defining the length of stay following percutaneous coronary intervention: an expert consensus document from the Society for Cardiovascular Angiography and Interventions. Endorsed by the American College of Cardiology Foundation

Percutaneous coronary intervention (PCI) is the most common method of coronary revascularization. Over time, as operator skills and technical advances have improved procedural outcomes, the length of stay (LOS) has decreased. However, standardization in the definition of LOS following PCI has been challenging due to significant physician, procedural, and patient variables. Given the increased focus on both patient safety as well as the cost of medical care, system process issues are a concern and provide a driving force for standardization while simultaneously maintaining the quality of patient care. This document: (1) provides a summary of the existing published data on same-day patient discharge following PCI, (2) reviews studies that developed methods to predict risk following PCI, and (3) provides clarification of the terms used to define care settings following PCI. In addition, a decision matrix is proposed for the care of patients following PCI. It is intended to provide both the interventional cardiologist as well as the facilities, in which they are associated, a guide to allow for the appropriate LOS for the appropriate patient who could be considered for early discharge or outpatient intervention.

Prediction of survival after coronary revascularization: modeling short-term, mid-term, and long-term survival

BACKGROUND

Many clinical prediction rules for short-term mortality after coronary revascularization have been developed, validated, and introduced into routine clinical practice. Few rules exist for predicting long-term survival in the modern era of coronary revascularization. We report on the development and validation of models for predicting long-term survival after coronary artery bypass graft surgery and percutaneous coronary intervention on the basis of recent experience.

METHODS

We linked 1987 through 2001 coronary artery bypass graft surgery and 1992 through 2001 percutaneous coronary intervention data from our northern New England registries on 35,000 patients with complete data on risk factors to the National Death Index, ascertaining 7,000 deaths. We partitioned time after revascularization into three periods on the basis of exploratory analysis using generalizations of Cox's semiparametric model to nonproportional hazards and nonlinear log-hazards. These periods were 0 to 3 months, 4 to 18 months, and 19 months and later. For each period, Cox's regression model was used to regress survival on risk factors yielding three models, which were then combined to make long-term predictions.

RESULTS

These models were incorporated into easy-to-use software that yields predicted survival for up to 8 years after revascularization. The Harrell concordance statistic ranged from 72% to 83% for these models.

CONCLUSIONS

We developed and internally validated models that accurately predict long-term survival after coronary artery bypass graft surgery and percutaneous coronary intervention as currently performed. These models using routine clinical data, can be solved with available software, and could be used to enhance informed, patient-centered clinical decision making on the choice of revascularization procedure.

The Toronto score for in-hospital mortality after percutaneous coronary interventions

BACKGROUND

Benchmarking the performance of providers is an increasing priority in many health care economies. In-hospital mortality represents an important and uniformly assessed measure on which to examine the outcome of percutaneous coronary intervention (PCI). Most existing prediction models of in-hospital mortality after PCI were derived from 1990s data, and their current relevance is uncertain.

METHODS

From consecutive PCIs performed during 2000-2008, derivation and validation cohorts of 10,694 and 5,347 patients, respectively, were analyzed. Logistic regression for in-hospital death yielded integer risk weights for each independent predictor variable. These were summed for each patient to create the Toronto PCI risk score.

RESULTS

Death occurred in 1.3% of patients. Independent predictors with associated risk weights in parentheses were as follows: age 40 to 49 y (1), 50 to 59 y (2), 60 to 69 y (3), 70 to 79 y (4), and > or =80 y (5); diabetes (2); renal insufficiency (2); New York Heart Association class 4 (3); left ventricular ejection fraction <20% (3); myocardial infarction in the previous month (3); multivessel disease (1); left main disease (2); rescue or facilitated PCI (3); primary PCI (4); and shock (6). The model had a receiver operator curve of 0.96 and Hosmer-Lemeshow goodness-of-fit P = .16 in the validation set. Four previously published external models were tested in the entire data set. Three models had ROC curves significantly less than the Toronto PCI score, and all 4 showed significant levels of imprecision.

CONCLUSIONS

The Toronto PCI mortality score is an accurate and contemporary predictive tool that permits evaluation of risk-stratified outcomes and aids counseling of patients undergoing PCI.

Validation of the Mayo Clinic Risk Score for In-Hospital Mortality After Percutaneous Coronary Interventions Using the National Cardiovascular Data Registry

Background— We sought to validate the recently developed Mayo Clinic Risk Score model for in-hospital mortality after percutaneous coronary intervention using an independent data set. The Mayo Clinic Risk Score has 7 simple clinical and noninvasive variables, available before coronary angiography, for prediction of in-hospital mortality. External validation using an independent data set would support broader applicability of the model.

Methods and Results— In-hospital mortality after percutaneous coronary intervention on 309 351 patients from the National Cardiovascular Data Registry admitted from January 1, 2004, to March, 30, 2006, was studied. Using the Mayo Clinic Risk Score equation, we assigned predicted probabilities of death to each patient. The area under the receiver-operating characteristics curve was 0.884, indicating excellent discrimination overall as well as among subgroups, including gender, diabetes mellitus, renal failure, low ejection fraction, different age groups, and multivessel disease. Ninety-seven percent of patients undergoing percutaneous coronary intervention had a Mayo Clinic Risk Score <10, indicating low to intermediate risk. The Mayo Clinic Risk Score model initially slightly underpredicted event rates when applied in National Cardiovascular Data Registry data (observed 1.23% versus predicted 1.10%), but this underprediction was corrected after recalibration. The recalibrated risk score discriminated (c index=0.885) and calibrated well in an National Cardiovascular Data Registry validation data set consisting of procedures performed between April 1, 2006, and March 30, 2007.

Conclusions— Seven variables can be combined into a convenient risk scoring system before coronary angiography is performed to predict in-hospital mortality after percutaneous coronary intervention. This model may be useful for providing patients with individualized, evidence-based estimates of procedural risk as part of the informed consent process before percutaneous coronary intervention.

Predicting major adverse cardiac events after percutaneous coronary intervention: the Texas Heart Institute risk score

BACKGROUND

Many models have been devised in the past to predict adverse outcomes after PCI, but with rapid advancements in this field, a new risk-prediction model may be needed. The purpose of our study was to identify the clinical and angiographic variables associated with adverse cardiac events after percutaneous coronary intervention (PCI) and to construct a simple bedside tool for risk stratification of PCI patients.

METHODS

Using our institution's database, we analyzed data from 9,494 patients who underwent PCI between January 1, 1996, and December 31, 2002 (ie, during the bare-metal stent era). Predictors of major adverse cardiac events--death, myocardial infarction, stroke, and repeat revascularization by emergent coronary artery bypass grafting or PCI--were identified by multivariate logistic regression analysis using baseline clinical, angiographic, and procedural variables. A simple integer score was constructed by multiplying the beta coefficient for each variable by a constant and rounding the result to the nearest integer. The score was validated in 5,545 patients who underwent PCI between January 1, 2003, and December 31, 2006 (ie, during the drug-eluting stent era).

RESULTS

Multivariate regression analysis identified emergent procedure, urgent procedure, unstable angina, acute myocardial infarction, renal insufficiency, hypertension, congestive heart failure, peripheral vascular disease, type C lesion, presence of thrombus, and number of stents placed as independent predictors of adverse events after PCI. The model had good overall discrimination (area under the receiver operator characteristic curve 0.701), and the model fitted the validation cohort adequately.

CONCLUSIONS

Risk of complications after PCI can be assessed with this simple tool, which may permit comparisons between different operators as well as different hospitals.

Comorbid conditions and outcomes after percutaneous coronary intervention

OBJECTIVE

To evaluate whether adding comorbid conditions to a risk model can help predict in-hospital outcome and long-term mortality after percutaneous coronary intervention (PCI).

DESIGN

Retrospective chart review

SETTING

Academic medical centre.

PATIENTS

7659 patients who had 9032 PCIs.

INTERVENTIONS

PCI performed at Mayo Clinic between 1 January 1999 and 30 June 2004.

MAIN OUTCOME MEASURES

The Mayo Clinic Risk Score (MCRS) and the coronary artery disease (CAD)-specific index for determination of comorbid conditions in all patients.

RESULTS

The mean (SD) MCRS score was 6.5 (2.9). The CAD-specific index was 0 or 1 in 46%, 2 or 3 in 30% and 4 or higher in 24%. The rate of in-hospital major adverse cardiovascular events (MACE) increased with higher MCRS and CAD-specific index (Cochran-Armitage test, p<0.001 for both models). The c-statistic for the MCRS for in-hospital MACE was 0.78; adding the CAD-specific index did not improve its discriminatory ability for in-hospital MACE (c-statistic = 0.78; likelihood ratio test, p = 0.29). A total of 707 deaths after dismissal occurred after 7253 successful procedures. The c-statistic for all-cause mortality was 0.69 for the MCRS model alone and 0.75 for the MCRS and CAD-specific indices together (likelihood ratio test, p<0.001), indicating significant improvement in the discriminatory ability.

CONCLUSIONS

Addition of comorbid conditions to the MCRS adds significant prognostic information for post-dismissal mortality but adds little prognostic information about in-hospital complications after PCI. Such health-status measures should be included in future risk stratification models that predict long-term mortality after PCI.

Prognostication in severe acute respiratory syndrome: a retrospective time-course analysis of 1312 laboratory-confirmed patients in Hong Kong

Background and objective:  The temporal importance of prognostic indicators for severe acute respiratory syndrome (SARS) has not been studied. This study identified the various clinical prognostic factors for SARS and described the temporal evolution of these factors in the course of the SARS illness in Hong Kong in 2003.

Methods:  A retrospective analysis of the entire Hong Kong cohort of 1312 laboratory‐confirmed SARS patients aged 15–74 years was undertaken. Demographic, clinical and laboratory data at presentation and investigative data during the first 10 days of illness from the time of symptom onset were compiled. Two adverse outcomes were examined: hospital mortality and the development of oxygenation failure based on the estimated PaO₂/FiO₂ ratio of <200 mm Hg. Logistic regression was used to identify the association between these prognostic factors and outcomes.

Results:  Based on adjusted odds ratios with a P‐value of <0.05, older age, male gender, elevated pulse rate and elevated neutrophil count were all predictive of oxygenation failure and death during the 10‐day illness. Raised serum albumin and creatinine phosphokinase (CPK) levels were predictive of hospital mortality during this period. The presenting ALT and CPK level and the day 7 and day 10 platelet counts were predictive of oxygenation failure while the day 7 LDH was predictive of death. Contact exposure outside health‐care institutions also appeared to carry higher risk of death.

Conclusion:  This large‐scale analysis identified important discriminatory parameters related to the patients’ demographic profile (age and gender), severity of illness (pulse rate and neutrophil count), and multisystem derangement (platelet count, CPK, ALT and LDH), all of which prognosticated adverse outcomes during the SARS episode. While age, pulse rate and neutrophil count consistently remained significant prognosticators during the first 10 days of illness, the prognostic impact of other derangements was more time‐course dependent. Clinicians should be aware of the time‐course evolution of these prognosticators.

Bedside estimation of risk from percutaneous coronary intervention: the new Mayo Clinic risk scores

OBJECTIVE

To derive risk models for percutaneous coronary intervention (PCI) outcomes from clinical and laboratory variables available before the procedure so they can be used for preprocedure risk stratification.

PATIENTS AND METHODS

Using the Mayo Clinic registry, we analyzed 9035 PCIs on 7640 unique patients from January 1, 2000, through April 30, 2005. We included only the first PCI per patient (n=7457). Logistic regression was used to model the calculated risk score and major procedural complications. Separate risk models were made for mortality and major adverse cardiovascular events (MACE) derived solely from baseline and laboratory characteristics. Final risk scores for procedural death, defined as any death during the index hospitalization, and MACE contained the same 7 variables (age, myocardial infarction less than or equal to 24 hours, preprocedural shock, serum creatinine level, left ventricular ejection fraction, congestive heart failure, and peripheral artery disease).

RESULTS

Models had adequate goodness of fit, and areas under the receiver operating characteristic curve were 0.74 and 0.89 for MACE and procedural death, respectively, indicating excellent overall discrimination. The model was robust across many subgroups, including those undergoing elective PCI, those having diabetes mellitus, and elderly patients. Bootstrap analysis indicated that the model was not overfit to the available data set.

CONCLUSION

Before coronary angiography is performed, a risk-scoring system based on 7 variables can be used conveniently to predict cardiovascular complications after PCI. This model may be useful for providing patients with individualized, evidence-based estimates of procedural risk as part of the informed consent process.

Effects of SVM parameter optimization on discrimination and calibration for post-procedural PCI mortality

Support vector machines (SVM) have become popular among machine learning researchers, but their applications in biomedicine have been somewhat limited. A number of methods, such as grid search and evolutionary algorithms, have been utilized to optimize model parameters of SVMs. The sensitivity of the results to changes in optimization methods has not been investigated in the context of medical applications. In this study, radial-basis kernel SVM and polynomial kernel SVM mortality prediction models for percutaneous coronary interventions were optimized using (a) mean-squared error, (b) mean cross-entropy error, (c) the area under the receiver operating characteristic, and (d) the Hosmer-Lemeshow goodness-of-fit test (HL chi(2)). A threefold cross-validation inner and outer loop method was used to select the best models using the training data, and evaluations were based on previously unseen test data. The results were compared to those produced by logistic regression models optimized using the same indices. The choice of optimization parameters had a significant impact on performance in both SVM kernel types.

Prognosis in critical care

Prognostic risk prediction models have been employed in the intensive care unit (ICU) setting since the 1980s and provide health care providers with important information to help inform decisions related to treatment and prognosis, as well as to compare outcomes across institutions. Prognostic models for critical care are among the most widely utilized and tested predictive models in healthcare. In this article, we review and compare mortality prediction models, including the APACHE (1981), SAPS (1984), APACHE-II (1985), MPM (1987), APACHE-III (1991), SAPS-II (1993), and MPM-II (1993). We emphasize the importance of model calibration in this domain. In addition, we present a brief review of the statistical methodology, multiple logistic regression, which underlies most of the models currently used in critical care.

Relation of preprocedural statin therapy to in-hospital procedural complications following percutaneous coronary interventions in patients with hyperlipidemia

We investigated whether the observed protective effect of hyperlipidemia is stronger in patients who take statins and, if so, whether that effect is likely due to patient characteristics associated with statin use. In-hospital complications and long-term outcomes of patients with hyperlipidemia (cholesterol level > or = 240 mg/dl) undergoing percutaneous coronary interventions (PCI) on statins (group Ia, n = 2,052) and not on statins (group Ib, n = 1,650) were compared with 726 patients with lower cholesterol levels (group II). Despite a higher prevalence of co-morbidities and worse angiographic characteristics in patients with hyperlipidemia, patients in group Ia had significantly lower in-hospital mortality (0% vs 2% in the other 2 groups, p < 0.001), a lower increase in the postprocedure creatine kinase-MB fraction (14% vs 27% in group Ib and 28% in group II, p < 0.001), and fewer PCI complications (15% vs 30% in groups Ib and II, p < 0.001). After adjustment, patients in group Ia had a significant decrease in complications (odds ratio 0.72, 95% confidence interval 0.65 to 0.92, p = 0.009). In contrast, those in group Ib had outcomes similar to those of patients with lower cholesterol. After application of propensity analysis to adjust for the likelihood of receiving statins based on clinical, angiographic, and procedural characteristics, group Ia had fewer in-hospital complications (odds ratio 0.75, 95% confidence interval 0.62 to 0.90, p = 0.002) and lower in-hospital mortality (odds ratio 0.32, 95% confidence interval 0.12 to 0.84, p = 0.021). After successful PCI, overall survival after dismissal and survival free of myocardial infarction and target vessel revascularization were similar. In conclusion, hyperlipidemia per se is not associated with lower in-hospital complications after PCI. The benefit is largely limited to patients on statin treatment.

A risk score to predict in-hospital mortality for percutaneous coronary interventions

OBJECTIVES

Our purpose was to develop a risk score to predict in-hospital mortality for percutaneous coronary intervention (PCI) using a statewide population-based PCI registry.

BACKGROUND

Risk scores predicting adverse outcomes after PCI have been developed from a single or a small group of hospitals, and their abilities to be generalized to other patient populations might be affected.

METHODS

A logistic regression model was developed to predict in-hospital mortality for PCI using data from 46,090 procedures performed in 41 hospitals in the New York State Percutaneous Coronary Intervention Reporting System in 2002. A risk score was derived from this model and was validated using 2003 data from New York.

RESULTS

The risk score included nine significant risk factors (age, gender, hemodynamic state, ejection fraction, pre-procedural myocardial infarction, peripheral arterial disease, congestive heart disease, renal failure, and left main disease) that were consistent with other reports. The point values for risk factors range from 1 to 9, and the total risk score ranges from 0 to 40. The observed and recalibrated predicted risks in 2003 were highly correlated for all PCI patients as well as for those in the higher-risk subgroup who suffered myocardial infarctions within 24 h before the procedure. The total risk score for mortality is strongly associated with complication rates and length of stay in the 2003 PCI data.

CONCLUSIONS

The risk score accurately predicted in-hospital death for PCI procedures using future New York data. Its performance in other patient populations needs to be further studied.

The impact of ejection fraction on outcomes after percutaneous coronary intervention in patients with congestive heart failure: an analysis of the National Heart, Lung, and Blood Institute Percutaneous Transluminal Coronary Angioplasty Registry and Dynamic Registry

BACKGROUND

Patients with congestive heart failure (CHF) have higher rates of adverse outcomes after percutaneous coronary intervention (PCI). A comprehensive analysis of outcomes in patients with CHF in the current era has not been done. We studied the outcomes of patients with CHF who underwent PCI in the National Heart, Lung, and Blood Institute-sponsored Percutaneous Transluminal Coronary Angioplasty (PTCA) and Dynamic registries.

METHODS

We evaluated demographic and angiographic characteristics and the clinical outcomes of patients with CHF in the Dynamic Registry and the PTCA Registry, excluding patients with acute myocardial infarction. In the Dynamic Registry, patients with CHF (n = 503) were compared with patients without CHF (n = 4194), and patients with CHF with a preserved ejection fraction (EF) (n = 134) were compared with patients with CHF who have a reduced EF (n = 199). The patients with CHF in the 1997 through 2001 Dynamic Registry (n = 236) were then similarly compared with patients with CHF in the earlier PTCA Registry (n = 117).

RESULTS

In the Dynamic Registry, compared with patients without CHF, patients with CHF had a higher-risk clinical and angiographic profile, and a higher mortality rate both inhospital (2.6% vs 0.4%, P < or = .001) and at 1 year (13.1% vs 3.0%, P < .001). Patients with reduced EF had higher inhospital mortality rates and a trend toward higher mortality at 1 year. The patients with CHF in the Dynamic Registry compared with those in the PTCA Registry had a higher risk profile yet had significantly higher procedural success rates and improved clinical outcomes.

CONCLUSIONS

Although CHF remains a strong predictor of adverse outcomes after PCI, significant improvement seen in the past decade is likely related to improved procedural techniques and improved medical therapy.

A global goodness-of-fit test for receiver operating characteristic curve analysis via the bootstrap method

OBJECTIVE

Medical classification accuracy studies often yield continuous data based on predictive models for treatment outcomes. A popular method for evaluating the performance of diagnostic tests is the receiver operating characteristic (ROC) curve analysis. The main objective was to develop a global statistical hypothesis test for assessing the goodness-of-fit (GOF) for parametric ROC curves via the bootstrap.

DESIGN

A simple log (or logit) and a more flexible Box-Cox normality transformations were applied to untransformed or transformed data from two clinical studies to predict complications following percutaneous coronary interventions (PCIs) and for image-guided neurosurgical resection results predicted by tumor volume, respectively. We compared a non-parametric with a parametric binormal estimate of the underlying ROC curve. To construct such a GOF test, we used the non-parametric and parametric areas under the curve (AUCs) as the metrics, with a resulting p value reported.

RESULTS

In the interventional cardiology example, logit and Box-Cox transformations of the predictive probabilities led to satisfactory AUCs (AUC=0.888; p=0.78, and AUC=0.888; p=0.73, respectively), while in the brain tumor resection example, log and Box-Cox transformations of the tumor size also led to satisfactory AUCs (AUC=0.898; p=0.61, and AUC=0.899; p=0.42, respectively). In contrast, significant departures from GOF were observed without applying any transformation prior to assuming a binormal model (AUC=0.766; p=0.004, and AUC=0.831; p=0.03), respectively.

CONCLUSIONS

In both studies the p values suggested that transformations were important to consider before applying any binormal model to estimate the AUC. Our analyses also demonstrated and confirmed the predictive values of different classifiers for determining the interventional complications following PCIs and resection outcomes in image-guided neurosurgery.

Discrimination and calibration of mortality risk prediction models in interventional cardiology

OBJECTIVES

Using a local percutaneous coronary intervention (PCI) data repository, we sought to compare the performance of a number of local and well-known mortality models with respect to discrimination and calibration.

BACKGROUND

Accurate risk prediction is important for a number of reasons including physician decision support, quality of care assessment, and patient education. Current evidence on the value of applying PCI risk models to individual cases drawn from a different population is controversial.

METHODS

Data were collected from January 01, 2002 to September 30, 2004 on 5216 consecutive percutaneous coronary interventions at Brigham and Women's Hospital (Boston, MA). Logistic regression was used to create a local risk model for in-hospital mortality in these procedures, and a number of statistical methods were used to compare the discrimination and calibration of this new and old local risk models, as well as the Northern New England Cooperative Group, New York State (1992 and 1997), University of Michigan consortium, American College of Cardiology-National Cardiovascular Data Registry, and The Cleveland Clinic Foundation risk prediction models. Areas under the ROC (AUC) curves were used to evaluate discrimination, and the Hosmer-Lemeshow (HL) goodness-of-fit test and calibration curves assessed applicability of the models to individual cases.

RESULTS

Multivariate risk factors included in the newly constructed local model were: age, prior intervention, diabetes, unstable angina, salvage versus elective procedure, cardiogenic shock, acute myocardial infarction (AMI), and left anterior descending artery intervention. The area under the ROC curve (AUC) was 0.929 (SE=0.017), and the p value for the Hosmer-Lemeshow (HL) goodness-of-fit was 0.473. This indicates good discrimination and calibration. Bootstrap re-sampling indicated AUC stability. Evaluation of the external models showed an AUC range from 0.82 to 0.90 indicating good discrimination across all models, but poor calibration (HL p value < or = 0.0001).

CONCLUSIONS

Validation of AUC values across all models suggests that certain risk factors have remained important over the last decade. However, the lack of calibration suggests that small changes in patient populations and data collection methods quickly reduce the accuracy of patient level estimations over time. Possible solutions to this problem involve either recalibration of models using local data or development of new local models.

Prediction of complications following nonemergency percutaneous coronary interventions

Previous models for prediction of complications after percutaneous coronary interventions (PCIs) have included in-hospital mortality and major in-hospital complications. In general, these models have excluded elevated cardiac biomarkers as a complication. We sought to determine whether a risk model could predict complications, including biomarker elevation, in patients undergoing nonemergency PCI. We examined the outcomes of nonemergency PCI performed on patients at Mayo Clinic from 2000 to 2003. The primary end point was in-hospital complications of death, myocardial infarction (MI) (Q-wave MI, or post-PCI creatine kinase-MB elevation >or=3 times the upper limit of normal), emergency coronary artery bypass grafting, or stroke. We used the Hosmer-Lemeshow test to demonstrate the adequacy of the model fit, and the c-index for discriminatory ability of the model. Of 2,894 nonemergency PCIs, the end point was noted in 232 (8%). The final prediction model included vein graft intervention (odds ratio [OR] 2.19), angiographic thrombus (OR 2.12), preprocedure stenosis of a minor (OR 1.98) or major (OR 1.62) side branch, and type C lesion (OR 1.48). The model had modest ability to discriminate between event and nonevent patients (c = 0.641). In the 500 bootstrap samples for internal validation, the c-index was 0.642 +/- 0.020, indicating only fair discriminatory ability. The average number of observed events was 232.0 +/- 14.7 compared with 232.1 +/- 2.5 expected events (average difference -0.06 +/- 14.5). In conclusion, the 5 risk variables associated with an increased risk of complications in patients undergoing elective PCI included vein graft intervention, presence of angiographic thrombus, stenosis of a major or minor side branch, and type C lesion; however, the discriminatory ability of the model derived from the variables was only modest.

Multivariate prediction of major adverse cardiac events after 9914 percutaneous coronary interventions in the north west of England

OBJECTIVE

To develop a multivariate prediction model for major adverse cardiac events (MACE) after percutaneous coronary interventions (PCIs) by using the North West Quality Improvement Programme in Cardiac Interventions (NWQIP) PCI Registry.

SETTING

All NHS centres undertaking adult PCIs in north west England.

METHODS

Retrospective analysis of prospectively collected data on 9914 consecutive patients undergoing adult PCI between 1 August 2001 and 31 December 2003. A multivariate logistic regression analysis was undertaken, with the forward stepwise technique, to identify independent risk factors for MACE. The area under the receiver operating characteristic (ROC) curve and the Hosmer-Lemeshow goodness of fit statistic were calculated to assess the performance and calibration of the model, respectively. The statistical model was internally validated by using the technique of bootstrap resampling.

MAIN OUTCOME MEASURES

MACE, which were in-hospital mortality, Q wave myocardial infarction, emergency coronary artery bypass graft surgery, and cerebrovascular accidents.

RESULTS

Independent variables identified with an increased risk of developing MACE were advanced age, female sex, cerebrovascular disease, cardiogenic shock, priority, and treatment of the left main stem or graft lesions during PCI. The ROC curve for the predicted probability of MACE was 0.76, indicating a good discrimination power. The prediction equation was well calibrated, predicting well at all levels of risk. Bootstrapping showed that estimates were stable.

CONCLUSIONS

A contemporaneous multivariate prediction model for MACE after PCI was developed. The NWQIP tool allows calculation of the risk of MACE permitting meaningful risk adjusted comparisons of performance between hospitals and operators.

A Prognostic Model for HIV Seroconversion Among Injection Drug Users as a Tool for Stratification in Clinical Trials

OBJECTIVE

The main goal of this study was to construct a prognostic model for HIV seroconversion among injection drug users (IDUs) using easy-to-measure risk indicators.

DESIGN

Cox proportional hazards regression modeling was used for risk stratification in a heterogeneous population of IDUs with regards to HIV risk-taking behaviors.

METHODS

Subjects were recruited in a prospective cohort of IDUs followed between September 1992 and October 2001. A total of 1602 men, seronegative at enrollment with at least 1 follow-up visit, were included in the analyses. Only variables that consistently predict HIV seroconversion in several settings were considered. The final model was used to assign a risk score for each participant.

RESULTS

Three risk indicators were included in the risk score to predict HIV seroconversion: unstable housing, average cocaine injections per day, and having shared a syringe with a known HIV-positive partner. Kaplan-Meier survival functions were generated and risk score values stratified in 3 groups. HIV incidence rates per 100 person-years were as follows: 0.91 (95% CI, 0.55-1.52) for the low-risk group, 3.10 (95% CI, 2.49-3.84) for the moderate-risk group, and 7.82 (95% CI, 6.30-9.73) for the high-risk group (log-rank P value < 0.0001).

CONCLUSION

If validated in other settings, this risk score may improve the prediction of outcome and allow more accurate stratification in clinical trials.