Clinical Prediction Models for Cardiovascular Disease: Tufts Predictive Analytics and Comparative Effectiveness Clinical Prediction Model Database

Research inefficiencies in external validation studies of the Framingham Wilson coronary heart disease risk rule: A systematic review

BACKGROUND

External validation studies create evidence about a clinical prediction rule's (CPR's) generalizability by evaluating and updating the CPR in populations different from those used in the derivation, and also by contributing to estimating its overall performance when meta-analysed in a systematic review. While most cardiovascular CPRs do not have any external validation, some CPRs have been externally validated repeatedly. Hence, we examined whether external validation studies of the Framingham Wilson coronary heart disease (CHD) risk rule contributed to generating evidence to their full potential.

METHODS

A forward citation search of the Framingham Wilson CHD risk rule's derivation study was conducted to identify studies that evaluated the Framingham Wilson CHD risk rule in different populations. For external validation studies of the Framingham Wilson CHD risk rule, we examined whether authors updated the Framingham Wilson CHD risk rule when it performed poorly. We also assessed the contribution of external validation studies to understanding the Predicted/Observed (P/O) event ratio and c statistic of the Framingham Wilson CHD risk rule.

RESULTS

We identified 98 studies that evaluated the Framingham Wilson CHD risk rule; 40 of which were external validation studies. Of these 40 studies, 27 (67.5%) concluded the Framingham Wilson CHD risk rule performed poorly but did not update it. Of 23 external validation studies conducted with data that could be included in meta-analyses, 13 (56.5%) could not fully contribute to the meta-analyses of P/O ratio and/or c statistic because these performance measures were neither reported nor could be calculated from provided data.

DISCUSSION

Most external validation studies failed to generate evidence about the Framingham Wilson CHD risk rule's generalizability to their full potential. Researchers might increase the value of external validation studies by presenting all relevant performance measures and by updating the CPR when it performs poorly.

The stroke outcome optimization project: Acute ischemic strokes from a comprehensive stroke center

Stroke is a leading cause of disability, and Magnetic Resonance Imaging (MRI) is routinely acquired for acute stroke management. Publicly sharing these datasets can aid in the development of machine learning algorithms, particularly for lesion identification, brain health quantification, and prognosis. These algorithms thrive on large amounts of information, but require diverse datasets to avoid overfitting to specific populations or acquisitions. While there are many large public MRI datasets, few of these include acute stroke. We describe clinical MRI using diffusion-weighted, fluid-attenuated and T1-weighted modalities for 1715 individuals admitted in the upstate of South Carolina, of whom 1461 have acute ischemic stroke. Demographic and impairment data are provided for 1106 of the stroke survivors from this cohort. Our validation demonstrates that machine learning can leverage the imaging data to predict stroke severity as measured by the NIH Stroke Scale/Score (NIHSS). We share not only the raw data, but also the scripts for replicating our findings. These tools can aid in education, and provide a benchmark for validating improved methods.

Artificial Intelligence in Cardiovascular Care-Part 2: Applications: JACC Review Topic of the Week

Recent artificial intelligence (AI) advancements in cardiovascular care offer potential enhancements in effective diagnosis, treatment, and outcomes. More than 600 U.S. Food and Drug Administration-approved clinical AI algorithms now exist, with 10% focusing on cardiovascular applications, highlighting the growing opportunities for AI to augment care. This review discusses the latest advancements in the field of AI, with a particular focus on the utilization of multimodal inputs and the field of generative AI. Further discussions in this review involve an approach to understanding the larger context in which AI-augmented care may exist, and include a discussion of the need for rigorous evaluation, appropriate infrastructure for deployment, ethics and equity assessments, regulatory oversight, and viable business cases for deployment. Embracing this rapidly evolving technology while setting an appropriately high evaluation benchmark with careful and patient-centered implementation will be crucial for cardiology to leverage AI to enhance patient care and the provider experience.

Development and validation of prediction models for papillary thyroid cancer structural recurrence using machine learning approaches

BACKGROUND

Although papillary thyroid cancer (PTC) patients are known to have an excellent prognosis, up to 30% of patients experience disease recurrence after initial treatment. Accurately predicting disease prognosis remains a challenge given that the predictive value of several predictors remains controversial. Thus, we investigated whether machine learning (ML) approaches based on comprehensive predictors can predict the risk of structural recurrence for PTC patients.

METHODS

A total of 2244 patients treated with thyroid surgery and radioiodine were included. Twenty-nine perioperative variables consisting of four dimensions (demographic characteristics and comorbidities, tumor-related variables, lymph node (LN)-related variables, and metabolic and inflammatory markers) were analyzed. We applied five ML algorithms-logistic regression (LR), support vector machine (SVM), extreme gradient boosting (XGBoost), random forest (RF), and neural network (NN)-to develop the models. The area under the receiver operating characteristic (AUC-ROC) curve, calibration curve, and variable importance were used to evaluate the models' performance.

RESULTS

During a median follow-up of 45.5 months, 179 patients (8.0%) experienced structural recurrence. The non-stimulated thyroglobulin, LN dissection, number of LNs dissected, lymph node metastasis ratio, N stage, comorbidity of hypertension, comorbidity of diabetes, body mass index, and low-density lipoprotein were used to develop the models. All models showed a greater AUC (AUC = 0.738 to 0.767) than did the ATA risk stratification (AUC = 0.620, DeLong test: P < 0.01). The SVM, XGBoost, and RF model showed greater sensitivity (0.568, 0.595, 0.676), specificity (0.903, 0.857, 0.784), accuracy (0.875, 0.835, 0.775), positive predictive value (PPV) (0.344, 0.272, 0.219), negative predictive value (NPV) (0.959, 0.959, 0.964), and F1 score (0.429, 0.373, 0.331) than did the ATA risk stratification (sensitivity = 0.432, specificity = 0.770, accuracy = 0.742, PPV = 0.144, NPV = 0.938, F1 score = 0.216). The RF model had generally consistent calibration compared with the other models. The Tg and the LNR were the top 2 important variables in all the models, the N stage was the top 5 important variables in all the models.

CONCLUSIONS

The RF model achieved the expected prediction performance with generally good discrimination, calibration and interpretability in this study. This study sheds light on the potential of ML approaches for improving the accuracy of risk stratification for PTC patients.

TRIAL REGISTRATION

Retrospectively registered at www.chictr.org.cn (trial registration number: ChiCTR2300075574, date of registration: 2023-09-08).

Heart rate complexity helps mortality prediction in the intensive care unit: A pilot study using artificial intelligence

BACKGROUND

In intensive care units (ICUs), accurate mortality prediction is crucial for effective patient management and resource allocation. The Simplified Acute Physiology Score II (SAPS-2), though commonly used, relies heavily on comprehensive clinical data and blood samples. This study sought to develop an artificial intelligence (AI) model utilizing key hemodynamic parameters to predict ICU mortality within the first 24 h and assess its performance relative to SAPS-2.

METHODS

We conducted an analysis of select hemodynamic parameters and the structure of heart rate curves to identify potential predictors of ICU mortality. A machine-learning model was subsequently trained and validated on distinct patient cohorts. The AI algorithm's performance was then compared to the SAPS-2, focusing on classification accuracy, calibration, and generalizability.

MEASUREMENTS AND MAIN RESULTS

The study included 1298 ICU admissions from March 27th, 2015, to March 27th, 2017. An additional cohort from 2022 to 2023 comprised 590 patients, resulting in a total dataset of 1888 patients. The observed mortality rate stood at 24.0%. Key determinants of mortality were the Glasgow Coma Scale score, heart rate complexity, patient age, duration of diastolic blood pressure below 50 mmHg, heart rate variability, and specific mean and systolic blood pressure thresholds. The AI model, informed by these determinants, exhibited a performance profile in predicting mortality that was comparable, if not superior, to the SAPS-2.

CONCLUSIONS

The AI model, which integrates heart rate and blood pressure curve analyses with basic clinical parameters, provides a methodological approach to predict in-hospital mortality in ICU patients. This model offers an alternative to existing tools that depend on extensive clinical data and laboratory inputs. Its potential integration into ICU monitoring systems may facilitate more streamlined mortality prediction processes.

The design of studies testing the effectiveness of risk-guided care has many challenges: a scoping review addressing key considerations

OBJECTIVES

Studies evaluating the effectiveness of care based on patients' risk of adverse outcomes (risk-guided care) use a variety of study designs. In this scoping review, using examples, we review characteristics of relevant studies and present key design features to optimize the trustworthiness of results.

STUDY DESIGN AND SETTING

We searched five online databases for studies evaluating the effect of risk-guided care among adults on clinical outcomes, process, or cost. Pairs of reviewers independently performed screening and data abstraction. We descriptively summarized the study design and characteristics.

RESULTS

Among 14,561 hits, we identified 116 eligible studies. Study designs included randomized controlled trials (RCTs), post hoc analysis of RCTs, and retrospective or prospective cohort studies. Challenges and sources of bias in the design included limited performance of predictive models, contamination, inadequacy to address the credibility of subgroup effects, absence of differences in care across risk strata, reporting only process measures as opposed to clinical outcomes, and failure to report benefits and harms.

CONCLUSION

To assess the benefit of risk-guided care, RCTs provide the most trustworthy evidence. Observational studies offer an alternative but are hampered by confounding and other limitations. Reaching valid conclusions when testing risk-guided care requires addressing the challenges identified in our review.

Psychiatric disorders and cardiovascular diseases: A mendelian randomization study

Background

Previous researches have demonstrated a connection between psychiatric disorders and cardiovascular diseases (CVDs), but the cause-and-effect relationship is still unclear. To that goal, the mendelian randomization (MR) method was used to study the causal link between psychiatric disorders and CVDs.

Methods

Genome-wide association studies (GWAS) data were collected for four CVDs, including coronary artery disease (n = 547,261), atrial fibrillation (n = 537,409), heart failure (n = 977,323) and ischemic stroke (n = 440,328). Summary data for four psychiatric disorders, including bipolar disorder (n = 51,710), major depressive disorder (n = 480,359), schizophrenia (n = 127,906) and attention deficit hyperactivity disorder (n = 55,374), came from the Psychiatric Genomics Consortium (PGC). All participants were European. The IVW method was mainly used, and the reliability of the results was increased using sensitivity analyses such as MR-Egger, Cochrane's Q test, MR-PRESSO and leave-one-out.

Results

MR revealed that the attention deficit hyperactivity disorder was linked to an increased risk of atrial fibrillation (OR, 1.085; 95% CI, 1.021-1.153; P = 0.008), heart failure (OR, 1.117; 95% CI, 1.044-1.195; P = 0.001), and ischemic stroke (OR, 1.146; 95% CI, 1.052-1.248; P = 0.002). The schizophrenia was linked to an increased risk of heart failure (OR, 1.035; 95% CI, 1.006-1.066; P = 0.017), but was found to be suggestively inverse associated with coronary artery disease (OR, 0.969; 95% CI, 0.941-0.997; P = 0.03). The major depressive disorder was associated with higher odds of coronary artery disease (OR, 1.109; 95% CI, 1.018-1.208; P = 0.018), while the bipolar disorder was linked to a reduced incidence of coronary artery disease (OR, 0.894; 95% CI, 0.831-0.961; P = 0.002) and heart failure (OR, 0.889; 95% CI, 0.829-0.955; P = 0.001). There were no clear relationships between other psychiatric disorders and CVDs.

Conclusion

The results provide genetic proof of a possible causal relationship between psychiatric disorders and CVDs. These results imply that psychiatric disorders may be the cause of some CVDs, and that some abnormal mental states may increase or reduce the likelihood of CVDs, providing guidance for the CVDs prevention.

Dynamic prediction modeling of postoperative mortality among patients undergoing surgical aortic valve replacement in a statewide cohort over a 12-year period

Objective

Clinical prediction models for surgical aortic valve replacement mortality, are valuable decision tools but are often limited in their ability to account for changes in medical practice, patient selection, and the risk of outcomes over time. Recent research has identified methods to update models as new data accrue, but their effect on model performance has not been rigorously tested.

Methods

The study population included 44,546 adults who underwent an isolated surgical aortic valve replacement from January 1, 1999, to December 31, 2018, statewide in Pennsylvania. After chronologically splitting the data into training and validation sets, we compared calibration, discrimination, and accuracy measures amongst a nonupdating model to 2 methods of model updating: calibration regression and the novel dynamic logistic state space model.

Results

The risk of mortality decreased significantly during the validation period (P < .01) and the nonupdating model demonstrated poor calibration and reduced accuracy over time. Both updating models maintained better calibration (Hosmer-Lemeshow χ2 statistic) than the nonupdating model: nonupdating (156.5), calibration regression (4.9), and dynamic logistic state space model (8.0). Overall accuracy (Brier score) was consistently better across both updating models: dynamic logistic state space model (0.0252), calibration regression (0.0253), and nonupdating (0.0256). Discrimination improved with the dynamic logistic state space model (area under the curve, 0.696) compared with the nonupdating model (area under the curve, 0.685) and calibration regression method (area under the curve, 0.687).

Conclusions

Dynamic model updating can improve model accuracy, discrimination, and calibration. The decision as to which method to use may depend on which measure is most important in each clinical context. Because competing therapies have emerged for valve replacement models, updating may guide clinical decision making.

Current role and future perspectives of artificial intelligence in echocardiography

Echocardiography is an essential tool in diagnostic cardiology and is fundamental to clinical care. Artificial intelligence (AI) can help health care providers serving as a valuable diagnostic tool for physicians in the field of echocardiography specially on the automation of measurements and interpretation of results. In addition, it can help expand the capabilities of research and discover alternative pathways in medical management specially on prognostication. In this review article, we describe the current role and future perspectives of AI in echocardiography.

Establishment and Validation of a Novel Prediction Model for Early Natural Biochemical Recurrence After Radical Prostatectomy Based on Post-Operative PSA at Sixth Week

Objective

Based on post-operative PSA at 6th week (PSA6w) after radical prostatectomy to establish an optimal model for predicting natural biochemical recurrence (BCR).

Methods

A total of 742 patients with post-operative PSA6w from PC-follow database, between January 2003 and October 2022, were included. All the patients had not received any hormone therapy and radiotherapy before operation and BCR. Of these patients, 588 cases operated by one surgeon were enrolled for modelling and another 154 cases operated by other surgeons were for external validation. After screened by Cox regression, the post-operative PSA6w, pathological stage, Gleason Grade and positive surgical margins were adopted for modelling. The R software was used to plot the nomogram of the prediction model for BCR. C-index and calibration curve were calculated to evaluate the new model. Finally, integrated discrimination improvement was adopted to evaluate the prediction performances of the new nomogram model and the classical Kattan nomogram.

Results

The C-index of the new model was 0.871 (95% CI: 0.830-0.912). The calibration curve of the new model demonstrated superior consistency between the predicted and actual value. The C-index of the external validation group was 0.850 (95% CI: 0.742-0.958), which demonstrated perfect universality. The integrated discrimination improvement showed a 12.61% improvement in prediction performance over that of the classical Kattan nomogram (P < 0.01). Based on the new nomogram, patients were divided to high and low BCR group with a 3 year BCR-free cutoff probability as 74.72%. Low-risk patients, accounting for 77.89% of the patients, have no need to follow up frequently with a false-negative rate only 5.24%, which will save medical resources to a large extent.

Conclusion

Post-operative PSA6w is a sensitive risk biomarker for early natural BCR. The new nomogram model could predict BCR probability with a higher accuracy and will further simplify the clinical follow-up strategies.

The role of modeling studies in asthma management and clinical decision-making: a Delphi survey of physician knowledge and perceptions

OBJECTIVE

To investigate the knowledge and perceptions of physicians on the role of modeling studies in asthma, using a modified Delphi procedure.

METHODS

Group opinions among a panel of respiratory experts were obtained using two online questionnaires and a virtual scientific workshop. A consensus was pre-defined as agreement by >75% of participants.

RESULTS

From 26 experts who agreed to participate, 22 completed both surveys. At the end of the process, the panel rated their own understanding of modeling as good (77%) but that among physicians in general as poor (77%). Participants agreed that data from modeling studies should be used, at least sometimes, to inform treatment guidelines (91%) and could be useful for guiding clinical decisions (100%). Perceived barriers to using modeling studies were 'A lack of understanding' (81%) and 'A lack of standardized methodology' (82%). Based on data from two modeling studies, no consensus was reached on physicians recommending regular inhaled corticosteroids (ICS) versus as-needed therapy for patients with mild asthma, whereas 77% agreed that they would recommend regular ICS over maintenance and reliever therapy for ≥80% of their patients with moderate asthma. No consensus was reached on the value of modeling data in relation to empirical data.

CONCLUSION

There is overall support among respiratory experts for the usefulness of modeling data to guide asthma treatment guidelines and clinical decision making. More publications on modeling data using robust models and accessible terminology will aid the understanding of physicians in general and help clarify the evidence-based value of modeling studies.

Cumulative vulnerabilities and smoking abstinence: A test from a randomized clinical trial

Smoking cessation is often associated with socioeconomic and intrapersonal vulnerabilities such as psychopathology. Yet, most research that focuses on predicting smoking cessation outcomes tends focus on a small number of possible vulnerabilities. In a secondary data analysis, we developed and empirically evaluated a comprehensive, cumulative vulnerability risk composite reflecting psychologically based transdiagnostic processes, social determinants of health, and psychopathology. Participants were adult smokers who responded to study advertisements (e.g., flyers, newspaper ads, radio announcements) for an in-person delivered 4-session smoking cessation trial (N = 267; 47% female; Mage = 39.4, SD = 13.8). Results indicated that the decline in point prevalence abstinence (PPA) from quit week to 6-month post-quit was statistically significant (p < .001). There were statistically significant effects of cumulative risk score on the intercept (p < .001) and slope (p = .01). These findings were evident in unadjusted and adjusted (controlling for sex, treatment condition, and nicotine dependence) models. The present results indicate smokers with greater cumulative vulnerability demonstrated poorer smoking cessation outcomes. There may be clinical advantages to better understanding cumulative vulnerability among treatment-seeking smokers and other smoking populations to enhance the impact of public health efforts to reduce smoking.

Performance and usability of pre-operative prediction models for 30-day peri-operative mortality risk: a systematic review

Estimating pre-operative mortality risk may inform clinical decision-making for peri-operative care. However, pre-operative mortality risk prediction models are rarely implemented in routine clinical practice. High predictive accuracy and clinical usability are essential for acceptance and clinical implementation. In this systematic review, we identified and appraised prediction models for 30-day postoperative mortality in non-cardiac surgical cohorts. PubMed and Embase were searched up to December 2022 for studies investigating pre-operative prediction models for 30-day mortality. We assessed predictive performance in terms of discrimination and calibration. Risk of bias was evaluated using a tool to assess the risk of bias and applicability of prediction model studies. To further inform potential adoption, we also assessed clinical usability for selected models. In all, 15 studies evaluating 10 prediction models were included. Discrimination ranged from a c-statistic of 0.82 (MySurgeryRisk) to 0.96 (extreme gradient boosting machine learning model). Calibration was reported in only six studies. Model performance was highest for the surgical outcome risk tool (SORT) and its external validations. Clinical usability was highest for the surgical risk pre-operative assessment system. The SORT and risk quantification index also scored high on clinical usability. We found unclear or high risk of bias in the development of all models. The SORT showed the best combination of predictive performance and clinical usability and has been externally validated in several heterogeneous cohorts. To improve clinical uptake, full integration of reliable models with sufficient face validity within the electronic health record is imperative.

A multi-subgroup predictive model based on clinical parameters and laboratory biomarkers to predict in-hospital outcomes of plasma exchange-centered artificial liver treatment in patients with hepatitis B virus-related acute-on-chronic liver failure

Background

Postoperative risk stratification is challenging in patients with hepatitis B virus-related acute-on-chronic liver failure (HBV-ACLF) who undergo artificial liver treatment. This study characterizes patients' clinical parameters and laboratory biomarkers with different in-hospital outcomes. The purpose was to establish a multi-subgroup combined predictive model and analyze its predictive capability.

Methods

We enrolled HBV-ACLF patients who received plasma exchange (PE)-centered artificial liver support system (ALSS) therapy from May 6, 2017, to April 6, 2022. There were 110 patients who died (the death group) and 110 propensity score-matched patients who achieved satisfactory outcomes (the survivor group). We compared baseline, before ALSS, after ALSS, and change ratios of laboratory biomarkers. Outcome prediction models were established by generalized estimating equations (GEE). The discrimination was assessed using receiver operating characteristic analyses. Calibration plots compared the mean predicted probability and the mean observed outcome.

Results

We built a multi-subgroup predictive model (at admission; before ALSS; after ALSS; change ratio) to predict in-hospital outcomes of HBV-ACLF patients who received PE-centered ALSS. There were 110 patients with 363 ALSS sessions who survived and 110 who did not, and 363 ALSS sessions were analyzed. The univariate GEE models revealed that several parameters were independent risk factors. Clinical parameters and laboratory biomarkers were entered into the multivariate GEE model. The discriminative power of the multivariate GEE models was excellent, and calibration showed better agreement between the predicted and observed probabilities than the univariate models.

Conclusions

The multi-subgroup combined predictive model generated accurate prognostic information for patients undergoing HBV-ACLF patients who received PE-centered ALSS.

Endovascular versus neurosurgical aneurysm treatment: study protocol for the development and validation of a clinical prediction tool for individualised decision making

INTRODUCTION

Treatment decisions for aneurysmal subarachnoid haemorrhage patients should be supported by individualised predictions of the effects of aneurysm treatment. We present a study protocol and analysis plan for the development and external validation of models to predict benefit of neurosurgical versus endovascular aneurysm treatment on functional outcome and durability of treatment.

METHODS AND ANALYSIS

We will use data from the International Subarachnoid Aneurysm Trial for model development. The outcomes are functional outcome, measured with modified Rankin Scale at 12 months, and any retreatment or rebleed of the target aneurysm during follow-up. We will develop an ordinal logistic regression model and Cox regression model, considering age, World Federation of Neurological Surgeons grade, Fisher grade, vasospasm at presentation, aneurysm lumen size, aneurysm neck size, aneurysm location and time-to-aneurysm-treatment as predictors. We will test for interactions with treatment and with baseline risk and derive individualised predicted probabilities of treatment benefit. A benefit of ≥5% will be considered clinically relevant. Discriminative performance of the outcome predictions will be assessed with the c-statistic. Calibration will be assessed with calibration plots. Discriminative performance of the benefit predictions will be assessed with the c-for benefit. We will assess internal validity with bootstrapping and external validity with leave-one-out internal-external cross-validation.

ETHICS AND DISSEMINATION

The medical ethical research committee of the Erasmus MC University Medical Center Rotterdam approved the study protocol under the exemption category and waived the need for written informed consent (MEC-2020-0810). We will disseminate our results through an open-access peer-reviewed scientific publication and with a web-based clinical prediction tool.

Artificial intelligence in the pediatric echocardiography laboratory: Automation, physiology, and outcomes

Artificial intelligence (AI) is frequently used in non-medical fields to assist with automation and decision-making. The potential for AI in pediatric cardiology, especially in the echocardiography laboratory, is very high. There are multiple tasks AI is designed to do that could improve the quality, interpretation, and clinical application of echocardiographic data at the level of the sonographer, echocardiographer, and clinician. In this state-of-the-art review, we highlight the pertinent literature on machine learning in echocardiography and discuss its applications in the pediatric echocardiography lab with a focus on automation of the pediatric echocardiogram and the use of echo data to better understand physiology and outcomes in pediatric cardiology. We also discuss next steps in utilizing AI in pediatric echocardiography.

Prediction of Mortality in Coronary Artery Disease: Role of Machine Learning and Maximal Exercise Capacity

OBJECTIVE

To develop a prediction model for survival of patients with coronary artery disease (CAD) using health conditions beyond cardiovascular risk factors, including maximal exercise capacity, through the application of machine learning (ML) techniques.

METHODS

Analysis of data from a retrospective cohort linking clinical, administrative, and vital status databases from 1995 to 2016 was performed. Inclusion criteria were age 18 years or older, diagnosis of CAD, referral to a cardiac rehabilitation program, and available baseline exercise test results. Primary outcome was death from any cause. Feature selection was performed using supervised and unsupervised ML techniques. The final prognostic model used the survival tree (ST) algorithm.

RESULTS

From the cohort of 13,362 patients (60±11 years; 2400 [18%] women), 1577 died during a median follow-up of 8 years (interquartile range, 4 to 13 years), with an estimated survival of 67% up to 21 years. Feature selection revealed age and peak metabolic equivalents (METs) as the features with the greatest importance for mortality prediction. Using these 2 features, the ST generated a long-term prediction with a C-index of 0.729 by splitting patients in 8 clusters with different survival probabilities (P<.001). The ST root node was split by peak METs of 6.15 or less or more than 6.15, and each patient's subgroup was further split by age or other peak METs cut points.

CONCLUSION

Applying ML techniques, age and maximal exercise capacity accurately predict mortality in patients with CAD and outperform variables commonly used for decision-making in clinical practice. A novel and simple prognostic model was established, and maximal exercise capacity was further suggested to be one of the most powerful predictors of mortality in CAD.

Systematic review and meta-analysis of prognostic models in Southeast Asian populations with acute myocardial infarction

Background

The cultural and genetic diversity of the Southeast Asian population has contributed to distinct cardiovascular disease risks, incidence, and prognosis compared to the Western population, thereby raising concerns about the accuracy of predicted risks of existing prognostic models.

Objectives

We aimed to evaluate the predictive performances of validated, recalibrated, and developed prognostic risk prediction tools used in the Southeast Asian population with acute myocardial infarction (AMI) events for secondary events

Methods

We searched MEDLINE and Cochrane Central databases until March 2022. We included prospective and retrospective cohort studies that exclusively evaluated populations in the Southeast Asian region with a confirmed diagnosis of an AMI event and evaluated for risk of secondary events such as mortality, recurrent AMI, and heart failure admission. The CHARMS and PRISMA checklists and PROBAST for risk of bias assessment were used in this review.

Results

We included 7 studies with 11 external validations, 3 recalibrations, and 3 new models from 4 countries. Both short- and long-term outcomes were assessed. Overall, we observed that the external validation studies provided a good predictive accuracy of the models in the respective populations. The pooled estimate of the C-statistic in the Southeast Asian population for GRACE risk score is 0.83 (95%CI 0.72–0.90, n = 6 validations) and for the TIMI risk score is 0.80 (95%CI: 0.772–0.83, n = 5 validations). Recalibrated and new models demonstrated marginal improvements in discriminative values. However, the method of predictive accuracy measurement in most studies was insufficient thereby contributing to the mixed accuracy effect. The evidence synthesis was limited due to the relatively low quality and heterogeneity of the available studies.

Conclusion

Both TIMI and GRACE risk scores demonstrated good predictive accuracies in the population. However, with the limited strength of evidence, these results should be interpreted with caution. Future higher-quality studies spanning various parts of the Asian region will help to understand the prognostic utility of these models better.

Systematic review registration

https://www.crd.york.ac.uk/prospero/display_record.php?%20RecordID=228486.

Validation of the myocardial-ischaemic-injury-index machine learning algorithm to guide the diagnosis of myocardial infarction in a heterogenous population: a prespecified exploratory analysis

BACKGROUND

Diagnostic pathways for myocardial infarction rely on fixed troponin thresholds, which do not recognise that troponin varies by age, sex, and time within individuals. To overcome this limitation, we recently introduced a machine learning algorithm that predicts the likelihood of myocardial infarction. Our aim was to evaluate whether this algorithm performs well in routine clinical practice and predicts subsequent events.

METHODS

The myocardial-ischaemic-injury-index (MI3) algorithm was validated in a prespecified exploratory analysis using data from a multi-centre randomised trial done in Scotland, UK that included consecutive patients with suspected acute coronary syndrome undergoing serial high-sensitivity cardiac troponin I measurement. Patients with ST-segment elevation myocardial infarction were excluded. MI3 incorporates age, sex, and two troponin measurements to compute a value (0-100) reflecting an individual's likelihood of myocardial infarction during the index visit and estimates diagnostic performance metrics (including area under the receiver-operating-characteristic curve, and the sensitivity, specificity, negative predictive value, and positive predictive value) at the computed score. Model performance for an index diagnosis of myocardial infarction (type 1 or type 4b), and for subsequent myocardial infarction or cardiovascular death at 1 year was determined using the previously defined low-probability threshold (1·6) and high-probability MI3 threshold (49·7). The trial is registered with ClinicalTrials.gov, NCT01852123.

FINDINGS

In total, 20 761 patients (64 years [SD 16], 9597 [46%] women) enrolled between June 10, 2013, and March 3, 2016, were included from the High-STEACS trial cohort, of whom 3272 (15·8%) had myocardial infarction. MI3 had an area under the receiver-operating-characteristic curve of 0·949 (95% CI 0·946-0·952) identifying 12 983 (62·5%) patients as low-probability for myocardial infarction at the pre-specified threshold (MI3 score <1·6; sensitivity 99·3% [95% CI 99·0-99·6], negative predictive value 99·8% [99·8-99·9]), and 2961 (14·3%) as high-probability at the pre-specified threshold (MI3 score ≥49·7; specificity 95·0% [94·6-95·3], positive predictive value 70·4% [68·7-72·0]). At 1 year, subsequent myocardial infarction or cardiovascular death occurred more often in high-probability patients than low-probability patients (520 [17·6%] of 2961 vs 197 [1·5%] of 12 983], p<0·0001).

INTERPRETATION

In consecutive patients undergoing serial cardiac troponin measurement for suspected acute coronary syndrome, the MI3 algorithm accurately estimated the likelihood of myocardial infarction and predicted subsequent adverse cardiovascular events. By providing individual probabilities the MI3 algorithm could improve the diagnosis and assessment of risk in patients with suspected acute coronary syndrome.

FUNDING

Medical Research Council, British Heart Foundation, National Institute for Health Research, and NHSX.

Generalizability of Cardiovascular Disease Clinical Prediction Models: 158 Independent External Validations of 104 Unique Models

Background:

While clinical prediction models (CPMs) are used increasingly commonly to guide patient care, the performance and clinical utility of these CPMs in new patient cohorts is poorly understood.

Methods:

We performed 158 external validations of 104 unique CPMs across 3 domains of cardiovascular disease (primary prevention, acute coronary syndrome, and heart failure). Validations were performed in publicly available clinical trial cohorts and model performance was assessed using measures of discrimination, calibration, and net benefit. To explore potential reasons for poor model performance, CPM-clinical trial cohort pairs were stratified based on relatedness, a domain-specific set of characteristics to qualitatively grade the similarity of derivation and validation patient populations. We also examined the model-based C-statistic to assess whether changes in discrimination were because of differences in case-mix between the derivation and validation samples. The impact of model updating on model performance was also assessed.

Results:

Discrimination decreased significantly between model derivation (0.76 [interquartile range 0.73–0.78]) and validation (0.64 [interquartile range 0.60–0.67], P<0.001), but approximately half of this decrease was because of narrower case-mix in the validation samples. CPMs had better discrimination when tested in related compared with distantly related trial cohorts. Calibration slope was also significantly higher in related trial cohorts (0.77 [interquartile range, 0.59–0.90]) than distantly related cohorts (0.59 [interquartile range 0.43–0.73], P=0.001). When considering the full range of possible decision thresholds between half and twice the outcome incidence, 91% of models had a risk of harm (net benefit below default strategy) at some threshold; this risk could be reduced substantially via updating model intercept, calibration slope, or complete re-estimation.

Conclusions:

There are significant decreases in model performance when applying cardiovascular disease CPMs to new patient populations, resulting in substantial risk of harm. Model updating can mitigate these risks. Care should be taken when using CPMs to guide clinical decision-making.

Cardiovascular risk in patients with type 2 diabetes: A systematic review of prediction models

AIMS

To identify all cardiovascular disease risk prediction models developed in patients with type 2 diabetes or in the general population with diabetes as a covariate updating previous studies, describing model performance and analysing both their risk of bias and their applicability METHODS: A systematic search for predictive models of cardiovascular risk was performed in PubMed. The CHARMS and PROBAST guidelines for data extraction and for the assessment of risk of bias and applicability were followed. Google Scholar citations of the selected articles were reviewed to identify studies that conducted external validations.

RESULTS

The titles of 10,556 references were extracted to ultimately identify 19 studies with models developed in a population with diabetes and 46 studies in the general population. Within models developed in a population with diabetes, only six were classified as having a low risk of bias, 17 had a favourable assessment of applicability, 11 reported complete model information, and also 11 were externally validated.

CONCLUSIONS

There exists an overabundance of cardiovascular risk prediction models applicable to patients with diabetes, but many have a high risk of bias due to methodological shortcomings and independent validations are scarce. We recommend following the existing guidelines to facilitate their applicability.

Prognostic Value of mRNAsi/Corrected mRNAsi Calculated by the One-Class Logistic Regression Machine-Learning Algorithm in Glioblastoma Within Multiple Datasets

Glioblastoma (GBM) is the most common glial tumour and has extremely poor prognosis. GBM stem-like cells drive tumorigenesis and progression. However, a systematic assessment of stemness indices and their association with immunological properties in GBM is lacking. We collected 874 GBM samples from four GBM cohorts (TCGA, CGGA, GSE4412, and GSE13041) and calculated the mRNA expression-based stemness indices (mRNAsi) and corrected mRNAsi (c_mRNAsi, mRNAsi/tumour purity) with OCLR algorithm. Then, mRNAsi/c_mRNAsi were used to quantify the stemness traits that correlated significantly with prognosis. Additionally, confounding variables were identified. We used discrimination, calibration, and model improvement capability to evaluate the established models. Finally, the CIBERSORTx algorithm and ssGSEA were implemented for functional analysis. Patients with high mRNAsi/c_mRNAsi GBM showed better prognosis among the four GBM cohorts. After identifying the confounding variables, c_mRNAsi still maintained its prognostic value. Model evaluation showed that the c_mRNAsi-based model performed well. Patients with high c_mRNAsi exhibited significant immune suppression. Moreover, c_mRNAsi correlated negatively with infiltrating levels of immune-related cells. In addition, ssGSEA revealed that immune-related pathways were generally activated in patients with high c_mRNAsi. We comprehensively evaluated GBM stemness indices based on large cohorts and established a c_mRNAsi-based classifier for prognosis prediction.

A Primer on the Present State and Future Prospects for Machine Learning and Artificial Intelligence Applications in Cardiology

The artificial intelligence (AI) revolution is well underway, including in the medical field, and has dramatically transformed our lives. An understanding of the basics of AI applications, their development, and challenges to their clinical implementation is important for clinicians to fully appreciate the possibilities of AI. Such a foundation would ensure that clinicians have a good grasp and realistic expectations for AI in medicine and prevent discrepancies between the promised and real-world impact. When quantifying the track record for AI applications in cardiology, we found that a substantial number of AI systems are never deployed in clinical practice, although there certainly are many success stories. Successful implementations shared the following: they came from clinical areas where large amount of training data was available; were deployable into a single diagnostic modality; prediction models generally had high performance on external validation; and most were developed as part of collaborations with medical device manufacturers who had substantial experience with implementation of new technology. When looking into the current processes used for developing AI-based systems, we suggest that expanding the analytic framework to address potential deployment and implementation issues at project outset will improve the rate of successful implementation, and will be a necessary next step for AI to achieve its full potential in cardiovascular medicine.

GRADE concept paper 2: Concepts for judging certainty on the calibration of prognostic models in a body of validation studies

``In this paper, we highlight key concepts...'' is background.The sentence ``IN this paper, we highlight key concepts in evaluating the certainty of evidence regarding the calibration of prognostic models'' is methods. The rest is results and conclusion. Brognostic models combine several prognostic factors to provide an estimate of the likelihood (or risk) of future events in individual patients, conditional on their prognostic factor values. A fundamental part of evaluating prognostic models is undertaking studies to determine whether their predictive performance, such as calibration and discrimination, is reproduced across settings. Systematic reviews and meta-analyses of studies evaluating prognostic models' performance are a necessary step for selection of models for clinical practice and for testing the underlying assumption that their use will improve outcomes, including patient's reassurance and optimal future planning. In this paper, we highlight key concepts in evaluating the certainty of evidence regarding the calibration of prognostic models. Four concepts are key to evaluating the certainty of evidence on prognostic models' performance regarding calibration. The first concept is that the inference regarding calibration may take 1 of 2 forms: deciding whether 1 is rating certainty that a model's performance is satisfactory or, instead, unsatisfactory, in either case defining the threshold for satisfactory (or unsatisfactory) model performance. Second, inconsistency is the critical GRADE domain to deciding whether we are rating certainty in the model performance being satisfactory or unsatisfactory. Third, depending on whether 1 is rating certainty in satisfactory or unsatisfactory performance, different patterns of inconsistency of results across studies will inform ratings of certainty of evidence. Fourth, exploring the distribution of point estimates of observed to expected ratio across individual studies, and its determinants, will bear on the need for and direction of future research.

Integrative Analyses and Verification of the Expression and Prognostic Significance for RCN1 in Glioblastoma Multiforme

Glioblastoma multiform is a lethal primary brain tumor derived from astrocytic, with a poor prognosis in adults. Reticulocalbin-1 (RCN1) is a calcium-binding protein, dysregulation of which contributes to tumorigenesis and progression in various cancers. The present study aimed to identify the impact of RCN1 on the outcomes of patients with Glioblastoma multiforme (GBM). The study applied two public databases to require RNA sequencing data of Glioblastoma multiform samples with clinical data for the construction of a training set and a validation set, respectively. We used bioinformatic analyses to determine that RCN1 could be an independent factor for the overall survival of Glioblastoma multiform patients. In the training set, the study constructed a predictive prognostic model based on the combination of RCN1 with various clinical parameters for overall survival at 0.5-, 1.0-, and 1.5-years, as well as developed a nomogram, which was further validated by validation set. Pathways analyses indicated that RCN1 was involved in KEAS and MYC pathways and apoptosis. In vitro experiments indicated that RCN1 promoted cell invasion of Glioblastoma multiform cells. These results illustrated the prognostic role of RCN1 for overall survival in Glioblastoma multiform patients, indicated the promotion of RCN1 in cell invasion, and suggested the probability of RCN1 as a potential targeted molecule for treatment in Glioblastoma multiform.

Deep Learning Algorithm Predicts Angiographic Coronary Artery Disease in Stable Patients Using Only a Standard 12-Lead Electrocardiogram

BACKGROUND

Current electrocardiogram analysis algorithms cannot predict the presence of coronary artery disease (CAD), especially in stable patients. This study assessed the ability of an artificial intelligence algorithm (ECGio; HEARTio Inc, Pittsburgh, PA) to predict the presence, location, and severity of coronary artery lesions in an unselected stable patient population.

METHODS

A cohort of 1659 stable outpatients was randomly divided into training (86%) and validation (14%) subsets, maintaining population characteristics. ECGio was trained and validated using electrocardiograms paired with retrospectively collected angiograms. Coronary artery lesions were classified in 2 analyses. The primary classification was no to mild (< 30% diameter stenosis [DS]) vs moderate (30%-70% DS) vs severe (≥ 70% DS) CAD. The secondary classification was yes/no based on ≥ 50% DS in any vessel.

RESULTS

In the primary analysis, 22 patients had no angiographic CAD and were grouped mild CAD (56 patients, DS < 30%), 31 had moderate CAD (DS 30%-70%), and 113 had severe CAD (DS ≥ 70%). Weighted average sensitivity was 93.2%, and weighted average specificity was 96.4%. In the secondary analysis, 93 had significant CAD, and 128 did not. There was sensitivity of 93.1% and specificity of 85.6% in determining the presence of clinically significant disease (≥ 50% DS) in any vessel. ECGio was able to predict stenosis with average vessel error in the left anterior descending coronary artery of 18%, the left circumflex coronary artery of 19%, the right coronary artery of 18%, and the left main coronary artery of 8%.

CONCLUSIONS

This study strongly suggests that it is possible to use an artificial intelligence algorithm to determine the presence and severity of CAD in stable patients, using data from a 12-lead electrocardiogram.

Promoting Prognostic Model Application: A Review Based on Gliomas

Malignant neoplasms are characterized by poor therapeutic efficacy, high recurrence rate, and extensive metastasis, leading to short survival. Previous methods for grouping prognostic risks are based on anatomic, clinical, and pathological features that exhibit lower distinguishing capability compared with genetic signatures. The update of sequencing techniques and machine learning promotes the genetic panels-based prognostic model development, especially the RNA-panel models. Gliomas harbor the most malignant features and the poorest survival among all tumors. Currently, numerous glioma prognostic models have been reported. We systematically reviewed all 138 machine-learning-based genetic models and proposed novel criteria in assessing their quality. Besides, the biological and clinical significance of some highly overlapped glioma markers in these models were discussed. This study screened out markers with strong prognostic potential and 27 models presenting high quality. Conclusively, we comprehensively reviewed 138 prognostic models combined with glioma genetic panels and presented novel criteria for the development and assessment of clinically important prognostic models. This will guide the genetic models in cancers from laboratory-based research studies to clinical applications and improve glioma patient prognostic management.

Promoting Prognostic Model Application: A Review Based on Gliomas

Malignant neoplasms are characterized by poor therapeutic efficacy, high recurrence rate, and extensive metastasis, leading to short survival. Previous methods for grouping prognostic risks are based on anatomic, clinical, and pathological features that exhibit lower distinguishing capability compared with genetic signatures. The update of sequencing techniques and machine learning promotes the genetic panels-based prognostic model development, especially the RNA-panel models. Gliomas harbor the most malignant features and the poorest survival among all tumors. Currently, numerous glioma prognostic models have been reported. We systematically reviewed all 138 machine-learning-based genetic models and proposed novel criteria in assessing their quality. Besides, the biological and clinical significance of some highly overlapped glioma markers in these models were discussed. This study screened out markers with strong prognostic potential and 27 models presenting high quality. Conclusively, we comprehensively reviewed 138 prognostic models combined with glioma genetic panels and presented novel criteria for the development and assessment of clinically important prognostic models. This will guide the genetic models in cancers from laboratory-based research studies to clinical applications and improve glioma patient prognostic management.

Large-scale validation of the prediction model risk of bias assessment Tool (PROBAST) using a short form: high risk of bias models show poorer discrimination

OBJECTIVE

To assess whether the Prediction model Risk Of Bias ASsessment Tool (PROBAST) and a shorter version of this tool can identify clinical prediction models (CPMs) that perform poorly at external validation.

STUDY DESIGN AND SETTING

We evaluated risk of bias (ROB) on 102 CPMs from the Tufts CPM Registry, comparing PROBAST to a short form consisting of six PROBAST items anticipated to best identify high ROB. We then applied the short form to all CPMs in the Registry with at least 1 validation (n=556) and assessed the change in discrimination (dAUC) in external validation cohorts (n=1,147).

RESULTS

PROBAST classified 98/102 CPMS as high ROB. The short form identified 96 of these 98 as high ROB (98% sensitivity), with perfect specificity. In the full CPM registry, 527 of 556 CPMs (95%) were classified as high ROB, 20 (3.6%) low ROB, and 9 (1.6%) unclear ROB. Only one model with unclear ROB was reclassified to high ROB after full PROBAST assessment of all low and unclear ROB models. Median change in discrimination was significantly smaller in low ROB models (dAUC -0.9%, IQR -6.2-4.2%) compared to high ROB models (dAUC -11.7%, IQR -33.3-2.6%; P<0.001).

CONCLUSION

High ROB is pervasive among published CPMs. It is associated with poor discriminative performance at validation, supporting the application of PROBAST or a shorter version in CPM reviews.

Appraising prediction research: a guide and meta-review on bias and applicability assessment using the Prediction model Risk Of Bias ASsessment Tool (PROBAST)

Over the past few years, a large number of prediction models have been published, often of poor methodological quality. Seemingly objective and straightforward, prediction models provide a risk estimate for the outcome of interest, usually based on readily available clinical information. Yet, using models of substandard methodological rigour, especially without external validation, may result in incorrect risk estimates and consequently misclassification. To assess and combat bias in prediction research the prediction model risk of bias assessment tool (PROBAST) was published in 2019. This risk of bias (ROB) tool includes four domains and 20 signalling questions highlighting methodological flaws, and provides guidance in assessing the applicability of the model. In this paper, the PROBAST will be discussed, along with an in‐depth review of two commonly encountered pitfalls in prediction modelling that may induce bias: overfitting and composite endpoints. We illustrate the prevalence of potential bias in prediction models with a meta‐review of 50 systematic reviews that used the PROBAST to appraise their included studies, thus including 1510 different studies on 2104 prediction models. All domains showed an unclear or high ROB; these results were markedly stable over time, highlighting the urgent need for attention on bias in prediction research. This article aims to do just that by providing (1) the clinician with tools to evaluate the (methodological) quality of a clinical prediction model, (2) the researcher working on a review with methods to appraise the included models, and (3) the researcher developing a model with suggestions to improve model quality.

Most published prediction models have limited clinical uptake, are not externally validated and come with methodological issues. The PROBAST (Prediction model Risk Of Bias ASssessment Tool) guides the researcher writing a review, or the clinician interested in a model for risk calculation in a clinical setting. This review examines the aspects of bias in prediction research, and provides information on the prevalence of bias in published models.

Prediction models for clinical outcome after cochlear implantation: a systematic review

OBJECTIVES

Cochlear implants (CIs) are implantable hearing devices with a wide variation in clinical outcome between patients. We aim to provide an overview of the literature on prediction models and their performance for clinical outcome after cochlear implantation in bilateral hearing loss or deafness.

STUDY DESIGN AND SETTING

In this systematic review, studies describing the development or external validation of a multivariable model for predicting clinical CI outcome were eligible for selection.

RESULTS

A total of 4,042 references were screened. We included nine development studies and one external validation study. The outcome measure of all development studies was speech perception performance after cochlear implantation. The most commonly used model predictors were duration of hearing loss or deafness (n = 7), different types of preoperative measurements (n = 5), and etiology (n = 3). In three studies, crucial information to enable the model to be used for individual risk prediction was missing. One study performed internal validation,two models were externally validated. One study reported specific discrimination or calibration performance measures.

CONCLUSION

Although many articles describe development studies of prediction models for speech perception performance after cochlear implantation, the value of most of these models for their application in clinical practice remains unclear. Therefore, research should focus on increasing the clinical relevance of existing CI outcome prediction models.

The Year in Perioperative Echocardiography: Selected Highlights from 2020

This article is the fifth of an annual series reviewing the research highlights of the year pertaining to the subspecialty of perioperative echocardiography for the Journal of Cardiothoracic and Vascular Anesthesia. The authors thank Editor-in-Chief Dr. Kaplan and the editorial board for the opportunity to continue this series. In most cases, these will be research articles that are targeted at the perioperative echocardiography diagnosis and treatment of patients after cardiothoracic surgery; but in some cases, these articles will target the use of perioperative echocardiography in general.

A tutorial on calibration measurements and calibration models for clinical prediction models

Our primary objective is to provide the clinical informatics community with an introductory tutorial on calibration measurements and calibration models for predictive models using existing R packages and custom implemented code in R on real and simulated data. Clinical predictive model performance is commonly published based on discrimination measures, but use of models for individualized predictions requires adequate model calibration. This tutorial is intended for clinical researchers who want to evaluate predictive models in terms of their applicability to a particular population. It is also for informaticians and for software engineers who want to understand the role that calibration plays in the evaluation of a clinical predictive model, and to provide them with a solid starting point to consider incorporating calibration evaluation and calibration models in their work. Covered topics include (1) an introduction to the importance of calibration in the clinical setting, (2) an illustration of the distinct roles that discrimination and calibration play in the assessment of clinical predictive models, (3) a tutorial and demonstration of selected calibration measurements, (4) a tutorial and demonstration of selected calibration models, and (5) a brief discussion of limitations of these methods and practical suggestions on how to use them in practice.

Design, methods, and reporting of impact studies of cardiovascular clinical prediction rules are suboptimal: a systematic review

OBJECTIVES

To evaluate design, methods, and reporting of impact studies of cardiovascular clinical prediction rules (CPRs).

STUDY DESIGN AND SETTING

We conducted a systematic review. Impact studies of cardiovascular CPRs were identified by forward citation and electronic database searches. We categorized the design of impact studies as appropriate for randomized and nonrandomized experiments, excluding uncontrolled before-after study. For impact studies with appropriate study design, we assessed the quality of methods and reporting. We compared the quality of methods and reporting between impact and matched control studies.

RESULTS

We found 110 impact studies of cardiovascular CPRs. Of these, 65 (59.1%) used inappropriate designs. Of 45 impact studies with appropriate design, 31 (68.9%) had substantial risk of bias. Mean number of reporting domains that impact studies with appropriate study design adhered to was 10.2 of 21 domains (95% confidence interval, 9.3 and 11.1). The quality of methods and reporting was not clearly different between impact and matched control studies.

CONCLUSION

We found most impact studies either used inappropriate study design, had substantial risk of bias, or poorly complied with reporting guidelines. This appears to be a common feature of complex interventions. Users of CPRs should critically evaluate evidence showing the effectiveness of CPRs.

Computational medicine, present and the future: obstetrics and gynecology perspective

Medicine is, in its essence, decision making under uncertainty; the decisions are made about tests to be performed and treatments to be administered. Traditionally, the uncertainty in decision making was handled using expertise collected by individual providers and, more recently, systematic appraisal of research in the form of evidence-based medicine. The traditional approach has been used successfully in medicine for a very long time. However, it has substantial limitations because of the complexity of the system of the human body and healthcare. The complex systems are a network of highly coupled components intensely interacting with each other. These interactions give those systems redundancy and thus robustness to failure and, at the same time, equifinality, that is, many different causative pathways leading to the same outcome. The equifinality of the complex systems of the human body and healthcare system demand the individualization of medical care, medicine, and medical decision making. Computational models excel in modeling complex systems and, consequently, enabling individualization of medical decision making and medicine. Computational models are theory- or knowledge-based models, data-driven models, or models that combine both approaches. Data are essential, although to a different degree, for computational models to successfully represent complex systems. The individualized decision making, made possible by the computational modeling of complex systems, has the potential to revolutionize the entire spectrum of medicine from individual patient care to policymaking. This approach allows applying tests and treatments to individuals who receive a net benefit from them, for whom benefits outweigh the risk, rather than treating all individuals in a population because, on average, the population benefits. Thus, the computational modeling-enabled individualization of medical decision making has the potential to both improve health outcomes and decrease the costs of healthcare.

Early and Supernormal Vascular Aging: Clinical Characteristics and Association With Incident Cardiovascular Events

Pulse wave velocity is an established marker of early vascular aging but may also help identifying individuals with supernormal vascular aging. We tested the hypothesis that individuals with the largest difference (Δ-age) between chronological and vascular age show the lowest rate of cardiovascular events and may thus be defined as supernormal vascular aging. Vascular age was defined as the predicted age in the best fitting multivariable regression model including classical risk factors and treatment and pulse wave velocity, in a subset of the Reference Values for Arterial Stiffness Collaboration Database (n=3347). Δ-age was then calculated as chronological age minus vascular age, and the 10th and 90th percentiles were used to define early (Δ-age<-5.7 years), normal (Δ-age -5.7 to 6.8 years) and supernormal vascular aging (Δ-age>6.8 years). The risk for fatal and nonfatal cardiovascular events associated with vascular aging categories was investigated in the Malmö Diet and Cancer Study cohort (n=2642). In the Malmö Diet and Cancer Study Cohort (6.6-year follow-up, 286 events), Δ-age was significantly (P<0.01) and inversely associated with cardiovascular events. Compared with normal vascular aging, supernormal vascular aging had lower risk (hazard ratio, 0.59 [95% CI, 0.41-0.85]), whereas early vascular aging had higher risk (hazard ratio, 2.70 [95% CI, 1.55-4.70]) of cardiovascular events, in particular coronary events. There was no significant association with all-cause mortality. This study represents the first validation of the clinical significance of the supernormal vascular aging concept, based on prospective data. Its further characterization may help discovering novel protective molecular pathways and providing preventive strategies for successful vascular aging.

Clinical Predictive Models of Sudden Cardiac Arrest: A Survey of the Current Science and Analysis of Model Performances

Background

More than 500 000 sudden cardiac arrests (SCAs) occur annually in the United States. Clinical predictive models (CPMs) may be helpful tools to differentiate between patients who are likely to survive or have good neurologic recovery and those who are not. However, which CPMs are most reliable for discriminating between outcomes in SCA is not known.

Methods and Results

We performed a systematic review of the literature using the Tufts PACE (Predictive Analytics and Comparative Effectiveness) CPM Registry through February 1, 2020, and identified 81 unique CPMs of SCA and 62 subsequent external validation studies. Initial cardiac rhythm, age, and duration of cardiopulmonary resuscitation were the 3 most commonly used predictive variables. Only 33 of the 81 novel SCA CPMs (41%) were validated at least once. Of 81 novel SCA CPMs, 56 (69%) and 61 of 62 validation studies (98%) reported discrimination, with median c‐statistics of 0.84 and 0.81, respectively. Calibration was reported in only 29 of 62 validation studies (41.9%). For those novel models that both reported discrimination and were validated (26 models), the median percentage change in discrimination was −1.6%. We identified 3 CPMs that had undergone at least 3 external validation studies: the out‐of‐hospital cardiac arrest score (9 validations; median c‐statistic, 0.79), the cardiac arrest hospital prognosis score (6 validations; median c‐statistic, 0.83), and the good outcome following attempted resuscitation score (6 validations; median c‐statistic, 0.76).

Conclusions

Although only a small number of SCA CPMs have been rigorously validated, the ones that have been demonstrate good discrimination.

Logistic regression has similar performance to optimised machine learning algorithms in a clinical setting: application to the discrimination between type 1 and type 2 diabetes in young adults

BACKGROUND

There is much interest in the use of prognostic and diagnostic prediction models in all areas of clinical medicine. The use of machine learning to improve prognostic and diagnostic accuracy in this area has been increasing at the expense of classic statistical models. Previous studies have compared performance between these two approaches but their findings are inconsistent and many have limitations. We aimed to compare the discrimination and calibration of seven models built using logistic regression and optimised machine learning algorithms in a clinical setting, where the number of potential predictors is often limited, and externally validate the models.

METHODS

We trained models using logistic regression and six commonly used machine learning algorithms to predict if a patient diagnosed with diabetes has type 1 diabetes (versus type 2 diabetes). We used seven predictor variables (age, BMI, GADA islet-autoantibodies, sex, total cholesterol, HDL cholesterol and triglyceride) using a UK cohort of adult participants (aged 18-50 years) with clinically diagnosed diabetes recruited from primary and secondary care (n = 960, 14% with type 1 diabetes). Discrimination performance (ROC AUC), calibration and decision curve analysis of each approach was compared in a separate external validation dataset (n = 504, 21% with type 1 diabetes).

RESULTS

Average performance obtained in internal validation was similar in all models (ROC AUC ≥ 0.94). In external validation, there were very modest reductions in discrimination with AUC ROC remaining ≥ 0.93 for all methods. Logistic regression had the numerically highest value in external validation (ROC AUC 0.95). Logistic regression had good performance in terms of calibration and decision curve analysis. Neural network and gradient boosting machine had the best calibration performance. Both logistic regression and support vector machine had good decision curve analysis for clinical useful threshold probabilities.

CONCLUSION

Logistic regression performed as well as optimised machine algorithms to classify patients with type 1 and type 2 diabetes. This study highlights the utility of comparing traditional regression modelling to machine learning, particularly when using a small number of well understood, strong predictor variables.

Predicting treatment effects in unipolar depression: A meta-review

There is increasing interest in clinical prediction models in psychiatry, which focus on developing multivariate algorithms to guide personalized diagnostic or management decisions. The main target of these models is the prediction of treatment response to different antidepressant therapies. This is because the ability to predict response based on patients' personal data may allow clinicians to make improved treatment decisions, and to provide more efficacious or more tolerable medications to the right patient. We searched the literature for systematic reviews about treatment prediction in the context of existing treatment modalities for adult unipolar depression, until July 2019. Treatment effect is defined broadly to include efficacy, safety, tolerability and acceptability outcomes. We first focused on the identification of individual predictor variables that might predict treatment response, and second, we considered multivariate clinical prediction models. Our meta-review included a total of 10 systematic reviews; seven (from 2014 to 2018) focusing on individual predictor variables and three focusing on clinical prediction models. These identified a number of sociodemographic, phenomenological, clinical, neuroimaging, remote monitoring, genetic and serum marker variables as possible predictor variables for treatment response, alongside statistical and machine-learning approaches to clinical prediction model development. Effect sizes for individual predictor variables were generally small and clinical prediction models had generally not been validated in external populations. There is a need for rigorous model validation in large external data-sets to prove the clinical utility of models. We also discuss potential future avenues in the field of personalized psychiatry, particularly the combination of multiple sources of data and the emerging field of artificial intelligence and digital mental health to identify new individual predictor variables.

Evaluating risk prediction models for adults with heart failure: A systematic literature review

Background

The ability to predict risk allows healthcare providers to propose which patients might benefit most from certain therapies, and is relevant to payers’ demands to justify clinical and economic value. To understand the robustness of risk prediction models for heart failure (HF), we conducted a systematic literature review to (1) identify HF risk-prediction models, (2) assess statistical approach and extent of validation, (3) identify common variables, and (4) assess risk of bias (ROB).

Methods

Literature databases were searched from March 2013 to May 2018 to identify risk prediction models conducted in an out-of-hospital setting in adults with HF. Distinct risk prediction variables were ranked according to outcomes assessed and incorporation into the studies. ROB was assessed using Prediction model Risk Of Bias ASsessment Tool (PROBAST).

Results

Of 4720 non-duplicated citations, 40 risk-prediction publications were deemed relevant. Within the 40 publications, 58 models assessed 55 (co)primary outcomes, including all-cause mortality (n = 17), cardiovascular death (n = 9), HF hospitalizations (n = 15), and composite endpoints (n = 14). Few publications reported detail on handling missing data (n = 11; 28%). The discriminatory ability for predicting all-cause mortality, cardiovascular death, and composite endpoints was generally better than for HF hospitalization. 105 distinct predictor variables were identified. Predictors included in >5 publications were: N-terminal prohormone brain-natriuretic peptide, creatinine, blood urea nitrogen, systolic blood pressure, sodium, NYHA class, left ventricular ejection fraction, heart rate, and characteristics including male sex, diabetes, age, and BMI. Only 11/58 (19%) models had overall low ROB, based on our application of PROBAST. In total, 26/58 (45%) models discussed internal validation, and 14/58 (24%) external validation.

Conclusions

The majority of the 58 identified risk-prediction models for HF present particular concerns according to ROB assessment, mainly due to lack of validation and calibration. The potential utility of novel approaches such as machine learning tools is yet to be determined.

Registration number

The SLR was registered in Prospero (ID: CRD42018100709).

The Predictive Approaches to Treatment effect Heterogeneity (PATH) Statement: Explanation and Elaboration

The PATH (Predictive Approaches to Treatment effect Heterogeneity) Statement was developed to promote the conduct of, and provide guidance for, predictive analyses of heterogeneity of treatment effects (HTE) in clinical trials. The goal of predictive HTE analysis is to provide patient-centered estimates of outcome risk with versus without the intervention, taking into account all relevant patient attributes simultaneously, to support more personalized clinical decision making than can be made on the basis of only an overall average treatment effect. The authors distinguished 2 categories of predictive HTE approaches (a "risk-modeling" and an "effect-modeling" approach) and developed 4 sets of guidance statements: criteria to determine when risk-modeling approaches are likely to identify clinically meaningful HTE, methodological aspects of risk-modeling methods, considerations for translation to clinical practice, and considerations and caveats in the use of effect-modeling approaches. They discuss limitations of these methods and enumerate research priorities for advancing methods designed to generate more personalized evidence. This explanation and elaboration document describes the intent and rationale of each recommendation and discusses related analytic considerations, caveats, and reservations.

Evidence-Based Utilization of Prognostic Prediction Models in Cardiovascular Medicine

Prediction models are combinations of predictors to assess the risks of specific endpoints such as the presence or prognosis of a disease. Many novel predictors have been developed, modelling techniques have been evolving, and prediction models are currently abundant in the medical literature, especially in cardiovascular medicine, but evidence is still lacking regarding how to use them. Recent methodological advances in systematic reviews and meta-analysis have enabled systematic evaluation of prediction model studies and quantitative analysis to identify determinants of model performance. Knowing what is critical to model performance, under what circumstances model performance remains adequate, and when a model might require further adjustment and improvement will facilitate effective utilization of prediction models and will enhance diagnostic and prognostic accuracy in clinical practice. In this review article, we provide a current methodological overview of the attempts to implement evidence-based utilization of prognostic prediction models for all potential model users, including patients and their families, health-care providers, administrators, researchers, guideline developers and policy makers.

Clinical Prediction Models for Valvular Heart Disease

Background

While many clinical prediction models (CPMs) exist to guide valvular heart disease treatment decisions, the relative performance of these CPMs is largely unknown. We systematically describe the CPMs available for patients with valvular heart disease with specific attention to performance in external validations.

Methods and Results

A systematic review identified 49 CPMs for patients with valvular heart disease treated with surgery (n=34), percutaneous interventions (n=12), or no intervention (n=3). There were 204 external validations of these CPMs. Only 35 (71%) CPMs have been externally validated. Sixty‐five percent (n=133) of the external validations were performed on distantly related populations. There was substantial heterogeneity in model performance and a median percentage change in discrimination of −27.1% (interquartile range, −49.4%–−5.7%). Nearly two‐thirds of validations (n=129) demonstrate at least a 10% relative decline in discrimination. Discriminatory performance of EuroSCORE II and Society of Thoracic Surgeons (2009) models (accounting for 73% of external validations) varied widely: EuroSCORE II validation c‐statistic range 0.50 to 0.95; Society of Thoracic Surgeons (2009) Models validation c‐statistic range 0.50 to 0.86. These models performed well when tested on related populations (median related validation c‐statistics: EuroSCORE II, 0.82 [0.76, 0.85]; Society of Thoracic Surgeons [2009], 0.72 [0.67, 0.79]). There remain few (n=9) external validations of transcatheter aortic valve replacement CPMs.

Conclusions

Many CPMs for patients with valvular heart disease have never been externally validated and isolated external validations appear insufficient to assess the trustworthiness of predictions. For surgical valve interventions, there are existing predictive models that perform reasonably well on related populations. For transcatheter aortic valve replacement (CPMs additional external validations are needed to broadly understand the trustworthiness of predictions.

Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature

Clinical prediction rules (CPRs) that predict the absolute risk of a clinical condition or future outcome for individual patients are abundant in the medical literature; however, systematic reviews have demonstrated shortcomings in the methodological quality and reporting of prediction studies. To maximise the potential and clinical usefulness of CPRs, they must be rigorously developed and validated, and their impact on clinical practice and patient outcomes must be evaluated. This review aims to present a comprehensive overview of the stages involved in the development, validation and evaluation of CPRs, and to describe in detail the methodological standards required at each stage, illustrated with examples where appropriate. Important features of the study design, statistical analysis, modelling strategy, data collection, performance assessment, CPR presentation and reporting are discussed, in addition to other, often overlooked aspects such as the acceptability, cost-effectiveness and longer-term implementation of CPRs, and their comparison with clinical judgement. Although the development and evaluation of a robust, clinically useful CPR is anything but straightforward, adherence to the plethora of methodological standards, recommendations and frameworks at each stage will assist in the development of a rigorous CPR that has the potential to contribute usefully to clinical practice and decision-making and have a positive impact on patient care.

Untapped potential of multicenter studies: a review of cardiovascular risk prediction models revealed inappropriate analyses and wide variation in reporting

BACKGROUND

Clinical prediction models are often constructed using multicenter databases. Such a data structure poses additional challenges for statistical analysis (clustered data) but offers opportunities for model generalizability to a broad range of centers. The purpose of this study was to describe properties, analysis, and reporting of multicenter studies in the Tufts PACE Clinical Prediction Model Registry and to illustrate consequences of common design and analyses choices.

METHODS

Fifty randomly selected studies that are included in the Tufts registry as multicenter and published after 2000 underwent full-text screening. Simulated examples illustrate some key concepts relevant to multicenter prediction research.

RESULTS

Multicenter studies differed widely in the number of participating centers (range 2 to 5473). Thirty-nine of 50 studies ignored the multicenter nature of data in the statistical analysis. In the others, clustering was resolved by developing the model on only one center, using mixed effects or stratified regression, or by using center-level characteristics as predictors. Twenty-three of 50 studies did not describe the clinical settings or type of centers from which data was obtained. Four of 50 studies discussed neither generalizability nor external validity of the developed model.

CONCLUSIONS

Regression methods and validation strategies tailored to multicenter studies are underutilized. Reporting on generalizability and potential external validity of the model lacks transparency. Hence, multicenter prediction research has untapped potential.

REGISTRATION

This review was not registered.