Reviewing the use and quality of machine learning in developing clinical prediction models for cardiovascular disease

Development of a convenient and effective hypertension risk prediction model and exploration of the relationship between Serum Ferritin and Hypertension Risk: a study based on NHANES 2017-March 2020

Background

Hypertension is a major public health problem, and its resulting other cardiovascular diseases are the leading cause of death worldwide. In this study, we constructed a convenient and high-performance hypertension risk prediction model to assist in clinical diagnosis and explore other important influencing factors.

Methods

We included 8,073 people from NHANES (2017-March 2020), using their 120 features to form the original dataset. After data pre-processing, we removed several redundant features through LASSO regression and correlation analysis. Thirteen commonly used machine learning methods were used to construct prediction models, and then, the methods with better performance were coupled with recursive feature elimination to determine the optimal feature subset. After data balancing through SMOTE, we integrated these better-performing learners to construct a fusion model based for predicting hypertension risk on stacking strategy. In addition, to explore the relationship between serum ferritin and the risk of hypertension, we performed a univariate analysis and divided it into four level groups (Q1 to Q4) by quartiles, with the lowest level group (Q1) as the reference, and performed multiple logistic regression analysis and trend analysis.

Results

The optimal feature subsets were: age, BMI, waist, SBP, DBP, Cre, UACR, serum ferritin, HbA1C, and doctors recommend reducing salt intake. Compared to other machine learning models, the constructed fusion model showed better predictive performance with precision, accuracy, recall, F1 value and AUC of 0.871, 0.873, 0.871, 0.869 and 0.966, respectively. For the analysis of the relationship between serum ferritin and hypertension, after controlling for all co-variates, OR and 95% CI from Q2 to Q4, compared to Q1, were 1.396 (1.176-1.658), 1.499 (1.254-1.791), and 1.645 (1.360-1.989), respectively, with P < 0.01 and P for trend <0.001.

Conclusion

The hypertension risk prediction model developed in this study is efficient in predicting hypertension with only 10 low-cost and easily accessible features, which is cost-effective in assisting clinical diagnosis. We also found a trend correlation between serum ferritin levels and the risk of hypertension.

An innovative model for predicting coronary heart disease using triglyceride-glucose index: a machine learning-based cohort study

BACKGROUND

Various predictive models have been developed for predicting the incidence of coronary heart disease (CHD), but none of them has had optimal predictive value. Although these models consider diabetes as an important CHD risk factor, they do not consider insulin resistance or triglyceride (TG). The unsatisfactory performance of these prediction models may be attributed to the ignoring of these factors despite their proven effects on CHD. We decided to modify standard CHD predictive models through machine learning to determine whether the triglyceride-glucose index (TyG-index, a logarithmized combination of fasting blood sugar (FBS) and TG that demonstrates insulin resistance) functions better than diabetes as a CHD predictor.

METHODS

Two-thousand participants of a community-based Iranian population, aged 20-74 years, were investigated with a mean follow-up of 9.9 years (range: 7.6-12.2). The association between the TyG-index and CHD was investigated using multivariate Cox proportional hazard models. By selecting common components of previously validated CHD risk scores, we developed machine learning models for predicting CHD. The TyG-index was substituted for diabetes in CHD prediction models. All components of machine learning models were explained in terms of how they affect CHD prediction. CHD-predicting TyG-index cut-off points were calculated.

RESULTS

The incidence of CHD was 14.5%. Compared to the lowest quartile of the TyG-index, the fourth quartile had a fully adjusted hazard ratio of 2.32 (confidence interval [CI] 1.16-4.68, p-trend 0.04). A TyG-index > 8.42 had the highest negative predictive value for CHD. The TyG-index-based support vector machine (SVM) performed significantly better than diabetes-based SVM for predicting CHD. The TyG-index was not only more important than diabetes in predicting CHD; it was the most important factor after age in machine learning models.

CONCLUSION

We recommend using the TyG-index in clinical practice and predictive models to identify individuals at risk of developing CHD and to aid in its prevention.

Cardiovascular disease incidence prediction by machine learning and statistical techniques: a 16-year cohort study from eastern Mediterranean region

BACKGROUND

Cardiovascular diseases (CVD) are the predominant cause of early death worldwide. Identification of people with a high risk of being affected by CVD is consequential in CVD prevention. This study adopts Machine Learning (ML) and statistical techniques to develop classification models for predicting the future occurrence of CVD events in a large sample of Iranians.

METHODS

We used multiple prediction models and ML techniques with different abilities to analyze the large dataset of 5432 healthy people at the beginning of entrance into the Isfahan Cohort Study (ICS) (1990-2017). Bayesian additive regression trees enhanced with "missingness incorporated in attributes" (BARTm) was run on the dataset with 515 variables (336 variables without and the remaining with up to 90% missing values). In the other used classification algorithms, variables with more than 10% missing values were excluded, and MissForest imputes the missing values of the remaining 49 variables. We used Recursive Feature Elimination (RFE) to select the most contributing variables. Random oversampling technique, recommended cut-point by precision-recall curve, and relevant evaluation metrics were used for handling unbalancing in the binary response variable.

RESULTS

This study revealed that age, systolic blood pressure, fasting blood sugar, two-hour postprandial glucose, diabetes mellitus, history of heart disease, history of high blood pressure, and history of diabetes are the most contributing factors for predicting CVD incidence in the future. The main differences between the results of classification algorithms are due to the trade-off between sensitivity and specificity. Quadratic Discriminant Analysis (QDA) algorithm presents the highest accuracy (75.50 ± 0.08) but the minimum sensitivity (49.84 ± 0.25); In contrast, decision trees provide the lowest accuracy (51.95 ± 0.69) but the top sensitivity (82.52 ± 1.22). BARTm.90% resulted in 69.48 ± 0.28 accuracy and 54.00 ± 1.66 sensitivity without any preprocessing step.

CONCLUSIONS

This study confirmed that building a prediction model for CVD in each region is valuable for screening and primary prevention strategies in that specific region. Also, results showed that using conventional statistical models alongside ML algorithms makes it possible to take advantage of both techniques. Generally, QDA can accurately predict the future occurrence of CVD events with a fast (inference speed) and stable (confidence values) procedure. The combined ML and statistical algorithm of BARTm provide a flexible approach without any need for technical knowledge about assumptions and preprocessing steps of the prediction procedure.

The role of modeling studies in asthma management and clinical decision-making: a Delphi survey of physician knowledge and perceptions

OBJECTIVE

To investigate the knowledge and perceptions of physicians on the role of modeling studies in asthma, using a modified Delphi procedure.

METHODS

Group opinions among a panel of respiratory experts were obtained using two online questionnaires and a virtual scientific workshop. A consensus was pre-defined as agreement by >75% of participants.

RESULTS

From 26 experts who agreed to participate, 22 completed both surveys. At the end of the process, the panel rated their own understanding of modeling as good (77%) but that among physicians in general as poor (77%). Participants agreed that data from modeling studies should be used, at least sometimes, to inform treatment guidelines (91%) and could be useful for guiding clinical decisions (100%). Perceived barriers to using modeling studies were 'A lack of understanding' (81%) and 'A lack of standardized methodology' (82%). Based on data from two modeling studies, no consensus was reached on physicians recommending regular inhaled corticosteroids (ICS) versus as-needed therapy for patients with mild asthma, whereas 77% agreed that they would recommend regular ICS over maintenance and reliever therapy for ≥80% of their patients with moderate asthma. No consensus was reached on the value of modeling data in relation to empirical data.

CONCLUSION

There is overall support among respiratory experts for the usefulness of modeling data to guide asthma treatment guidelines and clinical decision making. More publications on modeling data using robust models and accessible terminology will aid the understanding of physicians in general and help clarify the evidence-based value of modeling studies.

Development and Validation of Esophageal Squamous Cell Carcinoma Risk Prediction Models Based on an Endoscopic Screening Program

IMPORTANCE

Assessment tools are lacking for screening of esophageal squamous cell cancer (ESCC) in China, especially for the follow-up stage. Risk prediction to optimize the screening procedure is urgently needed.

OBJECTIVE

To develop and validate ESCC prediction models for identifying people at high risk for follow-up decision-making.

DESIGN, SETTING, AND PARTICIPANTS

This open, prospective multicenter diagnostic study has been performed since September 1, 2006, in Shandong Province, China. This study used baseline and follow-up data until December 31, 2021. The data were analyzed between April 6 and May 31, 2022. Eligibility criteria consisted of rural residents aged 40 to 69 years who had no contraindications for endoscopy. Among 161 212 eligible participants, those diagnosed with cancer or who had cancer at baseline, did not complete the questionnaire, were younger than 40 years or older than 69 years, or were detected with severe dysplasia or worse lesions were eliminated from the analysis.

EXPOSURES

Risk factors obtained by questionnaire and endoscopy.

MAIN OUTCOMES AND MEASURES

Pathological diagnosis of ESCC and confirmation by cancer registry data.

RESULTS

In this diagnostic study of 104 129 participants (56.39% women; mean [SD] age, 54.31 [7.64] years), 59 481 (mean [SD] age, 53.83 [7.64] years; 58.55% women) formed the derivation set while 44 648 (mean [SD] age, 54.95 [7.60] years; 53.51% women) formed the validation set. A total of 252 new cases of ESCC were diagnosed during 424 903.50 person-years of follow-up in the derivation cohort and 61 new cases from 177 094.10 person-years follow-up in the validation cohort. Model A included the covariates age, sex, and number of lesions; model B included age, sex, smoking status, alcohol use status, body mass index, annual household income, history of gastrointestinal tract diseases, consumption of pickled food, number of lesions, distinct lesions, and mild or moderate dysplasia. The Harrell C statistic of model A was 0.80 (95% CI, 0.77-0.83) in the derivation set and 0.90 (95% CI, 0.87-0.93) in the validation set; the Harrell C statistic of model B was 0.83 (95% CI, 0.81-0.86) and 0.91 (95% CI, 0.88-0.95), respectively. The models also had good calibration performance and clinical usefulness.

CONCLUSIONS AND RELEVANCE

The findings of this diagnostic study suggest that the models developed are suitable for selecting high-risk populations for follow-up decision-making and optimizing the cancer screening process.

Cardiovascular diseases prediction by machine learning incorporation with deep learning

It is yet unknown what causes cardiovascular disease (CVD), but we do know that it is associated with a high risk of death, as well as severe morbidity and disability. There is an urgent need for AI-based technologies that are able to promptly and reliably predict the future outcomes of individuals who have cardiovascular disease. The Internet of Things (IoT) is serving as a driving force behind the development of CVD prediction. In order to analyse and make predictions based on the data that IoT devices receive, machine learning (ML) is used. Traditional machine learning algorithms are unable to take differences in the data into account and have a low level of accuracy in their model predictions. This research presents a collection of machine learning models that can be used to address this problem. These models take into account the data observation mechanisms and training procedures of a number of different algorithms. In order to verify the efficacy of our strategy, we combined the Heart Dataset with other classification models. The proposed method provides nearly 96 percent of accuracy result than other existing methods and the complete analysis over several metrics has been analysed and provided. Research in the field of deep learning will benefit from additional data from a large number of medical institutions, which may be used for the development of artificial neural network structures.

A scoping methodological review of simulation studies comparing statistical and machine learning approaches to risk prediction for time-to-event data

BACKGROUND

There is substantial interest in the adaptation and application of so-called machine learning approaches to prognostic modelling of censored time-to-event data. These methods must be compared and evaluated against existing methods in a variety of scenarios to determine their predictive performance. A scoping review of how machine learning methods have been compared to traditional survival models is important to identify the comparisons that have been made and issues where they are lacking, biased towards one approach or misleading.

METHODS

We conducted a scoping review of research articles published between 1 January 2000 and 2 December 2020 using PubMed. Eligible articles were those that used simulation studies to compare statistical and machine learning methods for risk prediction with a time-to-event outcome in a medical/healthcare setting. We focus on data-generating mechanisms (DGMs), the methods that have been compared, the estimands of the simulation studies, and the performance measures used to evaluate them.

RESULTS

A total of ten articles were identified as eligible for the review. Six of the articles evaluated a method that was developed by the authors, four of which were machine learning methods, and the results almost always stated that this developed method's performance was equivalent to or better than the other methods compared. Comparisons were often biased towards the novel approach, with the majority only comparing against a basic Cox proportional hazards model, and in scenarios where it is clear it would not perform well. In many of the articles reviewed, key information was unclear, such as the number of simulation repetitions and how performance measures were calculated.

CONCLUSION

It is vital that method comparisons are unbiased and comprehensive, and this should be the goal even if realising it is difficult. Fully assessing how newly developed methods perform and how they compare to a variety of traditional statistical methods for prognostic modelling is imperative as these methods are already being applied in clinical contexts. Evaluations of the performance and usefulness of recently developed methods for risk prediction should be continued and reporting standards improved as these methods become increasingly popular.