Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting

Risk Prediction Models for Melanoma: A Systematic Review on the Heterogeneity in Model Development and Validation

The rising incidence of cutaneous melanoma over the past few decades has prompted substantial efforts to develop risk prediction models identifying people at high risk of developing melanoma to facilitate targeted screening programs. We review these models, regarding study characteristics, differences in risk factor selection and assessment, evaluation, and validation methods. Our systematic literature search revealed 40 studies comprising 46 different risk prediction models eligible for the review. Altogether, 35 different risk factors were part of the models with nevi being the most common one (n = 35, 78%); little consistency in other risk factors was observed. Results of an internal validation were reported for less than half of the studies (n = 18, 45%), and only 6 performed external validation. In terms of model performance, 29 studies assessed the discriminative ability of their models; other performance measures, e.g., regarding calibration or clinical usefulness, were rarely reported. Due to the substantial heterogeneity in risk factor selection and assessment as well as methodologic aspects of model development, direct comparisons between models are hardly possible. Uniform methodologic standards for the development and validation of risk prediction models for melanoma and reporting standards for the accompanying publications are necessary and need to be obligatory for that reason.

Prediction of type 2 diabetes risk in people with non-diabetic hyperglycaemia: model derivation and validation using UK primary care data

OBJECTIVE

Using primary care data, develop and validate sex-specific prognostic models that estimate the 10-year risk of people with non-diabetic hyperglycaemia developing type 2 diabetes.

DESIGN

Retrospective cohort study.

SETTING

Primary care.

PARTICIPANTS

154 705 adult patients with non-diabetic hyperglycaemia.

PRIMARY OUTCOME

Development of type 2 diabetes.

METHODS

This study used data routinely collected in UK primary care from general practices contributing to the Clinical Practice Research Datalink. Patients were split into development (n=109 077) and validation datasets (n=45 628). Potential predictor variables, including demographic and lifestyle factors, medical and family history, prescribed medications and clinical measures, were included in survival models following the imputation of missing data. Measures of calibration at 10 years and discrimination were determined using the validation dataset.

RESULTS

In the development dataset, 9332 patients developed type 2 diabetes during 293 238 person-years of follow-up (31.8 (95% CI 31.2 to 32.5) per 1000 person-years). In the validation dataset, 3783 patients developed type 2 diabetes during 115 113 person-years of follow-up (32.9 (95% CI 31.8 to 33.9) per 1000 person-years). The final prognostic models comprised 14 and 16 predictor variables for males and females, respectively. Both models had good calibration and high levels of discrimination. The performance statistics for the male model were: Harrell's C statistic of 0.700 in the development and 0.701 in the validation dataset, with a calibration slope of 0.974 (95% CI 0.905 to 1.042) in the validation dataset. For the female model, Harrell's C statistics were 0.720 and 0.718, respectively, while the calibration slope was 0.994 (95% CI 0.931 to 1.057) in the validation dataset.

CONCLUSION

These models could be used in primary care to identify those with non-diabetic hyperglycaemia most at risk of developing type 2 diabetes for targeted referral to the National Health Service Diabetes Prevention Programme.

A general presentation on how to carry out a CHARMS analysis for prognostic multivariate models

The CHARMS (critical appraisal and data extraction for systematic reviews of prediction modelling studies) checklist was created to provide methodological appraisals of predictive models, based on the best available scientific evidence and through systematic reviews. Our purpose is to give a general presentation on how to carry out a CHARMS analysis for prognostic multivariate models, making clear what the steps are and how they are applied individually to the studies included in the systematic review. This tutorial is aimed at providing such a resource. In addition to this explanation, we will apply the method to a real case: predictive models of atrial fibrillation in the community. This methodology could be applied to other predictive models using the steps provided in our review so as to have complete information for each included model and determine whether it can be implemented in daily clinical practice.

Prognostic models for chronic kidney disease: a systematic review and external validation

BACKGROUND

Accurate risk prediction is needed in order to provide personalized healthcare for chronic kidney disease (CKD) patients. An overload of prognosis studies is being published, ranging from individual biomarker studies to full prediction studies. We aim to systematically appraise published prognosis studies investigating multiple biomarkers and their role in risk predictions. Our primary objective was to investigate if the prognostic models that are reported in the literature were of sufficient quality and to externally validate them.

METHODS

We undertook a systematic review and appraised the quality of studies reporting multivariable prognosis models for end-stage renal disease (ESRD), cardiovascular (CV) events and mortality in CKD patients. We subsequently externally validated these models in a randomized trial that included patients from a broad CKD population.

RESULTS

We identified 91 papers describing 36 multivariable models for prognosis of ESRD, 50 for CV events, 46 for mortality and 17 for a composite outcome. Most studies were deemed of moderate quality. Moreover, they often adopted different definitions for the primary outcome and rarely reported full model equations (21% of the included studies). External validation was performed in the Multifactorial Approach and Superior Treatment Efficacy in Renal Patients with the Aid of Nurse Practitioners trial (n = 788, with 160 events for ESRD, 79 for CV and 102 for mortality). The 24 models that reported full model equations showed a great variability in their performance, although calibration remained fairly adequate for most models, except when predicting mortality (calibration slope >1.5).

CONCLUSIONS

This review shows that there is an abundance of multivariable prognosis models for the CKD population. Most studies were considered of moderate quality, and they were reported and analysed in such a manner that their results cannot directly be used in follow-up research or in clinical practice.

Risk and management of pre-diabetes

Type 2 diabetes mellitus (T2DM) is associated with a two- to four-fold increased risk of developing cardiovascular disease (CVD) and microvascular complications, which may already be present before diagnosis. It is, therefore, important to detect people with an increased risk of T2DM at an early stage. In order to identify individuals with so-called 'pre-diabetes', comprising impaired fasting glucose (IFG) and impaired glucose tolerance (IGT), current guidelines have developed definitions based on fasting plasma glucose, two-hour glucose concentrations and haemoglobin A1c. Subjects with pre-diabetes are at an increased risk of developing T2DM and CVD. This elevated risk seems similar according to the different criteria used to define pre-diabetes. The risk of progression to T2DM or CVD does, however, depend on other risk factors such as sex, body mass index and ethnicity. Based on the risk factors to develop T2DM, many risk assessment models have been developed to identify those at highest risk. These models perform well to identify those at risk and could be used to initiate preventive interventions. Many studies have shown that lifestyle modification and metformin are effective in preventing the development of T2DM, although lifestyle modification seems to have a more sustainable effect. In addition, lifestyle modification seems more effective in those with IGT than those with IFG. In this review, we will describe the different definitions used to define pre-diabetes, progression from pre-diabetes to T2DM or other vascular complications, risk factors associated with progressions and the management of progression to T2DM, ending with clinical recommendations.

TRIPOD statement: a preliminary pre-post analysis of reporting and methods of prediction models

OBJECTIVES

To assess the difference in completeness of reporting and methodological conduct of published prediction models before and after publication of the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement.

METHODS

In the seven general medicine journals with the highest impact factor, we compared the completeness of the reporting and the quality of the methodology of prediction model studies published between 2012 and 2014 (pre-TRIPOD) with studies published between 2016 and 2017 (post-TRIPOD). For articles published in the post-TRIPOD period, we examined whether there was improved reporting for articles (1) citing the TRIPOD statement, and (2) published in journals that published the TRIPOD statement.

RESULTS

A total of 70 articles was included (pre-TRIPOD: 32, post-TRIPOD: 38). No improvement was seen for the overall percentage of reported items after the publication of the TRIPOD statement (pre-TRIPOD 74%, post-TRIPOD 76%, 95% CI of absolute difference: -4% to 7%). For the individual TRIPOD items, an improvement was seen for 16 (44%) items, while 3 (8%) items showed no improvement and 17 (47%) items showed a deterioration. Post-TRIPOD, there was no improved reporting for articles citing the TRIPOD statement, nor for articles published in journals that published the TRIPOD statement. The methodological quality improved in the post-TRIPOD period. More models were externally validated in the same article (absolute difference 8%, post-TRIPOD: 39%), used measures of calibration (21%, post-TRIPOD: 87%) and discrimination (9%, post-TRIPOD: 100%), and used multiple imputation for handling missing data (12%, post-TRIPOD: 50%).

CONCLUSIONS

Since the publication of the TRIPOD statement, some reporting and methodological aspects have improved. Prediction models are still often poorly developed and validated and many aspects remain poorly reported, hindering optimal clinical application of these models. Long-term effects of the TRIPOD statement publication should be evaluated in future studies.

Development and internal validation of a prediction tool to aid the diagnosis of Cushing's syndrome in dogs attending primary-care practice

BACKGROUND

Novel methods to aid identification of dogs with spontaneous Cushing's syndrome are warranted to optimize case selection for diagnostics, avoid unnecessary testing, and ultimately aid decision-making for veterinarians.

HYPOTHESIS/OBJECTIVES

To develop and internally validate a prediction tool for dogs receiving a diagnosis of Cushing's syndrome using primary-care electronic health records.

ANIMALS

Three hundred and ninety-eight dogs diagnosed with Cushing's syndrome and 541 noncase dogs, tested for but not diagnosed with Cushing's syndrome, from a cohort of 905 544 dogs attending VetCompass participating practices.

METHODS

A cross-sectional study design was performed. A prediction model was developed using multivariable binary logistic regression taking the demography, presenting clinical signs and some routine laboratory results into consideration. Predictive performance of each model was assessed and internally validated through bootstrap resampling. A novel clinical prediction tool was developed from the final model.

RESULTS

The final model included predictor variables sex, age, breed, polydipsia, vomiting, potbelly/hepatomegaly, alopecia, pruritus, alkaline phosphatase, and urine specific gravity. The model demonstrated good discrimination (area under the receiver operating curve [AUROC] = 0.78 [95% CI = 0.75-0.81]; optimism-adjusted AUROC = 0.76) and calibration (C-slope = 0.86). A tool was developed from the model which calculates the predicted likelihood of a dog having Cushing's syndrome from 0% (score = -13) to 96% (score = 10).

CONCLUSIONS AND CLINICAL IMPORTANCE

A tool to predict a diagnosis of Cushing's syndrome at the point of first suspicion in dogs was developed, with good predictive performance. This tool can be used in practice to support decision-making and increase confidence in diagnosis.

Use and performance of machine learning models for type 2 diabetes prediction in community settings: A systematic review and meta-analysis

OBJECTIVE

We aimed to identify machine learning (ML) models for type 2 diabetes (T2DM) prediction in community settings and determine their predictive performance.

METHOD

Systematic review of ML predictive modelling studies in 13 databases since 2009 was conducted. Primary outcomes included metrics of discrimination, calibration, and classification. Secondary outcomes included important variables, level of validation, and intended use of models. Meta-analysis of c-indices, subgroup analyses, meta-regression, publication bias assessments and sensitivity analyses were conducted.

RESULTS

Twenty-three studies (40 prediction models) were included. Studies with high-, moderate-, and low- risk of bias were 3, 14, and 6 respectively. All studies conducted internal validation whereas none conducted external validation of their models. Twenty studies provided classification metrics to varying extents whereas only 7 studies performed model calibration. Eighteen studies reported information on both the variables used for model development and the feature importance. Twelve studies highlighted potential applicability of their models for T2DM screening. Meta-analysis produced a good pooled c-index (0.812). Sources of heterogeneity were identified through subgroup analyses and meta-regression. Issues pertaining to methodological quality and reporting were observed.

CONCLUSIONS

We found evidence of good performance of ML models for T2DM prediction in the community. Improvements to methodology, reporting and validation are needed before they can be used at scale.

A Newly Developed Diabetes Risk Index, Based on Lipoprotein Subfractions and Branched Chain Amino Acids, is Associated with Incident Type 2 Diabetes Mellitus in the PREVEND Cohort

Objective: Evaluate the ability of a newly developed diabetes risk score, the Diabetes Risk Index (DRI), to predict incident type 2 diabetes mellitus (T2D) in a large adult population. Methods: The DRI was developed by combining the Lipoprotein Insulin Resistance Index (LP-IR), calculated from 6 lipoprotein subspecies and size parameters, and the branched chain amino acids, valine and leucine, all of which have been shown previously to be associated with future T2D. DRI scores were calculated in a total of 6134 nondiabetic men and women in the Prevention of Renal and Vascular End-Stage Disease (PREVEND) Study. Cox proportional hazards regression was used to evaluate the association of DRI scores with incident T2D. Results: During a median follow-up of 8.5 years, 306 new T2D cases were ascertained. In analyses adjusted for age and sex, there was a significant association between DRI scores and incident T2D with the hazard ratio (HR) for the highest versus lowest quartile being 12.07 (95% confidence interval: 6.97–20.89, p < 0.001). After additional adjustment for body mass index (BMI), family history of T2D, alcohol consumption, diastolic blood pressure, total cholesterol, triglycerides, HDL cholesterol and HOMA-IR, the HR was attenuated but remained significant (HR 3.20 (1.73–5.95), p = 0.001). Similar results were obtained when DRI was analyzed as HR per 1 SD increase (HR 1.37 (1.14–1.65), p < 0.001). The Kaplan–Meier plot demonstrated that patients in the highest quartile of DRI scores presented at higher risk (p-value for log-rank test <0.001). Conclusions: Higher DRI scores are associated with an increased risk of T2D. The association is independent of clinical risk factors for T2D including HOMA-IR, BMI and conventional lipids.

The added value of periodontal measurements for identification of diabetes among Saudi adults

BACKGROUND

The aims of this study were to develop a prediction model for identification of individuals with diabetes based on clinical and perceived periodontal measurements; and to evaluate its added value when combined with standard diabetes screening tools.

METHODS

The study was carried out among 250 adults attending primary care clinics in Riyadh (Saudi Arabia). The study adopted a case-control approach, where diabetes status was first ascertained, and the Finnish Diabetes Risk Score (FINDRISC), Canadian Diabetes Risk questionnaire (CANRISK), and periodontal examinations were carried out afterward.

RESULTS

A periodontal prediction model (PPM) including three periodontal indicators (missing teeth, percentage of sites with pocket probing depth ≥6 mm, and mean pocket probing depth) had an area under the curve (AUC) of 0.694 (95% Confidence Interval: 0.612-0.776) and classified correctly 62.4% of participants. The FINDRISC and CANRISK tools had AUCs of 0.766 (95% CI: 0.690-0.843) and 0.821 (95% CI: 0.763-0.879), respectively. The addition of the PPM significantly improved the AUC of FINDRISC (P = 0.048) but not of CANRISK (P = 0.144), with 26.8% and 9.8% of participants correctly reclassified, respectively. Finally, decision curve analysis showed that adding the PPM to both tools would result in net benefits among patients with probability scores lower than 70%.

CONCLUSIONS

This study showed that periodontal measurements could play a role in identifying individuals with diabetes, and that addition of clinical periodontal measurements improved the performance of FINDRISC and CANRISK.

Improving Accuracy for Diabetes Mellitus Prediction by Using Deepnet

Diabetes is a salient issue and a significant health care concern for many nations. The forecast for the prevalence of diabetes is on the rise. Hence, building a prediction machine learning model to assist in the identification of diabetic patients is of great interest. This study aims to create a machine learning model that is capable of predicting diabetes with high performance. The following study used the BigML platform to train four machine learning algorithms, namely, Deepnet, Models (decision tree), Ensemble and Logistic Regression, on data sets collected from the Ministry of National Guard Hospital Affairs (MNGHA) in Saudi Arabia between the years of 2013 and 2015. The comparative evaluation criteria for the four algorithms examined included; Accuracy, Precision, Recall, F-measure and PhiCoefficient. Results show that the Deepnet algorithm achieved higher performance compared to other machine learning algorithms based on various evaluation matrices.

Interaction between sex and rurality on the prevalence of diabetes in Guyana: a nationally representative study

INTRODUCTION

Diabetes prevalence has never been measured in Guyana. We conducted a nationally representative cross-sectional study to estimate the prevalence of diabetes and pre-diabetes, and the association between sex and diabetes.

RESEARCH DESIGN AND METHODS

In 2016, the Ministry of Public Health led Guyana's first national STEPS survey among adults aged 18-69 years. Half of the participants were randomly selected for hemoglobin A1c and fasting blood glucose testing. We estimated the prevalence of diabetes and pre-diabetes and measured the association between sex and diabetes prevalence using logistic regression to compute adjusted ORs.

RESULTS

We included 805 adults (511 women, 294 men, mean age 41.8 (SD 14.4) years). The national prevalence of diabetes was 18.1% (95% CI: 15.4% to 20.8%), with higher rates among women (21.4%, 95% CI: 18.0% to 24.7%) than men (15.1%, 95% CI: 10.9% to 19.3%). Sex-specific diabetes prevalence varied significantly across urban and rural areas (p=0.002 for interaction). In rural areas, diabetes was twice as common among women (24.1%, 95% CI: 20.1% to 28.2%) compared with men (11.8%, 95% CI: 7.7% to 15.9%). After adjusting for prespecified covariates, rural women had double the odds of diabetes compared with rural men (OR 2.1, 95% CI: 1.20 to 3.82). This prevalence pattern was reversed in urban areas (diabetes prevalence, women: 13.9%, 95% CI: 8.7% to 19.0%; men: 22.0%, 95% CI: 12.9% to 31.1%), with urban women having half the odds of diabetes compared with urban men (OR 0.4, 95% CI: 0.20 to 0.99). We estimated that nearly one-third of women and over a quarter of men had diabetes or pre-diabetes.

CONCLUSIONS

The burden of diabetes in Guyana is considerably higher than previously estimated, with an unexpectedly high prevalence among women-particularly in rural areas.

Early warning scores for detecting deterioration in adult hospital patients: systematic review and critical appraisal of methodology

OBJECTIVE

To provide an overview and critical appraisal of early warning scores for adult hospital patients.

DESIGN

Systematic review.

DATA SOURCES

Medline, CINAHL, PsycInfo, and Embase until June 2019.

ELIGIBILITY CRITERIA FOR STUDY SELECTION

Studies describing the development or external validation of an early warning score for adult hospital inpatients.

RESULTS

13 171 references were screened and 95 articles were included in the review. 11 studies were development only, 23 were development and external validation, and 61 were external validation only. Most early warning scores were developed for use in the United States (n=13/34, 38%) and the United Kingdom (n=10/34, 29%). Death was the most frequent prediction outcome for development studies (n=10/23, 44%) and validation studies (n=66/84, 79%), with different time horizons (the most frequent was 24 hours). The most common predictors were respiratory rate (n=30/34, 88%), heart rate (n=28/34, 83%), oxygen saturation, temperature, and systolic blood pressure (all n=24/34, 71%). Age (n=13/34, 38%) and sex (n=3/34, 9%) were less frequently included. Key details of the analysis populations were often not reported in development studies (n=12/29, 41%) or validation studies (n=33/84, 39%). Small sample sizes and insufficient numbers of event patients were common in model development and external validation studies. Missing data were often discarded, with just one study using multiple imputation. Only nine of the early warning scores that were developed were presented in sufficient detail to allow individualised risk prediction. Internal validation was carried out in 19 studies, but recommended approaches such as bootstrapping or cross validation were rarely used (n=4/19, 22%). Model performance was frequently assessed using discrimination (development n=18/22, 82%; validation n=69/84, 82%), while calibration was seldom assessed (validation n=13/84, 15%). All included studies were rated at high risk of bias.

CONCLUSIONS

Early warning scores are widely used prediction models that are often mandated in daily clinical practice to identify early clinical deterioration in hospital patients. However, many early warning scores in clinical use were found to have methodological weaknesses. Early warning scores might not perform as well as expected and therefore they could have a detrimental effect on patient care. Future work should focus on following recommended approaches for developing and evaluating early warning scores, and investigating the impact and safety of using these scores in clinical practice.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42017053324.

Improved Landmark Dynamic Prediction Model to Assess Cardiovascular Disease Risk in On-Treatment Blood Pressure Patients: A Simulation Study and Post Hoc Analysis on SPRINT Data

Landmark model (LM) is a dynamic prediction model that uses a longitudinal biomarker in time-to-event data to make prognosis prediction. This study was designed to improve this model and to apply it to assess the cardiovascular risk in on-treatment blood pressure patients. A frailty parameter was used in LM, landmark frailty model (LFM), to account the frailty of the patients and measure the correlation between different landmarks. The proposed model was compared with LM in different scenarios respecting data missing status, sample size (100, 200, and 400), landmarks (6, 12, 24, and 48), and failure percentage (30, 50, and 100%). Bias of parameter estimation and mean square error as well as deviance statistic between models were compared. Additionally, discrimination and calibration capability as the goodness of fit of the model were evaluated using dynamic concordance index (DCI), dynamic prediction error (DPE), and dynamic relative prediction error (DRPE). The proposed model was performed on blood pressure data, obtained from systolic blood pressure intervention trial (SPRINT), in order to calculate the cardiovascular risk. Dynpred, coxme, and coxphw packages in the R.3.4.3 software were used. It was proved that our proposed model, LFM, had a better performance than LM. Parameter estimation in LFM was closer to true values in comparison to that in LM. Deviance statistic showed that there was a statistically significant difference between the two models. In the landmark numbers 6, 12, and 24, the LFM had a higher DCI over time and the three landmarks showed better performance in discrimination. Both DPE and DRPE in LFM were lower in comparison to those in LM over time. It was indicated that LFM had better calibration in comparison to its peer. Moreover, real data showed that the structure of prognostic process was predicted better in LFM than in LM. Accordingly, it is recommended to use the LFM model for assessing cardiovascular risk due to its better performance.

Comparative analysis of screening models for undiagnosed diabetes in Mexico

BACKGROUND

It is estimated that 37% of Mexican adults have undiagnosed diabetes, and are therefore at high risk of developing the severe and devastating complications associated to it. In recent years, a variety of screening tools based on the characteristics of the adult Mexican population have been proposed in order to reduce the negative effects of the disease.

OBJECTIVES

To assess the performance of screening models to diagnose diabetes in the Mexican adult population and to propose a screening model based on HbA1c measurements.

MATERIALS AND METHODS

Data from the 2016 Halfway National Health and Nutrition Survey (NHNS) were used to assess the screening models and to develop and validate the proposed 2016 NHNS model, built using a multivariate logistic regression model. Explanatory variables included in the 2016 NHNS 2016 model were selected through a stepwise backward procedure, using sensitivity and specificity as performance indicators.

RESULTS

Of the screening models assessed, only the model based on the 2006 NHNS survey showed a performance consistent with previous reports. The proposed 2016 NHNS model included age, waist circumference, and systolic blood pressure as explanatory variables and showed a sensitivity of 0.72 and a specificity of 0.80 in the validation data set.

CONCLUSIONS

Age, waist circumference, and systolic blood pressure are variables of special importance for early detection of undiagnosed diabetes in Mexican adults. Based on the consistent performance of the 2006 NHNS model in different data sets, its use as a screening tool for adults with undiagnosed diabetes in Mexico is recommended.

Durability of glycaemic control in patients with type 2 diabetes after metformin failure: Prognostic model derivation and validation using the DISCOVER study

AIM

To develop and internally validate prognostic models on the long-term durability of glycaemic control in patients with type 2 diabetes after metformin failure.

MATERIALS AND METHODS

DISCOVER is a 3-year, prospective observational study across six continents investigating second-line glucose-lowering therapies. In this analysis from 35 countries, we included patients on metformin initiating second-line glucose-lowering medication(s) because of physician-defined lack of efficacy. The outcome was durability of glycaemic control, defined as three consecutive levels of HbA1c at 6-, 12- and 24-month follow-up at target (HbA1c equal to or lower than the level when the physician initiated the second-line therapy in patients with baseline HbA1c ≤7% [53 mmol/mol]; and equal to or lower than 7% in those with baseline HbA1c >7%). We developed and internally validated two prognostic models: a base model, which included age, sex, ethnicity, country income group, baseline HbA1c and second-line therapy, and an advanced model, established through statistical variable selections from a model including base variables and 13 additional predictors selected from a literature review. We used logistic regression to develop and 500 bootstrapping samples to internally validate the models; discrimination and calibration were used to assess model performance.

RESULTS

Overall, 896 out of 2995 participants (29.9%) had sustained glycaemic control. The base model performed well: Nagelkerke R² was 0.13, C-index 0.70 (95% CI: 0.68, 0.71) and bias-corrected C-index 0.69 after internal validation. Diabetes duration, insurance type, estimated glomerular filtration rate and glucose self-monitoring were additionally selected in the advanced model, which had only a slightly better performance compared with the base model: Nagelkerke R² 0.20, C-index 0.71 (95% CI: 0.69, 0.73) and bias-corrected C-index 0.70. Calibration plots showed good calibrations of both validated models.

CONCLUSION

These prognostic models, which include simple demographic and routinely collected clinical information, enabled the estimation of the probability of 2-year sustained glycaemic control in patients after metformin failure. The models have been implemented into a web-based tool to support healthcare professionals in their decisions.

Systematic review of prediction models for gestational hypertension and preeclampsia

INTRODUCTION

Prediction models for gestational hypertension and preeclampsia have been developed with data and assumptions from developed countries. Their suitability and application for low resource settings have not been tested. This review aimed to identify and assess the methodological quality of prediction models for gestational hypertension and pre-eclampsia with reference to their application in low resource settings.

METHODS

Using combinations of keywords for gestational hypertension, preeclampsia and prediction models seven databases were searched to identify prediction models developed with maternal data obtained before 20 weeks of pregnancy and including at least three predictors (Prospero registration CRD 42017078786). Prediction model characteristics and performance measures were extracted using the CHARMS, STROBE and TRIPOD checklists. The National Institute of Health quality assessment tools for observational cohort and cross-sectional studies were used for study quality appraisal.

RESULTS

We retrieved 8,309 articles out of which 40 articles were eligible for review. Seventy-seven percent of all the prediction models combined biomarkers with maternal clinical characteristics. Biomarkers used as predictors in most models were pregnancy associated plasma protein-A (PAPP-A) and placental growth factor (PlGF). Only five studies were conducted in a low-and middle income country.

CONCLUSIONS

Most of the studies evaluated did not completely follow the CHARMS, TRIPOD and STROBE guidelines in prediction model development and reporting. Adherence to these guidelines will improve prediction modelling studies and subsequent application of prediction models in clinical practice. Prediction models using maternal characteristics, with good discrimination and calibration, should be externally validated for use in low and middle income countries where biomarker assays are not routinely available.

Targeting Inflammatory-Mitochondrial Response in Major Depression: Current Evidence and Further Challenges

The prevalence of psychiatric disorders has increased in recent years. Among existing mental disorders, major depressive disorder (MDD) has emerged as one of the leading causes of disability worldwide, affecting individuals throughout their lives. Currently, MDD affects 15% of adults in the Americas. Over the past 50 years, pharmacotherapy, psychotherapy, and brain stimulation have been used to treat MDD. The most common approach is still pharmacotherapy; however, studies show that about 40% of patients are refractory to existing treatments. Although the monoamine hypothesis has been widely accepted as a molecular mechanism to explain the etiology of depression, its relationship with other biochemical phenomena remains only partially understood. This is the case of the link between MDD and inflammation, mitochondrial dysfunction, and oxidative stress. Studies have found that depressive patients usually exhibit altered inflammatory markers, mitochondrial membrane depolarization, oxidized mitochondrial DNA, and thus high levels of both central and peripheral reactive oxygen species (ROS). The effect of antidepressants on these events remains unclear. Nevertheless, the effects of ROS on the brain are well known, including lipid peroxidation of neuronal membranes, accumulation of peroxidation products in neurons, protein and DNA damage, reduced antioxidant defenses, apoptosis induction, and neuroinflammation. Antioxidants such as ascorbic acid, tocopherols, and coenzyme Q have shown promise in some depressive patients, but without consensus on their efficacy. Hence, this paper provides a review of MDD and its association with inflammation, mitochondrial dysfunction, and oxidative stress and is aimed at thoroughly discussing the putative links between these events, which may contribute to the design and development of new therapeutic approaches for patients.

Development and Validation of a New Diabetes Risk Score in Guyana

INTRODUCTION

We present a new diabetes risk score developed and validated in a multi-ethnic population in Guyana, South America. Measurement of in-country diabetes prevalence is a vital epidemiologic tool to combat the pandemic. It is believed that for every person diagnosed with type 2 diabetes there is another undiagnosed. The International Diabetes Federation (IDF) recommends a two-step detection programme using a risk score questionnaire to identify high-risk individuals followed by glycaemic measure.

METHODS

Data on 798 persons from the 2016 STEPwise Approach to Chronic Disease Risk Factor Surveillance (STEPS) were used to correlate responses to 36 questions with glycated haemoglobin (HbA1C) and fasting plasma glucose (FPG) results. Bootstrapping was used to internally validate the derived seven-variable model. This model with the addition of family history questions was tested in a convenience sample of 659 Guyanese adults and externally validated in a cohort of another 528.

RESULTS

An 8-item Guyana Diabetes Risk Score (GDRS) was derived. The final model performed with an area under the curve (AUC) of 0.812 CONCLUSIONS: The validated eight-item Guyana Diabetes Risk Score will be extremely useful in identifying individuals at high risk of having diabetes in Caribbean, Black or East Indian populations.

FINDRISC in Latin America: a systematic review of diagnosis and prognosis models

This review aimed to assess whether the FINDRISC, a risk score for type 2 diabetes mellitus (T2DM), has been externally validated in Latin America and the Caribbean (LAC). We conducted a systematic review following the CHARMS (CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies) framework. Reports were included if they validated or re-estimated the FINDRISC in population-based samples, health facilities or administrative data. Reports were excluded if they only studied patients or at-risk individuals. The search was conducted in Medline, Embase, Global Health, Scopus and LILACS. Risk of bias was assessed with the PROBAST (Prediction model Risk of Bias ASsessment Tool) tool. From 1582 titles and abstracts, 4 (n=7502) reports were included for qualitative summary. All reports were from South America; there were slightly more women, and the mean age ranged from 29.5 to 49.7 years. Undiagnosed T2DM prevalence ranged from 2.6% to 5.1%. None of the studies conducted an independent external validation of the FINDRISC; conversely, they used the same (or very similar) predictors to fit a new model. None of the studies reported calibration metrics. The area under the receiver operating curve was consistently above 65.0%. All studies had high risk of bias. There has not been any external validation of the FINDRISC model in LAC. Selected reports re-estimated the FINDRISC, although they have several methodological limitations. There is a need for big data to develop-or improve-T2DM diagnostic and prognostic models in LAC. This could benefit T2DM screening and early diagnosis.

Evaluating the Performance of the Indian Diabetes Risk Score in Different Ethnic Groups

Aim: To evaluate the performance of Madras Diabetes Research Foundation-Indian Diabetes Risk Score (MDRF-IDRS) in different ethnic groups, including Indians, Hispanic, non-Hispanic whites, non-Hispanic blacks, and other American. Methods: The MDRF-IDRS is calculated based on a risk equation that includes age, waist circumference, family history of diabetes, and physical activity. The National Health and Nutrition Examination Survey data on American and Chennai Urban Rural Epidemiology Study data on Indians were used in this study. Study participants aged ≥20 years with and without type 2 diabetes were included. Performance of the MDRF-IDRS was assessed using sensitivity, specificity, positive predictive value, negative predictive value, and the area under the receiver operating characteristic curve (AUC) measures within each ethnic group. IDRSs' performance was then compared with existing noninvasive American diabetes risk scores. Results: Total number of participants included was 11,035 (2292 Indians and 8743 Americans). MDRF-IDRS (cutoff ≥60) performed well in Indians with an AUC, sensitivity, and specificity of 0.73, 80.2%, and 57.3%, respectively. MDRF-IDRS cutoff ≥70 had the highest discriminative performance among Hispanic, non-Hispanic whites, and non-Hispanic blacks with sensitivity and specificity of between 70.1%-86.9% and 61.2%-72.2%, respectively. The AUC for American was between 0.77 and 0.81 with the highest and lowest AUC in non-Hispanic black and non-Hispanic white, respectively. With a smaller number of variables, IDRS showed almost the same performance in predicting diabetes among American compared with the existing noninvasive American diabetes risk score. Conclusion: The MDRF-IDRS performs well among Indians and Americans, including Hispanic, non-Hispanic white, non-Hispanic black, and other American. It can be used as a screening tool to help in early diagnosis, management, and optimal control of diabetes mainly in mass screening programs in India and America.

Development and validation of nomograms to accurately predict risk of recurrence for patients with laryngeal squamous cell carcinoma: Cohort study

BACKGROUND

Recurrence is still major obstacle to long-term survival in laryngeal squamous cell carcinoma (LSCC). We aimed to establish and validate a nomogram to precisely predict recurrence probability in patients with LSCC.

METHODS

A total of 283 consecutive patients with LSCC received curative-intend surgery between 2011 and 2014 at were enrolled in this study. Subsequently, 283 LSCC patients were randomly assigned to a training cohort (N = 171) and a validation cohort (N = 112) in a 3:2 ratio. According to the results of multivariable Cox regression analysis in the training cohort, we developed a nomogram. The predictive accuracy and discriminative ability of the nomogram were evaluated by calibration curve and concordance index (C-index), and compared with TNM stage system by C-index, receiver operating characteristic (ROC) analysis. Decision curve analysis (DCA) was performed to estimate clinical value of our nomogram.

RESULTS

Six independent factors rooted in multivariable analysis of the training cohort to predict recurrence were age, tumor site, smoking, alcohol, N stage and hemoglobin, which were all integrated into the nomogram. The calibration curve for the probability of recurrence presented that the nomogram-based predictions were in good correspondence with actual observations. The C-index of the nomogram was 0.81 (0.75-0.88), and the area under curve (AUC) of nomogram in predicting recurrence free survival (RFS) was 0.894, which were significantly better than traditional TNM stage. Decision curve analysis further affirmed that our nomogram had a larger net benefit than TNM stage. The results were confirmed in the validation cohort.

CONCLUSION

A risk prediction nomogram for patients with LSCC, incorporating readily assessable clinicopathologic variables, generates more accurate estimations of the recurrence probability when compared TNM stage alone, but still needs additional data before being used in clinical implications.

Machine learning for characterizing risk of type 2 diabetes mellitus in a rural Chinese population: the Henan Rural Cohort Study

With the development of data mining, machine learning offers opportunities to improve discrimination by analyzing complex interactions among massive variables. To test the ability of machine learning algorithms for predicting risk of type 2 diabetes mellitus (T2DM) in a rural Chinese population, we focus on a total of 36,652 eligible participants from the Henan Rural Cohort Study. Risk assessment models for T2DM were developed using six machine learning algorithms, including logistic regression (LR), classification and regression tree (CART), artificial neural networks (ANN), support vector machine (SVM), random forest (RF) and gradient boosting machine (GBM). The model performance was measured in an area under the receiver operating characteristic curve, sensitivity, specificity, positive predictive value, negative predictive value and area under precision recall curve. The importance of variables was identified based on each classifier and the shapley additive explanations approach. Using all available variables, all models for predicting risk of T2DM demonstrated strong predictive performance, with AUCs ranging between 0.811 and 0.872 using laboratory data and from 0.767 to 0.817 without laboratory data. Among them, the GBM model performed best (AUC: 0.872 with laboratory data and 0.817 without laboratory data). Performance of models plateaued when introduced 30 variables to each model except CART model. Among the top-10 variables across all methods were sweet flavor, urine glucose, age, heart rate, creatinine, waist circumference, uric acid, pulse pressure, insulin, and hypertension. New important risk factors (urinary indicators, sweet flavor) were not found in previous risk prediction methods, but determined by machine learning in our study. Through the results, machine learning methods showed competence in predicting risk of T2DM, leading to greater insights on disease risk factors with no priori assumption of causality.

How valid are projections of the future prevalence of diabetes? Rapid reviews of prevalence-based and Markov chain models and comparisons of different models' projections for England

OBJECTIVES

To examine validity of prevalence-based models giving projections of prevalence of diabetes in adults, in England and the UK, and of Markov chain models giving estimates of economic impacts of interventions to prevent type 2 diabetes (T2D).

METHODS

Rapid reviews of both types of models. Estimation of the future prevalence of T2D in England by Markov chain models; and from the trend in the prevalence of diabetes, as reported in the Quality and Outcomes Framework (QOF), estimated by ordinary least squares regression analysis.

SETTING

Adult population in England and UK.

MAIN OUTCOME MEASURE

Prevalence of T2D in England and UK in 2025.

RESULTS

The prevalence-based models reviewed use sample estimates of past prevalence rates by age and sex and projected population changes. Three most recent models, including that of Public Health England (PHE), neither take account of increases in obesity, nor report Confidence Intervals (CIs). The Markov chain models reviewed use transition probabilities between states of risk and death, estimated from various sources. None of their accounts give the full matrix of transition probabilities, and only a minority report tests of validation. Their primary focus is on estimating the ratio of costs to benefits of preventive interventions in those with hyperglycaemia, only one reported estimates of those developing T2D in the absence of a preventive intervention in the general population.Projections of the prevalence of T2D in England in 2025 were (in millions) by PHE, 3.95; from the QOF trend, 4.91 and by two Markov chain models, based on our review, 5.64 and 9.07.

CONCLUSIONS

To inform national policies on preventing T2D, governments need validated models, designed to use available data, which estimate the scale of incidence of T2D and survival in the general population, with and without preventive interventions.

Three-component non-invasive risk score for undiagnosed diabetes in Chinese people: Development, validation and longitudinal evaluation

AIMS/INTRODUCTION

To develop a new non-invasive risk score for undiagnosed diabetes in Chinese people, and to evaluate the incident diabetes risk in those with high-risk scores, but no diabetes on initial testing.

MATERIALS AND METHODS

A total of 2,609 participants with no known diabetes (aged 25-74 years) who underwent oral glucose tolerance tests in Hong Kong (HK) were investigated for independent risk factors of diabetes to develop a categorization point scoring system, the Non-invasive Diabetes Score (NDS). This NDS was validated in a cross-sectional study of 2,746 participants in Shaanxi, China. HK participants tested to not have diabetes at baseline were assessed for subsequent incident diabetes rates.

RESULTS

In the HK cohort, hypertension, age and body mass index were the key independent risk factors selected to develop the NDS, with ≥28 out of 50 NDS points considered as high risk. The area under the receiver operating characteristic curve for undiagnosed diabetes was 0.818 and 0.720 for the HK and Shaanxi cohort, respectively. The negative predictive value was 97.4% (HK) and 95.8% (Shaanxi); the number needed to screen to identify one case of diabetes was five (HK) and 11 (Shaanxi), respectively. Among those that tested non-diabetes at baseline, individuals with NDS ≥28 had a threefold risk of incident diabetes during the subsequent 20.9 years, compared with those with NDS <28 (P < 0.001), with a steeper rise in incident diabetes observed in those with NDS at higher tertiles.

CONCLUSIONS

This new three-component risk score is a user-friendly tool for diabetes screening, and might inform the subsequent testing interval for high-risk non-diabetes individuals.

Efficient treatment of outliers and class imbalance for diabetes prediction

Learning from outliers and imbalanced data remains one of the major difficulties for machine learning classifiers. Among the numerous techniques dedicated to tackle this problem, data preprocessing solutions are known to be efficient and easy to implement. In this paper, we propose a selective data preprocessing approach that embeds knowledge of the outlier instances into artificially generated subset to achieve an even distribution. The Synthetic Minority Oversampling TEchnique (SMOTE) was used to balance the training data by introducing artificial minority instances. However, this was not before the outliers were identified and oversampled (irrespective of class). The aim is to balance the training dataset while controlling the effect of outliers. The experiments prove that such selective oversampling empowers SMOTE, ultimately leading to improved classification performance.

Predicting diabetes clinical outcomes using longitudinal risk factor trajectories

BACKGROUND

The ubiquity of electronic health records (EHR) offers an opportunity to observe trajectories of laboratory results and vital signs over long periods of time. This study assessed the value of risk factor trajectories available in the electronic health record to predict incident type 2 diabetes.

STUDY DESIGN AND METHODS

Analysis was based on a large 13-year retrospective cohort of 71,545 adult, non-diabetic patients with baseline in 2005 and median follow-up time of 8 years. The trajectories of fasting plasma glucose, lipids, BMI and blood pressure were computed over three time frames (2000-2001, 2002-2003, 2004) before baseline. A novel method, Cumulative Exposure (CE), was developed and evaluated using Cox proportional hazards regression to assess risk of incident type 2 diabetes. We used the Framingham Diabetes Risk Scoring (FDRS) Model as control.

RESULTS

The new model outperformed the FDRS Model (.802 vs .660; p-values <2e-16). Cumulative exposure measured over different periods showed that even short episodes of hyperglycemia increase the risk of developing diabetes. Returning to normoglycemia moderates the risk, but does not fully eliminate it. The longer an individual maintains glycemic control after a hyperglycemic episode, the lower the subsequent risk of diabetes.

CONCLUSION

Incorporating risk factor trajectories substantially increases the ability of clinical decision support risk models to predict onset of type 2 diabetes and provides information about how risk changes over time.

Can risk be predicted? An umbrella systematic review of current risk prediction models for cardiovascular diseases, diabetes and hypertension

OBJECTIVE

To provide an overview of the currently available risk prediction models (RPMs) for cardiovascular diseases (CVDs), diabetes and hypertension, and to compare their effectiveness in proper recognition of patients at risk of developing these diseases.

DESIGN

Umbrella systematic review.

DATA SOURCES

PubMed, Scopus, Cochrane Library.

ELIGIBILITY CRITERIA

Systematic reviews or meta-analysis examining and comparing performances of RPMs for CVDs, hypertension or diabetes in healthy adult (18-65 years old) population, published in English language.

DATA EXTRACTION AND SYNTHESIS

Data were extracted according to the following parameters: number of studies included, intervention (RPMs applied/assessed), comparison, performance, validation and outcomes. A narrative synthesis was performed. Data were reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.

STUDY SELECTION

3612 studies were identified. After title/abstract screening and removal of duplicate articles, 37 studies met the eligibility criteria. After reading the full text, 13 were deemed relevant for inclusion. Three further papers from the reference lists of these articles were then added.

STUDY APPRAISAL

The methodological quality of the included studies was assessed using the AMSTAR tool.

RISK OF BIAS IN INDIVIDUAL STUDIES

Risk of Bias evaluation was carried out using the ROBIS tool.

RESULTS

Sixteen studies met the inclusion criteria: six focused on diabetes, two on hypertension and eight on CVDs. Globally, prediction models for diabetes and hypertension showed no significant difference in effectiveness. Conversely, some promising differences among prediction tools were highlighted for CVDs. The Ankle-Brachial Index, in association with the Framingham tool, and QRISK scores provided some evidence of a certain superiority compared with Framingham alone.

LIMITATIONS

Due to the significant heterogeneity of the studies, it was not possible to perform a meta-analysis. The electronic search was limited to studies in English and to three major international databases (MEDLINE/PubMed, Scopus and Cochrane Library), with additional works derived from the reference list of other studies; grey literature with unpublished documents was not included in the search. Furthermore, no assessment of potential adverse effects of RPMs was carried out.

CONCLUSIONS

Consistent evidence is available only for CVD prediction: the Framingham score, alone or in combination with the Ankle-Brachial Index, and the QRISK score can be confirmed as the gold standard. Further efforts should not be concentrated on creating new scores, but rather on performing external validation of the existing ones, in particular on high-risk groups. Benefits could be further improved by supplementing existing models with information on lifestyle, personal habits, family and employment history, social network relationships, income and education.

PROSPERO REGISTRATION NUMBER

CRD42018088012.

Risk scores for type 2 diabetes mellitus in Latin America: a systematic review of population-based studies

AIM

To summarize the evidence on diabetes risk scores for Latin American populations.

METHODS

A systematic review was conducted (CRD42019122306) looking for diagnostic and prognostic models for type 2 diabetes mellitus among randomly selected adults in Latin America. Five databases (LILACS, Scopus, MEDLINE, Embase and Global Health) were searched. type 2 diabetes mellitus was defined using at least one blood biomarker and the reports needed to include information on the development and/or validation of a multivariable regression model. Risk of bias was assessed using the PROBAST guidelines.

RESULTS

Of the 1500 reports identified, 11 were studied in detail and five were included in the qualitative analysis. Two reports were from Mexico, two from Peru and one from Brazil. The number of diabetes cases varied from 48 to 207 in the derivations models, and between 29 and 582 in the validation models. The most common predictors were age, waist circumference and family history of diabetes, and only one study used oral glucose tolerance test as the outcome. The discrimination performance across studies was ~ 70% (range: 66-72%) as per the area under the receiving-operator curve, the highest metric was always the negative predictive value. Sensitivity was always higher than specificity.

CONCLUSION

There is no evidence to support the use of one risk score throughout Latin America. The development, validation and implementation of risk scores should be a research and public health priority in Latin America to improve type 2 diabetes mellitus screening and prevention.

Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data

Objective

To develop a conceptual prediction model framework containing standardized steps and describe the corresponding open-source software developed to consistently implement the framework across computational environments and observational healthcare databases to enable model sharing and reproducibility.

Methods

Based on existing best practices we propose a 5 step standardized framework for: (1) transparently defining the problem; (2) selecting suitable datasets; (3) constructing variables from the observational data; (4) learning the predictive model; and (5) validating the model performance. We implemented this framework as open-source software utilizing the Observational Medical Outcomes Partnership Common Data Model to enable convenient sharing of models and reproduction of model evaluation across multiple observational datasets. The software implementation contains default covariates and classifiers but the framework enables customization and extension.

Results

As a proof-of-concept, demonstrating the transparency and ease of model dissemination using the software, we developed prediction models for 21 different outcomes within a target population of people suffering from depression across 4 observational databases. All 84 models are available in an accessible online repository to be implemented by anyone with access to an observational database in the Common Data Model format.

Conclusions

The proof-of-concept study illustrates the framework’s ability to develop reproducible models that can be readily shared and offers the potential to perform extensive external validation of models, and improve their likelihood of clinical uptake. In future work the framework will be applied to perform an “all-by-all” prediction analysis to assess the observational data prediction domain across numerous target populations, outcomes and time, and risk settings.

Comparison Between Non-High-Density Lipoprotein Cholesterol and Low-Density Lipoprotein Cholesterol to Estimate Cardiovascular Risk Using a Multivariate Model

BACKGROUND

Although studies exist comparing low-density lipoprotein cholesterol (LDL-C) and non-high-density lipoprotein cholesterol (HDL-C) in the development of cardiovascular disease (CVD), most have limitations in the mathematical models used to evaluate their prognostic power adjusted for the other risk factors (cardiovascular risk).

OBJECTIVE

The aim of this study was to compare LDL-C and non-HDL-C in patients with CVD to determine whether both parameters predict CVD similarly.

METHODS

A cohort of 1322 subjects drawn from the general population of a Spanish region was followed between 1992 and 2006. The outcome was time to CVD. Secondary variables were gender, age, hypertension, diabetes, personal history of CVD, current smoker, body mass index, LDL-C, and non-HDL-C. Two CVD prediction models were constructed with the secondary variables, with only the lipid parameter varying (non-HDL-C or LDL-C). In the construction of the models, the following were considered: multiple imputation, events per variable of 10 or more, and continuous predictors as powers. The validation was conducted by bootstrapping obtaining the distribution of the C statistic (discrimination) and the probabilities observed by smooth curves. These results were compared in both models using graphical and analytical testing.

RESULTS

There were a total of 137 CVD events. The models showed no differences in the distributions of the C statistic (discrimination, P = .536) or in the calibration plot.

CONCLUSIONS

In our population, LDL-C and non-HDL-C were equivalent at predicting CVD. More studies using this methodology are needed to confirm these results.

Three myths about risk thresholds for prediction models

BACKGROUND

Clinical prediction models are useful in estimating a patient's risk of having a certain disease or experiencing an event in the future based on their current characteristics. Defining an appropriate risk threshold to recommend intervention is a key challenge in bringing a risk prediction model to clinical application; such risk thresholds are often defined in an ad hoc way. This is problematic because tacitly assumed costs of false positive and false negative classifications may not be clinically sensible. For example, when choosing the risk threshold that maximizes the proportion of patients correctly classified, false positives and false negatives are assumed equally costly. Furthermore, small to moderate sample sizes may lead to unstable optimal thresholds, which requires a particularly cautious interpretation of results.

MAIN TEXT

We discuss how three common myths about risk thresholds often lead to inappropriate risk stratification of patients. First, we point out the contexts of counseling and shared decision-making in which a continuous risk estimate is more useful than risk stratification. Second, we argue that threshold selection should reflect the consequences of the decisions made following risk stratification. Third, we emphasize that there is usually no universally optimal threshold but rather that a plausible risk threshold depends on the clinical context. Consequently, we recommend to present results for multiple risk thresholds when developing or validating a prediction model.

CONCLUSION

Bearing in mind these three considerations can avoid inappropriate allocation (and non-allocation) of interventions. Using discriminating and well-calibrated models will generate better clinical outcomes if context-dependent thresholds are used.

Development and validation of multivariable clinical diagnostic models to identify type 1 diabetes requiring rapid insulin therapy in adults aged 18-50 years

OBJECTIVE

To develop and validate multivariable clinical diagnostic models to assist distinguishing between type 1 and type 2 diabetes in adults aged 18-50.

DESIGN

Multivariable logistic regression analysis was used to develop classification models integrating five pre-specified predictor variables, including clinical features (age of diagnosis, body mass index) and clinical biomarkers (GADA and Islet Antigen 2 islet autoantibodies, Type 1 Diabetes Genetic Risk Score), to identify type 1 diabetes with rapid insulin requirement using data from existing cohorts.

SETTING

UK cohorts recruited from primary and secondary care.

PARTICIPANTS

1352 (model development) and 582 (external validation) participants diagnosed with diabetes between the age of 18 and 50 years of white European origin.

MAIN OUTCOME MEASURES

Type 1 diabetes was defined by rapid insulin requirement (within 3 years of diagnosis) and severe endogenous insulin deficiency (C-peptide <200 pmol/L). Type 2 diabetes was defined by either a lack of rapid insulin requirement or, where insulin treated within 3 years, retained endogenous insulin secretion (C-peptide >600 pmol/L at ≥5 years diabetes duration). Model performance was assessed using area under the receiver operating characteristic curve (ROC AUC), and internal and external validation.

RESULTS

Type 1 diabetes was present in 13% of participants in the development cohort. All five predictor variables were discriminative and independent predictors of type 1 diabetes (p<0.001 for all) with individual ROC AUC ranging from 0.82 to 0.85. Model performance was high: ROC AUC range 0.90 (95% CI 0.88 to 0.93) (clinical features only) to 0.97 (95% CI 0.96 to 0.98) (all predictors) with low prediction error. Results were consistent in external validation (clinical features and GADA ROC AUC 0.93 (0.90 to 0.96)).

CONCLUSIONS

Clinical diagnostic models integrating clinical features with biomarkers have high accuracy for identifying type 1 diabetes with rapid insulin requirement, and could assist clinicians and researchers in accurately identifying patients with type 1 diabetes.

Building Risk Prediction Models for Type 2 Diabetes Using Machine Learning Techniques

Introduction

As one of the most prevalent chronic diseases in the United States, diabetes, especially type 2 diabetes, affects the health of millions of people and puts an enormous financial burden on the US economy. We aimed to develop predictive models to identify risk factors for type 2 diabetes, which could help facilitate early diagnosis and intervention and also reduce medical costs.

Methods

We analyzed cross-sectional data on 138,146 participants, including 20,467 with type 2 diabetes, from the 2014 Behavioral Risk Factor Surveillance System. We built several machine learning models for predicting type 2 diabetes, including support vector machine, decision tree, logistic regression, random forest, neural network, and Gaussian Naive Bayes classifiers. We used univariable and multivariable weighted logistic regression models to investigate the associations of potential risk factors with type 2 diabetes.

Results

All predictive models for type 2 diabetes achieved a high area under the curve (AUC), ranging from 0.7182 to 0.7949. Although the neural network model had the highest accuracy (82.4%), specificity (90.2%), and AUC (0.7949), the decision tree model had the highest sensitivity (51.6%) for type 2 diabetes. We found that people who slept 9 or more hours per day (adjusted odds ratio [aOR] = 1.13, 95% confidence interval [CI], 1.03–1.25) or had checkup frequency of less than 1 year (aOR = 2.31, 95% CI, 1.86–2.85) had higher risk for type 2 diabetes.

Conclusion

Of the 8 predictive models, the neural network model gave the best model performance with the highest AUC value; however, the decision tree model is preferred for initial screening for type 2 diabetes because it had the highest sensitivity and, therefore, detection rate. We confirmed previously reported risk factors and also identified sleeping time and frequency of checkup as 2 new potential risk factors related to type 2 diabetes.

Which clinical and biochemical predictors should be used to screen for diabetes in patients with serious mental illness receiving antipsychotic medication? A large observational study

OBJECTIVE

We aimed to investigate which clinical and metabolic tests offer optimal accuracy and acceptability to help diagnose diabetes among a large sample of people with serious mental illness in receipt of antipsychotic medication.

METHODS

A prospective observational study design of biochemical and clinical factors was used. Biochemical measures were fasting glucose, insulin and lipids, oral glucose tolerance testing (OGTT), hemoglobin A1c, and insulin resistance assessed with the homeostatic model (HOMA-IR) were determined in a consecutive cohort of 798 adult psychiatric inpatients receiving antipsychotics. Clinical variables were gender, age, global assessment of functioning (GAF), mental health clinicians' global impression (CGI), duration of severe mental illness, height, weight, BMI and waist/hip ratio. In addition, we calculated the risk using combined clinical predictors using the Leicester Practice Risk Score (LPRS) and the Topics Diabetes Risk Score (TDRS). Diabetes was defined by older criteria (impaired fasting glucose (IFG) or OGTT) as well as2010 criteria (IFG or OGTT or Glycated haemoglobin (HBA1c)) at conventional cut-offs.

RESULTS

Using the older criteria, 7.8% had diabetes (men: 6.3%; women: 10.3%). Using the new criteria, 10.2% had diabetes (men: 8.2%, women: 13.2%), representing a 30.7% increase (p = 0.02) in the prevalence of diabetes. Regarding biochemical predictors, conventional OGTT, IFG, and HbA1c thresholds used to identify newly defined diabetes missed 25%, 50% and 75% of people with diabetes, respectively. The conventional HBA1c cut-point of ≥6.5% (48 mmol/mol) missed 7 of 10 newly defined cases of diabetes while a cut-point of ≥5.7% improved sensitivity from 44.4% to up to 85%. Specific algorithm approaches offered reasonable accuracy. Unfortunately no single clinical factor was able to accurately rule-in a diagnosis of diabetes. Three clinical factors were able to rule-out diabetes with good accuracy namely: BMI, waist/hip ratio and height. A BMI < 30 had a 92% negative predictive value in ruling-out diabetes. Of those not diabetic, 20% had a BMI ≥ 30. However, for complete diagnosis a specific biochemical protocol is still necessary.

CONCLUSIONS

Patients with SMI maintained on antipsychotic medication cannot be reliably screened for diabetes using clinical variables alone. Accurate assessment requires a two-step algorithm consisting of HBA1c ≥5.7% followed by both FG and OGTT which does not require all patients to have OGTT and FG.

Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature

Clinical prediction rules (CPRs) that predict the absolute risk of a clinical condition or future outcome for individual patients are abundant in the medical literature; however, systematic reviews have demonstrated shortcomings in the methodological quality and reporting of prediction studies. To maximise the potential and clinical usefulness of CPRs, they must be rigorously developed and validated, and their impact on clinical practice and patient outcomes must be evaluated. This review aims to present a comprehensive overview of the stages involved in the development, validation and evaluation of CPRs, and to describe in detail the methodological standards required at each stage, illustrated with examples where appropriate. Important features of the study design, statistical analysis, modelling strategy, data collection, performance assessment, CPR presentation and reporting are discussed, in addition to other, often overlooked aspects such as the acceptability, cost-effectiveness and longer-term implementation of CPRs, and their comparison with clinical judgement. Although the development and evaluation of a robust, clinically useful CPR is anything but straightforward, adherence to the plethora of methodological standards, recommendations and frameworks at each stage will assist in the development of a rigorous CPR that has the potential to contribute usefully to clinical practice and decision-making and have a positive impact on patient care.

Dynamic risk prediction for diabetes using biomarker change measurements

Background

Dynamic risk models, which incorporate disease-free survival and repeated measurements over time, might yield more accurate predictions of future health status compared to static models. The objective of this study was to develop and apply a dynamic prediction model to estimate the risk of developing type 2 diabetes mellitus.

Methods

Both a static prediction model and a dynamic landmark model were used to provide predictions of a 2-year horizon time for diabetes-free survival, updated at 1, 2, and 3 years post-baseline i.e., predicting diabetes-free survival to 2 years and predicting diabetes-free survival to 3 years, 4 years, and 5 years post-baseline, given the patient already survived past 1 year, 2 years, and 3 years post-baseline, respectively. Prediction accuracy was evaluated at each time point using robust non-parametric procedures. Data from 2057 participants of the Diabetes Prevention Program (DPP) study (1027 in metformin arm, 1030 in placebo arm) were analyzed.

Results

The dynamic landmark model demonstrated good prediction accuracy with area under curve (AUC) estimates ranging from 0.645 to 0.752 and Brier Score estimates ranging from 0.088 to 0.135. Relative to a static risk model, the dynamic landmark model did not significantly differ in terms of AUC but had significantly lower (i.e., better) Brier Score estimates for predictions at 1, 2, and 3 years (e.g. 0.167 versus 0.099; difference − 0.068 95% CI − 0.083 to − 0.053, at 3 years in placebo group) post-baseline.

Conclusions

Dynamic prediction models based on longitudinal, repeated risk factor measurements have the potential to improve the accuracy of future health status predictions.

Electronic supplementary material

The online version of this article (10.1186/s12874-019-0812-y) contains supplementary material, which is available to authorized users.

Evaluation and refinement of the PRESTARt tool for identifying 12-14 year olds at high lifetime risk of developing type 2 diabetes compared to a clinicians assessment of risk: a cross-sectional study

BACKGROUND

Traditionally Type 2 Diabetes Mellitus (T2DM) was associated with older age, but is now being increasingly diagnosed in younger populations due to the increasing prevalence of obesity and inactivity. We aimed to evaluate whether a tool developed for community use to identify adolescents at high lifetime risk of developing T2DM agreed with a risk assessment conducted by a clinician using data collected from five European countries. We also assessed whether the tool could be simplified.

METHODS

To evaluate the tool we collected data from 636 adolescents aged 12-14 years from five European countries. Each participant's data were then assessed by two clinicians independently, who judged each participant to be at either low or high risk of developing T2DM in their lifetime. This was used as the gold standard to which the tool was evaluated and refined.

RESULTS

The refined tool categorised adolescents at high risk if they were overweight/obese and had at least one other risk factor (High waist circumference, family history of diabetes, parental obesity, not breast fed, high sugar intake, high screen time, low physical activity and low fruit and vegetable intake). Of those found to be at high risk by the clinicians, 93% were also deemed high risk by the tool. The specificity shows that 67% of those deemed at low risk by the clinicians were also found to be a low risk by the tool.

CONCLUSIONS

We have evaluated a tool for identifying adolescents with risk factors associated with the development of T2DM in the future. Future work to externally validate the tool using prospective data including T2DM incidence is required.

Machine Learning Models in Type 2 Diabetes Risk Prediction: Results from a Cross-sectional Retrospective Study in Chinese Adults

Type 2 diabetes mellitus (T2DM) has become a prevalent health problem in China, especially in urban areas. Early prevention strategies are needed to reduce the associated mortality and morbidity. We applied the combination of rules and different machine learning techniques to assess the risk of development of T2DM in an urban Chinese adult population. A retrospective analysis was performed on 8000 people with non-diabetes and 3845 people with T2DM in Nanjing. Multilayer Perceptron (MLP), AdaBoost (AD), Trees Random Forest (TRF), Support Vector Machine (SVM), and Gradient Tree Boosting (GTB) machine learning techniques with 10 cross validation methods were used with the proposed model for the prediction of the risk of development of T2DM. The performance of these models was evaluated with accuracy, precision, sensitivity, specificity, and area under receiver operating characteristic (ROC) curve (AUC). After comparison, the prediction accuracy of the different five machine models was 0.87, 0.86, 0.86, 0.86 and 0.86 respectively. The combination model using the same voting weight of each component was built on T2DM, which was performed better than individual models. The findings indicate that, combining machine learning models could provide an accurate assessment model for T2DM risk prediction.

The use of rigorous methods was strongly warranted among prognostic prediction models for obstetric care

OBJECTIVE

The objective of the study was to examine methodological characteristics about the design and conduct in prognostic prediction models used for obstetric care.

STUDY DESIGN AND SETTING

We searched PubMed for studies on prognostic prediction models for obstetric care, published in top general medicine or major specialty journals between January 2011 and February 2018. Teams of method-trained investigators independently screened titles and abstracts and collected data using a prespecified, pilot-tested, structured questionnaire.

RESULTS

In total, 91 studies were eligible, of which two were published in top general medicine journals, 20 (22.0%) involved an epidemiologist or statistician, 18 (19.4%) published study protocols, 53 (58.2%) did not include any model validation, 20 (22.0%) did not clearly state the intended timing of use, 23 (25.3%) had no eligibility criteria, 15 (16.5%) did not use clear criteria for ascertaining outcome, and 69 (75.82%) did not apply blinding to outcome assessment. Among those models, 11 (12.1%) included participants fewer than 200 events, 41 (48.8%) had fewer than 100 events, and 19 (24.7%) had fewer than 10 events per variable.

CONCLUSION

The prognostic prediction models have important limitations in design and conduct. Substantial efforts are needed to strengthen the production of reliable prognostic prediction models for obstetric care.

Decision models of prediabetes populations: A systematic review

AIMS

With evidence supporting the use of preventive interventions for prediabetes populations and the use of novel biomarkers to stratify the risk of progression, there is a need to evaluate their cost-effectiveness across jurisdictions. Our aim is to summarize and assess the quality and validity of decision models and model-based economic evaluations of populations with prediabetes, to evaluate their potential use for the assessment of novel prevention strategies and to discuss the knowledge gaps, challenges and opportunities.

MATERIALS AND METHODS

We searched Medline, Embase, EconLit and NHS EED between 2000 and 2018 for studies reporting computer simulation models of the natural history of individuals with prediabetes and/or we used decision models to evaluate the impact of treatment strategies on these populations. Data were extracted following PRISMA guidelines and assessed using modelling checklists. Two reviewers independently assessed 50% of the titles and abstracts to determine whether a full text review was needed. Of these, 10% was assessed by each reviewer to cross-reference the decision to proceed to full review. Using a standardized form and double extraction, each of four reviewers extracted 50% of the identified studies.

RESULTS

A total of 29 published decision models that simulate prediabetes populations were identified. Studies showed large variations in the definition of prediabetes and model structure. The inclusion of complications in prediabetes (n = 8) and type 2 diabetes (n = 17) health states also varied. A minority of studies simulated annual changes in risk factors (glycaemia, HbA1c, blood pressure, BMI, lipids) as individuals progressed in the models (n = 7) and accounted for heterogeneity among individuals with prediabetes (n = 7).

CONCLUSIONS

Current prediabetes decision models have considerable limitations in terms of their quality and validity and do not allow evaluation of stratified strategies using novel biomarkers, highlighting a clear need for more comprehensive prediabetes decision models.

Empirical evidence of the impact of study characteristics on the performance of prediction models: a meta-epidemiological study

OBJECTIVES

To empirically assess the relation between study characteristics and prognostic model performance in external validation studies of multivariable prognostic models.

DESIGN

Meta-epidemiological study.

DATA SOURCES AND STUDY SELECTION

On 16 October 2018, we searched electronic databases for systematic reviews of prognostic models. Reviews from non-overlapping clinical fields were selected if they reported common performance measures (either the concordance (c)-statistic or the ratio of observed over expected number of events (OE ratio)) from 10 or more validations of the same prognostic model.

DATA EXTRACTION AND ANALYSES

Study design features, population characteristics, methods of predictor and outcome assessment, and the aforementioned performance measures were extracted from the included external validation studies. Random effects meta-regression was used to quantify the association between the study characteristics and model performance.

RESULTS

We included 10 systematic reviews, describing a total of 224 external validations, of which 221 reported c-statistics and 124 OE ratios. Associations between study characteristics and model performance were heterogeneous across systematic reviews. C-statistics were most associated with variation in population characteristics, outcome definitions and measurement and predictor substitution. For example, validations with eligibility criteria comparable to the development study were associated with higher c-statistics compared with narrower criteria (difference in logit c-statistic 0.21(95% CI 0.07 to 0.35), similar to an increase from 0.70 to 0.74). Using a case-control design was associated with higher OE ratios, compared with using data from a cohort (difference in log OE ratio 0.97(95% CI 0.38 to 1.55), similar to an increase in OE ratio from 1.00 to 2.63).

CONCLUSIONS

Variation in performance of prognostic models across studies is mainly associated with variation in case-mix, study designs, outcome definitions and measurement methods and predictor substitution. Researchers developing and validating prognostic models should realise the potential influence of these study characteristics on the predictive performance of prognostic models.

Identification of novel population clusters with different susceptibilities to type 2 diabetes and their impact on the prediction of diabetes

Type 2 diabetes is one of the subtypes of diabetes. However, previous studies have revealed its heterogeneous features. Here, we hypothesized that there would be heterogeneity in its development, resulting in higher susceptibility in some populations. We performed risk-factor based clustering (RFC), which is a hierarchical clustering of the population with profiles of five known risk factors for type 2 diabetes (age, gender, body mass index, hypertension, and family history of diabetes). The RFC identified six population clusters with significantly different prevalence rates of type 2 diabetes in the discovery data (N = 10,023), ranging from 0.09 to 0.44 (Chi-square test, P < 0.001). The machine learning method identified six clusters in the validation data (N = 215,083), which also showed the heterogeneity of prevalence between the clusters (P < 0.001). In addition to the prevalence of type 2 diabetes, the clusters showed different clinical features including biochemical profiles and prediction performance with the risk factors. SOur results seem to implicate a heterogeneous mechanism in the development of type 2 diabetes. These results will provide new insights for the development of more precise management strategy for type 2 diabetes.

Evaluating the utility of self-reported questionnaire data to screen for dysglycemia in young adults: Findings from the US National Health and Nutrition Examination Survey

Dysglycemia, including prediabetes and type 2 diabetes, is dangerous and widespread. Yet, the condition is transiently reversible and sequelae preventable, prompting the use of prediction algorithms to quickly assess dysglycemia status through self-reported data. However, as current algorithms have largely been developed in older populations, their application to younger adults is uncertain considering associations between risk factors and dysglycemia vary by age. We sought to identify sex-specific predictors of current dysglycemia among young adults and evaluate their ability to screen for prediabetes and undiagnosed diabetes. We analyzed 2005-2014 data from the National Health and Nutrition Examination Survey for 3251 participants aged 20-39, who completed an oral glucose tolerance test (OGTT), had not been diagnosed with diabetes, and, for females, were not pregnant. Sex-specific stepwise logistic models were fit with predictors identified from univariate analyses. Risk scores were developed using adjusted odds ratios and model performance was assessed using area under the curve (AUC) measures. The OGTT identified 906 (27.9%) and 78 (2.4%) participants with prediabetes or undiagnosed diabetes, respectively. Predictors of dysglycemia status for males were BMI, age, race, and first-degree family history of diabetes, and, in addition to those, education, delivered baby weight, waist circumference, and vigorous physical activity for females. Our male- and female-specific models demonstrated improved validity to assess dysglycemia presence among young adults relative to the widely-used American Diabetes Association test (AUC = 0.69 vs. 0.61; 0.92 vs. 0.71, respectively). Thus, age-specific scoring algorithms employing questionnaire data show promise and are effective in identifying dysglycemia among young adults.

Nomograms for incident risk of post-partum type 2 diabetes in Chinese women with prior gestational diabetes mellitus

OBJECTIVE

Counselling patients with gestational diabetes mellitus (GDM) on their individual risk of post-partum type 2 diabetes (T2D) is challenging. This study aimed to develop nomograms for predicting incident risk of post-partum T2D in women with GDM diagnosed by WHO 1998 criteria.

METHODS

We performed a retrospective cohort study in 1263 Chinese women with GDM, of whom 83 were diagnosed as T2D at 2.3 years post-partum. Multivariate Cox proportional hazards models were used to investigate the independent predictors for post-partum T2D. The results of multivariate analyses were used to formulate nomograms for predicting incident risk of post-partum T2D. The predictive accuracy was evaluated using the area under the receiver operating characteristic curve (AUROC).

RESULTS

On multivariate analysis, independent predictors of post-partum T2DM in women with GDM included family history of diabetes [hazard ratio (HR) and its 95% confidential interval (95% CI): 2.06 (95% CI: 1.32-3.22)], history of pregnancy-induced hypertension [3.11 (95% CI: 1.86-5.21)], pre-pregnancy BMI [1.00, 1.90 (95% CI: 1.14-3.16), and 3.67 (95% CI: 2.03-6.63) for BMI <24, 24-28, and ≥28 kg/m² ], and 2-hour glucose at 26-30 gestational weeks [1.00, 2.84 (95% CI: 1.42-5.69), and 9.42 (95% CI: 4.46-19.90) for 2-hour glucose at 7.8 ~ <8.5, 8.5 ~ <11.1, and ≥11.1 mmol/L). The overall AUROC of nomogram was 82.8% (95% CI: 78.1%-87.5%), with AUROCs of 85.9% (95% CI: 79.7%-92.1%) and 83.2% (95% CI: 77.9%-88.6%) for post-partum 2-year and 3-year risk of T2D, respectively.

CONCLUSIONS

This easy-to-use nomogram, with non-invasive clinical characteristics, can accurately predict the risk of post-partum T2D in women with GDM. It may facilitate risk communication between patients and clinicians.

Untapped potential of multicenter studies: a review of cardiovascular risk prediction models revealed inappropriate analyses and wide variation in reporting

BACKGROUND

Clinical prediction models are often constructed using multicenter databases. Such a data structure poses additional challenges for statistical analysis (clustered data) but offers opportunities for model generalizability to a broad range of centers. The purpose of this study was to describe properties, analysis, and reporting of multicenter studies in the Tufts PACE Clinical Prediction Model Registry and to illustrate consequences of common design and analyses choices.

METHODS

Fifty randomly selected studies that are included in the Tufts registry as multicenter and published after 2000 underwent full-text screening. Simulated examples illustrate some key concepts relevant to multicenter prediction research.

RESULTS

Multicenter studies differed widely in the number of participating centers (range 2 to 5473). Thirty-nine of 50 studies ignored the multicenter nature of data in the statistical analysis. In the others, clustering was resolved by developing the model on only one center, using mixed effects or stratified regression, or by using center-level characteristics as predictors. Twenty-three of 50 studies did not describe the clinical settings or type of centers from which data was obtained. Four of 50 studies discussed neither generalizability nor external validity of the developed model.

CONCLUSIONS

Regression methods and validation strategies tailored to multicenter studies are underutilized. Reporting on generalizability and potential external validity of the model lacks transparency. Hence, multicenter prediction research has untapped potential.

REGISTRATION

This review was not registered.