Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting

An Electronic Health Record-Compatible Model to Predict Personalized Treatment Effects From the Diabetes Prevention Program: A Cross-Evidence Synthesis Approach Using Clinical Trial and Real-World Data

OBJECTIVE

To develop an electronic health record (EHR)-based risk tool that provides point-of-care estimates of diabetes risk to support targeting interventions to patients most likely to benefit.

PATIENTS AND METHODS

A risk prediction model was developed and validated in a large observational database of patients with an index visit date between January 1, 2012, and December 31, 2016, with treatment effect estimates from risk-based reanalysis of clinical trial data. The risk model development cohort included 1.1 million patients with prediabetes from the OptumLabs Data Warehouse (OLDW); the validation cohort included a distinct sample of 1.1 million patients in OLDW. The randomly assigned clinical trial cohort included 3081 people from the Diabetes Prevention Program (DPP) study.

RESULTS

Eleven variables reliably obtainable from the EHR were used to predict diabetes risk. This model validated well in the OLDW (C statistic = 0.76; observed 3-year diabetes rate was 1.8% (95% confidence interval [CI], 1.7 to 1.9) in the lowest-risk quarter and 19.6% (19.4 to 19.8) in the highest-risk quarter). In the DPP, the hazard ratio (HR) for lifestyle modification was constant across all levels of risk (HR, 0.43; 95% CI, 0.35 to 0.53), whereas the HR for metformin was highly risk dependent (HR, 1.1; 95% CI, 0.61 to 2.0 in the lowest-risk quarter vs HR, 0.45; 95% CI, 0.35 to 0.59 in the highest-risk quarter). Fifty-three percent of the benefits of population-wide dissemination of the DPP lifestyle modification and 73% of the benefits of population-wide metformin therapy can be obtained by targeting the highest-risk quarter of patients.

CONCLUSION

The Tufts-Predictive Analytics and Comparative Effectiveness DPP Risk model is an EHR-compatible tool that might support targeted diabetes prevention to more efficiently realize the benefits of the DPP interventions.

Applicability of predictive models for 30-day unplanned hospital readmission risk in paediatrics: a systematic review

OBJECTIVES

To summarise multivariable predictive models for 30-day unplanned hospital readmissions (UHRs) in paediatrics, describe their performance and completeness in reporting, and determine their potential for application in practice.

DESIGN

Systematic review.

DATA SOURCE

CINAHL, Embase and PubMed up to 7 October 2021.

ELIGIBILITY CRITERIA

English or German language studies aiming to develop or validate a multivariable predictive model for 30-day paediatric UHRs related to all-cause, surgical conditions or general medical conditions were included.

DATA EXTRACTION AND SYNTHESIS

Study characteristics, risk factors significant for predicting readmissions and information about performance measures (eg, c-statistic) were extracted. Reporting quality was addressed by the 'Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis' (TRIPOD) adherence form. The study quality was assessed by applying six domains of potential biases. Due to expected heterogeneity among the studies, the data were qualitatively synthesised.

RESULTS

Based on 28 studies, 37 predictive models were identified, which could potentially be used for determining individual 30-day UHR risk in paediatrics. The number of study participants ranged from 190 children to 1.4 million encounters. The two most common significant risk factors were comorbidity and (postoperative) length of stay. 23 models showed a c-statistic above 0.7 and are primarily applicable at discharge. The median TRIPOD adherence of the models was 59% (P₂₅-P₇₅, 55%-69%), ranging from a minimum of 33% to a maximum of 81%. Overall, the quality of many studies was moderate to low in all six domains.

CONCLUSION

Predictive models may be useful in identifying paediatric patients at increased risk of readmission. To support the application of predictive models, more attention should be placed on completeness in reporting, particularly for those items that may be relevant for implementation in practice.

Composite Outcomes in Clinical Prediction Modeling: Are We Trying to Predict Apples and Oranges?

Composite outcomes are widely used in clinical research. Existing literature has considered the pros and cons of composite outcomes in clinical trials, but their extensive use in clinical prediction has received much less attention. Clinical prediction assists decision-making by directing patients with higher risks of adverse outcomes toward interventions that provide the greatest benefits to those at the greatest risk. In this article, we summarize our existing understanding of the advantages and disadvantages of composite outcomes, consider how these relate to clinical prediction, and highlight the problem of key predictors having markedly different associations with individual components of the composite outcome. We suggest that a "composite outcome fallacy" may occur when a clinical prediction model is based on strong associations between key predictors and one component of a composite outcome (such as mortality) and used to direct patients toward intervention when these predictors actually have an inverse association with a more relevant component of the composite outcome (such as the use of a lifesaving intervention). We propose that clinical prediction scores using composite outcomes should report their accuracy for key components of the composite outcome and examine for inconsistencies among predictor variables.

Endorsement of the TRIPOD statement and the reporting of studies developing contrast-induced nephropathy prediction models for the coronary angiography/percutaneous coronary intervention population: a cross-sectional study

OBJECTIVE

Clear and specific reporting of a research paper is essential for its validity and applicability. Some studies have revealed that the reporting of studies based on the clinical prediction models was generally insufficient based on the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) checklist. However, the reporting of studies on contrast-induced nephropathy (CIN) prediction models in the coronary angiography (CAG)/percutaneous coronary intervention (PCI) population has not been thoroughly assessed. Thus, the aim is to evaluate the reporting of the studies on CIN prediction models for the CAG/PCI population using the TRIPOD checklist.

DESIGN

A cross-sectional study.

METHODS

PubMed and Embase were systematically searched from inception to 30 September 2021. Only the studies on the development of CIN prediction models for the CAG/PCI population were included. The data were extracted into a standardised spreadsheet designed in accordance with the 'TRIPOD Adherence Assessment Form'. The overall completeness of reporting of each model and each TRIPOD item were evaluated, and the reporting before and after the publication of the TRIPOD statement was compared. The linear relationship between model performance and TRIPOD adherence was also assessed.

RESULTS

We identified 36 studies that developed CIN prediction models for the CAG/PCI population. Median TRIPOD checklist adherence was 60% (34%-77%), and no significant improvement was found since the publication of the TRIPOD checklist (p=0.770). There was a significant difference in adherence to individual TRIPOD items, ranging from 0% to 100%. Moreover, most studies did not specify critical information within the Methods section. Only 5 studies (14%) explained how they arrived at the study size, and only 13 studies (36%) described how to handle missing data. In the Statistical analysis section, how the continuous predictors were modelled, the cut-points of categorical or categorised predictors, and the methods to choose the cut-points were only reported in 7 (19%), 6 (17%) and 1 (3%) of the studies, respectively. Nevertheless, no relationship was found between model performance and TRIPOD adherence in both the development and validation datasets (r=-0.260 and r=-0.069, respectively).

CONCLUSIONS

The reporting of CIN prediction models for the CAG/PCI population still needs to be improved based on the TRIPOD checklist. In order to promote further external validation and clinical application of the prediction models, more information should be provided in future studies.

Completeness of reporting of clinical prediction models developed using supervised machine learning: a systematic review

BACKGROUND

While many studies have consistently found incomplete reporting of regression-based prediction model studies, evidence is lacking for machine learning-based prediction model studies. We aim to systematically review the adherence of Machine Learning (ML)-based prediction model studies to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Statement.

METHODS

We included articles reporting on development or external validation of a multivariable prediction model (either diagnostic or prognostic) developed using supervised ML for individualized predictions across all medical fields. We searched PubMed from 1 January 2018 to 31 December 2019. Data extraction was performed using the 22-item checklist for reporting of prediction model studies ( www.TRIPOD-statement.org ). We measured the overall adherence per article and per TRIPOD item.

RESULTS

Our search identified 24,814 articles, of which 152 articles were included: 94 (61.8%) prognostic and 58 (38.2%) diagnostic prediction model studies. Overall, articles adhered to a median of 38.7% (IQR 31.0-46.4%) of TRIPOD items. No article fully adhered to complete reporting of the abstract and very few reported the flow of participants (3.9%, 95% CI 1.8 to 8.3), appropriate title (4.6%, 95% CI 2.2 to 9.2), blinding of predictors (4.6%, 95% CI 2.2 to 9.2), model specification (5.2%, 95% CI 2.4 to 10.8), and model's predictive performance (5.9%, 95% CI 3.1 to 10.9). There was often complete reporting of source of data (98.0%, 95% CI 94.4 to 99.3) and interpretation of the results (94.7%, 95% CI 90.0 to 97.3).

CONCLUSION

Similar to prediction model studies developed using conventional regression-based techniques, the completeness of reporting is poor. Essential information to decide to use the model (i.e. model specification and its performance) is rarely reported. However, some items and sub-items of TRIPOD might be less suitable for ML-based prediction model studies and thus, TRIPOD requires extensions. Overall, there is an urgent need to improve the reporting quality and usability of research to avoid research waste.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO, CRD42019161764.

Relationships between fasting glucose levels, lifestyle factors, and metabolic parameters in Korean adults without diagnosis of diabetes mellitus

BACKGROUND

We investigated the associations between fasting glucose level ranges with lifestyle factors and metabolic profiles among adults without previous diagnosis of diabetes.

METHODS

We analyzed 13 625 adults without previous diagnosis of diabetes from the Korea National Health and Nutrition Examination Survey during 2016 to 2018. We categorized fasting glucose levels (mg/dl) as follows: <90, 90 to 99, 100 to 109, 110 to 124, and ≥125. We evaluated trends in the proportions of individuals with obesity, abdominal, obesity, current smoking, heavy drinking, and low physical activity according to these categories, and the odds for uncontrolled blood pressure (BP), low-density lipoprotein cholesterol (LDL-C), triglycerides (TG), and high-density lipoprotein cholesterol (HDL-C) for each fasting glucose level compared to a fasting glucose level of <90 mg/dl.

RESULTS

The proportions of individuals with obesity, abdominal obesity, and heavy drinking increased according to fasting glucose level (P for trend <.05). The odds for BP ≥140/90 mm Hg, TG ≥150 mg/dl, HDL-C < 40 mg/dl in men, and HDL-C < 50 mg/dl in women increased with increasing fasting glucose levels; however, the odds for LDL-C ≥ 130 mg/dl increased with increasing fasting glucose levels only in women. The increases in odds for uncontrolled BP and lipid profiles were mostly observed for fasting glucose levels ≥90 mg/dl.

CONCLUSIONS

Efforts are needed to prevent increased fasting glucose levels, as higher levels, even within normal range, were associated with poor metabolic profiles.

A Graphical Toolkit for Longitudinal Dataset Maintenance and Predictive Model Training in Health Care

BACKGROUND

Predictive analytic models, including machine learning (ML) models, are increasingly integrated into electronic health record (EHR)-based decision support tools for clinicians. These models have the potential to improve care, but are challenging to internally validate, implement, and maintain over the long term. Principles of ML operations (MLOps) may inform development of infrastructure to support the entire ML lifecycle, from feature selection to long-term model deployment and retraining.

OBJECTIVES

This study aimed to present the conceptual prototypes for a novel predictive model management system and to evaluate the acceptability of the system among three groups of end users.

METHODS

Based on principles of user-centered software design, human-computer interaction, and ethical design, we created graphical prototypes of a web-based MLOps interface to support the construction, deployment, and maintenance of models using EHR data. To assess the acceptability of the interface, we conducted semistructured user interviews with three groups of users (health informaticians, clinical and data stakeholders, chief information officers) and evaluated preliminary usability using the System Usability Scale (SUS). We subsequently revised prototypes based on user input and developed user case studies.

RESULTS

Our prototypes include design frameworks for feature selection, model training, deployment, long-term maintenance, visualization over time, and cross-functional collaboration. Users were able to complete 71% of prompted tasks without assistance. The average SUS score of the initial prototype was 75.8 out of 100, translating to a percentile range of 70 to 79, a letter grade of B, and an adjective rating of "good." We reviewed persona-based case studies that illustrate functionalities of this novel prototype.

CONCLUSION

The initial graphical prototypes of this MLOps system are preliminarily usable and demonstrate an unmet need within the clinical informatics landscape.

Use and performance of machine learning models for type 2 diabetes prediction in clinical and community care settings: Protocol for a systematic review and meta-analysis of predictive modeling studies

Objective

Machine learning involves the use of algorithms without explicit instructions. Of late, machine learning models have been widely applied for the prediction of type 2 diabetes. However, no evidence synthesis of the performance of these prediction models of type 2 diabetes is available. We aim to identify machine learning prediction models for type 2 diabetes in clinical and community care settings and determine their predictive performance.

Methods

The systematic review of English language machine learning predictive modeling studies in 12 databases will be conducted. Studies predicting type 2 diabetes in predefined clinical or community settings are eligible. Standard CHARMS and TRIPOD guidelines will guide data extraction. Methodological quality will be assessed using a predefined risk of bias assessment tool. The extent of validation will be categorized by Reilly–Evans levels. Primary outcomes include model performance metrics of discrimination ability, calibration, and classification accuracy. Secondary outcomes include candidate predictors, algorithms used, level of validation, and intended use of models. The random-effects meta-analysis of c-indices will be performed to evaluate discrimination abilities. The c-indices will be pooled per prediction model, per model type, and per algorithm. Publication bias will be assessed through funnel plots and regression tests. Sensitivity analysis will be conducted to estimate the effects of study quality and missing data on primary outcome. The sources of heterogeneity will be assessed through meta-regression. Subgroup analyses will be performed for primary outcomes.

Ethics and dissemination

No ethics approval is required, as no primary or personal data are collected. Findings will be disseminated through scientific sessions and peer-reviewed journals.

PROSPERO registration number

CRD42019130886

Characterizing screening strategies for type 2 diabetes in high-risk ethnic communities: a scoping review protocol

OBJECTIVE

This review aims to identify the characteristics of screening strategies for type 2 diabetes to determine the most pragmatic approach to improve relevance to high-risk community groups from ethnically diverse backgrounds.

INTRODUCTION

Type 2 diabetes is increasingly contributing to the global burden of disease and is more common in some community groups. Although screening underpins the success of primary prevention programs for type 2 diabetes, screening of high-risk community groups from ethnically diverse backgrounds require different screening protocols and can be challenging. These strategies have never been systematically scoped.

INCLUSION CRITERIA

This scoping review will consider screening strategies for type 2 diabetes that target high-risk ethnic community groups. Studies with adults older than 18 years will be considered for inclusion. Screening strategies may include, but are not limited to, risk-assessment questionnaires, blood tests, or both, using an opportunistic approach involving general practices or a targeted approach toward high-risk community groups from ethnically diverse backgrounds. Experimental and observational quantitative studies and mixed methods studies will be included.

METHODS

MEDLINE, CINAHL, PsycINFO, Informit, ProQuest, Web of Science, and Scopus will be searched. Studies will be screened for inclusion by two independent reviewers, and data will be extracted using the Reach, Effectiveness, Adoption, Implementation, and Maintenance framework. Results will be summarized in tables accompanied by narrative text.

Systematic review of studies examining contribution of oral health variables to risk prediction models for undiagnosed Type 2 diabetes and prediabetes

Objective

To conduct systematic review applying “preferred reporting items for systematic reviews and meta‐analyses statement” and “prediction model risk of assessment bias tool” to studies examining the performance of predictive models incorporating oral health‐related variables as candidate predictors for projecting undiagnosed diabetes mellitus (Type 2)/prediabetes risk.

Materials and Methods

Literature searches undertaken in PubMed, Web of Science, and Gray literature identified eligible studies published between January 1, 1980 and July 31, 2018. Systematically reviewed studies met inclusion criteria if studies applied multivariable regression modeling or informatics approaches to risk prediction for undiagnosed diabetes/prediabetes, and included dental/oral health‐related variables modeled either independently, or in combination with other risk variables.

Results

Eligibility for systematic review was determined for seven of the 71 studies screened. Nineteen dental/oral health‐related variables were examined across studies. “Periodontal pocket depth” and/or “missing teeth” were oral health variables consistently retained as predictive variables in models across all systematically reviewed studies. Strong performance metrics were reported for derived models by all systematically reviewed studies. The predictive power of independently modeled oral health variables was marginally amplified when modeled with point‐of‐care biological glycemic measures in dental settings. Meta‐analysis was precluded due to high inter‐study variability in study design and population diversity.

Conclusions

Predictive modeling consistently supported “periodontal measures” and “missing teeth” as candidate variables for predicting undiagnosed diabetes/prediabetes. Validation of predictive risk modeling for undiagnosed diabetes/prediabetes across diverse populations will test the feasibility of translating such models into clinical practice settings as noninvasive screening tools for identifying at‐risk individuals following demonstration of model validity within the defined population.

Obesity Measures as Predictors of Type 2 Diabetes and Cardiovascular Diseases among the Jordanian Population: A Cross-Sectional Study

Obesity is strongly associated with cardiovascular diseases (CVD) and type 2 diabetes (T2D). This study aimed to use obesity measures, body mass index (BMI) and waist circumference (WC) to predict the CVD and T2D risk and to determine the best predictor of these diseases among Jordanian adults. A cross-sectional study was conducted at the governmental and military hospitals across Jordan. The study participants were healthy or previously diagnosed with CVD or T2D. The continuous variables were compared using ANOVA, and the categorical variables were compared using the X2 test. The multivariate logistic regression was used to predict CVD and T2D risk through their association with BMI and WC. The final sample consisted of 6000 Jordanian adults with a mean age of 41.5 ± 14.7 years, 73.6% females. The BMI (OR = 1.7, CI: 1.30–2.30, p < 0.001) was associated with a higher risk of T2D compared to WC (OR = 1.3, CI: 1.04–1.52, p = 0.016). However, our results showed that BMI was not associated with CVD risk, while the WC was significantly and positively associated with CVD risk (OR = 1.9, CI: 1.47–2.47, p < 0.001). In conclusion, an elevated BMI predicts a higher risk of T2D, while WC is more efficient in predicting CVD risk. Our results can be used to construct a population-specific intervention to reduce the risk of CVD and T2D among adults in Jordan and other countries with similar backgrounds.

Transforming Anesthesia Data Into the Observational Medical Outcomes Partnership Common Data Model: Development and Usability Study

BACKGROUND

Electronic health records (EHRs, such as those created by an anesthesia management system) generate a large amount of data that can notably be reused for clinical audits and scientific research. The sharing of these data and tools is generally affected by the lack of system interoperability. To overcome these issues, Observational Health Data Sciences and Informatics (OHDSI) developed the Observational Medical Outcomes Partnership (OMOP) common data model (CDM) to standardize EHR data and promote large-scale observational and longitudinal research. Anesthesia data have not previously been mapped into the OMOP CDM.

OBJECTIVE

The primary objective was to transform anesthesia data into the OMOP CDM. The secondary objective was to provide vocabularies, queries, and dashboards that might promote the exploitation and sharing of anesthesia data through the CDM.

METHODS

Using our local anesthesia data warehouse, a group of 5 experts from 5 different medical centers identified local concepts related to anesthesia. The concepts were then matched with standard concepts in the OHDSI vocabularies. We performed structural mapping between the design of our local anesthesia data warehouse and the OMOP CDM tables and fields. To validate the implementation of anesthesia data into the OMOP CDM, we developed a set of queries and dashboards.

RESULTS

We identified 522 concepts related to anesthesia care. They were classified as demographics, units, measurements, operating room steps, drugs, periods of interest, and features. After semantic mapping, 353 (67.7%) of these anesthesia concepts were mapped to OHDSI concepts. Further, 169 (32.3%) concepts related to periods and features were added to the OHDSI vocabularies. Then, 8 OMOP CDM tables were implemented with anesthesia data and 2 new tables (EPISODE and FEATURE) were added to store secondarily computed data. We integrated data from 5,72,609 operations and provided the code for a set of 8 queries and 4 dashboards related to anesthesia care.

CONCLUSIONS

Generic data concerning demographics, drugs, units, measurements, and operating room steps were already available in OHDSI vocabularies. However, most of the intraoperative concepts (the duration of specific steps, an episode of hypotension, etc) were not present in OHDSI vocabularies. The OMOP mapping provided here enables anesthesia data reuse.

Handgrip strength as a simple measure for screening prediabetes and type 2 diabetes mellitus risk among adults in Malawi: A cross-sectional study

OBJECTIVE

Handgrip strength, a simple measure of muscle strength, has been reported as a predictor of both prediabetes and type 2 diabetes mellitus (T2DM) and has been suggested for screening prediabetes and T2DM risk. This study examined the relationship of handgrip strength with prediabetes and T2DM among rural- and urban-dwelling adults in Malawi.

METHODS

This was a cross-sectional study nested in a follow-up study of prediabetic and prehypertensive individuals identified during an extensive noncommunicable disease survey in Malawi. A total of 261 participants (women: 64%) were recruited. Univariate and multivariate binary logistic regression analyses were performed to examine the association of prediabetes and T2DM with relative handgrip strength.

RESULTS

The mean (SD) age of the participants was 49.7 (13.6) years, and 54.0% were between the ages of 40 and 59 years. The mean (SD) absolute handgrip strength and relative handgrip strength were 28.8 (7.3) kg and 1.16 (0.40) kg/BMI, respectively, and the mean relative handgrip strength differed significantly (p < 0.001) by T2DM status. In unadjusted model, the odds ratio (OR) of prediabetes and T2DM per unit increase in relative handgrip strength was 0.12 [95% CI; 0.04-0.33]. The result remained significant after adjusting for age (continuous), sex, place of study, hypertension, dyslipidaemia and level of education (aOR [95% CI]; 0.19 [0.03-0.95]).

CONCLUSIONS

The findings suggest that handgrip strength could be a relatively inexpensive, noninvasive measure for contributing to risk scores to identify high-risk individuals for screening diabetes in SSA.

Reporting of prognostic clinical prediction models based on machine learning methods in oncology needs to be improved

OBJECTIVE

Evaluate the completeness of reporting of prognostic prediction models developed using machine learning methods in the field of oncology.

STUDY DESIGN AND SETTING

We conducted a systematic review, searching the MEDLINE and Embase databases between 01/01/2019 and 05/09/2019, for non-imaging studies developing a prognostic clinical prediction model using machine learning methods (as defined by primary study authors) in oncology. We used the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement to assess the reporting quality of included publications. We described overall reporting adherence of included publications and by each section of TRIPOD.

RESULTS

Sixty-two publications met the inclusion criteria. 48 were development studies and 14 were development with validation studies. 152 models were developed across all publications. Median adherence to TRIPOD reporting items was 41% [range: 10%-67%] and at least 50% adherence was found in 19% (n=12/62) of publications. Adherence was lower in development only studies (median: 38% [range: 10%-67%]); and higher in development with validation studies (median: 49% [range: 33%-59%]).

CONCLUSION

Reporting of clinical prediction models using machine learning in oncology is poor and needs urgent improvement, so readers and stakeholders can appraise the study methods, understand study findings, and reduce research waste.

Dynamic prediction models improved the risk classification of type 2 diabetes compared with classical static models

OBJECTIVE

Dynamic prediction models use the repeated measurements of predictors to estimate coefficients that link the longitudinal predictors to a static model (i.e. Cox regression). This study aims to develop and validate a dynamic prediction for incident type 2 diabetes (T2DM) as the outcome.

STUDY DESIGN AND SETTING

Data from the Tehran lipid and glucose study was used to develop (n = 5291 individuals; phases 1 to 3) and validate (n = 3147 individuals; phases 3 to 6) the dynamic prediction model among individuals aged ≥ 20 years. We used repeated measurements of fasting plasma glucose (FPG) or waist circumference (WC) in the framework of the joint modeling (JM) of longitudinal and time-to-event analysis.

RESULTS

Compared with the Cox which used just baseline data, JM showed the same discrimination, better calibration, and higher clinical usefulness (i.e. with a net benefit considering both true and false positive decisions); all were shown with repeated measurements of FPG/WC. Additionally, in our study, the dynamic models improve the risk reclassification (net reclassification index 33% for FPG and 24% for WC model).

CONCLUSION

Dynamic prediction models, compared with the static one could yield significant improvements in the prediction of T2DM. The complexity of the dynamic models could be addressed by using decision support systems.

FTO-rs9939609 Polymorphism is a Predictor of Future Type 2 Diabetes: A Population-Based Prospective Study

The study aimed to evaluate the contribution of the FTO A/T polymorphism (rs9939609) to the prediction of the future type 2 diabetes (T2D). A population-based prospective study included 1443 nondiabetic subjects at baseline, and they were examined for developing T2D after 5-year follow-up. Cox proportional hazards model was used to evaluate the hazard ratio (HR) of rs9939609 to the future T2D in the models adjusted for the confounding factors including socio-economic status, lifestyle factors (smoking and drinking history, sporting habits, and leisure time), and clinical patterns (obese status, blood pressures, and dyslipidemia) at baseline. The area under receiver operating characteristic curve (AUC) was used to measure the power to predict individuals with T2D. The FTO-rs9939609 polymorphism was a significant predictor of future T2D in the model unadjusted, and it remained significant in the final model after adjustment for the confounding factors, showing an additive effect of the A-allele (HR = 1.35, 95% CI = 1.02–1.78, P = 0.036, AUC = 0.676). For normoglycemic subjects at baseline, the similar final adjusted model reported the increased HR per A-allele (HR = 1.50, 95% CI = 1.09–2.07, P = 0.012, AUC = 0.697). Five-year changes in BMI, waist circumference, and systolic blood pressure did not remove the contribution of rs9939609 to increased HR of T2D. The population attributable risk for risk genotype was 13.6%. In conclusion, the study indicates that the FTO-rs9939609 polymorphism is an important genetic predictor for future T2D in Vietnamese population.

Availability and reporting quality of external validations of machine-learning prediction models with orthopedic surgical outcomes: a systematic review

Background and purpose - External validation of machine learning (ML) prediction models is an essential step before clinical application. We assessed the proportion, performance, and transparent reporting of externally validated ML prediction models in orthopedic surgery, using the Transparent Reporting for Individual Prognosis or Diagnosis (TRIPOD) guidelines.Material and methods - We performed a systematic search using synonyms for every orthopedic specialty, ML, and external validation. The proportion was determined by using 59 ML prediction models with only internal validation in orthopedic surgical outcome published up until June 18, 2020, previously identified by our group. Model performance was evaluated using discrimination, calibration, and decision-curve analysis. The TRIPOD guidelines assessed transparent reporting.Results - We included 18 studies externally validating 10 different ML prediction models of the 59 available ML models after screening 4,682 studies. All external validations identified in this review retained good discrimination. Other key performance measures were provided in only 3 studies, rendering overall performance evaluation difficult. The overall median TRIPOD completeness was 61% (IQR 43-89), with 6 items being reported in less than 4/18 of the studies.Interpretation - Most current predictive ML models are not externally validated. The 18 available external validation studies were characterized by incomplete reporting of performance measures, limiting a transparent examination of model performance. Further prospective studies are needed to validate or refute the myriad of predictive ML models in orthopedics while adhering to existing guidelines. This ensures clinicians can take full advantage of validated and clinically implementable ML decision tools.

A critical appraisal of the prognostic predictive models for patients with sepsis: Which model can be applied in clinical practice?

BACKGROUND

Sepsis is associated with high mortality and predictive models can help in clinical decision-making. The objective of this study was to carry out a systematic review of these models.

METHODS

In 2019, we conducted a systematic review in MEDLINE and EMBASE (CDR42018111121:PROSPERO) of articles that developed predictive models for mortality in septic patients (inclusion criteria). We followed the CHARMS recommendations (Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies), extracting the information from its 11 domains (Source of data, Participants, etc). We determined the risk of bias and applicability (participants, outcome, predictors and analysis) through PROBAST (Prediction model Risk Of Bias ASsessment Tool).

RESULTS

A total of 14 studies were included. In the CHARMS extraction, the models found showed great variability in its 11 domains. Regarding the PROBAST checklist, only one article had an unclear risk of bias as it did not indicate how missing data were handled while the others all had a high risk of bias. This was mainly due to the statistical analysis (inadequate sample size, handling of continuous predictors, missing data and selection of predictors), since 13 studies had a high risk of bias. Applicability was satisfactory in six articles. Most of the models integrate predictors from routine clinical practice. Discrimination and calibration were assessed for almost all the models, with the area under the ROC curve ranging from 0.59 to 0.955 and no lack of calibration. Only three models were externally validated and their maximum discrimination values in the derivation were from 0.712 and 0.84. One of them (Osborn) had undergone multiple validation studies.

DISCUSSION

Despite most of the studies showing a high risk of bias, we very cautiously recommend applying the Osborn model, as this has been externally validated various times.

Using electronic health records to develop and validate a machine-learning tool to predict type 2 diabetes outcomes: a study protocol

INTRODUCTION

Type 2 diabetes mellitus (T2DM) is a major cause of blindness, kidney failure, myocardial infarction, stroke and lower limb amputation. We are still unable, however, to accurately predict or identify which patients are at a higher risk of deterioration. Most risk stratification tools do not account for novel factors such as sociodemographic determinants, self-management ability or access to healthcare. Additionally, most tools are based in clinical trials, with limited external generalisability.

OBJECTIVE

The aim of this work is to design and validate a machine learning-based tool to identify patients with T2DM at high risk of clinical deterioration, based on a comprehensive set of patient-level characteristics retrieved from a population health linked dataset.

SAMPLE AND DESIGN

Retrospective cohort study of patients with diagnosis of T2DM on 1 January 2015, with a 5-year follow-up. Anonymised electronic healthcare records from the Whole System Integrated Care (WSIC) database will be used.

PRELIMINARY OUTCOMES

Outcome variables of clinical deterioration will include retinopathy, chronic renal disease, myocardial infarction, stroke, peripheral arterial disease or death. Predictor variables will include sociodemographic and geographic data, patients' ability to self-manage disease, clinical and metabolic parameters and healthcare service usage. Prognostic models will be defined using multidependence Bayesian networks. The derivation cohort, comprising 80% of the patients, will be used to define the prognostic models. Model parameters will be internally validated by comparing the area under the receiver operating characteristic curve in the derivation cohort with those calculated from a leave-one-out and a 10 times twofold cross-validation.

ETHICS AND DISSEMINATION

The study has received approvals from the Information Governance Committee at the WSIC. Results will be made available to people with T2DM, their caregivers, the funders, diabetes care societies and other researchers.

A new pipeline for structural characterization and classification of RNA-Seq microbiome data

BACKGROUND

High-throughput sequencing enables the analysis of the composition of numerous biological systems, such as microbial communities. The identification of dependencies within these systems requires the analysis and assimilation of the underlying interaction patterns between all the variables that make up that system. However, this task poses a challenge when considering the compositional nature of the data coming from DNA-sequencing experiments because traditional interaction metrics (e.g., correlation) produce unreliable results when analyzing relative fractions instead of absolute abundances. The compositionality-associated challenges extend to the classification task, as it usually involves the characterization of the interactions between the principal descriptive variables of the datasets. The classification of new samples/patients into binary categories corresponding to dissimilar biological settings or phenotypes (e.g., control and cases) could help researchers in the development of treatments/drugs.

RESULTS

Here, we develop and exemplify a new approach, applicable to compositional data, for the classification of new samples into two groups with different biological settings. We propose a new metric to characterize and quantify the overall correlation structure deviation between these groups and a technique for dimensionality reduction to facilitate graphical representation. We conduct simulation experiments with synthetic data to assess the proposed method's classification accuracy. Moreover, we illustrate the performance of the proposed approach using Operational Taxonomic Unit (OTU) count tables obtained through 16S rRNA gene sequencing data from two microbiota experiments. Also, compare our method's performance with that of two state-of-the-art methods.

CONCLUSIONS

Simulation experiments show that our method achieves a classification accuracy equal to or greater than 98% when using synthetic data. Finally, our method outperforms the other classification methods with real datasets from gene sequencing experiments.

Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence

INTRODUCTION

The Transparent Reporting of a multivariable prediction model of Individual Prognosis Or Diagnosis (TRIPOD) statement and the Prediction model Risk Of Bias ASsessment Tool (PROBAST) were both published to improve the reporting and critical appraisal of prediction model studies for diagnosis and prognosis. This paper describes the processes and methods that will be used to develop an extension to the TRIPOD statement (TRIPOD-artificial intelligence, AI) and the PROBAST (PROBAST-AI) tool for prediction model studies that applied machine learning techniques.

METHODS AND ANALYSIS

TRIPOD-AI and PROBAST-AI will be developed following published guidance from the EQUATOR Network, and will comprise five stages. Stage 1 will comprise two systematic reviews (across all medical fields and specifically in oncology) to examine the quality of reporting in published machine-learning-based prediction model studies. In stage 2, we will consult a diverse group of key stakeholders using a Delphi process to identify items to be considered for inclusion in TRIPOD-AI and PROBAST-AI. Stage 3 will be virtual consensus meetings to consolidate and prioritise key items to be included in TRIPOD-AI and PROBAST-AI. Stage 4 will involve developing the TRIPOD-AI checklist and the PROBAST-AI tool, and writing the accompanying explanation and elaboration papers. In the final stage, stage 5, we will disseminate TRIPOD-AI and PROBAST-AI via journals, conferences, blogs, websites (including TRIPOD, PROBAST and EQUATOR Network) and social media. TRIPOD-AI will provide researchers working on prediction model studies based on machine learning with a reporting guideline that can help them report key details that readers need to evaluate the study quality and interpret its findings, potentially reducing research waste. We anticipate PROBAST-AI will help researchers, clinicians, systematic reviewers and policymakers critically appraise the design, conduct and analysis of machine learning based prediction model studies, with a robust standardised tool for bias evaluation.

ETHICS AND DISSEMINATION

Ethical approval has been granted by the Central University Research Ethics Committee, University of Oxford on 10-December-2020 (R73034/RE001). Findings from this study will be disseminated through peer-review publications.

PROSPERO REGISTRATION NUMBER

CRD42019140361 and CRD42019161764.

Predicting Type 2 Diabetes Using Logistic Regression and Machine Learning Approaches

Diabetes mellitus is one of the most common human diseases worldwide and may cause several health-related complications. It is responsible for considerable morbidity, mortality, and economic loss. A timely diagnosis and prediction of this disease could provide patients with an opportunity to take the appropriate preventive and treatment strategies. To improve the understanding of risk factors, we predict type 2 diabetes for Pima Indian women utilizing a logistic regression model and decision tree—a machine learning algorithm. Our analysis finds five main predictors of type 2 diabetes: glucose, pregnancy, body mass index (BMI), diabetes pedigree function, and age. We further explore a classification tree to complement and validate our analysis. The six-fold classification tree indicates glucose, BMI, and age are important factors, while the ten-node tree implies glucose, BMI, pregnancy, diabetes pedigree function, and age as the significant predictors. Our preferred specification yields a prediction accuracy of 78.26% and a cross-validation error rate of 21.74%. We argue that our model can be applied to make a reasonable prediction of type 2 diabetes, and could potentially be used to complement existing preventive measures to curb the incidence of diabetes and reduce associated costs.

Comparing non-invasive diabetes risk scores for detecting patients in clinical practice: a cross-sectional validation study

Background: Type 2 diabetes (T2DM) is a significant cause of morbidity and mortality, thus early identification is of paramount importance. A high proportion of T2DM cases are undiagnosed highlighting the importance of effective detection methods such as non-invasive diabetes risk scores (DRSs). Thus far, no DRS has been validated in an Irish population. Therefore, the aim of this study was to compare the ability of nine DRSs to detect T2DM cases in an Irish population. Methods: This was a cross-sectional study of 1,990 men and women aged 46–73 years. Data on DRS components were collected from questionnaires and clinical examinations. T2DM was determined according to a fasting plasma glucose level ≥7.0 mmol/l or a glycated haemoglobin A1c level ≥6.5% (≥48 mmol/mol). Receiver operating characteristic curve analysis assessed the ability of DRSs and their components to discriminate T2DM cases. Results: Among the examined scores, area under the curve (AUC) values ranged from 0.71–0.78, with the Cambridge Diabetes Risk Score (AUC=0.78, 95% CI: 0.75–0.82), Leicester Diabetes Risk Score (AUC=0.78, 95% CI: 0.75–0.82), Rotterdam Predictive Model 2 (AUC=0.78, 95% CI: 0.74–0.82) and the U.S. Diabetes Risk Score (AUC=0.78, 95% CI: 0.74–0.81) demonstrating the largest AUC values as continuous variables and at optimal cut-offs. Regarding individual DRS components, anthropometric measures displayed the largest AUC values. Conclusions: The best performing DRSs were broadly similar in terms of their components; all incorporated variables for age, sex, BMI, hypertension and family diabetes history. The Cambridge Diabetes Risk Score, had the largest AUC value at an optimal cut-off, can be easily accessed online for use in a clinical setting and may be the most appropriate and cost-effective method for case-finding in an Irish population.

Prediction Models for Type 2 Diabetes Risk in the General Population: A Systematic Review of Observational Studies

OBJECTIVES

This study aimed to provide an overview of prediction models of undiagnosed type 2 diabetes mellitus (U-T2DM) or the incident T2DM (I-T2DM) using the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) checklist and the prediction model risk of the bias assessment tool (PROBAST).

DATA SOURCES

Both PUBMED and EMBASE databases were searched to guarantee adequate and efficient coverage.

STUDY SELECTION

Articles published between December 2011 and October 2019 were considered.

DATA EXTRACTION

For each article, information on model development requirements, discrimination measures, calibration, overall performance, clinical usefulness, overfitting, and risk of bias (ROB) was reported.

RESULTS

The median (interquartile range; IQR) number of the 46 study populations for model development was 5711 (1971 - 27426) and 2457 (2060 - 6995) individuals for I-T2DM and U-T2DM, respectively. The most common reported predictors were age and body mass index, and only the Qrisk-2017 study included social factors (e.g., Townsend score). Univariable analysis was reported in 46% of the studies, and the variable selection procedure was not clear in 17.4% of them. Moreover, internal and external validation was reported in 43% the studies, while over 63% of them reported calibration. The median (IQR) of AUC for I-T2DM models was 0.78 (0.74 - 0.82); the corresponding value for studies derived before October 2011 was 0.80 (0.77 - 0.83). The highest discrimination index was reported for Qrisk-2017 with C-statistics of 0.89 for women and 0.87 for men. Low ROB for I-T2DM and U-T2DM was assessed at 18% and 41%, respectively.

CONCLUSIONS

Among prediction models, an intermediate to poor quality was reassessed in several aspects of model development and validation. Generally, despite its new risk factors or new methodological aspects, the newly developed model did not increase our capability in screening/predicting T2DM, mainly in the analysis part. It was due to the lack of external validation of the prediction models.

Accounting for Heterogeneity in Resource Allocation Decisions: Methods and Practice in UK Cancer Technology Appraisals

OBJECTIVES

The availability of novel, more efficacious and expensive cancer therapies is increasing, resulting in significant treatment effect heterogeneity and complicated treatment and disease pathways. The aim of this study is to review the extent to which UK cancer technology appraisals (TAs) consider the impact of patient and treatment effect heterogeneity.

METHODS

A systematic search of National Institute for Health and Care Excellence TAs of colorectal, lung and ovarian cancer was undertaken for the period up to April 2020. For each TA, the pivotal clinical studies and economic evaluations were reviewed for considerations of patient and treatment effect heterogeneity. The study critically reviews the use of subgroup analysis and real-world translation in economic evaluations, alongside specific attributes of the economic modeling framework.

RESULTS

The search identified 49 TAs including 49 economic models. In total, 804 subgroup analyses were reported across 69 clinical studies. The most common stratification factors were age, gender, and Eastern Cooperative Oncology Group performance score, with 15% (119 of 804) of analyses demonstrating significantly different clinical outcomes to the main population; economic subgroup analyses were undertaken in only 17 TAs. All economic models were cohort-level with the majority described as partitioned survival models (39) or Markov/semi-Markov models. The impact of real-world heterogeneity on disease progression estimates was only explored in 2 models.

CONCLUSION

The ability of current modeling approaches to capture patient and treatment effect heterogeneity is constrained by their limited flexibility and simplistic nature. This study highlights a need for the use of more sophisticated modeling methods that enable greater consideration of real-world heterogeneity.

A clinical diabetes risk prediction model for prediabetic women with prior gestational diabetes

Introduction

Without treatment, prediabetic women with a history of gestational diabetes mellitus (GDM) are at greater risk for developing type 2 diabetes compared with women without a history of GDM. Both intensive lifestyle intervention and metformin can reduce risk. To predict risk and treatment response, we developed a risk prediction model specifically for women with prior GDM.

Methods

The Diabetes Prevention Program was a randomized controlled trial to evaluate the effectiveness of intensive lifestyle intervention, metformin (850mg twice daily), and placebo in preventing diabetes. Data from the Diabetes Prevention Program (DPP) was used to conduct a secondary analysis to evaluate 11 baseline clinical variables of 317 women with prediabetes and a self-reported history of GDM to develop a 3-year diabetes risk prediction model using Cox proportional hazards regression. Reduced models were explored and compared with the main model.

Results

Within three years, 82 (25.9%) women developed diabetes. In our parsimonious model using 4 of 11 clinical variables, higher fasting glucose and hemoglobin A1C were each associated with greater risk for diabetes (each hazard ratio approximately 1.4), and there was an interaction between treatment arm and BMI suggesting that metformin was more effective relative to no treatment for BMI ≥ 35kg/m² than BMI < 30kg/m². The model had fair discrimination (bias corrected C index = 0.68) and was not significantly different from our main model using 11 clinical variables. The estimated incidence of diabetes without treatment was 37.4%, compared to 20.0% with intensive lifestyle intervention or metformin treatment for women with a prior GDM.

Conclusions

A clinical prediction model was developed for individualized decision making for prediabetes treatment in women with prior GDM.

Large-scale validation of the prediction model risk of bias assessment Tool (PROBAST) using a short form: high risk of bias models show poorer discrimination

OBJECTIVE

To assess whether the Prediction model Risk Of Bias ASsessment Tool (PROBAST) and a shorter version of this tool can identify clinical prediction models (CPMs) that perform poorly at external validation.

STUDY DESIGN AND SETTING

We evaluated risk of bias (ROB) on 102 CPMs from the Tufts CPM Registry, comparing PROBAST to a short form consisting of six PROBAST items anticipated to best identify high ROB. We then applied the short form to all CPMs in the Registry with at least 1 validation (n=556) and assessed the change in discrimination (dAUC) in external validation cohorts (n=1,147).

RESULTS

PROBAST classified 98/102 CPMS as high ROB. The short form identified 96 of these 98 as high ROB (98% sensitivity), with perfect specificity. In the full CPM registry, 527 of 556 CPMs (95%) were classified as high ROB, 20 (3.6%) low ROB, and 9 (1.6%) unclear ROB. Only one model with unclear ROB was reclassified to high ROB after full PROBAST assessment of all low and unclear ROB models. Median change in discrimination was significantly smaller in low ROB models (dAUC -0.9%, IQR -6.2-4.2%) compared to high ROB models (dAUC -11.7%, IQR -33.3-2.6%; P<0.001).

CONCLUSION

High ROB is pervasive among published CPMs. It is associated with poor discriminative performance at validation, supporting the application of PROBAST or a shorter version in CPM reviews.

Prediction of type 2 diabetes mellitus based on nutrition data

Numerous predictive models for the risk of type 2 diabetes mellitus (T2DM) exist, but a minority of them has implemented nutrition data so far, even though the significant effect of nutrition on the pathogenesis, prevention and management of T2DM has been established. Thus, in the present study, we aimed to build a predictive model for the risk of T2DM that incorporates nutrition data and calculates its predictive performance. We analysed cross-sectional data from 1591 individuals from the population-based Cooperative Health Research in the Region of Augsburg (KORA) FF4 study (2013–14) and used a bootstrap enhanced elastic net penalised multivariate regression method in order to build our predictive model and select among 193 food intake variables. After selecting the significant predictor variables, we built a logistic regression model with these variables as predictors and T2DM status as the outcome. The values of area under the receiver operating characteristic (ROC) curve, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and accuracy of our predictive model were calculated. Eleven out of the 193 food intake variables were selected for inclusion in our model, which yielded a value of area under the ROC curve of 0⋅79 and a maximum PPV, NPV and accuracy of 0⋅37, 0⋅98 and 0⋅91, respectively. The present results suggest that nutrition data should be implemented in predictive models to predict the risk of T2DM, since they improve their performance and they are easy to assess.

Using a machine learning approach to investigate factors associated with treatment-resistant depression among adults with chronic non-cancer pain conditions and major depressive disorder

OBJECTIVE

Presence of chronic non-cancer pain conditions (CNPC) among adults with major depressive disorder (MDD) may reduce benefits of antidepressant therapy, thereby increasing the possibility of treatment resistance. This study sought to investigate factors associated with treatment-resistant depression (TRD) among adults with MDD and CNPC using machine learning approaches.

METHODS

This retrospective cohort study was conducted using a US claims database which included adults with newly diagnosed MDD and CNPC (January 2007-June 2017). TRD was identified using a clinical staging algorithm for claims data. Random forest (RF), a machine learning method, and logistic regression was used to identify factors associated with TRD. Initial model development included 42 known and/or probable factors that may be associated with TRD. The final refined model included 20 factors.

RESULTS

Included in the sample were 23,645 patients (73% female mean age: 55 years; 78% with ≥2 CNPC, and 91% with joint pain/arthritis). Overall, 11.4% adults (N = 2684) met selected criteria for TRD. The five leading factors associated with TRD were the following: mental health specialist visits, polypharmacy (≥5 medications), psychotherapy use, anxiety, and age. Cross-validated logistic regression model indicated that those with TRD were younger, more likely to have anxiety, mental health specialist visits, polypharmacy, and psychotherapy use with adjusted odds ratios (AORs) ranging from 1.93 to 1.27 (all ps < .001).

CONCLUSION

Machine learning identified several factors that warrant further investigation and may serve as potential targets for clinical intervention to improve treatment outcomes in patients with TRD and CNPC.

Derivation and Validation of Essential Predictors and Risk Index for Early Detection of Diabetic Retinopathy Using Electronic Health Records

Diabetic retinopathy (DR) is a leading cause for blindness among working-aged adults. The growing prevalence of diabetes urges for cost-effective tools to improve the compliance of eye examinations for early detection of DR. The objective of this research is to identify essential predictors and develop predictive technologies for DR using electronic health records. We conducted a retrospective analysis on a derivation cohort with 3749 DR and 94,127 non-DR diabetic patients. In the analysis, an ensemble predictor selection method was employed to find essential predictors among 26 variables in demographics, duration of diabetes, complications and laboratory results. A predictive model and a risk index were built based on the selected, essential predictors, and then validated using another independent validation cohort with 869 DR and 6448 non-DR diabetic patients. Out of the 26 variables, 10 were identified to be essential for predicting DR. The predictive model achieved a 0.85 AUC on the derivation cohort and a 0.77 AUC on the validation cohort. For the risk index, the AUCs were 0.81 and 0.73 on the derivation and validation cohorts, respectively. The predictive technologies can provide an early warning sign that motivates patients to comply with eye examinations for early screening and potential treatments.

Aim for Clinical Utility, Not Just Predictive Accuracy

Supplemental Digital Content is available in the text.

The predictions from an accurate prognostic model can be of great interest to patients and clinicians. When predictions are reported to individuals, they may decide to take action to improve their health or they may simply be comforted by the knowledge. However, if there is a clearly defined space of actions in the clinical context, a formal decision rule based on the prediction has the potential to have a much broader impact. The use of a prediction-based decision rule should be formalized and preferably compared with the standard of care in a randomized trial to assess its clinical utility; however, evidence is needed to motivate such a trial. We outline how observational data can be used to propose a decision rule based on a prognostic prediction model. We then propose a framework for emulating a prediction driven trial to evaluate the clinical utility of a prediction-based decision rule in observational data. A split-sample structure is often feasible and useful to develop the prognostic model, define the decision rule, and evaluate its clinical utility. See video abstract at, http://links.lww.com/EDE/B656.

Exome Chip Analysis of 14,026 Koreans Reveals Known and Newly Discovered Genetic Loci Associated with Type 2 Diabetes Mellitus

BACKGROUND

Most loci associated with type 2 diabetes mellitus (T2DM) discovered to date are within noncoding regions of unknown functional significance. By contrast, exonic regions have advantages for biological interpretation.

METHODS

We analyzed the association of exome array data from 14,026 Koreans to identify susceptible exonic loci for T2DM. We used genotype information of 50,543 variants using the Illumina exome array platform.

RESULTS

In total, 7 loci were significant with a Bonferroni adjusted P=1.03×10-6. rs2233580 in paired box gene 4 (PAX4) showed the highest odds ratio of 1.48 (P=1.60×10-10). rs11960799 in membrane associated ring-CH-type finger 3 (MARCH3) and rs75680863 in transcobalamin 2 (TCN2) were newly identified loci. When we built a model to predict the incidence of diabetes with the 7 loci and clinical variables, area under the curve (AUC) of the model improved significantly (AUC=0.72, P<0.05), but marginally in its magnitude, compared with the model using clinical variables (AUC=0.71, P<0.05). When we divided the entire population into three groups-normal body mass index (BMI; <25 kg/m2), overweight (25≤ BMI <30 kg/m2), and obese (BMI ≥30 kg/m2) individuals-the predictive performance of the 7 loci was greatest in the group of obese individuals, where the net reclassification improvement was highly significant (0.51; P=8.00×10-5).

CONCLUSION

We found exonic loci having a susceptibility for T2DM. We found that such genetic information is advantageous for predicting T2DM in a subgroup of obese individuals.

Establishment of noninvasive diabetes risk prediction model based on tongue features and machine learning techniques

BACKGROUND

Diabetes is a chronic noncommunicable disease with high incidence rate. Diabetics without early diagnosis or standard treatment may contribute to serious multisystem complications, which can be life threatening. Timely detection and intervention of prediabetes is very important to prevent diabetes, because it is inevitable in the development and progress of the disease.

OBJECTIVE

Our objective was to establish the predictive model that can be applied to evaluate people with blood glucose in high and critical state.

METHODS

We established the diabetes risk prediction model formed by a combined TCM tongue diagnosis with machine learning techniques. 1512 subjects were recruited from the hospital. After data preprocessing, we got the dataset 1 and dataset 2. Dataset 1 was used to train classical machine learning model, while dataset 2 was used to train deep learning model. To evaluate the performance of the prediction model, we used Classification Accuracy(CA), Precision, Recall, F1-score, Precision-Recall curve(P-R curve), Area Under the Precision-Recall curve(AUPRC), Receiver Operating Characteristic curve(ROC curve), Area Under the Receiver Operating Characteristic curve(AUROC), then selected the best diabetes risk prediction model.

RESULTS

On the test set of dataset 1, the CA of non-invasive Stacking model was 71 %, micro average AUROC was 0.87, macro average AUROC was 0.84, and micro average AUPRC was 0.77. In the critical blood glucose group, the AUROC was 0.84, AUPRC was 0.67. In the high blood glucose group, AUROC was 0.87, AUPRC was 0.83. On the validation set of dataset 2, the CA of ResNet50 model was 69 %, micro average AUROC was 0.84, macro average AUROC was 0.83, and micro average AUPRC was 0.73. In the critical blood glucose group, AUROC was 0.88, AUPRC was 0.71. In the high blood glucose group, AUROC was 0.80, AUPRC was 0.76. On the test set of dataset 2, the CA of ResNet50 model was 65 %, micro average AUROC was 0.83, macro average AUROC was 0.82, and micro average AUPRC was 0.71. In the critical blood glucose group, the prediction of AUROC was 0.84, AUPRC was 0.60. In the high blood glucose group, AUROC was 0.87, AUPRC was 0.71.

CONCLUSIONS

Tongue features can improve the prediction accuracy of the diabetes risk prediction model formed by classical machine learning model significantly. In addition to the excellent performance, Stacking model and ResNet50 model which were recommended had non-invasive operation and were easy to use. Stacking model and ResNet50 model had high precision, low false positive rate and low misdiagnosis rate on detecting hyperglycemia. While on detecting blood glucose value in critical state, Stacking model and ResNet50 model had a high sensitivity, a low false negative rate and a low missed diagnosis rate. The study had proved that the differential changes of tongue features reflected the abnormal glucose metabolism, thus the diabetes risk prediction model formed by a combined TCM tongue diagnosis and machine learning technique was feasible.

Performance of Risk Assessment Models for Prevalent or Undiagnosed Type 2 Diabetes Mellitus in a Multi-Ethnic Population-The Helius Study

Background:

Most risk assessment models for type 2 diabetes (T2DM) have been developed in Caucasians and Asians; little is known about their performance in other ethnic groups.

Objective(s):

We aimed to identify existing models for the risk of prevalent or undiagnosed T2DM and externally validate them in a multi-ethnic population currently living in the Netherlands.

Methods:

A literature search to identify risk assessment models for prevalent or undiagnosed T2DM was performed in PubMed until December 2017. We validated these models in 4,547 Dutch, 3,035 South Asian Surinamese, 4,119 African Surinamese, 2,326 Ghanaian, 3,598 Turkish, and 3,894 Moroccan origin participants from the HELIUS (Healthy LIfe in an Urban Setting) cohort study performed in Amsterdam. Model performance was assessed in terms of discrimination (C-statistic) and calibration (Hosmer-Lemeshow test). We identified 25 studies containing 29 models for prevalent or undiagnosed T2DM. C-statistics varied between 0.77–0.92 in Dutch, 0.66–0.83 in South Asian Surinamese, 0.70–0.82 in African Surinamese, 0.61–0.81 in Ghanaian, 0.69–0.86 in Turkish, and 0.69–0.87 in the Moroccan populations. The C-statistics were generally lower among the South Asian Surinamese, African Surinamese, and Ghanaian populations and highest among the Dutch. Calibration was poor (Hosmer-Lemeshow p < 0.05) for all models except one.

Conclusions:

Generally, risk models for prevalent or undiagnosed T2DM show moderate to good discriminatory ability in different ethnic populations living in the Netherlands, but poor calibration. Therefore, these models should be recalibrated before use in clinical practice and should be adapted to the situation of the population they are intended to be used in.

Design, methods, and reporting of impact studies of cardiovascular clinical prediction rules are suboptimal: a systematic review

OBJECTIVES

To evaluate design, methods, and reporting of impact studies of cardiovascular clinical prediction rules (CPRs).

STUDY DESIGN AND SETTING

We conducted a systematic review. Impact studies of cardiovascular CPRs were identified by forward citation and electronic database searches. We categorized the design of impact studies as appropriate for randomized and nonrandomized experiments, excluding uncontrolled before-after study. For impact studies with appropriate study design, we assessed the quality of methods and reporting. We compared the quality of methods and reporting between impact and matched control studies.

RESULTS

We found 110 impact studies of cardiovascular CPRs. Of these, 65 (59.1%) used inappropriate designs. Of 45 impact studies with appropriate design, 31 (68.9%) had substantial risk of bias. Mean number of reporting domains that impact studies with appropriate study design adhered to was 10.2 of 21 domains (95% confidence interval, 9.3 and 11.1). The quality of methods and reporting was not clearly different between impact and matched control studies.

CONCLUSION

We found most impact studies either used inappropriate study design, had substantial risk of bias, or poorly complied with reporting guidelines. This appears to be a common feature of complex interventions. Users of CPRs should critically evaluate evidence showing the effectiveness of CPRs.

A diabetes risk score for Qatar utilizing a novel mathematical modeling approach to identify individuals at high risk for diabetes

We developed a diabetes risk score using a novel analytical approach and tested its diagnostic performance to detect individuals at high risk of diabetes, by applying it to the Qatari population. A representative random sample of 5,000 Qataris selected at different time points was simulated using a diabetes mathematical model. Logistic regression was used to derive the score using age, sex, obesity, smoking, and physical inactivity as predictive variables. Performance diagnostics, validity, and potential yields of a diabetes testing program were evaluated. In 2020, the area under the curve (AUC) was 0.79 and sensitivity and specificity were 79.0% and 66.8%, respectively. Positive and negative predictive values (PPV and NPV) were 36.1% and 93.0%, with 42.0% of Qataris being at high diabetes risk. In 2030, projected AUC was 0.78 and sensitivity and specificity were 77.5% and 65.8%. PPV and NPV were 36.8% and 92.0%, with 43.0% of Qataris being at high diabetes risk. In 2050, AUC was 0.76 and sensitivity and specificity were 74.4% and 64.5%. PPV and NPV were 40.4% and 88.7%, with 45.0% of Qataris being at high diabetes risk. This model-based score demonstrated comparable performance to a data-derived score. The derived self-complete risk score provides an effective tool for initial diabetes screening, and for targeted lifestyle counselling and prevention programs.

A combined strategy of feature selection and machine learning to identify predictors of prediabetes

OBJECTIVE

To identify predictors of prediabetes using feature selection and machine learning on a nationally representative sample of the US population.

MATERIALS AND METHODS

We analyzed n = 6346 men and women enrolled in the National Health and Nutrition Examination Survey 2013-2014. Prediabetes was defined using American Diabetes Association guidelines. The sample was randomly partitioned to training (n = 3174) and internal validation (n = 3172) sets. Feature selection algorithms were run on training data containing 156 preselected exposure variables. Four machine learning algorithms were applied on 46 exposure variables in original and resampled training datasets built using 4 resampling methods. Predictive models were tested on internal validation data (n = 3172) and external validation data (n = 3000) prepared from National Health and Nutrition Examination Survey 2011-2012. Model performance was evaluated using area under the receiver operating characteristic curve (AUROC). Predictors were assessed by odds ratios in logistic models and variable importance in others. The Centers for Disease Control (CDC) prediabetes screening tool was the benchmark to compare model performance.

RESULTS

Prediabetes prevalence was 23.43%. The CDC prediabetes screening tool produced 64.40% AUROC. Seven optimal (≥ 70% AUROC) models identified 25 predictors including 4 potentially novel associations; 20 by both logistic and other nonlinear/ensemble models and 5 solely by the latter. All optimal models outperformed the CDC prediabetes screening tool (P < 0.05).

DISCUSSION

Combined use of feature selection and machine learning increased predictive performance outperforming the recommended screening tool. A range of predictors of prediabetes was identified.

CONCLUSION

This work demonstrated the value of combining feature selection with machine learning to identify a wide range of predictors that could enhance prediabetes prediction and clinical decision-making.

Spatial Analysis of Incidence of Diagnosed Type 2 Diabetes Mellitus and Its Association With Obesity and Physical Inactivity

OBJECTIVES

To investigate the spatial distribution of 10-year incidence of diagnosed type 2 diabetes mellitus (T2DM) and its association with obesity and physical inactivity at a reginal level breakdown.

METHODS

Demographic, behavioral, medical and pharmaceutical and diagnosed T2DM incidence data were collected from a cohort of 232,064 participants who were free of diabetes at enrolment in the 45 and Up Study, conducted in the state of New South Wales (NSW), Australia. We examined the geographical trend and correlation between obesity prevalence, physical inactivity rate and age-and-gender-adjusted cumulative incidence of T2DM, aggregated based on geographical regions.

RESULT

The T2DM incidence, prevalence of obesity and physical inactivity rate at baseline were 6.32%, 20.24%, and 18.7%, respectively. The spatial variation of T2DM incidence was significant (Moran's I=0.52; p<0.01), with the lowest incidence of 2.76% in Richmond Valley-Coastal and the highest of 12.27% in Mount Druitt. T2DM incidence was significantly correlated with the prevalence of obesity (Spearman r=0.62, p<0.001), percentage of participants having five sessions of physical activities or less per week (r=0.79, p<0.001) and percentage of participants walked to work (r=-0.44, p<0.001). The geographical variations in obesity prevalence and physical inactivity rate resembled the geographical variation in the incidence of T2DM.

CONCLUSION

The spatial distribution of T2DM incidence is significantly associated with the geographical prevalence of obesity and physical inactivity rate. Regional campaigns advocating the importance of physical activities in response to the alarming T2DM epidemic should be promoted.

Models for estimating and projecting global, regional and national prevalence and disease burden of asthma: a systematic review

Background

Statistical models are increasingly being used to estimate and project the prevalence and burden of asthma. Given substantial variations in these estimates, there is a need to critically assess the properties of these models and assess their transparency and reproducibility. We aimed to critically appraise the strengths, limitations and reproducibility of existing models for estimating and projecting the global, regional and national prevalence and burden of asthma.

Methods

We undertook a systematic review, which involved searching Medline, Embase, World Health Organization Library and Information Services (WHOLIS) and Web of Science from 1980 to 2017 for modelling studies. Two reviewers independently assessed the eligibility of studies for inclusion and then assessed their strengths, limitations and reproducibility using pre-defined quality criteria. Data were descriptively and narratively synthesised.

Results

We identified 108 eligible studies, which employed a total of 51 models: 42 models were used to derive national level estimates, two models for regional estimates, four models for global and regional estimates and three models for global, regional and national estimates. Ten models were used to estimate the prevalence of asthma, 27 models estimated the burden of asthma – including, health care service utilisation, disability-adjusted life years, mortality and direct and indirect costs of asthma – and 14 models estimated both the prevalence and burden of asthma. Logistic and linear regression models were most widely used for national estimates. Different versions of the DisMod-MR- Bayesian meta-regression models and Cause Of Death Ensemble model (CODEm) were predominantly used for global, regional and national estimates. Most models suffered from a number of methodological limitations – in particular, poor reporting, insufficient quality and lack of reproducibility.

Conclusions

Whilst global, regional and national estimates of asthma prevalence and burden continue to inform health policy and investment decisions on asthma, most models used to derive these estimates lack the required reproducibility. There is a need for better-constructed models for estimating and projecting the prevalence and disease burden of asthma and a related need for better reporting of models, and making data and code available to facilitate replication.

A prediction nomogram for the 3-year risk of incident diabetes among Chinese adults

Identifying individuals at high risk for incident diabetes could help achieve targeted delivery of interventional programs. We aimed to develop a personalized diabetes prediction nomogram for the 3-year risk of diabetes among Chinese adults. This retrospective cohort study was among 32,312 participants without diabetes at baseline. All participants were randomly stratified into training cohort (n = 16,219) and validation cohort (n = 16,093). The least absolute shrinkage and selection operator model was used to construct a nomogram and draw a formula for diabetes probability. 500 bootstraps performed the receiver operating characteristic (ROC) curve and decision curve analysis resamples to assess the nomogram's determination and clinical use, respectively. 155 and 141 participants developed diabetes in the training and validation cohort, respectively. The area under curve (AUC) of the nomogram was 0.9125 (95% CI, 0.8887-0.9364) and 0.9030 (95% CI, 0.8747-0.9313) for the training and validation cohort, respectively. We used 12,545 Japanese participants for external validation, its AUC was 0.8488 (95% CI, 0.8126-0.8850). The internal and external validation showed our nomogram had excellent prediction performance. In conclusion, we developed and validated a personalized prediction nomogram for 3-year risk of incident diabetes among Chinese adults, identifying individuals at high risk of developing diabetes.

The external validity and performance of the no-laboratory American Diabetes Association screening tool for identifying undiagnosed type 2 diabetes among the Iranian population

AIMS

The aim of this study is to assess the American Diabetes Association (ADA) risk score as a self-assessment screening tool for undiagnosed type 2 diabetes (T2DM) in Iran.

METHODS

In a national survey of risk factors for non-communicable diseases, we included 3458 Iranian adults. The discrimination and validity were assessed using the area under the curve (AUC), sensitivity, specificity, Youden's index, positive and negative predictive values (PPV and NPV). The frequency of high-risk Iranian population who need a glucose test and those who need intervention were also estimated.

RESULTS

The AUC was 73.7% and the suggested ADA score of ≥5 yielded a sensitivity of 51.6%, specificity 82.4%, NPV 98.3%, and PPV 7.9%. This threshold results in classifying 18.6% of the Iranians, equals to 8.5 million, as high-risk and 1.5% of the population, about 700,000, would need intervention. However, our study suggested score ≥4 that identified 34% of the population as high-risk and 2% of the population would need intervention.

CONCLUSION

Our findings support the ADA suggested threshold for identifying high-risk individuals for undiagnosed T2DM; however, a lower threshold is also recommended for higher sensitivity. The ADA screening tool could help the public health system for low-cost screening.

Investigating spatial convergence of diagnosed dementia, depression and type 2 diabetes prevalence in West Adelaide, Australia

BACKGROUND

Comorbid depression and type 2 diabetes (T2D) is an important risk factor for dementia. This study investigates the factors associated with, the spatial variation and spatial convergence of diagnosed cases of these conditions. This approach may identify areas with unmet needs.

METHODS

We used cross-sectional data (2010 to 2014) from 16 general practices in west Adelaide, Australia. Multi-level modelling accounting for individual-level characteristics nested within statistical area level 1 (SA1) determined covariate associations with these three diseases. Getis-Ord Gi method was used to investigate spatial variation, hot spots and cold spots of these conditions.

RESULTS

1.4% of active patients in west Adelaide aged 45 and above were diagnosed with dementia, 9.6% with depression and 13.3% with T2D. Comorbidity was significant across all three diseases. Elderly age (65+ years) was significantly associated with diagnosed dementia and T2D. Hyperlipidemia or hypertension diagnosis and belonging to lower socioeconomic status were significantly associated with diagnosed T2D and depression. The spatial distribution of each disease varied across west Adelaide. Spatial convergence of the three diseases was observed in two large hot spot clusters and one main cluster of cold spots.

LIMITATIONS

Due to underreporting, potentially significant covariates like alcohol intake were unable to be assessed. There may be a bias towards health-conscious individuals or patients managing diagnosed diseases that actively visit their general practice.

CONCLUSIONS

Patterns of spatial convergence and the shared associations in dementia, depression and diabetes enable policymakers to tailor interventions to the areas where risk of these conditions are greater.

Electrochemically Activated Conductive Ni-Based MOFs for Non-enzymatic Sensors Toward Long-Term Glucose Monitoring

Continuous intensive monitoring of glucose is one of the most important approaches in recovering the quality of life of diabetic patients. One challenge for electrochemical enzymatic glucose sensors is their short lifespan for continuous glucose monitoring. Therefore, it is of great significance to develop non-enzymatic glucose sensors as an alternative approach for long-term glucose monitoring. This study presented a highly sensitive and selective electrochemical non-enzymatic glucose sensor using the electrochemically activated conductive Ni3(2,3,6,7,10,11-hexaiminotriphenylene)2 MOFs as sensing materials. The morphology and structure of the MOFs were investigated by scanning SEM and FTIR, respectively. The performance of the activated electrode toward the electrooxidation of glucose in alkaline solution was evaluated with cyclic voltammetry technology in the potential range from 0.2 V to 0.6 V. The electrochemical activated Ni-MOFs exhibited obvious anodic (0.46 V) and cathodic peaks (0.37 V) in the 0.1 M NaOH solution due to the Ni(II)/Ni(III) transfer. A linear relationship between the glucose concentrations (ranging from 0 to 10 mM) and anodic peak currents with R2 = 0.954 was obtained. It was found that the diffusion of glucose was the limiting step in the electrochemical reaction. The sensor exhibited good selectivity toward glucose in the presence of 10-folds uric acid and ascorbic acid. Moreover, this sensor showed good long-term stability for continuous glucose monitoring. The good selectivity, stability, and rapid response of this sensor suggests that it could have potential applications in long-term non-enzymatic blood glucose monitoring.

Protocol for a systematic review on the methodological and reporting quality of prediction model studies using machine learning techniques

INTRODUCTION

Studies addressing the development and/or validation of diagnostic and prognostic prediction models are abundant in most clinical domains. Systematic reviews have shown that the methodological and reporting quality of prediction model studies is suboptimal. Due to the increasing availability of larger, routinely collected and complex medical data, and the rising application of Artificial Intelligence (AI) or machine learning (ML) techniques, the number of prediction model studies is expected to increase even further. Prediction models developed using AI or ML techniques are often labelled as a 'black box' and little is known about their methodological and reporting quality. Therefore, this comprehensive systematic review aims to evaluate the reporting quality, the methodological conduct, and the risk of bias of prediction model studies that applied ML techniques for model development and/or validation.

METHODS AND ANALYSIS

A search will be performed in PubMed to identify studies developing and/or validating prediction models using any ML methodology and across all medical fields. Studies will be included if they were published between January 2018 and December 2019, predict patient-related outcomes, use any study design or data source, and available in English. Screening of search results and data extraction from included articles will be performed by two independent reviewers. The primary outcomes of this systematic review are: (1) the adherence of ML-based prediction model studies to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD), and (2) the risk of bias in such studies as assessed using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). A narrative synthesis will be conducted for all included studies. Findings will be stratified by study type, medical field and prevalent ML methods, and will inform necessary extensions or updates of TRIPOD and PROBAST to better address prediction model studies that used AI or ML techniques.

ETHICS AND DISSEMINATION

Ethical approval is not required for this study because only available published data will be analysed. Findings will be disseminated through peer-reviewed publications and scientific conferences.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO, CRD42019161764.

The Association Between Dietary Habits and Other Lifestyle Indicators and Dysglycemia in Saudi Adults Free of Previous Diagnosis of Diabetes

Objective:

Study the association of dietary habits and other indicators of lifestyle with dysglycemia in Saudi adults.

Methods:

In a cross-sectional design, data were obtained from 1403 Saudi adults (⩾20 years), not previously diagnosed with diabetes. Demographics, lifestyle variables and dietary habits were obtained using a predesigned questionnaire. Fasting plasma glucose, glycated hemoglobin and 1-hour oral glucose tolerance test were used to identify dysglycemia. Regression analysis was performed to determine the associations of dietary factors and other indicators of lifestyle with dysglycemia.

Results:

A total 1075 adults (596 men, and 479 women) had normoglycemia, and 328 (195 men, and 133 women) had dysglycemia. Following adjustment for age, BMI and waist circumference, in men the weekly intake of 5 portions or more of red meat and Turkish coffee were associated with decreased odds of having dysglycemia odds ratio (OR) 0.444 (95% CI: 0.223, 0.881; P = .02) and 0.387 (95% CI: 0.202, 0.74; P = .004), respectively. In women, the intake of fresh juice 1 to 4 portions per week and 5 portions or more were associated with OR 0.603 (95% CI: 0.369, 0.985; P = .043) and OR 0.511 (95% CI: 0.279, 0.935; P = .029) decreased odds of having dysglycemia, respectively compared with women who did not drink fresh juice. The intake of 5 times or more per week of hibiscus drink was associated with increased odds of having dysglycemia, OR 5.551 (95% CI: 1.576, 19.55, P = .008) compared with women not using such a drink. Other lifestyle factors were not associated with dysglycemia.

Conclusion:

Dietary practices by studied Saudis have some impact on risk of dysglycemia, with obvious sex differences.