Key steps and common pitfalls in developing and validating risk models

A Risk Score to Identify Low Bone Mineral Density for Age in Young Patients with Anorexia Nervosa

OBJECTIVE

Developing a scoring assessment tools for the determination of low bone mass for age at lumbar spine and hip in patients with anorexia nervosa (AN).

METHODS

The areal bone mineral density (aBMD) was determined with dual-energy X-ray absorptiometry (DXA). In 331 women with AN and 121 controls, aged from 14.5 to 34.9 years, univariate and multivariate logistic regression analyses were performed to address the association of Z-score aBMD evaluated at lumbar spine and hip with several parameters.

RESULTS

For the lumbar spine and hip, the three risk factors significantly and independently associated with Z-score aBMD were age of patients (variable in class ≥20 year vs. <20 year), minimal disease-related BMI (continuous variable), and duration of amenorrhea without contraceptive use (variable in class ≥18 months vs. <18 months), with close values for the odds ratio for the two bone sites. A simple risk score equation was developed and tested combining only these three parameters. The AUC's measuring the score's performance were, respectively, 0.85 [95% CI: 0.79-0.90] with a sensitivity of 83% and specificity of 71%, and 0.82 [95% CI: 0.76-0.86] with a sensitivity of 92% and specificity of 55% to detect low aBMD in lumbar spine and hip. The cut-off values for low bone mass for age were 0.9 and 1.33 for the two bone sites. The prediction model revealed that a minimum of 83% of the patients presenting low bone mass for age were correctly identified.

CONCLUSIONS

the study presents for the first time a risk score for diagnosing low bone mass for age in young patients with AN. Considering its excellent sensitivity, and its ease of use, requiring only three parameters that are well identified in this disease, this new score may be useful in clinical settings when DXA scans are not feasible.

Early prediction of mortality upon intensive care unit admission

BACKGROUND

We aimed to develop and validate models for predicting intensive care unit (ICU) mortality of critically ill adult patients as early as upon ICU admission.

METHODS

Combined data of 79,657 admissions from two teaching hospitals' ICU databases were used to train and validate the machine learning models to predict ICU mortality upon ICU admission and at 24 h after ICU admission by using logistic regression, gradient boosted trees (GBT), and deep learning algorithms.

RESULTS

In the testing dataset for the admission models, the ICU mortality rate was 7%, and 38.4% of patients were discharged alive or dead within 1 day of ICU admission. The area under the receiver operating characteristic curve (0.856, 95% CI 0.845-0.867) and area under the precision-recall curve (0.331, 95% CI 0.323-0.339) were the highest for the admission GBT model. The ICU mortality rate was 17.4% in the 24-hour testing dataset, and the performance was the highest for the 24-hour GBT model.

CONCLUSION

The ADM models can provide crucial information on ICU mortality as early as upon ICU admission. 24 H models can be used to improve the prediction of ICU mortality for patients discharged more than 1 day after ICU admission.

Calibration, Clinical Utility, and Specificity of Clinical Decision Support Tools in Inflammatory Bowel Disease

BACKGROUND & AIMS

Clinical decision support tools (CDSTs) have been developed to predict response to vedolizumab (VDZ) and ustekinumab (UST) in Crohn's disease (CD) and ulcerative colitis (UC). In addition to assessing their discrimination performance, our study aimed to evaluate their calibration, clinical utility, and specificity.

METHODS

We included 280 patients with CD and 218 patients with UC initiating VDZ, and 194 patients with CD initiating UST. We assessed discrimination by comparing rates of effectiveness outcomes between response probability groups forecasted by CDSTs. Calibration curves and decision curve analysis evaluated the calibration and clinical utility of VDZ-CDSTs. Additionally, we examined the agreement between UST-CDST and VDZ-CDST in assigning response probability groups among patients with CD starting UST.

RESULTS

In the overall cohort, CDSTs allocated 7.2%, 50.0%, and 42.8% of the patients to the low-, intermediate-, and high-response probability groups, respectively. VDZ-CDSTs groups demonstrated significant differences in the rates of clinical and endoscopic response and remission, whereas UST-CDST groups showed significant discrimination only for clinical remission. Although VDZ-CDSTs overestimated clinical remission rates, they more accurately predicted rates of VDZ persistence without need for surgery or dose escalation. Compared with empirically treating all patients with VDZ, VDZ-CDSTs yielded higher net benefits in selecting patients who would continue VDZ without need for surgery or dose escalation. Finally, the agreement between UST-CDST and VDZ-CDST in predicting response was 73.7%.

CONCLUSION

VDZ-CDSTs significantly discriminated response to VDZ and were more beneficial in identifying patients who would continue therapy without requiring surgery or dose escalation, compared with treating all patients empirically.

Point-of-Care Noninvasive Prediction of Liver-Related Events in Patients With Nonalcoholic Fatty Liver Disease

BACKGROUND & AIMS

Individual risk prediction of liver-related events (LRE) is needed for clinical assessment of nonalcoholic fatty liver disease (NAFLD)/nonalcoholic steatohepatitis (NASH) patients. We aimed to provide point-of-care validated liver stiffness measurement (LSM)-based risk prediction models for the development of LRE in patients with NAFLD, focusing on selecting patients for clinical trials at risk of clinical events.

METHODS

Two large multicenter cohorts were evaluated, 2638 NAFLD patients covering all LSM values as the derivation cohort and 679 more advanced patients as the validation cohort. We used Cox regression to develop and validate risk prediction models based on LSM alone, and the ANTICIPATE and ANTICIPATE-NASH models for clinically significant portal hypertension. The main outcome of the study was the rate of LRE in the first 3 years after initial assessment.

RESULTS

The 3 predictive models had similar performance in the derivation cohort with a very high discriminative value (c-statistic, 0.87-0.91). In the validation cohort, the LSM-LRE alone model had a significant inferior discrimination (c-statistic, 0.75) compared with the other 2 models, whereas the ANTICIPATE-NASH-LRE model (0.81) was significantly better than the ANTICIPATE-LRE model (0.79). In addition, the ANTICIPATE-NASH-LRE model presented very good calibration in the validation cohort (integrated calibration index, 0.016), and was better than the ANTICIPATE-LRE model.

CONCLUSIONS

The ANTICIPATE-LRE models, and especially the ANTICIPATE-NASH-LRE model, could be valuable validated clinical tools to individually assess the risk of LRE at 3 years in patients with NAFLD/NASH.

Prognostic factors for the development of upper limb dysfunctions after breast cancer: the UPLIFT-BC prospective longitudinal cohort study protocol

INTRODUCTION

Upper limb (UL) dysfunctions are highly prevalent in people after breast cancer and have a great impact on performing activities in daily living. To improve care, a more comprehensive understanding of the development and persistence of UL dysfunctions is needed. Therefore, the UPLIFT-BC study will primarily examine the prognostic value of different factors at the body functions and structures, environmental and personal level of the International Classification of Functioning, Disability and Health (ICF) framework at 1-month post-surgery for persisting UL dysfunctions at 6 months after finishing cancer treatment.

METHODS AND ANALYSIS

A prospective longitudinal cohort study, running from 1-week pre-surgery to 6 months post-local cancer treatment, is performed in a cohort of 250 women diagnosed with primary breast cancer. Different potentially prognostic factors to UL dysfunctions, covering body functions and structures, environmental and personal factors of the ICF, are assessed pre-surgically and at different time points post-surgery. The primary aim is to investigate the prognostic value of these factors at 1-month post-surgery for subjective UL function (ie, QuickDASH) at 6 months post-cancer treatment, that is, 6 months post-radiotherapy or post-surgery (T3), depending on the individuals' cancer treatment trajectory. In this, factors with relevant prognostic value pre-surgery are considered as well. Similar analyses are performed with an objective measure for UL function (ie, accelerometry) and a composite score of the combination of subjective and objective UL function. Second, in the subgroup of participants who receive radiotherapy, the prognostic value of the same factors is explored at 1-month post-radiotherapy and 6 months post-surgery. A forward stepwise selection strategy is used to obtain these multivariable prognostic models.

ETHICS AND DISSEMINATION

The study protocol was approved by the Ethics Committee of UZ/KU Leuven (reference number s66248). The results of this study will be published in peer-reviewed journals and will be presented at several research conferences.

TRIAL REGISTRATION NUMBER

NCT05297591.

Identification of Predictors for Progression of Scoliosis in Rett Syndrome

Rett syndrome is a neurodevelopmental disorder in which scoliosis is a common orthopedic complication. This explorative study aims to identify predictors for rapid progression of scoliosis in Rett syndrome to enable variable selection for future prediction model development. A univariable logistic regression model was used to identify variables that discriminate between individuals with and without rapid progression of scoliosis (>10  ∘ Cobb angle/6 months) based on multi-center data. Predictors were identified using univariable logistic regression with OR (95% CI) and AUC (95% CI). Age at inclusion, Cobb angle at baseline and epilepsy have the highest discriminative ability for rapid progression of scoliosis in Rett syndrome.

Interpretable Predictive Models to Understand Risk Factors for Maternal and Fetal Outcomes

Although most pregnancies result in a good outcome, complications are not uncommon and can be associated with serious implications for mothers and babies. Predictive modeling has the potential to improve outcomes through a better understanding of risk factors, heightened surveillance for high-risk patients, and more timely and appropriate interventions, thereby helping obstetricians deliver better care. We identify and study the most important risk factors for four types of pregnancy complications: (i) severe maternal morbidity, (ii) shoulder dystocia, (iii) preterm preeclampsia, and (iv) antepartum stillbirth. We use an Explainable Boosting Machine (EBM), a high-accuracy glass-box learning method, for the prediction and identification of important risk factors. We undertake external validation and perform an extensive robustness analysis of the EBM models. EBMs match the accuracy of other black-box ML methods, such as deep neural networks and random forests, and outperform logistic regression, while being more interpretable. EBMs prove to be robust. The interpretability of the EBM models reveal surprising insights into the features contributing to risk (e.g., maternal height is the second most important feature for shoulder dystocia) and may have potential for clinical application in the prediction and prevention of serious complications in pregnancy.

Development and validation of a risk model for hospital-acquired venous thrombosis: the Medical Inpatients Thrombosis and Hemostasis study

BACKGROUND

Regulatory organizations recommend assessing hospital-acquired (HA) venous thromboembolism (VTE) risk for medical inpatients.

OBJECTIVES

To develop and validate a risk assessment model (RAM) for HA-VTE in medical inpatients using objective and assessable risk factors knowable at admission.

METHODS

The development cohort included people admitted to medical services at the University of Vermont Medical Center (Burlington, Vermont) between 2010 and 2019, and the validation cohorts included people admitted to Hennepin County Medical Center (Minneapolis, Minnesota), University of Michigan Medical Center (Ann Arbor, Michigan), and Harris Health Systems (Houston, Texas). Individuals with VTE at admission, aged <18 years, and admitted for <1 midnight were excluded. We used a Bayesian penalized regression technique to select candidate HA-VTE risk factors for final inclusion in the RAM.

RESULTS

The development cohort included 60 633 admissions and 227 HA-VTE, and the validation cohorts included 111 269 admissions and 651 HA-VTE. Seven HA-VTE risk factors with t statistics ≥1.5 were included in the RAM: history of VTE, low hemoglobin level, elevated creatinine level, active cancer, hyponatremia, increased red cell distribution width, and malnutrition. The areas under the receiver operating characteristic curve and calibration slope were 0.72 and 1.10, respectively. The areas under the receiver operating characteristic curve and calibration slope were 0.70 and 0.93 at Hennepin County Medical Center, 0.70 and 0.87 at the University of Michigan Medical Center, and 0.71 and 1.00 at Harris Health Systems, respectively. The RAM performed well stratified by age, sex, and race.

CONCLUSION

We developed and validated a RAM for HA-VTE in medical inpatients. By quantifying risk, clinicians can determine the potential benefits of measures to reduce HA-VTE.

Using simple clinical and ultrasound variables to develop a model to predict first trimester pregnancy viability

INTRODUCTION

Early prediction of pregnancies destined to miscarry will allow couples to prepare for this common but often unexpected eventuality, and clinicians to allocate finite resources. We aimed to develop a prediction model combining clinical, demographic, and sonographic data as a clinical tool to aid counselling about first trimester pregnancy outcome.

MATERIAL AND METHODS

This is a prospective, observational cohort study conducted at Queen Charlotte's and Chelsea Hospital, UK from March 2014 to May 2019. Women with confirmed intrauterine pregnancies between 5 weeks and their dating scan (11-14 weeks) were recruited. Participants attended serial ultrasound scans in the first trimester and at each visit recorded symptoms of vaginal bleeding, pelvic pain, nausea and vomiting using validated scoring tools. Pregnancies were followed up until the dating scan (11-14 weeks). Univariate and multivariate analyses were performed to predict first trimester viability. A model was developed with multivariable logistic regression, variables limited by feature selection, and bootstrapping with multiple imputation was used for internal validation.

RESULTS

1403 women were recruited and after exclusions, data were available for 1105. 160 women (14.5 %) experienced first trimester miscarriage and 945 women (85.5 %) had viable pregnancies at 11-14 weeks' gestation. The average gestational age at the initial visit (calculated from the menstrual dates) was 7 + 1 weeks (+/-12.2 days). A multivariable logistic regression model was developed to predict first trimester viability and included the variables: mean gestational sac diameter, presence of fetal heart pulsations, difference in gestational age from last menstrual period and from mean sac diameter on ultrasonography, current folic acid usage and maternal age. The model demonstrated good performance (optimism-corrected area under curve (AUC) 0.84, 95 % CI 0.81-0.87; optimism-corrected calibration slope 0.969).

CONCLUSION

We have developed and internally validated a model to predict first trimester viability with good accuracy prior to the 11-14 week dating scan, which now needs to be externally validated prior to clinical use.

Inter-hospital variation in data collection, radiotherapy treatment, and survival in patients with head and neck cancer: A multisite study

BACKGROUND AND PURPOSE

Inter-hospital inequalities in head and neck cancer (HNC) survival may exist due to variation in radiotherapy treatment-related factors. This study investigated inter-hospital variation in data collection, primary radiotherapy treatment, and survival in HNC patients from an Australian setting.

MATERIALS AND METHODS

Data collected in oncology information systems (OIS) from seven Australian hospitals was extracted for 3,182 adults treated with curative radiotherapy, with or without surgery or chemotherapy, for primary, non-metastatic squamous cell carcinoma of the head and neck (2000-2017). Death data was sourced from the National Death Index using record linkage. Multivariable Cox regression was used to assess the association between survival and hospital.

RESULTS

Inter-hospital variation in data collection, primary radiotherapy dose, and five-year HNC-related death was detected. Completion of eleven fields ranged from 66%-98%. Primary radiotherapy treated Tis-T1N0 glottic and any stage oral cavity and oropharynx cancers received significantly different time-corrected biologically equivalent dose in two gray fractions (EQD2T) by hospital, with observed deviation from Australian radiotherapy guidelines. Increased EQD2T dose was associated with a reduced risk of five-year HNC-related death in all patients and those treated with primary radiotherapy. Hospital, tumour site, and T and N classification were also identified as independent prognostic factors for five-year HNC-related death in all patients treated with radiotherapy.

CONCLUSION

Unexplained variation exists in HNC-related death in patients treated at Australian hospitals. Available routinely collected data in OIS are insufficient to explain variation in survival. Innovative data collection, extraction, and classification practices are needed to inform clinical practice.

Prognostic models for surgical-site infection in gastrointestinal surgery: systematic review

BACKGROUND

Identification of patients at high risk of surgical-site infection may allow clinicians to target interventions and monitoring to minimize associated morbidity. The aim of this systematic review was to identify and evaluate prognostic tools for the prediction of surgical-site infection in gastrointestinal surgery.

METHODS

This systematic review sought to identify original studies describing the development and validation of prognostic models for 30-day SSI after gastrointestinal surgery (PROSPERO: CRD42022311019). MEDLINE, Embase, Global Health, and IEEE Xplore were searched from 1 January 2000 to 24 February 2022. Studies were excluded if prognostic models included postoperative parameters or were procedure specific. A narrative synthesis was performed, with sample-size sufficiency, discriminative ability (area under the receiver operating characteristic curve), and prognostic accuracy compared.

RESULTS

Of 2249 records reviewed, 23 eligible prognostic models were identified. A total of 13 (57 per cent) reported no internal validation and only 4 (17 per cent) had undergone external validation. Most identified operative contamination (57 per cent, 13 of 23) and duration (52 per cent, 12 of 23) as important predictors; however, there remained substantial heterogeneity in other predictors identified (range 2-28). All models demonstrated a high risk of bias due to the analytic approach, with overall low applicability to an undifferentiated gastrointestinal surgical population. Model discrimination was reported in most studies (83 per cent, 19 of 23); however, calibration (22 per cent, 5 of 23) and prognostic accuracy (17 per cent, 4 of 23) were infrequently assessed. Of externally validated models (of which there were four), none displayed 'good' discrimination (area under the receiver operating characteristic curve greater than or equal to 0.7).

CONCLUSION

The risk of surgical-site infection after gastrointestinal surgery is insufficiently described by existing risk-prediction tools, which are not suitable for routine use. Novel risk-stratification tools are required to target perioperative interventions and mitigate modifiable risk factors.

Prognostic models for heart failure in patients with type 2 diabetes: a systematic review and meta-analysis

OBJECTIVE

To provide a systematic review, critical appraisal, assessment of performance and generalisability of all the reported prognostic models for heart failure (HF) in patients with type 2 diabetes (T2D).

METHODS

We performed a literature search in Medline, Embase, Central Register of Controlled Trials, Cochrane Database of Systematic Reviews and Scopus (from inception to July 2022) and grey literature to identify any study developing and/or validating models predicting HF applicable to patients with T2D. We extracted data on study characteristics, modelling methods and measures of performance, and we performed a random-effects meta-analysis to pool discrimination in models with multiple validation studies. We also performed a descriptive synthesis of calibration and we assessed the risk of bias and certainty of evidence (high, moderate, low).

RESULTS

Fifty-five studies reporting on 58 models were identified: (1) models developed in patients with T2D for HF prediction (n=43), (2) models predicting HF developed in non-diabetic cohorts and externally validated in patients with T2D (n=3), and (3) models originally predicting a different outcome and externally validated for HF (n=12). RECODe (C-statistic=0.75 95% CI (0.72, 0.78), 95% prediction interval (PI) (0.68, 0.81); high certainty), TRS-HFDM (C-statistic=0.75 95% CI (0.69, 0.81), 95% PI (0.58, 0.87); low certainty) and WATCH-DM (C-statistic=0.70 95% CI (0.67, 0.73), 95% PI (0.63, 0.76); moderate certainty) showed the best performance. QDiabetes-HF demonstrated also good discrimination but was externally validated only once and not meta-analysed.

CONCLUSIONS

Among the prognostic models identified, four models showed promising performance and, thus, could be implemented in current clinical practice.

Prognostic models for predicting clinical disease progression, worsening and activity in people with multiple sclerosis

BACKGROUND

Multiple sclerosis (MS) is a chronic inflammatory disease of the central nervous system that affects millions of people worldwide. The disease course varies greatly across individuals and many disease-modifying treatments with different safety and efficacy profiles have been developed recently. Prognostic models evaluated and shown to be valid in different settings have the potential to support people with MS and their physicians during the decision-making process for treatment or disease/life management, allow stratified and more precise interpretation of interventional trials, and provide insights into disease mechanisms. Many researchers have turned to prognostic models to help predict clinical outcomes in people with MS; however, to our knowledge, no widely accepted prognostic model for MS is being used in clinical practice yet.

OBJECTIVES

To identify and summarise multivariable prognostic models, and their validation studies for quantifying the risk of clinical disease progression, worsening, and activity in adults with MS.

SEARCH METHODS

We searched MEDLINE, Embase, and the Cochrane Database of Systematic Reviews from January 1996 until July 2021. We also screened the reference lists of included studies and relevant reviews, and references citing the included studies.

SELECTION CRITERIA

We included all statistically developed multivariable prognostic models aiming to predict clinical disease progression, worsening, and activity, as measured by disability, relapse, conversion to definite MS, conversion to progressive MS, or a composite of these in adult individuals with MS. We also included any studies evaluating the performance of (i.e. validating) these models. There were no restrictions based on language, data source, timing of prognostication, or timing of outcome.

DATA COLLECTION AND ANALYSIS

Pairs of review authors independently screened titles/abstracts and full texts, extracted data using a piloted form based on the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS), assessed risk of bias using the Prediction Model Risk Of Bias Assessment Tool (PROBAST), and assessed reporting deficiencies based on the checklist items in Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD). The characteristics of the included models and their validations are described narratively. We planned to meta-analyse the discrimination and calibration of models with at least three external validations outside the model development study but no model met this criterion. We summarised between-study heterogeneity narratively but again could not perform the planned meta-regression.

MAIN RESULTS

We included 57 studies, from which we identified 75 model developments, 15 external validations corresponding to only 12 (16%) of the models, and six author-reported validations. Only two models were externally validated multiple times. None of the identified external validations were performed by researchers independent of those that developed the model. The outcome was related to disease progression in 39 (41%), relapses in 8 (8%), conversion to definite MS in 17 (18%), and conversion to progressive MS in 27 (28%) of the 96 models or validations. The disease and treatment-related characteristics of included participants, and definitions of considered predictors and outcome, were highly heterogeneous amongst the studies. Based on the publication year, we observed an increase in the percent of participants on treatment, diversification of the diagnostic criteria used, an increase in consideration of biomarkers or treatment as predictors, and increased use of machine learning methods over time. Usability and reproducibility All identified models contained at least one predictor requiring the skills of a medical specialist for measurement or assessment. Most of the models (44; 59%) contained predictors that require specialist equipment likely to be absent from primary care or standard hospital settings. Over half (52%) of the developed models were not accompanied by model coefficients, tools, or instructions, which hinders their application, independent validation or reproduction. The data used in model developments were made publicly available or reported to be available on request only in a few studies (two and six, respectively). Risk of bias We rated all but one of the model developments or validations as having high overall risk of bias. The main reason for this was the statistical methods used for the development or evaluation of prognostic models; we rated all but two of the included model developments or validations as having high risk of bias in the analysis domain. None of the model developments that were externally validated or these models' external validations had low risk of bias. There were concerns related to applicability of the models to our research question in over one-third (38%) of the models or their validations. Reporting deficiencies Reporting was poor overall and there was no observable increase in the quality of reporting over time. The items that were unclearly reported or not reported at all for most of the included models or validations were related to sample size justification, blinding of outcome assessors, details of the full model or how to obtain predictions from it, amount of missing data, and treatments received by the participants. Reporting of preferred model performance measures of discrimination and calibration was suboptimal.

AUTHORS' CONCLUSIONS

The current evidence is not sufficient for recommending the use of any of the published prognostic prediction models for people with MS in clinical routine today due to lack of independent external validations. The MS prognostic research community should adhere to the current reporting and methodological guidelines and conduct many more state-of-the-art external validation studies for the existing or newly developed models.

External Validation of Prediction Models for Surgical Complications in People Considering Total Hip or Knee Arthroplasty Was Successful for Delirium but Not for Surgical Site Infection, Postoperative Bleeding, and Nerve Damage: A Retrospective Cohort Study

Although several models for the prediction of surgical complications after primary total hip or total knee replacement (THA and TKA, respectively) are available, only a few models have been externally validated. The aim of this study was to externally validate four previously developed models for the prediction of surgical complications in people considering primary THA or TKA. We included 2614 patients who underwent primary THA or TKA in secondary care between 2017 and 2020. Individual predicted probabilities of the risk for surgical complication per outcome (i.e., surgical site infection, postoperative bleeding, delirium, and nerve damage) were calculated for each model. The discriminative performance of patients with and without the outcome was assessed with the area under the receiver operating characteristic curve (AUC), and predictive performance was assessed with calibration plots. The predicted risk for all models varied between <0.01 and 33.5%. Good discriminative performance was found for the model for delirium with an AUC of 84% (95% CI of 0.82-0.87). For all other outcomes, poor discriminative performance was found; 55% (95% CI of 0.52-0.58) for the model for surgical site infection, 61% (95% CI of 0.59-0.64) for the model for postoperative bleeding, and 57% (95% CI of 0.53-0.61) for the model for nerve damage. Calibration of the model for delirium was moderate, resulting in an underestimation of the actual probability between 2 and 6%, and exceeding 8%. Calibration of all other models was poor. Our external validation of four internally validated prediction models for surgical complications after THA and TKA demonstrated a lack of predictive accuracy when applied in another Dutch hospital population, with the exception of the model for delirium. This model included age, the presence of a heart disease, and the presence of a disease of the central nervous system as predictor variables. We recommend that clinicians use this simple and straightforward delirium model during preoperative counselling, shared decision-making, and early delirium precautionary interventions.

Machine Learning and Nomogram Prognostic Modeling for 2-Year Head and Neck Cancer-Specific Survival Using Electronic Health Record Data: A Multisite Study

PURPOSE

There is limited knowledge of the prediction of 2-year cancer-specific survival (CSS) in the head and neck cancer (HNC) population. The aim of this study is to develop and validate machine learning models and a nomogram for the prediction of 2-year CSS in patients with HNC using real-world data collected by major teaching and tertiary referral hospitals in New South Wales (NSW), Australia.

MATERIALS AND METHODS

Data collected in oncology information systems at multiple NSW Cancer Centres were extracted for 2,953 eligible adults diagnosed between 2000 and 2017 with squamous cell carcinoma of the head and neck. Death data were sourced from the National Death Index using record linkage. Machine learning and Cox regression/nomogram models were developed and internally validated in Python and R, respectively.

RESULTS

Machine learning models demonstrated highest performance (C-index) in the larynx and nasopharynx cohorts (0.82), followed by the oropharynx (0.79) and the hypopharynx and oral cavity cohorts (0.73). In the whole HNC population, C-indexes of 0.79 and 0.70 and Brier scores of 0.10 and 0.27 were reported for the machine learning and nomogram model, respectively. Cox regression analysis identified age, T and N classification, and time-corrected biologic equivalent dose in two gray fractions as independent prognostic factors for 2-year CSS. N classification was the most important feature used for prediction in the machine learning model followed by age.

CONCLUSION

Machine learning and nomogram analysis predicted 2-year CSS with high performance using routinely collected and complete clinical information extracted from oncology information systems. These models function as visual decision-making tools to guide radiotherapy treatment decisions and provide insight into the prediction of survival outcomes in patients with HNC.

Development and Validation of Predictive Model for a Diagnosis of First Episode Psychosis Using the Multinational EU-GEI Case-control Study and Modern Statistical Learning Methods

Background and Hypothesis

It is argued that availability of diagnostic models will facilitate a more rapid identification of individuals who are at a higher risk of first episode psychosis (FEP). Therefore, we developed, evaluated, and validated a diagnostic risk estimation model to classify individual with FEP and controls across six countries.

Study Design

We used data from a large multi-center study encompassing 2627 phenotypically well-defined participants (aged 18-64 years) recruited from six countries spanning 17 research sites, as part of the European Network of National Schizophrenia Networks Studying Gene-Environment Interactions study. To build the diagnostic model and identify which of important factors for estimating an individual risk of FEP, we applied a binary logistic model with regularization by the least absolute shrinkage and selection operator. The model was validated employing the internal-external cross-validation approach. The model performance was assessed with the area under the receiver operating characteristic curve (AUROC), calibration, sensitivity, and specificity.

Study Results

Having included preselected 22 predictor variables, the model was able to discriminate adults with FEP and controls with high accuracy across all six countries (rangesAUROC = 0.84-0.86). Specificity (range = 73.9-78.0%) and sensitivity (range = 75.6-79.3%) were equally good, cumulatively indicating an excellent model accuracy; though, calibration slope for the diagnostic model showed a presence of some overfitting when applied specifically to participants from France, the UK, and The Netherlands.

Conclusions

The new FEP model achieved a good discrimination and good calibration across six countries with different ethnic contributions supporting its robustness and good generalizability.

Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review

BACKGROUND

Clinical prediction models are often not evaluated properly in specific settings or updated, for instance, with information from new markers. These key steps are needed such that models are fit for purpose and remain relevant in the long-term. We aimed to present an overview of methodological guidance for the evaluation (i.e., validation and impact assessment) and updating of clinical prediction models.

METHODS

We systematically searched nine databases from January 2000 to January 2022 for articles in English with methodological recommendations for the post-derivation stages of interest. Qualitative analysis was used to summarize the 70 selected guidance papers.

RESULTS

Key aspects for validation are the assessment of statistical performance using measures for discrimination (e.g., C-statistic) and calibration (e.g., calibration-in-the-large and calibration slope). For assessing impact or usefulness in clinical decision-making, recent papers advise using decision-analytic measures (e.g., the Net Benefit) over simplistic classification measures that ignore clinical consequences (e.g., accuracy, overall Net Reclassification Index). Commonly recommended methods for model updating are recalibration (i.e., adjustment of intercept or baseline hazard and/or slope), revision (i.e., re-estimation of individual predictor effects), and extension (i.e., addition of new markers). Additional methodological guidance is needed for newer types of updating (e.g., meta-model and dynamic updating) and machine learning-based models.

CONCLUSION

Substantial guidance was found for model evaluation and more conventional updating of regression-based models. An important development in model evaluation is the introduction of a decision-analytic framework for assessing clinical usefulness. Consensus is emerging on methods for model updating.

Development of an Injury Burden Prediction Model in Professional Baseball Pitchers

Background

Baseball injuries are a significant problem and have increased in incidence over the last decade. Reporting injury incidence only gives context to rate but not in relation to severity or injury time loss.

Hypothesis/Purpose

The purpose of this study was to 1) incorporate both modifiable and non-modifiable factors to develop an arm injury burden prediction model in Minor League Baseball (MiLB) pitchers; and 2) understand how the model performs separately on elbow and shoulder injury burden.

Study Design

Prospective longitudinal study.

Methods

The study was conducted from 2013 to 2019 on MiLB pitchers. Pitchers were evaluated in spring training arm for shoulder range of motion and injuries were followed throughout the season. A model to predict arm injury burden was produced using zero inflated negative binomial regression. Internal validation was performed using ten-fold cross validation. Subgroup analyses were performed for elbow and shoulder separately. Model performance was assessed with root mean square error (RMSE), model fit (R2), and calibration with 95% confidence intervals (95% CI).

Results

Two-hundred, ninety-seven pitchers (94 injuries) were included with an injury incidence of 1.15 arm injuries per 1000 athletic exposures. Median days lost to an arm injury was 58 (11, 106). The final model demonstrated good prediction ability (RMSE: 11.9 days, R2: 0.80) and a calibration slope of 0.98 (95% CI: 0.92, 1.04). A separate elbow model demonstrated weaker predictive performance (RMSE: 21.3; R2: 0.42; calibration: 1.25 [1.16, 1.34]), as did a separate shoulder model (RMSE: 17.9; R2: 0.57; calibration: 1.01 [0.92, 1.10]).

Conclusions

The injury burden prediction model demonstrated excellent performance. Caution should be advised with predictions between one to 14 days lost to arm injury. Separate elbow and shoulder prediction models demonstrated decreased performance. The inclusion of both modifiable and non-modifiable factors into a comprehensive injury burden model provides the most accurate prediction of days lost in professional pitchers.

Level of Evidence

2.

Canadian Anaphylaxis Network-Predicting Recurrence after Emergency Presentation for Allergic REaction (CAN-PREPARE): a prospective, cohort study protocol

INTRODUCTION

Anaphylaxis is a severe, potentially fatal multiorgan system manifestation of an allergic reaction. The highest incidence of anaphylaxis is in children and adolescents. Biphasic anaphylaxis (BA) is defined as the recurrence of allergic symptoms after resolution of an initial reaction. It has been reported to occur in 10%-20% of cases within 1-48 hours from the onset of the initial reaction. The dilemma for physicians is determining which patients with resolved anaphylaxis should be observed for BA and for how long. Guidelines for duration of postanaphylaxis monitoring vary, are based on limited evidence and can have unintended negative impacts on patient safety, quality of life and healthcare resources. The objectives of this study are to derive a prognostic model for BA and to develop a risk-scoring system that informs disposition decisions of children who present to emergency departments (ED) with anaphylaxis.

METHODS AND ANALYSIS

This prospective multicentre cohort study will enrol 1682 patients from seven paediatric EDs that are members of the Paediatric Emergency Research Canada network. We will enrol patients younger than 18 years of age with an allergic reaction meeting anaphylaxis diagnostic criteria. Trained ED research assistants will screen, obtain consent and prospectively collect study data. Research assistants will follow patients during their ED visit and ascertain, in conjunction with the medical team, if the patient develops BA. A standardised follow-up survey conducted following study enrolment will determine if a biphasic reaction occurred after ED disposition. Model development will conform to the broad principles of the PROGRESS (Prognosis Research Strategy) framework and reporting will follow the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis Statement.

ETHICS AND DISSEMINATION

Ethics approval has been received from all participating centres. Our dissemination plan focuses on informing clinicians, policy makers and parents of the results through publication in peer-reviewed journals and broadcasting on multiple media platforms.

TRIAL REGISTRATION NUMBER

NCT05135377.

Development and validation of a prediction model for depression in adolescents with polycystic ovary syndrome: A study protocol

Introduction

The high prevalence and severity of depression in adolescents with polycystic ovary syndrome (PCOS) is a critical health threat that must be taken seriously. The identification of high-risk groups for depression in adolescents with PCOS is essential to preventing its development and improving its prognosis. At present, the routine screening of depression in adolescents with PCOS is mainly performed using scales, and there is no early identification method for high-risk groups of PCOS depression in adolescents. It is necessary to use a warning model to identify high-risk groups for depression with PCOS in adolescents.

Methods and analysis

Model development and validation will be conducted using a retrospective study. The study will involve normal adolescent girls as the control group and adolescent PCOS patients as the experimental group. We will collect not only general factors such as individual susceptibility factors, biological factors, and psychosocial environmental factors of depression in adolescence, but will also examine the pathological factors, illness perception factors, diagnosis and treatment factors, and symptom-related factors of PCOS, as well as the outcome of depression. LASSO will be used to fit a multivariate warning model of depression risk. Data collected between January 2022 and August 2022 will be used to develop and validate the model internally, and data collected between September 2022 and December 2022 will be used for external validation. We will use the C-statistic to measure the model's discrimination, the calibration plot to measure the model's risk prediction ability for depression, and the nomogram to visualize the model.

Discussion

The ability to calculate the absolute risk of depression outcomes in adolescents with PCOS would enable early and accurate predictions of depression risk among adolescents with PCOS, and provide the basis for the formulation of depression prevention and control strategies, which have important theoretical and practical implications.

Trial registration number

[ChiCTR2100050123]; Pre-results.

Black Box Prediction Methods in Sports Medicine Deserve a Red Card for Reckless Practice: A Change of Tactics is Needed to Advance Athlete Care

There is growing interest in the role of predictive analytics in sport, where such extensive data collection provides an exciting opportunity for the development and utilisation of prediction models for medical and performance purposes. Clinical prediction models have traditionally been developed using regression-based approaches, although newer machine learning methods are becoming increasingly popular. Machine learning models are considered 'black box'. In parallel with the increase in machine learning, there is also an emergence of proprietary prediction models that have been developed by researchers with the aim of becoming commercially available. Consequently, because of the profitable nature of proprietary systems, developers are often reluctant to transparently report (or make freely available) the development and validation of their prediction algorithms; the term 'black box' also applies to these systems. The lack of transparency and unavailability of algorithms to allow implementation by others of 'black box' approaches is concerning as it prevents independent evaluation of model performance, interpretability, utility, and generalisability prior to implementation within a sports medicine and performance environment. Therefore, in this Current Opinion article, we: (1) critically examine the use of black box prediction methodology and discuss its limited applicability in sport, and (2) argue that black box methods may pose a threat to delivery and development of effective athlete care and, instead, highlight why transparency and collaboration in prediction research and product development are essential to improve the integration of prediction models into sports medicine and performance.

Simple method for identification of women at risk of gestational diabetes mellitus in Arusha urban, Tanzania

BACKGROUND

Screening for gestational diabetes mellitus in Tanzania is challenged by limited resources. Therefore, this study aimed to develop a simple method for identification of women at risk of gestational diabetes mellitus in Arusha urban, Tanzania.

METHODS

This study used data from a cross sectional study, that was conducted between March and December 2018 in Arusha District involving 468 pregnant women who were not known to have diabetes before pregnancy. Urine glucose was tested using urine multistics and blood glucose levels by Gluco-Plus™ and diagnosed in accordance with the World Health Organization's criteria. Anthropometrics were measured using standard procedures and maternal characteristics were collected through face-to-face interviews using a questionnaire with structured questions. Univariate analysis assessed individual variables association with gestational diabetes mellitus where variables with p-value of < 0.05 were included in multivariable analysis and predictors with p-value < 0.1 remained in the final model. Each variable was scored based on its estimated coefficients and risk scores were calculated by multiplying the corresponding coefficients by ten to get integers. The model's performance was assessed using c-statistic. Data were analyzed using Statistical Package for Social Science™.

RESULTS

The risk score included body fat ≥ 38%, delivery to macrosomic babies, mid-upper arm circumference ≥ 28 cm, and family history of type 2 diabetes mellitus. The score correctly identified 98% of women with gestational diabetes with an area under the receiver operating characteristic curve of 0.97 (95% CI 0.96-0.99, p < 0.001), sensitivity of 0.98, and specificity of 0.46.

CONCLUSION

The developed screening tool is highly sensitive and correctly differentiates women with and without gestational diabetes mellitus in a Tanzanian sub-population.

Evaluation of prognostic risk models for postoperative pulmonary complications in adult patients undergoing major abdominal surgery: a systematic review and international external validation cohort study

BACKGROUND

Stratifying risk of postoperative pulmonary complications after major abdominal surgery allows clinicians to modify risk through targeted interventions and enhanced monitoring. In this study, we aimed to identify and validate prognostic models against a new consensus definition of postoperative pulmonary complications.

METHODS

We did a systematic review and international external validation cohort study. The systematic review was done in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. We searched MEDLINE and Embase on March 1, 2020, for articles published in English that reported on risk prediction models for postoperative pulmonary complications following abdominal surgery. External validation of existing models was done within a prospective international cohort study of adult patients (≥18 years) undergoing major abdominal surgery. Data were collected between Jan 1, 2019, and April 30, 2019, in the UK, Ireland, and Australia. Discriminative ability and prognostic accuracy summary statistics were compared between models for the 30-day postoperative pulmonary complication rate as defined by the Standardised Endpoints in Perioperative Medicine Core Outcome Measures in Perioperative and Anaesthetic Care (StEP-COMPAC). Model performance was compared using the area under the receiver operating characteristic curve (AUROCC).

FINDINGS

In total, we identified 2903 records from our literature search; of which, 2514 (86·6%) unique records were screened, 121 (4·8%) of 2514 full texts were assessed for eligibility, and 29 unique prognostic models were identified. Nine (31·0%) of 29 models had score development reported only, 19 (65·5%) had undergone internal validation, and only four (13·8%) had been externally validated. Data to validate six eligible models were collected in the international external validation cohort study. Data from 11 591 patients were available, with an overall postoperative pulmonary complication rate of 7·8% (n=903). None of the six models showed good discrimination (defined as AUROCC ≥0·70) for identifying postoperative pulmonary complications, with the Assess Respiratory Risk in Surgical Patients in Catalonia score showing the best discrimination (AUROCC 0·700 [95% CI 0·683-0·717]).

INTERPRETATION

In the pre-COVID-19 pandemic data, variability in the risk of pulmonary complications (StEP-COMPAC definition) following major abdominal surgery was poorly described by existing prognostication tools. To improve surgical safety during the COVID-19 pandemic recovery and beyond, novel risk stratification tools are required.

FUNDING

British Journal of Surgery Society.

Risk of major postoperative complications in breast reconstructive surgery with and without an acellular dermal matrix; Development of a prognostic prediction model

Introduction

Acellular dermal matrices (ADM) have been suggested to allow for different approaches and reduce the risk of postoperative complications in implant-based breast surgery. Surgeons seem to embrace ADMs around the world, although a lack of consistent evidence regarding the factors that increase the risk of major postoperative complications remains.

Purpose

To develop and internally validate a model to predict the risk of a major postoperative complication in breast reconstructive surgery with and without an ADM.

Methodology

The DBIR is an opt-out registry that holds characteristics of all breast implant surgeries in the Netherlands since 2015. Using a literature-driven preselection of predictors, multivariable mixed-effects logistic regression modelling was used to develop the prediction model.

Results

A total of 2939 breasts were eligible, of which 11% underwent an ADM-assisted procedure (single-stage or two-stage). However, 31% underwent a two-stage procedure (with or without the use of ADM). Of all breasts, 10.2% developed a major postoperative complication. Age (OR 1.01), delayed timing (OR 0.71), and two-stage technique (OR 4.46) were associated with the outcome.

Conclusion

The data suggest that ADM use was not associated with a major postoperative complication, while two-stage reconstructions were strongly associated with an increased risk of major complications. Despite these findings, ADMs are not as popular in the Netherlands as in the USA. The predictive capabilities of the developed model are mediocre to poor, but because of the above findings, we believe that the role of the two-stage technique as a golden standard should be put up for debate.

Underdiagnosis and underreporting of vertebral fractures on chest radiographs in men aged over 50 years or postmenopausal women with and without type 2 diabetes mellitus: a retrospective cohort study

Background

Osteoporotic vertebral fractures are often clinically silent and unrecognized. The present study aimed to determine whether routine chest radiographs could be a potential screening tool for identifying missed vertebral fractures in men aged over 50 years or postmenopausal women, especially those with type 2 diabetes mellitus (T2DM). In this study, we aimed to determine the prevalence of undetected vertebral fractures in elderly Chinese patients with and without T2DM.

Methods

Clinical data and chest radiographs of 567 individuals with T2DM (T2DM group) and 583 without diabetes (nondiabetic group) at a tertiary hospital in central south China were extracted from the records. Vertebral fractures were specifically looked for on chest radiographs and classified using the Genant semi-quantitative scale. Prevalence was compared between the two groups.

Results

Mean age and sex composition were comparable between the two groups. Mean weight and body mass index were significantly lower in the T2DM group. In both groups, fractures mostly involved the T11–12 and L1 vertebrae. Moderate/severe fractures were identified in 33.3% individuals in the T2DM group (31.4% men and 36.0% women) versus 23.2% individuals (20.9% men and 25.5% women) in the nondiabetic group.

Conclusions

Routine chest radiographs could be a useful screening tool for identifying asymptomatic vertebral fractures.

Trial registration The study was designed as an observational retrospective study, therefore a trial registration was not necessary.

A Unified Framework on Generalizability of Clinical Prediction Models

To be useful, clinical prediction models (CPMs) must be generalizable to patients in new settings. Evaluating generalizability of CPMs helps identify spurious relationships in data, provides insights on when they fail, and thus, improves the explainability of the CPMs. There are discontinuities in concepts related to generalizability of CPMs in the clinical research and machine learning domains. Specifically, conventional statistical reasons to explain poor generalizability such as inadequate model development for the purposes of generalizability, differences in coding of predictors and outcome between development and external datasets, measurement error, inability to measure some predictors, and missing data, all have differing and often complementary treatments, in the two domains. Much of the current machine learning literature on generalizability of CPMs is in terms of dataset shift of which several types have been described. However, little research exists to synthesize concepts in the two domains. Bridging this conceptual discontinuity in the context of CPMs can facilitate systematic development of CPMs and evaluation of their sensitivity to factors that affect generalizability. We survey generalizability and dataset shift in CPMs from both the clinical research and machine learning perspectives, and describe a unifying framework to analyze generalizability of CPMs and to explain their sensitivity to factors affecting it. Our framework leads to a set of signaling statements that can be used to characterize differences between datasets in terms of factors that affect generalizability of the CPMs.

Quantitative prediction error analysis to investigate predictive performance under predictor measurement heterogeneity at model implementation

BACKGROUND

When a predictor variable is measured in similar ways at the derivation and validation setting of a prognostic prediction model, yet both differ from the intended use of the model in practice (i.e., "predictor measurement heterogeneity"), performance of the model at implementation needs to be inferred. This study proposed an analysis to quantify the impact of anticipated predictor measurement heterogeneity.

METHODS

A simulation study was conducted to assess the impact of predictor measurement heterogeneity across validation and implementation setting in time-to-event outcome data. The use of the quantitative prediction error analysis was illustrated using an example of predicting the 6-year risk of developing type 2 diabetes with heterogeneity in measurement of the predictor body mass index.

RESULTS

In the simulation study, calibration-in-the-large of prediction models was poor and overall accuracy was reduced in all scenarios of predictor measurement heterogeneity. Model discrimination decreased with increasing random predictor measurement heterogeneity.

CONCLUSIONS

Heterogeneity of predictor measurements across settings of validation and implementation reduced predictive performance at implementation of prognostic models with a time-to-event outcome. When validating a prognostic model, the targeted clinical setting needs to be considered and analyses can be conducted to quantify the impact of anticipated predictor measurement heterogeneity on model performance at implementation.

Machine Learning and Statistical Prediction of Pitching Arm Kinetics

BACKGROUND

Over the past decade, research has attempted to elucidate the cause of throwing-related injuries in the baseball athlete. However, when considering the entire kinetic chain, full body mechanics, and pitching cycle sequencing, there are hundreds of variables that could influence throwing arm health, and there is a lack of quality investigations evaluating the relationship and influence of multiple variables on arm stress.

PURPOSE

To identify which variables have the most influence on elbow valgus torque and shoulder distraction force using a statistical model and a machine learning approach.

STUDY DESIGN

Cross-sectional study; Level of evidence, 3.

METHODS

A retrospective review was performed on baseball pitchers who underwent biomechanical evaluation at the university biomechanics laboratory. Regression models and 4 machine learning models were created for both elbow valgus torque and shoulder distraction force. All models utilized the same predictor variables, which included pitch velocity and 17 pitching mechanics.

RESULTS

The analysis included a total of 168 high school and collegiate pitchers with a mean age of 16.7 years (SD, 3.2 years) and BMI of 24.4 (SD, 1.2). For both elbow valgus torque and shoulder distraction force, the gradient boosting machine models demonstrated the smallest root mean square errors and the most precise calibrations compared with all other models. The gradient boosting model for elbow valgus torque reported the highest influence for pitch velocity (relative influence, 28.4), with 5 mechanical variables also having significant influence. The gradient boosting model for shoulder distraction force reported the highest influence for pitch velocity (relative influence, 20.4), with 6 mechanical variables also having significant influence.

CONCLUSION

The gradient boosting machine learning model demonstrated the best overall predictive performance for both elbow valgus torque and shoulder distraction force. Pitch velocity was the most influential variable in both models. However, both models also revealed that pitching mechanics, including maximum humeral rotation velocity, shoulder abduction at foot strike, and maximum shoulder external rotation, significantly influenced both elbow and shoulder stress.

CLINICAL RELEVANCE

The results of this study can be used to inform players, coaches, and clinicians on specific mechanical variables that may be optimized to mitigate elbow or shoulder stress that could lead to throwing-related injury.

Risk prediction models for discrete ordinal outcomes: Calibration and the impact of the proportional odds assumption

Calibration is a vital aspect of the performance of risk prediction models, but research in the context of ordinal outcomes is scarce. This study compared calibration measures for risk models predicting a discrete ordinal outcome, and investigated the impact of the proportional odds assumption on calibration and overfitting. We studied the multinomial, cumulative, adjacent category, continuation ratio, and stereotype logit/logistic models. To assess calibration, we investigated calibration intercepts and slopes, calibration plots, and the estimated calibration index. Using large sample simulations, we studied the performance of models for risk estimation under various conditions, assuming that the true model has either a multinomial logistic form or a cumulative logit proportional odds form. Small sample simulations were used to compare the tendency for overfitting between models. As a case study, we developed models to diagnose the degree of coronary artery disease (five categories) in symptomatic patients. When the true model was multinomial logistic, proportional odds models often yielded poor risk estimates, with calibration slopes deviating considerably from unity even on large model development datasets. The stereotype logistic model improved the calibration slope, but still provided biased risk estimates for individual patients. When the true model had a cumulative logit proportional odds form, multinomial logistic regression provided biased risk estimates, although these biases were modest. Nonproportional odds models require more parameters to be estimated from the data, and hence suffered more from overfitting. Despite larger sample size requirements, we generally recommend multinomial logistic regression for risk prediction modeling of discrete ordinal outcomes.

Clinical Prediction Models in Sports Medicine: A Guide for Clinicians and Researchers

SYNOPSIS

Participating in sport carries inherent risk of injury. Clinicians execute high-level clinical reasoning and decision making to support athletes to achieve the best outcomes. Accurately diagnosing a problem, estimating prognosis, or selecting the most suitable intervention for each athlete is challenging. Clinical prediction models are tools to assist clinicians in estimating the risk or probability of a health outcome for an individual by using data from multiple predictors. Although common in general medical literature, clinical prediction models are rare in sports medicine. The purpose of this article was to (1) describe the steps required to develop and validate (ie, evaluate) a clinical prediction model for clinical researchers, and (2) help sports medicine clinicians understand and interpret clinical prediction model studies. Using a case study to illustrate how to implement clinical prediction models in practice, we address the following issues in developing and validating a clinical prediction model: study design and data, sample size, missing data, selecting predictors, handling continuous predictors, model fitting, internal and external validation, performance measures, reporting, and model presentation. Our work builds on initiatives to improve diagnostic and prognostic clinical research, including the PROGnosis RESearch Strategy (PROGRESS) series of papers and textbook and the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement. J Orthop Sports Phys Ther 2021;51(10):517-525. doi:10.2519/jospt.2021.10697.

Determinants of Knowledge About Dietary Supplements Among Polish Internet Users: Nationwide Cross-sectional Study

Background

An accurate understanding of dietary supplements (DS) is a prerequisite for informed decisions regarding their intake. However, there is a need for studies on this understanding among the public based on validated research tools.

Objective

This study aims to assess the knowledge about DS among Polish internet users with no medical education and to identify its determinants and design an appropriate predictive model.

Methods

The study protocol was prospectively registered with a statistical analysis plan. Polish users of a web-based health service and a social networking service were administered a survey consisting of the recently developed questionnaire on knowledge about DS, the questionnaire on trust in advertising DS, the beliefs about medicines questionnaire, and several other health-related single-item measures and sociodemographic questions. The results were subjected to general linear modeling.

Results

A total of 6273 participants were included. Of the 17 yes or no questions in the questionnaire of knowledge about DS, the mean number of correct responses was 9.0 (95% CI 8.9-9.1). Health service users performed worse than social networking users by 2.3 points (95% CI 2.1-2.5) in an analysis adjusted for potential confounders. Internet users had fewer true beliefs about DS if they presented higher trust in their advertising (adjusted β=−.37; 95% CI −.39 to −.34), used DS (adjusted β=−.14; 95% CI −.17 to −.12), experienced their positive effect (adjusted β=−.16; 95% CI −.18 to −.13), were older or younger than 35 years (adjusted β=−.14; 95% CI −.17 to −.12), expressed interest in the topic of DS (adjusted β=−.10; 95% CI −.13 to −.08), reported getting information about the products from friends (adjusted β=−.13; 95% CI −.15 to −.11), and believed that medicines are harmful (adjusted β=−.12; 95% CI −.15 to −.10). The proposed 5-predictor model could explain 31.2% of the variance in knowledge about DS. The model appeared resistant to overfitting and was able to forecast most of the observed associations.

Conclusions

Polish internet users with no medical education exhibit some false beliefs regarding DS. Trusting the advertising of DS appears to conflict with knowledge about them. There is an urgent need for effective web-based educational campaigns on DS and the promotion of advertising literacy. After the proposed predictive model is externally validated, it may help identify the least informed target audience.

Predicting neonatal respiratory distress syndrome and hypoglycaemia prior to discharge: Leveraging health administrative data and machine learning

OBJECTIVES

A major challenge for hospitals and clinicians is the early identification of neonates at risk of developing adverse conditions. We develop a model based on routinely collected administrative data, which accurately predicts two common disorders among early term and preterm (<39 weeks) neonates prior to discharge.

STUDY DESIGN

The data included all inpatient live births born prior to 39 weeks (n = 154,755) occurring in the Australian state of Queensland between January 2009 and December 2015. Predictor variables included all maternal data captured in administrative records from the beginning of gestation up to, and including, the delivery, as well as neonatal data recorded at the delivery. Gradient boosted trees were used to predict neonatal respiratory distress syndrome and hypoglycaemia prior to discharge, with model performance benchmarked against a logistic regression models.

RESULTS

The gradient boosted trees model achieved very high discrimination for respiratory distress syndrome [AUC = 0.923, 95% CI (0.917, 0.928)] and good discrimination for hypoglycaemia [AUC = 0.832, 95% CI (0.827, 0.837)] in the validation data, as well as outperforming the logistic regression models.

CONCLUSION

Our study suggests that routinely collected health data have the potential to play an important role in assisting clinicians to identify neonates at risk of developing selected disorders shortly after birth. Despite achieving high levels of discrimination, many issues remain before such models can be implemented in practice, which we discuss in relation to our findings.

Protocol for development and validation of a clinical prediction model for adverse pregnancy outcomes in women with gestational diabetes

INTRODUCTION

Gestational diabetes (GDM) is a common yet highly heterogeneous condition. The ability to calculate the absolute risk of adverse pregnancy outcomes for an individual woman with GDM would allow preventative and therapeutic interventions to be delivered to women at high-risk, sparing women at low-risk from unnecessary care. The Prediction for Risk-Stratified care for women with GDM (PeRSonal GDM) study will develop, validate and evaluate the clinical utility of a prediction model for adverse pregnancy outcomes in women with GDM.

METHODS AND ANALYSIS

We undertook formative research to conceptualise and design the prediction model. Informed by these findings, we will conduct a model development and validation study using a retrospective cohort design with participant data collected as part of routine clinical care across three hospitals. The study will include all pregnancies resulting in births from 1 July 2017 to 31 December 2018 coded for a diagnosis of GDM (estimated sample size 2430 pregnancies). We will use a temporal split-sample development and validation strategy. A multivariable logistic regression model will be fitted. The performance of this model will be assessed, and the validated model will also be evaluated using decision curve analysis. Finally, we will explore modes of model presentation suited to clinical use, including electronic risk calculators.

ETHICS AND DISSEMINATION

This study was approved by the Human Research Ethics Committee of Monash Health (RES-19-0000713 L). We will disseminate results via presentations at scientific meetings and publication in peer-reviewed journals.

TRIAL REGISTRATION DETAILS

Systematic review proceeding this work was registered on PROSPERO (CRD42019115223) and the study was registered on the Australian and New Zealand Clinical Trials Registry (ACTRN12620000915954); Pre-results.

Predicting postpartum psychiatric admission using a machine learning approach

AIMS

The accurate identification of mothers at risk of postpartum psychiatric admission would allow for preventive intervention or more timely admission. We developed a prediction model to identify women at risk of postpartum psychiatric admission.

METHODS

Data included administrative health data of all inpatient live births in the Australian state of Queensland between January 2009 and October 2014. Analyses were restricted to mothers with one or more indicator of mental health problems during pregnancy (n = 75,054 births). The predictors included all maternal data up to and including the delivery, and neonatal data recorded at delivery. We used multiple machine learning methods to predict hospital admission in the 12 months following delivery in which the primary diagnosis was recorded as an ICD-10 psychotic, bipolar or depressive disorders.

RESULTS

The boosted trees algorithm produced the best performing model, predicting postpartum psychiatric admission in the validation data with good discrimination [AUC = 0.80; 95% CI = (0.76, 0.83)] and achieving good calibration. This model outperformed benchmark logistic regression model and an elastic net model. In addition to indicators of maternal metal health history, maternal and neonatal anthropometric measures and social/lifestyle factors were strong predictors.

CONCLUSION

Our results indicate the potential of a big data approach when aiming to identify mothers at risk of postpartum psychiatric admission. Mothers at risk could be followed-up and supported after neonatal discharge to either remove the need for admission or facilitate more timely admission.

External validation of models to predict the outcome of pregnancies of unknown location: a multicentre cohort study

Objective

To validate externally five approaches to predict ectopic pregnancy (EP) in pregnancies of unknown location (PUL): the M6P and M6NP risk models, the two‐step triage strategy (2ST, which incorporates M6P), the M4 risk model, and beta human chorionic gonadotropin ratio cut‐offs (BhCG‐RC).

Design

Secondary analysis of a prospective cohort study.

Setting

Eight UK early pregnancy assessment units.

Population

Women presenting with a PUL and BhCG >25 IU/l.

Methods

Women were managed using the 2ST protocol: PUL were classified as low risk of EP if presenting progesterone ≤2 nmol/l; the remaining cases returned 2 days later for triage based on M6P. EP risk ≥5% was used to classify PUL as high risk. Missing values were imputed, and predictions for the five approaches were calculated post hoc. We meta‐analysed centre‐specific results.

Main outcome measures

Discrimination, calibration and clinical utility (decision curve analysis) for predicting EP.

Results

Of 2899 eligible women, the primary analysis excluded 297 (10%) women who were lost to follow up. The area under the ROC curve for EP was 0.89 (95% CI 0.86–0.91) for M6P, 0.88 (0.86–0.90) for 2ST, 0.86 (0.83–0.88) for M6NP and 0.82 (0.78–0.85) for M4. Sensitivities for EP were 96% (M6P), 94% (2ST), 92% (N6NP), 80% (M4) and 58% (BhCG‐RC); false‐positive rates were 35%, 33%, 39%, 24% and 13%. M6P and 2ST had the best clinical utility and good overall calibration, with modest variability between centres.

Conclusions

2ST and M6P performed best for prediction and triage in PUL.

Tweetable abstract

The M6 model, as part of a two‐step triage strategy, is the best approach to characterise and triage PULs.

The M6 model, as part of a two‐step triage strategy, is the best approach to characterise and triage PULs.

Tutorial: a guide to performing polygenic risk score analyses

A polygenic score (PGS) or polygenic risk score (PRS) is an estimate of an individual's genetic liability to a trait or disease, calculated according to their genotype profile and relevant genome-wide association study (GWAS) data. While present PRSs typically explain only a small fraction of trait variance, their correlation with the single largest contributor to phenotypic variation-genetic liability-has led to the routine application of PRSs across biomedical research. Among a range of applications, PRSs are exploited to assess shared etiology between phenotypes, to evaluate the clinical utility of genetic data for complex disease and as part of experimental studies in which, for example, experiments are performed that compare outcomes (e.g., gene expression and cellular response to treatment) between individuals with low and high PRS values. As GWAS sample sizes increase and PRSs become more powerful, PRSs are set to play a key role in research and stratified medicine. However, despite the importance and growing application of PRSs, there are limited guidelines for performing PRS analyses, which can lead to inconsistency between studies and misinterpretation of results. Here, we provide detailed guidelines for performing and interpreting PRS analyses. We outline standard quality control steps, discuss different methods for the calculation of PRSs, provide an introductory online tutorial, highlight common misconceptions relating to PRS results, offer recommendations for best practice and discuss future challenges.

Risk Assessment Program of Highly Pathogenic Avian Influenza with Deep Learning Algorithm

Objectives

This study presents the development and validation of a risk assessment program of highly pathogenic avian influenza (HPAI). This program was developed by the Korean government (Animal and Plant Quarantine Agency) and a private corporation (Korea Telecom, KT), using a national database (Korean animal health integrated system, KAHIS).

Methods

Our risk assessment program was developed using the multilayer perceptron method using R Language. HPAI outbreaks on 544 poultry farms (307 with H5N6, and 237 with H5N8) that had available visit records of livestock-related vehicles amongst the 812 HPAI outbreaks that were confirmed between January 2014 and June 2017 were involved in this study.

Results

After 140,000 iterations without drop-out, a model with 3 hidden layers and 10 nodes per layer, were selected. The activation function of the model was hyperbolic tangent. Precision and recall of the test gave F1 measures of 0.41, 0.68 and 0.51, respectively, at validation. The predicted risk values were higher for the “outbreak” (average ± SD, 0.20 ± 0.31) than “non-outbreak” (0.18 ± 0.30) farms (p < 0.001).

Conclusion

The risk assessment model developed was employed during the epidemics of 2016/2017 (pilot version) and 2017/2018 (complementary version). This risk assessment model enhanced risk management activities by enabling preemptive control measures to prevent the spread of diseases.

The added value of periodontal measurements for identification of diabetes among Saudi adults

BACKGROUND

The aims of this study were to develop a prediction model for identification of individuals with diabetes based on clinical and perceived periodontal measurements; and to evaluate its added value when combined with standard diabetes screening tools.

METHODS

The study was carried out among 250 adults attending primary care clinics in Riyadh (Saudi Arabia). The study adopted a case-control approach, where diabetes status was first ascertained, and the Finnish Diabetes Risk Score (FINDRISC), Canadian Diabetes Risk questionnaire (CANRISK), and periodontal examinations were carried out afterward.

RESULTS

A periodontal prediction model (PPM) including three periodontal indicators (missing teeth, percentage of sites with pocket probing depth ≥6 mm, and mean pocket probing depth) had an area under the curve (AUC) of 0.694 (95% Confidence Interval: 0.612-0.776) and classified correctly 62.4% of participants. The FINDRISC and CANRISK tools had AUCs of 0.766 (95% CI: 0.690-0.843) and 0.821 (95% CI: 0.763-0.879), respectively. The addition of the PPM significantly improved the AUC of FINDRISC (P = 0.048) but not of CANRISK (P = 0.144), with 26.8% and 9.8% of participants correctly reclassified, respectively. Finally, decision curve analysis showed that adding the PPM to both tools would result in net benefits among patients with probability scores lower than 70%.

CONCLUSIONS

This study showed that periodontal measurements could play a role in identifying individuals with diabetes, and that addition of clinical periodontal measurements improved the performance of FINDRISC and CANRISK.

A clinical model predicting the risk of esophageal high-grade lesions in opportunistic screening: a multicenter real-world study in China

BACKGROUND AND AIMS

Prediction models for esophageal squamous cell carcinoma are not common, and no model targeting a clinical population has previously been developed and validated. We aimed to develop a prediction model for estimating the risk of high-grade esophageal lesions for application in clinical settings and to validate the performance of this model in an external population.

METHODS

The model was developed based on the results of endoscopic evaluation of 5624 outpatients in one hospital in a high-risk region in northern China and was validated using 5765 outpatients who had undergone endoscopy in another hospital in a non-high-risk region in southern China. Predictors were selected with unconditional logistic regression analysis. The Akaike information criterion was used to determine the final structure of the model. Discrimination was estimated using the area under the receiver operating characteristic curve (AUC). Calibration was assessed using a calibration plot with an intercept and slope.

RESULTS

The final prediction model contained 5 variables, including age, smoking, body mass index, dysphagia, and retrosternal pain. This model generated an AUC of 0.871 (95% confidence interval, 0.842-0.946) in the development set, with an AUC of 0.862 after bootstrapping. The 5-variable model was superior to a single age model. In the validation population, the AUC was 0.843 (95% confidence interval, 0.793-0.894). This model successfully stratified the clinical population into 3 risk groups and showed high ability for identifying concentrated groups of cases.

CONCLUSIONS

Our model for esophageal high-grade lesions has a high predictive value. It has the potential for application in clinical opportunistic screening to aid decision making for both health care professionals and individuals.

Absence of Iodine Staining Associates With Progression of Esophageal Lesions in a Prospective Endoscopic Surveillance Study in China

BACKGROUND & AIMS

Chromoendoscopy with iodine staining is used to identify esophageal squamous dysplasia and esophageal squamous cell carcinomas (ESCCs)-absence of staining indicates suspicious regions of dysplasia. However, screening detects precancerous lesions (mild and moderate dysplasia) that do not require immediate treatment; it is a challenge to which lesions are at risk for progression. We investigated the association between absence of iodine staining at chromoendoscopy screening and lesion progression using 6 years of follow-up data from a population-based randomized controlled trial in China. We then constructed and validated a model to calculate risk of progression to severe dysplasia, carcinoma in situ, or ESCC.

METHODS

We collected data from 1468 participants (45-69 years old) who were either negative for iodine staining at a baseline chromoendoscopy or found to have mild or moderate dysplasia in histologic analysis of biopsies in the Endoscopic Screening for Esophageal Cancer study in China, from January 2012 through September 2016; 788 of these participants were re-examined by endoscopy after a median interval of 4.2 years (development cohort). We investigated the association between absence of iodine staining and progression of esophageal lesions using Cox prediction models, considering corresponding baseline pathology findings and patient answers to a comprehensive questionnaire. Patients who did not receive a follow-up examination (n = 680) was used as the validation cohort; outcome events in these patients were identified by annual door to door active interviews or linkage with local electronic registry data. The primary outcome was incident esophageal severe dysplasia, carcinoma in situ, or ESCC.

RESULTS

In the development cohort, 11 lesions that did not stain with iodine but were classified as not dysplastic in the histology analysis were found to be severe dysplasia, carcinoma in situ, or ESCC at the follow-up evaluation. These lesions accounted for 39.3% of all progressed lesions (n = 28). In the validation cohort, 6 patients with lesions did not stain with iodine but were classified as not dysplastic by histology had a later diagnosis of ESCC, determined from medical records; these patients accounted for 50.0% of all patients with lesion progression (n = 12) until the closing date of this study. We developed a model based on patient age, body mass index, pathology findings, and baseline iodine staining to calculate risk for severe dysplasia, carcinoma in situ, or ESCC. It identified patients for severe dysplasia, carcinoma in situ, or ESCC in the development set with an area under the curve of 0.868 (95% CI, 0.817-0.920) and in the validation set with an area under the curve of 0.850 (95% CI, 0.748-0.952). Almost no cases would be missed if subjects determined to be high or intermediate-high risk subjects by the model were included in surveillance.

CONCLUSIONS

Absence of iodine staining at baseline chromoendoscopy identifies esophageal lesions at risk of progression with a high level of sensitivity. A model that combines results of iodine chromoendoscopy with other patient features identifies patients at risk of lesion progression with greater accuracy than histologic analysis of baseline biopsies.

Predicting disease severity and remission in juvenile idiopathic arthritis: are we getting closer?

PURPOSE OF REVIEW

To summarize current research on the prediction of severe disease or remission in children with juvenile arthritis, and define further steps needed towards developing prediction tools with sufficient accuracy for clinical use.

RECENT FINDINGS

High disease activity, poor patient-reported outcomes, ankle or wrist involvement, and a longer time from onset to the start of treatment herald a severe disease course and a low chance of remission. Other studies confirmed that age less than 7 years and positive ANA are the strongest predictors of uveitis development. Preliminary evidence suggests ultrasound findings may predict flare in patients with clinically inactive disease, and several new biomarkers show promise. A few prediction tools that combine predictors to estimate the chance of remission or a severe disease course in the medium-term to long-term have shown good accuracy when internally validated in the population in which they were developed.

SUMMARY

Promising candidate tools for predicting disease severity and long-term remission in juvenile arthritis are now available. These tools need external validation in other populations, and ideally formal trials to assess whether their use in practice improves patient outcomes. We are definitively getting closer, but we are not there yet.

Individual participant data validation of the PICNICC prediction model for febrile neutropenia

BACKGROUND

Risk-stratified approaches to managing cancer therapies and their consequent complications rely on accurate predictions to work effectively. The risk-stratified management of fever with neutropenia is one such very common area of management in paediatric practice. Such rules are frequently produced and promoted without adequate confirmation of their accuracy.

METHODS

An individual participant data meta-analytic validation of the 'Predicting Infectious ComplicatioNs In Children with Cancer' (PICNICC) prediction model for microbiologically documented infection in paediatric fever with neutropenia was undertaken. Pooled estimates were produced using random-effects meta-analysis of the area under the curve-receiver operating characteristic curve (AUC-ROC), calibration slope and ratios of expected versus observed cases (E/O).

RESULTS

The PICNICC model was poorly predictive of microbiologically documented infection (MDI) in these validation cohorts. The pooled AUC-ROC was 0.59, 95% CI 0.41 to 0.78, tau²=0, compared with derivation value of 0.72, 95% CI 0.71 to 0.76. There was poor discrimination (pooled slope estimate 0.03, 95% CI -0.19 to 0.26) and calibration in the large (pooled E/O ratio 1.48, 95% CI 0.87 to 2.1). Three different simple recalibration approaches failed to improve performance meaningfully.

CONCLUSION

This meta-analysis shows the PICNICC model should not be used at admission to predict MDI. Further work should focus on validating alternative prediction models. Validation across multiple cohorts from diverse locations is essential before widespread clinical adoption of such rules to avoid overtreating or undertreating children with fever with neutropenia.

Risk scores for type 2 diabetes mellitus in Latin America: a systematic review of population-based studies

AIM

To summarize the evidence on diabetes risk scores for Latin American populations.

METHODS

A systematic review was conducted (CRD42019122306) looking for diagnostic and prognostic models for type 2 diabetes mellitus among randomly selected adults in Latin America. Five databases (LILACS, Scopus, MEDLINE, Embase and Global Health) were searched. type 2 diabetes mellitus was defined using at least one blood biomarker and the reports needed to include information on the development and/or validation of a multivariable regression model. Risk of bias was assessed using the PROBAST guidelines.

RESULTS

Of the 1500 reports identified, 11 were studied in detail and five were included in the qualitative analysis. Two reports were from Mexico, two from Peru and one from Brazil. The number of diabetes cases varied from 48 to 207 in the derivations models, and between 29 and 582 in the validation models. The most common predictors were age, waist circumference and family history of diabetes, and only one study used oral glucose tolerance test as the outcome. The discrimination performance across studies was ~ 70% (range: 66-72%) as per the area under the receiving-operator curve, the highest metric was always the negative predictive value. Sensitivity was always higher than specificity.

CONCLUSION

There is no evidence to support the use of one risk score throughout Latin America. The development, validation and implementation of risk scores should be a research and public health priority in Latin America to improve type 2 diabetes mellitus screening and prevention.