Clinical prediction models: diagnosis versus prognosis

Psychosis Prognosis Predictor: A continuous and uncertainty-aware prediction of treatment outcome in first-episode psychosis

INTRODUCTION

Machine learning models have shown promising potential in individual-level outcome prediction for patients with psychosis, but also have several limitations. To address some of these limitations, we present a model that predicts multiple outcomes, based on longitudinal patient data, while integrating prediction uncertainty to facilitate more reliable clinical decision-making.

MATERIAL AND METHODS

We devised a recurrent neural network architecture incorporating long short-term memory (LSTM) units to facilitate outcome prediction by leveraging multimodal baseline variables and clinical data collected at multiple time points. To account for model uncertainty, we employed a novel fuzzy logic approach to integrate the level of uncertainty into individual predictions. We predicted antipsychotic treatment outcomes in 446 first-episode psychosis patients in the OPTiMiSE study, for six different clinical scenarios. The treatment outcome measures assessed at both week 4 and week 10 encompassed symptomatic remission, clinical global remission, and functional remission.

RESULTS

Using only baseline predictors to predict different outcomes at week 4, leave-one-site-out validation AUC ranged from 0.62 to 0.66; performance improved when clinical data from week 1 was added (AUC = 0.66-0.71). For outcome at week 10, using only baseline variables, the models achieved AUC = 0.56-0.64; using data from more time points (weeks 1, 4, and 6) improved the performance to AUC = 0.72-0.74. After incorporating prediction uncertainties and stratifying the model decisions based on model confidence, we could achieve accuracies above 0.8 for ~50% of patients in five out of the six clinical scenarios.

CONCLUSION

We constructed prediction models utilizing a recurrent neural network architecture tailored to clinical scenarios derived from a time series dataset. One crucial aspect we incorporated was the consideration of uncertainty in individual predictions, which enhances the reliability of decision-making based on the model's output. We provided evidence showcasing the significance of leveraging time series data for achieving more accurate treatment outcome prediction in the field of psychiatry.

Diagnostic models to predict structural spinal osteoarthritis on lumbar radiographs in older adults with back pain: Development and internal validation

Objective

It is difficult for health care providers to diagnose structural spinal osteoarthritis (OA), because current guidelines recommend against imaging in patients with back pain. Therefore, the aim of this study was to develop and internally validate multivariable diagnostic prediction models based on a set of clinical and demographic features to be used for the diagnosis of structural spinal OA on lumbar radiographs in older patients with back pain.

Design

Three diagnostic prediction models, for structural spinal OA on lumbar radiographs (i.e. multilevel osteophytes, multilevel disc space narrowing (DSN), and both combined), were developed and internally validated in the 'Back Complaints in Older Adults' (BACE) cohort (N ​= ​669). Model performance (i.e. overall performance, discrimination and calibration) and clinical utility (i.e. decision curve analysis) were assessed. Internal validation was performed by bootstrapping.

Results

Mean age of the cohort was 66.9 years (±7.6 years) and 59% were female. All three models included age, gender, back pain duration and duration of spinal morning stiffness as predictors. The combined model additionally included restricted lateral flexion and spinal morning stiffness severity, and exhibited the best model performance (optimism adjusted c-statistic 0.661; good calibration with intercept -0.030 and slope of 0.886) and acceptable clinical utility. The other models showed suboptimal discrimination, good calibration and acceptable decision curves.

Conclusion

All three models for structural spinal OA displayed lesuboptimal discrimination and need improvement. However, these internally validated models have potential to inform primary care clinicians about a patient with risk of having structural spinal OA on lumbar radiographs. External validation before implementation in clinical care is recommended.

Predicting Individual Hearing-Aid Preference From Self-Reported Listening Experiences in Daily Life

OBJECTIVES

The study compared the utility of two approaches for collecting real-world listening experiences to predict hearing-aid preference: a retrospective questionnaire (Speech, Spatial, and Qualities of Hearing Scale [SSQ]) and in-situ Ecological Momentary Assessment (EMA). The rationale being that each approach likely provides different and yet complementary information. In addition, it was examined how self-reported listening activity and hearing-aid data-logging can augment EMAs for individualized and contextualized hearing outcome assessments.

DESIGN

Experienced hearing-aid users (N = 40) with mild-to-moderate symmetrical sensorineural hearing loss completed the SSQ questionnaire and gave repeated EMAs for two wear periods of 2-weeks each with two different hearing-aid models that differed mainly in their noise reduction technology. The EMAs were linked to a self-reported listening activity and sound environment parameters (from hearing-aid data-logging) recorded at the time of EMA completion. Wear order was randomized by hearing-aid model. Linear mixed-effects models and Random Forest models with five-fold cross-validation were used to assess the statistical associations between listening experiences and end-of-trial preferences, and to evaluate how accurately EMAs predicted preference within individuals.

RESULTS

Only 6 of the 49 SSQ items significantly discriminated between responses made for the end-of-trial preferred versus nonpreferred hearing-aid model. For the EMAs, questions related to perception of the sound from the hearing aids were all significantly associated with preference, and these associations were strongest in EMAs completed in sound environments with predominantly low SNR and listening activities related to television, people talking, nonspecific listening, and music listening. Mean differences in listening experiences from SSQ and EMA correctly predicted preference in 71.8% and 72.5% of included participants, respectively. However, a prognostic classification of single EMAs into end-of-trial preference with a Random Forest model achieved a 93.8% accuracy when contextual information was included.

CONCLUSIONS

SSQ and EMA predicted preference equally well when considering mean differences, however, EMAs had a high prognostic classifications accuracy due to the repeated-measures nature, which make them ideal for individualized hearing outcome investigations, especially when responses are combined with contextual information about the sound environment.

Planned but ever published? A retrospective analysis of clinical prediction model studies registered on clinicaltrials.gov since 2000

OBJECTIVES

To describe the characteristics and publication outcomes of clinical prediction model studies registered on clinicaltrials.gov since 2000.

STUDY DESIGN AND SETTING

Observational studies registered on clinicaltrials.gov between January 1, 2000, and March 2, 2022, describing the development of a new clinical prediction model or the validation of an existing model for predicting individual-level prognostic or diagnostic risk were analyzed. Eligible clinicaltrials.gov records were classified by modeling study type (development, validation) and the model outcome being predicted (prognostic, diagnostic). Recorded characteristics included study status, sample size information, Medical Subject Headings, and plans to share individual participant data. Publication outcomes were analyzed by linking National Clinical Trial numbers for eligible records with PubMed abstracts.

RESULTS

Nine hundred twenty-eight records were analyzed from a possible 89,896 observational study records. Publications searches found 170 matching peer-reviewed publications for 137 clinicaltrials.gov records. The estimated proportion of records with 1 or more matching publications after accounting for time since study start was 2.8% at 2 years (95% CI: 1.7%, 3.9%), 12.3% at 5 years (9.8% to 14.9%) and 27% at 10 years (23% to 33%). Stratifying records by study start year indicated that publication proportions improved over time. Records tended to prioritize the development of new prediction models over the validation of existing models (76%; 704/928 vs. 24%; 182/928). At the time of download, 27% of records were marked as complete, 35% were still recruiting, and 14.7% had unknown status. Only 7.4% of records stated plans to share individual participant data.

CONCLUSION

Published clinical prediction model studies are only a fraction of overall research efforts, with many studies planned but not completed or published. Improving the uptake of study preregistration and follow-up will increase the visibility of planned research. Introducing additional registry features and guidance may improve the identification of clinical prediction model studies posted to clinical registries.

Decision Curve Analysis of In-Hospital Mortality Prediction Models: The Relative Value of Pre- and Intraoperative Data For Decision-Making

BACKGROUND

Clinical prediction modeling plays a pivotal part in modern clinical care, particularly in predicting the risk of in-hospital mortality. Recent modeling efforts have focused on leveraging intraoperative data sources to improve model performance. However, the individual and collective benefit of pre- and intraoperative data for clinical decision-making remains unknown. We hypothesized that pre- and intraoperative predictors contribute equally to the net benefit in a decision curve analysis (DCA) of in-hospital mortality prediction models that include pre- and intraoperative predictors.

METHODS

Data from the VitalDB database featuring a subcohort of 6043 patients were used. A total of 141 predictors for in-hospital mortality were grouped into preoperative (demographics, intervention characteristics, and laboratory measurements) and intraoperative (laboratory and monitor data, drugs, and fluids) data. Prediction models using either preoperative, intraoperative, or all data were developed with multiple methods (logistic regression, neural network, random forest, gradient boosting machine, and a stacked learner). Predictive performance was evaluated by the area under the receiver-operating characteristic curve (AUROC) and under the precision-recall curve (AUPRC). Clinical utility was examined with a DCA in the predefined risk preference range (denoted by so-called treatment threshold probabilities) between 0% and 20%.

RESULTS

AUROC performance of the prediction models ranged from 0.53 to 0.78. AUPRC values ranged from 0.02 to 0.25 (compared to the incidence of 0.09 in our dataset) and high AUPRC values resulted from prediction models based on preoperative laboratory values. A DCA of pre- and intraoperative prediction models highlighted that preoperative data provide the largest overall benefit for decision-making, whereas intraoperative values provide only limited benefit for decision-making compared to preoperative data. While preoperative demographics, comorbidities, and surgery-related data provide the largest benefit for low treatment thresholds up to 5% to 10%, preoperative laboratory measurements become the dominant source for decision support for higher thresholds.

CONCLUSIONS

When it comes to predicting in-hospital mortality and subsequent decision-making, preoperative demographics, comorbidities, and surgery-related data provide the largest benefit for clinicians with risk-averse preferences, whereas preoperative laboratory values provide the largest benefit for decision-makers with more moderate risk preferences. Our decision-analytic investigation of different predictor categories moves beyond the question of whether certain predictors provide a benefit in traditional performance metrics (eg, AUROC). It offers a nuanced perspective on for whom these predictors might be beneficial in clinical decision-making. Follow-up studies requiring larger datasets and dedicated deep-learning models to handle continuous intraoperative data are essential to examine the robustness of our results.

Risk prediction model for in-stent restenosis following PCI: a systematic review

Introduction

The morbidity and mortality rates of coronary heart disease are significant, with PCI being the primary treatment. The high incidence of ISR following PCI poses a challenge to its effectiveness. Currently, there are numerous studies on ISR risk prediction models after PCI, but the quality varies and there is still a lack of systematic evaluation and analysis.

Methods

To systematically retrieve and evaluate the risk prediction models for ISR after PCI. A comprehensive search was conducted across 9 databases from inception to March 1, 2024. The screening of literature and extraction of data were independently carried out by two investigators, utilizing the checklist for critical appraisal and data extraction for systematic reviews of prediction modeling studies (CHARMS). Additionally, the risk of bias and applicability were evaluated using the Prediction Model Risk of Bias Assessment Tool (PROBAST).

Results

A total of 17 studies with 29 models were included, with a sample size of 175-10,004 cases, and the incidence of outcome events was 5.79%-58.86%. The area under the receiver operating characteristic curve was 0.530-0.953. The top 5 predictors with high frequency were diabetes, number of diseased vessels, age, LDL-C and stent diameter. Bias risk assessment into the research of the risk of higher bias the applicability of the four study better.

Discussion

The overall risk of bias in the current ISR risk prediction model post-PCI is deemed high. Moving forward, it is imperative to enhance study design and specify the reporting process, optimize and validate the model, and enhance its performance.

Improvements Are Needed in the Adherence to the TRIPOD Statement for Clinical Prediction Models for Patients With Spinal Pain or Osteoarthritis: A Metaresearch Study

This metaresearch study aimed to evaluate the completeness of reporting of prediction model studies in patients with spinal pain or osteoarthritis (OA) in terms of adherence to the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) statement. We searched for prognostic and diagnostic prediction models in patients with spinal pain or OA in MEDLINE, Embase, Web of Science, and CINAHL. Using a standardized assessment form, we assessed the adherence to the TRIPOD of the included studies. Two independent reviewers performed the study selection and data extraction phases. We included 66 studies. Approximately 35% of the studies declared to have used the TRIPOD. The median adherence to the TRIPOD was 59% overall (interquartile range (IQR): 21.8), with the items of the methods and results sections having the worst reporting. Studies on neck pain had better adherence to the TRIPOD than studies on back pain and OA (medians of 76.5%, 59%, and 53%, respectively). External validation studies had the highest total adherence (median: 79.5%, IQR: 12.8) of all the study types. The median overall adherence was 4 points higher in studies that declared TRIPOD use than those that did not. Finally, we did not observe any improvement in adherence over the years. The adherence to the TRIPOD of prediction models in the spinal and OA fields is low, with the methods and results sections being the most poorly reported. Future studies on prediction models in spinal pain and OA should follow the TRIPOD to improve their reporting completeness. PERSPECTIVE: This article provides data about adherence to the TRIPOD statement in 66 prediction model studies for spinal pain or OA. The adherence to the TRIPOD statement was found to be low (median adherence of 59%). This inadequate reporting may negatively impact the effective use of the models in clinical practice.

Development of a diagnostic predictive model for determining child stunting in Malawi: a comparative analysis of variable selection approaches

BACKGROUND

Childhood stunting is a major indicator of child malnutrition and a focus area of Global Nutrition Targets for 2025 and Sustainable Development Goals. Risk factors for childhood stunting are well studied and well known and could be used in a risk prediction model for assessing whether a child is stunted or not. However, the selection of child stunting predictor variables is a critical step in the development and performance of any such prediction model. This paper compares the performance of child stunting diagnostic predictive models based on predictor variables selected using a set of variable selection methods.

METHODS

Firstly, we conducted a subjective review of the literature to identify determinants of child stunting in Sub-Saharan Africa. Secondly, a multivariate logistic regression model of child stunting was fitted using the identified predictors on stunting data among children aged 0-59 months in the Malawi Demographic Health Survey (MDHS 2015-16) data. Thirdly, several reduced multivariable logistic regression models were fitted depending on the predictor variables selected using seven variable selection algorithms, namely backward, forward, stepwise, random forest, Least Absolute Shrinkage and Selection Operator (LASSO), and judgmental. Lastly, for each reduced model, a diagnostic predictive model for the childhood stunting risk score, defined as the child propensity score based on derived coefficients, was calculated for each child. The prediction risk models were assessed using discrimination measures, including area under-receiver operator curve (AUROC), sensitivity and specificity.

RESULTS

The review identified 68 predictor variables of child stunting, of which 27 were available in the MDHS 2016-16 data. The common risk factors selected by all the variable selection models include household wealth index, age of the child, household size, type of birth (singleton/multiple births), and birth weight. The best cut-off point on the child stunting risk prediction model was 0.37 based on risk factors determined by the judgmental variable selection method. The model's accuracy was estimated with an AUROC value of 64% (95% CI: 60%-67%) in the test data. For children residing in urban areas, the corresponding AUROC was AUC = 67% (95% CI: 58-76%), as opposed to those in rural areas, AUC = 63% (95% CI: 59-67%).

CONCLUSION

The derived child stunting diagnostic prediction model could be useful as a first screening tool to identify children more likely to be stunted. The identified children could then receive necessary nutritional interventions.

External validation of the computer aided risk scoring system in predicting in-hospital mortality following emergency medical admissions

BACKGROUND

Clinical prediction models have the potential to improve the quality of care and enhance patient safety outcomes. A Computer-aided Risk Scoring system (CARSS) was previously developed to predict in-hospital mortality following emergency admissions based on routinely collected blood tests and vitals. We aimed to externally validate the CARSS model.

METHODS

In this retrospective external validation study, we considered all adult (≥18 years) emergency medical admissions discharged between 11/11/2020 and 11/11/2022 from The Rotherham Foundation Trust (TRFT), UK. We assessed the predictive performance of the CARSS model based on its discriminative (c-statistic) and calibration characteristics (calibration slope and calibration plots).

RESULTS

Out of 32,774 admissions, 20,422 (62.3 %) admissions were included. The TRFT sample had similar demographic characteristics to the development sample but had higher mortality (6.1 % versus 5.7 %). The CARSS model demonstrated good discrimination (c-statistic 0.87 [95 % CI 0.86-0.88]) and good calibration to the TRFT dataset (slope = 1.03 [95 % CI 0.98-1.08] intercept = 0 [95 % CI -0.06-0.07]) after re-calibrating for differences in baseline mortality (intercept = 0.96 [95 % CI 0.90-1.03] before re-calibration).

CONCLUSION

In summary, the CARSS model is externally validated after correcting the baseline risk of death between development and validation datasets. External validation of the CARSS model showed that it under-predicted in-hospital mortality. Re-calibration of this model showed adequate performance in the TRFT dataset.

Prediction models for the risk of ventilator-associated pneumonia in patients on mechanical ventilation: A systematic review and meta-analysis

BACKGROUND

Identifying patients at risk of ventilator-associated pneumonia through prediction models can facilitate medical decision-making. Our objective was to evaluate the current models for ventilator-associated pneumonia in patients with mechanical ventilation.

METHODS

Nine databases systematically retrieved from establishment to March 6, 2024. Two independent reviewers performed study selection, data extraction, and quality assessment, respectively. The Prediction Model Risk of Bias Assessment Tool was used to evaluate the risk of model bias and applicability. Stata 17.0 was used to conduct a meta-analysis of discrimination of model validation.

RESULTS

The total of 34 studies were included, with reported 52 prediction models. The most frequent predictors in the models were mechanical ventilation duration, length of intensive care unit stay, and age. Each study was essentially considered having a high risk of bias. A meta-analysis of 17 studies containing 33 models with validation was performed with a pooled area under the receiver-operating curve of 0.80 (95% confidence interval: 0.78-0.83).

CONCLUSIONS

Despite the relatively excellent performance of the models, there is a high risk of bias of the model development process. Enhancing the methodological quality, especially the external validation, practical application, and optimization of the models need urgent attention.

Rehabilitation in subjects with frozen shoulder: a survey of current (2023) clinical practice of Italian physiotherapists

OBJECTIVE

Frozen Shoulder (FS) is a musculoskeletal pathology that leads to disability, functional decline, and a worsening in quality of life. Physiotherapists are the primary professionals involved in the treatment of FS, and it is essential to determine if their practice aligns with evidence-based suggestions.

AIM

The aim is to assess the knowledge, skills, and operational strategies of Italian physiotherapists regarding FS and compare them with the existing literature.

METHODS

A web-based, anonymous, and voluntary cross-sectional survey was developed and administered to Italian physiotherapists to evaluate their clinical practices.

RESULTS

A total of 501 physiotherapists (38.5% female), completed the survey. More than half were under 35 years old (67.8%), declared working in private practice settings or being self-employed (57.1%), and were primarily engaged with musculoskeletal patients (81.8%). For subjects with FS at their first access, 21.4% identified X-rays as the most useful imaging technique to recognize pathologies beyond rehabilitation competence. In terms of general management, the majority reported working with an orthopaedic or physiatrist (47.5%) or in a multidisciplinary team (33.5%). Regarding manual therapy techniques, 63.3% of physiotherapists preferred intense degree mobilization, posterior direction, and moderate pain at the end of the range of motion for low irritable/high stiffness FS; however, there is a lack of consensus for managing very irritable/low stiffness FS. The majority of physiotherapists (57.7%) concurred that stretching improves the balance between metalloproteinase and its inhibitors. Additionally, 48.3% of physiotherapists selected mobile phone videos and messages to improve patients' compliance with exercises at home and for motivational/educational purposes.

DISCUSSION AND CONCLUSION

The clinical practices of Italian physiotherapists in FS subjects sometimes deviate from evidence-based recommendations. While some discrepancies may be attributed to the existing uncertainties in the literature regarding knowledge and management strategies for FS patients, the authors recommend a stronger adherence to evidence-based practice.

Artificial Intelligence-Enabled Electrocardiography Predicts Future Pacemaker Implantation and Adverse Cardiovascular Events

Medical advances prolonging life have led to more permanent pacemaker implants. When pacemaker implantation (PMI) is commonly caused by sick sinus syndrome or conduction disorders, predicting PMI is challenging, as patients often experience related symptoms. This study was designed to create a deep learning model (DLM) for predicting future PMI from ECG data and assess its ability to predict future cardiovascular events. In this study, a DLM was trained on a dataset of 158,471 ECGs from 42,903 academic medical center patients, with additional validation involving 25,640 medical center patients and 26,538 community hospital patients. Primary analysis focused on predicting PMI within 90 days, while all-cause mortality, cardiovascular disease (CVD) mortality, and the development of various cardiovascular conditions were addressed with secondary analysis. The study's raw ECG DLM achieved area under the curve (AUC) values of 0.870, 0.878, and 0.883 for PMI prediction within 30, 60, and 90 days, respectively, along with sensitivities exceeding 82.0% and specificities over 81.9% in the internal validation. Significant ECG features included the PR interval, corrected QT interval, heart rate, QRS duration, P-wave axis, T-wave axis, and QRS complex axis. The AI-predicted PMI group had higher risks of PMI after 90 days (hazard ratio [HR]: 7.49, 95% CI: 5.40-10.39), all-cause mortality (HR: 1.91, 95% CI: 1.74-2.10), CVD mortality (HR: 3.53, 95% CI: 2.73-4.57), and new-onset adverse cardiovascular events. External validation confirmed the model's accuracy. Through ECG analyses, our AI DLM can alert clinicians and patients to the possibility of future PMI and related mortality and cardiovascular risks, aiding in timely patient intervention.

Acceptability of data linkage to identify women at risk of postnatal complication for the development of digital risk prediction tools and interventions to better optimise postnatal care, a qualitative descriptive study design

BACKGROUND

Pregnancy acts as a cardiovascular stress test. Although many complications resolve following birth, women with hypertensive disorder of pregnancy have an increased risk of developing cardiovascular disease (CVD) long-term. Monitoring postnatal health can reduce this risk but requires better methods to identity high-risk women for timely interventions.

METHODS

Employing a qualitative descriptive study design, focus groups and/or interviews were conducted, separately engaging public contributors and clinical professionals. Diverse participants were recruited through social media convenience sampling. Semi-structured, facilitator-led discussions explored perspectives of current postnatal assessment and attitudes towards linking patient electronic healthcare data to develop digital tools for identifying postpartum women at risk of CVD. Participant perspectives were gathered using post-it notes or a facilitator scribe and analysed thematically.

RESULTS

From 27 public and seven clinical contributors, five themes regarding postnatal check expectations versus reality were developed, including 'limited resources', 'low maternal health priority', 'lack of knowledge', 'ineffective systems' and 'new mum syndrome'. Despite some concerns, all supported data linkage to identify women postnatally, targeting intervention to those at greater risk of CVD. Participants outlined potential benefits of digitalisation and risk prediction, highlighting design and communication needs for diverse communities.

CONCLUSIONS

Current health system constraints in England contribute to suboptimal postnatal care. Integrating data linkage and improving education on data and digital tools for maternal healthcare shows promise for enhanced monitoring and improved future health. Recognised for streamlining processes and risk prediction, digital tools may enable more person-centred care plans, addressing the gaps in current postnatal care practice.

Understanding random resampling techniques for class imbalance correction and their consequences on calibration and discrimination of clinical risk prediction models

OBJECTIVE

Class imbalance is sometimes considered a problem when developing clinical prediction models and assessing their performance. To address it, correction strategies involving manipulations of the training dataset, such as random undersampling or oversampling, are frequently used. The aim of this article is to illustrate the consequences of these class imbalance correction strategies on clinical prediction models' internal validity in terms of calibration and discrimination performances.

METHODS

We used both heuristic intuition and formal mathematical reasoning to characterize the relations between conditional probabilities of interest and probabilities targeted when using random undersampling or oversampling. We propose a plug-in estimator that represents a natural correction for predictions obtained from models that have been trained on artificially balanced datasets ("naïve" models). We conducted a Monte Carlo simulation with two different data generation processes and present a real-world example using data from the International Stroke Trial database to empirically demonstrate the consequences of applying random resampling techniques for class imbalance correction on calibration and discrimination (in terms of Area Under the ROC, AUC) for logistic regression and tree-based prediction models.

RESULTS

Across our simulations and in the real-world example, calibration of the naïve models was very poor. The models using the plug-in estimator generally outperformed the models relying on class imbalance correction in terms of calibration while achieving the same discrimination performance.

CONCLUSION

Random resampling techniques for class imbalance correction do not generally improve discrimination performance (i.e., AUC), and their use is hard to justify when aiming at providing calibrated predictions. Improper use of such class imbalance correction techniques can lead to suboptimal data usage and less valid risk prediction models.

The suboptimal clinical applicability of prognostic prediction models for severe postpartum hemorrhage: a meta-epidemiological study

OBJECTIVES

To systematically investigate clinical applicability of the current prognostic prediction models for severe postpartum hemorrhage (SPPH).

STUDY DESIGN AND SETTING

A meta-epidemiological study of prognostic prediction models was conducted for SPPH. A pre-designed structured questionnaire was adopted to extract the study characteristics, predictors and the outcome, modeling methods, predictive performance, the classification ability for high-risk individuals, and clinical use scenarios. The risk of bias among studies was assessed by the Prediction model Risk Of Bias ASsessment Tool (PROBAST).

RESULTS

Twenty-two studies containing 27 prediction models were included. The number of predictors in the final models varied from 3 to 53. However, one-third of the models (11) did not clearly specify the timing of predictor measurement. Calibration was found to be lacking in 10 (37.0%) models. Among the 20 models with an incidence rate of predicted outcomes below 15.0%, none of the models estimated the area under the precision-recall curve, and all reported positive predictive values were below 40.0%. Only two (7.4%) models specified the target clinical setting, while seven (25.9%) models clarified the intended timing of model use. Lastly, all 22 studies were deemed to be at high risk of bias.

CONCLUSION

Current SPPH prediction models have limited clinical applicability due to methodological flaws, including unclear predictor measurement, inadequate calibration assessment, and insufficient evaluation of classification ability. Additionally, there is a lack of clarity regarding the timing for model use, target users, and clinical settings. These limitations raise concerns about the reliability and usefulness of these models in real-world clinical practice.

Predictive Models for Assessing Patients' Response to Treatment in Metastatic Prostate Cancer: A Systematic Review

Background and objective

The treatment landscape of metastatic prostate cancer (mPCa) has evolved significantly over the past two decades. Despite this, the optimal therapy for patients with mPCa has not been determined. This systematic review identifies available predictive models that assess mPCa patients' response to treatment.

Methods

We critically reviewed MEDLINE and CENTRAL in December 2022 according to the Preferred Reporting Items for Systematic Reviews and Meta-analyses statement. Only quantitative studies in English were included with no time restrictions. The quality of the included studies was assessed using the PROBAST tool. Data were extracted following the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews criteria.

Key findings and limitations

The search identified 616 citations, of which 15 studies were included in our review. Nine of the included studies were validated internally or externally. Only one study had a low risk of bias and a low risk concerning applicability. Many studies failed to detail model performance adequately, resulting in a high risk of bias. Where reported, the models indicated good or excellent performance.

Conclusions and clinical implications

Most of the identified predictive models require additional evaluation and validation in properly designed studies before these can be implemented in clinical practice to assist with treatment decision-making for men with mPCa.

Patient summary

In this review, we evaluate studies that predict which treatments will work best for which metastatic prostate cancer patients. We found that existing studies need further improvement before these can be used by health care professionals.

Machine learning algorithms to predict outcomes in children and adolescents with COVID-19: A systematic review

BACKGROUND AND OBJECTIVES

We aimed to analyze the study designs, modeling approaches, and performance evaluation metrics in studies using machine learning techniques to develop clinical prediction models for children and adolescents with COVID-19.

METHODS

We searched four databases for articles published between 01/01/2020 and 10/25/2023, describing the development of multivariable prediction models using any machine learning technique for predicting several outcomes in children and adolescents who had COVID-19.

RESULTS

We included ten articles, six (60 % [95 % confidence interval (CI) 0.31 - 0.83]) were predictive diagnostic models and four (40% [95 % CI 0.170.69]) were prognostic models. All models were developed to predict a binary outcome (n= 10/10, 100 % [95 % CI 0.72-1]). The most frequently predicted outcome was disease detection (n=3/10, 30% [95 % CI 0.11-0.60]). The most commonly used machine learning models in the studies were tree-based (n=12/33, 36.3% [95 % CI 0.17-0.47]) and neural networks (n=9/27, 33.2% [95% CI 0.15-0.44]).

CONCLUSION

Our review revealed that attention is required to address problems including small sample sizes, inconsistent reporting practices on data preparation, biases in data sources, lack of reporting metrics such as calibration and discrimination, hyperparameters and other aspects that allow reproducibility by other researchers and might improve the methodology.

External validation of six COVID-19 prognostic models for predicting mortality risk in older populations in a hospital, primary care, and nursing home setting

OBJECTIVES

To systematically evaluate the performance of COVID-19 prognostic models and scores for mortality risk in older populations across three health-care settings: hospitals, primary care, and nursing homes.

STUDY DESIGN AND SETTING

This retrospective external validation study included 14,092 older individuals of ≥70 years of age with a clinical or polymerase chain reaction-confirmed COVID-19 diagnosis from March 2020 to December 2020. The six validation cohorts include three hospital-based (CliniCo, COVID-OLD, COVID-PREDICT), two primary care-based (Julius General Practitioners Network/Academisch network huisartsgeneeskunde/Network of Academic general Practitioners, PHARMO), and one nursing home cohort (YSIS) in the Netherlands. Based on a living systematic review of COVID-19 prediction models using Prediction model Risk Of Bias ASsessment Tool for quality and risk of bias assessment and considering predictor availability in validation cohorts, we selected six prognostic models predicting mortality risk in adults with COVID-19 infection (GAL-COVID-19 mortality, 4C Mortality Score, National Early Warning Score 2-extended model, Xie model, Wang clinical model, and CURB65 score). All six prognostic models were validated in the hospital cohorts and the GAL-COVID-19 mortality model was validated in all three healthcare settings. The primary outcome was in-hospital mortality for hospitals and 28-day mortality for primary care and nursing home settings. Model performance was evaluated in each validation cohort separately in terms of discrimination, calibration, and decision curves. An intercept update was performed in models indicating miscalibration followed by predictive performance re-evaluation.

MAIN OUTCOME MEASURE

In-hospital mortality for hospitals and 28-day mortality for primary care and nursing home setting.

RESULTS

All six prognostic models performed poorly and showed miscalibration in the older population cohorts. In the hospital settings, model performance ranged from calibration-in-the-large -1.45 to 7.46, calibration slopes 0.24-0.81, and C-statistic 0.55-0.71 with 4C Mortality Score performing as the most discriminative and well-calibrated model. Performance across health-care settings was similar for the GAL-COVID-19 model, with a calibration-in-the-large in the range of -2.35 to -0.15 indicating overestimation, calibration slopes of 0.24-0.81 indicating signs of overfitting, and C-statistic of 0.55-0.71.

CONCLUSION

Our results show that most prognostic models for predicting mortality risk performed poorly in the older population with COVID-19, in each health-care setting: hospital, primary care, and nursing home settings. Insights into factors influencing predictive model performance in the older population are needed for pandemic preparedness and reliable prognostication of health-related outcomes in this demographic.

A comparison of hyperparameter tuning procedures for clinical prediction models: A simulation study

Tuning hyperparameters, such as the regularization parameter in Ridge or Lasso regression, is often aimed at improving the predictive performance of risk prediction models. In this study, various hyperparameter tuning procedures for clinical prediction models were systematically compared and evaluated in low-dimensional data. The focus was on out-of-sample predictive performance (discrimination, calibration, and overall prediction error) of risk prediction models developed using Ridge, Lasso, Elastic Net, or Random Forest. The influence of sample size, number of predictors and events fraction on performance of the hyperparameter tuning procedures was studied using extensive simulations. The results indicate important differences between tuning procedures in calibration performance, while generally showing similar discriminative performance. The one-standard-error rule for tuning applied to cross-validation (1SE CV) often resulted in severe miscalibration. Standard non-repeated and repeated cross-validation (both 5-fold and 10-fold) performed similarly well and outperformed the other tuning procedures. Bootstrap showed a slight tendency to more severe miscalibration than standard cross-validation-based tuning procedures. Differences between tuning procedures were larger for smaller sample sizes, lower events fractions and fewer predictors. These results imply that the choice of tuning procedure can have a profound influence on the predictive performance of prediction models. The results support the application of standard 5-fold or 10-fold cross-validation that minimizes out-of-sample prediction error. Despite an increased computational burden, we found no clear benefit of repeated over non-repeated cross-validation for hyperparameter tuning. We warn against the potentially detrimental effects on model calibration of the popular 1SE CV rule for tuning prediction models in low-dimensional settings.

Cross-institution natural language processing for reliable clinical association studies: a methodological exploration

OBJECTIVES

Natural language processing (NLP) of clinical notes in electronic medical records is increasingly used to extract otherwise sparsely available patient characteristics, to assess their association with relevant health outcomes. Manual data curation is resource intensive and NLP methods make these studies more feasible. However, the methodology of using NLP methods reliably in clinical research is understudied. The objective of this study is to investigate how NLP models could be used to extract study variables (specifically exposures) to reliably conduct exposure-outcome association studies.

STUDY DESIGN AND SETTING

In a convenience sample of patients admitted to the intensive care unit of a US academic health system, multiple association studies are conducted, comparing the association estimates based on NLP-extracted vs. manually extracted exposure variables. The association studies varied in NLP model architecture (Bidirectional Encoder Decoder from Transformers, Long Short-Term Memory), training paradigm (training a new model, fine-tuning an existing external model), extracted exposures (employment status, living status, and substance use), health outcomes (having a do-not-resuscitate/intubate code, length of stay, and in-hospital mortality), missing data handling (multiple imputation vs. complete case analysis), and the application of measurement error correction (via regression calibration).

RESULTS

The study was conducted on 1,174 participants (median [interquartile range] age, 61 [50, 73] years; 60.6% male). Additionally, up to 500 discharge reports of participants from the same health system and 2,528 reports of participants from an external health system were used to train the NLP models. Substantial differences were found between the associations based on NLP-extracted and manually extracted exposures under all settings. The error in association was only weakly correlated with the overall F1 score of the NLP models.

CONCLUSION

Associations estimated using NLP-extracted exposures should be interpreted with caution. Further research is needed to set conditions for reliable use of NLP in medical association studies.

Clinical Predictive Modeling of Heart Failure: Domain Description, Models' Characteristics and Literature Review

This study attempts to identify and briefly describe the current directions in applied and theoretical clinical prediction research. Context-rich chronic heart failure syndrome (CHFS) telemedicine provides the medical foundation for this effort. In the chronic stage of heart failure, there are sudden exacerbations of syndromes with subsequent hospitalizations, which are called acute decompensation of heart failure (ADHF). These decompensations are the subject of diagnostic and prognostic predictions. The primary purpose of ADHF predictions is to clarify the current and future health status of patients and subsequently optimize therapeutic responses. We proposed a simplified discrete-state disease model as an attempt at a typical summarization of a medical subject before starting predictive modeling. The study tries also to structure the essential common characteristics of quantitative models in order to understand the issue in an application context. The last part provides an overview of prediction works in the field of CHFS. These three parts provide the reader with a comprehensive view of quantitative clinical predictive modeling in heart failure telemedicine with an emphasis on several key general aspects. The target community is medical researchers seeking to align their clinical studies with prognostic or diagnostic predictive modeling, as well as other predictive researchers. The study was written by a non-medical expert.

The Role of Artificial Intelligence in Improving Patient Outcomes and Future of Healthcare Delivery in Cardiology: A Narrative Review of the Literature

Cardiovascular diseases exert a significant burden on the healthcare system worldwide. This narrative literature review discusses the role of artificial intelligence (AI) in the field of cardiology. AI has the potential to assist healthcare professionals in several ways, such as diagnosing pathologies, guiding treatments, and monitoring patients, which can lead to improved patient outcomes and a more efficient healthcare system. Moreover, clinical decision support systems in cardiology have improved significantly over the past decade. The addition of AI to these clinical decision support systems can improve patient outcomes by processing large amounts of data, identifying subtle associations, and providing a timely, evidence-based recommendation to healthcare professionals. Lastly, the application of AI allows for personalized care by utilizing predictive models and generating patient-specific treatment plans. However, there are several challenges associated with the use of AI in healthcare. The application of AI in healthcare comes with significant cost and ethical considerations. Despite these challenges, AI will be an integral part of healthcare delivery in the near future, leading to personalized patient care, improved physician efficiency, and anticipated better outcomes.

Diagnostic prediction models to identify patients at risk for healthcare-facility-onset Clostridioides difficile: A systematic review of methodology and reporting

OBJECTIVE

To systematically review the methodology, performance, and generalizability of diagnostic models for predicting the risk of healthcare-facility-onset (HO) Clostridioides difficile infection (CDI) in adult hospital inpatients (aged ≥18 years).

BACKGROUND

CDI is the most common cause of healthcare-associated diarrhea. Prediction models that identify inpatients at risk of HO-CDI have been published; however, the quality and utility of these models remain uncertain.

METHODS

Two independent reviewers evaluated articles describing the development and/or validation of multivariable HO-CDI diagnostic models in an inpatient setting. All publication dates, languages, and study designs were considered. Model details (eg, sample size and source, outcome, and performance) were extracted from the selected studies based on the CHARMS checklist. The risk of bias was further assessed using PROBAST.

RESULTS

Of the 3,030 records evaluated, 11 were eligible for final analysis, which described 12 diagnostic models. Most studies clearly identified the predictors and outcomes but did not report how missing data were handled. The most frequent predictors across all models were advanced age, receipt of high-risk antibiotics, history of hospitalization, and history of CDI. All studies reported the area under the receiver operating characteristic curve (AUROC) as a measure of discriminatory ability. However, only 3 studies reported the model calibration results, and only 2 studies were externally validated. All of the studies had a high risk of bias.

CONCLUSION

The studies varied in their ability to predict the risk of HO-CDI. Future models will benefit from the validation on a prospective external cohort to maximize external validity.

Clinical prediction models and the multiverse of madness

BACKGROUND

Each year, thousands of clinical prediction models are developed to make predictions (e.g. estimated risk) to inform individual diagnosis and prognosis in healthcare. However, most are not reliable for use in clinical practice.

MAIN BODY

We discuss how the creation of a prediction model (e.g. using regression or machine learning methods) is dependent on the sample and size of data used to develop it-were a different sample of the same size used from the same overarching population, the developed model could be very different even when the same model development methods are used. In other words, for each model created, there exists a multiverse of other potential models for that sample size and, crucially, an individual's predicted value (e.g. estimated risk) may vary greatly across this multiverse. The more an individual's prediction varies across the multiverse, the greater the instability. We show how small development datasets lead to more different models in the multiverse, often with vastly unstable individual predictions, and explain how this can be exposed by using bootstrapping and presenting instability plots. We recommend healthcare researchers seek to use large model development datasets to reduce instability concerns. This is especially important to ensure reliability across subgroups and improve model fairness in practice.

CONCLUSIONS

Instability is concerning as an individual's predicted value is used to guide their counselling, resource prioritisation, and clinical decision making. If different samples lead to different models with very different predictions for the same individual, then this should cast doubt into using a particular model for that individual. Therefore, visualising, quantifying and reporting the instability in individual-level predictions is essential when proposing a new model.

Five critical quality criteria for artificial intelligence-based prediction models

To raise the quality of clinical artificial intelligence (AI) prediction modelling studies in the cardiovascular health domain and thereby improve their impact and relevancy, the editors for digital health, innovation, and quality standards of the European Heart Journal propose five minimal quality criteria for AI-based prediction model development and validation studies: complete reporting, carefully defined intended use of the model, rigorous validation, large enough sample size, and openness of code and software.

The relevance of competing risk adjustment in cardiovascular risk prediction models for clinical practice

BACKGROUND

Many models developed for predicting the risk of cardiovascular disease (CVD) are adjusted for the competing risk of non-CVD mortality, which has been suggested to reduce potential overestimation of cumulative incidence in populations where the risk of competing events is high. The objective was to evaluate and illustrate the clinical impact of competing risk adjustment when deriving a CVD prediction model in a high-risk population.

METHODS AND RESULTS

Individuals with established atherosclerotic CVD were included from the Utrecht Cardiovascular Cohort-Secondary Manifestations of ARTerial disease (UCC-SMART). In 8355 individuals, followed for a median of 8.2 years (IQR 4.2-12.5), two similar prediction models for the estimation of 10-year residual CVD risk were derived: with competing risk adjustment using a Fine and Gray model and without competing risk adjustment using a Cox proportional hazards model. On average, predictions were higher from the Cox model. The Cox model predictions overestimated the cumulative incidence [predicted-observed ratio 1.14 (95% CI 1.09-1.20)], which was most apparent in the highest risk quartiles and in older persons. Discrimination of both models was similar. When determining treatment eligibility on thresholds of predicted risks, more individuals would be treated based on the Cox model predictions. If, for example, individuals with a predicted risk > 20% were considered eligible for treatment, 34% of the population would be treated according to the Fine and Gray model predictions and 44% according to the Cox model predictions.

INTERPRETATION

Individual predictions from the model unadjusted for competing risks were higher, reflecting the different interpretations of both models. For models aiming to accurately predict absolute risks, especially in high-risk populations, competing risk adjustment must be considered.

Integrative bioinformatics analysis of WDHD1: a potential biomarker for pan-cancer prognosis, diagnosis, and immunotherapy

BACKGROUND

Although WD repeat and high-mobility group box DNA binding protein 1 (WDHD1) played an essential role in DNA replication, chromosome stability, and DNA damage repair, the panoramic picture of WDHD1 in human tumors remains unclear. Hence, this study aims to comprehensively characterize WDHD1 across 33 human cancers.

METHODS

Based on publicly available databases such as TCGA, GTEx, and HPA, we used a bioinformatics approach to systematically explore the genomic features and biological functions of WDHD1 in pan-cancer.

RESULTS

WDHD1 mRNA levels were significantly increased in more than 20 types of tumor tissues. Elevated WDHD1 expression was associated with significantly shorter overall survival (OS) in 10 tumors. Furthermore, in uterine corpus endometrial carcinoma (UCEC) and liver hepatocellular carcinoma (LIHC), WDHD1 expression was significantly associated with higher histological grades and pathological stages. In addition, WDHD1 had a high diagnostic value among 16 tumors (area under the ROC curve [AUC] > 0.9). Functional enrichment analyses suggested that WDHD1 probably participated in many oncogenic pathways such as E2F and MYC targets (false discovery rate [FDR] < 0.05), and it was involved in the processes of DNA replication and DNA damage repair (p.adjust < 0.05). WDHD1 expression also correlated with the half-maximal inhibitory concentrations (IC50) of rapamycin (4 out of 10 cancers) and paclitaxel (10 out of 10 cancers). Overall, WDHD1 was negatively associated with immune cell infiltration and might promote tumor immune escape. Our analysis of genomic alterations suggested that WDHD1 was altered in 1.5% of pan-cancer cohorts and the "mutation" was the predominant type of alteration. Finally, through correlation analysis, we found that WDHD1 might be closely associated with tumor heterogeneity, tumor stemness, mismatch repair (MMR), and RNA methylation modification, which were all processes associated with the tumor progression.

CONCLUSIONS

Our pan-cancer analysis of WDHD1 provides valuable insights into the genomic characterization and biological functions of WDHD1 in human cancers and offers some theoretical support for the future use of WDHD1-targeted therapies, immunotherapies, and chemotherapeutic combinations for the management of tumors.

Evidence of questionable research practices in clinical prediction models

BACKGROUND

Clinical prediction models are widely used in health and medical research. The area under the receiver operating characteristic curve (AUC) is a frequently used estimate to describe the discriminatory ability of a clinical prediction model. The AUC is often interpreted relative to thresholds, with "good" or "excellent" models defined at 0.7, 0.8 or 0.9. These thresholds may create targets that result in "hacking", where researchers are motivated to re-analyse their data until they achieve a "good" result.

METHODS

We extracted AUC values from PubMed abstracts to look for evidence of hacking. We used histograms of the AUC values in bins of size 0.01 and compared the observed distribution to a smooth distribution from a spline.

RESULTS

The distribution of 306,888 AUC values showed clear excesses above the thresholds of 0.7, 0.8 and 0.9 and shortfalls below the thresholds.

CONCLUSIONS

The AUCs for some models are over-inflated, which risks exposing patients to sub-optimal clinical decision-making. Greater modelling transparency is needed, including published protocols, and data and code sharing.

Development and evaluation of a risk prediction model for diabetes mellitus type 2 patients with vision-threatening diabetic retinopathy

Objective

This study aims to develop and evaluate a non-imaging clinical data-based nomogram for predicting the risk of vision-threatening diabetic retinopathy (VTDR) in diabetes mellitus type 2 (T2DM) patients.

Methods

Based on the baseline data of the Guangdong Shaoguan Diabetes Cohort Study conducted by the Zhongshan Ophthalmic Center (ZOC) in 2019, 2294 complete data of T2DM patients were randomly divided into a training set (n=1605) and a testing set (n=689). Independent risk factors were selected through univariate and multivariate logistic regression analysis on the training dataset, and a nomogram was constructed for predicting the risk of VTDR in T2DM patients. The model was evaluated using receiver operating characteristic (ROC) curves and area under the curve (AUC) in the training and testing datasets to assess discrimination, and Hosmer-Lemeshow test and calibration curves to assess calibration.

Results

The results of the multivariate logistic regression analysis showed that Age (OR = 0.954, 95% CI: 0.940-0.969, p = 0.000), BMI (OR = 0.942, 95% CI: 0.902-0.984, p = 0.007), systolic blood pressure (SBP) (OR =1.014, 95% CI: 1.007-1.022, p = 0.000), diabetes duration (10-15y: OR =3.126, 95% CI: 2.087-4.682, p = 0.000; >15y: OR =3.750, 95% CI: 2.362-5.954, p = 0.000), and glycated hemoglobin (HbA1C) (OR = 1.325, 95% CI: 1.221-1.438, p = 0.000) were independent risk factors for T2DM patients with VTDR. A nomogram was constructed using these variables. The model discrimination results showed an AUC of 0.7193 for the training set and 0.6897 for the testing set. The Hosmer-Lemeshow test results showed a high consistency between the predicted and observed probabilities for both the training set (Chi-square=2.2029, P=0.9742) and the testing set (Chi-square=7.6628, P=0.4671).

Conclusion

The introduction of Age, BMI, SBP, Duration, and HbA1C as variables helps to stratify the risk of T2DM patients with VTDR.

Developing prediction models to estimate the risk of two survival outcomes both occurring: A comparison of techniques

INTRODUCTION

This study considers the prediction of the time until two survival outcomes have both occurred. We compared a variety of analytical methods motivated by a typical clinical problem of multimorbidity prognosis.

METHODS

We considered five methods: product (multiply marginal risks), dual-outcome (directly model the time until both events occur), multistate models (msm), and a range of copula and frailty models. We assessed calibration and discrimination under a variety of simulated data scenarios, varying outcome prevalence, and the amount of residual correlation. The simulation focused on model misspecification and statistical power. Using data from the Clinical Practice Research Datalink, we compared model performance when predicting the risk of cardiovascular disease and type 2 diabetes both occurring.

RESULTS

Discrimination was similar for all methods. The product method was poorly calibrated in the presence of residual correlation. The msm and dual-outcome models were the most robust to model misspecification but suffered a drop in performance at small sample sizes due to overfitting, which the copula and frailty model were less susceptible to. The copula and frailty model's performance were highly dependent on the underlying data structure. In the clinical example, the product method was poorly calibrated when adjusting for 8 major cardiovascular risk factors.

DISCUSSION

We recommend the dual-outcome method for predicting the risk of two survival outcomes both occurring. It was the most robust to model misspecification, although was also the most prone to overfitting. The clinical example motivates the use of the methods considered in this study.

Imputation and missing indicators for handling missing data in the development and deployment of clinical prediction models: A simulation study

Background: In clinical prediction modelling, missing data can occur at any stage of the model pipeline; development, validation or deployment. Multiple imputation is often recommended yet challenging to apply at deployment; for example, the outcome cannot be in the imputation model, as recommended under multiple imputation. Regression imputation uses a fitted model to impute the predicted value of missing predictors from observed data, and could offer a pragmatic alternative at deployment. Moreover, the use of missing indicators has been proposed to handle informative missingness, but it is currently unknown how well this method performs in the context of clinical prediction models. Methods: We simulated data under various missing data mechanisms to compare the predictive performance of clinical prediction models developed using both imputation methods. We consider deployment scenarios where missing data is permitted or prohibited, imputation models that use or omit the outcome, and clinical prediction models that include or omit missing indicators. We assume that the missingness mechanism remains constant across the model pipeline. We also apply the proposed strategies to critical care data. Results: With complete data available at deployment, our findings were in line with existing recommendations; that the outcome should be used to impute development data when using multiple imputation and omitted under regression imputation. When missingness is allowed at deployment, omitting the outcome from the imputation model at the development was preferred. Missing indicators improved model performance in many cases but can be harmful under outcome-dependent missingness. Conclusion: We provide evidence that commonly taught principles of handling missing data via multiple imputation may not apply to clinical prediction models, particularly when data can be missing at deployment. We observed comparable predictive performance under multiple imputation and regression imputation. The performance of the missing data handling method must be evaluated on a study-by-study basis, and the most appropriate strategy for handling missing data at development should consider whether missing data are allowed at deployment. Some guidance is provided.

Systematic review of preoperative and intraoperative colorectal Anastomotic Leak Prediction Scores (ALPS)

OBJECTIVE

To systematically review preoperative and intraoperative Anastomotic Leak Prediction Scores (ALPS) and validation studies to evaluate performance and utility in surgical decision-making. Anastomotic leak (AL) is the most feared complication of colorectal surgery. Individualised leak risk could guide anastomosis and/or diverting stoma.

METHODS

Systematic search of Ovid MEDLINE and Embase databases, 30 October 2020, identified existing ALPS and validation studies. All records including >1 risk factor, used to develop new, or to validate existing models for preoperative or intraoperative use to predict colorectal AL, were selected. Data extraction followed CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies guidelines. Models were assessed for applicability for surgical decision-making and risk of bias using Prediction model Risk Of Bias ASsessment Tool.

RESULTS

34 studies were identified containing 31 individual ALPS (12 colonic/colorectal, 19 rectal) and 6 papers with validation studies only. Development dataset patient populations were heterogeneous in terms of numbers, indication for surgery, urgency and stoma inclusion. Heterogeneity precluded meta-analysis. Definitions and timeframe for AL were available in only 22 and 11 ALPS, respectively. 26/31 studies used some form of multivariable logistic regression in their modelling. Models included 3-33 individual predictors. 27/31 studies reported model discrimination performance but just 18/31 reported calibration. 15/31 ALPS were reported with external validation, 9/31 with internal validation alone and 4 published without any validation. 27/31 ALPS and every validation study were scored high risk of bias in model analysis.

CONCLUSIONS

Poor reporting practices and methodological shortcomings limit wider adoption of published ALPS. Several models appear to perform well in discriminating patients at highest AL risk but all raise concerns over risk of bias, and nearly all over wider applicability. Large-scale, precisely reported external validation studies are required.

PROSPERO REGISTRATION NUMBER

CRD42020164804.

Predictive models for predicting the risk of maternal postpartum depression: A systematic review and evaluation

OBJECTIVES

Clinical prediction models have been widely used to screen and diagnose postpartum depression (PPD). This study systematically reviews and evaluates the risk of bias and the applicability of PPD prediction models.

METHODS

A systematic search was performed in eight databases from inception to June 1, 2022. The literature was independently screened, and data were extracted by two investigators using the checklist for critical appraisal and data extraction for systematic reviews of prediction modeling studies (CHARMS). The risk of bias and applicability was assessed using the Prediction Model Risk of Bias Assessment Tool (PROBAST).

RESULTS

After the screening, 12 studies of PPD risk prediction models were included, with the area under the ROC curve of the models ranging from 0.611 to 0.937. The most-reported predictors of PPD included several aspects, including prenatal mood disorders, endocrine and hormonal influences, psychosocial aspects, the influence of family factors, and somatic illness factors. The applicability of all studies was good. However, there was some bias, mainly due to inadequate outcome events, missing data not appropriately handled, lack of model performance assessment, and overfitting of the models.

CONCLUSIONS

This systematic review and evaluation indicate that most present PPD prediction models have a high risk of bias during development and validation. Despite some models' predictive solid performance, the models' clinical practice rate is low. Therefore, future research should develop predictive models with excellent performance in all aspects and clinical applicability to better inform maternal medical decisions.

Description, Prediction, and Causation in Sport and Exercise Medicine Research: Resolving the Confusion to Improve Research Quality and Patient Outcomes

SYNOPSIS: Researchers often assign a label (such as a risk factor or predictor) to a characteristic that is statistically associated with an outcome (such as future injury). Labeling signifies that the characteristic has an established clinical value. More often than not, these labels are assigned prematurely and haphazardly. The rampant practice conflates research goals, the ultimate clinical value of the findings, and many risk factors/predictors that may not warrant the label. To address these issues and improve injury prevention research, we (1) outline the problem; (2) clarify the key differences between the research goals of description, causation, and prediction/prognosis (along with labeling conventions); (3) differentiate the clinical implications for each label; and (4) frame an appropriate scientific process to follow before applying a label. J Orthop Sports Phys Ther 2023;53(7):1-7. Epub: 26 April 2023. doi:10.2519/jospt.2023.11773.

Artificial Intelligence Applications in Cardiovascular Magnetic Resonance Imaging: Are We on the Path to Avoiding the Administration of Contrast Media?

In recent years, cardiovascular imaging examinations have experienced exponential growth due to technological innovation, and this trend is consistent with the most recent chest pain guidelines. Contrast media have a crucial role in cardiovascular magnetic resonance (CMR) imaging, allowing for more precise characterization of different cardiovascular diseases. However, contrast media have contraindications and side effects that limit their clinical application in determinant patients. The application of artificial intelligence (AI)-based techniques to CMR imaging has led to the development of non-contrast models. These AI models utilize non-contrast imaging data, either independently or in combination with clinical and demographic data, as input to generate diagnostic or prognostic algorithms. In this review, we provide an overview of the main concepts pertaining to AI, review the existing literature on non-contrast AI models in CMR, and finally, discuss the strengths and limitations of these AI models and their possible future development.

Predictive Models for Health Deterioration: Understanding Disease Pathways for Personalized Medicine

Artificial intelligence (AI) and machine learning (ML) methods are currently widely employed in medicine and healthcare. A PubMed search returns more than 100,000 articles on these topics published between 2018 and 2022 alone. Notwithstanding several recent reviews in various subfields of AI and ML in medicine, we have yet to see a comprehensive review around the methods' use in longitudinal analysis and prediction of an individual patient's health status within a personalized disease pathway. This review seeks to fill that gap. After an overview of the AI and ML methods employed in this field and of specific medical applications of models of this type, the review discusses the strengths and limitations of current studies and looks ahead to future strands of research in this field. We aim to enable interested readers to gain a detailed impression of the research currently available and accordingly plan future work around predictive models for deterioration in health status.

Recurrence of low back pain: A difficult outcome to predict. Development and validation of a multivariable prediction model for recurrence in patients recently recovered from an episode of non-specific low back pain

BACKGROUND

Recurrence of low back pain (LBP) is common. If clinicians could identify an individual's risk of recurrence, this would enhance clinical decision-making and tailored patient care.

OBJECTIVE/DESIGN

To develop and validate a simple tool to predict the probability of a recurrence of LBP by 3- or 12-months following recovery.

METHODS

Data utilised for the prediction model development came from a prospective inception cohort study of participants (n = 250) recently recovered from LBP, who had sought care from chiropractic or physiotherapy services. The outcome measure was a recurrence of activity-limiting LBP. Candidate predictor variables (e.g., basic demographics, LBP history, levels of physical activity, etc) collected at baseline were considered for inclusion in a multivariable Cox model. The model's performance was tested in a separate validation dataset of participants (n = 261) involved in a randomised controlled trial investigating exercise for the prevention of LBP recurrences.

RESULTS

The final model included the number of previous episodes, total sitting time, and level of education. In the development sample, discrimination was acceptable (Harrell's C-statistic = 0.61, 95% CI, 0.59-0.62), but in the validation sample, discrimination was poor (0.56, 95% CI, 0.54-0.58). Calibration of the model in the validation dataset was acceptable at 3 months but was less precise at 12 months.

CONCLUSION

The developed prediction model, which included number of previous episodes, total sitting time, and level of education, did not perform adequately in the validation sample to recommend its use in clinical practice. Predicting recurrence of LBP in clinical practice remains challenging.

Minimum sample size for developing a multivariable prediction model using multinomial logistic regression

AIMS

Multinomial logistic regression models allow one to predict the risk of a categorical outcome with > 2 categories. When developing such a model, researchers should ensure the number of participants (n ) is appropriate relative to the number of events (E k ) and the number of predictor parameters (p k ) for each category k. We propose three criteria to determine the minimum n required in light of existing criteria developed for binary outcomes.

PROPOSED CRITERIA

The first criterion aims to minimise the model overfitting. The second aims to minimise the difference between the observed and adjusted R 2 Nagelkerke. The third criterion aims to ensure the overall risk is estimated precisely. For criterion (i), we show the sample size must be based on the anticipated Cox-snell R 2 of distinct 'one-to-one' logistic regression models corresponding to the sub-models of the multinomial logistic regression, rather than on the overall Cox-snell R 2 of the multinomial logistic regression.

EVALUATION OF CRITERIA

We tested the performance of the proposed criteria (i) through a simulation study and found that it resulted in the desired level of overfitting. Criterion (ii) and (iii) were natural extensions from previously proposed criteria for binary outcomes and did not require evaluation through simulation.

SUMMARY

We illustrated how to implement the sample size criteria through a worked example considering the development of a multinomial risk prediction model for tumour type when presented with an ovarian mass. Code is provided for the simulation and worked example. We will embed our proposed criteria within the pmsampsize R library and Stata modules.

Risk prediction of nephropathy by integrating clinical and genetic information among adult patients with type 2 diabetes

AIMS

Diabetic nephropathy (DN) is a major healthcare challenge. We developed and internally and externally validated a risk prediction model of DN by integrating clinical factors and SNPs from genes of multiple CKD-related pathways in the Han Chinese population.

MATERIALS AND METHODS

A total of 1526 patients with type 2 diabetes were randomly allocated into derivation (n = 1019) or validation (n = 507) sets. External validation was performed with 3899 participants from the Taiwan Biobank. We selected 66 SNPs identified from literature review for building our weighted genetic risk score (wGRS). The steps for prediction model development integrating clinical and genetic information were based on the Framingham Heart Study.

RESULTS

The AUROC (95% CI) for this DN prediction model with combined clinical factors and wGRS was 0.81 (0.78, 0.84) in the derivation set. Furthermore, by directly using the information of these 66 SNPs, our final prediction model had AUROC values of 0.85 (0.82, 0.87), 0.89 (0.86, 0.91), and 0.77 (0.74, 0.80) in the derivation, internal validation, and external validation sets, respectively. Under the combined model, the results with a cutoff point of 30% showed 70.91% sensitivity, 67.84% specificity, 51.54% positive predictive value, and 82.86% negative predictive value.

CONCLUSIONS

We developed and internally and externally validated a model with clinical factors and SNPs from genes of multiple CKD-related pathways to predict DN in Taiwan. This model can be used in clinical risk management practice as a screening tool to identify persons who are genetically predisposed to DN for early intervention and prevention.

Risk prediction models for breast cancer-related lymphedema: A systematic review and meta-analysis

PURPOSE

To review and critically evaluate currently available risk prediction models for breast cancer-related lymphedema (BCRL).

METHODS

PubMed, Embase, CINAHL, Scopus, Web of Science, the Cochrane Library, CNKI, SinoMed, WangFang Data, VIP Database were searched from inception to April 1, 2022, and updated on November 8, 2022. Study selection, data extraction and quality assessment were conducted by two independent reviewers. The Prediction Model Risk of Bias Assessment Tool was used to assess the risk of bias and applicability. Meta-analysis of AUC values of model external validations was performed using Stata 17.0.

RESULTS

Twenty-one studies were included, reporting twenty-two prediction models, with the AUC or C-index ranging from 0.601 to 0.965. Only two models were externally validated, with the pooled AUC of 0.70 (n = 3, 95%CI: 0.67 to 0.74), and 0.80 (n = 3, 95%CI: 0.75 to 0.86), respectively. Most models were developed using classical regression methods, with two studies using machine learning. Predictors most frequently used in included models were radiotherapy, body mass index before surgery, number of lymph nodes dissected, and chemotherapy. All studies were judged as high overall risk of bias and poorly reported.

CONCLUSIONS

Current models for predicting BCRL showed moderate to good predictive performance. However, all models were at high risk of bias and poorly reported, and their performance is probably optimistic. None of these models is suitable for recommendation in clinical practice. Future research should focus on validating, optimizing, or developing new models in well-designed and reported studies, following the methodology guidance and reporting guidelines.

Systematic review identifies the design and methodological conduct of studies on machine learning-based prediction models

BACKGROUND AND OBJECTIVES

We sought to summarize the study design, modelling strategies, and performance measures reported in studies on clinical prediction models developed using machine learning techniques.

METHODS

We search PubMed for articles published between 01/01/2018 and 31/12/2019, describing the development or the development with external validation of a multivariable prediction model using any supervised machine learning technique. No restrictions were made based on study design, data source, or predicted patient-related health outcomes.

RESULTS

We included 152 studies, 58 (38.2% [95% CI 30.8-46.1]) were diagnostic and 94 (61.8% [95% CI 53.9-69.2]) prognostic studies. Most studies reported only the development of prediction models (n = 133, 87.5% [95% CI 81.3-91.8]), focused on binary outcomes (n = 131, 86.2% [95% CI 79.8-90.8), and did not report a sample size calculation (n = 125, 82.2% [95% CI 75.4-87.5]). The most common algorithms used were support vector machine (n = 86/522, 16.5% [95% CI 13.5-19.9]) and random forest (n = 73/522, 14% [95% CI 11.3-17.2]). Values for area under the Receiver Operating Characteristic curve ranged from 0.45 to 1.00. Calibration metrics were often missed (n = 494/522, 94.6% [95% CI 92.4-96.3]).

CONCLUSION

Our review revealed that focus is required on handling of missing values, methods for internal validation, and reporting of calibration to improve the methodological conduct of studies on machine learning-based prediction models.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO, CRD42019161764.

Clinical Prediction Models for Pancreatic Cancer in General and At-Risk Populations: A Systematic Review

INTRODUCTION

Identifying high-risk individuals using a risk prediction model could be a crucial first stage of screening pathways to improve the early detection of pancreatic cancer. A systematic review was conducted to critically evaluate the published primary literature on the development or validation of clinical risk prediction models for pancreatic cancer risk.

METHODS

MEDLINE, Embase, and Web of Science were searched for relevant articles from the inception of each database up to November 2021. Study selection and data extraction were conducted by 2 independent reviewers. The Prediction model Risk Of Bias Assessment Tool (PROBAST) was applied to assess risk of bias.

RESULTS

In total, 33 studies were included, describing 38 risk prediction models. Excluding studies with an overlapping population, this study consist of 15,848,100 participants, of which 58,313 were diagnosed with pancreatic cancer. Eight studies externally validated their model, and 13 performed internal validation. The studies described risk prediction models for pancreatic cancer in the general population (n = 14), patients with diabetes (n = 8), and individuals with gastrointestinal (and other) symptoms (symptoms included abdominal pain, unexplained weight loss, jaundice, and change in bowel habits and indigestion; n = 11). The commonly used clinical risk factors in the model were cigarette smoking (n = 27), age (n = 25), diabetes history (n = 22), chronic pancreatitis (n = 18), and body mass index (n = 14). In the 25 studies that assessed model performance, C-statistics ranged from 0.61 to 0.98. Of the 33 studies included, 6 were rated as being at a low risk of bias based on PROBAST.

DISCUSSION

Many clinical risk prediction models for pancreatic cancer had been developed for different target populations. Although low risk-of-bias studies were identified, these require external validation and implementation studies to ensure that these will benefit clinical decision making.

Serum uric acid is a predictive biomarker of incident metabolic syndrome at the Brazilian longitudinal study of adult Health (ELSA - Brasil)

AIM

To investigate whether serum uric acid (SUA) levels and hyperuricemia can be predictive biomarkers of incident metabolic syndrome(MS) among different body mass index(BMI) categories, and to investigate SUA cutoffs that best discriminate individuals with incident MS.

METHODS

We analyzed 7,789 participants without MS at baseline of ELSA-Brasil study. Logistic regression models were performed to evaluate associations between incident MS and SUA levels/hyperuricemia, expressed by odds ratios(ORs) and confidence intervals(95 % CI).

RESULTS

We found 1,646 incident MS cases after a median follow-up of 3.8[3.5-4.1] years. Incident MS was present among 8.3 % (n = 290) of participants with normal weight, 28.3 % (n = 850) with overweight, 39.8 % (n = 506) with obesity. Among incident MS participants of total sample, 33.0 % had hyperuricemia [SUA > 6.0 mg/dL (356.9 μmol/L)]. After all adjustments, SUA was independently prognostic of incident MS: for each 1 mg/dL increase in SUA the odds of incident MS were 45 % higher (OR1.45[CI95 %1.34-1.55 p <.01]). Associations were found for those presenting normal weight, overweight and obesity (OR1.43[CI95 %1.31-1.57 p <.01; OR1.22[CI95 %1.13-1.32 p <.01]; and OR1.16[CI95 %1.04-1.29 p <.05]) respectively. Hyperuricemia was independently associated with incident MS (OR1.88[CI95 %1.49-0.2.36 p <.01]). The SUA cut point level maximizing sensitivity and specificity in the discrimination of incident MS was 5.0 mg/dL.

CONCLUSIONS

SUA level is an independent predictive biomarker of incident MS at all BMI categories.

Critical appraisal of artificial intelligence-based prediction models for cardiovascular disease

The medical field has seen a rapid increase in the development of artificial intelligence (AI)-based prediction models. With the introduction of such AI-based prediction model tools and software in cardiovascular patient care, the cardiovascular researcher and healthcare professional are challenged to understand the opportunities as well as the limitations of the AI-based predictions. In this article, we present 12 critical questions for cardiovascular health professionals to ask when confronted with an AI-based prediction model. We aim to support medical professionals to distinguish the AI-based prediction models that can add value to patient care from the AI that does not.