Unexpected predictor-outcome associations in clinical prediction research: causes and solutions

Association between bronchopulmonary dysplasia severity and its risk factors and long-term outcomes in three definitions: a historical cohort study

OBJECTIVE

To compare the association of the severity categories of the 2001-National Institutes of Health (NIH), the 2018-NIH and the 2019-Jensen bronchopulmonary dysplasia (BPD) definitions with neurodevelopmental and respiratory outcomes at 2 and 5 years' corrected age (CA), and several BPD risk factors.

DESIGN

Single-centre historical cohort study with retrospective data collection.

SETTING

Infants born between 2009 and 2015 at the Amsterdam University Medical Centers, location Amsterdam Medical Center.

PATIENTS

Preterm infants born at gestational age (GA) <30 weeks and surviving up to 36 weeks' postmenstrual age.

INTERVENTIONS

Perinatal characteristics, (social) demographics and comorbidities were collected from the electronic patient records.

MAIN OUTCOME MEASURES

The primary outcomes were neurodevelopmental impairment (NDI) or late death, and respiratory morbidity at 2 and 5 years' CA. Using logistic regression and Brier scores, we investigated if the ordinal grade severity is associated with incremental increase of adverse long-term outcomes.

RESULTS

584 preterm infants (median GA: 28.1 weeks) were included and classified according to the three BPD definitions. None of the definitions showed a clear ordinal incremental increase of risk for any of the outcomes with increasing severity classification. No significant differences were found between the three BPD definitions (Brier scores 0.169-0.230). Respiratory interventions, but not GA, birth weight or small for GA, showed an ordinal relationship with BPD severity in all three BPD definitions.

CONCLUSION

The severity classification of three BPD definitions showed low accuracy of the probability forecast on NDI or late death and respiratory morbidity at 2 and 5 years' CA, with no differences between the definitions.

Machine learning in predicting outcomes for stroke patients following rehabilitation treatment: A systematic review

OBJECTIVE

This review aimed to summarize the use of machine learning for predicting the potential benefits of stroke rehabilitation treatments, to evaluate the risk of bias of predictive models, and to provide recommendations for future models.

MATERIALS AND METHODS

This systematic review was conducted in accordance with the PRISMA statement and the CHARMS checklist. The PubMed, Embase, Cochrane Library, Scopus, and CNKI databases were searched up to April 08, 2023. The PROBAST tool was used to assess the risk of bias of the included models.

RESULTS

Ten studies within 32 models met our inclusion criteria. The optimal AUC value of the included models ranged from 0.63 to 0.91, and the optimal R2 value ranged from 0.64 to 0.91. All of the included models were rated as having a high or unclear risk of bias, and most of them were downgraded due to inappropriate data sources or analysis processes.

DISCUSSION AND CONCLUSION

There remains much room for improvement in future modeling studies, such as high-quality data sources and model analysis. Reliable predictive models should be developed to improve the efficacy of rehabilitation treatment by clinicians.

External validation of six clinical models for prediction of chronic kidney disease in a German population

Background

Chronic kidney disease (CKD) is responsible for large personal health and societal burdens. Screening populations at higher risk for CKD is effective to initiate earlier treatment and decelerate disease progress. We externally validated clinical prediction models for unknown CKD that might be used in population screening.

Methods

We validated six risk models for prediction of CKD using only non-invasive parameters. Validation data came from 4,185 participants of the German Heinz-Nixdorf-Recall study (HNR), drawn in 2000 from a general population aged 45–75 years. We estimated discrimination and calibration using the full model information, and calculated the diagnostic properties applying the published scoring algorithms of the models using various thresholds for the sum of scores.

Results

The risk models used four to nine parameters. Age and hypertension were included in all models. Five out of six c-values ranged from 0.71 to 0.73, indicating fair discrimination. Positive predictive values ranged from 15 to 19%, negative predictive values were > 93% using score thresholds that resulted in values for sensitivity and specificity above 60%.

Conclusions

Most of the selected CKD prediction models show fair discrimination in a German general population. The estimated diagnostic properties indicate that the models are suitable for identifying persons at higher risk for unknown CKD without invasive procedures.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12882-022-02899-0.

Fear of movement and emotional distress as prognostic factors for disability in patients with shoulder pain: a prospective cohort study

Background

Shoulder pain is a prevalent and often long-lasting musculoskeletal disorder. The overall prognosis of shoulder pain is highly variable with 40–50% of patients reporting persistent pain 6–12 months after consulting a clinician. The evidence for psychological prognostic factors for patients with shoulder pain is inconsistent. Therefore, the objective of this study was to investigate the association between fear of movement and emotional distress at presentation and self-reported disability over one year of follow-up.

Methods

This is a prospective cohort study of consecutive patients referred to secondary outpatient care due to shoulder pain. Consenting patients underwent a physical examination and completed a comprehensive questionnaire at baseline, three months-, and one-year follow-up. Associations between baseline fear of movement (0–10) or emotional distress (1–4), respectively, and patient reported disability measured using Quick Disability of the Arm and Shoulder (QuickDASH, 0–100) over one year were analyzed with linear mixed-effects models (LMM) for repeated measures (baseline, 3 months and 1 year), adjusting for established prognostic factors.

Results

A total of 138 patients were recruited between March 2015 and January 2018, with response rates of 84.7% (n = 117) and 79.7% (n = 100) at three months and one year, respectively. Adjusted associations revealed that for every point increase in baseline fear of movement, the QuickDASH score increased (worsened) by 1.10 points (95% CI 0.2–2.0) over the follow-up year. For every point increase in baseline emotional distress, the QuickDASH score increased by 19.9 points (95% CI 13.9–25.9) from baseline over the follow-up year.

Conclusion

Higher fear of movement and emotional distress scores at baseline were significantly associated with higher disability over one year in patients with shoulder pain referred to secondary care. Our study indicates that these psychological factors affect prognosis and should be considered by clinicians and researchers working with patients with shoulder pain.

Performance of binary prediction models in high-correlation low-dimensional settings: a comparison of methods

BACKGROUND

Clinical prediction models are developed widely across medical disciplines. When predictors in such models are highly collinear, unexpected or spurious predictor-outcome associations may occur, thereby potentially reducing face-validity of the prediction model. Collinearity can be dealt with by exclusion of collinear predictors, but when there is no a priori motivation (besides collinearity) to include or exclude specific predictors, such an approach is arbitrary and possibly inappropriate.

METHODS

We compare different methods to address collinearity, including shrinkage, dimensionality reduction, and constrained optimization. The effectiveness of these methods is illustrated via simulations.

RESULTS

In the conducted simulations, no effect of collinearity was observed on predictive outcomes (AUC, R², Intercept, Slope) across methods. However, a negative effect of collinearity on the stability of predictor selection was found, affecting all compared methods, but in particular methods that perform strong predictor selection (e.g., Lasso). Methods for which the included set of predictors remained most stable under increased collinearity were Ridge, PCLR, LAELR, and Dropout.

CONCLUSIONS

Based on the results, we would recommend refraining from data-driven predictor selection approaches in the presence of high collinearity, because of the increased instability of predictor selection, even in relatively high events-per-variable settings. The selection of certain predictors over others may disproportionally give the impression that included predictors have a stronger association with the outcome than excluded predictors.

Invited Commentary: Treatment Drop-in-Making the Case for Causal Prediction

Clinical prediction models (CPMs) are often used to guide treatment initiation, with individuals at high risk offered treatment. This implicitly assumes that the probability quoted from a CPM represents the risk to an individual of an adverse outcome in absence of treatment. However, for a CPM to correctly target this estimand requires careful causal thinking. One problem that needs to be overcome is treatment drop-in: where individuals in the development data commence treatment after the time of prediction but before the outcome occurs. In this issue of the Journal, Xu et al. (Am J Epidemiol. 2021;190(10):2000-2014) use causal estimates from external data sources, such as clinical trials, to adjust CPMs for treatment drop-in. This represents a pragmatic and promising approach to address this issue, and it illustrates the value of utilizing causal inference in prediction. Building causality into the prediction pipeline can also bring other benefits. These include the ability to make and compare hypothetical predictions under different interventions, to make CPMs more explainable and transparent, and to improve model generalizability. Enriching CPMs with causal inference therefore has the potential to add considerable value to the role of prediction in healthcare.

Prediction of Cardiovascular Disease Risk Accounting for Future Initiation of Statin Treatment

Cardiovascular disease (CVD) risk-prediction models are used to identify high-risk individuals and guide statin initiation. However, these models are usually derived from individuals who might initiate statins during follow-up. We present a simple approach to address statin initiation to predict “statin-naive” CVD risk. We analyzed primary care data (2004–2017) from the UK Clinical Practice Research Datalink for 1,678,727 individuals (aged 40–85 years) without CVD or statin treatment history at study entry. We derived age- and sex-specific prediction models including conventional risk factors and a time-dependent effect of statin initiation constrained to 25% risk reduction (from trial results). We compared predictive performance and measures of public-health impact (e.g., number needed to screen to prevent 1 event) against models ignoring statin initiation. During a median follow-up of 8.9 years, 103,163 individuals developed CVD. In models accounting for (versus ignoring) statin initiation, 10-year CVD risk predictions were slightly higher; predictive performance was moderately improved. However, few individuals were reclassified to a high-risk threshold, resulting in negligible improvements in number needed to screen to prevent 1 event. In conclusion, incorporating statin effects from trial results into risk-prediction models enables statin-naive CVD risk estimation and provides moderate gains in predictive ability but had a limited impact on treatment decision-making under current guidelines in this population.

Development and validation of a risk prediction model of preterm birth for women with preterm labour symptoms (the QUIDS study): A prospective cohort study and individual participant data meta-analysis

BACKGROUND

Timely interventions in women presenting with preterm labour can substantially improve health outcomes for preterm babies. However, establishing such a diagnosis is very challenging, as signs and symptoms of preterm labour are common and can be nonspecific. We aimed to develop and externally validate a risk prediction model using concentration of vaginal fluid fetal fibronectin (quantitative fFN), in combination with clinical risk factors, for the prediction of spontaneous preterm birth and assessed its cost-effectiveness.

METHODS AND FINDINGS

Pregnant women included in the analyses were 22+0 to 34+6 weeks gestation with signs and symptoms of preterm labour. The primary outcome was spontaneous preterm birth within 7 days of quantitative fFN test. The risk prediction model was developed and internally validated in an individual participant data (IPD) meta-analysis of 5 European prospective cohort studies (2009 to 2016; 1,783 women; mean age 29.7 years; median BMI 24.8 kg/m2; 67.6% White; 11.7% smokers; 51.8% nulliparous; 10.4% with multiple pregnancy; 139 [7.8%] with spontaneous preterm birth within 7 days). The model was then externally validated in a prospective cohort study in 26 United Kingdom centres (2016 to 2018; 2,924 women; mean age 28.2 years; median BMI 25.4 kg/m2; 88.2% White; 21% smokers; 35.2% nulliparous; 3.5% with multiple pregnancy; 85 [2.9%] with spontaneous preterm birth within 7 days). The developed risk prediction model for spontaneous preterm birth within 7 days included quantitative fFN, current smoking, not White ethnicity, nulliparity, and multiple pregnancy. After internal validation, the optimism adjusted area under the curve was 0.89 (95% CI 0.86 to 0.92), and the optimism adjusted Nagelkerke R2 was 35% (95% CI 33% to 37%). On external validation in the prospective UK cohort population, the area under the curve was 0.89 (95% CI 0.84 to 0.94), and Nagelkerke R2 of 36% (95% CI: 34% to 38%). Recalibration of the model's intercept was required to ensure overall calibration-in-the-large. A calibration curve suggested close agreement between predicted and observed risks in the range of predictions 0% to 10%, but some miscalibration (underprediction) at higher risks (slope 1.24 (95% CI 1.23 to 1.26)). Despite any miscalibration, the net benefit of the model was higher than "treat all" or "treat none" strategies for thresholds up to about 15% risk. The economic analysis found the prognostic model was cost effective, compared to using qualitative fFN, at a threshold for hospital admission and treatment of ≥2% risk of preterm birth within 7 days. Study limitations include the limited number of participants who are not White and levels of missing data for certain variables in the development dataset.

CONCLUSIONS

In this study, we found that a risk prediction model including vaginal fFN concentration and clinical risk factors showed promising performance in the prediction of spontaneous preterm birth within 7 days of test and has potential to inform management decisions for women with threatened preterm labour. Further evaluation of the risk prediction model in clinical practice is required to determine whether the risk prediction model improves clinical outcomes if used in practice.

TRIAL REGISTRATION

The study was approved by the West of Scotland Research Ethics Committee (16/WS/0068). The study was registered with ISRCTN Registry (ISRCTN 41598423) and NIHR Portfolio (CPMS: 31277).

Retrospective study on the possible existence of a treatment paradox in sepsis scores in the emergency department

OBJECTIVE

The quick Sequential Organ Failure Assessment (qSOFA) is developed as a tool to identify patients with infection with increased risk of dying from sepsis in non-intensive care unit settings, like the emergency department (ED). An abnormal score may trigger the initiation of appropriate therapy to reduce that risk. This study assesses the risk of a treatment paradox: the effect of a strong predictor for mortality will be reduced if that predictor also acts as a trigger for initiating treatment to prevent mortality.

DESIGN

Retrospective analysis on data from a large observational cohort.

SETTING

ED of a tertiary medical centre in the Netherlands.

PARTICIPANTS

3178 consecutive patients with suspected infection.

PRIMARY OUTCOME

To evaluate the existence of a treatment paradox by determining the influence of baseline qSOFA on treatment decisions within the first 24 hours after admission.

RESULTS

226 (7.1%) had a qSOFA ≥2, of which 51 (22.6%) died within 30 days. Area under receiver operating characteristics of qSOFA for 30-day mortality was 0.68 (95% CI 0.61 to 0.75). Patients with a qSOFA ≥2 had higher odds of receiving any form of intensive therapy (OR 11.4 (95% CI 7.5 to 17.1)), such as aggressive fluid resuscitation (OR 8.8 95% CI 6.6 to 11.8), fast antibiotic administration (OR 8.5, 95% CI 5.7 to 12.3) or vasopressic therapy (OR 17.3, 95% CI 11.2 to 26.8), compared with patients with qSOFA <2.

CONCLUSION

In ED patients with suspected infection, a qSOFA ≥2 was associated with more intensive treatment. This could lead to inadequate prediction of 30-day mortality due to the presence of a treatment paradox.

TRIAL REGISTRATION NUMBER

6916.

Samarasekera N, Lerpiniere C, Perry LA, Moullaali TJ, J.M. Loan J, Wardlaw JM, Al‐Shahi Salman R. Association between Computed Tomographic Biomarkers of Cerebral Small Vessel Diseases and Long-Term Outcome after Spontaneous Intracerebral Hemorrhage

OBJECTIVE

A study was undertaken to assess whether cerebral small vessel disease (SVD) computed tomographic (CT) biomarkers are associated with long-term outcome after intracerebral hemorrhage.

METHODS

We performed a prospective, community-based cohort study of adults diagnosed with spontaneous intracerebral hemorrhage between June 1, 2010 and May 31, 2013. A neuroradiologist rated the diagnostic brain CT for acute intracerebral hemorrhage features and SVD biomarkers. We used severity of white matter lucencies and cerebral atrophy, and the number of lacunes to calculate the CT SVD score. We assessed the association between CT SVD biomarkers and either death, or death or dependence (modified Rankin Scale scores = 4-6) 1 year after first-ever intracerebral hemorrhage using logistic regression, adjusting for known predictors of outcome.

RESULTS

Within 1 year of intracerebral hemorrhage, 224 (56%) of 402 patients died. In separate models, 1-year death was associated with severe atrophy (adjusted odds ratio [aOR] = 2.54, 95% confidence interval [CI] = 1.44-4.49, p = 0.001) but not lacunes or severe white matter lucencies, and CT SVD sum score ≥ 1 (aOR = 2.50, 95% CI = 1.40-4.45, p = 0.002). Two hundred seventy-seven (73%) of 378 patients with modified Rankin Scale data were dead or dependent at 1 year. In separate models, 1-year death or dependence was associated with severe atrophy (aOR = 3.67, 95% CI = 1.71-7.89, p = 0.001) and severe white matter lucencies (aOR = 2.18, 95% CI = 1.06-4.51, p = 0.035) but not lacunes, and CT SVD sum score ≥ 1 (aOR = 2.81, 95% CI = 1.45-5.46, p = 0.002).

INTERPRETATION

SVD biomarkers on the diagnostic brain CT are associated with 1-year death and dependence after intracerebral hemorrhage, independent of known predictors of outcome. ANN NEUROL 2021;89:266-279.

Training Load and Injury Part 2: Questionable Research Practices Hijack the Truth and Mislead Well-Intentioned Clinicians

BACKGROUND

In this clinical commentary, we highlight issues related to conceptual foundations and methods used in training load and injury research. We focus on sources of degrees of freedom that can favor questionable research practices such as P hacking and hypothesizing after the results are known, which can undermine the trustworthiness of research findings.

CLINICAL QUESTION

Is the methodological rigor of studies in the training load and injury field sufficient to inform training-related decisions in clinical practice?

KEY RESULTS

The absence of a clear conceptual framework, causal structure, and reliable methods can promote questionable research practices, selective reporting, and confirmation bias. The fact that well-accepted training principles (eg, overload progression) are in line with some study findings may simply be a consequence of confirmation bias, resulting from cherry picking and emphasizing results that align with popular beliefs. Identifying evidence-based practical applications, grounded in high-quality research, is not currently possible. The strongest recommendation we can make for the clinician is grounded in common sense: "Do not train too much, too soon"-not because it has been confirmed by studies, but because it reflects accepted generic training principles.

CLINICAL APPLICATION

The training load and injury research field has fundamental conceptual and methodological weaknesses. Therefore, making decisions about planning and modifying training programs for injury reduction in clinical practice, based on available studies, is premature. Clinicians should continue to rely on best practice, experience, and well-known training principles, and consider the potential influence of contextual factors when planning and monitoring training loads. J Orthop Sports Phys Ther 2020;50(10):577-584. Epub 1 Aug 2020. doi:10.2519/jospt.2020.9211.

Training Load and Its Role in Injury Prevention, Part 2: Conceptual and Methodologic Pitfalls

In part 2 of this clinical commentary, we highlight the conceptual and methodologic pitfalls evident in current training-load-injury research. These limitations make these studies unsuitable for determining how to use new metrics such as acute workload, chronic workload, and their ratio for reducing injury risk. The main overarching concerns are the lack of a conceptual framework and reference models that do not allow for appropriate interpretation of the results to define a causal structure. The lack of any conceptual framework also gives investigators too many degrees of freedom, which can dramatically increase the risk of false discoveries and confirmation bias by forcing the interpretation of results toward common beliefs and accepted training principles. Specifically, we underline methodologic concerns relating to (1) measure of exposures, (2) pitfalls of using ratios, (3) training-load measures, (4) time windows, (5) discretization and reference category, (6) injury definitions, (7) unclear analyses, (8) sample size and generalizability, (9) missing data, and (10) standards and quality of reporting. Given the pitfalls of previous studies, we need to return to our practices before this research influx began, when practitioners relied on traditional training principles (eg, overload progression) and adjusted training loads based on athletes' responses. Training-load measures cannot tell us whether the variations are increasing or decreasing the injury risk; we recommend that practitioners still rely on their expert knowledge and experience.

Acute:Chronic Workload Ratio: Conceptual Issues and Fundamental Pitfalls

The number of studies examining associations between training load and injury has increased exponentially. As a result, many new measures of exposure and training-load-based prognostic factors have been created. The acute:chronic workload ratio (ACWR) is the most popular. However, when recommending the manipulation of a prognostic factor in order to alter the likelihood of an event, one assumes a causal effect. This introduces a series of additional conceptual and methodological considerations that are problematic and should be considered. Because no studies have even tried to estimate causal effects properly, manipulating ACWR in practical settings in order to change injury rates remains a conjecture and an overinterpretation of the available data. Furthermore, there are known issues with the use of ratio data and unrecognized assumptions that negatively affect the ACWR metric for use as a causal prognostic factor. ACWR use in practical settings can lead to inappropriate recommendations, because its causal relation to injury has not been established, it is an inaccurate metric (failing to normalize the numerator by the denominator even when uncoupled), it has a lack of background rationale to support its causal role, it is an ambiguous metric, and it is not consistently and unidirectionally related to injury risk. Conclusion: There is no evidence supporting the use of ACWR in training-load-management systems or for training recommendations aimed at reducing injury risk. The statistical properties of the ratio make the ACWR an inaccurate metric and complicate its interpretation for practical applications. In addition, it adds noise and creates statistical artifacts.

The extent and predictors of phthalate exposure among couples undergoing in vitro fertilization treatment

Phthalates are chemicals used as plasticizers and solvents in many consumer products but are suspected of disrupting the endocrine system and are known for their reproductive/developmental health risks. This study examined the extent and predictors of phthalate exposure among 599 couples undergoing in vitro fertilization. A questionnaire was administered to obtain sociodemographic, health, and lifestyle data, and two spot urine samples were collected from the couples to analyze eight phthalate metabolites, cotinine (COT) as a smoking index, and creatinine to adjust for urine dilution. Seven phthalate metabolites were detected in > 94% of the urine samples, and monobenzyl phthalate (MBzP) was found in 24% of the women and 26% of their male partners. Median phthalate levels were highest for monoethyl phthalate (MEP), at 333.26 μg/l in women and 290 μg/l in male partners, and lowest for MBzP, at 1.17 μg/l in women and 1.14 μg/l in male partners. Correlation coefficients of ≥ 0.4 between the women and their male partners for the eight urinary phthalate metabolites may indicate a shared source of exposure. A multivariate regression model was used to assess the association between predictors and each urinary phthalate metabolite. Several potential predictors for the variations in specific urinary phthalate metabolites were identified, including the body mass index, age, socioeconomic status, and regional distribution for both women and their male partners but with slightly different patterns. Women with a history of breastfeeding, using bottled water for cooking and storing food in plastic bags had lower MEP (8.7%), mono-(2-ethyl-5-carboxypentyl) phthalate (MECPP) (9.2%), and both mono-iso-butyl phthalate and MECPP (8.2 and 8.1%). A history of contraceptive use was associated with an increase in MECPP (8.7%), mono-(2-ethyl-5-hydroxyhexyl) phthalate (11.4%), mono-(2-ethyl-5-oxohexyl) phthalate (7.6%), and the molar sum of bis (2-ethylhexyl) phthalate metabolites (8.9%). Urinary COT levels were associated with an increase of 10-16% in all urinary metabolites in women but of only 10.5% in mono-(2-ethylhexyl) phthalate in male partners. More than 95% of the couples reported the use of cosmetics, perfumes, and personal-care products, but we were not able to find associations with urinary phthalate metabolites, perhaps due to their short half-lives. MEP levels associated with the use of household cleaning products were 11.2% higher in male partners. Our levels were generally higher than those reported elsewhere, perhaps due to different lifestyles, cultural practices, dietary habits, use of personal-care products, and governmental legislation.

PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration

Prediction models in health care use predictors to estimate for an individual the probability that a condition or disease is already present (diagnostic model) or will occur in the future (prognostic model). Publications on prediction models have become more common in recent years, and competing prediction models frequently exist for the same outcome or target population. Health care providers, guideline developers, and policymakers are often unsure which model to use or recommend, and in which persons or settings. Hence, systematic reviews of these studies are increasingly demanded, required, and performed. A key part of a systematic review of prediction models is examination of risk of bias and applicability to the intended population and setting. To help reviewers with this process, the authors developed PROBAST (Prediction model Risk Of Bias ASsessment Tool) for studies developing, validating, or updating (for example, extending) prediction models, both diagnostic and prognostic. PROBAST was developed through a consensus process involving a group of experts in the field. It includes 20 signaling questions across 4 domains (participants, predictors, outcome, and analysis). This explanation and elaboration document describes the rationale for including each domain and signaling question and guides researchers, reviewers, readers, and guideline developers in how to use them to assess risk of bias and applicability concerns. All concepts are illustrated with published examples across different topics. The latest version of the PROBAST checklist, accompanying documents, and filled-in examples can be downloaded from www.probast.org.

Adjustment for unmeasured confounding through informative priors for the confounder-outcome relation

Background

Observational studies of medical interventions or risk factors are potentially biased by unmeasured confounding. In this paper we propose a Bayesian approach by defining an informative prior for the confounder-outcome relation, to reduce bias due to unmeasured confounding. This approach was motivated by the phenomenon that the presence of unmeasured confounding may be reflected in observed confounder-outcome relations being unexpected in terms of direction or magnitude.

Methods

The approach was tested using simulation studies and was illustrated in an empirical example of the relation between LDL cholesterol levels and systolic blood pressure. In simulated data, a comparison of the estimated exposure-outcome relation was made between two frequentist multivariable linear regression models and three Bayesian multivariable linear regression models, which varied in the precision of the prior distributions. Simulated data contained information on a continuous exposure, a continuous outcome, and two continuous confounders (one considered measured one unmeasured), under various scenarios.

Results

In various scenarios the proposed Bayesian analysis with an correctly specified informative prior for the confounder-outcome relation substantially reduced bias due to unmeasured confounding and was less biased than the frequentist model with covariate adjustment for one of the two confounding variables. Also, in general the MSE was smaller for the Bayesian model with informative prior, compared to the other models.

Conclusions

As incorporating (informative) prior information for the confounder-outcome relation may reduce the bias due to unmeasured confounding, we consider this approach one of many possible sensitivity analyses of unmeasured confounding.

Electronic supplementary material

The online version of this article (10.1186/s12874-018-0634-3) contains supplementary material, which is available to authorized users.

Using marginal structural models to adjust for treatment drop-in when developing clinical prediction models

Clinical prediction models (CPMs) can inform decision making about treatment initiation, which requires predicted risks assuming no treatment is given. However, this is challenging since CPMs are usually derived using data sets where patients received treatment, often initiated postbaseline as "treatment drop-ins." This study proposes the use of marginal structural models (MSMs) to adjust for treatment drop-in. We illustrate the use of MSMs in the CPM framework through simulation studies that represent randomized controlled trials and real-world observational data and the example of statin initiation for cardiovascular disease prevention. The simulations include a binary treatment and a covariate, each recorded at two timepoints and having a prognostic effect on a binary outcome. The bias in predicted risk was examined in a model ignoring treatment, a model fitted on treatment-naïve patients (at baseline), a model including baseline treatment, and the MSM. In all simulation scenarios, all models except the MSM underestimated the risk of outcome given absence of treatment. These results were supported in the statin initiation example, which showed that ignoring statin initiation postbaseline resulted in models that significantly underestimated the risk of a cardiovascular disease event occurring within 10 years. Consequently, CPMs that do not acknowledge treatment drop-in can lead to underallocation of treatment. In conclusion, when developing CPMs to predict treatment-naïve risk, researchers should consider using MSMs to adjust for treatment drop-in, and also seek to exploit the ability of MSMs to allow estimation of individual treatment effects.

Prognosis of sciatica and back-related leg pain in primary care: the ATLAS cohort

BACKGROUND CONTEXT

Evidence is lacking on the prognosis and prognostic factors of back-related leg pain and sciatica in patients seeing their primary care physicians. This evidence could guide timely appropriate treatment and referral decisions.

PURPOSE

The present study aims to describe the prognosis and prognostic factors in primary care patients with low back-related leg pain and sciatica.

STUDY DESIGN

This is a prospective cohort study.

PATIENT SAMPLE

The present study included adults visiting their family doctor with back-related leg pain in the United Kingdom.

OUTCOME MEASURES

Information about pain, function, psychological, and clinical variables, was collected. Good outcome was defined as 30% or more reduction in disability (Roland-Morris Disability Questionnaire).

METHODS

Participants completed the questionnaires, underwent clinical assessments, received a magnetic resonance imaging scan, and were followed-up 12 months later. Mixed-effects logistic regression evaluated the prognostic value of six a priori defined variable sets (leg pain duration, pain intensity, neuropathic pain, psychological factors, clinical examination, and imaging variables). A combined model, including variables from all models, examined independent effects. The National Institute for Health Research funded the study. There are no conflicts of interest.

RESULTS

A total of 609 patients were included. At 12 months, 55% of patients improved in both the total sample and the sciatica group. For the whole cohort, longer leg pain duration (odds ratio [OR] 0.41; confidence interval [CI] 0.19-0.90), higher identity score (OR 0.70; CI 0.53-0.93), and patient's belief that the problem will last a long time (OR 0.27; CI 0.13-0.57) were the strongest independent prognostic factors negatively associated with improvement. These last two factors were similarly negatively associated with improvement in the sciatica subgroup.

CONCLUSIONS

The present study provides new evidence on the prognosis and prognostic factors of back-related leg pain and sciatica in primary care. Just over half of patients improved at 12 months. Patient's belief of recovery timescale and number of other symptoms attributed to the pain are independent prognostic factors. These factors can be used to inform and direct decisions about timing and intensity of available therapeutic options.

A Roadmap for Optimizing Asthma Care Management via Computational Approaches

Asthma affects 9% of Americans and incurs US $56 billion in cost, 439,000 hospitalizations, and 1.8 million emergency room visits annually. A small fraction of asthma patients with high vulnerabilities, severe disease, or great barriers to care consume most health care costs and resources. An effective approach is urgently needed to identify high-risk patients and intervene to improve outcomes and to reduce costs and resource use. Care management is widely used to implement tailored care plans for this purpose, but it is expensive and has limited service capacity. To maximize benefit, we should enroll only patients anticipated to have the highest costs or worst prognosis. Effective care management requires correctly identifying high-risk patients, but current patient identification approaches have major limitations. This paper pinpoints these limitations and outlines multiple machine learning techniques to address them, providing a roadmap for future research.

Bayesian joint modeling of bivariate longitudinal and competing risks data: An application to study patient-ventilator asynchronies in critical care patients

Mechanical ventilation is a common procedure of life support in intensive care. Patient-ventilator asynchronies (PVAs) occur when the timing of the ventilator cycle is not simultaneous with the timing of the patient respiratory cycle. The association between severity markers and the events death or alive discharge has been acknowledged before, however, little is known about the addition of PVAs data to the analyses. We used an index of asynchronies (AI) to measure PVAs and the SOFA (sequential organ failure assessment) score to assess overall severity. To investigate the added value of including the AI, we propose a Bayesian joint model of bivariate longitudinal and competing risks data. The longitudinal process includes a mixed effects model for the SOFA score and a mixed effects beta regression model for the AI. The survival process is defined in terms of a cause-specific hazards model for the competing risks death or alive discharge. Our model indicates that the SOFA score is strongly related to vital status. PVAs are positively associated with alive discharge but there is not enough evidence that PVAs provide a more accurate indication of death prognosis than the SOFA score alone.

Evaluation of the STarT Back Screening Tool for Prediction of Low Back Pain Intensity in an Outpatient Physical Therapy Setting

Study Design Prospective cohort study. Background Optimal management of patients with low back pain (LBP) relies on accurate prognosis of future clinical outcomes. The STarT Back Screening Tool (SBT), a prognostic index developed and validated in the primary care setting, has 3 scoring measures: SBT overall, psychosocial, and categorical scores. Objective Our study aimed to compare the predictive validity of 3 SBT measures with future pain intensity in patients receiving physical therapy for LBP. Methods Two hundred seven patients with LBP receiving physical therapy completed the SBT at initial (baseline) evaluation and were evaluated 12 weeks later for their pain intensity. Multivariable proportional odds regression was used to evaluate the associations of the various SBT measures with pain intensity at follow-up. Results Adjusting for covariates, all SBT measures were positively and significantly associated with the odds of greater pain intensity at follow-up evaluation (P<.01). Adding SBT psychosocial scores to a covariate-only model improved its predictive accuracy (concordance statistic increase, 0.03; 95% confidence interval: 0.01, 0.09), while improvements in prediction were smaller or negligible with the SBT overall and categorical scores (concordance statistic increase, 0.02 and 0.007, respectively). In mutually adjusted analyses, SBT psychosocial scores added incremental predictive value over SBT overall scores in predicting future pain intensity (P = .03). Conclusion Among the 3 SBT measures, the SBT psychosocial subscale was a significant predictor of future pain intensity in patients with LBP and had comparable, if not better, prognostic significance compared with the SBT overall score. Level of Evidence Prognosis, level 4. J Orthop Sports Phys Ther 2017;47(4):261-267. Epub 3 Mar 2017. doi:10.2519/jospt.2017.7284.

Prediction models: the right tool for the right problem

PURPOSE OF REVIEW

Perioperative prediction models can help to improve personalized patient care by providing individual risk predictions to both patients and providers. However, the scientific literature on prediction model development and validation can be quite technical and challenging to understand. This article aims to provide the necessary insight for clinicians to assess the value of a prediction model that they intend to use in their clinical practice.

RECENT FINDINGS

Recent developments in prediction model research include the continuous development of new performance characteristics for prediction models, increasing insight into the limitations of old characteristics, as well as an improved understanding of the generalizability of prediction models to new populations and practices.

SUMMARY

Clinicians can assess the value of a prediction model for their practice by first identifying what the usage of the model will be. Second, they can recognize which performance characteristics are relevant to their assessment of the model. Finally, they need to decide whether the available scientific evidence sufficiently matches their clinical practice to proceed with implementation.

Childhood asthma prediction models: a systematic review

Early identification of children at risk of developing asthma at school age is crucial, but the usefulness of childhood asthma prediction models in clinical practice is still unclear. We systematically reviewed all existing prediction models to identify preschool children with asthma-like symptoms at risk of developing asthma at school age. Studies were included if they developed a new prediction model or updated an existing model in children aged 4 years or younger with asthma-like symptoms, with assessment of asthma done between 6 and 12 years of age. 12 prediction models were identified in four types of cohorts of preschool children: those with health-care visits, those with parent-reported symptoms, those at high risk of asthma, or children in the general population. Four basic models included non-invasive, easy-to-obtain predictors only, notably family history, allergic disease comorbidities or precursors of asthma, and severity of early symptoms. Eight extended models included additional clinical tests, mostly specific IgE determination. Some models could better predict asthma development and other models could better rule out asthma development, but the predictive performance of no single model stood out in both aspects simultaneously. This finding suggests that there is a large proportion of preschool children with wheeze for which prediction of asthma development is difficult.

Predictive Factors for Delivery within 7 Days after Successful 48-Hour Treatment of Threatened Preterm Labor

Objective The aim of this study was to assess which characteristics and results of vaginal examination are predictive for delivery within 7 days, in women with threatened preterm labor after initial treatment. Study Design A secondary analysis of a randomized controlled trial on maintenance nifedipine includes women who remained undelivered after threatened preterm labor for 48 hours. We developed one model for women with premature prelabor rupture of membranes (PPROM) and one without PPROM. The predictors were identified by backward selection. We assessed calibration and discrimination and used bootstrapping techniques to correct for potential overfitting. Results For women with PPROM (model 1), nulliparity, history of preterm birth, and vaginal bleeding were included in the multivariable analysis. For women without PPROM (model 2), maternal age, vaginal bleeding, cervical length, and fetal fibronectin (fFN) status were in the multivariable analysis. Discriminative capability was moderate to good (c-statistic 0.68; 95% confidence interval [CI] 0.60-0.77 for model 1 and 0.89; 95% CI, 0.84-0.93 for model 2). Conclusion PPROM and vaginal bleeding in the current pregnancy are relevant predictive factors in all women, as are maternal age, cervical length, and fFN in women without PPROM and nulliparity, history of preterm birth in women with PPROM.

Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration

The TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) Statement includes a 22-item checklist, which aims to improve the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. This explanation and elaboration document describes the rationale; clarifies the meaning of each item; and discusses why transparent reporting is important, with a view to assessing risk of bias and clinical usefulness of the prediction model. Each checklist item of the TRIPOD Statement is explained in detail and accompanied by published examples of good reporting. The document also provides a valuable reference of issues to consider when designing, conducting, and analyzing prediction model studies. To aid the editorial process and help peer reviewers and, ultimately, readers and systematic reviewers of prediction model studies, it is recommended that authors include a completed checklist in their submission. The TRIPOD checklist can also be downloaded from www.tripod-statement.org.

Prediction of recurrence of hypertensive disorders of pregnancy in the term period, a retrospective cohort study

OBJECTIVES

To assess the recurrence risk of term hypertensive disease of pregnancy and to determine which potential risk factors are predictive of recurrence.

STUDY DESIGN

We performed a retrospective cohort study in two secondary and one tertiary care hospitals in the Netherlands. We identified women with a hypertensive disorder in the index pregnancy and delivery after 37weeks of gestation between January 2000 and December 2002. Data were extracted from medical files and women were approached for additional information on subsequent pregnancies. Adverse outcome was defined as recurrence of a hypertensive disorder in the next subsequent pregnancy.

MAIN OUTCOME MEASURES

The absolute risk of recurrence and a prediction model containing demographic and clinical factors predictive of recurrence.

RESULTS

We identified 638 women for potential inclusion, of whom 503 could be contacted. Of these women, 312 (62%) had a subsequent pregnancy. Hypertensive disorders recurred in 120 (38%, 95% CI 33-44) women, of whom 15 (5%, 95% CI 3-7) delivered preterm. Women undergoing recurrence were more at risk to develop chronic hypertension after pregnancy (35% versus 16%, OR 2.8, 95% CI 1.5-5.3). Body mass index, non-White European origin, chronic hypertension, maximum diastolic blood pressure, no use of anticonvulsive medication and interpregnancy interval were predictors for recurrence.

CONCLUSIONS

Women with hypertensive disorders and term delivery have a substantial chance of recurrence, but a small risk of preterm delivery. A number of predictors for recurrence could be identified and women with a recurrence more often developed chronic hypertension.