Counterfactual prediction is not only for causal inference

Leisure-Time Physical Activity in Subjects with Metabolic-Dysfunction-Associated Steatotic Liver Disease: An All-Cause Mortality Study

Background: Metabolic-dysfunction-associated steatotic liver disease (MASLD) affects 30% of adults worldwide and is associated with obesity and cardiovascular risk factors. If left untreated, it can progress to severe liver disease. Lifestyle changes such as physical activity and weight loss help to reduce the severity and risk of mortality. This study estimated the impact of MASLD and leisure-time physical activity (LTPA) on mortality and examined how gender mediates this effect in a Southern Italian population. Methods: This work is a population-based prospective cohort study of inhabitants of Castellana Grotte (>30 years old) in Southern Italy, which began in 1985. Participants provided general health information, underwent anthropometric measurements and ultrasonography, and completed a validated questionnaire on their food intake and LTPA. The vital status was tracked through local municipalities Results: In total, 1826 participants (39% with MASLD) were enrolled in this study, drawn from 2970 eligible subjects; the mean age was 51.91 (±14.76) years and 56.2% were men. Subjects with MASLD who practiced low LTPA had a significantly higher risk of death than those who did not have MASLD and practiced high LTPA. In addition, subjects with MASLD who practiced low LTPA were about 19% less likely to survive to the age of 82 years. As regards gender, both men and women with MASLD and low LTPA showed a significant risk of death, but this was higher in women. Conclusions: The presence of MASLD, especially in women, increases the risk of death from all causes. LTPA plays a key role in the disease and reduces mortality in these individuals.

Making predictions under interventions: a case study from the PREDICT-CVD cohort in New Zealand primary care

Background

Most existing clinical prediction models do not allow predictions under interventions. Such predictions allow predicted risk under different proposed strategies to be compared and are therefore useful to support clinical decision making. We aimed to compare methodological approaches for predicting individual level cardiovascular risk under three interventions: smoking cessation, reducing blood pressure, and reducing cholesterol.

Methods

We used data from the PREDICT prospective cohort study in New Zealand to calculate cardiovascular risk in a primary care setting. We compared three strategies to estimate absolute risk under intervention: (a) conditioning on hypothetical interventions in non-causal models; (b) combining existing prediction models with causal effects estimated using observational causal inference methods; and (c) combining existing prediction models with causal effects reported in published literature.

Results

The median absolute cardiovascular risk among smokers was 3.9%; our approaches predicted that smoking cessation reduced this to a median between a non-causal estimate of 2.5% and a causal estimate of 2.8%, depending on estimation methods. For reducing blood pressure, the proposed approaches estimated a reduction of absolute risk from a median of 4.9% to a median between 3.2% and 4.5% (both derived from causal estimation). Reducing cholesterol was estimated to reduce median absolute risk from 3.1% to between 2.2% (non-causal estimate) and 2.8% (causal estimate).

Conclusions

Estimated absolute risk reductions based on non-causal methods were different to those based on causal methods, and there was substantial variation in estimates within the causal methods. Researchers wishing to estimate risk under intervention should be explicit about their causal modelling assumptions and conduct sensitivity analysis by considering a range of possible approaches.

Flexible Machine Learning Estimation of Conditional Average Treatment Effects: A Blessing and a Curse

Causal inference from observational data requires untestable identification assumptions. If these assumptions apply, machine learning methods can be used to study complex forms of causal effect heterogeneity. Recently, several machine learning methods were developed to estimate the conditional average treatment effect (ATE). If the features at hand cannot explain all heterogeneity, the individual treatment effects can seriously deviate from the conditional ATE. In this work, we demonstrate how the distributions of the individual treatment effect and the conditional ATE can differ when a causal random forest is applied. We extend the causal random forest to estimate the difference in conditional variance between treated and controls. If the distribution of the individual treatment effect equals that of the conditional ATE, this estimated difference in variance should be small. If they differ, an additional causal assumption is necessary to quantify the heterogeneity not captured by the distribution of the conditional ATE. The conditional variance of the individual treatment effect can be identified when the individual effect is independent of the outcome under no treatment given the measured features. Then, in the cases where the individual treatment effect and conditional ATE distributions differ, the extended causal random forest can appropriately estimate the variance of the individual treatment effect distribution, whereas the causal random forest fails to do so.

Should Cognitive Screening Tests Be Corrected for Age and Education? Insights From a Causal Perspective

Cognitive screening tests such as the Mini-Mental State Examination are widely used in clinical routine to predict cognitive impairment. The raw test scores are often corrected for age and education, although documented poorer discrimination performance of corrected scores has challenged this practice. Nonetheless, test correction persists, perhaps due to the seemingly counterintuitive nature of the underlying problem. We used a causal framework to inform the long-standing debate from a more intuitive angle. We illustrate and quantify the consequences of applying the age-education correction of cognitive tests on discrimination performance. In an effort to bridge theory and practical implementation, we computed differences in discrimination performance under plausible causal scenarios using Open Access Series of Imaging Studies (OASIS)-1 data. We show that when age and education are causal risk factors for cognitive impairment and independently also affect the test score, correcting test scores for age and education removes meaningful information, thereby diminishing discrimination performance.

All-Cause Mortality Risk Prediction in Older Adults with Cancer: Practical Approaches and Limitations

PURPOSE OF REVIEW

The prediction of all-cause mortality is an important component of shared decision-making across the cancer care continuum, particularly in older adults with limited life expectancy, for whom there is an increased risk of over-diagnosis and treatment.

RECENT FINDINGS

Currently, several international societies recommend the use of all-cause mortality risk prediction tools when making decisions regarding screening and treatment in geriatric oncology. Here, we review some practical aspects of the utilization of those tools and dissect the characteristics of those most employed in geriatric oncology, highlighting both their advantages and their limitations.

Validating the knowledge bank approach for personalized prediction of survival in acute myeloid leukemia: a reproducibility study

Reproducibility is not only essential for the integrity of scientific research but is also a prerequisite for model validation and refinement for the future application of predictive algorithms. However, reproducible research is becoming increasingly challenging, particularly in high-dimensional genomic data analyses with complex statistical or algorithmic techniques. Given that there are no mandatory requirements in most biomedical and statistical journals to provide the original data, analytical source code, or other relevant materials for publication, accessibility to these supplements naturally suggests a greater credibility of the published work. In this study, we performed a reproducibility assessment of the notable paper by Gerstung et al. (Nat Genet 49:332–340, 2017) by rerunning the analysis using their original code and data, which are publicly accessible. Despite an open science setting, it was challenging to reproduce the entire research project; reasons included: incomplete data and documentation, suboptimal code readability, coding errors, limited portability of intensive computing performed on a specific platform, and an R computing environment that could no longer be re-established. We learn that the availability of code and data does not guarantee transparency and reproducibility of a study; paradoxically, the source code is still liable to error and obsolescence, essentially due to methodological and computational complexity, a lack of reproducibility checking at submission, and updates for software and operating environment. The complex code may also hide problematic methodological aspects of the proposed research. Building on the experience gained, we discuss the best programming and software engineering practices that could have been employed to improve reproducibility, and propose practical criteria for the conduct and reporting of reproducibility studies for future researchers.

Predicting counterfactual risks under hypothetical treatment strategies: an application to HIV

The accuracy of a prediction algorithm depends on contextual factors that may vary across deployment settings. To address this inherent limitation of prediction, we propose an approach to counterfactual prediction based on the g-formula to predict risk across populations that differ in their distribution of treatment strategies. We apply this to predict 5-year risk of mortality among persons receiving care for HIV in the U.S. Veterans Health Administration under different hypothetical treatment strategies. First, we implement a conventional approach to develop a prediction algorithm in the observed data and show how the algorithm may fail when transported to new populations with different treatment strategies. Second, we generate counterfactual data under different treatment strategies and use it to assess the robustness of the original algorithm's performance to these differences and to develop counterfactual prediction algorithms. We discuss how estimating counterfactual risks under a particular treatment strategy is more challenging than conventional prediction as it requires the same data, methods, and unverifiable assumptions as causal inference. However, this may be required when the alternative assumption of constant treatment patterns across deployment settings is unlikely to hold and new data is not yet available to retrain the algorithm.

Methods for Evaluating Environmental Health Impacts at Different Stages of the Policy Process in Cities

PURPOSE OF REVIEW

Evaluating the environmental health impacts of urban policies is critical for developing and implementing policies that lead to more healthy and equitable cities. This article aims to (1) identify research questions commonly used when evaluating the health impacts of urban policies at different stages of the policy process, (2) describe commonly used methods, and (3) discuss challenges, opportunities, and future directions.

RECENT FINDINGS

In the diagnosis and design stages of the policy process, research questions aim to characterize environmental problems affecting human health and to estimate the potential impacts of new policies. Simulation methods using existing exposure-response information to estimate health impacts predominate at these stages of the policy process. In subsequent stages, e.g., during implementation, research questions aim to understand the actual policy impacts. Simulation methods or observational methods, which rely on experimental data gathered in the study area to assess the effectiveness of the policy, can be applied at these stages. Increasingly, novel techniques fuse both simulation and observational methods to enhance the robustness of impact evaluations assessing implemented policies. The policy process consists of interdependent stages, from inception to end, but most reviewed studies focus on single stages, neglecting the continuity of the policy life cycle. Studies assessing the health impacts of policies using a multi-stage approach are lacking. Most studies investigate intended impacts of policies; focusing also on unintended impacts may provide a more comprehensive evaluation of policies.

Harrell Jr F, Moons KGM, Debray TPA, Reitsma JB. A tutorial on individualized treatment effect prediction from randomized trials with a binary endpoint

Randomized trials typically estimate average relative treatment effects, but decisions on the benefit of a treatment are possibly better informed by more individualized predictions of the absolute treatment effect. In case of a binary outcome, these predictions of absolute individualized treatment effect require knowledge of the individual's risk without treatment and incorporation of a possibly differential treatment effect (ie, varying with patient characteristics). In this article, we lay out the causal structure of individualized treatment effect in terms of potential outcomes and describe the required assumptions that underlie a causal interpretation of its prediction. Subsequently, we describe regression models and model estimation techniques that can be used to move from average to more individualized treatment effect predictions. We focus mainly on logistic regression-based methods that are both well-known and naturally provide the required probabilistic estimates. We incorporate key components from both causal inference and prediction research to arrive at individualized treatment effect predictions. While the separate components are well known, their successful amalgamation is very much an ongoing field of research. We cut the problem down to its essentials in the setting of a randomized trial, discuss the importance of a clear definition of the estimand of interest, provide insight into the required assumptions, and give guidance with respect to modeling and estimation options. Simulated data illustrate the potential of different modeling options across scenarios that vary both average treatment effect and treatment effect heterogeneity. Two applied examples illustrate individualized treatment effect prediction in randomized trial data.

Invited Commentary: Treatment Drop-in-Making the Case for Causal Prediction

Clinical prediction models (CPMs) are often used to guide treatment initiation, with individuals at high risk offered treatment. This implicitly assumes that the probability quoted from a CPM represents the risk to an individual of an adverse outcome in absence of treatment. However, for a CPM to correctly target this estimand requires careful causal thinking. One problem that needs to be overcome is treatment drop-in: where individuals in the development data commence treatment after the time of prediction but before the outcome occurs. In this issue of the Journal, Xu et al. (Am J Epidemiol. 2021;190(10):2000-2014) use causal estimates from external data sources, such as clinical trials, to adjust CPMs for treatment drop-in. This represents a pragmatic and promising approach to address this issue, and it illustrates the value of utilizing causal inference in prediction. Building causality into the prediction pipeline can also bring other benefits. These include the ability to make and compare hypothetical predictions under different interventions, to make CPMs more explainable and transparent, and to improve model generalizability. Enriching CPMs with causal inference therefore has the potential to add considerable value to the role of prediction in healthcare.

The data-collection on adverse effects of anti-HIV drugs (D:A:D) model for predicting cardiovascular events: External validation in a diverse cohort of people living with HIV

OBJECTIVES

Little is known about the external validity of the Data-collection on Adverse Effects of Anti-HIV Drugs (D:A:D) model for predicting cardiovascular disease (CVD) risk among people living with HIV (PLWH). We aimed to evaluate the performance of the updated D:A:D model for 5-year CVD risk in a diverse group of PLWH engaged in HIV care.

METHODS

We used data from an institutional HIV registry, which includes PLWH engaged in care at a safety-net HIV clinic. Eligible individuals had a baseline clinical encounter between 1 January 2013 and 31 December 2014, with follow-up through to 31 December 2019. We estimated 5-year predicted risks of CVD as a function of the prognostic index and baseline survival of the D:A:D model, which were used to assess model discrimination (C-index), calibration and net benefit.

RESULTS

Our evaluable population comprised 1029 PLWH, of whom 30% were female, 50% were non-Hispanic black, and median age was 45 years. The C-index was 0.70 [95% confidence limits (CL): 0.64-0.75]. The predicted 5-year CVD risk was 3.0% and the observed 5-year risk was 8.9% (expected/observed ratio = 0.33, 95% CL: 0.26-0.54). The model had a greater net benefit than treating all or treating none at a risk threshold of 10%.

CONCLUSIONS

The D:A:D model was miscalibrated for CVD risk among PLWH engaged in HIV care at an urban safety-net HIV clinic, which may be related to differences in case-mix and baseline CVD risk. Nevertheless, the HIV D:A:D model may be useful for decisions about CVD intervention for high-risk patients.

Prediction or causality? A scoping review of their conflation within current observational research

Etiological research aims to uncover causal effects, whilst prediction research aims to forecast an outcome with the best accuracy. Causal and prediction research usually require different methods, and yet their findings may get conflated when reported and interpreted. The aim of the current study is to quantify the frequency of conflation between etiological and prediction research, to discuss common underlying mistakes and provide recommendations on how to avoid these. Observational cohort studies published in January 2018 in the top-ranked journals of six distinct medical fields (Cardiology, Clinical Epidemiology, Clinical Neurology, General and Internal Medicine, Nephrology and Surgery) were included for the current scoping review. Data on conflation was extracted through signaling questions. In total, 180 studies were included. Overall, 26% (n = 46) contained conflation between etiology and prediction. The frequency of conflation varied across medical field and journal impact factor. From the causal studies 22% was conflated, mainly due to the selection of covariates based on their ability to predict without taking the causal structure into account. Within prediction studies 38% was conflated, the most frequent reason was a causal interpretation of covariates included in a prediction model. Conflation of etiology and prediction is a common methodological error in observational medical research and more frequent in prediction studies. As this may lead to biased estimations and erroneous conclusions, researchers must be careful when designing, interpreting and disseminating their research to ensure this conflation is avoided.

Multiple imputation to quantify misclassification in observational studies of the cognitively impaired: an application for pain assessment in nursing home residents

Background

Despite experimental evidence suggesting that pain sensitivity is not impaired by cognitive impairment, observational studies in nursing home residents have observed an inverse association between cognitive impairment and resident-reported or staff-assessed pain. Under the hypothesis that the inverse association may be partially attributable to differential misclassification due to recall and communication limitations, this study implemented a missing data approach to quantify the absolute magnitude of misclassification of pain, pain frequency, and pain intensity by level of cognitive impairment.

Methods

Using the 2016 Minimum Data Set 3.0, we conducted a cross-sectional study among newly admitted US nursing home residents. Pain presence, severity, and frequency is assessed via resident-reported measures. For residents unable to communicate their pain, nursing home staff document pain based on direct resident observation and record review. We estimate a counterfactual expected level of pain in the absence of cognitive impairment by multiply imputing modified pain indicators for which the values were retained for residents with no/mild cognitive impairment and set to missing for residents with moderate/severe cognitive impairment. Absolute differences (∆) in the presence and magnitude of pain were calculated as the difference between documented pain and the expected level of pain.

Results

The difference between observed and expected resident reported pain was greater in residents with severe cognitive impairment (∆ = -10.2%, 95% Confidence Interval (CI): -10.9% to -9.4%) than those with moderate cognitive impairment (∆ = -4.5%, 95% CI: -5.4% to -3.6%). For staff-assessed pain, the magnitude of apparent underreporting was similar between residents with moderate impairment (∆ = -7.2%, 95% CI: -8.3% to -6.0%) and residents with severe impairment (∆ = -7.2%, 95% CI: -8.0% to -6.3%). Pain characterized as “mild” had the highest magnitude of apparent underreporting.

Conclusions

In residents with moderate to severe cognitive impairment, documentation of any pain was lower than expected in the absence of cognitive impairment. This finding supports the hypothesis that an inverse association between pain and cognitive impairment may be explained by differential misclassification. This study highlights the need to develop analytic and/or procedural solutions to correct for recall/reporter bias resulting from cognitive impairment.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01327-5.

Prospective Validation of an Electronic Health Record-Based, Real-Time Suicide Risk Model

IMPORTANCE

Numerous prognostic models of suicide risk have been published, but few have been implemented outside of integrated managed care systems.

OBJECTIVE

To evaluate performance of a suicide attempt risk prediction model implemented in a vendor-supplied electronic health record to predict subsequent (1) suicidal ideation and (2) suicide attempt.

DESIGN, SETTING, AND PARTICIPANTS

This observational cohort study evaluated implementation of a suicide attempt prediction model in live clinical systems without alerting. The cohort comprised patients seen for any reason in adult inpatient, emergency department, and ambulatory surgery settings at an academic medical center in the mid-South from June 2019 to April 2020.

MAIN OUTCOMES AND MEASURES

Primary measures assessed external, prospective, and concurrent validity. Manual medical record validation of coded suicide attempts confirmed incident behaviors with intent to die. Subgroup analyses were performed based on demographic characteristics, relevant clinical context/setting, and presence or absence of universal screening. Performance was evaluated using discrimination (number needed to screen, C statistics, positive/negative predictive values) and calibration (Spiegelhalter z statistic). Recalibration was performed with logistic calibration.

RESULTS

The system generated 115 905 predictions for 77 973 patients (42 490 [54%] men, 35 404 [45%] women, 60 586 [78%] White, 12 620 [16%] Black). Numbers needed to screen in highest risk quantiles were 23 and 271 for suicidal ideation and attempt, respectively. Performance was maintained across demographic subgroups. Numbers needed to screen for suicide attempt by sex were 256 for men and 323 for women; and by race: 373, 176, and 407 for White, Black, and non-White/non-Black patients, respectively. Model C statistics were, across the health system: 0.836 (95% CI, 0.836-0.837); adult hospital: 0.77 (95% CI, 0.77-0.772); emergency department: 0.778 (95% CI, 0.777-0.778); psychiatry inpatient settings: 0.634 (95% CI, 0.633-0.636). Predictions were initially miscalibrated (Spiegelhalter z = -3.1; P = .001) with improvement after recalibration (Spiegelhalter z = 1.1; P = .26).

CONCLUSIONS AND RELEVANCE

In this study, this real-time predictive model of suicide attempt risk showed reasonable numbers needed to screen in nonpsychiatric specialty settings in a large clinical system. Assuming that research-valid models will translate without performing this type of analysis risks inaccuracy in clinical practice, misclassification of risk, wasted effort, and missed opportunity to correct and prevent such problems. The next step is careful pairing with low-cost, low-harm preventive strategies in a pragmatic trial of effectiveness in preventing future suicidality.

Covariates adjustment questioned conclusions of predictive analyses: an illustration with the Kidney Donor Risk Index

OBJECTIVES

We aimed to illustrate that considering covariates can lead to meaningful interpretation of the discriminative capacities of a prognostic marker. For this, we evaluated the ability of the Kidney Donor Risk Index (KDRI) to discriminate kidney graft failure risk.

STUDY DESIGN AND SETTING

From 4114 French patients, we estimated the adjusted area under the time-dependent ROC curve by standardizing the marker and weighting the observations. By weighting the contributions, we also studied the impact of KDRI-based transplantations on the patient and graft survival.

RESULTS

The covariate-adjusted AUC varied from 55% (95% confidence interval [CI]: 51-60%) for a prognostic up to 1 year post-transplantation to 56% (95% CI: 52-59%) up to 7 years. The Restricted Mean Survival Time (RMST) was 6.44 years for high-quality graft recipients (95% CI: 6.30-6.56) and would have been 6.31 years (95% CI: 6.13-6.46) if they had medium-quality transplants. The RMST was 5.10 years for low-quality graft recipients (95% CI: 4.90-5.31) and would have been 5.52 years (95% CI: 5.17-5.83) if they had medium-quality transplants.

CONCLUSION

We demonstrated that the KDRI discriminative capacities were mainly explained by the recipient characteristics. We also showed that counterfactual estimations, often used in causal studies, are also interesting in predictive studies, especially regarding the new available methods.

A scoping review of causal methods enabling predictions under hypothetical interventions

BACKGROUND

The methods with which prediction models are usually developed mean that neither the parameters nor the predictions should be interpreted causally. For many applications, this is perfectly acceptable. However, when prediction models are used to support decision making, there is often a need for predicting outcomes under hypothetical interventions.

AIMS

We aimed to identify published methods for developing and validating prediction models that enable risk estimation of outcomes under hypothetical interventions, utilizing causal inference. We aimed to identify the main methodological approaches, their underlying assumptions, targeted estimands, and potential pitfalls and challenges with using the method. Finally, we aimed to highlight unresolved methodological challenges.

METHODS

We systematically reviewed literature published by December 2019, considering papers in the health domain that used causal considerations to enable prediction models to be used for predictions under hypothetical interventions. We included both methodologies proposed in statistical/machine learning literature and methodologies used in applied studies.

RESULTS

We identified 4919 papers through database searches and a further 115 papers through manual searches. Of these, 87 papers were retained for full-text screening, of which 13 were selected for inclusion. We found papers from both the statistical and the machine learning literature. Most of the identified methods for causal inference from observational data were based on marginal structural models and g-estimation.

CONCLUSIONS

There exist two broad methodological approaches for allowing prediction under hypothetical intervention into clinical prediction models: (1) enriching prediction models derived from observational studies with estimated causal effects from clinical trials and meta-analyses and (2) estimating prediction models and causal effects directly from observational data. These methods require extending to dynamic treatment regimes, and consideration of multiple interventions to operationalise a clinical decision support system. Techniques for validating 'causal prediction models' are still in their infancy.

Prediction models for covid-19 outcomes

Reasons to be cautious