Prediction meets causal inference: the role of treatment in clinical prediction models

Dynamic prediction of time to wound healing at routine wound care visits

Objective Having a wound decreases patients' quality of life and brings uncertainty, especially if the wound does not show a healing tendency. The objective of this study was to develop and validate a model to dynamically predict time to wound healing at subsequent routine wound care visits. Approach A dynamic prediction model was developed in a cohort of wounds treated by nurse practitioners between 2017-2022. Potential predictors were selected based on literature, expert opinion, and availability in the routine care setting. To assess performance for future wound care visits, the model was validated in a new cohort of wounds visited in early 2023. Reporting followed TRIPOD guidelines. Results We analyzed data from 92,098 visits, corresponding to 14,248 wounds and 7,221 patients. At external validation, discriminative performance of our developed model was comparable to internal validation (c-statistic = 0.70 [95% CI 0.69, 0.71]) and the model remained well-calibrated. Strong predictors were wound-level characteristics and indicators of the healing process so far (e.g., wound surface area). Innovation Going beyond previous prediction studies in the field, the developed model dynamically predicts the remaining time to wound healing for many wound types at subsequent wound care visits, in line with the dynamic nature of wound care. In addition, the model was externally validated and showed stable performance. Conclusion: The developed model can potentially contribute to patient satisfaction and reduce uncertainty around wound healing times when implemented in practice. When the predicted time of wound healing remains high, practitioners can consider adapting their wound management.

Prediction Under Interventions: Evaluation of Counterfactual Performance Using Longitudinal Observational Data

Predictions under interventions are estimates of what a person's risk of an outcome would be if they were to follow a particular treatment strategy, given their individual characteristics. Such predictions can give important input to medical decision-making. However, evaluating the predictive performance of interventional predictions is challenging. Standard ways of evaluating predictive performance do not apply when using observational data, because prediction under interventions involves obtaining predictions of the outcome under conditions that are different from those that are observed for a subset of individuals in the validation dataset. This work describes methods for evaluating counterfactual performance of predictions under interventions for time-to-event outcomes. This means we aim to assess how well predictions would match the validation data if all individuals had followed the treatment strategy under which predictions are made. We focus on counterfactual performance evaluation using longitudinal observational data, and under treatment strategies that involve sustaining a particular treatment regime over time. We introduce an estimation approach using artificial censoring and inverse probability weighting that involves creating a validation dataset mimicking the treatment strategy under which predictions are made. We extend measures of calibration, discrimination (c-index and cumulative/dynamic AUCt) and overall prediction error (Brier score) to allow assessment of counterfactual performance. The methods are evaluated using a simulation study, including scenarios in which the methods should detect poor performance. Applying our methods in the context of liver transplantation shows that our procedure allows quantification of the performance of predictions supporting crucial decisions on organ allocation.

From algorithms to action: improving patient care requires causality

In cancer research there is much interest in building and validating outcome prediction models to support treatment decisions. However, because most outcome prediction models are developed and validated without regard to the causal aspects of treatment decision making, many published outcome prediction models may cause harm when used for decision making, despite being found accurate in validation studies. Guidelines on prediction model validation and the checklist for risk model endorsement by the American Joint Committee on Cancer do not protect against prediction models that are accurate during development and validation but harmful when used for decision making. We explain why this is the case and how to build and validate models that are useful for decision making.

Making predictions under interventions: a case study from the PREDICT-CVD cohort in New Zealand primary care

Background

Most existing clinical prediction models do not allow predictions under interventions. Such predictions allow predicted risk under different proposed strategies to be compared and are therefore useful to support clinical decision making. We aimed to compare methodological approaches for predicting individual level cardiovascular risk under three interventions: smoking cessation, reducing blood pressure, and reducing cholesterol.

Methods

We used data from the PREDICT prospective cohort study in New Zealand to calculate cardiovascular risk in a primary care setting. We compared three strategies to estimate absolute risk under intervention: (a) conditioning on hypothetical interventions in non-causal models; (b) combining existing prediction models with causal effects estimated using observational causal inference methods; and (c) combining existing prediction models with causal effects reported in published literature.

Results

The median absolute cardiovascular risk among smokers was 3.9%; our approaches predicted that smoking cessation reduced this to a median between a non-causal estimate of 2.5% and a causal estimate of 2.8%, depending on estimation methods. For reducing blood pressure, the proposed approaches estimated a reduction of absolute risk from a median of 4.9% to a median between 3.2% and 4.5% (both derived from causal estimation). Reducing cholesterol was estimated to reduce median absolute risk from 3.1% to between 2.2% (non-causal estimate) and 2.8% (causal estimate).

Conclusions

Estimated absolute risk reductions based on non-causal methods were different to those based on causal methods, and there was substantial variation in estimates within the causal methods. Researchers wishing to estimate risk under intervention should be explicit about their causal modelling assumptions and conduct sensitivity analysis by considering a range of possible approaches.

Causal machine learning for predicting treatment outcomes

Causal machine learning (ML) offers flexible, data-driven methods for predicting treatment outcomes including efficacy and toxicity, thereby supporting the assessment and safety of drugs. A key benefit of causal ML is that it allows for estimating individualized treatment effects, so that clinical decision-making can be personalized to individual patient profiles. Causal ML can be used in combination with both clinical trial data and real-world data, such as clinical registries and electronic health records, but caution is needed to avoid biased or incorrect predictions. In this Perspective, we discuss the benefits of causal ML (relative to traditional statistical or ML approaches) and outline the key components and steps. Finally, we provide recommendations for the reliable use of causal ML and effective translation into the clinic.

Delayed versus primary closure to minimize risk of surgical-site infection for complicated appendicitis: A secondary analysis of a randomized trial using counterfactual prediction modeling

OBJECTIVE

To evaluate the risk of surgical site infection (SSI) following complicated appendectomy in individual patients receiving delayed primary closure (DPC) versus primary closure (PC) after adjustment for individual risk factors.

DESIGN

Secondary analysis of randomized controlled trial (RCT) with prediction model.

SETTING

Referral centers across Thailand.

PARTICIPANTS

Adult patients who underwent appendectomy via a lower-right-quadrant abdominal incision due to complicated appendicitis.

METHODS

A secondary analysis of a published RCT was performed applying a counterfactual prediction model considering interventions (PC vs DPC) and other significant predictors. A multivariable logistic regression was applied, and a likelihood-ratio test was used to select significant predictors to retain in a final model. Factual versus counterfactual SSI risks for individual patients along with individual treatment effect (iTE) were estimated.

RESULTS

In total, 546 patients (271 PC vs 275 DPC) were included in the analysis. The individualized prediction model consisted of allocated intervention, diabetes, type of complicated appendicitis, fecal contamination, and incision length. The iTE varied between 0.4% and 7% for PC compared to DPC; ∼38.1% of patients would have ≥2.1% lower SSI risk following PC compared to DPC. The greatest risk reduction was identified in diabetes with ruptured appendicitis, fecal contamination, and incision length of 10 cm, where SSI risks were 47.1% and 54.1% for PC and DPC, respectively.

CONCLUSIONS

In this secondary analysis, we found that most patients benefited from early PC versus DPC. Findings may be used to inform SSI prevention strategies for patients with complicated appendicitis.

Limited incremental predictive value of the frailty index and other vulnerability measures from routine care data for mortality risk prediction in older patients with COVID-19 in primary care

BACKGROUND

During the COVID-19 pandemic, older patients in primary care were triaged based on their frailty or assumed vulnerability for poor outcomes, while evidence on the prognostic value of vulnerability measures in COVID-19 patients in primary care was lacking. Still, knowledge on the role of vulnerability is pivotal in understanding the resilience of older people during acute illness, and hence important for future pandemic preparedness. Therefore, we assessed the predictive value of different routine care-based vulnerability measures in addition to age and sex for 28-day mortality in an older primary care population of patients with COVID-19.

METHODS

From primary care medical records using three routinely collected Dutch primary care databases, we included all patients aged 70 years or older with a COVID-19 diagnosis registration in 2020 and 2021. All-cause mortality was predicted using logistic regression based on age and sex only (basic model), and separately adding six vulnerability measures: renal function, cognitive impairment, number of chronic drugs, Charlson Comorbidity Index, Chronic Comorbidity Score, and a Frailty Index. Predictive performance of the basic model and the six vulnerability models was compared in terms of area under the receiver operator characteristic curve (AUC), index of prediction accuracy and the distribution of predicted risks.

RESULTS

Of the 4,065 included patients, 9% died within 28 days after COVID-19 diagnosis. Predicted mortality risk ranged between 7-26% for the basic model including age and sex, changing to 4-41% by addition of comorbidity-based vulnerability measures (Charlson Comorbidity Index, Chronic Comorbidity Score), more reflecting impaired organ functioning. Similarly, the AUC of the basic model slightly increased from 0.69 (95%CI 0.66 - 0.72) to 0.74 (95%CI 0.71 - 0.76) by addition of either of these comorbidity scores. Addition of a Frailty Index, renal function, the number of chronic drugs or cognitive impairment yielded no substantial change in predictions.

CONCLUSION

In our dataset of older COVID-19 patients in primary care, the 28-day mortality fraction was substantial at 9%. Six different vulnerability measures had little incremental predictive value in addition to age and sex in predicting short-term mortality.

Flexible Machine Learning Estimation of Conditional Average Treatment Effects: A Blessing and a Curse

Causal inference from observational data requires untestable identification assumptions. If these assumptions apply, machine learning methods can be used to study complex forms of causal effect heterogeneity. Recently, several machine learning methods were developed to estimate the conditional average treatment effect (ATE). If the features at hand cannot explain all heterogeneity, the individual treatment effects can seriously deviate from the conditional ATE. In this work, we demonstrate how the distributions of the individual treatment effect and the conditional ATE can differ when a causal random forest is applied. We extend the causal random forest to estimate the difference in conditional variance between treated and controls. If the distribution of the individual treatment effect equals that of the conditional ATE, this estimated difference in variance should be small. If they differ, an additional causal assumption is necessary to quantify the heterogeneity not captured by the distribution of the conditional ATE. The conditional variance of the individual treatment effect can be identified when the individual effect is independent of the outcome under no treatment given the measured features. Then, in the cases where the individual treatment effect and conditional ATE distributions differ, the extended causal random forest can appropriately estimate the variance of the individual treatment effect distribution, whereas the causal random forest fails to do so.

Development and Validation of a Novel Model for Predicting Coronary Heart Disease in Snoring Hypertensive Patients with Hyperhomocysteinemia

Hypertensive patients with snoring and elevated plasma homocysteine levels are common. When these factors are combined, the risk of coronary heart disease (CHD) is high. Herein, we developed and validated an easy-to-use nomogram to predict high-risk CHD in snoring hypertensive patients with elevated plasma homocysteine.Snoring patients (n = 1,962) with hyperhomocysteinemia and hypertension were divided into training (n = 1,373, 70%) and validation (n = 589, 30%) sets. We extracted CHD predictors using multivariate Cox regression analysis, then constructed a nomogram model. Internal validation using 1,000 bootstrap resampling was performed to assess the consistency and discrimination of the predictive model using the area under the receiver operating characteristic curve (AUC) and calibration plots.We constructed a nomogram model with the extracted predictors, including age, waist-height ratio, smoking, and low-density lipoprotein cholesterol levels. The AUCs of the training and validation cohorts at 80 months were 0.735 (95% CI: 0.678-0.792) and 0.646 (95% CI: 0.547-0.746), respectively. The consistency between the observed CHD survival and the probability of CHD survival in the training and validation sets was acceptable based on the calibration plots. A total of more than 151 points in the nomogram can be used in the identification of high-risk patients for CHD among snoring hypertensive patients with elevated plasma homocysteine.We developed a CHD risk prediction model for snoring hypertension patients with hyperhomocysteinemia. Our findings provide a useful clinical tool for the rapid identification of high-risk CHD at an early stage.

Validation of Risk Stratification for Cardiac Events in Pregnant Women With Valvular Heart Disease

BACKGROUND

Most risk stratification tools for pregnant patients with heart disease were developed in high-income countries and in populations with predominantly congenital heart disease, and therefore, may not be generalizable to those with valvular heart disease (VHD).

OBJECTIVES

The purpose of this study was to validate and establish the clinical utility of 2 risk stratification tools-DEVI (VHD-specific tool) and CARPREG-II-for predicting adverse cardiac events in pregnant patients with VHD.

METHODS

We conducted a cohort study involving consecutive pregnancies complicated with VHD admitted to a tertiary center in a middle-income setting from January 2019 to April 2022. Individual risk for adverse composite cardiac events was calculated using DEVI and CARPREG-II models. Performance was assessed through discrimination and calibration characteristics. Clinical utility was evaluated with Decision Curve Analysis.

RESULTS

Of 577 eligible pregnancies, 69 (12.1%) experienced a component of the composite outcome. A majority (94.7%) had rheumatic etiology, with mitral regurgitation as the predominant lesion (48.2%). The area under the receiver-operating characteristic curve was 0.884 (95% CI: 0.844-0.923) for the DEVI and 0.808 (95% CI: 0.753-0.863) for the CARPREG-II models. Calibration plots suggested that DEVI score overestimates risk at higher probabilities, whereas CARPREG-II score overestimates risk at both extremes and underestimates risk at middle probabilities. Decision curve analysis demonstrated that both models were useful across predicted probability thresholds between 10% and 50%.

CONCLUSIONS

In pregnant patients with VHD, DEVI and CARPREG-II scores showed good discriminative ability and clinical utility across a range of probabilities. The DEVI score showed better agreement between predicted probabilities and observed events.

Fixing the Leaky Pipe: How to Improve the Uptake of Patient-Reported Outcomes-Based Prognostic and Predictive Models in Cancer Clinical Practice

PURPOSE

This discussion paper outlines challenges and proposes solutions for successfully implementing prediction models that incorporate patient-reported outcomes (PROs) in cancer practice.

METHODS

We organized a full-day multidisciplinary meeting of people with expertise in cancer care delivery, PRO collection, PRO use in prediction modeling, computing, implementation, and decision science. The discussions presented here focused on identifying challenges to the development, implementation and use of prediction models incorporating PROs, and suggesting possible solutions.

RESULTS

Specific challenges and solutions were identified across three broad areas. (1) Understanding decision making and implementation: necessitating multidisciplinary collaboration in the early stages and throughout; early stakeholder engagement to define the decision problem and ensure acceptability of PROs in prediction; understanding patient/clinician interpretation of PRO predictions and uncertainty to optimize prediction impact; striving for model integration into existing electronic health records; and early regulatory alignment. (2) Recognizing the limitations to PRO collection and their impact on prediction: incorporating validated, clinically important PROs to maximize model generalizability and clinical engagement; and minimizing missing PRO data (resulting from both structural digital exclusion and time-varying factors) to avoid exacerbating existing inequalities. (3) Statistical and modeling challenges: incorporating statistical methods to address missing data; ensuring predictive modeling recognizes complex causal relationships; and considering temporal and geographic recalibration so that model predictions reflect the relevant population.

CONCLUSION

Developing and implementing PRO-based prediction models in cancer care requires extensive multidisciplinary working from the earliest stages, recognition of implementation challenges because of PRO collection and model presentation, and robust statistical methods to manage missing data, causality, and calibration. Prediction models incorporating PROs should be viewed as complex interventions, with their development and impact assessment carried out to reflect this.

Development and internal-external validation of statistical and machine learning models for breast cancer prognostication: cohort study

OBJECTIVE

To develop a clinically useful model that estimates the 10 year risk of breast cancer related mortality in women (self-reported female sex) with breast cancer of any stage, comparing results from regression and machine learning approaches.

DESIGN

Population based cohort study.

SETTING

QResearch primary care database in England, with individual level linkage to the national cancer registry, Hospital Episodes Statistics, and national mortality registers.

PARTICIPANTS

141 765 women aged 20 years and older with a diagnosis of invasive breast cancer between 1 January 2000 and 31 December 2020.

MAIN OUTCOME MEASURES

Four model building strategies comprising two regression (Cox proportional hazards and competing risks regression) and two machine learning (XGBoost and an artificial neural network) approaches. Internal-external cross validation was used for model evaluation. Random effects meta-analysis that pooled estimates of discrimination and calibration metrics, calibration plots, and decision curve analysis were used to assess model performance, transportability, and clinical utility.

RESULTS

During a median 4.16 years (interquartile range 1.76-8.26) of follow-up, 21 688 breast cancer related deaths and 11 454 deaths from other causes occurred. Restricting to 10 years maximum follow-up from breast cancer diagnosis, 20 367 breast cancer related deaths occurred during a total of 688 564.81 person years. The crude breast cancer mortality rate was 295.79 per 10 000 person years (95% confidence interval 291.75 to 299.88). Predictors varied for each regression model, but both Cox and competing risks models included age at diagnosis, body mass index, smoking status, route to diagnosis, hormone receptor status, cancer stage, and grade of breast cancer. The Cox model's random effects meta-analysis pooled estimate for Harrell's C index was the highest of any model at 0.858 (95% confidence interval 0.853 to 0.864, and 95% prediction interval 0.843 to 0.873). It appeared acceptably calibrated on calibration plots. The competing risks regression model had good discrimination: pooled Harrell's C index 0.849 (0.839 to 0.859, and 0.821 to 0.876, and evidence of systematic miscalibration on summary metrics was lacking. The machine learning models had acceptable discrimination overall (Harrell's C index: XGBoost 0.821 (0.813 to 0.828, and 0.805 to 0.837); neural network 0.847 (0.835 to 0.858, and 0.816 to 0.878)), but had more complex patterns of miscalibration and more variable regional and stage specific performance. Decision curve analysis suggested that the Cox and competing risks regression models tested may have higher clinical utility than the two machine learning approaches.

CONCLUSION

In women with breast cancer of any stage, using the predictors available in this dataset, regression based methods had better and more consistent performance compared with machine learning approaches and may be worthy of further evaluation for potential clinical use, such as for stratified follow-up.

How to Develop and Validate Prediction Models for Orthopedic Outcomes

Prediction models are common in medicine for predicting outcomes such as mortality, complications, or response to treatment. Despite the growing interest in these models in arthroplasty (and orthopaedics in general), few have been adopted in clinical practice. If robustly built and validated, prediction models can be excellent tools to support surgical decision making. In this paper, we provide an overview of the statistical concepts surrounding prediction models and outline practical steps for prediction model development and validation in arthroplasty research. Please visit the followinghttps://www.youtube.com/watch?v=9Yrit23Rkicfor a video that explains the highlights of the paper in practical terms.

A connection between survival multistate models and causal inference for external treatment interruptions

Recently, treatment interruptions such as a clinical hold in randomized clinical trials have been investigated by using a multistate model approach. The phase III clinical trial START (Stimulating Targeted Antigenic Response To non-small-cell cancer) with primary endpoint overall survival was temporarily placed on hold for enrollment and treatment by the US Food and Drug Administration (FDA). Multistate models provide a flexible framework to account for treatment interruptions induced by a time-dependent external covariate. Extending previous work, we propose a censoring and a filtering approach both aimed at estimating the initial treatment effect on overall survival in the hypothetical situation of no clinical hold. A special focus is on creating a link to causal inference. We show that calculating the matrix of transition probabilities in the multistate model after application of censoring (or filtering) yields the desired causal interpretation. Assumptions in support of the identification of a causal effect by censoring (or filtering) are discussed. Thus, we provide the basis to apply causal censoring (or filtering) in more general settings such as the COVID-19 pandemic. A simulation study demonstrates that both causal censoring and filtering perform favorably compared to a naïve method ignoring the external impact.

A recalibrated prediction model can identify level-1 trauma patients at risk of nosocomial pneumonia

INTRODUCTION

Nosocomial pneumonia has poor prognosis in hospitalized trauma patients. Croce et al. published a model to predict post-traumatic ventilator-associated pneumonia, which achieved high discrimination and reasonable sensitivity. We aimed to externally validate Croce's model to predict nosocomial pneumonia in patients admitted to a Dutch level-1 trauma center.

MATERIALS AND METHODS

This retrospective study included all trauma patients (≥ 16y) admitted for > 24 h to our level-1 trauma center in 2017. Exclusion criteria were pneumonia or antibiotic treatment upon hospital admission, treatment elsewhere > 24 h, or death < 48 h. Croce's model used eight clinical variables-on trauma severity and treatment, available in the emergency department-to predict nosocomial pneumonia risk. The model's predictive performance was assessed through discrimination and calibration before and after re-estimating the model's coefficients. In sensitivity analysis, the model was updated using Ridge regression.

RESULTS

809 Patients were included (median age 51y, 67% male, 97% blunt trauma), of whom 86 (11%) developed nosocomial pneumonia. Pneumonia patients were older, more severely injured, and underwent more emergent interventions. Croce's model showed good discrimination (AUC 0.83, 95% CI 0.79-0.87), yet predicted probabilities were too low (mean predicted risk 6.4%), and calibration was suboptimal (calibration slope 0.63). After full model recalibration, discrimination (AUC 0.84, 95% CI 0.80-0.88) and calibration improved. Adding age to the model increased the AUC to 0.87 (95% CI 0.84-0.91). Prediction parameters were similar after the models were updated using Ridge regression.

CONCLUSION

The externally validated and intercept-recalibrated models show good discrimination and have the potential to predict nosocomial pneumonia. At this time, clinicians could apply these models to identify high-risk patients, increase patient monitoring, and initiate preventative measures. Recalibration of Croce's model improved the predictive performance (discrimination and calibration). The recalibrated model provides a further basis for nosocomial pneumonia prediction in level-1 trauma patients. Several models are accessible via an online tool.

LEVEL OF EVIDENCE

Level III, Prognostic/Epidemiological Study.

Assessment of Spinal Metastases Surgery Risk Stratification Tools in Breast Cancer by Molecular Subtype

BACKGROUND

Breast cancer molecular features and modern therapies are not included in spine metastasis prediction algorithms.

OBJECTIVE

To examine molecular differences and the impact of postoperative systemic therapy to improve prognosis prediction for spinal metastases surgery and aid surgical decision making.

METHODS

This is a retrospective multi-institutional study of patients who underwent spine surgery for symptomatic breast cancer spine metastases from 2008 to 2021 at the Massachusetts General Hospital and Brigham and Women's Hospital. We studied overall survival, stratified by breast cancer molecular subtype, and calculated hazard ratios (HRs) adjusting for demographics, tumor characteristics, treatments, and laboratory values. We tested the performance of established models (Tokuhashi, Bauer, Skeletal Oncology Research Group, New England Spinal Metastases Score) to predict and compare all-cause.

RESULTS

A total of 98 patients surgically treated for breast cancer spine metastases were identified (100% female sex; median age, 56 years [IQR, 36-84 years]). The 1-year probabilities of survival for hormone receptor positive, hormone receptor positive/human epidermal growth factor receptor 2+, human epidermal growth factor receptor 2+, and triple-negative breast cancer were 63% (45 of 71), 83% (10 of 12), 0% (0 of 3), and 12% (1 of 8), respectively ( P < .001). Patients with triple-negative breast cancer had a higher proportion of visceral metastases, brain metastases, and poor physical activity at baseline. Postoperative chemotherapy and endocrine therapy were associated with prolonged survival. The Skeletal Oncology Research Group prognostic model had the highest discrimination (area under the receiver operating characteristic, 0.77 [95% CI, 0.73-0.81]). The performance of all prognostic scores improved when preoperative molecular data and postoperative systemic treatment plans was considered.

CONCLUSION

Spine metastases risk tools were able to predict prognosis at a significantly higher degree after accounting for molecular features which guide treatment response.

Accounting for treatment during the development or validation of prediction models

Clinical prediction models are widely used to predict adverse outcomes in patients, and are often employed to guide clinical decision-making. Clinical data typically consist of patients who received different treatments. Many prediction modeling studies fail to account for differences in patient treatment appropriately, which results in the development of prediction models that show poor accuracy and generalizability. In this paper, we list the most common methods used to handle patient treatments and discuss certain caveats associated with each method. We believe that proper handling of differences in patient treatment is crucial for the development of accurate and generalizable models. As different treatment strategies are employed for different diseases, the best approach to properly handle differences in patient treatment is specific to each individual situation. We use the Ma-Spore acute lymphoblastic leukemia data set as a case study to demonstrate the complexities associated with differences in patient treatment, and offer suggestions on incorporating treatment information during evaluation of prediction models. In clinical data, patients are typically treated on a case by case basis, with unique cases occurring more frequently than expected. Hence, there are many subtleties to consider during the analysis and evaluation of clinical prediction models.

Development and external validation of a risk prediction model for falls in patients with an indication for antihypertensive treatment: retrospective cohort study

OBJECTIVE

To develop and externally validate the STRAtifying Treatments In the multi-morbid Frail elderlY (STRATIFY)-Falls clinical prediction model to identify the risk of hospital admission or death from a fall in patients with an indication for antihypertensive treatment.

DESIGN

Retrospective cohort study.

SETTING

Primary care data from electronic health records contained within the UK Clinical Practice Research Datalink (CPRD).

PARTICIPANTS

Patients aged 40 years or older with at least one blood pressure measurement between 130 mm Hg and 179 mm Hg.

MAIN OUTCOME MEASURE

First serious fall, defined as hospital admission or death with a primary diagnosis of a fall within 10 years of the index date (12 months after cohort entry). Model development was conducted using a Fine-Gray approach in data from CPRD GOLD, accounting for the competing risk of death from other causes, with subsequent recalibration at one, five, and 10 years using pseudo values. External validation was conducted using data from CPRD Aurum, with performance assessed through calibration curves and the observed to expected ratio, C statistic, and D statistic, pooled across general practices, and clinical utility using decision curve analysis at thresholds around 10%.

RESULTS

Analysis included 1 772 600 patients (experiencing 62 691 serious falls) from CPRD GOLD used in model development, and 3 805 366 (experiencing 206 956 serious falls) from CPRD Aurum in the external validation. The final model consisted of 24 predictors, including age, sex, ethnicity, alcohol consumption, living in an area of high social deprivation, a history of falls, multiple sclerosis, and prescriptions of antihypertensives, antidepressants, hypnotics, and anxiolytics. Upon external validation, the recalibrated model showed good discrimination, with pooled C statistics of 0.833 (95% confidence interval 0.831 to 0.835) and 0.843 (0.841 to 0.844) at five and 10 years, respectively. Original model calibration was poor on visual inspection and although this was improved with recalibration, under-prediction of risk remained (observed to expected ratio at 10 years 1.839, 95% confidence interval 1.811 to 1.865). Nevertheless, decision curve analysis suggests potential clinical utility, with net benefit larger than other strategies.

CONCLUSIONS

This prediction model uses commonly recorded clinical characteristics and distinguishes well between patients at high and low risk of falls in the next 1-10 years. Although miscalibration was evident on external validation, the model still had potential clinical utility around risk thresholds of 10% and so could be useful in routine clinical practice to help identify those at high risk of falls who might benefit from closer monitoring or early intervention to prevent future falls. Further studies are needed to explore the appropriate thresholds that maximise the model's clinical utility and cost effectiveness.

Should we adopt a prognosis-based approach to unexplained infertility?

The treatment of unexplained infertility is a contentious topic that continues to attract a great deal of interest amongst clinicians, patients and policy makers. The inability to identify an underlying pathology makes it difficult to devise effective treatments for this condition. Couples with unexplained infertility can conceive on their own and any proposed intervention needs to offer a better chance of having a baby. Over the years, several prognostic and prediction models based on routinely collected clinical data have been developed, but these are not widely used by clinicians and patients. In this opinion paper, we propose a prognosis-based approach such that a decision to access treatment is based on the estimated chances of natural and treatment-related conception, which, in the same couple, can change over time. This approach avoids treating all couples as a homogeneous group and minimizes unnecessary treatment whilst ensuring access to those who need it early.

LACE Index to Predict the High Risk of 30-Day Readmission in Patients With Acute Myocardial Infarction at a University Affiliated Hospital

Background

The LACE index (length of stay, acuity of admission, comorbidity index, and emergency room visit in the past 6 months) has been used to predict the risk of 30-day readmission after hospital discharge in both medical and surgical patients. This study aimed to utilize the LACE index to predict the risk of 30-day readmission in hospitalized patients with acute myocardial infraction (AMI).

Methods

This was a retrospective study. Data were extracted from the hospital's electronic medical records of patients admitted with AMI between 2015 and 2019. LACE index was built on admission patient demographic data, and clinical and laboratory findings during the index of admission. The multivariate logistic regression was performed to determine the association and the risk prediction ability of the LACE index, and 30-day readmission were analyzed by receiver operator characteristic curves with C-statistic.

Results

Of the 3,607 patients included in the study, 5.7% (205) were readmitted within 30 days of discharge from the hospital. The adjusted odds ratio based on logistic regression of all baseline variables showed a statistically significant association with the LACE score and revealed an increased risk of readmission within 30 days of hospital discharge. However, patients with high LACE scores (≥10) had a significantly higher rate of emergency revisits within 30 days from the index discharge than those with low LACE scores. Despite this, analysis of the receiver operating characteristic curve indicated that the LACE index had favorable discrimination ability C-statistic 0.78 (95%CI; 0.75–0.81). The Hosmer–Lemeshow goodness- of-fit test P value was p = 0.920, indicating that the model was well-calibrated to predict risk of the 30-day readmission.

Conclusion

The LACE index demonstrated the good discrimination power to predict the risk of 30-day readmissions for hospitalized patients with AMI. These results can help clinicians to predict the risk of 30-day readmission at the early stage of hospitalization and pay attention during the care of high-risk patients. Future work is to be focused on additional factors to predict the risk of 30-day readmissions; they should be considered to improve the model performance of the LACE index with other acute conditions by using administrative data.

The Disconnect Between Development and Intended Use of Clinical Prediction Models for Covid-19: A Systematic Review and Real-World Data Illustration

Background

The SARS-CoV-2 pandemic has boosted the appearance of clinical predictions models in medical literature. Many of these models aim to provide guidance for decision making on treatment initiation. Special consideration on how to account for post-baseline treatments is needed when developing such models. We examined how post-baseline treatment was handled in published Covid-19 clinical prediction models and we illustrated how much estimated risks may differ according to how treatment is handled.

Methods

Firstly, we reviewed 33 Covid-19 prognostic models published in literature in the period up to 5 May 2020. We extracted: (1) the reported intended use of the model; (2) how treatment was incorporated during model development and (3) whether the chosen analysis strategy was in agreement with the intended use. Secondly, we used nationwide Dutch data on hospitalized patients who tested positive for SARS-CoV-2 in 2020 to illustrate how estimated mortality risks will differ when using four different analysis strategies to model ICU treatment.

Results

Of the 33 papers, 21 (64%) had misalignment between intended use and analysis strategy, 7 (21%) were unclear about the estimated risk and only 5 (15%) had clear alignment between intended use and analysis strategy. We showed with real data how different approaches to post-baseline treatment yield different estimated mortality risks, ranging between 33 and 46% for a 75 year-old patient with two medical conditions.

Conclusions

Misalignment between intended use and analysis strategy is common in reported Covid-19 clinical prediction models. This can lead to considerable under or overestimation of intended risks.

Prospective validation of the 4C prognostic models for adults hospitalised with COVID-19 using the ISARIC WHO Clinical Characterisation Protocol

PURPOSE

To prospectively validate two risk scores to predict mortality (4C Mortality) and in-hospital deterioration (4C Deterioration) among adults hospitalised with COVID-19.

METHODS

Prospective observational cohort study of adults (age ≥18 years) with confirmed or highly suspected COVID-19 recruited into the International Severe Acute Respiratory and emerging Infections Consortium (ISARIC) WHO Clinical Characterisation Protocol UK (CCP-UK) study in 306 hospitals across England, Scotland and Wales. Patients were recruited between 27 August 2020 and 17 February 2021, with at least 4 weeks follow-up before final data extraction. The main outcome measures were discrimination and calibration of models for in-hospital deterioration (defined as any requirement of ventilatory support or critical care, or death) and mortality, incorporating predefined subgroups.

RESULTS

76 588 participants were included, of whom 27 352 (37.4%) deteriorated and 12 581 (17.4%) died. Both the 4C Mortality (0.78 (0.77 to 0.78)) and 4C Deterioration scores (pooled C-statistic 0.76 (95% CI 0.75 to 0.77)) demonstrated consistent discrimination across all nine National Health Service regions, with similar performance metrics to the original validation cohorts. Calibration remained stable (4C Mortality: pooled slope 1.09, pooled calibration-in-the-large 0.12; 4C Deterioration: 1.00, -0.04), with no need for temporal recalibration during the second UK pandemic wave of hospital admissions.

CONCLUSION

Both 4C risk stratification models demonstrate consistent performance to predict clinical deterioration and mortality in a large prospective second wave validation cohort of UK patients. Despite recent advances in the treatment and management of adults hospitalised with COVID-19, both scores can continue to inform clinical decision making.

TRIAL REGISTRATION NUMBER

ISRCTN66726260.

The Predictive Individual Effect for Survival Data

BACKGROUND

The call for patient-focused drug development is loud and clear, as expressed in the twenty-first Century Cures Act and in recent guidelines and initiatives of regulatory agencies. Among the factors contributing to modernized drug development and improved health-care activities are easily interpretable measures of clinical benefit. In addition, special care is needed for cancer trials with time-to-event endpoints if the treatment effect is not constant over time.

OBJECTIVE

To quantify the potential clinical survival benefit for a new patient, would he/she be treated with the test or control treatment.

METHODS

We propose the predictive individual effect which is a patient-centric and tangible measure of clinical benefit under a wide variety of scenarios. It can be obtained by standard predictive calculations under a rank preservation assumption that has been used previously in trials with treatment switching.

RESULTS

We discuss four recent Oncology trials that cover situations with proportional as well as non-proportional hazards (delayed treatment effect or crossing of survival curves). It is shown that the predictive individual effect offers valuable insights beyond p-values, estimates of hazard ratios or differences in median survival.

CONCLUSION

Compared to standard statistical measures, the predictive individual effect is a direct, easily interpretable measure of clinical benefit. It facilitates communication among clinicians, patients, and other parties and should therefore be considered in addition to standard statistical results.

Classical Regression and Predictive Modeling

BACKGROUND

With the advent of personalized and stratified medicine, there has been much discussion about predictive modeling and the role of classical regression in modern medical research. We describe and distinguish the goals in these 2 frameworks for analysis.

METHODS

The assumptions underlying and utility of classical regression are reviewed for continuous and binary outcomes. The tenets of predictive modeling are then discussed and contrasted. Principles are illustrated by simulation and through application of methods to a neurosurgical study.

RESULTS

Classical regression can be used for insights into causal mechanisms if careful thought is given to the role of variables of interest and potential confounders. In predictive modeling, interest lies more in accuracy of predictions and so alternative metrics are used to judge adequacy of models and methods; methods which average predictions over several contending models can improve predictive performance but these do not admit a single risk score.

CONCLUSIONS

Both classical regression and predictive modeling have important roles in modern medical research. Understanding the distinction between the 2 frameworks for analysis is important to place them in their appropriate context and interpreting findings from published studies appropriately.

Validating the knowledge bank approach for personalized prediction of survival in acute myeloid leukemia: a reproducibility study

Reproducibility is not only essential for the integrity of scientific research but is also a prerequisite for model validation and refinement for the future application of predictive algorithms. However, reproducible research is becoming increasingly challenging, particularly in high-dimensional genomic data analyses with complex statistical or algorithmic techniques. Given that there are no mandatory requirements in most biomedical and statistical journals to provide the original data, analytical source code, or other relevant materials for publication, accessibility to these supplements naturally suggests a greater credibility of the published work. In this study, we performed a reproducibility assessment of the notable paper by Gerstung et al. (Nat Genet 49:332–340, 2017) by rerunning the analysis using their original code and data, which are publicly accessible. Despite an open science setting, it was challenging to reproduce the entire research project; reasons included: incomplete data and documentation, suboptimal code readability, coding errors, limited portability of intensive computing performed on a specific platform, and an R computing environment that could no longer be re-established. We learn that the availability of code and data does not guarantee transparency and reproducibility of a study; paradoxically, the source code is still liable to error and obsolescence, essentially due to methodological and computational complexity, a lack of reproducibility checking at submission, and updates for software and operating environment. The complex code may also hide problematic methodological aspects of the proposed research. Building on the experience gained, we discuss the best programming and software engineering practices that could have been employed to improve reproducibility, and propose practical criteria for the conduct and reporting of reproducibility studies for future researchers.

Quantitative prediction error analysis to investigate predictive performance under predictor measurement heterogeneity at model implementation

BACKGROUND

When a predictor variable is measured in similar ways at the derivation and validation setting of a prognostic prediction model, yet both differ from the intended use of the model in practice (i.e., "predictor measurement heterogeneity"), performance of the model at implementation needs to be inferred. This study proposed an analysis to quantify the impact of anticipated predictor measurement heterogeneity.

METHODS

A simulation study was conducted to assess the impact of predictor measurement heterogeneity across validation and implementation setting in time-to-event outcome data. The use of the quantitative prediction error analysis was illustrated using an example of predicting the 6-year risk of developing type 2 diabetes with heterogeneity in measurement of the predictor body mass index.

RESULTS

In the simulation study, calibration-in-the-large of prediction models was poor and overall accuracy was reduced in all scenarios of predictor measurement heterogeneity. Model discrimination decreased with increasing random predictor measurement heterogeneity.

CONCLUSIONS

Heterogeneity of predictor measurements across settings of validation and implementation reduced predictive performance at implementation of prognostic models with a time-to-event outcome. When validating a prognostic model, the targeted clinical setting needs to be considered and analyses can be conducted to quantify the impact of anticipated predictor measurement heterogeneity on model performance at implementation.

How measurements affected by medication use are reported and handled in observational research: a literature review

Purpose

In epidemiological research, measurements affected by medication, for example, blood pressure lowered by antihypertensives, are common. Different ways of handling medication are required depending on the research questions and whether the affected measurement is the exposure, the outcome, or a confounder. This study aimed to review handling of medication use in observational research.

Methods

PubMed was searched for etiological studies published between 2015 and 2019 in 15 high‐ranked journals from cardiology, diabetes, and epidemiology. We selected studies that analyzed blood pressure, glucose, or lipid measurements (whether exposure, outcome or confounder) by linear or logistic regression. Two reviewers independently recorded how medication use was handled and assessed whether the methods used were in accordance with the research aim. We reported the methods used per variable category (exposure, outcome, confounder).

Results

A total of 127 articles were included. Most studies did not perform any method to account for medication use (exposure 58%, outcome 53%, and confounder 45%). Restriction (exposure 22%, outcome 23%, and confounders 10%), or adjusting for medication use using a binary indicator were also used frequently (exposure: 18%, outcome: 19%, confounder: 45%). No advanced methods were applied. In 60% of studies, the methods' validity could not be judged due to ambiguous reporting of the research aim. Invalid approaches were used in 28% of the studies, mostly when the affected variable was the outcome (36%).

Conclusion

Many studies ambiguously stated the research aim and used invalid methods to handle medication use. Researchers should consider a valid methodological approach based on their research question.

Predicting counterfactual risks under hypothetical treatment strategies: an application to HIV

The accuracy of a prediction algorithm depends on contextual factors that may vary across deployment settings. To address this inherent limitation of prediction, we propose an approach to counterfactual prediction based on the g-formula to predict risk across populations that differ in their distribution of treatment strategies. We apply this to predict 5-year risk of mortality among persons receiving care for HIV in the U.S. Veterans Health Administration under different hypothetical treatment strategies. First, we implement a conventional approach to develop a prediction algorithm in the observed data and show how the algorithm may fail when transported to new populations with different treatment strategies. Second, we generate counterfactual data under different treatment strategies and use it to assess the robustness of the original algorithm's performance to these differences and to develop counterfactual prediction algorithms. We discuss how estimating counterfactual risks under a particular treatment strategy is more challenging than conventional prediction as it requires the same data, methods, and unverifiable assumptions as causal inference. However, this may be required when the alternative assumption of constant treatment patterns across deployment settings is unlikely to hold and new data is not yet available to retrain the algorithm.

The web-based application "QUiPP v.2" for the prediction of preterm birth in symptomatic women is not yet ready for worldwide clinical use: ten reflections on development, validation and use

PURPOSE

In this correspondence, we highlight general and domain-specific caveats in the development and validation of prediction models.

METHODS

Development and use of the "QUiPP" application, a tool for preterm birth prediction which is supported by the United Kingdom National Health Service, is scrutinised and commented on.

RESULTS

We highlight and elaborate ten points which may be perceived to be unclear or potentially misleading.

CONCLUSION

While the QUiPP application has high potential, it lacks transparency (on certain aspects related to model development) and proper validation. This precludes transportability to settings with other treatment policies and to other countries where the app has been made publicly available.

Lessons learnt when accounting for competing events in the external validation of time-to-event prognostic models

Background

External validation of prognostic models is necessary to assess the accuracy and generalizability of the model to new patients. If models are validated in a setting in which competing events occur, these competing risks should be accounted for when comparing predicted risks to observed outcomes.

Methods

We discuss existing measures of calibration and discrimination that incorporate competing events for time-to-event models. These methods are illustrated using a clinical-data example concerning the prediction of kidney failure in a population with advanced chronic kidney disease (CKD), using the guideline-recommended Kidney Failure Risk Equation (KFRE). The KFRE was developed using Cox regression in a diverse population of CKD patients and has been proposed for use in patients with advanced CKD in whom death is a frequent competing event.

Results

When validating the 5-year KFRE with methods that account for competing events, it becomes apparent that the 5-year KFRE considerably overestimates the real-world risk of kidney failure. The absolute overestimation was 10%age points on average and 29%age points in older high-risk patients.

Conclusions

It is crucial that competing events are accounted for during external validation to provide a more reliable assessment the performance of a model in clinical settings in which competing risks occur.

Pragmatic Precision Psychiatry-A New Direction for Optimizing Treatment Selection

IMPORTANCE

Clinical trials have identified numerous prescriptive predictors of mental disorder treatment response, ie, predictors of which treatments are best for which patients. However, none of these prescriptive predictors is strong enough alone to guide precision treatment planning. This has prompted growing interest in developing precision treatment rules (PTRs) that combine information across multiple prescriptive predictors, but this work has been much less successful in psychiatry than some other areas of medicine. Study designs and analysis schemes used in research on PTR development in other areas of medicine are reviewed, key challenges for implementing similar studies of mental disorders are highlighted, and recent methodological advances to address these challenges are described here.

OBSERVATIONS

Discovering prescriptive predictors requires large samples. Three approaches have been used in other areas of medicine to do this: conduct very large randomized clinical trials, pool individual-level results across multiple smaller randomized clinical trials, and develop preliminary PTRs in large observational treatment samples that are then tested in smaller randomized clinical trials. The third approach is most feasible for research on mental disorders. This approach requires working with large real-world observational electronic health record databases; carefully selecting samples to emulate trials; extracting information about prescriptive predictors from electronic health records along with other inexpensive data augmentation strategies; estimating preliminary PTRs in the observational data using appropriate methods; implementing pragmatic trials to validate the preliminary PTRs; and iterating between subsequent observational studies and quality improvement pragmatic trials to refine and expand the PTRs. New statistical methods exist to address the methodological challenges of implementing this approach.

CONCLUSIONS AND RELEVANCE

Advances in pragmatic precision psychiatry will require moving beyond the current focus on randomized clinical trials and adopting an iterative discovery-confirmation process that integrates observational and experimental studies in real-world clinical populations.

Harrell Jr F, Moons KGM, Debray TPA, Reitsma JB. A tutorial on individualized treatment effect prediction from randomized trials with a binary endpoint

Randomized trials typically estimate average relative treatment effects, but decisions on the benefit of a treatment are possibly better informed by more individualized predictions of the absolute treatment effect. In case of a binary outcome, these predictions of absolute individualized treatment effect require knowledge of the individual's risk without treatment and incorporation of a possibly differential treatment effect (ie, varying with patient characteristics). In this article, we lay out the causal structure of individualized treatment effect in terms of potential outcomes and describe the required assumptions that underlie a causal interpretation of its prediction. Subsequently, we describe regression models and model estimation techniques that can be used to move from average to more individualized treatment effect predictions. We focus mainly on logistic regression-based methods that are both well-known and naturally provide the required probabilistic estimates. We incorporate key components from both causal inference and prediction research to arrive at individualized treatment effect predictions. While the separate components are well known, their successful amalgamation is very much an ongoing field of research. We cut the problem down to its essentials in the setting of a randomized trial, discuss the importance of a clear definition of the estimand of interest, provide insight into the required assumptions, and give guidance with respect to modeling and estimation options. Simulated data illustrate the potential of different modeling options across scenarios that vary both average treatment effect and treatment effect heterogeneity. Two applied examples illustrate individualized treatment effect prediction in randomized trial data.

Joint modeling of liver transplant candidates outperforms the model for end-stage liver disease: The effect of disease development over time on patient outcome

Liver function is measured regularly in liver transplantation (LT) candidates. Currently, these previous disease development data are not used for survival prediction. By constructing and validating joint models (JMs), we aimed to predict the outcome based on all available data, using both disease severity and its rate of change over time. Adult LT candidates listed in Eurotransplant between 2007 and 2018 (n = 16 283) and UNOS between 2016 and 2019 (n = 30 533) were included. Patients with acute liver failure, exception points, or priority status were excluded. Longitudinal MELD(-Na) data were modeled using spline-based mixed effects. Waiting list survival was modeled with Cox proportional hazards models. The JMs combined the longitudinal and survival analysis. JM 90-day mortality prediction performance was compared to MELD(-Na) in the validation cohorts. MELD(-Na) score and its rate of change over time significantly influenced patient survival. The JMs significantly outperformed the MELD(-Na) score at baseline and during follow-up. At baseline, MELD-JM AUC and MELD AUC were 0.94 (0.92-0.95) and 0.87 (0.85-0.89), respectively. MELDNa-JM AUC was 0.91 (0.89-0.93) and MELD-Na AUC was 0.84 (0.81-0.87). The JMs were significantly (p < .001) more accurate than MELD(-Na). After 90 days, we ranked patients for LT based on their MELD-Na and MELDNa-JM survival rates, showing that MELDNa-JM-prioritized patients had three times higher waiting list mortality.

Prediction of Cardiovascular Disease Risk Accounting for Future Initiation of Statin Treatment

Cardiovascular disease (CVD) risk-prediction models are used to identify high-risk individuals and guide statin initiation. However, these models are usually derived from individuals who might initiate statins during follow-up. We present a simple approach to address statin initiation to predict “statin-naive” CVD risk. We analyzed primary care data (2004–2017) from the UK Clinical Practice Research Datalink for 1,678,727 individuals (aged 40–85 years) without CVD or statin treatment history at study entry. We derived age- and sex-specific prediction models including conventional risk factors and a time-dependent effect of statin initiation constrained to 25% risk reduction (from trial results). We compared predictive performance and measures of public-health impact (e.g., number needed to screen to prevent 1 event) against models ignoring statin initiation. During a median follow-up of 8.9 years, 103,163 individuals developed CVD. In models accounting for (versus ignoring) statin initiation, 10-year CVD risk predictions were slightly higher; predictive performance was moderately improved. However, few individuals were reclassified to a high-risk threshold, resulting in negligible improvements in number needed to screen to prevent 1 event. In conclusion, incorporating statin effects from trial results into risk-prediction models enables statin-naive CVD risk estimation and provides moderate gains in predictive ability but had a limited impact on treatment decision-making under current guidelines in this population.

Prediction or causality? A scoping review of their conflation within current observational research

Etiological research aims to uncover causal effects, whilst prediction research aims to forecast an outcome with the best accuracy. Causal and prediction research usually require different methods, and yet their findings may get conflated when reported and interpreted. The aim of the current study is to quantify the frequency of conflation between etiological and prediction research, to discuss common underlying mistakes and provide recommendations on how to avoid these. Observational cohort studies published in January 2018 in the top-ranked journals of six distinct medical fields (Cardiology, Clinical Epidemiology, Clinical Neurology, General and Internal Medicine, Nephrology and Surgery) were included for the current scoping review. Data on conflation was extracted through signaling questions. In total, 180 studies were included. Overall, 26% (n = 46) contained conflation between etiology and prediction. The frequency of conflation varied across medical field and journal impact factor. From the causal studies 22% was conflated, mainly due to the selection of covariates based on their ability to predict without taking the causal structure into account. Within prediction studies 38% was conflated, the most frequent reason was a causal interpretation of covariates included in a prediction model. Conflation of etiology and prediction is a common methodological error in observational medical research and more frequent in prediction studies. As this may lead to biased estimations and erroneous conclusions, researchers must be careful when designing, interpreting and disseminating their research to ensure this conflation is avoided.

Retrospective study on the possible existence of a treatment paradox in sepsis scores in the emergency department

OBJECTIVE

The quick Sequential Organ Failure Assessment (qSOFA) is developed as a tool to identify patients with infection with increased risk of dying from sepsis in non-intensive care unit settings, like the emergency department (ED). An abnormal score may trigger the initiation of appropriate therapy to reduce that risk. This study assesses the risk of a treatment paradox: the effect of a strong predictor for mortality will be reduced if that predictor also acts as a trigger for initiating treatment to prevent mortality.

DESIGN

Retrospective analysis on data from a large observational cohort.

SETTING

ED of a tertiary medical centre in the Netherlands.

PARTICIPANTS

3178 consecutive patients with suspected infection.

PRIMARY OUTCOME

To evaluate the existence of a treatment paradox by determining the influence of baseline qSOFA on treatment decisions within the first 24 hours after admission.

RESULTS

226 (7.1%) had a qSOFA ≥2, of which 51 (22.6%) died within 30 days. Area under receiver operating characteristics of qSOFA for 30-day mortality was 0.68 (95% CI 0.61 to 0.75). Patients with a qSOFA ≥2 had higher odds of receiving any form of intensive therapy (OR 11.4 (95% CI 7.5 to 17.1)), such as aggressive fluid resuscitation (OR 8.8 95% CI 6.6 to 11.8), fast antibiotic administration (OR 8.5, 95% CI 5.7 to 12.3) or vasopressic therapy (OR 17.3, 95% CI 11.2 to 26.8), compared with patients with qSOFA <2.

CONCLUSION

In ED patients with suspected infection, a qSOFA ≥2 was associated with more intensive treatment. This could lead to inadequate prediction of 30-day mortality due to the presence of a treatment paradox.

TRIAL REGISTRATION NUMBER

6916.

Covariates adjustment questioned conclusions of predictive analyses: an illustration with the Kidney Donor Risk Index

OBJECTIVES

We aimed to illustrate that considering covariates can lead to meaningful interpretation of the discriminative capacities of a prognostic marker. For this, we evaluated the ability of the Kidney Donor Risk Index (KDRI) to discriminate kidney graft failure risk.

STUDY DESIGN AND SETTING

From 4114 French patients, we estimated the adjusted area under the time-dependent ROC curve by standardizing the marker and weighting the observations. By weighting the contributions, we also studied the impact of KDRI-based transplantations on the patient and graft survival.

RESULTS

The covariate-adjusted AUC varied from 55% (95% confidence interval [CI]: 51-60%) for a prognostic up to 1 year post-transplantation to 56% (95% CI: 52-59%) up to 7 years. The Restricted Mean Survival Time (RMST) was 6.44 years for high-quality graft recipients (95% CI: 6.30-6.56) and would have been 6.31 years (95% CI: 6.13-6.46) if they had medium-quality transplants. The RMST was 5.10 years for low-quality graft recipients (95% CI: 4.90-5.31) and would have been 5.52 years (95% CI: 5.17-5.83) if they had medium-quality transplants.

CONCLUSION

We demonstrated that the KDRI discriminative capacities were mainly explained by the recipient characteristics. We also showed that counterfactual estimations, often used in causal studies, are also interesting in predictive studies, especially regarding the new available methods.

A scoping review of causal methods enabling predictions under hypothetical interventions

BACKGROUND

The methods with which prediction models are usually developed mean that neither the parameters nor the predictions should be interpreted causally. For many applications, this is perfectly acceptable. However, when prediction models are used to support decision making, there is often a need for predicting outcomes under hypothetical interventions.

AIMS

We aimed to identify published methods for developing and validating prediction models that enable risk estimation of outcomes under hypothetical interventions, utilizing causal inference. We aimed to identify the main methodological approaches, their underlying assumptions, targeted estimands, and potential pitfalls and challenges with using the method. Finally, we aimed to highlight unresolved methodological challenges.

METHODS

We systematically reviewed literature published by December 2019, considering papers in the health domain that used causal considerations to enable prediction models to be used for predictions under hypothetical interventions. We included both methodologies proposed in statistical/machine learning literature and methodologies used in applied studies.

RESULTS

We identified 4919 papers through database searches and a further 115 papers through manual searches. Of these, 87 papers were retained for full-text screening, of which 13 were selected for inclusion. We found papers from both the statistical and the machine learning literature. Most of the identified methods for causal inference from observational data were based on marginal structural models and g-estimation.

CONCLUSIONS

There exist two broad methodological approaches for allowing prediction under hypothetical intervention into clinical prediction models: (1) enriching prediction models derived from observational studies with estimated causal effects from clinical trials and meta-analyses and (2) estimating prediction models and causal effects directly from observational data. These methods require extending to dynamic treatment regimes, and consideration of multiple interventions to operationalise a clinical decision support system. Techniques for validating 'causal prediction models' are still in their infancy.

Multivariate prediction of mixed, multilevel, sequential outcomes arising from in vitro fertilisation

In vitro fertilisation (IVF) comprises a sequence of interventions concerned with the creation and culture of embryos which are then transferred to the patient's uterus. While the clinically important endpoint is birth, the responses to each stage of treatment contain additional information about the reasons for success or failure. As such, the ability to predict not only the overall outcome of the cycle, but also the stage-specific responses, can be useful. This could be done by developing separate models for each response variable, but recent work has suggested that it may be advantageous to use a multivariate approach to model all outcomes simultaneously. Here, joint analysis of the sequential responses is complicated by mixed outcome types defined at two levels (patient and embryo). A further consideration is whether and how to incorporate information about the response at each stage in models for subsequent stages. We develop a case study using routinely collected data from a large reproductive medicine unit in order to investigate the feasibility and potential utility of multivariate prediction in IVF. We consider two possible scenarios. In the first, stage-specific responses are to be predicted prior to treatment commencement. In the second, responses are predicted dynamically, using the outcomes of previous stages as predictors. In both scenarios, we fail to observe benefits of joint modelling approaches compared to fitting separate regression models for each response variable.

Performance of prediction models for COVID-19: the Caudine Forks of the external validation

Healthcare systems worldwide have observed significant changes to meet demands due to the coronavirus disease 2019 (COVID-19) pandemic. The uncertainty surrounding optimal treatment, the rapid public health urgency and clinical emergencies have caused a chaotic disruption of the cases and their related contacts at inpatient and outpatient settings. Developing more tailored healthcare plans based on the currently available scientific evidence, could help improve clinical efficacy, treatment outcomes, prognosis, and health efficiency.

Existing evidence suggests that none of the COVID-19 prediction models can be supported for clinical use. Here we discuss “what next” in COVID-19 prediction.https://bit.ly/2SMtoLV

Survival and analysis of predictors of mortality in patients undergoing replacement renal therapy: a 20-year cohort

Background

optimal management of end-stage renal disease (ESRD) in hemodialysis (HD) patients should be more studied because it is a serious risk factor for mortality, being considered an unquestionable global priority.

Methods

we performed a retrospective cohort study from the Nephrology Service in Brazil evaluating the survival of patients with ESRD in HD during 20 years. Kaplan-Meier method with the Log-Rank and Cox’s proportional hazards model explored the association between survival time and demographic factors, quality of treatment and laboratory values.

Results

Data from 422 patients were included. The mean survival time was 6.79 ± 0.37. The overall survival rates at first year was 82,3%. The survival time correlated significantly with clinical prognostic factors. Prognostic analyses with the Cox proportional hazards regression model and Kaplan-Meier survival curves further identified that leukocyte count (HR = 2.665, 95% CI: 1.39–5.12), serum iron (HR = 8.396, 95% CI: 2.02–34.96), serum calcium (HR = 4.102, 95% CI: 1.35–12.46) and serum protein (HR = 4.630, 95% CI: 2.07–10.34) as an independent risk factor for the prognosis of survival time, while patients with chronic obstructive pyelonephritis (HR = 0.085, 95% CI: 0.01–0.74), high ferritin values (HR = 0.392, 95% CI: 0.19–0.80), serum phosphorus (HR = 0.290, 95% CI: 0.19–0.61) and serum albumin (HR = 0.230, 95% CI: 0.10–0.54) were less risk to die.

Conclusion

survival remains low in the early years of ESRD treatment. The present study identified that elevated values of ferritin, serum calcium, phosphorus, albumin, leukocyte, serum protein and serum iron values as a useful prognostic factor for the survival time.

Prediction models for covid-19 outcomes

Reasons to be cautious