Predictive approaches to heterogeneous treatment effects: a scoping review

Estimating and presenting hazard ratios and absolute risks from a Cox model with complex nonlinear interactions

Interaction analysis is a critical component of clinical and public health research and represents a key topic in precision health and medicine. In applied settings, however, interaction assessment is usually limited to the test of a product term in a regression model and to the presentation of results stratified by levels of additional covariates. Stratification of results often relies on categorizing or making linearity assumptions for continuous covariates, with substantial loss of precision and of relevant information. In time-to-event analysis, moreover, interaction assessment is often limited to the multiplicative hazard scale by inclusion of a product term in a Cox regression model, disregarding the clinically relevant information that is captured by the absolute risk scale. In this paper we present a user-friendly procedure, based on the prediction of individual absolute risks from the Cox model, for the estimation and presentation of interactive effects on both the multiplicative and additive scales in survival analysis. We describe how to flexibly incorporate interactions with continuous covariates, which potentially operate in a nonlinear fashion, provide software for replicating our procedure, and discuss different approaches to deriving CIs. The presented approach will allow clinical and public health researchers to assess complex relationships between multiple covariates as they relate to a clinical endpoint, and to provide a more intuitive and precise depiction of the results in applied research papers focusing on interaction and effect stratification.

Evaluating individualized treatment effect predictions: A model-based perspective on discrimination and calibration assessment

In recent years, there has been a growing interest in the prediction of individualized treatment effects. While there is a rapidly growing literature on the development of such models, there is little literature on the evaluation of their performance. In this paper, we aim to facilitate the validation of prediction models for individualized treatment effects. The estimands of interest are defined based on the potential outcomes framework, which facilitates a comparison of existing and novel measures. In particular, we examine existing measures of discrimination for benefit (variations of the c-for-benefit), and propose model-based extensions to the treatment effect setting for discrimination and calibration metrics that have a strong basis in outcome risk prediction. The main focus is on randomized trial data with binary endpoints and on models that provide individualized treatment effect predictions and potential outcome predictions. We use simulated data to provide insight into the characteristics of the examined discrimination and calibration statistics under consideration, and further illustrate all methods in a trial of acute ischemic stroke treatment. The results show that the proposed model-based statistics had the best characteristics in terms of bias and accuracy. While resampling methods adjusted for the optimism of performance estimates in the development data, they had a high variance across replications that limited their accuracy. Therefore, individualized treatment effect models are best validated in independent data. To aid implementation, a software implementation of the proposed methods was made available in R.

Computational simulation of the potential improvement in clinical outcomes of cardiovascular diseases with the use of a personalized predictive medicine approach

IMPORTANCE AND OBJECTIVES

The current medical paradigm of evidence-based medicine relies on clinical guidelines derived from randomized clinical trials (RCTs), but these guidelines often overlook individual variations in treatment effects. Approaches have been proposed to develop models predicting the effects of individualized management, such as predictive allocation, individualizing treatment allocation. It is currently unknown whether widespread implementation of predictive allocation could result in better population-level outcomes over guideline-based therapy. We sought to simulate the potential effect of predictive allocation using data from previously conducted RCTs.

METHODS AND RESULTS

Data from 3 RCTs (positive trial, negative trial, trial stopped for futility) in pediatric cardiology were used in a computational simulation study to quantify the potential benefits of a personalized approach based on predictive allocation. Outcomes were compared when using a universal approach vs predictive allocation where each patient was allocated to the treatment associated with the lowest predicted probability of negative outcome. Compared to results from RCTs, predictive allocation yielded absolute risk reductions of 13.8% (95% confidence interval [CI] -1.9 to 29.5), 13.9% (95% CI 4.5-23.2), and 15.6% (95% CI 1.5-29.6), respectively, corresponding to a number needed to treat of 7.3, 7.2, and 6.4. The net benefit of predictive allocation was directly proportional to the performance of the prediction models and disappeared as model performance degraded below an area under the curve of 0.55.

DISCUSSION

These findings highlight that predictive allocation could result in improved group-level outcomes, particularly when highly predictive models are available. These findings will need to be confirmed in simulations of other trials with varying conditions and eventually in RCTs of predictive vs guideline-based treatment allocation.

Combining randomized and non-randomized data to predict heterogeneous effects of competing treatments

Some patients benefit from a treatment while others may do so less or do not benefit at all. We have previously developed a two-stage network meta-regression prediction model that synthesized randomized trials and evaluates how treatment effects vary across patient characteristics. In this article, we extended this model to combine different sources of types in different formats: aggregate data (AD) and individual participant data (IPD) from randomized and non-randomized evidence. In the first stage, a prognostic model is developed to predict the baseline risk of the outcome using a large cohort study. In the second stage, we recalibrated this prognostic model to improve our predictions for patients enrolled in randomized trials. In the third stage, we used the baseline risk as effect modifier in a network meta-regression model combining AD, IPD randomized clinical trial to estimate heterogeneous treatment effects. We illustrated the approach in the re-analysis of a network of studies comparing three drugs for relapsing-remitting multiple sclerosis. Several patient characteristics influence the baseline risk of relapse, which in turn modifies the effect of the drugs. The proposed model makes personalized predictions for health outcomes under several treatment options and encompasses all relevant randomized and non-randomized evidence.

Do machine learning methods lead to similar individualized treatment rules? A comparison study on real data

Identifying patients who benefit from a treatment is a key aspect of personalized medicine, which allows the development of individualized treatment rules (ITRs). Many machine learning methods have been proposed to create such rules. However, to what extent the methods lead to similar ITRs, that is, recommending the same treatment for the same individuals is unclear. In this work, we compared 22 of the most common approaches in two randomized control trials. Two classes of methods can be distinguished. The first class of methods relies on predicting individualized treatment effects from which an ITR is derived by recommending the treatment evaluated to the individuals with a predicted benefit. In the second class, methods directly estimate the ITR without estimating individualized treatment effects. For each trial, the performance of ITRs was assessed by various metrics, and the pairwise agreement between all ITRs was also calculated. Results showed that the ITRs obtained via the different methods generally had considerable disagreements regarding the patients to be treated. A better concordance was found among akin methods. Overall, when evaluating the performance of ITRs in a validation sample, all methods produced ITRs with limited performance, suggesting a high potential for optimism. For non-parametric methods, this optimism was likely due to overfitting. The different methods do not lead to similar ITRs and are therefore not interchangeable. The choice of the method strongly influences for which patients a certain treatment is recommended, drawing some concerns about their practical use.

Assessing the properties of patient-specific treatment effect estimates from causal forest algorithms under essential heterogeneity

BACKGROUND

Treatment variation from observational data has been used to estimate patient-specific treatment effects. Causal Forest Algorithms (CFAs) developed for this task have unknown properties when treatment effect heterogeneity from unmeasured patient factors influences treatment choice - essential heterogeneity.

METHODS

We simulated eleven populations with identical treatment effect distributions based on patient factors. The populations varied in the extent that treatment effect heterogeneity influenced treatment choice. We used the generalized random forest application (CFA-GRF) to estimate patient-specific treatment effects for each population. Average differences between true and estimated effects for patient subsets were evaluated.

RESULTS

CFA-GRF performed well across the population when treatment effect heterogeneity did not influence treatment choice. Under essential heterogeneity, however, CFA-GRF yielded treatment effect estimates that reflected true treatment effects only for treated patients and were on average greater than true treatment effects for untreated patients.

CONCLUSIONS

Patient-specific estimates produced by CFAs are sensitive to why patients in real-world practice make different treatment choices. Researchers using CFAs should develop conceptual frameworks of treatment choice prior to estimation to guide estimate interpretation ex post.

Use of conditional estimates of effect in cancer epidemiology: An application to lung cancer treatment

BACKGROUND

In oncology clinical trials, there is the assumption that randomization sufficiently balances confounding covariates and therefore average treatment effects are usually reported. This paper explores the wider benefits provided by conditioning on covariates for reasons other than mitigation of confounding.

METHODS

We reanalyzed the data from primary randomized controlled trials listed in two meta-analyses to explore the significance of conditioning on smoking status in terms of the effect magnitude of treatment on progression free survival in non-small cell lung cancer.

RESULTS

The reanalysis revealed that conditioning on smoking status using sub-group analyses provided the closest empiric estimate of individual treatment effect based on smoking status and significantly reduced the heterogeneity of treatment effect observed across studies. In addition, smoking status was determined to be a modifier of the effect of treatment.

CONCLUSION

Conditioning on prognostic covariates in randomized trials in oncology helps generate the closest empiric estimates of individual treatment benefit, addresses heterogeneity due to varying covariate distributions across trials and facilitates future decision making as well as evidence synthesis. Conditioning using sub-group analyses also allows examination for effect modification in meta-analysis.

Machine learning in precision diabetes care and cardiovascular risk prediction

Artificial intelligence and machine learning are driving a paradigm shift in medicine, promising data-driven, personalized solutions for managing diabetes and the excess cardiovascular risk it poses. In this comprehensive review of machine learning applications in the care of patients with diabetes at increased cardiovascular risk, we offer a broad overview of various data-driven methods and how they may be leveraged in developing predictive models for personalized care. We review existing as well as expected artificial intelligence solutions in the context of diagnosis, prognostication, phenotyping, and treatment of diabetes and its cardiovascular complications. In addition to discussing the key properties of such models that enable their successful application in complex risk prediction, we define challenges that arise from their misuse and the role of methodological standards in overcoming these limitations. We also identify key issues in equity and bias mitigation in healthcare and discuss how the current regulatory framework should ensure the efficacy and safety of medical artificial intelligence products in transforming cardiovascular care and outcomes in diabetes.

Performance metrics for models designed to predict treatment effect

BACKGROUND

Measuring the performance of models that predict individualized treatment effect is challenging because the outcomes of two alternative treatments are inherently unobservable in one patient. The C-for-benefit was proposed to measure discriminative ability. However, measures of calibration and overall performance are still lacking. We aimed to propose metrics of calibration and overall performance for models predicting treatment effect in randomized clinical trials (RCTs).

METHODS

Similar to the previously proposed C-for-benefit, we defined observed pairwise treatment effect as the difference between outcomes in pairs of matched patients with different treatment assignment. We match each untreated patient with the nearest treated patient based on the Mahalanobis distance between patient characteristics. Then, we define the Eavg-for-benefit, E50-for-benefit, and E90-for-benefit as the average, median, and 90th quantile of the absolute distance between the predicted pairwise treatment effects and local-regression-smoothed observed pairwise treatment effects. Furthermore, we define the cross-entropy-for-benefit and Brier-for-benefit as the logarithmic and average squared distance between predicted and observed pairwise treatment effects. In a simulation study, the metric values of deliberately "perturbed models" were compared to those of the data-generating model, i.e., "optimal model". To illustrate these performance metrics, different modeling approaches for predicting treatment effect are applied to the data of the Diabetes Prevention Program: 1) a risk modelling approach with restricted cubic splines; 2) an effect modelling approach including penalized treatment interactions; and 3) the causal forest.

RESULTS

As desired, performance metric values of "perturbed models" were consistently worse than those of the "optimal model" (Eavg-for-benefit ≥ 0.043 versus 0.002, E50-for-benefit ≥ 0.032 versus 0.001, E90-for-benefit ≥ 0.084 versus 0.004, cross-entropy-for-benefit ≥ 0.765 versus 0.750, Brier-for-benefit ≥ 0.220 versus 0.218). Calibration, discriminative ability, and overall performance of three different models were similar in the case study. The proposed metrics were implemented in a publicly available R-package "HTEPredictionMetrics".

CONCLUSION

The proposed metrics are useful to assess the calibration and overall performance of models predicting treatment effect in RCTs.

Estimating Patient-Specific Relative Benefit of Adding Biologics to Conventional Rheumatoid Arthritis Treatment: An Individual Participant Data Meta-Analysis

Importance

Current evidence remains ambiguous regarding whether biologics should be added to conventional treatment of rheumatoid arthritis for specific patients, which may cause potential overuse or treatment delay.

Objectives

To estimate the benefit of adding biologics to conventional antirheumatic drugs for the treatment of rheumatoid arthritis given baseline characteristics.

Data Sources

Cochrane CENTRAL, Scopus, MEDLINE, and the World Health Organization International Clinical Trials Registry Platform were searched for articles published from database inception to March 2, 2022.

Study Selection

Randomized clinical trials comparing certolizumab plus conventional antirheumatic drugs with placebo plus conventional drugs were selected.

Data Extraction and Synthesis

Individual participant data of the prespecified outcomes and covariates were acquired from the Vivli database. A 2-stage model was fitted to estimate patient-specific relative outcomes of adding certolizumab vs conventional drugs only. Stage 1 was a penalized logistic regression model to estimate the baseline expected probability of the outcome regardless of treatment using baseline characteristics. Stage 2 was a bayesian individual participant data meta-regression model to estimate the relative outcomes for a particular baseline expected probability. Patient-specific results were displayed interactively on an application based on a 2-stage model.

Main Outcomes and Measures

The primary outcome was low disease activity or remission at 3 months, defined by 3 disease activity indexes (ie, Disease Activity Score based on the evaluation of 28 joints, Clinical Disease Activity Index, or Simplified Disease Activity Index).

Results

Individual participant data were obtained from 3790 patients (2996 female [79.1%] and 794 male [20.9%]; mean [SD] age, 52.7 [12.3] years) from 5 large randomized clinical trials for moderate to high activity rheumatoid arthritis with usable data for 22 prespecified baseline covariates. Overall, adding certolizumab was associated with a higher probability of reaching low disease activity. The odds ratio for patients with an average baseline expected probability of the outcome was 6.31 (95% credible interval, 2.22-15.25). However, the benefits differed in patients with different baseline characteristics. For example, the estimated risk difference was smaller than 10% for patients with either low or high baseline expected probability.

Conclusions and Relevance

In this individual participant data meta-analysis, adding certolizumab was associated with more effectiveness for rheumatoid arthritis in general. However, the benefit was uncertain for patients with low or high baseline expected probability, for whom other evaluations were necessary. The interactive application displaying individual estimates may help with treatment selection.

Methodological concerns about "concordance-statistic for benefit" as a measure of discrimination in predicting treatment benefit

Prediction algorithms that quantify the expected benefit of a given treatment conditional on patient characteristics can critically inform medical decisions. Quantifying the performance of treatment benefit prediction algorithms is an active area of research. A recently proposed metric, the concordance statistic for benefit (cfb), evaluates the discriminative ability of a treatment benefit predictor by directly extending the concept of the concordance statistic from a risk model with a binary outcome to a model for treatment benefit. In this work, we scrutinize cfb on multiple fronts. Through numerical examples and theoretical developments, we show that cfb is not a proper scoring rule. We also show that it is sensitive to the unestimable correlation between counterfactual outcomes and to the definition of matched pairs. We argue that measures of statistical dispersion applied to predicted benefits do not suffer from these issues and can be an alternative metric for the discriminatory performance of treatment benefit predictors.

Measuring the performance of prediction models to personalize treatment choice

When data are available from individual patients receiving either a treatment or a control intervention in a randomized trial, various statistical and machine learning methods can be used to develop models for predicting future outcomes under the two conditions, and thus to predict treatment effect at the patient level. These predictions can subsequently guide personalized treatment choices. Although several methods for validating prediction models are available, little attention has been given to measuring the performance of predictions of personalized treatment effect. In this article, we propose a range of measures that can be used to this end. We start by defining two dimensions of model accuracy for treatment effects, for a single outcome: discrimination for benefit and calibration for benefit. We then amalgamate these two dimensions into an additional concept, decision accuracy, which quantifies the model's ability to identify patients for whom the benefit from treatment exceeds a given threshold. Subsequently, we propose a series of performance measures related to these dimensions and discuss estimating procedures, focusing on randomized data. Our methods are applicable for continuous or binary outcomes, for any type of prediction model, as long as it uses baseline covariates to predict outcomes under treatment and control. We illustrate all methods using two simulated datasets and a real dataset from a trial in depression. We implement all methods in the R package predieval. Results suggest that the proposed measures can be useful in evaluating and comparing the performance of competing models in predicting individualized treatment effect.

Decision Curve Analysis for Personalized Treatment Choice between Multiple Options

BACKGROUND

Decision curve analysis can be used to determine whether a personalized model for treatment benefit would lead to better clinical decisions. Decision curve analysis methods have been described to estimate treatment benefit using data from a single randomized controlled trial.

OBJECTIVES

Our main objective is to extend the decision curve analysis methodology to the scenario in which several treatment options exist and evidence about their effects comes from a set of trials, synthesized using network meta-analysis (NMA).

METHODS

We describe the steps needed to estimate the net benefit of a prediction model using evidence from studies synthesized in an NMA. We show how to compare personalized versus one-size-fit-all treatment decision-making strategies, such as "treat none" or "treat all patients with a specific treatment" strategies. First, threshold values for each included treatment need to be defined (i.e., the minimum risk difference compared with control that renders a treatment worth taking). The net benefit per strategy can then be plotted for a plausible range of threshold values to reveal the most clinically useful strategy. We applied our methodology to an NMA prediction model for relapsing-remitting multiple sclerosis, which can be used to choose between natalizumab, dimethyl fumarate, glatiramer acetate, and placebo.

RESULTS

We illustrated the extended decision curve analysis methodology using several threshold value combinations for each available treatment. For the examined threshold values, the "treat patients according to the prediction model" strategy performs either better than or close to the one-size-fit-all treatment strategies. However, even small differences may be important in clinical decision making. As the advantage of the personalized model was not consistent across all thresholds, improving the existing model (by including, for example, predictors that will increase discrimination) is needed before advocating its clinical usefulness.

CONCLUSIONS

This novel extension of decision curve analysis can be applied to NMA-based prediction models to evaluate their use to aid treatment decision making.

HIGHLIGHTS

Decision curve analysis is extended into a (network) meta-analysis framework.Personalized models predicting treatment benefit are evaluated when several treatment options are available and evidence about their effects comes from a set of trials.Detailed steps to compare personalized versus one-size-fit-all treatment decision-making strategies are outlined.This extension of decision curve analysis can be applied to (network) meta-analysis-based prediction models to evaluate their use to aid treatment decision making.

A standardized framework for risk-based assessment of treatment effect heterogeneity in observational healthcare databases

Treatment effects are often anticipated to vary across groups of patients with different baseline risk. The Predictive Approaches to Treatment Effect Heterogeneity (PATH) statement focused on baseline risk as a robust predictor of treatment effect and provided guidance on risk-based assessment of treatment effect heterogeneity in a randomized controlled trial. The aim of this study is to extend this approach to the observational setting using a standardized scalable framework. The proposed framework consists of five steps: (1) definition of the research aim, i.e., the population, the treatment, the comparator and the outcome(s) of interest; (2) identification of relevant databases; (3) development of a prediction model for the outcome(s) of interest; (4) estimation of relative and absolute treatment effect within strata of predicted risk, after adjusting for observed confounding; (5) presentation of the results. We demonstrate our framework by evaluating heterogeneity of the effect of thiazide or thiazide-like diuretics versus angiotensin-converting enzyme inhibitors on three efficacy and nine safety outcomes across three observational databases. We provide a publicly available R software package for applying this framework to any database mapped to the Observational Medical Outcomes Partnership Common Data Model. In our demonstration, patients at low risk of acute myocardial infarction receive negligible absolute benefits for all three efficacy outcomes, though they are more pronounced in the highest risk group, especially for acute myocardial infarction. Our framework allows for the evaluation of differential treatment effects across risk strata, which offers the opportunity to consider the benefit-harm trade-off between alternative treatments.

Estimating individualized treatment effects from randomized controlled trials: a simulation study to compare risk-based approaches

BACKGROUND

Baseline outcome risk can be an important determinant of absolute treatment benefit and has been used in guidelines for "personalizing" medical decisions. We compared easily applicable risk-based methods for optimal prediction of individualized treatment effects.

METHODS

We simulated RCT data using diverse assumptions for the average treatment effect, a baseline prognostic index of risk, the shape of its interaction with treatment (none, linear, quadratic or non-monotonic), and the magnitude of treatment-related harms (none or constant independent of the prognostic index). We predicted absolute benefit using: models with a constant relative treatment effect; stratification in quarters of the prognostic index; models including a linear interaction of treatment with the prognostic index; models including an interaction of treatment with a restricted cubic spline transformation of the prognostic index; an adaptive approach using Akaike's Information Criterion. We evaluated predictive performance using root mean squared error and measures of discrimination and calibration for benefit.

RESULTS

The linear-interaction model displayed optimal or close-to-optimal performance across many simulation scenarios with moderate sample size (N = 4,250; ~ 785 events). The restricted cubic splines model was optimal for strong non-linear deviations from a constant treatment effect, particularly when sample size was larger (N = 17,000). The adaptive approach also required larger sample sizes. These findings were illustrated in the GUSTO-I trial.

CONCLUSIONS

An interaction between baseline risk and treatment assignment should be considered to improve treatment effect predictions.

Analgesic and Anxiolytic Effects of Virtual Reality During Minor Procedures in an Emergency Department: A Randomized Controlled Study

STUDY OBJECTIVE

We aimed to assess the analgesic and anxiolytic efficacy of distraction, a nonpharmacologic intervention provided by 3-dimensional (3D) virtual reality (VR) compared with that provided by 2-dimensional (2D) VR during minor emergency department (ED) procedures.

METHODS

This randomized controlled study conducted in the ED of a teaching hospital included patients aged more than or equal to 18 years undergoing minor procedures. The patients watched the same computer-generated VR world either in 3D in a head-mounted display (intervention) or in 2D on a laptop screen (control). Our main outcomes were pain and anxiety during the procedure, assessed on a 100-mm visual analog scale. Secondary outcomes included the impression of telepresence in the computer-generated world assessed using the Igroup Presence Questionnaire, and the prevalence and intensity of cybersickness measured on a 100-mm visual analog scale.

RESULTS

The final analysis included 117 patients. The differences in median procedural pain and anxiety levels between the 2D and 3D VR groups were not significant: -3 mm (95% confidence interval [CI] -14 to 8) and -4 mm (95% CI -15 to 3), respectively; the difference in telepresence was 2.0 point (95% CI 0 to 2.0), and the proportion difference of cybersickness was -4% (95% CI -22 to 14), with an intensity difference of -5 mm (95% CI -9 to 3).

CONCLUSION

During minor procedures in adult patients in the ED, distraction by viewing a 3D virtual world in a head-mounted VR display did not result in lower average levels of procedural pain and anxiety than that by 2D viewing on a screen despite a higher sense of telepresence. There were no significant differences in the prevalence and intensity of cybersickness between the 2 groups.

Redefining effect modification

The odds ratio (OR) has been misunderstood in evidence based medicine and clinical epidemiology. Currently, "noncollapsibility" is considered a problem with interpretation of the OR and it is thought that the OR is rarely the parameter of interest for causal inference or interpretation of effect modification. The current focus on the relative risk (RR) and risk difference (RD) suffers from an important limitation: they are not solely measures of effect and vary numerically with baseline risk. In this paper, generalized linear models are examined in terms of the three binary effect measures commonly used in epidemiology to demonstrate that ORs may be the only way to interpret effect modification and have properties that should make them the parameter of interest for causal inference. We look forward to discussion, debate, and counter-views on this issue from the epidemiology community.

Assessing the ability of an instrumental variable causal forest algorithm to personalize treatment evidence using observational data: the case of early surgery for shoulder fracture

Background

Comparative effectiveness research (CER) using observational databases has been suggested to obtain personalized evidence of treatment effectiveness. Inferential difficulties remain using traditional CER approaches especially related to designating patients to reference classes a priori. A novel Instrumental Variable Causal Forest Algorithm (IV-CFA) has the potential to provide personalized evidence using observational data without designating reference classes a priori, but the consistency of the evidence when varying key algorithm parameters remains unclear. We investigated the consistency of IV-CFA estimates through application to a database of Medicare beneficiaries with proximal humerus fractures (PHFs) that previously revealed heterogeneity in the effects of early surgery using instrumental variable estimators.

Methods

IV-CFA was used to estimate patient-specific early surgery effects on both beneficial and detrimental outcomes using different combinations of algorithm parameters and estimate variation was assessed for a population of 72,751 fee-for-service Medicare beneficiaries with PHFs in 2011. Classification and regression trees (CART) were applied to these estimates to create ex-post reference classes and the consistency of these classes were assessed. Two-stage least squares (2SLS) estimators were applied to representative ex-post reference classes to scrutinize the estimates relative to known 2SLS properties.

Results

IV-CFA uncovered substantial early surgery effect heterogeneity across PHF patients, but estimates for individual patients varied with algorithm parameters. CART applied to these estimates revealed ex-post reference classes consistent across algorithm parameters. 2SLS estimates showed that ex-post reference classes containing older, frailer patients with more comorbidities, and lower utilizers of healthcare were less likely to benefit and more likely to have detriments from higher rates of early surgery.

Conclusions

IV-CFA provides an illuminating method to uncover ex-post reference classes of patients based on treatment effects using observational data with a strong instrumental variable. Interpretation of treatment effect estimates within each ex-post reference class using traditional CER methods remains conditional on the extent of measured information in the data.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-022-01663-0.

Reported Challenges in Health Technology Assessment of Complex Health Technologies

OBJECTIVES

With complex health technologies entering the market, methods for health technology assessment (HTA) may require changes. This study aimed to identify challenges in HTA of complex health technologies.

METHODS

A survey was sent to European HTA organizations participating in European Network for HTA (EUnetHTA). The survey contained open questions and used predefined potentially complex health technologies and 7 case studies to identify types of complex health technologies and challenges faced during HTA. The survey was validated, tested for reliability by an expert panel, and pilot tested before dissemination.

RESULTS

A total of 22 HTA organizations completed the survey (67%). Advanced therapeutic medicinal products (ATMPs) and histology-independent therapies were considered most challenging based on the predefined complex health technologies and case studies. For the case studies, more than half of the reported challenges were "methodological," equal in relative effectiveness assessments as in cost-effectiveness assessments. Through the open questions, we found that most of these challenges actually rooted in data unavailability. Data were reported as "absent," "insufficient," "immature," or "low quality" by 18 of 20 organizations (90%), in particular data on quality of life. Policy and organizational challenges and challenges because of societal or political pressure were reported by 8 (40%) and 4 organizations (20%), respectively. Modeling issues were reported least often (n = 2, 4%).

CONCLUSIONS

Most challenges in HTA of complex health technologies root in data insufficiencies rather than in the complexity of health technologies itself. As the number of complex technologies grows, the urgency for new methods and policies to guide HTA decision making increases.

Prediction of treatment outcome in clinical trials under a personalized medicine perspective

A central problem in most data-driven personalized medicine scenarios is the estimation of heterogeneous treatment effects to stratify individuals into subpopulations that differ in their susceptibility to a particular disease or response to a specific treatment. In this work, with an illustrative example on type 2 diabetes we showed how the increasing ability to access and analyzed open data from randomized clinical trials (RCTs) allows to build Machine Learning applications in a framework of personalized medicine. An ensemble machine learning predictive model is first developed and then applied to estimate the expected treatment response according to the medication that would be prescribed. Machine learning is quickly becoming indispensable to bridge science and clinical practice, but it is not sufficient on its own. A collaborative effort is requested to clinicians, statisticians, and computer scientists to strengthen tools built on machine learning to take advantage of this evidence flow.

Pragmatic Precision Psychiatry-A New Direction for Optimizing Treatment Selection

IMPORTANCE

Clinical trials have identified numerous prescriptive predictors of mental disorder treatment response, ie, predictors of which treatments are best for which patients. However, none of these prescriptive predictors is strong enough alone to guide precision treatment planning. This has prompted growing interest in developing precision treatment rules (PTRs) that combine information across multiple prescriptive predictors, but this work has been much less successful in psychiatry than some other areas of medicine. Study designs and analysis schemes used in research on PTR development in other areas of medicine are reviewed, key challenges for implementing similar studies of mental disorders are highlighted, and recent methodological advances to address these challenges are described here.

OBSERVATIONS

Discovering prescriptive predictors requires large samples. Three approaches have been used in other areas of medicine to do this: conduct very large randomized clinical trials, pool individual-level results across multiple smaller randomized clinical trials, and develop preliminary PTRs in large observational treatment samples that are then tested in smaller randomized clinical trials. The third approach is most feasible for research on mental disorders. This approach requires working with large real-world observational electronic health record databases; carefully selecting samples to emulate trials; extracting information about prescriptive predictors from electronic health records along with other inexpensive data augmentation strategies; estimating preliminary PTRs in the observational data using appropriate methods; implementing pragmatic trials to validate the preliminary PTRs; and iterating between subsequent observational studies and quality improvement pragmatic trials to refine and expand the PTRs. New statistical methods exist to address the methodological challenges of implementing this approach.

CONCLUSIONS AND RELEVANCE

Advances in pragmatic precision psychiatry will require moving beyond the current focus on randomized clinical trials and adopting an iterative discovery-confirmation process that integrates observational and experimental studies in real-world clinical populations.

Leveraging machine learning predictive biomarkers to augment the statistical power of clinical trials with baseline magnetic resonance imaging

A key factor in designing randomized clinical trials is the sample size required to achieve a particular level of power to detect the benefit of a treatment. Sample size calculations depend upon the expected benefits of a treatment (effect size), the accuracy of measurement of the primary outcome, and the level of power specified by the investigators. In this study, we show that radiomic models, which leverage complex brain MRI patterns and machine learning, can be utilized in clinical trials with protocols that incorporate baseline MR imaging to significantly increase statistical power to detect treatment effects. Akin to the historical control paradigm, we propose to utilize a radiomic prediction model to generate a pseudo-control sample for each individual in the trial of interest. Because the variability of expected outcome across patients can mask our ability to detect treatment effects, we can increase the power to detect a treatment effect in a clinical trial by reducing that variability through using radiomic predictors as surrogates. We illustrate this method with simulations based on data from two cohorts in different neurologic diseases, Alzheimer's disease and glioblastoma multiforme. We present sample size requirements across a range of effect sizes using conventional analysis and models that include a radiomic predictor. For our Alzheimer's disease cohort, at an effect size of 0.35, total sample size requirements for 80% power declined from 246 to 212 for the endpoint cognitive decline. For our glioblastoma multiforme cohort, at an effect size of 1.65 with the endpoint survival time, total sample size requirements declined from 128 to 74. This methodology can decrease the required sample sizes by as much as 50%, depending on the strength of the radiomic predictor. The power of this method grows with increased accuracy of radiomic prediction, and furthermore, this method is most helpful when treatment effect sizes are small. Neuroimaging biomarkers are a powerful and increasingly common suite of tools that are, in many cases, highly predictive of disease outcomes. Here, we explore the possibility of using MRI-based radiomic biomarkers for the purpose of improving statistical power in clinical trials in the contexts of brain cancer and prodromal Alzheimer's disease. These methods can be applied to a broad range of neurologic diseases using a broad range of predictors of outcome to make clinical trials more efficient.

Harrell Jr F, Moons KGM, Debray TPA, Reitsma JB. A tutorial on individualized treatment effect prediction from randomized trials with a binary endpoint

Randomized trials typically estimate average relative treatment effects, but decisions on the benefit of a treatment are possibly better informed by more individualized predictions of the absolute treatment effect. In case of a binary outcome, these predictions of absolute individualized treatment effect require knowledge of the individual's risk without treatment and incorporation of a possibly differential treatment effect (ie, varying with patient characteristics). In this article, we lay out the causal structure of individualized treatment effect in terms of potential outcomes and describe the required assumptions that underlie a causal interpretation of its prediction. Subsequently, we describe regression models and model estimation techniques that can be used to move from average to more individualized treatment effect predictions. We focus mainly on logistic regression-based methods that are both well-known and naturally provide the required probabilistic estimates. We incorporate key components from both causal inference and prediction research to arrive at individualized treatment effect predictions. While the separate components are well known, their successful amalgamation is very much an ongoing field of research. We cut the problem down to its essentials in the setting of a randomized trial, discuss the importance of a clear definition of the estimand of interest, provide insight into the required assumptions, and give guidance with respect to modeling and estimation options. Simulated data illustrate the potential of different modeling options across scenarios that vary both average treatment effect and treatment effect heterogeneity. Two applied examples illustrate individualized treatment effect prediction in randomized trial data.

A two-stage prediction model for heterogeneous effects of treatments

Treatment effects vary across different patients, and estimation of this variability is essential for clinical decision‐making. We aimed to develop a model estimating the benefit of alternative treatment options for individual patients, extending a risk modeling approach in a network meta‐analysis framework. We propose a two‐stage prediction model for heterogeneous treatment effects by combining prognosis research and network meta‐analysis methods where individual patient data are available. In the first stage, a prognostic model to predict the baseline risk of the outcome. In the second stage, we use the baseline risk score from the first stage as a single prognostic factor and effect modifier in a network meta‐regression model. We apply the approach to a network meta‐analysis of three randomized clinical trials comparing the relapses in Natalizumab, Glatiramer Acetate, and Dimethyl Fumarate, including 3590 patients diagnosed with relapsing‐remitting multiple sclerosis. We find that the baseline risk score modifies the relative and absolute treatment effects. Several patient characteristics, such as age and disability status, impact the baseline risk of relapse, which in turn moderates the benefit expected for each of the treatments. For high‐risk patients, the treatment that minimizes the risk of relapse in 2 years is Natalizumab, whereas Dimethyl Fumarate might be a better option for low‐risk patients. Our approach can be easily extended to all outcomes of interest and has the potential to inform a personalized treatment approach.