Assessing heterogeneity of treatment effect in a clinical trial with the proportional interactions model

Design of a multicenter randomized controlled trial of a post-discharge suicide prevention intervention for high-risk psychiatric inpatients: The Veterans Coordinated Community Care Study

BACKGROUND

The period after psychiatric hospital discharge is one of elevated risk for suicide-related behaviors (SRBs). Post-discharge clinical outreach, although potentially effective in preventing SRBs, would be more cost-effective if targeted at high-risk patients. To this end, a machine learning model was developed to predict post-discharge suicides among Veterans Health Administration (VHA) psychiatric inpatients and target a high-risk preventive intervention.

METHODS

The Veterans Coordinated Community Care (3C) Study is a multicenter randomized controlled trial using this model to identify high-risk VHA psychiatric inpatients (n = 850) randomized with equal allocation to either the Coping Long Term with Active Suicide Program (CLASP) post-discharge clinical outreach intervention or treatment-as-usual (TAU). The primary outcome is SRBs over a 6-month follow-up. We will estimate average treatment effects adjusted for loss to follow-up and investigate the possibility of heterogeneity of treatment effects.

RESULTS

Recruitment is underway and will end September 2024. Six-month follow-up will end and analysis will begin in Summer 2025.

CONCLUSION

Results will provide information about the effectiveness of CLASP versus TAU in reducing post-discharge SRBs and provide guidance to VHA clinicians and policymakers about the implications of targeted use of CLASP among high-risk psychiatric inpatients in the months after hospital discharge.

CLINICAL TRIALS REGISTRATION

ClinicalTrials.Gov identifier: NCT05272176 (https://www.

CLINICALTRIALS

gov/ct2/show/NCT05272176).

Model-based adaptive randomization procedures for heteroscedasticity of treatment responses

In clinical trials, the responses of patients usually depend on the assigned treatment as well as some important covariates, which may cause heteroscedasticity in treatment responses. As clinical trials are generally designed to demonstrate efficacy for the overall population, they are usually not adequately powered for detecting interactions. To improve the power of interaction tests, this article develops two model-based adaptive randomization procedures for heteroscedasticity of treatment responses, and derives their limiting allocation proportions, which are generalizations of the Neyman allocation. Issues of hypothesis testing and sample size estimation are also addressed. Simulation studies show that compared with complete randomization, the two model-based randomization procedures have greater power to detect differences in systematic effects, main treatment effects and treatment-covariate interactions. In addition, the validity of limiting allocation proportion is also verified through simulations.

Assessing Heterogeneity of Treatment Effect in Real-World Data

Increasing availability of real-world data (RWD) generated from patient care enables the generation of evidence to inform clinical decisions for subpopulations of patients and perhaps even individuals. There is growing opportunity to identify important heterogeneity of treatment effects (HTE) in these subgroups. Thus, HTE is relevant to all with interest in patients' responses to interventions, including regulators who must make decisions about products when signals of harms arise postapproval and payers who make coverage decisions based on expected net benefit to their beneficiaries. Prior work discussed HTE in randomized studies. Here, we address methodological considerations when investigating HTE in observational studies. We propose 4 primary goals of HTE analyses and the corresponding approaches in the context of RWD: to confirm subgroup effects, to describe the magnitude of HTE, to discover clinically important subgroups, and to predict individual effects. We discuss other possible goals including exploring prognostic score- and propensity score-based treatment effects, and testing the transportability of trial results to populations different from trial participants. Finally, we outline methodological needs for enhancing real-world HTE analysis.

Bayesian Statistics for Medical Devices: Progress Since 2010

The use of Bayesian statistics to support regulatory evaluation of medical devices began in the late 1990s. We review the literature, focusing on recent developments of Bayesian methods, including hierarchical modeling of studies and subgroups, borrowing strength from prior data, effective sample size, Bayesian adaptive designs, pediatric extrapolation, benefit-risk decision analysis, use of real-world evidence, and diagnostic device evaluation. We illustrate how these developments were utilized in recent medical device evaluations. In Supplementary Material, we provide a list of medical devices for which Bayesian statistics were used to support approval by the US Food and Drug Administration (FDA), including those since 2010, the year the FDA published their guidance on Bayesian statistics for medical devices. We conclude with a discussion of current and future challenges and opportunities for Bayesian statistics, including artificial intelligence/machine learning (AI/ML) Bayesian modeling, uncertainty quantification, Bayesian approaches using propensity scores, and computational challenges for high dimensional data and models.

Evaluating the heterogeneous effect of a modifiable risk factor on suicide: The case of vitamin D deficiency

OBJECTIVES

To illustrate the use of machine learning methods to search for heterogeneous effects of a target modifiable risk factor on suicide in observational studies. The illustration focuses on secondary analysis of a matched case-control study of vitamin D deficiency predicting subsequent suicide.

METHODS

We describe a variety of machine learning methods to search for prescriptive predictors; that is, predictors of significant variation in the association between a target risk factor and subsequent suicide. In each case, the purpose is to evaluate the potential value of selective intervention on the target risk factor to prevent the outcome based on the provisional assumption that the target risk factor is causal. The approaches illustrated include risk modeling based on the super learner ensemble machine learning method, Least Absolute Shrinkage and Selection Operator (Lasso) penalized regression, and the causal forest algorithm.

RESULTS

The logic of estimating heterogeneous intervention effects is exposited along with the illustration of some widely used methods for implementing this logic.

CONCLUSIONS

In addition to describing best practices in using the machine learning methods considered here, we close with a discussion of broader design and analysis issues in planning an observational study to investigate heterogeneous effects of a modifiable risk factor.

Enrichment Bayesian design for randomized clinical trials using categorical biomarkers and a binary outcome

Background

Adaptive clinical trials have been increasingly commonly employed to select a potential target population for one trial without conducting trials separately. Such enrichment designs typically consist of two or three stages, where the first stage serves as a screening process for selecting a specific subpopulation.

Methods

We propose a Bayesian design for randomized clinical trials with a binary outcome that focuses on restricting the inclusion to a subset of patients who are likely to benefit the most from the treatment during trial accrual. Several Bayesian measures of efficacy and treatment-by-subset interactions were used to dictate the enrichment, either based on Gail and Simon’s or Millen’s criteria. A simulation study was used to assess the performance of our design. The method is exemplified in a real randomized clinical trial conducted in patients with respiratory failure that failed to show any benefit of high flow oxygen supply compared with standard oxygen.

Results

The use of the enrichment rules allowed the detection of the existence of a treatment-by-subset interaction more rapidly compared with Gail and Simon’s criteria, with decreasing proportions of enrollment in the whole sample, and the proportions of enrichment lower, in the presence of interaction based on Millen’s criteria. In the real dataset, this may have allowed the detection of the potential interest of high flow oxygen in patients with a SOFA neurological score ≥ 1.

Conclusion

Enrichment designs that handle the uncertainty in treatment efficacy by focusing on the target population offer a promising balance for trial efficiency and ease of interpretation.

Supplementary Information

The online version contains supplementary material available at (10.1186/s12874-022-01513-z).

Pragmatic Precision Psychiatry-A New Direction for Optimizing Treatment Selection

IMPORTANCE

Clinical trials have identified numerous prescriptive predictors of mental disorder treatment response, ie, predictors of which treatments are best for which patients. However, none of these prescriptive predictors is strong enough alone to guide precision treatment planning. This has prompted growing interest in developing precision treatment rules (PTRs) that combine information across multiple prescriptive predictors, but this work has been much less successful in psychiatry than some other areas of medicine. Study designs and analysis schemes used in research on PTR development in other areas of medicine are reviewed, key challenges for implementing similar studies of mental disorders are highlighted, and recent methodological advances to address these challenges are described here.

OBSERVATIONS

Discovering prescriptive predictors requires large samples. Three approaches have been used in other areas of medicine to do this: conduct very large randomized clinical trials, pool individual-level results across multiple smaller randomized clinical trials, and develop preliminary PTRs in large observational treatment samples that are then tested in smaller randomized clinical trials. The third approach is most feasible for research on mental disorders. This approach requires working with large real-world observational electronic health record databases; carefully selecting samples to emulate trials; extracting information about prescriptive predictors from electronic health records along with other inexpensive data augmentation strategies; estimating preliminary PTRs in the observational data using appropriate methods; implementing pragmatic trials to validate the preliminary PTRs; and iterating between subsequent observational studies and quality improvement pragmatic trials to refine and expand the PTRs. New statistical methods exist to address the methodological challenges of implementing this approach.

CONCLUSIONS AND RELEVANCE

Advances in pragmatic precision psychiatry will require moving beyond the current focus on randomized clinical trials and adopting an iterative discovery-confirmation process that integrates observational and experimental studies in real-world clinical populations.

Lasso estimation of hierarchical interactions for analyzing heterogeneity of treatment effect

Individuals differ in how they respond to a given treatment. In an effort to predict the treatment response and analyze the heterogeneity of treatment effect, we propose a general modeling framework by identifying treatment-covariate interactions honoring a hierarchical condition. We construct a single-step l 1 norm penalty procedure that maintains the hierarchical structure of interactions in the sense that a treatment-covariate interaction term is included in the model only when either the covariate or both the covariate and treatment have nonzero main effects. We developed a constrained Lasso approach with two parameterization schemes that enforce the hierarchical interaction restriction differently. We solved the resulting constrained optimization problem using a spectral projected gradient method. We compared our methods to the unstructured Lasso using simulation studies including a scenario that violates the hierarchical condition (misspecified model). The simulations showed that our methods yielded more parsimonious models and outperformed the unstructured Lasso for correctly identifying nonzero treatment-covariate interactions. The superior performance of our methods are also corroborated by an application to a large randomized clinical trial data investigating a drug for treating congestive heart failure (N = 2569). Our methods provide a well-suited approach for doing secondary analysis in clinical trials to analyze heterogeneous treatment effects and to identify predictive biomarkers.

Preventing asthma in high risk kids (PARK) with omalizumab: Design, rationale, methods, lessons learned and adaptation

Asthma remains one of the most important challenges to pediatric public health in the US. A large majority of children with persistent and chronic asthma demonstrate aeroallergen sensitization, which remains a pivotal risk factor associated with the development of persistent, progressive asthma throughout life. In individuals with a tendency toward Type 2 inflammation, sensitization and exposure to high concentrations of offending allergens is associated with increased risk for development of, and impairment from, asthma. The cascade of biological responses to allergens is primarily mediated through IgE antibodies and their production is further stimulated by IgE responses to antigen exposure. In addition, circulating IgE impairs innate anti-viral immune responses. The latter effect could magnify the effects of another early life exposure associated with increased risk of the development of asthma - viral infections. Omalizumab binds to circulating IgE and thus ablates antigen signaling through IgE-related mechanisms. Further, it has been shown restore IFN-α response to rhinovirus and to reduce asthma exacerbations during the viral season. We therefore hypothesized that early blockade of IgE and IgE mediated responses with omalizumab would prevent the development and reduce the severity of asthma in those at high risk for developing asthma. Herein, we describe a double-blind, placebo-controlled trial of omalizumab in 2-3 year old children at high risk for development of asthma to prevent the development and reduce the severity of asthma. We describe the rationale, methods, and lessons learned in implementing this potentially transformative trial aimed at prevention of asthma.

Predictive approaches to heterogeneous treatment effects: a scoping review

BACKGROUND

Recent evidence suggests that there is often substantial variation in the benefits and harms across a trial population. We aimed to identify regression modeling approaches that assess heterogeneity of treatment effect within a randomized clinical trial.

METHODS

We performed a literature review using a broad search strategy, complemented by suggestions of a technical expert panel.

RESULTS

The approaches are classified into 3 categories: 1) Risk-based methods (11 papers) use only prognostic factors to define patient subgroups, relying on the mathematical dependency of the absolute risk difference on baseline risk; 2) Treatment effect modeling methods (9 papers) use both prognostic factors and treatment effect modifiers to explore characteristics that interact with the effects of therapy on a relative scale. These methods couple data-driven subgroup identification with approaches to prevent overfitting, such as penalization or use of separate data sets for subgroup identification and effect estimation. 3) Optimal treatment regime methods (12 papers) focus primarily on treatment effect modifiers to classify the trial population into those who benefit from treatment and those who do not. Finally, we also identified papers which describe model evaluation methods (4 papers).

CONCLUSIONS

Three classes of approaches were identified to assess heterogeneity of treatment effect. Methodological research, including both simulations and empirical evaluations, is required to compare the available methods in different settings and to derive well-informed guidance for their application in RCT analysis.

Why clinical trials may not help patients make treatment decisions: results from focus group discussions with 22 patients

Aim: Despite broad interest in advancing personalized medicine, most evidence is currently derived from average results of clinical trials that may obscure heterogeneity of trial participants. Little is known currently about how patients view heterogeneity in trials and whether they can participate in methodological discussions about this concept. Materials & methods: In structured discussions with three focus groups involving 22 participants, we assessed how representatives of patient communities have used research to guide individual treatment decisions. Discussion themes were organized into a framework describing patient decision-making in four steps: decisions patients make in the course of care; information used to make decisions; sources for information; and quality of information. Results/conclusion: Patients prioritize information that reflects their own characteristics, preferences and values. They struggle applying clinical research to their own case.

The Predictive Approaches to Treatment effect Heterogeneity (PATH) Statement

Heterogeneity of treatment effect (HTE) refers to the nonrandom variation in the magnitude or direction of a treatment effect across levels of a covariate, as measured on a selected scale, against a clinical outcome. In randomized controlled trials (RCTs), HTE is typically examined through a subgroup analysis that contrasts effects in groups of patients defined "1 variable at a time" (for example, male vs. female or old vs. young). The authors of this statement present guidance on an alternative approach to HTE analysis, "predictive HTE analysis." The goal of predictive HTE analysis is to provide patient-centered estimates of outcome risks with versus without the intervention, taking into account all relevant patient attributes simultaneously. The PATH (Predictive Approaches to Treatment effect Heterogeneity) Statement was developed using a multidisciplinary technical expert panel, targeted literature reviews, simulations to characterize potential problems with predictive approaches, and a deliberative process engaging the expert panel. The authors distinguish 2 categories of predictive HTE approaches: a "risk-modeling" approach, wherein a multivariable model predicts the risk for an outcome and is applied to disaggregate patients within RCTs to define risk-based variation in benefit, and an "effect-modeling" approach, wherein a model is developed on RCT data by incorporating a term for treatment assignment and interactions between treatment and baseline covariates. Both approaches can be used to predict differential absolute treatment effects, the most relevant scale for clinical decision making. The authors developed 4 sets of guidance: criteria to determine when risk-modeling approaches are likely to identify clinically important HTE, methodological aspects of risk-modeling methods, considerations for translation to clinical practice, and considerations and caveats in the use of effect-modeling approaches. The PATH Statement, together with its explanation and elaboration document, may guide future analyses and reporting of RCTs.

The Predictive Approaches to Treatment effect Heterogeneity (PATH) Statement: Explanation and Elaboration

The PATH (Predictive Approaches to Treatment effect Heterogeneity) Statement was developed to promote the conduct of, and provide guidance for, predictive analyses of heterogeneity of treatment effects (HTE) in clinical trials. The goal of predictive HTE analysis is to provide patient-centered estimates of outcome risk with versus without the intervention, taking into account all relevant patient attributes simultaneously, to support more personalized clinical decision making than can be made on the basis of only an overall average treatment effect. The authors distinguished 2 categories of predictive HTE approaches (a "risk-modeling" and an "effect-modeling" approach) and developed 4 sets of guidance statements: criteria to determine when risk-modeling approaches are likely to identify clinically meaningful HTE, methodological aspects of risk-modeling methods, considerations for translation to clinical practice, and considerations and caveats in the use of effect-modeling approaches. They discuss limitations of these methods and enumerate research priorities for advancing methods designed to generate more personalized evidence. This explanation and elaboration document describes the intent and rationale of each recommendation and discusses related analytic considerations, caveats, and reservations.

Machine learning methods for developing precision treatment rules with observational data

Clinical trials have identified a variety of predictor variables for use in precision treatment protocols, ranging from clinical biomarkers and symptom profiles to self-report measures of various sorts. Although such variables are informative collectively, none has proven sufficiently powerful to guide optimal treatment selection individually. This has prompted growing interest in the development of composite precision treatment rules (PTRs) that are constructed by combining information across a range of predictors. But this work has been hampered by the generally small samples in randomized clinical trials and the use of suboptimal analysis methods to analyze the resulting data. In this paper, we propose to address the sample size problem by: working with large observational electronic medical record databases rather than controlled clinical trials to develop preliminary PTRs; validating these preliminary PTRs in subsequent pragmatic trials; and using ensemble machine learning methods rather than individual algorithms to carry out statistical analyses to develop the PTRs. The major challenges in this proposed approach are that treatment are not randomly assigned in observational databases and that these databases often lack measures of key prescriptive predictors and mental disorder treatment outcomes. We proposed a tiered case-cohort design approach that uses innovative methods for measuring and balancing baseline covariates and estimating PTRs to address these challenges.

Subgroup identification in clinical trials via the predicted individual treatment effect

Identifying subgroups of treatment responders through the different phases of clinical trials has the potential to increase success in drug development. Recent developments in subgroup analysis consider subgroups that are defined in terms of the predicted individual treatment effect, i.e. the difference between the predicted outcome under treatment and the predicted outcome under control for each individual, which in turn may depend on multiple biomarkers. In this work, we study the properties of different modelling strategies to estimate the predicted individual treatment effect. We explore linear models and compare different estimation methods, such as maximum likelihood and the Lasso with and without randomized response. For the latter, we implement confidence intervals based on the selective inference framework to account for the model selection stage. We illustrate the methods in a dataset of a treatment for Alzheimer disease (normal response) and in a dataset of a treatment for prostate cancer (survival outcome). We also evaluate via simulations the performance of using the predicted individual treatment effect to identify subgroups where a novel treatment leads to better outcomes compared to a control treatment.

Development and evaluating multimarker models for guiding treatment decisions

Background

Despite the growing interest in developing markers for predicting treatment response and optimizing treatment decisions, an appropriate methodology to identify, combine and evaluate such markers has been slow to develop. We propose a step-by-step strategy for analysing data from existing randomised trials with the aim of identifying a multi-marker model for guiding decisions about treatment.

Methods

We start with formulating the treatment selection problem, continue with defining the treatment threshold, prepare a list of candidate markers, develop the model, apply the model to estimate individual treatment effects, and evaluate model performance in the study group of patients who meet the trial eligibility criteria. In this process, we rely on some well-known techniques for multivariable prediction modelling, but focus on predicting benefit from treatment, rather than outcome itself. We present our approach using data from a randomised trial in which 808 women with multiple pregnancy were assigned to cervical pessary or control, to prevent adverse perinatal outcomes. Overall, cervical pessary did not reduce the risk of adverse perinatal outcomes.

Results

The treatment threshold was zero. We had a preselected list of 5 potential markers and developed a logistic model including the markers, treatment and all marker-by-treatment interaction terms. The model was well calibrated and identified 35% (95% confidence interval (CI) 32 to 39%) of the trial participants as benefitting from pessary insertion. We estimated that the risk of adverse outcome could be reduced from 13.5 to 8.1% (5.4% risk reduction; 95% CI 2.1 to 8.6%) through model-based selective pessary insertion. The next step is external validation upon existence of independent trial data.

Conclusions

We suggest revisiting existing trials data to explore whether differences in treatment benefit can be explained by differences in baseline characteristics of patients. This could lead to treatment selection tools which, after validation in comparable existing trials, can be introduced into clinical practice for guiding treatment decisions in future patients.

Evaluation of biomarkers for treatment selection using individual participant data from multiple clinical trials

Biomarkers that predict treatment effects may be used to guide treatment decisions, thus improving patient outcomes. A meta-analysis of individual participant data (IPD) is potentially more powerful than a single-study data analysis in evaluating markers for treatment selection. Our study was motivated by the IPD that were collected from 2 randomized controlled trials of hypertension and preeclampsia among pregnant women to evaluate the effect of labor induction over expectant management of the pregnancy in preventing progression to severe maternal disease. The existing literature on statistical methods for biomarker evaluation in IPD meta-analysis have evaluated a marker's performance in terms of its ability to predict risk of disease outcome, which do not directly apply to the treatment selection problem. In this study, we propose a statistical framework for evaluating a marker for treatment selection given IPD from a small number of individual clinical trials. We derive marker-based treatment rules by minimizing the average expected outcome across studies. The application of the proposed methods to the IPD from 2 studies in women with hypertension in pregnancy is presented.

Bayesian analysis of heterogeneous treatment effects for patient-centered outcomes research

Evaluation of heterogeneity of treatment effect (HTE) is an essential aspect of personalized medicine and patient-centered outcomes research. Our goal in this article is to promote the use of Bayesian methods for subgroup analysis and to lower the barriers to their implementation by describing the ways in which the companion software beanz can facilitate these types of analyses. To advance this goal, we describe several key Bayesian models for investigating HTE and outline the ways in which they are well-suited to address many of the commonly cited challenges in the study of HTE. Topics highlighted include shrinkage estimation, model choice, sensitivity analysis, and posterior predictive checking. A case study is presented in which we demonstrate the use of the methods discussed.

Subgroup analyses in confirmatory clinical trials: time to be specific about their purposes

Background

It is well recognized that treatment effects may not be homogeneous across the study population. Subgroup analyses constitute a fundamental step in the assessment of evidence from confirmatory (Phase III) clinical trials, where conclusions for the overall study population might not hold. Subgroup analyses can have different and distinct purposes, requiring specific design and analysis solutions. It is relevant to evaluate methodological developments in subgroup analyses against these purposes to guide health care professionals and regulators as well as to identify gaps in current methodology.

Methods

We defined four purposes for subgroup analyses: (1) Investigate the consistency of treatment effects across subgroups of clinical importance, (2) Explore the treatment effect across different subgroups within an overall non-significant trial, (3) Evaluate safety profiles limited to one or a few subgroup(s), (4) Establish efficacy in the targeted subgroup when included in a confirmatory testing strategy of a single trial. We reviewed the methodology in line with this “purpose-based” framework. The review covered papers published between January 2005 and April 2015 and aimed to classify them in none, one or more of the aforementioned purposes.

Results

In total 1857 potentially eligible papers were identified. Forty-eight papers were selected and 20 additional relevant papers were identified from their references, leading to 68 papers in total. Nineteen were dedicated to purpose 1, 16 to purpose 4, one to purpose 2 and none to purpose 3. Seven papers were dedicated to more than one purpose, the 25 remaining could not be classified unambiguously. Purposes of the methods were often not specifically indicated, methods for subgroup analysis for safety purposes were almost absent and a multitude of diverse methods were developed for purpose (1).

Conclusions

It is important that researchers developing methodology for subgroup analysis explicitly clarify the objectives of their methods in terms that can be understood from a patient’s, health care provider’s and/or regulator’s perspective. A clear operational definition for consistency of treatment effects across subgroups is lacking, but is needed to improve the usability of subgroup analyses in this setting. Finally, methods to particularly explore benefit-risk systematically across subgroups need more research.

Electronic supplementary material

The online version of this article (doi:10.1186/s12874-016-0122-6) contains supplementary material, which is available to authorized users.

Methods for identification and confirmation of targeted subgroups in clinical trials: A systematic review

Important objectives in the development of stratified medicines include the identification and confirmation of subgroups of patients with a beneficial treatment effect and a positive benefit-risk balance. We report the results of a literature review on methodological approaches to the design and analysis of clinical trials investigating a potential heterogeneity of treatment effects across subgroups. The identified approaches are classified based on certain characteristics of the proposed trial designs and analysis methods. We distinguish between exploratory and confirmatory subgroup analysis, frequentist, Bayesian and decision-theoretic approaches and, last, fixed-sample, group-sequential, and adaptive designs and illustrate the available trial designs and analysis strategies with published case studies.

Using internally developed risk models to assess heterogeneity in treatment effects in clinical trials

BACKGROUND

Recent proposals suggest that risk-stratified analyses of clinical trials be routinely performed to better enable tailoring of treatment decisions to individuals. Trial data can be stratified using externally developed risk models (eg, Framingham risk score), but such models are not always available. We sought to determine whether internally developed risk models, developed directly on trial data, introduce bias compared with external models.

METHODS AND RESULTS

We simulated a large patient population with known risk factors and outcomes. Clinical trials were then simulated by repeatedly drawing from the patient population assuming a specified relative treatment effect in the experimental arm, which either did or did not vary according to a subject's baseline risk. For each simulated trial, 2 internal risk models were developed on either the control population only (internal controls only) or the whole trial population blinded to treatment (internal whole trial). Bias was estimated for the internal models by comparing treatment effect predictions to predictions from the external model. Under all treatment assumptions, internal models introduced only modest bias compared with external models. The magnitude of these biases was slightly smaller for internal whole trial models than for internal controls only models. Internal whole trial models were also slightly less sensitive to bias introduced by overfitting and less sensitive to falsely identifying the existence of variability in treatment effect across the risk spectrum compared with internal controls only models.

CONCLUSIONS

Appropriately developed internal models produce relatively unbiased estimates of treatment effect across the spectrum of risk. When estimating treatment effect, internally developed risk models using both treatment arms should, in general, be preferred to models developed on the control population.