Mastering variation: variance components and personalised medicine

Does evidence support the high expectations placed in precision medicine? A bibliographic review

Background: Precision medicine is the Holy Grail of interventions that are tailored to a patient's individual characteristics. However, conventional clinical trials are designed to find differences in averages, and interpreting these differences depends on untestable assumptions. Although only an ideal, a constant effect of treatment would facilitate individual management. A direct consequence of a constant effect is that the variance of the outcome measure would be the same in the treated and control arms. We reviewed the literature to explore the similarity of these variances as a foundation for examining whether and how often precision medicine is definitively required. Methods: We reviewed parallel clinical trials with numerical primary endpoints published in 2004, 2007, 2010 and 2013. We collected the baseline and final standard deviations of the main outcome measure. We assessed homoscedasticity by comparing the variance of the primary endpoint between arms through the outcome variance ratio (treated to control group). Results: The review provided 208 articles with enough information to conduct the analysis. One out of five studies (n = 40, 19.2%) had statistically different variances between groups, implying a non-constant-effect. The adjusted point estimate of the mean outcome variance ratio (treated to control group) is 0.89 (95% CI 0.81 to 0.97). Conclusions: The mean variance ratio is significantly lower than 1 and the lower variance was found more often in the intervention group than in the control group, suggesting it is more usual for treated patients to be stable. This observed reduction in variance might also imply that there could be a subgroup of less ill patients who derive no benefit from treatment. This would require further study as to whether the treatment effect outweighs the side effects as well as the economic costs. We have shown that there are ways to analyze the apparently unobservable constant effect.

Does evidence support the high expectations placed in precision medicine? A bibliographic review

Background: Precision medicine is the Holy Grail of interventions that are tailored to a patient’s individual characteristics. However, conventional clinical trials are designed to find differences in averages, and interpreting these differences depends on untestable assumptions. Although only an ideal, a constant effect of treatment would facilitate individual management. A direct consequence of a constant effect is that the variance of the outcome measure would be the same in the treated and control arms. We reviewed the literature to explore the similarity of these variances as a foundation for examining whether and how often precision medicine is definitively required.

Methods: We reviewed parallel clinical trials with numerical primary endpoints published in 2004, 2007, 2010 and 2013. We collected the baseline and final standard deviations of the main outcome measure. We assessed homoscedasticity by comparing the variance of the primary endpoint between arms through the outcome variance ratio (treated to control group).

Results: The review provided 208 articles with enough information to conduct the analysis. One out of five studies (n = 40, 19.2%) had statistically different variances between groups, implying a non-constant-effect. The adjusted point estimate of the mean outcome variance ratio (treated to control group) is 0.89 (95% CI 0.81 to 0.97).

Conclusions: The mean variance ratio is significantly lower than 1 and the lower variance was found more often in the intervention group than in the control group, suggesting it is more usual for treated patients to be stable. This observed reduction in variance might also imply that there could be a subgroup of less ill patients who derive no benefit from treatment. This would require further study as to whether the treatment effect outweighs the side effects as well as the economic costs. We have shown that there are ways to analyze the apparently unobservable constant effect.

Does evidence support the high expectations placed in precision medicine? A bibliographic review

Background: Precision medicine is the Holy Grail of interventions that are tailored to a patient's individual characteristics. However, conventional clinical trials are designed to find differences in averages, and interpreting these differences depends on untestable assumptions. Although only an ideal, a constant effect of treatment would facilitate individual management. A direct consequence of a constant effect is that the variance of the outcome measure would be the same in the treated and control arms. We reviewed the literature to explore the similarity of these variances as a foundation for examining whether and how often precision medicine is definitively required. Methods: We reviewed parallel clinical trials with numerical primary endpoints published in 2004, 2007, 2010 and 2013. We collected the baseline and final standard deviations of the main outcome measure. We assessed homoscedasticity by comparing the variance of the primary endpoint between arms through the outcome variance ratio (treated to control group). Results: The review provided 208 articles with enough information to conduct the analysis. One out of five studies (n = 40, 19.2%) had statistically different variances between groups, implying a non-constant-effect. The adjusted point estimate of the mean outcome variance ratio (treated to control group) is 0.89 (95% CI 0.81 to 0.97). Conclusions: The mean variance ratio is significantly lower than 1 and the lower variance was found more often in the intervention group than in the control group, suggesting it is more usual for treated patients to be stable. This observed reduction in variance might also imply that there could be a subgroup of less ill patients who derive no benefit from treatment. This would require further study as to whether the treatment effect outweighs the side effects as well as the economic costs. We have shown that there are ways to analyze the apparently unobservable constant effect.

How much variation in oocyte yield after controlled ovarian stimulation can be explained? A multilevel modelling study

STUDY QUESTION

How much variation in oocyte yield after controlled ovarian stimulation (COS) can be accounted for by known patient and treatment characteristics?

SUMMARY ANSWER

There is substantial variation in the COS responses of similar women and in repeated COS episodes undertaken by the same woman, which cannot be accounted for at present.

WHAT IS ALREADY KNOWN

The goal of individualized COS is to safely collect enough oocytes to maximize the chance of success in an ART cycle. Personalization of treatment rests on the ability to reduce variation in response through modifiable factors.

STUDY DESIGN, SIZE, DURATION

Multilevel modelling of a routine ART database covering the period 1 October 2008–8 August 2012 was employed to estimate the amount of variation in COS response and the extent to which this could be explained by immutable patient characteristics and by manipulable treatment variables. A total of 1851 treatment cycles undertaken by 1430 patients were included. The study was not subject to attrition, as cancelled cycles were included in the analysis.

PARTICIPANTS/MATERIALS, SETTING, METHODS

Women aged 21–43 years undergoing ovarian stimulation for IVF (possibly with ICSI) using their own eggs at a tertiary care centre.

MAIN RESULTS AND THE ROLE OF CHANCE

Substantial unexplained variation in COS response (oocyte yield): was observed (3.4-fold (95% CI: 3.12 to 3.61)). Only a relatively small amount of this variation (around 19%) can be explained by modifiable factors. A significant, previously undescribed predictor of response was the practitioner performing oocyte retrieval, with 1.5-fold variation between surgeons with the highest and lowest yields.

LIMITATIONS REASONS FOR CAUTION

Although a large number of covariables were adjusted for in the analysis, including those that were used for dosing and determination of the stimulation regimen, this study is subject to confounding due to unmeasured variables and measurement error.

WIDER IMPLICATIONS OF THE FINDINGS

The present study suggests that there are limits to the extent that COS response can be predicted on the basis of known factors, or controlled by manipulation of treatment factors. Moreover, modifiable variation in response appears to be partially attributable to differences between surgeons performing oocyte retrieval. Consequently, consistent prevention of ineffective or unsafe responses to COS is not likely to be possible at present. Our results highlight the importance of blinding surgeons in RCTs. The data also suggest that there is likely to be limited scope for personalized treatment unless additional predictors of ovarian response can be identified.

STUDY FUNDING/COMPETING INTERESTS

J.W. is funded by a Doctoral Research Fellowship from the National Institute for Health Research (DRF-2014-07-050) supervised by S.A.R. The views expressed in this publication are those of the authors and not necessarily those of the NHS, the National Institute for Health Research or the Department of Health. J.W. is a statistical editor of the Cochrane Gynaecology and Fertility Group. S.A.R. is a statistical editor for Human Reproduction. J.W. also declares that publishing peer-reviewed articles benefits his career. A.L.M. has received consultation fees from MSD, Merck Serono, Ferring, TEVA, Roche, Beckman Coulter.

Estimating individual benefits of medical or behavioral treatments in severely ill patients

There is a need for statistical methods appropriate for the analysis of clinical trials from a personalized-medicine viewpoint as opposed to the common statistical practice that simply examines average treatment effects. This article proposes an approach to quantifying, reporting and analyzing individual benefits of medical or behavioral treatments to severely ill patients with chronic conditions, using data from clinical trials. The approach is a new development of a published framework for measuring the severity of a chronic disease and the benefits treatments provide to individuals, which utilizes regression models with random coefficients. Here, a patient is considered to be severely ill if the patient’s basal severity is close to one. This allows the derivation of a very flexible family of probability distributions of individual benefits that depend on treatment duration and the covariates included in the regression model. Our approach may enrich the statistical analysis of clinical trials of severely ill patients because it allows investigating the probability distribution of individual benefits in the patient population and the variables that influence it, and we can also measure the benefits achieved in specific patients including new patients. We illustrate our approach using data from a clinical trial of the anti-depressant imipramine.

Sample size considerations for n-of-1 trials

N-of-1 trials are trials in which patients are treated with two or more treatments on multiple occasions. They can have many different purposes and can be analysed in different frameworks. In this note, five different criteria for planning sample sizes for n-of-1 trials are identified, and formulae and advice to address the associated tasks are provided. The basic design addressed is that of randomisation to treatment and control within cycles of pairs of episodes and the model assumed is that of a Normal-Normal mixture with variance components corresponding to within-cycle within-patient variation and treatment-by-patient interaction. The code to accomplish the tasks has been written in GenStat®, SAS® and R® and the application of the approaches is illustrated.

Aggregated N-of-1 trials for unlicensed medicines for small populations: an assessment of a trial with ephedrine for myasthenia gravis

Background

Inexpensive medicines with a long history of use may currently be prescribed off-label for rare indications. Reimbursement is at the discretion of health insurance companies, and may be unpredictable. The example addressed was ephedrine as add-on treatment for myasthenia gravis. Stakeholders from academia, a patient organization, the Dutch National Health Care Institute (NHCI) and Dutch Medicines Evaluation Board (MEB) advised on the trial design. The NHCI and MEB agreed to provide scientific advice on the suitability of the evidence generated by the trial, for regulatory decisions. This paper describes the feasibility of the trial and the utility of its aggregated results.

Results

The trialists experienced the trial as feasible. Retrospective interviews showed that the trial as performed was acceptable to patients. The treatment effect in the primary outcome measure, muscle strength, was statistically significant when inferred to the population level, though the effect size was modest. Secondary outcomes were statistically significant in a preplanned, fixed effects analysis within the four patients. The NHCI advised that it could potentially make reimbursement decisions based on the Fitting Evidence framework, should the trialists decide to apply for reimbursement. The MEB advised that for a licensing decision, the N-of-1 design is a last-resort option for demonstrating treatment benefit in a rare disease. N-of-1 trials alone do not provide enough evidence on potential risk. The MEB found the current trial inconclusive. It suggested doing a 2-armed trial of longer duration, possibly with a different outcome measure (postponement of corticosteroid use). It suggested engaging a consultancy or commercial sponsor, should the trialists decide to seek market authorization of the drug.

Conclusions

In theory, evidence from aggregated N-of-1 trials is suitable for use in licensing and reimbursement decisions. The current example illustrates differences in interpretation of N-of-1 results by health authorities. In the era of personalized medicine, consensus is required on the interpretation of data from study designs geared to small groups. Demonstrating effectiveness of inexpensive medicines in small populations may require involvement of non-commercial parties, to preserve affordability.

Electronic supplementary material

The online version of this article (doi:10.1186/s13023-017-0636-y) contains supplementary material, which is available to authorized users.

Understanding Variation in Sets of N-of-1 Trials

A recent paper in this journal by Chen and Chen has used computer simulations to examine a number of approaches to analysing sets of n-of-1 trials. We have examined such designs using a more theoretical approach based on considering the purpose of analysis and the structure as regards randomisation that the design uses. We show that different purposes require different analyses and that these in turn may produce quite different results. Our approach to incorporating the randomisation employed when the purpose is to test a null hypothesis of strict equality of the treatment makes use of Nelder’s theory of general balance. However, where the purpose is to make inferences about the effects for individual patients, we show that a mixed model is needed. There are strong parallels to the difference between fixed and random effects meta-analyses and these are discussed.

Measuring the individual benefit of a medical or behavioral treatment using generalized linear mixed-effects models

We propose statistical definitions of the individual benefit of a medical or behavioral treatment and of the severity of a chronic illness. These definitions are used to develop a graphical method that can be used by statisticians and clinicians in the data analysis of clinical trials from the perspective of personalized medicine. The method focuses on assessing and comparing individual effects of treatments rather than average effects and can be used with continuous and discrete responses, including dichotomous and count responses. The method is based on new developments in generalized linear mixed-effects models, which are introduced in this article. To illustrate, analyses of data from the Sequenced Treatment Alternatives to Relieve Depression clinical trial of sequences of treatments for depression and data from a clinical trial of respiratory treatments are presented. The estimation of individual benefits is also explained. Copyright © 2016 John Wiley & Sons, Ltd.

Mastering variation: variance components and personalised medicine

Various sources of variation in observed response in clinical trials and clinical practice are considered, and ways in which the corresponding components of variation might be estimated are discussed. Although the issues have been generally well‐covered in the statistical literature, they seem to be poorly understood in the medical literature and even the statistical literature occasionally shows some confusion. To increase understanding and communication, some simple graphical approaches to illustrating issues are proposed. It is also suggested that reducing variation in medical practice might make as big a contribution to improving health outcome as personalising its delivery according to the patient. It is concluded that the common belief that there is a strong personal element in response to treatment is not based on sound statistical evidence. © 2015 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.