Maximum type I error rate inflation from sample size reassessment when investigators are blind to treatment labels

Consider a parallel group trial for the comparison of an experimental treatment to a control, where the second‐stage sample size may depend on the blinded primary endpoint data as well as on additional blinded data from a secondary endpoint. For the setting of normally distributed endpoints, we demonstrate that this may lead to an inflation of the type I error rate if the null hypothesis holds for the primary but not the secondary endpoint.

We derive upper bounds for the inflation of the type I error rate, both for trials that employ random allocation and for those that use block randomization. We illustrate the worst‐case sample size reassessment rule in a case study. For both randomization strategies, the maximum type I error rate increases with the effect size in the secondary endpoint and the correlation between endpoints. The maximum inflation increases with smaller block sizes if information on the block size is used in the reassessment rule. Based on our findings, we do not question the well‐established use of blinded sample size reassessment methods with nuisance parameter estimates computed from the blinded interim data of the primary endpoint. However, we demonstrate that the type I error rate control of these methods relies on the application of specific, binding, pre‐planned and fully algorithmic sample size reassessment rules and does not extend to general or unplanned sample size adjustments based on blinded data. © 2015 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.

[Impact of Self-Reported Fatigue on ACL Injuries in Alpine Skiing: A Sex Comparison]

INTRODUCTION

In recreational alpine skiing, about one third of all injuries affect the knee joint, and the most common diagnosis in adult male and female skiers is a tear of the anterior cruciate ligament (ACL), which makes up 15 - 21 % of all injuries. General preventive recommendations to reduce the incidence of ski injuries include avoiding fatigue. However, it seems unclear to what extent ACL injuries in male and female recreational skiers are related to perceived fatigue.

METHODS

This study was conducted as a prospective questionnaire-based investigation in two Austrian ski injury clinics during the five winter seasons between 2009/2010 and 2013/2014. In total, 588 skiers (67.9 % females) with a mean age of 42.1 ± 10.9 years were interviewed about demographics, skiing ability, skiing behaviour, fitness, day and time of accident, skiing duration and perceived fatigue at the moment of accident.

RESULTS

ACL injured males reported a significantly higher skiing ability and fitness level as well as a more risky behaviour on ski slopes compared to females. About one third of males and females injured their ACL within the first day of the ski trip and about 57 % within the first two days, with no sex differences. However, a significantly higher number of female skiers sustained an ACL injury during the first hour of skiing (28 vs. 17 %) as well as during the first two hours of skiing compared to males (52 vs. 44 %). About 81 % of males and females felt no fatigue or just a trace of fatigue in their legs at the time of accident, with no sex differences.

CONCLUSION

Based on the findings of this study, fatigue does not seem to be a major risk factor for an ACL injury among male and female recreational skiers.

Sample size reassessment for a two-stage design controlling the false discovery rate

Sample size calculations for gene expression microarray and NGS-RNA-Seq experiments are challenging because the overall power depends on unknown quantities as the proportion of true null hypotheses and the distribution of the effect sizes under the alternative. We propose a two-stage design with an adaptive interim analysis where these quantities are estimated from the interim data. The second stage sample size is chosen based on these estimates to achieve a specific overall power. The proposed procedure controls the power in all considered scenarios except for very low first stage sample sizes. The false discovery rate (FDR) is controlled despite of the data dependent choice of sample size. The two-stage design can be a useful tool to determine the sample size of high-dimensional studies if in the planning phase there is high uncertainty regarding the expected effect sizes and variability.

Bridging the gap: a review of dose investigations in paediatric investigation plans

Aims

In the EU, development of new medicines for children should follow a prospectively agreed paediatric investigation plan (PIP). Finding the right dose for children is crucial but challenging due to the variability of pharmacokinetics across age groups and the limited sample sizes available. We examined strategies adopted in PIPs to support paediatric dosing recommendations to identify common assumptions underlying dose investigations and the attempts planned to verify them in children.

Methods

We extracted data from 73 PIP opinions recently adopted by the Paediatric Committee of the European Medicines Agency. These opinions represented 79 medicinal development programmes and comprised a total of 97 dose investigation studies. We identified the design of these dose investigation studies, recorded the analyses planned and determined the criteria used to define target doses.

Results

Most dose investigation studies are clinical trials (83 of 97) that evaluate a single dosing rule. Sample sizes used to investigate dose are highly variable across programmes, with smaller numbers used in younger children (< 2 years). Many studies (40 of 97) do not pre-specify a target dose criterion. Of those that do, most (33 of 57 studies) guide decisions using pharmacokinetic data alone.

Conclusions

Common assumptions underlying dose investigation strategies include dose proportionality and similar exposure−response relationships in adults and children. Few development programmes pre-specify steps to verify assumptions in children. There is scope for the use of Bayesian methods as a framework for synthesizing existing information to quantify prior uncertainty about assumptions. This process can inform the design of optimal drug development strategies.

Decision-theoretic designs for small trials and pilot studies: A review

Pilot studies and other small clinical trials are often conducted but serve a variety of purposes and there is little consensus on their design. One paradigm that has been suggested for the design of such studies is Bayesian decision theory. In this article, we review the literature with the aim of summarizing current methodological developments in this area. We find that decision-theoretic methods have been applied to the design of small clinical trials in a number of areas. We divide our discussion of published methods into those for trials conducted in a single stage, those for multi-stage trials in which decisions are made through the course of the trial at a number of interim analyses, and those that attempt to design a series of clinical trials or a drug development programme. In all three cases, a number of methods have been proposed, depending on the decision maker’s perspective being considered and the details of utility functions that are used to construct the optimal design.

Prominin-1 (CD133, AC133) and dipeptidyl-peptidase IV (CD26) are indicators of infinitive growth in colon cancer cells

Advanced colorectal cancer is characterized by uncontrolled growth and resistance against anti-cancer agents, including ErbB inhibitors. Recent data suggest that cancer stem cells (CSC) are particularly resistant. These cells may reside within a CD133+ fraction of the malignant cells. Using HCT116 cells we explored the role of CD133 and other CSC markers in drug resistance in colon cancer cells. CD133+ cells outnumbered CD133- cells over time in long-term culture. Both populations displayed the KRAS mutation 38G > A and an almost identical target profile, including EGFR/ErbB1, ErbB2, and ErbB4. Microarray analyses and flow cytometry identified CD26 as additional CSC marker co-expressed on CD133+ cells. However, knock-down of CD133 or CD26 did not affect short-term growth of HCT116 cells, and both cell-populations were equally resistant to various targeted drugs except irreversible ErbB inhibitors, which blocked growth and ERK1/2 phosphorylation in CD133- cells more efficiently than in CD133+ cells. Moreover, the MEK inhibitor AS703026 was found to overcome resistance against ErbB blockers in CD133+ cells. Together, CD133 and CD26 are markers of long-term growth and resistance to ErbB blockers in HCT116 cells, which may be mediated by constitutive ERK activity.

Adaptive designs for subpopulation analysis optimizing utility functions

If the response to treatment depends on genetic biomarkers, it is important to identify predictive biomarkers that define (sub-)populations where the treatment has a positive benefit risk balance. One approach to determine relevant subpopulations are subgroup analyses where the treatment effect is estimated in biomarker positive and biomarker negative groups. Subgroup analyses are challenging because several types of risks are associated with inference on subgroups. On the one hand, by disregarding a relevant subpopulation a treatment option may be missed due to a dilution of the treatment effect in the full population. Furthermore, even if the diluted treatment effect can be demonstrated in an overall population, it is not ethical to treat patients that do not benefit from the treatment when they can be identified in advance. On the other hand, selecting a spurious subpopulation increases the risk to restrict an efficacious treatment to a too narrow fraction of a potential benefiting population. We propose to quantify these risks with utility functions and investigate nonadaptive study designs that allow for inference on subgroups using multiple testing procedures as well as adaptive designs, where subgroups may be selected in an interim analysis. The characteristics of such adaptive and nonadaptive designs are compared for a range of scenarios.

Adaptive graph-based multiple testing procedures

Multiple testing procedures defined by directed, weighted graphs have recently been proposed as an intuitive visual tool for constructing multiple testing strategies that reflect the often complex contextual relations between hypotheses in clinical trials. Many well-known sequentially rejective tests, such as (parallel) gatekeeping tests or hierarchical testing procedures are special cases of the graph based tests. We generalize these graph-based multiple testing procedures to adaptive trial designs with an interim analysis. These designs permit mid-trial design modifications based on unblinded interim data as well as external information, while providing strong family wise error rate control. To maintain the familywise error rate, it is not required to prespecify the adaption rule in detail. Because the adaptive test does not require knowledge of the multivariate distribution of test statistics, it is applicable in a wide range of scenarios including trials with multiple treatment comparisons, endpoints or subgroups, or combinations thereof. Examples of adaptations are dropping of treatment arms, selection of subpopulations, and sample size reassessment. If, in the interim analysis, it is decided to continue the trial as planned, the adaptive test reduces to the originally planned multiple testing procedure. Only if adaptations are actually implemented, an adjusted test needs to be applied. The procedure is illustrated with a case study and its operating characteristics are investigated by simulations.

Adaptive clinical trial designs for European marketing authorization: a survey of scientific advice letters from the European Medicines Agency

Background

Since the first methodological publications on adaptive study design approaches in the 1990s, the application of these approaches in drug development has raised increasing interest among academia, industry and regulators. The European Medicines Agency (EMA) as well as the Food and Drug Administration (FDA) have published guidance documents addressing the potentials and limitations of adaptive designs in the regulatory context. Since there is limited experience in the implementation and interpretation of adaptive clinical trials, early interaction with regulators is recommended. The EMA offers such interactions through scientific advice and protocol assistance procedures.

Methods

We performed a text search of scientific advice letters issued between 1 January 2007 and 8 May 2012 that contained relevant key terms. Letters containing questions related to adaptive clinical trials in phases II or III were selected for further analysis. From the selected letters, important characteristics of the proposed design and its context in the drug development program, as well as the responses of the Committee for Human Medicinal Products (CHMP)/Scientific Advice Working Party (SAWP), were extracted and categorized. For 41 more recent procedures (1 January 2009 to 8 May 2012), additional details of the trial design and the CHMP/SAWP responses were assessed. In addition, case studies are presented as examples.

Results

Over a range of 5½ years, 59 scientific advices were identified that address adaptive study designs in phase II and phase III clinical trials. Almost all were proposed as confirmatory phase III or phase II/III studies. The most frequently proposed adaptation was sample size reassessment, followed by dropping of treatment arms and population enrichment. While 12 (20%) of the 59 proposals for an adaptive clinical trial were not accepted, the great majority of proposals were accepted (15, 25%) or conditionally accepted (32, 54%). In the more recent 41 procedures, the most frequent concerns raised by CHMP/SAWP were insufficient justifications of the adaptation strategy, type I error rate control and bias.

Conclusions

For the majority of proposed adaptive clinical trials, an overall positive opinion was given albeit with critical comments. Type I error rate control, bias and the justification of the design are common issues raised by the CHMP/SAWP.

Connections between permutation and t-tests: relevance to adaptive methods

A permutation test assigns a p-value by conditioning on the data and treating the different possible treatment assignments as random. The fact that the conditional type I error rate given the data is controlled at level α ensures validity of the test even if certain adaptations are made. We show the connection between permutation and t-tests, and use this connection to explain why certain adaptations are valid in a t-test setting as well. We illustrate this with an example of blinded sample size recalculation.

Sharing clinical trial data on patient level: opportunities and challenges

In recent months one of the most controversially discussed topics among regulatory agencies, the pharmaceutical industry, journal editors, and academia has been the sharing of patient-level clinical trial data. Several projects have been started such as the European Medicines Agency´s (EMA) “proactive publication of clinical trial data”, the BMJ open data campaign, or the AllTrials initiative. The executive director of the EMA, Dr. Guido Rasi, has recently announced that clinical trial data on patient level will be published from 2014 onwards (although it has since been delayed). The EMA draft policy on proactive access to clinical trial data was published at the end of June 2013 and open for public consultation until the end of September 2013. These initiatives will change the landscape of drug development and publication of medical research. They provide unprecedented opportunities for research and research synthesis, but pose new challenges for regulatory authorities, sponsors, scientific journals, and the public. Besides these general aspects, data sharing also entails intricate biostatistical questions such as problems of multiplicity. An important issue in this respect is the interpretation of multiple statistical analyses, both prospective and retrospective. Expertise in biostatistics is needed to assess the interpretation of such multiple analyses, for example, in the context of regulatory decision-making by optimizing procedural guidance and sophisticated analysis methods.

Effect of intensive care after cardiac arrest on patient outcome: a database analysis

Introduction

The study aimed to determine the impact of treatment frequency, hospital size, and capability on mortality of patients admitted after cardiac arrest for postresuscitation care to different intensive care units.

Methods

Prospectively recorded data from 242,588 adults consecutively admitted to 87 Austrian intensive care units over a period of 13 years (1998 to 2010) were analyzed retrospectively. Multivariate analysis was used to assess the effect of the frequency of postresuscitation care on mortality, correcting for baseline parameters, severity of illness, hospital size, and capability to perform coronary angiography and intervention.

Results

In total, 5,857 patients had had cardiac arrest and were admitted to an intensive care unit. Observed hospital mortality was 56% in the cardiac-arrest cohort (3,302 nonsurvivors). Patients treated in intensive care units with a high frequency of postresuscitation care generally had high severity of illness (median Simplified Acute Physiology Score (SAPS II), 65). Intensive care units with a higher frequency of care showed improved risk-adjusted mortality. The SAPS II adjusted, observed-to-expected mortality ratios (O/E-Ratios) in the three strata (<18; 18 to 26; >26 resuscitations per ICU per year) were 0.869 (95% confidence interval, 0.844 to 894), 0.876 (0.850 to 0.902), and 0.808 (0.784 to 0.833).

Conclusions

In this database analysis, a high frequency of post-cardiac arrest care at an intensive care unit seemed to be associated with improved outcome of cardiac-arrest patients. We were able to identify patients who seemed to profit more from high frequency of care, namely, those with an intermediate severity of illness. Considering these findings, cardiac-arrest care centers might be a reasonable step to improve outcome in this specific population of cardiac-arrest patients.

Detection of epistatic effects with logic regression and a classical linear regression model

To locate multiple interacting quantitative trait loci (QTL) influencing a trait of interest within experimental populations, usually methods as the Cockerham's model are applied. Within this framework, interactions are understood as the part of the joined effect of several genes which cannot be explained as the sum of their additive effects. However, if a change in the phenotype (as disease) is caused by Boolean combinations of genotypes of several QTLs, this Cockerham's approach is often not capable to identify them properly. To detect such interactions more efficiently, we propose a logic regression framework. Even though with the logic regression approach a larger number of models has to be considered (requiring more stringent multiple testing correction) the efficient representation of higher order logic interactions in logic regression models leads to a significant increase of power to detect such interactions as compared to a Cockerham's approach. The increase in power is demonstrated analytically for a simple two-way interaction model and illustrated in more complex settings with simulation study and real data analysis.

Predicting nitrogen and acidity effects on long-term dynamics of dissolved organic matter

Increases in dissolved organic carbon (DOC) fluxes may relate to changes in sulphur and nitrogen pollution. We integrated existing models of vegetation growth and soil organic matter turnover, acid-base dynamics, and organic matter mobility, to form the 'MADOC' model. After calibrating parameters governing interactions between pH and DOC dissolution using control treatments on two field experiments, MADOC reproduced responses of pH and DOC to additions of acidifying and alkalising solutions. Long-term trends in a range of acid waters were also reproduced. The model suggests that the sustained nature of observed DOC increases can best be explained by a continuously replenishing potentially-dissolved carbon pool, rather than dissolution of a large accumulated store. The simulations informed the development of hypotheses that: DOC increase is related to plant productivity increase as well as to pH change; DOC increases due to nitrogen pollution will become evident, and be sustained, after soil pH has stabilised.

Adaptive Budgets in Clinical Trials

We consider situations where a drug developer gets access to additional financial resources when a promising result has been observed in a pre-planned interim analysis during a clinical trial which should lead to the registration of the drug. First the option that the drug developer completely puts the additional resources into increasing the second stage sample size has been investigated. If investors invest the more the larger the observed interim effect, this may not be a reasonable strategy: Then additional sample sizes are applied when the conditional power is already very large and hardly any impact on the overall power can be expected. Nevertheless, further reducing the type II error rate in promising situations may be of interest for a drug developer. In a second step, sample size was based on a utility function including the reward of registration (which was allowed to depend on the observed effect size at the end of the trial) and sampling costs. Utility as a function of the sample size may have more than one local maximum, one of them at the lowest per group sample size. For small effects an optimal strategy could be to apply the smallest sample size accepted by regulators.

Simultaneous confidence intervals that are compatible with closed testing in adaptive designs

We describe a general method for finding a confidence region for a parameter vector that is compatible with the decisions of a two-stage closed test procedure in an adaptive experiment. The closed test procedure is characterized by the fact that rejection or nonrejection of a null hypothesis may depend on the decisions for other hypotheses and the compatible confidence region will, in general, have a complex, nonrectangular shape. We find the smallest cross-product of simultaneous confidence intervals containing the region and provide computational shortcuts for calculating the lower bounds on parameters corresponding to the rejected null hypotheses. We illustrate the method with an adaptive phase II/III clinical trial.

Abstract 4257: Genome-wide CpG island methylation analysis identifies tumor specifically methylated genes in non-small cell lung cancer patients

DNA methylation is part of the epigenetic gene regulation complex and it has been shown that methylation of certain genes occurs frequently in non-small cell lung cancers (NSCLC). We performed a genome-wide search for methylated CpG islands in primary tumors and corresponding non-malignant lung tissue samples of 101 stage I-III NSCLC patients by combining methylated DNA immunoprecipitation and microarray analysis using NimbleGen's 385K Human CpG Island plus Promoter arrays (MeDIP-chip). To test for differences in methylation between tumors and corresponding non-malignant lung tissues, we calculated paired t-statistics with permutation adjusted p-values for step down multiple testing. Overall, we identified 2.414 genomic positions differentially methylated between tumor and corresponding non-malignant lung tissue samples by MeDIP-chip analyses. Ninety-seven % of them were found to be tumor-specifically methylated. Annotation of these genomic positions resulted in the identification of 477 tumor-specifically methylated genes. These genes were classified according to Gene Ontology (GO) categories and over-representation of certain GO terms was calculated. Interestingly, we found that a large number of tumor-specifically methylated genes act as regulators of gene expression or mediate homophilic cell adhesion. Tumor-specific methylation of selected genes was confirmed by methylation-sensitive high-resolution melting analysis and ROC curve analyses revealed that primary tumors may be distinguished from non-malignant lung tissue samples by methylation of certain genes. In addition, in the majority of tumors methylation of certain genes was associated with loss of their protein expression determined by immunohistochemistry. Moreover, treatment of NSCLC cells with epigenetically active drugs resulted in upregulated expression of many tumor-specifically methylated genes analysed by gene expression microarrays. In conclusion, we identified a large number of tumor-specifically methylated genes in NSCLC patients. Expression of many of them is regulated by methylation. Overall, our findings emphasize the impact of methylation on the pathogenesis of NSCLCs.

Citation Format: Gerwin Heller, Valerie Babinsky, Barbara Ziegler, Marlene Weinzierl, Christian Noll, Corinna Altenberger, Leonhard Muellauer, Gerhard Dekan, Yuliya Grin, Gyoergy Lang, Adelheid End-Pfützenreuter, Irene Steiner, Sonja Zehetmayer, Balazs Doeme, Madeleine Arns, Kwun M. Fong, Casey M. Wright, Ian A. Yang, Walter Klepetko, Martin Posch, Christoph C. Zielinski, Sabine Zoechbauer-Mueller. Genome-wide CpG island methylation analysis identifies tumor specifically methylated genes in non-small cell lung cancer patients. [abstract]. In: Proceedings of the 104th Annual Meeting of the American Association for Cancer Research; 2013 Apr 6-10; Washington, DC. Philadelphia (PA): AACR; Cancer Res 2013;73(8 Suppl):Abstract nr 4257. doi:10.1158/1538-7445.AM2013-4257

Modeling and simulation to optimize the design and analysis of confirmatory trials, characterize risk-benefit, and support label claims

The role of modeling and simulation (M&S) in the design and interpretation of phase III studies, from break out session 4 of the European Medicines Agency (EMA)/European Federation of Pharmaceutical Industries and Associations (EFPIA) M&S workshop, was divided into themes illustrated with case studies (Table 1): (1) M&S being conducted to support the design of confirmatory trials; (2) longitudinal model‐based test as primary inferential analysis (biosimilarity and disease progression trials); (3) assessment of benefit–risk ratio, approval and labeling of an unstudied dose or dosing regimen, and development of future regulatory guidance.

CPT: Pharmacometrics & Systems Pharmacology (2013) 2, e27; doi:10.1038/psp.2013.4; advance online publication 27 February 2013