Post hoc power estimation in large-scale multiple testing problems

BACKGROUND

The statistical power or multiple Type II error rate in large-scale multiple testing problems as, for example, in gene expression microarray experiments, depends on typically unknown parameters and is therefore difficult to assess a priori. However, it has been suggested to estimate the multiple Type II error rate post hoc, based on the observed data.

METHODS

We consider a class of post hoc estimators that are functions of the estimated proportion of true null hypotheses among all hypotheses. Numerous estimators for this proportion have been proposed and we investigate the statistical properties of the derived multiple Type II error rate estimators in an extensive simulation study.

RESULTS

The performance of the estimators in terms of the mean squared error depends sensitively on the distributional scenario. Estimators based on empirical distributions of the null hypotheses are superior in the presence of strongly correlated test statistics.

AVAILABILITY

R-code to compute all considered estimators based on P-values and supplementary material is available on the authors web page http://statistics.msi.meduniwien.ac.at/index.php?page=pageszfnr.

Selection and bias--two hostile brothers

We consider the situation where in a first stage of a clinical trial several treatments are compared with a single control and the 'best' treatment(s) are selected in an interim analysis to be carried on to the second stage. We quantify the mean bias and mean square error of the conventional estimates after selection depending on the number of treatments and the selection time during the trial. The cases without or with reshuffling the planned sample size of the dropped treatments to the selected ones are investigated. The mean bias shows very different patterns depending on the selection rule and the unknown parameter values. We stress the fact that the quantification of the bias is possible only in designs with planned adaptivity where the design allows reacting to new evidence, but the decision rules are laid down in advance. Finally, we calculate the mean bias which arises in a simple but influential regulatory selection rule, to register a new medical therapy only when two pivotal trials have both proven an effect by a statistical test.

Use of dynamic soil-vegetation models to assess impacts of nitrogen deposition on plant species composition: an overview

Field observations and experimental data of effects of nitrogen (N) deposition on plant species diversity have been used to derive empirical critical N loads for various ecosystems. The great advantage of such an approach is the inclusion of field evidence, but there are also restrictions, such as the absence of explicit criteria regarding significant effects on the vegetation, and the impossibility to predict future impacts when N deposition changes. Model approaches can account for this. In this paper, we review the possibilities of static and dynamic multispecies models in combination with dynamic soil-vegetation models to (1) predict plant species composition as a function of atmospheric N deposition and (2) calculate critical N loads in relation to a prescribed protection level of the species composition. The similarities between the models are presented, but also several important differences, including the use of different indicators for N and acidity and the prediction of individual plant species vs. plant communities. A summary of the strengths and weaknesses of the various models, including their validation status, is given. Furthermore, examples are given of critical load calculations with the model chains and their comparison with empirical critical N loads. We show that linked biogeochemistry-biodiversity models for N have potential for applications to support European policy to reduce N input, but the definition of damage thresholds for terrestrial biodiversity represents a major challenge. There is also a clear need for further testing and validation of the models against long-term monitoring or long-term experimental data sets and against large-scale survey data. This requires a focused data collection in Europe, combing vegetation descriptions with variables affecting the species diversity, such as soil acidity, nutrient status and water availability. Finally, there is a need for adaptation and upscaling of the models beyond the regions for which dose-response relationships have been parameterized, to make them generally applicable.

Optimal choice of the number of treatments to be included in a clinical trial

It is common for a number of potentially effective treatments to be available for clinical evaluation. Limitations on resources mean that this inevitably leads to a decision as to how many, and which, treatments should be considered for inclusion in a clinical trial. This paper considers the problem of selection of possible treatments for inclusion in a phase III clinical trial. We assume that treatments will be compared using a standard frequentist hypothesis test, and propose a Bayesian decision-theoretic approach that leads to minimization of the total sample size of the trial subject to controlling the familywise type I error rate and the expected probability of rejecting at least one null hypothesis. The method is illustrated in the simplest situation, in which two experimental treatments could be included in the clinical trial, exploring the levels of evidence that are required to lead to an optimal trial that includes one or both of these treatments.

Respective potencies of Botox and Dysport in a human skin model: a randomized, double-blind study

Mouse units used to quantify the activity of botulinum A toxin preparations are not equivalent and issues concerning efficacy and safety remain with regard to their respective potencies and diffusion qualities in human tissue. We compared the effects of Botox (BOT) and Dysport (DYS) in different doses and dilutions in a human skin model. Eighteen (8 women, 10 men) healthy volunteers, aged 28.4 years +/- 5.7 years were injected intradermally with pure saline, BOT and DYS at 16 points in the abdomen in random order and in a double-blind condition, using two conversion ratios (1:3 and 1:4) and three different dilution schemes. For an objective outcome, the Ninhydrin sweat test was used to compare the anhidrotic areas. Both preparations showed a linear dose and dilution relationship with similar variances of responses for anhidrosis and hypohidrosis, indicating the same reliability of response. The dose equivalence conversion ratios (BOT: DYS) were 1:1.3 for anhidrosis and 1:1.6 for hypohidrosis (1:1.1-1.5 and 1:1.4-1.8 95% confidence intervals). The diffusion characteristics of both products were similar. A dose equivalence factor of more than 1:2 (BOT:DYS) is not supported by these objective and reproducible data.

Adaptive designs for confirmatory clinical trials

Adaptive designs play an increasingly important role in clinical drug development. Such designs use accumulating data of an ongoing trial to decide how to modify design aspects without undermining the validity and integrity of the trial. Adaptive designs thus allow for a number of possible adaptations at midterm: Early stopping either for futility or success, sample size reassessment, change of population, etc. A particularly appealing application is the use of adaptive designs in combined phase II/III studies with treatment selection at interim. The expectation has arisen that carefully planned and conducted studies based on adaptive designs increase the efficiency of the drug development process by making better use of the observed data, thus leading to a higher information value per patient.In this paper we focus on adaptive designs for confirmatory clinical trials. We review the adaptive design methodology for a single null hypothesis and how to perform adaptive designs with multiple hypotheses using closed test procedures. We report the results of an extensive simulation study to evaluate the operational characteristics of the various methods. A case study and related numerical examples are used to illustrate the key results. In addition we provide a detailed discussion of current methods to calculate point estimates and confidence intervals for relevant parameters.

A graphical approach to sequentially rejective multiple test procedures

For clinical trials with multiple treatment arms or endpoints a variety of sequentially rejective, weighted Bonferroni-type tests have been proposed, such as gatekeeping procedures, fixed sequence tests, and fallback procedures. They allow to map the difference in importance as well as the relationship between the various research questions onto an adequate multiple test procedure. Since these procedures rely on the closed test principle, they usually require the explicit specification of a large number of intersection hypotheses tests. The underlying test strategy may therefore be difficult to communicate. We propose a simple iterative graphical approach to construct and perform such Bonferroni-type tests. The resulting multiple test procedures are represented by directed, weighted graphs, where each node corresponds to an elementary hypothesis, together with a simple algorithm to generate such graphs while sequentially testing the individual hypotheses. The approach is illustrated with the visualization of several common gatekeeping strategies. A case study is used to illustrate how the methods from this article can be used to tailor a multiple test procedure to given study objectives.

T cell senescence and contraction of T cell repertoire diversity in patients with chronic obstructive pulmonary disease

Pathogenetic mechanisms leading to chronic obstructive pulmonary disease (COPD) remain poorly understood. Because clonogenic T cells (CD4(+)CD28(null)) were shown to be increased in autoimmune diseases we hypothesized that CD4(+)CD28(null) T cells play a role in COPD. Here we describe that enhanced presence of CD4(+)CD28(null) cells is associated with impaired lung function. Sixty-four patients and controls were included. T cell phenotype was analysed using flow cytometry. Enzyme-linked immunosorbent assays were utilized to determine cytokines. Statistical evaluations were performed using non-parametric group comparisons and correlations. A logistic regression model was used to determine predictive values of CD4(+)CD28(null) in the diagnosis of COPD. Populations of CD4(+) T cells lacking surface co-stimulatory CD28 were enlarged significantly in evaluated patients when compared with controls. Natural killer (NK)-like T cell receptors (CD94, 158) and intracellular perforin, granzyme B were increased in CD4(+)CD28(null) cells. Cytokine production after triggering of peripheral blood mononuclear cells (PBMCs) was elevated in patients at early disease stages. Receiver operating characteristic curve plotting revealed that presence of CD4(+)CD28(null) T cells has a diagnostic value. These CD4(+)CD28(null) T cells show increased expression of NK-like T cell receptors (CD94, 158) and intracellular perforin and granzyme B. Furthermore, triggering of PBMCs obtained from patients with mild COPD led to increased interferon-gamma and tumour necrosis factor-alpha production in vitro compared with controls. Our finding of increased CD4(+)CD28(null) T cells in COPD indicates that chronic antigen exposure, e.g. through contents of smoke, leads to loss of CD28 and up-regulation of NK cell receptors expression on T cells in susceptible patients.

Optimized multi-stage designs controlling the false discovery or the family-wise error rate

When a large number of hypotheses are investigated, we propose multi-stage designs where in each interim analysis promising hypotheses are screened, which are investigated in further stages. Given a fixed overall number of observations, this allows one to spend more observations for promising hypotheses than with single-stage designs, where the observations are equally distributed among all considered hypotheses. We propose multi-stage procedures controlling either the family-wise error rate (FWER) or the false discovery rate (FDR) and derive asymptotically optimal stopping boundaries and sample size allocations (across stages) to maximize the power of the procedure. Optimized two-stage designs lead to a considerable increase in power compared with the classical single-stage design. Going from two to three stages additionally leads to a distinctive increase in power. Adding a fourth stage leads to a further improvement, which is, however, less pronounced. Surprisingly, we found only small differences in power between optimized integrated designs, where the data of all stages are used in the final test statistics, and optimized pilot designs where only the data from the final stage are used for testing. However, the integrated design controlling the FDR appeared to be more robust against misspecifications in the planning phase. Additionally, we found that with increasing number of stages the drop in power when controlling the FWER instead of the FDR becomes negligible. Our investigations show that the crucial point is not the choice of the error rate or the type of design, but the sequential nature of the trial where non-promising hypotheses are dropped in the early phases of the experiment.

Modelling and mapping long-term risks due to reactive nitrogen effects: an overview of LRTAP convention activities

Long-range transboundary air pollution has caused severe environmental effects in Europe. European air pollution abatement policy, in the framework of the UNECE Convention on Long-range Transboundary Air Pollution (LRTAP Convention) and the European Union Clean Air for Europe (CAFE) programme, has used critical loads and their exceedances by atmospheric deposition to design emission abatement targets and strategies. The LRTAP Convention International Cooperative Programme on Modelling and Mapping Critical Loads and Levels and Air Pollution Effects, Risks and Trends (ICP M&M) generates European critical loads datasets to enable this work. Developing dynamic nitrogen flux models and using them for a prognosis and assessment of nitrogen effects remains a challenge. Further research is needed on links between nitrogen deposition effects, climate change, and biodiversity.

Adaptive Dunnett tests for treatment selection

Clinical trials incorporating treatment selection at pre-specified interim analyses allow to integrate two clinical studies into a single, confirmatory study. In an adaptive interim analysis, treatment arms are selected based on interim data as well as external information. The specific selection rule does not need to be pre-specified in advance in order to control the multiple type I error rate. We propose an adaptive Dunnett test procedure based on the conditional error rate of the single-stage Dunnett test. The adaptive procedure uniformly improves the classical Dunnett test, which is shown to be strictly conservative if treatments are dropped at interim. The adaptive Dunnett test is compared in a simulation with the classical Dunnett test as well as with adaptive combination tests based on the closure principle. The method is illustrated with a real-data example.

Effects of N-acetylcysteine against systemic and renal hemodynamic effects of endotoxin in healthy humans

OBJECTIVE

Systemic inflammation causes vasodilation and impairs the vascular response to catecholamines. There is evidence that altered vasoreactivity is associated with increased production of free radicals. We studied the influence of systemic doses of the antioxidant N-acetylcysteine on inflammatory cytokines and renal plasma flow and on the systemic pressor response to norepinephrine during experimental endotoxemia.

DESIGN

A double-blind, placebo-controlled crossover study.

SETTING

Medical University of Vienna, Clinical Pharmacology, Vienna General Hospital, AKH.

SUBJECTS

Eight healthy, male humans.

INTERVENTIONS

Intravenous administration of Escherichia coli endotoxin (lipopolysaccharide, 20 IU/kg) on two separate study days with concomitant intravenous infusion of placebo or N-acetylcysteine (150 mg/kg loading dose; 15 mg/kg/hr continuous infusion), respectively.

MEASUREMENTS AND MAIN RESULTS

Measurements of inflammatory cytokines, of renal plasma flow by the para-aminohippurate-clearance method, and of the systemic pressor response to norepinephrine were taken at baseline and after endotoxin. Lipopolysaccharide increased body temperature and plasma concentrations of tumor necrosis factor-alpha, which was mitigated during N-acetylcysteine infusions. Likewise, the lipopolysaccharide-induced increases in renal plasma flow and decreases in blood pressure were attenuated, and the hyporeactivity of pulse rate to norepinephrine 4 hrs after lipopolysaccharide was improved by N-acetylcysteine.

CONCLUSION

High doses of N-acetylcysteine might exert protective effects on systemic hemodynamics and on the reactivity to catecholamines in humans challenged by lipopolysaccharide. This action of the antioxidant N-acetylcysteine is paralleled by humoral anti-inflammatory mechanisms and may be useful in patients with systemic inflammation.

Physiotherapy-based rehabilitation following disc herniation operation: results of a randomized clinical trial

STUDY DESIGN

Three-group, randomized, single blinded, controlled trial.

OBJECTIVE

To test the effectiveness of physiotherapy-based rehabilitation starting 1 week after lumbar disc surgery. In addition, we tried to estimate the contribution of specific effects to the observed outcome (efficacy).

SUMMARY OF BACKGROUND DATA

Physiotherapy-based rehabilitation is usually recommended for patients following lumbar disc surgery. Few and conflicting data exist for the relative effectiveness of this intervention.

METHODS

A total of 120 patients following first-time, uncomplicated lumbar disc surgery were randomly assigned to "comprehensive" physiotherapy, "sham" neck massage, or no therapy. Before enrollment, all subjects completed a minimal physiotherapeutic intervention. Physiotherapy was administered by experienced physiotherapists and consisted of 20 sessions per patient over 12 weeks. Masseurs administered "sham massage" to the neck. The amount of treatment time was equal to that of physiotherapy. The main outcome measure was the Low Back Pain Rating Score (LBPRS) at 6 and 12 weeks, and 1.5 years after randomization. Secondary parameters were patients' overall satisfaction with treatment outcome and socioeconomic and psychologic measures.

RESULTS

At the end of therapy (12 weeks), the LBPRS revealed a significantly better improvement in the physiotherapy group than in the untreated group. LBPRS outcome, however, did not significantly differ between physiotherapy and "sham" therapy. There was a tendency toward significance between the sham therapy and no therapy. Within the 1.5-year follow-up, LBP rating scales remained significantly improved compared with baseline, but there were no significant outcome differences. No statistically significant between-group differences were found for the secondary outcome parameters.

CONCLUSION

As compared with no therapy, physiotherapy following first-time disc herniation operation is effective in the short-term. Because of the limited benefits of physiotherapy relative to "sham" therapy, it is open to question whether this treatment acts primarily physiologically in patients following first-time lumbar disc surgery, but psychological factors may contribute substantially to the benefits observed.

Elevated levels of interleukin-1beta-converting enzyme and caspase-cleaved cytokeratin-18 in acute myocardial infarction

BACKGROUND

Systemic inflammation and apoptosis-specific immune activation play a major role in acute coronary syndromes (ACS) including acute myocardial infarction (AMI). The role of systemic and coronary obtained inflammatory plasma protein interleukin-1beta precursor (IL-1betap), IL-1beta-converting enzyme (ICE) and the apoptosis-specific caspase-cleaved cytokeratin-18 (ccCK-18) are not known in ACS.

MATERIALS AND METHODS

Plasma samples were obtained from stable angina (SA, n = 34), unstable angina (UA, n = 37) and patients with AMI (n = 39). Coronary blood was acquired by means of thrombectomy devices (X-sizer) in AMI patients. IL-1betap, ICE and ccCK-18 were determined by enzyme-linked immunosorbent assay (ELISA). Group comparisons were evaluated by parametric Tukey test. Multivariate logistic regression analysis was performed to determine predictive values of IL-1betap, ICE and ccCK-18 as compared to creatine kinase (CK) and troponin T (TnT) in order to relate these markers with the occurrence of myocardial damage.

RESULTS

IL-1betap, ICE and ccCK-18 were identified to be significantly altered in the peripheral blood of patients suffering from AMI as compared to SA and UA. ROC curves were plotted and revealed that ccCK-18 is a novel sensitive marker for the detection of myocardial damage as compared to TnT or CK. (AUC ccCK-18 0.925, TnT AUC 0.62 and CK AUC 0.858.) Moreover, ICE and ccCK-18 were significantly increased at the site of coronary occlusion as compared to peripheral blood samples in AMI patients (both P < 0.001).

CONCLUSION

Our data suggest that ACS is related to increased concentration of systemic soluble ICE and ccCK-18. Moreover, soluble ccCK-18 was identified to be a superior marker as compared to TnT or CK, for detection of myocardial damage.