Ensuring the comparability of comparison groups: is randomization enough?

A statistical method for removing unbalanced trials with multiple covariates in meta-analysis

In meta-analysis literature, there are several checklists describing the procedures necessary to evaluate studies from a qualitative point of view, whereas preliminary quantitative and statistical investigations on the "combinability" of trials have been neglected. Covariate balance is an important prerequisite to conduct meta-analysis. We propose a method to identify unbalanced trials with respect to a set of covariates, in presence of covariate imbalance, namely when the randomized controlled trials generate a meta-sample that cannot satisfy the requisite of randomization/combinability in meta-analysis. The method is able to identify the unbalanced trials, through four stages aimed at achieving combinability. The studies responsible for the imbalance are identified, and then they can be eliminated. The proposed procedure is simple and relies on the combined Anderson-Darling test applied to the Empirical Cumulative Distribution Functions of both experimental and control meta-arms. To illustrate the method in practice, two datasets from well-known meta-analyses in the literature are used.

Risk of selection bias assessment in the NINDS rt-PA stroke study

OBJECTIVES

The NINDS rt-PA Stroke Study is frequently cited in support of alteplase for acute ischemic stroke within 3 h of symptom onset. Multiple post-hoc reanalyses of this trial have been published to adjust for a baseline imbalance in stroke severity. We performed a risk of selection bias assessment and reanalyzed trial data to determine if the etiology of this baseline imbalance was more likely due to random chance or randomization errors.

METHODS

A risk of selection bias assessment was conducted using signaling questions from the Cochrane Risk of Bias 2 (ROB 2) tool. Four sensitivity analyses were conducted on the trial data based on the randomization process: assessment of imbalances in allocation in unique strata; adherence to a pre-specified restriction on randomization between time strata at each randomization center; assessment of differences in baseline computed tomography (CT) results in unique strata; and comparison of baseline characteristics between allocation groups within each time strata. A multivariable logistic regression model was used to compare reported treatment effects with revised treatment effects after adjustment of baseline imbalances identified in the sensitivity analyses.

RESULTS

Based on criteria from the ROB 2 tool, the risk of bias arising from the randomization process was high. Sensitivity analyses found 11 of 16 unique strata deviated from the expected 1:1 allocation ratio. Three randomization centers violated an apriori rule regarding a maximum difference in allocation between the time strata. Three unique strata had imbalances in baseline CT results that prognostically favored alteplase. Four imbalances in baseline characteristics were identified in the 91-180-min time stratum that all prognostically favored alteplase and were consistent with a larger alteplase treatment effect size compared to the 0-90-min time stratum. After adjustments for baseline imbalances, all reported treatment effects were reduced. Three out of seven originally positive reported results were revised to non-significant.

CONCLUSION

This risk of selection bias assessment revealed a high risk of selection bias in the NINDS rt-PA Stroke Study. Sensitivity analyses conducted based on the randomization process supported this assessment. Baseline imbalances in the trial were more likely due to randomization errors than random chance. Adjusted analyses accounting for baseline imbalances revealed a reduction in reported treatment effects supporting the presence of selection bias in the trial. Treatment decisions and guideline recommendations based on the original treatment effect reported in the NINDS rt-PA Stroke Study should be done cautiously.

A roadmap to using randomization in clinical trials

BACKGROUND

Randomization is the foundation of any clinical trial involving treatment comparison. It helps mitigate selection bias, promotes similarity of treatment groups with respect to important known and unknown confounders, and contributes to the validity of statistical tests. Various restricted randomization procedures with different probabilistic structures and different statistical properties are available. The goal of this paper is to present a systematic roadmap for the choice and application of a restricted randomization procedure in a clinical trial.

METHODS

We survey available restricted randomization procedures for sequential allocation of subjects in a randomized, comparative, parallel group clinical trial with equal (1:1) allocation. We explore statistical properties of these procedures, including balance/randomness tradeoff, type I error rate and power. We perform head-to-head comparisons of different procedures through simulation under various experimental scenarios, including cases when common model assumptions are violated. We also provide some real-life clinical trial examples to illustrate the thinking process for selecting a randomization procedure for implementation in practice.

RESULTS

Restricted randomization procedures targeting 1:1 allocation vary in the degree of balance/randomness they induce, and more importantly, they vary in terms of validity and efficiency of statistical inference when common model assumptions are violated (e.g. when outcomes are affected by a linear time trend; measurement error distribution is misspecified; or selection bias is introduced in the experiment). Some procedures are more robust than others. Covariate-adjusted analysis may be essential to ensure validity of the results. Special considerations are required when selecting a randomization procedure for a clinical trial with very small sample size.

CONCLUSIONS

The choice of randomization design, data analytic technique (parametric or nonparametric), and analysis strategy (randomization-based or population model-based) are all very important considerations. Randomization-based tests are robust and valid alternatives to likelihood-based tests and should be considered more frequently by clinical investigators.

Assessing risk of bias: a proposal for a unified framework for observational studies and randomized trials

BACKGROUND

Evidence based medicine aims to integrate scientific evidence, clinical experience, and patient values and preferences. Individual health care professionals need to appraise the evidence from randomized trials and observational studies when guidelines are not yet available. To date, tools for assessment of bias and terminologies for bias are specific for each study design. Moreover, most tools appeal only to methodological knowledge to detect bias, not to subject matter knowledge, i.e. in-depth medical knowledge about a topic. We propose a unified framework that enables the coherent assessment of bias across designs.

METHODS

Epidemiologists traditionally distinguish between three types of bias in observational studies: confounding, information bias, and selection bias. These biases result from a common cause, systematic error in the measurement or common effect of the intervention and outcome respectively. We applied this conceptual framework to randomized trials and show how it can be used to identify bias. The three sources of bias were illustrated with graphs that visually represent researchers' assumptions about the relationships between the investigated variables (causal diagrams).

RESULTS

Critical appraisal of evidence started with the definition of the research question in terms of the population of interest, the compared interventions and the main outcome. Next, we used causal diagrams to illustrate how each source of bias can lead to over- or underestimated treatment effects. Then, we discussed how randomization, blinded outcome measurement and intention-to-treat analysis minimize bias in trials. Finally, we identified study aspects that can only be appraised with subject matter knowledge, irrespective of study design.

CONCLUSIONS

The unified framework encompassed the three main sources of bias for the effect of an assigned intervention on an outcome. It facilitated the integration of methodological and subject matter knowledge in the assessment of bias. We hope that graphical diagrams will help clarify debate among professionals by reducing misunderstandings based on different terminology for bias.

Detecting the extent of control over selection bias relating to oral health and otorhinolaryngology: cross-sectional study

BACKGROUND

The authors of randomized controlled trials will usually claim that they have met the randomization process criterion. However, sequence generation schemes differ and some schemes that are claimed to be randomized are not genuinely randomized. Even less well understood, and often more difficult to ascertain, is whether the allocation was really concealed.

OBJECTIVE

To detect the extent of control over selection bias, in a comparison between two Cochrane groups: oral health and otorhinolaryngology; and to describe the methods used to control for this bias.

DESIGN AND SETTING

Cross-sectional study conducted in a public university in São Paulo, Brazil.

METHODS

The risk of selection bias in 1,714 records indexed in Medline database up to 2018 was assessed, independent of language and access. Two dimensions implicated in the allocation were considered: generation of the allocation sequence; and allocation concealment.

RESULTS

We included 420 randomized controlled trials and all of them were evaluated to detect selection bias. In the sample studied, only 28 properly controlled the selection bias. Lack of control over selection bias was present in 80% of the studies evaluated in both groups.

CONCLUSION

The two groups were similar regarding control over selection bias. They are also similar to the methods used. The dimension of allocation concealment appears to be a limiting factor with regard to production of randomized controlled trials with low risk of selection bias. The quality of reporting in studies on oral health and otorhinolaryngology is suboptimal and needs to be improved, in line with other fields of healthcare.

Effect of Autologous Bone Marrow Stem Cell Therapy in Patients with Liver Cirrhosis: A Meta-analysis

Background and Aims: Although autologous bone marrow stem cell (BMSC) transplantation is an effective treatment for liver cirrhosis, there are few reports describing the optimal delivery route and number of injected BMSCs. Methods: A literature search was conducted using PubMed, ISI Web of Science, Cochrane Central Register of Controlled Trials, and EBSCO. A meta-analysis was performed to assess the effect of BMSCs on liver and coagulation function indices. Subgroup analysis was performed based on number of injected BMSCs, delivery route, and length of follow-up. Results: A total of 15 studies were selected from among 1903 potential studies for analysis. Autologous BMSC transplantation significantly improved aspartate aminotransferase, total bilirubin, albumin, prothrombin time, prothrombin activity, prothrombin concentration, Child-Pugh score, and model for end-stage liver disease. In the subgroup analysis of cell numbers, all four of the indices were significantly improved when the number of BMSCs was >4 × 10⁸. The subgroup analysis referring to the delivery route showed that arterial infusion increased the therapeutic effect over venous infusion. Finally, in the subgroup analysis of follow-up length, the results showed that BMSC therapy significantly improved liver function at 2 weeks after transplantation. In addition, this therapy improved coagulation 4 weeks after the transplant, with a maintenance of efficacy for up to 24 weeks. Conclusions: Autologous BMSC therapy is beneficial for liver improvement and coagulation in patients with liver cirrhosis. The therapeutic effect was generated at 2-4 weeks after transplantation. The effect lasted for 24 weeks but no more than 48 weeks. The greatest benefit to patients was observed with a 4 × 10⁸ autologous BMSC transplant via the hepatic artery.

The Use of (Network) Meta-Analysis in Clinical Oncology

Meta-analysis is important in oncological research to provide a more reliable answer to a clinical research question that was assessed in multiple studies but with inconsistent results. Pair-wise meta-analysis can be applied when comparing two treatments at once, whereas it is possible to compare multiple treatments at once with network meta-analysis (NMA). After careful systematic review of the literature and quality assessment of the identified studies, there are several assumptions in the use of meta-analysis. First, the added value of meta-analysis should be evaluated by examining the comparability of study populations. Second, the appropriate comparator in meta-analysis should be chosen according to the types of comparisons made in individual studies: (1) Experimental and comparator arms are different treatments (A vs. B); (2) Substitution of a conventional treatment by an experimental treatment (A+B vs. A+C); or (3) Addition of an experimental treatment (A+B vs. B). Ideally there is one common comparator treatment, but when there are multiple common comparators, the most efficacious comparator is preferable. Third, treatments can only be adequately pooled in meta-analysis or merged into one treatment node in NMA when considering likewise mechanism of action and similar setting in which treatment is indicated. Fourth, for both pair-wise meta-analysis and NMA, adequate assessment of heterogeneity should be performed and sub-analysis and sensitivity analysis can be applied to objectify a possible confounding factor. Network inconsistency, as statistical manifestation of violating the transitivity assumption, can best be evaluated by node-split modeling. NMA has advantages over pair-wise meta-analysis, such as clarification of inconsistent outcomes from multiple studies including multiple common comparators and indirect effect calculation of missing direct comparisons between important treatments. Also, NMA can provide increased statistical power and cross-validation of the observed treatment effect of weak connections with reasonable network connectivity and sufficient sample-sizes. However, inappropriate use of NMA can cause misleading results, and may emerge when there is low network connectivity, and therefore low statistical power. Furthermore, indirect evidence is still observational and should be interpreted with caution. NMA should therefore preferably be conducted and interpreted by both expert clinicians in the field and an experienced statistician. Finally, the use of meta-analysis can be extended to other areas, for example the identification of prognostic and predictive factors. Also, the integration of evidence from both meta-analysis and expert opinion can improve the construction of prognostic models in real-world databases.

Influence on Adiposity and Atherogenic Lipaemia of Fatty Meals and Snacks in Daily Life

The present work reviewed the connections of changes in consumption of high-fat food with changes in adiposity and lipaemia in adults with overweight or obesity. Hyperlipaemia from higher fat meals and excessive adiposity contributes to atherogenic process. Low-fat diet interventions decrease body fat, lipaemia, and atherosclerosis markers. Inaccuracy of physical estimates of dietary fat intake remains, however, a limit to establishing causal connections. To fill this gap, tracking fat-rich eating episodes at short intervals quantifies the behavioural frequency suggested to measure (by regression of changes in real time) direct effects of this eating pattern on adiposity and atherogenic lipaemia. Such evidence will provide the basis for an approach focused on a sustained decrease in frequency of fatty meals or snacks to reduce obesity, hyperlipaemia, and atherosclerosis.

Mechanisms and direction of allocation bias in randomised clinical trials

BACKGROUND

Selective allocation of patients into the compared groups of a randomised trial may cause allocation bias, but the mechanisms behind the bias and its directionality are incompletely understood. We therefore analysed the mechanisms and directionality of allocation bias in randomised clinical trials.

METHODS

Two systematic reviews and a theoretical analysis. We conducted one systematic review of empirical studies of motives/methods for deciphering patient allocation sequences; and another review of methods publications commenting on allocation bias. We theoretically analysed the mechanisms of allocation bias and hypothesised which main factors predicts its direction.

RESULTS

Three empirical studies addressed motives/methods for deciphering allocation sequences. Main motives included ensuring best care for patients and ensuring best outcome for the trial. Main methods included various manipulations with randomisation envelopes. Out of 57 methods publications 11 (19 %) mentioned explicitly that allocation bias can go in either direction. We hypothesised that the direction of allocation bias is mainly decided by the interaction between the patient allocators' motives and treatment preference.

CONCLUSION

Inadequate allocation concealment may exaggerate treatment effects in some trials while underestimate effects in others. Our hypothesis provides a theoretical overview of the main factors responsible for the direction of allocation bias.

Use and reporting of restricted randomization: a review

RATIONALE, AIMS AND OBJECTIVES

Restricted randomization, such as blocking or minimization, allows for the creation of balanced groups and even distribution of covariates, but it increases the risk of selection bias and technical error. Various methods are available to reduce these risks but there is limited evidence about their current usage, and there are also indications that reporting of these methods may not be adequate. This review aims to identify how frequently different methods of restriction are being used and to assess the reporting of these methods against established reporting standards.

METHODS

82 reports of randomized controlled trial were reviewed. For each trial, the reported method of randomization was recorded and the reporting of randomization was assessed. Where the method of randomization was not clear from the main paper, protocols and other published materials were also reviewed, and authors were contacted for further information.

RESULTS

For 11% of trials the method of randomization was not reported in either the paper or a published protocol, and in a further 39% of cases the report omitted key details so that the predictability of the method could not be evaluated. In total, 88% of trials appear to have used some form of restricted randomization, and all of those that report the exact methods used either blocking or minimization. 15% of trials reported using blocks of six or less and 4% used minimization with no random element reported, both of which are highly predictable.

CONCLUSION

Our results indicate that the majority of trials use some form of restriction, with many using relatively predictable methods that put them at greater risk of selection bias and technical error. Reporting of randomization methods often falls short of the minimum requirements set out by the CONSORT statement, leaving the reader unable to make an informed judgement about the risk of bias.

Risk of selection bias in randomised trials

BACKGROUND

Selection bias occurs when recruiters selectively enrol patients into the trial based on what the next treatment allocation is likely to be. This can occur even if appropriate allocation concealment is used if recruiters can guess the next treatment assignment with some degree of accuracy. This typically occurs in unblinded trials when restricted randomisation is implemented to force the number of patients in each arm or within each centre to be the same. Several methods to reduce the risk of selection bias have been suggested; however, it is unclear how often these techniques are used in practice.

METHODS

We performed a review of published trials which were not blinded to assess whether they utilised methods for reducing the risk of selection bias. We assessed the following techniques: (a) blinding of recruiters; (b) use of simple randomisation; (c) avoidance of stratification by site when restricted randomisation is used; (d) avoidance of permuted blocks if stratification by site is used; and (e) incorporation of prognostic covariates into the randomisation procedure when restricted randomisation is used. We included parallel group, individually randomised phase III trials published in four general medical journals (BMJ, Journal of the American Medical Association, The Lancet, and New England Journal of Medicine) in 2010.

RESULTS

We identified 152 eligible trials. Most trials (98%) provided no information on whether recruiters were blind to previous treatment allocations. Only 3% of trials used simple randomisation; 63% used some form of restricted randomisation, and 35% did not state the method of randomisation. Overall, 44% of trials were stratified by site of recruitment; 27% were not, and 29% did not report this information. Most trials that did stratify by site of recruitment used permuted blocks (58%), and only 15% reported using random block sizes. Many trials that used restricted randomisation also included prognostic covariates in the randomisation procedure (56%).

CONCLUSIONS

The risk of selection bias could not be ascertained for most trials due to poor reporting. Many trials which did provide details on the randomisation procedure were at risk of selection bias due to a poorly chosen randomisation methods. Techniques to reduce the risk of selection bias should be more widely implemented.

Guidelines for the Reporting of Treatment Trials for Alcohol Use Disorders

BACKGROUND

The primary goals in conducting clinical trials of treatments for alcohol use disorders (AUDs) are to identify efficacious treatments and determine which treatments are most efficacious for which patients. Accurate reporting of study design features and results is imperative to enable readers of research reports to evaluate to what extent a study has achieved these goals. Guidance on quality of clinical trial reporting has evolved substantially over the past 2 decades, primarily through the publication and widespread adoption of the Consolidated Standards of Reporting Trials statement. However, there is room to improve the adoption of those standards in reporting the design and findings of treatment trials for AUD.

METHODS

This paper provides a narrative review of guidance on reporting quality in AUD treatment trials.

RESULTS

Despite improvements in the reporting of results of treatment trials for AUD over the past 2 decades, many published reports provide insufficient information on design or methods.

CONCLUSIONS

The reporting of alcohol treatment trial design, analysis, and results requires improvement in 4 primary areas: (i) trial registration, (ii) procedures for recruitment and retention, (iii) procedures for randomization and intervention design considerations, and (iv) statistical methods used to assess treatment efficacy. Improvements in these areas and the adoption of reporting standards by authors, reviewers, and editors are critical to an accurate assessment of the reliability and validity of treatment effects. Continued developments in this area are needed to move AUD treatment research forward via systematic reviews and meta-analyses that maximize the utility of completed studies.

Randomization Does Not Help Much, Comparability Does

According to R.A. Fisher, randomization "relieves the experimenter from the anxiety of considering innumerable causes by which the data may be disturbed." Since, in particular, it is said to control for known and unknown nuisance factors that may considerably challenge the validity of a result, it has become very popular. This contribution challenges the received view. First, looking for quantitative support, we study a number of straightforward, mathematically simple models. They all demonstrate that the optimism surrounding randomization is questionable: In small to medium-sized samples, random allocation of units to treatments typically yields a considerable imbalance between the groups, i.e., confounding due to randomization is the rule rather than the exception. In the second part of this contribution, the reasoning is extended to a number of traditional arguments in favour of randomization. This discussion is rather non-technical, and sometimes touches on the rather fundamental Frequentist/Bayesian debate. However, the result of this analysis turns out to be quite similar: While the contribution of randomization remains doubtful, comparability contributes much to a compelling conclusion. Summing up, classical experimentation based on sound background theory and the systematic construction of exchangeable groups seems to be advisable.

Identifying items to assess methodological quality in physical therapy trials: a factor analysis

BACKGROUND

Numerous tools and individual items have been proposed to assess the methodological quality of randomized controlled trials (RCTs). The frequency of use of these items varies according to health area, which suggests a lack of agreement regarding their relevance to trial quality or risk of bias.

OBJECTIVE

The objectives of this study were: (1) to identify the underlying component structure of items and (2) to determine relevant items to evaluate the quality and risk of bias of trials in physical therapy by using an exploratory factor analysis (EFA).

DESIGN

A methodological research design was used, and an EFA was performed.

METHODS

Randomized controlled trials used for this study were randomly selected from searches of the Cochrane Database of Systematic Reviews. Two reviewers used 45 items gathered from 7 different quality tools to assess the methodological quality of the RCTs. An exploratory factor analysis was conducted using the principal axis factoring (PAF) method followed by varimax rotation.

RESULTS

Principal axis factoring identified 34 items loaded on 9 common factors: (1) selection bias; (2) performance and detection bias; (3) eligibility, intervention details, and description of outcome measures; (4) psychometric properties of the main outcome; (5) contamination and adherence to treatment; (6) attrition bias; (7) data analysis; (8) sample size; and (9) control and placebo adequacy.

LIMITATION

Because of the exploratory nature of the results, a confirmatory factor analysis is needed to validate this model.

CONCLUSIONS

To the authors' knowledge, this is the first factor analysis to explore the underlying component items used to evaluate the methodological quality or risk of bias of RCTs in physical therapy. The items and factors represent a starting point for evaluating the methodological quality and risk of bias in physical therapy trials. Empirical evidence of the association among these items with treatment effects and a confirmatory factor analysis of these results are needed to validate these items.

How should we evaluate the risk of bias of physical therapy trials?: a psychometric and meta-epidemiological approach towards developing guidelines for the design, conduct, and reporting of RCTs in Physical Therapy (PT) area: a study protocol

BACKGROUND

Numerous tools and items have been developed in all health areas to assess the risk of bias of randomized controlled trials (RCTs). The Cochrane Collaboration (CC) released a new tool to assess bias in RCTs, based on empirical evidence quantifying the association between some design features and estimates of treatment effects (TEs). However, this evidence is limited to medicine and investigating a selected set of components. No such studies have been conducted in other health areas such as Physical Therapy (PT) and allied health professions. Evidence specific to the PT area is needed to understand and quantify the association between design features and TE estimates to inform practice and decision-making in this field. The overall goal of this project is to provide direction for the design, conduct, reporting and bias assessment of PT RCTs. We will achieve this through the following specific objectives and methods.

METHODS/DESIGN

1) to measure the association between methodological components and other factors (for example, PT area, type of intervention, type of outcomes) and TE estimates in RCTs in PT, 40 randomly selected meta-analyses of RCTs involving PT interventions will be identified from the Cochrane Database of Systematic Reviews. Trials will be evaluated independently by two reviewers using the most commonly used tools in the PT field. A two-level analysis will be conducted using a meta-meta-analytic approach; 2) to identify relevant items to evaluate risk of bias of PT trials, an exploratory factor analysis (EFA) will be used to identify the latent structure of the items; 3) to develop guidelines for the design, conduct, reporting, and risk of bias assessment of PT RCTs, items obtained from the factor analysis and the meta-epidemiological approach will be further evaluated by experts in PT through a web-based survey following a Delphi procedure.

DISCUSSION

The results of this project will have a direct impact on research and practice in PT and are valuable to a number of stakeholders: researchers when designing, conducting, and reporting trials; systematic reviewers and meta-analysts when synthesizing trial results; physiotherapists when making day-to-day treatment decision; and, other healthcare decision-makers, such as those developing policy or practice guidelines.

Generalized method for adaptive randomization in clinical trials

A flexible, generalized method of treatment allocation is proposed. The method uses a set of controlling parameters that enables the generic algorithm to produce a family of possible outcomes ranging from simple randomization to deterministic allocation. The method controls balance at stratum level, stratification level and overall without detriment to the predictability of the method. The paper lists the desirable characteristics of allocation methods and shows that the proposed method fulfils the majority and is easy to use in the clinical context, once the coding has been established. An explanation of the method for 2, 3 and 4 treatment group allocations is given. Simulations demonstrate the flexibility of the method.

Self-medication of migraine and tension-type headache: summary of the evidence-based recommendations of the Deutsche Migräne und Kopfschmerzgesellschaft (DMKG), the Deutsche Gesellschaft für Neurologie (DGN), the Österreichische Kopfschmerzgesellschaft (ÖKSG) and the Schweizerische Kopfwehgesellschaft (SKG)

The current evidence-based guideline on self-medication in migraine and tension-type headache of the German, Austrian and Swiss headache societies and the German Society of Neurology is addressed to physicians engaged in primary care as well as pharmacists and patients. The guideline is especially concerned with the description of the methodology used, the selection process of the literature used and which evidence the recommendations are based upon. The following recommendations about self-medication in migraine attacks can be made: The efficacy of the fixed-dose combination of acetaminophen, acetylsalicylic acid and caffeine and the monotherapies with ibuprofen or naratriptan or acetaminophen or phenazone are scientifically proven and recommended as first-line therapy. None of the substances used in self-medication in migraine prophylaxis can be seen as effective. Concerning the self-medication in tension-type headache, the following therapies can be recommended as first-line therapy: the fixed-dose combination of acetaminophen, acetylsalicylic acid and caffeine as well as the fixed combination of acetaminophen and caffeine as well as the monotherapies with ibuprofen or acetylsalicylic acid or diclofenac. The four scientific societies hope that this guideline will help to improve the treatment of headaches which largely is initiated by the patients themselves without any consultation with their physicians.

Cluster size variability and imbalance in cluster randomized controlled trials

Cluster randomized controlled trials are increasingly used to evaluate medical interventions. Research has found that cluster size variability leads to a reduction in the overall effective sample size. Although reporting standards of cluster trials have started to evolve, a far greater degree of transparency is needed to ensure that robust evidence is presented. The use of the numbers of patients recruited to summarize recruitment rate should be avoided in favour of an improved metric that illustrates cumulative power and accounts for cluster variability. Data from four trials is included to show the link between cluster size variability and imbalance. Furthermore, using simulations it is demonstrated that by randomising using a two block randomization strategy and weighting the second by cluster size recruitment, chance imbalance can be minimized.

Assessing performance of a randomized versus a non-randomized study design

INTRODUCTION

Randomization is the most optimal design for evaluating program-effectiveness. In practice, however, conducting a randomized controlled trial is not always feasible. For a non-randomized study into the effect of a parent management training, predefined intervention and control groups of families were matched on six key characteristics. The quality of this match was then compared with the quality which is to be expected from a randomized study.

METHODS

The performance of matching intervention and control families for predefined and randomized groups was evaluated by simulating new hypothetical intervention and control groups. The Mahalanobis metric was used to assess the distance between families in the intervention and the control groups and pairwise matching was performed. The global distance between these groups was used as measure of the balance of covariates in all matched pairs, with a smaller distance indicating a higher match quality.

RESULTS

In the ideal situation, when predefined groups are actually equal to randomized groups, the expected probability of a more equal balance of characteristics in the former groups than in the latter groups is 0.50. Using the data obtained in our study, and our predefined groups, this expected probability was 0.34.

CONCLUSION

Even when randomized groups are more balanced than predefined groups, using the latter groups for analyses might still be acceptable when the differences in group means are small. Findings suggest that matching can be a viable alternative to randomization for situations in which randomization is not feasible due to pragmatic constraints. However, a more accurate judgment on the value of the results obtained in this study requires results from similar analyses performed in other studies for comparison.

Randomized trials for the real world: making as few and as reasonable assumptions as possible

The strength of the randomized trial to yield conclusions not dependent on assumptions applies only in an ideal setting. In the real world various complications such as loss-to-follow-up, missing outcomes, noncompliance and nonrandom selection into a trial force a reliance on assumptions. To handle real world complications, it is desirable to make as few and as reasonable assumptions as possible. This article reviews four techniques for using a few reasonable assumptions to design or analyse randomized trials in the presence of specific real world complications: 1) a double sampling design for survival data to avoid strong assumptions about informative censoring, 2) sensitivity analysis for partially missing binary outcomes that uses the randomization to reduce the number of parameters specified by the investigator, 3) an estimate of the effect of treatment received in the presence of all-or-none compliance that requires reasonable assumptions, and 4) statistics for binary outcomes that avoid some assumptions for generalizing results to a target population.

Efficacy and Safety of TachoSil ® as Haemostatic Treatment versus Standard Suturing in Kidney Tumour Resection: A Randomised Prospective Study

OBJECTIVE

Elective nephron-sparing surgery (NSS) for renal cell carcinoma (RCC) has gained general acceptance as an alternative to radical nephrectomy. To achieve haemostasis without risk of local ischaemia and necrosis of kidney parenchyma after standard haemostatic suturing, we investigated TachoSil's efficacy and safety as atraumatic haemostatic treatment after kidney tumour resection.

METHODS

A total of 185 patients scheduled for NSS for small, superficial kidney tumours were included in an open, randomised, prospective, multicentre, parallel-group trial. Primary objectives were to test haemostatic efficacy and safety of TachoSil versus standard suturing. Efficacy was tested by comparing intraoperative time to haemostasis (primary end point). Secondary objectives included proportion of subjects with haemostasis after 10 min of trial treatment, occurrence of haematoma on day 2 after surgery, volume and haemoglobin concentration of postoperative drainage fluid, and surgeon's rating of usefulness of trial treatments. Safety was evaluated by occurrence of adverse events.

RESULTS

In the intent-to-treat population, time to haemostasis was significantly shorter with TachoSil versus standard suturing (mean: 5.3 vs. 9.5 min [p<0.0001]). Haemostasis was obtained within 10 min in 92% of patients in the TachoSil group and in 67% in the standard treatment group (p<0.0001). Differences in other secondary end points were not statistically significant. Both treatments were well tolerated. Surgeons rated TachoSil higher in terms of convenience to prepare and apply, and impression of efficacy.

CONCLUSION

TachoSil was superior to standard suturing in obtaining intraoperative control of haemorrhage and was as well tolerated as standard haemostatic treatment during NSS.

Enhancing the validity and utility of randomized clinical trials in addictions treatment research: III. Data processing and statistical analysis

PURPOSE

This is the third paper in a series that reviews strategies for optimizing the validity and utility of randomized clinical trials (RCTs) in addictions treatment research. Whereas the two previous papers focused on design and implementation, here we address issues pertaining to data processing and statistical analysis.

SCOPE

Recommendations for enhancing data quality and utility are offered in sections on data coding and entry; and data format, structure and management. We discuss the need for preliminary data analyses that examine statistical power; patterns of attrition; between-group equivalence; and treatment integrity and discriminability. We discuss tests of treatment efficacy, as well as ancillary analyses aimed at explicating treatment processes.

CONCLUSIONS

Safeguards are necessary to protect data quality, and advance planning is needed to ensure that data formats are compatible with statistical objectives. In addition to treatment efficacy, statistical analyses should evaluate study internal and external validity, and investigate the change mechanisms that underlie treatment effects.

The impact of trial baseline imbalances should be considered in systematic reviews: a methodological case study

OBJECTIVES

It is possible for baseline imbalances to occur between treatment groups for one or more variables in a randomized controlled trial, although the identification and detection of baseline imbalances remain controversial. If trials with baseline imbalances are combined in a meta-analysis, then this may result in misleading conclusions.

STUDY DESIGN AND SETTING

The identification and consequences of baseline imbalances in meta-analyses are discussed. Metaregression using mean baseline scores as a covariate is proposed as a potential method for adjusting baseline imbalances within meta-analysis. We will use a recent systematic review looking at the effect of calcium supplements on weight as an illustrative case study.

RESULTS

Meta-analysis conducted using the mean final values of the treatment groups as the outcome resulted in an apparent, statistically significant, treatment effect. However, using a meta-analysis of baseline values, this was shown to be due to the baseline imbalance between treatment groups, rather than as a result of any intervention received by the participants. Applying the method of metaregression demonstrated that there was in fact a smaller, statistically insignificant effect between treatment groups.

CONCLUSION

The meta-analyst should always consider the possibility of baseline imbalances and adjustments should be made wherever possible.

Enhancing the validity and utility of randomized clinical trials in addictions treatment research: I. Treatment implementation and research design

PURPOSE

This paper is the first in a series that examines methods for improving the validity and utility of randomized clinical trials (RCTs) in addictions treatment research. The specific foci of this article are treatment implementation and research design.

SCOPE

We begin by considering the conditions under which the RCT provides an appropriate design choice. Sections that follow discuss methodological issues with respect to RCT structure and collaborative arrangements; treatment specification, delivery and cost; experimental design; and randomization/blinding procedures. We emphasize the importance of advance planning; treatment integrity and discriminability; treatment standardization; staff training and supervision; client compliance; maintenance of between-group equivalence across study conditions; and inclusion of appropriate comparison groups in study designs.

CONCLUSIONS

Investigators are encouraged to maximize the internal validity of RCTs, but also to consider methods for enhancing external validity. The utility of addictions RCTs for advancing theory and improving clinical practice can be enhanced by investigating underlying mechanisms of action.

What does biostatistics mean to us

It is human nature to try to recognize patterns and to make sense of that which we observe. Unfortunately, our intuition is often wrong, and so there is a need to impose some objectivity on the methods by which observations are converted into knowledge. One definition of biostatistics could be precisely this, the rigorous and objective conversion of medical and/or biological observations into knowledge. Both consumers of biostatistical principles and biostatisticians themselves vary in the extent to which they recognize the need to continue the improvement. Some may not recognize the need for (some or all of) the methods that have already been developed; others may accept these as they find them completely sufficient; still others recognize both the value and the shortcomings of these methods, and seek to develop even better methods to ensure that future medical conclusions are less subject to biases than current ones are.

The feasibility of Whole Body Vibration in institutionalised elderly persons and its influence on muscle performance, balance and mobility: a randomised controlled trial [ISRCTN62535013]

BACKGROUND

Fatigue or lack of interest can reduce the feasibility of intensive physical exercise in nursing home residents. Low-volume exercise interventions with similar training effects might be an alternative. The aim of this randomised controlled trial was to investigate the feasibility of Whole Body Vibration (WBV) in institutionalised elderly, and its impact on functional capacity and muscle performance.

METHODS

Twenty-four nursing home residents (15 female, 9 male; mean age 77.5 +/- 11.0 years) were randomised (stratification for age, gender and ADL-category) to 6 weeks static WBV exercise (WBV+, N = 13) or control (only static exercise; N = 11). Outcome measures were exercise compliance, timed up-and-go, Tinetti-test, back scratch, chair sit-and-reach, handgrip strength and linear isokinetic leg extension.

RESULTS

At baseline, WBV+ and control groups were similar for all outcome variables. Twenty-one participants completed the program and attended respectively 96% and 86% of the exercise sessions for the WBV+ and control groups. Training-induced changes in timed up-and-go and Tinetti-test were better for WBV+ compared to control (p = 0.029 for timed up-and-go, p = 0.001 and p = 0.002 for Tinetti body balance and total score respectively). In an alternative analysis (Worst Rank Score & Last Observation Carried Forward) the differences in change remained significant on the Tinetti body balance and total score. No other significant differences in change between both groups were observed.

CONCLUSION

In nursing home residents with limited functional dependency, six weeks static WBV exercise is feasible, and is beneficial for balance and mobility. The supplementary benefit of WBV on muscle performance compared to classic exercise remains to be explored further.

Intraocular pressure-lowering effects of all commonly used glaucoma drugs: a meta-analysis of randomized clinical trials

OBJECTIVE

To estimate the intraocular pressure (IOP) reduction achieved by the most frequently prescribed glaucoma drugs and a placebo in a meta-analysis of randomized clinical trials.

DESIGN

Meta-analysis of randomized clinical trials.

PARTICIPANTS

Twenty-seven articles reporting on 28 randomized clinical trials. These articles reported 6953 participants for the trough and 6841 for the peak.

METHODS

Articles published up to December 2003 were identified in the following data sources: Medline, Embase, and the Cochrane Controlled Trials Register, and references from relevant articles. Over 85% of the patients had to be diagnosed with primary open-angle glaucoma (POAG) or ocular hypertension (OH), and articles had to be written in English, German, French, or Dutch. Quality of trials was assessed by a Delphi list with additions. The pooled 1-month IOP-lowering effect from baseline at peak and trough was calculated by performing meta-analysis using the random effects model.

MAIN OUTCOME MEASURES

Absolute and relative change in IOP from baseline, for peak and trough moments.

RESULTS

Relative IOP reductions from baseline [mean (95% confidence interval)] were -23% (-25% to -22%) for a peak and -20% (-23% to -17%) for a trough for 0.5% betaxolol; peak, -27% (-29% to -25%), and trough, -26% (-28% to -25%), for 0.5% timolol; peak, -22% (-24% to -20%), and trough, -17% (-19% to -15%), for 2.0% dorzolamide; peak, -17% (-19% to -15%), and trough, -17% (-19% to -15%) for 1.0% brinzolamide; peak, -25% (-28% to -22%), and trough, -18% (-21% to -14%) for 0.2% brimonidine; peak, -31% (-33% to -29%), and trough, -28% (-30% to -26%) for 0.005% latanoprost; peak, -31% (-32% to -29%), and trough, -29% (-32% to -25%) for 0.004% travoprost; peak, -33% (-35% to -31%), and trough, -28% (-29% to -27%) for 0.03% bimatoprost; and peak, -5% (-9% to -1%), and trough, -5% (-10% to -0%) for the placebo. The difference in absolute IOP reduction from baseline between timolol and prostaglandin analogs or prostamide varied from -0.4 to 0.1 mmHg at trough and from 1.0 to 1.5 mmHg at peak. Quality scores of included studies were generally high, a mean of 14.2 on a scale from 0 to 20 (interquartile range, 13-16).

CONCLUSION

This meta-analysis suggests that bimatoprost, travoprost, latanoprost, and timolol are the most effective intraocular pressure-reducing agents in POAG and OH patients.

Quantifying the Magnitude of Baseline Covariate Imbalances Resulting from Selection Bias in Randomized Clinical Trials

Selection bias is most common in observational studies, when patients select their own treatments or treatments are assigned based on patient characteristics, such as disease severity. This first-order selection bias, as we call it, is eliminated by randomization, but there is residual selection bias that may occur even in randomized trials which occurs when, subconsciously or otherwise, an investigator uses advance knowledge of upcoming treatment allocations as the basis for deciding whom to enroll. For example, patients more likely to respond may be preferentially enrolled when the active treatment is due to be allocated, and patients less likely to respond may be enrolled when the control group is due to be allocated. If the upcoming allocations can be observed in their entirety, then we will call the resulting selection bias second-order selection bias. Allocation concealment minimizes the ability to observe upcoming allocations, yet upcoming allocations may still be predicted (imperfectly), or even determined with certainty, if at least some of the previous allocations are known, and if restrictions (such as randomized blocks) were placed on the randomization. This mechanism, based on prediction but not observation of upcoming allocations, is the third-order selection bias that is controlled by perfectly successful masking, but without perfect masking is not controlled even by the combination of advance randomization and allocation concealment. Our purpose is to quantify the magnitude of baseline imbalance that can result from third-order selection bias when the randomized block procedure is used. The smaller the block sizes, the more accurately one can predict future treatment assignments in the same block as known previous assignments, so this magnitude will depend on the block size, as well as on the level of certainty about upcoming allocations required to bias the patient selection. We find that a binary covariate can, on average, be up to 50% unbalanced by third-order selection bias.

The reverse propensity score to detect selection bias and correct for baseline imbalances

The propensity score has been proposed, and for the most part accepted, as a tool to allow for the evaluation of medical interventions in the presence of baseline imbalances arising in the context of observational studies. The lack of an analogous tool to allow for the evaluation of medical interventions in the presence of potentially systematic baseline imbalances in randomized trials has required the use of ad hoc methods. This, in turn, leads to challenges to the conclusions. For example, much of the controversy surrounding recommendations for or against mammography for some age groups stems from the fact that all the randomized trials to study mammography had baseline imbalances, to some extent, in important prognostic covariates. While some of these trials used cluster randomization, baseline imbalances are prevalent also in individually randomized trials. We provide a systematic approach for evaluating medical interventions in the presence of potentially systematic baseline imbalances in individually randomized trials with allocation concealment. Specifically, we define the reverse propensity score as the probability, conditional on all previous allocations and the allocation procedure (restrictions on the randomization), that a given patient will receive a given treatment. We demonstrate how the reverse propensity score allows for both detection of and correction for selection bias, or systematic baseline imbalances.