The methodological quality of cluster randomised controlled trials for managing tropical parasitic disease: a review of trials published from 1998 to 2007

Influential methods reports for group-randomized trials and related designs

BACKGROUND

This article identifies the most influential methods reports for group-randomized trials and related designs published through 2020. Many interventions are delivered to participants in real or virtual groups or in groups defined by a shared interventionist so that there is an expectation for positive correlation among observations taken on participants in the same group. These interventions are typically evaluated using a group- or cluster-randomized trial, an individually randomized group treatment trial, or a stepped wedge group- or cluster-randomized trial. These trials face methodological issues beyond those encountered in the more familiar individually randomized controlled trial.

METHODS

PubMed was searched to identify candidate methods reports; that search was supplemented by reports known to the author. Candidate reports were reviewed by the author to include only those focused on the designs of interest. Citation counts and the relative citation ratio, a new bibliometric tool developed at the National Institutes of Health, were used to identify influential reports. The relative citation ratio measures influence at the article level by comparing the citation rate of the reference article to the citation rates of the articles cited by other articles that also cite the reference article.

RESULTS

In total, 1043 reports were identified that were published through 2020. However, 55 were deemed to be the most influential based on their relative citation ratio or their citation count using criteria specific to each of the three designs, with 32 group-randomized trial reports, 7 individually randomized group treatment trial reports, and 16 stepped wedge group-randomized trial reports. Many of the influential reports were early publications that drew attention to the issues that distinguish these designs from the more familiar individually randomized controlled trial. Others were textbooks that covered a wide range of issues for these designs. Others were "first reports" on analytic methods appropriate for a specific type of data (e.g. binary data, ordinal data), for features commonly encountered in these studies (e.g. unequal cluster size, attrition), or for important variations in study design (e.g. repeated measures, cohort versus cross-section). Many presented methods for sample size calculations. Others described how these designs could be applied to a new area (e.g. dissemination and implementation research). Among the reports with the highest relative citation ratios were the CONSORT statements for each design.

CONCLUSIONS

Collectively, the influential reports address topics of great interest to investigators who might consider using one of these designs and need guidance on selecting the most appropriate design for their research question and on the best methods for design, analysis, and sample size.

Analysis and reporting of stratified cluster randomized trials-a systematic survey

BACKGROUND

In order to correctly assess the effect of intervention from stratified cluster randomized trials (CRTs), it is necessary to adjust for both clustering and stratification, as failure to do so leads to misleading conclusions about the intervention effect. We have conducted a systematic survey to examine the current practices about analysis and reporting of stratified CRTs.

METHOD

We used the search terms to identify the stratified CRTs from MEDLINE since the inception to July 2019. In phase 1, we screened the title and abstract for English-only studies and selected, including the main results paper of the identified protocols, for the next phase. In phase 2, we screened the full text and selected studies for data abstraction. The data abstraction form was piloted and developed using the REDCap. We abstracted data on multiple design and methodological aspects of the study including whether the primary method adjusted for both clustering and stratification, reporting of sample size, randomization, and results.

RESULTS

We screened 2686 studies in the phase 1 and selected 286 studies for phase 2-among them 185 studies were selected for data abstraction. Most of the selected studies were two-arm 140/185 (76%) and parallel-group 165/185 (89%) trials. Among these 185 studies, 27 (15%) of them did not provide any sample size or power calculation, while 105 (57%) studies did not mention any method used for randomization within each stratum. Further, 43 (23%) and 150 (81%) of 185 studies did not provide the definition of all the strata, while more than 60% of the studies did not include all the stratification variable(s) in the flow chart or baseline characteristics table. More than half 114/185 (62%) of the studies did not adjust the primary method for both clustering and stratification.

CONCLUSION

Stratification helps to achieve the balance among intervention groups. However, to correctly assess the intervention effect from stratified CRTs, it is important to adjust the primary analysis for both stratification and clustering. There are significant deficiencies in the reporting of methodological aspects of stratified CRTs, which require substantial improvements in several areas including definition of strata, inclusion of stratification variable(s) in the flow chart or baseline characteristics table, and reporting the stratum-specific number of clusters and individuals in the intervention groups.

Essential Ingredients and Innovations in the Design and Analysis of Group-Randomized Trials

This article reviews the essential ingredients and innovations in the design and analysis of group-randomized trials. The methods literature for these trials has grown steadily since they were introduced to the biomedical research community in the late 1970s, and we summarize those developments. We review, in addition to the group-randomized trial, methods for two closely related designs, the individually randomized group treatment trial and the stepped-wedge group-randomized trial. After describing the essential ingredients for these designs, we review the most important developments in the evolution of their methods using a new bibliometric tool developed at the National Institutes of Health. We then discuss the questions to be considered when selecting from among these designs or selecting the traditional randomized controlled trial. We close with a review of current methods for the analysis of data from these designs, a case study to illustrate each design, and a brief summary.

Vector and reservoir control for preventing leishmaniasis

BACKGROUND

Leishmaniasis is caused by the Leishmania parasite, and transmitted by infected phlebotomine sandflies. Of the two distinct clinical syndromes, cutaneous leishmaniasis (CL) affects the skin and mucous membranes, and visceral leishmaniasis (VL) affects internal organs. Approaches to prevent transmission include vector control by reducing human contact with infected sandflies, and reservoir control, by reducing the number of infected animals.

OBJECTIVES

To assess the effects of vector and reservoir control interventions for cutaneous and for visceral leishmaniasis.

SEARCH METHODS

We searched the following databases to 13 January 2015: Cochrane Infectious Diseases Group Specialized Register, CENTRAL, MEDLINE, EMBASE, LILACS and WHOLIS, Web of Science, and RePORTER. We also searched trials registers for ongoing trials.

SELECTION CRITERIA

Randomized controlled trials (RCTs) evaluating the effects of vector and reservoir control interventions in leishmaniasis-endemic regions.

DATA COLLECTION AND ANALYSIS

Two review authors independently searched for trials and extracted data from included RCTs. We resolved any disagreements by discussion with a third review author. We assessed the quality of the evidence using the GRADE approach.

MAIN RESULTS

We included 14 RCTs that evaluated a range of interventions across different settings. The study methods were generally poorly described, and consequently all included trials were judged to be at high or unclear risk of selection and reporting bias. Only seven trials reported clinical outcome data which limits our ability to make broad generalizations to different epidemiological settings and cultures. Cutaneous leishmaniasisOne four-arm RCT from Afghanistan compared indoor residual spraying (IRS), insecticide-treated bednets (ITNs), and insecticide-treated bedsheets, with no intervention. Over 15 months follow-up, all three insecticide-based interventions had a lower incidence of CL than the control area (IRS: risk ratio (RR) 0.61, 95% confidence interval (CI) 0.38 to 0.97, 2892 participants, moderate quality evidence; ITNs: RR 0.32, 95% CI 0.18 to 0.56, 2954 participants, low quality evidence; ITS: RR 0.34, 95% CI 0.20 to 0.57, 2784 participants, low quality evidence). No difference was detected between the three interventions (low quality evidence). One additional trial of ITNs from Iran was underpowered to show a difference.Insecticide treated curtains were compared with no intervention in one RCT from Venezuela, where there were no CL episodes in the intervention areas over 12 months follow-up compared to 142 in control areas (RR 0.00, 95% CI 0.00 to 0.49, one trial, 2938 participants, low quality evidence).Personal protection using insecticide treated clothing was evaluated by two RCTs in soldiers, but the trials were underpowered to reliably detect effects on the incidence of CL (RR 0.40, 95% CI 0.13 to 1.20, two trials, 558 participants, low quality evidence). Visceral leishmaniasisIn a single RCT of ITNs versus no intervention from India and Nepal, the incidence of VL was low in both groups and no difference was detected (RR 0.99, 95% CI 0.46 to 2.15, one trial, 19,810 participants, moderate quality evidence).Two trials from Brazil evaluated the effects of culling infected dogs compared to no intervention or IRS. Although they report a reduction in seroconversion over 18 months follow-up, they did not measure or report effects on clinical disease.

AUTHORS' CONCLUSIONS

Using insecticides to reduce phlebotomine sandfly numbers may be effective at reducing the incidence of CL, but there is insufficient evidence from trials to know whether it is better to spray the internal walls of houses or to treat bednets, curtains, bedsheets or clothing.

A systematic review of cluster randomised trials in residential facilities for older people suggests how to improve quality

BACKGROUND

Previous reviews of cluster randomised trials have been critical of the quality of the trials reviewed, but none has explored determinants of the quality of these trials in a specific field over an extended period of time. Recent work suggests that correct conduct and reporting of these trials may require more than published guidelines. In this review, our aim was to assess the quality of cluster randomised trials conducted in residential facilities for older people, and to determine whether (1) statistician involvement in the trial and (2) strength of journal endorsement of the Consolidated Standards of Reporting Trials (CONSORT) statement influence quality.

METHODS

We systematically identified trials randomising residential facilities for older people, or parts thereof, without language restrictions, up to the end of 2010, using National Library of Medicine (Medline) via PubMed and hand-searching. We based quality assessment criteria largely on the extended CONSORT statement for cluster randomised trials. We assessed statistician involvement based on statistician co-authorship, and strength of journal endorsement of the CONSORT statement from journal websites.

RESULTS

73 trials met our inclusion criteria. Of these, 20 (27%) reported accounting for clustering in sample size calculations and 54 (74%) in the analyses. In 29 trials (40%), methods used to identify/recruit participants were judged by us to have potentially caused bias or reporting was unclear to reach a conclusion. Some elements of quality improved over time but this appeared not to be related to the publication of the extended CONSORT statement for these trials. Trials with statistician/epidemiologist co-authors were more likely to account for clustering in sample size calculations (unadjusted odds ratio 5.4, 95% confidence interval 1.1 to 26.0) and analyses (unadjusted OR 3.2, 1.2 to 8.5). Journal endorsement of the CONSORT statement was not associated with trial quality.

CONCLUSIONS

Despite international attempts to improve methods in cluster randomised trials, important quality limitations remain amongst these trials in residential facilities. Statistician involvement on trial teams may be more effective in promoting quality than further journal endorsement of the CONSORT statement. Funding bodies and journals should promote statistician involvement and co-authorship in addition to adherence to CONSORT guidelines.

Illustrating Problems Faced by Stroke Researchers: A Review of Cluster-Randomized Controlled Trials

The cluster-randomized controlled trial is a design increasingly used in prevention and health care evaluation studies and is highly relevant to stroke research. However, there are methodological issues that make it complex to implement. These are not always fully appreciated, with reviews continuing to reveal deficiencies. We searched PUBMED and CENTRAL databases to March 31, 2011 for cluster-randomized controlled trials in stroke. Two investigators independently reviewed citations for eligibility and extracted data on key aspects of each trial. Fifteen trials met the eligibility criteria. No trial fully met CONSORT cluster-randomized controlled trial guidelines, although good design and reporting practice were usually present. Twelve trials included the term ‘cluster-randomized’ (or ‘group-randomized’) in the title, and 12 trials stated the intraclass correlation coefficient used to plan the number of clusters and cluster size. However, few provided a clear, evidence-based justification for the choice of intraclass correlation coefficient, and only two-thirds reported the intraclass correlation coefficient for primary outcomes. Several trials appeared underpowered because of problems in determining an appropriate sample size, defining appropriate clusters, and recruiting and retaining clusters and patients. Cluster-randomized controlled trials are difficult to design and perform due to the combination of methodological and practical difficulties. It is important that further improvements are made to reporting cluster-randomized trials and intraclass correlation coefficients should be estimated using a standardized approach and reported consistently; this would be beneficial for stroke researchers when designing future cluster-randomized trials.

Quality of cluster randomized controlled trials in oral health: a systematic review of reports published between 2005 and 2009

OBJECTIVES

To assess the quality of methods and reporting of recently published cluster randomized trials (CRTs) in oral health.

METHODS

We searched PubMed for CRTs that included at least one oral health-related outcome and were published from 2005 to 2009 inclusive. We developed a list of criteria for assessing trial quality and reporting. This was influenced largely by the extended CONSORT statement for CRTs but also included criteria suggested by other authors. We examined the extent to which trials were consistent with these criteria.

RESULTS

Twenty-three trials were included in the review. In 15 (65%) trials, clustering had been accounted for in sample size calculations, and in 18 (78%) authors had accounted for clustering in analysis. Intraclass correlation coefficients (ICCs) were reported for eight (35%) trials; the outcome assessor was reported as having been blinded to allocation in 12 (52%) trials; 17 (74%) described eligibility criteria at individual level, but only nine (39%) described such criteria at cluster level. Sixteen of 20 trials (80%), in which individuals were recruited, reported that individual informed consent was obtained.

CONCLUSIONS

These results suggest that the quality of recent CRTs in oral health is relatively high and appears to compare favourably with other fields. However, there remains room for improvement. Authors of future trials should endeavour to ensure sample size calculations and analyses properly account for clustering (and are reported as such), consider the potential for recruitment/identification bias at the design stage, describe the steps taken to avoid this in the final report and report observed ICCs and cluster-level eligibility criteria.

Impact of CONSORT extension for cluster randomised trials on quality of reporting and study methodology: review of random sample of 300 trials, 2000-8

OBJECTIVE

To assess the impact of the 2004 extension of the CONSORT guidelines on the reporting and methodological quality of cluster randomised trials.

DESIGN

Methodological review of 300 randomly sampled cluster randomised trials. Two reviewers independently abstracted 14 criteria related to quality of reporting and four methodological criteria specific to cluster randomised trials. We compared manuscripts published before CONSORT (2000-4) with those published after CONSORT (2005-8). We also investigated differences by journal impact factor, type of journal, and trial setting.

DATA SOURCES

A validated Medline search strategy. Eligibility criteria for selecting studies Cluster randomised trials published in English language journals, 2000-8.

RESULTS

There were significant improvements in five of 14 reporting criteria: identification as cluster randomised; justification for cluster randomisation; reporting whether outcome assessments were blind; reporting the number of clusters randomised; and reporting the number of clusters lost to follow-up. No significant improvements were found in adherence to methodological criteria. Trials conducted in clinical rather than non-clinical settings and studies published in medical journals with higher impact factor or general medical journals were more likely to adhere to recommended reporting and methodological criteria overall, but there was no evidence that improvements after publication of the CONSORT extension for cluster trials were more likely in trials conducted in clinical settings nor in trials published in either general medical journals or in higher impact factor journals.

CONCLUSION

The quality of reporting of cluster randomised trials improved in only a few aspects since the publication of the extension of CONSORT for cluster randomised trials, and no improvements at all were observed in essential methodological features. Overall, the adherence to reporting and methodological guidelines for cluster randomised trials remains suboptimal, and further efforts are needed to improve both reporting and methodology.

Cluster randomized trials of cancer screening interventions: are appropriate statistical methods being used?

The design and analysis of cluster randomized trials can require more sophistication than individually randomized trials. However, the need for statistical methods that account for the clustered design has not always been appreciated, and past reviews have found widespread deficiencies in methodology and reporting. We reviewed cluster randomized trials of cancer screening interventions published in 1995-2010 to determine whether the use of appropriate statistical methods had increased over time. Literature searches yielded 50 articles reporting outcome analyses of cluster randomized trials of breast, cervix and colorectal cancer screening interventions. Of studies published in 1995-1999, 2000-2002, 2003-2006 and 2007-2010, 55% (6/11), 82% (9/11), 92% (12/13) and 60% (9/15) used appropriate analytic methods, respectively. Results were suggestive of a peak in 2003-2006 (p =.06) followed by a decline in 2007-2010 (p =.08). While the sample of studies was small, these results indicate that many cluster randomized trials of cancer screening interventions have had deficiencies in the application of correct statistical procedures for the outcome analysis, and that increased adoption of appropriate methods in the early and mid-2000's may not have been sustained.