Managing clustering effects and learning effects in the design and analysis of randomised surgical trials: a review of existing guidance

BACKGROUND

The complexities associated with delivering randomised surgical trials, such as clustering effects, by centre or surgeon, and surgical learning, are well known. Despite this, approaches used to manage these complexities, and opinions on these, vary. Guidance documents have been developed to support clinical trial design and reporting. This work aimed to identify and examine existing guidance and consider its relevance to clustering effects and learning curves within surgical trials.

METHODS

A review of existing guidelines, developed to inform the design and analysis of randomised controlled trials, is undertaken. Guidelines were identified using an electronic search, within the Equator Network, and by a targeted search of those endorsed by leading UK funding bodies, regulators, and medical journals. Eligible documents were compared against pre-specified key criteria to identify gaps or inconsistencies in recommendations.

RESULTS

Twenty-eight documents were eligible (12 Equator Network; 16 targeted search). Twice the number of guidance documents targeted design (n/N=20/28, 71%) than analysis (n/N=10/28, 36%). Managing clustering by centre through design was well documented. Clustering by surgeon had less coverage and contained some inconsistencies. Managing the surgical learning curve, or changes in delivery over time, through design was contained within several documents (n/N=8/28, 29%), of which one provided guidance on reporting this and restricted to early phase studies only. Methods to analyse clustering effects and learning were provided in five and four documents respectively (N=28).

CONCLUSIONS

To our knowledge, this is the first review as to the extent to which existing guidance for designing and analysing randomised surgical trials covers the management of clustering, by centre or surgeon, and the surgical learning curve. Twice the number of identified documents targeted design aspects than analysis. Most notably, no single document exists for use when designing these studies, which may lead to inconsistencies in practice. The development of a single document, with agreed principles to guide trial design and analysis across a range of realistic clinical scenarios, is needed.

A randomised controlled trial comparing palate surgery at 6 months versus 12 months of age (the TOPS trial): a statistical analysis plan

Background

Cleft palate is among the most common birth abnormalities. The success of primary surgery in the early months of life is crucial for successful feeding, hearing, dental development, and facial growth. Over recent decades, age at palatal surgery in infancy has reduced. The Timing Of Primary Surgery for cleft palate (TOPS) trial aims to determine whether, in infants with cleft palate, it is better to perform primary surgery at age 6 or 12 months (corrected for gestational age).

Methods/design

The TOPS trial is an international, two-arm, parallel group, randomised controlled trial. The primary outcome is insufficient velopharyngeal function at 5 years of age. Secondary outcomes, measured at 12 months, 3 years, and 5 years of age, include measures of speech development, safety of the procedure, hearing level, middle ear function, dentofacial development, and growth. The analysis approaches for primary and secondary outcomes are described here, as are the descriptive statistics which will be reported. The TOPS protocol has been published previously.

Discussion

This paper provides details of the planned statistical analyses for the TOPS trial and will reduce the risk of outcome reporting bias and data-driven results.

Trial registration

ClinicalTrials.gov NCT00993551. Registered on 9 October 2009.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13063-020-04886-y.

Managing clustering effects and learning effects in the design and analysis of multicentre randomised trials: a survey to establish current practice

Background

Patient outcomes can depend on the treating centre, or health professional, delivering the intervention. A health professional’s skill in delivery improves with experience, meaning that outcomes may be associated with learning. Considering differences in intervention delivery at trial design will ensure that any appropriate adjustments can be made during analysis. This work aimed to establish practice for the allowance of clustering and learning effects in the design and analysis of randomised multicentre trials.

Methods

A survey that drew upon quotes from existing guidelines, references to relevant publications and example trial scenarios was delivered. Registered UK Clinical Research Collaboration Registered Clinical Trials Units were invited to participate.

Results

Forty-four Units participated (N = 50). Clustering was managed through design by stratification, more commonly by centre than by treatment provider. Managing learning by design through defining a minimum expertise level for treatment provider was common (89%). One-third reported experience in expertise-based designs. The majority of Units had adjusted for clustering during analysis, although approaches varied. Analysis of learning was rarely performed for the main analysis (n = 1), although it was explored by other means. The insight behind the approaches used within and reasons for, or against, alternative approaches were provided.

Conclusions

Widespread awareness of challenges in designing and analysing multicentre trials is identified. Approaches used, and opinions on these, vary both across and within Units, indicating that approaches are dependent on the type of trial. Agreeing principles to guide trial design and analysis across a range of realistic clinical scenarios should be considered.

A framework for the design, conduct and interpretation of randomised controlled trials in the presence of treatment changes

Background

When a randomised trial is subject to deviations from randomised treatment, analysis according to intention-to-treat does not estimate two important quantities: relative treatment efficacy and effectiveness in a setting different from that in the trial. Even in trials of a predominantly pragmatic nature, there may be numerous reasons to consider the extent, and impact on analysis, of such deviations from protocol. Simple methods such as per-protocol or as-treated analyses, which exclude or censor patients on the basis of their adherence, usually introduce selection and confounding biases. However, there exist appropriate causal estimation methods which seek to overcome these inherent biases, but these methods remain relatively unfamiliar and are rarely implemented in trials.

Methods

This paper demonstrates when it may be of interest to look beyond intention-to-treat analysis for answers to alternative causal research questions through illustrative case studies. We seek to guide trialists on how to handle treatment changes in the design, conduct and planning the analysis of a trial; these changes may be planned or unplanned, and may or may not be permitted in the protocol. We highlight issues that must be considered at the trial planning stage relating to: the definition of nonadherence and the causal research question of interest, trial design, data collection, monitoring, statistical analysis and sample size.

Results and conclusions

During trial planning, trialists should define their causal research questions of interest, anticipate the likely extent of treatment changes and use these to inform trial design, including the extent of data collection and data monitoring. A series of concise recommendations is presented to guide trialists when considering undertaking causal analyses.

How individual participant data meta-analyses have influenced trial design, conduct, and analysis

Objectives

To demonstrate how individual participant data (IPD) meta-analyses have impacted directly on the design and conduct of trials and highlight other advantages IPD might offer.

Study Design and Setting

Potential examples of the impact of IPD meta-analyses on trials were identified at an international workshop, attended by individuals with experience in the conduct of IPD meta-analyses and knowledge of trials in their respective clinical areas. Experts in the field who did not attend were asked to provide any further examples. We then examined relevant trial protocols, publications, and Web sites to verify the impacts of the IPD meta-analyses. A subgroup of workshop attendees sought further examples and identified other aspects of trial design and conduct that may inform IPD meta-analyses.

Results

We identified 52 examples of IPD meta-analyses thought to have had a direct impact on the design or conduct of trials. After screening relevant trial protocols and publications, we identified 28 instances where IPD meta-analyses had clearly impacted on trials. They have influenced the selection of comparators and participants, sample size calculations, analysis and interpretation of subsequent trials, and the conduct and analysis of ongoing trials, sometimes in ways that would not possible with systematic reviews of aggregate data. We identified additional potential ways that IPD meta-analyses could be used to influence trials.

Conclusions

IPD meta-analysis could be better used to inform the design, conduct, analysis, and interpretation of trials.