Centre selection for clinical trials and the generalisability of results: a mixed methods study

Assessing the representativeness of cluster randomized trials: Evidence from two large pragmatic trials in United States nursing homes

BACKGROUND/AIMS

When the randomized clusters in a cluster randomized trial are selected based on characteristics that influence treatment effectiveness, results from the trial may not be directly applicable to the target population. We used data from two large nursing home-based pragmatic cluster randomized trials to compare nursing home and resident characteristics in randomized facilities to eligible non-randomized and ineligible facilities.

METHODS

We linked data from the high-dose influenza vaccine trial and the Music & Memory Pragmatic TRIal for Nursing Home Residents with ALzheimer's Disease (METRICaL) to nursing home assessments and Medicare fee-for-service claims. The target population for the high-dose trial comprised Medicare-certified nursing homes; the target population for the METRICaL trial comprised nursing homes in one of four US-based nursing home chains. We used standardized mean differences to compare facility and individual characteristics across the three groups and logistic regression to model the probability of nursing home trial participation.

RESULTS

In the high-dose trial, 4476 (29%) of the 15,502 nursing homes in the target population were eligible for the trial, of which 818 (18%) were randomized. Of the 1,361,122 residents, 91,179 (6.7%) were residents of randomized facilities, 463,703 (34.0%) of eligible non-randomized facilities, and 806,205 (59.3%) of ineligible facilities. In the METRICaL trial, 160 (59%) of the 270 nursing homes in the target population were eligible for the trial, of which 80 (50%) were randomized. Of the 20,262 residents, 973 (34.4%) were residents of randomized facilities, 7431 (36.7%) of eligible non-randomized facilities, and 5858 (28.9%) of ineligible facilities. In the high-dose trial, randomized facilities differed from eligible non-randomized and ineligible facilities by the number of beds (132.5 vs 145.9 and 91.9, respectively), for-profit status (91.8% vs 66.8% and 68.8%), belonging to a nursing home chain (85.8% vs 49.9% and 54.7%), and presence of a special care unit (19.8% vs 25.9% and 14.4%). In the METRICaL trial randomized facilities differed from eligible non-randomized and ineligible facilities by the number of beds (103.7 vs 110.5 and 67.0), resource-poor status (4.6% vs 10.0% and 18.8%), and presence of a special care unit (26.3% vs 33.8% and 10.9%). In both trials, the characteristics of residents in randomized facilities were similar across the three groups.

CONCLUSION

In both trials, facility-level characteristics of randomized nursing homes differed considerably from those of eligible non-randomized and ineligible facilities, while there was little difference in resident-level characteristics across the three groups. Investigators should assess the characteristics of clusters that participate in cluster randomized trials, not just the individuals within the clusters, when examining the applicability of trial results beyond participating clusters.

Costs of fall injuries in the STRIDE study: an economic evaluation of healthcare system heterogeneity and heterogeneity of treatment effect

OBJECTIVES

The Strategies to Reduce Injuries and Develop Confidence in Elders (STRIDE) Study cluster-randomized 86 primary care practices in 10 healthcare systems to a patient-centered multifactorial fall injury prevention intervention or enhanced usual care, enrolling 5451 participants. We estimated total healthcare costs from participant-reported fall injuries receiving medical attention (FIMA) that were averted by the STRIDE intervention and tested for healthcare-system-level heterogeneity and heterogeneity of treatment effect (HTE).

METHODS

Participants were community-dwelling adults age ≥ 70 at increased fall injury risk. We estimated practice-level total costs per person-year of follow-up (PYF), assigning unit costs to FIMA with and without an overnight hospital stay. Using independent variables for treatment arm, healthcare system, and their interaction, we fit a generalized linear model with log link, log follow-up time offset, and Tweedie error distribution.

RESULTS

Unadjusted total costs per PYF were $2,034 (intervention) and $2,289 (control). The adjusted (intervention minus control) cost difference per PYF was -$167 (95% confidence interval (CI), -$491, $216). Cost heterogeneity by healthcare system was present (p = 0.035), as well as HTE (p = 0.090). Adjusted total costs per PYF in control practices varied from $1,529 to $3,684 for individual healthcare systems; one system with mean intervention minus control costs of -$2092 (95% CI, -$3,686 to -$944) per PYF accounted for HTE, but not healthcare system cost heterogeneity.

CONCLUSIONS

We observed substantial heterogeneity of healthcare system costs in the STRIDE study, with small reductions in healthcare costs for FIMA in the STRIDE intervention accounted for by a single healthcare system.

TRIAL REGISTRATION

Clinicaltrials.gov (NCT02475850).

Improving the external validity of Antenatal Late Preterm Steroids trial findings

BACKGROUND

The external validity of randomised trials can be compromised when trial participants differ from real-world populations. In the Antenatal Late Preterm Steroids (ALPS) trial of antenatal corticosteroids at late preterm ages, participants had systematically younger gestational ages than those outside the trial setting. As risk of respiratory morbidity (the primary trial outcome) is higher at younger gestations, absolute benefits of corticosteroids calculated in the trial population may overestimate real-world treatment benefits.

OBJECTIVES

To estimate the real-world absolute risk reduction and number-needed-to-treat (NNT) for antenatal corticosteroids at late preterm ages, accounting for gestational age differences between the ALPS and real-world populations.

METHODS

Individual participant data from the ALPS trial (which recruited 2831 women with imminent preterm birth at 34+0 to 36+5 weeks') was appended to population-based data for 15,741 women admitted for delivery between 34+0 and 36+5 weeks' from British Columbia, Canada, 2000-2013. We used logistic regression to calculate inverse odds of sampling weights for each trial participant and re-estimated treatment effects of corticosteroids on neonatal respiratory morbidity in ALPS participants, weighted to reflect the gestational age distribution of the population-based (real-world) sample.

RESULTS

The real-world absolute risk reduction was estimated to be -2.2 (95% CI -4.6, 0.0) cases of respiratory morbidity per 100, compared with -2.8 (95% CI -5.3, -0.3) in original trial data. Corresponding NNTs were 46 in the real-world setting vs 35 in the trial. Our focus on absolute measures also highlighted that the benefits of antenatal corticosteroids may be meaningfully greater at 34 weeks vs. 36 weeks (e.g., risk reductions of -3.7 vs. -1.2 per 100 respectively).

CONCLUSIONS

The absolute risk reductions and NNTs associated with antenatal corticosteroid administration at late preterm ages estimated in our study may be more appropriate for patient counselling as they better reflect the anticipated benefits of treatment when used in a real-world situation.

Shifting research culture to address the mismatch between where trials recruit and where populations with the most disease live: a qualitative study

Background

Research participation is beneficial to patients, clinicians and healthcare services. There is currently poor alignment between UK clinical research activity and local prevalence of disease. The National Institute of Health Research is keen to encourage chief investigators (CIs) to base their research activity in areas of high patient need, to support equity, efficiency and capacity building. We explored how CIs choose sites for their trials and suggest ways to encourage them to recruit from areas with the heaviest burden of disease.

Methods

Qualitative, semi-structured telephone interviews with a purposive sample of 30 CIs of ongoing or recently completed multi-centre trials, all of which were funded by the UK National Institute of Health Research.

Results

CIs want to deliver world-class trials to time and budget. Approaching newer, less research-active sites appears risky, potentially compromising trial success. CIs fear that funders may close the trial if recruitment (or retention) is low, with potential damage to their research reputation.

We consider what might support a shift in CI behaviour. The availability of ‘heat maps’ showing the disparity between disease prevalence and current research activity will help to inform site selection. Embedded qualitative research during trial set up and early, appropriate patient and public involvement and engagement can provide useful insights for a more nuanced and inclusive approach to recruitment. Public sector funders could request more granularity in recruitment reports and incentivise research activity in areas of greater patient need. Accounts from the few CIs who had ‘broken the mould’ suggest that nurturing new sites can be very successful in terms of efficient recruitment and retention.

Conclusion

While improvements in equity and capacity building certainly matter to CIs, most are primarily motivated by their commitment to delivering successful trials. Highlighting the benefits to trial delivery is therefore likely to be the best way to encourage CIs to focus their research activity in areas of greatest need.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01268-z.

Using datasets to ascertain the generalisability of clinical cohorts: the example of European QUALity Study on the treatment of advanced chronic kidney disease (EQUAL)

BACKGROUND

Cohort studies are among the most robust of observational studies but have issues with external validity. This study assesses threats to external validity (generalisability) in the European QUALity (EQUAL) study, a cohort study of people over 65 years with stage 4/5 chronic kidney disease.

METHODS

Patients meeting the EQUAL inclusion criteria were identified in The Health Improvement Network database and stratified into those attending renal units (secondary care cohort-SCC) and not (primary care cohort-PCC). Survival, progression to renal replacement therapy (RRT), and hospitalisation were compared.

RESULTS

The analysis included 250, 633, and 2,464 patients in EQUAL, PCC, and SCC. EQUAL had a higher proportion of men in comparison to PCC and SCC (60.0% vs. 34.8% vs. 51.4%). Increasing age (≥85 years odds ratio (OR) 0.25 (95% confidence interval (CI) 0.15-0.40)) and comorbidity (Charlson Comorbidity Index ≥ 4 OR 0.69 (CI 0.52-0.91)) were associated with non-participation in EQUAL. EQUAL had a higher proportion of patients starting RRT at 1 year compared to SCC (8.1% vs. 2.1%%, p < 0.001). Patients in the PCC and SCC had increased risk of Hospitalisation (incidence rate ratio=1.76 (95% CI 1.27-2.47) & 2.13 (95% CI 1.59-2.86)) and mortality at one year (hazard ratio=3.48 (95% CI 2.1-5.7) & 1.7 (95% CI 1.1-2.7)) compared to EQUAL.

CONCLUSIONS

This study provides evidence of how participants in a cohort study can differ from the broader population of patients, which is essential when considering external validity and applying to local practice.

Re-inventing the randomized controlled trial in medical oncology: The registry-based trial

Substantial progress has recently been made in optimizing the management of cancer patients, resulting in major gains in survival and quality of life. Much of this progress has resulted from the serial testing of promising treatment strategies, typically using prospective randomized controlled trials to compare outcomes achieved with the new approach versus the current standard(s) of care. However, there is an ever-expanding list of important questions that are difficult to investigate, particularly with respect to determining the optimal sequencing and combination of proven active agents. With the rapidly growing list of clinical, pathologic and molecular characteristics that promise to predict treatment benefit and/or risk for defined patient subsets, many new questions regarding how best to personalize our approach to treatment selection are emerging. These questions can be investigated in the context of registry-based randomized clinical trials. Recently, the potential of registry-based randomized clinical trials was demonstrated in cardiology, highlighting the ability to rapidly recruit large numbers of patients to a trial addressing an important clinical question, with minimal cost and high external validity. In this review, we discuss the challenges and limitations of conventional clinical trials in multidisciplinary cancer care, describe the potential advantages of registry-based randomized trials, and highlight several registry-based oncology studies that are already underway to demonstrate the feasibility of this approach.

Intervention studies to foster resilience - A systematic review and proposal for a resilience framework in future intervention studies

Psychological resilience refers to the phenomenon that many people are able to adapt to the challenges of life and maintain mental health despite exposure to adversity. This has stimulated research on training programs to foster psychological resilience. We evaluated concepts, methods and designs of 43 randomized controlled trials published between 1979 and 2014 which assessed the efficacy of such training programs and propose standards for future intervention research based on recent developments in the field. We found that concepts, methods and designs in current resilience intervention studies are of limited use to properly assess efficacy of interventions to foster resilience. Major problems are the use of definitions of resilience as trait or a composite of resilience factors, the use of unsuited assessment instruments, and inappropriate study designs. To overcome these challenges, we propose 1) an outcome-oriented definition of resilience, 2) an outcome-oriented assessment of resilience as change in mental health in relation to stressor load, and 3) methodological standards for suitable study designs of future intervention studies. Our proposals may contribute to an improved quality of resilience intervention studies and may stimulate further progress in this growing research field.

Series: Pragmatic trials and real world evidence: Paper 2. Setting, sites, and investigator selection

This second article in the series on pragmatic trials describes the challenges in selection of sites for pragmatic clinical trials and the impact on validity, precision, and generalizability of the results. The selection of sites is an important factor for the successful execution of a pragmatic trial and impacts the extent to which the results are applicable to future patients in clinical practice. The first step is to define usual care and understand the heterogeneity of sites, patient demographics, disease prevalence and country choice. Next, specific site characteristics are important to consider such as interest in the objectives of the trial, the level of research experience, availability of resources, and the expected number of eligible patients. It can be advisable to support the sites with implementing the trial-related activities and minimize the additional burden that the research imposes on routine clinical practice. Health care providers should be involved in an early phase of protocol development to generate engagement and ensure an appropriate selection of sites with patients who are representative of the future drug users.

Cluster Sampling Bias in Government-Sponsored Evaluations: A Correlational Study of Employment and Welfare Pilots in England

For pilot or experimental employment programme results to apply beyond their test bed, researchers must select ‘clusters’ (i.e. the job centres delivering the new intervention) that are reasonably representative of the whole territory. More specifically, this requirement must account for conditions that could artificially inflate the effect of a programme, such as the fluidity of the local labour market or the performance of the local job centre. Failure to achieve representativeness results in Cluster Sampling Bias (CSB). This paper makes three contributions to the literature. Theoretically, it approaches the notion of CSB as a human behaviour. It offers a comprehensive theory, whereby researchers with limited resources and conflicting priorities tend to oversample ‘effect-enhancing’ clusters when piloting a new intervention. Methodologically, it advocates for a ‘narrow and deep’ scope, as opposed to the ‘wide and shallow’ scope, which has prevailed so far. The PILOT-2 dataset was developed to test this idea. Empirically, it provides evidence on the prevalence of CSB. In conditions similar to the PILOT-2 case study, investigators (1) do not sample clusters with a view to maximise generalisability; (2) do not oversample ‘effect-enhancing’ clusters; (3) consistently oversample some clusters, including those with higher-than-average client caseloads; and (4) report their sampling decisions in an inconsistent and generally poor manner. In conclusion, although CSB is prevalent, it is still unclear whether it is intentional and meant to mislead stakeholders about the expected effect of the intervention or due to higher-level constraints or other considerations.

LEADER 7: cardiovascular risk profiles of US and European participants in the LEADER diabetes trial differ

AIMS

To determine whether US and European participants in the Liraglutide Effect and Action in Diabetes: Evaluation of cardiovascular outcome Results (LEADER) trial differ regarding risk factors for cardiovascular mortality and morbidity.

METHODS

Baseline data, stratified for prior cardiovascular disease (CVD), were compared using multivariable logistic regression analysis to establish whether region is an independent determinant of achieved targets for glycated hemoglobin (HbA1c), blood pressure (BP), and low-density lipoprotein (LDL)-cholesterol.

RESULTS

Independent of CVD history, US participants were more often of non-White origin and had a longer history of type 2 diabetes, higher body weight, and higher baseline HbA1c. They had substantially lower systolic and diastolic BP, and a marginally lower LDL-cholesterol level. Fewer US participants were diagnosed with left ventricular dysfunction. In the largest group of patients, those with prior CVD and the highest cardiovascular risk, US participants were more often female, had a higher waist circumference, and had a decreased estimated glomerular filtration rate, but less frequently prior myocardial infarction or angina pectoris.

CONCLUSIONS

There were baseline differences between US and European participants. These differences may result from variation in regional targets for cardiovascular risk factor management, and should be considered in the analysis and reporting of the trial results. Clinical trial identifier: ClinicalTrials.gov, NCT01179048.

Evaluating the Generalisability of Trial Results: Introducing a Centre- and Trial-Level Generalisability Index

BACKGROUND

Few randomised controlled trials (RCTs) recruit centres representatively, which may limit the external validity of trial results.

OBJECTIVE

The aim of this study was to propose a proof-of-concept method of assessing the generalisability of the clinical and cost-effectiveness findings of a given RCT.

METHODS

We developed a generalisability index (Gix), informed by centre-level characteristics, as a measure of centre and trial representativeness. The centre-level Gix quantifies how representative a centre is in relation to its jurisdiction, e.g. a country or health authority. The trial-level Gix quantifies how representative trial recruitment is in relation to clinical practice in the jurisdiction. Taking a real-world RCT as a case study and assuming trial-wide results to represent 'true jurisdiction values', we used simulation methods to recreate 5000 RCTs and investigate the relationship between trial representativeness, reflected by the standardised trial-Gix, and the deviation of simulated trial results from the 'true values'.

RESULTS

The simulation study provides evidence that trial results (odds ratio for the primary outcome and incremental quality-adjusted life-years) were influenced by the representativeness of the sample of recruiting centres. Simulated RCTs with the closest results to the 'true values' were those whose recruitment closely mirrored the jurisdiction-wide context. Results appeared robust to six alternative specifications of the Gix.

CONCLUSIONS

Our findings suggest that an unrepresentative selection of centres limits the external validity of trial results. The Gix may be a valuable tool to help facilitate rational selection of trial centres and ensure the generalisability of results at the jurisdiction level.

An Approach to Assess Generalizability in Comparative Effectiveness Research: A Case Study of the Whole Systems Demonstrator Cluster Randomized Trial Comparing Telehealth with Usual Care for Patients with Chronic Health Conditions

BACKGROUND

Policy makers require estimates of comparative effectiveness that apply to the population of interest, but there has been little research on quantitative approaches to assess and extend the generalizability of randomized controlled trial (RCT)-based evaluations. We illustrate an approach using observational data.

METHODS

Our example is the Whole Systems Demonstrator (WSD) trial, in which 3230 adults with chronic conditions were assigned to receive telehealth or usual care. First, we used novel placebo tests to assess whether outcomes were similar between the RCT control group and a matched subset of nonparticipants who received usual care. We matched on 65 baseline variables obtained from the electronic medical record. Second, we conducted sensitivity analysis to consider whether the estimates of treatment effectiveness were robust to alternative assumptions about whether "usual care" is defined by the RCT control group or nonparticipants. Thus, we provided alternative estimates of comparative effectiveness by contrasting the outcomes of the RCT telehealth group and matched nonparticipants.

RESULTS

For some endpoints, such as the number of outpatient attendances, the placebo tests passed, and the effectiveness estimates were robust to the choice of comparison group. However, for other endpoints, such as emergency admissions, the placebo tests failed and the estimates of treatment effect differed markedly according to whether telehealth patients were compared with RCT controls or matched nonparticipants.

CONCLUSIONS

The proposed placebo tests indicate those cases when estimates from RCTs do not generalize to routine clinical practice and motivate complementary estimates of comparative effectiveness that use observational data. Future RCTs are recommended to incorporate these placebo tests and the accompanying sensitivity analyses to enhance their relevance to policy making.

Economic Evaluation alongside Multinational Studies: A Systematic Review of Empirical Studies

Purpose of the study

This study seeks to explore methods for conducting economic evaluations alongside multinational trials by conducting a systematic review of the methods used in practice and the challenges that are typically faced by the researchers who conducted the economic evaluations.

Methods

A review was conducted for the period 2002 to 2012, with potentially relevant articles identified by searching the Medline, Embase and NHS EED databases. Studies were included if they were full economic evaluations conducted alongside a multinational trial.

Results

A total of 44 studies out of a possible 2667 met the inclusion criteria. Methods used for the analyses varied between studies, indicating a lack of consensus on how economic evaluation alongside multinational studies should be carried out. The most common challenge appeared to be related to addressing differences between countries, which potentially hinders the generalisability and transferability of results. Other challenges reported included inadequate sample sizes and choosing cost-effectiveness thresholds.

Conclusions

It is recommended that additional guidelines be developed to aid researchers in this area and that these be based on an understanding of the challenges associated with multinational trials and the strengths and limitations of alternative approaches. Guidelines should focus on ensuring that results will aid decision makers in their individual countries.

Rational centre selection for RCTs with a parallel economic evaluation--the next step towards increased generalisability?

The paper discusses the impact of centre selection on the generalisability of randomised controlled trial (RCT)-based economic evaluations and suggests a future research agenda. The first section briefly reviews the current methods for addressing generalisability. We argue that these methods make no verifiable assumptions about how representative the recruiting centres are to the population of centres in the jurisdiction. The second section uses data from a multicentre RCT to illustrate that cost-effectiveness estimates can be influenced by the sample of recruiting centres. Finally, we propose two concepts that may advance generalisability research. First, we distinguish between the 'research space' and the 'policy space' and argue that policy makers are interested in the latter, while current methods describe the former. Second, we propose a centre-specific generalisability index used at RCT design stage to address generalisability. We conclude that future research should focus on generalisability at RCT design stage rather than on post hoc analyses.

The need for increased harmonisation of clinical trials and economic evaluations

Despite the increasing number of protocol and reporting guidelines available to trialists, there is still little guidance for protocol writers on the incorporation of patient-reported outcomes and economic assessments alongside clinical trials. It is unsurprising, therefore, that trial protocols present disproportionately less information for the economic evaluation component than for clinical outcomes. Costing methodologies, generalisability considerations, methods to address sensitive patient-reported outcome information and missing data are often insufficiently described in the trial protocol. The paper illustrates these shortcomings with specific examples and makes a case for shifting researchers' attention from the reporting to the design stage of trial-based economic evaluation to promote the validity, generalisability and accountability of trial-based economic evaluations.