Monitoring performance of sites within multicentre randomised trials: a systematic review of performance metrics

Development of the clinical trial site performance metrics instrument: A study protocol

Clinical trials (CTs) are essential for medical advancements, yet their increasing complexity and cost demand improved efficiency in trial management. One major challenge in multicenter studies is the inconsistency in evaluating site performance. This study aims to develop and validate a Clinical Trial Site Performance Measure (CT-SPM) to assess "good performance" across trials. The tool will be tested and refined through psychometric analysis, resulting in both a comprehensive scale and a short form for universal application. The study is conducted in three phases: Phase 1 involves metric selection through expert consensus; Phase 2 focuses on psychometric testing to evaluate the reliability and validity of the instrument; and Phase 3 defines a cut-off for "good performance" using statistical models. This protocol aims to standardize site performance evaluation, potentially reducing research costs and enhancing trial quality.•The study develops and validates a Clinical Trial Site Performance Measure (CT-SPM) using expert consensus and psychometric testing.•A comprehensive and short-form tool will be created to evaluate site performance in multicenter clinical trials.•A cut-off for "good performance" will be established using statistical models, facilitating consistent and efficient site evaluations.

Protocol of the CONSORT and SPIRIT Extension for multicenter clinical trials

Background

Multicenter clinical trials play an indispensable role for assessing the efficacy of a new intervention or treatment, particularly in Phase II or III studies. Previous studies have shown that these studies often suffer from inadequate reporting of key details related to their design, implementation, and analysis, both in the protocol and final reports. This limitation reduces the practical and scientific value of the findings. Furthermore, the lack of guidance on how to report multicenter features can contribute to poor reporting. Therefore, this study aims to develop guidelines to improve the reporting of multicenter trials, including two Extensions of the CONSORT 2010 and the SPIRIT 2013.

Methods/design

The standard methodology for developing health research reporting guidelines involves the following steps: (i) Identifying the need for development and launching the research project; (ii) Preparing the registration and reviewing the literatures; (iii) Proposing the initial Checklists and conducting the Delphi exercise; (iv) Arranging the consensus meeting and formulating the Checklists; (v) Conducting the pilot test and drafting explanatory documents (E&E); (vi) Seeking comments from advisory group and finalizing the guidelines; and (vii) Developing the publication and dissemination strategies.

Conclusion

By using the CONSORT and SPIRIT checklists as starting points, the development of extensions specific to multicenter trials can help researchers design and report high-quality clinical research. This, in turn, can facilitate the application of study findings in the current evidence-based healthcare system.

Monitoring metrics over time: Why clinical trialists need to systematically collect site performance metrics

Background

Over the last decade, there has been an increasing interest in risk-based monitoring (RBM) in clinical trials, resulting in a number of guidelines from regulators and its inclusion in ICH GCP. However, there is a lack of detail on how to approach RBM from a practical perspective, and insufficient understanding of best practice.

Purpose

We present a method for clinical trials units to track their metrics within clinical trials using descriptive statistics and visualisations.

Research Design

We suggest descriptive statistics and visualisations within a SWAT methodology.

Study Sample

We illustrate this method using the metrics from TEMPER, a monitoring study carried out in three trials at the MRC Clinical Trials Unit at UCL.

Data Collection

The data collection for TEMPER is described in DOI: 10.1177/1740774518793379.

Results

We show the results and discuss a protocol for a Study-Within-A-Trial (SWAT 167) for those wishing to use the method.

Conclusions

The potential benefits metric tracking brings to clinical trials include enhanced assessment of sites for potential corrective action, improved evaluation and contextualisation of the influence of metrics and their thresholds, and the establishment of best practice in RBM. The standardisation of the collection of such monitoring data would benefit both individual trials and the clinical trials community.

The importance of methods for site performance evaluation in REPRIEVE, a longitudinal, global, multicenter trial

BACKGROUND

REPRIEVE, the Randomized Trial to Prevent Vascular Events in HIV, is a multicenter, primary prevention trial evaluating whether a statin can prevent major cardiovascular events in people with HIV. REPRIEVE is conducted at >100 clinical research sites (CRSs) globally. Detailed, comprehensive, and novel methods for evaluating and communicating CRS performance are required to ensure trial integrity and data quality. In this analysis we describe a comprehensive multidimensional methodology for evaluating CRS performance.

METHODS

The REPRIEVE Data Coordinating and Clinical Coordinating Centers developed a robust system for evaluation of and communication with CRSs, designed to identify potential issues and obstacles to performance, provide real-time technical support, and make recommendations for process improvements to facilitate efficient trial execution. We describe these systems and evaluate their impact on participant retention, data management, and specimen management from 2019 to 2022, corresponding to the period from end of recruitment to present. This evaluation was based on pre-defined metrics, regular reviews, and bidirectional communication.

RESULTS

Participant retention, data management, and specimen management all remained steady over the three-year period, although metrics varied by country of enrollment. Targeted messaging relating to certain performance metrics was effective.

CONCLUSION

Site performance is vital to ensure trial integrity and achievement of key trial goals. This analysis demonstrates that utilization of a comprehensive approach allows for a thorough evaluation of CRS performance, facilitates data and specimen management, and enhances participant retention. Our approach may serve as a guidepost for maximizing future large-scale clinical trials' operational success and scientific rigor.

CLINICALTRIALS

gov Identifier: NCT02344290.

A Cross-sectional literature survey showed the reporting quality of multicenter randomized controlled trials should be improved

OBJECTIVE

To assess the reporting quality of randomized controlled trials (RCTs) with multicenter design, particularly whether necessary information related to multicenter characteristics was adequately reported.

STUDY DESIGN AND SETTING

Through a search of 4 international electronic databases, we identified multicenter RCTs published in English from 1975 to 2019. Reporting quality was assessed by the CONSORT (Consolidated Standards of Reporting Trials) checklist (37 items) and by a self-designed multicenter-specific checklist (27 items covering multicenter design, implement and analysis). The scores of trials published in three time periods (1975-1995; 1996-2009; and 2010-2019) were also compared.

RESULTS

A total of 2,844 multicenter RCTs were included. For the CONSORT checklist, the mean (standard deviation) reporting score was 24.1 (5.5), 12 items were assessed as excellent (>90%), 12 items as good (50%-90%), and 13 items as poor (<50%). For the multicenter checklist, the reporting score was 3.9 (2.2), only 3 items were excellent or good, and the remaining 24 items were poor. Time period comparison showed that reporting quality improved over time, especially after the CONSORT 2010 issued.

CONCLUSION

Although CONSORT appears to have enhanced the reporting quality of multicenter RCTs, further improvement is needed. A "CONSORT extension for multicenter trials" should be developed.

Recruitment planning for clinical trials with a vulnerable perinatal adolescent population using the Clinical Trials Transformative Initiative framework and principles of partner and community engagement

Recruitment planning is needed to establish a foundation for obesity prevention research with high risk, disadvantaged perinatal adolescent populations. In the context of developing clinical trial protocols, investigators partnered with Mississippi's Nutrition Program for Women, Infants and Children (WIC) and adopted the Clinical Trials Transformative Initiative (CTTI) framework for recruitment planning to identify and mitigate challenges to recruitment early in the clinical trial development process. The recruitment protocol consisted of 20 passive strategies grounded in principles of partner and community engagement and was flexible, accommodating, altruistic, community-focused, and minimally burdensome to partners and participants. The recruitment goal included 150 adolescent-coparticipant dyads and 145 dyads (96.7%) were successfully recruited. Investigators demonstrated the feasibility of recruiting a disadvantaged and vulnerable perinatal adolescent population that is underrepresented in health research, in one of the most persistently impoverished and poor health regions in the U.S. Four important aspects of recruitment planning using the CTTI framework are discussed including: (1) establishing partnerships with trusted community resources is a paramount investment; (2) dedicating time and resources to know and go to your community is invaluable; (3) fostering trust by offering convenient, continuous and clear communication; and (4) encouraging collaboration and participation through limiting partner and participant burden. Establishing organizational and community partnership requires a substantial amount of invaluable time and fosters recruitment success. Following the CTTI recommendations for recruitment planning led to a robust recruitment protocol that will be used in future intervention trials with an understudied perinatal adolescent population with high risk for poor maternal and fetal health outcomes.

Test evaluation trials present different challenges for trial managers compared to intervention trials

INTRODUCTION

Test evaluation trials present different challenges for trial managers compared to intervention trials. There has been very little research on the management of test evaluation trials and how this impacts on trial success, in comparison with intervention trials. Evaluations of medical tests present specific challenges, because they are a pivot point bridging the complexities of pathways prompting testing with treatment decision-making. We systematically explored key differences in the trial design and management of test evaluation trials compared to intervention trials at the different stages of study design and delivery. We identified challenges in test evaluation trials that were more pronounced than in intervention trials, based on experience from 10 test evaluation trials.

METHODS

We formed a focus group of 7 trial managers and a statistician who had been involved in the day-to-day management of both test evaluation trials and intervention trials. We used discussion and content analysis to group challenges from 10 trials into a structured thematic format. The trials covered a range of medical conditions, diagnostic tests, clinical pathways and conditions including chronic kidney disease, chronic pelvic pain, colitis, detrusor over-activity, group B streptococcal colonisation, tuberculosis and colorectal, lung, ovarian and thyroid cancers.

RESULTS

We identified 10 common themes underlying challenges that are more pronounced in test evaluation compared to intervention trials. We illustrate these themes with examples from 10 trials, including with 31 specific challenges we experienced. The themes were ethics/governance; accessing patient populations; recruitment; patient preference; test processes, clinical pathways and samples storage; uncertainty of diagnostic results; verifying diagnosis (reference standard); follow-up; adverse effects; and diagnostic impact.

CONCLUSION

We present 10 common themes, including 31 challenges, in test evaluation trials that will be helpful to others designing and managing future test evaluation trials. Proactive identification of potential challenges at the design and planning stages of test evaluation trials will enable strategies to improve trial design and management that may be different from standard strategies used for intervention trials. Future work could extend this topic to include challenges for other trial stakeholders including participants, clinicians, statisticians and funders.

TRIAL REGISTRATION

All trials reviewed in this project were registered and are provided in Table 1.

Results of the 2017 inspection campaign of French phase I/II research sites in Île-de-France following the BIA 10-2474 accident: Medical vs. regulatory relevance

AIMS

Following the serious adverse events that occurred in January 2016 during the BIA 10-2474 First-in-Human study, the French Ministry of Health asked the Regional Health Agencies to inspect operations at all research sites conducting phase I/II clinical trials of experimental drugs. The aim of this study was to assess the medical relevance of the inspections made in Île-de-France (Paris region) in 2017.

METHODS

All 30 sites of Île-de-France region fully authorized to perform phase I/II trials were inspected by a public health physician and a public health pharmacist. Their reported list of observations was submitted to three physicians with longstanding experience of early pharmacology studies performed in academic or private research facilities. These physicians were asked to adjudicate each observation according to their perceived medical importance regarding safety. Adjudications were first performed separately and disagreements were later settled during a final adjudication meeting.

RESULTS

At least one disagreement occurred initially among the 3 adjudicators for 84 of the 120 observations (70%) reported by the inspectors. Following reconciliation, the 3 physicians agreed that 20% of the observations were likely to have potentially serious medical consequences. These observations mainly concerned the management of emergencies and of serious adverse events and the continuity of care.

CONCLUSIONS

Maintenance of on-site inspections periodically carried out by regulatory authorities granting authorisations to perform phase I/II trials are justified. However, the medical relevance of these inspections can be improved with more emphasis on factors affecting the safety of research participants than on administrative or purely regulatory issues.

Development of a standardised set of metrics for monitoring site performance in multicentre randomised trials: a Delphi study

BACKGROUND

Site performance is key to the success of large multicentre randomised trials. A standardised set of clear and accessible summaries of site performance could facilitate the timely identification and resolution of potential problems, minimising their impact. The aim of this study was to identify and agree a core set of key performance metrics for managing multicentre randomised trials.

METHODS

We used a mixed methods approach to identify potential metrics and to achieve consensus about the final set, adapting methods that are recommended by the COMET Initiative for developing core outcome sets in health care. We used performance metrics identified from our systematic search and focus groups to create an online Delphi survey. We invited respondents to score each metric for inclusion in the final core set, over three survey rounds. Metrics scored as critical by ≥70% and unimportant by <15% of respondents were taken forward to a consensus meeting of representatives from key UK-based stakeholders. Participants in the consensus meeting discussed and voted on each metric, using anonymous electronic voting. Metrics with >50% of participants voting for inclusion were retained.

RESULTS

Round 1 of the Delphi survey presented 28 performance metrics, and a further six were added in round 2. Of 294 UK-based stakeholders who registered for the Delphi survey, 211 completed all three rounds. At the consensus meeting, 17 metrics were discussed and voted on: 15 metrics were retained following survey round 3, plus two others that were preferred by consensus meeting participants. Consensus was reached on a final core set of eight performance metrics in three domains: (1) recruitment and retention, (2) data quality and (3) protocol compliance. A simple tool for visual reporting of the metrics is available from the Nottingham Clinical Trials Unit website.

CONCLUSIONS

We have established a core set of metrics for measuring the performance of sites in multicentre randomised trials. These metrics could improve trial conduct by enabling researchers to identify and address problems before trials are adversely affected. Future work could evaluate the effectiveness of using the metrics and reporting tool.