Triggered or routine site monitoring visits for randomised controlled trials: results of TEMPER, a prospective, matched-pair study

Approaches and experiences implementing remote, electronic consent at the Leeds Clinical Trials Research Unit

BACKGROUND

Use of electronic methods to support informed consent ('eConsent') is increasingly popular in clinical research. This commentary reports the approach taken to implement electronic consent methods and subsequent experiences from a range of studies at the Leeds Clinical Trials Research Unit (CTRU), a large clinical trials unit in the UK.

MAIN TEXT

We implemented a remote eConsent process using the REDCap platform. The process can be used in trials of investigational medicinal products and other intervention types or research designs. Our standard eConsent system focuses on documenting informed consent, with other aspects of consent (e.g. providing information to potential participants and a recruiter discussing the study with each potential participant) occurring outside the system, though trial teams can use electronic methods for these activities where they have ethical approval. Our overall process includes a verbal consent step prior to confidential information being entered onto REDCap and an identity verification step in line with regulator guidance. We considered the regulatory requirements around the system's generation of source documents, how to ensure data protection standards were upheld and how to monitor informed consent within the system. We present four eConsent case studies from the CTRU: two randomised clinical trials and two other health research studies. These illustrate the ways eConsent can be implemented, and lessons learned, including about differences in uptake.

CONCLUSIONS

We successfully implemented a remote eConsent process at the CTRU across multiple studies. Our case studies highlight benefits of study participants being able to give consent without having to be present at the study site. This may better align with patient preferences and trial site needs and therefore improve recruitment and resilience against external shocks (such as pandemics). Variation in uptake of eConsent may be influenced more by site-level factors than patient preferences, which may not align well with the aspiration towards patient-centred research. Our current process has some limitations, including the provision of all consent-related text in more than one language, and scalability of implementing more than one consent form version at a time. We consider how enhancements in CTRU processes, or external developments, might affect our approach.

Monitoring metrics over time: Why clinical trialists need to systematically collect site performance metrics

Background

Over the last decade, there has been an increasing interest in risk-based monitoring (RBM) in clinical trials, resulting in a number of guidelines from regulators and its inclusion in ICH GCP. However, there is a lack of detail on how to approach RBM from a practical perspective, and insufficient understanding of best practice.

Purpose

We present a method for clinical trials units to track their metrics within clinical trials using descriptive statistics and visualisations.

Research Design

We suggest descriptive statistics and visualisations within a SWAT methodology.

Study Sample

We illustrate this method using the metrics from TEMPER, a monitoring study carried out in three trials at the MRC Clinical Trials Unit at UCL.

Data Collection

The data collection for TEMPER is described in DOI: 10.1177/1740774518793379.

Results

We show the results and discuss a protocol for a Study-Within-A-Trial (SWAT 167) for those wishing to use the method.

Conclusions

The potential benefits metric tracking brings to clinical trials include enhanced assessment of sites for potential corrective action, improved evaluation and contextualisation of the influence of metrics and their thresholds, and the establishment of best practice in RBM. The standardisation of the collection of such monitoring data would benefit both individual trials and the clinical trials community.

Lack of transparent reporting of trial monitoring approaches in randomised controlled trials: A systematic review of contemporary protocol papers

BACKGROUND

Monitoring is essential to ensure patient safety and data integrity in clinical trials as per Good Clinical Practice. The Standard Protocol Items: Recommendations for Interventional Trials Statement and its checklist guides authors to include monitoring in their protocols. We investigated how well monitoring was reported in published 'protocol papers' for contemporary randomised controlled trials.

METHODS

A systematic search was conducted in PubMed to identify eligible protocol papers published in selected journals between 1 January 2020 and 31 May 2020. Protocol papers were classified by whether they reported monitoring and, if so, by the details of monitoring. Data were summarised descriptively.

RESULTS

Of 811 protocol papers for randomised controlled trials, 386 (48%; 95% CI: 44%-51%) explicitly reported some monitoring information. Of these, 20% (77/386) reported monitoring information consistent with an on-site monitoring approach, and 39% (152/386) with central monitoring, 26% (101/386) with a mixed approach, while 14% (54/386) did not provide sufficient information to specify an approach. Only 8% (30/386) of randomised controlled trials reported complete details about all of scope, frequency and organisation of monitoring; frequency of monitoring was the least reported. However, 6% (25/386) of papers used the term 'audit' to describe 'monitoring'.

DISCUSSION

Monitoring information was reported in only approximately half of the protocol papers. Suboptimal reporting of monitoring hinders the clinical community from having the full information on which to judge the validity of a trial and jeopardises the value of protocol papers and the credibility of the trial itself. Greater efforts are needed to promote the transparent reporting of monitoring to journal editors and authors.

An approach to virtual clinical trial site visits: Lessons from the MeTeOR trial

Objective

To provide a framework for conducting clinical trial site visits virtually over videoconference, and to report on our experience doing so during the twelve-year follow-up of the Meniscal Tear in Osteoarthritis Research (MeTeOR) trial.

Design

Using published FDA guidance and prior literature, we created a structure for virtual site visits that prioritized monitoring for protocol compliance, safety, and data integrity. We conducted site visits in three stages: preparation for the visit, the virtual meeting itself, and follow-up. The preparation phase involved a review of relevant site-specific documents and a written report on the findings prior to the visit. The virtual visit itself was focused on any questions the site staff had about the pre-visit report, observing a mock study visit, touring physical spaces, and understanding the site staff's work environment. In the follow-up phase, we wrote a post-visit report summarizing the discussion during the visit and feedback given by the coordinating site.

Results

We found that the virtual site visits conducted as part of the MeTeOR trial follow-up ran smoothly. Although we could not directly compare in-person and virtual site visits, site staff unanimously appreciated the efficiency and effectiveness of the virtual site visits. We noted that displaying physical workspaces over videoconferencing was difficult, and a notable drawback to this method.

Conclusions

To our knowledge, this is the first published framework for conducting virtual clinical trial site visits. Conducting these visits virtually confer several advantages in terms of time, money, and efficiency.

What is the purpose of clinical trial monitoring?

BACKGROUND

The sources of information on clinical trial monitoring do not give information in an accessible language and do not give detailed guidance. In order to enable communication and to build clinical trial monitoring tools on a strong easily communicated foundation, we identified the need to define monitoring in accessible language.

METHODS

In a three-step process, the material from sources that describe clinical trial monitoring were synthesised into principles of monitoring. A poll regarding their applicability was run at a UK national academic clinical trials monitoring meeting.

RESULTS

The process derived 5 key principles of monitoring: keeping participants safe and respecting their rights, having data we can trust, making sure the trial is being run as it was meant to be, improving the way the trial is run and preventing problems before they happen.

CONCLUSION

From the many sources mentioning monitoring of clinical trials, the purpose of monitoring can be summarised simply as 5 principles. These principles, given in accessible language, should form a firm basis for discussion of monitoring of clinical trials.

Panoramic quality assessment tool for investigator initiated trials

Objectives

Quality can be a challenge for Investigator initiated trials (IITs) since these trials are scarcely overseen by a sponsor or monitoring team. Therefore, quality assessment for departments managing clinical research grants program is important and urgently needed. Our study aims at developing a handy quality assessment tool for IITs that can be applied by both departments and project teams.

Methods

The framework of the quality assessment tool was developed based on the literature studies, accepted guidelines and the Delphi method. A total of 272 ongoing IITs funded by Shanghai non-profit organizations in 2015 and 2016 were used to extract quality indexes. Confirmatory factor analysis (CFA) was used to further evaluate the validity and feasibility of the conceptual quality assessment tool.

Results

The tool consisted of 4 critical quality attributes, including progress, quality, regulation, scientificity, and 13 observed quality indexes. A total of 257 IITs were included in the validity and feasibility assessment. The majority (60.29%) were Randomized Controlled Trial (RCT), and 41.18% were multi-center studies. In order to test the validity and feasibility of IITs quality assessment tool, CFA showed that the model fit the data adequately. (CMIN/DF = 1.868, GFI = 0.916; CFI = 0.936; TLI = 0.919; RMSEA = 0.063; SRMR = 0.076). Different types of clinical studies fit well in the tool. However, RCT scored lower than prospective cohort and retrospective study in enrollment progress (7.02 vs. 7.43, 9.63, respectively).

Conclusion

This study established a panoramic quality assessment tool based on the Delphi method and CFA, and the feasibility and effectiveness of the tool were verified through clinical research examples. The use of this tool can help project management departments effectively and dynamically manage research projects, rationally allocate resources, and ensure the quality of IITs.

Monitoring and auditing protocol adherence, data integrity and ethical conduct of a randomized clinical trial: A case study

PURPOSE

To categorize, quantify and interpret findings documented in feedback letters of monitoring or auditing visits for an investigator-initiated, peer-review funded multicenter randomized trial testing probiotics for critically ill patients.

MATERIALS & METHODS

In 37 Canadian centers, monitoring and auditing visits were performed by 3 trained individuals; findings were reported in feedback letters. At trial termination, we performed duplicate content analysis on letters, categorizing observations first into unique findings, followed by 10 pre-determined trial quality management domains. We further classified each observation into a) missing operational records, b) errors in process, and potential threats to c) data integrity, d) patient privacy or e) safety.

RESULTS

Across 37 monitoring or auditing visits, 75 unique findings were categorized into 10 domains. Most frequently, observations were in domains of training documentation (180/566 [32%]) and the informed consent process (133/566 [23%]). Most observations were missing operational records (438/566 [77%]) rather than errors in process (128/566 [23%]). Of 75 findings, 13 (62/566 observations [11%]) posed a potential threat to data integrity, 1 (1/566 observation [0.18%]) to patient privacy, and 9 (49/566 observations [8.7%]) to patient safety.

CONCLUSIONS

Monitoring and auditing findings predominantly concerned missing documentation with minimal threats to data integrity, patient privacy or safety.

TRIAL REGISTRATION

PROSPECT (Probiotics: Prevention of Severe Pneumonia and Endotracheal Colonization Trial): NCT02462590.

Monitoring strategies for clinical intervention studies

BACKGROUND

Trial monitoring is an important component of good clinical practice to ensure the safety and rights of study participants, confidentiality of personal information, and quality of data. However, the effectiveness of various existing monitoring approaches is unclear. Information to guide the choice of monitoring methods in clinical intervention studies may help trialists, support units, and monitors to effectively adjust their approaches to current knowledge and evidence.

OBJECTIVES

To evaluate the advantages and disadvantages of different monitoring strategies (including risk-based strategies and others) for clinical intervention studies examined in prospective comparative studies of monitoring interventions.

SEARCH METHODS

We systematically searched CENTRAL, PubMed, and Embase via Ovid for relevant published literature up to March 2021. We searched the online 'Studies within A Trial' (SWAT) repository, grey literature, and trial registries for ongoing or unpublished studies.

SELECTION CRITERIA

We included randomized or non-randomized prospective, empirical evaluation studies of different monitoring strategies in one or more clinical intervention studies. We applied no restrictions for language or date of publication.

DATA COLLECTION AND ANALYSIS

We extracted data on the evaluated monitoring methods, countries involved, study population, study setting, randomization method, and numbers and proportions in each intervention group. Our primary outcome was critical and major monitoring findings in prospective intervention studies. Monitoring findings were classified according to different error domains (e.g. major eligibility violations) and the primary outcome measure was a composite of these domains. Secondary outcomes were individual error domains, participant recruitment and follow-up, and resource use. If we identified more than one study for a comparison and outcome definitions were similar across identified studies, we quantitatively summarized effects in a meta-analysis using a random-effects model. Otherwise, we qualitatively summarized the results of eligible studies stratified by different comparisons of monitoring strategies. We used the GRADE approach to assess the certainty of the evidence for different groups of comparisons.

MAIN RESULTS

We identified eight eligible studies, which we grouped into five comparisons. 1. Risk-based versus extensive on-site monitoring: based on two large studies, we found moderate certainty of evidence for the combined primary outcome of major or critical findings that risk-based monitoring is not inferior to extensive on-site monitoring. Although the risk ratio was close to 'no difference' (1.03 with a 95% confidence interval [CI] of 0.81 to 1.33, below 1.0 in favor of the risk-based strategy), the high imprecision in one study and the small number of eligible studies resulted in a wide CI of the summary estimate. Low certainty of evidence suggested that monitoring strategies with extensive on-site monitoring were associated with considerably higher resource use and costs (up to a factor of 3.4). Data on recruitment or retention of trial participants were not available. 2. Central monitoring with triggered on-site visits versus regular on-site visits: combining the results of two eligible studies yielded low certainty of evidence with a risk ratio of 1.83 (95% CI 0.51 to 6.55) in favor of triggered monitoring intervention. Data on recruitment, retention, and resource use were not available. 3. Central statistical monitoring and local monitoring performed by site staff with annual on-site visits versus central statistical monitoring and local monitoring only: based on one study, there was moderate certainty of evidence that a small number of major and critical findings were missed with the central monitoring approach without on-site visits: 3.8% of participants in the group without on-site visits and 6.4% in the group with on-site visits had a major or critical monitoring finding (odds ratio 1.7, 95% CI 1.1 to 2.7; P = 0.03). The absolute number of monitoring findings was very low, probably because defined major and critical findings were very study specific and central monitoring was present in both intervention groups. Very low certainty of evidence did not suggest a relevant effect on participant retention, and very low certainty evidence indicated an extra cost for on-site visits of USD 2,035,392. There were no data on recruitment. 4. Traditional 100% source data verification (SDV) versus targeted or remote SDV: the two studies assessing targeted and remote SDV reported findings only related to source documents. Compared to the final database obtained using the full SDV monitoring process, only a small proportion of remaining errors on overall data were identified using the targeted SDV process in the MONITORING study (absolute difference 1.47%, 95% CI 1.41% to 1.53%). Targeted SDV was effective in the verification of source documents, but increased the workload on data management. The other included study was a pilot study, which compared traditional on-site SDV versus remote SDV and found little difference in monitoring findings and the ability to locate data values despite marked differences in remote access in two clinical trial networks. There were no data on recruitment or retention. 5. Systematic on-site initiation visit versus on-site initiation visit upon request: very low certainty of evidence suggested no difference in retention and recruitment between the two approaches. There were no data on critical and major findings or on resource use.

AUTHORS' CONCLUSIONS

The evidence base is limited in terms of quantity and quality. Ideally, for each of the five identified comparisons, more prospective, comparative monitoring studies nested in clinical trials and measuring effects on all outcomes specified in this review are necessary to draw more reliable conclusions. However, the results suggesting risk-based, targeted, and mainly central monitoring as an efficient strategy are promising. The development of reliable triggers for on-site visits is ongoing; different triggers might be used in different settings. More evidence on risk indicators that identify sites with problems or the prognostic value of triggers is needed to further optimize central monitoring strategies. In particular, approaches with an initial assessment of trial-specific risks that need to be closely monitored centrally during trial conduct with triggered on-site visits should be evaluated in future research.

Clinical trial recruiters' experiences working with trial eligibility criteria: results of an exploratory, cross-sectional, online survey in the UK

BACKGROUND

Eligibility criteria are a fundamental element of clinical trial design, defining who can and who should not participate in a trial. Problems with the design or application of criteria are known to occur and pose risks to participants' safety and trial integrity, sometimes also negatively impacting on trial recruitment and generalisability. We conducted a short, exploratory survey to gather evidence on UK recruiters' experiences interpreting and applying eligibility criteria and their views on how criteria are communicated and developed.

METHODS

Our survey included topics informed by a wider programme of work at the Clinical Trials Research Unit, University of Leeds, on assuring eligibility criteria quality. Respondents were asked to answer based on all their trial experience, not only on experiences with our trials. The survey was disseminated to recruiters collaborating on trials run at our trials unit, and via other mailing lists and social media. The quantitative responses were descriptively analysed, with inductive analysis of free-text responses to identify themes.

RESULTS

A total of 823 eligible respondents participated. In total, 79% of respondents reported finding problems with eligibility criteria in some trials, and 9% in most trials. The main themes in the types of problems experienced were criteria clarity (67% of comments), feasibility (34%), and suitability (14%). In total, 27% of those reporting some level of problem said these problems had led to patients being incorrectly included in trials; 40% said they had led to incorrect exclusions. Most respondents (56%) reported accessing eligibility criteria mainly in the trial protocol. Most respondents (74%) supported the idea of recruiter review of eligibility criteria earlier in the protocol development process.

CONCLUSIONS

Our survey corroborates other evidence about the existence of suboptimal trial eligibility criteria. Problems with clarity were the most often reported, but the number of comments on feasibility and suitability suggest some recruiters feel eligibility criteria and associated assessments can hinder recruitment to trials. Our proposal for more recruiter involvement in protocol development has strong support and some potential benefits, but questions remain about how best to implement this. We invite other trialists to consider our other suggestions for how to assure quality in trial eligibility criteria.

Dynamic methods for ongoing assessment of site-level risk in risk-based monitoring of clinical trials: A scoping review

Background/Aims

It is increasingly recognised that reliance on frequent site visits for monitoring clinical trials is inefficient. Regulators and trialists have recently encouraged more risk-based monitoring. Risk assessment should take place before a trial begins to define the overarching monitoring strategy. It can also be done on an ongoing basis, to target sites for monitoring activity. Various methods have been proposed for such prioritisation, often using terms like ‘central statistical monitoring’, ‘triggered monitoring’ or, as in the International Conference on Harmonization Good Clinical Practice guidance, ‘targeted on-site monitoring’. We conducted a scoping review to identify such methods, to establish if any were supported by adequate evidence to allow wider implementation, and to guide future developments in this field of research.

Methods

We used seven publication databases, two sets of methodological conference abstracts and an Internet search engine to identify methods for using centrally held trial data to assess site conduct during a trial. We included only reports in English, and excluded reports published before 1996 or not directly relevant to our research question. We used reference and citation searches to find additional relevant reports. We extracted data using a predefined template. We contacted authors to request additional information about included reports.

Results

We included 30 reports in our final dataset, of which 21 were peer-reviewed publications. In all, 20 reports described central statistical monitoring methods (of which 7 focussed on detection of fraud or misconduct) and 9 described triggered monitoring methods; 21 reports included some assessment of their methods’ effectiveness, typically exploring the methods’ characteristics using real trial data without known integrity issues. Of the 21 with some effectiveness assessment, most contained limited information about whether or not concerns identified through central monitoring constituted meaningful problems. Several reports demonstrated good classification ability based on more than one classification statistic, but never without caveats of unclear reporting or other classification statistics being low or unavailable. Some reports commented on cost savings from reduced on-site monitoring, but none gave detailed costings for the development and maintenance of central monitoring methods themselves.

Conclusion

Our review identified various proposed methods, some of which could be combined within the same trial. The apparent emphasis on fraud detection may not be proportionate in all trial settings. Despite some promising evidence and some self-justifying benefits for data cleaning activity, many proposed methods have limitations that may currently prevent their routine use for targeting trial monitoring activity. The implementation costs, or uncertainty about these, may also be a barrier. We make recommendations for how the evidence-base supporting these methods could be improved.

Assessing the potential for prevention or earlier detection of on-site monitoring findings from randomised controlled trials: Further analyses of findings from the prospective TEMPER triggered monitoring study

BACKGROUND/AIMS

Clinical trials should be designed and managed to minimise important errors with potential to compromise patient safety or data integrity, employ monitoring practices that detect and correct important errors quickly, and take robust action to prevent repetition. Regulators highlight the use of risk-based monitoring, making greater use of centralised monitoring and reducing reliance on centre visits. The TEMPER study was a prospective evaluation of triggered monitoring (a risk-based monitoring method), whereby centres are prioritised for visits based on central monitoring results. Conducted in three UK-based randomised cancer treatment trials of investigational medicine products with time-to-event outcomes, it found high levels of serious findings at triggered centre visits but also at visits to matched control centres that, based on central monitoring, were not of concern. Here, we report a detailed review of the serious findings from TEMPER centre visits. We sought to identify feasible, centralised processes which might detect or prevent these findings without a centre visit.

METHODS

The primary outcome of this study was the proportion of all 'major' and 'critical' TEMPER centre visit findings theoretically detectable or preventable through a feasible, centralised process. To devise processes, we considered a representative example of each finding type through an internal consensus exercise. This involved (a) agreeing the potential, by some described process, for each finding type to be centrally detected or prevented and (b) agreeing a proposed feasibility score for each proposed process. To further assess feasibility, we ran a consultation exercise, whereby the proposed processes were reviewed and rated for feasibility by invited external trialists.

RESULTS

In TEMPER, 312 major or critical findings were identified at 94 visits. These findings comprised 120 distinct issues, for which we proposed 56 different centralised processes. Following independent review of the feasibility of the proposed processes by 87 consultation respondents across eight different trial stakeholder groups, we conclude that 306/312 (98%) findings could theoretically be prevented or identified centrally. Of the processes deemed feasible, those relating to informed consent could have the most impact. Of processes not currently deemed feasible, those involving use of electronic health records are among those with the largest potential benefit.

CONCLUSIONS

This work presents a best-case scenario, where a large majority of monitoring findings were deemed theoretically preventable or detectable by central processes. Caveats include the cost of applying all necessary methods, and the resource implications of enhanced central monitoring for both centre and trials unit staff. Our results will inform future monitoring plans and emphasise the importance of continued critical review of monitoring processes and outcomes to ensure they remain appropriate.

Time-consuming and expensive data quality monitoring procedures persist in clinical trials: A national survey

INTRODUCTION

The Good Clinical Practice guideline identifies that data monitoring is an essential research activity. However, limited evidence exists on how to perform monitoring including the amount or frequency that is needed to ensure data quality. This study aims to explore the monitoring procedures that are implemented to ensure data quality in Australian clinical research studies.

MATERIAL AND METHODS

Clinical studies listed on the Australian and New Zealand Clinical Trials Registry were invited to participate in a national cross-sectional, mixed-mode, multi-contact (postal letter and e-mail) web-based survey. Information was gathered about the types of data quality monitoring procedures being implemented.

RESULTS

Of the 3689 clinical studies contacted, 589 (16.0%) responded, of which 441 (77.4%) completed the survey. Over half (55%) of the studies applied source data verification (SDV) compared to risk-based targeted and triggered monitoring (10-11%). Conducting 100% on-site monitoring was most common for those who implemented the traditional approach. Respondents who did not conduct 100% monitoring, included 1-25% of data points for SDV, centralized or on-site monitoring. The incidence of adverse events and protocol deviations were the most likely factors to trigger a site visit for risk-based triggered (63% and 44%) and centralized monitoring (48% and 44%), respectively.

CONCLUSION

Instead of using more optimal risk-based approaches, small single-site clinical studies are conducting traditional monitoring procedures which are time consuming and expensive. Formal guidelines need to be improved and provided to all researchers for 'new' risk-based monitoring approaches.

Clinical trial monitoring effectiveness: Remote risk-based monitoring versus on-site monitoring with 100% source data verification

BACKGROUND/AIMS

Traditional on-site monitoring of clinical trials via frequent site visits and 100% source data verification is cost-consuming, and it still cannot guarantee data quality effectively. Depending on the types and designs of clinical trials, an alternative would be combining several monitoring methods, such as risk-based monitoring and remote monitoring. However, there is insufficient evidence of its effectiveness. This research compared the effectiveness of risk-based monitoring with a remote monitoring system with that of traditional on-site monitoring.

METHODS

With a cloud-based remote monitoring system called beagle View^®, we created a remote risk-based monitoring methodology that focused only on critical data and processes. We selected a randomized controlled trial conducted at Tohoku University Hospital and randomly sampled 11 subjects whose case report forms had already been reviewed by data managers. Critical data and processes were verified retrospectively by remote risk-based monitoring; later, all data and processes were confirmed by on-site monitoring. We compared the ability of remote risk-based monitoring to detect critical data and process errors with that of on-site monitoring with 100% source data verification, including an examination of clinical trial staff workload and potential cost savings.

RESULTS

Of the total data points (n = 5617), 19.7% (n = 1105, 95% confidence interval = 18.7-20.7) were identified as critical. The error rates of critical data detected by on-site monitoring, remote risk-based monitoring, and data review by data managers were 7.6% (n = 84, 95% CI = 6.2-9.3), 7.6% (n = 84, 95% confidence interval = 6.2-9.3), and 3.9% (n = 43, 95% confidence interval = 2.9-5.2), respectively. The total number of critical process errors detected by on-site monitoring was 14. Of these 14, 92.9% (n = 13, 95% confidence interval = 68.5-98.7) and 42.9% (n = 6, 95% confidence interval = 21.4-67.4) of critical process errors were detected by remote risk-based monitoring and data review by data managers, respectively. The mean time clinical trial staff spent dealing with remote risk-based monitoring was 9.9 ± 5.3 (mean ± SD) min per visit per subject. Our calculations show that remote risk-based monitoring saved between 9 and 41 on-site monitoring visits, corresponding to a cost of between US$13,500 and US$61,500 per trial site.

CONCLUSION

Remote risk-based monitoring was able to detect critical data and process errors as well as on-site monitoring with 100% source data verification, saving travel time and monitoring costs. Remote risk-based monitoring offers an effective alternative to traditional on-site monitoring of clinical trials.

Improving Stem Cell Clinical Trial Design and Conduct: Development of a Quality Assessment Tool for Stem Cell Clinical Trials

Background

Clinical trials are at the cornerstone of evidence-based stem cell therapies, but the quality assessment for designing and conduct these sometimes-complex studies are scarce of evidence. This study is aimed at developing a handy quality assessment tool for stem cell clinical trials, enhancing capacity of the self-regulate overall quality, and participating protection.

Methods

The framework of quality assessment tool was based on the PQRS (progress-quality-regulation-scientific) quality assessment tool, and detailed quality indicators were developed by leader group discussion, expert consulting, and literature review. Stem cell clinical trials were retrieved from the International Clinical Trials Registry Platform, and corresponding quality indicators were assessed and extracted. The validity and feasibility of conceptual quality assessment tool were further evaluated by using structural equation modeling.

Results

The quality assessment tool for stem cell clinical trials contains four critical quality attributes, including participant protection, scientific value, quality control, and stem cell products, and 9 observed quality indicators. From 11 primary clinical trial registries in the International Clinical Trials Registry Platform, 9410 stem cell trial registrations were identified, and 1036 studies were eligible for publications and protocols screening. After reviewed full text, 37 studies were included in the validity and feasibility evaluation: 32 studies were completed, and 3 studies terminated early. Most of the studies (83.79%) were in the early phase, and 63.16% of the studies were investigator-initiated trial. To further tested for validity, the critical quality attributes and quality indicators (QIs) between expertise further validated by the SEM method, which showed a good fit for the model (chi − square = 26.008; P = 0.353; TLI = 0.967; CFI = 0.978; RMSEA = 0.048). Compared with exploratory trials, evaluating using the quality assessment tool, confirmatory trials performed similarly in participant protection, scientific value, and quality control, but lower in stem cell products.

Conclusions

The results of critical quality attributes and quality indicators between expertise and confirmatory validation analysis are basically consistent, indicating the feasibility and validity of applying this quality assessment tool for overall quality evaluation of stem cell trials.

Central statistical monitoring of investigator-led clinical trials in oncology

Investigator-led clinical trials are pragmatic trials that aim to investigate the benefits and harms of treatments in routine clinical practice. These much-needed trials represent the majority of all trials currently conducted. They are however threatened by the rising costs of clinical research, which are in part due to extensive trial monitoring processes that focus on unimportant details. Risk-based quality management focuses, instead, on “things that really matter”. We discuss the role of central statistical monitoring as part of risk-based quality management. We describe the principles of central statistical monitoring, provide examples of its use, and argue that it could help drive down the cost of randomized clinical trials, especially investigator-led trials, whilst improving their quality.

A randomized evaluation of on-site monitoring nested in a multinational randomized trial

BACKGROUND

Evidence from prospectively designed studies to guide on-site monitoring practices for randomized trials is limited. A cluster randomized study, nested within the Strategic Timing of AntiRetroviral Treatment (START) trial, was conducted to evaluate on-site monitoring.

METHODS

Sites were randomized to either annual on-site monitoring or no on-site monitoring. All sites were centrally monitored, and local monitoring was carried out twice each year. Randomization was stratified by country and projected enrollment in START. The primary outcome was a participant-level composite outcome including components for eligibility errors, consent violations, use of antiretroviral treatment not recommended by protocol, late reporting of START primary and secondary clinical endpoints (defined as the event being reported more than 6 months from occurrence), and data alteration and fraud. Logistic regression fixed effect hierarchical models were used to compare on-site versus no on-site monitoring for the primary composite outcome and its components. Odds ratios and 95% confidence intervals comparing on-site monitoring versus no on-site monitoring are cited.

RESULTS

In total, 99 sites (2107 participants) were randomized to receive annual on-site monitoring and 97 sites (2264 participants) were randomized to be monitored only centrally and locally. The two monitoring groups were well balanced at entry. In the on-site monitoring group, 469 annual on-site monitoring visits were conducted, and 134 participants (6.4%) in 56 of 99 sites (57%) had a primary monitoring outcome. In the no on-site monitoring group, 85 participants (3.8%) in 34 of 97 sites (35%) had a primary monitoring outcome (odds ratio = 1.7; 95% confidence interval: 1.1-2.7; p = 0.03). Informed consent violations accounted for most outcomes in each group (56 vs 41 participants). The largest odds ratio was for eligibility violations (odds ratio = 12.2; 95% confidence interval: 1.8-85.2; p = 0.01). The number of participants with a late START primary endpoint was similar for each monitoring group (23 vs 16 participants). Late START grade 4 and unscheduled hospitalization events were found for 34 participants in the on-site monitoring group and 19 participants in the no on-site monitoring group (odds ratio = 2.0; 95% confidence interval: 1.1-3.7; p = 0.02). There were no cases of data alteration or fraud. Based on the travel budget for on-site monitoring and the hours spent conducting on-site monitoring, the estimated cost of on-site monitoring was over US$2 million.

CONCLUSION

On-site monitoring led to the identification of more eligibility and consent violations and START clinical events being reported more than 6 months from occurrence as compared to no on-site monitoring. Considering the nature of the excess monitoring outcomes identified at sites receiving on-site monitoring, as well as the cost of on-site monitoring, the value to the START study was limited.

Monitoring in practice - How are UK academic clinical trials monitored? A survey

Background

Despite the US Food and Drug Administration (FDA) and European Medicines Agency (EMA) encouraging the use of risk-based monitoring for trials in 2013, there remains a lack of evidence-based guidelines on how to monitor. We surveyed the academic United Kingdom Clinical Research Collaboration (UKCRC) registered clinical trials units (CTUs) to find out their policy on monitoring of phase III randomised clinical trials of an investigational medicinal product (CTIMPs).

Methods

An online survey of monitoring policy with sections on the CTU, central monitoring and on-site monitoring was sent to all 50 UKCRC registered CTUs in November 2018.

Descriptive data analysis and tabulations are reported using the total number answering each question.

Results

A total of 43/50 (86%) of CTUs responded with 38 conducting phase III randomised CTIMP trials. Of these 38 CTUs, 34 finished the survey. Most CTUs (36/37, 97%) use a central monitoring process to guide, target or supplement site visits. More than half (19/36, 53%) of CTUs do not use an automated monitoring report when centrally monitoring trials and all units use trial team knowledge to make a final decision on whether an on-site visit is required.

A total of 31/34 (91%) CTUs used triggers to decide whether or not to conduct an on-site monitoring visit. On-site, a mixture of source data verification and checking of processes was carried out.

The CTUs overwhelmingly (27/34, 79%) selected optimising central monitoring as their most pressing concern.

Conclusion

The survey showed a wide variation in phase III randomised CTIMP trial monitoring practices by academic clinical trials units within a single research-active country. We urgently need to develop evidence-based regulator-agreed guidance for CTUs on best practice for both central and on-site monitoring and to develop tools for all CTUs to use.

Impact of a targeted monitoring on data-quality and data-management workload of randomized controlled trials: A prospective comparative study

AIMS

Monitoring risk-based approaches in clinical trials are encouraged by regulatory guidance. However, the impact of a targeted source data verification (SDV) on data-management (DM) workload and on final data quality needs to be addressed.

METHODS

MONITORING was a prospective study aiming at comparing full SDV (100% of data verified for all patients) and targeted SDV (only key data verified for all patients) followed by the same DM program (detecting missing data and checking consistency) on final data quality, global workload and staffing costs.

RESULTS

In all, 137 008 data including 18 124 key data were collected for 126 patients from 6 clinical trials. Compared to the final database obtained using the full SDV monitoring process, the final database obtained using the targeted SDV monitoring process had a residual error rate of 1.47% (95% confidence interval, 1.41-1.53%) on overall data and 0.78% (95% confidence interval, 0.65-0.91%) on key data. There were nearly 4 times more queries per study with targeted SDV than with full SDV (mean ± standard deviation: 132 ± 101 vs 34 ± 26; P = .03). For a handling time of 15 minutes per query, the global workload of the targeted SDV monitoring strategy remained below that of the full SDV monitoring strategy. From 25 minutes per query it was above, increasing progressively to represent a 50% increase for 45 minutes per query.

CONCLUSION

Targeted SDV monitoring is accompanied by increased workload for DM, which allows to obtain a small proportion of remaining errors on key data (<1%), but may substantially increase trial costs.

Early warnings and repayment plans: novel trial management methods for monitoring and managing data return rates in a multi-centre phase III randomised controlled trial with paper Case Report Forms

Background

Monitoring and managing data returns in multi-centre randomised controlled trials is an important aspect of trial management. Maintaining consistently high data return rates has various benefits for trials, including enhancing oversight, improving reliability of central monitoring techniques and helping prepare for database lock and trial analyses. Despite this, there is little evidence to support best practice, and current standard methods may not be optimal.

Methods

We report novel methods from the Trial of Imaging and Schedule in Seminoma Testis (TRISST), a UK-based, multi-centre, phase III trial using paper Case Report Forms to collect data over a 6-year follow-up period for 669 patients. Using an automated database report which summarises the data return rate overall and per centre, we developed a Microsoft Excel-based tool to allow observation of per-centre trends in data return rate over time. The tool allowed us to distinguish between forms that can and cannot be completed retrospectively, to inform understanding of issues at individual centres. We reviewed these statistics at regular trials unit team meetings. We notified centres whose data return rate appeared to be falling, even if they had not yet crossed the pre-defined acceptability threshold of an 80% data return rate. We developed a set method for agreeing targets for gradual improvement with centres having persistent data return problems. We formalised a detailed escalation policy to manage centres who failed to meet agreed targets. We conducted a post-hoc, descriptive analysis of the effectiveness of the new processes.

Results

The new processes were used from April 2015 to September 2016. By May 2016, data return rates were higher than they had been at any time previously, and there were no centres with return rates below 80%, which had never been the case before. In total, 10 centres out of 35 were contacted regarding falling data return rates. Six out of these 10 showed improved rates within 6–8 weeks, and the remainder within 4 months.

Conclusions

Our results constitute preliminary effectiveness evidence for novel methods in monitoring and managing data return rates in randomised controlled trials. We encourage other researchers to work on generating better evidence-based methods in this area, whether through more robust evaluation of our methods or of others.

Implementing monitoring triggers and matching of triggered and control sites in the TEMPER study: a description and evaluation of a triggered monitoring management system

BACKGROUND

Triggered monitoring in clinical trials is a risk-based monitoring approach where triggers (centrally monitored, predefined key risk and performance indicators) drive the extent, timing, and frequency of monitoring visits. The TEMPER study used a prospective, matched-pair design to evaluate the use of a triggered monitoring strategy, comparing findings from triggered monitoring visits with those from matched control sites. To facilitate this study, we developed a bespoke risk-based monitoring system: the TEMPER Management System.

METHODS

The TEMPER Management System comprises a web application (the front end), an SQL server database (the back end) to store the data generated for TEMPER, and a reporting function to aid users in study processes such as the selection of triggered sites. Triggers based on current practice were specified for three clinical trials and were implemented in the system. Trigger data were generated in the system using data extracted from the trial databases to inform the selection of triggered sites to visit. Matching of the chosen triggered sites with untriggered control sites was also performed in the system, while data entry screens facilitated the collection and management of the data from findings gathered at monitoring visits.

RESULTS

There were 38 triggers specified for the participating trials. Using these, 42 triggered sites were chosen and matched with control sites. Monitoring visits were carried out to all sites, and visit findings were entered into the TEMPER Management System. Finally, data extracted from the system were used for analysis.

CONCLUSIONS

The TEMPER Management System made possible the completion of the TEMPER study. It implemented an approach of standardising the automation of current-practice triggers, and the generation of trigger data to inform the selection of triggered sites to visit. It also implemented a matching algorithm informing the selection of matched control sites. We hope that by publishing this paper it encourages other trialists to share their approaches to, and experiences of, triggered monitoring and other risk-based monitoring systems.