The impact of clinical trial monitoring approaches on data integrity and cost--a review of current literature

Navigating the challenges of clinical trial professionals in the healthcare sector

Clinical trials (CTs) are essential for medical advancements but face significant challenges, particularly in professional training and role clarity. Principal investigators, clinical research coordinators (CRCs), nurses, clinical trial pharmacists, and monitors are key players. Each faces unique challenges, such as maintaining protocol compliance, managing investigational products, and ensuring data integrity. Clinical trials' complexity and evolving nature demand specialized and ongoing training for these professionals. Addressing these challenges requires clear role delineation, continuous professional development, and supportive workplace environments to improve retention and trial outcomes. Enhanced training programs and a collaborative approach are essential for the successful conduct of clinical trials and the advancement of medical research.

Evaluating registry-based trial economics: Results from the STRESS clinical trial

Background

Registry-based trials have the potential to reduce randomized clinical trial (RCT) costs. However, observed cost differences also may be achieved through pragmatic trial designs. A systematic comparison of trial costs across different designs has not been previously performed.

Methods

We conducted a study to compare the current Steroids to Reduce Systemic inflammation after infant heart surgery (STRESS) registry-based RCT vs. two established designs: pragmatic RCT and explanatory RCT. The primary outcome was total RCT design costs. Secondary outcomes included: RCT duration and personnel hours. Costs were estimated using the Duke Clinical Research Institute's pricing model.

Results

The Registry-Based RCT estimated duration was 31.9 weeks greater than the other designs (259.5 vs. 227.6 weeks). This delay was caused by the Registry-Based design's periodic data harvesting that delayed site closing and statistical reporting. Total personnel hours were greatest for the Explanatory design followed by the Pragmatic design and the Registry-Based design (52,488 vs 29,763 vs. 24,480 h, respectively). Total costs were greatest for the Explanatory design followed by the Pragmatic design and the Registry-Based design ($10,140,263 vs. $4,164,863 vs. $3,268,504, respectively). Thus, Registry-Based total costs were 32 % of the Explanatory and 78 % of the Pragmatic design.

Conclusion

Total costs for the STRESS RCT with a registry-based design were less than those for a pragmatic design and much less than an explanatory design. Cost savings reflect design elements and leveraging of registry resources to improve cost efficiency, but delays to trial completion should be considered.

Research integrity in randomized clinical trials: A scoping umbrella review

BACKGROUND

Randomized clinical trials (RCTs) are experiencing a crisis of confidence in their trustworthiness. Although a comprehensive literature search yielded several reviews on RCT integrity, an overarching overview is lacking.

OBJECTIVES

The authors undertook a scoping umbrella review of the research integrity literature concerning RCTs.

SEARCH STRATEGY AND SELECTION CRITERIA

Following prospective registration (https://osf.io/3ursn), two reviewers independently searched PubMed, Scopus, The Cochrane Library, and Google Scholar, without language or time restrictions, until November 2021. The authors included systematic reviews covering any aspect of research integrity throughout the RCT lifecycle.

DATA COLLECTION AND ANALYSIS

The authors assessed methodological quality using a modified AMSTAR 2 (A Measurement Tool to Assess Systematic Reviews) tool and collated the main findings.

MAIN RESULTS

A total of 55 relevant reviews, summarizing 6001 studies (median per review, 63; range, 8-1106) from 1964 to 2021, had an overall critically low quality of 96% (53 reviews). Topics covered included general aspects (15%), design and approval (22%), conduct and monitoring (11%), reporting (38%), postpublication concerns (2%), and future research (13%). The most common integrity issues covered were ethics (18%) and transparency (18%).

CONCLUSIONS

Low-quality reviews identified various integrity issues across the RCT lifecycle, emphasizing the importance of high ethical standards and professionalism while highlighting gaps in the integrity landscape. Multistakeholder consensus is needed to develop specific RCT integrity standards.

Performance of four centralized statistical monitoring methods for early detection of an atypical center in a multicenter study

Background

Ensuring the quality of data is essential for the credibility of a multicenter clinical trial. Centralized Statistical Monitoring (CSM) of data allows the detection of a center in which the distribution of a specific variable is atypical compared to other centers. The ideal CSM method should allow early detection of problem and therefore involve the fewest possible participants.

Methods

We simulated clinical trials and compared the performance of four CSM methods (Student, Hatayama, Desmet, Distance) to detect whether the distribution of a quantitative variable was atypical in one center in relation to the others, with different numbers of participants and different mean deviation amplitudes.

Results

The Student and Hatayama methods had good sensitivity but poor specificity, which disqualifies them for practical use in CSM. The Desmet and Distance methods had very high specificity for detecting all the mean deviations tested (including small values) but low sensitivity with mean deviations less than 50%.

Conclusion

Although the Student and Hatayama methods are more sensitive, their low specificity would lead to too many alerts being triggered, which would result in additional unnecessary control work to ensure data quality. The Desmet and Distance methods have low sensitivity when the deviation from the mean is low, suggesting that the CSM should be used alongside other conventional monitoring procedures rather than replacing them. However, they have excellent specificity, which suggests they can be applied routinely, since using them takes up no time at central level and does not cause any unnecessary workload in investigating centers.

Guidance on interim analysis methods in clinical trials

Interim analyses in clinical trials can take on a multitude of forms. They are often used to guide Data and Safety Monitoring Board (DSMB) recommendations to study teams regarding recruitment targets for large, later-phase clinical trials. As collaborative biostatisticians working and teaching in multiple fields of research and across a broad array of trial phases, we note the large heterogeneity and confusion surrounding interim analyses in clinical trials. Thus, in this paper, we aim to provide a general overview and guidance on interim analyses for a nonstatistical audience. We explain each of the following types of interim analyses: efficacy, futility, safety, and sample size re-estimation, and we provide the reader with reasoning, examples, and implications for each. We emphasize that while the types of interim analyses employed may differ depending on the nature of the study, we would always recommend prespecification of the interim analytic plan to the extent possible with risk mitigation and trial integrity remaining a priority. Finally, we posit that interim analyses should be used as tools to help the DSMB make informed decisions in the context of the overarching study. They should generally not be deemed binding, and they should not be reviewed in isolation.

Revitalising cancer trials post-pandemic: time for reform

The COVID-19 pandemic posed significant risk to the health of cancer patients, compromised standard cancer care and interrupted clinical cancer trials, prompting dramatic streamlining of services. From this health crisis has emerged the opportunity to carry forward an unexpected legacy of positive reforms to clinical cancer research, where conventionally convoluted approvals processes, inefficient trial design, procedures and data gathering could benefit from the lessons in rationalisation learned during the pandemic.

Tools for large-scale data analytics of an international multi-center study in radiation oncology for cervical cancer

PURPOSE

To develop and implement a software that enables centers, treating patients with state-of-the-art radiation oncology, to compare their patient, treatment, and outcome data to a reference cohort, and to assess the quality of their treatment approach.

MATERIALS AND METHODS

A comprehensive data dashboard was designed, which al- lowed holistic assessment of institutional treatment approaches. The software was tested in the ongoing EMBRACE-II study for locally advanced cervical cancer. The tool created individualized dashboards and automatic analysis scripts, verified pro- tocol compliance and checked data for inconsistencies. Identified quality assurance (QA) events were analysed. A survey among users was conducted to assess usability.

RESULTS

The survey indicated favourable feedback to the prototype and highlighted its value for internal monitoring. Overall, 2302 QA events were identified (0.4% of all collected data). 54% were due to missing or incomplete data, and 46% originated from other causes. At least one QA event was found in 519/1001 (52%) of patients. QA events related to primary study endpoints were found in 16% of patients. Sta- tistical methods demonstrated good performance in detecting anomalies, with precisions ranging from 71% to 100%. Most frequent QA event categories were Treatment Technique (27%), Patient Characteristics (22%), Dose Reporting (17%), Outcome 156 (15%), Outliers (12%), and RT Structures (8%).

CONCLUSION

A software tool was developed and tested within a clinical trial in radia- tion oncology. It enabled the quantitative and qualitative comparison of institutional patient and treatment parameters with a large multi-center reference cohort. We demonstrated the value of using statistical methods to automatically detect implau- sible data points and highlighted common pitfalls and uncertainties in radiotherapy for cervical cancer.

Effect of intensive versus limited monitoring on clinical trial conduct and outcomes: A randomized trial

BACKGROUND

Regulatory agencies have endorsed more limited approaches to clinical trial site monitoring. However, the impact of different monitoring strategies on trial conduct and outcomes is unclear.

METHODS

We conducted a patient-level block-randomized controlled trial evaluating the effect of intensive versus limited monitoring on cardiovascular clinical trial conduct and outcomes nested within the CoreValve Continued Access and Expanded Use Studies. Intensive monitoring included complete source data verification of all critical datapoints whereas limited monitoring included automated data checks only. This study's endpoints included clinical trial outcome ascertainment as well as monitoring action items, protocol deviations, and adverse event ascertainment.

RESULTS

A total of 2,708 patients underwent transcatheter aortic valve replacement (TAVR) and were randomized to either intensive monitoring (n = 1,354) or limited monitoring (n = 1,354). Monitoring action items were more common with intensive monitoring (52% vs 15%; P < .001), but there was no difference in the percentage of patients with any protocol deviation (91.6% vs 90.4%; P = .314). The reported incidence of trial outcomes between intensive and limited monitoring was similar for mortality (30 days: 4.8% vs 5.5%, P = .442; 1 year: 20.3% vs 21.3%, P = .473) and stroke (30 days: 2.8% vs 2.4%, P = .458), as well as most secondary trial outcomes with the exception of bleeding (intensive: 36.3% vs limited: 32.0% at 30 days, P = .019). There was a higher reported incidence of cardiac adverse events reported in the intensive monitoring group at 1 year (76.7% vs 72.4%; P = .019).

CONCLUSIONS

Tailored limited monitoring strategies can be implemented without influencing the integrity of TAVR trial outcomes.

Monitoring strategies for clinical intervention studies

BACKGROUND

Trial monitoring is an important component of good clinical practice to ensure the safety and rights of study participants, confidentiality of personal information, and quality of data. However, the effectiveness of various existing monitoring approaches is unclear. Information to guide the choice of monitoring methods in clinical intervention studies may help trialists, support units, and monitors to effectively adjust their approaches to current knowledge and evidence.

OBJECTIVES

To evaluate the advantages and disadvantages of different monitoring strategies (including risk-based strategies and others) for clinical intervention studies examined in prospective comparative studies of monitoring interventions.

SEARCH METHODS

We systematically searched CENTRAL, PubMed, and Embase via Ovid for relevant published literature up to March 2021. We searched the online 'Studies within A Trial' (SWAT) repository, grey literature, and trial registries for ongoing or unpublished studies.

SELECTION CRITERIA

We included randomized or non-randomized prospective, empirical evaluation studies of different monitoring strategies in one or more clinical intervention studies. We applied no restrictions for language or date of publication.

DATA COLLECTION AND ANALYSIS

We extracted data on the evaluated monitoring methods, countries involved, study population, study setting, randomization method, and numbers and proportions in each intervention group. Our primary outcome was critical and major monitoring findings in prospective intervention studies. Monitoring findings were classified according to different error domains (e.g. major eligibility violations) and the primary outcome measure was a composite of these domains. Secondary outcomes were individual error domains, participant recruitment and follow-up, and resource use. If we identified more than one study for a comparison and outcome definitions were similar across identified studies, we quantitatively summarized effects in a meta-analysis using a random-effects model. Otherwise, we qualitatively summarized the results of eligible studies stratified by different comparisons of monitoring strategies. We used the GRADE approach to assess the certainty of the evidence for different groups of comparisons.

MAIN RESULTS

We identified eight eligible studies, which we grouped into five comparisons. 1. Risk-based versus extensive on-site monitoring: based on two large studies, we found moderate certainty of evidence for the combined primary outcome of major or critical findings that risk-based monitoring is not inferior to extensive on-site monitoring. Although the risk ratio was close to 'no difference' (1.03 with a 95% confidence interval [CI] of 0.81 to 1.33, below 1.0 in favor of the risk-based strategy), the high imprecision in one study and the small number of eligible studies resulted in a wide CI of the summary estimate. Low certainty of evidence suggested that monitoring strategies with extensive on-site monitoring were associated with considerably higher resource use and costs (up to a factor of 3.4). Data on recruitment or retention of trial participants were not available. 2. Central monitoring with triggered on-site visits versus regular on-site visits: combining the results of two eligible studies yielded low certainty of evidence with a risk ratio of 1.83 (95% CI 0.51 to 6.55) in favor of triggered monitoring intervention. Data on recruitment, retention, and resource use were not available. 3. Central statistical monitoring and local monitoring performed by site staff with annual on-site visits versus central statistical monitoring and local monitoring only: based on one study, there was moderate certainty of evidence that a small number of major and critical findings were missed with the central monitoring approach without on-site visits: 3.8% of participants in the group without on-site visits and 6.4% in the group with on-site visits had a major or critical monitoring finding (odds ratio 1.7, 95% CI 1.1 to 2.7; P = 0.03). The absolute number of monitoring findings was very low, probably because defined major and critical findings were very study specific and central monitoring was present in both intervention groups. Very low certainty of evidence did not suggest a relevant effect on participant retention, and very low certainty evidence indicated an extra cost for on-site visits of USD 2,035,392. There were no data on recruitment. 4. Traditional 100% source data verification (SDV) versus targeted or remote SDV: the two studies assessing targeted and remote SDV reported findings only related to source documents. Compared to the final database obtained using the full SDV monitoring process, only a small proportion of remaining errors on overall data were identified using the targeted SDV process in the MONITORING study (absolute difference 1.47%, 95% CI 1.41% to 1.53%). Targeted SDV was effective in the verification of source documents, but increased the workload on data management. The other included study was a pilot study, which compared traditional on-site SDV versus remote SDV and found little difference in monitoring findings and the ability to locate data values despite marked differences in remote access in two clinical trial networks. There were no data on recruitment or retention. 5. Systematic on-site initiation visit versus on-site initiation visit upon request: very low certainty of evidence suggested no difference in retention and recruitment between the two approaches. There were no data on critical and major findings or on resource use.

AUTHORS' CONCLUSIONS

The evidence base is limited in terms of quantity and quality. Ideally, for each of the five identified comparisons, more prospective, comparative monitoring studies nested in clinical trials and measuring effects on all outcomes specified in this review are necessary to draw more reliable conclusions. However, the results suggesting risk-based, targeted, and mainly central monitoring as an efficient strategy are promising. The development of reliable triggers for on-site visits is ongoing; different triggers might be used in different settings. More evidence on risk indicators that identify sites with problems or the prognostic value of triggers is needed to further optimize central monitoring strategies. In particular, approaches with an initial assessment of trial-specific risks that need to be closely monitored centrally during trial conduct with triggered on-site visits should be evaluated in future research.

Capturing Data in Rare Disease Registries to Support Regulatory Decision Making: A Survey Study Among Industry and Other Stakeholders

Introduction

In rare diseases, registry-based studies can be used to provide natural history data pre-approval and complement drug efficacy and/or safety knowledge post-approval.

Objective

The objective of this study was to investigate the opinion of stakeholders about key aspects of rare disease registries that are used to support regulatory decision making and to compare the responses of employees from industry to other stakeholders.

Methods

A web-based survey was used to gauge the importance of (1) common data elements (including safety outcomes), (2) data quality and (3) governance aspects that are generic across different rare diseases. The survey included 47 questions. The data were collected in the period April-October 2019.

Results

Seventy-three respondents completed ≥ 80% of the survey. Most of the respondents were from the industry (n = 42, 57%). For safety data, 31 (42%) respondents were in favour of collecting all adverse events. For data quality, the respondents found a level of 30% reasonable for source data verification. For missing data, a level of 20% was considered acceptable. Compared to responders from industry, the other stakeholders found it less relevant to share data with industry and found it less acceptable if the registry is financed by industry.

Conclusions

This study showed that the opinion towards data and governance is well aligned across parties, and issues of data and governance on their own should not pose a barrier to collaboration. This finding is supportive of the European Medicines Agency’s efforts to encourage stakeholders to work with existing registries when collecting data to support regulatory decision making.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40264-021-01081-z.

Verification of data in a nationwide transanal total mesorectal excision registry in China

BACKGROUND AND OBJECTIVES

Transanal total mesorectal excision is a surgical procedure for mid- and low rectal cancer. The Chinese TaTME Registry Collaborative is a nationwide database collecting information on patients who have undergone this procedure.

METHODS

Centers were invited by the registry committee to participate in a three-part data audit project: remote audits for data completeness and deviation values, onsite source verification of data accuracy, and an online survey of the characteristics of data managers.

RESULTS

Twenty-three tertiary centers participated in this project. The median case volume registered by the centers was 51 (interquartile range, 25-89). The overall data completeness for 30 verified variables was 89.1%. Eight centers achieved a high data completeness rate (>95%). The source data of eight centers were verified onsite. The overall accuracy rate was 90.4% (85.3%-97.6% across centers). Postoperative complications, mortality, and proximal/distal resection margin involvement were accurately reported in >95% of cases. The data completeness rate was higher if the data manager was a surgeon/surgical resident (94.2% vs. 84.8%, p = 0.045).

CONCLUSIONS

The completeness and accuracy of the data in the Chinese TaTME Registry Collaborative are acceptable. The quality of the data is highest when entered by colorectal surgeons and residents.

Dynamic methods for ongoing assessment of site-level risk in risk-based monitoring of clinical trials: A scoping review

Background/Aims

It is increasingly recognised that reliance on frequent site visits for monitoring clinical trials is inefficient. Regulators and trialists have recently encouraged more risk-based monitoring. Risk assessment should take place before a trial begins to define the overarching monitoring strategy. It can also be done on an ongoing basis, to target sites for monitoring activity. Various methods have been proposed for such prioritisation, often using terms like ‘central statistical monitoring’, ‘triggered monitoring’ or, as in the International Conference on Harmonization Good Clinical Practice guidance, ‘targeted on-site monitoring’. We conducted a scoping review to identify such methods, to establish if any were supported by adequate evidence to allow wider implementation, and to guide future developments in this field of research.

Methods

We used seven publication databases, two sets of methodological conference abstracts and an Internet search engine to identify methods for using centrally held trial data to assess site conduct during a trial. We included only reports in English, and excluded reports published before 1996 or not directly relevant to our research question. We used reference and citation searches to find additional relevant reports. We extracted data using a predefined template. We contacted authors to request additional information about included reports.

Results

We included 30 reports in our final dataset, of which 21 were peer-reviewed publications. In all, 20 reports described central statistical monitoring methods (of which 7 focussed on detection of fraud or misconduct) and 9 described triggered monitoring methods; 21 reports included some assessment of their methods’ effectiveness, typically exploring the methods’ characteristics using real trial data without known integrity issues. Of the 21 with some effectiveness assessment, most contained limited information about whether or not concerns identified through central monitoring constituted meaningful problems. Several reports demonstrated good classification ability based on more than one classification statistic, but never without caveats of unclear reporting or other classification statistics being low or unavailable. Some reports commented on cost savings from reduced on-site monitoring, but none gave detailed costings for the development and maintenance of central monitoring methods themselves.

Conclusion

Our review identified various proposed methods, some of which could be combined within the same trial. The apparent emphasis on fraud detection may not be proportionate in all trial settings. Despite some promising evidence and some self-justifying benefits for data cleaning activity, many proposed methods have limitations that may currently prevent their routine use for targeting trial monitoring activity. The implementation costs, or uncertainty about these, may also be a barrier. We make recommendations for how the evidence-base supporting these methods could be improved.

Assessing the potential for prevention or earlier detection of on-site monitoring findings from randomised controlled trials: Further analyses of findings from the prospective TEMPER triggered monitoring study

BACKGROUND/AIMS

Clinical trials should be designed and managed to minimise important errors with potential to compromise patient safety or data integrity, employ monitoring practices that detect and correct important errors quickly, and take robust action to prevent repetition. Regulators highlight the use of risk-based monitoring, making greater use of centralised monitoring and reducing reliance on centre visits. The TEMPER study was a prospective evaluation of triggered monitoring (a risk-based monitoring method), whereby centres are prioritised for visits based on central monitoring results. Conducted in three UK-based randomised cancer treatment trials of investigational medicine products with time-to-event outcomes, it found high levels of serious findings at triggered centre visits but also at visits to matched control centres that, based on central monitoring, were not of concern. Here, we report a detailed review of the serious findings from TEMPER centre visits. We sought to identify feasible, centralised processes which might detect or prevent these findings without a centre visit.

METHODS

The primary outcome of this study was the proportion of all 'major' and 'critical' TEMPER centre visit findings theoretically detectable or preventable through a feasible, centralised process. To devise processes, we considered a representative example of each finding type through an internal consensus exercise. This involved (a) agreeing the potential, by some described process, for each finding type to be centrally detected or prevented and (b) agreeing a proposed feasibility score for each proposed process. To further assess feasibility, we ran a consultation exercise, whereby the proposed processes were reviewed and rated for feasibility by invited external trialists.

RESULTS

In TEMPER, 312 major or critical findings were identified at 94 visits. These findings comprised 120 distinct issues, for which we proposed 56 different centralised processes. Following independent review of the feasibility of the proposed processes by 87 consultation respondents across eight different trial stakeholder groups, we conclude that 306/312 (98%) findings could theoretically be prevented or identified centrally. Of the processes deemed feasible, those relating to informed consent could have the most impact. Of processes not currently deemed feasible, those involving use of electronic health records are among those with the largest potential benefit.

CONCLUSIONS

This work presents a best-case scenario, where a large majority of monitoring findings were deemed theoretically preventable or detectable by central processes. Caveats include the cost of applying all necessary methods, and the resource implications of enhanced central monitoring for both centre and trials unit staff. Our results will inform future monitoring plans and emphasise the importance of continued critical review of monitoring processes and outcomes to ensure they remain appropriate.

Time-consuming and expensive data quality monitoring procedures persist in clinical trials: A national survey

INTRODUCTION

The Good Clinical Practice guideline identifies that data monitoring is an essential research activity. However, limited evidence exists on how to perform monitoring including the amount or frequency that is needed to ensure data quality. This study aims to explore the monitoring procedures that are implemented to ensure data quality in Australian clinical research studies.

MATERIAL AND METHODS

Clinical studies listed on the Australian and New Zealand Clinical Trials Registry were invited to participate in a national cross-sectional, mixed-mode, multi-contact (postal letter and e-mail) web-based survey. Information was gathered about the types of data quality monitoring procedures being implemented.

RESULTS

Of the 3689 clinical studies contacted, 589 (16.0%) responded, of which 441 (77.4%) completed the survey. Over half (55%) of the studies applied source data verification (SDV) compared to risk-based targeted and triggered monitoring (10-11%). Conducting 100% on-site monitoring was most common for those who implemented the traditional approach. Respondents who did not conduct 100% monitoring, included 1-25% of data points for SDV, centralized or on-site monitoring. The incidence of adverse events and protocol deviations were the most likely factors to trigger a site visit for risk-based triggered (63% and 44%) and centralized monitoring (48% and 44%), respectively.

CONCLUSION

Instead of using more optimal risk-based approaches, small single-site clinical studies are conducting traditional monitoring procedures which are time consuming and expensive. Formal guidelines need to be improved and provided to all researchers for 'new' risk-based monitoring approaches.

Clinical trial monitoring effectiveness: Remote risk-based monitoring versus on-site monitoring with 100% source data verification

BACKGROUND/AIMS

Traditional on-site monitoring of clinical trials via frequent site visits and 100% source data verification is cost-consuming, and it still cannot guarantee data quality effectively. Depending on the types and designs of clinical trials, an alternative would be combining several monitoring methods, such as risk-based monitoring and remote monitoring. However, there is insufficient evidence of its effectiveness. This research compared the effectiveness of risk-based monitoring with a remote monitoring system with that of traditional on-site monitoring.

METHODS

With a cloud-based remote monitoring system called beagle View^®, we created a remote risk-based monitoring methodology that focused only on critical data and processes. We selected a randomized controlled trial conducted at Tohoku University Hospital and randomly sampled 11 subjects whose case report forms had already been reviewed by data managers. Critical data and processes were verified retrospectively by remote risk-based monitoring; later, all data and processes were confirmed by on-site monitoring. We compared the ability of remote risk-based monitoring to detect critical data and process errors with that of on-site monitoring with 100% source data verification, including an examination of clinical trial staff workload and potential cost savings.

RESULTS

Of the total data points (n = 5617), 19.7% (n = 1105, 95% confidence interval = 18.7-20.7) were identified as critical. The error rates of critical data detected by on-site monitoring, remote risk-based monitoring, and data review by data managers were 7.6% (n = 84, 95% CI = 6.2-9.3), 7.6% (n = 84, 95% confidence interval = 6.2-9.3), and 3.9% (n = 43, 95% confidence interval = 2.9-5.2), respectively. The total number of critical process errors detected by on-site monitoring was 14. Of these 14, 92.9% (n = 13, 95% confidence interval = 68.5-98.7) and 42.9% (n = 6, 95% confidence interval = 21.4-67.4) of critical process errors were detected by remote risk-based monitoring and data review by data managers, respectively. The mean time clinical trial staff spent dealing with remote risk-based monitoring was 9.9 ± 5.3 (mean ± SD) min per visit per subject. Our calculations show that remote risk-based monitoring saved between 9 and 41 on-site monitoring visits, corresponding to a cost of between US$13,500 and US$61,500 per trial site.

CONCLUSION

Remote risk-based monitoring was able to detect critical data and process errors as well as on-site monitoring with 100% source data verification, saving travel time and monitoring costs. Remote risk-based monitoring offers an effective alternative to traditional on-site monitoring of clinical trials.

Monitoring in clinical trials of complementary and alternative medicine

Background

Clinical trial monitoring is an essential activity for quality assurance (QA) to ensure the protection of human rights and the reliability and transparency of the data collection process. The purpose of this article is to enhance the understanding of monitoring process and major findings in clinical trials of complementary and alternative medicine (CAM).

Methods

Based on International Conference on Harmonization of technical requirements for registration of pharmaceuticals for human use (ICH-GCP), we summarized main concept of monitoring process. Personal experiences on monitoring for CAM studies were also narratively described.

Results

In this brief article, the basic concept of QA and quality control (QC), various monitoring activities during the study process, and major findings regarding clinical trials of CAM are suggested in an effort to improve understanding of monitoring in clinical research on CAM.

Conclusion

When performing clinical trials for CAM-related interventions, the monitoring recommended in GCP is needed to be recognized as a mandatory element in the course of CAM research.

Improving Study Conduct and Data Quality in Clinical Trials of Chronic Pain Treatments: IMMPACT Recommendations

The estimated probability of progressing from phase 3 analgesic clinical trials to regulatory approval is approximately 57%, suggesting that a considerable number of treatments with phase 2 trial results deemed sufficiently successful to progress to phase 3 do not yield positive phase 3 results. Deficiencies in the quality of clinical trial conduct could account for some of this failure. An Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials meeting was convened to identify potential areas for improvement in trial conduct in order to improve assay sensitivity (ie, ability of trials to detect a true treatment effect). We present recommendations based on presentations and discussions at the meeting, literature reviews, and iterative revisions of this article. The recommendations relate to the following areas: 1) study design (ie, to promote feasibility), 2) site selection and staff training, 3) participant selection and training, 4) treatment adherence, 5) data collection, and 6) data and study monitoring. Implementation of these recommendations may improve the quality of clinical trial data and thus the validity and assay sensitivity of clinical trials. Future research regarding the effects of these strategies will help identify the most efficient use of resources for conducting high quality clinical trials. PERSPECTIVE: Every effort should be made to optimize the quality of clinical trial data. This manuscript discusses considerations to improve conduct of pain clinical trials based on research in multiple medical fields and the expert consensus of pain researchers and stakeholders from academia, regulatory agencies, and industry.

Feasibility of a Hybrid Risk-Adapted Monitoring System in Investigator-Sponsored Trials in Cancer

BACKGROUND

We assessed the feasibility of a hybrid monitoring system (minimal on-site monitoring + strategic central monitoring) used at the academic research office at Asan Medical Center (Seoul, Korea) in monitoring investigator-sponsored oncology trials.

METHODS

Monitoring findings in three oncology trials conducted between 2014 and 2017 were compared. A confirmatory source data verification (SDV) was carried out in the low-risk trial and compared with the central monitoring findings. The economic advantages of central monitoring were tested by calculating the monitoring hours per patient.

RESULTS

A total of 50, 118, 228 patients were enrolled in the high-, intermediate-, and low-risk trials, respectively. The high-risk trial was monitored through 42 on-site visits (1299 findings); the intermediate-risk trial had 79 monitorings (on-site, 24%; central, 76%; 1464 findings); the low-risk trial had 197 monitorings (on-site, 4%; central, 96%; 3364 findings). Central monitoring was more effective than on-site monitoring in revealing minor errors such as "missing case report forms" and "data outliers" (both P < 0.0001), and showed comparable results in revealing major issues such as investigational product compliance and delayed reporting of serious adverse events (both P > 0.05). Confirmatory SDV in the low-risk trial revealed more findings than central monitoring in the "inconsistent data" and "inappropriate adverse event" categories. The total monitoring hours per patient were lower in the intermediate- and low-risk trials than in the high-risk trial (8.1 and 7.3 vs. 14.3 h, respectively).

CONCLUSION

Our hybrid monitoring system showed acceptable feasibility in revealing both major and minor issues in multi-center oncology investigator-sponsored trials.

Data Validation and Verification Using Blockchain in a Clinical Trial for Breast Cancer: Regulatory Sandbox

Background

The integrity of data in a clinical trial is essential, but the current data management process is too complex and highly labor-intensive. As a result, clinical trials are prone to consuming a lot of budget and time, and there is a risk for human-induced error and data falsification. Blockchain technology has the potential to address some of these challenges.

Objective

The aim of the study was to validate a system that enables the security of medical data in a clinical trial using blockchain technology.

Methods

We have developed a blockchain-based data management system for clinical trials and tested the system through a clinical trial for breast cancer. The project was conducted to demonstrate clinical data management using blockchain technology under the regulatory sandbox enabled by the Japanese Cabinet Office.

Results

We verified and validated the data in the clinical trial using the validation protocol and tested its resilience to data tampering. The robustness of the system was also proven by survival with zero downtime for clinical data registration during a Amazon Web Services disruption event in the Tokyo region on August 23, 2019.

Conclusions

We show that our system can improve clinical trial data management, enhance trust in the clinical research process, and ease regulator burden. The system will contribute to the sustainability of health care services through the optimization of cost for clinical trials.

Bayesian central statistical monitoring using finite mixture models in multicenter clinical trials

Background

Central monitoring (CM), in which data across all clinical sites are monitored, has an important role in risk-based monitoring. Several statistical methods have been proposed to compare patient outcomes among the sites for detecting atypical sites that have different trends in observed data. These methods assume that the number of clinical sites is not small, e.g., 100 or more. In addition, the proportion of atypical sites is assumed to be relatively small. However, in actuality, the central statistical monitoring (CSM) has to be implemented in small or moderate sized clinical trials such as small phase II clinical trials. The number of sites is no longer large in such situations. Therefore, it is of concern that existing methods may not work efficiently in CM of small or moderate sized clinical trials. In the light of this problem, we propose a Bayesian CSM method to detect atypical sites as the robust method against the existence of atypical sites.

Methods

We use Bayesian finite mixture models (FMM) to model patient outcome values of both atypical and typical sites. In the method, the distributions of outcome values in normal sites are determined by choosing the body distribution, which has the largest mixture parameter value of finite mixture models based on the assumption that normal sites are in the majority. Atypical sites are detected by the criterion based on the posterior predictive distribution of normal site's outcome values derived from only the chosen body distribution.

Results

Proposed method is evaluated by cumulative detection probability and type I error averaged over sites every round of CSM under the various scenarios, being compared with the conventional type analysis. If the total number of patients enrolled is 48, the proposed method is superior at least 10% for any shift sizes at the 2nd and the 3rd rounds. If the total number of patients is 96, both methods show similar detection probability for only one atypical site and large shift size. However, the proposed method is superior for the other scenarios. It is observed that all the type I errors averaged over sites are little difference between the methods at all the scenarios.

Conclusion

We propose a Bayesian CSM method which works efficiently in a practical use of CM. It is shown that our method detects atypical sites with high probability regardless of the proportion of the atypical sites under the small clinical trial settings which is the target of our proposed method.

A randomized evaluation of on-site monitoring nested in a multinational randomized trial

BACKGROUND

Evidence from prospectively designed studies to guide on-site monitoring practices for randomized trials is limited. A cluster randomized study, nested within the Strategic Timing of AntiRetroviral Treatment (START) trial, was conducted to evaluate on-site monitoring.

METHODS

Sites were randomized to either annual on-site monitoring or no on-site monitoring. All sites were centrally monitored, and local monitoring was carried out twice each year. Randomization was stratified by country and projected enrollment in START. The primary outcome was a participant-level composite outcome including components for eligibility errors, consent violations, use of antiretroviral treatment not recommended by protocol, late reporting of START primary and secondary clinical endpoints (defined as the event being reported more than 6 months from occurrence), and data alteration and fraud. Logistic regression fixed effect hierarchical models were used to compare on-site versus no on-site monitoring for the primary composite outcome and its components. Odds ratios and 95% confidence intervals comparing on-site monitoring versus no on-site monitoring are cited.

RESULTS

In total, 99 sites (2107 participants) were randomized to receive annual on-site monitoring and 97 sites (2264 participants) were randomized to be monitored only centrally and locally. The two monitoring groups were well balanced at entry. In the on-site monitoring group, 469 annual on-site monitoring visits were conducted, and 134 participants (6.4%) in 56 of 99 sites (57%) had a primary monitoring outcome. In the no on-site monitoring group, 85 participants (3.8%) in 34 of 97 sites (35%) had a primary monitoring outcome (odds ratio = 1.7; 95% confidence interval: 1.1-2.7; p = 0.03). Informed consent violations accounted for most outcomes in each group (56 vs 41 participants). The largest odds ratio was for eligibility violations (odds ratio = 12.2; 95% confidence interval: 1.8-85.2; p = 0.01). The number of participants with a late START primary endpoint was similar for each monitoring group (23 vs 16 participants). Late START grade 4 and unscheduled hospitalization events were found for 34 participants in the on-site monitoring group and 19 participants in the no on-site monitoring group (odds ratio = 2.0; 95% confidence interval: 1.1-3.7; p = 0.02). There were no cases of data alteration or fraud. Based on the travel budget for on-site monitoring and the hours spent conducting on-site monitoring, the estimated cost of on-site monitoring was over US$2 million.

CONCLUSION

On-site monitoring led to the identification of more eligibility and consent violations and START clinical events being reported more than 6 months from occurrence as compared to no on-site monitoring. Considering the nature of the excess monitoring outcomes identified at sites receiving on-site monitoring, as well as the cost of on-site monitoring, the value to the START study was limited.

Impact of a targeted monitoring on data-quality and data-management workload of randomized controlled trials: A prospective comparative study

AIMS

Monitoring risk-based approaches in clinical trials are encouraged by regulatory guidance. However, the impact of a targeted source data verification (SDV) on data-management (DM) workload and on final data quality needs to be addressed.

METHODS

MONITORING was a prospective study aiming at comparing full SDV (100% of data verified for all patients) and targeted SDV (only key data verified for all patients) followed by the same DM program (detecting missing data and checking consistency) on final data quality, global workload and staffing costs.

RESULTS

In all, 137 008 data including 18 124 key data were collected for 126 patients from 6 clinical trials. Compared to the final database obtained using the full SDV monitoring process, the final database obtained using the targeted SDV monitoring process had a residual error rate of 1.47% (95% confidence interval, 1.41-1.53%) on overall data and 0.78% (95% confidence interval, 0.65-0.91%) on key data. There were nearly 4 times more queries per study with targeted SDV than with full SDV (mean ± standard deviation: 132 ± 101 vs 34 ± 26; P = .03). For a handling time of 15 minutes per query, the global workload of the targeted SDV monitoring strategy remained below that of the full SDV monitoring strategy. From 25 minutes per query it was above, increasing progressively to represent a 50% increase for 45 minutes per query.

CONCLUSION

Targeted SDV monitoring is accompanied by increased workload for DM, which allows to obtain a small proportion of remaining errors on key data (<1%), but may substantially increase trial costs.

Triggered or routine site monitoring visits for randomised controlled trials: results of TEMPER, a prospective, matched-pair study

BACKGROUND/AIMS

In multi-site clinical trials, where trial data and conduct are scrutinised centrally with pre-specified triggers for visits to sites, targeted monitoring may be an efficient way to prioritise on-site monitoring. This approach is widely used in academic trials, but has never been formally evaluated.

METHODS

TEMPER assessed the ability of targeted monitoring, as used in three ongoing phase III randomised multi-site oncology trials, to distinguish sites at which higher and lower rates of protocol and/or Good Clinical Practice violations would be found during site visits. Using a prospective, matched-pair design, sites that had been prioritised for visits after having activated 'triggers' were matched with a control ('untriggered') site, which would not usually have been visited at that time. The paired sites were visited within 4 weeks of each other, and visit findings are recorded and categorised according to the seriousness of the deviation. The primary outcome measure was the proportion of sites with ≥1 'Major' or 'Critical' finding not previously identified centrally. The study was powered to detect an absolute difference of ≥30% between triggered and untriggered visits. A sensitivity analysis, recommended by the study's blinded endpoint review committee, excluded findings related to re-consent. Additional analyses assessed the prognostic value of individual triggers and data from pre-visit questionnaires completed by site and trials unit staff.

RESULTS

In total, 42 matched pairs of visits took place between 2013 and 2016. In the primary analysis, 88.1% of triggered visits had ≥1 new Major/Critical finding, compared to 81.0% of untriggered visits, an absolute difference of 7.1% (95% confidence interval -8.3%, +22.5%; p = 0.365). When re-consent findings were excluded, these figures reduced to 85.7% versus 59.5%, (difference = 26.2%, 95% confidence interval 8.0%, 44.4%; p = 0.007). Individual triggers had modest prognostic value but knowledge of the trial-related activities carried out by site staff may be useful.

CONCLUSION

Triggered monitoring approaches, as used in these trials, were not sufficiently discriminatory. The rate of Major and Critical findings was higher than anticipated, but the majority related to consent and re-consent with no indication of systemic problems that would impact trial-wide safety issues or integrity of the results in any of the three trials. Sensitivity analyses suggest triggered monitoring may be of potential use, but needs improvement and investigation of further central monitoring triggers is warranted. TEMPER highlights the need to question and evaluate methods in trial conduct, and should inform further developments in this area.

Evolving quality standards for large-scale registries: the GARFIELD-AF experience

Aims

Registries have the potential to capture treatment practices and outcomes in populations beyond the constraints of clinical trial settings. The value of data obtained depend critically upon robust quality standards (including source data verification [SDV] and training); features that are commonly absent from registries. This article outlines the quality standards developed for Global Anticoagulant Registry in the FIELD-Atrial Fibrillation (GARFIELD-AF).

Methods and Results

GARFIELD-AF comprises ∼57 000 patients prospectively recruited over 6.5 years in 35 countries in five successive cohorts. The registry employs a combination of remote and onsite monitoring to ascertain completeness and accuracy of records and by design, SDV is performed on 20% of cases (i.e. ∼11 400 patients). Four performance measures for ranking sites according to data quality and other performance indicators were evaluated (including data quality for 13 quantifiable variables, late data locking, number of missing critical variables, and history of poor data quality from the previous monitoring phase). These criteria facilitated the identification of sites with potentially suboptimal data quality for onsite monitoring. During early phases of the registry, critical variables for data checking were also identified. SDV using these variables (partial SDV in 902 patients) showed similar concordance to SDV of all fields (110 patients): 94.4% vs. 93.1%, respectively. This standard formed the baseline against which ongoing quality improvements were assessed, facilitating corrective action on data quality issues. In consequence, concordance was improved in the next monitoring phase (95.6%; n = 1172).

Conclusion

The quality standards in GARFIELD-AF have the potential to inform a future 'reference' for registries.

Exploring Data Quality Management within Clinical Trials

BACKGROUND

Clinical trials are an important research method for improving medical knowledge and patient care. Multiple international and national guidelines stipulate the need for data quality and assurance. Many strategies and interventions are developed to reduce error in trials, including standard operating procedures, personnel training, data monitoring, and design of case report forms. However, guidelines are nonspecific in the nature and extent of necessary methods.

OBJECTIVE

This article gathers information about current data quality tools and procedures used within Australian clinical trial sites, with the aim to develop standard data quality monitoring procedures to ensure data integrity.

METHODS

Relevant information about data quality management methods and procedures, error levels, data monitoring, staff training, and development were collected. Staff members from 142 clinical trials listed on the National Health and Medical Research Council (NHMRC) clinical trials Web site were invited to complete a short self-reported semiquantitative anonymous online survey.

RESULTS

Twenty (14%) clinical trials completed the survey. Results from the survey indicate that procedures to ensure data quality varies among clinical trial sites. Centralized monitoring (65%) was the most common procedure to ensure high-quality data. Ten (50%) trials reported having a data management plan in place and two sites utilized an error acceptance level to minimize discrepancy, set at <5% and 5 to 10%, respectively. The quantity of data variables checked (10-100%), the frequency of visits (once-a-month to annually), and types of variables (100%, critical data or critical and noncritical data audits) for data monitoring varied among respondents. The average time spent on staff training per person was 11.58 hours over a 12-month period and the type of training was diverse.

CONCLUSION

Clinical trial sites are implementing ad hoc methods pragmatically to ensure data quality. Findings highlight the necessity for further research into "standard practice" focusing on developing and implementing publically available data quality monitoring procedures.

The changing face of clinical trials in the personalized medicine and immuno-oncology era: report from the international congress on clinical trials in Oncology & Hemato-Oncology (ICTO 2017)

In the past decade, the oncology community has witnessed major advances in the understanding of cancer biology and major breakthroughs in several different therapeutic areas, from solid tumors to hematological malignancies; moreover, the advent of effective immunotherapy approaches, such as immune-checkpoint blockade, is revolutionizing treatment algorithms in almost all oncology disease areas. As knowledge evolves and new weapons emerge in the “war against cancer”, clinical and translational research need to adapt to a rapidly changing environment to effectively translate novel concepts into sustainable and accessible therapeutic options for cancer patients.

With this in mind, translational cancer researchers, oncology professionals, treatment experts, CRO and industry leaders, as well as patient representatives gathered in London, 16-17 March 2017, for The International Congress on Clinical Trials in Oncology and Hemato-Oncology (ICTO2017), to discuss the changing face of oncology clinical trials in the new era of personalized medicine and immuno-oncology. A wide range of topics, including clinical trial design in immuno-oncology, biomarker-oriented drug development paths, statistical design and endpoint selection, challenges in the design and conduct of personalized medicine clinical trials, risk-based monitoring, financing and reimbursement, as well as best operational practices, were discussed in an open, highly interactive format, favoring networking among all relevant stakeholders. The most relevant data, approaches and issues emerged and discussed during the conference are summarized in this report.

An evaluation of the effectiveness of a risk-based monitoring approach implemented with clinical trials involving implantable cardiac medical devices

Background

Risk-based monitoring is a concept endorsed by the Food and Drug Administration to improve clinical trial data quality by focusing monitoring efforts on critical data elements and higher risk investigator sites. BIOTRONIK approached this by implementing a comprehensive strategy that assesses risk and data quality through a combination of operational controls and data surveillance. This publication demonstrates the effectiveness of a data-driven risk assessment methodology when used in conjunction with a tailored monitoring plan.

Methods

We developed a data-driven risk assessment system to rank 133 investigator sites comprising 3442 subjects and identify those sites that pose a potential risk to the integrity of data collected in implantable cardiac device clinical trials. This included identification of specific risk factors and a weighted scoring mechanism. We conducted trend analyses for risk assessment data collected over 1 year to assess the overall impact of our data surveillance process combined with other operational monitoring efforts.

Results

Trending analyses of key risk factors revealed an improvement in the quality of data collected during the observation period. The three risk factors follow-up compliance rate, unavailability of critical data, and noncompliance rate correspond closely with Food and Drug Administration’s risk-based monitoring guidance document. Among these three risk factors, 100% (12/12) of quantiles analyzed showed an increase in data quality. Of these, 67% (8/12) of the improving trends in worst performing quantiles had p-values less than 0.05, and 17% (2/12) had p-values between 0.05 and 0.06. Among the poorest performing site quantiles, there was a statistically significant decrease in subject follow-up noncompliance rates, protocol noncompliance rates, and incidence of missing critical data.

Conclusion

One year after implementation of a comprehensive strategy for risk-based monitoring, including a data-driven risk assessment methodology to target on-site monitoring visits, statistically significant improvement was seen in a majority of measurable risk factors at the worst performing site quantiles. For the three risk factors which are most critical to the overall compliance of cardiac rhythm management medical device studies: follow-up compliance rate, unavailability of critical data, and noncompliance rate, we measured significant improvement in data quality. Although the worst performing site quantiles improved but not significantly in some risk factors such as subject attrition, the data-driven risk assessment highlighted key areas on which to continue focusing both on-site and centralized monitoring efforts. Data-driven surveillance of clinical trial performance provides actionable observations that can improve site performance. Clinical trials utilizing risk-based monitoring by leveraging a data-driven quality assessment combined with specific operational procedures may lead to an improvement in data quality and resource efficiencies.

Generating evidence on a risk-based monitoring approach in the academic setting - lessons learned

BACKGROUND

In spite of efforts to employ risk-based strategies to increase monitoring efficiency in the academic setting, empirical evidence on their effectiveness remains sparse. This mixed-methods study aimed to evaluate the risk-based on-site monitoring approach currently followed at our academic institution.

METHODS

We selected all studies monitored by the Clinical Trial Unit (CTU) according to Risk ADApted MONitoring (ADAMON) at the University Hospital Basel, Switzerland, between 01.01.2012 and 31.12.2014. We extracted study characteristics and monitoring information from the CTU Enterprise Resource Management system and from monitoring reports of all selected studies. We summarized the data descriptively. Additionally, we conducted semi-structured interviews with the three current CTU monitors.

RESULTS

During the observation period, a total of 214 monitoring visits were conducted in 43 studies resulting in 2961 documented monitoring findings. Our risk-based approach predominantly identified administrative (46.2%) and patient right findings (49.1%). We identified observational study design, high ADAMON risk category, industry sponsorship, the presence of an electronic database, experienced site staff, and inclusion of vulnerable study population to be factors associated with lower numbers of findings. The monitors understand the positive aspects of a risk-based approach but fear missing systematic errors due to the low frequency of visits.

CONCLUSIONS

We show that the factors mostly increasing the risk for on-site monitoring findings are underrepresented in the current risk analysis scheme. Our risk-based on-site approach should further be complemented by centralized data checks, allowing monitors to transform their role towards partners for overall trial quality, and success.