A statistical approach to central monitoring of data quality in clinical trials

Assessing the performance of methods for central statistical monitoring of a binary or continuous outcome in multi-center trials: A simulation study

BACKGROUND

Quality study monitoring is fundamental to patient safety and data integrity. Regulators and industry consortia have increasingly advocated for risk-based monitoring (RBM) and central statistical monitoring (CSM) for more effective and efficient monitoring. Assessing which statistical methods underpin these approaches can best identify unusual data patterns in multi-center clinical trials that may be driven by potential systematic errors is important.

METHODS

We assessed various CSM techniques, including cross-tests, fixed-effects, mixed-effects, and finite mixture models, across scenarios with different sample sizes, contamination rates, and overdispersion via simulation. Our evaluation utilized threshold-independent metrics such as the area under the curve (AUC) and average precision (AP), offering a fuller picture of CSM performance.

RESULTS

All CSM methods showed consistent characteristics across center sizes or overdispersion. The adaptive finite mixture model outperformed others in AUC and AP, especially at 30% contamination, upholding high specificity unless converging to a single-component model due to low contamination or deviation. The mixed-effects model performed well at lower contamination rates. However, it became conservative in specificity and exhibited declined performance for binary outcomes under high deviation. Cross-tests and fixed-effects methods underperformed, especially when deviation increased.

CONCLUSION

Our evaluation explored the merits and drawbacks of multiple CSM methods, and found that relying on sensitivity and specificity alone is likely insufficient to fully measure predictive performance. The finite mixture method demonstrated more consistent performance across scenarios by mitigating the influence of outliers. In practice, considering the study-specific costs of false positives/negatives with available resources for monitoring is important.

Practical Guidelines for Standardised Resolution of Important Protocol Deviations in Clinical Trials Conducted in Sub-Saharan Africa

A clinical trial is any research on human subjects that involves an investigational medicinal product or device. Investigational medicinal products include unlicensed drugs or drugs used outside the product license (e.g. for a new indication) (ICH-GCP). As per the internationally accepted ICH-GCP guidelines, clinical trials should be conducted strictly per the approved protocol. However, during the lifecycle of a trial, protocol deviations may occur. Under ICH efficacy guidelines, protocol deviations are divided into non-important (minor) or important (major), and the latter can jeopardise the participant's rights, safety or the quality of data generated by the study. Existing guidelines on protocol deviation management do not detail or standardise actions to be taken for participants, investigational products, data or samples as part of a holistic management of important protocol deviations. Herein, we propose guidelines to address the current literature gap and promote the standardisation of actions to address important protocol deviations in clinical trials. The advised actions should complement the existing local institutional review board and national regulatory authority requirements.

Does Central Statistical Monitoring Improve Data Quality? An Analysis of 1,111 Sites in 159 Clinical Trials

BACKGROUND

Central monitoring aims at improving the quality of clinical research by pro-actively identifying risks and remediating emerging issues in the conduct of a clinical trial that may have an adverse impact on patient safety and/or the reliability of trial results. This paper, focusing on statistical data monitoring (SDM), is the second of a series that attempts to quantify the impact of central monitoring in clinical trials.

MATERIAL AND METHODS

Quality improvement was assessed in studies using SDM from a single large central monitoring platform. The analysis focused on a total of 1111 sites that were identified as at-risk by the SDM tests and for which the study teams conducted a follow-up investigation. These sites were taken from 159 studies conducted by 23 different clinical development organizations (including both sponsor companies and contract research organizations). Two quality improvement metrics were assessed for each selected site, one based on a site data inconsistency score (DIS, overall -log10 P-value of the site compared with all other sites) and the other based on the observed metric value associated with each risk signal.

RESULTS

The SDM quality metrics showed improvement in 83% (95% CI, 80-85%) of the sites across therapeutic areas and study phases (primarily phases 2 and 3). In contrast, only 56% (95% CI, 41-70%) of sites showed improvement in 2 historical studies that did not use SDM during study conduct.

CONCLUSION

The results of this analysis provide clear quantitative evidence supporting the hypothesis that the use of SDM in central monitoring is leading to improved quality in clinical trial conduct and associated data across participating sites.

A code for clinical trials centralized monitoring, sharing open-science solutions to high-quality data

Monitoring of clinical trials is critical to the protection of human subjects and the conduct of high-quality research. Even though the adoption of risk-based monitoring (RBM) has been suggested for many years, the RBM approach has been less widespread than expected. Centralized monitoring is one of the RMB pillars, together with remote-site monitoring visits, reduced Source Data Verification (SDV) and Source Document Reviews (SDR). The COVID-19 pandemic promoted disruptions in the conduction of clinical trials, as on-site monitoring visits were adjourned. In this context, the transition to RBM by all actors involved in clinical trials has been encouraged. In order to ensure the highest quality of data within a COVID-19 clinical trial, a centralized monitoring tool alongside Case Report Forms (CRFs) and synchronous automated routines were developed at the clinical research platform, Fiocruz, Brazilian Ministry of Health. This paper describes how these tools were developed, their features, advantages, and limitations. The software codes, and the CRFs are available at the Fiocruz Data Repository for Research-Arca Dados, reaffirming Fiocruz's commitment to Open Science practices.

Application of Statistical Methods for Central Statistical Monitoring and Implementations on the German Multiple Sclerosis Registry

Monitoring of clinical trials is a fundamental process required by regulatory agencies. It assures the compliance of a center to the required regulations and the trial protocol. Traditionally, monitoring teams relied on extensive on-site visits and source data verification. However, this is costly, and the outcome is limited. Thus, central statistical monitoring (CSM) is an additional approach recently embraced by the International Council for Harmonisation (ICH) to detect problematic or erroneous data by using visualizations and statistical control measures. Existing implementations have been primarily focused on detecting inlier and outlier data. Other approaches include principal component analysis and distribution of the data. Here we focus on the utilization of comparisons of centers to the Grand mean for different model types and assumptions for common data types, such as binomial, ordinal, and continuous response variables. We implement the usage of multiple comparisons of single centers to the Grand mean of all centers. This approach is also available for various non-normal data types that are abundant in clinical trials. Further, using confidence intervals, an assessment of equivalence to the Grand mean can be applied. In a Monte Carlo simulation study, the applied statistical approaches have been investigated for their ability to control type I error and the assessment of their respective power for balanced and unbalanced designs which are common in registry data and clinical trials. Data from the German Multiple Sclerosis Registry (GMSR) including proportions of missing data, adverse events and disease severity scores were used to verify the results on Real-World-Data (RWD).

Performance of four centralized statistical monitoring methods for early detection of an atypical center in a multicenter study

Background

Ensuring the quality of data is essential for the credibility of a multicenter clinical trial. Centralized Statistical Monitoring (CSM) of data allows the detection of a center in which the distribution of a specific variable is atypical compared to other centers. The ideal CSM method should allow early detection of problem and therefore involve the fewest possible participants.

Methods

We simulated clinical trials and compared the performance of four CSM methods (Student, Hatayama, Desmet, Distance) to detect whether the distribution of a quantitative variable was atypical in one center in relation to the others, with different numbers of participants and different mean deviation amplitudes.

Results

The Student and Hatayama methods had good sensitivity but poor specificity, which disqualifies them for practical use in CSM. The Desmet and Distance methods had very high specificity for detecting all the mean deviations tested (including small values) but low sensitivity with mean deviations less than 50%.

Conclusion

Although the Student and Hatayama methods are more sensitive, their low specificity would lead to too many alerts being triggered, which would result in additional unnecessary control work to ensure data quality. The Desmet and Distance methods have low sensitivity when the deviation from the mean is low, suggesting that the CSM should be used alongside other conventional monitoring procedures rather than replacing them. However, they have excellent specificity, which suggests they can be applied routinely, since using them takes up no time at central level and does not cause any unnecessary workload in investigating centers.

Guidance on interim analysis methods in clinical trials

Interim analyses in clinical trials can take on a multitude of forms. They are often used to guide Data and Safety Monitoring Board (DSMB) recommendations to study teams regarding recruitment targets for large, later-phase clinical trials. As collaborative biostatisticians working and teaching in multiple fields of research and across a broad array of trial phases, we note the large heterogeneity and confusion surrounding interim analyses in clinical trials. Thus, in this paper, we aim to provide a general overview and guidance on interim analyses for a nonstatistical audience. We explain each of the following types of interim analyses: efficacy, futility, safety, and sample size re-estimation, and we provide the reader with reasoning, examples, and implications for each. We emphasize that while the types of interim analyses employed may differ depending on the nature of the study, we would always recommend prespecification of the interim analytic plan to the extent possible with risk mitigation and trial integrity remaining a priority. Finally, we posit that interim analyses should be used as tools to help the DSMB make informed decisions in the context of the overarching study. They should generally not be deemed binding, and they should not be reviewed in isolation.

Development of a risk-tailored approach and dashboard for efficient management and monitoring of investigator-initiated trials

BACKGROUND

Most randomized controlled trials (RCTs) in the academic setting have limited resources for clinical trial management and monitoring. Inefficient conduct of trials was identified as an important source of waste even in well-designed studies. Thoroughly identifying trial-specific risks to enable focussing of monitoring and management efforts on these critical areas during trial conduct may allow for the timely initiation of corrective action and to improve the efficiency of trial conduct. We developed a risk-tailored approach with an initial risk assessment of an individual trial that informs the compilation of monitoring and management procedures in a trial dashboard.

METHODS

We performed a literature review to identify risk indicators and trial monitoring approaches followed by a contextual analysis involving local, national and international stakeholders. Based on this work we developed a risk-tailored management approach with integrated monitoring for RCTs and including a visualizing trial dashboard. We piloted the approach and refined it in an iterative process based on feedback from stakeholders and performed formal user testing with investigators and staff of two clinical trials.

RESULTS

The developed risk assessment comprises four domains (patient safety and rights, overall trial management, intervention management, trial data). An accompanying manual provides rationales and detailed instructions for the risk assessment. We programmed two trial dashboards tailored to one medical and one surgical RCT to manage identified trial risks based on daily exports of accumulating trial data. We made the code for a generic dashboard available on GitHub that can be adapted to individual trials.

CONCLUSIONS

The presented trial management approach with integrated monitoring enables user-friendly, continuous checking of critical elements of trial conduct to support trial teams in the academic setting. Further work is needed in order to show effectiveness of the dashboard in terms of safe trial conduct and successful completion of clinical trials.

Lack of transparent reporting of trial monitoring approaches in randomised controlled trials: A systematic review of contemporary protocol papers

BACKGROUND

Monitoring is essential to ensure patient safety and data integrity in clinical trials as per Good Clinical Practice. The Standard Protocol Items: Recommendations for Interventional Trials Statement and its checklist guides authors to include monitoring in their protocols. We investigated how well monitoring was reported in published 'protocol papers' for contemporary randomised controlled trials.

METHODS

A systematic search was conducted in PubMed to identify eligible protocol papers published in selected journals between 1 January 2020 and 31 May 2020. Protocol papers were classified by whether they reported monitoring and, if so, by the details of monitoring. Data were summarised descriptively.

RESULTS

Of 811 protocol papers for randomised controlled trials, 386 (48%; 95% CI: 44%-51%) explicitly reported some monitoring information. Of these, 20% (77/386) reported monitoring information consistent with an on-site monitoring approach, and 39% (152/386) with central monitoring, 26% (101/386) with a mixed approach, while 14% (54/386) did not provide sufficient information to specify an approach. Only 8% (30/386) of randomised controlled trials reported complete details about all of scope, frequency and organisation of monitoring; frequency of monitoring was the least reported. However, 6% (25/386) of papers used the term 'audit' to describe 'monitoring'.

DISCUSSION

Monitoring information was reported in only approximately half of the protocol papers. Suboptimal reporting of monitoring hinders the clinical community from having the full information on which to judge the validity of a trial and jeopardises the value of protocol papers and the credibility of the trial itself. Greater efforts are needed to promote the transparent reporting of monitoring to journal editors and authors.

Interactive statistical monitoring to optimize review of potential clinical trial issues during study conduct

Background

Statistical monitoring involves the review of prospective study data collected in participating sites to detect intra/inter patients and sites inconsistencies. We report methods and results of statistical monitoring in a phase IV clinical trial.

Method

PRO-MSACTIVE is a study evaluating ocrelizumab in active relapsing multiple sclerosis (RMS) patients in France. Specific statistical methods (volcano plots, mahalanobis distance, funnel plot …) have been applied to a SDTM database to detect potential issues. R-Shiny application was developed to generate an interactive web application in order to ease site and/or patients identification during statistical data review meetings.

Results

The PRO-MSACTIVE study enrolled 422 patients in 46 centers between July 2018 and August 2019. Three data review meetings were held between April and October 2019 and 14 standard and planned tests were run on study data, with a total of 15 (32.6%) sites identified as needing review or investigation. Overall 36 findings were identified during the meetings: duplicate records, outliers, inconsistent delays between dates.

Conclusion

Statistical monitoring is useful to identify unusual or clustered data patterns that might be revealing issues that could impact the data integrity and/or may potentially impact patients' safety. With anticipated and appropriate interactive data visualization, early signals can easily be identified or reviewed by the study team and appropriate actions be set up and assigned to the most appropriate function for a close follow-up and resolution. Interactive statistical monitoring is time consuming to initiate using R-Shiny, but is time saving after the 1st data review meeting (DRV).(ClinicalTrials.gov identifier: NCT03589105; EudraCT identifier: 2018-000780-91).

Does Central Monitoring Lead to Higher Quality? An Analysis of Key Risk Indicator Outcomes

BACKGROUND

Central monitoring, which typically includes the use of key risk indicators (KRIs), aims at improving the quality of clinical research by pro-actively identifying and remediating emerging issues in the conduct of a clinical trial that may have an adverse impact on patient safety and/or the reliability of trial results. However, there has to-date been a relative lack of direct quantitative evidence published supporting the claim that central monitoring actually leads to improved quality.

MATERIAL AND METHODS

Nine commonly used KRIs were analyzed for evidence of quality improvement using data retrieved from a large central monitoring platform. A total of 212 studies comprising 1676 sites with KRI signals were used in the analysis, representing central monitoring activity from 23 different sponsor organizations. Two quality improvement metrics were assessed for each KRI, one based on a statistical score (p-value) and the other based on a KRI's observed value.

RESULTS

Both KRI quality metrics showed improvement in a vast majority of sites (82.9% for statistical score, 81.1% for observed KRI value). Additionally, the statistical score and the observed KRI values improved, respectively by 66.1% and 72.4% on average towards the study average for those sites showing improvement.

CONCLUSION

The results of this analysis provide clear quantitative evidence supporting the hypothesis that use of KRIs in central monitoring is leading to improved quality in clinical trial conduct and associated data across participating sites.

Monitoring strategies for clinical intervention studies

BACKGROUND

Trial monitoring is an important component of good clinical practice to ensure the safety and rights of study participants, confidentiality of personal information, and quality of data. However, the effectiveness of various existing monitoring approaches is unclear. Information to guide the choice of monitoring methods in clinical intervention studies may help trialists, support units, and monitors to effectively adjust their approaches to current knowledge and evidence.

OBJECTIVES

To evaluate the advantages and disadvantages of different monitoring strategies (including risk-based strategies and others) for clinical intervention studies examined in prospective comparative studies of monitoring interventions.

SEARCH METHODS

We systematically searched CENTRAL, PubMed, and Embase via Ovid for relevant published literature up to March 2021. We searched the online 'Studies within A Trial' (SWAT) repository, grey literature, and trial registries for ongoing or unpublished studies.

SELECTION CRITERIA

We included randomized or non-randomized prospective, empirical evaluation studies of different monitoring strategies in one or more clinical intervention studies. We applied no restrictions for language or date of publication.

DATA COLLECTION AND ANALYSIS

We extracted data on the evaluated monitoring methods, countries involved, study population, study setting, randomization method, and numbers and proportions in each intervention group. Our primary outcome was critical and major monitoring findings in prospective intervention studies. Monitoring findings were classified according to different error domains (e.g. major eligibility violations) and the primary outcome measure was a composite of these domains. Secondary outcomes were individual error domains, participant recruitment and follow-up, and resource use. If we identified more than one study for a comparison and outcome definitions were similar across identified studies, we quantitatively summarized effects in a meta-analysis using a random-effects model. Otherwise, we qualitatively summarized the results of eligible studies stratified by different comparisons of monitoring strategies. We used the GRADE approach to assess the certainty of the evidence for different groups of comparisons.

MAIN RESULTS

We identified eight eligible studies, which we grouped into five comparisons. 1. Risk-based versus extensive on-site monitoring: based on two large studies, we found moderate certainty of evidence for the combined primary outcome of major or critical findings that risk-based monitoring is not inferior to extensive on-site monitoring. Although the risk ratio was close to 'no difference' (1.03 with a 95% confidence interval [CI] of 0.81 to 1.33, below 1.0 in favor of the risk-based strategy), the high imprecision in one study and the small number of eligible studies resulted in a wide CI of the summary estimate. Low certainty of evidence suggested that monitoring strategies with extensive on-site monitoring were associated with considerably higher resource use and costs (up to a factor of 3.4). Data on recruitment or retention of trial participants were not available. 2. Central monitoring with triggered on-site visits versus regular on-site visits: combining the results of two eligible studies yielded low certainty of evidence with a risk ratio of 1.83 (95% CI 0.51 to 6.55) in favor of triggered monitoring intervention. Data on recruitment, retention, and resource use were not available. 3. Central statistical monitoring and local monitoring performed by site staff with annual on-site visits versus central statistical monitoring and local monitoring only: based on one study, there was moderate certainty of evidence that a small number of major and critical findings were missed with the central monitoring approach without on-site visits: 3.8% of participants in the group without on-site visits and 6.4% in the group with on-site visits had a major or critical monitoring finding (odds ratio 1.7, 95% CI 1.1 to 2.7; P = 0.03). The absolute number of monitoring findings was very low, probably because defined major and critical findings were very study specific and central monitoring was present in both intervention groups. Very low certainty of evidence did not suggest a relevant effect on participant retention, and very low certainty evidence indicated an extra cost for on-site visits of USD 2,035,392. There were no data on recruitment. 4. Traditional 100% source data verification (SDV) versus targeted or remote SDV: the two studies assessing targeted and remote SDV reported findings only related to source documents. Compared to the final database obtained using the full SDV monitoring process, only a small proportion of remaining errors on overall data were identified using the targeted SDV process in the MONITORING study (absolute difference 1.47%, 95% CI 1.41% to 1.53%). Targeted SDV was effective in the verification of source documents, but increased the workload on data management. The other included study was a pilot study, which compared traditional on-site SDV versus remote SDV and found little difference in monitoring findings and the ability to locate data values despite marked differences in remote access in two clinical trial networks. There were no data on recruitment or retention. 5. Systematic on-site initiation visit versus on-site initiation visit upon request: very low certainty of evidence suggested no difference in retention and recruitment between the two approaches. There were no data on critical and major findings or on resource use.

AUTHORS' CONCLUSIONS

The evidence base is limited in terms of quantity and quality. Ideally, for each of the five identified comparisons, more prospective, comparative monitoring studies nested in clinical trials and measuring effects on all outcomes specified in this review are necessary to draw more reliable conclusions. However, the results suggesting risk-based, targeted, and mainly central monitoring as an efficient strategy are promising. The development of reliable triggers for on-site visits is ongoing; different triggers might be used in different settings. More evidence on risk indicators that identify sites with problems or the prognostic value of triggers is needed to further optimize central monitoring strategies. In particular, approaches with an initial assessment of trial-specific risks that need to be closely monitored centrally during trial conduct with triggered on-site visits should be evaluated in future research.

Detection of Fraud in a Clinical Trial Using Unsupervised Statistical Monitoring

BACKGROUND

A central statistical assessment of the quality of data collected in clinical trials can improve the quality and efficiency of sponsor oversight of clinical investigations.

MATERIAL AND METHODS

The database of a large randomized clinical trial with known fraud was reanalyzed with a view to identifying, using only statistical monitoring techniques, the center where fraud had been confirmed. The analysis was conducted with an unsupervised statistical monitoring software using mixed-effects statistical models. The statistical analyst was unaware of the location, nature, and extent of the fraud.

RESULTS

Five centers were detected as atypical, including the center with known fraud (which was ranked 2). An incremental analysis showed that the center with known fraud could have been detected after only 25% of its data had been reported.

CONCLUSION

An unsupervised approach to central monitoring, using mixed-effects statistical models, is effective at detecting centers with fraud or other data anomalies in clinical trials.

Central data monitoring in the multicentre randomised SafeBoosC-III trial - a pragmatic approach

BACKGROUND

Data monitoring of clinical trials is a tool aimed at reducing the risks of random errors (e.g. clerical errors) and systematic errors, which include misinterpretation, misunderstandings, and fabrication. Traditional 'good clinical practice data monitoring' with on-site monitors increases trial costs and is time consuming for the local investigators. This paper aims to outline our approach of time-effective central data monitoring for the SafeBoosC-III multicentre randomised clinical trial and present the results from the first three central data monitoring meetings.

METHODS

The present approach to central data monitoring was implemented for the SafeBoosC-III trial, a large, pragmatic, multicentre, randomised clinical trial evaluating the benefits and harms of treatment based on cerebral oxygenation monitoring in preterm infants during the first days of life versus monitoring and treatment as usual. We aimed to optimise completeness and quality and to minimise deviations, thereby limiting random and systematic errors. We designed an automated report which was blinded to group allocation, to ease the work of data monitoring. The central data monitoring group first reviewed the data using summary plots only, and thereafter included the results of the multivariate Mahalanobis distance of each centre from the common mean. The decisions of the group were manually added to the reports for dissemination, information, correcting errors, preventing furture errors and documentation.

RESULTS

The first three central monitoring meetings identified 156 entries of interest, decided upon contacting the local investigators for 146 of these, which resulted in correction of 53 entries. Multiple systematic errors and protocol violations were identified, one of these included 103/818 randomised participants. Accordingly, the electronic participant record form (ePRF) was improved to reduce ambiguity.

DISCUSSION

We present a methodology for central data monitoring to optimise quality control and quality development. The initial results included identification of random errors in data entries leading to correction of the ePRF, systematic protocol violations, and potential protocol adherence issues. Central data monitoring may optimise concurrent data completeness and may help timely detection of data deviations due to misunderstandings or fabricated data.

Methods to assess research misconduct in health-related research: A scoping review

OBJECTIVE

To give an overview of the available methods to investigate research misconduct in health-related research.

STUDY DESIGN AND SETTING

In this scoping review, we conducted a literature search in MEDLINE, Embase, The Cochrane CENTRAL Register of Studies Online (CRSO), and The Virtual Health Library portal up to July 2020. We included papers that mentioned and/or described methods for screening or assessing research misconduct in health-related research. We categorized identified methods into the following four groups according to their scopes: overall concern, textual concern, image concern, and data concern.

RESULTS

We included 57 papers reporting on 27 methods: two on overall concern, four on textual concern, three on image concern, and 18 on data concern. Apart from the methods to locate textual plagiarism and image manipulation, all other methods, be it theoretical or empirical, are based on examples, are not standardized, and lack formal validation.

CONCLUSION

Existing methods cover a wide range of issues regarding research misconduct. Although measures to counteract textual plagiarism are well implemented, tools to investigate other forms of research misconduct are rudimentary and labour-intensive. To cope with the rising challenge of research misconduct, further development of automatic tools and routine validation of these methods is needed.

TRIAL REGISTRATION NUMBER

Center for Open Science (OSF) (https://osf.io/mq89w).

Risk-based centralized data monitoring of clinical trials at the time of COVID-19 pandemic

OBJECTIVES

COVID-19 pandemic caused several alarming challenges for clinical trials. On-site source data verification (SDV) in the multicenter clinical trial became difficult due to travel ban and social distancing. For multicenter clinical trials, centralized data monitoring is an efficient and cost-effective method of data monitoring. Centralized data monitoring reduces the risk of COVID-19 infections and provides additional capabilities compared to on-site monitoring. The key steps for on-site monitoring include identifying key risk factors and thresholds for the risk factors, developing a monitoring plan, following up the risk factors, and providing a management plan to mitigate the risk.

METHODS

For analysis purposes, we simulated data similar to our clinical trial data. We classified the data monitoring process into two groups, such as the Supervised analysis process, to follow each patient remotely by creating a dashboard and an Unsupervised analysis process to identify data discrepancy, data error, or data fraud. We conducted several risk-based statistical analysis techniques to avoid on-site source data verification to reduce time and cost, followed up with each patient remotely to maintain social distancing, and created a centralized data monitoring dashboard to ensure patient safety and maintain the data quality.

CONCLUSION

Data monitoring in clinical trials is a mandatory process. A risk-based centralized data review process is cost-effective and helpful to ignore on-site data monitoring at the time of the pandemic. We summarized how different statistical methods could be implemented and explained in SAS to identify various data error or fabrication issues in multicenter clinical trials.

Central monitoring in a randomized, open-label, controlled phase 3 clinical trial for a treatment-shortening regimen for pulmonary tuberculosis

INTRODUCTION

With the growing use of online study management systems and rapid availability of data, timely data review and quality assessments are necessary to ensure proper clinical trial implementation. In this report we describe central monitoring used to ensure protocol compliance and accurate data reporting, implemented during a large phase 3 clinical trial.

MATERIAL AND METHODS

The Tuberculosis Trials Consortium (TBTC) Study 31/AIDS Clinical Trials Group (ACTG) study A5349 (S31) is an international, multi-site, randomized, open-label, controlled, non-inferiority phase 3 clinical trial comparing two 4-month regimens to a standard 6 month regimen for treatment of drug-susceptible tuberculosis (TB) among adolescents and adults with a sample size of 2500 participants.

RESULTS

Central monitoring utilized primary study data in a five-tiered approach, including (1) real-time data checks & topic-specific intervention reports, (2) missing forms reports, (3) quality assurance metrics, (4) critical data reports and (5) protocol deviation identification, aimed to detect and resolve quality challenges. Over the course of the study, 240 data checks and reports were programed across the five tiers used.

DISCUSSION

This use of primary study data to identify issues rapidly allowed the study sponsor to focus quality assurance and data cleaning activities on prioritized data, related to protocol compliance and accurate reporting of study results. Our approach enabled us to become more efficient and effective as we informed sites about deviations, resolved missing or inconsistent data, provided targeted guidance, and gained a deeper understanding of challenges experienced at clinical trial sites.

TRIAL REGISTRATION

This trial was registered with ClinicalTrials.gov (Identifier: NCT02410772) on April 8, 2015.

A method for interoperable knowledge-based data quality assessment

BACKGROUND

Assessing the quality of healthcare data is a complex task including the selection of suitable measurement methods (MM) and adequately assessing their results.

OBJECTIVES

To present an interoperable data quality (DQ) assessment method that formalizes MMs based on standardized data definitions and intends to support collaborative governance of DQ-assessment knowledge, e.g. which MMs to apply and how to assess their results in different situations.

METHODS

We describe and explain central concepts of our method using the example of its first real world application in a study on predictive biomarkers for rejection and other injuries of kidney transplants. We applied our open source tool-openCQA-that implements our method utilizing the openEHR specifications. Means to support collaborative governance of DQ-assessment knowledge are the version-control system git and openEHR clinical information models.

RESULTS

Applying the method on the study's dataset showed satisfactory practicability of the described concepts and produced useful results for DQ-assessment.

CONCLUSIONS

The main contribution of our work is to provide applicable concepts and a tested exemplary open source implementation for interoperable and knowledge-based DQ-assessment in healthcare that considers the need for flexible task and domain specific requirements.

Clinical trial preparations for the next pandemic

This paper describes the need to prepare for the development of antiviral therapeutics for the next pandemic. Preparation would consist of a stockpiling of best practices for clinical trial design, analysis and operations during the current SARS-CoV-2 pandemic as well as continuous development of treatments and methodology between pandemics. This development would be facilitated by a global clinical trial pandemic reserve similar to the military reserves consisting of medical and quantitative methods professionals who would remain engaged between pandemics. Continuous identification of potential antiviral drugs and diagnostic methods would also be needed. Specific methodology addressed includes the importance of large simple trials, follow up time, efficacy endpoint, appropriate estimands, non-inferiority trials, more sophisticated patient accrual models and procedures for data sharing between clinical trials.

Dynamic methods for ongoing assessment of site-level risk in risk-based monitoring of clinical trials: A scoping review

Background/Aims

It is increasingly recognised that reliance on frequent site visits for monitoring clinical trials is inefficient. Regulators and trialists have recently encouraged more risk-based monitoring. Risk assessment should take place before a trial begins to define the overarching monitoring strategy. It can also be done on an ongoing basis, to target sites for monitoring activity. Various methods have been proposed for such prioritisation, often using terms like ‘central statistical monitoring’, ‘triggered monitoring’ or, as in the International Conference on Harmonization Good Clinical Practice guidance, ‘targeted on-site monitoring’. We conducted a scoping review to identify such methods, to establish if any were supported by adequate evidence to allow wider implementation, and to guide future developments in this field of research.

Methods

We used seven publication databases, two sets of methodological conference abstracts and an Internet search engine to identify methods for using centrally held trial data to assess site conduct during a trial. We included only reports in English, and excluded reports published before 1996 or not directly relevant to our research question. We used reference and citation searches to find additional relevant reports. We extracted data using a predefined template. We contacted authors to request additional information about included reports.

Results

We included 30 reports in our final dataset, of which 21 were peer-reviewed publications. In all, 20 reports described central statistical monitoring methods (of which 7 focussed on detection of fraud or misconduct) and 9 described triggered monitoring methods; 21 reports included some assessment of their methods’ effectiveness, typically exploring the methods’ characteristics using real trial data without known integrity issues. Of the 21 with some effectiveness assessment, most contained limited information about whether or not concerns identified through central monitoring constituted meaningful problems. Several reports demonstrated good classification ability based on more than one classification statistic, but never without caveats of unclear reporting or other classification statistics being low or unavailable. Some reports commented on cost savings from reduced on-site monitoring, but none gave detailed costings for the development and maintenance of central monitoring methods themselves.

Conclusion

Our review identified various proposed methods, some of which could be combined within the same trial. The apparent emphasis on fraud detection may not be proportionate in all trial settings. Despite some promising evidence and some self-justifying benefits for data cleaning activity, many proposed methods have limitations that may currently prevent their routine use for targeting trial monitoring activity. The implementation costs, or uncertainty about these, may also be a barrier. We make recommendations for how the evidence-base supporting these methods could be improved.

Time-consuming and expensive data quality monitoring procedures persist in clinical trials: A national survey

INTRODUCTION

The Good Clinical Practice guideline identifies that data monitoring is an essential research activity. However, limited evidence exists on how to perform monitoring including the amount or frequency that is needed to ensure data quality. This study aims to explore the monitoring procedures that are implemented to ensure data quality in Australian clinical research studies.

MATERIAL AND METHODS

Clinical studies listed on the Australian and New Zealand Clinical Trials Registry were invited to participate in a national cross-sectional, mixed-mode, multi-contact (postal letter and e-mail) web-based survey. Information was gathered about the types of data quality monitoring procedures being implemented.

RESULTS

Of the 3689 clinical studies contacted, 589 (16.0%) responded, of which 441 (77.4%) completed the survey. Over half (55%) of the studies applied source data verification (SDV) compared to risk-based targeted and triggered monitoring (10-11%). Conducting 100% on-site monitoring was most common for those who implemented the traditional approach. Respondents who did not conduct 100% monitoring, included 1-25% of data points for SDV, centralized or on-site monitoring. The incidence of adverse events and protocol deviations were the most likely factors to trigger a site visit for risk-based triggered (63% and 44%) and centralized monitoring (48% and 44%), respectively.

CONCLUSION

Instead of using more optimal risk-based approaches, small single-site clinical studies are conducting traditional monitoring procedures which are time consuming and expensive. Formal guidelines need to be improved and provided to all researchers for 'new' risk-based monitoring approaches.

Statistical Considerations for Trials in Adjuvant Treatment of Colorectal Cancer

The design of the best possible clinical trials of adjuvant interventions in colorectal cancer will entail the use of both time-tested and novel methods that allow efficient, reliable and patient-relevant therapeutic development. The ultimate goal of this endeavor is to safely and expeditiously bring to clinical practice novel interventions that impact patient lives. In this paper, we discuss statistical aspects and provide suggestions to optimize trial design, data collection, study implementation, and the use of predictive biomarkers and endpoints in phase 3 trials of systemic adjuvant therapy. We also discuss the issues of collaboration and patient centricity, expecting that several novel agents with activity in the (neo)adjuvant therapy of colon and rectal cancers will become available in the near future.

Conduct of Clinical Trials in the Era of COVID-19: JACC Scientific Expert Panel

The coronavirus disease-2019 (COVID-19) pandemic has profoundly changed clinical care and research, including the conduct of clinical trials, and the clinical research ecosystem will need to adapt to this transformed environment. The Heart Failure Academic Research Consortium is a partnership between the Heart Failure Collaboratory and the Academic Research Consortium, composed of academic investigators from the United States and Europe, patients, the U.S. Food and Drug Administration, the National Institutes of Health, and industry members. A series of meetings were convened to address the challenges caused by the COVID-19 pandemic, review options for maintaining or altering best practices, and establish key recommendations for the conduct and analysis of clinical trials for cardiovascular disease and heart failure. This paper summarizes the discussions and expert consensus recommendations.

Improving Study Conduct and Data Quality in Clinical Trials of Chronic Pain Treatments: IMMPACT Recommendations

The estimated probability of progressing from phase 3 analgesic clinical trials to regulatory approval is approximately 57%, suggesting that a considerable number of treatments with phase 2 trial results deemed sufficiently successful to progress to phase 3 do not yield positive phase 3 results. Deficiencies in the quality of clinical trial conduct could account for some of this failure. An Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials meeting was convened to identify potential areas for improvement in trial conduct in order to improve assay sensitivity (ie, ability of trials to detect a true treatment effect). We present recommendations based on presentations and discussions at the meeting, literature reviews, and iterative revisions of this article. The recommendations relate to the following areas: 1) study design (ie, to promote feasibility), 2) site selection and staff training, 3) participant selection and training, 4) treatment adherence, 5) data collection, and 6) data and study monitoring. Implementation of these recommendations may improve the quality of clinical trial data and thus the validity and assay sensitivity of clinical trials. Future research regarding the effects of these strategies will help identify the most efficient use of resources for conducting high quality clinical trials. PERSPECTIVE: Every effort should be made to optimize the quality of clinical trial data. This manuscript discusses considerations to improve conduct of pain clinical trials based on research in multiple medical fields and the expert consensus of pain researchers and stakeholders from academia, regulatory agencies, and industry.

Central statistical monitoring of investigator-led clinical trials in oncology

Investigator-led clinical trials are pragmatic trials that aim to investigate the benefits and harms of treatments in routine clinical practice. These much-needed trials represent the majority of all trials currently conducted. They are however threatened by the rising costs of clinical research, which are in part due to extensive trial monitoring processes that focus on unimportant details. Risk-based quality management focuses, instead, on “things that really matter”. We discuss the role of central statistical monitoring as part of risk-based quality management. We describe the principles of central statistical monitoring, provide examples of its use, and argue that it could help drive down the cost of randomized clinical trials, especially investigator-led trials, whilst improving their quality.

Bayesian central statistical monitoring using finite mixture models in multicenter clinical trials

Background

Central monitoring (CM), in which data across all clinical sites are monitored, has an important role in risk-based monitoring. Several statistical methods have been proposed to compare patient outcomes among the sites for detecting atypical sites that have different trends in observed data. These methods assume that the number of clinical sites is not small, e.g., 100 or more. In addition, the proportion of atypical sites is assumed to be relatively small. However, in actuality, the central statistical monitoring (CSM) has to be implemented in small or moderate sized clinical trials such as small phase II clinical trials. The number of sites is no longer large in such situations. Therefore, it is of concern that existing methods may not work efficiently in CM of small or moderate sized clinical trials. In the light of this problem, we propose a Bayesian CSM method to detect atypical sites as the robust method against the existence of atypical sites.

Methods

We use Bayesian finite mixture models (FMM) to model patient outcome values of both atypical and typical sites. In the method, the distributions of outcome values in normal sites are determined by choosing the body distribution, which has the largest mixture parameter value of finite mixture models based on the assumption that normal sites are in the majority. Atypical sites are detected by the criterion based on the posterior predictive distribution of normal site's outcome values derived from only the chosen body distribution.

Results

Proposed method is evaluated by cumulative detection probability and type I error averaged over sites every round of CSM under the various scenarios, being compared with the conventional type analysis. If the total number of patients enrolled is 48, the proposed method is superior at least 10% for any shift sizes at the 2nd and the 3rd rounds. If the total number of patients is 96, both methods show similar detection probability for only one atypical site and large shift size. However, the proposed method is superior for the other scenarios. It is observed that all the type I errors averaged over sites are little difference between the methods at all the scenarios.

Conclusion

We propose a Bayesian CSM method which works efficiently in a practical use of CM. It is shown that our method detects atypical sites with high probability regardless of the proportion of the atypical sites under the small clinical trial settings which is the target of our proposed method.

A randomized evaluation of on-site monitoring nested in a multinational randomized trial

BACKGROUND

Evidence from prospectively designed studies to guide on-site monitoring practices for randomized trials is limited. A cluster randomized study, nested within the Strategic Timing of AntiRetroviral Treatment (START) trial, was conducted to evaluate on-site monitoring.

METHODS

Sites were randomized to either annual on-site monitoring or no on-site monitoring. All sites were centrally monitored, and local monitoring was carried out twice each year. Randomization was stratified by country and projected enrollment in START. The primary outcome was a participant-level composite outcome including components for eligibility errors, consent violations, use of antiretroviral treatment not recommended by protocol, late reporting of START primary and secondary clinical endpoints (defined as the event being reported more than 6 months from occurrence), and data alteration and fraud. Logistic regression fixed effect hierarchical models were used to compare on-site versus no on-site monitoring for the primary composite outcome and its components. Odds ratios and 95% confidence intervals comparing on-site monitoring versus no on-site monitoring are cited.

RESULTS

In total, 99 sites (2107 participants) were randomized to receive annual on-site monitoring and 97 sites (2264 participants) were randomized to be monitored only centrally and locally. The two monitoring groups were well balanced at entry. In the on-site monitoring group, 469 annual on-site monitoring visits were conducted, and 134 participants (6.4%) in 56 of 99 sites (57%) had a primary monitoring outcome. In the no on-site monitoring group, 85 participants (3.8%) in 34 of 97 sites (35%) had a primary monitoring outcome (odds ratio = 1.7; 95% confidence interval: 1.1-2.7; p = 0.03). Informed consent violations accounted for most outcomes in each group (56 vs 41 participants). The largest odds ratio was for eligibility violations (odds ratio = 12.2; 95% confidence interval: 1.8-85.2; p = 0.01). The number of participants with a late START primary endpoint was similar for each monitoring group (23 vs 16 participants). Late START grade 4 and unscheduled hospitalization events were found for 34 participants in the on-site monitoring group and 19 participants in the no on-site monitoring group (odds ratio = 2.0; 95% confidence interval: 1.1-3.7; p = 0.02). There were no cases of data alteration or fraud. Based on the travel budget for on-site monitoring and the hours spent conducting on-site monitoring, the estimated cost of on-site monitoring was over US$2 million.

CONCLUSION

On-site monitoring led to the identification of more eligibility and consent violations and START clinical events being reported more than 6 months from occurrence as compared to no on-site monitoring. Considering the nature of the excess monitoring outcomes identified at sites receiving on-site monitoring, as well as the cost of on-site monitoring, the value to the START study was limited.

Creating an academic research organization to efficiently design, conduct, coordinate, and analyze clinical trials: The Center for Clinical Trials & Data Coordination

When properly executed, the randomized controlled trial is one of the best vehicles for assessing the effectiveness of one or more interventions. However, numerous challenges may emerge in the areas of study startup, recruitment, data quality, cost, and reporting of results. The use of well-run coordinating centers could help prevent these issues, but very little exists in the literature describing their creation or the guiding principles behind their inception. The Center for Clinical Trials & Data Coordination (CCDC) was established in 2015 through institutional funds with the intent of 1) providing relevant expertise in clinical trial design, conduct, coordination, and analysis; 2) advancing the careers of clinical investigators and CCDC-affiliated faculty; and 3) obtaining large data coordinating center (DCC) grants. We describe the organizational structure of the CCDC as well as the homegrown clinical trial management system integrating nine crucial elements: electronic data capture, eligibility and randomization, drug and external data tracking, safety reporting, outcome adjudication, data and safety monitoring, statistical analysis and reporting, data sharing, and regulatory compliance. Lastly, we share numerous lessons that can be taken from our experience. Specifically, we focus on 1) funding for DCCs, 2) the importance of DCCs to clinical researchers, 3) the expertise of DCC personnel, and 4) continually striving to improve. In conclusion, the CCDC strives to provide high-quality support for the design, conduct, coordination, and analyses of clinical trials, and we hope this paper will serve as a blueprint for future clinical trialists involved in DCCs.

Comparing methods for clinical investigator site inspection selection: a comparison of site selection methods of investigators in clinical trials

Background During the past two decades, the number and complexity of clinical trials have risen dramatically increasing the difficulty of choosing sites for inspection. FDA's resources are limited and so sites should be chosen with care. Purpose To determine if data mining techniques and/or unsupervised statistical monitoring can assist with the process of identifying potential clinical sites for inspection. Methods Five summary-level clinical site datasets from four new drug applications (NDA) and one biologics license application (BLA), where the FDA had performed or had planned site inspections, were used. The number of sites inspected and the results of the inspections were blinded to the researchers. Five supervised learning models from the previous two years (2016-2017) of an on-going research project were used to predict site inspections results, i.e., No Action Indicated (NAI), Voluntary Action Indicated (VAI), or Official Action Indicated (OAI). Statistical Monitoring Applied to Research Trials (SMART^TM) software for unsupervised statistical monitoring software developed by CluePoints (Mont-Saint-Guibert, Belgium) was utilized to identify atypical centers (via a p-value approach) within a study.Finally, Clinical Investigator Site Selection Tool (CISST), developed by the Center for Drug Evaluation and Research (CDER), was used to calculate the total risk of each site thereby providing a framework for site selection. The agreement between the predictions of these methods was compared. The overall accuracy and sensitivity of the methods were graphically compared. Results Spearman's rank order correlation was used to examine the agreement between the SMART^TM analysis (CluePoints' software) and the CISST analysis. The average aggregated correlation between the p-values (SMART^TM) and total risk scores (CISST) for all five studies was 0.21, and range from -0.41 to 0.50. The Random Forest models for 2016 and 2017 showed the highest aggregated mean agreement (65.1%) amongst outcomes (NAI, VAI, OAI) for the three available studies. While there does not appear to be a single most accurate approach, the performance of methods under certain circumstances is discussed later in this paper. Limitations Classifier models based on data mining techniques require historical data (i.e., training data) to develop the model. There is a possibility that sites in the five-summary level datasets were included in the training datasets for the models from the previous year's research which could result in spurious confirmation of predictive ability. Additionally, the CISST was utilized in three of the five site selection processes, possibly biasing the data. Conclusion The agreement between methods was lower than expected and no single method emerged as the most accurate.

Detection of atypical data in multicenter clinical trials using unsupervised statistical monitoring

BACKGROUND/AIMS

A risk-based approach to clinical research may include a central statistical assessment of data quality. We investigated the operating characteristics of unsupervised statistical monitoring aimed at detecting atypical data in multicenter experiments. The approach is premised on the assumption that, save for random fluctuations and natural variations, data coming from all centers should be comparable and statistically consistent. Unsupervised statistical monitoring consists of performing as many statistical tests as possible on all trial data, in order to detect centers whose data are inconsistent with data from other centers.

METHODS

We conducted simulations using data from a large multicenter trial conducted in Japan for patients with advanced gastric cancer. The actual trial data were contaminated in computer simulations for varying percentages of centers, percentages of patients modified within each center and numbers and types of modified variables. The unsupervised statistical monitoring software was run by a blinded team on the contaminated data sets, with the purpose of detecting the centers with contaminated data. The operating characteristics (sensitivity, specificity and Youden's J-index) were calculated for three detection methods: one using the p-values of individual statistical tests after adjustment for multiplicity, one using a summary of all p-values for a given center, called the Data Inconsistency Score, and one using both of these methods.

RESULTS

The operating characteristics of the three methods were satisfactory in situations of data contamination likely to occur in practice, specifically when a single or a few centers were contaminated. As expected, the sensitivity increased for increasing proportions of patients and increasing numbers of variables contaminated. The three methods showed a specificity better than 93% in all scenarios of contamination. The method based on the Data Inconsistency Score and individual p-values adjusted for multiplicity generally had slightly higher sensitivity at the expense of a slightly lower specificity.

CONCLUSIONS

The use of brute force (a computer-intensive approach that generates large numbers of statistical tests) is an effective way to check data quality in multicenter clinical trials. It can provide a cost-effective complement to other data-management and monitoring techniques.

Implementing monitoring triggers and matching of triggered and control sites in the TEMPER study: a description and evaluation of a triggered monitoring management system

BACKGROUND

Triggered monitoring in clinical trials is a risk-based monitoring approach where triggers (centrally monitored, predefined key risk and performance indicators) drive the extent, timing, and frequency of monitoring visits. The TEMPER study used a prospective, matched-pair design to evaluate the use of a triggered monitoring strategy, comparing findings from triggered monitoring visits with those from matched control sites. To facilitate this study, we developed a bespoke risk-based monitoring system: the TEMPER Management System.

METHODS

The TEMPER Management System comprises a web application (the front end), an SQL server database (the back end) to store the data generated for TEMPER, and a reporting function to aid users in study processes such as the selection of triggered sites. Triggers based on current practice were specified for three clinical trials and were implemented in the system. Trigger data were generated in the system using data extracted from the trial databases to inform the selection of triggered sites to visit. Matching of the chosen triggered sites with untriggered control sites was also performed in the system, while data entry screens facilitated the collection and management of the data from findings gathered at monitoring visits.

RESULTS

There were 38 triggers specified for the participating trials. Using these, 42 triggered sites were chosen and matched with control sites. Monitoring visits were carried out to all sites, and visit findings were entered into the TEMPER Management System. Finally, data extracted from the system were used for analysis.

CONCLUSIONS

The TEMPER Management System made possible the completion of the TEMPER study. It implemented an approach of standardising the automation of current-practice triggers, and the generation of trigger data to inform the selection of triggered sites to visit. It also implemented a matching algorithm informing the selection of matched control sites. We hope that by publishing this paper it encourages other trialists to share their approaches to, and experiences of, triggered monitoring and other risk-based monitoring systems.

Triggered or routine site monitoring visits for randomised controlled trials: results of TEMPER, a prospective, matched-pair study

BACKGROUND/AIMS

In multi-site clinical trials, where trial data and conduct are scrutinised centrally with pre-specified triggers for visits to sites, targeted monitoring may be an efficient way to prioritise on-site monitoring. This approach is widely used in academic trials, but has never been formally evaluated.

METHODS

TEMPER assessed the ability of targeted monitoring, as used in three ongoing phase III randomised multi-site oncology trials, to distinguish sites at which higher and lower rates of protocol and/or Good Clinical Practice violations would be found during site visits. Using a prospective, matched-pair design, sites that had been prioritised for visits after having activated 'triggers' were matched with a control ('untriggered') site, which would not usually have been visited at that time. The paired sites were visited within 4 weeks of each other, and visit findings are recorded and categorised according to the seriousness of the deviation. The primary outcome measure was the proportion of sites with ≥1 'Major' or 'Critical' finding not previously identified centrally. The study was powered to detect an absolute difference of ≥30% between triggered and untriggered visits. A sensitivity analysis, recommended by the study's blinded endpoint review committee, excluded findings related to re-consent. Additional analyses assessed the prognostic value of individual triggers and data from pre-visit questionnaires completed by site and trials unit staff.

RESULTS

In total, 42 matched pairs of visits took place between 2013 and 2016. In the primary analysis, 88.1% of triggered visits had ≥1 new Major/Critical finding, compared to 81.0% of untriggered visits, an absolute difference of 7.1% (95% confidence interval -8.3%, +22.5%; p = 0.365). When re-consent findings were excluded, these figures reduced to 85.7% versus 59.5%, (difference = 26.2%, 95% confidence interval 8.0%, 44.4%; p = 0.007). Individual triggers had modest prognostic value but knowledge of the trial-related activities carried out by site staff may be useful.

CONCLUSION

Triggered monitoring approaches, as used in these trials, were not sufficiently discriminatory. The rate of Major and Critical findings was higher than anticipated, but the majority related to consent and re-consent with no indication of systemic problems that would impact trial-wide safety issues or integrity of the results in any of the three trials. Sensitivity analyses suggest triggered monitoring may be of potential use, but needs improvement and investigation of further central monitoring triggers is warranted. TEMPER highlights the need to question and evaluate methods in trial conduct, and should inform further developments in this area.

Rethinking the Clinically Based Thresholds of TransCelerate BioPharma for Risk-Based Monitoring

BACKGROUND

The quality of data from clinical trials has received a great deal of attention in recent years. Of central importance is the need to protect the well-being of study participants and maintain the integrity of final analysis results. However, traditional approaches to assess data quality have come under increased scrutiny as providing little benefit for the substantial cost. Numerous regulatory guidance documents and industry position papers have described risk-based approaches to identify quality and safety issues. In particular, the position paper of TransCelerate BioPharma recommends defining risk thresholds to assess safety and quality risks based on past clinical experience. This exercise can be extremely time-consuming, and the resulting thresholds may only be relevant to a particular therapeutic area, patient or clinical site population. In addition, predefined thresholds cannot account for safety or quality issues where the underlying rate of observing a particular problem may change over the course of a clinical trial, and often do not consider varying patient exposure.

METHODS

In this manuscript, we appropriate rules commonly utilized for funnel plots to define a traffic-light system for risk indicators based on statistical criteria that consider the duration of patient follow-up. Further, we describe how these methods can be adapted to assess changing risk over time. Finally, we illustrate numerous graphical approaches to summarize and communicate risk, and discuss hybrid clinical-statistical approaches to allow for the assessment of risk at sites with low patient enrollment.

RESULTS

We illustrate the aforementioned methodologies for a clinical trial in patients with schizophrenia.

CONCLUSIONS

Funnel plots are a flexible graphical technique that can form the basis for a risk-based strategy to assess data integrity, while considering site sample size, patient exposure, and changing risk across time.

Central and statistical data monitoring in the Clinical Randomisation of an Antifibrinolytic in Significant Haemorrhage (CRASH-2) trial

Background The purpose of monitoring in clinical trials is to ensure the rights, safety, and well-being of trial patients and the accuracy of the trial data. In the Clinical Randomisation of an Antifibrinolytic in Significant Haemorrhage (CRASH-2) trial, which recruited over 20,000 adult trauma patients worldwide, the nature and extent of monitoring was based on a risk assessment undertaken before recruitment started. Purpose We report the methods used for central and statistical monitoring in the CRASH-2 trial and explain how central monitoring was used to target on-site investigations. Methods To ensure that trial participants met the inclusion criteria, we monitored event rates for the primary (death) and secondary outcomes (blood transfusion given). We monitored four quantitative variables (systolic blood pressure (SBP), heart rate (HR), respiratory rate, and capillary refill time) as indicators of the severity of bleeding. We used the coefficient of variation (CV) to identify sites with too much or too little variability. To ensure the accuracy of the data on side effects, we monitored thromboembolic events at each site. Sites with higher or lower than expected event rates were identified for further evaluation. Results A total of 274 sites recruited patients: 145 sites recruited ≥20; patients, and 52 sites recruited ≥100 patients. Sites with low case fatality and low blood transfusion rates were found to be including patients with relatively mild haemorrhage. One site with a high rate of thromboembolic events was found to be using clinical judgement alone. Measurements of SBP and HR varied by about one-fifth of their average value, and capillary refill time measurements varied by around one-third of their average; between-site variation was lowest for blood pressure. Limitations A comparison of mean and median CV indicated that the distributions are slightly skewed to the right. Our simple approach to calculating 95% confidence intervals for the CV may be improved by using a logarithmic transformation of CV for each variable. Conclusions Central and statistical monitoring of data can be used to monitor clinical trials, particularly large, pragmatic, international trials where 100% on-site monitoring is neither necessary nor cost-effective. In the CRASH-2 trial, re-education about trial protocol and the development of guidance helped resolve the issues identified during monitoring.

Truths, lies, and statistics

Distribution of valuable research discoveries are needed for the continual advancement of patient care. Publication and subsequent reliance of false study results would be detrimental for patient care. Unfortunately, research misconduct may originate from many sources. While there is evidence of ongoing research misconduct in all it's forms, it is challenging to identify the actual occurrence of research misconduct, which is especially true for misconduct in clinical trials. Research misconduct is challenging to measure and there are few studies reporting the prevalence or underlying causes of research misconduct among biomedical researchers. Reported prevalence estimates of misconduct are probably underestimates, and range from 0.3% to 4.9%. There have been efforts to measure the prevalence of research misconduct; however, the relatively few published studies are not freely comparable because of varying characterizations of research misconduct and the methods used for data collection. There are some signs which may point to an increased possibility of research misconduct, however there is a need for continued self-policing by biomedical researchers. There are existing resources to assist in ensuring appropriate statistical methods and preventing other types of research fraud. These included the "Statistical Analyses and Methods in the Published Literature", also known as the SAMPL guidelines, which help scientists determine the appropriate method of reporting various statistical methods; the "Strengthening Analytical Thinking for Observational Studies", or the STRATOS, which emphases on execution and interpretation of results; and the Committee on Publication Ethics (COPE), which was created in 1997 to deliver guidance about publication ethics. COPE has a sequence of views and strategies grounded in the values of honesty and accuracy.

The impact of data errors on the outcome of randomized clinical trials

Background/aims Considerable human and financial resources are typically spent to ensure that data collected for clinical trials are free from errors. We investigated the impact of random and systematic errors on the outcome of randomized clinical trials. Methods We used individual patient data relating to response endpoints of interest in two published randomized clinical trials, one in ophthalmology and one in oncology. These randomized clinical trials enrolled 1186 patients with age-related macular degeneration and 736 patients with metastatic colorectal cancer. The ophthalmology trial tested the benefit of pegaptanib for the treatment of age-related macular degeneration and identified a statistically significant treatment benefit, whereas the oncology trial assessed the benefit of adding cetuximab to a regimen of capecitabine, oxaliplatin, and bevacizumab for the treatment of metastatic colorectal cancer and failed to identify a statistically significant treatment difference. We simulated trial results by adding errors that were independent of the treatment group (random errors) and errors that favored one of the treatment groups (systematic errors). We added such errors to the data for the response endpoint of interest for increasing proportions of randomly selected patients. Results Random errors added to up to 50% of the cases produced only slightly inflated variance in the estimated treatment effect of both trials, with no qualitative change in the p-value. In contrast, systematic errors produced bias even for very small proportions of patients with added errors. Conclusion A substantial amount of random errors is required before appreciable effects on the outcome of randomized clinical trials are noted. In contrast, even a small amount of systematic errors can severely bias the estimated treatment effects. Therefore, resources devoted to randomized clinical trials should be spent primarily on minimizing sources of systematic errors which can bias the analyses, rather than on random errors which result only in a small loss in power.

Improving public health by improving clinical trial guidelines and their application

Evidence generated from randomized controlled trials forms the foundation of cardiovascular therapeutics and has led to the adoption of numerous drugs and devices that prolong survival and reduce morbidity, as well as the avoidance of interventions that have been shown to be ineffective or even unsafe. Many aspects of cardiovascular research have evolved considerably since the first randomized trials in cardiology were conducted. In order to be large enough to provide reliable evidence about effects on major outcomes, cardiovascular trials may now involve thousands of patients recruited from hundreds of clinical sites in many different countries. Costly infrastructure has developed to meet the increasingly complex organizational and operational requirements of these clinical trials. Concerns have been raised that this approach is unsustainable, inhibiting the reliable evaluation of new and existing treatments, to the detriment of patient care. These issues were considered by patients, regulators, funders, and trialists at a meeting of the European Society of Cardiology Cardiovascular Roundtable in October 2015. This paper summarizes the key insights and discussions from the workshop, highlights subsequent progress, and identifies next steps to produce meaningful change in the conduct of cardiovascular clinical research.

DQe-v: A Database-Agnostic Framework for Exploring Variability in Electronic Health Record Data Across Time and Site Location

Data variability is a commonly observed phenomenon in Electronic Health Records (EHR) data networks. A common question asked in scientific investigations of EHR data is whether the cross-site and -time variability reflects an underlying data quality error at one or more contributing sites versus actual differences driven by various idiosyncrasies in the healthcare settings. Although research analysts and data scientists have commonly used various statistical methods to detect and account for variability in analytic datasets, self service tools to facilitate exploring cross-organizational variability in EHR data warehouses are lacking and could benefit from meaningful data visualizations. DQ^e-v, an interactive, database-agnostic tool for visually exploring variability in EHR data provides such a solution. DQ^e-v is built on an open source platform, R statistical software, with annotated scripts and a readme document that makes it fully reproducible. To illustrate and describe functionality of DQ^e-v, we describe the DQ^e-v’s readme document which includes a complete guide to installation, running the program, and interpretation of the outputs. We also provide annotated R scripts and an example dataset as supplemental materials. DQ^e-v offers a self service tool to visually explore data variability within EHR datasets irrespective of the data model. GitHub and CIELO offer hosting and distribution of the tool and can facilitate collaboration across any interested community of users as we target improving usability, efficiency, and interoperability.

Generating evidence on a risk-based monitoring approach in the academic setting - lessons learned

BACKGROUND

In spite of efforts to employ risk-based strategies to increase monitoring efficiency in the academic setting, empirical evidence on their effectiveness remains sparse. This mixed-methods study aimed to evaluate the risk-based on-site monitoring approach currently followed at our academic institution.

METHODS

We selected all studies monitored by the Clinical Trial Unit (CTU) according to Risk ADApted MONitoring (ADAMON) at the University Hospital Basel, Switzerland, between 01.01.2012 and 31.12.2014. We extracted study characteristics and monitoring information from the CTU Enterprise Resource Management system and from monitoring reports of all selected studies. We summarized the data descriptively. Additionally, we conducted semi-structured interviews with the three current CTU monitors.

RESULTS

During the observation period, a total of 214 monitoring visits were conducted in 43 studies resulting in 2961 documented monitoring findings. Our risk-based approach predominantly identified administrative (46.2%) and patient right findings (49.1%). We identified observational study design, high ADAMON risk category, industry sponsorship, the presence of an electronic database, experienced site staff, and inclusion of vulnerable study population to be factors associated with lower numbers of findings. The monitors understand the positive aspects of a risk-based approach but fear missing systematic errors due to the low frequency of visits.

CONCLUSIONS

We show that the factors mostly increasing the risk for on-site monitoring findings are underrepresented in the current risk analysis scheme. Our risk-based on-site approach should further be complemented by centralized data checks, allowing monitors to transform their role towards partners for overall trial quality, and success.

Monitoring clinical trials: a practical guide

This article describes the processes and procedures involved in planning, conducting and reporting monitoring activities for large Clinical Trials of Investigational Medicinal Products (CTIMPs), focusing on those conducted in resource-limited settings.

The impact of clinical trial monitoring approaches on data integrity and cost--a review of current literature

PURPOSE

Monitoring is a costly requirement when conducting clinical trials. New regulatory guidance encourages the industry to consider alternative monitoring methods to the traditional 100 % source data verification (SDV) approach. The purpose of this literature review is to provide an overview of publications on different monitoring methods and their impact on subject safety data, data integrity, and monitoring cost.

METHODS

The literature search was performed by keyword searches in MEDLINE and hand search of key journals. All publications were reviewed for details on how a monitoring approach impacted subject safety data, data integrity, or monitoring costs.

RESULTS

Twenty-two publications were identified. Three publications showed that SDV has some value for detection of not initially reported adverse events and centralized statistical monitoring (CSM) captures atypical trends. Fourteen publications showed little objective evidence of improved data integrity with traditional monitoring such as 100 % SDV and sponsor queries as compared to reduced SDV, CSM, and remote monitoring. Eight publications proposed a potential for significant cost reductions of monitoring by reducing SDV without compromising the validity of the trial results.

CONCLUSIONS

One hundred percent SDV is not a rational method of ensuring data integrity and subject safety based on the high cost, and this literature review indicates that reduced SDV is a viable monitoring method. Alternative methods of monitoring such as centralized monitoring utilizing statistical tests are promising alternatives but have limitations as stand-alone tools. Reduced SDV combined with a centralized, risk-based approach may be the ideal solution to reduce monitoring costs while improving essential data quality.

Efficient Source Data Verification Using Statistical Acceptance Sampling: A Simulation Study

BACKGROUND

One approach to increase the efficiency of clinical trial monitoring is to replace 100% source data verification (SDV) by verification of samples of source data. An intuitive strategy for determining appropriate sampling plans (ie, sample sizes and the maximum tolerable number of transcription errors in the samples) is to use acceptance sampling methodology. Expanding upon earlier work in which the use of acceptance sampling strategies for sampling-based SDV was proposed, we describe an alternative acceptance sampling strategy that, instead of relying on sampling standards, evaluates all possible sampling plans algorithmically, thereby ensuring that selected sampling plans conform to prespecified criteria.

METHODS

Empirical trial data guided the design of the proposed strategy. In addition, extensive simulations, also based on the empirical data, were performed to assess the performance in terms of workload reductions and the post-SDV error proportion of applying the proposed strategy.

RESULTS

13 different scenarios were simulated, but results of the default scenario show that the average pre-SDV error proportion per trial of .056 was reduced to .023 by inspecting only 40.5% of the case report form entries. Of the inspected data entries, almost half (18.0/40.5) was, on average, SDV-ed as part of the sampling process; remaining entries were inspected during full inspections after too many errors were observed in the samples.

CONCLUSION

Our results suggest that major reductions in workload can be achieved, while maintaining acceptable data quality levels. However, the results also indicate that the proposed strategy is conservative and further improvement is possible.

Statistical monitoring of data quality and consistency in the Stomach Cancer Adjuvant Multi-institutional Trial Group Trial

INTRODUCTION

Data quality may impact the outcome of clinical trials; hence, there is a need to implement quality control strategies for the data collected. Traditional approaches to quality control have primarily used source data verification during on-site monitoring visits, but these approaches are hugely expensive as well as ineffective. There is growing interest in central statistical monitoring (CSM) as an effective way to ensure data quality and consistency in multicenter clinical trials.

METHODS

CSM with SMART™ uses advanced statistical tools that help identify centers with atypical data patterns which might be the sign of an underlying quality issue. This approach was used to assess the quality and consistency of the data collected in the Stomach Cancer Adjuvant Multi-institutional Trial Group Trial, involving 1495 patients across 232 centers in Japan.

RESULTS

In the Stomach Cancer Adjuvant Multi-institutional Trial Group Trial, very few atypical data patterns were found among the participating centers, and none of these patterns were deemed to be related to a quality issue that could significantly affect the outcome of the trial.

DISCUSSION

CSM can be used to provide a check of the quality of the data from completed multicenter clinical trials before analysis, publication, and submission of the results to regulatory agencies. It can also form the basis of a risk-based monitoring strategy in ongoing multicenter trials. CSM aims at improving data quality in clinical trials while also reducing monitoring costs.

Statistical challenges for central monitoring in clinical trials: a review

Recently, the complexity and costs of clinical trials have increased dramatically, especially in the area of new drug development. Risk-based monitoring (RBM) has been attracting attention as an efficient and effective trial monitoring approach, which can be applied irrespectively of the trial sponsor, i.e., academic institution or pharmaceutical company. In the RBM paradigm, it is expected that a statistical approach to central monitoring can help improve the effectiveness of on-site monitoring by prioritizing and guiding site visits according to central statistical data checks, as evidenced by examples of actual trial datasets. In this review, several statistical methods for central monitoring are presented. It is important to share knowledge about the role and performance capabilities of statistical methodology among clinical trial team members (i.e., sponsors, investigators, data managers, monitors, and biostatisticians) in order to adopt central statistical monitoring for assessing data quality in the actual clinical trial.

Data-driven risk identification in phase III clinical trials using central statistical monitoring

Our interest lies in quality control for clinical trials, in the context of risk-based monitoring (RBM). We specifically study the use of central statistical monitoring (CSM) to support RBM. Under an RBM paradigm, we claim that CSM has a key role to play in identifying the "risks to the most critical data elements and processes" that will drive targeted oversight. In order to support this claim, we first see how to characterize the risks that may affect clinical trials. We then discuss how CSM can be understood as a tool for providing a set of data-driven key risk indicators (KRIs), which help to organize adaptive targeted monitoring. Several case studies are provided where issues in a clinical trial have been identified thanks to targeted investigation after the identification of a risk using CSM. Using CSM to build data-driven KRIs helps to identify different kinds of issues in clinical trials. This ability is directly linked with the exhaustiveness of the CSM approach and its flexibility in the definition of the risks that are searched for when identifying the KRIs. In practice, a CSM assessment of the clinical database seems essential to ensure data quality. The atypical data patterns found in some centers and variables are seen as KRIs under a RBM approach. Targeted monitoring or data management queries can be used to confirm whether the KRIs point to an actual issue or not.

Closeout of the HALT-PKD trials

BACKGROUND

The HALT Polycystic Kidney Disease Trials Network consisted of two randomized, double blind, placebo-controlled trials among patients with autosomal dominant polycystic kidney disease. The trials involved 5-8years of participant follow-up with interventions in blood pressure and antihypertensive therapy. We provide a framework for designing and implementing closeout near the end of a trial while ensuring patient safety and maintaining scientific rigor and study morale.

METHODS

We discuss issues and resolutions for determining the last visit, tapering medications, and unblinding of participants to study allocation and results. We also discuss closure of clinical sites and Data Coordinating Center responsibilities to ensure timely release of study results and meeting the requirements of regulatory and funding authorities.

RESULTS

Just over 90% of full participants had a 6-month study visit prior to their last visit preparing them for trial closeout. Nearly all patients wanted notification of study results (99%) and treatment allocation (99%). All participants were safely tapered off study and open label blood pressure medications. Within 6months, the trials were closed, primary papers published, and 805 letters distributed to participants with results and allocation. DCC obligations for data repository and clinicaltrials.gov reporting were completed within 12months of the last study visit.

CONCLUSIONS

Closeout of our trials involved years of planning and significant human and financial resources. We provide questions for investigators to consider when planning closeout of their trials with focus on (1) patient safety, (2) dissemination of study results and (3) compliance with regulatory and funding responsibilities.