Good Statistical Monitoring: A Flexible Open-Source Tool to Detect Risks in Clinical Trials

BACKGROUND

Risk-based quality management is a regulatory-recommended approach to manage risk in a clinical trial. A key element of this strategy is to conduct risk-based monitoring to detect potential risks to critical data and processes earlier. However, there are limited publicly available tools to perform the analytics required for this purpose. Good Statistical Monitoring is a new open-source solution developed to help address this need.

METHODS

A team of statisticians, data scientists, clinicians, data managers, clinical operations, regulatory, and quality compliance staff collaborated to design Good Statistical Monitoring, an R package, to flexibly and efficiently implement end-to-end analyses of key risks. The package currently supports the mapping of clinical trial data from a variety of formats, evaluation of 12 key risk indicators, interactive visualization of analysis results, and creation of standardized reports.

RESULTS

The Good Statistical Monitoring package is freely available on GitHub and empowers clinical study teams to proactively monitor key risks. It employs a modular workflow to perform risk assessments that can be customized by replacing any workflow component with a study-specific alternative. Results can be exported to other clinical systems or can be viewed as an interactive report to facilitate follow-up risk mitigation. Rigorous testing and qualification are performed as part of each release to ensure package quality.

CONCLUSIONS

Good Statistical Monitoring is an open-source solution designed to enable clinical study teams to implement statistical monitoring of critical risks, as part of a comprehensive risk-based quality management strategy.

A survey of experts to identify methods to detect problematic studies: stage 1 of the INveStigating ProblEmatic Clinical Trials in Systematic Reviews project

BACKGROUND AND OBJECTIVE

Randomized controlled trials (RCTs) inform health-care decisions. Unfortunately, some published RCTs contain false data, and some appear to have been entirely fabricated. Systematic reviews are performed to identify and synthesize all RCTs which have been conducted on a given topic. This means that any of these 'problematic studies' are likely to be included, but there are no agreed methods for identifying them. The INveStigating ProblEmatic Clinical Trials in Systematic Reviews (INSPECT-SR) project is developing a tool to identify problematic RCTs in systematic reviews of health care-related interventions. The tool will guide the user through a series of 'checks' to determine a study's authenticity. The first objective in the development process is to assemble a comprehensive list of checks to consider for inclusion.

METHODS

We assembled an initial list of checks for assessing the authenticity of research studies, with no restriction to RCTs, and categorized these into five domains: Inspecting results in the paper; Inspecting the research team; Inspecting conduct, governance, and transparency; Inspecting text and publication details; Inspecting the individual participant data. We implemented this list as an online survey, and invited people with expertise and experience of assessing potentially problematic studies to participate through professional networks and online forums. Participants were invited to provide feedback on the checks on the list, and were asked to describe any additional checks they knew of, which were not featured in the list.

RESULTS

Extensive feedback on an initial list of 102 checks was provided by 71 participants based in 16 countries across five continents. Fourteen new checks were proposed across the five domains, and suggestions were made to reword checks on the initial list. An updated list of checks was constructed, comprising 116 checks. Many participants expressed a lack of familiarity with statistical checks, and emphasized the importance of feasibility of the tool.

CONCLUSION

A comprehensive list of trustworthiness checks has been produced. The checks will be evaluated to determine which should be included in the INSPECT-SR tool.

PLAIN LANGUAGE SUMMARY

Systematic reviews draw upon evidence from randomized controlled trials (RCTs) to find out whether treatments are safe and effective. The conclusions from systematic reviews are often very influential, and inform both health-care policy and individual treatment decisions. However, it is now clear that the results of many published RCTs are not genuine. In some cases, the entire study may have been fabricated. It is not usual for the veracity of RCTs to be questioned during the process of compiling a systematic review. As a consequence, these "problematic studies" go unnoticed, and are allowed to contribute to the conclusions of influential systematic reviews, thereby influencing patient care. This prompts the question of how these problematic studies could be identified. In this study, we created an extensive list of checks that could be performed to try to identify these studies. We started by assembling a list of checks identified in previous research, and conducting a survey of experts to ask whether they were aware of any additional methods, and to give feedback on the list. As a result, a list of 116 potential "trustworthiness checks" was created. In subsequent research, we will evaluate these checks to see which should be included in a tool, INveStigating ProblEmatic Clinical Trials in Systematic Reviews, which can be used to detect problematic studies.

Assessing the performance of methods for central statistical monitoring of a binary or continuous outcome in multi-center trials: A simulation study

BACKGROUND

Quality study monitoring is fundamental to patient safety and data integrity. Regulators and industry consortia have increasingly advocated for risk-based monitoring (RBM) and central statistical monitoring (CSM) for more effective and efficient monitoring. Assessing which statistical methods underpin these approaches can best identify unusual data patterns in multi-center clinical trials that may be driven by potential systematic errors is important.

METHODS

We assessed various CSM techniques, including cross-tests, fixed-effects, mixed-effects, and finite mixture models, across scenarios with different sample sizes, contamination rates, and overdispersion via simulation. Our evaluation utilized threshold-independent metrics such as the area under the curve (AUC) and average precision (AP), offering a fuller picture of CSM performance.

RESULTS

All CSM methods showed consistent characteristics across center sizes or overdispersion. The adaptive finite mixture model outperformed others in AUC and AP, especially at 30% contamination, upholding high specificity unless converging to a single-component model due to low contamination or deviation. The mixed-effects model performed well at lower contamination rates. However, it became conservative in specificity and exhibited declined performance for binary outcomes under high deviation. Cross-tests and fixed-effects methods underperformed, especially when deviation increased.

CONCLUSION

Our evaluation explored the merits and drawbacks of multiple CSM methods, and found that relying on sensitivity and specificity alone is likely insufficient to fully measure predictive performance. The finite mixture method demonstrated more consistent performance across scenarios by mitigating the influence of outliers. In practice, considering the study-specific costs of false positives/negatives with available resources for monitoring is important.

A survey of experts to identify methods to detect problematic studies: Stage 1 of the INSPECT-SR Project

Background

Randomised controlled trials (RCTs) inform healthcare decisions. Unfortunately, some published RCTs contain false data, and some appear to have been entirely fabricated. Systematic reviews are performed to identify and synthesise all RCTs which have been conducted on a given topic. This means that any of these 'problematic studies' are likely to be included, but there are no agreed methods for identifying them. The INSPECT-SR project is developing a tool to identify problematic RCTs in systematic reviews of healthcare-related interventions. The tool will guide the user through a series of 'checks' to determine a study's authenticity. The first objective in the development process is to assemble a comprehensive list of checks to consider for inclusion.

Methods

We assembled an initial list of checks for assessing the authenticity of research studies, with no restriction to RCTs, and categorised these into five domains: Inspecting results in the paper; Inspecting the research team; Inspecting conduct, governance, and transparency; Inspecting text and publication details; Inspecting the individual participant data. We implemented this list as an online survey, and invited people with expertise and experience of assessing potentially problematic studies to participate through professional networks and online forums. Participants were invited to provide feedback on the checks on the list, and were asked to describe any additional checks they knew of, which were not featured in the list.

Results

Extensive feedback on an initial list of 102 checks was provided by 71 participants based in 16 countries across five continents. Fourteen new checks were proposed across the five domains, and suggestions were made to reword checks on the initial list. An updated list of checks was constructed, comprising 116 checks. Many participants expressed a lack of familiarity with statistical checks, and emphasized the importance of feasibility of the tool.

Conclusions

A comprehensive list of trustworthiness checks has been produced. The checks will be evaluated to determine which should be included in the INSPECT-SR tool.

Application of Statistical Methods for Central Statistical Monitoring and Implementations on the German Multiple Sclerosis Registry

Monitoring of clinical trials is a fundamental process required by regulatory agencies. It assures the compliance of a center to the required regulations and the trial protocol. Traditionally, monitoring teams relied on extensive on-site visits and source data verification. However, this is costly, and the outcome is limited. Thus, central statistical monitoring (CSM) is an additional approach recently embraced by the International Council for Harmonisation (ICH) to detect problematic or erroneous data by using visualizations and statistical control measures. Existing implementations have been primarily focused on detecting inlier and outlier data. Other approaches include principal component analysis and distribution of the data. Here we focus on the utilization of comparisons of centers to the Grand mean for different model types and assumptions for common data types, such as binomial, ordinal, and continuous response variables. We implement the usage of multiple comparisons of single centers to the Grand mean of all centers. This approach is also available for various non-normal data types that are abundant in clinical trials. Further, using confidence intervals, an assessment of equivalence to the Grand mean can be applied. In a Monte Carlo simulation study, the applied statistical approaches have been investigated for their ability to control type I error and the assessment of their respective power for balanced and unbalanced designs which are common in registry data and clinical trials. Data from the German Multiple Sclerosis Registry (GMSR) including proportions of missing data, adverse events and disease severity scores were used to verify the results on Real-World-Data (RWD).

Performance of four centralized statistical monitoring methods for early detection of an atypical center in a multicenter study

Background

Ensuring the quality of data is essential for the credibility of a multicenter clinical trial. Centralized Statistical Monitoring (CSM) of data allows the detection of a center in which the distribution of a specific variable is atypical compared to other centers. The ideal CSM method should allow early detection of problem and therefore involve the fewest possible participants.

Methods

We simulated clinical trials and compared the performance of four CSM methods (Student, Hatayama, Desmet, Distance) to detect whether the distribution of a quantitative variable was atypical in one center in relation to the others, with different numbers of participants and different mean deviation amplitudes.

Results

The Student and Hatayama methods had good sensitivity but poor specificity, which disqualifies them for practical use in CSM. The Desmet and Distance methods had very high specificity for detecting all the mean deviations tested (including small values) but low sensitivity with mean deviations less than 50%.

Conclusion

Although the Student and Hatayama methods are more sensitive, their low specificity would lead to too many alerts being triggered, which would result in additional unnecessary control work to ensure data quality. The Desmet and Distance methods have low sensitivity when the deviation from the mean is low, suggesting that the CSM should be used alongside other conventional monitoring procedures rather than replacing them. However, they have excellent specificity, which suggests they can be applied routinely, since using them takes up no time at central level and does not cause any unnecessary workload in investigating centers.

Guidance on interim analysis methods in clinical trials

Interim analyses in clinical trials can take on a multitude of forms. They are often used to guide Data and Safety Monitoring Board (DSMB) recommendations to study teams regarding recruitment targets for large, later-phase clinical trials. As collaborative biostatisticians working and teaching in multiple fields of research and across a broad array of trial phases, we note the large heterogeneity and confusion surrounding interim analyses in clinical trials. Thus, in this paper, we aim to provide a general overview and guidance on interim analyses for a nonstatistical audience. We explain each of the following types of interim analyses: efficacy, futility, safety, and sample size re-estimation, and we provide the reader with reasoning, examples, and implications for each. We emphasize that while the types of interim analyses employed may differ depending on the nature of the study, we would always recommend prespecification of the interim analytic plan to the extent possible with risk mitigation and trial integrity remaining a priority. Finally, we posit that interim analyses should be used as tools to help the DSMB make informed decisions in the context of the overarching study. They should generally not be deemed binding, and they should not be reviewed in isolation.

Development of a risk-tailored approach and dashboard for efficient management and monitoring of investigator-initiated trials

BACKGROUND

Most randomized controlled trials (RCTs) in the academic setting have limited resources for clinical trial management and monitoring. Inefficient conduct of trials was identified as an important source of waste even in well-designed studies. Thoroughly identifying trial-specific risks to enable focussing of monitoring and management efforts on these critical areas during trial conduct may allow for the timely initiation of corrective action and to improve the efficiency of trial conduct. We developed a risk-tailored approach with an initial risk assessment of an individual trial that informs the compilation of monitoring and management procedures in a trial dashboard.

METHODS

We performed a literature review to identify risk indicators and trial monitoring approaches followed by a contextual analysis involving local, national and international stakeholders. Based on this work we developed a risk-tailored management approach with integrated monitoring for RCTs and including a visualizing trial dashboard. We piloted the approach and refined it in an iterative process based on feedback from stakeholders and performed formal user testing with investigators and staff of two clinical trials.

RESULTS

The developed risk assessment comprises four domains (patient safety and rights, overall trial management, intervention management, trial data). An accompanying manual provides rationales and detailed instructions for the risk assessment. We programmed two trial dashboards tailored to one medical and one surgical RCT to manage identified trial risks based on daily exports of accumulating trial data. We made the code for a generic dashboard available on GitHub that can be adapted to individual trials.

CONCLUSIONS

The presented trial management approach with integrated monitoring enables user-friendly, continuous checking of critical elements of trial conduct to support trial teams in the academic setting. Further work is needed in order to show effectiveness of the dashboard in terms of safe trial conduct and successful completion of clinical trials.

Interactive statistical monitoring to optimize review of potential clinical trial issues during study conduct

Background

Statistical monitoring involves the review of prospective study data collected in participating sites to detect intra/inter patients and sites inconsistencies. We report methods and results of statistical monitoring in a phase IV clinical trial.

Method

PRO-MSACTIVE is a study evaluating ocrelizumab in active relapsing multiple sclerosis (RMS) patients in France. Specific statistical methods (volcano plots, mahalanobis distance, funnel plot …) have been applied to a SDTM database to detect potential issues. R-Shiny application was developed to generate an interactive web application in order to ease site and/or patients identification during statistical data review meetings.

Results

The PRO-MSACTIVE study enrolled 422 patients in 46 centers between July 2018 and August 2019. Three data review meetings were held between April and October 2019 and 14 standard and planned tests were run on study data, with a total of 15 (32.6%) sites identified as needing review or investigation. Overall 36 findings were identified during the meetings: duplicate records, outliers, inconsistent delays between dates.

Conclusion

Statistical monitoring is useful to identify unusual or clustered data patterns that might be revealing issues that could impact the data integrity and/or may potentially impact patients' safety. With anticipated and appropriate interactive data visualization, early signals can easily be identified or reviewed by the study team and appropriate actions be set up and assigned to the most appropriate function for a close follow-up and resolution. Interactive statistical monitoring is time consuming to initiate using R-Shiny, but is time saving after the 1st data review meeting (DRV).(ClinicalTrials.gov identifier: NCT03589105; EudraCT identifier: 2018-000780-91).

Efficacy and safety of a quadruple ultra-low-dose treatment for hypertension (QUARTET USA): Rationale and design for a randomized controlled trial

BACKGROUND

Over half of patients with elevated blood pressure require multi-drug treatment to achieve blood pressure control. However, multi-drug treatment may lead to lower adherence and more adverse drug effects compared with monotherapy.

OBJECTIVE

The Quadruple Ultra-low-dose Treatment for Hypertension (QUARTET) USA trial was designed to evaluate whether initiating treatment with ultra-low-dose quadruple-combination therapy will lower office blood pressure more effectively, and with fewer side effects, compared with initiating standard dose monotherapy in treatment naive patients with SBP < 180 and DBP < 110 mm Hg and patients on monotherapy with SBP < 160 and DBP < 100 mm Hg.

METHODS/DESIGN

QUARTET USA was a prospective, randomized, double-blind trial (ClinicalTrials.gov NCT03640312) conducted in federally qualified health centers in a large city in the US. Patients were randomly assigned (1:1) to either ultra-low-dose quadruple combination therapy or standard dose monotherapy. The primary outcome was mean change from baseline in office systolic blood pressure at 12-weeks, adjusted for baseline values. Secondary outcomes included measures of blood pressure change and variability, medication adherence, and health related quality of life. Safety outcomes included occurrence of serious adverse events, relevant adverse drug effects, and electrolyte abnormalities. A process evaluation aimed to understand provider experiences of implementation and participant experiences around side effects, adherence, and trust with clinical care.

DISCUSSION

QUARTET USA was designed to evaluate whether a novel approach to blood pressure control would lower office blood pressure more effectively, and with fewer side effects, compared with standard dose monotherapy. QUARTET USA was conducted within a network of federally qualified healthcare centers with the aim of generating information on the safety and efficacy of ultra-low-dose quadruple-combination therapy in diverse groups that experience a high burden of hypertension.

Consistency of covid-19 trial preprints with published reports and impact for decision making: retrospective review

Objective

To assess the trustworthiness (ie, complete and consistent reporting of key methods and results between preprint and published trial reports) and impact (ie, effects of preprints on meta-analytic estimates and the certainty of evidence) of preprint trial reports during the covid-19 pandemic.

Design

Retrospective review.

Data sources

World Health Organization covid-19 database and the Living Overview of the Evidence (L-OVE) covid-19 platform by the Epistemonikos Foundation (up to 3 August 2021).

Main outcome measures

Comparison of characteristics of covid-19 trials with and without preprints, estimates of time to publication of covid-19 preprints, and description of differences in reporting of key methods and results between preprints and their later publications. For the effects of eight treatments on mortality and mechanical ventilation, the study comprised meta-analyses including preprints and excluding preprints at one, three, and six months after the first trial addressing the treatment became available either as a preprint or publication (120 meta-analyses in total, 60 of which included preprints and 60 of which excluded preprints) and assessed the certainty of evidence using the GRADE framework.

Results

Of 356 trials included in the study, 101 were only available as preprints, 181 as journal publications, and 74 as preprints first and subsequently published in journals. The median time to publication of preprints was about six months. Key methods and results showed few important differences between trial preprints and their subsequent published reports. Apart from two (3.3%) of 60 comparisons, point estimates were consistent between meta-analyses including preprints versus those excluding preprints as to whether they indicated benefit, no appreciable effect, or harm. For nine (15%) of 60 comparisons, the rating of the certainty of evidence was different when preprints were included versus being excluded-the certainty of evidence including preprints was higher in four comparisons and lower in five comparisons.

Conclusion

No compelling evidence indicates that preprints provide results that are inconsistent with published papers. Preprints remain the only source of findings of many trials for several months-an unsuitable length of time in a health emergency that is not conducive to treating patients with timely evidence. The inclusion of preprints could affect the results of meta-analyses and the certainty of evidence. Evidence users should be encouraged to consider data from preprints.

Assessing Research Misconduct in Randomized Controlled Trials

Randomized controlled trials (RCTs) serve as the pillar of evidence-based medicine and guide medical practice. Compromised data integrity in RCTs undermines the authority of this valuable tool for science and puts patients at risk. Although a large number of retractions due to data issues in obstetrics and gynecology have occurred in the past few years, many problematic RCTs could still go uncovered because in general there is insufficient willingness to envisage and confront research misconduct. In this article, we discuss the necessity of assessing research misconduct, summarize methods that have been applied in detecting previous cases of misconduct, and propose potential solutions. There is no established mechanism to monitor feedback on published articles and the current system that handles potential research misconduct is unsatisfactory. Fortunately, there are methods to assess data integrity in RCTs both with and without individual participant data. Investigations into research misconduct can be facilitated by assessing all publications from a leading author or author group to identify duplication and patterns of ongoing misconduct. There is a pressing need to improve the mechanism that investigates data manipulation. The mechanism that handles misconduct should prioritize the interests of patients and readers rather than trial authors and their institutions. An equally urgent issue is to establish mechanisms that prevent compromised trials from polluting evidence synthesis or misguiding practice.

Central data monitoring in the multicentre randomised SafeBoosC-III trial - a pragmatic approach

BACKGROUND

Data monitoring of clinical trials is a tool aimed at reducing the risks of random errors (e.g. clerical errors) and systematic errors, which include misinterpretation, misunderstandings, and fabrication. Traditional 'good clinical practice data monitoring' with on-site monitors increases trial costs and is time consuming for the local investigators. This paper aims to outline our approach of time-effective central data monitoring for the SafeBoosC-III multicentre randomised clinical trial and present the results from the first three central data monitoring meetings.

METHODS

The present approach to central data monitoring was implemented for the SafeBoosC-III trial, a large, pragmatic, multicentre, randomised clinical trial evaluating the benefits and harms of treatment based on cerebral oxygenation monitoring in preterm infants during the first days of life versus monitoring and treatment as usual. We aimed to optimise completeness and quality and to minimise deviations, thereby limiting random and systematic errors. We designed an automated report which was blinded to group allocation, to ease the work of data monitoring. The central data monitoring group first reviewed the data using summary plots only, and thereafter included the results of the multivariate Mahalanobis distance of each centre from the common mean. The decisions of the group were manually added to the reports for dissemination, information, correcting errors, preventing furture errors and documentation.

RESULTS

The first three central monitoring meetings identified 156 entries of interest, decided upon contacting the local investigators for 146 of these, which resulted in correction of 53 entries. Multiple systematic errors and protocol violations were identified, one of these included 103/818 randomised participants. Accordingly, the electronic participant record form (ePRF) was improved to reduce ambiguity.

DISCUSSION

We present a methodology for central data monitoring to optimise quality control and quality development. The initial results included identification of random errors in data entries leading to correction of the ePRF, systematic protocol violations, and potential protocol adherence issues. Central data monitoring may optimise concurrent data completeness and may help timely detection of data deviations due to misunderstandings or fabricated data.

Methods to assess research misconduct in health-related research: A scoping review

OBJECTIVE

To give an overview of the available methods to investigate research misconduct in health-related research.

STUDY DESIGN AND SETTING

In this scoping review, we conducted a literature search in MEDLINE, Embase, The Cochrane CENTRAL Register of Studies Online (CRSO), and The Virtual Health Library portal up to July 2020. We included papers that mentioned and/or described methods for screening or assessing research misconduct in health-related research. We categorized identified methods into the following four groups according to their scopes: overall concern, textual concern, image concern, and data concern.

RESULTS

We included 57 papers reporting on 27 methods: two on overall concern, four on textual concern, three on image concern, and 18 on data concern. Apart from the methods to locate textual plagiarism and image manipulation, all other methods, be it theoretical or empirical, are based on examples, are not standardized, and lack formal validation.

CONCLUSION

Existing methods cover a wide range of issues regarding research misconduct. Although measures to counteract textual plagiarism are well implemented, tools to investigate other forms of research misconduct are rudimentary and labour-intensive. To cope with the rising challenge of research misconduct, further development of automatic tools and routine validation of these methods is needed.

TRIAL REGISTRATION NUMBER

Center for Open Science (OSF) (https://osf.io/mq89w).

Risk-based centralized data monitoring of clinical trials at the time of COVID-19 pandemic

OBJECTIVES

COVID-19 pandemic caused several alarming challenges for clinical trials. On-site source data verification (SDV) in the multicenter clinical trial became difficult due to travel ban and social distancing. For multicenter clinical trials, centralized data monitoring is an efficient and cost-effective method of data monitoring. Centralized data monitoring reduces the risk of COVID-19 infections and provides additional capabilities compared to on-site monitoring. The key steps for on-site monitoring include identifying key risk factors and thresholds for the risk factors, developing a monitoring plan, following up the risk factors, and providing a management plan to mitigate the risk.

METHODS

For analysis purposes, we simulated data similar to our clinical trial data. We classified the data monitoring process into two groups, such as the Supervised analysis process, to follow each patient remotely by creating a dashboard and an Unsupervised analysis process to identify data discrepancy, data error, or data fraud. We conducted several risk-based statistical analysis techniques to avoid on-site source data verification to reduce time and cost, followed up with each patient remotely to maintain social distancing, and created a centralized data monitoring dashboard to ensure patient safety and maintain the data quality.

CONCLUSION

Data monitoring in clinical trials is a mandatory process. A risk-based centralized data review process is cost-effective and helpful to ignore on-site data monitoring at the time of the pandemic. We summarized how different statistical methods could be implemented and explained in SAS to identify various data error or fabrication issues in multicenter clinical trials.

Dynamic methods for ongoing assessment of site-level risk in risk-based monitoring of clinical trials: A scoping review

Background/Aims

It is increasingly recognised that reliance on frequent site visits for monitoring clinical trials is inefficient. Regulators and trialists have recently encouraged more risk-based monitoring. Risk assessment should take place before a trial begins to define the overarching monitoring strategy. It can also be done on an ongoing basis, to target sites for monitoring activity. Various methods have been proposed for such prioritisation, often using terms like ‘central statistical monitoring’, ‘triggered monitoring’ or, as in the International Conference on Harmonization Good Clinical Practice guidance, ‘targeted on-site monitoring’. We conducted a scoping review to identify such methods, to establish if any were supported by adequate evidence to allow wider implementation, and to guide future developments in this field of research.

Methods

We used seven publication databases, two sets of methodological conference abstracts and an Internet search engine to identify methods for using centrally held trial data to assess site conduct during a trial. We included only reports in English, and excluded reports published before 1996 or not directly relevant to our research question. We used reference and citation searches to find additional relevant reports. We extracted data using a predefined template. We contacted authors to request additional information about included reports.

Results

We included 30 reports in our final dataset, of which 21 were peer-reviewed publications. In all, 20 reports described central statistical monitoring methods (of which 7 focussed on detection of fraud or misconduct) and 9 described triggered monitoring methods; 21 reports included some assessment of their methods’ effectiveness, typically exploring the methods’ characteristics using real trial data without known integrity issues. Of the 21 with some effectiveness assessment, most contained limited information about whether or not concerns identified through central monitoring constituted meaningful problems. Several reports demonstrated good classification ability based on more than one classification statistic, but never without caveats of unclear reporting or other classification statistics being low or unavailable. Some reports commented on cost savings from reduced on-site monitoring, but none gave detailed costings for the development and maintenance of central monitoring methods themselves.

Conclusion

Our review identified various proposed methods, some of which could be combined within the same trial. The apparent emphasis on fraud detection may not be proportionate in all trial settings. Despite some promising evidence and some self-justifying benefits for data cleaning activity, many proposed methods have limitations that may currently prevent their routine use for targeting trial monitoring activity. The implementation costs, or uncertainty about these, may also be a barrier. We make recommendations for how the evidence-base supporting these methods could be improved.

Design and conduct of confirmatory chronic pain clinical trials

The purpose of this article is to provide readers with a basis for understanding the emerging science of clinical trials and to provide a set of practical, evidence-based suggestions for designing and executing confirmatory clinical trials in a manner that minimizes measurement error. The most important step in creating a mindset of quality clinical research is to abandon the antiquated concept that clinical trials are a method for capturing data from clinical practice and shifting to a concept of the clinical trial as a measurement system, consisting of an interconnected set of processes, each of which must be in calibration for the trial to generate an accurate and reliable estimate of the efficacy (and safety) of a given treatment. The status quo of inaccurate, unreliable, and protracted clinical trials is unacceptable and unsustainable. This article gathers aspects of study design and conduct under a single broad umbrella of techniques available to improve the accuracy and reliability of confirmatory clinical trials across traditional domain boundaries.

Improving Study Conduct and Data Quality in Clinical Trials of Chronic Pain Treatments: IMMPACT Recommendations

The estimated probability of progressing from phase 3 analgesic clinical trials to regulatory approval is approximately 57%, suggesting that a considerable number of treatments with phase 2 trial results deemed sufficiently successful to progress to phase 3 do not yield positive phase 3 results. Deficiencies in the quality of clinical trial conduct could account for some of this failure. An Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials meeting was convened to identify potential areas for improvement in trial conduct in order to improve assay sensitivity (ie, ability of trials to detect a true treatment effect). We present recommendations based on presentations and discussions at the meeting, literature reviews, and iterative revisions of this article. The recommendations relate to the following areas: 1) study design (ie, to promote feasibility), 2) site selection and staff training, 3) participant selection and training, 4) treatment adherence, 5) data collection, and 6) data and study monitoring. Implementation of these recommendations may improve the quality of clinical trial data and thus the validity and assay sensitivity of clinical trials. Future research regarding the effects of these strategies will help identify the most efficient use of resources for conducting high quality clinical trials. PERSPECTIVE: Every effort should be made to optimize the quality of clinical trial data. This manuscript discusses considerations to improve conduct of pain clinical trials based on research in multiple medical fields and the expert consensus of pain researchers and stakeholders from academia, regulatory agencies, and industry.

Bayesian central statistical monitoring using finite mixture models in multicenter clinical trials

Background

Central monitoring (CM), in which data across all clinical sites are monitored, has an important role in risk-based monitoring. Several statistical methods have been proposed to compare patient outcomes among the sites for detecting atypical sites that have different trends in observed data. These methods assume that the number of clinical sites is not small, e.g., 100 or more. In addition, the proportion of atypical sites is assumed to be relatively small. However, in actuality, the central statistical monitoring (CSM) has to be implemented in small or moderate sized clinical trials such as small phase II clinical trials. The number of sites is no longer large in such situations. Therefore, it is of concern that existing methods may not work efficiently in CM of small or moderate sized clinical trials. In the light of this problem, we propose a Bayesian CSM method to detect atypical sites as the robust method against the existence of atypical sites.

Methods

We use Bayesian finite mixture models (FMM) to model patient outcome values of both atypical and typical sites. In the method, the distributions of outcome values in normal sites are determined by choosing the body distribution, which has the largest mixture parameter value of finite mixture models based on the assumption that normal sites are in the majority. Atypical sites are detected by the criterion based on the posterior predictive distribution of normal site's outcome values derived from only the chosen body distribution.

Results

Proposed method is evaluated by cumulative detection probability and type I error averaged over sites every round of CSM under the various scenarios, being compared with the conventional type analysis. If the total number of patients enrolled is 48, the proposed method is superior at least 10% for any shift sizes at the 2nd and the 3rd rounds. If the total number of patients is 96, both methods show similar detection probability for only one atypical site and large shift size. However, the proposed method is superior for the other scenarios. It is observed that all the type I errors averaged over sites are little difference between the methods at all the scenarios.

Conclusion

We propose a Bayesian CSM method which works efficiently in a practical use of CM. It is shown that our method detects atypical sites with high probability regardless of the proportion of the atypical sites under the small clinical trial settings which is the target of our proposed method.

A randomized evaluation of on-site monitoring nested in a multinational randomized trial

BACKGROUND

Evidence from prospectively designed studies to guide on-site monitoring practices for randomized trials is limited. A cluster randomized study, nested within the Strategic Timing of AntiRetroviral Treatment (START) trial, was conducted to evaluate on-site monitoring.

METHODS

Sites were randomized to either annual on-site monitoring or no on-site monitoring. All sites were centrally monitored, and local monitoring was carried out twice each year. Randomization was stratified by country and projected enrollment in START. The primary outcome was a participant-level composite outcome including components for eligibility errors, consent violations, use of antiretroviral treatment not recommended by protocol, late reporting of START primary and secondary clinical endpoints (defined as the event being reported more than 6 months from occurrence), and data alteration and fraud. Logistic regression fixed effect hierarchical models were used to compare on-site versus no on-site monitoring for the primary composite outcome and its components. Odds ratios and 95% confidence intervals comparing on-site monitoring versus no on-site monitoring are cited.

RESULTS

In total, 99 sites (2107 participants) were randomized to receive annual on-site monitoring and 97 sites (2264 participants) were randomized to be monitored only centrally and locally. The two monitoring groups were well balanced at entry. In the on-site monitoring group, 469 annual on-site monitoring visits were conducted, and 134 participants (6.4%) in 56 of 99 sites (57%) had a primary monitoring outcome. In the no on-site monitoring group, 85 participants (3.8%) in 34 of 97 sites (35%) had a primary monitoring outcome (odds ratio = 1.7; 95% confidence interval: 1.1-2.7; p = 0.03). Informed consent violations accounted for most outcomes in each group (56 vs 41 participants). The largest odds ratio was for eligibility violations (odds ratio = 12.2; 95% confidence interval: 1.8-85.2; p = 0.01). The number of participants with a late START primary endpoint was similar for each monitoring group (23 vs 16 participants). Late START grade 4 and unscheduled hospitalization events were found for 34 participants in the on-site monitoring group and 19 participants in the no on-site monitoring group (odds ratio = 2.0; 95% confidence interval: 1.1-3.7; p = 0.02). There were no cases of data alteration or fraud. Based on the travel budget for on-site monitoring and the hours spent conducting on-site monitoring, the estimated cost of on-site monitoring was over US$2 million.

CONCLUSION

On-site monitoring led to the identification of more eligibility and consent violations and START clinical events being reported more than 6 months from occurrence as compared to no on-site monitoring. Considering the nature of the excess monitoring outcomes identified at sites receiving on-site monitoring, as well as the cost of on-site monitoring, the value to the START study was limited.

Creating an academic research organization to efficiently design, conduct, coordinate, and analyze clinical trials: The Center for Clinical Trials & Data Coordination

When properly executed, the randomized controlled trial is one of the best vehicles for assessing the effectiveness of one or more interventions. However, numerous challenges may emerge in the areas of study startup, recruitment, data quality, cost, and reporting of results. The use of well-run coordinating centers could help prevent these issues, but very little exists in the literature describing their creation or the guiding principles behind their inception. The Center for Clinical Trials & Data Coordination (CCDC) was established in 2015 through institutional funds with the intent of 1) providing relevant expertise in clinical trial design, conduct, coordination, and analysis; 2) advancing the careers of clinical investigators and CCDC-affiliated faculty; and 3) obtaining large data coordinating center (DCC) grants. We describe the organizational structure of the CCDC as well as the homegrown clinical trial management system integrating nine crucial elements: electronic data capture, eligibility and randomization, drug and external data tracking, safety reporting, outcome adjudication, data and safety monitoring, statistical analysis and reporting, data sharing, and regulatory compliance. Lastly, we share numerous lessons that can be taken from our experience. Specifically, we focus on 1) funding for DCCs, 2) the importance of DCCs to clinical researchers, 3) the expertise of DCC personnel, and 4) continually striving to improve. In conclusion, the CCDC strives to provide high-quality support for the design, conduct, coordination, and analyses of clinical trials, and we hope this paper will serve as a blueprint for future clinical trialists involved in DCCs.

Detection of atypical data in multicenter clinical trials using unsupervised statistical monitoring

BACKGROUND/AIMS

A risk-based approach to clinical research may include a central statistical assessment of data quality. We investigated the operating characteristics of unsupervised statistical monitoring aimed at detecting atypical data in multicenter experiments. The approach is premised on the assumption that, save for random fluctuations and natural variations, data coming from all centers should be comparable and statistically consistent. Unsupervised statistical monitoring consists of performing as many statistical tests as possible on all trial data, in order to detect centers whose data are inconsistent with data from other centers.

METHODS

We conducted simulations using data from a large multicenter trial conducted in Japan for patients with advanced gastric cancer. The actual trial data were contaminated in computer simulations for varying percentages of centers, percentages of patients modified within each center and numbers and types of modified variables. The unsupervised statistical monitoring software was run by a blinded team on the contaminated data sets, with the purpose of detecting the centers with contaminated data. The operating characteristics (sensitivity, specificity and Youden's J-index) were calculated for three detection methods: one using the p-values of individual statistical tests after adjustment for multiplicity, one using a summary of all p-values for a given center, called the Data Inconsistency Score, and one using both of these methods.

RESULTS

The operating characteristics of the three methods were satisfactory in situations of data contamination likely to occur in practice, specifically when a single or a few centers were contaminated. As expected, the sensitivity increased for increasing proportions of patients and increasing numbers of variables contaminated. The three methods showed a specificity better than 93% in all scenarios of contamination. The method based on the Data Inconsistency Score and individual p-values adjusted for multiplicity generally had slightly higher sensitivity at the expense of a slightly lower specificity.

CONCLUSIONS

The use of brute force (a computer-intensive approach that generates large numbers of statistical tests) is an effective way to check data quality in multicenter clinical trials. It can provide a cost-effective complement to other data-management and monitoring techniques.

Implementing monitoring triggers and matching of triggered and control sites in the TEMPER study: a description and evaluation of a triggered monitoring management system

BACKGROUND

Triggered monitoring in clinical trials is a risk-based monitoring approach where triggers (centrally monitored, predefined key risk and performance indicators) drive the extent, timing, and frequency of monitoring visits. The TEMPER study used a prospective, matched-pair design to evaluate the use of a triggered monitoring strategy, comparing findings from triggered monitoring visits with those from matched control sites. To facilitate this study, we developed a bespoke risk-based monitoring system: the TEMPER Management System.

METHODS

The TEMPER Management System comprises a web application (the front end), an SQL server database (the back end) to store the data generated for TEMPER, and a reporting function to aid users in study processes such as the selection of triggered sites. Triggers based on current practice were specified for three clinical trials and were implemented in the system. Trigger data were generated in the system using data extracted from the trial databases to inform the selection of triggered sites to visit. Matching of the chosen triggered sites with untriggered control sites was also performed in the system, while data entry screens facilitated the collection and management of the data from findings gathered at monitoring visits.

RESULTS

There were 38 triggers specified for the participating trials. Using these, 42 triggered sites were chosen and matched with control sites. Monitoring visits were carried out to all sites, and visit findings were entered into the TEMPER Management System. Finally, data extracted from the system were used for analysis.

CONCLUSIONS

The TEMPER Management System made possible the completion of the TEMPER study. It implemented an approach of standardising the automation of current-practice triggers, and the generation of trigger data to inform the selection of triggered sites to visit. It also implemented a matching algorithm informing the selection of matched control sites. We hope that by publishing this paper it encourages other trialists to share their approaches to, and experiences of, triggered monitoring and other risk-based monitoring systems.

Monitoring performance of sites within multicentre randomised trials: a systematic review of performance metrics

Background

Large multicentre trials are complex and expensive projects. A key factor for their successful planning and delivery is how well sites meet their targets in recruiting and retaining participants, and in collecting high-quality, complete data in a timely manner. Collecting and monitoring easily accessible data relevant to performance of sites has the potential to improve trial management efficiency. The aim of this systematic review was to identify metrics that have either been proposed or used for monitoring site performance in multicentre trials.

Methods

We searched the Cochrane Library, five biomedical bibliographic databases (CINAHL, EMBASE, Medline, PsychINFO and SCOPUS) and Google Scholar for studies describing ways of monitoring or measuring individual site performance in multicentre randomised trials. Records identified were screened for eligibility. For included studies, data on study content were extracted independently by two reviewers, and disagreements resolved by discussion.

Results

After removing duplicate citations, we identified 3188 records. Of these, 21 were eligible for inclusion and yielded 117 performance metrics. The median number of metrics reported per paper was 8, range 1–16. Metrics broadly fell into six categories: site potential; recruitment; retention; data collection; trial conduct and trial safety.

Conclusions

This review identifies a list of metrics to monitor site performance within multicentre randomised trials. Those that would be easy to collect, and for which monitoring might trigger actions to mitigate problems at site level, merit further evaluation.

Harmonization, data management, and statistical issues related to prospective multicenter studies in Ankylosing spondylitis (AS): Experience from the Prospective Study Of Ankylosing Spondylitis (PSOAS) cohort

Ankylosing spondylitis (AS) is characterized by inflammation of the spine and sacroiliac joints causing pain and stiffness and, in some patients, ultimately new bone formation, and progressive joint ankyloses. The classical definition of AS is based on the modified New York (mNY) criteria. Limited data have been reported regarding data quality assurance procedure for multicenter or multisite prospective cohort of patients with AS. Since 2002, 1272 qualified AS patients have been enrolled from five sites (4 US sites and 1 Australian site) in the Prospective Study Of Ankylosing Spondylitis (PSOAS). In 2012, a Data Management and Statistical Core (DMSC) was added to the PSOAS team to assist in study design, establish a systematic approach to data management and data quality, and develop and apply appropriate statistical analysis of data. With assistance from the PSOAS investigators, DMSC modified Case Report Forms and developed database in Research Electronic Data Capture (REDCap). DMSC also developed additional data quality assurance procedure to assure data quality. The error rate for various forms in PSOAS databases ranged from 0.07% for medications data to 1.1% for arthritis activity questionnaire-Global pain. Furthermore, based on data from a sub study of 48 patients with AS, we showed a strong level (90.0%) of agreement between the two readers of X-rays with respect to modified Stoke Ankylosing Spondylitis Spine Score (mSASSS). This paper not only could serve as reference for future publications from PSOAS cohort but also could serve as a basic guide to ensuring data quality for multicenter clinical studies.

Effect of a Quality Improvement Intervention on Clinical Outcomes in Patients in India With Acute Myocardial Infarction: The ACS QUIK Randomized Clinical Trial

IMPORTANCE

Wide heterogeneity exists in acute myocardial infarction treatment and outcomes in India.

OBJECTIVE

To evaluate the effect of a locally adapted quality improvement tool kit on clinical outcomes and process measures in Kerala, a southern Indian state.

DESIGN, SETTING, AND PARTICIPANTS

Cluster randomized, stepped-wedge clinical trial conducted between November 10, 2014, and November 9, 2016, in 63 hospitals in Kerala, India, with a last date of follow-up of December 31, 2016. During 5 predefined steps over the study period, hospitals were randomly selected to move in a 1-way crossover from the control group to the intervention group. Consecutively presenting patients with acute myocardial infarction were offered participation.

INTERVENTIONS

Hospitals provided either usual care (control group; n = 10 066 participants [step 0: n = 2915; step 1: n = 2649; step 2: n = 2251; step 3: n = 1422; step 4; n = 829; step 5: n = 0]) or care using a quality improvement tool kit (intervention group; n = 11 308 participants [step 0: n = 0; step 1: n = 662; step 2: n = 1265; step 3: n = 2432; step 4: n = 3214; step 5: n = 3735]) that consisted of audit and feedback, checklists, patient education materials, and linkage to emergency cardiovascular care and quality improvement training.

MAIN OUTCOMES AND MEASURES

The primary outcome was the composite of all-cause death, reinfarction, stroke, or major bleeding using standardized definitions at 30 days. Secondary outcomes included the primary outcome's individual components, 30-day cardiovascular death, medication use, and tobacco cessation counseling. Mixed-effects logistic regression models were used to account for clustering and temporal trends.

RESULTS

Among 21 374 eligible randomized participants (mean age, 60.6 [SD, 12.0] years; n = 16 183 men [76%] ; n = 13 689 [64%] with ST-segment elevation myocardial infarction), 21 079 (99%) completed the trial. The primary composite outcome was observed in 5.3% of the intervention participants and 6.4% of the control participants. The observed difference in 30-day major adverse cardiovascular event rates between the groups was not statistically significant after adjustment (adjusted risk difference, -0.09% [95% CI, -1.32% to 1.14%]; adjusted odds ratio, 0.98 [95% CI, 0.80-1.21]). The intervention group had a higher rate of medication use including reperfusion but no effect on tobacco cessation counseling. There were no unexpected adverse events reported.

CONCLUSIONS AND RELEVANCE

Among patients with acute myocardial infarction in Kerala, India, use of a quality improvement intervention compared with usual care did not decrease a composite of 30-day major adverse cardiovascular events. Further research is needed to understand the lack of efficacy.

TRIAL REGISTRATION

clinicaltrials.gov Identifier: NCT02256657.

Truths, lies, and statistics

Distribution of valuable research discoveries are needed for the continual advancement of patient care. Publication and subsequent reliance of false study results would be detrimental for patient care. Unfortunately, research misconduct may originate from many sources. While there is evidence of ongoing research misconduct in all it's forms, it is challenging to identify the actual occurrence of research misconduct, which is especially true for misconduct in clinical trials. Research misconduct is challenging to measure and there are few studies reporting the prevalence or underlying causes of research misconduct among biomedical researchers. Reported prevalence estimates of misconduct are probably underestimates, and range from 0.3% to 4.9%. There have been efforts to measure the prevalence of research misconduct; however, the relatively few published studies are not freely comparable because of varying characterizations of research misconduct and the methods used for data collection. There are some signs which may point to an increased possibility of research misconduct, however there is a need for continued self-policing by biomedical researchers. There are existing resources to assist in ensuring appropriate statistical methods and preventing other types of research fraud. These included the "Statistical Analyses and Methods in the Published Literature", also known as the SAMPL guidelines, which help scientists determine the appropriate method of reporting various statistical methods; the "Strengthening Analytical Thinking for Observational Studies", or the STRATOS, which emphases on execution and interpretation of results; and the Committee on Publication Ethics (COPE), which was created in 1997 to deliver guidance about publication ethics. COPE has a sequence of views and strategies grounded in the values of honesty and accuracy.

An evaluation of the effectiveness of a risk-based monitoring approach implemented with clinical trials involving implantable cardiac medical devices

Background

Risk-based monitoring is a concept endorsed by the Food and Drug Administration to improve clinical trial data quality by focusing monitoring efforts on critical data elements and higher risk investigator sites. BIOTRONIK approached this by implementing a comprehensive strategy that assesses risk and data quality through a combination of operational controls and data surveillance. This publication demonstrates the effectiveness of a data-driven risk assessment methodology when used in conjunction with a tailored monitoring plan.

Methods

We developed a data-driven risk assessment system to rank 133 investigator sites comprising 3442 subjects and identify those sites that pose a potential risk to the integrity of data collected in implantable cardiac device clinical trials. This included identification of specific risk factors and a weighted scoring mechanism. We conducted trend analyses for risk assessment data collected over 1 year to assess the overall impact of our data surveillance process combined with other operational monitoring efforts.

Results

Trending analyses of key risk factors revealed an improvement in the quality of data collected during the observation period. The three risk factors follow-up compliance rate, unavailability of critical data, and noncompliance rate correspond closely with Food and Drug Administration’s risk-based monitoring guidance document. Among these three risk factors, 100% (12/12) of quantiles analyzed showed an increase in data quality. Of these, 67% (8/12) of the improving trends in worst performing quantiles had p-values less than 0.05, and 17% (2/12) had p-values between 0.05 and 0.06. Among the poorest performing site quantiles, there was a statistically significant decrease in subject follow-up noncompliance rates, protocol noncompliance rates, and incidence of missing critical data.

Conclusion

One year after implementation of a comprehensive strategy for risk-based monitoring, including a data-driven risk assessment methodology to target on-site monitoring visits, statistically significant improvement was seen in a majority of measurable risk factors at the worst performing site quantiles. For the three risk factors which are most critical to the overall compliance of cardiac rhythm management medical device studies: follow-up compliance rate, unavailability of critical data, and noncompliance rate, we measured significant improvement in data quality. Although the worst performing site quantiles improved but not significantly in some risk factors such as subject attrition, the data-driven risk assessment highlighted key areas on which to continue focusing both on-site and centralized monitoring efforts. Data-driven surveillance of clinical trial performance provides actionable observations that can improve site performance. Clinical trials utilizing risk-based monitoring by leveraging a data-driven quality assessment combined with specific operational procedures may lead to an improvement in data quality and resource efficiencies.

The impact of data errors on the outcome of randomized clinical trials

Background/aims Considerable human and financial resources are typically spent to ensure that data collected for clinical trials are free from errors. We investigated the impact of random and systematic errors on the outcome of randomized clinical trials. Methods We used individual patient data relating to response endpoints of interest in two published randomized clinical trials, one in ophthalmology and one in oncology. These randomized clinical trials enrolled 1186 patients with age-related macular degeneration and 736 patients with metastatic colorectal cancer. The ophthalmology trial tested the benefit of pegaptanib for the treatment of age-related macular degeneration and identified a statistically significant treatment benefit, whereas the oncology trial assessed the benefit of adding cetuximab to a regimen of capecitabine, oxaliplatin, and bevacizumab for the treatment of metastatic colorectal cancer and failed to identify a statistically significant treatment difference. We simulated trial results by adding errors that were independent of the treatment group (random errors) and errors that favored one of the treatment groups (systematic errors). We added such errors to the data for the response endpoint of interest for increasing proportions of randomly selected patients. Results Random errors added to up to 50% of the cases produced only slightly inflated variance in the estimated treatment effect of both trials, with no qualitative change in the p-value. In contrast, systematic errors produced bias even for very small proportions of patients with added errors. Conclusion A substantial amount of random errors is required before appreciable effects on the outcome of randomized clinical trials are noted. In contrast, even a small amount of systematic errors can severely bias the estimated treatment effects. Therefore, resources devoted to randomized clinical trials should be spent primarily on minimizing sources of systematic errors which can bias the analyses, rather than on random errors which result only in a small loss in power.

Acute coronary syndrome quality improvement in Kerala (ACS QUIK): Rationale and design for a cluster-randomized stepped-wedge trial

Ischemic heart disease is the leading cause of death in India, and there are likely more myocardial infarctions in India than in any other country in the world. We have previously reported heterogeneous care for patients with myocardial infarction in Kerala, a state in southern India, including both gaps in optimal care and inappropriate care. Based on that prior work, limitations from previous nonrandomized quality improvement studies and promising gains in process of care measures demonstrated from previous randomized trials, we and the Cardiological Society of India-Kerala chapter sought to develop, implement, and evaluate a quality improvement intervention to improve process of care measures and clinical outcomes for these patients. In this article, we report the rationale and study design for the ACS QUIK cluster-randomized stepped-wedge clinical trial (NCT02256657) in which we aim to enroll 15,750 participants with acute coronary syndromes across 63 hospitals. To date, most participants are men (76%) and have ST-segment elevation myocardial infarction (63%). The primary outcome is 30-day major adverse cardiovascular events defined as death, recurrent infarction, stroke, or major bleeding. Our secondary outcomes include health-related quality of life and individual- and household-level costs. We also describe the principal features and limitations of the stepped-wedge study design, which may be important for other investigators or sponsors considering cluster-randomized stepped-wedge trials.

The impact of clinical trial monitoring approaches on data integrity and cost--a review of current literature

PURPOSE

Monitoring is a costly requirement when conducting clinical trials. New regulatory guidance encourages the industry to consider alternative monitoring methods to the traditional 100 % source data verification (SDV) approach. The purpose of this literature review is to provide an overview of publications on different monitoring methods and their impact on subject safety data, data integrity, and monitoring cost.

METHODS

The literature search was performed by keyword searches in MEDLINE and hand search of key journals. All publications were reviewed for details on how a monitoring approach impacted subject safety data, data integrity, or monitoring costs.

RESULTS

Twenty-two publications were identified. Three publications showed that SDV has some value for detection of not initially reported adverse events and centralized statistical monitoring (CSM) captures atypical trends. Fourteen publications showed little objective evidence of improved data integrity with traditional monitoring such as 100 % SDV and sponsor queries as compared to reduced SDV, CSM, and remote monitoring. Eight publications proposed a potential for significant cost reductions of monitoring by reducing SDV without compromising the validity of the trial results.

CONCLUSIONS

One hundred percent SDV is not a rational method of ensuring data integrity and subject safety based on the high cost, and this literature review indicates that reduced SDV is a viable monitoring method. Alternative methods of monitoring such as centralized monitoring utilizing statistical tests are promising alternatives but have limitations as stand-alone tools. Reduced SDV combined with a centralized, risk-based approach may be the ideal solution to reduce monitoring costs while improving essential data quality.

Statistical monitoring of data quality and consistency in the Stomach Cancer Adjuvant Multi-institutional Trial Group Trial

INTRODUCTION

Data quality may impact the outcome of clinical trials; hence, there is a need to implement quality control strategies for the data collected. Traditional approaches to quality control have primarily used source data verification during on-site monitoring visits, but these approaches are hugely expensive as well as ineffective. There is growing interest in central statistical monitoring (CSM) as an effective way to ensure data quality and consistency in multicenter clinical trials.

METHODS

CSM with SMART™ uses advanced statistical tools that help identify centers with atypical data patterns which might be the sign of an underlying quality issue. This approach was used to assess the quality and consistency of the data collected in the Stomach Cancer Adjuvant Multi-institutional Trial Group Trial, involving 1495 patients across 232 centers in Japan.

RESULTS

In the Stomach Cancer Adjuvant Multi-institutional Trial Group Trial, very few atypical data patterns were found among the participating centers, and none of these patterns were deemed to be related to a quality issue that could significantly affect the outcome of the trial.

DISCUSSION

CSM can be used to provide a check of the quality of the data from completed multicenter clinical trials before analysis, publication, and submission of the results to regulatory agencies. It can also form the basis of a risk-based monitoring strategy in ongoing multicenter trials. CSM aims at improving data quality in clinical trials while also reducing monitoring costs.

Statistical challenges for central monitoring in clinical trials: a review

Recently, the complexity and costs of clinical trials have increased dramatically, especially in the area of new drug development. Risk-based monitoring (RBM) has been attracting attention as an efficient and effective trial monitoring approach, which can be applied irrespectively of the trial sponsor, i.e., academic institution or pharmaceutical company. In the RBM paradigm, it is expected that a statistical approach to central monitoring can help improve the effectiveness of on-site monitoring by prioritizing and guiding site visits according to central statistical data checks, as evidenced by examples of actual trial datasets. In this review, several statistical methods for central monitoring are presented. It is important to share knowledge about the role and performance capabilities of statistical methodology among clinical trial team members (i.e., sponsors, investigators, data managers, monitors, and biostatisticians) in order to adopt central statistical monitoring for assessing data quality in the actual clinical trial.

Data-driven risk identification in phase III clinical trials using central statistical monitoring

Our interest lies in quality control for clinical trials, in the context of risk-based monitoring (RBM). We specifically study the use of central statistical monitoring (CSM) to support RBM. Under an RBM paradigm, we claim that CSM has a key role to play in identifying the "risks to the most critical data elements and processes" that will drive targeted oversight. In order to support this claim, we first see how to characterize the risks that may affect clinical trials. We then discuss how CSM can be understood as a tool for providing a set of data-driven key risk indicators (KRIs), which help to organize adaptive targeted monitoring. Several case studies are provided where issues in a clinical trial have been identified thanks to targeted investigation after the identification of a risk using CSM. Using CSM to build data-driven KRIs helps to identify different kinds of issues in clinical trials. This ability is directly linked with the exhaustiveness of the CSM approach and its flexibility in the definition of the risks that are searched for when identifying the KRIs. In practice, a CSM assessment of the clinical database seems essential to ensure data quality. The atypical data patterns found in some centers and variables are seen as KRIs under a RBM approach. Targeted monitoring or data management queries can be used to confirm whether the KRIs point to an actual issue or not.

Data fraud in clinical trials

Highly publicized cases of fabrication or falsification of data in clinical trials have occurred in recent years and it is likely that there are additional undetected or unreported cases. We review the available evidence on the incidence of data fraud in clinical trials, describe several prominent cases, present information on motivation and contributing factors and discuss cost-effective ways of early detection of data fraud as part of routine central statistical monitoring of data quality. Adoption of these clinical trial monitoring procedures can identify potential data fraud not detected by conventional on-site monitoring and can improve overall data quality.