Detection of atypical data in multicenter clinical trials using unsupervised statistical monitoring

Does Central Statistical Monitoring Improve Data Quality? An Analysis of 1,111 Sites in 159 Clinical Trials

BACKGROUND

Central monitoring aims at improving the quality of clinical research by pro-actively identifying risks and remediating emerging issues in the conduct of a clinical trial that may have an adverse impact on patient safety and/or the reliability of trial results. This paper, focusing on statistical data monitoring (SDM), is the second of a series that attempts to quantify the impact of central monitoring in clinical trials.

MATERIAL AND METHODS

Quality improvement was assessed in studies using SDM from a single large central monitoring platform. The analysis focused on a total of 1111 sites that were identified as at-risk by the SDM tests and for which the study teams conducted a follow-up investigation. These sites were taken from 159 studies conducted by 23 different clinical development organizations (including both sponsor companies and contract research organizations). Two quality improvement metrics were assessed for each selected site, one based on a site data inconsistency score (DIS, overall -log10 P-value of the site compared with all other sites) and the other based on the observed metric value associated with each risk signal.

RESULTS

The SDM quality metrics showed improvement in 83% (95% CI, 80-85%) of the sites across therapeutic areas and study phases (primarily phases 2 and 3). In contrast, only 56% (95% CI, 41-70%) of sites showed improvement in 2 historical studies that did not use SDM during study conduct.

CONCLUSION

The results of this analysis provide clear quantitative evidence supporting the hypothesis that the use of SDM in central monitoring is leading to improved quality in clinical trial conduct and associated data across participating sites.

Assessing the impact of risk-based data monitoring on outcomes for a paediatric multicentre randomised controlled trial

BACKGROUND/AIMS

Regulatory guidelines recommend that sponsors develop a risk-based approach to monitoring clinical trials. However, there is a lack of evidence to guide the effective implementation of monitoring activities encompassed in this approach. The aim of this study was to assess the efficiency and impact of the risk-based monitoring approach used for a multicentre randomised controlled trial comparing treatments in paediatric patients undergoing cardiac bypass surgery.

METHODS

This is a secondary analysis of data from a randomised controlled trial that implemented targeted source data verification as part of the risk-based monitoring approach. Monitoring duration and source to database error rates were calculated across the monitored trial dataset. The monitored and unmonitored trial dataset, and simulated trial datasets with differing degrees of source data verification and cohort sizes were compared for their effect on trial outcomes.

RESULTS

In total, 106,749 critical data points across 1,282 participants were verified from source data either remotely or on-site during the trial. The total time spent monitoring was 365 hours, with a median (interquartile range) of 10 (7, 16) minutes per participant. An overall source to database error rate of 3.1% was found, and this did not differ between treatment groups. A low rate of error was found for all outcomes undergoing 100% source data verification, with the exception of two secondary outcomes with error rates >10%. Minimal variation in trial outcomes were found between the unmonitored and monitored datasets. Reduced degrees of source data verification and reduced cohort sizes assessed using simulated trial datasets had minimal impact on trial outcomes.

CONCLUSIONS

Targeted source data verification of data critical to trial outcomes, which carried with it a substantial time investment, did not have an impact on study outcomes in this trial. This evaluation of the cost-effectiveness of targeted source data verification contributes to the evidence-base regarding the context where reduced emphasis should be placed on source data verification as the foremost monitoring activity.

Application of Statistical Methods for Central Statistical Monitoring and Implementations on the German Multiple Sclerosis Registry

Monitoring of clinical trials is a fundamental process required by regulatory agencies. It assures the compliance of a center to the required regulations and the trial protocol. Traditionally, monitoring teams relied on extensive on-site visits and source data verification. However, this is costly, and the outcome is limited. Thus, central statistical monitoring (CSM) is an additional approach recently embraced by the International Council for Harmonisation (ICH) to detect problematic or erroneous data by using visualizations and statistical control measures. Existing implementations have been primarily focused on detecting inlier and outlier data. Other approaches include principal component analysis and distribution of the data. Here we focus on the utilization of comparisons of centers to the Grand mean for different model types and assumptions for common data types, such as binomial, ordinal, and continuous response variables. We implement the usage of multiple comparisons of single centers to the Grand mean of all centers. This approach is also available for various non-normal data types that are abundant in clinical trials. Further, using confidence intervals, an assessment of equivalence to the Grand mean can be applied. In a Monte Carlo simulation study, the applied statistical approaches have been investigated for their ability to control type I error and the assessment of their respective power for balanced and unbalanced designs which are common in registry data and clinical trials. Data from the German Multiple Sclerosis Registry (GMSR) including proportions of missing data, adverse events and disease severity scores were used to verify the results on Real-World-Data (RWD).

Performance of four centralized statistical monitoring methods for early detection of an atypical center in a multicenter study

Background

Ensuring the quality of data is essential for the credibility of a multicenter clinical trial. Centralized Statistical Monitoring (CSM) of data allows the detection of a center in which the distribution of a specific variable is atypical compared to other centers. The ideal CSM method should allow early detection of problem and therefore involve the fewest possible participants.

Methods

We simulated clinical trials and compared the performance of four CSM methods (Student, Hatayama, Desmet, Distance) to detect whether the distribution of a quantitative variable was atypical in one center in relation to the others, with different numbers of participants and different mean deviation amplitudes.

Results

The Student and Hatayama methods had good sensitivity but poor specificity, which disqualifies them for practical use in CSM. The Desmet and Distance methods had very high specificity for detecting all the mean deviations tested (including small values) but low sensitivity with mean deviations less than 50%.

Conclusion

Although the Student and Hatayama methods are more sensitive, their low specificity would lead to too many alerts being triggered, which would result in additional unnecessary control work to ensure data quality. The Desmet and Distance methods have low sensitivity when the deviation from the mean is low, suggesting that the CSM should be used alongside other conventional monitoring procedures rather than replacing them. However, they have excellent specificity, which suggests they can be applied routinely, since using them takes up no time at central level and does not cause any unnecessary workload in investigating centers.

Does Central Monitoring Lead to Higher Quality? An Analysis of Key Risk Indicator Outcomes

BACKGROUND

Central monitoring, which typically includes the use of key risk indicators (KRIs), aims at improving the quality of clinical research by pro-actively identifying and remediating emerging issues in the conduct of a clinical trial that may have an adverse impact on patient safety and/or the reliability of trial results. However, there has to-date been a relative lack of direct quantitative evidence published supporting the claim that central monitoring actually leads to improved quality.

MATERIAL AND METHODS

Nine commonly used KRIs were analyzed for evidence of quality improvement using data retrieved from a large central monitoring platform. A total of 212 studies comprising 1676 sites with KRI signals were used in the analysis, representing central monitoring activity from 23 different sponsor organizations. Two quality improvement metrics were assessed for each KRI, one based on a statistical score (p-value) and the other based on a KRI's observed value.

RESULTS

Both KRI quality metrics showed improvement in a vast majority of sites (82.9% for statistical score, 81.1% for observed KRI value). Additionally, the statistical score and the observed KRI values improved, respectively by 66.1% and 72.4% on average towards the study average for those sites showing improvement.

CONCLUSION

The results of this analysis provide clear quantitative evidence supporting the hypothesis that use of KRIs in central monitoring is leading to improved quality in clinical trial conduct and associated data across participating sites.

Machine learning for detecting centre-level irregularities in randomized controlled trials: A pilot study

Centralized statistical monitoring is sometimes employed as an alternative to onsite monitoring for randomized control trials. Current central monitoring methods have limitations, in that they are relatively resource intensive and do not necessarily generalize to studies where an irregularity pattern has not been observed before. Machine learning has been effective in detecting irregularities in industries such as finance and manufacturing, but to date none have been applied to clinical trials. We conducted a pilot study for the use of machine learning to identify center-level irregularities in data from multicenter clinical trials. We employed unsupervised machine learning methods, which do not rely on labelled data, and therefore allow for the automated discovery of previously unseen irregularity patterns while maintaining flexibility when applied to new data with different structures. This pilot study employs unsupervised machine learning to compute distance matrices between centres, which we used to produce centre-level continuous features. We then used a one-class support vector machine to learn the underlying distribution of each data set to identify data that was substantially different from these distributions. We evaluated our approach against current automatable centralized monitoring methods on two trials with known irregularities. While current approaches performed well on one trial (AUROC 0.752 for monitoring vs. 0.584 for machine learning), our techniques performed substantially better on the other (AUROC 0.140 for monitoring vs 0.728 for machine learning). The results of this pilot study suggest both the feasibility and the potential value of a machine learning-based approach to irregularity detection in RCTs.

Detection of Fraud in a Clinical Trial Using Unsupervised Statistical Monitoring

BACKGROUND

A central statistical assessment of the quality of data collected in clinical trials can improve the quality and efficiency of sponsor oversight of clinical investigations.

MATERIAL AND METHODS

The database of a large randomized clinical trial with known fraud was reanalyzed with a view to identifying, using only statistical monitoring techniques, the center where fraud had been confirmed. The analysis was conducted with an unsupervised statistical monitoring software using mixed-effects statistical models. The statistical analyst was unaware of the location, nature, and extent of the fraud.

RESULTS

Five centers were detected as atypical, including the center with known fraud (which was ranked 2). An incremental analysis showed that the center with known fraud could have been detected after only 25% of its data had been reported.

CONCLUSION

An unsupervised approach to central monitoring, using mixed-effects statistical models, is effective at detecting centers with fraud or other data anomalies in clinical trials.

Central statistical monitoring of investigator-led clinical trials in oncology

Investigator-led clinical trials are pragmatic trials that aim to investigate the benefits and harms of treatments in routine clinical practice. These much-needed trials represent the majority of all trials currently conducted. They are however threatened by the rising costs of clinical research, which are in part due to extensive trial monitoring processes that focus on unimportant details. Risk-based quality management focuses, instead, on “things that really matter”. We discuss the role of central statistical monitoring as part of risk-based quality management. We describe the principles of central statistical monitoring, provide examples of its use, and argue that it could help drive down the cost of randomized clinical trials, especially investigator-led trials, whilst improving their quality.

Comparing methods for clinical investigator site inspection selection: a comparison of site selection methods of investigators in clinical trials

Background During the past two decades, the number and complexity of clinical trials have risen dramatically increasing the difficulty of choosing sites for inspection. FDA's resources are limited and so sites should be chosen with care. Purpose To determine if data mining techniques and/or unsupervised statistical monitoring can assist with the process of identifying potential clinical sites for inspection. Methods Five summary-level clinical site datasets from four new drug applications (NDA) and one biologics license application (BLA), where the FDA had performed or had planned site inspections, were used. The number of sites inspected and the results of the inspections were blinded to the researchers. Five supervised learning models from the previous two years (2016-2017) of an on-going research project were used to predict site inspections results, i.e., No Action Indicated (NAI), Voluntary Action Indicated (VAI), or Official Action Indicated (OAI). Statistical Monitoring Applied to Research Trials (SMART^TM) software for unsupervised statistical monitoring software developed by CluePoints (Mont-Saint-Guibert, Belgium) was utilized to identify atypical centers (via a p-value approach) within a study.Finally, Clinical Investigator Site Selection Tool (CISST), developed by the Center for Drug Evaluation and Research (CDER), was used to calculate the total risk of each site thereby providing a framework for site selection. The agreement between the predictions of these methods was compared. The overall accuracy and sensitivity of the methods were graphically compared. Results Spearman's rank order correlation was used to examine the agreement between the SMART^TM analysis (CluePoints' software) and the CISST analysis. The average aggregated correlation between the p-values (SMART^TM) and total risk scores (CISST) for all five studies was 0.21, and range from -0.41 to 0.50. The Random Forest models for 2016 and 2017 showed the highest aggregated mean agreement (65.1%) amongst outcomes (NAI, VAI, OAI) for the three available studies. While there does not appear to be a single most accurate approach, the performance of methods under certain circumstances is discussed later in this paper. Limitations Classifier models based on data mining techniques require historical data (i.e., training data) to develop the model. There is a possibility that sites in the five-summary level datasets were included in the training datasets for the models from the previous year's research which could result in spurious confirmation of predictive ability. Additionally, the CISST was utilized in three of the five site selection processes, possibly biasing the data. Conclusion The agreement between methods was lower than expected and no single method emerged as the most accurate.