Dynamic methods for ongoing assessment of site-level risk in risk-based monitoring of clinical trials: A scoping review

Assessing the performance of methods for central statistical monitoring of a binary or continuous outcome in multi-center trials: A simulation study

BACKGROUND

Quality study monitoring is fundamental to patient safety and data integrity. Regulators and industry consortia have increasingly advocated for risk-based monitoring (RBM) and central statistical monitoring (CSM) for more effective and efficient monitoring. Assessing which statistical methods underpin these approaches can best identify unusual data patterns in multi-center clinical trials that may be driven by potential systematic errors is important.

METHODS

We assessed various CSM techniques, including cross-tests, fixed-effects, mixed-effects, and finite mixture models, across scenarios with different sample sizes, contamination rates, and overdispersion via simulation. Our evaluation utilized threshold-independent metrics such as the area under the curve (AUC) and average precision (AP), offering a fuller picture of CSM performance.

RESULTS

All CSM methods showed consistent characteristics across center sizes or overdispersion. The adaptive finite mixture model outperformed others in AUC and AP, especially at 30% contamination, upholding high specificity unless converging to a single-component model due to low contamination or deviation. The mixed-effects model performed well at lower contamination rates. However, it became conservative in specificity and exhibited declined performance for binary outcomes under high deviation. Cross-tests and fixed-effects methods underperformed, especially when deviation increased.

CONCLUSION

Our evaluation explored the merits and drawbacks of multiple CSM methods, and found that relying on sensitivity and specificity alone is likely insufficient to fully measure predictive performance. The finite mixture method demonstrated more consistent performance across scenarios by mitigating the influence of outliers. In practice, considering the study-specific costs of false positives/negatives with available resources for monitoring is important.

A risk-based monitoring approach to source data monitoring and documenting monitoring findings

BACKGROUND

Clinical trial monitoring is evolving from labor-intensive to targeted approaches. The traditional 100% Source Data Monitoring (SDM) approach fails to prioritize data by significance, diverting attention from critical elements. Despite regulatory guidance on Risk-Based Monitoring (RBM), its widespread implementation has been slow.

METHODS

Our study teams assess the study's overall risk, document heightened and critical risks, and create a study-specific risk-based monitoring plan, integrating SDM and Central Data Monitoring (CDM). SDM combines a fixed list of pre-identified variables and a list of randomly identified variables to monitor. Identifying variables follows a two-step approach: first, a random sample of participants is selected, second, a random set of variables for each participant selected is identified. Sampling weights prioritize critical variables. Regular team meetings are held to discuss and compile significant findings into a Study Monitoring Report.

RESULTS

We present a random SDM sample and a Study Monitoring Report. The random SDM output includes a look-up table for selected database elements. The report provides a holistic view of the study issues and overall health.

CONCLUSIONS

The proposed random sampling method is used to monitor a representative set of critical variables, while the Study Monitoring Report is written to summarize significant monitoring findings and data trends. The report allows the sponsor to assess the current status of the study and data effectively. Communicating and sharing emerging insights facilitates timely adjustments of future monitoring activities, optimizing efficiencies, and study outcomes.

Assessing the impact of risk-based data monitoring on outcomes for a paediatric multicentre randomised controlled trial

BACKGROUND/AIMS

Regulatory guidelines recommend that sponsors develop a risk-based approach to monitoring clinical trials. However, there is a lack of evidence to guide the effective implementation of monitoring activities encompassed in this approach. The aim of this study was to assess the efficiency and impact of the risk-based monitoring approach used for a multicentre randomised controlled trial comparing treatments in paediatric patients undergoing cardiac bypass surgery.

METHODS

This is a secondary analysis of data from a randomised controlled trial that implemented targeted source data verification as part of the risk-based monitoring approach. Monitoring duration and source to database error rates were calculated across the monitored trial dataset. The monitored and unmonitored trial dataset, and simulated trial datasets with differing degrees of source data verification and cohort sizes were compared for their effect on trial outcomes.

RESULTS

In total, 106,749 critical data points across 1,282 participants were verified from source data either remotely or on-site during the trial. The total time spent monitoring was 365 hours, with a median (interquartile range) of 10 (7, 16) minutes per participant. An overall source to database error rate of 3.1% was found, and this did not differ between treatment groups. A low rate of error was found for all outcomes undergoing 100% source data verification, with the exception of two secondary outcomes with error rates >10%. Minimal variation in trial outcomes were found between the unmonitored and monitored datasets. Reduced degrees of source data verification and reduced cohort sizes assessed using simulated trial datasets had minimal impact on trial outcomes.

CONCLUSIONS

Targeted source data verification of data critical to trial outcomes, which carried with it a substantial time investment, did not have an impact on study outcomes in this trial. This evaluation of the cost-effectiveness of targeted source data verification contributes to the evidence-base regarding the context where reduced emphasis should be placed on source data verification as the foremost monitoring activity.

Critical path activities in clinical trial setup and conduct: How to avoid bottlenecks and accelerate clinical trials

Most clinical trials are delayed due to scientific and/or operational challenges. Any effort to minimize delays can generate value for patients and sponsors. This article reviews critical path process steps commonly identified by practitioners, such as during protocol development, site contracting, or patient recruitment. Commonly considered measures, such as adding more trial sites or countries, were contrasted with less frequented measures, such as evidence-based feasibility or real-world evidence analysis, to help validate assumptions before clinical trial initiation. In a broad analysis, we integrated a literature review with a practitioner survey into a framework to help decision makers on the most critical process steps when setting up or conducting clinical trials in order to bring critical treatments to patients faster.

Development of a risk-tailored approach and dashboard for efficient management and monitoring of investigator-initiated trials

BACKGROUND

Most randomized controlled trials (RCTs) in the academic setting have limited resources for clinical trial management and monitoring. Inefficient conduct of trials was identified as an important source of waste even in well-designed studies. Thoroughly identifying trial-specific risks to enable focussing of monitoring and management efforts on these critical areas during trial conduct may allow for the timely initiation of corrective action and to improve the efficiency of trial conduct. We developed a risk-tailored approach with an initial risk assessment of an individual trial that informs the compilation of monitoring and management procedures in a trial dashboard.

METHODS

We performed a literature review to identify risk indicators and trial monitoring approaches followed by a contextual analysis involving local, national and international stakeholders. Based on this work we developed a risk-tailored management approach with integrated monitoring for RCTs and including a visualizing trial dashboard. We piloted the approach and refined it in an iterative process based on feedback from stakeholders and performed formal user testing with investigators and staff of two clinical trials.

RESULTS

The developed risk assessment comprises four domains (patient safety and rights, overall trial management, intervention management, trial data). An accompanying manual provides rationales and detailed instructions for the risk assessment. We programmed two trial dashboards tailored to one medical and one surgical RCT to manage identified trial risks based on daily exports of accumulating trial data. We made the code for a generic dashboard available on GitHub that can be adapted to individual trials.

CONCLUSIONS

The presented trial management approach with integrated monitoring enables user-friendly, continuous checking of critical elements of trial conduct to support trial teams in the academic setting. Further work is needed in order to show effectiveness of the dashboard in terms of safe trial conduct and successful completion of clinical trials.

Monitoring strategies for clinical intervention studies

BACKGROUND

Trial monitoring is an important component of good clinical practice to ensure the safety and rights of study participants, confidentiality of personal information, and quality of data. However, the effectiveness of various existing monitoring approaches is unclear. Information to guide the choice of monitoring methods in clinical intervention studies may help trialists, support units, and monitors to effectively adjust their approaches to current knowledge and evidence.

OBJECTIVES

To evaluate the advantages and disadvantages of different monitoring strategies (including risk-based strategies and others) for clinical intervention studies examined in prospective comparative studies of monitoring interventions.

SEARCH METHODS

We systematically searched CENTRAL, PubMed, and Embase via Ovid for relevant published literature up to March 2021. We searched the online 'Studies within A Trial' (SWAT) repository, grey literature, and trial registries for ongoing or unpublished studies.

SELECTION CRITERIA

We included randomized or non-randomized prospective, empirical evaluation studies of different monitoring strategies in one or more clinical intervention studies. We applied no restrictions for language or date of publication.

DATA COLLECTION AND ANALYSIS

We extracted data on the evaluated monitoring methods, countries involved, study population, study setting, randomization method, and numbers and proportions in each intervention group. Our primary outcome was critical and major monitoring findings in prospective intervention studies. Monitoring findings were classified according to different error domains (e.g. major eligibility violations) and the primary outcome measure was a composite of these domains. Secondary outcomes were individual error domains, participant recruitment and follow-up, and resource use. If we identified more than one study for a comparison and outcome definitions were similar across identified studies, we quantitatively summarized effects in a meta-analysis using a random-effects model. Otherwise, we qualitatively summarized the results of eligible studies stratified by different comparisons of monitoring strategies. We used the GRADE approach to assess the certainty of the evidence for different groups of comparisons.

MAIN RESULTS

We identified eight eligible studies, which we grouped into five comparisons. 1. Risk-based versus extensive on-site monitoring: based on two large studies, we found moderate certainty of evidence for the combined primary outcome of major or critical findings that risk-based monitoring is not inferior to extensive on-site monitoring. Although the risk ratio was close to 'no difference' (1.03 with a 95% confidence interval [CI] of 0.81 to 1.33, below 1.0 in favor of the risk-based strategy), the high imprecision in one study and the small number of eligible studies resulted in a wide CI of the summary estimate. Low certainty of evidence suggested that monitoring strategies with extensive on-site monitoring were associated with considerably higher resource use and costs (up to a factor of 3.4). Data on recruitment or retention of trial participants were not available. 2. Central monitoring with triggered on-site visits versus regular on-site visits: combining the results of two eligible studies yielded low certainty of evidence with a risk ratio of 1.83 (95% CI 0.51 to 6.55) in favor of triggered monitoring intervention. Data on recruitment, retention, and resource use were not available. 3. Central statistical monitoring and local monitoring performed by site staff with annual on-site visits versus central statistical monitoring and local monitoring only: based on one study, there was moderate certainty of evidence that a small number of major and critical findings were missed with the central monitoring approach without on-site visits: 3.8% of participants in the group without on-site visits and 6.4% in the group with on-site visits had a major or critical monitoring finding (odds ratio 1.7, 95% CI 1.1 to 2.7; P = 0.03). The absolute number of monitoring findings was very low, probably because defined major and critical findings were very study specific and central monitoring was present in both intervention groups. Very low certainty of evidence did not suggest a relevant effect on participant retention, and very low certainty evidence indicated an extra cost for on-site visits of USD 2,035,392. There were no data on recruitment. 4. Traditional 100% source data verification (SDV) versus targeted or remote SDV: the two studies assessing targeted and remote SDV reported findings only related to source documents. Compared to the final database obtained using the full SDV monitoring process, only a small proportion of remaining errors on overall data were identified using the targeted SDV process in the MONITORING study (absolute difference 1.47%, 95% CI 1.41% to 1.53%). Targeted SDV was effective in the verification of source documents, but increased the workload on data management. The other included study was a pilot study, which compared traditional on-site SDV versus remote SDV and found little difference in monitoring findings and the ability to locate data values despite marked differences in remote access in two clinical trial networks. There were no data on recruitment or retention. 5. Systematic on-site initiation visit versus on-site initiation visit upon request: very low certainty of evidence suggested no difference in retention and recruitment between the two approaches. There were no data on critical and major findings or on resource use.

AUTHORS' CONCLUSIONS

The evidence base is limited in terms of quantity and quality. Ideally, for each of the five identified comparisons, more prospective, comparative monitoring studies nested in clinical trials and measuring effects on all outcomes specified in this review are necessary to draw more reliable conclusions. However, the results suggesting risk-based, targeted, and mainly central monitoring as an efficient strategy are promising. The development of reliable triggers for on-site visits is ongoing; different triggers might be used in different settings. More evidence on risk indicators that identify sites with problems or the prognostic value of triggers is needed to further optimize central monitoring strategies. In particular, approaches with an initial assessment of trial-specific risks that need to be closely monitored centrally during trial conduct with triggered on-site visits should be evaluated in future research.

Evidence and risk indicators of non-random sampling in clinical trials in implant dentistry: A systematic appraisal

Aim

Analysis of distribution of p‐values of continuous differences between test and controls after randomization provides evidence of unintentional error, non‐random sampling, or data fabrication in randomized controlled trials (RCTs). We assessed evidence of highly unusual distributions of baseline characteristics of subjects enrolled in clinical trials in implant dentistry.

Materials and methods

RCTs published between 2005 and 2020 were systematically searched in Pubmed, Embase, and Cochrane databases. Baseline patient data were extracted from full text articles by two independent assessors. The hypothesis of non‐random sampling was tested by comparing the expected and the observed distribution of the p‐values of differences between test and controls after randomization.

Results

One‐thousand five‐hundred and thirty‐eight unique RCTs were identified, of which 409 (26.6%) did not report baseline characteristics of the population, and 671 (43.6%) reported data in forms other than mean and standard deviation and could not be used to assess their random sampling. Four‐hundred and fifty‐eight trials with 1449 baseline variables in the form of mean and standard deviation were assessed. The study observed an over‐representation of very small p‐values [<.001, 1.38%, 95% confidence interval (CI) 0.85–2.12 compared to the expected 0.10%, 95% CI 0.00–0.26]. No evidence of over‐representation of larger p‐values was observed. Unusual distributions were present in 2.38% of RCTs and more frequent in non‐registered trials, in studies supported by non‐industry funding, and in multi‐centre RCTs.

Conclusions

The inability to assess random sampling due to insufficient reporting in 26.6% of trials requires attention. In trials reporting suitable baseline data, unusual distributions were uncommon, and no evidence of data fabrication was detected, but there was evidence of non‐random sampling. Continued efforts are necessary to ensure high integrity and trust in the evidence base of the field.