Generating evidence on a risk-based monitoring approach in the academic setting - lessons learned

Development of a risk-tailored approach and dashboard for efficient management and monitoring of investigator-initiated trials

BACKGROUND

Most randomized controlled trials (RCTs) in the academic setting have limited resources for clinical trial management and monitoring. Inefficient conduct of trials was identified as an important source of waste even in well-designed studies. Thoroughly identifying trial-specific risks to enable focussing of monitoring and management efforts on these critical areas during trial conduct may allow for the timely initiation of corrective action and to improve the efficiency of trial conduct. We developed a risk-tailored approach with an initial risk assessment of an individual trial that informs the compilation of monitoring and management procedures in a trial dashboard.

METHODS

We performed a literature review to identify risk indicators and trial monitoring approaches followed by a contextual analysis involving local, national and international stakeholders. Based on this work we developed a risk-tailored management approach with integrated monitoring for RCTs and including a visualizing trial dashboard. We piloted the approach and refined it in an iterative process based on feedback from stakeholders and performed formal user testing with investigators and staff of two clinical trials.

RESULTS

The developed risk assessment comprises four domains (patient safety and rights, overall trial management, intervention management, trial data). An accompanying manual provides rationales and detailed instructions for the risk assessment. We programmed two trial dashboards tailored to one medical and one surgical RCT to manage identified trial risks based on daily exports of accumulating trial data. We made the code for a generic dashboard available on GitHub that can be adapted to individual trials.

CONCLUSIONS

The presented trial management approach with integrated monitoring enables user-friendly, continuous checking of critical elements of trial conduct to support trial teams in the academic setting. Further work is needed in order to show effectiveness of the dashboard in terms of safe trial conduct and successful completion of clinical trials.

Monitoring and auditing protocol adherence, data integrity and ethical conduct of a randomized clinical trial: A case study

PURPOSE

To categorize, quantify and interpret findings documented in feedback letters of monitoring or auditing visits for an investigator-initiated, peer-review funded multicenter randomized trial testing probiotics for critically ill patients.

MATERIALS & METHODS

In 37 Canadian centers, monitoring and auditing visits were performed by 3 trained individuals; findings were reported in feedback letters. At trial termination, we performed duplicate content analysis on letters, categorizing observations first into unique findings, followed by 10 pre-determined trial quality management domains. We further classified each observation into a) missing operational records, b) errors in process, and potential threats to c) data integrity, d) patient privacy or e) safety.

RESULTS

Across 37 monitoring or auditing visits, 75 unique findings were categorized into 10 domains. Most frequently, observations were in domains of training documentation (180/566 [32%]) and the informed consent process (133/566 [23%]). Most observations were missing operational records (438/566 [77%]) rather than errors in process (128/566 [23%]). Of 75 findings, 13 (62/566 observations [11%]) posed a potential threat to data integrity, 1 (1/566 observation [0.18%]) to patient privacy, and 9 (49/566 observations [8.7%]) to patient safety.

CONCLUSIONS

Monitoring and auditing findings predominantly concerned missing documentation with minimal threats to data integrity, patient privacy or safety.

TRIAL REGISTRATION

PROSPECT (Probiotics: Prevention of Severe Pneumonia and Endotracheal Colonization Trial): NCT02462590.

Monitoring strategies for clinical intervention studies

BACKGROUND

Trial monitoring is an important component of good clinical practice to ensure the safety and rights of study participants, confidentiality of personal information, and quality of data. However, the effectiveness of various existing monitoring approaches is unclear. Information to guide the choice of monitoring methods in clinical intervention studies may help trialists, support units, and monitors to effectively adjust their approaches to current knowledge and evidence.

OBJECTIVES

To evaluate the advantages and disadvantages of different monitoring strategies (including risk-based strategies and others) for clinical intervention studies examined in prospective comparative studies of monitoring interventions.

SEARCH METHODS

We systematically searched CENTRAL, PubMed, and Embase via Ovid for relevant published literature up to March 2021. We searched the online 'Studies within A Trial' (SWAT) repository, grey literature, and trial registries for ongoing or unpublished studies.

SELECTION CRITERIA

We included randomized or non-randomized prospective, empirical evaluation studies of different monitoring strategies in one or more clinical intervention studies. We applied no restrictions for language or date of publication.

DATA COLLECTION AND ANALYSIS

We extracted data on the evaluated monitoring methods, countries involved, study population, study setting, randomization method, and numbers and proportions in each intervention group. Our primary outcome was critical and major monitoring findings in prospective intervention studies. Monitoring findings were classified according to different error domains (e.g. major eligibility violations) and the primary outcome measure was a composite of these domains. Secondary outcomes were individual error domains, participant recruitment and follow-up, and resource use. If we identified more than one study for a comparison and outcome definitions were similar across identified studies, we quantitatively summarized effects in a meta-analysis using a random-effects model. Otherwise, we qualitatively summarized the results of eligible studies stratified by different comparisons of monitoring strategies. We used the GRADE approach to assess the certainty of the evidence for different groups of comparisons.

MAIN RESULTS

We identified eight eligible studies, which we grouped into five comparisons. 1. Risk-based versus extensive on-site monitoring: based on two large studies, we found moderate certainty of evidence for the combined primary outcome of major or critical findings that risk-based monitoring is not inferior to extensive on-site monitoring. Although the risk ratio was close to 'no difference' (1.03 with a 95% confidence interval [CI] of 0.81 to 1.33, below 1.0 in favor of the risk-based strategy), the high imprecision in one study and the small number of eligible studies resulted in a wide CI of the summary estimate. Low certainty of evidence suggested that monitoring strategies with extensive on-site monitoring were associated with considerably higher resource use and costs (up to a factor of 3.4). Data on recruitment or retention of trial participants were not available. 2. Central monitoring with triggered on-site visits versus regular on-site visits: combining the results of two eligible studies yielded low certainty of evidence with a risk ratio of 1.83 (95% CI 0.51 to 6.55) in favor of triggered monitoring intervention. Data on recruitment, retention, and resource use were not available. 3. Central statistical monitoring and local monitoring performed by site staff with annual on-site visits versus central statistical monitoring and local monitoring only: based on one study, there was moderate certainty of evidence that a small number of major and critical findings were missed with the central monitoring approach without on-site visits: 3.8% of participants in the group without on-site visits and 6.4% in the group with on-site visits had a major or critical monitoring finding (odds ratio 1.7, 95% CI 1.1 to 2.7; P = 0.03). The absolute number of monitoring findings was very low, probably because defined major and critical findings were very study specific and central monitoring was present in both intervention groups. Very low certainty of evidence did not suggest a relevant effect on participant retention, and very low certainty evidence indicated an extra cost for on-site visits of USD 2,035,392. There were no data on recruitment. 4. Traditional 100% source data verification (SDV) versus targeted or remote SDV: the two studies assessing targeted and remote SDV reported findings only related to source documents. Compared to the final database obtained using the full SDV monitoring process, only a small proportion of remaining errors on overall data were identified using the targeted SDV process in the MONITORING study (absolute difference 1.47%, 95% CI 1.41% to 1.53%). Targeted SDV was effective in the verification of source documents, but increased the workload on data management. The other included study was a pilot study, which compared traditional on-site SDV versus remote SDV and found little difference in monitoring findings and the ability to locate data values despite marked differences in remote access in two clinical trial networks. There were no data on recruitment or retention. 5. Systematic on-site initiation visit versus on-site initiation visit upon request: very low certainty of evidence suggested no difference in retention and recruitment between the two approaches. There were no data on critical and major findings or on resource use.

AUTHORS' CONCLUSIONS

The evidence base is limited in terms of quantity and quality. Ideally, for each of the five identified comparisons, more prospective, comparative monitoring studies nested in clinical trials and measuring effects on all outcomes specified in this review are necessary to draw more reliable conclusions. However, the results suggesting risk-based, targeted, and mainly central monitoring as an efficient strategy are promising. The development of reliable triggers for on-site visits is ongoing; different triggers might be used in different settings. More evidence on risk indicators that identify sites with problems or the prognostic value of triggers is needed to further optimize central monitoring strategies. In particular, approaches with an initial assessment of trial-specific risks that need to be closely monitored centrally during trial conduct with triggered on-site visits should be evaluated in future research.

Clinical researchers' lived experiences with data quality monitoring in clinical trials: a qualitative study

BACKGROUND

Fundamental to the success of clinical research that involves human participants is the quality of the data that is generated. To ensure data quality, clinical trials must comply with the Good Clinical Practice guideline which recommends data monitoring. To date, the guideline is broad, requires technology for enforcement, follows strict industry standards, mostly designed for drug-registration trials and based on informal consensus. It is also unknown what challenges clinical trials and researchers face in implementing data monitoring procedures. Thus, this study aimed to describe researcher experiences with data quality monitoring in clinical trials.

METHODS

We conducted semi-structured telephone interviews following a guided-phenomenological approach. Participants were recruited from the Australian and New Zealand Clinical Trials Registry and were researchers affiliated with a listed clinical study. Each transcript was analysed with inductive thematic analysis before thematic categorisation of themes from all transcripts. Primary, secondary and subthemes were categorised according to the emerging relationships.

RESULTS

Data saturation were reached after interviewing seven participants. Five primary themes, two secondary themes and 21 subthemes in relation to data quality monitoring emerged from the data. The five primary themes included: education and training, ways of working, working with technology, working with data, and working within regulatory requirements. The primary theme 'education and training' influenced the other four primary themes. While 'working with technology' influenced the 'way of working'. All other themes had reciprocal relationships. There was no relationship reported between 'working within regulatory requirements' and 'working with technology'. The researchers experienced challenges in meeting regulatory requirements, using technology and fostering working relationships for data quality monitoring.

CONCLUSION

Clinical trials implemented a variety of data quality monitoring procedures tailored to their situation and study context. Standardised frameworks that are accessible to all types of clinical trials are needed with an emphasis on education and training.

Central monitoring in a randomized, open-label, controlled phase 3 clinical trial for a treatment-shortening regimen for pulmonary tuberculosis

INTRODUCTION

With the growing use of online study management systems and rapid availability of data, timely data review and quality assessments are necessary to ensure proper clinical trial implementation. In this report we describe central monitoring used to ensure protocol compliance and accurate data reporting, implemented during a large phase 3 clinical trial.

MATERIAL AND METHODS

The Tuberculosis Trials Consortium (TBTC) Study 31/AIDS Clinical Trials Group (ACTG) study A5349 (S31) is an international, multi-site, randomized, open-label, controlled, non-inferiority phase 3 clinical trial comparing two 4-month regimens to a standard 6 month regimen for treatment of drug-susceptible tuberculosis (TB) among adolescents and adults with a sample size of 2500 participants.

RESULTS

Central monitoring utilized primary study data in a five-tiered approach, including (1) real-time data checks & topic-specific intervention reports, (2) missing forms reports, (3) quality assurance metrics, (4) critical data reports and (5) protocol deviation identification, aimed to detect and resolve quality challenges. Over the course of the study, 240 data checks and reports were programed across the five tiers used.

DISCUSSION

This use of primary study data to identify issues rapidly allowed the study sponsor to focus quality assurance and data cleaning activities on prioritized data, related to protocol compliance and accurate reporting of study results. Our approach enabled us to become more efficient and effective as we informed sites about deviations, resolved missing or inconsistent data, provided targeted guidance, and gained a deeper understanding of challenges experienced at clinical trial sites.

TRIAL REGISTRATION

This trial was registered with ClinicalTrials.gov (Identifier: NCT02410772) on April 8, 2015.

Time-consuming and expensive data quality monitoring procedures persist in clinical trials: A national survey

INTRODUCTION

The Good Clinical Practice guideline identifies that data monitoring is an essential research activity. However, limited evidence exists on how to perform monitoring including the amount or frequency that is needed to ensure data quality. This study aims to explore the monitoring procedures that are implemented to ensure data quality in Australian clinical research studies.

MATERIAL AND METHODS

Clinical studies listed on the Australian and New Zealand Clinical Trials Registry were invited to participate in a national cross-sectional, mixed-mode, multi-contact (postal letter and e-mail) web-based survey. Information was gathered about the types of data quality monitoring procedures being implemented.

RESULTS

Of the 3689 clinical studies contacted, 589 (16.0%) responded, of which 441 (77.4%) completed the survey. Over half (55%) of the studies applied source data verification (SDV) compared to risk-based targeted and triggered monitoring (10-11%). Conducting 100% on-site monitoring was most common for those who implemented the traditional approach. Respondents who did not conduct 100% monitoring, included 1-25% of data points for SDV, centralized or on-site monitoring. The incidence of adverse events and protocol deviations were the most likely factors to trigger a site visit for risk-based triggered (63% and 44%) and centralized monitoring (48% and 44%), respectively.

CONCLUSION

Instead of using more optimal risk-based approaches, small single-site clinical studies are conducting traditional monitoring procedures which are time consuming and expensive. Formal guidelines need to be improved and provided to all researchers for 'new' risk-based monitoring approaches.

Feasibility of a Hybrid Risk-Adapted Monitoring System in Investigator-Sponsored Trials in Cancer

BACKGROUND

We assessed the feasibility of a hybrid monitoring system (minimal on-site monitoring + strategic central monitoring) used at the academic research office at Asan Medical Center (Seoul, Korea) in monitoring investigator-sponsored oncology trials.

METHODS

Monitoring findings in three oncology trials conducted between 2014 and 2017 were compared. A confirmatory source data verification (SDV) was carried out in the low-risk trial and compared with the central monitoring findings. The economic advantages of central monitoring were tested by calculating the monitoring hours per patient.

RESULTS

A total of 50, 118, 228 patients were enrolled in the high-, intermediate-, and low-risk trials, respectively. The high-risk trial was monitored through 42 on-site visits (1299 findings); the intermediate-risk trial had 79 monitorings (on-site, 24%; central, 76%; 1464 findings); the low-risk trial had 197 monitorings (on-site, 4%; central, 96%; 3364 findings). Central monitoring was more effective than on-site monitoring in revealing minor errors such as "missing case report forms" and "data outliers" (both P < 0.0001), and showed comparable results in revealing major issues such as investigational product compliance and delayed reporting of serious adverse events (both P > 0.05). Confirmatory SDV in the low-risk trial revealed more findings than central monitoring in the "inconsistent data" and "inappropriate adverse event" categories. The total monitoring hours per patient were lower in the intermediate- and low-risk trials than in the high-risk trial (8.1 and 7.3 vs. 14.3 h, respectively).

CONCLUSION

Our hybrid monitoring system showed acceptable feasibility in revealing both major and minor issues in multi-center oncology investigator-sponsored trials.

A randomized evaluation of on-site monitoring nested in a multinational randomized trial

BACKGROUND

Evidence from prospectively designed studies to guide on-site monitoring practices for randomized trials is limited. A cluster randomized study, nested within the Strategic Timing of AntiRetroviral Treatment (START) trial, was conducted to evaluate on-site monitoring.

METHODS

Sites were randomized to either annual on-site monitoring or no on-site monitoring. All sites were centrally monitored, and local monitoring was carried out twice each year. Randomization was stratified by country and projected enrollment in START. The primary outcome was a participant-level composite outcome including components for eligibility errors, consent violations, use of antiretroviral treatment not recommended by protocol, late reporting of START primary and secondary clinical endpoints (defined as the event being reported more than 6 months from occurrence), and data alteration and fraud. Logistic regression fixed effect hierarchical models were used to compare on-site versus no on-site monitoring for the primary composite outcome and its components. Odds ratios and 95% confidence intervals comparing on-site monitoring versus no on-site monitoring are cited.

RESULTS

In total, 99 sites (2107 participants) were randomized to receive annual on-site monitoring and 97 sites (2264 participants) were randomized to be monitored only centrally and locally. The two monitoring groups were well balanced at entry. In the on-site monitoring group, 469 annual on-site monitoring visits were conducted, and 134 participants (6.4%) in 56 of 99 sites (57%) had a primary monitoring outcome. In the no on-site monitoring group, 85 participants (3.8%) in 34 of 97 sites (35%) had a primary monitoring outcome (odds ratio = 1.7; 95% confidence interval: 1.1-2.7; p = 0.03). Informed consent violations accounted for most outcomes in each group (56 vs 41 participants). The largest odds ratio was for eligibility violations (odds ratio = 12.2; 95% confidence interval: 1.8-85.2; p = 0.01). The number of participants with a late START primary endpoint was similar for each monitoring group (23 vs 16 participants). Late START grade 4 and unscheduled hospitalization events were found for 34 participants in the on-site monitoring group and 19 participants in the no on-site monitoring group (odds ratio = 2.0; 95% confidence interval: 1.1-3.7; p = 0.02). There were no cases of data alteration or fraud. Based on the travel budget for on-site monitoring and the hours spent conducting on-site monitoring, the estimated cost of on-site monitoring was over US$2 million.

CONCLUSION

On-site monitoring led to the identification of more eligibility and consent violations and START clinical events being reported more than 6 months from occurrence as compared to no on-site monitoring. Considering the nature of the excess monitoring outcomes identified at sites receiving on-site monitoring, as well as the cost of on-site monitoring, the value to the START study was limited.

Triggered or routine site monitoring visits for randomised controlled trials: results of TEMPER, a prospective, matched-pair study

BACKGROUND/AIMS

In multi-site clinical trials, where trial data and conduct are scrutinised centrally with pre-specified triggers for visits to sites, targeted monitoring may be an efficient way to prioritise on-site monitoring. This approach is widely used in academic trials, but has never been formally evaluated.

METHODS

TEMPER assessed the ability of targeted monitoring, as used in three ongoing phase III randomised multi-site oncology trials, to distinguish sites at which higher and lower rates of protocol and/or Good Clinical Practice violations would be found during site visits. Using a prospective, matched-pair design, sites that had been prioritised for visits after having activated 'triggers' were matched with a control ('untriggered') site, which would not usually have been visited at that time. The paired sites were visited within 4 weeks of each other, and visit findings are recorded and categorised according to the seriousness of the deviation. The primary outcome measure was the proportion of sites with ≥1 'Major' or 'Critical' finding not previously identified centrally. The study was powered to detect an absolute difference of ≥30% between triggered and untriggered visits. A sensitivity analysis, recommended by the study's blinded endpoint review committee, excluded findings related to re-consent. Additional analyses assessed the prognostic value of individual triggers and data from pre-visit questionnaires completed by site and trials unit staff.

RESULTS

In total, 42 matched pairs of visits took place between 2013 and 2016. In the primary analysis, 88.1% of triggered visits had ≥1 new Major/Critical finding, compared to 81.0% of untriggered visits, an absolute difference of 7.1% (95% confidence interval -8.3%, +22.5%; p = 0.365). When re-consent findings were excluded, these figures reduced to 85.7% versus 59.5%, (difference = 26.2%, 95% confidence interval 8.0%, 44.4%; p = 0.007). Individual triggers had modest prognostic value but knowledge of the trial-related activities carried out by site staff may be useful.

CONCLUSION

Triggered monitoring approaches, as used in these trials, were not sufficiently discriminatory. The rate of Major and Critical findings was higher than anticipated, but the majority related to consent and re-consent with no indication of systemic problems that would impact trial-wide safety issues or integrity of the results in any of the three trials. Sensitivity analyses suggest triggered monitoring may be of potential use, but needs improvement and investigation of further central monitoring triggers is warranted. TEMPER highlights the need to question and evaluate methods in trial conduct, and should inform further developments in this area.

Perceived barriers and facilitators to Risk Based Monitoring in academic-led clinical trials: a mixed methods study

Background

In November 2016, the ICH published a requirement for sponsors to develop a systematic, prioritised, risk-based approach to monitoring clinical trials. This approach is more commonly known as risk-based monitoring (RBM). However, recent evidence suggests that a ‘gold standard’, validated approach to RBM does not exist and it is unclear how sponsors will introduce RBM into their organisations. A first step needed to inform the implementation of RBM is to explore academic trialists’ readiness and ability to perform RBM. The aim of this paper is to identify the attitudes and perceived barriers and facilitators to the implementation of RBM in academic-led clinical trials in Ireland.

Methods

This is a mixed-methods, explanatory sequential design, with quantitative survey followed by semistructured interviews. Academic clinical researchers (N = 132) working in Ireland were surveyed to examine their use and perceptions of RBM. A purposive sample of survey participants (n = 22) were then interviewed to gain greater insight into the quantitative findings. The survey and interview data were merged to generate a list of perceived barriers and facilitators to RBM implementation, with suggestions for, and solutions to, these issues.

Results

Survey response rate was 49% (132/273). Thirteen percent (n = 18) of responders were not familiar with the term risk-based monitoring and less than a quarter of respondents (21%, n = 28) had performed RBM in a clinical trial. Barriers to RBM implementation included lack of RBM knowledge/training, increased costs caused by greater IT demands, increased workload for trial staff and lack of evidence to support RBM as an effective monitoring approach. Facilitators included participants’ legal obligation to perform RBM under the new ICH-GCP guidelines, availability of RBM guidance and perception of cost savings by performing RBM in future trials.

Conclusion

The results of this study demonstrate a need for training and regulatory-endorsed guidelines to support the implementation of RBM in academic-led clinical trials. The study provides valuable insights to inform interventions and strategies by policy-makers and clinical trial regulators to improve RBM uptake.

Electronic supplementary material

The online version of this article (doi:10.1186/s13063-017-2148-4) contains supplementary material, which is available to authorized users.