1
|
Ge L, Wang Z, Liu CC, Childress S, Wildfire J, Wu G. Assessing the performance of methods for central statistical monitoring of a binary or continuous outcome in multi-center trials: A simulation study. Contemp Clin Trials 2024; 143:107580. [PMID: 38796099 DOI: 10.1016/j.cct.2024.107580] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/03/2024] [Revised: 04/29/2024] [Accepted: 05/21/2024] [Indexed: 05/28/2024]
Abstract
BACKGROUND Quality study monitoring is fundamental to patient safety and data integrity. Regulators and industry consortia have increasingly advocated for risk-based monitoring (RBM) and central statistical monitoring (CSM) for more effective and efficient monitoring. Assessing which statistical methods underpin these approaches can best identify unusual data patterns in multi-center clinical trials that may be driven by potential systematic errors is important. METHODS We assessed various CSM techniques, including cross-tests, fixed-effects, mixed-effects, and finite mixture models, across scenarios with different sample sizes, contamination rates, and overdispersion via simulation. Our evaluation utilized threshold-independent metrics such as the area under the curve (AUC) and average precision (AP), offering a fuller picture of CSM performance. RESULTS All CSM methods showed consistent characteristics across center sizes or overdispersion. The adaptive finite mixture model outperformed others in AUC and AP, especially at 30% contamination, upholding high specificity unless converging to a single-component model due to low contamination or deviation. The mixed-effects model performed well at lower contamination rates. However, it became conservative in specificity and exhibited declined performance for binary outcomes under high deviation. Cross-tests and fixed-effects methods underperformed, especially when deviation increased. CONCLUSION Our evaluation explored the merits and drawbacks of multiple CSM methods, and found that relying on sensitivity and specificity alone is likely insufficient to fully measure predictive performance. The finite mixture method demonstrated more consistent performance across scenarios by mitigating the influence of outliers. In practice, considering the study-specific costs of false positives/negatives with available resources for monitoring is important.
Collapse
Affiliation(s)
- Li Ge
- Gilead Sciences, Foster City 94404, CA, USA; Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison, Madison 53703, WI, USA
| | | | | | | | | | - George Wu
- Gilead Sciences, Foster City 94404, CA, USA.
| |
Collapse
|
2
|
de Viron S, Trotta L, Steijn W, Young S, Buyse M. Does Central Statistical Monitoring Improve Data Quality? An Analysis of 1,111 Sites in 159 Clinical Trials. Ther Innov Regul Sci 2024; 58:483-494. [PMID: 38334868 PMCID: PMC11043176 DOI: 10.1007/s43441-024-00613-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2023] [Accepted: 01/08/2024] [Indexed: 02/10/2024]
Abstract
BACKGROUND Central monitoring aims at improving the quality of clinical research by pro-actively identifying risks and remediating emerging issues in the conduct of a clinical trial that may have an adverse impact on patient safety and/or the reliability of trial results. This paper, focusing on statistical data monitoring (SDM), is the second of a series that attempts to quantify the impact of central monitoring in clinical trials. MATERIAL AND METHODS Quality improvement was assessed in studies using SDM from a single large central monitoring platform. The analysis focused on a total of 1111 sites that were identified as at-risk by the SDM tests and for which the study teams conducted a follow-up investigation. These sites were taken from 159 studies conducted by 23 different clinical development organizations (including both sponsor companies and contract research organizations). Two quality improvement metrics were assessed for each selected site, one based on a site data inconsistency score (DIS, overall -log10 P-value of the site compared with all other sites) and the other based on the observed metric value associated with each risk signal. RESULTS The SDM quality metrics showed improvement in 83% (95% CI, 80-85%) of the sites across therapeutic areas and study phases (primarily phases 2 and 3). In contrast, only 56% (95% CI, 41-70%) of sites showed improvement in 2 historical studies that did not use SDM during study conduct. CONCLUSION The results of this analysis provide clear quantitative evidence supporting the hypothesis that the use of SDM in central monitoring is leading to improved quality in clinical trial conduct and associated data across participating sites.
Collapse
Affiliation(s)
- Sylviane de Viron
- CluePoints S.A, Avenue Albert Einstein, 2a 1348, Louvain-la-Neuve, Belgium.
| | - Laura Trotta
- CluePoints S.A, Avenue Albert Einstein, 2a 1348, Louvain-la-Neuve, Belgium
| | - William Steijn
- CluePoints S.A, Avenue Albert Einstein, 2a 1348, Louvain-la-Neuve, Belgium
| | | | - Marc Buyse
- CluePoints S.A, Avenue Albert Einstein, 2a 1348, Louvain-la-Neuve, Belgium
- International Drug Development Institute (IDDI), Louvain-la-Neuve, Belgium
- Interuniversity Institute for Biostatistics and Statistical Bioinformatics (I-BioStat), Hasselt University, Hasselt, Belgium
| |
Collapse
|
3
|
Klatte K, Subramaniam S, Benkert P, Schulz A, Ehrlich K, Rösler A, Deschodt M, Fabbro T, Pauli-Magnus C, Briel M. Development of a risk-tailored approach and dashboard for efficient management and monitoring of investigator-initiated trials. BMC Med Res Methodol 2023; 23:84. [PMID: 37020207 PMCID: PMC10074803 DOI: 10.1186/s12874-023-01902-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/10/2022] [Accepted: 03/23/2023] [Indexed: 04/07/2023] Open
Abstract
BACKGROUND Most randomized controlled trials (RCTs) in the academic setting have limited resources for clinical trial management and monitoring. Inefficient conduct of trials was identified as an important source of waste even in well-designed studies. Thoroughly identifying trial-specific risks to enable focussing of monitoring and management efforts on these critical areas during trial conduct may allow for the timely initiation of corrective action and to improve the efficiency of trial conduct. We developed a risk-tailored approach with an initial risk assessment of an individual trial that informs the compilation of monitoring and management procedures in a trial dashboard. METHODS We performed a literature review to identify risk indicators and trial monitoring approaches followed by a contextual analysis involving local, national and international stakeholders. Based on this work we developed a risk-tailored management approach with integrated monitoring for RCTs and including a visualizing trial dashboard. We piloted the approach and refined it in an iterative process based on feedback from stakeholders and performed formal user testing with investigators and staff of two clinical trials. RESULTS The developed risk assessment comprises four domains (patient safety and rights, overall trial management, intervention management, trial data). An accompanying manual provides rationales and detailed instructions for the risk assessment. We programmed two trial dashboards tailored to one medical and one surgical RCT to manage identified trial risks based on daily exports of accumulating trial data. We made the code for a generic dashboard available on GitHub that can be adapted to individual trials. CONCLUSIONS The presented trial management approach with integrated monitoring enables user-friendly, continuous checking of critical elements of trial conduct to support trial teams in the academic setting. Further work is needed in order to show effectiveness of the dashboard in terms of safe trial conduct and successful completion of clinical trials.
Collapse
Affiliation(s)
- Katharina Klatte
- Department of Clinical Research, University Hospital Basel and University of Basel, Spitalstrasse 12, Basel, CH- 4031, Switzerland.
| | - Suvitha Subramaniam
- Department of Clinical Research, University Hospital Basel and University of Basel, Spitalstrasse 12, Basel, CH- 4031, Switzerland
| | - Pascal Benkert
- Department of Clinical Research, University Hospital Basel and University of Basel, Spitalstrasse 12, Basel, CH- 4031, Switzerland
| | - Alexandra Schulz
- Department of Clinical Research, University Hospital Basel and University of Basel, Spitalstrasse 12, Basel, CH- 4031, Switzerland
| | - Klaus Ehrlich
- Department of Clinical Research, University Hospital Basel and University of Basel, Spitalstrasse 12, Basel, CH- 4031, Switzerland
| | - Astrid Rösler
- Department of Clinical Research, University Hospital Basel and University of Basel, Spitalstrasse 12, Basel, CH- 4031, Switzerland
| | - Mieke Deschodt
- Department of Public Health & Primary Care, KU Leuven, Leuven, Belgium
- Competence Centre of Nursing, University Hospitals Leuven, Leuven, Belgium
| | - Thomas Fabbro
- Department of Clinical Research, University Hospital Basel and University of Basel, Spitalstrasse 12, Basel, CH- 4031, Switzerland
| | - Christiane Pauli-Magnus
- Department of Clinical Research, University Hospital Basel and University of Basel, Spitalstrasse 12, Basel, CH- 4031, Switzerland
| | - Matthias Briel
- Department of Clinical Research, University Hospital Basel and University of Basel, Spitalstrasse 12, Basel, CH- 4031, Switzerland
- Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, ON, Canada
| |
Collapse
|
4
|
de Viron S, Trotta L, Steijn W, Young S, Buyse M. Does Central Monitoring Lead to Higher Quality? An Analysis of Key Risk Indicator Outcomes. Ther Innov Regul Sci 2023; 57:295-303. [PMID: 36269551 PMCID: PMC9589525 DOI: 10.1007/s43441-022-00470-5] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/08/2022] [Accepted: 09/30/2022] [Indexed: 11/21/2022]
Abstract
BACKGROUND Central monitoring, which typically includes the use of key risk indicators (KRIs), aims at improving the quality of clinical research by pro-actively identifying and remediating emerging issues in the conduct of a clinical trial that may have an adverse impact on patient safety and/or the reliability of trial results. However, there has to-date been a relative lack of direct quantitative evidence published supporting the claim that central monitoring actually leads to improved quality. MATERIAL AND METHODS Nine commonly used KRIs were analyzed for evidence of quality improvement using data retrieved from a large central monitoring platform. A total of 212 studies comprising 1676 sites with KRI signals were used in the analysis, representing central monitoring activity from 23 different sponsor organizations. Two quality improvement metrics were assessed for each KRI, one based on a statistical score (p-value) and the other based on a KRI's observed value. RESULTS Both KRI quality metrics showed improvement in a vast majority of sites (82.9% for statistical score, 81.1% for observed KRI value). Additionally, the statistical score and the observed KRI values improved, respectively by 66.1% and 72.4% on average towards the study average for those sites showing improvement. CONCLUSION The results of this analysis provide clear quantitative evidence supporting the hypothesis that use of KRIs in central monitoring is leading to improved quality in clinical trial conduct and associated data across participating sites.
Collapse
Affiliation(s)
- Sylviane de Viron
- CluePoints S.A., Avenue Albert Einstein, 2a, 1348, Louvain-la-Neuve, Belgium.
| | - Laura Trotta
- CluePoints S.A., Avenue Albert Einstein, 2a, 1348 Louvain-la-Neuve, Belgium
| | - William Steijn
- CluePoints S.A., Avenue Albert Einstein, 2a, 1348 Louvain-la-Neuve, Belgium
| | | | - Marc Buyse
- CluePoints S.A., Avenue Albert Einstein, 2a, 1348 Louvain-la-Neuve, Belgium ,grid.482598.aInternational Drug Development Institute (IDDI), Louvain-la-Neuve, Belgium ,grid.12155.320000 0001 0604 5662Interuniversity Institute for Biostatistics and Statistical Bioinformatics (I-BioStat), Hasselt University, Hasselt, Belgium
| |
Collapse
|
5
|
Cragg WJ, Hurley C, Yorke-Edwards V, Stenning SP. Assessing the potential for prevention or earlier detection of on-site monitoring findings from randomised controlled trials: Further analyses of findings from the prospective TEMPER triggered monitoring study. Clin Trials 2021; 18:115-126. [PMID: 33231127 PMCID: PMC7876652 DOI: 10.1177/1740774520972650] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/22/2022]
Abstract
BACKGROUND/AIMS Clinical trials should be designed and managed to minimise important errors with potential to compromise patient safety or data integrity, employ monitoring practices that detect and correct important errors quickly, and take robust action to prevent repetition. Regulators highlight the use of risk-based monitoring, making greater use of centralised monitoring and reducing reliance on centre visits. The TEMPER study was a prospective evaluation of triggered monitoring (a risk-based monitoring method), whereby centres are prioritised for visits based on central monitoring results. Conducted in three UK-based randomised cancer treatment trials of investigational medicine products with time-to-event outcomes, it found high levels of serious findings at triggered centre visits but also at visits to matched control centres that, based on central monitoring, were not of concern. Here, we report a detailed review of the serious findings from TEMPER centre visits. We sought to identify feasible, centralised processes which might detect or prevent these findings without a centre visit. METHODS The primary outcome of this study was the proportion of all 'major' and 'critical' TEMPER centre visit findings theoretically detectable or preventable through a feasible, centralised process. To devise processes, we considered a representative example of each finding type through an internal consensus exercise. This involved (a) agreeing the potential, by some described process, for each finding type to be centrally detected or prevented and (b) agreeing a proposed feasibility score for each proposed process. To further assess feasibility, we ran a consultation exercise, whereby the proposed processes were reviewed and rated for feasibility by invited external trialists. RESULTS In TEMPER, 312 major or critical findings were identified at 94 visits. These findings comprised 120 distinct issues, for which we proposed 56 different centralised processes. Following independent review of the feasibility of the proposed processes by 87 consultation respondents across eight different trial stakeholder groups, we conclude that 306/312 (98%) findings could theoretically be prevented or identified centrally. Of the processes deemed feasible, those relating to informed consent could have the most impact. Of processes not currently deemed feasible, those involving use of electronic health records are among those with the largest potential benefit. CONCLUSIONS This work presents a best-case scenario, where a large majority of monitoring findings were deemed theoretically preventable or detectable by central processes. Caveats include the cost of applying all necessary methods, and the resource implications of enhanced central monitoring for both centre and trials unit staff. Our results will inform future monitoring plans and emphasise the importance of continued critical review of monitoring processes and outcomes to ensure they remain appropriate.
Collapse
Affiliation(s)
- William J Cragg
- MRC Clinical Trials Unit at UCL, London,
UK
- Clinical Trials Research Unit, Leeds
Institute of Clinical Trials Research, University of Leeds, Leeds, UK
| | - Caroline Hurley
- Health Research Board-Trials Methodology
Research Network (HRB-TMRN), National University of Ireland, Galway, Ireland
| | | | | |
Collapse
|
6
|
Trotta L, Kabeya Y, Buyse M, Doffagne E, Venet D, Desmet L, Burzykowski T, Tsuburaya A, Yoshida K, Miyashita Y, Morita S, Sakamoto J, Praveen P, Oba K. Detection of atypical data in multicenter clinical trials using unsupervised statistical monitoring. Clin Trials 2019; 16:512-522. [PMID: 31331195 DOI: 10.1177/1740774519862564] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
BACKGROUND/AIMS A risk-based approach to clinical research may include a central statistical assessment of data quality. We investigated the operating characteristics of unsupervised statistical monitoring aimed at detecting atypical data in multicenter experiments. The approach is premised on the assumption that, save for random fluctuations and natural variations, data coming from all centers should be comparable and statistically consistent. Unsupervised statistical monitoring consists of performing as many statistical tests as possible on all trial data, in order to detect centers whose data are inconsistent with data from other centers. METHODS We conducted simulations using data from a large multicenter trial conducted in Japan for patients with advanced gastric cancer. The actual trial data were contaminated in computer simulations for varying percentages of centers, percentages of patients modified within each center and numbers and types of modified variables. The unsupervised statistical monitoring software was run by a blinded team on the contaminated data sets, with the purpose of detecting the centers with contaminated data. The operating characteristics (sensitivity, specificity and Youden's J-index) were calculated for three detection methods: one using the p-values of individual statistical tests after adjustment for multiplicity, one using a summary of all p-values for a given center, called the Data Inconsistency Score, and one using both of these methods. RESULTS The operating characteristics of the three methods were satisfactory in situations of data contamination likely to occur in practice, specifically when a single or a few centers were contaminated. As expected, the sensitivity increased for increasing proportions of patients and increasing numbers of variables contaminated. The three methods showed a specificity better than 93% in all scenarios of contamination. The method based on the Data Inconsistency Score and individual p-values adjusted for multiplicity generally had slightly higher sensitivity at the expense of a slightly lower specificity. CONCLUSIONS The use of brute force (a computer-intensive approach that generates large numbers of statistical tests) is an effective way to check data quality in multicenter clinical trials. It can provide a cost-effective complement to other data-management and monitoring techniques.
Collapse
Affiliation(s)
| | - Yuusuke Kabeya
- Department of Biostatistics, The University of Tokyo, Tokyo, Japan.,EPS Corporation, Tokyo, Japan
| | - Marc Buyse
- International Drug Development Institute (IDDI), San Francisco, CA, USA.,CluePoints, Wayne, PA, USA
| | | | - David Venet
- Institut de Recherches Interdisciplinaires et de Développements en Intelligence Artificielle (IRIDIA), University of Brussels, Brussels, Belgium
| | - Lieven Desmet
- Institute of Statistics, Biostatistics and Actuarial Sciences (ISBA), University of Louvain, Louvain-la-Neuve, Belgium
| | - Tomasz Burzykowski
- International Drug Development Institute (IDDI), Louvain-la-Neuve, Belgium.,Interuniversity Institute for Biostatistics and Statistical Bioinformatics (I-BioStat), University of Hasselt, Hasselt, Belgium
| | - Akira Tsuburaya
- Department of Surgery, Jizankai Medical Foundation, Tsuboi Cancer Center Hospital, Koriyama, Japan
| | - Kazuhiro Yoshida
- Department of Surgical Oncology, Graduate School of Medicine, Gifu University, Gifu, Japan
| | - Yumi Miyashita
- Epidemiological and Clinical Research Information Network (ECRIN), Okazaki, Japan
| | - Satoshi Morita
- Department of Biomedical Statistics and Bioinformatics, Graduate School of Medicine, Kyoto University, Kyoto, Japan
| | - Junichi Sakamoto
- Epidemiological and Clinical Research Information Network (ECRIN), Okazaki, Japan.,Tokai Central Hospital, Kakamigahara, Japan
| | | | - Koji Oba
- Department of Biostatistics, The University of Tokyo, Tokyo, Japan.,Interfaculty Initiative in Information Studies, The University of Tokyo, Tokyo, Japan
| |
Collapse
|
7
|
Walker KF, Turzanski J, Whitham D, Montgomery A, Duley L. Monitoring performance of sites within multicentre randomised trials: a systematic review of performance metrics. Trials 2018; 19:562. [PMID: 30326948 PMCID: PMC6192157 DOI: 10.1186/s13063-018-2941-8] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2018] [Accepted: 09/26/2018] [Indexed: 11/11/2022] Open
Abstract
Background Large multicentre trials are complex and expensive projects. A key factor for their successful planning and delivery is how well sites meet their targets in recruiting and retaining participants, and in collecting high-quality, complete data in a timely manner. Collecting and monitoring easily accessible data relevant to performance of sites has the potential to improve trial management efficiency. The aim of this systematic review was to identify metrics that have either been proposed or used for monitoring site performance in multicentre trials. Methods We searched the Cochrane Library, five biomedical bibliographic databases (CINAHL, EMBASE, Medline, PsychINFO and SCOPUS) and Google Scholar for studies describing ways of monitoring or measuring individual site performance in multicentre randomised trials. Records identified were screened for eligibility. For included studies, data on study content were extracted independently by two reviewers, and disagreements resolved by discussion. Results After removing duplicate citations, we identified 3188 records. Of these, 21 were eligible for inclusion and yielded 117 performance metrics. The median number of metrics reported per paper was 8, range 1–16. Metrics broadly fell into six categories: site potential; recruitment; retention; data collection; trial conduct and trial safety. Conclusions This review identifies a list of metrics to monitor site performance within multicentre randomised trials. Those that would be easy to collect, and for which monitoring might trigger actions to mitigate problems at site level, merit further evaluation.
Collapse
Affiliation(s)
- Kate F Walker
- Nottingham Clinical Trials Unit, QMC, Nottingham, NG7 2UH, UK.
| | - Julie Turzanski
- Nottingham Clinical Trials Unit, QMC, Nottingham, NG7 2UH, UK
| | - Diane Whitham
- Nottingham Clinical Trials Unit, QMC, Nottingham, NG7 2UH, UK
| | - Alan Montgomery
- Nottingham Clinical Trials Unit, QMC, Nottingham, NG7 2UH, UK
| | - Lelia Duley
- Nottingham Clinical Trials Unit, QMC, Nottingham, NG7 2UH, UK
| |
Collapse
|
8
|
McNamara C, Engelhardt N, Potter W, Yavorsky C, Masotti M, Di Clemente G. Risk-Based Data Monitoring: Quality Control in Central Nervous System (CNS) Clinical Trials. Ther Innov Regul Sci 2018; 53:176-182. [PMID: 29758992 DOI: 10.1177/2168479018774325] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
Monitoring the quality of clinical trial efficacy outcome data has received increased attention in the past decade, with regulatory guidance encouraging it to be conducted proactively, and remotely. However, the methods utilized to develop and implement risk-based data monitoring (RBDM) programs vary, and there is a dearth of published material to guide these processes in the context of central nervous system (CNS) trials. We reviewed regulatory guidance published within the past 6 years, generic white papers, and studies applying RBDM to data from CNS clinical trials. Methodologic considerations and system requirements necessary to establish an effective, real-time risk-based monitoring platform in CNS trials are presented. Key RBDM terms are defined in the context of CNS trial data, such as "critical data," "risk indicators," "noninformative data," and "mitigation of risk." Additionally, potential benefits of, and challenges associated with implementation of data quality monitoring are highlighted. Application of methodological and system requirement considerations to real-time monitoring of clinical ratings in CNS trials has the potential to minimize risk and enhance the quality of clinical trial data.
Collapse
Affiliation(s)
- Cynthia McNamara
- 1 Cronos Clinical Consulting Services, Inc, Lambertville, NJ, USA
| | - Nina Engelhardt
- 1 Cronos Clinical Consulting Services, Inc, Lambertville, NJ, USA
| | - William Potter
- 2 National Institute of Mental Health, Bethesda, MD, USA
| | | | - Matthew Masotti
- 1 Cronos Clinical Consulting Services, Inc, Lambertville, NJ, USA
| | | |
Collapse
|
9
|
Zink RC, Dmitrienko A, Dmitrienko A. Rethinking the Clinically Based Thresholds of TransCelerate BioPharma for Risk-Based Monitoring. Ther Innov Regul Sci 2018; 52:560-571. [PMID: 29714565 DOI: 10.1177/2168479017738981] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
BACKGROUND The quality of data from clinical trials has received a great deal of attention in recent years. Of central importance is the need to protect the well-being of study participants and maintain the integrity of final analysis results. However, traditional approaches to assess data quality have come under increased scrutiny as providing little benefit for the substantial cost. Numerous regulatory guidance documents and industry position papers have described risk-based approaches to identify quality and safety issues. In particular, the position paper of TransCelerate BioPharma recommends defining risk thresholds to assess safety and quality risks based on past clinical experience. This exercise can be extremely time-consuming, and the resulting thresholds may only be relevant to a particular therapeutic area, patient or clinical site population. In addition, predefined thresholds cannot account for safety or quality issues where the underlying rate of observing a particular problem may change over the course of a clinical trial, and often do not consider varying patient exposure. METHODS In this manuscript, we appropriate rules commonly utilized for funnel plots to define a traffic-light system for risk indicators based on statistical criteria that consider the duration of patient follow-up. Further, we describe how these methods can be adapted to assess changing risk over time. Finally, we illustrate numerous graphical approaches to summarize and communicate risk, and discuss hybrid clinical-statistical approaches to allow for the assessment of risk at sites with low patient enrollment. RESULTS We illustrate the aforementioned methodologies for a clinical trial in patients with schizophrenia. CONCLUSIONS Funnel plots are a flexible graphical technique that can form the basis for a risk-based strategy to assess data integrity, while considering site sample size, patient exposure, and changing risk across time.
Collapse
Affiliation(s)
- Richard C Zink
- 1 JMP Life Sciences, SAS Institute Inc, Cary, NC, USA.,2 Department of Biostatistics, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | | | | |
Collapse
|
10
|
Brosteanu O, Schwarz G, Houben P, Paulus U, Strenge-Hesse A, Zettelmeyer U, Schneider A, Hasenclever D. Risk-adapted monitoring is not inferior to extensive on-site monitoring: Results of the ADAMON cluster-randomised study. Clin Trials 2017; 14:584-596. [PMID: 28786330 PMCID: PMC5718334 DOI: 10.1177/1740774517724165] [Citation(s) in RCA: 34] [Impact Index Per Article: 4.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Background According to Good Clinical Practice, clinical trials must protect rights and
safety of patients and make sure that the trial results are valid and
interpretable. Monitoring on-site has an important role in achieving these
objectives; it controls trial conduct at trial sites and informs the sponsor
on systematic problems. In the past, extensive on-site monitoring with a
particular focus on formal source data verification often lost sight of
systematic problems in study procedures that endanger Good Clinical Practice
objectives. ADAMON is a prospective, stratified, cluster-randomised,
controlled study comparing extensive on-site monitoring with risk-adapted
monitoring according to a previously published approach. Methods In all, 213 sites from 11 academic trials were cluster-randomised between
extensive on-site monitoring (104) and risk-adapted monitoring (109).
Independent post-trial audits using structured manuals were performed to
determine the frequency of major Good Clinical Practice findings at the
patient level. The primary outcome measure is the proportion of audited
patients with at least one major audit finding. Analysis relies on logistic
regression incorporating trial and monitoring arm as fixed effects and site
as random effect. The hypothesis was that risk-adapted monitoring is
non-inferior to extensive on-site monitoring with a non-inferiority margin
of 0.60 (logit scale). Results Average number of monitoring visits and time spent on-site was 2.1 and 2.7
times higher in extensive on-site monitoring than in risk-adapted
monitoring, respectively. A total of 156 (extensive on-site monitoring: 76;
risk-adapted monitoring: 80) sites were audited. In 996 of 1618 audited
patients, a total of 2456 major audit findings were documented. Depending on
the trial, findings were identified in 18%–99% of the audited patients, with
no marked monitoring effect in any of the trials. The estimated monitoring
effect is −0.04 on the logit scale with two-sided 95% confidence interval
(−0.40; 0.33), demonstrating that risk-adapted monitoring is non-inferior to
extensive on-site monitoring. At most, extensive on-site monitoring could
reduce the frequency of major Good Clinical Practice findings by 8.2%
compared with risk-adapted monitoring. Conclusion Compared with risk-adapted monitoring, the potential benefit of extensive
on-site monitoring is small relative to overall finding rates, although
risk-adapted monitoring requires less than 50% of extensive on-site
monitoring resources. Clusters of findings within trials suggest that
complicated, overly specific or not properly justified protocol requirements
contributed to the overall frequency of findings. Risk-adapted monitoring in
only a sample of patients appears sufficient to identify systematic problems
in the conduct of clinical trials. Risk-adapted monitoring has a part to
play in quality control. However, no monitoring strategy can remedy defects
in quality of design. Monitoring should be embedded in a comprehensive
quality management approach covering the entire trial lifecycle.
Collapse
Affiliation(s)
- Oana Brosteanu
- 1 Clinical Trial Centre Leipzig, Leipzig University, Leipzig, Germany
| | - Gabriele Schwarz
- 2 Federal Institute for Drugs and Medical Devices, Bonn, Germany
| | - Peggy Houben
- 1 Clinical Trial Centre Leipzig, Leipzig University, Leipzig, Germany
| | - Ursula Paulus
- 3 Clinical Trials Centre Cologne, University of Cologne, Cologne, Germany
| | - Anke Strenge-Hesse
- 4 KKS-Network/National ECRIN Office, University of Cologne, Cologne, Germany
| | - Ulrike Zettelmeyer
- 3 Clinical Trials Centre Cologne, University of Cologne, Cologne, Germany
| | - Anja Schneider
- 1 Clinical Trial Centre Leipzig, Leipzig University, Leipzig, Germany
| | - Dirk Hasenclever
- 5 Institute for Medical Informatics, Statistics, and Epidemiology, Leipzig University, Leipzig, Germany
| |
Collapse
|
11
|
Gough J, Wilson B, Zerola M, Wallis P, Mork L, Knepper D, Achenbach H. Defining a Central Monitoring Capability: Sharing the Experience of TransCelerate BioPharma's Approach, Part 2. Ther Innov Regul Sci 2016; 50:8-14. [PMID: 30236019 DOI: 10.1177/2168479015618696] [Citation(s) in RCA: 13] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
BACKGROUND TransCelerate's model approach to risk-based monitoring (RBM) includes the application of the appropriate monitoring activities to enable both the early detection and timely resolution of issues. This article is a follow-up to part 1, published in the September 2014 issue with the same title. METHODS The intent of this paper is to share information on what has been learned by various companies' applications of central monitoring activities based on different RBM operating models. A library of risk indicators has been created, and this paper provides additional guidance on what has been learned in the application of these tools. RESULTS The goal is to share the needs related to people, process, and technology as experienced by TransCelerate member companies. CONCLUSIONS One of the primary issue detection methods of central monitoring is the proactive identification of areas of focus through the use of risk indicators.
Collapse
Affiliation(s)
- Jacqueline Gough
- 1 Clinical Risk Management, Eli Lilly and Company, Indianapolis, IN, USA
| | - Brett Wilson
- 2 Global Development Operations, Research & Development, Bristol-Myers Squibb, Princeton, NJ, USA
| | - Mireille Zerola
- 3 Biometrics and Data Management, Boehringer Ingelheim, Bracknell, Berkshire, United Kingdom
| | - Phil Wallis
- 4 Global Development Operations, Research & Development, Pfizer, Groton, CT, USA
| | - Laila Mork
- 5 Global Clinical Trial Management, Drug Development Operations, Allergan, Irvine, CA, USA
| | - David Knepper
- 6 Business Operations, Allergan, Jersey City, NJ, USA
| | | |
Collapse
|