Assessing the performance of methods for central statistical monitoring of a binary or continuous outcome in multi-center trials: A simulation study

BACKGROUND

Quality study monitoring is fundamental to patient safety and data integrity. Regulators and industry consortia have increasingly advocated for risk-based monitoring (RBM) and central statistical monitoring (CSM) for more effective and efficient monitoring. Assessing which statistical methods underpin these approaches can best identify unusual data patterns in multi-center clinical trials that may be driven by potential systematic errors is important.

METHODS

We assessed various CSM techniques, including cross-tests, fixed-effects, mixed-effects, and finite mixture models, across scenarios with different sample sizes, contamination rates, and overdispersion via simulation. Our evaluation utilized threshold-independent metrics such as the area under the curve (AUC) and average precision (AP), offering a fuller picture of CSM performance.

RESULTS

All CSM methods showed consistent characteristics across center sizes or overdispersion. The adaptive finite mixture model outperformed others in AUC and AP, especially at 30% contamination, upholding high specificity unless converging to a single-component model due to low contamination or deviation. The mixed-effects model performed well at lower contamination rates. However, it became conservative in specificity and exhibited declined performance for binary outcomes under high deviation. Cross-tests and fixed-effects methods underperformed, especially when deviation increased.

CONCLUSION

Our evaluation explored the merits and drawbacks of multiple CSM methods, and found that relying on sensitivity and specificity alone is likely insufficient to fully measure predictive performance. The finite mixture method demonstrated more consistent performance across scenarios by mitigating the influence of outliers. In practice, considering the study-specific costs of false positives/negatives with available resources for monitoring is important.

Widening participation - recruitment methods in mental health randomised controlled trials: a qualitative study

BACKGROUND

Barriers to mental health research participation are well documented including distrust of services and research; and stigma surrounding mental health. They can contribute to a lack of diversity amongst participants in mental health research, which threatens the generalisability of knowledge. Given the recent widespread use of the internet in medical research, this study aimed to explore the perspectives of key partners on the use of online (e.g. social media) and offline (e.g. in-person) recruitment as an approach to improving diversity in mental health randomised controlled trials (RCTs).

METHODS

Face-to-face and online interviews/focus groups with researchers working in mental health and Patient and Public Involvement partners in the United Kingdom. Recordings were transcribed and analysed using a combination of inductive and deductive thematic analysis.

RESULTS

Three focus groups and three interviews were conducted with a total N = 23 participants. Four overarching themes were identified: (1) recruitment reach; (2) Demographic factors that affect selection of recruitment method; (3) safety of technology, and; (4) practical challenges. Five main factors were identified that affect the choice of recruitment method: age, complexity of mental health problem and stigma, cultural and ethnicity differences and digital divide. The use of online methods was considered more accessible to people who may feel stigmatised by their mental health condition and with a benefit of reaching a wider population. However, a common view amongst participants was that online methods require closer data monitoring for quality of responders, are not fully secure and less trustworthy compared to offline methods that enable participants to build relationships with health providers. Funding, staff time and experience, organisational support, and technical issues such as spam or phishing emails were highlighted as practical challenges facing online recruitment. All participants agreed that using a hybrid approach tailored to the population under study is paramount.

CONCLUSIONS

This study highlighted the importance of offering a flexible and multifaceted recruitment approach by integrating online with offline methods to support inclusivity and widening participation in mental health research. The findings will be used to develop considerations for researchers designing RCTs to improve recruitment in mental health research.

Data management in diabetes clinical trials: a qualitative study

Background

Clinical trials play an important role in expanding the knowledge of diabetes prevention, diagnosis, and treatment, and data management is one of the main issues in clinical trials. Lack of appropriate planning for data management in clinical trials may negatively influence achieving the desired results. The aim of this study was to explore data management processes in diabetes clinical trials in three research institutes in Iran.

Method

This was a qualitative study conducted in 2019. In this study, data were collected through in-depth semi-structured interviews with 16 researchers in three endocrinology and metabolism research institutes. To analyze data, the method of thematic analysis was used.

Results

The five themes that emerged from data analysis included (1) clinical trial data collection, (2) technologies used in data management, (3) data security and confidentiality management, (4) data quality management, and (5) data management standards. In general, the findings indicated that no clear and standard process was used for data management in diabetes clinical trials, and each research center executed its own methods and processes.

Conclusion

According to the results, the common methods of data management in diabetes clinical trials included a set of paper-based processes. It seems that using information technology can help facilitate data management processes in a variety of clinical trials, including diabetes clinical trials.

Bots and nots: Safeguarding online survey research with underrepresented and diverse populations

The COVID-19 pandemic has intensified the use of online recruitment and data collection for reaching historically underrepresented minorities (URMs) and other diverse groups. Preventing and detecting responses from automated accounts "bots" and those who misrepresent themselves is one challenge in utilizing online approaches. Through internet-mediated methods, interested LGBTQ+ and non-LGBTQ+ couples facing advanced cancer completed an interest form via REDCap®. Eligible participants received a direct link to electronic consent and surveys in REDCap®. Once responses to the interest form (N = 619) were received, the study PI: 1) assessed participants' entries and non-response survey data (time of completion, rate of recruitment, etc.), 2) temporarily postponed recruitment, 3) sent eligibility questionnaires, consent documents, and validated surveys to N= 10 couples and scrutinized these data for suspicious patterns or indications of untrustworthy data, 4) responded to potential participants via email, and 5) implemented additional strategies for detecting and preventing untrustworthy survey responses. Investigators must consider multi-step eligibility screening processes to detect and prevent the collection of untrustworthy data. Investigators' reliance on internet-mediated approaches for conducting research with diverse, hard-to-reach populations increases the importance of addressing threats to data validity. Ultimately, safeguarding internet-mediated research supports research accessibility and inclusion for URMs while also protecting participant data integrity.

Detecting fabrication in large-scale molecular omics data

Fraud is a pervasive problem and can occur as fabrication, falsification, plagiarism, or theft. The scientific community is not exempt from this universal problem and several studies have recently been caught manipulating or fabricating data. Current measures to prevent and deter scientific misconduct come in the form of the peer-review process and on-site clinical trial auditors. As recent advances in high-throughput omics technologies have moved biology into the realm of big-data, fraud detection methods must be updated for sophisticated computational fraud. In the financial sector, machine learning and digit-frequencies are successfully used to detect fraud. Drawing from these sources, we develop methods of fabrication detection in biomedical research and show that machine learning can be used to detect fraud in large-scale omic experiments. Using the gene copy-number data as input, machine learning models correctly predicted fraud with 58-100% accuracy. With digit frequency as input features, the models detected fraud with 82%-100% accuracy. All of the data and analysis scripts used in this project are available at https://github.com/MSBradshaw/FakeData.

Time-consuming and expensive data quality monitoring procedures persist in clinical trials: A national survey

INTRODUCTION

The Good Clinical Practice guideline identifies that data monitoring is an essential research activity. However, limited evidence exists on how to perform monitoring including the amount or frequency that is needed to ensure data quality. This study aims to explore the monitoring procedures that are implemented to ensure data quality in Australian clinical research studies.

MATERIAL AND METHODS

Clinical studies listed on the Australian and New Zealand Clinical Trials Registry were invited to participate in a national cross-sectional, mixed-mode, multi-contact (postal letter and e-mail) web-based survey. Information was gathered about the types of data quality monitoring procedures being implemented.

RESULTS

Of the 3689 clinical studies contacted, 589 (16.0%) responded, of which 441 (77.4%) completed the survey. Over half (55%) of the studies applied source data verification (SDV) compared to risk-based targeted and triggered monitoring (10-11%). Conducting 100% on-site monitoring was most common for those who implemented the traditional approach. Respondents who did not conduct 100% monitoring, included 1-25% of data points for SDV, centralized or on-site monitoring. The incidence of adverse events and protocol deviations were the most likely factors to trigger a site visit for risk-based triggered (63% and 44%) and centralized monitoring (48% and 44%), respectively.

CONCLUSION

Instead of using more optimal risk-based approaches, small single-site clinical studies are conducting traditional monitoring procedures which are time consuming and expensive. Formal guidelines need to be improved and provided to all researchers for 'new' risk-based monitoring approaches.

Comparison of Data on Serious Adverse Events and Mortality in ClinicalTrials.gov, Corresponding Journal Articles, and FDA Medical Reviews: Cross-Sectional Analysis

INTRODUCTION

Inconsistencies in data on serious adverse events (SAEs) and mortality in ClinicalTrials.gov and corresponding journal articles pose a challenge to research transparency.

OBJECTIVE

The objective of this study was to compare data on SAEs and mortality from clinical trials reported in ClinicalTrials.gov and corresponding journal articles with US Food and Drug Administration (FDA) medical reviews.

METHODS

We conducted a cross-sectional study of a randomly selected sample of new molecular entities approved during the study period 1 January 2013 to 31 December 2015. We extracted data on SAEs and mortality from 15 pivotal trials from ClinicalTrials.gov and corresponding journal articles (the two index resources), and FDA medical reviews (reference standard). We estimated the magnitude of deviations in rates of SAEs and mortality between the index resources and the reference standard.

RESULTS

We found deviations in rates of SAEs (30% in ClinicalTrials.gov and 30% in corresponding journal articles) and mortality (72% in ClinicalTrials.gov and 53% in corresponding journal articles) when compared with the reference standard. The intra-class correlation coefficient between the three resources was 0.99 (95% confidence interval [CI] 0.98-0.99) for SAE rates and 0.99 (95% CI 0.97-0.99) for mortality rates.

CONCLUSION

There are differences in data on rates of SAEs and mortality in randomized clinical trials in both ClinicalTrials.gov and journal articles compared with FDA reviews. Further efforts should focus on decreasing existing discrepancies to enhance the transparency and reproducibility of data reporting in clinical trials.

Investigation of the Factors Affecting Risk-Based Quality Management of Investigator-Initiated Investigational New-Drug Trials for Unapproved Anticancer Drugs in Japan

BACKGROUND

With an increase in the complexity and cost of clinical trials and the advances in information technology, monitoring guidance issued by regulatory authorities recommends risk-adapted monitoring. To introduce the monitoring method for investigator-initiated investigational new drug (IND) trials using unapproved anticancer drugs, we performed exploratory retrospective analyses to identify risk factors for data quality.

METHODS

To select investigator-initiated IND trials using unapproved anticancer drugs, we set the trial selection criteria. Data collection was performed by using audit trails and monitoring reports. Collected data were analyzed by univariate and multivariate analyses to identify the independent risk factors related to error.

RESULTS

By trial selection criteria, 5 investigator-initiated IND trials using unapproved anticancer drugs were selected. The error rates of the total data, critical data, and noncritical data were 7.4%, 9.7%, and 5.9%, respectively. There was no difference between clinical research core hospitals certified by the Ministry of Health, Labour and Welfare and other hospitals in univariate analysis (odds ratio [OR], 1.00; 99% confidence interval [CI], 0.96-1.05; P = .9179). As the main independent risk factors related to error, critical data in the importance of data (OR, 1.28; 99% CI, 1.24-1.33; P < .0001) and groups with ≤3 patients after registration (OR, 1.12; 99% CI, 1.10-1.15; P < .0001) were significantly related to errors in multivariate analysis.

CONCLUSIONS

The results of this research suggest that the feasibility of risk-based monitoring and sampling source data verification was indicated for noncritical data and patients after the third case.

Risk-Based Data Monitoring: Quality Control in Central Nervous System (CNS) Clinical Trials

Monitoring the quality of clinical trial efficacy outcome data has received increased attention in the past decade, with regulatory guidance encouraging it to be conducted proactively, and remotely. However, the methods utilized to develop and implement risk-based data monitoring (RBDM) programs vary, and there is a dearth of published material to guide these processes in the context of central nervous system (CNS) trials. We reviewed regulatory guidance published within the past 6 years, generic white papers, and studies applying RBDM to data from CNS clinical trials. Methodologic considerations and system requirements necessary to establish an effective, real-time risk-based monitoring platform in CNS trials are presented. Key RBDM terms are defined in the context of CNS trial data, such as "critical data," "risk indicators," "noninformative data," and "mitigation of risk." Additionally, potential benefits of, and challenges associated with implementation of data quality monitoring are highlighted. Application of methodological and system requirement considerations to real-time monitoring of clinical ratings in CNS trials has the potential to minimize risk and enhance the quality of clinical trial data.

Rethinking the Clinically Based Thresholds of TransCelerate BioPharma for Risk-Based Monitoring

BACKGROUND

The quality of data from clinical trials has received a great deal of attention in recent years. Of central importance is the need to protect the well-being of study participants and maintain the integrity of final analysis results. However, traditional approaches to assess data quality have come under increased scrutiny as providing little benefit for the substantial cost. Numerous regulatory guidance documents and industry position papers have described risk-based approaches to identify quality and safety issues. In particular, the position paper of TransCelerate BioPharma recommends defining risk thresholds to assess safety and quality risks based on past clinical experience. This exercise can be extremely time-consuming, and the resulting thresholds may only be relevant to a particular therapeutic area, patient or clinical site population. In addition, predefined thresholds cannot account for safety or quality issues where the underlying rate of observing a particular problem may change over the course of a clinical trial, and often do not consider varying patient exposure.

METHODS

In this manuscript, we appropriate rules commonly utilized for funnel plots to define a traffic-light system for risk indicators based on statistical criteria that consider the duration of patient follow-up. Further, we describe how these methods can be adapted to assess changing risk over time. Finally, we illustrate numerous graphical approaches to summarize and communicate risk, and discuss hybrid clinical-statistical approaches to allow for the assessment of risk at sites with low patient enrollment.

RESULTS

We illustrate the aforementioned methodologies for a clinical trial in patients with schizophrenia.

CONCLUSIONS

Funnel plots are a flexible graphical technique that can form the basis for a risk-based strategy to assess data integrity, while considering site sample size, patient exposure, and changing risk across time.

Risk-adapted monitoring is not inferior to extensive on-site monitoring: Results of the ADAMON cluster-randomised study

Background

According to Good Clinical Practice, clinical trials must protect rights and safety of patients and make sure that the trial results are valid and interpretable. Monitoring on-site has an important role in achieving these objectives; it controls trial conduct at trial sites and informs the sponsor on systematic problems. In the past, extensive on-site monitoring with a particular focus on formal source data verification often lost sight of systematic problems in study procedures that endanger Good Clinical Practice objectives. ADAMON is a prospective, stratified, cluster-randomised, controlled study comparing extensive on-site monitoring with risk-adapted monitoring according to a previously published approach.

Methods

In all, 213 sites from 11 academic trials were cluster-randomised between extensive on-site monitoring (104) and risk-adapted monitoring (109). Independent post-trial audits using structured manuals were performed to determine the frequency of major Good Clinical Practice findings at the patient level. The primary outcome measure is the proportion of audited patients with at least one major audit finding. Analysis relies on logistic regression incorporating trial and monitoring arm as fixed effects and site as random effect. The hypothesis was that risk-adapted monitoring is non-inferior to extensive on-site monitoring with a non-inferiority margin of 0.60 (logit scale).

Results

Average number of monitoring visits and time spent on-site was 2.1 and 2.7 times higher in extensive on-site monitoring than in risk-adapted monitoring, respectively. A total of 156 (extensive on-site monitoring: 76; risk-adapted monitoring: 80) sites were audited. In 996 of 1618 audited patients, a total of 2456 major audit findings were documented. Depending on the trial, findings were identified in 18%–99% of the audited patients, with no marked monitoring effect in any of the trials. The estimated monitoring effect is −0.04 on the logit scale with two-sided 95% confidence interval (−0.40; 0.33), demonstrating that risk-adapted monitoring is non-inferior to extensive on-site monitoring. At most, extensive on-site monitoring could reduce the frequency of major Good Clinical Practice findings by 8.2% compared with risk-adapted monitoring.

Conclusion

Compared with risk-adapted monitoring, the potential benefit of extensive on-site monitoring is small relative to overall finding rates, although risk-adapted monitoring requires less than 50% of extensive on-site monitoring resources. Clusters of findings within trials suggest that complicated, overly specific or not properly justified protocol requirements contributed to the overall frequency of findings. Risk-adapted monitoring in only a sample of patients appears sufficient to identify systematic problems in the conduct of clinical trials. Risk-adapted monitoring has a part to play in quality control. However, no monitoring strategy can remedy defects in quality of design. Monitoring should be embedded in a comprehensive quality management approach covering the entire trial lifecycle.