Toward development of a comprehensive external quality assurance program for polyfunctional intracellular cytokine staining assays

The External Quality Assurance Program Oversight Laboratory (EQAPOL) Flow Cytometry Program assesses the proficiency of NIH/NIAID/DAIDS-supported and potentially other interested research laboratories in performing Intracellular Cytokine Staining (ICS) assays. The goal of the EQAPOL Flow Cytometry External Quality Assurance Program (EQAP) is to provide proficiency testing and remediation for participating sites. The program is not punitive; rather, EQAPOL aims to help sites identify areas for improvement. EQAPOL utilizes a highly standardized ICS assay to minimize variability and readily identify those sites experiencing technical difficulties with their assays. Here, we report the results of External Proficiency 3 (EP3) where participating sites performed a 7-color ICS assay. On average, sites perform well in the Flow Cytometry EQAP (median score is "Good"). The most common technical issues identified by the program involve protocol adherence and data analysis; these areas have been the focus of site remediation. The EQAPOL Flow Cytometry team is now in the process of expanding the program to 8-color ICS assays. Evaluating polyfunctional ICS responses would align the program with assays currently being performed in support of HIV immune monitoring assays.

Monitoring performance of sites within multicentre randomised trials: a systematic review of performance metrics

Background

Large multicentre trials are complex and expensive projects. A key factor for their successful planning and delivery is how well sites meet their targets in recruiting and retaining participants, and in collecting high-quality, complete data in a timely manner. Collecting and monitoring easily accessible data relevant to performance of sites has the potential to improve trial management efficiency. The aim of this systematic review was to identify metrics that have either been proposed or used for monitoring site performance in multicentre trials.

Methods

We searched the Cochrane Library, five biomedical bibliographic databases (CINAHL, EMBASE, Medline, PsychINFO and SCOPUS) and Google Scholar for studies describing ways of monitoring or measuring individual site performance in multicentre randomised trials. Records identified were screened for eligibility. For included studies, data on study content were extracted independently by two reviewers, and disagreements resolved by discussion.

Results

After removing duplicate citations, we identified 3188 records. Of these, 21 were eligible for inclusion and yielded 117 performance metrics. The median number of metrics reported per paper was 8, range 1–16. Metrics broadly fell into six categories: site potential; recruitment; retention; data collection; trial conduct and trial safety.

Conclusions

This review identifies a list of metrics to monitor site performance within multicentre randomised trials. Those that would be easy to collect, and for which monitoring might trigger actions to mitigate problems at site level, merit further evaluation.

Improving data quality in observational research studies: Report of the Cure Glomerulonephropathy (CureGN) network

BACKGROUND

High data quality is of crucial importance to the integrity of research projects. In the conduct of multi-center observational cohort studies with increasing types and quantities of data, maintaining data quality is challenging, with few published guidelines.

METHODS

The Cure Glomerulonephropathy (CureGN) Network has established numerous quality control procedures to manage the 70 participating sites in the United States, Canada, and Europe. This effort is supported and guided by the activities of several committees, including Data Quality, Recruitment and Retention, and Central Review, that work in tandem with the Data Coordinating Center to monitor the study. We have implemented coordinator training and feedback channels, data queries of questionable or missing data, and developed performance metrics for recruitment, retention, visit completion, data entry, recording of patient-reported outcomes, collection, shipping and accessing of biological samples and pathology materials, and processing, cataloging and accessing genetic data and materials.

RESULTS

We describe the development of data queries and site Report Cards, and their use in monitoring and encouraging excellence in site performance. We demonstrate improvements in data quality and completeness over 4 years after implementing these activities. We describe quality initiatives addressing specific challenges in collecting and cataloging whole slide images and other kidney pathology data, and novel methods of data quality assessment.

CONCLUSIONS

This paper reports the CureGN experience in optimizing data quality and underscores the importance of general and study-specific data quality initiatives to maintain excellence in the research measures of a multi-center observational study.

Quantifying and visualizing site performance in clinical trials

Background

One of the keys to running a successful clinical trial is the selection of high quality clinical sites, i.e., sites that are able to enroll patients quickly, engage them on an ongoing basis to prevent drop-out, and execute the trial in strict accordance to the clinical protocol. Intuitively, the historical track record of a site is one of the strongest predictors of its future performance; however, issues such as data availability and wide differences in protocol complexity can complicate interpretation. Here, we demonstrate how operational data derived from central laboratory services can provide key insights into the performance of clinical sites and help guide operational planning and site selection for new clinical trials.

Methods

Our methodology uses the metadata associated with laboratory kit shipments to clinical sites (such as trial and anonymized patient identifiers, investigator names and addresses, sample collection and shipment dates, etc.) to reconstruct the complete schedule of patient visits and derive insights about the operational performance of those sites, including screening, enrollment, and drop-out rates and other quality indicators. This information can be displayed in its raw form or normalized to enable direct comparison of site performance across studies of varied design and complexity.

Results

Leveraging Covance's market leadership in central laboratory services, we have assembled a database of operational metrics that spans more than 14,000 protocols, 1400 indications, 230,000 unique investigators, and 23 million patient visits and represents a significant fraction of all clinical trials run globally in the last few years. By analyzing this historical data, we are able to assess and compare the performance of clinical investigators across a wide range of therapeutic areas and study designs. This information can be aggregated across trials and geographies to gain further insights into country and regional trends, sometimes with surprising results.

Conclusions

The use of operational data from Covance Central Laboratories provides a unique perspective into the performance of clinical sites with respect to many important metrics such as patient enrollment and retention. These metrics can, in turn, be used to guide operational planning and site selection for new clinical trials, thereby accelerating recruitment, improving quality, and reducing cost.

Does Central Monitoring Lead to Higher Quality? An Analysis of Key Risk Indicator Outcomes

BACKGROUND

Central monitoring, which typically includes the use of key risk indicators (KRIs), aims at improving the quality of clinical research by pro-actively identifying and remediating emerging issues in the conduct of a clinical trial that may have an adverse impact on patient safety and/or the reliability of trial results. However, there has to-date been a relative lack of direct quantitative evidence published supporting the claim that central monitoring actually leads to improved quality.

MATERIAL AND METHODS

Nine commonly used KRIs were analyzed for evidence of quality improvement using data retrieved from a large central monitoring platform. A total of 212 studies comprising 1676 sites with KRI signals were used in the analysis, representing central monitoring activity from 23 different sponsor organizations. Two quality improvement metrics were assessed for each KRI, one based on a statistical score (p-value) and the other based on a KRI's observed value.

RESULTS

Both KRI quality metrics showed improvement in a vast majority of sites (82.9% for statistical score, 81.1% for observed KRI value). Additionally, the statistical score and the observed KRI values improved, respectively by 66.1% and 72.4% on average towards the study average for those sites showing improvement.

CONCLUSION

The results of this analysis provide clear quantitative evidence supporting the hypothesis that use of KRIs in central monitoring is leading to improved quality in clinical trial conduct and associated data across participating sites.

Utilization of a structured research site mentorship model to facilitate site performance in a clinical research network

Background

Research site mentorship has a positive impact on study enrollment. The VA Cooperative Studies Program's (CSP) Network of Dedicated Enrollment Sites (NODES) utilized an existing site mentorship model to onboard 13 new expansion sites. We describe the successes, challenges, and lessons learned during the development and implementation of this model in this paper.

Methods

NODES established a "Site Mentorship/Expansion Workgroup (SWG)" in October 2020 to plan and guide the consortium on providing mentorship and services to other clinical research networks, non-Node CSP study sites, and NODES expansion sites. In 2021, the SWG developed a 12-month implementation plan to onboard 13 new sites by pairing original Node (mentor) sites with expansion Node (mentee) sites. Mentors offered prompt guidance and solutions to mentees on site-level challenges by working with them closely. Implementation of the plan occurred from February 2022 through September 2023.

Results

Data from the implementation of this mentorship plan demonstrated a 32.7 % increase (from 54.8 % in 2022 to 87.5 % in 2023) in the expansion sites' achievement of their Objectives & Key Results (OKRs). From October 2020-September 2021, prior to mentorship assignments, the original sites (mentors) achieved an average of 88 % of their OKRs and attained an average of 86.7 % and 80.9 % of those OKRs in October-September of 2022 and 2023 respectively during the mentorship implementation phase.

Conclusions

The results demonstrate that developing and implementing a research site mentorship model to facilitate onboarding and performance of research sites into an established network was feasible and contributed to the success of those sites.

The importance of methods for site performance evaluation in REPRIEVE, a longitudinal, global, multicenter trial

BACKGROUND

REPRIEVE, the Randomized Trial to Prevent Vascular Events in HIV, is a multicenter, primary prevention trial evaluating whether a statin can prevent major cardiovascular events in people with HIV. REPRIEVE is conducted at >100 clinical research sites (CRSs) globally. Detailed, comprehensive, and novel methods for evaluating and communicating CRS performance are required to ensure trial integrity and data quality. In this analysis we describe a comprehensive multidimensional methodology for evaluating CRS performance.

METHODS

The REPRIEVE Data Coordinating and Clinical Coordinating Centers developed a robust system for evaluation of and communication with CRSs, designed to identify potential issues and obstacles to performance, provide real-time technical support, and make recommendations for process improvements to facilitate efficient trial execution. We describe these systems and evaluate their impact on participant retention, data management, and specimen management from 2019 to 2022, corresponding to the period from end of recruitment to present. This evaluation was based on pre-defined metrics, regular reviews, and bidirectional communication.

RESULTS

Participant retention, data management, and specimen management all remained steady over the three-year period, although metrics varied by country of enrollment. Targeted messaging relating to certain performance metrics was effective.

CONCLUSION

Site performance is vital to ensure trial integrity and achievement of key trial goals. This analysis demonstrates that utilization of a comprehensive approach allows for a thorough evaluation of CRS performance, facilitates data and specimen management, and enhances participant retention. Our approach may serve as a guidepost for maximizing future large-scale clinical trials' operational success and scientific rigor.

CLINICALTRIALS

gov Identifier: NCT02344290.

Does Central Statistical Monitoring Improve Data Quality? An Analysis of 1,111 Sites in 159 Clinical Trials

BACKGROUND

Central monitoring aims at improving the quality of clinical research by pro-actively identifying risks and remediating emerging issues in the conduct of a clinical trial that may have an adverse impact on patient safety and/or the reliability of trial results. This paper, focusing on statistical data monitoring (SDM), is the second of a series that attempts to quantify the impact of central monitoring in clinical trials.

MATERIAL AND METHODS

Quality improvement was assessed in studies using SDM from a single large central monitoring platform. The analysis focused on a total of 1111 sites that were identified as at-risk by the SDM tests and for which the study teams conducted a follow-up investigation. These sites were taken from 159 studies conducted by 23 different clinical development organizations (including both sponsor companies and contract research organizations). Two quality improvement metrics were assessed for each selected site, one based on a site data inconsistency score (DIS, overall -log10 P-value of the site compared with all other sites) and the other based on the observed metric value associated with each risk signal.

RESULTS

The SDM quality metrics showed improvement in 83% (95% CI, 80-85%) of the sites across therapeutic areas and study phases (primarily phases 2 and 3). In contrast, only 56% (95% CI, 41-70%) of sites showed improvement in 2 historical studies that did not use SDM during study conduct.

CONCLUSION

The results of this analysis provide clear quantitative evidence supporting the hypothesis that the use of SDM in central monitoring is leading to improved quality in clinical trial conduct and associated data across participating sites.

Development of the clinical trial site performance metrics instrument: A study protocol

Clinical trials (CTs) are essential for medical advancements, yet their increasing complexity and cost demand improved efficiency in trial management. One major challenge in multicenter studies is the inconsistency in evaluating site performance. This study aims to develop and validate a Clinical Trial Site Performance Measure (CT-SPM) to assess "good performance" across trials. The tool will be tested and refined through psychometric analysis, resulting in both a comprehensive scale and a short form for universal application. The study is conducted in three phases: Phase 1 involves metric selection through expert consensus; Phase 2 focuses on psychometric testing to evaluate the reliability and validity of the instrument; and Phase 3 defines a cut-off for "good performance" using statistical models. This protocol aims to standardize site performance evaluation, potentially reducing research costs and enhancing trial quality.•The study develops and validates a Clinical Trial Site Performance Measure (CT-SPM) using expert consensus and psychometric testing.•A comprehensive and short-form tool will be created to evaluate site performance in multicenter clinical trials.•A cut-off for "good performance" will be established using statistical models, facilitating consistent and efficient site evaluations.

Enhancing construction site performance through technology and management practices as material waste mitigation in the Malaysian construction industry

The construction industry, increasingly prioritizing sustainability, necessitates an exploration of technology and management's role in mitigating material waste at construction sites. This study examines the impact of 3R, IBS, BIM, and MMA in enhancing Construction Site Performance (CSP) in the Malaysian construction sector. Seven hypotheses were formulated to assess the relationship between technology adoption, material management practices, and the moderating influence of Material Management Adoption (MMA) on CSP. Data were collected through an online survey from 295 valid responses in the Malaysian construction sector, focusing on professionals involved in solid waste management. Utilizing Partial Least Squares - Structural Equation Modeling (PLS-SEM) and Statistical Package for the Social Sciences (SPSS), the findings highlight the importance of technological integration, efficient material management, and competitive strategies in effective material waste mitigation. Furthermore, the qualitative aspect of the study, conducted among 6 solid waste organizations in Malaysia, enriches the findings by providing nuanced insights into local practices and challenges. Emphasizing the importance of contextual insights, the study addresses professionals involved in solid waste management within the Malaysian construction industry. The geographical specificity adds depth to the analysis, offering a comprehensive understanding of regional dynamics. Despite acknowledging limitations in technology and material usage, the study offers recommendations for refining waste mitigation and improving construction site performance. This research model offers actionable insights for construction site stakeholders, emphasizing the criticality of waste mitigation and CSP. The results, both quantitative and qualitative, underscore the potential of these practices within the Malaysian construction industry to foster innovation and drive positive change.