Assessing data quality and the variability of source data verification auditing methods in clinical research settings

Assessing the impact of risk-based data monitoring on outcomes for a paediatric multicentre randomised controlled trial

BACKGROUND/AIMS

Regulatory guidelines recommend that sponsors develop a risk-based approach to monitoring clinical trials. However, there is a lack of evidence to guide the effective implementation of monitoring activities encompassed in this approach. The aim of this study was to assess the efficiency and impact of the risk-based monitoring approach used for a multicentre randomised controlled trial comparing treatments in paediatric patients undergoing cardiac bypass surgery.

METHODS

This is a secondary analysis of data from a randomised controlled trial that implemented targeted source data verification as part of the risk-based monitoring approach. Monitoring duration and source to database error rates were calculated across the monitored trial dataset. The monitored and unmonitored trial dataset, and simulated trial datasets with differing degrees of source data verification and cohort sizes were compared for their effect on trial outcomes.

RESULTS

In total, 106,749 critical data points across 1,282 participants were verified from source data either remotely or on-site during the trial. The total time spent monitoring was 365 hours, with a median (interquartile range) of 10 (7, 16) minutes per participant. An overall source to database error rate of 3.1% was found, and this did not differ between treatment groups. A low rate of error was found for all outcomes undergoing 100% source data verification, with the exception of two secondary outcomes with error rates >10%. Minimal variation in trial outcomes were found between the unmonitored and monitored datasets. Reduced degrees of source data verification and reduced cohort sizes assessed using simulated trial datasets had minimal impact on trial outcomes.

CONCLUSIONS

Targeted source data verification of data critical to trial outcomes, which carried with it a substantial time investment, did not have an impact on study outcomes in this trial. This evaluation of the cost-effectiveness of targeted source data verification contributes to the evidence-base regarding the context where reduced emphasis should be placed on source data verification as the foremost monitoring activity.

Web scraping of user-simulated online nutrition information for people with multiple sclerosis

BACKGROUND

People diagnosed with multiple sclerosis (MS) often seek to modify their diet guided by online advice, however this advice may not align with national dietary guidelines. The aim of this study was to simulate an online search for dietary advice conducted by a person with MS and evaluate the content. It was hypothesised that a variety of eating patterns are promoted for MS online and these dietary approaches can be contradictory.

METHODS

An online search was simulated using Google Trends-informed search terms and Google and Bing search engines. URLs were extracted using R. Nutrition data were extracted including recommendations for diets, foods, supplements, and health professional consultation. Statistical analyses were conducted using R.

RESULTS

73 URLs from 49 websites were extracted, with only 14 results common to both search engines. Dietary recommendations included overall eating patterns (58 webpages, 79%), individual foods (55 webpages, 75%), and supplements (33 webpages, 45%). The most promoted eating pattern for MS was a balanced diet (33 recommendations, 48%), more likely by nonprofit organisations and health information websites (14 and 17 recommendations, 100% and 89%); lifestyle program websites were more likely to recommend restrictive diets (19 recommendations, 100%) (p<0.001). 52% pages advised consulting a health professional, most often a doctor or dietitian.

CONCLUSION

A balanced diet is the most recommended eating pattern for MS online, though advice promoting restrictive diets persists.

Data quality assessment of the Enhanced Gonococcal Antimicrobial Surveillance Programme (EGASP), Thailand, 2015-2021

BACKGROUND

Quality assessments of gonococcal surveillance data are critical to improve data validity and to enhance the value of surveillance findings. Detecting data errors by systematic audits identifies areas for quality improvement. We designed and implemented an internal audit process to evaluate the accuracy and completeness of surveillance data for the Thailand Enhanced Gonococcal Antimicrobial Surveillance Programme (EGASP).

METHODS

We conducted a data quality audit of source records by comparison with the data stored in the EGASP database for five audit cycles from 2015-2021. Ten percent of culture-confirmed cases of Neisseria gonorrhoeae were randomly sampled along with any cases identified with elevated antimicrobial susceptibility testing results and cases with repeat infections. Incorrect and incomplete data were investigated, and corrective action and preventive actions (CAPA) were implemented. Accuracy was defined as the percentage of identical data in both the source records and the database. Completeness was defined as the percentage of non-missing data from either the source document or the database. Statistical analyses were performed using the t-test and the Fisher's exact test.

RESULTS

We sampled and reviewed 70, 162, 85, 68, and 46 EGASP records during the five audit cycles. Overall accuracy and completeness in the five audit cycles ranged from 93.6% to 99.4% and 95.0% to 99.9%, respectively. Overall, completeness was significantly higher than accuracy (p = 0.017). For each laboratory and clinical data element, concordance was >85% in all audit cycles except for two laboratory data elements in two audit cycles. These elements significantly improved following identification and CAPA implementation.

DISCUSSION

We found a high level of data accuracy and completeness in the five audit cycles. The implementation of the audit process identified areas for improvement. Systematic quality assessments of laboratory and clinical data ensure high quality EGASP surveillance data to monitor for antimicrobial resistant Neisseria gonorrhoeae in Thailand.

Data Quality in Health Research: Integrative Literature Review

BACKGROUND

Decision-making and strategies to improve service delivery must be supported by reliable health data to generate consistent evidence on health status. The data quality management process must ensure the reliability of collected data. Consequently, various methodologies to improve the quality of services are applied in the health field. At the same time, scientific research is constantly evolving to improve data quality through better reproducibility and empowerment of researchers and offers patient groups tools for secured data sharing and privacy compliance.

OBJECTIVE

Through an integrative literature review, the aim of this work was to identify and evaluate digital health technology interventions designed to support the conducting of health research based on data quality.

METHODS

A search was conducted in 6 electronic scientific databases in January 2022: PubMed, SCOPUS, Web of Science, Institute of Electrical and Electronics Engineers Digital Library, Cumulative Index of Nursing and Allied Health Literature, and Latin American and Caribbean Health Sciences Literature. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses checklist and flowchart were used to visualize the search strategy results in the databases.

RESULTS

After analyzing and extracting the outcomes of interest, 33 papers were included in the review. The studies covered the period of 2017-2021 and were conducted in 22 countries. Key findings revealed variability and a lack of consensus in assessing data quality domains and metrics. Data quality factors included the research environment, application time, and development steps. Strategies for improving data quality involved using business intelligence models, statistical analyses, data mining techniques, and qualitative approaches.

CONCLUSIONS

The main barriers to health data quality are technical, motivational, economical, political, legal, ethical, organizational, human resources, and methodological. The data quality process and techniques, from precollection to gathering, postcollection, and analysis, are critical for the final result of a study or the quality of processes and decision-making in a health care organization. The findings highlight the need for standardized practices and collaborative efforts to enhance data quality in health research. Finally, context guides decisions regarding data quality strategies and techniques.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.1101/2022.05.31.22275804.

Investigating and preventing scientific misconduct using Benford's Law

Integrity and trust in that integrity are fundamental to academic research. However, procedures for monitoring the trustworthiness of research, and for investigating cases where concern about possible data fraud have been raised are not well established. Here we suggest a practical approach for the investigation of work suspected of fraudulent data manipulation using Benford's Law. This should be of value to both individual peer-reviewers and academic institutions and journals. In this, we draw inspiration from well-established practices of financial auditing. We provide synthesis of the literature on tests of adherence to Benford's Law, culminating in advice of a single initial test for digits in each position of numerical strings within a dataset. We also recommend further tests which may prove useful in the event that specific hypotheses regarding the nature of data manipulation can be justified. Importantly, our advice differs from the most common current implementations of tests of Benford's Law. Furthermore, we apply the approach to previously-published data, highlighting the efficacy of these tests in detecting known irregularities. Finally, we discuss the results of these tests, with reference to their strengths and limitations.

Nursing Care in Peripheral Intravenous Catheter (PIVC): Protocol of a Best Practice Implementation Project

Background: The use of a peripheral intravenous catheters (PIVC) is a common invasive practice in healthcare settings. It is estimated that about 70% of people with PIVCs will develop associated complications, such as infections. It is the consensus that best practices could reduce the appearance of such complications and reduce the length of stay in hospital. Methods: A project will be applied to implement the best approach in peripheral venous catheterization, provided by clinical nurses from an inland hospital in Portugal. The Joanna Briggs Institute methodology will be used on evidence implementation projects, which will be developed in three phases. First, a baseline audit will be performed. The second phase implements corrective measures, and the third phase is a follow-up audit. Conclusions: This project will improve the practice of the nursing team on peripheral venous catheterization nursing cares, positively influencing the quality of nursing care and patient safety. The implementation and dissemination of this project could boost its replication in other centres.

Research monitoring practices in critical care research: a survey of current state and attitudes

Background/Aims

In 2016, international standards governing clinical research recommended that the approach to monitoring a research project should be undertaken based on risk, however it is unknown whether this approach has been adopted in Australia and New Zealand (ANZ) throughout critical care research. The aims of the project were to: 1) Gain an understanding of current research monitoring practices in academic-led clinical trials in the field of critical care research, 2) Describe the perceived barriers and enablers to undertaking research monitoring.

Methods

Electronic survey distributed to investigators, research co-ordinators and other research staff currently undertaking and supporting academic-led clinical trials in the field of critical care in ANZ.

Results

Of the 118 respondents, 70 were involved in the co-ordination of academic trials; the remaining results pertain to this sub-sample. Fifty-eight (83%) were working in research units associated with hospitals, 29 (41%) were experienced Research Coordinators and 19 (27%) Principal Investigators; 31 (44%) were primarily associated with paediatric research. Fifty-six (80%) develop monitoring plans with 33 (59%) of these undertaking a risk assessment; the most common barrier reported was lack of expertise. Nineteen (27%) indicated that centralised monitoring was used, noting that technology to support centralised monitoring (45/51; 88%) along with support from data managers and statisticians (45/52; 87%) were key enablers. Coronavirus disease-19 (COVID-19) impacted monitoring for 82% (45/55) by increasing remote (25/45; 56%) and reducing onsite (29/45; 64%) monitoring.

Conclusions

Contrary to Good Clinical Practice guidance, risk assessments to inform monitoring plans are not being consistently performed due to lack of experience and guidance. There is an urgent need to enhance risk assessment methodologies and develop technological solutions for centralised statistical monitoring.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-022-01551-7.

Accuracy of Simulated Research Tasks by Community Hospitals Participating in a Multicenter Telemedicine Trial

Background/Aims: Clinical trials evaluating facility-to-facility telemedicine may include sites that have limited research experience. For the trial to be successful, these sites must correctly perform research-related tasks. This study aimed to determine whether health care professionals at community hospitals could accurately identify simulated study eligible patients and submit data to a research coordinating center. Methods: Twenty-seven community hospitals in the United States and Canada participated in this study. An electronic survey was sent to one designated health care professional at each site. The survey included a description of trial eligibility criteria and five written neonatal resuscitation scenarios. For each scenario, the participant determined whether the neonate was study eligible. One scenario required participants to submit 14 data elements to the coordinating center. Accuracy of study eligibility and data submission was summarized using standard descriptive statistics. Results: The survey response rate was 100% (27/27). Overall accuracy in determining study eligibility was 89% (120/135), and accuracy varied across the five scenarios (range 82-93%). Overall accuracy of data submission was 92% (310/336). Data were >95% accurate for 9 of the 14 data elements, with 100% accuracy achieved for 6 data elements. These results were used to clarify eligibility criteria, inform database design, and improve training materials for the subsequent clinical trial. Conclusions: Health care professionals at community hospitals accurately determined trial eligibility and submitted study data based on written clinical scenarios. Research teams conducting telemedicine trials with community hospitals should consider completing pre-trial simulation activities to identify opportunities for improving trial processes and materials.

Developing a systematic approach to assessing data quality in secondary use of clinical data based on intended use

INTRODUCTION

Secondary use of electronic health record (EHR) data for research requires that the data are fit for use. Data quality (DQ) frameworks have traditionally focused on structural conformance and completeness of clinical data extracted from source systems. In this paper, we propose a framework for evaluating semantic DQ that will allow researchers to evaluate fitness for use prior to analyses.

METHODS

We reviewed current DQ literature, as well as experience from recent multisite network studies, and identified gaps in the literature and current practice. Derived principles were used to construct the conceptual framework with attention to both analytic fitness and informatics practice.

RESULTS

We developed a systematic framework that guides researchers in assessing whether a data source is fit for use for their intended study or project. It combines tools for evaluating clinical context with DQ principles, as well as factoring in the characteristics of the data source, in order to develop semantic DQ checks.

CONCLUSIONS

Our framework provides a systematic process for DQ development. Further work is needed to codify practices and metadata around both structural and semantic data quality.

Data Quality of Longitudinally Collected Patient-Reported Outcomes After Thoracic Surgery: Comparison of Paper- and Web-Based Assessments

Background

High-frequency patient-reported outcome (PRO) assessments are used to measure patients' symptoms after surgery for surgical research; however, the quality of those longitudinal PRO data has seldom been discussed.

Objective

The aim of this study was to determine data quality-influencing factors and to profile error trajectories of data longitudinally collected via paper-and-pencil (P&P) or web-based assessment (electronic PRO [ePRO]) after thoracic surgery.

Methods

We extracted longitudinal PRO data with 678 patients scheduled for lung surgery from an observational study (n=512) and a randomized clinical trial (n=166) on the evaluation of different perioperative care strategies. PROs were assessed by the MD Anderson Symptom Inventory Lung Cancer Module and single-item Quality of Life Scale before surgery and then daily after surgery until discharge or up to 14 days of hospitalization. Patient compliance and data error were identified and compared between P&P and ePRO. Generalized estimating equations model and 2-piecewise model were used to describe trajectories of error incidence over time and to identify the risk factors.

Results

Among 678 patients, 629 with at least 2 PRO assessments, 440 completed 3347 P&P assessments and 189 completed 1291 ePRO assessments. In total, 49.4% of patients had at least one error, including (1) missing items (64.69%, 1070/1654), (2) modifications without signatures (27.99%, 463/1654), (3) selection of multiple options (3.02%, 50/1654), (4) missing patient signatures (2.54%, 42/1654), (5) missing researcher signatures (1.45%, 24/1654), and (6) missing completion dates (0.30%, 5/1654). Patients who completed ePRO had fewer errors than those who completed P&P assessments (ePRO: 30.2% [57/189] vs. P&P: 57.7% [254/440]; P<.001). Compared with ePRO patients, those using P&P were older, less educated, and sicker. Common risk factors of having errors were a lower education level (P&P: odds ratio [OR] 1.39, 95% CI 1.20-1.62; P<.001; ePRO: OR 1.82, 95% CI 1.22-2.72; P=.003), treated in a provincial hospital (P&P: OR 3.34, 95% CI 2.10-5.33; P<.001; ePRO: OR 4.73, 95% CI 2.18-10.25; P<.001), and with severe disease (P&P: OR 1.63, 95% CI 1.33-1.99; P<.001; ePRO: OR 2.70, 95% CI 1.53-4.75; P<.001). Errors peaked on postoperative day (POD) 1 for P&P, and on POD 2 for ePRO.

Conclusions

It is possible to improve data quality of longitudinally collected PRO through ePRO, compared with P&P. However, ePRO-related sampling bias needs to be considered when designing clinical research using longitudinal PROs as major outcomes.

Identifying relapses and stem cell transplants in pediatric acute lymphoblastic leukemia using administrative data: Capturing national outcomes irrespective of trial enrollment

INTRODUCTION

Our objectives were to design and validate methods to identify relapse and hematopoietic stem cell transplantation (HSCT) in children with acute lymphoblastic leukemia (ALL) using administrative data representing hospitalizations at US pediatric institutions.

METHODS

We developed daily billing and ICD-9 code definitions to identify relapses and HSCTs within a cohort of children with newly diagnosed ALL between January 1, 2004, and December 31, 2013, previously assembled from the Pediatric Health Information System (PHIS) database. Chart review for children with ALL at the Children's Hospital of Philadelphia (CHOP) and Texas Children's Hospital (TCH) was performed to establish relapse and HSCT gold standards for sensitivity and positive predictive value (PPV) calculations. We estimated incidences of relapse and HSCT in the PHIS ALL cohort.

RESULTS

We identified 362 CHOP and 314 TCH ALL patients in PHIS and established true positives by chart review. Sensitivity and PPV for identifying both relapse and HSCT in PHIS were > 90% at both hospitals. Five-year relapse incidence in the 10 150-patient PHIS cohort was 10.3% (95% CI 9.8%-10.9%) with 7.1% (6.6%-7.6%) of children underwent HSCTs. Patients in higher-risk demographic groups had higher relapse and HSCT rates. Our analysis also identified differences in incidences of relapse and HSCT by race, ethnicity, and insurance status.

CONCLUSIONS

Administrative data can be used to identify relapse and HSCT accurately in children with ALL whether they occur on- or off-therapy, in contrast with published approaches. This method has wide potential applicability for estimating these incidences in pediatric ALL, including patients not enrolled on clinical trials.

Developing a Small Baby Program for the Extremely Low Birth Weight: The Wee CARE Team

The Children's Hospital at Providence (TCHaP) is a hospital within a hospital, in the heart of Alaska's biggest city, Anchorage. TCHaP admits up to 60 extremely low birth weight (ELBW) neonates per year. The ELBW population, although small in number, contributes disproportionately to rates of death or serious morbidities. Nationally, ELBW is defined as a neonate born at a gestational age between 22 and 29 weeks. In 2014, only 38 percent of neonates born in Alaska <28 weeks survived without experiencing major morbidities. For those born <26 weeks, morbidity-free survival dropped to 25 percent. Discussions were held among NICU nursing leaders, clinical nurses, and physicians about current co-morbidities and potentially best practices to improve outcomes. Subsequently, the group decided to develop best practices for managing the care of the ELBW, which started by organizing a group of specialists. This group at TCHaP is called the Wee CARE team.

Effectiveness of data auditing as a tool to reinforce good research data management (RDM) practice: a Singapore study

Background

Institutions, funding agencies and publishers are placing increasing emphasis on good research data management (RDM). RDM lapses in medical science can result in questionable data and cause the public’s confidence in the scientific community to crumble. A fledgling medical school in a young university in Singapore has mandated every funded research project to have a data management plan (DMP). However, researchers’ adherence to their DMPs was unknown until the school embarked on routine data auditing. We hypothesize that research data auditing improves RDM awareness, compliance and reception in the school.

Methods

We conducted surveys with research PIs and researchers before and after data auditing to evaluate differences in self-reported RDM awareness, compliance and reception. As it is mandatory to deposit research data in a central data repository system in the school, we tracked data deposition by each laboratory from 2 weeks before to 3 months after data auditing as a marker of actual RDM compliance.

Results

Research data auditing had an overall positive effect on self-reported RDM awareness, compliance and reception for both research PIs and researchers. Research PIs agreed more that RDM was important to scientific reproducibility, were more aware of proper RDM, had higher RDM strength in their laboratories and were more compliant with the DMP. Both research PIs and researchers believed data auditing helped them to be more compliant with data deposition in the repository. However, data auditing had no significant impact on laboratories’ data deposition rates over time, which could be due to the short sampling period.

Conclusions

Research PIs and researchers generally felt that data auditing was effective in improving RDM practices. It helped to evaluate their RDM practices objectively, propose corrective actions for RDM lapses and spread awareness of the university’s data management policies. Our findings corroborated other studies in medical research, geosciences, engineering and ethics that data auditing promotes good RDM practices. Hence, we recommend research institutions worldwide to adopt data auditing as a tool to reinforce research integrity.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12910-021-00662-y.

Implementing an automated monitoring process in a digital, longitudinal observational cohort study

Background

Clinical data collection requires correct and complete data sets in order to perform correct statistical analysis and draw valid conclusions. While in randomized clinical trials much effort concentrates on data monitoring, this is rarely the case in observational studies- due to high numbers of cases and often-restricted resources. We have developed a valid and cost-effective monitoring tool, which can substantially contribute to an increased data quality in observational research.

Methods

An automated digital monitoring system for cohort studies developed by the German Rheumatism Research Centre (DRFZ) was tested within the disease register RABBIT-SpA, a longitudinal observational study including patients with axial spondyloarthritis and psoriatic arthritis. Physicians and patients complete electronic case report forms (eCRF) twice a year for up to 10 years. Automatic plausibility checks were implemented to verify all data after entry into the eCRF. To identify conflicts that cannot be found by this approach, all possible conflicts were compiled into a catalog. This “conflict catalog” was used to create queries, which are displayed as part of the eCRF. The proportion of queried eCRFs and responses were analyzed by descriptive methods. For the analysis of responses, the type of conflict was assigned to either a single conflict only (affecting individual items) or a conflict that required the entire eCRF to be queried.

Results

Data from 1883 patients was analyzed. A total of n = 3145 eCRFs submitted between baseline (T0) and T3 (12 months) had conflicts (40–64%). Fifty-six to 100% of the queries regarding eCRFs that were completely missing were answered. A mean of 1.4 to 2.4 single conflicts occurred per eCRF, of which 59–69% were answered. The most common missing values were CRP, ESR, Schober’s test, data on systemic glucocorticoid therapy, and presence of enthesitis.

Conclusion

Providing high data quality in large observational cohort studies is a major challenge, which requires careful monitoring. An automated monitoring process was successfully implemented and well accepted by the study centers. Two thirds of the queries were answered with new data. While conventional manual monitoring is resource-intensive and may itself create new sources of errors, automated processes are a convenient way to augment data quality.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13075-021-02563-2.

Anomaly Detection Algorithm for Real-World Data and Evidence in Clinical Research: Implementation, Evaluation, and Validation Study

BACKGROUND

Statistical analysis, which has become an integral part of evidence-based medicine, relies heavily on data quality that is of critical importance in modern clinical research. Input data are not only at risk of being falsified or fabricated, but also at risk of being mishandled by investigators.

OBJECTIVE

The urgent need to assure the highest data quality possible has led to the implementation of various auditing strategies designed to monitor clinical trials and detect errors of different origin that frequently occur in the field. The objective of this study was to describe a machine learning-based algorithm to detect anomalous patterns in data created as a consequence of carelessness, systematic error, or intentionally by entering fabricated values.

METHODS

A particular electronic data capture (EDC) system, which is used for data management in clinical registries, is presented including its architecture and data structure. This EDC system features an algorithm based on machine learning designed to detect anomalous patterns in quantitative data. The detection algorithm combines clustering with a series of 7 distance metrics that serve to determine the strength of an anomaly. For the detection process, the thresholds and combinations of the metrics were used and the detection performance was evaluated and validated in the experiments involving simulated anomalous data and real-world data.

RESULTS

Five different clinical registries related to neuroscience were presented-all of them running in the given EDC system. Two of the registries were selected for the evaluation experiments and served also to validate the detection performance on an independent data set. The best performing combination of the distance metrics was that of Canberra, Manhattan, and Mahalanobis, whereas Cosine and Chebyshev metrics had been excluded from further analysis due to the lowest performance when used as single distance metric-based classifiers.

CONCLUSIONS

The experimental results demonstrate that the algorithm is universal in nature, and as such may be implemented in other EDC systems, and is capable of anomalous data detection with a sensitivity exceeding 85%.

Verification of data in a nationwide transanal total mesorectal excision registry in China

BACKGROUND AND OBJECTIVES

Transanal total mesorectal excision is a surgical procedure for mid- and low rectal cancer. The Chinese TaTME Registry Collaborative is a nationwide database collecting information on patients who have undergone this procedure.

METHODS

Centers were invited by the registry committee to participate in a three-part data audit project: remote audits for data completeness and deviation values, onsite source verification of data accuracy, and an online survey of the characteristics of data managers.

RESULTS

Twenty-three tertiary centers participated in this project. The median case volume registered by the centers was 51 (interquartile range, 25-89). The overall data completeness for 30 verified variables was 89.1%. Eight centers achieved a high data completeness rate (>95%). The source data of eight centers were verified onsite. The overall accuracy rate was 90.4% (85.3%-97.6% across centers). Postoperative complications, mortality, and proximal/distal resection margin involvement were accurately reported in >95% of cases. The data completeness rate was higher if the data manager was a surgeon/surgical resident (94.2% vs. 84.8%, p = 0.045).

CONCLUSIONS

The completeness and accuracy of the data in the Chinese TaTME Registry Collaborative are acceptable. The quality of the data is highest when entered by colorectal surgeons and residents.

Time-consuming and expensive data quality monitoring procedures persist in clinical trials: A national survey

INTRODUCTION

The Good Clinical Practice guideline identifies that data monitoring is an essential research activity. However, limited evidence exists on how to perform monitoring including the amount or frequency that is needed to ensure data quality. This study aims to explore the monitoring procedures that are implemented to ensure data quality in Australian clinical research studies.

MATERIAL AND METHODS

Clinical studies listed on the Australian and New Zealand Clinical Trials Registry were invited to participate in a national cross-sectional, mixed-mode, multi-contact (postal letter and e-mail) web-based survey. Information was gathered about the types of data quality monitoring procedures being implemented.

RESULTS

Of the 3689 clinical studies contacted, 589 (16.0%) responded, of which 441 (77.4%) completed the survey. Over half (55%) of the studies applied source data verification (SDV) compared to risk-based targeted and triggered monitoring (10-11%). Conducting 100% on-site monitoring was most common for those who implemented the traditional approach. Respondents who did not conduct 100% monitoring, included 1-25% of data points for SDV, centralized or on-site monitoring. The incidence of adverse events and protocol deviations were the most likely factors to trigger a site visit for risk-based triggered (63% and 44%) and centralized monitoring (48% and 44%), respectively.

CONCLUSION

Instead of using more optimal risk-based approaches, small single-site clinical studies are conducting traditional monitoring procedures which are time consuming and expensive. Formal guidelines need to be improved and provided to all researchers for 'new' risk-based monitoring approaches.

Self-audits as alternatives to travel-audits for improving data quality in the Caribbean, Central and South America network for HIV epidemiology

INTRODUCTION

Audits play a critical role in maintaining the integrity of observational cohort data. While previous work has validated the audit process, sending trained auditors to sites ("travel-audits") can be costly. We investigate the efficacy of training sites to conduct "self-audits."

METHODS

In 2017, eight research groups in the Caribbean, Central, and South America network for HIV Epidemiology each audited a subset of their patient records randomly selected by the data coordinating center at Vanderbilt. Designated investigators at each site compared abstracted research data to the original clinical source documents and captured audit findings electronically. Additionally, two Vanderbilt investigators performed on-site travel-audits at three randomly selected sites (one adult and two pediatric) in late summer 2017.

RESULTS

Self- and travel-auditors, respectively, reported that 93% and 92% of 8919 data entries, captured across 28 unique clinical variables on 65 patients, were entered correctly. Across all entries, 8409 (94%) received the same assessment from self- and travel-auditors (7988 correct and 421 incorrect). Of 421 entries mutually assessed as "incorrect," 304 (82%) were corrected by both self- and travel-auditors and 250 of these (72%) received the same corrections. Reason for changing antiretroviral therapy (ART) regimen, ART end date, viral load value, CD4%, and HIV diagnosis date had the most mismatched corrections.

CONCLUSIONS

With similar overall error rates, findings suggest that data audits conducted by trained local investigators could provide an alternative to on-site audits by external auditors to ensure continued data quality. However, discrepancies observed between corrections illustrate challenges in determining correct values even with audits.

The impact of data quality and source data verification on epidemiologic inference: a practical application using HIV observational data

BACKGROUND

Data audits are often evaluated soon after completion, even though the identification of systematic issues may lead to additional data quality improvements in the future. In this study, we assess the impact of the entire data audit process on subsequent statistical analyses.

METHODS

We conducted on-site audits of datasets from nine international HIV care sites. Error rates were quantified for key demographic and clinical variables among a subset of records randomly selected for auditing. Based on audit results, some sites were tasked with targeted validation of high-error-rate variables resulting in a post-audit dataset. We estimated the times from antiretroviral therapy initiation until death and first AIDS-defining event using the pre-audit data, the audit data, and the post-audit data.

RESULTS

The overall discrepancy rate between pre-audit and audit data (n = 250) across all audited variables was 17.1%. The estimated probability of mortality and an AIDS-defining event over time was higher in the audited data relative to the pre-audit data. Among patients represented in both the post-audit and pre-audit cohorts (n = 18,999), AIDS and mortality estimates also were higher in the post-audit data.

CONCLUSION

Though some changes may have occurred independently, our findings suggest that improved data quality following the audit may impact epidemiological inferences.

Impact of a targeted monitoring on data-quality and data-management workload of randomized controlled trials: A prospective comparative study

AIMS

Monitoring risk-based approaches in clinical trials are encouraged by regulatory guidance. However, the impact of a targeted source data verification (SDV) on data-management (DM) workload and on final data quality needs to be addressed.

METHODS

MONITORING was a prospective study aiming at comparing full SDV (100% of data verified for all patients) and targeted SDV (only key data verified for all patients) followed by the same DM program (detecting missing data and checking consistency) on final data quality, global workload and staffing costs.

RESULTS

In all, 137 008 data including 18 124 key data were collected for 126 patients from 6 clinical trials. Compared to the final database obtained using the full SDV monitoring process, the final database obtained using the targeted SDV monitoring process had a residual error rate of 1.47% (95% confidence interval, 1.41-1.53%) on overall data and 0.78% (95% confidence interval, 0.65-0.91%) on key data. There were nearly 4 times more queries per study with targeted SDV than with full SDV (mean ± standard deviation: 132 ± 101 vs 34 ± 26; P = .03). For a handling time of 15 minutes per query, the global workload of the targeted SDV monitoring strategy remained below that of the full SDV monitoring strategy. From 25 minutes per query it was above, increasing progressively to represent a 50% increase for 45 minutes per query.

CONCLUSION

Targeted SDV monitoring is accompanied by increased workload for DM, which allows to obtain a small proportion of remaining errors on key data (<1%), but may substantially increase trial costs.