Measuring the quality of observational study data in an international HIV research network

Lessons learned from over a decade of data audits in international observational HIV cohorts in Latin America and East Africa

Introduction

Routine patient care data are increasingly used for biomedical research, but such "secondary use" data have known limitations, including their quality. When leveraging routine care data for observational research, developing audit protocols that can maximize informational return and minimize costs is paramount.

Methods

For more than a decade, the Latin America and East Africa regions of the International epidemiology Databases to Evaluate AIDS (IeDEA) consortium have been auditing the observational data drawn from participating human immunodeficiency virus clinics. Since our earliest audits, where external auditors used paper forms to record audit findings from paper medical records, we have streamlined our protocols to obtain more efficient and informative audits that keep up with advancing technology while reducing travel obligations and associated costs.

Results

We present five key lessons learned from conducting data audits of secondary-use data from resource-limited settings for more than 10 years and share eight recommendations for other consortia looking to implement data quality initiatives.

Conclusion

After completing multiple audit cycles in both the Latin America and East Africa regions of the IeDEA consortium, we have established a rich reference for data quality in our cohorts, as well as large, audited analytical datasets that can be used to answer important clinical questions with confidence. By sharing our audit processes and how they have been adapted over time, we hope that others can develop protocols informed by our lessons learned from more than a decade of experience in these large, diverse cohorts.

Multiwave validation sampling for error-prone electronic health records

Electronic health record (EHR) data are increasingly used for biomedical research, but these data have recognized data quality challenges. Data validation is necessary to use EHR data with confidence, but limited resources typically make complete data validation impossible. Using EHR data, we illustrate prospective, multiwave, two-phase validation sampling to estimate the association between maternal weight gain during pregnancy and the risks of her child developing obesity or asthma. The optimal validation sampling design depends on the unknown efficient influence functions of regression coefficients of interest. In the first wave of our multiwave validation design, we estimate the influence function using the unvalidated (phase 1) data to determine our validation sample; then in subsequent waves, we re-estimate the influence function using validated (phase 2) data and update our sampling. For efficiency, estimation combines obesity and asthma sampling frames while calibrating sampling weights using generalized raking. We validated 996 of 10,335 mother-child EHR dyads in six sampling waves. Estimated associations between childhood obesity/asthma and maternal weight gain, as well as other covariates, are compared to naïve estimates that only use unvalidated data. In some cases, estimates markedly differ, underscoring the importance of efficient validation sampling to obtain accurate estimates incorporating validated data.

Efficient odds ratio estimation under two-phase sampling using error-prone data from a multi-national HIV research cohort

Persons living with HIV engage in routine clinical care, generating large amounts of data in observational HIV cohorts. These data are often error-prone, and directly using them in biomedical research could bias estimation and give misleading results. A cost-effective solution is the two-phase design, under which the error-prone variables are observed for all patients during Phase I, and that information is used to select patients for data auditing during Phase II. For example, the Caribbean, Central, and South America network for HIV epidemiology (CCASAnet) selected a random sample from each site for data auditing. Herein, we consider efficient odds ratio estimation with partially audited, error-prone data. We propose a semiparametric approach that uses all information from both phases and accommodates a number of error mechanisms. We allow both the outcome and covariates to be error-prone and these errors to be correlated, and selection of the Phase II sample can depend on Phase I data in an arbitrary manner. We devise a computationally efficient, numerically stable EM algorithm to obtain estimators that are consistent, asymptotically normal, and asymptotically efficient. We demonstrate the advantages of the proposed methods over existing ones through extensive simulations. Finally, we provide applications to the CCASAnet cohort.

A narrative review on the validity of electronic health record-based research in epidemiology

Electronic health records (EHRs) are widely used in epidemiological research, but the validity of the results is dependent upon the assumptions made about the healthcare system, the patient, and the provider. In this review, we identify four overarching challenges in using EHR-based data for epidemiological analysis, with a particular emphasis on threats to validity. These challenges include representativeness of the EHR to a target population, the availability and interpretability of clinical and non-clinical data, and missing data at both the variable and observation levels. Each challenge reveals layers of assumptions that the epidemiologist is required to make, from the point of patient entry into the healthcare system, to the provider documenting the results of the clinical exam and follow-up of the patient longitudinally; all with the potential to bias the results of analysis of these data. Understanding the extent of as well as remediating potential biases requires a variety of methodological approaches, from traditional sensitivity analyses and validation studies, to newer techniques such as natural language processing. Beyond methods to address these challenges, it will remain crucial for epidemiologists to engage with clinicians and informaticians at their institutions to ensure data quality and accessibility by forming multidisciplinary teams around specific research projects.

Pregnancies among women living with HIV using contraceptives and antiretroviral therapy in western Kenya: a retrospective, cohort study

BACKGROUND

Preventing unintended pregnancies is paramount for women living with HIV (WLHIV). Previous studies have suggested that efavirenz-containing antiretroviral therapy (ART) reduces contraceptive effectiveness of implants, but there are uncertainties regarding the quality of the electronic medical record (EMR) data used in these prior studies.

METHODS

We conducted a retrospective, cohort study of EMR data from 2011 to 2015 among WLHIV of reproductive age accessing HIV care in public facilities in western Kenya. We validated a large subsample of records with manual chart review and telephone interviews. We estimated adjusted incidence rate ratios (aIRRs) with Poisson regression accounting for the validation sampling using inverse probability weighting and generalized raking.

RESULTS

A total of 85,324 women contributed a total of 170,845 women-years (w-y) of observation time; a subset of 5080 women had their charts reviewed, and 1285 underwent interviews. Among implant users, the aIRR of pregnancy for efavirenz- vs. nevirapine-containing ART was 1.9 (95% CI 1.6, 2.4) using EMR data only and 3.2 (95% CI 1.8, 5.7) when additionally using both chart review and interview validated data. Among efavirenz users, the aIRR of pregnancy for depomedroxyprogesterone acetate (DMPA) vs. implant use was 1.8 (95% CI 1.5, 2.1) in EMR only and 2.4 (95% CI 1.0, 6.1) using validated data.

CONCLUSION

Pregnancy rates are higher when contraceptive implants are concomitantly used with efavirenz-containing ART, though rates were similar to leading alternative contraceptive methods such as DMPA. Our data provides policymakers, program staff, and WLHIV greater confidence in guiding their decision-making around contraceptive and ART options. Our novel, 3-phase validation sampling provides an innovative tool for using routine EMR data to improve the robustness of data quality.

Improved generalized raking estimators to address dependent covariate and failure-time outcome error

Biomedical studies that use electronic health records (EHR) data for inference are often subject to bias due to measurement error. The measurement error present in EHR data is typically complex, consisting of errors of unknown functional form in covariates and the outcome, which can be dependent. To address the bias resulting from such errors, generalized raking has recently been proposed as a robust method that yields consistent estimates without the need to model the error structure. We provide rationale for why these previously proposed raking estimators can be expected to be inefficient in failure-time outcome settings involving misclassification of the event indicator. We propose raking estimators that utilize multiple imputation, to impute either the target variables or auxiliary variables, to improve the efficiency. We also consider outcome-dependent sampling designs and investigate their impact on the efficiency of the raking estimators, either with or without multiple imputation. We present an extensive numerical study to examine the performance of the proposed estimators across various measurement error settings. We then apply the proposed methods to our motivating setting, in which we seek to analyze HIV outcomes in an observational cohort with EHR data from the Vanderbilt Comprehensive Care Clinic.

Improving data quality in observational research studies: Report of the Cure Glomerulonephropathy (CureGN) network

Background

High data quality is of crucial importance to the integrity of research projects. In the conduct of multi-center observational cohort studies with increasing types and quantities of data, maintaining data quality is challenging, with few published guidelines.

Methods

The Cure Glomerulonephropathy (CureGN) Network has established numerous quality control procedures to manage the 70 participating sites in the United States, Canada, and Europe. This effort is supported and guided by the activities of several committees, including Data Quality, Recruitment and Retention, and Central Review, that work in tandem with the Data Coordinating Center to monitor the study. We have implemented coordinator training and feedback channels, data queries of questionable or missing data, and developed performance metrics for recruitment, retention, visit completion, data entry, recording of patient-reported outcomes, collection, shipping and accessing of biological samples and pathology materials, and processing, cataloging and accessing genetic data and materials.

Results

We describe the development of data queries and site Report Cards, and their use in monitoring and encouraging excellence in site performance. We demonstrate improvements in data quality and completeness over 4 years after implementing these activities. We describe quality initiatives addressing specific challenges in collecting and cataloging whole slide images and other kidney pathology data, and novel methods of data quality assessment.

Conclusions

This paper reports the CureGN experience in optimizing data quality and underscores the importance of general and study-specific data quality initiatives to maintain excellence in the research measures of a multi-center observational study.

Efficient semiparametric inference for two-phase studies with outcome and covariate measurement errors

In modern observational studies using electronic health records or other routinely collected data, both the outcome and covariates of interest can be error-prone and their errors often correlated. A cost-effective solution is the two-phase design, under which the error-prone outcome and covariates are observed for all subjects during the first phase and that information is used to select a validation subsample for accurate measurements of these variables in the second phase. Previous research on two-phase measurement error problems largely focused on scenarios where there are errors in covariates only or the validation sample is a simple random sample of study subjects. Herein, we propose a semiparametric approach to general two-phase measurement error problems with a quantitative outcome, allowing for correlated errors in the outcome and covariates and arbitrary second-phase selection. We devise a computationally efficient and numerically stable expectation-maximization algorithm to maximize the nonparametric likelihood function. The resulting estimators possess desired statistical properties. We demonstrate the superiority of the proposed methods over existing approaches through extensive simulation studies, and we illustrate their use in an observational HIV study.

Raking and regression calibration: Methods to address bias from correlated covariate and time-to-event error

Medical studies that depend on electronic health records (EHR) data are often subject to measurement error, as the data are not collected to support research questions under study. These data errors, if not accounted for in study analyses, can obscure or cause spurious associations between patient exposures and disease risk. Methodology to address covariate measurement error has been well developed; however, time-to-event error has also been shown to cause significant bias, but methods to address it are relatively underdeveloped. More generally, it is possible to observe errors in both the covariate and the time-to-event outcome that are correlated. We propose regression calibration (RC) estimators to simultaneously address correlated error in the covariates and the censored event time. Although RC can perform well in many settings with covariate measurement error, it is biased for nonlinear regression models, such as the Cox model. Thus, we additionally propose raking estimators which are consistent estimators of the parameter defined by the population estimating equation. Raking can improve upon RC in certain settings with failure-time data, require no explicit modeling of the error structure, and can be utilized under outcome-dependent sampling designs. We discuss features of the underlying estimation problem that affect the degree of improvement the raking estimator has over the RC approach. Detailed simulation studies are presented to examine the performance of the proposed estimators under varying levels of signal, error, and censoring. The methodology is illustrated on observational EHR data on HIV outcomes from the Vanderbilt Comprehensive Care Clinic.

Errors in multiple variables in human immunodeficiency virus (HIV) cohort and electronic health record data: statistical challenges and opportunities

Objectives: Observational data derived from patient electronic health records (EHR) data are increasingly used for human immunodeficiency virus/acquired immunodeficiency syndrome (HIV/AIDS) research. There are challenges to using these data, in particular with regards to data quality; some are recognized, some unrecognized, and some recognized but ignored. There are great opportunities for the statistical community to improve inference by incorporating validation subsampling into analyses of EHR data.Methods: Methods to address measurement error, misclassification, and missing data are relevant, as are sampling designs such as two-phase sampling. However, many of the existing statistical methods for measurement error, for example, only address relatively simple settings, whereas the errors seen in these datasets span multiple variables (both predictors and outcomes), are correlated, and even affect who is included in the study.Results/Conclusion: We will discuss some preliminary methods in this area with a particular focus on time-to-event outcomes and outline areas of future research.

Frequency of non-communicable diseases in people 50 years of age and older receiving HIV care in Latin America

BACKGROUND

A growing population of older adults with HIV will increase demands on HIV-related healthcare. Nearly a quarter of people receiving care for HIV in Latin America are currently 50 years or older, yet little is known about the frequency of comorbidities in this population. We estimated the prevalence and incidence of non-communicable diseases (NCDs) among people 50 years of age or older (≥50yo) receiving HIV care during 2000-2015 in six centers affiliated with the Caribbean, Central and South American network for HIV epidemiology (CCASAnet).

METHODS

We estimated the annual prevalence, and overall prevalence and incidence of cardiovascular diseases, diabetes, hypertension, dyslipidemia, psychiatric disorders, chronic liver and renal diseases, and non-AIDS-defining cancers, and multimorbidity (more than one NCD) of people ≥50yo receiving care for HIV. Analyses were performed according to age at enrollment into HIV care (<50yo and ≥50yo).

RESULTS

We included 3,415 patients ≥50yo, of whom 1,487(43%) were enrolled at age ≥50 years. The annual prevalence of NCDs increased from 32% to 68% and multimorbidity from 30% to 40% during 2000-2015. At the last registered visit, 53% of patients enrolled <50yo and 50% of those enrolled ≥50yo had at least one NCD. Most common NCDs at the last visit in each age-group at enrollment were dyslipidemia (36% in <50yo and 28% in ≥50yo), hypertension (17% and 18%), psychiatric disorders (15% and 10%), and diabetes (11% and 12%).

CONCLUSIONS

The prevalence of NCDs and multimorbidity in people ≥50 years receiving care for HIV in CCASAnet centers in Latin America increased substantially in the last 15 years. Our results make evident the need of planning for provision of complex, primary care for aging adults living with HIV.

ACCOUNTING FOR DEPENDENT ERRORS IN PREDICTORS AND TIME-TO-EVENT OUTCOMES USING ELECTRONIC HEALTH RECORDS, VALIDATION SAMPLES, AND MULTIPLE IMPUTATION

Data from electronic health records (EHR) are prone to errors, which are often correlated across multiple variables. The error structure is further complicated when analysis variables are derived as functions of two or more error-prone variables. Such errors can substantially impact estimates, yet we are unaware of methods that simultaneously account for errors in covariates and time-to-event outcomes. Using EHR data from 4217 patients, the hazard ratio for an AIDS-defining event associated with a 100 cell/mm3 increase in CD4 count at ART initiation was 0.74 (95%CI: 0.68-0.80) using unvalidated data and 0.60 (95%CI: 0.53-0.68) using fully validated data. Our goal is to obtain unbiased and efficient estimates after validating a random subset of records. We propose fitting discrete failure time models to the validated subsample and then multiply imputing values for unvalidated records. We demonstrate how this approach simultaneously addresses dependent errors in predictors, time-to-event outcomes, and inclusion criteria. Using the fully validated dataset as a gold standard, we compare the mean squared error of our estimates with those from the unvalidated dataset and the corresponding subsample-only dataset for various subsample sizes. By incorporating reasonably sized validated subsamples and appropriate imputation models, our approach had improved estimation over both the naive analysis and the analysis using only the validation subsample.

Self-audits as alternatives to travel-audits for improving data quality in the Caribbean, Central and South America network for HIV epidemiology

INTRODUCTION

Audits play a critical role in maintaining the integrity of observational cohort data. While previous work has validated the audit process, sending trained auditors to sites ("travel-audits") can be costly. We investigate the efficacy of training sites to conduct "self-audits."

METHODS

In 2017, eight research groups in the Caribbean, Central, and South America network for HIV Epidemiology each audited a subset of their patient records randomly selected by the data coordinating center at Vanderbilt. Designated investigators at each site compared abstracted research data to the original clinical source documents and captured audit findings electronically. Additionally, two Vanderbilt investigators performed on-site travel-audits at three randomly selected sites (one adult and two pediatric) in late summer 2017.

RESULTS

Self- and travel-auditors, respectively, reported that 93% and 92% of 8919 data entries, captured across 28 unique clinical variables on 65 patients, were entered correctly. Across all entries, 8409 (94%) received the same assessment from self- and travel-auditors (7988 correct and 421 incorrect). Of 421 entries mutually assessed as "incorrect," 304 (82%) were corrected by both self- and travel-auditors and 250 of these (72%) received the same corrections. Reason for changing antiretroviral therapy (ART) regimen, ART end date, viral load value, CD4%, and HIV diagnosis date had the most mismatched corrections.

CONCLUSIONS

With similar overall error rates, findings suggest that data audits conducted by trained local investigators could provide an alternative to on-site audits by external auditors to ensure continued data quality. However, discrepancies observed between corrections illustrate challenges in determining correct values even with audits.

The impact of data quality and source data verification on epidemiologic inference: a practical application using HIV observational data

BACKGROUND

Data audits are often evaluated soon after completion, even though the identification of systematic issues may lead to additional data quality improvements in the future. In this study, we assess the impact of the entire data audit process on subsequent statistical analyses.

METHODS

We conducted on-site audits of datasets from nine international HIV care sites. Error rates were quantified for key demographic and clinical variables among a subset of records randomly selected for auditing. Based on audit results, some sites were tasked with targeted validation of high-error-rate variables resulting in a post-audit dataset. We estimated the times from antiretroviral therapy initiation until death and first AIDS-defining event using the pre-audit data, the audit data, and the post-audit data.

RESULTS

The overall discrepancy rate between pre-audit and audit data (n = 250) across all audited variables was 17.1%. The estimated probability of mortality and an AIDS-defining event over time was higher in the audited data relative to the pre-audit data. Among patients represented in both the post-audit and pre-audit cohorts (n = 18,999), AIDS and mortality estimates also were higher in the post-audit data.

CONCLUSION

Though some changes may have occurred independently, our findings suggest that improved data quality following the audit may impact epidemiological inferences.

Impact of a targeted monitoring on data-quality and data-management workload of randomized controlled trials: A prospective comparative study

AIMS

Monitoring risk-based approaches in clinical trials are encouraged by regulatory guidance. However, the impact of a targeted source data verification (SDV) on data-management (DM) workload and on final data quality needs to be addressed.

METHODS

MONITORING was a prospective study aiming at comparing full SDV (100% of data verified for all patients) and targeted SDV (only key data verified for all patients) followed by the same DM program (detecting missing data and checking consistency) on final data quality, global workload and staffing costs.

RESULTS

In all, 137 008 data including 18 124 key data were collected for 126 patients from 6 clinical trials. Compared to the final database obtained using the full SDV monitoring process, the final database obtained using the targeted SDV monitoring process had a residual error rate of 1.47% (95% confidence interval, 1.41-1.53%) on overall data and 0.78% (95% confidence interval, 0.65-0.91%) on key data. There were nearly 4 times more queries per study with targeted SDV than with full SDV (mean ± standard deviation: 132 ± 101 vs 34 ± 26; P = .03). For a handling time of 15 minutes per query, the global workload of the targeted SDV monitoring strategy remained below that of the full SDV monitoring strategy. From 25 minutes per query it was above, increasing progressively to represent a 50% increase for 45 minutes per query.

CONCLUSION

Targeted SDV monitoring is accompanied by increased workload for DM, which allows to obtain a small proportion of remaining errors on key data (<1%), but may substantially increase trial costs.

Assessing data quality and the variability of source data verification auditing methods in clinical research settings

INTRODUCTION

Data audits within clinical settings are extensively used as a major strategy to identify errors, monitor study operations and ensure high-quality data. However, clinical trial guidelines are non-specific in regards to recommended frequency, timing and nature of data audits. The absence of a well-defined data quality definition and method to measure error undermines the reliability of data quality assessment. This review aimed to assess the variability of source data verification (SDV) auditing methods to monitor data quality in a clinical research setting.

MATERIAL AND METHODS

The scientific databases MEDLINE, Scopus and Science Direct were searched for English language publications, with no date limits applied. Studies were considered if they included data from a clinical trial or clinical research setting and measured and/or reported data quality using a SDV auditing method.

RESULTS

In total 15 publications were included. The nature and extent of SDV audit methods in the articles varied widely, depending upon the complexity of the source document, type of study, variables measured (primary or secondary), data audit proportion (3-100%) and collection frequency (6-24 months). Methods for coding, classifying and calculating error were also inconsistent. Transcription errors and inexperienced personnel were the main source of reported error. Repeated SDV audits using the same dataset demonstrated ∼ 40% improvement in data accuracy and completeness over time. No description was given in regards to what determines poor data quality in clinical trials.

CONCLUSIONS

A wide range of SDV auditing methods are reported in the published literature though no uniform SDV auditing method could be determined for "best practice" in clinical trials. Published audit methodology articles are warranted for the development of a standardised SDV auditing method to monitor data quality in clinical research settings.

Survival after cancer diagnosis in a cohort of HIV-positive individuals in Latin America

BACKGROUND

This study aimed to evaluate trends and predictors of survival after cancer diagnosis in persons living with HIV in the Caribbean, Central, and South America network for HIV epidemiology cohort.

METHODS

Demographic, cancer, and HIV-related data from HIV-positive adults diagnosed with cancer ≤ 1 year before or any time after HIV diagnosis from January 1, 2000-June 30, 2015 were retrospectively collected. Cancer cases were classified as AIDS-defining cancers (ADC) and non-AIDS-defining cancers (NADC). The association of mortality with cancer- and HIV-related factors was assessed using Kaplan-Meier curves and Cox proportional hazards models stratified by clinic site and cancer type.

RESULTS

Among 15,869 patients, 783 had an eligible cancer diagnosis; 82% were male and median age at cancer diagnosis was 39 years (interquartile range [IQR]: 32-47). Patients were from Brazil (36.5%), Argentina (19.9%), Chile (19.7%), Mexico (19.3%), and Honduras (4.6%). A total of 564 ADC and 219 NADC were diagnosed. Patients with NADC had similar survival probabilities as those with ADC at one year (81% vs. 79%) but lower survival at five years (60% vs. 69%). In the adjusted analysis, risk of mortality increased with detectable viral load (adjusted hazard ratio [aHR] = 1.63, p = 0.02), age (aHR = 1.02 per year, p = 0.002) and time between HIV and cancer diagnoses (aHR = 1.03 per year, p = 0.01).

CONCLUSION

ADC remain the most frequent cancers in the region. Overall mortality was related to detectable viral load and age. Longer-term survival was lower after diagnosis of NADC than for ADC, which may be due to factors unrelated to HIV.

Considerations for analysis of time-to-event outcomes measured with error: Bias and correction with SIMEX

For time-to-event outcomes, a rich literature exists on the bias introduced by covariate measurement error in regression models, such as the Cox model, and methods of analysis to address this bias. By comparison, less attention has been given to understanding the impact or addressing errors in the failure time outcome. For many diseases, the timing of an event of interest (such as progression-free survival or time to AIDS progression) can be difficult to assess or reliant on self-report and therefore prone to measurement error. For linear models, it is well known that random errors in the outcome variable do not bias regression estimates. With nonlinear models, however, even random error or misclassification can introduce bias into estimated parameters. We compare the performance of 2 common regression models, the Cox and Weibull models, in the setting of measurement error in the failure time outcome. We introduce an extension of the SIMEX method to correct for bias in hazard ratio estimates from the Cox model and discuss other analysis options to address measurement error in the response. A formula to estimate the bias induced into the hazard ratio by classical measurement error in the event time for a log-linear survival model is presented. Detailed numerical studies are presented to examine the performance of the proposed SIMEX method under varying levels and parametric forms of the error in the outcome. We further illustrate the method with observational data on HIV outcomes from the Vanderbilt Comprehensive Care Clinic.

Virologic failure and mortality in older ART initiators in a multisite Latin American and Caribbean Cohort

INTRODUCTION

The "greying" of the HIV epidemic necessitates a better understanding of the healthcare needs of older HIV-positive adults. As these individuals age, it is unclear whether comorbidities and their associated therapies or the ageing process itself alter the response to antiretroviral therapy (ART). In this study, HIV treatment outcomes and corresponding risk factors were compared between older ART initiators and those who were younger using data from the Caribbean, Central and South America Network for HIV Epidemiology (CCASAnet).

METHODS

HIV-positive adults (≥18 years) initiating ART at nine sites in Argentina, Brazil, Chile, Haiti, Honduras, Mexico and Peru were included. Patients were classified as older (≥50 years) or younger (<50 years) based on age at ART initiation. ART effectiveness was measured using three outcomes: death, virologic failure and ART treatment modification. Cox regression models for each outcome compared risk between older and younger patients, adjusting for other covariates.

RESULTS

Among 26,311 patients initiating ART between 1996 and 2016, 3389 (13%) were ≥50 years. The majority of patients in both ≥50 and <50 age groups received a non-nucleoside reverse transcriptase inhibitor-based regimen (89% vs. 87%), did not have AIDS at baseline (63% vs. 62%), and were male (59% vs. 58%). Older patients had a higher risk of death (adjusted hazard ratio (aHR) 1.64; 95% confidence intervals (CI): 1.48 to 1.83) and a lower risk of virologic failure (aHR: 0.73; 95% CI: 0.63 to 0.84). There was no difference in risk of ART modification (aHR: 1.00; 95% CI: 0.94 to 1.06). Risk factors for death, virologic failure and treatment modification were similar for each group.

CONCLUSIONS

Older age at ART initiation was associated with increased mortality and decreased risk of virologic failure in our cohort of more than 26,000 ART initiators in Latin America and the Caribbean. To the best of our knowledge this is the first study from the region to evaluate ART outcomes in this growing and important population. Given the complexity of issues related to ageing with HIV, a greater understanding is needed in order to properly respond to this shifting epidemic.

Mortality in Children with Human Immunodeficiency Virus Initiating Treatment: A Six-Cohort Study in Latin America

OBJECTIVES

To assess the risks of and factors associated with mortality, loss to follow-up, and changing regimens after children with HIV infected perinatally initiate combination antiretroviral therapy (cART) in Latin America and the Caribbean.

STUDY DESIGN

This 1997-2013 retrospective cohort study included 1174 antiretroviral therapy-naïve, perinatally infected children who started cART age when they were younger than 18 years of age (median 4.7 years; IQR 1.7-8.8) at 1 of 6 cohorts from Argentina, Brazil, Haiti, and Honduras, within the Caribbean, Central and South America Network for HIV Epidemiology. Median follow-up was 5.6 years (IQR 2.3-9.3). Study outcomes were all-cause mortality, loss to follow-up, and major changes in cART. We used Cox proportional hazards models stratified by site to examine the association between predictors and times to death or changing regimens.

RESULTS

Only 52% started cART at younger than 5 years of age; 19% began a protease inhibitor. At cART initiation, median CD4 count was 472 cells/mm³ (IQR 201-902); median CD4% was 16% (IQR 10-23). Probability of death was high in the first year of cART: 0.06 (95% CI 0.04-0.07). Five years after cART initiation, the cumulative mortality incidence was 0.12 (95% CI 0.10-0.14). Cumulative incidences for loss to follow-up and regimen change after 5 years were 0.16 (95% 0.14-0.18) and 0.30 (95% 0.26-0.34), respectively. Younger children had the greatest risk of mortality, whereas older children had the greatest risk of being lost to follow-up or changing regimens.

CONCLUSIONS

Innovative clinical and community approaches are needed for quality improvement in the pediatric care of HIV in the Americas.

The HIV care cascade in Buenos Aires, Argentina: results in a tertiary referral hospital

OBJECTIVE

To determine rates of retention, antiretroviral therapy (ART) use, and viral suppression in an adult cohort from a public tertiary referral hospital in the city of Buenos Aires, Argentina.

METHODS

HIV-positive ART-naïve patients ≥ 18 years old starting care 2011-2013 contributed data until the end of 2014. Three outcomes were assessed in 2014: retention in care, ART use, and viral suppression. Patient characteristics associated with each outcome were assessed through logistic regression.

RESULTS

A total of 1 031 patients were included. By the end of 2014, 1.5% had died and 14.8% were transferred to a different center. Of the remaining 859 patients, 563 (65.5%) were retained in 2014. Among those retained, 459 (81.5%) were on ART in 2014. Of those 459 on ART, 270 (58.8%) were virologically suppressed. Younger age was associated with lower retention (OR (odds ratio): 0.67; 95% CI (confidence interval): 0.44-0.92 for ≥ 35 vs. < 35 years), but unrelated with ART use or viral suppression. Low CD4 count at first visit was associated with ART use (OR: 35.72 for CD4 < 200, 7.13 for CD4 200-499 vs. ≥ 500, P < 0.001) and with virologic suppression (OR: 2.17 for CD4 < 200, 2.46 for CD4 200-499 vs. ≥ 500, P: 0.023).

CONCLUSIONS

Our hospital in Buenos Aires is still below the recommended 90-90-90 targets of the Joint United Nations Programme on HIV/AIDS (UNAIDS) for ART use and viral suppression. We found a major gap in retention in care. Identifying younger age as being associated with worse retention will help in the design of targeted interventions.

An assessment of data quality in a multi-site electronic medical record system in Haiti

OBJECTIVES

Strong data quality (DQ) is a precursor to strong data use. In resource limited settings, routine DQ assessment (DQA) within electronic medical record (EMR) systems can be resource-intensive using manual methods such as audit and chart review; automated queries offer an efficient alternative. This DQA focused on Haiti's national EMR - iSanté - and included longitudinal data for over 100,000 persons living with HIV (PLHIV) enrolled in HIV care and treatment services at 95 health care facilities (HCF).

METHODS

This mixed-methods evaluation used a qualitative Delphi process to identify DQ priorities among local stakeholders, followed by a quantitative DQA on these priority areas. The quantitative DQA examined 13 indicators of completeness, accuracy, and timeliness of retrospective data collected from 2005 to 2013. We described levels of DQ for each indicator over time, and examined the consistency of within-HCF performance and associations between DQ and HCF and EMR system characteristics.

RESULTS

Over all iSanté data, age was incomplete in <1% of cases, while height, pregnancy status, TB status, and ART eligibility were more incomplete (approximately 20-40%). Suspicious data flags were present for <3% of cases of male sex, ART dispenses, CD4 values, and visit dates, but for 26% of cases of age. Discontinuation forms were available for about half of all patients without visits for 180 or more days, and >60% of encounter forms were entered late. For most indicators, DQ tended to improve over time. DQ was highly variable across HCF, and within HCFs DQ was variable across indicators. In adjusted analyses, HCF and system factors with generally favorable and statistically significant associations with DQ were University hospital category, private sector governance, presence of local iSante server, greater HCF experience with the EMR, greater maturity of the EMR itself, and having more system users but fewer new users. In qualitative feedback, local stakeholders emphasized lack of stable power supply as a key challenge to data quality and use of the iSanté EMR.

CONCLUSIONS

Variable performance on key DQ indicators across HCF suggests that excellent DQ is achievable in Haiti, but further effort is needed to systematize and routinize DQ approaches within HCFs. A dynamic, interactive "DQ dashboard" within iSanté could bring transparency and motivate improvement. While the results of the study are specific to Haiti's iSanté data system, the study's methods and thematic lessons learned holdgeneralized relevance for other large-scale EMR systems in resource-limited countries.

Updating a patient-level ART database covering remote health facilities in Zomba district, Malawi: lessons learned

SETTING

A non-governmental organization, Dignitas International, working in partnership with the Ministry of Health in Malawi, adopted innovative, low-technology methods to collect, capture, and manage patient-level antiretroviral therapy (ART) data in a district database covering 26 remote low-resource facilities in Zomba District, Malawi.

OBJECTIVE

To establish a longitudinal, observational database of routinely collected program data that could serve as a program monitoring and evaluation tool as well as a platform to conduct effective operational research.

DESIGN

This article describes the processes developed for digital capture of paper-based ART clinical records at health facilities and updating them in a central electronic database. It documents and focuses on lessons learned during the implementation and review of processes.

CONCLUSIONS

Data quality can only be ensured with regular review of, and compliance with, clearly delineated workflow protocols and adequate staffing and supervision. Through the implementation of this procedure, we expect to improve data quality, completeness, and use of routine ART clinical data in low-resource settings.

The MANGUA Project: A Population-Based HIV Cohort in Guatemala

Introduction. The MANGUA cohort is an ongoing multicenter, observational study of people living with HIV/AIDS in Guatemala. The cohort is based on the MANGUA application which is an electronic database to capture essential data from the medical records of HIV patients in care. Methods. The cohort enrolls HIV-positive adults ≥16 years of age. A predefined set of sociodemographic, behavioral, clinical, and laboratory data are registered at entry to the cohort study. Results. As of October 1st, 2012, 21 697 patients had been included in the MANGUA cohort (median age: 33 years, 40.3% female). At enrollment 74.1% had signs of advanced HIV infection and only 56.3% had baseline CD4 cell counts. In the first 12 months after starting antiretroviral treatment 26.9% (n = 3938) of the patients were lost to the program. Conclusions. The implementation of a cohort of HIV-positive patients in care in Guatemala is feasible and has provided national HIV indicators to monitor and evaluate the HIV epidemic. The identified percentages of late presenters and high rates of LTFU will help the Ministry to target their current efforts in improving access to diagnosis and care.

Mortality and loss to follow-up among HIV-infected persons on long-term antiretroviral therapy in Latin America and the Caribbean

Introduction

Long-term survival of HIV patients after initiating highly active antiretroviral therapy (ART) has not been sufficiently described in Latin America and the Caribbean, as compared to other regions. The aim of this study was to describe the incidence of mortality, loss to follow-up (LTFU) and associated risk factors for patients enrolled in the Caribbean, Central and South America Network (CCASAnet).

Methods

We assessed time from ART initiation (baseline) to death or LTFU between 2000 and 2014 among ART-naïve adults (≥18 years) from sites in seven countries included in CCASAnet: Argentina, Brazil, Chile, Haiti, Honduras, Mexico and Peru. Kaplan-Meier techniques were used to estimate the probability of mortality over time. Risk factors for death were assessed using Cox regression models stratified by site and adjusted for sex, baseline age, nadir pre-ART CD4 count, calendar year of ART initiation, clinical AIDS at baseline and type of ART regimen.

Results

A total of 16,996 ART initiators were followed for a median of 3.5 years (interquartile range (IQR): 1.6–6.2). The median age at ART initiation was 36 years (IQR: 30–44), subjects were predominantly male (63%), median CD4 count was 156 cells/µL (IQR: 60–251) and 26% of subjects had clinical AIDS prior to starting ART. Initial ART regimens were predominantly non-nucleoside reverse transcriptase inhibitor based (86%). The cumulative incidence of LTFU five years after ART initiation was 18.2% (95% confidence interval (CI) 17.5–18.8%). A total of 1582 (9.3%) subjects died; the estimated probability of death one, three and five years after ART initiation was 5.4, 8.3 and 10.3%, respectively. The estimated five-year mortality probability varied substantially across sites, from 3.5 to 14.0%. Risk factors for death were clinical AIDS at baseline (adjusted hazard ratio (HR)=1.65 (95% CI 1.47–1.87); p<0.001), lower baseline CD4 (HR=1.95 (95% CI 1.63–2.32) for 50 vs. 350 cells/µL; p<0.001) and older age (HR=1.47 (95% CI 1.29–1.69) for 50 vs. 30 years at ART initiation; p<0.001).

Conclusions

In this large, long-term study of mortality among HIV-positive adults initiating ART in Latin America and the Caribbean, overall estimates of mortality were heterogeneous, generally falling between those reported in high-income countries and sub-Saharan Africa.

Reviewing progress: 7 year trends in characteristics of adults and children enrolled at HIV care and treatment clinics in the United Republic of Tanzania

BACKGROUND

To evaluate the on-going scale-up of HIV programs, we assessed trends in patient characteristics at enrolment and ART initiation over 7 years of implementation.

METHODS

Data were from Optimal Models, a prospective open cohort study of HIV-infected (HIV+) adults (≥15 years) and children (<15 years) enrolled from January 2005 to December 2011 at 44 HIV clinics in 3 regions of mainland Tanzania (Kagera, Kigoma, Pwani) and Zanzibar. Comparative statistics for trends in characteristics of patients enrolled in 2005-2007, 2008-2009 and 2010-2011 were examined.

RESULTS

Overall 62,801 HIV + patients were enrolled: 58,102(92.5%) adults, (66.5% female); 4,699(7.5%) children.Among adults, pregnant women enrolment increased: 6.8%, 2005-2007; 12.1%, 2008-2009; 17.2%, 2010-2011; as did entry into care from prevention of mother-to-child HIV transmission (PMTCT) programs: 6.6%, 2005-2007; 9.5%, 2008-2009; 12.6%, 2010-2011. WHO stage IV at enrolment declined: 27.1%, 2005-2007; 20.2%, 2008-2009; 11.1% 2010-2011. Of the 42.5% and 29.5% with CD4+ data at enrolment and ART initiation respectively, median CD4+ count increased: 210 cells/μL, 2005-2007; 262 cells/μL, 2008-2009; 266 cells/μL 2010-2011; but median CD4+ at ART initiation did not change (148 cells/μL overall). Stavudine initiation declined: 84.9%, 2005-2007; 43.1%, 2008-2009; 19.7%, 2010-2011.Among children, median age (years) at enrolment decreased from 6.1(IQR:2.7-10.0) in 2005-2007 to 4.8(IQR:1.9-8.6) in 2008-2009, and 4.1(IQR:1.5-8.1) in 2010-2011 and children <24 months increased from 18.5% to 26.1% and 31.5% respectively. Entry from PMTCT was 7.0%, 2005-2007; 10.7%, 2008-2009; 15.0%, 2010-2011. WHO stage IV at enrolment declined from 22.9%, 2005-2007, to 18.3%, 2008-2009 to 13.9%, 2010-2011. Proportion initiating stavudine was 39.8% 2005-2007; 39.5%, 2008-2009; 26.1%, 2010-2011. Median age at ART initiation also declined significantly.

CONCLUSIONS

Over time, the proportion of pregnant women and of adults and children enrolled from PMTCT programs increased. There was a decline in adults and children with advanced HIV disease at enrolment and initiation of stavudine. Pediatric age at enrolment and ART initiation declined. Results suggest HIV program maturation from an emergency response.

Implementation of provider-based electronic medical records and improvement of the quality of data in a large HIV program in Sub-Saharan Africa

Introduction

Starting in June 2010 the Infectious Diseases Institute (IDI) clinic (a large urban HIV out-patient facility) switched to provider-based Electronic Medical Records (EMR) from paper EMR entered in the database by data-entry clerks. Standardized clinics forms were eliminated but providers still fill free text clinical notes in physical patients’ files. The objective of this study was to compare the rate of errors in the database before and after the introduction of the provider-based EMR.

Methods and Findings

Data in the database pre and post provider-based EMR was compared with the information in the patients’ files and classified as correct, incorrect, and missing. We calculated the proportion of incorrect, missing and total error for key variables (toxicities, opportunistic infections, reasons for treatment change and interruption). Proportions of total errors were compared using chi-square test. A survey of the users of the EMR was also conducted. We compared data from 2,382 visits (from 100 individuals) of a retrospective validation conducted in 2007 with 34,957 visits (from 10,920 individuals) of a prospective validation conducted in April–August 2011. The total proportion of errors decreased from 66.5% in 2007 to 2.1% in 2011 for opportunistic infections, from 51.9% to 3.5% for ART toxicity, from 82.8% to 12.5% for reasons for ART interruption and from 94.1% to 0.9% for reasons for ART switch (all P<0.0001). The survey showed that 83% of the providers agreed that provider-based EMR led to improvement of clinical care, 80% reported improved access to patients’ records, and 80% appreciated the automation of providers’ tasks.

Conclusions

The introduction of provider-based EMR improved the quality of data collected with a significant reduction in missing and incorrect information. The majority of providers and clients expressed satisfaction with the new system. We recommend the use of provider-based EMR in large HIV programs in Sub-Saharan Africa.

Using audit information to adjust parameter estimates for data errors in clinical trials

BACKGROUND

Audits are often performed to assess the quality of clinical trial data, but beyond detecting fraud or sloppiness, the audit data are generally ignored. In an earlier study, using data from a nonrandomized study, Shepherd and Yu developed statistical methods to incorporate audit results into study estimates and demonstrated that audit data could be used to eliminate bias.

PURPOSE

In this article, we examine the usefulness of audit-based error-correction methods in clinical trial settings where a continuous outcome is of primary interest.

METHODS

We demonstrate the bias of multiple linear regression estimates in general settings with an outcome that may have errors and a set of covariates for which some may have errors and others, including treatment assignment, are recorded correctly for all subjects. We study this bias under different assumptions, including independence between treatment assignment, covariates, and data errors (conceivable in a double-blinded randomized trial) and independence between treatment assignment and covariates but not data errors (possible in an unblinded randomized trial). We review moment-based estimators to incorporate the audit data and propose new multiple imputation estimators. The performance of estimators is studied in simulations.

RESULTS

When treatment is randomized and unrelated to data errors, estimates of the treatment effect using the original error-prone data (i.e., ignoring the audit results) are unbiased. In this setting, both moment and multiple imputation estimators incorporating audit data are more variable than standard analyses using the original data. In contrast, in settings where treatment is randomized but correlated with data errors and in settings where treatment is not randomized, standard treatment-effect estimates will be biased. And in all settings, parameter estimates for the original, error-prone covariates will be biased. The treatment and covariate effect estimates can be corrected by incorporating audit data using either the multiple imputation or moment-based approaches. Bias, precision, and coverage of confidence intervals improve as the audit size increases.

LIMITATIONS

The extent of bias and the performance of methods depend on the extent and nature of the error as well as the size of the audit. This study only considers methods for the linear model. Settings much different than those considered here need further study.

CONCLUSIONS

In randomized trials with continuous outcomes and treatment assignment independent of data errors, standard analyses of treatment effects will be unbiased and are recommended. However, if treatment assignment is correlated with data errors or other covariates, naive analyses may be biased. In these settings, and when covariate effects are of interest, approaches for incorporating audit results should be considered.