Reporting of abstracts in studies that used routinely collected data for exploring drug treatment effects: a cross-sectional survey

Validation and impact of algorithms for identifying variables in observational studies of routinely collected data

BACKGROUND AND OBJECTIVES

Among observational studies of routinely collected health data (RCD) for exploring treatment effects, algorithms are used to identify study variables. However, the extent to which algorithms are reliable and impact the credibility of effect estimates is far from clear. This study aimed to investigate the validation of algorithms for identifying study variables from RCD, and examine the impact of alternative algorithms on treatment effects.

METHODS

We searched PubMed for observational studies published in 2018 that used RCD to explore drug treatment effects. Information regarding the reporting, validation, and interpretation of algorithms was extracted. We summarized the reporting and methodological characteristics of algorithms and validation. We also assessed the divergence in effect estimates given alternative algorithms by calculating the ratio of estimates of the primary vs. alternative analyses.

RESULTS

A total of 222 studies were included, of which 93 (41.9%) provided a complete list of algorithms for identifying participants, 36 (16.2%) for exposure, and 132 (59.5%) for outcomes, and 15 (6.8%) for all study variables including population, exposure, and outcomes. Fifty-nine (26.6%) studies stated that the algorithms were validated, and 54 (24.3%) studies reported methodological characteristics of 66 validations, among which 61 validations in 49 studies were from the cross-referenced validation studies. Of those 66 validations, 22 (33.3%) reported sensitivity and 16 (24.2%) reported specificity. A total of 63.6% of studies reporting sensitivity and 56.3% reporting specificity used test-result-based sampling, an approach that potentially biases effect estimates. Twenty-eight (12.6%) studies used alternative algorithms to identify study variables, and 24 reported the effects estimated by primary analyses and sensitivity analyses. Of these, 20% had differential effect estimates when using alternative algorithms for identifying population, 18.2% for identifying exposure, and 45.5% for classifying outcomes. Only 32 (14.4%) studies discussed how the algorithms may affect treatment estimates.

CONCLUSION

In observational studies of RCD, the algorithms for variable identification were not regularly validated, and-even if validated-the methodological approach and performance of the validation were often poor. More seriously, different algorithms may yield differential treatment effects, but their impact is often ignored by researchers. Strong efforts, including recommendations, are warranted to improve good practice.

Reporting, handling, and interpretation of time-varying drug treatments in observational studies using routinely collected healthcare data

BACKGROUND

Time-varying drug treatments are common in studies using routinely collected health data (RCD) for assessing treatment effects. This study aimed to examine how these studies reported, handled, and interpreted time-varying drug treatments.

METHODS

A systematic search was conducted on PubMed from 2018 to 2020. Eligible studies were those used RCD to explore drug treatment effects. We summarized the reporting characteristics and methods employed for handling time-varying treatments. Logistic regressions were performed to investigate the association between study characteristics and the reporting of time-varying treatments.

RESULTS

Two hundred and fifty-six studies were included, and 225 (87.9%) studies involved time-varying treatments. Of these, 24 (10.7%) reported the proportion of time-varying treatments and 105 (46.7%) reported methods used to handle time-varying treatments. Multivariable logistic regression showed that medical studies, prespecified protocol, and involvement of methodologists were associated with a higher likelihood of reporting the methods applied to handle time-varying treatments. Among the 105 studies that reported methods, as-treated analyses were the most commonly used analysis sets, which were employed in 73.9%, 75.3% and 88.2% of studies that reported approaches for treatment discontinuation, treatment switching and treatment add-on. Among the 225 studies involved time-varying treatments, 27 (12.0%) acknowledged the potential bias introduced by treatment change, of which 14 (51.9%) suggested that potential biases may impact acceptance or rejection of the null hypothesis.

CONCLUSIONS

Among observational studies using RCD, the underreporting about the presence and methods for handling time-varying treatments was largely common. The potential biases due to time-varying treatments have frequently been disregarded. Collaborative endeavors are strongly needed to enhance the prevailing practices.

Data source profile reporting by studies that use routinely collected health data to explore the effects of drug treatment

BACKGROUND

Routinely collected health data (RCD) are important resource for exploring drug treatment effects. Adequate reporting of data source profiles may increase the credibility of evidence generated from these data. This study conducted a systematic literature review to evaluate the reporting characteristics of databases used by RCD studies to explore the effects of drug treatment.

METHODS

Observational studies published in 2018 that used RCD to explore the effects of drug treatment were identified by searching PubMed. We categorized eligible reports into two groups by journal impact factor (IF), including the top 5 general medical journals (NEJM, Lancet, JAMA, BMJ and JAMA Internal Medicine) and the other journals. The reporting characteristics of the databases used were described and compared between the two groups and between studies citing and not citing database references.

RESULTS

A total of 222 studies were included, of which 53 (23.9%) reported that they applied data linkage, 202 (91.0%) reported the type of database, and 211 (95.0%) reported the coverage of the data source. Only 81 (36.5%) studies reported the timeframe of the database. Studies in high-impact journals were more likely to report that they applied data linkage (65.1% vs. 20.2%) and used electronic medical records (EMR) (73.7% vs. 30.0%) and national data sources (77.8% vs. 51.3%) than those published in other medical journals. There were 137/222 (61.7%) cited database references. Studies with database-specific citations had better reporting of the data sources and were more likely to publish in high-impact journals than those without (mean IF, 6.08 vs. 4.09).

CONCLUSIONS

Some deficits were found in the reporting quality of databases in studies that used RCD to explore the effects of drug treatment. Studies citing database-specific references may provide detailed information regarding data source characteristics. The adoption of reporting guidelines and education on their use is urgently needed to promote transparency by research groups.