1
|
Wang W, Liu M, He Q, Wang M, Xu J, Li L, Li G, He L, Zou K, Sun X. Validation and impact of algorithms for identifying variables in observational studies of routinely collected data. J Clin Epidemiol 2024; 166:111232. [PMID: 38043830 DOI: 10.1016/j.jclinepi.2023.111232] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2023] [Revised: 11/23/2023] [Accepted: 11/28/2023] [Indexed: 12/05/2023]
Abstract
BACKGROUND AND OBJECTIVES Among observational studies of routinely collected health data (RCD) for exploring treatment effects, algorithms are used to identify study variables. However, the extent to which algorithms are reliable and impact the credibility of effect estimates is far from clear. This study aimed to investigate the validation of algorithms for identifying study variables from RCD, and examine the impact of alternative algorithms on treatment effects. METHODS We searched PubMed for observational studies published in 2018 that used RCD to explore drug treatment effects. Information regarding the reporting, validation, and interpretation of algorithms was extracted. We summarized the reporting and methodological characteristics of algorithms and validation. We also assessed the divergence in effect estimates given alternative algorithms by calculating the ratio of estimates of the primary vs. alternative analyses. RESULTS A total of 222 studies were included, of which 93 (41.9%) provided a complete list of algorithms for identifying participants, 36 (16.2%) for exposure, and 132 (59.5%) for outcomes, and 15 (6.8%) for all study variables including population, exposure, and outcomes. Fifty-nine (26.6%) studies stated that the algorithms were validated, and 54 (24.3%) studies reported methodological characteristics of 66 validations, among which 61 validations in 49 studies were from the cross-referenced validation studies. Of those 66 validations, 22 (33.3%) reported sensitivity and 16 (24.2%) reported specificity. A total of 63.6% of studies reporting sensitivity and 56.3% reporting specificity used test-result-based sampling, an approach that potentially biases effect estimates. Twenty-eight (12.6%) studies used alternative algorithms to identify study variables, and 24 reported the effects estimated by primary analyses and sensitivity analyses. Of these, 20% had differential effect estimates when using alternative algorithms for identifying population, 18.2% for identifying exposure, and 45.5% for classifying outcomes. Only 32 (14.4%) studies discussed how the algorithms may affect treatment estimates. CONCLUSION In observational studies of RCD, the algorithms for variable identification were not regularly validated, and-even if validated-the methodological approach and performance of the validation were often poor. More seriously, different algorithms may yield differential treatment effects, but their impact is often ignored by researchers. Strong efforts, including recommendations, are warranted to improve good practice.
Collapse
Affiliation(s)
- Wen Wang
- Institute of Integrated Traditional Chinese and Western Medicine, Chinese Evidence-based Medicine and Cochrane China Center, West China Hospital, Sichuan University, Chengdu 610041, China; NMPA Key Laboratory for Real World Data Research and Evaluation in Hainan, Chengdu 610041, China; Sichuan Center of Technology Innovation for Real World Data, Chengdu 610041, China.
| | - Mei Liu
- Institute of Integrated Traditional Chinese and Western Medicine, Chinese Evidence-based Medicine and Cochrane China Center, West China Hospital, Sichuan University, Chengdu 610041, China; NMPA Key Laboratory for Real World Data Research and Evaluation in Hainan, Chengdu 610041, China; Sichuan Center of Technology Innovation for Real World Data, Chengdu 610041, China
| | - Qiao He
- Institute of Integrated Traditional Chinese and Western Medicine, Chinese Evidence-based Medicine and Cochrane China Center, West China Hospital, Sichuan University, Chengdu 610041, China; NMPA Key Laboratory for Real World Data Research and Evaluation in Hainan, Chengdu 610041, China; Sichuan Center of Technology Innovation for Real World Data, Chengdu 610041, China
| | - Mingqi Wang
- Institute of Integrated Traditional Chinese and Western Medicine, Chinese Evidence-based Medicine and Cochrane China Center, West China Hospital, Sichuan University, Chengdu 610041, China; NMPA Key Laboratory for Real World Data Research and Evaluation in Hainan, Chengdu 610041, China; Sichuan Center of Technology Innovation for Real World Data, Chengdu 610041, China
| | - Jiayue Xu
- Institute of Integrated Traditional Chinese and Western Medicine, Chinese Evidence-based Medicine and Cochrane China Center, West China Hospital, Sichuan University, Chengdu 610041, China; NMPA Key Laboratory for Real World Data Research and Evaluation in Hainan, Chengdu 610041, China; Sichuan Center of Technology Innovation for Real World Data, Chengdu 610041, China
| | - Ling Li
- Institute of Integrated Traditional Chinese and Western Medicine, Chinese Evidence-based Medicine and Cochrane China Center, West China Hospital, Sichuan University, Chengdu 610041, China; NMPA Key Laboratory for Real World Data Research and Evaluation in Hainan, Chengdu 610041, China; Sichuan Center of Technology Innovation for Real World Data, Chengdu 610041, China
| | - Guowei Li
- Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario L8S 4L8, Canada; Center for Clinical Epidemiology and Methodology, Guangdong Second Provincial General Hospital, Guangzhou, Guangdong 510317, China; Biostatistics Unit, Research Institute at St. Joseph's Healthcare Hamilton, Hamilton, Ontario L8N 4A6, Canada
| | - Lin He
- Intelligence Library Center, West China Hospital, Sichuan University, Chengdu, Sichuan 610041, China
| | - Kang Zou
- Institute of Integrated Traditional Chinese and Western Medicine, Chinese Evidence-based Medicine and Cochrane China Center, West China Hospital, Sichuan University, Chengdu 610041, China; NMPA Key Laboratory for Real World Data Research and Evaluation in Hainan, Chengdu 610041, China; Sichuan Center of Technology Innovation for Real World Data, Chengdu 610041, China
| | - Xin Sun
- Institute of Integrated Traditional Chinese and Western Medicine, Chinese Evidence-based Medicine and Cochrane China Center, West China Hospital, Sichuan University, Chengdu 610041, China; NMPA Key Laboratory for Real World Data Research and Evaluation in Hainan, Chengdu 610041, China; Sichuan Center of Technology Innovation for Real World Data, Chengdu 610041, China.
| |
Collapse
|