Validation studies in epidemiologic research: estimation of the positive predictive value

Detection Algorithms for Gastrointestinal Perforation Cases in the Medical Information Database Network (MID-NET®) in Japan

BACKGROUND

The Medical Information Database Network (MID-NET®) in Japan is a vast repository providing an essential pharmacovigilance tool. Gastrointestinal perforation (GIP) is a critical adverse drug event, yet no well-established GIP identification algorithm exists in MID-NET®.

METHODS

This study evaluated 12 identification algorithms by combining ICD-10 codes with GIP therapeutic procedures. Two sites contributed 200 inpatients with GIP-suggestive ICD-10 codes (100 inpatients each), while a third site contributed 165 inpatients with GIP-suggestive ICD-10 codes and antimicrobial prescriptions. The positive predictive values (PPVs) of the algorithms were determined, and the relative sensitivity (rSn) among the 165 inpatients at the third institution was evaluated.

RESULTS

A trade-off between PPV and rSn was observed. For instance, ICD-10 code-based definitions yielded PPVs of 59.5%, whereas ICD-10 codes with CT scan and antimicrobial information gave PPVs of 56.0% and an rSn of 97.0%, and ICD-10 codes with CT scan and antimicrobial information as well as three types of operation codes produced PPVs of 84.2% and an rSn of 24.2%. The same algorithms produced statistically significant differences in PPVs among the three institutions. Combining diagnostic and procedure codes improved the PPVs. The algorithm combining ICD-10 codes with CT scan and antimicrobial information and 80 different operation codes offered the optimal balance (PPV: 61.6%, rSn: 92.4%).

CONCLUSION

This study developed valuable GIP identification algorithms for MID-NET®, revealing the trade-offs between accuracy and sensitivity. The algorithm with the most reasonable balance was determined. These findings enhance pharmacovigilance efforts and facilitate further research to optimize adverse event detection algorithms.

Validation and impact of algorithms for identifying variables in observational studies of routinely collected data

BACKGROUND AND OBJECTIVES

Among observational studies of routinely collected health data (RCD) for exploring treatment effects, algorithms are used to identify study variables. However, the extent to which algorithms are reliable and impact the credibility of effect estimates is far from clear. This study aimed to investigate the validation of algorithms for identifying study variables from RCD, and examine the impact of alternative algorithms on treatment effects.

METHODS

We searched PubMed for observational studies published in 2018 that used RCD to explore drug treatment effects. Information regarding the reporting, validation, and interpretation of algorithms was extracted. We summarized the reporting and methodological characteristics of algorithms and validation. We also assessed the divergence in effect estimates given alternative algorithms by calculating the ratio of estimates of the primary vs. alternative analyses.

RESULTS

A total of 222 studies were included, of which 93 (41.9%) provided a complete list of algorithms for identifying participants, 36 (16.2%) for exposure, and 132 (59.5%) for outcomes, and 15 (6.8%) for all study variables including population, exposure, and outcomes. Fifty-nine (26.6%) studies stated that the algorithms were validated, and 54 (24.3%) studies reported methodological characteristics of 66 validations, among which 61 validations in 49 studies were from the cross-referenced validation studies. Of those 66 validations, 22 (33.3%) reported sensitivity and 16 (24.2%) reported specificity. A total of 63.6% of studies reporting sensitivity and 56.3% reporting specificity used test-result-based sampling, an approach that potentially biases effect estimates. Twenty-eight (12.6%) studies used alternative algorithms to identify study variables, and 24 reported the effects estimated by primary analyses and sensitivity analyses. Of these, 20% had differential effect estimates when using alternative algorithms for identifying population, 18.2% for identifying exposure, and 45.5% for classifying outcomes. Only 32 (14.4%) studies discussed how the algorithms may affect treatment estimates.

CONCLUSION

In observational studies of RCD, the algorithms for variable identification were not regularly validated, and-even if validated-the methodological approach and performance of the validation were often poor. More seriously, different algorithms may yield differential treatment effects, but their impact is often ignored by researchers. Strong efforts, including recommendations, are warranted to improve good practice.

Validation of ICD-9-CM codes for major genitourinary birth defects in Military Health System administrative data, 2006-2014

BACKGROUND

The Department of Defense Birth and Infant Health Research program is dedicated to birth defects research and surveillance among military families. Here, we assess and refine the validity of International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes for selected genitourinary birth defects in the Military Health System (MHS). We additionally outline methods for the calculation of positive predictive value (PPV) and negative predictive value (NPV), sensitivity, and specificity using a stratified sampling design.

METHODS

Among military infants born from 2006 through 2014, a random sample of ICD-9-CM screen-positive cases (for six genitourinary birth defects) and screen-negative cases were selected for chart review. PPV, NPV, sensitivity, and specificity were calculated for individual defects and any included defect (i.e., overall); measures were weighted by the inverse probability of being sampled.

RESULTS

Of 461,557 infants, 686 were sampled for chart review. Bladder exstrophy was accurately reported (PPV: >90%), while the accuracy of renal dysplasia, renal agenesis/hypoplasia, and hypospadias was moderate (PPVs: 66%-68%) and congenital hydronephrosis was low (PPV: 20%). Specificity and NPVs always exceeded 98%. The overall PPV was 50%; however, excluding congenital hydronephrosis screen-positive cases and requiring at least two inpatient or outpatient diagnostic codes resulted in a PPV of 85%.

CONCLUSIONS

The validity of major genitourinary birth defect codes varied in MHS administrative data. The accuracy of an overall defect measure improved by omitting congenital hydronephrosis and requiring at least two diagnostic codes. Although PPV is generally useful for research, additional calculation of NPV, sensitivity, and specificity better informs the identification of appropriate selection criteria across surveillance and research settings.

Diagnostic accuracy of algorithms to define incident and second hip fractures: A Taiwan validation study

BACKGROUND

Previous epidemiological researchers have used various algorithms to identify a second hip fracture; however, there has been no validation of these algorithms to date. This study aimed to verify existing algorithms for identifying second hip fracture under the International Classification of Diseases diagnostic coding systems. Furthermore, we examined the validity of two newly proposed algorithms that integrated the concept of periprosthetic fractures and laterality of the ICD-10 coding system.

METHODS

Claims data of patients hospitalized for hip fracture from National Taiwan University Hospitals between 2007 and 2020 were retrieved. Hip fracture was confirmed by 2 orthopaedic surgeons with medical records and imaging data as gold standards. The validity of 9 existing and 2 newly proposed algorithms for identifying second hip fracture was evaluated.

RESULTS

The positive predictive value (PPV) range between 84% and 90% in existing algorithms for identifying second hip fractures. Noteworthy, the longer time interval for discrimination resulted in slightly increased PPV (from 87% to 90%), while decreased sensitivity noticeably (from 87% to 72%). When considering the information about periprosthetic fracture, the PPV increased to 91% without diminished sensitivity. The PPV of the newly proposed ICD-10-specific algorithm was 100%.

CONCLUSION

Algorithms integrated clinical insights of periprosthetic fractures and laterality concept of ICD-10 coding system provided satisfactory validity and help precisely define second hip fracture in future database research.

Early identification of persistent somatic symptoms in primary care: data-driven and theory-driven predictive modelling based on electronic medical records of Dutch general practices

OBJECTIVE

The present study aimed to early identify patients with persistent somatic symptoms (PSS) in primary care by exploring routine care data-based approaches.

DESIGN/SETTING

A cohort study based on routine primary care data from 76 general practices in the Netherlands was executed for predictive modelling.

PARTICIPANTS

Inclusion of 94 440 adult patients was based on: at least 7-year general practice enrolment, having more than one symptom/disease registration and >10 consultations.

METHODS

Cases were selected based on the first PSS registration in 2017-2018. Candidate predictors were selected 2-5 years prior to PSS and categorised into data-driven approaches: symptoms/diseases, medications, referrals, sequential patterns and changing lab results; and theory-driven approaches: constructed factors based on literature and terminology in free text. Of these, 12 candidate predictor categories were formed and used to develop prediction models by cross-validated least absolute shrinkage and selection operator regression on 80% of the dataset. Derived models were internally validated on the remaining 20% of the dataset.

RESULTS

All models had comparable predictive values (area under the receiver operating characteristic curves=0.70 to 0.72). Predictors are related to genital complaints, specific symptoms (eg, digestive, fatigue and mood), healthcare utilisation, and number of complaints. Most fruitful predictor categories are literature-based and medications. Predictors often had overlapping constructs, such as digestive symptoms (symptom/disease codes) and drugs for anti-constipation (medication codes), indicating that registration is inconsistent between general practitioners (GPs).

CONCLUSIONS

The findings indicate low to moderate diagnostic accuracy for early identification of PSS based on routine primary care data. Nonetheless, simple clinical decision rules based on structured symptom/disease or medication codes could possibly be an efficient way to support GPs in identifying patients at risk of PSS. A full data-based prediction currently appears to be hampered by inconsistent and missing registrations. Future research on predictive modelling of PSS using routine care data should focus on data enrichment or free-text mining to overcome inconsistent registrations and improve predictive accuracy.