Mini-Sentinel's systematic reviews of validated methods for identifying health outcomes using administrative and claims data: methods and lessons learned

Development and validation of a code-based algorithm using in-hospital medical records to identify patients with pulmonary arterial hypertension in a French healthcare database

Introduction

Pulmonary arterial hypertension (PAH) is a rare and severe disease for which most of the evidence about prognostic factors, evolution and treatment efficacy comes from cohorts, registries and clinical trials. We therefore aimed to develop and validate a new PAH identification algorithm that can be used in the French healthcare database "Système National des Données de Santé (SNDS)".

Methods

We developed and validated the algorithm using the Grenoble Alpes University Hospital medical charts. We first identified PAH patients following a previously validated algorithm, using in-hospital ICD-10 (10th revision of the International Statistical Classification of Diseases) codes, right heart catheterisation procedure and PAH-specific treatment dispensing. Then, we refined the latter with the exclusion of chronic thromboembolic pulmonary hypertension procedures and treatment, the main misclassification factor. Second, we validated this algorithm using a gold standard review of in-hospital medical charts and calculated sensitivity, specificity, positive and negative predictive value (PPV and NPV) and accuracy. Finally, we applied this algorithm in the French healthcare database and described the characteristics of the identified patients.

Results

In the Grenoble University Hospital, we identified 252 unique patients meeting all the algorithm's criteria between 1 January 2010 and 30 June 2022, and reviewed all medical records. The sensitivity, specificity, PPV, NPV and accuracy were 91.0%, 74.3%, 67.9%, 93.3% and 80.6%, respectively. Application of this algorithm to the SNDS yielded the identification of 9931 patients with consistent characteristics compared to PAH registries.

Conclusion

Overall, we propose a new PAH identification algorithm developed and adapted to the French specificities that can be used in future studies using the French healthcare database.

Validation of diagnosis codes in healthcare databases in Taiwan, a literature review

PURPOSE

To compile validation findings of diagnosis codes and related algorithms for health outcomes of interest from National Health Insurance (NHI) or electronic medical records in Taiwan.

METHODS

We carried out a literature review of English articles in PubMed® and Embase from 2000 through July 2022 with appropriate search terms. Potentially relevant articles were identified through review of article titles and abstracts, full text search of methodology terms "validation", "positive predictive value", and "algorithm" in Subjects & Methods (or Methods) and Results sections of articles, followed by full text review of potentially eligible articles.

RESULTS

We identified 50 published reports with validation findings of diagnosis codes and related algorithms for a wide range of health outcomes of interest in Taiwan, including cardiovascular diseases, stroke, renal impairment, malignancy, diabetes, mental health diseases, respiratory diseases, viral (B and C) hepatitis, and tuberculosis. Most of the reported PPVs were in the 80% ~ 99% range. Assessment of algorithms based on ICD-10 systems were reported in 8 articles, all published in 2020 or later.

CONCLUSIONS

Investigators have published validation reports that may serve as empirical evidence to evaluate the utility of secondary health data environment in Taiwan for research and regulatory purpose.

Patterns of Tyrosine Kinase Inhibitor Utilization in Newly Treated Patients With Chronic Myeloid Leukemia: An Exhaustive Population-Based Study in France

We analyzed demographic characteristics, comorbidities and patterns of treatment with tyrosine kinase inhibitors (TKIs) in a cohort of 3,633 incident cases of chronic myeloid leukemia (CML) identified across France from 1 January 2011 to 31 December 2014. Patients were identified through a specific algorithm in the French Healthcare Data System and were followed up 12 months after inclusion in the cohort. The estimated incidence rate of CML for this period in France was 1.37 per 100,000 person-years (95% Confidence Interval 1.36-1.38) and was higher in men, with a peak at age 75-79 years. At baseline, the median age of the cohort was 60 years (Inter Quartile Range 47-71), the Male/Female ratio was 1.2, and 25% presented with another comorbidity. Imatinib was the first-line TKI for 77.6% of the patients, followed by nilotinib (18.3%) and dasatinib (4.1%). Twelve months after initiation, 86% of the patients remained on the same TKI, 13% switched to another TKI and 1% received subsequently three different TKIs. During the follow-up, 23% discontinued and 52% suspended the TKI. Patients received a mean of 16.7 (Standard Deviation (SD) 9.6) medications over the first year of follow-up, and a mean of 2.7 (SD 2.3) concomitant medications on the day of first TKI prescription: 24.4% of the patients received allopurinol, 6.4% proton pump inhibitors (PPI) and 6.5% antihypertensive agents. When treatment with TKI was initiated, incident CML patients presented with comorbidities and polypharmacy, which merits attention because of the persistent use of these concomitant drugs and the potential increased risk of drug-drug interactions.

Intra-database validation of case-identifying algorithms using reconstituted electronic health records from healthcare claims data

BACKGROUND

Diagnosis performances of case-identifying algorithms developed in healthcare database are usually assessed by comparing identified cases with an external data source. When this is not feasible, intra-database validation can present an appropriate alternative.

OBJECTIVES

To illustrate through two practical examples how to perform intra-database validations of case-identifying algorithms using reconstituted Electronic Health Records (rEHRs).

METHODS

Patients with 1) multiple sclerosis (MS) relapses and 2) metastatic castration-resistant prostate cancer (mCRPC) were identified in the French nationwide healthcare database (SNDS) using two case-identifying algorithms. A validation study was then conducted to estimate diagnostic performances of these algorithms through the calculation of their positive predictive value (PPV) and negative predictive value (NPV). To that end, anonymized rEHRs were generated based on the overall information captured in the SNDS over time (e.g. procedure, hospital stays, drug dispensing, medical visits) for a random selection of patients identified as cases or non-cases according to the predefined algorithms. For each disease, an independent validation committee reviewed the rEHRs of 100 cases and 100 non-cases in order to adjudicate on the status of the selected patients (true case/ true non-case), blinded with respect to the result of the corresponding algorithm.

RESULTS

Algorithm for relapses identification in MS showed a 95% PPV and 100% NPV. Algorithm for mCRPC identification showed a 97% PPV and 99% NPV.

CONCLUSION

The use of rEHRs to conduct an intra-database validation appears to be a valuable tool to estimate the performances of a case-identifying algorithm and assess its validity, in the absence of alternative.

Development and Validation of Algorithms to Identify Pulmonary Arterial Hypertension in Administrative Data

BACKGROUND

Pulmonary arterial hypertension (PAH) is a rare disease, and much of our understanding stems from single-center studies, which are limited by sample size and generalizability. Administrative data offer an appealing opportunity to inform clinical, research, and quality improvement efforts for PAH. Yet, currently no standardized, validated method exists to distinguish PAH from other subgroups of pulmonary hypertension (PH) within this data source.

RESEARCH QUESTION

Can a collection of algorithms be developed and validated to detect PAH in administrative data in two diverse settings: all Veterans Health Administration (VA) hospitals and Boston Medical Center (BMC), a PAH referral center.

STUDY DESIGN AND METHODS

In each setting, we identified all adult patients with incident PH from 2006 through 2017 using International Classification of Diseases PH diagnosis codes. From this baseline cohort of all PH subgroups, we sequentially applied the following criteria: diagnosis codes for PAH-associated conditions, procedure codes for right heart catheterizations (RHCs), and pharmacy claims for PAH-specific therapy. We then validated each algorithm using a gold standard review of primary clinical data and calculated sensitivity, specificity, positive predictive values (PPVs), and negative predictive values.

RESULTS

From our baseline cohort, we identified 12,012 PH patients in all VA hospitals and 503 patients in BMC. Sole use of PH diagnosis codes performed poorly in identifying PAH (PPV, 16.0% in VA hospitals and 36.0% in BMC). The addition of PAH-associated conditions to the algorithm modestly improved PPV. The best performing algorithm required ICD diagnosis codes, RHC codes, and PAH-specific therapy (VA hospitals: specificity, 97.1%; PPV, 70.0%; BMC: specificity, 95.0%; PPV, 86.0%).

INTERPRETATION

This set of validated algorithms to identify PAH in administrative data can be used by the PAH scientific and clinical community to enhance the reliability and value of research findings, to inform quality improvement initiatives, and ultimately to improve health for PAH patients.

Validity of administrative claims-based algorithms for ventricular arrhythmia and cardiac arrest in the pediatric population

PURPOSE

Identify administrative claims-based algorithms for capturing out-of-hospital ventricular arrhythmias (VA) and cardiac arrests (CA) due to cardiac causes in the pediatric population with high positive-predictive value (PPV).

METHODS

Within a single pediatric center, a retrospective cohort of patients hospitalized or seen in the emergency room for VA or CA were identified from the electronic health records. Eligible encounters were blindly reviewed and linked to administrative data, including ICD-9/ICD-10 codes. Test characteristics, including PPV, for different diagnostic and procedure codes were generated using a 50% training sample. The gold standard was definite or suspected out-of-hospital VA or CA due to cardiac cause verified based on clinical criteria. Algorithms with the highest PPV were then applied to a 50% validation sample to validate performance.

RESULTS

From 2004-2017, 598 encounters met eligibility criteria. 174 (29%) had an outcome of interest, with remainder being an inpatient event or CA due to other cause. Within the training sample (n = 263), VA codes in primary position had a PPV 94% (95%CI 81%-99%) with low sensitivity (44%, 95%CI 33%-56%). CA codes in any position or VA codes in nonprimary positions had low PPV (18%-19%, 31% respectively). Applying the top three performing algorithms to the validation sample (n = 252) yielded similar PPV values.

CONCLUSIONS

Contrary to adults, algorithms including a CA code do not perform well for identifying out-of-hospital VA and CA due to cardiac cause in the pediatric populations. Researchers should be aware of the potential implications for future pediatric drug safety studies for these outcomes.

Diagnostic Algorithms for Cardiovascular Death in Administrative Claims Databases: A Systematic Review

INTRODUCTION

Valid algorithms for identification of cardiovascular (CV) deaths allow researchers to reliably assess the CV safety of medications, which is of importance to regulatory science, patient safety, and public health.

OBJECTIVE

The aim was to conduct a systematic review of algorithms to identify CV death in administrative health plan claims databases.

METHODS

We searched MEDLINE, EMBASE, and Cochrane Library for English-language studies published between January 1, 2012 and October 17, 2017. We examined references in systematic reviews to identify earlier studies. Selection included any observational study using electronic health care data to evaluate the sensitivity, specificity, positive predictive value (PPV), or negative predictive value (NPV) of algorithms for CV death (sudden cardiac death [SCD], myocardial infarction [MI]-related death, or stroke-related death) among adults aged ≥ 18 years in the United States. Data were extracted by two independent reviewers, with disagreements resolved through further discussion and consensus. The Quality Assessment of Diagnostic Accuracy Studies-2 instrument was used to assess the risk of bias.

RESULTS

Five studies (n = 4 on SCD, n = 1 on MI- and stroke-related death) were included after a review of 2053 citations. All studies reported algorithm PPVs, with incomplete reporting on other accuracy parameters. One study was at low risk of bias, three studies were at moderate risk of bias, and one study was at unclear risk of bias. Two studies identified community-occurring SCD: one identified events using International Classification of Disease, Ninth Revision (ICD-9) codes on death certificates and other criteria from medical claims (PPV = 86.8%) and the other identified events resulting in hospital presentation using first-listed ICD-9 codes on emergency department or inpatient medical claims (PPV = 92.3%). Two studies used death certificates alone to identify SCD (PPV = 27% and 32%, respectively). One study used medical claims to identify CV death (PPV = 36.4%), coronary heart disease mortality (PPV = 28.3%), and stroke mortality (PPV = 34.5%).

CONCLUSION

Two existing algorithms based on medical claims diagnoses with or without death certificates can accurately identify SCD to support pharmacoepidemiologic studies. Developing valid algorithms identifying MI- and stroke-related death should be a research priority. PROSPERO 2017 CRD42017078745.

Validating the use of registries and claims data to support randomized trials: Rationale and design of the Extending Trial-Based Evaluations of Medical Therapies Using Novel Sources of Data (EXTEND) Study

BACKGROUND

Randomized controlled trials are the "gold standard" for comparing the safety and efficacy of therapies but may be limited due to high costs, lack of feasibility, and difficulty enrolling "real-world" patient populations. The Extending Trial-Based Evaluations of Medical Therapies Using Novel Sources of Data (EXTEND) Study seeks to evaluate whether data collected within procedural registries and claims databases can reproduce trial results by substituting surrogate non-trial-based variables for exposures and outcomes.

METHODS AND RESULTS

Patient-level data from 2 clinical trial programs-the Dual Antiplatelet Therapy Study and the United States CoreValve Studies-will be linked to a combination of national registry, administrative claims, and health system data. The concordance between baseline and outcomes data collected within nontrial data sets and trial information, including adjudicated end point events, will be assessed. We will compare the study results obtained using these alternative data sources to those derived using trial-ascertained variables and end points using trial-adjudicated end points and covariates.

CONCLUSIONS

Linkage of trials to registries and claims data represents an opportunity to use alternative data sources in place of and as adjuncts to randomized clinical trial data but requires further validation. The results of this research will help determine how these data sources can be used to improve our present and future understanding of new medical treatments.

Data Challenges With Real-Time Safety Event Detection And Clinical Decision Support

BACKGROUND

The continued digitization and maturation of health care information technology has made access to real-time data easier and feasible for more health care organizations. With this increased availability, the promise of using data to algorithmically detect health care-related events in real-time has become more of a reality. However, as more researchers and clinicians utilize real-time data delivery capabilities, it has become apparent that simply gaining access to the data is not a panacea, and some unique data challenges have emerged to the forefront in the process.

OBJECTIVE

The aim of this viewpoint was to highlight some of the challenges that are germane to real-time processing of health care system-generated data and the accurate interpretation of the results.

METHODS

Distinct challenges related to the use and processing of real-time data for safety event detection were compiled and reported by several informatics and clinical experts at a quaternary pediatric academic institution. The challenges were collated from the experiences of the researchers implementing real-time event detection on more than half a dozen distinct projects. The challenges have been presented in a challenge category-specific challenge-example format.

RESULTS

In total, 8 major types of challenge categories were reported, with 13 specific challenges and 9 specific examples detailed to provide a context for the challenges. The examples reported are anchored to a specific project using medication order, medication administration record, and smart infusion pump data to detect discrepancies and errors between the 3 datasets.

CONCLUSIONS

The use of real-time data to drive safety event detection and clinical decision support is extremely powerful, but it presents its own set of challenges that include data quality and technical complexity. These challenges must be recognized and accommodated for if the full promise of accurate, real-time safety event clinical decision support is to be realized.

Using healthcare claims data to analyze the prevalence of BCR-ABL-positive chronic myeloid leukemia in France: A nationwide population-based study

Background

Data on Chronic Myeloid Leukemia (CML) prevalence are scarce. Here we provide an estimation of the prevalence of CML in France for the year 2014 using French national health insurance data.

Methods

We selected patients claiming reimbursement for tyrosine kinase inhibitors (TKI) or with hospital discharge diagnoses for CML, BCR/ABL‐positive or with full reimbursement of health care expenses for myeloid leukemia. We built an algorithm which we validated on a random sample of 100 potential CML patients by comparing the results obtained using the algorithm and the opinion of two hematologists who reviewed the patient demographics and sequence of care abstracted from claims data (internal validity). For external validity, we compared the number of incident CML patients identified using the algorithm with those recorded in French population‐based cancer registries in departments covered by such a registry.

Results

We identified 10 789 prevalent CML patients in 2014, corresponding to a crude prevalence rate of 16.3 per 100 000 inhabitants [95% confidence interval (CI) 16.0‐16.6]: 18.5 in men [18.0‐19.0] and 14.2 in women [13.8‐14.6]. The crude CML prevalence was less than 1.6 per 100 000 [1.2‐2.0] under age 20, increasing to a maximum of 48.2 [45.4‐51.2) at ages 75‐79. It varied from 10.2 to 23.8 per 100 000 across French departments. The algorithm showed high internal and external validity. Concordance rate between the algorithm and the hematologists was 96%, and the numbers of incident CML patients identified using the algorithm and the registries were 162 and 150, respectively.

Conclusion

We built and validated an algorithm to identify CML patients in administrative healthcare databases. In addition to prevalence estimation, the algorithm could be used for future economic evaluations or pharmaco‐epidemiological studies in this population.

Accuracy of Algorithms to Identify Pulmonary Arterial Hypertension in Administrative Data: A Systematic Review

BACKGROUND

The diagnosis of pulmonary arterial hypertension (PAH) is challenging, and there is significant overlap with the more heterogenous diagnosis of pulmonary hypertension (PH). Clinical and research efforts that rely on administrative data are limited by current coding systems that do not adequately reflect the clinical classification scheme. The aim of this systematic review is to investigate current algorithms to detect PAH using administrative data and to appraise the diagnostic accuracy of these algorithms against a reference standard.

METHODS

We conducted comprehensive searches of Medline, Embase, and Web of Science from their inception. We included English-language articles that applied an algorithm to an administrative or electronic health record database to identify PAH in adults.

RESULTS

Of 2,669 unique citations identified, 32 studies met all inclusion criteria. Only four of these studies validated their algorithm against a reference standard. Algorithms varied widely, ranging from single International Classification of Diseases (ICD) codes to combinations of visit, procedure, and pharmacy codes. ICD codes alone performed poorly, with positive predictive values ranging from 3.3% to 66.7%. The addition of PAH-specific therapy and diagnostic procedures to the algorithm improved the diagnostic accuracy.

CONCLUSIONS

Algorithms to identify PAH in administrative databases vary widely, and few are validated. The sole use of ICD codes performs poorly, potentially leading to biased results. ICD codes should be revised to better discriminate between PH groups, and universally accepted algorithms need to be developed and validated to capture PAH in administrative data, better informing research and clinical efforts.

Comparative safety of NSAIDs for gastrointestinal events in Asia-Pacific populations: A multi-database, international cohort study

PURPOSE

The safety of nonsteroidal anti-inflammatory drugs (NSAIDs) commonly used in Asia-Pacific countries has had limited study. We assessed the risk of hospitalization for gastrointestinal events with loxoprofen and mefenamic acid compared with other NSAIDs in Asia-Pacific populations.

METHODS

We conducted a cohort study using a distributed network with a common data model in Australia, Hong Kong, Japan, Korea, and Taiwan. We included patients who initiated diclofenac, loxoprofen, mefenamic acid, or celecoxib and followed them until their first gastrointestinal hospitalization, switch or discontinuation of medication, disenrollment, or end of database coverage. We used Cox proportional hazards models to assess hospitalization risk.

RESULTS

We identified 9879 patients in Japan, 70 492 in Taiwan, 263 741 in Korea, and 246 in Hong Kong who initiated an NSAID, and 44 013 patients in Australia, a predominantly Caucasian population. The incidence of gastrointestinal hospitalization was 25.6 per 1000 person-years in Japan, 32.8 in Taiwan, 11.5 in Korea, 484.5 in Hong Kong, and 35.6 in Australia. Compared with diclofenac, the risk of gastrointestinal events with loxoprofen was significantly lower in Korea (hazards ratio, 0.37; 95% CI, 0.25-0.54) but not in Japan (1.65; 95% CI, 0.47-5.78). The risk of gastrointestinal events with mefenamic acid was significantly lower in Taiwan (0.45; 95% CI, 0.26-0.78) and Korea (0.11; 95% CI, 0.05-0.27) but not Hong Kong (2.16; 95% CI, 0.28-16.87), compared with diclofenac.

CONCLUSIONS

Compared with diclofenac, loxoprofen was associated with a lower risk of gastrointestinal hospitalizations in Korea and mefenamic acid with a lower risk in Taiwan and Korea.

Accuracy of administrative databases in detecting primary breast cancer diagnoses: a systematic review

OBJECTIVE

To define the accuracy of administrative datasets to identify primary diagnoses of breast cancer based on the International Classification of Diseases (ICD) 9th or 10th revision codes.

DESIGN

Systematic review.

DATA SOURCES

MEDLINE, EMBASE, Web of Science and the Cochrane Library (April 2017).

ELIGIBILITY CRITERIA

The inclusion criteria were: (a) the presence of a reference standard; (b) the presence of at least one accuracy test measure (eg, sensitivity) and (c) the use of an administrative database.

DATA EXTRACTION

Eligible studies were selected and data extracted independently by two reviewers; quality was assessed using the Standards for Reporting of Diagnostic accuracy criteria.

DATA ANALYSIS

Extracted data were synthesised using a narrative approach.

RESULTS

From 2929 records screened 21 studies were included (data collection period between 1977 and 2011). Eighteen studies evaluated ICD-9 codes (11 of which assessed both invasive breast cancer (code 174.x) and carcinoma in situ (ICD-9 233.0)); three studies evaluated invasive breast cancer-related ICD-10 codes. All studies except one considered incident cases.The initial algorithm results were: sensitivity ≥80% in 11 of 17 studies (range 57%-99%); positive predictive value was ≥83% in 14 of 19 studies (range 15%-98%) and specificity ≥98% in 8 studies. The combination of the breast cancer diagnosis with surgical procedures, chemoradiation or radiation therapy, outpatient data or physician claim may enhance the accuracy of the algorithms in some but not all circumstances. Accuracy for breast cancer based on outpatient or physician's data only or breast cancer diagnosis in secondary position diagnosis resulted low.

CONCLUSION

Based on the retrieved evidence, administrative databases can be employed to identify primary breast cancer. The best algorithm suggested is ICD-9 or ICD-10 codes located in primary position.

TRIAL REGISTRATION NUMBER

CRD42015026881.

Real-World Data Analytics Fit for Regulatory Decision-Making

Healthcare database analyses (claims, electronic health records) have been identified by various regulatory initiatives, including the 21^st Century Cures Act and Prescription Drug User Fee Act ("PDUFA"), as useful supplements to randomized clinical trials to generate evidence on the effectiveness, harm, and value of medical products in routine care. Specific applications include accelerated drug approval pathways and secondary indications for approved medical products. Such real-world data ("RWD") analyses reflect how medical products impact health outside a highly controlled research environment. A constant stream of data from the routine operation of modern healthcare systems that can be analyzed in rapid cycles enables incremental evidence development for regulatory decision-making. Key evidentiary needs by regulators include 1) monitoring of medication performance in routine care, including the effectiveness, safety and value; 2) identifying new patient strata in which a drug may have added value or unacceptable harms; and 3) monitoring targeted utilization. Four broad requirements have been proposed to enable successful regulatory decision-making based on healthcare database analyses (collectively, "MVET"): Meaningful evidence that provides relevant and context-informed evidence sufficient for interpretation, drawing conclusions, and making decisions; valid evidence that meets scientific and technical quality standards to allow causal interpretations; expedited evidence that provides incremental evidence that is synchronized with the decision-making process; and transparent evidence that is audible, reproducible, robust, and ultimately trusted by decision-makers. Evidence generation systems that satisfy MVET requirements to a high degree will contribute to effective regulatory decision-making. Rapid-cycle analytics of healthcare databases is maturing at a time when regulatory overhaul increasingly demands such evidence. Governance, regulations, and data quality are catching up as the utility of this resource is demonstrated in multiple contexts.

A novel approach for medical research on lymphomas: A study validation of claims-based algorithms to identify incident cases

The use of claims database to study lymphomas in real-life conditions is a crucial issue in the future. In this way, it is essential to develop validated algorithms for the identification of lymphomas in these databases. The aim of this study was to assess the validity of diagnosis codes in the French health insurance database to identify incident cases of lymphomas according to results of a regional cancer registry, as the gold standard.Between 2010 and 2013, incident lymphomas were identified in hospital data through 2 algorithms of selection. The results of the identification process and characteristics of incident lymphomas cases were compared with data from the Tarn Cancer Registry. Each algorithm's performance was assessed by estimating sensitivity, predictive positive value, specificity (SPE), and negative predictive value.During the period, the registry recorded 476 incident cases of lymphomas, of which 52 were Hodgkin lymphomas and 424 non-Hodgkin lymphomas. For corresponding area and period, algorithm 1 provides a number of incident cases close to the Registry, whereas algorithm 2 overestimated the number of incident cases by approximately 30%. Both algorithms were highly specific (SPE = 99.9%) but moderately sensitive. The comparative analysis illustrates that similar distribution and characteristics are observed in both sources.Given these findings, the use of claims database can be consider as a pertinent and powerful tool to conduct medico-economic or pharmacoepidemiological studies in lymphomas.

Identifying multiple myeloma patients using data from the French health insurance databases: Validation using a cancer registry

This study aimed to assess the performance of several algorithms based on hospital diagnoses and the long-term diseases scheme to identify multiple myeloma patients.Potential multiple myeloma patients in 2010 to 2013 were identified using the presence of hospital records with at least 1 main diagnosis code for multiple myeloma (ICD-10 "C90"). Alternative algorithms also considered related and associated diagnoses, combination with long-term conditions, or at least 2 diagnoses. Incident patients were those with no previous "C90" codes in the past 24 or 12 months. The sensitivity, specificity, and positive and negative predictive values (PPVs and NPVs) were computed, using a French cancer registry for the corresponding area and period as the criterion standard.Long-term conditions data extracted concerned 11,559 patients (21,846 for hospital data). The registry contained 125 cases of multiple myeloma. Sensitivity was 70% when using only main hospital diagnoses (specificity 100%, PPV 79%), 76% when also considering related diagnoses (specificity 100%, PPV 74%), and 90% with associated diagnoses included (100% specificity, 64% PPV).In relation with their good performance, selected algorithms can be used to study the benefit and risk of drugs in treated multiple myeloma patients.

Self-controlled designs in pharmacoepidemiology involving electronic healthcare databases: a systematic review

Background

Observational studies are widely used in pharmacoepidemiology. Several designs can be used, in particular self-controlled designs (case-crossover and self-controlled case series). These designs offer the advantage of controlling for time-invariant confounders, which may not be collected in electronic healthcare databases. They are particularly useful in pharmacoepidemiology involving healthcare database. To be valid, they require the presence of some characteristics (key validity assumptions), and in such situations, these designs should be preferred. We aimed at describing the appropriate use and reporting of the key validity assumptions in self-controlled design studies.

Methods

Articles published between January 2011 and December 2014, and describing a self-controlled study design involving electronic healthcare databases were retrieved. The appropriate use (fulfilment of key assumptions) was studied in terms of major (abrupt onset event, rare or recurrent event, and intermittent exposure) and minor assumptions (those for which the design can be adapted).

Results

Among the 107 articles describing a self-controlled design, 35/53 (66%) case-crossover studies, and 48/55 (87%) self-controlled case series fulfilled the major validity assumptions for use of the design; 4/35 and 14/48 respectively did not fulfill the minor assumptions. Overall, 31/53 (58%) case-crossover studies and 34/55 (62%) self-controlled case series fulfilled both major and minor assumptions. The reporting of the methodology or the results was appropriate, except for power calculation.

Conclusions

Self-controlled designs were not appropriately used in34% and 13% of the articles we reviewed that described a case-crossover or a self-controlled case series design, respectively. We encourage better use of these designs in situations in which major validity assumptions are fulfilled (i.e., for which they are recommended), accounting for situations for which the design can be adapted.

Electronic supplementary material

The online version of this article (doi:10.1186/s12874-016-0278-0) contains supplementary material, which is available to authorized users.

Validity of peptic ulcer disease and upper gastrointestinal bleeding diagnoses in administrative databases: a systematic review protocol

INTRODUCTION

Administrative healthcare databases are useful to investigate the epidemiology, health outcomes, quality indicators and healthcare utilisation concerning peptic ulcers and gastrointestinal bleeding, but the databases need to be validated in order to be a reliable source for research. The aim of this protocol is to perform the first systematic review of studies reporting the validation of International Classification of Diseases, 9th Revision and 10th version (ICD-9 and ICD-10) codes for peptic ulcer and upper gastrointestinal bleeding diagnoses.

METHODS AND ANALYSIS

MEDLINE, EMBASE, Web of Science and the Cochrane Library databases will be searched, using appropriate search strategies. We will include validation studies that used administrative data to identify peptic ulcer disease and upper gastrointestinal bleeding diagnoses or studies that evaluated the validity of peptic ulcer and upper gastrointestinal bleeding codes in administrative data. The following inclusion criteria will be used: (a) the presence of a reference standard case definition for the diseases of interest; (b) the presence of at least one test measure (eg, sensitivity, etc) and (c) the use of an administrative database as a source of data. Pairs of reviewers will independently abstract data using standardised forms and will evaluate quality using the checklist of the Standards for Reporting of Diagnostic Accuracy (STARD) criteria. This systematic review protocol has been produced in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocol (PRISMA-P) 2015 statement.

ETHICS AND DISSEMINATION

Ethics approval is not required given that this is a protocol for a systematic review. We will submit results of this study to a peer-reviewed journal for publication. The results will serve as a guide for researchers validating administrative healthcare databases to determine appropriate case definitions for peptic ulcer disease and upper gastrointestinal bleeding, as well as to perform outcome research using administrative healthcare databases of these conditions.

TRIAL REGISTRATION NUMBER

CRD42015029216.

Diagnostic accuracy of administrative data algorithms in the diagnosis of osteoarthritis: a systematic review

Background

Administrative health care data are frequently used to study disease burden and treatment outcomes in many conditions including osteoarthritis (OA). OA is a chronic condition with significant disease burden affecting over 27 million adults in the US. There are few studies examining the performance of administrative data algorithms to diagnose OA. The purpose of this study is to perform a systematic review of administrative data algorithms for OA diagnosis; and, to evaluate the diagnostic characteristics of algorithms based on restrictiveness and reference standards.

Methods

Two reviewers independently screened English-language articles published in Medline, Embase, PubMed, and Cochrane databases that used administrative data to identify OA cases. Each algorithm was classified as restrictive or less restrictive based on number and type of administrative codes required to satisfy the case definition. We recorded sensitivity and specificity of algorithms and calculated positive likelihood ratio (LR+) and positive predictive value (PPV) based on assumed OA prevalence of 0.1, 0.25, and 0.50.

Results

The search identified 7 studies that used 13 algorithms. Of these 13 algorithms, 5 were classified as restrictive and 8 as less restrictive. Restrictive algorithms had lower median sensitivity and higher median specificity compared to less restrictive algorithms when reference standards were self-report and American college of Rheumatology (ACR) criteria. The algorithms compared to reference standard of physician diagnosis had higher sensitivity and specificity than those compared to self-reported diagnosis or ACR criteria.

Conclusions

Restrictive algorithms are more specific for OA diagnosis and can be used to identify cases when false positives have higher costs e.g. interventional studies. Less restrictive algorithms are more sensitive and suited for studies that attempt to identify all cases e.g. screening programs.

Electronic supplementary material

The online version of this article (doi:10.1186/s12911-016-0319-y) contains supplementary material, which is available to authorized users.

Validation of chronic obstructive pulmonary disease (COPD) diagnoses in healthcare databases: a systematic review protocol

INTRODUCTION

Healthcare databases are useful sources to investigate the epidemiology of chronic obstructive pulmonary disease (COPD), to assess longitudinal outcomes in patients with COPD, and to develop disease management strategies. However, in order to constitute a reliable source for research, healthcare databases need to be validated. The aim of this protocol is to perform the first systematic review of studies reporting the validation of codes related to COPD diagnoses in healthcare databases.

METHODS AND ANALYSIS

MEDLINE, EMBASE, Web of Science and the Cochrane Library databases will be searched using appropriate search strategies. Studies that evaluated the validity of COPD codes (such as the International Classification of Diseases 9th Revision and 10th Revision system; the Real codes system or the International Classification of Primary Care) in healthcare databases will be included. Inclusion criteria will be: (1) the presence of a reference standard case definition for COPD; (2) the presence of at least one test measure (eg, sensitivity, positive predictive values, etc); and (3) the use of a healthcare database (including administrative claims databases, electronic healthcare databases or COPD registries) as a data source. Pairs of reviewers will independently abstract data using standardised forms and will assess quality using a checklist based on the Standards for Reporting of Diagnostic accuracy (STARD) criteria. This systematic review protocol has been produced in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocol (PRISMA-P) 2015 statement.

ETHICS AND DISSEMINATION

Ethics approval is not required. Results of this study will be submitted to a peer-reviewed journal for publication. The results from this systematic review will be used for outcome research on COPD and will serve as a guide to identify appropriate case definitions of COPD, and reference standards, for researchers involved in validating healthcare databases.

TRIAL REGISTRATION NUMBER

CRD42015029204.

Validity of breast, lung and colorectal cancer diagnoses in administrative databases: a systematic review protocol

INTRODUCTION

Breast, lung and colorectal cancers constitute the most common cancers worldwide and their epidemiology, related health outcomes and quality indicators can be studied using administrative healthcare databases. To constitute a reliable source for research, administrative healthcare databases need to be validated. The aim of this protocol is to perform the first systematic review of studies reporting the validation of International Classification of Diseases 9th and 10th revision codes to identify breast, lung and colorectal cancer diagnoses in administrative healthcare databases.

METHODS AND ANALYSIS

This review protocol has been developed according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocol (PRISMA-P) 2015 statement. We will search the following databases: MEDLINE, EMBASE, Web of Science and the Cochrane Library, using appropriate search strategies. We will include validation studies that used administrative data to identify breast, lung and colorectal cancer diagnoses or studies that evaluated the validity of breast, lung and colorectal cancer codes in administrative data. The following inclusion criteria will be used: (1) the presence of a reference standard case definition for the disease of interest; (2) the presence of at least one test measure (eg, sensitivity, positive predictive values, etc) and (3) the use of data source from an administrative database. Pairs of reviewers will independently abstract data using standardised forms and will assess quality using a checklist based on the Standards for Reporting of Diagnostic accuracy (STARD) criteria.

ETHICS AND DISSEMINATION

Ethics approval is not required. We will submit results of this study to a peer-reviewed journal for publication. The results will serve as a guide to identify appropriate case definitions and algorithms of breast, lung and colorectal cancers for researchers involved in validating administrative healthcare databases as well as for outcome research on these conditions that used administrative healthcare databases.

TRIAL REGISTRATION NUMBER

CRD42015026881.

Harmonizing post-market surveillance of prescription drug misuse: a systematic review of observational studies using routinely collected data (2000-2013)

BACKGROUND

Prescription drug misuse is a growing public health concern globally. Routinely collected data provide a valuable tool for quantifying prescription drug misuse.

OBJECTIVE

To synthesize the global literature investigating prescription drug misuse utilizing routinely collected, person-level prescription/dispensing data to examine reported measures, documented extent of misuse and associated factors.

METHODS

The MEDLINE, EMBASE, CINAHL, MEDLINE In Process, Scopus citations and Google Scholar databases were searched for relevant articles published between 1 January 2000 and 31 July 2013. A total of 10,803 abstracts were screened and 281 full-text manuscripts were retrieved. Fifty-two peer-reviewed, English-language manuscripts met our inclusion criteria-an aim/method investigating prescription drug misuse in adults and a measure of misuse derived exclusively from prescription/dispensing data.

RESULTS

Four proxies of prescription drug misuse were commonly used across studies: number of prescribers, number of dispensing pharmacies, early refills and volume of drugs dispensed. Overall, 89 unique measures of misuse were identified across the 52 studies, reflecting the heterogeneity in how measures are constructed: single or composite; different thresholds, cohort definitions and time period of assessment. Consequently, it was not possible to make definitive comparisons about the extent (range reported 0.01-93.5 %), variations and factors associated with prescription drug misuse.

CONCLUSIONS

Routine data collections are relatively consistent across jurisdictions. Despite the heterogeneity of the current literature, our review identifies the capacity to develop universally accepted metrics of misuse applied to a core set of variables in prescription/dispensing claims. Our timely recommendations have the potential to unify the global research field and increase the capacity for routine surveillance of prescription drug misuse.

Data Extraction and Management in Networks of Observational Health Care Databases for Scientific Research: A Comparison of EU-ADR, OMOP, Mini-Sentinel and MATRICE Strategies

Introduction:

We see increased use of existing observational data in order to achieve fast and transparent production of empirical evidence in health care research. Multiple databases are often used to increase power, to assess rare exposures or outcomes, or to study diverse populations. For privacy and sociological reasons, original data on individual subjects can’t be shared, requiring a distributed network approach where data processing is performed prior to data sharing.

Case Descriptions and Variation Among Sites:

We created a conceptual framework distinguishing three steps in local data processing: (1) data reorganization into a data structure common across the network; (2) derivation of study variables not present in original data; and (3) application of study design to transform longitudinal data into aggregated data sets for statistical analysis. We applied this framework to four case studies to identify similarities and differences in the United States and Europe: Exploring and Understanding Adverse Drug Reactions by Integrative Mining of Clinical Records and Biomedical Knowledge (EU-ADR), Observational Medical Outcomes Partnership (OMOP), the Food and Drug Administration’s (FDA’s) Mini-Sentinel, and the Italian network—the Integration of Content Management Information on the Territory of Patients with Complex Diseases or with Chronic Conditions (MATRICE).

Findings:

National networks (OMOP, Mini-Sentinel, MATRICE) all adopted shared procedures for local data reorganization. The multinational EU-ADR network needed locally defined procedures to reorganize its heterogeneous data into a common structure. Derivation of new data elements was centrally defined in all networks but the procedure was not shared in EU-ADR. Application of study design was a common and shared procedure in all the case studies. Computer procedures were embodied in different programming languages, including SAS, R, SQL, Java, and C++.

Conclusion:

Using our conceptual framework we found several areas that would benefit from research to identify optimal standards for production of empirical knowledge from existing databases.an opportunity to advance evidence-based care management. In addition, formalized CM outcomes assessment methodologies will enable us to compare CM effectiveness across health delivery settings.

Enrollment factors and bias of disease prevalence estimates in administrative claims data

PURPOSE

Considerations for using administrative claims data in research have not been well-described. To increase awareness of how enrollment factors and insurance benefit use may contribute to prevalence estimates, we evaluated how differences in operational definitions of the cohort impact observed estimates.

METHODS

We conducted a cross-sectional study estimating the prevalence of five gastrointestinal conditions using MarketScan claims data for 73.1 million enrollees. We extracted data obtained from 2009 to 2012 to identify cohorts meeting various enrollment, prescription drug benefit, or health care utilization characteristics. Next, we identified patients meeting the case definition for each of the diseases of interest. We compared the estimates obtained to evaluate the influence of enrollment period, drug benefit, and insurance usage.

RESULTS

As the criteria for inclusion in the cohort became increasingly restrictive the estimated prevalence increased, as much as 45% to 77% depending on the disease condition and the definition for inclusion. Requiring use of the insurance benefit and a longer period of enrollment had the greatest influence on the estimates observed.

CONCLUSIONS

Individuals meeting case definition were more likely to meet the more stringent definition for inclusion in the study cohort. This may be considered a form of selection bias, where overly restrictive inclusion criteria definitions may result in selection of a source population that may no longer represent the population from which cases arose.

A formal concept analysis and semantic query expansion cooperation to refine health outcomes of interest

Background

Electronic Health Records (EHRs) are frequently used by clinicians and researchers to search for, extract, and analyze groups of patients by defining Health Outcome of Interests (HOI). The definition of an HOI is generally considered a complex and time consuming task for health care professionals.

Methods

In our clinical note-based pharmacovigilance research, we often operate upon potentially hundreds of ontologies at once, expand query inputs, and we also increase the search space over clinical text as well as structured data. Such a method implies to specify an initial set of seed concepts, which are based on concept unique identifiers. This paper presents a novel method based on Formal Concept Analysis (FCA) and Semantic Query Expansion (SQE) to assist the end-user in defining their seed queries and in refining the expanded search space that it encompasses.

Results

We evaluate our method over a gold-standard corpus from the 2008 i2b2 Obesity Challenge. This experimentation emphasizes positive results for sensitivity and specificity measures. Our new approach provides better recall with high precision of the obtained results. The most promising aspect of this approach consists in the discovery of positive results not present our Obesity NLP reference set.

Conclusions

Together with a Web graphical user interface, our FCA and SQE cooperation end up being an efficient approach for refining health outcome of interest using plain terms. We consider that this approach can be extended to support other domains such as cohort building tools.

Adverse reactions to oncologic drugs: spontaneous reporting and signal detection

Oncology is one of the areas of medicine with the most active research being conducted on new drugs. New pharmacological entities frequently enter the clinical arena, and therefore, the safety profile of anticancer products deserves continuous monitoring. However, only very severe and (unusual) suspected adverse drug reactions (ADRs) are usually reported, since cancer patients develop ADRs very frequently and some practical selectivity must be used. Notably, a recent study was able to identify 76 serious ADRs reported in updated drug labels of oncologic drugs and 50% of them (n = 38) were potentially fatal. Of these, 49 and 58%, respectively, were not described in initial drug labels. The aims of this article are to provide an overview about spontaneous reporting of ADRs of oncologic drugs and to discuss the available methods to analyze the safety of anticancer drugs using databases of spontaneous ADR reporting.

Alternative outcome definitions and their effect on the performance of methods for observational outcome studies

BACKGROUND

A systematic risk identification system has the potential to test marketed drugs for important Health Outcomes of Interest or HOI. For each HOI, multiple definitions are used in the literature, and some of them are validated for certain databases. However, little is known about the effect of different definitions on the ability of methods to estimate their association with medical products.

OBJECTIVES

Alternative definitions of HOI were studied for their effect on the performance of analytical methods in observational outcome studies.

METHODS

A set of alternative definitions for three HOI were defined based on literature review and clinical diagnosis guidelines: acute kidney injury, acute liver injury and acute myocardial infarction. The definitions varied by the choice of diagnostic codes and the inclusion of procedure codes and lab values. They were then used to empirically study an array of analytical methods with various analytical choices in four observational healthcare databases. The methods were executed against predefined drug-HOI pairs to generate an effect estimate and standard error for each pair. These test cases included positive controls (active ingredients with evidence to suspect a positive association with the outcome) and negative controls (active ingredients with no evidence to expect an effect on the outcome). Three different performance metrics where used: (i) Area Under the Receiver Operator Characteristics (ROC) curve (AUC) as a measure of a method's ability to distinguish between positive and negative test cases, (ii) Measure of bias by estimation of distribution of observed effect estimates for the negative test pairs where the true effect can be assumed to be one (no relative risk), and (iii) Minimal Detectable Relative Risk (MDRR) as a measure of whether there is sufficient power to generate effect estimates.

RESULTS

In the three outcomes studied, different definitions of outcomes show comparable ability to differentiate true from false control cases (AUC) and a similar bias estimation. However, broader definitions generating larger outcome cohorts allowed more drugs to be studied with sufficient statistical power.

CONCLUSIONS

Broader definitions are preferred since they allow studying drugs with lower prevalence than the more precise or narrow definitions while showing comparable performance characteristics in differentiation of signal vs. no signal as well as effect size estimation.

Clinical research informatics and electronic health record data

OBJECTIVES

The goal of this survey is to discuss the impact of the growing availability of electronic health record (EHR) data on the evolving field of Clinical Research Informatics (CRI), which is the union of biomedical research and informatics.

RESULTS

Major challenges for the use of EHR-derived data for research include the lack of standard methods for ensuring that data quality, completeness, and provenance are sufficient to assess the appropriateness of its use for research. Areas that need continued emphasis include methods for integrating data from heterogeneous sources, guidelines (including explicit phenotype definitions) for using these data in both pragmatic clinical trials and observational investigations, strong data governance to better understand and control quality of enterprise data, and promotion of national standards for representing and using clinical data.

CONCLUSIONS

The use of EHR data has become a priority in CRI. Awareness of underlying clinical data collection processes will be essential in order to leverage these data for clinical research and patient care, and will require multi-disciplinary teams representing clinical research, informatics, and healthcare operations. Considerations for the use of EHR data provide a starting point for practical applications and a CRI research agenda, which will be facilitated by CRI's key role in the infrastructure of a learning healthcare system.

A systematic review of validated methods to capture acute bronchospasm using administrative or claims data

PURPOSE

To identify and assess billing, procedural, or diagnosis code, or pharmacy claim-based algorithms used to identify acute bronchospasm in administrative and claims databases.

METHODS

We searched the MEDLINE database from 1991 to September 2012 using controlled vocabulary and key terms related to bronchospasm, wheeze and acute asthma. We also searched the reference lists of included studies. Two investigators independently assessed the full text of studies against pre-determined inclusion criteria. Two reviewers independently extracted data regarding participant and algorithm characteristics.

RESULTS

Our searches identified 677 citations of which 38 met our inclusion criteria. In these 38 studies, the most commonly used ICD-9 code was 493.x. Only 3 studies reported any validation methods for the identification of bronchospasm, wheeze or acute asthma in administrative and claims databases; all were among pediatric populations and only 2 offered any validation statistics. Some of the outcome definitions utilized were heterogeneous and included other disease based diagnoses, such as bronchiolitis and pneumonia, which are typically of an infectious etiology. One study offered the validation of algorithms utilizing Emergency Department triage chief complaint codes to diagnose acute asthma exacerbations with ICD-9 786.07 (wheezing) revealing the highest sensitivity (56%), specificity (97%), PPV (93.5%) and NPV (76%).

CONCLUSIONS

There is a paucity of studies reporting rigorous methods to validate algorithms for the identification of bronchospasm in administrative data. The scant validated data available are limited in their generalizability to broad-based populations.

A systematic review of validated methods for identifying patients with rheumatoid arthritis using administrative or claims data

PURPOSE

To review the evidence supporting the validity of billing, procedural, or diagnosis code, or pharmacy claim-based algorithms used to identify patients with rheumatoid arthritis (RA) in administrative and claim databases.

METHODS

We searched the MEDLINE database from 1991 to September 2012 using controlled vocabulary and key terms related to RA and reference lists of included studies were searched. Two investigators independently assessed the full text of studies against pre-determined inclusion criteria and extracted the data. Data collected included participant and algorithm characteristics.

RESULTS

Nine studies reported validation of computer algorithms based on International Classification of Diseases (ICD) codes with or without free-text, medication use, laboratory data and the need for a diagnosis by a rheumatologist. These studies yielded positive predictive values (PPV) ranging from 34 to 97% to identify patients with RA. Higher PPVs were obtained with the use of at least two ICD and/or procedure codes (ICD-9 code 714 and others), the requirement of a prescription of a medication used to treat RA, or requirement of participation of a rheumatologist in patient care. For example, the PPV increased from 66 to 97% when the use of disease-modifying antirheumatic drugs and the presence of a positive rheumatoid factor were required.

CONCLUSIONS

There have been substantial efforts to propose and validate algorithms to identify patients with RA in automated databases. Algorithms that include more than one code and incorporate medications or laboratory data and/or required a diagnosis by a rheumatologist may increase the PPV.

A systematic review of validated methods to capture several rare conditions using administrative or claims data

PURPOSE

To identify and assess billing, procedural, or diagnosis code, or pharmacy claim-based algorithms used to identify the following health outcomes in administrative and claims databases: acute disseminated encephalomyelitis (ADEM), optic neuritis, tics, and Henoch Schönlein purpura (HSP).

METHODS

We searched the MEDLINE database from 1991 to September 2012 using controlled vocabulary and key terms related to the conditions. We also searched the reference lists of included studies. Two investigators independently assessed the full text of studies against pre-determined inclusion criteria and extracted case validation data from those studies meeting inclusion criteria.

RESULTS

Two eligible studies addressed ADEM, two addressed optic neuritis, and four studies addressed tics. Only one study addressed HSP. Among these, one study of ADEM reported a positive predictive value of 66%, however the identification algorithm contained a combination of International Classification of Diseases (ICD) codes and other identification methods and the performance of the ICD-9 codes alone was not reported. No other studies reported validation data.

CONCLUSIONS

The lack of data on the validity of algorithms to identify these conditions may hamper our ability to determine incidence patterns with respect to infection and vaccination exposures. Further epidemiologic research to define validated methods of identifying cases could improve surveillance using large linked healthcare databases.

Ascertainment of acute liver injury in two European primary care databases

PURPOSE

The purpose of this study was to ascertain acute liver injury (ALI) in primary care databases using different computer algorithms. The aim of this investigation was to study and compare the incidence of ALI in different primary care databases and using different definitions of ALI.

METHODS

The Clinical Practice Research Datalink (CPRD) in UK and the Spanish "Base de datos para la Investigación Farmacoepidemiológica en Atención Primaria" (BIFAP) were used. Both are primary care databases from which we selected individuals of all ages registered between January 2004 and December 2009. We developed two case definitions of idiopathic ALI using computer algorithms: (i) restrictive definition (definite cases) and (ii) broad definition (definite and probable cases). Patients presenting prior liver conditions were excluded. Manual review of potential cases was performed to confirm diagnosis, in a sample in CPRD (21%) and all potential cases in BIFAP. Incidence rates of ALI by age, sex and calendar year were calculated.

RESULTS

In BIFAP, all cases considered definite after manual review had been detected with the computer algorithm as potential cases, and none came from the non-cases group. The restrictive definition of ALI had a low sensitivity but a very high specificity (95% in BIFAP) and showed higher rates of agreement between computer search and manual review compared to the broad definition. Higher incidence rates of definite ALI in 2008 were observed in BIFAP (3.01 (95% confidence interval (CI) 2.13-4.25) per 100,000 person-years than CPRD (1.35 (95% CI 1.03-1.78)).

CONCLUSIONS

This study shows that it is feasible to identify ALI cases if restrictive selection criteria are used and the possibility to review additional information to rule out differential diagnoses. Our results confirm that idiopathic ALI is a very rare disease in the general population. Finally, the construction of a standard definition with predefined criteria facilitates the timely comparison across databases.

Methods for systematic reviews of administrative database studies capturing health outcomes of interest

This report provides an overview of methods used to conduct systematic reviews for the US Food and Drug Administration (FDA) Mini-Sentinel project, which is designed to inform the development of safety monitoring tools for FDA-regulated products including vaccines. The objective of these reviews was to summarize the literature describing algorithms (e.g., diagnosis or procedure codes) to identify health outcomes in administrative and claims data. A particular focus was the validity of the algorithms when compared to reference standards such as diagnoses in medical records. The overarching goal was to identify algorithms that can accurately identify the health outcomes for safety surveillance. We searched the MEDLINE database via PubMed and required dual review of full text articles and of data extracted from studies. We also extracted data on each study's methods for case validation. We reviewed over 5600 abstracts/full text studies across 15 health outcomes of interest. Nearly 260 studies met our initial criteria (conducted in the US or Canada, used an administrative database, reported case-finding algorithm). Few studies (N=45), however, reported validation of case-finding algorithms (sensitivity, specificity, positive or negative predictive value). Among these, the most common approach to validation was to calculate positive predictive values, based on a review of medical records as the reference standard. Of the studies reporting validation, the ease with which a given clinical condition could be identified in administrative records varied substantially, both by the clinical condition and by other factors such as the clinical setting, which relates to the disease prevalence.

A systematic review of validated methods for identifying systemic lupus erythematosus (SLE) using administrative or claims data

PURPOSE

To examine the validity of billing, procedural, or diagnosis code, or pharmacy claim-based algorithms used to identify patients with systemic lupus erythematosus (SLE) in administrative and claims databases.

METHODS

We searched the MEDLINE database from 1991 to September 2012 using controlled vocabulary and key terms related to SLE. We also searched the reference lists of included studies. Two investigators independently assessed the full text of studies against pre-determined inclusion criteria. The two reviewers independently extracted data regarding participant and algorithm characteristics and assessed a study's methodologic rigor using a pre-defined approach.

RESULTS

Twelve studies included validation statistics for the identification of SLE in administrative and claims databases. Seven of these studies used the ICD-9 code of 710.0 in selected populations of patients seen by a rheumatologist or patients who had experienced the complication of SLE-associated nephritis, other kidney disease, or pregnancy. The other studies looked at limited data in general populations. The algorithm in the selected populations had a positive predictive value (PPV) in the range of 70-90% and of the limited data in general populations it was in the range of 50-60%.

CONCLUSIONS

Few studies use rigorous methods to validate an algorithm for the identification of SLE in general populations. Algorithms including ICD-9 code of 710.0 in physician billing and hospitalization records have a PPV of approximately 60%. A requirement that the code is obtained from a record based on treatment by a rheumatologist increases the PPV of the algorithm but limits the generalizability in the general population.

A systematic review of validated methods for identifying uveitis using administrative or claims data

PURPOSE

To review algorithms used to identify uveitis in administrative and claims databases.

METHODS

We searched the MEDLINE database via PubMed from 1991 to September 2012 using vocabulary and key terms related to uveitis. We also searched the reference lists of included studies. Two investigators independently assessed studies against pre-determined inclusion criteria. The same two investigators independently extracted data regarding participant and algorithm characteristics and assessed a study's methodological rigor using a pre-defined approach.

RESULTS

Seven studies met inclusion criteria. Variability exists among algorithms employed in these studies for finding cases of uveitis and related conditions as well as in use and implementation of validation methods. Of the seven included studies, three involved case validation. One used a narrow algorithm in addition to text mining of electronic medical records to identify incident cases and found a positive predictive value of 52.1%. The other two, which used broader uveitis definitions and included both incident and prevalent cases, found positive predictive values of 24.8% and 52.6%.

CONCLUSIONS

Further research, with case as well as individual code validation, is needed to determine appropriate uveitis algorithms for purposes of active surveillance in administrative data. Decisions about which algorithm to use will depend on the desired balance of sensitivity and specificity.

A systematic review of validated methods to capture stillbirth and spontaneous abortion using administrative or claims data

PURPOSE

To identify and assess diagnosis, procedure and pharmacy dispensing codes used to identify stillbirths and spontaneous abortion in administrative and claims databases from the United States or Canada.

METHODS

We searched the MEDLINE database from 1991 to September 2012 using controlled vocabulary and key terms related to stillbirth or spontaneous abortion. We also searched the reference lists of included studies. Two investigators independently assessed the full text of studies against pre-determined inclusion criteria. Two reviewers independently extracted data regarding participant and algorithm characteristics and assessed each study's methodological rigor using a pre-defined approach.

RESULTS

Ten publications addressing stillbirth and four addressing spontaneous abortion met our inclusion criteria. The International Classification of Diseases, Ninth Revision (ICD-9) codes most commonly used in algorithms for stillbirth were those for intrauterine death (656.4) and stillborn outcomes of delivery (V27.1, V27.3-V27.4, and V27.6-V27.7). Papers identifying spontaneous abortion used codes for missed abortion and spontaneous abortion: 632, 634.x, as well as V27.0-V27.7. Only two studies identifying stillbirth reported validation of algorithms. The overall positive predictive value of the algorithms was high (99%-100%), and one study reported an algorithm with 86% sensitivity. However, the predictive value of individual codes was not assessed and study populations were limited to specific geographic areas.

CONCLUSIONS

Additional validation studies with a nationally representative sample are needed to confirm the optimal algorithm to identify stillbirths or spontaneous abortion in administrative and claims databases.'

Observation stays in administrative claims databases: underestimation of hospitalized cases

PURPOSE

Recent policy changes in the USA have led to an increasing number of patients being placed into observation units rather than admitted directly to the hospital. Studies of administrative data that use inpatient diagnosis codes to identify cohorts, outcomes, or covariates may be affected by this change in practice. To understand the potential impact of observation stays on research using administrative healthcare data, we examine the trends of observation stays, short (≤2 days) inpatient admissions, and all inpatient admissions.

METHODS

We examined a large administrative claims database of commercially insured individuals in the USA between 2002 and 2011. Observation stays were defined on the basis of the procedure codes reimbursable by Medicare or commercial insurers. We report monthly rates of observation stays and short inpatient admissions overall and by patient demographics.

RESULTS

We identified 5 355 752 observation stays from 2002 to 2011. Over the course of study, the rate of observation stays increased, whereas the rate of short inpatient stays declined. The most common reason for observation stays was nonspecific chest pain, also the third most common reason for short inpatient stays. The increasing trend of observation stays related to circulatory diseases mirrors the decreasing trend of short inpatient stays.

CONCLUSIONS

The use of observation stays has increased in patients with commercial insurance. Failure to account for observation stays may lead to an under-ascertainment of hospitalizations in contemporary administrative healthcare data from the USA.

A systematic review of validated methods to capture myopericarditis using administrative or claims data

PURPOSE

To identify algorithms that can capture incident cases of myocarditis and pericarditis in administrative and claims databases; these algorithms can eventually be used to identify cardiac inflammatory adverse events following vaccine administration.

METHODS

We searched MEDLINE from 1991 to September 2012 using controlled vocabulary and key terms related to myocarditis. We also searched the reference lists of included studies. Two investigators independently assessed the full text of studies against pre-determined inclusion criteria. Two reviewers independently extracted data regarding participant and algorithm characteristics as well as study conduct.

RESULTS

Nine publications (including one study reported in two publications) met criteria for inclusion. Two studies performed medical record review in order to confirm that these coding algorithms actually captured patients with the disease of interest. One of these studies identified five potential cases, none of which were confirmed as acute myocarditis upon review. The other study, which employed a search algorithm based on diagnostic surveillance (using ICD-9 codes 420.90, 420.99, 422.90, 422.91 and 429.0) and sentinel reporting, identified 59 clinically confirmed cases of myopericarditis among 492,671 United States military service personnel who received smallpox vaccine between 2002 and 2003. Neither study provided algorithm validation statistics (positive predictive value, sensitivity, or specificity).

CONCLUSIONS

A validated search algorithm is currently unavailable for identifying incident cases of pericarditis or myocarditis. Several authors have published unvalidated ICD-9-based search algorithms that appear to capture myocarditis events occurring in the context of other underlying cardiac or autoimmune conditions.

A systematic review of validated methods for identifying Bell's palsy using administrative or claims data

PURPOSE

To identify and assess billing, procedural, or diagnosis code, or pharmacy claims-based algorithms used to identify Bell's palsy in administrative and claims databases.

METHODS

We searched the MEDLINE database via PubMed from 1991 to September 2012 using controlled vocabulary and key terms related to Bell's palsy. We also searched the reference lists of included studies. Two investigators independently assessed the full text of studies against pre-determined inclusion criteria. Two reviewers independently extracted data regarding participant and algorithm characteristics and assessed a study's methodologic rigor.

RESULTS

One study identified Bell's palsy using an algorithm that included ICD-9 code 351.x and H-ICDA code 350.x, and two other studies analyzed a dataset for ICD-9 code 351.0. The positive predictive values of these studies were 0.81 and 0.88, based on case adjudication of ICD-9 matches. Two further studies calculated incidence rates without validation of their methods, also including ICD-9 code 351.0. No study reported the sensitivity of algorithms to identify Bell's palsy.

CONCLUSIONS

Few publications used rigorous methods to identify a validated algorithm that could identify cases of Bell's palsy from an administrative database. The best evidence from two different datasets in the literature addressed in this review used ICD-9 code 351.0 or a collection of ICD-9 codes 351.x for facial nerve disorders including Bell's palsy, along with other ICD-9 and H-ICDA codes for facial weakness. Each study had acceptable PPV, suggesting that ICD-9 based-algorithms have some utility in detecting Bell's palsy cases.