Chart validation of inpatient ICD-9-CM administrative diagnosis codes for ischemic stroke among IGIV users in the Sentinel Distributed Database

A scoping review of the methodological approaches used in retrospective chart reviews to validate adverse event rates in administrative data

Patient safety is a key quality issue for health systems. Healthcare acquired adverse events (AEs) compromise safety and quality; therefore, their reporting and monitoring is a patient safety priority. Although administrative datasets are potentially efficient tools for monitoring rates of AEs, concerns remain over the accuracy of their data. Chart review validation studies are required to explore the potential of administrative data to inform research and health policy. This review aims to present an overview of the methodological approaches and strategies used to validate rates of AEs in administrative data through chart review. This review was conducted in line with the Joanna Briggs Institute methodological framework for scoping reviews. Through database searches, 1054 sources were identified, imported into Covidence, and screened against the inclusion criteria. Articles that validated rates of AEs in administrative data through chart review were included. Data were extracted, exported to Microsoft Excel, arranged into a charting table, and presented in a tabular and descriptive format. Fifty-six studies were included. Most sources reported on surgical AEs; however, other medical specialties were also explored. Chart reviews were used in all studies; however, few agreed on terminology for the study design. Various methodological approaches and sampling strategies were used. Some studies used the Global Trigger Tool, a two-stage chart review method, whilst others used alternative single-, two-stage, or unclear approaches. The sources used samples of flagged charts (n = 24), flagged and random charts (n = 11), and random charts (n = 21). Most studies reported poor or moderate accuracy of AE rates. Some studies reported good accuracy of AE recording which highlights the potential of using administrative data for research purposes. This review highlights the potential for administrative data to provide information on AE rates and improve patient safety and healthcare quality. Nonetheless, further work is warranted to ensure that administrative data are accurate. The variation of methodological approaches taken, and sampling techniques used demonstrate a lack of consensus on best practice; therefore, further clarity and consensus are necessary to develop a more systematic approach to chart reviewing.

Validation of ICD-10 hospital discharge diagnosis codes to identify incident and recurrent ischemic stroke from a US integrated healthcare system

PURPOSE

This study validated incident and recurrent ischemic stroke identified by International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10) hospital discharge diagnosis codes.

METHODS

Using electronic health records (EHR) of adults (≥18 years) receiving care from Kaiser Permanente Southern California with ICD-10 hospital discharge diagnosis codes of ischemic stroke (I63.x, G46.3, and G46.4) between October 2015 and September 2020, we identified 75 patients with both incident and recurrent stroke events (total 150 cases). Two neurologists independently evaluated validity of ICD-10 codes through chart reviews.

RESULTS

The positive predictive value (PPV, 95% CI) for incident stroke was 93% (95% CI: 88%, 99%) and the PPV for recurrent stroke was 72% (95% CI: 62%, 82%). The PPV for recurrent stroke improved after applying a gap of 20 days (PPV of 75%; 95% CI: 63%, 87%) or removing hospital admissions related to stroke-related procedures (PPV of 78%; 95% CI: 68%, 88%).

CONCLUSION

The ICD-10 hospital discharge diagnosis codes for ischemic stroke showed a high PPV for incident cases, while the PPV for recurrent cases were less optimal. Algorithms to improve the accuracy of ICD-10 codes for recurrent ischemic stroke may be necessary.

Validation of Claims-Based Algorithm for Lyme Disease, Massachusetts, USA

Compared with notifiable disease surveillance, claims-based algorithms estimate higher Lyme disease incidence, but their accuracy is unknown. We applied a previously developed Lyme disease algorithm (diagnosis code plus antimicrobial drug prescription dispensing within 30 days) to an administrative claims database in Massachusetts, USA, to identify a Lyme disease cohort during July 2000-June 2019. Clinicians reviewed and adjudicated medical charts from a cohort subset by using national surveillance case definitions. We calculated positive predictive values (PPVs). We identified 12,229 Lyme disease episodes in the claims database and reviewed and adjudicated 128 medical charts. The algorithm's PPV for confirmed, probable, or suspected cases was 93.8% (95% CI 88.1%-97.3%); the PPV was 66.4% (95% CI 57.5%-74.5%) for confirmed and probable cases only. In a high incidence setting, a claims-based algorithm identified cases with a high PPV, suggesting it can be used to assess Lyme disease burden and supplement traditional surveillance data.

Development and Evaluation of the Algorithm CErtaInty Tool (ACE-IT) to Assess Electronic Medical Record and Claims-based Algorithms' Fit for Purpose for Safety Outcomes

INTRODUCTION

Electronic health record (EHR) or medical claims-based algorithms (i.e., operational definitions) can be used to define safety outcomes using real-world data. However, existing tools do not allow researchers and decision-makers to adequately appraise whether a particular algorithm is fit for purpose (FFP) to support regulatory decisions on drug safety surveillance. Our objective was to develop a tool to enable regulatory decision-makers and other stakeholders to appraise whether a given algorithm is FFP for a specific decision context.

METHODS

We drafted a set of 77 generic items informed by regulatory guidance documents, existing instruments, and publications. The outcome of ischemic stroke served as an exemplar to inform the development of draft items. The items were designed to be outcome independent. We conducted a three-round online Delphi panel to develop and refine the tool and achieve consensus on items (> 70% agreement) among panel participants composed of regulators, researchers from pharmaceutical organizations, academic clinicians, methodologists, pharmacoepidemiologists, and cardiologists. We conducted a qualitative analysis of panel responses. Five pairs of reviewers independently evaluated two ischemic stroke algorithm validation studies to test its application. We developed a user guide, with explanation and elaboration for each item, guidance on essential and additional elements for user responses, and an illustrative example of a complete assessment. Furthermore, we conducted a 2-h online stakeholder panel of 16 participants from regulatory agencies, academic institutions, and industry. We solicited input on key factors for an FFP assessment, their general reaction to the Algorithm CErtaInty Tool (ACE-IT), limitations of the tool, and its potential use.

RESULTS

The expert panel reviewed and made changes to the initial list of 77 items. The panel achieved consensus on 38 items, and the final version of the ACE-IT includes 34 items after removal of duplicate items. Applying the tool to two ischemic stroke algorithms demonstrated challenges in its application and identified shared concepts addressed by more than one item. The ACE-IT was viewed positively by the majority of stakeholders. They identified that the tool could serve as an educational resource as well as an information-sharing platform. The time required to complete the assessment was identified as an important limitation. We consolidated items with shared concepts and added a preliminary screen section and a summary assessment box based on their input. The final version of the ACE-IT is a 34-item tool for assessing whether algorithm validation studies on safety outcomes are FFP. It comprises the domains of internal validity (24 items), external validity (seven items), and ethical conduct and reporting of the validation study (three items). The internal validity domain includes sections on objectives, data sources, population, outcomes, design and setting, statistical methods, reference standard, accuracy, and strengths and limitations. The external validity domain includes items that assess the generalizability to a proposed target study. The domain on ethics and transparency includes items on ethical conduct and reporting of the validation study.

CONCLUSION

The ACE-IT supports a structured, transparent, and flexible approach for decision-makers to appraise whether electronic health record or medical claims-based algorithms for safety outcomes are FFP for a specific decision context. Reliability and validity testing using a larger sample of participants in other therapeutic areas and further modifications to reduce the time needed to complete the assessment are needed to fully evaluate its utility for regulatory decision-making.

Electronic Medical Record-Based Case Phenotyping for the Charlson Conditions: Scoping Review

Background

Electronic medical records (EMRs) contain large amounts of rich clinical information. Developing EMR-based case definitions, also known as EMR phenotyping, is an active area of research that has implications for epidemiology, clinical care, and health services research.

Objective

This review aims to describe and assess the present landscape of EMR-based case phenotyping for the Charlson conditions.

Methods

A scoping review of EMR-based algorithms for defining the Charlson comorbidity index conditions was completed. This study covered articles published between January 2000 and April 2020, both inclusive. Embase (Excerpta Medica database) and MEDLINE (Medical Literature Analysis and Retrieval System Online) were searched using keywords developed in the following 3 domains: terms related to EMR, terms related to case finding, and disease-specific terms. The manuscript follows the Preferred Reporting Items for Systematic reviews and Meta-analyses extension for Scoping Reviews (PRISMA) guidelines.

Results

A total of 274 articles representing 299 algorithms were assessed and summarized. Most studies were undertaken in the United States (181/299, 60.5%), followed by the United Kingdom (42/299, 14.0%) and Canada (15/299, 5.0%). These algorithms were mostly developed either in primary care (103/299, 34.4%) or inpatient (168/299, 56.2%) settings. Diabetes, congestive heart failure, myocardial infarction, and rheumatology had the highest number of developed algorithms. Data-driven and clinical rule–based approaches have been identified. EMR-based phenotype and algorithm development reflect the data access allowed by respective health systems, and algorithms vary in their performance.

Conclusions

Recognizing similarities and differences in health systems, data collection strategies, extraction, data release protocols, and existing clinical pathways is critical to algorithm development strategies. Several strategies to assist with phenotype-based case definitions have been proposed.

Evaluation of Use of Technologies to Facilitate Medical Chart Review

INTRODUCTION

While medical chart review remains the gold standard to validate health conditions or events identified in administrative claims and electronic health record databases, it is time consuming, expensive and can involve subjective decisions.

AIM

The aim of this study was to describe the landscape of technology-enhanced approaches that could be used to facilitate medical chart review within and across distributed data networks.

METHOD

We conducted a semi-structured survey regarding processes for medical chart review with organizations that either routinely do medical chart review or use technologies that could facilitate chart review.

RESULTS

Fifteen out of 17 interviewed organizations used optical character recognition (OCR) or natural language processing (NLP) in their chart review process. None used handwriting recognition software. While these organizations found OCR and NLP to be useful for expediting extraction of useful information from medical charts, they also mentioned several challenges. Quality of medical scans can be variable, interfering with the accuracy of OCR. Additionally, linguistic complexity in medical notes and heterogeneity in reporting templates used by different healthcare systems can reduce the transportability of NLP-based algorithms to diverse healthcare settings.

CONCLUSION

New technologies including OCR and NLP are currently in use by various organizations involved in medical chart review. While technology-enhanced approaches could scale up capacity to validate key variables and make information about important clinical variables from medical records more generally available for research purposes, they often require considerable customization when employed in a distributed data environment with multiple, diverse healthcare settings.

Validating the use of registries and claims data to support randomized trials: Rationale and design of the Extending Trial-Based Evaluations of Medical Therapies Using Novel Sources of Data (EXTEND) Study

BACKGROUND

Randomized controlled trials are the "gold standard" for comparing the safety and efficacy of therapies but may be limited due to high costs, lack of feasibility, and difficulty enrolling "real-world" patient populations. The Extending Trial-Based Evaluations of Medical Therapies Using Novel Sources of Data (EXTEND) Study seeks to evaluate whether data collected within procedural registries and claims databases can reproduce trial results by substituting surrogate non-trial-based variables for exposures and outcomes.

METHODS AND RESULTS

Patient-level data from 2 clinical trial programs-the Dual Antiplatelet Therapy Study and the United States CoreValve Studies-will be linked to a combination of national registry, administrative claims, and health system data. The concordance between baseline and outcomes data collected within nontrial data sets and trial information, including adjudicated end point events, will be assessed. We will compare the study results obtained using these alternative data sources to those derived using trial-ascertained variables and end points using trial-adjudicated end points and covariates.

CONCLUSIONS

Linkage of trials to registries and claims data represents an opportunity to use alternative data sources in place of and as adjuncts to randomized clinical trial data but requires further validation. The results of this research will help determine how these data sources can be used to improve our present and future understanding of new medical treatments.