Demystifying probabilistic linkage: Common myths and misconceptions

Many of the distinctions made between probabilistic and deterministic linkage are misleading. While these two approaches to record linkage operate in different ways and can produce different outputs, the distinctions between them are more a result of how they are implemented than because of any intrinsic differences. In the way they are generally applied, probabilistic and deterministic procedures can be little more than alternative means to similar ends-or they can arrive at very different ends depending on choices that are made during implementation. Misconceptions about probabilistic linkage contribute to reluctance for implementing it and mistrust of its outputs. We aim to explain how the outputs of either approach can be tailored to suit the intended application, but also to highlight the ways in which probabilistic linkage is generally more flexible, more powerful and more informed by the data. This is accomplished by examining common misconceptions about probabilistic linkage and its difference from deterministic linkage, highlighting the potential impact of design choices on the outputs of either approach. We hope that better understanding of linkage designs will help to allay concerns about probabilistic linkage, and help data linkers to select and tailor procedures to produce outputs that are appropriate for their intended use.

Estimation of socioeconomic inequalities in mortality in Japan using national census-linked longitudinal mortality data

BACKGROUND

We aimed to develop census-linked longitudinal mortality data for Japan and assess its validity as a new resource for estimating socioeconomic inequalities in health.

METHODS

Using deterministic linkage, we identified, from national censuses for 2000 and 2010 and national death records, persons and deceased persons who had unique personal identifiers (generated using sex, birth year/month, address, and marital status). For the period 2010-2015, 1 537 337 Japanese men and women aged 30-79 years (1.9% in national census) were extracted to represent the sample population. This population was weighted to adjust for confounding factors. We estimated age-standardized mortality rates (ASMRs) by education level and occupational class. The slope index of inequality (SII) and relative index inequality (RII) by educational level were calculated as inequality measures.

RESULTS

The reweighted sample population's mortality rates were somewhat higher than those of the complete registry, especially in younger age-groups and for external causes. All-cause ASMRs (per 100 000 person-years) for individuals aged 40-79 years with high, middle, and low education levels were 1078 (95% confidence interval: 1051-1105), 1299 (1279-1320), and 1670 (1634-1707) for men, and 561 (536-587), 601 (589-613), and 777 (745-808) for women, respectively, during 2010-2015. SII and RII by educational level increased among both sexes between 2000-2005 and 2010-2015, which indicates mortality inequalities increased.

CONCLUSIONS

The developed census-linked longitudinal mortality data provide new estimates of socioeconomic inequalities in Japan that can be triangulated with estimates obtained with other methods.

Internal deterministic record linkage using indirect identifiers for matching of same-patient hospital transfers and early readmissions after acute coronary syndrome in a nationwide hospital discharge database: a retrospective observational validation study

OBJECTIVES

To assess validity of record linkage using multiple indirect personal identifiers to identify same-patient hospitalisations and definition of episode of care (EC) due to acute coronary syndrome (ACS).

METHODS

Using national hospital discharge data to identify all admissions due to ACS, we used six different linkage rules using indirect identifiers with increasing level of detail and compared validity against a pseudonymised unique identifier used as gold standard (GS). Contiguous hospitalisations within each matched group of hospitalizations occurring within 28 days of each other were considered one EC. We classified hospitalisations according to time between the first pair of hospitalisations as hospital transfer (HT: ≤1 day), early readmission (ER: 2-28 days) or recurrent cases (>28 days).

RESULTS

There were 146 671 hospitalisations (unlinked), 121 987 ACS 28-day EC (linked GS), with 18 398 HTs (≤1 day), and 6286 ERs (≤28 days). Linkage rules using demographic and residence code variables produced linkage rates with highest validity for rule using sex, date of birth and four-digit residence code with sensitivity of 98.4 (95% CI: 98.4 to 98.5); specificity of 97.8 (95% CI: 97.6 to 98.0) and Cohen's κ of 0.9 to detect ACS-EC, compared with GS linkage rule. Similarly, validity for HT and ER was high and of similar magnitude, with sensitivity ranging between 97.2% and 98.1%, and specificity between 98.8% and 99.9%, respectively.

CONCLUSIONS

Our internal linkage validation study using indirect patient identifiers will allow calibration of incidence rates and performance indicators, accounting for the effect of HT and readmissions.

Exploring the Complexity of Real-World Health Data Record Linkage-An Exemplary Study Linking Cancer Registry and Claims Data

PURPOSE

Record linkage based on quasi-identifiers remains an important approach as not every data source provides a comprehensive unique identifier. In this study, the reasons for the failure of a linkage based on quasi-identifiers were examined. Furthermore, informed algorithms using information on gold standard links were developed to investigate the potentially achievable linkage quality based on quasi-identifiers.

METHODS

The study population includes patients from an antidiabetic cohort from German claims and colorectal cancer patients from two German cancer registries. Linkage algorithms were applied using information on gold standard links. Informed linkage algorithms based on deterministic linkage, logistic regression, random forests, gradient boosting, and neural networks were derived and compared. Descriptive analyses were performed to identify reasons for the failure of linkage, such as discrepancies between data sources.

RESULTS

A gradient boosting-based linkage approach performed best, achieving a precision (positive predictive value) of 77%, a recall (sensitivity) of 81%, and an F*-measure (combining precision and recall) of 64%. Of 641 patients in GePaRD, 8% were not uniquely identifiable using birth year, sex, area of residence, and year and quarter of diagnosis, whereas 33% of 42 817 cancer registry patients were not uniquely identifiable with these quasi-identifiers.

CONCLUSIONS

Linkage of German claims and cancer registry data based on quasi-identifiers does result in insufficient linkage quality since subjects cannot be uniquely identified. It is advisable to use unique identifiers from a subsample, if available, to derive informed linkage algorithms for the entire sample. In this case, the machine learning technique gradient boosting has been found to outperform other methods.

Lessons in Linkage: combining administrative data using deterministic linkage for surveillance of sports and recreation injuries in Florida, United States

Objectives

Previous and ongoing epidemiological surveillance of sports and recreation injuries (SRI) has been cross-sectional in nature, utilised a subset of injuries based on athletic trainer availability, or focused on elite and professional athletes. In the United States, surveillance is often prohibitively expensive and not well funded by national organisations or agencies, except for the case of some professional and elite sports. This paper details the methodology, barriers, and successes of using deterministic linkage to combine emergency department and hospitalisation data with a single identifier for use in surveilling sports injuries for persons aged 5 to 18 years.

Design

Data linkage of a population cohort.

Methods

We performed deterministic linkage of administrative emergency department and hospitalisation data from the state of Florida in the US. Data was acquired from the Florida Agency for Health Care Administration. With limited identifiers available due to privacy, we combined data across multiple years using a near universal identifier. We identified sport and recreation injuries using a modified External Cause of Injury Morbidity Matrix and ICD codes across all possible diagnoses. Finally, we obtained descriptive statistics of records that were successfully linked and those that were not to assess similarities between the groups.

Results

We found 384,731 visits for SRI over a seven-year period. We were able to link approximately 70% of the records using a single identifier. There were statistically significant differences by age, sex, payer, and race/ethnicity for the records that were linked compared to the records that were not linked.

Conclusions

This study is significant because while similar methods have been used to examine other conditions (e.g. asthma), few have linked multiple types of administrative data especially with nearly no identifiers to examine sports and recreation injuries. This method was found useful to identify injuries over time for the same individuals seeking care in emergency departments, or in hospital inpatient settings, though future work will need to address the limitations of this method. If we expect to move health surveillance forward as budgets for it become even more limited, we must develop and improve methods to do it with fewer resources, including using data that has great limitations.

FIRLA: A Fast Incremental Record Linkage Algorithm

Record linkage is an important problem studied widely in many domains including biomedical informatics. A standard version of this problem is to cluster records from several datasets, such that each cluster has records pertinent to just one individual. Typically, datasets are huge in size. Hence, existing record linkage algorithms take a very long time. It is thus essential to develop novel fast algorithms for record linkage. The incremental version of this problem is to link previously clustered records with new records added to the input datasets. A novel algorithm has been created to efficiently perform standard and incremental record linkage. This algorithm leverages a set of efficient techniques that significantly restrict the number of record pair comparisons and distance computations. Our algorithm shows an average speed-up of 2.4x (up to 4x) for the standard linkage problem as compared to the state-of-the-art, without any drop in linkage performance at all. On average, our algorithm can incrementally link records in just 33% of the time required for linking them from scratch. Our algorithms achieve comparable or superior linkage performance and outperform the state-of-the-art in terms of linking time in all cases where the number of comparison attributes is greater than two. In practice, more than two comparison attributes are quite common. The proposed algorithm is very efficient and could be used in practice for record linkage applications especially when records are being added over time and linkage output needs to be updated frequently.

Record linkage of claims and cancer registries data-Evaluation of a deterministic linkage approach based on indirect personal identifiers

PURPOSE

In Germany, record linkage of claims and cancer registry data is cost- and time-consuming, since up until recently no unique personal identifier was available in both data sources. The aim of this study was to evaluate the feasibility and performance of a deterministic linkage procedure based on indirect personal identifiers included in the data sources.

METHODS

We identified users of glucose-lowering drugs with residence in four federal states in Northern and Southern Germany (Bavaria, Bremen, Hamburg, Lower Saxony) in the German Pharmacoepidemiological Research Database (GePaRD) and assessed colorectal and thyroid cancer cases. Cancer registries of the federal states selected all colorectal and thyroid cancer cases between 2004 and 2015. A deterministic linkage approach was performed based on indirect personal identifiers such as year of birth, sex, area of residence, type of cancer and an absolute difference between the dates of cancer diagnosis in both data sources of at most 90 days. Results were compared to a probabilistic linkage using "direct" personal identifiers (gold standard).

RESULTS

The deterministic linkage procedure yielded a sensitivity of 71.8% for colorectal cancer and 66.6% for thyroid cancer. For thyroid cancer, the sensitivity improved when using only inpatient diagnosis to define cancer in GePaRD (71.4%). Specificity was always above 99%. Using the probabilistic linkage to define cancer cases, the risk for colorectal cancer was estimated 10 percentage points lower than when using the deterministic approach.

CONCLUSIONS

Sensitivity of the deterministic linkage approach appears to be too low to be considered as reasonable alternative to the probabilistic linkage procedure.