A machine learning approach to create blocking criteria for record linkage

Towards malaria elimination: a case-control study to assess associated factors to malaria relapses in the extra-Amazon Region of Brazil from 2008 to 2019

BACKGROUND

Malaria is an infectious disease caused by the Plasmodium species and is a global burden. When not treated correctly, it can reemerge as a relapse or recrudescence. Malaria relapse cases can contribute to maintaining active transmission chains and can influence the patient to develop severe malaria, potentially leading to hospitalization or death. The objective of this study is to estimate the number of malaria relapse cases in the extra-Amazon region of Brazil and to investigate the associated factors.

METHODS

This is a case-control study that analyses malaria infections caused by Plasmodium vivax, as reported in Notifiable Diseases Information System (Sinan) for the Brazilian extra-Amazon region (an area not endemic for the disease) from 2008 to 2019. For the identification of relapse cases, deduplication record linkage processes in R software were used. Malaria relapses were defined as the case group, and new malaria infections were defined as the control group. Logistic regression models were used to assess associated factors.

RESULTS

Of the 711 malaria relapses, 589 (82.8%) were first relapses. Most relapses (71.6%) occurred between 30 and 120 days after the previous infection. Malaria relapses are spread throughout the extra-Amazon region, with a higher concentration near big cities. Driver occupation was found to be a common risk factor compared to other occupations, along with asymptomatic individuals. Other associated factors were: being infected in the Brazilian Amazon region, having follow-ups for malaria relapses, and having parasite density of the previous infection higher than 10,000 parasites per mm3.

CONCLUSIONS

This study provides evidence that allows malaria health surveillance services to direct their efforts to monitor cases of malaria in the highest risk segments identified in this study, particularly in the period between 30 and 120 days after being infected and treated. Relapses were associated to driver occupation, absence of symptoms, infection in endemic areas of Brazil, being detected through active surveillance or routine follow-up actions, and with parasitaemia greater than 10,000 parasites per mm3 in the previous infection. Improving cases follow-up is essential for preventing relapses.

Locality sensitive blocking (LSB): A robust blocking technique for data deduplication

Data deduplication is process of discovering multiple representations of same entity in an information system. Blocking has been a benchmark technique for avoiding the pair-wise record comparisons in data deduplication. Standard blocking (SB) aims at putting the potential duplicate records in the same block on the basis of a blocking key. Afterwards, the detailed comparisons are made only among the records residing in the same block. The selection of blocking key is a tedious process that involves exponential alternatives. The outcome of SB varies considerably with a change in blocking key. To this end, we have proposed a robust blocking technique called Locality Sensitive Blocking (LSB) that does not require the selection of blocking key. The experimental results show an increase of up to 0.448 in F-score as compared with SB. Furthermore, it is found that LSB is more robust towards blocking parameters and data noise.

Using supervised machine learning to identify efficient blocking schemes for record linkage

Record linkage enables survey data to be integrated with other data sources, expanding the analytic potential of both sources. However, depending on the number of records being linked, the processing time can be prohibitive. This paper describes a case study using a supervised machine learning algorithm, known as the Sequential Coverage Algorithm (SCA). The SCA was used to develop the join strategy for two data sources, the National Center for Health Statistics' (NCHS) 2016 National Hospital Care Survey (NHCS) and the Center for Medicare & Medicaid Services (CMS) Enrollment Database (EDB), during record linkage. Due to the size of the CMS data, common record joining methods (i.e. blocking) were used to reduce the number of pairs that need to be evaluated to identify the vast majority of matches. NCHS conducted a case study examining how the SCA improved the efficiency of blocking. This paper describes how the SCA was used to design the blocking used in this linkage.

A proficient cost reduction framework for de-duplication of records in data integration

Background

Record de-duplication is a process of identifying the records referring to the same entity. It has a pivotal role in data mining applications, which involves the integration of multiple data sources and data cleansing. It has been a challenging task due to its computational complexity and variations in data representations across different data sources. Blocking and windowing are the commonly used methods for reducing the number of record comparisons during record de-duplication. Both blocking and windowing require tuning of a certain set of parameters, such as the choice of a particular variant of blocking or windowing, the selection of appropriate window size for different datasets etc.

Methods

In this paper, we have proposed a framework that employs blocking and windowing techniques in succession, such that figuring out the parameters is not required. We have also evaluated the impact of different configurations on dirty and massively dirty datasets. To evaluate the proposed framework, experiments are performed using Febrl (Freely Extensible Biomedical Record Linkage).

Results

The proposed framework is comprehensively evaluated using a variety of quality and complexity parameters such as reduction ratio, precision, recall etc. It is observed that the proposed framework significantly reduces the number of record comparisons.

Conclusions

The selection of the linkage key is a critical performance factor for record linkage.

Electronic supplementary material

The online version of this article (doi:10.1186/s12911-016-0280-9) contains supplementary material, which is available to authorized users.