Validation of a novel method to identify healthcare-associated infections

Validation studies of Japanese administrative health care data: A scoping review

PURPOSE

Large-scale administrative health care databases are increasingly being utilized for research. However, there has not been much literature that validated administrative data in Japan; a previous review identified six validation studies published between 2011 and 2017. We conducted a literature review of studies that assessed the validity of Japanese administrative health care data.

METHODS

We searched for studies published by March 2022 that compared individual-level administrative data with a reference standard from another data source, as well as studies that validated administrative data using other data within the same database. The eligible studies were also summarized based on characteristics which included data types, settings, reference standard used, numbers of patients, and conditions validated.

RESULTS

There were 36 eligible studies, including 29 that used external reference standard and seven that validated administrative data using other data within the same database. Chart review was the reference standard in 21 studies (range of the numbers of patients, 72-1674; 11 studies conducted in single institutions and nine studies in 2-5 institutions). Five studies used a disease registry as the reference standard. Diagnoses of cardiovascular diseases, cancer, and diabetes were frequently evaluated.

CONCLUSIONS

Validation studies are being conducted at an increasing rate in Japan, although most of them are small scale. Further large-scale comprehensive validation studies are necessary to effectively utilize the databases for research.

Validation study on definition of cause of death in Japanese claims data

Identifying the cause of death is important for the study of end-of-life patients using claims data in Japan. However, the validity of how cause of death is identified using claims data remains unknown. Therefore, this study aimed to verify the validity of the method used to identify the cause of death based on Japanese claims data. Our study population included patients who died at two institutions between January 1, 2018 and December 31, 2019. Claims data consisted of medical data and Diagnosis Procedure Combination (DPC) data, and five definitions developed from disease classification in each dataset were compared with death certificates. Nine causes of death, including cancer, were included in the study. The definition with the highest positive predictive values (PPVs) and sensitivities in this study was the combination of "main disease" in both medical and DPC data. For cancer, these definitions had PPVs and sensitivities of > 90%. For heart disease, these definitions had PPVs of > 50% and sensitivities of > 70%. For cerebrovascular disease, these definitions had PPVs of > 80% and sensitivities of> 70%. For other causes of death, PPVs and sensitivities were < 50% for most definitions. Based on these results, we recommend definitions with a combination of "main disease" in both medical and DPC data for cancer and cerebrovascular disease. However, a clear argument cannot be made for other causes of death because of the small sample size. Therefore, the results of this study can be used with confidence for cancer and cerebrovascular disease but should be used with caution for other causes of death.

Accuracy of algorithms to identify patients with a diagnosis of major cancers and cancer-related adverse events in an administrative database: a validation study in an acute care hospital in Japan

OBJECTIVES

Validation studies in oncology are limited in Japan. This study was conducted to evaluate the accuracy of diagnosis and adverse event (AE) definitions for specific cancers in a Japanese health administrative real-world database (RWD).

DESIGN AND SETTING

Retrospective observational validation study to assess the diagnostic accuracy of electronic medical records (EMRs) and claim coding regarding oncology diagnosis and AEs based on medical record review in the RWD. The sensitivity and positive predictive value (PPV) with 95% CIs were calculated.

PARTICIPANTS

The validation cohort included patients with lung (n=2257), breast (n=1121), colorectal (n=1773), ovarian (n=216) and bladder (n=575) cancer who visited the hospital between January 2014 and December 2018, and those with prostate cancer (n=3491) visiting between January 2009 and December 2018, who were identified using EMRs.

OUTCOMES

Key outcomes included primary diagnosis, deaths and AEs.

RESULTS

For primary diagnosis, sensitivity and PPV for the respective cancers were as follows: lung, 100.0% (96.6 to 100.0) and 81.0% (74.9 to 86.2); breast, 100.0% (96.3 to 100.0) and 74.0% (67.3 to 79.9); colorectal, 100.0% (96.6 to 100.0) and 80.5% (74.3 to 85.8); ovarian, 89.8% (77.8 to 96.6) and 75.9% (62.8 to 86.1); bladder, 78.6% (63.2 to 89.7) and 67.3% (52.5 to 0.1); prostate, 100.0% (93.2 to 100.0) and 79.0% (69.7 to 86.5). Sensitivity and PPV for death were as follows: lung, 97.0% (84.2 to 99.9) and 100.0% (84.2 to 100.0); breast, 100.0% (1.3 to 100.0) and 100.0% (1.3 to 100.0); colorectal, 100.0% (28.4 to 100.0) and 100.0% (28.4 to 100.0); ovarian, 100.0% (35.9 to 100.0) and 100.0% (35.9 to 100.0); bladder, 100.0% (9.4-100.0) and 100.0% (9.4 to 100.0); prostate, 75.0% (19.4 to 99.4) and 100.0% (19.4 to 100.0). Overall, PPV tended to be low, with the definition based on International Classification of Diseases, 10th revision alone for AEs.

CONCLUSION

Diagnostic accuracy was not so high, and therefore needs to be further investigated.

TRIAL REGISTRATION NUMBER

University Hospital Medical Information Network (UMIN) Clinical Trials Registry (UMIN000039345).

Validation Study of Algorithms to Identify Malignant Tumors and Serious Infections in a Japanese Administrative Healthcare Database

BACKGROUND

This retrospective observational study validated case-finding algorithms for malignant tumors and serious infections in a Japanese administrative healthcare database.

METHODS

Random samples of possible cases of each disease (January 2015-January 2018) from two hospitals participating in the Medical Data Vision Co., Ltd. (MDV) database were identified using combinations of ICD-10 diagnostic codes and other procedural/billing codes. For each disease, two physicians identified true cases among the random samples of possible cases by medical record review; a third physician made the final decision in cases where the two physicians disagreed. The accuracy of case-finding algorithms was assessed using positive predictive value (PPV) and sensitivity.

RESULTS

There were 2,940 possible cases of malignant tumor; 180 were randomly selected and 108 were identified as true cases after medical record review. One case-finding algorithm gave a high PPV (64.1%) without substantial loss in sensitivity (90.7%) and included ICD-10 codes for malignancy and photographing/imaging. There were 3,559 possible cases of serious infection; 200 were randomly selected and 167 were identified as true cases after medical record review. Two case-finding algorithms gave a high PPV (85.6%) with no loss in sensitivity (100%). Both case-finding algorithms included the relevant diagnostic code and immunological infection test/other related test and, of these, one also included pathological diagnosis within 1 month of hospitalization.

CONCLUSIONS

The case-finding algorithms in this study showed good PPV and sensitivity for identification of cases of malignant tumors and serious infections from an administrative healthcare database in Japan.

History and Profile of Diagnosis Procedure Combination (DPC): Development of a Real Data Collection System for Acute Inpatient Care in Japan

DPC, which is an acronym for “Diagnosis Procedure Combination,” is a patient classification method developed in Japan for inpatients in the acute phase of illness. It was developed as a measuring tool intended to make acute inpatient care transparent, aiming at standardization of Japanese medical care, as well as evaluation and improvement of its quality. Subsequently, this classification method came to be used in the Japanese medical service reimbursement system for acute inpatient care and appropriate allocation of medical resources. Furthermore, it has recently contributed to the development and maintenance of an appropriate medical care provision system at a regional level, which is accomplished based on DPC data used for patient classification. In this paper, we first provide an overview of DPC. Next, we will look back at over 15 years of DPC history; in particular, we will explore how DPC has been refined to become an appropriate medical service reimbursement system. Finally, we will introduce an outline of DPC-related research, starting with research using DPC data.

Effect of methicillin-resistant Staphylococcus aureus in Japan

BACKGROUND

Methicillin-resistant Staphylococcus aureus (MRSA) is the most common antimicrobial-resistant organism identified in Japanese health care facilities. This study analyzed the clinical and economic burdens attributable to methicillin resistance in S aureus in Japanese hospitals.

METHODS

We retrospectively investigated data from 14,905 inpatients of 57 hospitals combined with data from nosocomial infection surveillance and administrative claim databases. The participants were inpatients with admission from April 1, 2014, to discharge on March 31, 2016. The outcomes were evaluated according to length of stay, hospital charges, and in-hospital mortality. We compared the disease burden of MRSA infections with methicillin-susceptible S aureus (MSSA) infections based on patients' characteristics and onset periods.

RESULTS

We categorized 7,188 and 7,717 patients into MRSA and MSSA groups, respectively. The adjusted effects of the MRSA group were 1.03-fold (95% confidence interval [CI] 1.01-1.05) and 1.04-fold (95% CI, 1.01-1.06), respectively, with an odds ratio of 1.14 (95% CI, 1.02-1.27).

CONCLUSIONS

The results of this study found that patient severity and onset delays were positively associated with both MRSA and burden and that the effect of methicillin resistance remained significant after adjustment.

Estimating the disease burden of methicillin-resistant Staphylococcus aureus in Japan: Retrospective database study of Japanese hospitals

OBJECTIVES

The nationwide impact of antimicrobial-resistant infections on healthcare facilities throughout Japan has yet to be examined. This study aimed to estimate the disease burden of methicillin-resistant Staphylococcus aureus (MRSA) infections in Japanese hospitals.

DESIGN

Retrospective analysis of inpatients comparing outcomes between subjects with and without MRSA infection.

DATA SOURCE

A nationwide administrative claims database.

SETTING

1133 acute care hospitals throughout Japan.

PARTICIPANTS

All surgical and non-surgical inpatients who were discharged between April 1, 2014 and March 31, 2015.

MAIN OUTCOME MEASURES

Disease burden was assessed using hospitalization costs, length of stay, and in-hospital mortality. Using a unique method of infection identification, we categorized patients into an anti-MRSA drug group and a control group based on anti-MRSA drug utilization. To estimate the burden of MRSA infections, we calculated the differences in outcome measures between these two groups. The estimates were extrapolated to all 1584 acute care hospitals in Japan that have adopted a prospective payment system.

RESULTS

We categorized 93 838 patients into the anti-MRSA drug group and 2 181 827 patients into the control group. The mean hospitalization costs, length of stay, and in-hospital mortality of the anti-MRSA drug group were US$33 548, 75.7 days, and 22.9%, respectively; these values were 3.43, 2.95, and 3.66 times that of the control group, respectively. When extrapolated to the 1584 hospitals, the total incremental burden of MRSA was estimated to be US$2 billion (3.41% of total hospitalization costs), 4.34 million days (3.02% of total length of stay), and 14.3 thousand deaths (3.62% of total mortality).

CONCLUSIONS

This study quantified the approximate disease burden of MRSA infections in Japan. These findings can inform policymakers on the burden of antimicrobial-resistant infections and support the application of infection prevention programs.

Accuracy of administrative data for surveillance of healthcare-associated infections: a systematic review

OBJECTIVE

Measuring the incidence of healthcare-associated infections (HAI) is of increasing importance in current healthcare delivery systems. Administrative data algorithms, including (combinations of) diagnosis codes, are commonly used to determine the occurrence of HAI, either to support within-hospital surveillance programmes or as free-standing quality indicators. We conducted a systematic review evaluating the diagnostic accuracy of administrative data for the detection of HAI.

METHODS

Systematic search of Medline, Embase, CINAHL and Cochrane for relevant studies (1995-2013). Methodological quality assessment was performed using QUADAS-2 criteria; diagnostic accuracy estimates were stratified by HAI type and key study characteristics.

RESULTS

57 studies were included, the majority aiming to detect surgical site or bloodstream infections. Study designs were very diverse regarding the specification of their administrative data algorithm (code selections, follow-up) and definitions of HAI presence. One-third of studies had important methodological limitations including differential or incomplete HAI ascertainment or lack of blinding of assessors. Observed sensitivity and positive predictive values of administrative data algorithms for HAI detection were very heterogeneous and generally modest at best, both for within-hospital algorithms and for formal quality indicators; accuracy was particularly poor for the identification of device-associated HAI such as central line associated bloodstream infections. The large heterogeneity in study designs across the included studies precluded formal calculation of summary diagnostic accuracy estimates in most instances.

CONCLUSIONS

Administrative data had limited and highly variable accuracy for the detection of HAI, and their judicious use for internal surveillance efforts and external quality assessment is recommended. If hospitals and policymakers choose to rely on administrative data for HAI surveillance, continued improvements to existing algorithms and their robust validation are imperative.

Which Kind of Provider's Operation Volumes Matters? Associations between CABG Surgical Site Infection Risk and Hospital and Surgeon Operation Volumes among Medical Centers in Taiwan

BACKGROUND

Volume-infection relationships have been examined for high-risk surgical procedures, but the conclusions remain controversial. The inconsistency might be due to inaccurate identification of cases of infection and different methods of categorizing service volumes. This study takes coronary artery bypass graft (CABG) surgical site infections (SSIs) as an example to examine whether a relationship exists between operation volumes and SSIs, when different SSIs case identification, definitions and categorization methods of operation volumes were implemented.

METHODS

A population-based cross-sectional multilevel study was conducted. A total of 7,007 patients who received CABG surgery between 2006 and 2008 from 19 medical centers in Taiwan were recruited. SSIs associated with CABG surgery were identified using International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9 CM) codes and a Classification and Regression Trees (CART) model. Two definitions of surgeon and hospital operation volumes were used: (1) the cumulative CABG operation volumes within the study period; and (2) the cumulative CABG operation volumes in the previous one year before each CABG surgery. Operation volumes were further treated in three different ways: (1) a continuous variable; (2) a categorical variable based on the quartile; and (3) a data-driven categorical variable based on k-means clustering algorithm. Furthermore, subgroup analysis for comorbidities was also conducted.

RESULTS

This study showed that hospital volumes were not significantly associated with SSIs, no matter which definitions or categorization methods of operation volume, or SSIs case identification approaches were used. On the contrary, the relationships between surgeon's volumes varied. Most of the models demonstrated that the low-volume surgeons had higher risk than high-volume surgeons.

CONCLUSION

Surgeon volumes were more important than hospital volumes in exploring the relationship between CABG operation volumes and SSIs in Taiwan. However, the relationships were not robust. Definitions and categorization methods of operation volume and correct identification of SSIs are important issues for future research.

Is it possible to identify cases of coronary artery bypass graft postoperative surgical site infection accurately from claims data?

Background

Claims data has usually been used in recent studies to identify cases of healthcare-associated infection. However, several studies have indicated that the ICD-9-CM codes might be inappropriate for identifying such cases from claims data; therefore, several researchers developed alternative identification models to correctly identify more cases from claims data. The purpose of this study was to investigate three common approaches to develop alternative models for the identification of cases of coronary artery bypass graft (CABG) surgical site infection, and to compare the performance between these models and the ICD-9-CM model.

Methods

The 2005–2008 National Health Insurance claims data and healthcare-associated infection surveillance data from two medical centers were used in this study for model development and model verification. In addition to the use of ICD-9-CM codes, this study also used classification algorithms, a multivariable regression model, and a decision tree model in the development of alternative identification models. In the classification algorithms, we defined three levels (strict, moderate, and loose) of the criteria in terms of their strictness. Sensitivity, specificity, positive predictive value, negative predictive value, and accuracy were used to evaluate the performance of each model.

Results

The ICD-9-CM-based model showed good specificity and negative predictive value, but sensitivity and positive predictive value were poor. Performances of the other models were varied, except for negative predictive value. Among the models, the performance of the decision tree model was excellent, especially in terms of positive predictive value.

Conclusion

The accuracy of identification of cases of CABG surgical site infection is an important issue in claims data. Use of the decision tree model to identify such cases can improve the accuracy of patient-level outcome research. This model should be considered when performing future research using claims data.