Accuracy of identifying hospital acquired venous thromboembolism by administrative coding: implications for big data and machine learning research

Implementing a Quality Intervention to Improve Confidence in Outpatient Venous Thromboembolism Management

INTRODUCTION

Guidelines recommend that patients with acute venous thromboembolism (VTE) represented by low-risk deep vein thrombosis (DVT) and pulmonary embolism (PE) receive initial treatment at home versus at the hospital, but a large percentage of these patients are not managed at home. This study assessed the effectiveness of a quality intervention on provider knowledge and confidence in evaluating outpatient treatment for patients with VTE in the emergency department (ED).

METHODS

A pilot program to overcome obstacles to outpatient VTE treatment in appropriate patients was initiated at Baylor Scott & White Health Temple ED. Subsequently, a formalized quality intervention with a targeted educational program was developed and delivered to ED providers. Provider surveys were administered pre- and post-quality intervention in order to assess clinical knowledge, confidence levels, and perceived barriers. Patient discharge information was extracted from electronic health records.

RESULTS

Twenty-five ED providers completed the pre- and post-surveys; 690 and 356 patients with VTE were included in the pre- and post-pilot and pre- and post-quality intervention periods, respectively. Many ED providers reported that a major barrier to discharging patients to outpatient care was the lack of available and adequate patient follow-up appointments. Notably, after the quality intervention, an increase in provider clinical knowledge and confidence scores was observed. Discharge rates for patients with VTE increased from 25.6% to 27.5% after the pilot intervention and increased from 28.5% to 29.9% after the quality intervention, but these differences were not statistically significant. Despite instantaneous uptick in discharge rates after the interventions, there was not a long-lasting effect.

CONCLUSION

Although the quality intervention led to improvements in provider clinical knowledge and confidence and identified barriers to discharging patients with VTE, discharge rates remained stable, underscoring the need for additional endeavors.

A scoping review of the methodological approaches used in retrospective chart reviews to validate adverse event rates in administrative data

Patient safety is a key quality issue for health systems. Healthcare acquired adverse events (AEs) compromise safety and quality; therefore, their reporting and monitoring is a patient safety priority. Although administrative datasets are potentially efficient tools for monitoring rates of AEs, concerns remain over the accuracy of their data. Chart review validation studies are required to explore the potential of administrative data to inform research and health policy. This review aims to present an overview of the methodological approaches and strategies used to validate rates of AEs in administrative data through chart review. This review was conducted in line with the Joanna Briggs Institute methodological framework for scoping reviews. Through database searches, 1054 sources were identified, imported into Covidence, and screened against the inclusion criteria. Articles that validated rates of AEs in administrative data through chart review were included. Data were extracted, exported to Microsoft Excel, arranged into a charting table, and presented in a tabular and descriptive format. Fifty-six studies were included. Most sources reported on surgical AEs; however, other medical specialties were also explored. Chart reviews were used in all studies; however, few agreed on terminology for the study design. Various methodological approaches and sampling strategies were used. Some studies used the Global Trigger Tool, a two-stage chart review method, whilst others used alternative single-, two-stage, or unclear approaches. The sources used samples of flagged charts (n = 24), flagged and random charts (n = 11), and random charts (n = 21). Most studies reported poor or moderate accuracy of AE rates. Some studies reported good accuracy of AE recording which highlights the potential of using administrative data for research purposes. This review highlights the potential for administrative data to provide information on AE rates and improve patient safety and healthcare quality. Nonetheless, further work is warranted to ensure that administrative data are accurate. The variation of methodological approaches taken, and sampling techniques used demonstrate a lack of consensus on best practice; therefore, further clarity and consensus are necessary to develop a more systematic approach to chart reviewing.

Evaluation of venous thromboembolism risk assessment models for hospital inpatients: the VTEAM evidence synthesis

Background

Pharmacological prophylaxis during hospital admission can reduce the risk of acquired blood clots (venous thromboembolism) but may cause complications, such as bleeding. Using a risk assessment model to predict the risk of blood clots could facilitate selection of patients for prophylaxis and optimise the balance of benefits, risks and costs.

Objectives

We aimed to identify validated risk assessment models and estimate their prognostic accuracy, evaluate the cost-effectiveness of different strategies for selecting hospitalised patients for prophylaxis, assess the feasibility of using efficient research methods and estimate key parameters for future research.

Design

We undertook a systematic review, decision-analytic modelling and observational cohort study conducted in accordance with Enhancing the QUAlity and Transparency Of health Research (EQUATOR) guidelines.

Setting

NHS hospitals, with primary data collection at four sites.

Participants

Medical and surgical hospital inpatients, excluding paediatric, critical care and pregnancy-related admissions.

Interventions

Prophylaxis for all patients, none and according to selected risk assessment models.

Main outcome measures

Model accuracy for predicting blood clots, lifetime costs and quality-adjusted life-years associated with alternative strategies, accuracy of efficient methods for identifying key outcomes and proportion of inpatients recommended prophylaxis using different models.

Results

We identified 24 validated risk assessment models, but low-quality heterogeneous data suggested weak accuracy for prediction of blood clots and generally high risk of bias in all studies. Decision-analytic modelling showed that pharmacological prophylaxis for all eligible is generally more cost-effective than model-based strategies for both medical and surgical inpatients, when valuing a quality-adjusted life-year at £20,000. The findings were more sensitive to uncertainties in the surgical population; strategies using risk assessment models were more cost-effective if the model was assumed to have a very high sensitivity, or the long-term risks of post-thrombotic complications were lower. Efficient methods using routine data did not accurately identify blood clots or bleeding events and several pre-specified feasibility criteria were not met. Theoretical prophylaxis rates across an inpatient cohort based on existing risk assessment models ranged from 13% to 91%.

Limitations

Existing studies may underestimate the accuracy of risk assessment models, leading to underestimation of their cost-effectiveness. The cost-effectiveness findings do not apply to patients with an increased risk of bleeding. Mechanical thromboprophylaxis options were excluded from the modelling. Primary data collection was predominately retrospective, risking case ascertainment bias.

Conclusions

Thromboprophylaxis for all patients appears to be generally more cost-effective than using a risk assessment model, in hospitalised patients at low risk of bleeding. To be cost-effective, any risk assessment model would need to be highly sensitive. Current evidence on risk assessment models is at high risk of bias and our findings should be interpreted in this context. We were unable to demonstrate the feasibility of using efficient methods to accurately detect relevant outcomes for future research.

Future work

Further research should evaluate routine prophylaxis strategies for all eligible hospitalised patients. Models that could accurately identify individuals at very low risk of blood clots (who could discontinue prophylaxis) warrant further evaluation.

Study registration

This study is registered as PROSPERO CRD42020165778 and Researchregistry5216.

Funding

This award was funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme (NIHR award ref: NIHR127454) and will be published in full in Health Technology Assessment; Vol. 28, No. 20. See the NIHR Funding and Awards website for further award information.

ClotCatcher: a novel natural language model to accurately adjudicate venous thromboembolism from radiology reports

INTRODUCTION

Accurate identification of venous thromboembolism (VTE) is critical to develop replicable epidemiological studies and rigorous predictions models. Traditionally, VTE studies have relied on international classification of diseases (ICD) codes which are inaccurate - leading to misclassification bias. Here, we developed ClotCatcher, a novel deep learning model that uses natural language processing to detect VTE from radiology reports.

METHODS

Radiology reports to detect VTE were obtained from patients admitted to Emory University Hospital (EUH) and Grady Memorial Hospital (GMH). Data augmentation was performed using the Google PEGASUS paraphraser. This data was then used to fine-tune ClotCatcher, a novel deep learning model. ClotCatcher was validated on both the EUH dataset alone and GMH dataset alone.

RESULTS

The dataset contained 1358 studies from EUH and 915 studies from GMH (n = 2273). The dataset contained 1506 ultrasound studies with 528 (35.1%) studies positive for VTE, and 767 CT studies with 91 (11.9%) positive for VTE. When validated on the EUH dataset, ClotCatcher performed best (AUC = 0.980) when trained on both EUH and GMH dataset without paraphrasing. When validated on the GMH dataset, ClotCatcher performed best (AUC = 0.995) when trained on both EUH and GMH dataset with paraphrasing.

CONCLUSION

ClotCatcher, a novel deep learning model with data augmentation rapidly and accurately adjudicated the presence of VTE from radiology reports. Applying ClotCatcher to large databases would allow for rapid and accurate adjudication of incident VTE. This would reduce misclassification bias and form the foundation for future studies to estimate individual risk for patient to develop incident VTE.

Diabetes status and other factors as correlates of risk for thrombotic and thromboembolic events during SARS-CoV-2 infection: A nationwide retrospective case-control study using Cerner Real-World Data™

OBJECTIVES

We sought to examine in individuals with SARS-CoV-2 infection whether risk for thrombotic and thromboembolic events (TTE) is modified by presence of a diabetes diagnosis. Furthermore, we analysed whether differential risk for TTEs exists in type 1 diabetes mellitus (T1DM) versus type 2 diabetes mellitus (T2DM).

DESIGN

Retrospective case-control study.

SETTING

The December 2020 version of the Cerner Real-World Data COVID-19 database is a deidentified, nationwide database containing electronic medical record (EMR) data from 87 US-based health systems.

PARTICIPANTS

We analysed EMR data for 322 482 patients >17 years old with suspected or confirmed SARS-CoV-2 infection who received care between December 2019 and mid-September 2020. Of these, 2750 had T1DM; 57 811 had T2DM; and 261 921 did not have diabetes.

OUTCOME

TTE, defined as presence of a diagnosis code for myocardial infarction, thrombotic stroke, pulmonary embolism, deep vein thrombosis or other TTE.

RESULTS

Odds of TTE were substantially higher in patients with T1DM (adjusted OR (AOR) 2.23 (1.93-2.59)) and T2DM (AOR 1.52 (1.46-1.58)) versus no diabetes. Among patients with diabetes, odds of TTE were lower in T2DM versus T1DM (AOR 0.84 (0.72-0.98)).

CONCLUSIONS

Risk of TTE during COVID-19 illness is substantially higher in patients with diabetes. Further, risk for TTEs is higher in those with T1DM versus T2DM. Confirmation of increased diabetes-associated clotting risk in future studies may warrant incorporation of diabetes status into SARS-CoV-2 infection treatment algorithms.

Accuracy of efficient data methods to determine the incidence of hospital-acquired thrombosis and major bleeding in medical and surgical inpatients: a multicentre observational cohort study in four UK hospitals

OBJECTIVES

We evaluated the accuracy of using routine health service data to identify hospital-acquired thrombosis (HAT) and major bleeding events (MBE) compared with a reference standard of case note review.

DESIGN

A multicentre observational cohort study.

SETTING

Four acute hospitals in the UK.

PARTICIPANTS

A consecutive unselective cohort of general medical and surgical patients requiring hospitalisation for a period of >24 hours during the calendar year 2021. We excluded paediatric, obstetric and critical care patients due to differential risk profiles.

INTERVENTIONS

We compared preidentified sources of routinely collected information (using hospital coding data and local contractually mandated thrombosis datasets) to data extracted from case notes using a predesigned workflow methodology.

PRIMARY AND SECONDARY OUTCOME MEASURES

We defined HAT as objectively confirmed venous thromboembolism occurring during hospital stay or within 90 days of discharge and MBE as per international consensus.

RESULTS

We were able to source all necessary routinely collected outcome data for 87% of 2008 case episodes reviewed. The sensitivity of hospital coding data (International Classification of Diseases 10th Revision, ICD-10) for the diagnosis of HAT and MBE was 62% (95% CI, 54 to 69) and 38% (95% CI, 27 to 50), respectively. Sensitivity improved to 81% (95% CI, 75 to 87) when using local thrombosis data sets.

CONCLUSIONS

Using routinely collected data appeared to miss a substantial proportion of outcome events, when compared with case note review. Our study suggests that currently available routine data collection methods in the UK are inadequate to support efficient study designs in venous thromboembolism research.

TRIAL REGISTRATION NUMBER

NIHR127454.

Real-world Health Data and Precision for the Diagnosis of Acute Kidney Injury, Acute-on-Chronic Kidney Disease, and Chronic Kidney Disease: Observational Study

Background

The criteria for the diagnosis of kidney disease outlined in the Kidney Disease: Improving Global Outcomes guidelines are based on a patient’s current, historical, and baseline data. The diagnosis of acute kidney injury, chronic kidney disease, and acute-on-chronic kidney disease requires previous measurements of creatinine, back-calculation, and the interpretation of several laboratory values over a certain period. Diagnoses may be hindered by unclear definitions of the individual creatinine baseline and rough ranges of normal values that are set without adjusting for age, ethnicity, comorbidities, and treatment. The classification of correct diagnoses and sufficient staging improves coding, data quality, reimbursement, the choice of therapeutic approach, and a patient’s outcome.

Objective

In this study, we aim to apply a data-driven approach to assign diagnoses of acute, chronic, and acute-on-chronic kidney diseases with the help of a complex rule engine.

Methods

Real-time and retrospective data from the hospital’s clinical data warehouse of inpatient and outpatient cases treated between 2014 and 2019 were used. Delta serum creatinine, baseline values, and admission and discharge data were analyzed. A Kidney Disease: Improving Global Outcomes–based SQL algorithm applied specific diagnosis-based International Classification of Diseases (ICD) codes to inpatient stays. Text mining on discharge documentation was also conducted to measure the effects on diagnosis.

Results

We show that this approach yielded an increased number of diagnoses (4491 cases in 2014 vs 11,124 cases of ICD-coded kidney disease and injury in 2019) and higher precision in documentation and coding. The percentage of unspecific ICD N19-coded diagnoses of N19 codes generated dropped from 19.71% (1544/7833) in 2016 to 4.38% (416/9501) in 2019. The percentage of specific ICD N18-coded diagnoses of N19 codes generated increased from 50.1% (3924/7833) in 2016 to 62.04% (5894/9501) in 2019.

Conclusions

Our data-driven method supports the process and reliability of diagnosis and staging and improves the quality of documentation and data. Measuring patient outcomes will be the next step in this project.