Prediction Accuracy With Electronic Medical Records Versus Administrative Claims

Association Between Neighborhood Context and Psychotropic Polypharmacy Use Among High-Need Children

OBJECTIVE

The authors explored whether neighborhood context is associated with psychotropic polypharmacy and psychotherapy among a cohort of children with high needs for psychiatric and general medical care.

METHODS

Electronic health record data from a large health care system were used in a cross-sectional design to examine psychotropic polypharmacy and psychotherapy in 2015-2019 among children ages 2-17 years (N=4,017) with geocoded addresses. Inclusion criteria were a diagnosis of a mental health condition, an intellectual and developmental disability, or a complex medical condition and one or more clinical encounters annually over the study period. Polypharmacy was defined as two or more psychotropic drug class prescriptions concurrently for ≥60 days. Psychotherapy was defined as receipt of any psychotherapy or adaptive behavior treatment. Neighborhood context (health, environment, education, and wealth) was measured with the Child Opportunity Index. Multilevel generalized linear mixed models with random intercept for census tracts were used to assess the associations between individual and neighborhood characteristics and psychotropic polypharmacy and psychotherapy.

RESULTS

Moderate (vs. low) child opportunity was associated with higher odds of polypharmacy (adjusted OR [AOR]=1.79, 95% CI=1.19-2.67). High (vs. low) child opportunity was associated with higher odds of psychotherapy (AOR=2.15, 95% CI=1.43-3.21). Black (vs. White) race (AOR=0.51, 95% CI=0.37-0.71) and Hispanic ethnicity (AOR=0.44, 95% CI=0.26-0.73) were associated with lower odds of polypharmacy.

CONCLUSIONS

Among high-need children, neighborhood Child Opportunity Index, race, and ethnicity were significantly associated with treatment outcomes in analyses adjusted for clinical factors. The findings underscore concerns about structural disparities and systemic racism and raise questions about access.

Short- and Long-Term Mortality of Hospitalized Patients With Autoimmune Rheumatic Diseases and Serious Infections: A National Cohort Study

OBJECTIVE

Infectious conditions are a significant cause of mortality in autoimmune rheumatic diseases (ARD). Among patients hospitalized with an infection, we compared in-hospital and long-term (3-year) mortality between those with and without ARD.

METHODS

This retrospective analysis included members of the largest health maintenance organization in Israel, aged > 18 years at the first episode of infection, who required hospitalization during 2003-2019. We compared in-hospital mortality and the results of a 3-year landmark analysis of those who survived the index hospitalization between patients with ARD, according to disease subgroups, and patients without ARD. Additionally, we compared mortality outcomes among patients with ARD, according to subgroup diagnosis, matched in a 1:3 ratio by age, sex, and ethnicity to patients without ARD.

RESULTS

Included were 365,247 patients who were admitted for the first time with the diagnosis of a serious infection. Of these, we identified 9755 with rheumatoid arthritis (RA), 1351 with systemic lupus erythematosus, 2120 with spondyloarthritis (SpA), 584 with systemic sclerosis, and 3214 with vasculitis. In a matched multivariate analysis, the risk for in-hospital mortality was lower among patients with RA (odds ratio [OR] 0.89, 95% CI 0.81-0.97) and SpA (OR 0.77, 95% CI 0.63-0.94). In a similar analysis, the risk of 3-year mortality was lower among patients with RA (hazard ratio [HR] 0.82, 95% CI 0.78-0.86) and vasculitis (HR 0.86, 95% CI 0.80-0.93).

CONCLUSION

Among patients hospitalized for an infection, the risk of in-hospital and 3-year mortality was not increased among those with ARD compared to those without ARD.

The validity of electronic health data for measuring smoking status: a systematic review and meta-analysis

BACKGROUND

Smoking is a risk factor for many chronic diseases. Multiple smoking status ascertainment algorithms have been developed for population-based electronic health databases such as administrative databases and electronic medical records (EMRs). Evidence syntheses of algorithm validation studies have often focused on chronic diseases rather than risk factors. We conducted a systematic review and meta-analysis of smoking status ascertainment algorithms to describe the characteristics and validity of these algorithms.

METHODS

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines were followed. We searched articles published from 1990 to 2022 in EMBASE, MEDLINE, Scopus, and Web of Science with key terms such as validity, administrative data, electronic health records, smoking, and tobacco use. The extracted information, including article characteristics, algorithm characteristics, and validity measures, was descriptively analyzed. Sources of heterogeneity in validity measures were estimated using a meta-regression model. Risk of bias (ROB) in the reviewed articles was assessed using the Quality Assessment of Diagnostic Accuracy Studies-2 tool.

RESULTS

The initial search yielded 2086 articles; 57 were selected for review and 116 algorithms were identified. Almost three-quarters (71.6%) of algorithms were based on EMR data. The algorithms were primarily constructed using diagnosis codes for smoking-related conditions, although prescription medication codes for smoking treatments were also adopted. About half of the algorithms were developed using machine-learning models. The pooled estimates of positive predictive value, sensitivity, and specificity were 0.843, 0.672, and 0.918 respectively. Algorithm sensitivity and specificity were highly variable and ranged from 3 to 100% and 36 to 100%, respectively. Model-based algorithms had significantly greater sensitivity (p = 0.006) than rule-based algorithms. Algorithms for EMR data had higher sensitivity than algorithms for administrative data (p = 0.001). The ROB was low in most of the articles (76.3%) that underwent the assessment.

CONCLUSIONS

Multiple algorithms using different data sources and methods have been proposed to ascertain smoking status in electronic health data. Many algorithms had low sensitivity and positive predictive value, but the data source influenced their validity. Algorithms based on machine-learning models for multiple linked data sources have improved validity.

The Shieh Score as a Risk Assessment Instrument for Reducing Hospital-Acquired Pressure Injuries: A Prospective Cohort Study

PURPOSE

The purpose of this study was to evaluate the Shieh Score's effectiveness in decreasing the rate of hospital-acquired pressure injuries when combined with an early warning notification system and standard order set of preventative measures.

DESIGN

This was a prospective cohort study.

SUBJECTS AND SETTING

This target population was nonpregnant, adult, hospitalized patients on inpatient and observation status at a tertiary hospital (Kaiser Permanente, Baldwin Park, California) during the 2020 year of the COVID-19 pandemic.

METHODS

A new, risk assessment instrument, the Shieh Score, was developed in 2019 to predict hospitalized patients at high risk for pressure injuries. Data collection occurred between January 21, 2020, and December 31, 2020. When a hospital patient met the high-risk criteria for the Shieh Score, a provider-ordered pink-colored sheet of paper titled "Skin at Risk" was hung at the head of the bed and a standard order set of pressure injury preventative measures was implemented by nursing staff.

RESULTS

Implementation of the program (Shieh Score, early warning system, and standard order set for preventive interventions) resulted in a 38% reduction in the annual hospital-acquired pressure injury rate from a mean incidence rate of 1.03 to 0.64 hospital-acquired pressure injuries per 1000 patient-days measured for the year 2020.

CONCLUSION

The Shieh Score is a pressure injury risk assessment instrument, which effectively identifies patients at high risk for hospital-acquired pressure injuries and decreases the hospital-acquired pressure injury rate when combined with an early warning notification system and standard order set.

The shaky foundations of large language models and foundation models for electronic health records

The success of foundation models such as ChatGPT and AlphaFold has spurred significant interest in building similar models for electronic medical records (EMRs) to improve patient care and hospital operations. However, recent hype has obscured critical gaps in our understanding of these models' capabilities. In this narrative review, we examine 84 foundation models trained on non-imaging EMR data (i.e., clinical text and/or structured data) and create a taxonomy delineating their architectures, training data, and potential use cases. We find that most models are trained on small, narrowly-scoped clinical datasets (e.g., MIMIC-III) or broad, public biomedical corpora (e.g., PubMed) and are evaluated on tasks that do not provide meaningful insights on their usefulness to health systems. Considering these findings, we propose an improved evaluation framework for measuring the benefits of clinical foundation models that is more closely grounded to metrics that matter in healthcare.

Real-world utilization of SARS-CoV-2 serological testing in RNA positive patients across the United States

BACKGROUND

As diagnostic tests for COVID-19 were broadly deployed under Emergency Use Authorization, there emerged a need to understand the real-world utilization and performance of serological testing across the United States.

METHODS

Six health systems contributed electronic health records and/or claims data, jointly developed a master protocol, and used it to execute the analysis in parallel. We used descriptive statistics to examine demographic, clinical, and geographic characteristics of serology testing among patients with RNA positive for SARS-CoV-2.

RESULTS

Across datasets, we observed 930,669 individuals with positive RNA for SARS-CoV-2. Of these, 35,806 (4%) were serotested within 90 days; 15% of which occurred <14 days from the RNA positive test. The proportion of people with a history of cardiovascular disease, obesity, chronic lung, or kidney disease; or presenting with shortness of breath or pneumonia appeared higher among those serotested compared to those who were not. Even in a population of people with active infection, race/ethnicity data were largely missing (>30%) in some datasets-limiting our ability to examine differences in serological testing by race. In datasets where race/ethnicity information was available, we observed a greater distribution of White individuals among those serotested; however, the time between RNA and serology tests appeared shorter in Black compared to White individuals. Test manufacturer data was available in half of the datasets contributing to the analysis.

CONCLUSION

Our results inform the underlying context of serotesting during the first year of the COVID-19 pandemic and differences observed between claims and EHR data sources-a critical first step to understanding the real-world accuracy of serological tests. Incomplete reporting of race/ethnicity data and a limited ability to link test manufacturer data, lab results, and clinical data challenge the ability to assess the real-world performance of SARS-CoV-2 tests in different contexts and the overall U.S. response to current and future disease pandemics.

Predicting mortality and visualizing health care spending by predicted mortality in Danes over age 65

Health care expenditure in the last year of life makes up a high proportion of medical spending across the world. This is often framed as waste, but this framing is only meaningful if it is known at the time of treatment who will go on to die. We analyze the distribution of health care spending by predicted mortality for the Danish population over age 65 over the year 2016, with one-year mortality predicted by a machine learning model based on sociodemographics and use of health care services for the two years before entry into follow-up. While a reasonably good model can be built, extremely few individuals have high ex-ante probability of dying, and those with a predicted mortality of more than 50% account for only 2.8% of total health care expenditure. Decedents outspent survivors by a factor of more than ten, but compared to survivors with similar predicted mortality they spent only 2.5 times as much. Our results suggest that while spending in the last year of life is indeed high, this is nearly all spent in situations where there is a reasonable expectation that the patient can survive.

Administrative Data Is Insufficient to Identify Near-Future Critical Illness: A Population-Based Retrospective Cohort Study

Background

Prediction of future critical illness could render it practical to test interventions seeking to avoid or delay the coming event.

Objective

Identify adults having >33% probability of near-future critical illness.

Research Design

Retrospective cohort study, 2013-2015.

Subjects

Community-dwelling residents of Manitoba, Canada, aged 40-89 years.

Measures

The outcome was a near-future critical illness, defined as intensive care unit admission with invasive mechanical ventilation, or non-palliative death occurring 30-180 days after 1 April each year. By dividing the data into training and test cohorts, a Classification and Regression Tree analysis was used to identify subgroups with ≥33% probability of the outcome. We considered 72 predictors including sociodemographics, chronic conditions, frailty, and health care utilization. Sensitivity analysis used logistic regression methods.

Results

Approximately 0.38% of each yearly cohort experienced near-future critical illness. The optimal Tree identified 2,644 mutually exclusive subgroups. Socioeconomic status was the most influential variable, followed by nursing home residency and frailty; age was sixth. In the training data, the model performed well; 41 subgroups containing 493 subjects had ≥33% members who developed the outcome. However, in the test data, those subgroups contained 429 individuals, with 20 (4.7%) experiencing the outcome, which comprised 0.98% of all subjects with the outcome. While logistic regression showed less model overfitting, it likewise failed to achieve the stated objective.

Conclusions

High-fidelity prediction of near-future critical illness among community-dwelling adults was not successful using population-based administrative data. Additional research is needed to ascertain whether the inclusion of additional types of data can achieve this goal.

Development of Deep Learning Models for Predicting In-Hospital Mortality Using an Administrative Claims Database: Retrospective Cohort Study

BACKGROUND

Administrative claims databases have been used widely in studies because they have large sample sizes and are easily available. However, studies using administrative databases lack information on disease severity, so a risk adjustment method needs to be developed.

OBJECTIVE

We aimed to develop and validate deep learning-based prediction models for in-hospital mortality of acute care patients.

METHODS

The main model was developed using only administrative claims data (age, sex, diagnoses, and procedures on the day of admission). We also constructed disease-specific models for acute myocardial infarction, heart failure, stroke, and pneumonia using common severity indices for these diseases. Using the Japanese Diagnosis Procedure Combination data from July 2010 to March 2017, we identified 46,665,933 inpatients and divided them into derivation and validation cohorts in a ratio of 95:5. The main model was developed using a 9-layer deep neural network with 4 hidden dense layers that had 1000 nodes and were fully connected to adjacent layers. We evaluated model discrimination ability by an area under the receiver operating characteristic curve (AUC) and calibration ability by calibration plot.

RESULTS

Among the eligible patients, 2,005,035 (4.3%) died. Discrimination and calibration of the models were satisfactory. The AUC of the main model in the validation cohort was 0.954 (95% CI 0.954-0.955). The main model had higher discrimination ability than the disease-specific models.

CONCLUSIONS

Our deep learning-based model using diagnoses and procedures produced valid predictions of in-hospital mortality.

Predicting Unplanned Health Care Utilization and Cost: Comparing Patient-reported Outcomes Measurement Information System and Claims

OBJECTIVES

There is little literature describing if and how payers are utilizing patient-reported outcomes to predict future costs. This study assessed if Patient-reported Outcomes Measurement Information System (PROMIS) domain scores, collected in routine practice at neurology clinics, improved payer predictive models for unplanned care utilization and cost.

STUDY DESIGN

Retrospective cohort analysis of private Health Plan-insured patients with visits at 18 Health Plan-affiliated neurology clinics.

METHODS

PROMIS domains (Anxiety v1.0, Cognitive Function Abilities v2.0, Depression v1.0, Fatigue v1.0, Pain Interference v1.0, Physical Function v2.0, Sleep Disturbance v1.0, and Ability to Participate in Social Roles and Activities v2.0) are collected as part of routine care. Data from patients' first PROMIS measures between June 27, 2018 and April 16, 2019 were extracted and combined with claims data. Using (1) claims data alone and (2) PROMIS and claims data, we examined the association of covariates to utilization (using a logit model) and cost (using a generalized linear model). We evaluated model fit using area under the receiver operating characteristic curve (for unplanned care utilization), akaike information criterion (for unplanned care costs), and sensitivity and specificity in predicting top 15% of unplanned care costs.

RESULTS

Area under the receiver operating curve values were slightly higher, and akaike information criterion values were similar, for PROMIS plus claims covariates compared with claims alone. The PROMIS plus claims model had slightly higher sensitivity and equivalent specificity compared with claims-only models.

CONCLUSION

One-time PROMIS measure data combined with claims data slightly improved predictive model performance compared with claims alone, but likely not to an extent that indicates improved practical utility for payers.

Electronic Health Record Network Research in Infectious Diseases

With the marked increases in electronic health record (EHR) use for providing clinical care, there have been parallel efforts to leverage EHR data for research. EHR repositories offer the promise of vast amounts of clinical data not easily captured with traditional research methods and facilitate clinical epidemiology and comparative effectiveness research, including analyses to identify patients at higher risk for complications or who are better candidates for treatment. These types of studies have been relatively slow to penetrate the field of infectious diseases, but the need for rapid turnaround during the COVID-19 global pandemic has accelerated the uptake. This review discusses the rationale for her network projects, opportunities and challenges that such networks present, and some prior studies within the field of infectious diseases.

A multi-layer model for the early detection of COVID-19

Current COVID-19 screening efforts mainly rely on reported symptoms and the potential exposure to infected individuals. Here, we developed a machine-learning model for COVID-19 detection that uses four layers of information: (i) sociodemographic characteristics of the individual, (ii) spatio-temporal patterns of the disease, (iii) medical condition and general health consumption of the individual and (iv) information reported by the individual during the testing episode. We evaluated our model on 140 682 members of Maccabi Health Services who were tested for COVID-19 at least once between February and October 2020. These individuals underwent, in total, 264 516 COVID-19 PCR tests, out of which 16 512 were positive. Our multi-layer model obtained an area under the curve (AUC) of 81.6% when evaluated over all the individuals in the dataset, and an AUC of 72.8% when only individuals who did not report any symptom were included. Furthermore, considering only information collected before the testing episode-i.e. before the individual had the chance to report on any symptom-our model could reach a considerably high AUC of 79.5%. Our ability to predict early on the outcomes of COVID-19 tests is pivotal for breaking transmission chains, and can be used for a more efficient testing policy.

Data Science Methods for Nursing-Relevant Patient Outcomes and Clinical Processes: The 2019 Literature Year in Review

Data science continues to be recognized and used within healthcare due to the increased availability of large data sets and advanced analytics. It can be challenging for nurse leaders to remain apprised of this rapidly changing landscape. In this article, we describe our findings from a scoping literature review of papers published in 2019 that use data science to explore, explain, and/or predict 15 phenomena of interest to nurses. Fourteen of the 15 phenomena were associated with at least one paper published in 2019. We identified the use of many contemporary data science methods (eg, natural language processing, neural networks) for many of the outcomes. We found many studies exploring Readmissions and Pressure Injuries. The topics of Artificial Intelligence/Machine Learning Acceptance, Burnout, Patient Safety, and Unit Culture were poorly represented. We hope that the studies described in this article help readers: (1) understand the breadth and depth of data science's ability to improve clinical processes and patient outcomes that are relevant to nurses and (2) identify gaps in the literature that are in need of exploration.

The Shieh Score as a Risk Assessment Tool for Hospital-Acquired Pressure Injuries: A Retrospective Cohort Study

OBJECTIVE

To design a new risk assessment tool to identify patients at high risk for hospital-acquired pressure injuries.

METHODS

The researchers developed the Shieh Score using retrospective data of 406,032 hospital admissions from January 2014 to December 2016 with 1,299 pressure injury cases from the pressure injury registry. A decision tree and best subset logistic regression were used to select predictors from demographic and clinical candidate variables, which were then used to construct the Shieh Score.

RESULTS

The final Shieh Score included the following measures: sex, age, diabetes, glomerular filtration rate, albumin level, level of function, use of IV norepinephrine, mechanical ventilation, and level of consciousness. The Shieh Score had a higher Youden Index, specificity, and positive predictive value than the Braden Scale. However, the Braden Scale had a higher sensitivity compared with the Shieh Score.

CONCLUSIONS

The Shieh Score is an alternative risk assessment tool that may effectively identify a smaller number of patients at high risk for hospital-acquired pressure injuries with a higher specificity and positive predictive value than the Braden Scale.

Comparing the Predictive Effects of Patient Medication Adherence Indices in Electronic Health Record and Claims-Based Risk Stratification Models

Multiple indices are available to measure medication adherence behaviors. Medication adherence measures, however, have rarely been extracted from electronic health records (EHRs) for population-level risk predictions. This study assessed the value of medication adherence indices in improving predictive models of cost and hospitalization. This study included a 2-year retrospective cohort of patients younger than age 65 years with linked EHR and insurance claims data. Three medication adherence measures were calculated: medication regimen complexity index (MRCI), medication possession ratio (MPR), and prescription fill rate (PFR). The authors examined the effects of adding these measures to 3 predictive models of utilization: a demographics model, a conventional model (Charlson index), and an advanced diagnosis-based model. Models were trained using EHR and claims data. The study population had an overall MRCI, MPR, and PFR of 14.6 ± 17.8, .624 ± .310, and .810 ± .270, respectively. Adding MRCI and MPR to the demographic and the morbidity models using claims data improved forecasting of next-year hospitalization substantially (eg, AUC of the demographic model increased from .605 to .656 using MRCI). Nonetheless, such boosting effects were attenuated for the advanced diagnosis-based models. Although EHR models performed inferior to claims models, adding adherence indices improved EHR model performances at a larger scale (eg, adding MRCI increased AUC by 4.4% for the Charlson model using EHR data compared to 3.8% using claims). This study shows that medication adherence measures can modestly improve EHR- and claims-derived predictive models of cost and hospitalization in non-elderly patients; however, the improvements are minimal for advanced diagnosis-based models.

Adverse events in women switching from olaparib capsules to tablets: retrospective observational study of US claims data

Aim: Describe rates of prespecified adverse events in patients who switched from olaparib capsules to tablets. Patients & methods: Retrospective, observational cohort analysis using self-controlled, pre-post design. Data on patients with ovarian cancer who switched from olaparib capsules to tablets between January 2015 and February 2019 were obtained from a US claims database. Results: Among all patients (n = 48), proportion with any prespecified adverse event was 45.8% (95% confidence interval: 31.4-60.8) during initial 90 days' capsule use and 35.4% (22.2-50.5) during initial 90 days' tablets use; difference -10.4% (-28.8-9.0). Conclusion: Switching from olaparib capsules to tablets was manageable with no evidence of increased toxicity. This real-world study supports the manageable tolerability of olaparib in women with ovarian cancer.