A roadmap to artificial intelligence (AI): Methods for designing and building AI ready data to promote fairness

OBJECTIVES

We evaluated methods for preparing electronic health record data to reduce bias before applying artificial intelligence (AI).

METHODS

We created methods for transforming raw data into a data framework for applying machine learning and natural language processing techniques for predicting falls and fractures. Strategies such as inclusion and reporting for multiple races, mixed data sources such as outpatient, inpatient, structured codes, and unstructured notes, and addressing missingness were applied to raw data to promote a reduction in bias. The raw data was carefully curated using validated definitions to create data variables such as age, race, gender, and healthcare utilization. For the formation of these variables, clinical, statistical, and data expertise were used. The research team included a variety of experts with diverse professional and demographic backgrounds to include diverse perspectives.

RESULTS

For the prediction of falls, information extracted from radiology reports was converted to a matrix for applying machine learning. The processing of the data resulted in an input of 5,377,673 reports to the machine learning algorithm, out of which 45,304 were flagged as positive and 5,332,369 as negative for falls. Processed data resulted in lower missingness and a better representation of race and diagnosis codes. For fractures, specialized algorithms extracted snippets of text around keywork "femoral" from dual x-ray absorptiometry (DXA) scans to identify femoral neck T-scores that are important for predicting fracture risk. The natural language processing algorithms yielded 98% accuracy and 2% error rate The methods to prepare data for input to artificial intelligence processes are reproducible and can be applied to other studies.

CONCLUSION

The life cycle of data from raw to analytic form includes data governance, cleaning, management, and analysis. When applying artificial intelligence methods, input data must be prepared optimally to reduce algorithmic bias, as biased output is harmful. Building AI-ready data frameworks that improve efficiency can contribute to transparency and reproducibility. The roadmap for the application of AI involves applying specialized techniques to input data, some of which are suggested here. This study highlights data curation aspects to be considered when preparing data for the application of artificial intelligence to reduce bias.

Deploying a national clinical text processing infrastructure

OBJECTIVES

Clinical text processing offers a promising avenue for improving multiple aspects of healthcare, though operational deployment remains a substantial challenge. This case report details the implementation of a national clinical text processing infrastructure within the Department of Veterans Affairs (VA).

METHODS

Two foundational use cases, cancer case management and suicide and overdose prevention, illustrate how text processing can be practically implemented at scale for diverse clinical applications using shared services.

RESULTS

Insights from these use cases underline both commonalities and differences, providing a replicable model for future text processing applications.

CONCLUSIONS

This project enables more efficient initiation, testing, and future deployment of text processing models, streamlining the integration of these use cases into healthcare operations. This project implementation is in a large integrated health delivery system in the United States, but we expect the lessons learned to be relevant to any health system, including smaller local and regional health systems in the United States.

Spatiotemporal trends in COVID-19 vaccine sentiments on a social media platform and correlations with reported vaccine coverage

Objective

To assess spatiotemporal trends in, and determinants of, the acceptance of coronavirus disease 2019 (COVID-19) vaccination globally, as expressed on the social media platform X (formerly Twitter).

Methods

We collected over 13 million posts on the platform regarding COVID-19 vaccination made between November 2020 and March 2022 in 90 languages. Multilingual deep learning XLM-RoBERTa models annotated all posts using an annotation framework after being fine-tuned on 8125 manually annotated, English-language posts. The annotation results were used to assess spatiotemporal trends in COVID-19 vaccine acceptance and confidence as expressed by platform users in 135 countries and territories. We identified associations between spatiotemporal trends in vaccine acceptance and country-level characteristics and public policies by using univariate and multivariate regression analysis.

Findings

A greater proportion of platform users in the World Health Organization's South-East Asia, Eastern Mediterranean and Western Pacific Regions expressed vaccine acceptance than users in the rest of the world. Countries in which a greater proportion of platform users expressed vaccine acceptance had higher COVID-19 vaccine coverage rates. Trust in government was also associated with greater vaccine acceptance. Internationally, vaccine acceptance and confidence declined among platform users as: (i) vaccination eligibility was extended to adolescents; (ii) vaccine supplies became sufficient; (iii) nonpharmaceutical interventions were relaxed; and (iv) global reports on adverse events following vaccination appeared.

Conclusion

Social media listening could provide an effective and expeditious means of informing public health policies during pandemics, and could supplement existing public health surveillance approaches in addressing global health issues.

A Roadmap to Artificial Intelligence (AI): Methods for Designing and Building AI ready Data for Women's Health Studies

Objectives

Evaluating methods for building data frameworks for application of AI in large scale datasets for women's health studies.

Methods

We created methods for transforming raw data to a data framework for applying machine learning (ML) and natural language processing (NLP) techniques for predicting falls and fractures.

Results

Prediction of falls was higher in women compared to men. Information extracted from radiology reports was converted to a matrix for applying machine learning. For fractures, by applying specialized algorithms, we extracted snippets from dual x-ray absorptiometry (DXA) scans for meaningful terms usable for predicting fracture risk.

Discussion

Life cycle of data from raw to analytic form includes data governance, cleaning, management, and analysis. For applying AI, data must be prepared optimally to reduce algorithmic bias.

Conclusion

Algorithmic bias is harmful for research using AI methods. Building AI ready data frameworks that improve efficiency can be especially valuable for women's health.

Emergency care and the patient experience: Using sentiment analysis and topic modeling to understand the impact of the COVID-19 pandemic

The COVID-19 pandemic has presented many unique challenges to patient care especially in emergency medicine. These challenges result in an altered patient experience. Patient experience refers to the cumulative impression made on patients during their medical visit and is measured by a standardized survey tool. Patient experience is considered a key measure of quality of care. The volume of survey data received makes it difficult to spot trends and concerns in patient comments. Topic modeling and sentiment analysis are well documented analytic techniques that can be used to gain insight into patient experience and make sense of vast quantities of data. This study examined three periods of time, pre, during and post-COVID-19 first wave in order to identify key trends in sentiment and topics related to patient experience. Previously collected, anonymized Press Ganey (PG) survey data was used from three northeastern emergency department that make up an academic emergency department. Data was collected for three contiguous time periods: Pre-COVID-19 (12/10/2019- 3/10/2020), During COVID-19: (3/11/2020–6/10/2020), and Post-first wave COVID-19 (6/11/2020- 9/10/2020). Preprocessing of the data was carried out then a sentiment label (i.e., positive, negative, neutral, mixed) was assigned by the tool. These labels were used to assess the validity of Press Ganey labels. Next, a topic modeling approach from machine learning was used to analyze the contents of the patient comments and uncover concerns and perceptions of patient experiences. Themes that emerged from the analysis of patient comments included concerns over personal safety and exposure to the virus, exclusion of family from decision making and care and high levels of scrutiny over systems issues, care, and treatment protocols. Topic modeling showed shifting priorities and concerns throughout the three periods examined. Prior to the pandemic, patient comments were largely positive and focused on technical expertise and perceptions of competence. New topics and concerns that patients reported relevant to the pandemic were identified during-COVID-19. Comments on systems issues regarding processes to limit viral spread and concerns over family/visitor restrictions were dominant. Although there was evidence of praise and appreciation of the efforts of staff there was also a high level of scrutiny of the processes encountered during the emergency visit. Sentiment analysis and topic modeling offer a unique method for organizing and analyzing the shifting concerns of patients and families. Suggestions of interventions are made to address these evolving concerns. The automation of analysis using artificial intelligence would allow for rapid and accurate analysis of patient feedback.

Complementary and Integrative Health Approaches Are Underused Among Older Veterans With Musculoskeletal Pain

Complementary and integrative health (CIH) approaches are recommended in national policy guidelines as viable options for managing chronic pain, yet their use among Veterans has been suboptimal, especially for older Veterans. We identified 64,444 Veterans with a diagnosis of musculoskeletal disorders (MSD) who reported a moderate to severe pain intensity during primary care visits in 2013 from the Veterans Health Administration (VHA) electronic records. Using natural language processing (NLP), CIH use (acupuncture, chiropractic care and massage) was documented for 8169 (6.5%) of 125408 primary care visits in providers’ progress notes. Compared to their younger counterparts, older Veterans aged ≥ 65 years had 21% lower likelihood of using CIH during the year [Odds Ratio (OR): 0.79; 95% Confidence Intervals (CI): 0.73, 0.86] after accounting for demographic, clinical, temporal and spatial confounding using a generalized estimating equation logistic model. Non-white race/ethnicity, tobacco use, medical comorbidities and diagnosis of alcohol or substance use disorders were independently associated with less CIH use (ORs ranging 0.97-0.80, p<0.03-0.0001); whereas female gender, being married and number of MSD diagnoses were associated with greater CIH use (ORs ranging 1.13-1.30, p<0.0001). Redefining CIH use as chiropractic care alone [4.8% person-visits; OR: 0.78 (95% CI: 0.70, 0.86)] or incorporating structured data [9.0% person-visits; OR: 0.76 (95% CI: 0.70-0.82)] in the adjusted GEE model derived consistent results. Research to identify and address barriers to CIH use among older Veterans is encouraged.

Identification of Patients with Nontraumatic Intracranial Hemorrhage Using Administrative Claims Data

INTRODUCTION

Nontraumatic intracranial hemorrhage (ICH) is a neurological emergency of research interest; however, unlike ischemic stroke, has not been well studied in large datasets due to the lack of an established administrative claims-based definition. We aimed to evaluate both explicit diagnosis codes and machine learning methods to create a claims-based definition for this clinical phenotype.

METHODS

We examined all patients admitted to our tertiary medical center with a primary or secondary International Classification of Disease version 9 (ICD-9) or 10 (ICD-10) code for ICH in claims from any portion of the hospitalization in 2014-2015. As a gold standard, we defined the nontraumatic ICH phenotype based on manual chart review. We tested explicit definitions based on ICD-9 and ICD-10 that had been previously published in the literature as well as four machine learning classifiers including support vector machine (SVM), logistic regression with LASSO, random forest and xgboost. We report five standard measures of model performance for each approach.

RESULTS

A total of 1830 patients with 2145 unique ICD-10 codes were included in the initial dataset, of which 437 (24%) were true positive based on manual review. The explicit ICD-10 definition performed best (Sensitivity = 0.89 (95% CI 0.85-0.92), Specificity = 0.83 (0.81-0.85), F-score = 0.73 (0.69-0.77)) and improves on an explicit ICD-9 definition (Sensitivity = 0.87 (0.83-0.90), Specificity = 0.77 (0.74-0.79), F-score = 0.67 (0.63-0.71). Among machine learning classifiers, SVM performed best (Sensitivity = 0.78 (0.75-0.82), Specificity = 0.84 (0.81-0.87), AUC = 0.89 (0.87-0.92), F-score = 0.66 (0.62-0.69)).

CONCLUSIONS

An explicit ICD-10 definition can be used to accurately identify patients with a nontraumatic ICH phenotype with substantially better performance than ICD-9. An explicit ICD-10 based definition is easier to implement and quantitatively not appreciably improved with the additional application of machine learning classifiers. Future research utilizing large datasets should utilize this definition to address important research gaps.

Exploring supervised machine learning approaches to predicting Veterans Health Administration chiropractic service utilization

BACKGROUND

Chronic spinal pain conditions affect millions of US adults and carry a high healthcare cost burden, both direct and indirect. Conservative interventions for spinal pain conditions, including chiropractic care, have been associated with lower healthcare costs and improvements in pain status in different clinical populations, including veterans. Little is currently known about predicting healthcare service utilization in the domain of conservative interventions for spinal pain conditions, including the frequency of use of chiropractic services. The purpose of this retrospective cohort study was to explore the use of supervised machine learning approaches to predicting one-year chiropractic service utilization by veterans receiving VA chiropractic care.

METHODS

We included 19,946 veterans who entered the Musculoskeletal Diagnosis Cohort between October 1, 2003 and September 30, 2013 and utilized VA chiropractic services within one year of cohort entry. The primary outcome was one-year chiropractic service utilization following index chiropractic visit, split into quartiles represented by the following classes: 1 visit, 2 to 3 visits, 4 to 6 visits, and 7 or greater visits. We compared the performance of four multiclass classification algorithms (gradient boosted classifier, stochastic gradient descent classifier, support vector classifier, and artificial neural network) in predicting visit quartile using 158 sociodemographic and clinical features.

RESULTS

The selected algorithms demonstrated poor prediction capabilities. Subset accuracy was 42.1% for the gradient boosted classifier, 38.6% for the stochastic gradient descent classifier, 41.4% for the support vector classifier, and 40.3% for the artificial neural network. The micro-averaged area under the precision-recall curve for each one-versus-rest classifier was 0.43 for the gradient boosted classifier, 0.38 for the stochastic gradient descent classifier, 0.43 for the support vector classifier, and 0.42 for the artificial neural network. Performance of each model yielded only a small positive shift in prediction probability (approximately 15%) compared to naïve classification.

CONCLUSIONS

Using supervised machine learning to predict chiropractic service utilization remains challenging, with only a small shift in predictive probability over naïve classification and limited clinical utility. Future work should examine mechanisms to improve model performance.

Estimating healthcare mobility in the Veterans Affairs Healthcare System

BACKGROUND

Healthcare mobility, defined as healthcare utilization in more than one distinct healthcare system, may have detrimental effects on outcomes of care. We characterized healthcare mobility and associated characteristics among a national sample of Veterans.

METHODS

Using the Veterans Health Administration Electronic Health Record, we conducted a retrospective cohort study to quantify healthcare mobility within a four year period. We examined the association between sociodemographic and clinical characteristics and healthcare mobility, and characterized possible temporal and geographic patterns of healthcare mobility.

RESULTS

Approximately nine percent of the sample were healthcare mobile. Younger Veterans, divorced or separated Veterans, and those with hepatitis C virus and psychiatric disorders were more likely to be healthcare mobile. We demonstrated two possible patterns of healthcare mobility, related to specialty care and lifestyle, in which Veterans repeatedly utilized two different healthcare systems.

CONCLUSIONS

Healthcare mobility is associated with young age, marital status changes, and also diseases requiring intensive management. This type of mobility may affect disease prevention and management and has implications for healthcare systems that seek to improve population health.