Data extraction for evidence synthesis using a large language model: A proof-of-concept study

Data extraction is a crucial, yet labor-intensive and error-prone part of evidence synthesis. To date, efforts to harness machine learning for enhancing efficiency of the data extraction process have fallen short of achieving sufficient accuracy and usability. With the release of large language models (LLMs), new possibilities have emerged to increase efficiency and accuracy of data extraction for evidence synthesis. The objective of this proof-of-concept study was to assess the performance of an LLM (Claude 2) in extracting data elements from published studies, compared with human data extraction as employed in systematic reviews. Our analysis utilized a convenience sample of 10 English-language, open-access publications of randomized controlled trials included in a single systematic review. We selected 16 distinct types of data, posing varying degrees of difficulty (160 data elements across 10 studies). We used the browser version of Claude 2 to upload the portable document format of each publication and then prompted the model for each data element. Across 160 data elements, Claude 2 demonstrated an overall accuracy of 96.3% with a high test-retest reliability (replication 1: 96.9%; replication 2: 95.0% accuracy). Overall, Claude 2 made 6 errors on 160 data items. The most common errors (n = 4) were missed data items. Importantly, Claude 2's ease of use was high; it required no technical expertise or labeled training data for effective operation (i.e., zero-shot learning). Based on findings of our proof-of-concept study, leveraging LLMs has the potential to substantially enhance the efficiency and accuracy of data extraction for evidence syntheses.

A New Type 2 Diabetes Microsimulation Model to Estimate Long-Term Health Outcomes, Costs, and Cost-Effectiveness

OBJECTIVES

This study aimed to develop a microsimulation model to estimate the health effects, costs, and cost-effectiveness of public health and clinical interventions for preventing/managing type 2 diabetes.

METHODS

We combined newly developed equations for complications, mortality, risk factor progression, patient utility, and cost-all based on US studies-in a microsimulation model. We performed internal and external validation of the model. To demonstrate the model's utility, we predicted remaining life-years, quality-adjusted life-years (QALYs), and lifetime medical cost for a representative cohort of 10 000 US adults with type 2 diabetes. We then estimated the cost-effectiveness of reducing hemoglobin A1c from 9% to 7% among adults with type 2 diabetes, using low-cost, generic, oral medications.

RESULTS

The model performed well in internal validation; the average absolute difference between simulated and observed incidence for 17 complications was < 8%. In external validation, the model was better at predicting outcomes in clinical trials than in observational studies. The cohort of US adults with type 2 diabetes was projected to have an average of 19.95 remaining life-years (from mean age 61), incur $187 729 in discounted medical costs, and accrue 8.79 discounted QALYs. The intervention to reduce hemoglobin A1c increased medical costs by $1256 and QALYs by 0.39, yielding an incremental cost-effectiveness ratio of $9103 per QALY.

CONCLUSIONS

Using equations exclusively derived from US studies, this new microsimulation model achieves good prediction accuracy in US populations. The model can be used to estimate the long-term health impact, costs, and cost-effectiveness of interventions for type 2 diabetes in the United States.

The smashHitCore Ontology for GDPR-Compliant Sensor Data Sharing in Smart Cities

The adoption of the General Data Protection Regulation (GDPR) has resulted in a significant shift in how the data of European Union citizens is handled. A variety of data sharing challenges in scenarios such as smart cities have arisen, especially when attempting to semantically represent GDPR legal bases, such as consent, contracts and the data types and specific sources related to them. Most of the existing ontologies that model GDPR focus mainly on consent. In order to represent other GDPR bases, such as contracts, multiple ontologies need to be simultaneously reused and combined, which can result in inconsistent and conflicting knowledge representation. To address this challenge, we present the smashHitCore ontology. smashHitCore provides a unified and coherent model for both consent and contracts, as well as the sensor data and data processing associated with them. The ontology was developed in response to real-world sensor data sharing use cases in the insurance and smart city domains. The ontology has been successfully utilised to enable GDPR-complaint data sharing in a connected car for insurance use cases and in a city feedback system as part of a smart city use case.

Data Protection by Design Tool for Automated GDPR Compliance Verification Based on Semantically Modeled Informed Consent

The enforcement of the GDPR in May 2018 has led to a paradigm shift in data protection. Organizations face significant challenges, such as demonstrating compliance (or auditability) and automated compliance verification due to the complex and dynamic nature of consent, as well as the scale at which compliance verification must be performed. Furthermore, the GDPR’s promotion of data protection by design and industrial interoperability requirements has created new technical challenges, as they require significant changes in the design and implementation of systems that handle personal data. We present a scalable data protection by design tool for automated compliance verification and auditability based on informed consent that is modeled with a knowledge graph. Automated compliance verification is made possible by implementing a regulation-to-code process that translates GDPR regulations into well-defined technical and organizational measures and, ultimately, software code. We demonstrate the effectiveness of the tool in the insurance and smart cities domains. We highlight ways in which our tool can be adapted to other domains.

Rural and urban differences in undersupply of buprenorphine provider availability in the United States, 2018

BACKGROUND

Medications to treat opioid use disorder (OUD) including buprenorphine products are evidence-based and cost-effective tools for combating the opioid crisis. However, limited availability to buprenorphine is pervasive in the United States (US) and may serve to exacerbate the deadly epidemic. Although prior research points to rural counties as especially needy of strategies that improve buprenorphine availability, it is important to investigate the availability of waivered providers according to treatment need as defined by the county-level rate of opioid-overdose deaths (OOD). This study examined differences in buprenorphine provider availability relative to treatment need among rural and urban counties in the US.

METHODS

Buprenorphine provider availability relative to need in each county was defined as the number of waivered providers divided by the rate of OODs (i.e., number of OODs/100,000 population), according to 2018 data. Counties with ratios in the bottom tertile of their state were classified as buprenorphine undersupplied. We estimated logit models to statistically test the association of rurality and state main effects and their interaction terms (independent variables) and the county classified as buprenorphine undersupplied (dependent variable).

RESULTS

A total of 38 states and 2595 counties had sufficient non-suppressed data to remain in the analysis. A larger percent of urban counties (36.43%) than rural counties (32.01%) were classified as buprenorphine undersupplied (p  = 0.001). The likelihood of a rural county being undersupplied varied considerably by state (Chi Square  = 82.88, p  = 0.000). All states with significant (p  < 0.05 or p  < 0.10) interaction terms showed lower likelihood of buprenorphine undersupply in rural counties.

CONCLUSIONS

The rural-urban distribution in undersupply of waivered buprenorphine providers relative to need varied markedly by state. Strategies for improving access to buprenorphine-waivered providers should be state-centric and informed by county-specific indicators of need.

Modeling Infectious Diseases in Healthcare Network (MInD-Healthcare) Framework for Describing and Reporting Multidrug-resistant Organism and Healthcare-Associated Infections Agent-based Modeling Methods

Mathematical modeling of healthcare-associated infections and multidrug-resistant organisms improves our understanding of pathogen transmission dynamics and provides a framework for evaluating prevention strategies. One way of improving the communication among modelers is by providing a standardized way of describing and reporting models, thereby instilling confidence in the reproducibility and generalizability of such models. We updated the Overview, Design concepts, and Details protocol developed by Grimm et al [11] for describing agent-based models (ABMs) to better align with elements commonly included in healthcare-related ABMs. The Modeling Infectious Diseases in Healthcare Network (MInD-Healthcare) framework includes the following 9 key elements: (1) Purpose and scope; (2) Entities, state variables, and scales; (3) Initialization; (4) Process overview and scheduling; (5) Input data; (6) Agent interactions and organism transmission; (7) Stochasticity; (8) Submodels; and (9) Model verification, calibration, and validation. Our objective is that this framework will improve the quality of evidence generated utilizing these models.

Modeling inpatient and outpatient antibiotic stewardship interventions to reduce the burden of Clostridioides difficile infection in a regional healthcare network

Antibiotic exposure can lead to unintended outcomes, including drug-drug interactions, adverse drug events, and healthcare-associated infections like Clostridioides difficile infection (CDI). Improving antibiotic use is critical to reduce an individual's CDI risk. Antibiotic stewardship initiatives can reduce inappropriate antibiotic prescribing (e.g., unnecessary antibiotic prescribing, inappropriate antibiotic selection), impacting both hospital (healthcare)-onset (HO)-CDI and community-associated (CA)-CDI. Previous computational and mathematical modeling studies have demonstrated a reduction in CDI incidence associated with antibiotic stewardship initiatives in hospital settings. Although the impact of antibiotic stewardship initiatives in long-term care facilities (LTCFs), including nursing homes, and in outpatient settings have been documented, the effects of specific interventions on CDI incidence are not well understood. We examined the relative effectiveness of antibiotic stewardship interventions on CDI incidence using a geospatially explicit agent-based model of a regional healthcare network in North Carolina. We simulated reductions in unnecessary antibiotic prescribing and inappropriate antibiotic selection with intervention scenarios at individual and network healthcare facilities, including short-term acute care hospitals (STACHs), nursing homes, and outpatient locations. Modeled antibiotic prescription rates were calculated using patient-level data on antibiotic length of therapy for the 10 modeled network STACHs. By simulating a 30% reduction in antibiotics prescribed across all inpatient and outpatient locations, we found the greatest reductions on network CDI incidence among tested scenarios, namely a 17% decrease in HO-CDI incidence and 7% decrease in CA-CDI. Among intervention scenarios of reducing inappropriate antibiotic selection, we found a greater impact on network CDI incidence when modeling this reduction in nursing homes alone compared to the same intervention in STACHs alone. These results support the potential importance of LTCF and outpatient antibiotic stewardship efforts on network CDI burden and add to the evidence that a coordinated approach to antibiotic stewardship across multiple facilities, including inpatient and outpatient settings, within a regional healthcare network could be an effective strategy to reduce network CDI burden.

Creation of a Geospatially Explicit, Agent-based Model of a Regional Healthcare Network with Application to Clostridioides difficile Infection

Agent-based models (ABMs) describe and simulate complex systems comprising unique agents, or individuals, while accounting for geospatial and temporal variability among dynamic processes. ABMs are increasingly used to study healthcare-associated infections (ie, infections acquired during admission to a healthcare facility), including Clostridioides difficile infection, currently the most common healthcare-associated infection in the United States. The overall burden and transmission dynamics of healthcare-associated infections, including C difficile infection, may be influenced by community sources and movement of people among healthcare facilities and communities. These complex dynamics warrant geospatially explicit ABMs that extend beyond single healthcare facilities to include entire systems (eg, hospitals, nursing homes and extended care facilities, the community). The agents in ABMs can be built on a synthetic population, a model-generated representation of the actual population with associated spatial (eg, home residence), temporal (eg, change in location over time), and nonspatial (eg, sociodemographic features) attributes. We describe our methods to create a geospatially explicit ABM of a major regional healthcare network using a synthetic population as microdata input. We illustrate agent movement in the healthcare network and the community, informed by patient-level medical records, aggregate hospital discharge data, healthcare facility licensing data, and published literature. We apply the ABM output to visualize agent movement in the healthcare network and the community served by the network. We provide an application example of the ABM to C difficile infection using a natural history submodel. We discuss the ABM's potential to detect network areas where disease risk is high; simulate and evaluate interventions to protect public health; adapt to other geographic locations and healthcare-associated infections, including emerging pathogens; and meaningfully translate results to public health practitioners, healthcare providers, and policymakers.

2440. Using a geospatially explicit agent-based model of a regional healthcare network to assess varied antibiotic risk on Clostridioides difficile infection incidence

Background

Different antibiotic classes are associated with different Clostridioides difficile infection (CDI) risk. The impact of varied antibiotic risk on CDI incidence can be explored using agent-based models (ABMs). ABMs can simulate complete systems (e.g., regional healthcare networks) comprised of discrete, unique agents (e.g., patients) which can be represented using a synthetic population, or model-generated representation of the population. We used an ABM of a North Carolina (NC) regional healthcare network to assess the impact of increasing antibiotic risk ratios (RRs) across network locations on healthcare-associated (HA) and community-associated (CA) CDI incidence.

Methods

The ABM describes CDI acquisition and patient movement across 14 network locations (i.e., nodes) (11 short-term acute care hospitals, 1 long-term acute care hospital, 1 nursing home, and the community). We used a sample of 2 million synthetic NC residents as ABM microdata. We updated agent states (i.e., location, antibiotic exposure, C. difficile colonization, CDI status) daily. We applied antibiotic RRs of 1, 5, 8.9 (original model RR), 15, and 20 to agents across the network to simulate varied risk corresponding to different antibiotic classes. We determined network HA-CDI and CA-CDI incidence and percent mean change for each RR.

Results

In this simulation study, HA-CDI incidence increased with increasing antibiotic risk, ranging from 11.3 to 81.4 HA-CDI cases/100,000 person-years for antibiotic RRs of 1 to 20, respectively. On average, the per unit increase in antibiotic RR was 33% for HA-CDI and 6% for CA-CDI (figure).

Conclusion

We used a geospatially explicit ABM to simulate increasing antibiotic risk, corresponding to different antibiotic classes, and to explore the impact on CDI incidence. The per unit increase in antibiotic risk was greater for HA-CDI than CA-CDI due to the higher probability of receiving antibiotics and higher concentration of agents with other CDI risk factors in the healthcare facilities of the ABM. These types of analyses, which demonstrate the interconnectedness of network healthcare facilities and the associated community served by the network, might help inform targeted antibiotic stewardship efforts in certain network locations.

Disclosures

All authors: No reported disclosures.

Cost-effectiveness of risk stratified medication management for reducing premature cardiovascular mortality in Kenya

Background

Cardiovascular disease (CVD) is a major contributor to the burden from non-communicable diseases in Sub-Saharan Africa and hypertension is the leading risk factor for CVD. The objective of this modeling study is to assess the cost-effectiveness of a risk stratified approach to medication management in Kenya in order to achieve adequate blood pressure control to reduce CVD events.

Methods

We developed a microsimulation model to evaluate CVD risk over the lifetime of a cohort of individuals. Risk groups were assigned utilizing modified Framingham study distributions based on individual level risk factors from the Kenya STEPwise survey which collected details on blood pressure, blood glucose, tobacco and alcohol use and cholesterol levels. We stratified individuals into 4 risk groups: very low, low, moderate and high risk. Mortality could occur due to acute CVD events, subsequent future events (for individual who survive the initial event) and other causes. We present cost and DALYs gained due to medication management for men and women 25 to 69 years.

Results

Treating high risk individuals only was generally more cost-effective that treating high and moderate risk individuals. At the anticipated base levels of effectiveness, medication management was only cost-effective under the low cost scenario. The incremental cost per DALY gained with low cost ranged from $1,505 to $3,608, which is well under $4,785 (3 times GPD per capita) threshold for Kenya. Under the low cost scenario, even lower levels of effectiveness of medication management are likely to be cost-effective for high-risk men and women.

Conclusions

In Kenya, our results indicate that the risk stratified approach to treating hypertension may be cost-effective especially for men and women at a high risk for CVD events, but these results are highly sensitive to the cost of medications. Medication management would require significant financial investment and therefore other interventions, including lifestyle changes, should be evaluated especially for those with elevated blood pressure and overall 10-year risk that is less than 20%.

Sexual content in video games: an analysis of the Entertainment Software Rating Board classification from 1994 to 2013

Background Video games are widely used by children and adolescents and have become a significant source of exposure to sexual content. Despite evidence of the important role of media in the development of sexual attitudes and behaviours, little attention has been paid to monitor sexual content in video games.

METHODS

Data was obtained about sexual content and rating for 23722 video games from 1994 to 2013 from the Entertainment Software Rating Board database; release dates and information on the top 100 selling video games was also obtained. A yearly prevalence of sexual content according to rating categories was calculated. Trends and comparisons were estimated using Joinpoint regression.

RESULTS

Sexual content was present in 13% of the video games. Games rated 'Mature' had the highest prevalence of sexual content (34.5%) followed by 'Teen' (30.7%) and 'E10+' (21.3%). Over time, sexual content decreased in the 'Everyone' category, 'E10+' maintained a low prevalence and 'Teen' and 'Mature' showed a marked increase. Both top and non-top video games showed constant increases, with top selling video games having 10.1% more sexual content across the period of study.

CONCLUSION

Over the last 20 years, the prevalence of sexual content has increased in video games with a 'Teen' or 'Mature' rating. Further studies are needed to quantify the potential association between sexual content in video games and sexual behaviour in children and adolescents.

Molecular biological analysis of a bidirectional hydrogenase from cyanobacteria

An 8.9-kb segment with hydrogenase genes from the cyanobacterium Anabaena variabilis has been cloned and sequenced. The sequences show homology to the methyl-viologen-reducing hydrogenases from archaebacteria and, even more striking, to the NAD(+)-reducing enzymes from Alcaligenes eutrophus and Nocardia opaca as well as to the NADP(+)-dependent protein from Desulfovibrio fructosovorans. The cluster from A. variabilis contains genes coding for both the hydrogenase heterodimer (hoxH and hoxY) and for the diaphorase moiety (hoxU and hoxF) described for the A. eutrophus enzyme. In A. variabilis the gene cluster is split by two open reading frames (between hoxY and hoxH and between hoxU and hoxY, respectively), and a probably non-coding 0.9-kb segment in an unusual way. The hoxH partial sequence from Anabaena 7119 and Anacystis nidulans was amplified by PCR. Using the labeled segment from A. 7119 as probe, Southern analysis revealed homologous gene segments in the cyanobacteria A. 7119, Anabaena cylindrica, Anacystis nidulans and A. variabilis. The bidirectional hydrogenase from A. nidulans was purified and digests were sequenced. The amino acid sequences obtained showed partial identities to the amino acid sequences deduced from the DNA data of the 8.9-kb segment from A. variabilis. Therefore the 8.9-kb segment contains the genes coding for the bidirectional, reversible hydrogenase from cyanobacteria. Crude extracts from A. nidulans perform NAD(P)H-dependent H2 evolution corroborating the molecular biological demonstration of the NAD(P)(+)-dependent hydrogenase in cyanobacteria.