Large language models facilitate the generation of electronic health record phenotyping algorithms

OBJECTIVES

Phenotyping is a core task in observational health research utilizing electronic health records (EHRs). Developing an accurate algorithm demands substantial input from domain experts, involving extensive literature review and evidence synthesis. This burdensome process limits scalability and delays knowledge discovery. We investigate the potential for leveraging large language models (LLMs) to enhance the efficiency of EHR phenotyping by generating high-quality algorithm drafts.

MATERIALS AND METHODS

We prompted four LLMs-GPT-4 and GPT-3.5 of ChatGPT, Claude 2, and Bard-in October 2023, asking them to generate executable phenotyping algorithms in the form of SQL queries adhering to a common data model (CDM) for three phenotypes (ie, type 2 diabetes mellitus, dementia, and hypothyroidism). Three phenotyping experts evaluated the returned algorithms across several critical metrics. We further implemented the top-rated algorithms and compared them against clinician-validated phenotyping algorithms from the Electronic Medical Records and Genomics (eMERGE) network.

RESULTS

GPT-4 and GPT-3.5 exhibited significantly higher overall expert evaluation scores in instruction following, algorithmic logic, and SQL executability, when compared to Claude 2 and Bard. Although GPT-4 and GPT-3.5 effectively identified relevant clinical concepts, they exhibited immature capability in organizing phenotyping criteria with the proper logic, leading to phenotyping algorithms that were either excessively restrictive (with low recall) or overly broad (with low positive predictive values).

CONCLUSION

GPT versions 3.5 and 4 are capable of drafting phenotyping algorithms by identifying relevant clinical criteria aligned with a CDM. However, expertise in informatics and clinical experience is still required to assess and further refine generated algorithms.

Large Language Models Facilitate the Generation of Electronic Health Record Phenotyping Algorithms

Objectives

Phenotyping is a core task in observational health research utilizing electronic health records (EHRs). Developing an accurate algorithm demands substantial input from domain experts, involving extensive literature review and evidence synthesis. This burdensome process limits scalability and delays knowledge discovery. We investigate the potential for leveraging large language models (LLMs) to enhance the efficiency of EHR phenotyping by generating high-quality algorithm drafts.

Materials and Methods

We prompted four LLMs-GPT-4 and GPT-3.5 of ChatGPT, Claude 2, and Bard-in October 2023, asking them to generate executable phenotyping algorithms in the form of SQL queries adhering to a common data model (CDM) for three phenotypes (i.e., type 2 diabetes mellitus, dementia, and hypothyroidism). Three phenotyping experts evaluated the returned algorithms across several critical metrics. We further implemented the top-rated algorithms and compared them against clinician-validated phenotyping algorithms from the Electronic Medical Records and Genomics (eMERGE) network.

Results

GPT-4 and GPT-3.5 exhibited significantly higher overall expert evaluation scores in instruction following, algorithmic logic, and SQL executability, when compared to Claude 2 and Bard. Although GPT-4 and GPT-3.5 effectively identified relevant clinical concepts, they exhibited immature capability in organizing phenotyping criteria with the proper logic, leading to phenotyping algorithms that were either excessively restrictive (with low recall) or overly broad (with low positive predictive values).

Conclusion

GPT versions 3.5 and 4 are capable of drafting phenotyping algorithms by identifying relevant clinical criteria aligned with a CDM. However, expertise in informatics and clinical experience is still required to assess and further refine generated algorithms.

Glycolysis: An early marker for vancomycin-specific T-cell activation

BACKGROUND

Vancomycin, a glycopeptide antibiotic used for Gram-positive bacterial infections, has been linked with drug reaction with eosinophilia and systemic symptoms (DRESS) in HLA-A*32:01-expressing individuals. This is associated with activation of T lymphocytes, for which glycolysis has been isolated as a fuel pathway following antigenic stimulation. However, the metabolic processes that underpin drug-reactive T-cell activation are currently undefined and may shed light on the energetic conditions needed for the elicitation of drug hypersensitivity or tolerogenic pathways. Here, we sought to characterise the immunological and metabolic pathways involved in drug-specific T-cell activation within the context of DRESS pathogenesis using vancomycin as model compound and drug-reactive T-cell clones (TCCs) generated from healthy donors and vancomycin-hypersensitive patients.

METHODS

CD4+ and CD8+ vancomycin-responsive TCCs were generated by serial dilution. The Seahorse XFe96 Analyzer was used to measure the extracellular acidification rate (ECAR) as an indicator of glycolytic function. Additionally, T-cell proliferation and cytokine release (IFN-γ) assay were utilised to correlate the bioenergetic characteristics of T-cell activation with in vitro assays.

RESULTS

Model T-cell stimulants induced non-specific T-cell activation, characterised by immediate augmentation of ECAR and rate of ATP production (JATPglyc). There was a dose-dependent and drug-specific glycolytic shift when vancomycin-reactive TCCs were exposed to the drug. Vancomycin-reactive TCCs did not exhibit T-cell cross-reactivity with structurally similar compounds within proliferative and cytokine readouts. However, cross-reactivity was observed when analysing energetic responses; TCCs with prior specificity for vancomycin were also found to exhibit glycolytic switching after exposure to teicoplanin. Glycolytic activation of TCC was HLA restricted, as exposure to HLA blockade attenuated the glycolytic induction.

CONCLUSION

These studies describe the glycolytic shift of CD4+ and CD8+ T cells following vancomycin exposure. Since similar glycolytic switching is observed with teicoplanin, which did not activate T cells, it is possible the master switch for T-cell activation is located upstream of metabolic signalling.

Drug Reaction With Eosinophilia and Systemic Symptoms

This Patient Page describes the symptoms, diagnosis, and treatment of drug reaction with eosinophilia and systemic symptoms.

Updates on the immunopathology and genomics of severe cutaneous adverse drug reactions

Severe cutaneous adverse reactions (SCARs) such as Stevens-Johnson syndrome, toxic epidermal necrolysis (SJS/TEN), and drug reaction with eosinophilia and systemic symptoms (DRESS)/drug-induced hypersensitivity syndrome (DIHS) cause significant morbidity and mortality and impede new drug development. HLA class I associations with SJS/TEN and drug reaction with eosinophilia and systemic symptoms/drug-induced hypersensitivity syndrome have aided preventive efforts and provided insights into immunopathogenesis. In SJS/TEN, HLA class I-restricted oligoclonal CD8+ T-cell responses occur at the tissue level. However, specific HLA risk allele(s) and antigens driving this response have not been identified for most drugs. HLA risk alleles also have incomplete positive and negative predictive values, making truly comprehensive screening currently challenging. Although, there have been key paradigm shifts in knowledge regarding drug hypersensitivity, there are still many open and unanswered questions about SCAR immunopathogenesis, as well as genetic and environmental risk. In addition to understanding the cellular and molecular basis of SCAR at the single-cell level, identification of the MHC-restricted drug-reactive self- or viral peptides driving the hypersensitivity reaction will also be critical to advancing premarketing strategies to predict risk at an individual and drug level. This will also enable identification of biologic markers for earlier diagnosis and accurate prognosis, as well as drug causality and targeted therapeutics.

Clinical Phenotypes of Immediate First-Dose Reactions to mRNA COVID-19: A Multicenter Latent Class Analysis

BACKGROUND

Although immediate potentially allergic reactions have been reported after dose 1 of mRNA coronavirus disease 2019 (COVID-19) vaccines, comprehensively defined subtypes have not been clearly distinguished.

OBJECTIVE

To define distinct clinical phenotypes of immediate reactions after dose 1 of mRNA COVID-19 vaccination, and to assess the relation of clinical phenotype to mRNA COVID-19 vaccine second dose tolerance.

METHODS

This retrospective study included patients with 1 or more potentially allergic symptoms or signs within 4 hours of receiving dose 1 of an mRNA COVID-19 vaccine and assessed by allergy/immunology specialists from 5 U.S. academic medical centers (January-June 2021). We used latent class analysis-an unbiased, machine-learning modeling method-to define novel clinical phenotypes. We assessed demographic, clinical, and reaction characteristics associated with phenotype membership. Using log-binomial regression, we assessed the relation between phenotype membership and second dose tolerance, defined as either no symptoms or mild, self-limited symptoms resolving with antihistamines alone. A sensitivity analysis considered second dose tolerance as objective signs only.

RESULTS

We identified 265 patients with dose-1 immediate reactions with 3 phenotype clusters: (1) Limited or Predominantly Cutaneous, (2) Sensory, and (3) Systemic. A total of 223 patients (84%) received a second dose and 200 (90%) tolerated their second dose. Sensory cluster (all patients had the symptom of numbness or tingling) was associated with a higher likelihood of second dose intolerance, but this finding did not persist when accounting for objective signs.

CONCLUSIONS

Three novel clinical phenotypes of immediate-onset reactions after dose 1 of mRNA COVID-19 vaccines were identified using latent class analysis: (1) Limited or Predominantly Cutaneous, (2) Sensory, and (3) Systemic. Whereas these clinical phenotypes may indicate differential mechanistic etiologies or associations with subsequent dose tolerance, most individuals proceeding to their second dose tolerated it.

Diagnostic accuracy of vaccine and vaccine excipient testing in the setting of allergic reactions to COVID-19 vaccines: A systematic review and meta-analysis

For persons with immediate allergic reactions to mRNA COVID-19 vaccines, skin testing (ST) to the vaccine/excipients (polyethylene glycol[PEG] and polysorbate 80 [PS]) has been recommended, but has unknown accuracy. To assess vaccine/excipient ST accuracy in predicting all-severity immediate allergic reactions upon re-vaccination, systematic review was performed searching Medline, EMBASE, Web of Science, and the WHO global coronavirus database (inception-Oct 4, 2021) for studies addressing immediate (≤4 h post-vaccination) all-severity allergic reactions to 2nd mRNA COVID-19 vaccination in persons with 1st dose immediate allergic reactions. Cases evaluating delayed reactions, change of vaccine platform, or revaccination without vaccine/excipient ST were excluded. Meta-analysis of diagnostic testing accuracy was performed using Bayesian methods. The GRADE approach evaluated certainty of the evidence, and QUADAS-2 assessed risk of bias. Among 20 studies of mRNA COVID-19 first dose vaccine reactions, 317 individuals underwent 578 ST to any one or combination of vaccine, PEG, or PS, and were re-vaccinated with the same vaccine. Test sensitivity for either mRNA vaccine was 0.2 (95%CrI 0.01-0.52) and specificity 0.97 (95%CrI 0.9-1). PEG test sensitivity was 0.02 (95%CrI 0.00-0.07) and specificity 0.99 (95%CrI 0.96-1). PS test sensitivity was 0.03 (95%CrI 0.00-0.0.11) and specificity 0.97 (95%CrI 0.91-1). Combined for use of any of the 3 testing agents, sensitivity was 0.03 (95%CrI 0.00-0.08) and specificity was 0.98 (95%CrI 0.95-1.00). Certainty of evidence was moderate. ST has low sensitivity but high specificity in predicting all-severity repeat immediate allergic reactions to the same agent, among persons with 1st dose immediate allergic reactions to mRNA COVID-19 vaccines. mRNA COVID-19 vaccine or excipient ST has limited risk assessment utility.

Novel Analysis Methods to Mine Immune-Mediated Phenotypes and Find Genetic Variation Within the Electronic Health Record (Roadmap for Phenotype to Genotype: Immunogenomics)

The field of immunogenomics has the opportunity for accelerated genetic discovery aided by the maturation of electronic health records (EHRs) linked to DNA biobanks. Novel analysis methods in deep phenotyping of EHR data allow the full realization of the paired and increasingly dense genetic/phenotypic information available. This enables researchers to uncover genetic risk factors for the prevention and optimal treatment of immune-mediated diseases and immune-mediated adverse drug reactions. This article reviews the background of EHRs linked to DNA biobanks, potential applications to immunogenomic discovery, and current and emerging techniques in EHR-based deep phenotyping.

Risk of Second Allergic Reaction to SARS-CoV-2 Vaccines: A Systematic Review and Meta-analysis

IMPORTANCE

Vaccination against SARS-CoV-2 is a highly effective strategy to prevent infection and severe COVID-19 outcomes. The best strategy for a second dose of vaccine among persons who had an immediate allergic reaction to their first SARS CoV-2 vaccination is unclear.

OBJECTIVE

To assess the risk of severe immediate allergic reactions (eg, anaphylaxis) to a second dose of SARS-CoV-2 mRNA vaccine among persons with immediate allergic reactions to their first vaccine dose.

DATA SOURCES

MEDLINE, Embase, Web of Science, and the World Health Organization Global Coronavirus database were searched from inception through October 4, 2021.

STUDY SELECTION

Included studies addressed immediate allergic reactions of any severity to a second SARS-CoV-2 vaccine dose in persons with a known or suspected immediate allergic reaction (<4 hours after vaccination) after their first SARS-CoV-2 vaccine dose. Studies describing a second vaccine dose among persons reporting delayed reactions (>4 hours after vaccination) were excluded.

DATA EXTRACTION AND SYNTHESIS

Paired reviewers independently selected studies, extracted data, and assessed risk of bias. Random-effects models were used for meta-analysis. The GRADE (Grading of Recommendation, Assessment, Development, and Evaluation) approach evaluated certainty of the evidence.

MAIN OUTCOMES AND MEASURES

Risk of severe immediate allergic reaction and repeated severe immediate allergic reactions with a second vaccine dose. Reaction severity was defined by the reporting investigator, using Brighton Collaboration Criteria, Ring and Messmer criteria, World Allergy Organization criteria, or National Institute of Allergy and Infectious Diseases criteria.

RESULTS

Among 22 studies of SARS-CoV-2 mRNA vaccines, 1366 individuals (87.8% women; mean age, 46.1 years) had immediate allergic reactions to their first vaccination. Analysis using the pooled random-effects model found that 6 patients developed severe immediate allergic reactions after their second vaccination (absolute risk, 0.16% [95% CI, 0.01%-2.94%]), 232 developed mild symptoms (13.65% [95% CI, 7.76%-22.9%]), and, conversely, 1360 tolerated the dose (99.84% [95% CI, 97.09%-99.99%]). Among 78 persons with severe immediate allergic reactions to their first SARS-CoV-2 mRNA vaccination, 4 people (4.94% [95% CI, 0.93%-22.28%]) had a second severe immediate reaction, and 15 had nonsevere symptoms (9.54% [95% CI, 2.18%-33.34%]). There were no deaths. Graded vaccine dosing, skin testing, and premedication as risk-stratification strategies did not alter the findings. Certainty of evidence was moderate for those with any allergic reaction to the first dose and low for those with severe allergic reactions to the first dose.

CONCLUSIONS AND RELEVANCE

In this systematic review and meta-analysis of case studies and case reports, the risk of immediate allergic reactions and severe immediate reactions or anaphylaxis associated with a second dose of an SARS-CoV-2 mRNA vaccine was low among persons who experienced an immediate allergic reaction to their first dose. These findings suggest that revaccination of individuals with an immediate allergic reaction to a first SARS-CoV-2 mRNA vaccine dose in a supervised setting equipped to manage severe allergic reactions can be safe.

The Admission Checklist: The key steps and responsibilities for the admitting resident

The process of admitting patients from the emergency department to the general medicine floor is foundational to the medical training process and medical practice more generally. Yet this process is rife with potential error if not approached systematically, and residents rarely receive explicit teaching in this area. The creation of an “Admission Checklist” proposed by the authors could serve the function of reducing error and enhancing inter-provider communication throughout this process. Such a checklist could improve trainee experience and education, and ultimately allow for improved outcomes for patients during transitions of care.

Safety Evaluation of the Second Dose of Messenger RNA COVID-19 Vaccines in Patients With Immediate Reactions to the First Dose

This case series study examines the safety of second doses of messenger RNA COVID-19 vaccines after first-dose allergic reactions.

Hidden Dangers: Recognizing Excipients as Potential Causes of Drug and Vaccine Hypersensitivity Reactions

Excipients are necessary as a support to the active ingredients in drugs, vaccines, and other products, and they contribute to their stability, preservation, pharmacokinetics, bioavailability, appearance, and acceptability. For both drugs and vaccines, these are rare reactions; however, for vaccines, they are the primary cause of immediate hypersensitivity. Suspicion for these "hidden dangers" should be high, in particular, when anaphylaxis has occurred in association with multiple chemically distinct drugs. Common excipients implicated include gelatin, carboxymethylcellulose, polyethylene glycols, and products related to polyethylene glycols in immediate hypersensitivity reactions and propylene glycol in delayed hypersensitivity reactions. Complete evaluation of a suspected excipient reaction requires detailed information from the product monograph and package insert to identify all ingredients that are present and to understand the function and structure for these chemicals. This knowledge helps develop a management plan that may include allergy testing to identify the implicated component and to give patients detailed information for future avoidance of relevant foods, drugs, and vaccines. Excipient reactions should be particularly considered for specific classes of drugs where they have been commonly found to be the culprit (eg, corticosteroids, injectable hormones, immunotherapies, monoclonal antibodies, and vaccines). We provide a review of the evidence-based literature outlining epidemiology and mechanisms of excipient reactions and provide strategies for heightened recognition and allergy testing.

Emerging Causes of Drug-Induced Anaphylaxis: A Review of Anaphylaxis-Associated Reports in the FDA Adverse Event Reporting System (FAERS)

BACKGROUND

Drug-induced anaphylaxis is a well-known adverse drug reaction for some drug classes, but emerging drug causes of anaphylaxis and novel mechanisms may contribute in unrecognized ways.

OBJECTIVE

We sought to determine the top drugs reported in association with anaphylaxis and anaphylaxis followed by death in the Food and Drug Administration Adverse Event Reporting System (FAERS).

METHODS

We reviewed the publicly available FAERS database from 1999 to 2019. Using search terms "anaphylactic shock" or "anaphylactic reaction" and sorting cases by generic drug names, we counted and trended reports to FAERS in which a drug was associated with anaphylaxis or anaphylaxis followed by death.

RESULTS

From 1999 to 2019, there were 17,506,002 adverse drug events reported in FAERS, of which 47,496 (0.27%) were reported as anaphylaxis. Excluding patients without age, sex, or country data, respectively, the median age of patients in reports of anaphylaxis was 52 (interquartile range: 28), 62.71% were female, and 13,899 of 34,381 (40.43%) reports were from the United States. There were 2984 of 47,496 (6.28%) reports of anaphylaxis followed by death. Top drug classes associated with anaphylaxis in FAERS were antibiotics, monoclonal antibodies (mAbs), nonsteroidal anti-inflammatory drugs, and acetaminophen. Top drug classes associated with anaphylaxis deaths were antibiotics, radiocontrast agents, and intraoperative agents. Linear regression demonstrated reports of anaphylaxis to mAbs increasing at an average rate of 0.77% of total anaphylaxis reports per year (95% confidence interval: 0.65, 0.88) from 2.00% in 1999 to 17.37% in 2019, faster than any other drug class.

CONCLUSION

Antibiotics were highly reported for anaphylaxis overall and anaphylaxis followed by death. Increasing reports were noted for anaphylaxis to mAb therapies.