PI Prob: A risk prediction and clinical guidance system for evaluating patients with recurrent infections

Artificial Intelligence and Machine Learning for Inborn Errors of Immunity: Current State and Future Promise

Artificial intelligence (AI) and machine learning (ML) research within medicine has exponentially increased over the last decade, with studies showcasing the potential of AI/ML algorithms to improve clinical practice and outcomes. Ongoing research and efforts to develop AI-based models have expanded to aid in the identification of inborn errors of immunity (IEI). The use of larger electronic health record data sets, coupled with advances in phenotyping precision and enhancements in ML techniques, has the potential to significantly improve the early recognition of IEI, thereby increasing access to equitable care. In this review, we provide a comprehensive examination of AI/ML for IEI, covering the spectrum from data preprocessing for AI/ML analysis to current applications within immunology, and address the challenges associated with implementing clinical decision support systems to refine the diagnosis and management of IEI.

Validating inborn error of immunity prevalence and risk with nationally representative electronic health record data

BACKGROUND

The 10 Warning Signs of Primary Immunodeficiency were created 30 years ago to advance recognition of inborn errors of immunity (IEI). However, no population-level assessment of their utility applied to electronic health record (EHR) data has been conducted.

OBJECTIVE

We sought to quantify the value of having ≥2 warning signs (WS) toward diagnosing IEI using a highly representative real-world US cohort. A secondary goal was estimating the US prevalence of IEI.

METHODS

In this cohort study, we accessed normalized and de-identified EHR data on 152 million US patients. An IEI cohort (n = 41,080), in which patients were defined by having at least 1 verifiable IEI diagnosis placed ≥2 times in their record, was compared with a matched set of controls (n = 250,262). WS were encoded along with relevant diagnoses, relative weights were calculated, and the proportion of IEI cases versus controls with ≥2 WS was compared.

RESULTS

The proportion of IEI cases with ≥2 WS significantly differed from controls (0.33 vs 0.031; P < .0005, χ2 test). We also estimated a US IEI prevalence of 6 per 10,000 individuals (41,080/73,165,655; 0.056%). WS 9 (≥2 deep-seated infections), 7 (fungal infections), 5 (failure to thrive) and 4 (≥2 pneumonias in 1 year) were the most heavily weighted among the IEI cohort.

CONCLUSIONS

This nationally representative US-based cohort study demonstrates that presence of WS and associated clinical diagnoses can facilitate identification of patients with IEI from EHR data. In addition, we estimate that 6 in 10,000, or approximately 150,000 to 200,000 individuals are affected by IEI across the United States.

Objectivizing issues in the diagnosis of complex rare diseases: lessons learned from testing existing diagnosis support systems on ciliopathies

BACKGROUND

There are approximately 8,000 different rare diseases that affect roughly 400 million people worldwide. Many of them suffer from delayed diagnosis. Ciliopathies are rare monogenic disorders characterized by a significant phenotypic and genetic heterogeneity that raises an important challenge for clinical diagnosis. Diagnosis support systems (DSS) applied to electronic health record (EHR) data may help identify undiagnosed patients, which is of paramount importance to improve patients' care. Our objective was to evaluate three online-accessible rare disease DSSs using phenotypes derived from EHRs for the diagnosis of ciliopathies.

METHODS

Two datasets of ciliopathy cases, either proven or suspected, and two datasets of controls were used to evaluate the DSSs. Patient phenotypes were automatically extracted from their EHRs and converted to Human Phenotype Ontology terms. We tested the ability of the DSSs to diagnose cases in contrast to controls based on Orphanet ontology.

RESULTS

A total of 79 cases and 38 controls were selected. Performances of the DSSs on ciliopathy real world data (best DSS with area under the ROC curve = 0.72) were not as good as published performances on the test set used in the DSS development phase. None of these systems obtained results which could be described as "expert-level". Patients with multisystemic symptoms were generally easier to diagnose than patients with isolated symptoms. Diseases easily confused with ciliopathy generally affected multiple organs and had overlapping phenotypes. Four challenges need to be considered to improve the performances: to make the DSSs interoperable with EHR systems, to validate the performances in real-life settings, to deal with data quality, and to leverage methods and resources for rare and complex diseases.

CONCLUSION

Our study provides insights into the complexities of diagnosing highly heterogenous rare diseases and offers lessons derived from evaluation existing DSSs in real-world settings. These insights are not only beneficial for ciliopathy diagnosis but also hold relevance for the enhancement of DSS for various complex rare disorders, by guiding the development of more clinically relevant rare disease DSSs, that could support early diagnosis and finally make more patients eligible for treatment.

Natural language processing of clinical notes enables early inborn error of immunity risk ascertainment

Background

There are now approximately 450 discrete inborn errors of immunity (IEI) described; however, diagnostic rates remain suboptimal. Use of structured health record data has proven useful for patient detection but may be augmented by natural language processing (NLP). Here we present a machine learning model that can distinguish patients from controls significantly in advance of ultimate diagnosis date.

Objective

We sought to create an NLP machine learning algorithm that could identify IEI patients early during the disease course and shorten the diagnostic odyssey.

Methods

Our approach involved extracting a large corpus of IEI patient clinical-note text from a major referral center's electronic health record (EHR) system and a matched control corpus for comparison. We built text classifiers with simple machine learning methods and trained them on progressively longer time epochs before date of diagnosis.

Results

The top performing NLP algorithm effectively distinguished cases from controls robustly 36 months before ultimate clinical diagnosis (area under precision recall curve > 0.95). Corpus analysis demonstrated that statistically enriched, IEI-relevant terms were evident 24+ months before diagnosis, validating that clinical notes can provide a signal for early prediction of IEI.

Conclusion

Mining EHR notes with NLP holds promise for improving early IEI patient detection.

Proceedings from the inaugural Artificial Intelligence in Primary Immune Deficiencies (AIPID) conference

Here, we summarize the proceedings of the inaugural Artificial Intelligence in Primary Immune Deficiencies conference, during which experts and advocates gathered to advance research into the applications of artificial intelligence (AI), machine learning, and other computational tools in the diagnosis and management of inborn errors of immunity (IEIs). The conference focused on the key themes of expediting IEI diagnoses, challenges in data collection, roles of natural language processing and large language models in interpreting electronic health records, and ethical considerations in implementation. Innovative AI-based tools trained on electronic health records and claims databases have discovered new patterns of warning signs for IEIs, facilitating faster diagnoses and enhancing patient outcomes. Challenges in training AIs persist on account of data limitations, especially in cases of rare diseases, overlapping phenotypes, and biases inherent in current data sets. Furthermore, experts highlighted the significance of ethical considerations, data protection, and the necessity for open science principles. The conference delved into regulatory frameworks, equity in access, and the imperative for collaborative efforts to overcome these obstacles and harness the transformative potential of AI. Concerted efforts to successfully integrate AI into daily clinical immunology practice are still needed.

Performance and clinical utility of a new supervised machine-learning pipeline in detecting rare ciliopathy patients based on deep phenotyping from electronic health records and semantic similarity

BACKGROUND

Rare diseases affect approximately 400 million people worldwide. Many of them suffer from delayed diagnosis. Among them, NPHP1-related renal ciliopathies need to be diagnosed as early as possible as potential treatments have been recently investigated with promising results. Our objective was to develop a supervised machine learning pipeline for the detection of NPHP1 ciliopathy patients from a large number of nephrology patients using electronic health records (EHRs).

METHODS AND RESULTS

We designed a pipeline combining a phenotyping module re-using unstructured EHR data, a semantic similarity module to address the phenotype dependence, a feature selection step to deal with high dimensionality, an undersampling step to address the class imbalance, and a classification step with multiple train-test split for the small number of rare cases. The pipeline was applied to thirty NPHP1 patients and 7231 controls and achieved good performances (sensitivity 86% with specificity 90%). A qualitative review of the EHRs of 40 misclassified controls showed that 25% had phenotypes belonging to the ciliopathy spectrum, which demonstrates the ability of our system to detect patients with similar conditions.

CONCLUSIONS

Our pipeline reached very encouraging performance scores for pre-diagnosing ciliopathy patients. The identified patients could then undergo genetic testing. The same data-driven approach can be adapted to other rare diseases facing underdiagnosis challenges.

Large-scale deep learning analysis to identify adult patients at risk for combined and common variable immunodeficiencies

BACKGROUND

Primary immunodeficiency (PI) is a group of heterogeneous disorders resulting from immune system defects. Over 70% of PI is undiagnosed, leading to increased mortality, co-morbidity and healthcare costs. Among PI disorders, combined immunodeficiencies (CID) are characterized by complex immune defects. Common variable immunodeficiency (CVID) is among the most common types of PI. In light of available treatments, it is critical to identify adult patients at risk for CID and CVID, before the development of serious morbidity and mortality.

METHODS

We developed a deep learning-based method (named "TabMLPNet") to analyze clinical history from nationally representative medical claims from electronic health records (Optum® data, covering all US), evaluated in the setting of identifying CID/CVID in adults. Further, we revealed the most important CID/CVID-associated antecedent phenotype combinations. Four large cohorts were generated: a total of 47,660 PI cases and (1:1 matched) controls.

RESULTS

The sensitivity/specificity of TabMLPNet modeling ranges from 0.82-0.88/0.82-0.85 across cohorts. Distinctive combinations of antecedent phenotypes associated with CID/CVID are identified, consisting of respiratory infections/conditions, genetic anomalies, cardiac defects, autoimmune diseases, blood disorders and malignancies, which can possibly be useful to systematize the identification of CID and CVID.

CONCLUSIONS

We demonstrated an accurate method in terms of CID and CVID detection evaluated on large-scale medical claims data. Our predictive scheme can potentially lead to the development of new clinical insights and expanded guidelines for identification of adult patients at risk for CID and CVID as well as be used to improve patient outcomes on population level.

Clinical Validation of a Primary Antibody Deficiency Screening Algorithm for Primary Care

PURPOSE

The diagnostic delay of primary antibody deficiencies (PADs) is associated with increased morbidity, mortality, and healthcare costs. Therefore, a screening algorithm was previously developed for the early detection of patients at risk of PAD in primary care. We aimed to clinically validate and optimize the PAD screening algorithm by applying it to a primary care database in the Netherlands.

METHODS

The algorithm was applied to a data set of 61,172 electronic health records (EHRs). Four hundred high-scoring EHRs were screened for exclusion criteria, and remaining patients were invited for serum immunoglobulin analysis and referred if clinically necessary.

RESULTS

Of the 104 patients eligible for inclusion, 16 were referred by their general practitioner for suspected PAD, of whom 10 had a PAD diagnosis. In patients selected by the screening algorithm and included for laboratory analysis, prevalence of PAD was ~ 1:10 versus 1:1700-1:25,000 in the general population. To optimize efficiency of the screening process, we refitted the algorithm with the subset of high-risk patients, which improved the area under the curve-receiver operating characteristics curve value to 0.80 (95% confidence interval 0.63-0.97). We propose a two-step screening process, first applying the original algorithm to distinguish high-risk from low-risk patients, then applying the optimized algorithm to select high-risk patients for serum immunoglobulin analysis.

CONCLUSION

Using the screening algorithm, we were able to identify 10 new PAD patients from a primary care population, thus reducing diagnostic delay. Future studies should address further validation in other populations and full cost-effectiveness analyses.

REGISTRATION

Clinicaltrials.gov record number NCT05310604, first submitted 25 March 2022.

Development of a primary care screening algorithm for the early detection of patients at risk of primary antibody deficiency

BACKGROUND

Primary antibody deficiencies (PAD) are characterized by a heterogeneous clinical presentation and low prevalence, contributing to a median diagnostic delay of 3-10 years. This increases the risk of morbidity and mortality from undiagnosed PAD, which may be prevented with adequate therapy. To reduce the diagnostic delay of PAD, we developed a screening algorithm using primary care electronic health record (EHR) data to identify patients at risk of PAD. This screening algorithm can be used as an aid to notify general practitioners when further laboratory evaluation of immunoglobulins should be considered, thereby facilitating a timely diagnosis of PAD.

METHODS

Candidate components for the algorithm were based on a broad range of presenting signs and symptoms of PAD that are available in primary care EHRs. The decision on inclusion and weight of the components in the algorithm was based on the prevalence of these components among PAD patients and control groups, as well as clinical rationale.

RESULTS

We analyzed the primary care EHRs of 30 PAD patients, 26 primary care immunodeficiency patients and 58,223 control patients. The median diagnostic delay of PAD patients was 9.5 years. Several candidate components showed a clear difference in prevalence between PAD patients and controls, most notably the mean number of antibiotic prescriptions in the 4 years prior to diagnosis (5.14 vs. 0.48). The final algorithm included antibiotic prescriptions, diagnostic codes for respiratory tract and other infections, gastro-intestinal complaints, auto-immune symptoms, malignancies and lymphoproliferative symptoms, as well as laboratory values and visits to the general practitioner.

CONCLUSIONS

In this study, we developed a screening algorithm based on a broad range of presenting signs and symptoms of PAD, which is suitable to implement in primary care. It has the potential to considerably reduce diagnostic delay in PAD, and will be validated in a prospective study. Trial registration The consecutive prospective study is registered at clinicaltrials.gov under NCT05310604.

A validated artificial intelligence-based pipeline for population-wide primary immunodeficiency screening

BACKGROUND

Identification of patients with underlying inborn errors of immunity and inherent susceptibility to infection remains challenging. The ensuing protracted diagnostic odyssey for such patients often results in greater morbidity and suboptimal outcomes, underscoring a need to develop systematic methods for improving diagnostic rates.

OBJECTIVE

The principal aim of this study is to build and validate a generalizable analytical pipeline for population-wide detection of infection susceptibility and risk of primary immunodeficiency.

METHODS

This prospective, longitudinal cohort study coupled weighted rules with a machine learning classifier for risk stratification. Claims data were analyzed from a diverse population (n = 427,110) iteratively over 30 months. Cohort outcomes were enumerated for new diagnoses, hospitalizations, and acute care visits. This study followed TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) standards.

RESULTS

Cohort members initially identified as high risk were proportionally more likely to receive a diagnosis of primary immunodeficiency compared to those at low-medium risk or those without claims of interest respectively (9% vs 1.5% vs 0.2%; P < .001, chi-square test). Subsequent machine learning stratification enabled an annualized individual snapshot of complexity for triaging referrals. This study's top-performing machine learning model for visit-level prediction used a single dense layer neural network architecture (area under the receiver-operator characteristic curve = 0.98; F1 score = 0.98).

CONCLUSIONS

A 2-step analytical pipeline can facilitate identification of individuals with primary immunodeficiency and accurately quantify clinical risk.

Early Diagnosis of Primary Immunodeficiency Disease Using Clinical Data and Machine Learning

BACKGROUND

Primary immunodeficiency diseases (PIDD) are a group of immune-related disorders that have a current median delay of diagnosis between 6 and 9 years. Early diagnosis and treatment of PIDD has been associated with improved patient outcomes.

OBJECTIVE

To develop a machine learning model using elements within the electronic health record data that are related to prior symptomatic treatment to predict PIDD.

METHODS

We conducted a retrospective study of patients with PIDD identified using inclusion criteria of PIDD-related diagnoses, immunodeficiency-specific medications, and low immunoglobulin levels. We constructed a control group of age-, sex-, and race-matched patients with asthma. The primary outcome was the diagnosis of PIDD. We considered comorbidities, laboratory tests, medications, and radiological orders as features, all before diagnosis and indicative of symptom-related treatment. Features were presented sequentially to logistic regression, elastic net, and random forest classifiers, which were trained using a nested cross-validation approach.

RESULTS

Our cohort consisted of 6422 patients, of whom 247 (4%) were diagnosed with PIDD. Our logistic regression model with comorbidities demonstrated good discrimination between patients with PIDD and those with asthma (c-statistic: 0.62 [0.58-0.65]). Adding laboratory results, medications, and radiological orders improved discrimination (c-statistic: 0.70 vs 0.62, P < .001), sensitivity, and specificity. Extending to the advanced machine learning models did not improve performance.

CONCLUSIONS

We developed a prediction model for early diagnosis of PIDD using historical data that are related to symptomatic care, which has potential to fill an important need in reducing the time to diagnose PIDD, leading to better outcomes for immunodeficient patients.

"Common variable immunodeficiency: Challenges for diagnosis"

Common variable immunodeficiency is a heterogeneous condition characterized by B cell dysfunction with reduced serum immunoglobulin levels and a highly variable spectrum of clinical manifestations ranging from recurrent infections to autoimmune disease. The diagnosis of CVID is often challenging due to the diverse clinical presentation of patients and the existence of multiple diagnostic criteria without a universally adopted consensus. Laboratory evaluation to assist with diagnosis currently includes serum immunoglobulin testing, immunophenotyping, assessment of vaccine response, and genetic testing. Additional emerging techniques include investigation of the B cell repertoire and the use of machine learning algorithms. Advances in our understanding of common variable immunodeficiency will ultimately contribute to earlier diagnosis and novel interventions with the goal of improving prognosis for these patients.

Respiratory Infections in Patients with Primary Immunodeficiency

Recurrent and life-threatening respiratory infections are nearly universal in patients with primary immunodeficiency diseases (PIDD). Early recognition, aggressive treatment, and prophylaxis with antimicrobials and immunoglobulin replacement have been the mainstays of management and will be reviewed here with an emphasis on respiratory infections. Genetic discoveries have allowed direct translation of research to clinical practice, improving our understanding of clinical patterns of pathogen susceptibilities and guiding prophylaxis. The recent identification of inborn errors in type I interferon signaling as a basis for life-threatening viral infections in otherwise healthy individuals suggests another targetable pathway for treatment and/or prophylaxis. The future of PIDD diagnosis will certainly involve early genetic identification by newborn screening before onset of infections, with early treatment offering the potential of preventing disease complications such as chronic lung changes. Gene editing approaches offer tremendous therapeutic potential, with rapidly emerging delivery systems. Antiviral therapies are desperately needed, and specific cellular therapies show promise in patients requiring hematopoietic stem cell transplantation. The introduction of approved therapies for clinical use in PIDD is limited by the difficulty of studying outcomes in rare patients/conditions with conventional clinical trials.

Seeing the Forest for the Trees: Evaluating Population Data in Allergy-Immunology

A population-level study is essential for understanding treatment effects, epidemiologic phenomena, and health care best practices. Evaluating large populations and associated data requires an analytic framework, which is commonly used by statisticians, epidemiologists, and data scientists. This document will serve to provide an overview of these commonly employed methods in allergy and immunology research. We will draw upon recent examples from the allergy-immunology literature to contextualize discrete principles of relevance to population-level analysis that include statistical features of a study population, elements of statistical inference, regression analysis, and an overview of machine learning practices. Our intent is to guide the reader through a practical description of this important quantitative discipline and facilitate greater understanding about data and result display in the medical literature.