Reducing diagnostic delays in Acute Hepatic Porphyria using electronic health records data and machine learning: a multicenter development and validation study

Importance

Acute Hepatic Porphyria (AHP) is a group of rare but treatable conditions associated with diagnostic delays of fifteen years on average. The advent of electronic health records (EHR) data and machine learning (ML) may improve the timely recognition of rare diseases like AHP. However, prediction models can be difficult to train given the limited case numbers, unstructured EHR data, and selection biases intrinsic to healthcare delivery.

Objective

To train and characterize models for identifying patients with AHP.

Design Setting and Participants

This diagnostic study used structured and notes-based EHR data from two centers at the University of California, UCSF (2012-2022) and UCLA (2019-2022). The data were split into two cohorts (referral, diagnosis) and used to develop models that predict: 1) who will be referred for testing of acute porphyria, amongst those who presented with abdominal pain (a cardinal symptom of AHP), and 2) who will test positive, amongst those referred. The referral cohort consisted of 747 patients referred for testing and 99,849 contemporaneous patients who were not. The diagnosis cohort consisted of 72 confirmed AHP cases and 347 patients who tested negative. Cases were female predominant and 6-75 years old at the time of diagnosis. Candidate models used a range of architectures. Feature selection was semi-automated and incorporated publicly available data from knowledge graphs.

Main Outcomes and Measures

F-score on an outcome-stratified test set.

Results

The best center-specific referral models achieved an F-score of 86-91%. The best diagnosis model achieved an F-score of 92%. To further test our model, we contacted 372 current patients who lack an AHP diagnosis but were predicted by our models as potentially having it (≥ 10% probability of referral, ≥ 50% of testing positive). However, we were only able to recruit 10 of these patients for biochemical testing, all of whom were negative. Nonetheless, post hoc evaluations suggested that these models could identify 71% of cases earlier than their diagnosis date, saving 1.2 years.

Conclusions and Relevance

ML can reduce diagnostic delays in AHP and other rare diseases. Robust recruitment strategies and multicenter coordination will be needed to validate these models before they can be deployed.

Quantifying the impact of symptomatic acute hepatic porphyria on well-being via patient-reported outcomes: Results from the Porphyria Worldwide Patient Experience Research (POWER) study

Acute hepatic porphyria (AHP) is a group of rare genetic diseases of heme biosynthesis resulting in severe neurovisceral attacks and chronic complications that negatively impact patients' well-being. This study evaluated the impacts of AHP on patients' physical and emotional health from a global perspective. Adult patients from the United States, Italy, Spain, Australia, Mexico, and Brazil with AHP with >1 porphyria attack within the past 2 years or receiving intravenous hemin and/or glucose for attack prevention completed an online survey assessing demographics, health characteristics, and patient-reported outcomes. Results were analyzed collectively and by patient subgroups. Ninety-two patients with AHP across the six countries completed the survey. More than 70% of patients reported that their physical, emotional, and financial health was fair or poor. Among patients who reported pain, fatigue, and muscle weakness, 94.3%, 95.6%, and 91.4%, respectively, reported that these symptoms limited daily activities. Moderate to severe depression was present in 58.7% of patients, and moderate to severe anxiety in 48.9% of patients. Of the 47% of patients who were employed, 36.8% reported loss in productivity while at work. Among patients, 85.9% reported that they had to change or modify goals that were important to them because of AHP. Aside from differences in healthcare utilization and pain severity, scores did not significantly vary with attack rate or use of hemin or glucose prophylactic treatments. AHP substantially impacts patients' physical and emotional well-being, regardless of hemin or glucose prophylactic treatment or frequency of attacks. This multinational study demonstrates that there is substantial disease burden for patients with AHP, even among those experiencing sporadic attacks or using prophylactic treatment.

Patient and caregiver experiences of living with acute hepatic porphyria in the UK: a mixed-methods study

Background

This study used quantitative and qualitative research methods to analyze how acute hepatic porphyria (AHP) affects patients with varying annualized porphyria attack rates. The overall impact of AHP on patients and caregivers, including their quality of life, was explored. The nature and treatment of acute attacks, experiences of long-term heme arginate treatment and access to other appropriate treatment, and the extent of and treatment for chronic symptoms were also investigated within this study.

Methods

Patient and caregiver data were collected via an online survey of members of the British Porphyria Association, followed by an optional 1-h telephone interview.

Results

Thirty-eight patients and 10 caregivers responded to the survey. Of those, 10 patients and three caregivers completed follow-up interviews. Overall, 19 patients (50%) had experienced an acute attack within the previous 2 years, and the severity and types of symptoms experienced during or between acute attacks varied considerably. There were no clear definitions among patients for ‘mild’ or ‘severe’ attacks. Treatments and treatment settings used to manage attacks also varied. Following unsatisfactory care experiences at hospitals, some patients reported avoiding further hospital services for later attacks. Therefore, using settings of care as a measure of attack severity should be avoided. Ninety-four percent of patients also experienced chronic symptoms, which were as varied as acute attacks. Pain was the predominant chronic symptom and was managed with opioids in severe cases. Regardless of AAR, porphyria heavily impacted the daily lives of patients and caregivers. Although patients experiencing frequent attacks generally endured a greater impact on their daily life, patients with less frequent attacks also experienced impacts on all domains (social, leisure activities, relationship with family, relationships, psychological wellbeing, finances, employment, and study). Caregivers were most affected in the finance, relationships with family, and employment domains, and just over half of the caregivers reported a moderate impact on their psychological wellbeing.

Conclusions/implications

The burden of illness with AHP is high across all patients, regardless of frequency of attacks, and AHP negatively affects patients and caregivers alike.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13023-021-01816-2.

Detecting rare diseases in electronic health records using machine learning and knowledge engineering: Case study of acute hepatic porphyria

BACKGROUND

With the growing adoption of the electronic health record (EHR) worldwide over the last decade, new opportunities exist for leveraging EHR data for detection of rare diseases. Rare diseases are often not diagnosed or delayed in diagnosis by clinicians who encounter them infrequently. One such rare disease that may be amenable to EHR-based detection is acute hepatic porphyria (AHP). AHP consists of a family of rare, metabolic diseases characterized by potentially life-threatening acute attacks and chronic debilitating symptoms. The goal of this study was to apply machine learning and knowledge engineering to a large extract of EHR data to determine whether they could be effective in identifying patients not previously tested for AHP who should receive a proper diagnostic workup for AHP.

METHODS AND FINDINGS

We used an extract of the complete EHR data of 200,000 patients from an academic medical center and enriched it with records from an additional 5,571 patients containing any mention of porphyria in the record. After manually reviewing the records of all 47 unique patients with the ICD-10-CM code E80.21 (Acute intermittent [hepatic] porphyria), we identified 30 patients who were positive cases for our machine learning models, with the rest of the patients used as negative cases. We parsed the record into features, which were scored by frequency of appearance and filtered using univariate feature analysis. We manually choose features not directly tied to provider attributes or suspicion of the patient having AHP. We trained on the full dataset, with the best cross-validation performance coming from support vector machine (SVM) algorithm using a radial basis function (RBF) kernel. The trained model was applied back to the full data set and patients were ranked by margin distance. The top 100 ranked negative cases were manually reviewed for symptom complexes similar to AHP, finding four patients where AHP diagnostic testing was likely indicated and 18 patients where AHP diagnostic testing was possibly indicated. From the top 100 ranked cases of patients with mention of porphyria in their record, we identified four patients for whom AHP diagnostic testing was possibly indicated and had not been previously performed. Based solely on the reported prevalence of AHP, we would have expected only 0.002 cases out of the 200 patients manually reviewed.

CONCLUSIONS

The application of machine learning and knowledge engineering to EHR data may facilitate the diagnosis of rare diseases such as AHP. Further work will recommend clinical investigation to identified patients' clinicians, evaluate more patients, assess additional feature selection and machine learning algorithms, and apply this methodology to other rare diseases. This work provides strong evidence that population-level informatics can be applied to rare diseases, greatly improving our ability to identify undiagnosed patients, and in the future improve the care of these patients and our ability study these diseases. The next step is to learn how best to apply these EHR-based machine learning approaches to benefit individual patients with a clinical study that provides diagnostic testing and clinical follow up for those identified as possibly having undiagnosed AHP.

A166 ANALYSIS OF SYMPTOMS, DIAGNOSTIC PATTERNS, AND CANADIAN PROVIDER PERSPECTIVE OF ACUTE HEPATIC PORPHYRIA

Background

Acute hepatic porphyria (AHP) is a family of rare genetic diseases, the most common being acute intermittent porphyria (AIP). AHP results from enzyme deficiencies involved in haem synthesis, leading to accumulation of neurotoxic haem intermediates, aminolaevulinic acid (ALA) and porphobilinogen (PBG), causing potentially life-threatening attacks and chronic symptoms. Patients afflicted by AHP often remain without a proper diagnosis for up to 15 years due to lack of awareness and testing. First-line diagnostic biochemical tests include measuring spot urinary ALA and PBG as both are elevated in the majority of AHP patients.

Aims

The study aimed to describe physicians experience diagnosing AHP and characterize patients globally, including Canada.

Methods

Physicians (n=175) who actively managed AHP patients (with and without recurrent attacks) in the preceding year were recruited from 9/2017–10/2017 to complete an online survey collecting information on demographics, familiarity with AHP and diagnostic tests, perspective on symptoms important to diagnosis, referral patterns, and treatment preferences. Physicians also completed a chart review of 546 patients and reported anonymized data on demographics, medical history, attacks, and symptoms. Data was analysed using descriptive statistics.

Results

Canadian physicians (n=15) practiced a mean of 19.7 years, 67% worked in community settings, and 53% were gastroenterologists. Symptoms informing AHP diagnosis included fatigue (93%), sensory loss (87%), mental confusion (87%), Abdominal pain (80%), red/dark urine (80%), vomiting (73%). AHP diagnostic tests considered informative for diagnosis included urinary ALA (87%) and PBG (80%); however, several non-specific tests were also commonly considered informative of AHP. Chronic symptoms reported included fatigue (75%), nausea/vomiting (70%), weakness (66%), pain (58%), anxiety (54%), diarrhea (41%), constipation (40%). Canadian physicians reported a mean of 58% AHP patients they manage being initially misdiagnosed. Global patients (n=546) were aged 40 years (mean), female (52%), with AIP (83%). Canadian patients (n=38) were aged 41 years (mean), female (61%), with AIP (78%). Patients had mean of 3.4 attacks and 1.6 hospitalizations in the past year.

Conclusions

This study highlights the challenges diagnosing AHP due to non-specificity of symptoms and limited understanding of diagnostic procedures. Due to the frequent presentation of gastrointestinal symptoms, AHP should be included in gastroenterologists’ differential diagnosis of patients presenting with non-specific abdominal pain. Among patients diagnosed with AHP, both acute attacks and chronic symptoms were reported, implicating both in the disease.

Funding Agencies

Alnylam Pharmaceuticals

1216. Cost-Effectiveness of Penicillin Skin Allergy Testing in Methicillin-Sensitive Staphylococcus aureus (MSSA) Bacteremia

Background

β-Lactams remain the gold standard for treatment of MSSA bacteremia due to superior outcomes compared with vancomycin. Approximately nine in 10 patients receiving penicillin skin testing (PST) will be de-labeled of a penicillin allergy and able to receive a β-lactam antibiotic. The study aims to evaluate the cost-effectiveness of penicillin allergy confirmation during acute care admission for methicillin-sensitive staphylococcus aureus (MSSA) bacteremia through a PST service.

Methods

A decision tree analysis was used to compare a PST intervention in patients with a registeredpenicillin allergy during an inpatient admission for MSSA bacteremia vs. usual care (No PST). The model was created from the health sector perspective with a 1-year time horizon. Patients with a penicillin allergy label were expected to receive vancomycin while patients with no penicillin allergy were expected to receive cefazolin. Potential inpatient, outpatient, and adverse reaction costs were considered in all arms of the model. The effects were measured in quality adjusted life years (QALY) and were calculated for patients who were cured, hospitalized, experienced severe adverse events, or died from MSSA infection.

Results

Patients who received PST services had a mean yearly cost of $12,802, mean quality adjusted life years (QALY) of 0.70, and mean cost/QALY of $18,311. The comparator group not receiving PST services had a mean yearly cost of $12,264, mean quality adjusted life years (QALY) of 0.64, and mean cost/QALY of $19,192. The model produced a final base case ICERof $8,966/QALY for receiving a PST during a hospital admission for the treatment of methicillin-sensitive staphylococcus aureus (MSSA) bacteremia.

Conclusion

Penicillin allergy confirmation through PST services was cost-effective for patients with a reported penicillin allergy admitted for MSSA bacteremia. Additional research to determine potential benefits of PST services beyond one year could further improve the cost-effectiveness of this intervention.

Disclosures

S. Meninger, ALK-Abelló: Grant Investigator, Research grant. E. Heil, ALK-Abelló: Grant Investigator, Research grant. T. J. Mattingly II, ALK-Abelló: Grant Investigator, Research grant.