Identification of undiagnosed atrial fibrillation patients using a machine learning risk prediction algorithm and diagnostic testing (PULsE-AI): Study protocol for a randomised controlled trial

Beyond Clinical Factors: Harnessing Artificial Intelligence and Multimodal Cardiac Imaging to Predict Atrial Fibrillation Recurrence Post-Catheter Ablation

Atrial fibrillation (AF) is the most common type of cardiac arrhythmia, with catheter ablation being a key alternative to medical treatment for restoring normal sinus rhythm. Despite advances in understanding AF pathogenesis, approximately 35% of patients experience AF recurrence at 12 months after catheter ablation. Therefore, accurate prediction of AF recurrence occurring after catheter ablation is important for patient selection and management. Conventional methods for predicting post-catheter ablation AF recurrence, which involve the use of univariate predictors and scoring systems, have played a supportive role in clinical decision-making. In an ever-changing landscape where technology is becoming ubiquitous within medicine, cardiac imaging and artificial intelligence (AI) could prove pivotal in enhancing AF recurrence predictions by providing data with independent predictive power and identifying key relationships in the data. This review comprehensively explores the existing methods for predicting the recurrence of AF following catheter ablation from different perspectives, including conventional predictors and scoring systems, cardiac imaging-based methods, and AI-based methods developed using a combination of demographic and imaging variables. By summarising state-of-the-art technologies, this review serves as a roadmap for developing future prediction models with enhanced accuracy, generalisability, and explainability, potentially contributing to improved care for patients with AF.

Availability of Evidence for Predictive Machine Learning Algorithms in Primary Care: A Systematic Review

Importance

The aging and multimorbid population and health personnel shortages pose a substantial burden on primary health care. While predictive machine learning (ML) algorithms have the potential to address these challenges, concerns include transparency and insufficient reporting of model validation and effectiveness of the implementation in the clinical workflow.

Objectives

To systematically identify predictive ML algorithms implemented in primary care from peer-reviewed literature and US Food and Drug Administration (FDA) and Conformité Européene (CE) registration databases and to ascertain the public availability of evidence, including peer-reviewed literature, gray literature, and technical reports across the artificial intelligence (AI) life cycle.

Evidence Review

PubMed, Embase, Web of Science, Cochrane Library, Emcare, Academic Search Premier, IEEE Xplore, ACM Digital Library, MathSciNet, AAAI.org (Association for the Advancement of Artificial Intelligence), arXiv, Epistemonikos, PsycINFO, and Google Scholar were searched for studies published between January 2000 and July 2023, with search terms that were related to AI, primary care, and implementation. The search extended to CE-marked or FDA-approved predictive ML algorithms obtained from relevant registration databases. Three reviewers gathered subsequent evidence involving strategies such as product searches, exploration of references, manufacturer website visits, and direct inquiries to authors and product owners. The extent to which the evidence for each predictive ML algorithm aligned with the Dutch AI predictive algorithm (AIPA) guideline requirements was assessed per AI life cycle phase, producing evidence availability scores.

Findings

The systematic search identified 43 predictive ML algorithms, of which 25 were commercially available and CE-marked or FDA-approved. The predictive ML algorithms spanned multiple clinical domains, but most (27 [63%]) focused on cardiovascular diseases and diabetes. Most (35 [81%]) were published within the past 5 years. The availability of evidence varied across different phases of the predictive ML algorithm life cycle, with evidence being reported the least for phase 1 (preparation) and phase 5 (impact assessment) (19% and 30%, respectively). Twelve (28%) predictive ML algorithms achieved approximately half of their maximum individual evidence availability score. Overall, predictive ML algorithms from peer-reviewed literature showed higher evidence availability compared with those from FDA-approved or CE-marked databases (45% vs 29%).

Conclusions and Relevance

The findings indicate an urgent need to improve the availability of evidence regarding the predictive ML algorithms' quality criteria. Adopting the Dutch AIPA guideline could facilitate transparent and consistent reporting of the quality criteria that could foster trust among end users and facilitating large-scale implementation.

The Artificial Intelligence Revolution in Stroke Care: A Decade of Scientific Evidence in Review

BACKGROUND

The emergence of artificial intelligence (AI) has significantly influenced the diagnostic evaluation of stroke and has revolutionized acute stroke care delivery. The scientific evidence evaluating the role of AI, especially in areas of stroke treatment and rehabilitation is limited but continues to accumulate. We performed a systemic review of current scientific evidence evaluating the use of AI in stroke evaluation and care and examined the publication trends during the past decade.

METHODS

A systematic search of electronic databases was conducted to identify all studies published from 2012 to 2022 that incorporated AI in any aspect of stroke care. Studies not directly relevant to stroke care in the context of AI and duplicate studies were excluded. The level of evidence and publication trends were examined.

RESULTS

A total of 623 studies were examined, including 101 reviews (16.2%), 9 meta-analyses (1.4%), 140 original articles on AI methodology (22.5%), 2 case reports (0.3%), 2 case series (0.3%), 31 case-control studies (5%), 277 cohort studies (44.5%), 16 cross-sectional studies (2.6%), and 45 experimental studies (7.2%). The highest published area of AI in stroke was diagnosis (44.1%) and the lowest was rehabilitation (12%). A 10-year trend analysis revealed a significant increase in AI literature in stroke care.

CONCLUSIONS

Most research on AI is in the diagnostic area of stroke care, with a recent noteworthy trend of increased research focus on stroke treatment and rehabilitation.

Pulse wave-based evaluation of the blood-supply capability of patients with heart failure via machine learning

Pulse wave, as a message carrier in the cardiovascular system (CVS), enables inferring CVS conditions while diagnosing cardiovascular diseases (CVDs). Heart failure (HF) is a major CVD, typically requiring expensive and time-consuming treatments for health monitoring and disease deterioration; it would be an effective and patient-friendly tool to facilitate rapid and precise non-invasive evaluation of the heart's blood-supply capability by means of powerful feature-abstraction capability of machine learning (ML) based on pulse wave, which remains untouched yet. Here we present an ML-based methodology, which is verified to accurately evaluate the blood-supply capability of patients with HF based on clinical data of 237 patients, enabling fast prediction of five representative cardiovascular function parameters comprising left ventricular ejection fraction (LVEF), left ventricular end-diastolic diameter (LVDd), left ventricular end-systolic diameter (LVDs), left atrial dimension (LAD), and peripheral oxygen saturation (SpO2). Two ML networks were employed and optimized based on high-quality pulse wave datasets, and they were validated consistently through statistical analysis based on the summary independent-samples t-test (p > 0.05), the Bland-Altman analysis with clinical measurements, and the error-function analysis. It is proven that evaluation of the SpO2, LAD, and LVDd performance can be achieved with the maximum error < 15%. While our findings thus demonstrate the potential of pulse wave-based, non-invasive evaluation of the blood-supply capability of patients with HF, they also set the stage for further refinements in health monitoring and deterioration prevention applications.

Undertaking multi-centre randomised controlled trials in primary care: learnings and recommendations from the PULsE-AI trial researchers

BACKGROUND

Conducting effective and translational research can be challenging and few trials undertake formal reflection exercises and disseminate learnings from them. Following completion of our multicentre randomised controlled trial, which was impacted by the COVID-19 pandemic, we sought to reflect on our experiences and share our thoughts on challenges, lessons learned, and recommendations for researchers undertaking or considering research in primary care.

METHODS

Researchers involved in the Prediction of Undiagnosed atriaL fibrillation using a machinE learning AlgorIthm (PULsE-AI) trial, conducted in England from June 2019 to February 2021 were invited to participate in a qualitative reflection exercise. Members of the Trial Steering Committee (TSC) were invited to attend a semi-structured focus group session, Principal Investigators and their research teams at practices involved in the trial were invited to participate in a semi-structured interview. Following transcription, reflexive thematic analysis was undertaken based on pre-specified themes of recruitment, challenges, lessons learned, and recommendations that formed the structure of the focus group/interview sessions, whilst also allowing the exploration of new themes that emerged from the data.

RESULTS

Eight of 14 members of the TSC, and one of six practices involved in the trial participated in the reflection exercise. Recruitment was highlighted as a major challenge encountered by trial researchers, even prior to disruption due to the COVID-19 pandemic. Researchers also commented on themes such as the need to consider incentivisation, and challenges associated with using technology in trials, especially in older age groups.

CONCLUSIONS

Undertaking a formal reflection exercise following the completion of the PULsE-AI trial enabled us to review experiences encountered whilst undertaking a prospective randomised trial in primary care. In sharing our learnings, we hope to support other clinicians undertaking research in primary care to ensure that future trials are of optimal value for furthering knowledge, streamlining pathways, and benefitting patients.

Machine learning derived ECG risk score improves cardiovascular risk assessment in conjunction with coronary artery calcium scoring

Background

Precision estimation of cardiovascular risk remains the cornerstone of atherosclerotic cardiovascular disease (ASCVD) prevention. While coronary artery calcium (CAC) scoring is the best available non-invasive quantitative modality to evaluate risk of ASCVD, it excludes risk related to prior myocardial infarction, cardiomyopathy, and arrhythmia which are implicated in ASCVD. The high-dimensional and inter-correlated nature of ECG data makes it a good candidate for analysis using machine learning techniques and may provide additional prognostic information not captured by CAC. In this study, we aimed to develop a quantitative ECG risk score (eRiS) to predict major adverse cardiovascular events (MACE) alone, or when added to CAC. Further, we aimed to construct and validate a novel nomogram incorporating ECG, CAC and clinical factors for ASCVD.

Methods

We analyzed 5,864 patients with at least 1 cardiovascular risk factor who underwent CAC scoring and a standard ECG as part of the CLARIFY study (ClinicalTrials.gov Identifier: NCT04075162). Events were defined as myocardial infarction, coronary revascularization, stroke or death. A total of 649 ECG features, consisting of measurements such as amplitude and interval measurements from all deflections in the ECG waveform (53 per lead and 13 overall) were automatically extracted using a clinical software (GE Muse™ Cardiology Information System, GE Healthcare). The data was split into 4 training (S_tr) and internal validation (S_v) sets [S_tr (1): S_v (1): 50:50; S_tr (2): S_v (2): 60:40; S_tr (3): S_v (3): 70:30; S_tr (4): S_v (4): 80:20], and the results were compared across all the subsets. We used the ECG features derived from S_tr to develop eRiS. A least absolute shrinkage and selection operator-Cox (LASSO-Cox) regularization model was used for data dimension reduction, feature selection, and eRiS construction. A Cox-proportional hazards model was used to assess the benefit of using an eRiS alone (M_ecg), CAC alone (M_cac) and a combination of eRiS and CAC (M_ecg+cac) for MACE prediction. A nomogram (M_nom) was further constructed by integrating eRiS with CAC and demographics (age and sex). The primary endpoint of the study was the assessment of the performance of M_ecg, M_cac, M_ecg+cac and M_nom in predicting CV disease-free survival in ASCVD.

Findings

Over a median follow-up of 14 months, 494 patients had MACE. The feature selection strategy preserved only about 18% of the features that were consistent across the various strata (S_tr). The M_ecg model, comprising of eRiS alone was found to be significantly associated with MACE and had good discrimination of MACE (C-Index: 0.7, p = <2e-16). eRiS could predict time-to MACE (C-Index: 0.6, p = <2e-16 across all S_v). The M_ecg+cac model was associated with MACE (C-index: 0.71). Model comparison showed that M_ecg+cac was superior to M_ecg (p = 1.8e-10) or M_cac (p < 2.2e-16) alone. The M_nom, comprising of eRiS, CAC, age and sex was associated with MACE (C-index 0.71). eRiS had the most significant contribution, followed by CAC score and other clinical variables. Further, M_nom was able to identify unique patient risk-groups based on eRiS, CAC and clinical variables.

Conclusion

The use of ECG features in conjunction with CAC may allow for improved prognostication and identification of populations at risk. Future directions will involve prospective validation of the risk score and the nomogram across diverse populations with a heterogeneity of treatment effects.

Performance of an electronic health record-based predictive model to identify patients with atrial fibrillation across countries

Background

Atrial fibrillation (AF) burden on patients and healthcare systems warrants innovative strategies for screening asymptomatic individuals.

Objective

We sought to externally validate a predictive model originally developed in a German population to detect unidentified incident AF utilising real-world primary healthcare databases from countries in Europe and Australia.

Methods

This retrospective cohort study used anonymized, longitudinal patient data from 5 country-level primary care databases, including Australia, Belgium, France, Germany, and the UK. The study eligibility included adult patients (≥45 years) with either an AF diagnosis (cases) or no diagnosis (controls) who had continuous enrolment in the respective database prior to the study period. Logistic regression was fitted to a binary response (yes/no) for AF diagnosis using pre-determined risk factors.

Results

AF patients were from Germany (n = 63,562), the UK (n = 42,652), France (n = 7,213), Australia (n = 2,753), and Belgium (n = 1,371). Cases were more likely to have hypertension or other cardiac conditions than controls in all validation datasets compared to the model development data. The area under the receiver operating characteristic (ROC) curve in the validation datasets ranged from 0.79 (Belgium) to 0.84 (Germany), comparable to the German study model, which had an area under the curve of 0.83. Most validation sets reported similar specificity at approximately 80% sensitivity, ranging from 67% (France) to 71% (United Kingdom). The positive predictive value (PPV) ranged from 2% (Belgium) to 16% (Germany), and the number needed to be screened was 50 in Belgium and 6 in Germany. The prevalence of AF varied widely between these datasets, which may be related to different coding practices. Low prevalence affected PPV, but not sensitivity, specificity, and ROC curves.

Conclusions

AF risk prediction algorithms offer targeted ways to identify patients using electronic health records, which could improve screening number and the cost-effectiveness of AF screening if implemented in clinical practice.

Identification of undiagnosed atrial fibrillation using a machine learning risk-prediction algorithm and diagnostic testing (PULsE-AI) in primary care: a multi-centre randomized controlled trial in England

Aims

The aim of the PULsE-AI trial was to assess the effectiveness of a machine learning risk-prediction algorithm in conjunction with diagnostic testing for identifying undiagnosed atrial fibrillation (AF) in primary care in England.

Methods and results

Eligible participants (aged ≥30 years without AF diagnosis; n = 23 745) from six general practices in England were randomized into intervention and control arms. Intervention arm participants, identified by the algorithm as high risk of undiagnosed AF (n = 944), were invited for diagnostic testing (n = 256 consented); those who did not accept the invitation, and all control arm participants, were managed routinely. The primary endpoint was the proportion of AF, atrial flutter, and fast atrial tachycardia diagnoses during the trial (June 2019-February 2021) in high-risk participants. Atrial fibrillation and related arrhythmias were diagnosed in 5.63% and 4.93% of high-risk participants in intervention and control arms, respectively {odds ratio (OR) [95% confidence interval (CI)]: 1.15 (0.77-1.73), P = 0.486}. Among intervention arm participants who underwent diagnostic testing (28.1%), 9.41% received AF and related arrhythmia diagnoses [vs. 4.93% (control); OR (95% CI): 2.24 (1.31-3.73), P = 0.003].

Conclusion

The AF risk-prediction algorithm accurately identified high-risk participants in both arms. While the proportions of AF and related arrhythmia diagnoses were not significantly different between high-risk arms, intervention arm participants who underwent diagnostic testing were twice as likely to receive arrhythmia diagnoses compared with routine care. The algorithm could be a valuable tool to select primary care groups at high risk of undiagnosed AF who may benefit from diagnostic testing.

Development and validation pathways of artificial intelligence tools evaluated in randomised clinical trials

Objective

Given the complexities of testing the translational capability of new artificial intelligence (AI) tools, we aimed to map the pathways of training/validation/testing in development process and external validation of AI tools evaluated in dedicated randomised controlled trials (AI-RCTs).

Methods

We searched for peer-reviewed protocols and completed AI-RCTs evaluating the clinical effectiveness of AI tools and identified development and validation studies of AI tools. We collected detailed information, and evaluated patterns of development and external validation of AI tools.

Results

We found 23 AI-RCTs evaluating the clinical impact of 18 unique AI tools (2009–2021). Standard-of-care interventions were used in the control arms in all but one AI-RCT. Investigators did not provide access to the software code of the AI tool in any of the studies. Considering the primary outcome, the results were in favour of the AI intervention in 82% of the completed AI-RCTs (14 out of 17). We identified significant variation in the patterns of development, external validation and clinical evaluation approaches among different AI tools. A published development study was found only for 10 of the 18 AI tools. Median time from the publication of a development study to the respective AI-RCT was 1.4 years (IQR 0.2–2.2).

Conclusions

We found significant variation in the patterns of development and validation for AI tools before their evaluation in dedicated AI-RCTs. Published peer-reviewed protocols and completed AI-RCTs were also heterogeneous in design and reporting. Upcoming guidelines providing guidance for the development and clinical translation process aim to improve these aspects.

Identification of undiagnosed atrial fibrillation using a machine learning risk prediction algorithm and diagnostic testing (PULsE-AI) in primary care: cost-effectiveness of a screening strategy evaluated in a randomized controlled trial in England

OBJECTIVE

The PULsE-AI trial sought to determine the effectiveness of a screening strategy that included a machine learning risk prediction algorithm in conjunction with diagnostic testing for identification of undiagnosed atrial fibrillation (AF) in primary care. This study aimed to evaluate the cost-effectiveness of implementing the screening strategy in a real-world setting.

METHODS

Data from the PULsE-AI trial - a prospective, randomized, controlled trial conducted across six general practices in England from June 2019 to February 2021 - were used to inform a cost-effectiveness analysis that included a hybrid screening decision tree and Markov AF disease progression model. Model outcomes were reported at both individual- and population-level (estimated UK population ≥30 years of age at high-risk of undiagnosed AF) and included number of patients screened, number of AF cases identified, mean total and incremental costs (screening, events, treatment), quality-adjusted-life-years (QALYs), and incremental cost-effectiveness ratio (ICER).

RESULTS

The screening strategy was estimated to result in 45,493 new diagnoses of AF across the high-risk population in the UK (3.3 million), and an estimated additional 14,004 lifetime diagnoses compared with routine care only. Per-patient costs for high-risk individuals who underwent the screening strategy were estimated at £1,985 (vs £1,888 for individuals receiving routine care only). At a population-level, the screening strategy was associated with a cost increase of approximately £322 million and an increase of 81,000 QALYs. The screening strategy demonstrated cost-effectiveness versus routine care only at an accepted ICER threshold of £20,000 per QALY-gained, with an ICER of £3,994/QALY.

CONCLUSIONS

Compared with routine care only, it is cost-effective to target individuals at high risk of undiagnosed AF, through an AF risk prediction algorithm, who should then undergo diagnostic testing. This AF risk prediction algorithm can reduce the number of patients needed to be screened to identify undiagnosed AF, thus alleviating primary care burden.

Usefulness of Mobile Electrocardiographic Devices to Reduce Urgent Healthcare Visits

Mobile electrocardiogram (mECG) devices are being used increasingly, supplying recordings to providers and providing automatic rhythm interpretation. Given the intermittent nature of certain cardiac arrhythmias, mECGs allow instant access to a recording device. In the current COVID-19 pandemic, efforts to limit in-person patient interactions and avoid overwhelming emergency and inpatient services would add value. Our goal was to evaluate whether a mECG device would reduce healthcare utilization overall, particularly those of urgent nature. We identified a cohort of KardiaMobile (AliveCor, USA) mECG users and compared their healthcare utilization 1 year prior to obtaining the device and 1 year after. One hundred and twenty-eight patients were studied (mean age 64, 47% female). Mean duration of follow-up pre-intervention was 9.8 months. One hundred and twenty-three of 128 individuals completed post-intervention follow-up. Patients were less likely to have cardiac monitors ordered (30 vs 6; p <0.01), outpatient office visits (525 vs 382; p <0.01), cardiac-specific ED visits (51 vs 30; p <0.01), arrhythmia related ED visits (45 vs 20; p <0.01), and unplanned arrhythmia admissions (34 vs 11; p <0.01) in the year after obtaining a KardiaMobile device compared to the year prior to obtaining the device. Mobile technology is available for heart rhythm monitoring and can give feedback to the user. This study showed a reduction of in-person, healthcare utilization with mECG device use. In conclusion, this strategy would be expected to decrease the risk of exposure to patients and providers and would avoid overwhelming emergency and inpatient services.

Digital health solutions in the screening of subclinical atrial fibrillation

Atrial fibrillation (AF) will become one of the biggest challenges in cardiovascular medicine in the near future. Attempting an improvement in future patient care calls explicitly for the screening of subclinical AF. Digital health solutions implementing communication technologies for the collection and analysis of digitally assessable data will most likely serve this need. Several new rapidly developing methods were introduced in the past decade. Although the vast majority still require scientific validation, the body of evidence is growing and several randomized controlled trials are planned. This review aims to give an overview of current technologies with a specific focus on mobile health (mHealth) and appraise their value with regard to the available scientific data.