Pragmatic considerations for fostering reproducible research in artificial intelligence

Diagnostic and Prognostic Electrocardiogram-Based Models for Rapid Clinical Applications

Leveraging artificial intelligence (AI) for the analysis of electrocardiograms (ECGs) has the potential to transform diagnosis and estimate the prognosis of not only cardiac but, increasingly, noncardiac conditions. In this review, we summarize clinical studies and AI-enhanced ECG-based clinical applications in the early detection, diagnosis, and estimating prognosis of cardiovascular diseases in the past 5 years (2019-2023). With advancements in deep learning and the rapid increased use of ECG technologies, a large number of clinical studies have been published. However, most of these studies are single-centre, retrospective, proof-of-concept studies that lack external validation. Prospective studies that progress from development toward deployment in clinical settings account for < 15% of the studies. Successful implementations of ECG-based AI applications that have received approval from the Food and Drug Administration have been developed through commercial collaborations, with approximately half of them being for mobile or wearable devices. The field is in its early stages, and overcoming several obstacles is essential, such as prospective validation in multicentre large data sets, addressing technical issues, bias, privacy, data security, model generalizability, and global scalability. This review concludes with a discussion of these challenges and potential solutions. By providing a holistic view of the state of AI in ECG analysis, this review aims to set a foundation for future research directions, emphasizing the need for comprehensive, clinically integrated, and globally deployable AI solutions in cardiovascular disease management.

Data-centric challenges with the application and adoption of artificial intelligence for drug discovery

INTRODUCTION

Artificial intelligence (AI) is exhibiting tremendous potential to reduce the massive costs and long timescales of drug discovery. There are however important challenges currently limiting the impact and scope of AI models.

AREAS COVERED

In this perspective, the authors discuss a range of data issues (bias, inconsistency, skewness, irrelevance, small size, high dimensionality), how they challenge AI models, and which issue-specific mitigations have been effective. Next, they point out the challenges faced by uncertainty quantification techniques aimed at enhancing and trusting the predictions from these AI models. They also discuss how conceptual errors, unrealistic benchmarks and performance misestimation can confound the evaluation of models and thus their development. Lastly, the authors explain how human bias, whether from AI experts or drug discovery experts, constitutes another challenge that can be alleviated by gaining more prospective experience.

EXPERT OPINION

AI models are often developed to excel on retrospective benchmarks unlikely to anticipate their prospective performance. As a result, only a few of these models are ever reported to have prospective value (e.g. by discovering potent and innovative drug leads for a therapeutic target). The authors have discussed what can go wrong in practice with AI for drug discovery. The authors hope that this will help inform the decisions of editors, funders investors, and researchers working in this area.

Challenges and Potential of Artificial Intelligence in Neuroradiology

PURPOSE

Artificial intelligence (AI) has emerged as a transformative force in medical research and is garnering increased attention in the public consciousness. This represents a critical time period in which medical researchers, healthcare providers, insurers, regulatory agencies, and patients are all developing and shaping their beliefs and policies regarding the use of AI in the healthcare sector. The successful deployment of AI will require support from all these groups. This commentary proposes that widespread support for medical AI must be driven by clear and transparent scientific reporting, beginning at the earliest stages of scientific research.

METHODS

A review of relevant guidelines and literature describing how scientific reporting plays a central role at key stages in the life cycle of an AI software product was conducted. To contextualize this principle within a specific medical domain, we discuss the current state of predictive tissue outcome modeling in acute ischemic stroke and the unique challenges presented therein.

RESULTS AND CONCLUSION

Translating AI methods from the research to the clinical domain is complicated by challenges related to model design and validation studies, medical product regulations, and healthcare providers' reservations regarding AI's efficacy and affordability. However, each of these limitations is also an opportunity for high-impact research that will help to accelerate the clinical adoption of state-of-the-art medical AI. In all cases, establishing and adhering to appropriate reporting standards is an important responsibility that is shared by all of the parties involved in the life cycle of a prospective AI software product.

Transparency in Artificial Intelligence Research: a Systematic Review of Availability Items Related to Open Science in Radiology and Nuclear Medicine

RATIONALE AND OBJECTIVES

Reproducibility of artificial intelligence (AI) research has become a growing concern. One of the fundamental reasons is the lack of transparency in data, code, and model. In this work, we aimed to systematically review the radiology and nuclear medicine papers on AI in terms of transparency and open science.

MATERIALS AND METHODS

A systematic literature search was performed in PubMed to identify original research studies on AI. The search was restricted to studies published in Q1 and Q2 journals that are also indexed on the Web of Science. A random sampling of the literature was performed. Besides six baseline study characteristics, a total of five availability items were evaluated. Two groups of independent readers including eight readers participated in the study. Inter-rater agreement was analyzed. Disagreements were resolved with consensus.

RESULTS

Following eligibility criteria, we included a final set of 194 papers. The raw data was available in about one-fifth of the papers (34/194; 18%). However, the authors made their private data available only in one paper (1/161; 1%). About one-tenth of the papers made their pre-modeling (25/194; 13%), modeling (28/194; 14%), or post-modeling files (15/194; 8%) available. Most of the papers (189/194; 97%) did not attempt to create a ready-to-use system for real-world usage. Data origin, use of deep learning, and external validation had statistically significantly different distributions. The use of private data alone was negatively associated with the availability of at least one item (p<0.001).

CONCLUSION

Overall rates of availability for items were poor, leaving room for substantial improvement.

Organ-on-a-chip meets artificial intelligence in drug evaluation

Drug evaluation has always been an important area of research in the pharmaceutical industry. However, animal welfare protection and other shortcomings of traditional drug development models pose obstacles and challenges to drug evaluation. Organ-on-a-chip (OoC) technology, which simulates human organs on a chip of the physiological environment and functionality, and with high fidelity reproduction organ-level of physiology or pathophysiology, exhibits great promise for innovating the drug development pipeline. Meanwhile, the advancement in artificial intelligence (AI) provides more improvements for the design and data processing of OoCs. Here, we review the current progress that has been made to generate OoC platforms, and how human single and multi-OoCs have been used in applications, including drug testing, disease modeling, and personalized medicine. Moreover, we discuss issues facing the field, such as large data processing and reproducibility, and point to the integration of OoCs and AI in data analysis and automation, which is of great benefit in future drug evaluation. Finally, we look forward to the opportunities and challenges faced by the coupling of OoCs and AI. In summary, advancements in OoCs development, and future combinations with AI, will eventually break the current state of drug evaluation.

Generalizability of an acute kidney injury prediction model across health systems

Delays in the identification of acute kidney injury (AKI) in hospitalized patients are a major barrier to the development of effective interventions to treat AKI. A recent study by Tomasev and colleagues at DeepMind described a model that achieved a state-of-the-art performance in predicting AKI up to 48 hours in advance.1 Because this model was trained in a population of US Veterans that was 94% male, questions have arisen about its reproducibility and generalizability. In this study, we aimed to reproduce key aspects of this model, trained and evaluated it in a similar population of US Veterans, and evaluated its generalizability in a large academic hospital setting. We found that the model performed worse in predicting AKI in females in both populations, with miscalibration in lower stages of AKI and worse discrimination (a lower area under the curve) in higher stages of AKI. We demonstrate that while this discrepancy in performance can be largely corrected in non-Veterans by updating the original model using data from a sex-balanced academic hospital cohort, the worse model performance persists in Veterans. Our study sheds light on the importance of reproducing artificial intelligence studies, and on the complexity of discrepancies in model performance in subgroups that cannot be explained simply on the basis of sample size.

Real-world data: a brief review of the methods, applications, challenges and opportunities

Background

The increased adoption of the internet, social media, wearable devices, e-health services, and other technology-driven services in medicine and healthcare has led to the rapid generation of various types of digital data, providing a valuable data source beyond the confines of traditional clinical trials, epidemiological studies, and lab-based experiments.

Methods

We provide a brief overview on the type and sources of real-world data and the common models and approaches to utilize and analyze real-world data. We discuss the challenges and opportunities of using real-world data for evidence-based decision making This review does not aim to be comprehensive or cover all aspects of the intriguing topic on RWD (from both the research and practical perspectives) but serves as a primer and provides useful sources for readers who interested in this topic.

Results and Conclusions

Real-world hold great potential for generating real-world evidence for designing and conducting confirmatory trials and answering questions that may not be addressed otherwise. The voluminosity and complexity of real-world data also call for development of more appropriate, sophisticated, and innovative data processing and analysis techniques while maintaining scientific rigor in research findings, and attentions to data ethics to harness the power of real-world data.

Clinical significance, challenges and limitations in using artificial intelligence for electrocardiography-based diagnosis

Cardiovascular diseases are one of the leading global causes of mortality. Currently, clinicians rely on their own analyses or automated analyses of the electrocardiogram (ECG) to obtain a diagnosis. However, both approaches can only include a finite number of predictors and are unable to execute complex analyses. Artificial intelligence (AI) has enabled the introduction of machine and deep learning algorithms to compensate for the existing limitations of current ECG analysis methods, with promising results. However, it should be prudent to recognize that these algorithms also associated with their own unique set of challenges and limitations, such as professional liability, systematic bias, surveillance, cybersecurity, as well as technical and logistical challenges. This review aims to increase familiarity with and awareness of AI algorithms used in ECG diagnosis, and to ultimately inform the interested stakeholders on their potential utility in addressing present clinical challenges.

Machine learning based reanalysis of clinical scores for distinguishing between ischemic and hemorrhagic stroke in low resource setting

BACKGROUND

Identifying ischemic or hemorrhagic strokes clinically may help in situations where neuroimaging is unavailable to provide primary-care prior to referring to stroke-ready facility. Stroke classification-based solely on clinical scores faces two unresolved issues. One pertains to overestimation of score performance, while other is biased performance due to class-imbalance inherent in stroke datasets. After correcting the issues using Machine Learning theory, we quantitatively compared existing scores to study the capabilities of clinical attributes for stroke classification.

METHODS

We systematically searched PubMed, ERIC, ScienceDirect, and IEEE-Xplore from 2001 to 2021 for studies that validated the Siriraj, Guys Hospital/Allen, Greek, and Besson scores for stroke classification. From included studies we extracted the reported cross-tabulation to identify and correct the above listed issues for an accurate comparative analysis of the performance of clinical scores.

RESULTS

A total of 21 studies were included. Comparative analysis demonstrates Siriraj Score outperforms others. For Siriraj Score the reported sensitivity range (Ischemic Stroke-diagnosis) 43-97% (Median = 78% [IQR 65-88%]) is significantly higher than our calculated range 40-90% (Median = 70% [IQR 57-73%]), also the reported sensitivity range (Hemorrhagic Stroke-diagnosis) 50-95% (Median = 71% [IQR 64-82%]) is higher than our calculated range 34-86% (Median = 59% [IQR 50-79%]) which indicates overestimation of performance by the included studies. Guys Hospital/Allen and Greek Scores show similar trends. Recommended weighted-accuracy metric provides better estimate of the performance.

CONCLUSION

We demonstrate that clinical attributes have a potential for stroke classification, however the performance of all scores varies across demographics, indicating the need to fine-tune scores for different demographics. To improve this variability, we suggest creating global data pool with statistically significant attributes. Machine Learning classifiers trained over such dataset may perform better and generalise at scale.

Artificial Intelligence for Education, Proctoring, and Credentialing in Cardiovascular Medicine

Artificial intelligence and machine learning are rapidly gaining popularity in every aspect of cardiovascular medicine. This review discusses the past, present, and future of artificial intelligence in education, remote proctoring, credentialing, research, and publication as they pertain to cardiovascular procedures. This review describes the benefits and limitations of artificial intelligence and machine learning and the exciting potential of integrating advanced simulation, holography, virtual reality, and extended reality into disease diagnosis and patient care, as well as their roles in cardiovascular research and education. Nonetheless, much of the available data resides in electronic medical records or within industry-sponsored proprietary programs that are not compatible or standardized for current clinical application. Many areas in cardiovascular medicine would benefit from the introduction or increased use of artificial intelligence. Web-based artificial intelligence applications could be used to address unmet needs for education, on-demand procedural proctoring, credentialing, and recredentialing for interventionists and physicians in remote locations. Further progress in artificial intelligence will require further collaboration among computer scientists and researchers in order to identify and correct the most relevant problems and to implement the best data-based approach to achieving this goal. The future success of artificial intelligence in cardiovascular medicine will depend on the degree of collaboration between all pertinent experts in this field. This will undoubtedly be a prolonged, stepwise process.

A Platform and Multisided Market for Translational, Software-Defined Medical Procedures in the Operating Room (OP 4.1): Proof-of-Concept Study

Background

Although digital and data-based technologies are widespread in various industries in the context of Industry 4.0, the use of smart connected devices in health care is still in its infancy. Innovative solutions for the medical environment are affected by difficult access to medical device data and high barriers to market entry because of proprietary systems.

Objective

In the proof-of-concept project OP 4.1, we show the business viability of connecting and augmenting medical devices and data through software add-ons by giving companies a technical and commercial platform for the development, implementation, distribution, and billing of innovative software solutions.

Methods

The creation of a central platform prototype requires the collaboration of several independent market contenders, including medical users, software developers, medical device manufacturers, and platform providers. A dedicated consortium of clinical and scientific partners as well as industry partners was set up.

Results

We demonstrate the successful development of the prototype of a user-centric, open, and extensible platform for the intelligent support of processes starting with the operating room. By connecting heterogeneous data sources and medical devices from different manufacturers and making them accessible for software developers and medical users, the cloud-based platform OP 4.1 enables the augmentation of medical devices and procedures through software-based solutions. The platform also allows for the demand-oriented billing of apps and medical devices, thus permitting software-based solutions to fast-track their economic development and become commercially successful.

Conclusions

The technology and business platform OP 4.1 creates a multisided market for the successful development, implementation, distribution, and billing of new software solutions in the operating room and in the health care sector in general. Consequently, software-based medical innovation can be translated into clinical routine quickly, efficiently, and cost-effectively, optimizing the treatment of patients through smartly assisted procedures.

Browser-based Data Annotation, Active Learning, and Real-Time Distribution of Artificial Intelligence Models: From Tumor Tissue Microarrays to COVID-19 Radiology

BACKGROUND

Artificial intelligence (AI) is fast becoming the tool of choice for scalable and reliable analysis of medical images. However, constraints in sharing medical data outside the institutional or geographical space, as well as difficulties in getting AI models and modeling platforms to work across different environments, have led to a "reproducibility crisis" in digital medicine.

METHODS

This study details the implementation of a web platform that can be used to mitigate these challenges by orchestrating a digital pathology AI pipeline, from raw data to model inference, entirely on the local machine. We discuss how this federated platform provides governed access to data by consuming the Application Program Interfaces exposed by cloud storage services, allows the addition of user-defined annotations, facilitates active learning for training models iteratively, and provides model inference computed directly in the web browser at practically zero cost. The latter is of particular relevance to clinical workflows because the code, including the AI model, travels to the user's data, which stays private to the governance domain where it was acquired.

RESULTS

We demonstrate that the web browser can be a means of democratizing AI and advancing data socialization in medical imaging backed by consumer-facing cloud infrastructure such as Box.com. As a case study, we test the accompanying platform end-to-end on a large dataset of digital breast cancer tissue microarray core images. We also showcase how it can be applied in contexts separate from digital pathology by applying it to a radiology dataset containing COVID-19 computed tomography images.

CONCLUSIONS

The platform described in this report resolves the challenges to the findable, accessible, interoperable, reusable stewardship of data and AI models by integrating with cloud storage to maintain user-centric governance over the data. It also enables distributed, federated computation for AI inference over those data and proves the viability of client-side AI in medical imaging.

AVAILABILITY

The open-source application is publicly available at , with a short video demonstration at .

Application of Artificial Intelligence Methods to Pharmacy Data for Cancer Surveillance and Epidemiology Research: A Systematic Review

PURPOSE

The implementation and utilization of electronic health records is generating a large volume and variety of data, which are difficult to process using traditional techniques. However, these data could help answer important questions in cancer surveillance and epidemiology research. Artificial intelligence (AI) data processing methods are capable of evaluating large volumes of data, yet current literature on their use in this context of pharmacy informatics is not well characterized.

METHODS

A systematic literature review was conducted to evaluate relevant publications within four domains (cancer, pharmacy, AI methods, population science) across PubMed, EMBASE, Scopus, and the Cochrane Library and included all publications indexed between July 17, 2008, and December 31, 2018. The search returned 3,271 publications, which were evaluated for inclusion.

RESULTS

There were 36 studies that met criteria for full-text abstraction. Of those, only 45% specifically identified the pharmacy data source, and 55% specified drug agents or drug classes. Multiple AI methods were used; 25% used machine learning (ML), 67% used natural language processing (NLP), and 8% combined ML and NLP.

CONCLUSION

This review demonstrates that the application of AI data methods for pharmacy informatics and cancer epidemiology research is expanding. However, the data sources and representations are often missing, challenging study replicability. In addition, there is no consistent format for reporting results, and one of the preferred metrics, F-score, is often missing. There is a resultant need for greater transparency of original data sources and performance of AI methods with pharmacy data to improve the translation of these results into meaningful outcomes.

Quantitative Molecular Positron Emission Tomography Imaging Using Advanced Deep Learning Techniques

The widespread availability of high-performance computing and the popularity of artificial intelligence (AI) with machine learning and deep learning (ML/DL) algorithms at the helm have stimulated the development of many applications involving the use of AI-based techniques in molecular imaging research. Applications reported in the literature encompass various areas, including innovative design concepts in positron emission tomography (PET) instrumentation, quantitative image reconstruction and analysis techniques, computer-aided detection and diagnosis, as well as modeling and prediction of outcomes. This review reflects the tremendous interest in quantitative molecular imaging using ML/DL techniques during the past decade, ranging from the basic principles of ML/DL techniques to the various steps required for obtaining quantitatively accurate PET data, including algorithms used to denoise or correct for physical degrading factors as well as to quantify tracer uptake and metabolic tumor volume for treatment monitoring or radiation therapy treatment planning and response prediction.This review also addresses future opportunities and current challenges facing the adoption of ML/DL approaches and their role in multimodality imaging.

Machine Learning Assessment of Left Ventricular Diastolic Function Based on Electrocardiographic Features

BACKGROUND

Left ventricular (LV) diastolic dysfunction is recognized as playing a major role in the pathophysiology of heart failure; however, clinical tools for identifying diastolic dysfunction before echocardiography remain imprecise.

OBJECTIVES

This study sought to develop machine-learning models that quantitatively estimate myocardial relaxation using clinical and electrocardiography (ECG) variables as a first step in the detection of LV diastolic dysfunction.

METHODS

A multicenter prospective study was conducted at 4 institutions in North America enrolling a total of 1,202 subjects. Patients from 3 institutions (n = 814) formed an internal cohort and were randomly divided into training and internal test sets (80:20). Machine-learning models were developed using signal-processed ECG, traditional ECG, and clinical features and were tested using the test set. Data from the fourth institution was reserved as an external test set (n = 388) to evaluate the model generalizability.

RESULTS

Despite diversity in subjects, the machine-learning model predicted the quantitative values of the LV relaxation velocities (e') measured by echocardiography in both internal and external test sets (mean absolute error: 1.46 and 1.93 cm/s; adjusted R² = 0.57 and 0.46, respectively). Analysis of the area under the receiver operating characteristic curve (AUC) revealed that the estimated e' discriminated the guideline-recommended thresholds for abnormal myocardial relaxation and diastolic and systolic dysfunction (LV ejection fraction) the internal (area under the curve [AUC]: 0.83, 0.76, and 0.75) and external test sets (0.84, 0.80, and 0.81), respectively. Moreover, the estimated e' allowed prediction of LV diastolic dysfunction based on multiple age- and sex-adjusted reference limits (AUC: 0.88 and 0.94 in the internal and external sets, respectively).

CONCLUSIONS

A quantitative prediction of myocardial relaxation can be performed using easily obtained clinical and ECG features. This cost-effective strategy may be a valuable first clinical step for assessing the presence of LV dysfunction and may potentially aid in the early diagnosis and management of heart failure patients.

Big Data and Artificial Intelligence: Opportunities and Threats in Electrophysiology

The combination of big data and artificial intelligence (AI) is having an increasing impact on the field of electrophysiology. Algorithms are created to improve the automated diagnosis of clinical ECGs or ambulatory rhythm devices. Furthermore, the use of AI during invasive electrophysiological studies or combining several diagnostic modalities into AI algorithms to aid diagnostics are being investigated. However, the clinical performance and applicability of created algorithms are yet unknown. In this narrative review, opportunities and threats of AI in the field of electrophysiology are described, mainly focusing on ECGs. Current opportunities are discussed with their potential clinical benefits as well as the challenges. Challenges in data acquisition, model performance, (external) validity, clinical implementation, algorithm interpretation as well as the ethical aspects of AI research are discussed. This article aims to guide clinicians in the evaluation of new AI applications for electrophysiology before their clinical implementation.

Artificial Intelligence in Cardiology: Present and Future

Artificial intelligence (AI) is a nontechnical, popular term that refers to machine learning of various types but most often to deep neural networks. Cardiology is at the forefront of AI in medicine. For this review, we searched PubMed and MEDLINE databases with no date restriction using search terms related to AI and cardiology. Articles were selected for inclusion on the basis of relevance. We highlight the major achievements in recent years in nearly all areas of cardiology and underscore the mounting evidence suggesting how AI will take center stage in the field. Artificial intelligence requires a close collaboration among computer scientists, clinical investigators, clinicians, and other users in order to identify the most relevant problems to be solved. Best practices in the generation and implementation of AI include the selection of ideal data sources, taking into account common challenges during the interpretation, validation, and generalizability of findings, and addressing safety and ethical concerns before final implementation. The future of AI in cardiology and in medicine in general is bright as the collaboration between investigators and clinicians continues to excel.

Ethical Use of Electronic Health Record Data and Artificial Intelligence: Recommendations of the Primary Care Informatics Working Group of the International Medical Informatics Association

Objective: To create practical recommendations for the curation of routinely collected health data and artificial intelligence (AI) in primary care with a focus on ensuring their ethical use.

Methods: We defined data curation as the process of management of data throughout its lifecycle to ensure it can be used into the future. We used a literature review and Delphi exercises to capture insights from the Primary Care Informatics Working Group (PCIWG) of the International Medical Informatics Association (IMIA).

Results: We created six recommendations: (1) Ensure consent and formal process to govern access and sharing throughout the data life cycle; (2) Sustainable data creation/collection requires trust and permission; (3) Pay attention to Extract-Transform-Load (ETL) processes as they may have unrecognised risks; (4) Integrate data governance and data quality management to support clinical practice in integrated care systems; (5) Recognise the need for new processes to address the ethical issues arising from AI in primary care; (6) Apply an ethical framework mapped to the data life cycle, including an assessment of data quality to achieve effective data curation.

Conclusions: The ethical use of data needs to be integrated within the curation process, hence running throughout the data lifecycle. Current information systems may not fully detect the risks associated with ETL and AI; they need careful scrutiny. With distributed integrated care systems where data are often used remote from documentation, harmonised data quality assessment, management, and governance is important. These recommendations should help maintain trust and connectedness in contemporary information systems and planned developments.