The role of patient-reported outcome measures in trials of artificial intelligence health technologies: a systematic evaluation of ClinicalTrials.gov records (1997-2022)

Trial Factors Associated With Completion of Clinical Trials Evaluating AI: Retrospective Case-Control Study

BACKGROUND

Evaluation of artificial intelligence (AI) tools in clinical trials remains the gold standard for translation into clinical settings. However, design factors associated with successful trial completion and the common reasons for trial failure are unknown.

OBJECTIVE

This study aims to compare trial design factors of complete and incomplete clinical trials testing AI tools. We conducted a case-control study of complete (n=485) and incomplete (n=51) clinical trials that evaluated AI as an intervention of ClinicalTrials.gov.

METHODS

Trial design factors, including area of clinical application, intended use population, and intended role of AI, were extracted. Trials that did not evaluate AI as an intervention and active trials were excluded. The assessed trial design factors related to AI interventions included the domain of clinical application related to organ systems; intended use population for patients or health care providers; and the role of AI for different applications in patient-facing clinical workflows, such as diagnosis, screening, and treatment. In addition, we also assessed general trial design factors including study type, allocation, intervention model, masking, age, sex, funder, continent, length of time, sample size, number of enrollment sites, and study start year. The main outcome was the completion of the clinical trial. Odds ratio (OR) and 95% CI values were calculated for all trial design factors using propensity-matched, multivariable logistic regression.

RESULTS

We queried ClinicalTrials.gov on December 23, 2023, using AI keywords to identify complete and incomplete trials testing AI technologies as a primary intervention, yielding 485 complete and 51 incomplete trials for inclusion in this study. Our nested propensity-matched, case-control results suggest that trials conducted in Europe were significantly associated with trial completion when compared with North American trials (OR 2.85, 95% CI 1.14-7.10; P=.03), and the trial sample size was positively associated with trial completion (OR 1.00, 95% CI 1.00-1.00; P=.02).

CONCLUSIONS

Our case-control study is one of the first to identify trial design factors associated with completion of AI trials and catalog study-reported reasons for AI trial failure. We observed that trial design factors positively associated with trial completion include trials conducted in Europe and sample size. Given the promising clinical use of AI tools in health care, our results suggest that future translational research should prioritize addressing the design factors of AI clinical trials associated with trial incompletion and common reasons for study failure.

Behavioural digital biomarkers enable real-time monitoring of patient-reported outcomes: a substudy of the multicentre, prospective observational SafeHeart study

AIMS

Patient-reported outcome measures (PROMs) serve multiple purposes, including shared decision-making and patient communication, treatment monitoring, and health technology assessment. Patient monitoring using PROMs is constrained by recall and non-response bias, respondent burden, and missing data. We evaluated the potential of behavioural digital biomarkers obtained from a wearable accelerometer to achieve personalized predictions of PROMs.

METHODS AND RESULTS

Data from the multicentre, prospective SafeHeart study conducted at Amsterdam University Medical Center in the Netherlands and Copenhagen University Hospital, Rigshospitalet in Copenhagen, Denmark, were used. The study enrolled patients with an implantable cardioverter defibrillator between May 2021 and September 2022 who then wore wearable devices with raw acceleration output to capture digital biomarkers reflecting physical behaviour. To collect PROMs, patients received the Kansas City Cardiomyopathy Questionnaire (KCCQ) and EuroQoL 5-Dimensions 5-Level (EQ5D-5L) questionnaire at two instances: baseline and after six months. Multivariable Tobit regression models were used to explore associations between digital biomarkers and PROMs, specifically whether digital biomarkers could enable PROM prediction. The study population consisted of 303 patients (mean age 62.9 ± 10.9 years, 81.2% male). Digital biomarkers showed significant correlations to patient-reported physical and social limitations, severity and frequency of symptoms, and quality of life. Prospective validation of the Tobit models indicated moderate correlations between the observed and predicted scores for KCCQ [concordance correlation coefficient (CCC) = 0.49, mean difference: 1.07 points] and EQ5D-5L (CCC = 0.38, mean difference: 0.02 points).

CONCLUSION

Wearable digital biomarkers correlate with PROMs, and may be leveraged for real-time prediction. These findings hold promise for monitoring of PROMs through wearable accelerometers.

Conceptual review of outcome metrics and measures used in clinical evaluation of artificial intelligence in radiology

Artificial intelligence (AI) has numerous applications in radiology. Clinical research studies to evaluate the AI models are also diverse. Consequently, diverse outcome metrics and measures are employed in the clinical evaluation of AI, presenting a challenge for clinical radiologists. This review aims to provide conceptually intuitive explanations of the outcome metrics and measures that are most frequently used in clinical research, specifically tailored for clinicians. While we briefly discuss performance metrics for AI models in binary classification, detection, or segmentation tasks, our primary focus is on less frequently addressed topics in published literature. These include metrics and measures for evaluating multiclass classification; those for evaluating generative AI models, such as models used in image generation or modification and large language models; and outcome measures beyond performance metrics, including patient-centered outcome measures. Our explanations aim to guide clinicians in the appropriate use of these metrics and measures.

H-Wave® Device Stimulation for Chronic Neck Pain: A Patient-Reported Outcome Measures (PROMs) Study

INTRODUCTION

Chronic neck pain (cNP) is one of the leading causes of disability worldwide, often being refractory to conventional forms of treatment. Various forms of electrical stimulation have been proposed to decrease pain and improve function. Patient-reported outcome measures (PROMs) for treatment of cNP have rarely been published.

METHODS

An independent retrospective statistical analysis of PROMs data for users of H-Wave® device stimulation (HWDS), prospectively collected by the device manufacturer over a 4-year period, was conducted. Final surveys for 34,192 pain management patients were filtered for pain chronicity limited to 3-24 months and device use of 22-365 days, resulting in 11,503 patients with "all diagnoses"; this number was further reduced to 1482 patients with cNP, sprain, or strain.

RESULTS

Neck pain was reduced by 3.13 points (0-10 pain scale), with significant (≥ 20%) relief in 86.6%. Function/activities of daily living (ADL) improved in 96.19%, while improved work performance was reported in 84.76%. Medication use decreased or stopped in 65.42% and sleep improved in 60.39%. Over 95% reported having expectations met or exceeded, service satisfaction, and confidence in device use, while no adverse events were reported. Subgroup analyses found positive benefit associations with longer duration of device use.

CONCLUSION

Near-equivalent outcomes were self-reported by cNP HWDS patients as for (previously published) chronic low back pain (cLBP) patients. HWDS provided effective and safe cNP relief, improvements in function and ADL, along with additional benefits including decreased medication use, better sleep, and improved work performance.

An online evidence-based dictionary of common adverse events of antidepressants: a new tool to empower patients and clinicians in their shared decision-making process

BACKGROUND

Adverse events (AEs) are commonly reported in clinical studies using the Medical Dictionary for Regulatory Activities (MedDRA), an international standard for drug safety monitoring. However, the technical language of MedDRA makes it challenging for patients and clinicians to share understanding and therefore to make shared decisions about medical interventions. In this project, people with lived experience of depression and antidepressant treatment worked with clinicians and researchers to co-design an online dictionary of AEs associated with antidepressants, taking into account its ease of use and applicability to real-world settings.

METHODS

Through a pre-defined literature search, we identified MedDRA-coded AEs from randomised controlled trials of antidepressants used in the treatment of depression. In collaboration with the McPin Foundation, four co-design workshops with a lived experience advisory panel (LEAP) and one independent focus group (FG) were conducted to produce user-friendly translations of AE terms. Guiding principles for translation were co-designed with McPin/LEAP members and defined before the finalisation of Clinical Codes (CCs, or non-technical terms to represent specific AE concepts). FG results were thematically analysed using the Framework Method.

RESULTS

Starting from 522 trials identified by the search, 736 MedDRA-coded AE terms were translated into 187 CCs, which balanced key factors identified as important to the LEAP and FG (namely, breadth, specificity, generalisability, patient-understandability and acceptability). Work with the LEAP showed that a user-friendly language of AEs should aim to mitigate stigma, acknowledge the multiple levels of comprehension in 'lay' language and balance the need for semantic accuracy with user-friendliness. Guided by these principles, an online dictionary of AEs was co-designed and made freely available ( https://thesymptomglossary.com ). The digital tool was perceived by the LEAP and FG as a resource which could feasibly improve antidepressant treatment by facilitating the accurate, meaningful expression of preferences about potential harms through a shared decision-making process.

CONCLUSIONS

This dictionary was developed in English around AEs from antidepressants in depression but it can be adapted to different languages and cultural contexts, and can also become a model for other interventions and disorders (i.e., antipsychotics in schizophrenia). Co-designed digital resources may improve the patient experience by helping to deliver personalised information on potential benefits and harms in an evidence-based, preference-sensitive way.

Perspectives of healthcare professionals and people living with HIV in dialogue: on information sharing to improve communication at the consultation

We report on a qualitative Group Survey study involving four healthcare professionals (HCPs) and eight people living with HIV who were recipients of care in the United Kingdom (UK). The survey aimed to bring participants' perspectives into dialogue and establish consensus about how communication between HCPs delivering HIV care and their patients could be improved in the context of the routine care consultation. Responses from both parties were anonymously collated, thematically analysed, and shared back with participants in two subsequent survey rounds to support consensus-building on matters of concern and identify thematic insights. In this paper, we report three themes for informing future designs of tools and services to support communication between patients and HCPs: Patient-clinician relationship for trusted sharing; Self-reporting psychosocial information to support Whole-person care; and Perceived barriers for online trusted sharing with HCPs. Our findings highlight key areas of concern and further investigation is needed to understand how self-reported information may be meaningfully captured, interpreted and processed by HCPs in ways that are trusted by patients who voice privacy and security concerns.

Charting a course for global progress in PIDs by 2030 - proceedings from the IPOPI global multi-stakeholders' summit (September 2023)

The International Patient Organisation for Primary Immunodeficiencies (IPOPI) held its second Global Multi-Stakeholders' Summit, an annual stimulating and forward-thinking meeting uniting experts to anticipate pivotal upcoming challenges and opportunities in the field of primary immunodeficiency (PID). The 2023 summit focused on three key identified discussion points: (i) How can immunoglobulin (Ig) therapy meet future personalized patient needs? (ii) Pandemic preparedness: what's next for public health and potential challenges for the PID community? (iii) Diagnosing PIDs in 2030: what needs to happen to diagnose better and to diagnose more? Clinician-Scientists, patient representatives and other stakeholders explored avenues to improve Ig therapy through mechanistic insights and tailored Ig preparations/products according to patient-specific needs and local exposure to infectious agents, amongst others. Urgency for pandemic preparedness was discussed, as was the threat of shortage of antibiotics and increasing antimicrobial resistance, emphasizing the need for representation of PID patients and other vulnerable populations throughout crisis and care management. Discussion also covered the complexities of PID diagnosis, addressing issues such as global diagnostic disparities, the integration of patient-reported outcome measures, and the potential of artificial intelligence to increase PID diagnosis rates and to enhance diagnostic precision. These proceedings outline the outcomes and recommendations arising from the 2023 IPOPI Global Multi-Stakeholders' Summit, offering valuable insights to inform future strategies in PID management and care. Integral to this initiative is its role in fostering collaborative efforts among stakeholders to prepare for the multiple challenges facing the global PID community.

Randomised controlled trials evaluating artificial intelligence in clinical practice: a scoping review

This scoping review of randomised controlled trials on artificial intelligence (AI) in clinical practice reveals an expanding interest in AI across clinical specialties and locations. The USA and China are leading in the number of trials, with a focus on deep learning systems for medical imaging, particularly in gastroenterology and radiology. A majority of trials (70 [81%] of 86) report positive primary endpoints, primarily related to diagnostic yield or performance; however, the predominance of single-centre trials, little demographic reporting, and varying reports of operational efficiency raise concerns about the generalisability and practicality of these results. Despite the promising outcomes, considering the likelihood of publication bias and the need for more comprehensive research including multicentre trials, diverse outcome measures, and improved reporting standards is crucial. Future AI trials should prioritise patient-relevant outcomes to fully understand AI's true effects and limitations in health care.

Survey on Value Elements Provided by Artificial Intelligence and Their Eligibility for Insurance Coverage With an Emphasis on Patient-Centered Outcomes

OBJECTIVE

This study aims to explore the opinions on the insurance coverage of artificial intelligence (AI), as categorized based on the distinct value elements offered by AI, with a specific focus on patient-centered outcomes (PCOs). PCOs are distinguished from traditional clinical outcomes and focus on patient-reported experiences and values such as quality of life, functionality, well-being, physical or emotional status, and convenience.

MATERIALS AND METHODS

We classified the value elements provided by AI into four dimensions: clinical outcomes, economic aspects, organizational aspects, and non-clinical PCOs. The survey comprised three sections: 1) experiences with PCOs in evaluating AI, 2) opinions on the coverage of AI by the National Health Insurance of the Republic of Korea when AI demonstrated benefits across the four value elements, and 3) respondent characteristics. The opinions regarding AI insurance coverage were assessed dichotomously and semi-quantitatively: non-approval (0) vs. approval (on a 1-10 weight scale, with 10 indicating the strongest approval). The survey was conducted from July 4 to 26, 2023, using a web-based method. Responses to PCOs and other value elements were compared.

RESULTS

Among 200 respondents, 44 (22%) were patients/patient representatives, 64 (32%) were industry/developers, 60 (30%) were medical practitioners/doctors, and 32 (16%) were government health personnel. The level of experience with PCOs regarding AI was low, with only 7% (14/200) having direct experience and 10% (20/200) having any experience (either direct or indirect). The approval rate for insurance coverage for PCOs was 74% (148/200), significantly lower than the corresponding rates for other value elements (82.5%-93.5%; P ≤ 0.034). The approval strength was significantly lower for PCOs, with a mean weight ± standard deviation of 5.1 ± 3.5, compared to other value elements (P ≤ 0.036).

CONCLUSION

There is currently limited demand for insurance coverage for AI that demonstrates benefits in terms of non-clinical PCOs.

Digital health interventions for postoperative recovery in children: a systematic review

BACKGROUND

Digital health interventions offer a promising approach for monitoring during postoperative recovery. However, the effectiveness of these interventions remains poorly understood, particularly in children. The objective of this study was to assess the efficacy of digital health interventions for postoperative recovery in children.

METHODS

A systematic review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, with the use of automation tools for searching and screening. We searched five electronic databases for randomised controlled trials or non-randomised studies of interventions that utilised digital health interventions to monitor postoperative recovery in children. The study quality was assessed using Cochrane Collaboration's Risk of Bias tools. The systematic review protocol was prospectively registered with PROSPERO (CRD42022351492).

RESULTS

The review included 16 studies involving 2728 participants from six countries. Tonsillectomy was the most common surgery and smartphone apps (WeChat) were the most commonly used digital health interventions. Digital health interventions resulted in significant improvements in parental knowledge about the child's condition and satisfaction regarding perioperative instructions (standard mean difference=2.16, 95% confidence interval 1.45-2.87; z=5.98, P<0.001; I2=88%). However, there was no significant effect on children's pain intensity (standard mean difference=0.09, 95% confidence interval -0.95 to 1.12; z=0.16, P=0.87; I2=98%).

CONCLUSIONS

Digital health interventions hold promise for improving parental postoperative knowledge and satisfaction. However, more research is needed for child-centric interventions with validated outcome measures. Future work should focus development and testing of user-friendly digital apps and wearables to ease the healthcare burden and improve outcomes for children.

SYSTEMATIC REVIEW PROTOCOL

PROSPERO (CRD42022351492).

Adapting and Evaluating an AI-Based Chatbot Through Patient and Stakeholder Engagement to Provide Information for Different Health Conditions: Master Protocol for an Adaptive Platform Trial (the MARVIN Chatbots Study)

BACKGROUND

Artificial intelligence (AI)-based chatbots could help address some of the challenges patients face in acquiring information essential to their self-health management, including unreliable sources and overburdened health care professionals. Research to ensure the proper design, implementation, and uptake of chatbots is imperative. Inclusive digital health research and responsible AI integration into health care require active and sustained patient and stakeholder engagement, yet corresponding activities and guidance are limited for this purpose.

OBJECTIVE

In response, this manuscript presents a master protocol for the development, testing, and implementation of a chatbot family in partnership with stakeholders. This protocol aims to help efficiently translate an initial chatbot intervention (MARVIN) to multiple health domains and populations.

METHODS

The MARVIN chatbots study has an adaptive platform trial design consisting of multiple parallel individual chatbot substudies with four common objectives: (1) co-construct a tailored AI chatbot for a specific health care setting, (2) assess its usability with a small sample of participants, (3) measure implementation outcomes (usability, acceptability, appropriateness, adoption, and fidelity) within a large sample, and (4) evaluate the impact of patient and stakeholder partnerships on chatbot development. For objective 1, a needs assessment will be conducted within the setting, involving four 2-hour focus groups with 5 participants each. Then, a co-construction design committee will be formed with patient partners, health care professionals, and researchers who will participate in 6 workshops for chatbot development, testing, and improvement. For objective 2, a total of 30 participants will interact with the prototype for 3 weeks and assess its usability through a survey and 3 focus groups. Positive usability outcomes will lead to the initiation of objective 3, whereby the public will be able to access the chatbot for a 12-month real-world implementation study using web-based questionnaires to measure usability, acceptability, and appropriateness for 150 participants and meta-use data to inform adoption and fidelity. After each objective, for objective 4, focus groups will be conducted with the design committee to better understand their perspectives on the engagement process.

RESULTS

From July 2022 to October 2023, this master protocol led to four substudies conducted at the McGill University Health Centre or the Centre hospitalier de l'Université de Montréal (both in Montreal, Quebec, Canada): (1) MARVIN for HIV (large-scale implementation expected in mid-2024), (2) MARVIN-Pharma for community pharmacists providing HIV care (usability study planned for mid-2024), (3) MARVINA for breast cancer, and (4) MARVIN-CHAMP for pediatric infectious conditions (both in preparation, with development to begin in early 2024).

CONCLUSIONS

This master protocol offers an approach to chatbot development in partnership with patients and health care professionals that includes a comprehensive assessment of implementation outcomes. It also contributes to best practice recommendations for patient and stakeholder engagement in digital health research.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05789901; https://classic.clinicaltrials.gov/ct2/show/NCT05789901.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/54668.

Constructing a finer-grained representation of clinical trial results from ClinicalTrials.gov

Randomized controlled trials are essential for evaluating clinical interventions; however, selective reporting and publication bias in medical journals have undermined the integrity of the clinical evidence system. ClinicalTrials.gov serves as a valuable and complementary repository, yet synthesizing information from it remains challenging. This study introduces a curated dataset that extends beyond the traditional PICO framework. It links efficacy with safety results at the experimental arm group level within each trial, and connects them across all trials through a knowledge graph. This novel representation effectively bridges the gap between generally described searchable information and specifically detailed yet underutilized reported results, and promotes a dual-faceted understanding of interventional effects. Adhering to the "calculate once, use many times" principle, the structured dataset will enhance the reuse and interpretation of ClinicalTrials.gov results data. It aims to facilitate more systematic evidence synthesis and health technology assessment, by incorporating both positive and negative results, distinguishing biomarkers, patient-reported outcomes, and clinical endpoints, while also balancing both efficacy and safety outcomes for a given medical intervention.

DENTAL PATIENT-REPORTED OUTCOMES UPDATE 2023

The emergence and rapid development of disruptive innovations are quickly turning our profession into personalized dentistry, built upon evidence-based, data-oriented, and patient-centered research. In order to help improve the quality and quantity of patient-centered evidence in dentistry, further promote the wide and standard use of dental patient-reported outcomes (dPROs) and dental patient-reported outcome measures (dPROMs), the Journal of Evidence-Based Dental Practice has put together this special issue, the third of a series entitled Dental Patient-Reported Outcomes Update. A total of 7 solicited articles are collected in this issue. To put them into a broader perspective, this review provides a concise summary of key, selected PRO and dPRO articles published during 2023. A brief introduction to those articles included in this Special Issue follows. Four main domains are covered in this Special Issue: (1) dPROs and digital dentistry, (2) standardization of dPRO-related methodology, (3) current usage of dPROs and dPROMs in published research, and (iv) the significance and relevance of dPRO usage.

Revolutionizing Spinal Care: Current Applications and Future Directions of Artificial Intelligence and Machine Learning

Artificial intelligence (AI) and machine learning (ML) are rapidly becoming integral components of modern healthcare, offering new avenues for diagnosis, treatment, and outcome prediction. This review explores their current applications and potential future in the field of spinal care. From enhancing imaging techniques to predicting patient outcomes, AI and ML are revolutionizing the way we approach spinal diseases. AI and ML have significantly improved spinal imaging by augmenting detection and classification capabilities, thereby boosting diagnostic accuracy. Predictive models have also been developed to guide treatment plans and foresee patient outcomes, driving a shift towards more personalized care. Looking towards the future, we envision AI and ML further ingraining themselves in spinal care with the development of algorithms capable of deciphering complex spinal pathologies to aid decision making. Despite the promise these technologies hold, their integration into clinical practice is not without challenges. Data quality, integration hurdles, data security, and ethical considerations are some of the key areas that need to be addressed for their successful and responsible implementation. In conclusion, AI and ML represent potent tools for transforming spinal care. Thoughtful and balanced integration of these technologies, guided by ethical considerations, can lead to significant advancements, ushering in an era of more personalized, effective, and efficient healthcare.

Embedding patient-reported outcomes at the heart of artificial intelligence health-care technologies

Integration of patient-reported outcome measures (PROMs) in artificial intelligence (AI) studies is a critical part of the humanisation of AI for health. It allows AI technologies to incorporate patients' own views of their symptoms and predict outcomes, reflecting a more holistic picture of health and wellbeing and ultimately helping patients and clinicians to make the best health-care decisions together. By positioning patient-reported outcomes (PROs) as a model input or output we propose a framework to embed PROMs within the function and evaluation of AI health care. However, the integration of PROs in AI systems presents several challenges. These challenges include (1) fragmentation of PRO data collection; (2) validation of AI systems trained and validated against clinician performance, rather than outcome data; (3) scarcity of large-scale PRO datasets; (4) inadequate selection of PROMs for the target population and inadequate infrastructure for collecting PROs; and (5) clinicians might not recognise the value of PROs and therefore not prioritise their adoption; and (6) studies involving PRO or AI frequently present suboptimal design. Notwithstanding these challenges, we propose considerations for the inclusion of PROs in AI health-care technologies to avoid promoting survival at the expense of wellbeing.