Evaluation of a Health Information Technology-Enabled Collective Intelligence Platform to Improve Diagnosis in Primary Care and Urgent Care Settings: Protocol for a Pragmatic Randomized Controlled Trial

Health information systems with technology acceptance model approach: A systematic review

In the rapidly evolving landscape of information technologies, individuals and organizations must adapt to the digital age. Given the diversity in users' knowledge and experience with technology, their acceptance levels also vary. Over the past 30 years, various theoretical models have been introduced to provide a framework for understanding user acceptance of technology. Among these, the Technology Acceptance Model (TAM) stands out as a key theoretical framework, offering insights into why new technologies are either accepted or rejected. Analyzing user acceptance of technology has thus become a critical area of study. Healthcare organizations aim to assess the perceived efficacy and user-friendliness of a given technology. This will help health organisations design and implement HIS that meet users' needs and preferences. In this context, how does the TAM clarify the acceptance and use of Health Information Systems (HIS)? To address this inquiry, a comprehensive literature review will be carried out. The systematic review involved 29 studies issued between 2018 and 2023 and searched the databases Pubmed, Scopus, Wos and Ulakbim TR Index. The PRISMA flowchart was used to identify the included studies. According to the results, some variables stand out in the acceptance and utilisation of HIS. Among the users of HIS, it can be said that the results relating to nurses stand out. In particular, there are studies which emphasise that 'gender' is a crucial factor in explaining the models. Another crucial finding of the current systematic review is the need to train users in the acceptance and use of HIS.

Virtual versus usual in-office care for multiple sclerosis: The VIRTUAL-MS multi-site randomized clinical trial study protocol

BACKGROUND

Multiple sclerosis (MS) affects nearly 1 million people and is estimated to cost $85.4 billion in the United States annually. People with MS have significant barriers to receiving care and telemedicine could substantially improve access to specialized, comprehensive care. In cross-sectional analyses, telemedicine has been shown to be feasible, have high patient and clinician satisfaction, reduce patient costs and burden, and enable a reasonable assessment of disability. However, no studies exist evaluating the longitudinal impact of telemedicine care for MS. Here we describe the study protocol for VIRtual versus UsuAL In-office care for Multiple Sclerosis (VIRTUAL-MS). The main objective of the study is to evaluate the impact of telemedicine for MS care on: patient clinical outcomes, economic costs, patient, and clinician experience.

METHODS

This two-site randomized clinical trial will enroll 120 adults with a recent diagnosis of MS and randomize 1:1 to receive in-clinic vs. telemedicine care for 24 months. The primary outcome of the study is worsening in any one of the four Multiple Sclerosis Functional Composite 4 (MSFC4) measures at 24 months. Other study outcomes include patient and clinician satisfaction, major healthcare costs, Expanded Disability Status Scale, treatment adherence, and digital outcomes.

CONCLUSION

The results of this study will directly address the key gaps in knowledge about longitudinal telemedicine-enabled care in an MS population. It will inform clinical care implementation as well as design of trials in MS and other chronic conditions.

TRIAL REGISTRATION

NCT05660187.

A Closed-Loop Digital Health Tool to Improve Depression Care in Multiple Sclerosis: Iterative Design and Cross-Sectional Pilot Randomized Controlled Trial and its Impact on Depression Care

BACKGROUND

People living with multiple sclerosis (MS) face a higher likelihood of being diagnosed with a depressive disorder than the general population. Although many low-cost screening tools and evidence-based interventions exist, depression in people living with MS is underreported, underascertained by clinicians, and undertreated.

OBJECTIVE

This study aims to design a closed-loop tool to improve depression care for these patients. It would support regular depression screening, tie into the point of care, and support shared decision-making and comprehensive follow-up. After an initial development phase, this study involved a proof-of-concept pilot randomized controlled trial (RCT) validation phase and a detailed human-centered design (HCD) phase.

METHODS

During the initial development phase, the technological infrastructure of a clinician-facing point-of-care clinical dashboard for MS management (BRIDGE) was leveraged to incorporate features that would support depression screening and comprehensive care (Care Technology to Ascertain, Treat, and Engage the Community to Heal Depression in people living with MS [MS CATCH]). This linked a patient survey, in-basket messages, and a clinician dashboard. During the pilot RCT phase, a convenience sample of 50 adults with MS was recruited from a single MS center with 9-item Patient Health Questionnaire scores of 5-19 (mild to moderately severe depression). During the routine MS visit, their clinicians were either asked or not to use MS CATCH to review their scores and care outcomes were collected. During the HCD phase, the MS CATCH components were iteratively modified based on feedback from stakeholders: people living with MS, MS clinicians, and interprofessional experts.

RESULTS

MS CATCH links 3 features designed to support mood reporting and ascertainment, comprehensive evidence-based management, and clinician and patient self-management behaviors likely to lead to sustained depression relief. In the pilot RCT (n=50 visits), visits in which the clinician was randomized to use MS CATCH had more notes documenting a discussion of depressive symptoms than those in which MS CATCH was not used (75% vs 34.6%; χ21=8.2; P=.004). During the HCD phase, 45 people living with MS, clinicians, and other experts participated in the design and refinement. The final testing round included 20 people living with MS and 10 clinicians including 5 not affiliated with our health system. Most scoring targets for likeability and usability, including perceived ease of use and perceived effectiveness, were met. Net Promoter Scale was 50 for patients and 40 for clinicians.

CONCLUSIONS

Created with extensive stakeholder feedback, MS CATCH is a closed-loop system aimed to increase communication about depression between people living with MS and their clinicians, and ultimately improve depression care. The pilot findings showed evidence of enhanced communication. Stakeholders also advised on trial design features of a full year long Department of Defense-funded feasibility and efficacy trial, which is now underway.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05865405; http://tinyurl.com/4zkvru9x.

Pragmatic phase II clinical trial to improve depression care in a real-world diverse MS cohort from an academic MS centre in Northern California: MS CATCH study protocol

INTRODUCTION

Depression occurs in over 50% of individuals living with multiple sclerosis (MS) and can be treated using many modalities. Yet, it remains: under-reported by patients, under-ascertained by clinicians and under-treated. To enhance these three behaviours likely to promote evidence-based depression care, we engaged multiple stakeholders to iteratively design a first-in-kind digital health tool. The tool, MS CATCH (Care technology to Ascertain, Treat, and engage the Community to Heal depression in patients with MS), closes the communication loop between patients and clinicians. Between clinical visits, the tool queries patients monthly about mood symptoms, supports patient self-management and alerts clinicians to worsening mood via their electronic health record in-basket. Clinicians can also access an MS CATCH dashboard displaying patients' mood scores over the course of their disease, and providing comprehensive management tools (contributing factors, antidepressant pathway, resources in patient's neighbourhood). The goal of the current trial is to evaluate the clinical effect and usability of MS CATCH in a real-world clinical setting.

METHODS AND ANALYSIS

MS CATCH is a single-site, phase II randomised, delayed start, trial enrolling 125 adults with MS and mild to moderately severe depression. Arm 1 will receive MS CATCH for 12 months, and arm 2 will receive usual care for 6 months, then MS CATCH for 6 months. Clinicians will be randomised to avoid practice effects. The effectiveness analysis is superiority intent-to-treat comparing MS CATCH to usual care over 6 months (primary outcome: evidence of screening and treatment; secondary outcome: Hospital Anxiety Depression Scale-Depression scores). The usability of the intervention will also be evaluated (primary outcome: adoption; secondary outcomes: adherence, engagement, satisfaction).

ETHICS AND DISSEMINATION

University of California, San Francisco Institutional Review Board (22-36620). The findings of the study are planned to be shared through conferences and publishments in a peer-reviewed journal. The deidentified dataset will be shared with qualified collaborators on request, provision of CITI and other certifications, and data sharing agreement. We will share the results, once the data are complete and analysed, with the scientific community and patient/clinician participants through abstracts, presentations and manuscripts.

TRIAL REGISTRATION NUMBER

NCT05865405.

Monitoring Diagnostic Safety Risks in Emergency Departments: Protocol for a Machine Learning Study

BACKGROUND

Diagnostic decision making, especially in emergency departments, is a highly complex cognitive process that involves uncertainty and susceptibility to errors. A combination of factors, including patient factors (eg, history, behaviors, complexity, and comorbidity), provider-care team factors (eg, cognitive load and information gathering and synthesis), and system factors (eg, health information technology, crowding, shift-based work, and interruptions) may contribute to diagnostic errors. Using electronic triggers to identify records of patients with certain patterns of care, such as escalation of care, has been useful to screen for diagnostic errors. Once errors are identified, sophisticated data analytics and machine learning techniques can be applied to existing electronic health record (EHR) data sets to shed light on potential risk factors influencing diagnostic decision making.

OBJECTIVE

This study aims to identify variables associated with diagnostic errors in emergency departments using large-scale EHR data and machine learning techniques.

METHODS

This study plans to use trigger algorithms within EHR data repositories to generate a large data set of records that are labeled trigger-positive or trigger-negative, depending on whether they meet certain criteria. Samples from both data sets will be validated using medical record reviews, upon which we expect to find a higher number of diagnostic safety events in the trigger-positive subset. Machine learning will be used to evaluate relationships between certain patient factors, provider-care team factors, and system-level risk factors and diagnostic safety signals in the statistically matched groups of trigger-positive and trigger-negative charts.

RESULTS

This federally funded study was approved by the institutional review board of 2 academic medical centers with affiliated community hospitals. Trigger queries are being developed at both organizations, and sample cohorts will be labeled using the triggers. Machine learning techniques such as association rule mining, chi-square automated interaction detection, and classification and regression trees will be used to discover important variables that could be incorporated within future clinical decision support systems to help identify and reduce risks that contribute to diagnostic errors.

CONCLUSIONS

The use of large EHR data sets and machine learning to investigate risk factors (related to the patient, provider-care team, and system-level) in the diagnostic process may help create future mechanisms for monitoring diagnostic safety.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/24642.

Contemporary use of real-world data for clinical trial conduct in the United States: a scoping review

OBJECTIVE

Real-world data (RWD), defined as routinely collected healthcare data, can be a potential catalyst for addressing challenges faced in clinical trials. We performed a scoping review of database-specific RWD applications within clinical trial contexts, synthesizing prominent uses and themes.

MATERIALS AND METHODS

Querying 3 biomedical literature databases, research articles using electronic health records, administrative claims databases, or clinical registries either within a clinical trial or in tandem with methodology related to clinical trials were included. Articles were required to use at least 1 US RWD source. All abstract screening, full-text screening, and data extraction was performed by 1 reviewer. Two reviewers independently verified all decisions.

RESULTS

Of 2020 screened articles, 89 qualified: 59 articles used electronic health records, 29 used administrative claims, and 26 used registries. Our synthesis was driven by the general life cycle of a clinical trial, culminating into 3 major themes: trial process tasks (51 articles); dissemination strategies (6); and generalizability assessments (34). Despite a diverse set of diseases studied, <10% of trials using RWD for trial process tasks evaluated medications or procedures (5/51). All articles highlighted data-related challenges, such as missing values.

DISCUSSION

Database-specific RWD have been occasionally leveraged for various clinical trial tasks. We observed underuse of RWD within conducted medication or procedure trials, though it is subject to the confounder of implicit report of RWD use.

CONCLUSION

Enhanced incorporation of RWD should be further explored for medication or procedure trials, including better understanding of how to handle related data quality issues to facilitate RWD use.

Unity Is Intelligence: A Collective Intelligence Experiment on ECG Reading to Improve Diagnostic Performance in Cardiology

Medical errors have a huge impact on clinical practice in terms of economic and human costs. As a result, technology-based solutions, such as those grounded in artificial intelligence (AI) or collective intelligence (CI), have attracted increasing interest as a means of reducing error rates and their impacts. Previous studies have shown that a combination of individual opinions based on rules, weighting mechanisms, or other CI solutions could improve diagnostic accuracy with respect to individual doctors. We conducted a study to investigate the potential of this approach in cardiology and, more precisely, in electrocardiogram (ECG) reading. To achieve this aim, we designed and conducted an experiment involving medical students, recent graduates, and residents, who were asked to annotate a collection of 10 ECGs of various complexity and difficulty. For each ECG, we considered groups of increasing size (from three to 30 members) and applied three different CI protocols. In all cases, the results showed a statistically significant improvement (ranging from 9% to 88%) in terms of diagnostic accuracy when compared to the performance of individual readers; this difference held for not only large groups, but also smaller ones. In light of these results, we conclude that CI approaches can support the tasks mentioned above, and possibly other similar ones as well. We discuss the implications of applying CI solutions to clinical settings, such as cases of augmented ‘second opinions’ and decision-making.

Impact of digitally acquired peer diagnostic input on diagnostic confidence in outpatient cases: A pragmatic randomized trial

OBJECTIVE

The study sought to evaluate if peer input on outpatient cases impacted diagnostic confidence.

MATERIALS AND METHODS

This randomized trial of a peer input intervention occurred among 28 clinicians with case-level randomization. Encounters with diagnostic uncertainty were entered onto a digital platform to collect input from ≥5 clinicians. The primary outcome was diagnostic confidence. We used mixed-effects logistic regression analyses to assess for intervention impact on diagnostic confidence.

RESULTS

Among the 509 cases (255 control; 254 intervention), the intervention did not impact confidence (odds ratio [OR], 1.46; 95% confidence interval [CI], 0.999-2.12), but after adjusting for clinician and case traits, the intervention was associated with higher confidence (OR, 1.53; 95% CI, 1.01-2.32). The intervention impact was greater in cases with high uncertainty (OR, 3.23; 95% CI, 1.09- 9.52).

CONCLUSIONS

Peer input increased diagnostic confidence primarily in high-uncertainty cases, consistent with findings that clinicians desire input primarily in cases with continued uncertainty.