User-centered design of a scalable, electronic health record-integrated remote symptom monitoring intervention for patients with asthma and providers in primary care

A Collection of Components to Design Clinical Dashboards Incorporating Patient-Reported Outcome Measures: Qualitative Study

BACKGROUND

A clinical dashboard is a data-driven clinical decision support tool visualizing multiple key performance indicators in a single report while minimizing time and effort for data gathering. Studies have shown that including patient-reported outcome measures (PROMs) in clinical dashboards supports the clinician's understanding of how treatments impact patients' health status, helps identify changes in health-related quality of life at an early stage, and strengthens patient-physician communication.

OBJECTIVE

This study aims to determine design components for clinical dashboards incorporating PROMs to inform software producers and users (ie, physicians).

METHODS

We conducted interviews with software producers and users to test preselected design components. Furthermore, the interviews allowed us to derive additional components that are not outlined in existing literature. Finally, we used inductive and deductive coding to derive a guide on which design components need to be considered when building a clinical dashboard incorporating PROMs.

RESULTS

A total of 25 design components were identified, of which 16 were already surfaced during the literature search. Furthermore, 9 additional components were derived inductively during our interviews. The design components are clustered in a generic dashboard, PROM-related, adjacent information, and requirements for adoption components. Both software producers and users agreed on the primary purpose of a clinical dashboard incorporating PROMs to enhance patient communication in outpatient settings. Dashboard benefits include enhanced data visualization and improved workflow efficiency, while interoperability and data collection were named as adoption challenges. Consistency in dashboard design components is preferred across different episodes of care, with adaptations only for disease-specific PROMs.

CONCLUSIONS

Clinical dashboards have the potential to facilitate informed treatment decisions if certain design components are followed. This study establishes a comprehensive framework of design components to guide the development of effective clinical dashboards incorporating PROMs in health care practice.

Development of an electronic health record-integrated patient-reported outcome-based shared decision-making dashboard in oncology

Objectives

Patient-reported outcomes (PROs) describe a patient's unique experiences with disease or treatment, yet effective use of this information during clinical encounters remains challenging. This project sought to build a PRO based dashboard within the electronic health record (EHR), prioritizing interpretability and utility of PROs for clinical decision-making.

Materials and Methods

Codesign principles were used to define the goal, features, and visualization of the data elements on the dashboard. Codesign sessions occurred between February 2019 and May 2020 and involved a diverse group of stakeholders. Pilot evaluation of dashboard usability was performed with patients and clinicians not involved in the codesign process through qualitative interviews and the Systems Usability Scale.

Results

The dashboard was placed into a single tab in the EHR and included select PROM scores, clinical data elements, and goals of care questions. Real-time data analytics and enhanced visualization of data was necessary for the dashboard to provide meaningful feedback to clinicians and patients for decision-making during clinic visits. During soft launch, the dashboard demonstrated "good" usability in patients and clinicians at 3 and 6 months (mean total SUS score >70).

Discussion

The current dashboard had good usability and made PRO scores more clinically understandable to patients and clinicians. This paper highlights the development, necessary data elements, and workflow considerations to implement this dashboard at an academic cancer center.

Conclusion

As the use of PROs in clinical care is increasing, patient- and clinician-centered tools are needed to ensure that this information is used in meaningful ways.

Realizing the Full Potential of Clinical Decision Support: Translating Usability Testing into Routine Practice in Health Care Operations

BACKGROUND

 Clinical Decision Support (CDS) tools have a mixed record of effectiveness, often due to inadequate alignment with clinical workflows and poor usability. While there is a consensus that usability testing methods address these issues, in practice, usability testing is generally only used for selected projects (such as funded research studies). There is a critical need for CDS operations to apply usability testing to all CDS implementations.

OBJECTIVES

 In this State of the Art/Best Practice paper, we share challenges with scaling usability in health care operations and alternative methods and CDS governance structures to enable usability testing as a routine practice.

METHODS

 We coalesce our experience and results of applying guerilla in situ usability testing to over 20 projects in a 1-year period with the proposed solution.

RESULTS

 We demonstrate the feasibility of adopting "guerilla in situ usability testing" in operations and their effectiveness in incorporating user feedback and improving design.

CONCLUSION

 Although some methodological rigor was relaxed to accommodate operational speed, the benefits outweighed the limitations. Broader adoption of usability testing may transform CDS implementation and improve health outcomes.

Evaluation of a Primary Care-Integrated Mobile Health Intervention to Monitor between-Visit Asthma Symptoms

OBJECTIVES

 This study aimed to evaluate implementation of a digital remote symptom monitoring intervention that delivered weekly symptom questionnaires and included the option to receive nurse callbacks via a mobile app for asthma patients in primary care.

METHODS

 Research questions were structured by the NASSS (Nonadoption, Abandonment, Scale-up Spread, and Sustainability) framework. Quantitative and qualitative methods assessed scalability of the electronic health record (EHR)-integrated app intervention implemented in a 12-month randomized controlled trial. Data sources included patient asthma control questionnaires; app usage logs; EHRs; and interviews and discussions with patients, primary care providers (PCPs), and nurses.

RESULTS

 We included app usage data from 190 patients and interview data from 21 patients and several clinician participants. Among 190 patients, average questionnaire completion rate was 72.3% and retention was 78.9% (i.e., 150 patients continued to use the app at the end of the trial period). App use was lower among Hispanic and younger patients and those with fewer years of education. Of 1,185 nurse callback requests offered to patients. Thirty-three (2.8%) were requested. Of 84 PCP participants, 14 (16.7%) accessed the patient-reported data in the EHR. Analyses showed that the intervention was appropriate for all levels of asthma control; had no major technical barriers; was desirable and useful for patient treatment; involved achievable tasks for patients; required modest role changes for clinicians; and was a minimal burden on the organization.

CONCLUSION

 A clinically integrated symptom monitoring intervention has strong potential for sustained adoption. Inequitable adoption remains a concern. PCP use of patient-reported data during visits could improve intervention adoption but may not be required for patient benefits.

Evaluation of a Digital Scribe: Conversation Summarization for Emergency Department Consultation Calls

OBJECTIVE

We present a proof-of-concept digital scribe system as an Emergency Department (ED) consultation call-based clinical conversation summarization pipeline to support clinical documentation, and report its performance.

MATERIALS AND METHODS

We use four pre-trained large language models to establish the digital scribe system: T5-small, T5-base, PEGASUS-PubMed, and BART-Large-CNN via zero-shot and fine-tuning approaches. Our dataset includes 100 referral conversations among ED clinicians and medical records. We report the ROUGE-1, ROUGE-2, and ROUGE-L to compare model performance. In addition, we annotated transcriptions to assess the quality of generated summaries.

RESULTS

The fine-tuned BART-Large-CNN model demonstrates greater performance in summarization tasks with the highest ROUGE scores (F1ROUGE-1=0.49, F1ROUGE-2=0.23, F1ROUGE-L=0.35) scores. In contrast, PEGASUS-PubMed lags notably (F1ROUGE-1=0.28, F1ROUGE-2=0.11, F1ROUGE-L=0.22). BART-Large-CNN's performance decreases by more than 50% with the zero-shot approach. Annotations show that BART-Large-CNN performs 71.4% recall in identifying key information and a 67.7% accuracy rate.

DISCUSSION

The BART-Large-CNN model demonstrates a high level of understanding of clinical dialogue structure, indicated by its performance with and without fine-tuning. Despite some instances of high recall, there is variability in the model's performance, particularly in achieving consistent correctness, suggesting room for refinement. The model's recall ability varies across different information categories.

CONCLUSION

The study provides evidence towards the potential of AI-assisted tools in assisting clinical documentation. Future work is suggested on expanding the research scope with additional language models and hybrid approaches, and comparative analysis to measure documentation burden and human factors.

Electronic Collection of Patient-Reported Outcomes to Improve Kidney Care: Benefits, Drawbacks, and Next Steps

Kidney services worldwide are increasingly using digital health technologies to deliver care. This includes kidney electronic patient-reported outcome (ePRO) systems: ambulatory digital technologies that enable the capture of PRO data electronically from people with kidney disease remotely and in real time to be shared with their kidney care team. Current kidney ePRO systems commonly aim to support the monitoring and management of symptoms in patients with kidney disease. The majority have thus far only been implemented in research settings and are not yet routinely used in clinical practice, leaving their readiness for real-world implementation largely unknown. Compared with paper-based PRO collection, ePRO systems have certain advantages, which we categorize as efficiency benefits (e.g., lower administrative burden), direct patient care benefits (e.g., automated PRO-based patient education), and health system and research benefits (e.g., collecting ePRO data once for multiple purposes). At the same time, kidney ePRO systems come with drawbacks, such as their potential to exacerbate existing inequities in care and outcomes and to negatively affect staff burden and patients' experience of kidney care. Areas that hold promise for expediting the development and uptake of kidney ePRO systems at the local, organizational, and national level include harnessing national kidney registries as enabling infrastructures; using novel data-driven technologies (e.g., computerized adaptive test systems, configurable dashboards); applying implementation science and action research approaches to enhance translation of ePRO research findings into clinical practice; and engaging stakeholders, including patients and carers, health care professionals, policymakers, payers, ePRO experts, technology providers, and organizations that monitor and improve the quality of kidney services.

Development and Usability Testing of an Exercise-Based Primary Care Fall Prevention Clinical Decision Support Tool

For older patients, falls are the leading cause offatal and nonfatal injuries. Guidelines recommend that at-risk older adults are referred to appropriate fall-prevention exercise programs, but many do not receive support for fall-risk management in the primary care setting. Advances in health information technology may be able to address this gap. This article describes the development and usability testing of a clinical decision support (CDS) tool for fall prevention exercise. Using rapid qualitative analysis and human-centered design, our team developed and tested the usability of our CDS prototype with primary care team members. Across 31 Health-Information Technology Usability Evaluation Scale surveys, our CDS prototype received a median score of 5.0, mean (SD) of 4.5 (0.8), and a range of 4.1-4.9. This study highlights the features and usability offall prevention CDS for helping primary care providers deliver patient-centeredfall prevention care.

Development and Evaluation of a Digital Scribe: Conversation Summarization Pipeline for Emergency Department Counseling Sessions towards Reducing Documentation Burden

Objective

We present a proof-of-concept digital scribe system as an ED clinical conversation summarization pipeline and report its performance.

Materials and Methods

We use four pre-trained large language models to establish the digital scribe system: T5-small, T5-base, PEGASUS-PubMed, and BART-Large-CNN via zero-shot and fine-tuning approaches. Our dataset includes 100 referral conversations among ED clinicians and medical records. We report the ROUGE-1, ROUGE-2, and ROUGE-L to compare model performance. In addition, we annotated transcriptions to assess the quality of generated summaries.

Results

The fine-tuned BART-Large-CNN model demonstrates greater performance in summarization tasks with the highest ROUGE scores (F1ROUGE-1=0.49, F1ROUGE-2=0.23, F1ROUGE-L=0.35) scores. In contrast, PEGASUS-PubMed lags notably (F1ROUGE-1=0.28, F1ROUGE-2=0.11, F1ROUGE-L=0.22). BART-Large-CNN's performance decreases by more than 50% with the zero-shot approach. Annotations show that BART-Large-CNN performs 71.4% recall in identifying key information and a 67.7% accuracy rate.

Discussion

The BART-Large-CNN model demonstrates a high level of understanding of clinical dialogue structure, indicated by its performance with and without fine-tuning. Despite some instances of high recall, there is variability in the model's performance, particularly in achieving consistent correctness, suggesting room for refinement. The model's recall ability varies across different information categories.

Conclusion

The study provides evidence towards the potential of AI-assisted tools in reducing clinical documentation burden. Future work is suggested on expanding the research scope with larger language models, and comparative analysis to measure documentation efforts and time.

Examining primary care provider experiences with using a clinical decision support tool for pain management

Objective

To evaluate primary care provider (PCP) experiences using a clinical decision support (CDS) tool over 16 months following a user-centered design process and implementation.

Materials and Methods

We conducted a qualitative evaluation of the Chronic Pain OneSheet (OneSheet), a chronic pain CDS tool. OneSheet provides pain- and opioid-related risks, benefits, and treatment information for patients with chronic pain to PCPs. Using the 5 Rights of CDS framework, we conducted and analyzed semi-structured interviews with 19 PCPs across 2 academic health systems.

Results

PCPs stated that OneSheet mostly contained the right information required to treat patients with chronic pain and was correctly located in the electronic health record. PCPs used OneSheet for distinct subgroups of patients with chronic pain, including patients prescribed opioids, with poorly controlled pain, or new to a provider or clinic. PCPs reported variable workflow integration and selective use of certain OneSheet features driven by their preferences and patient population. PCPs recommended broadening OneSheet access to clinical staff and patients for data entry to address clinician time constraints.

Discussion

Differences in patient subpopulations and workflow preferences had an outsized effect on CDS tool use even when the CDS contained the right information identified in a user-centered design process.

Conclusions

To increase adoption and use, CDS design and implementation processes may benefit from increased tailoring that accommodates variation and dynamics among patients, visits, and providers.

Are older adults considered in asthma and chronic obstructive pulmonary disease mobile health research? A scoping review

BACKGROUND

The use of mobile health (mHealth) for asthma and chronic obstructive pulmonary disease (COPD) is rapidly growing and may help address the complex respiratory care needs of our ageing population. However, little is currently known about how airways mHealth is developed and used among older adults (≥65 years).

OBJECTIVE

To identify if and how older adults with asthma and COPD have been incorporated across the mHealth research cycle.

METHODS

We searched Ovid MEDLINE, EMBASE, CINAHL and the Cochrane Central Registry of Controlled Trials for studies pertaining to the development or evaluation of asthma and COPD mHealth for adults published after 2010. Study, participant and mHealth details, including any considerations of older age, were extracted, synthesised and charted.

RESULTS

A total of 334 studies of 191 mHealth tools were identified. Adults ≥65 years old were included in 33.3% of asthma mHealth studies and 85.3% of COPD studies. Discussions of older age focused on barriers to technology use. Methodologic and/or analytic considerations of older age were mostly absent throughout the research cycle. Among the 28 instances quantitative age-related analyses were detailed, 12 described positive mHealth use and satisfaction outcomes in older adults versus negative or equivocal outcomes.

CONCLUSION

We identified an overall lack of consideration for older age throughout the airways mHealth research cycle, even among COPD mHealth studies that predominantly included older adults. We also found a contrast between the perceptions of how older age might negatively influence mHealth use and available quantitative evaluations. Future airways mHealth research must better integrate the needs and concerns of older adults.

Rapid implementation of remote digital primary care in Stockholm and implications for further system-wide implementation: practitioner's and manager's experience of the Always Open mobile application

OBJECTIVE

To contribute actionable knowledge how to increase appropriate use of digital technologies in primary care by understanding clinical managers experiences with a digital connection system, Always Open, during the COVID-19 pandemic.

DESIGN AND SUBJECTS

The overall design was a qualitative study with directed content analysis method. Data were collected from documents and focus group (n = 12) interviews with clinical managers (n = 99) of primary care. The seven domains of the Non-adoption, Abandonment, and challenges to the Scale-up, Spread and Sustainability (NASSS) framework was used to understand the implementation process, as described by the clinical managers.

RESULTS

Focus group participants reported that their units made their own local decisions to make more use of the technology provided by the health system. Most participants considered that the technology was ready to use, despite some limitations, that included individual clinician's and patient preferences, and how ready their unit was for making changes to practice and organization. Some raised concerns about how standardizing some aspects possibly conflicted with the decentralized management model of the organization. The overall experience was reported to be positive, with an intention to sustain the achievements.

CONCLUSION

Focus group interviews found that clinical unit managers reported that they and their staff were positive about the digital technology system for remote care. For the future, they wanted changes to be made at different levels of the health system to better combine digital and physical care. Possibilities to use digital technology to integrate primary and hospital health care were identified.

Rudin R, Sulca Flores J, Goolkasian G, Sousa J, Rodriguez J, Lipsitz S, Foer D, K. Dalal A. Assessing Equitable Recruitment in a Digital Health Trial for Asthma

OBJECTIVE

This study aimed to assess a multipronged strategy using primarily digital methods to equitably recruit asthma patients into a clinical trial of a digital health intervention.

METHODS

We approached eligible patients using at least one of eight recruitment strategies. We recorded approach dates and the strategy that led to completion of a web-based eligibility questionnaire that was reported during the verbal consent phone call. Study team members conducted monthly sessions using a structured guide to identify recruitment barriers and facilitators. The proportion of participants who reported being recruited by a portal or nonportal strategy was measured as our outcomes. We used Fisher's exact test to compare outcomes by equity variable, and multivariable logistic regression to control for each covariate and adjust effect size estimates. Using grounded theory, we coded and extracted themes regarding recruitment barriers and facilitators.

RESULTS

The majority (84.4%) of patients who met study inclusion criteria were patient portal enrollees. Of 6,366 eligible patients who were approached, 627 completed the eligibility questionnaire and were less frequently Hispanic, less frequently Spanish-speaking, and more frequently patient portal enrollees. Of 445 patients who consented to participate, 241 (54.2%) reported completing the eligibility questionnaire after being contacted by a patient portal message. In adjusted analysis, only race (odds ratio [OR]: 0.46, 95% confidence interval [CI]: 0.28-0.77, p = 0.003) and college education (OR: 0.60, 95% CI: 0.39-0.91, p = 0.016) remained significant. Key recruitment barriers included technology issues (e.g., lack of email access) and facilitators included bilingual study staff, Spanish-language recruitment materials, targeted phone calls, and clinician-initiated "1-click" referrals.

CONCLUSION

A primarily digital strategy to recruit patients into a digital health trial is unlikely to achieve equitable participation, even in a population overrepresented by patient portal enrollees. Nondigital recruitment methods that address racial and educational disparities and less active portal enrollees are necessary to ensure equity in clinical trial enrollment.

The relationship between patient enablement and help-seeking in the context of blood cancer symptoms

OBJECTIVE

Approaches to improve earlier diagnosis of cancer often focus on symptom awareness as a key driver of help-seeking behaviour and other psychological influences are less well understood. This is the first study to explore the role of patient enablement on help-seeking for people experiencing potential blood cancer symptoms.

METHODS

A cross-sectional, nationally representative survey was completed by 434 respondents (>18 years). Questions asked about symptom experiences, medical help-seeking and re-consultation. Existing patient enablement items were included in the newly developed Blood Cancer Awareness Measure. We collected data on patient socio-demographic characteristics.

RESULTS

Of those responding to the survey 224/434 (51.6%) reported experiencing at least one potential blood cancer symptom. Half of those experiencing symptoms (112/224) had sought medical help. Results from logistic regression analysis showed that higher scores on patient enablement were associated with being less likely to seek help (Odds Ratio [OR] 0.89, Confidence Interval [CI] 0.81-0.98) after controlling for socio-demographics. Separate analyses showed that higher enablement was associated with being more comfortable to re-consult if symptoms didn't go away or got worse (OR 1.31, CI 1.16-1.48); after a test result suggested there was nothing to worry about, but symptoms persisted (OR 1.23, CI 1.12-1.34) or to request further tests, scans or investigations (OR 1.31, CI 1.19-1.44).

CONCLUSIONS

Contrary to our hypotheses, patient enablement was associated with lower likelihood of help-seeking for potential blood cancer symptoms. Yet enablement appears to play an important role in likelihood of re-consulting when symptoms persist, get worse or need further investigation.

Top Ten Tips Palliative Care Clinicians Should Know About Leveraging the Electronic Health Record for Data Collection and Quality Improvement

As palliative care (PC) programs rapidly grow and expand across settings, the need to measure, improve, and standardize high-quality PC has also grown. The electronic health record (EHR) is a key component of these efforts as a central hub of care delivery and a repository of patient and system data. Deliberate efforts to leverage the EHR for PC quality improvement (QI) can help PC programs and health systems improve care for patients with serious illnesses. This article, written by clinicians with experience in QI, informatics, and clinical program development, provides practical tips and guidance on EHR strategies and tools for QI and quality measurement.

Current challenges for evaluating mobile health applications

As mobile health applications continue to proliferate without clear regulation, the need for app evaluation frameworks to offer guidance to patients and clinicians also expands. However, this expanding number of app evaluation frameworks itself can be a source of confusion and often contradictory recommendations. In pursuit of better frameworks that offer innovation for app evaluation, we present 4 challenges that app evaluation frameworks must overcome as well as examples from our own experience toward overcoming them. The recommendations are applicable to all health apps from any field of medicine, although we use examples from mental health as they are illustrative.

Development and Testing of an Electronic Health Record-Integrated Patient-Reported Outcome Application and Intervention to Improve Efficiency of Rheumatoid Arthritis Care

OBJECTIVE

Many patients with rheumatoid arthritis (RA) have difficulty finding clinicians to treat them because of workforce shortages. We developed an app to address this problem by improving care efficiency. The app collects patient-reported outcomes (PROs) and can be used to inform visit timing, potentially reducing the volume of low-value visits. We describe the development process, intervention design, and planned study for testing the app.

METHODS

We employed user-centered design, interviewing patients and clinicians, to develop the app. To improve visit efficiency, symptom tracking logic alerts clinicians to PRO trends: worsening PROs generate alerts suggesting an earlier visit, and stable or improving PROs generate notifications that scheduled visits could be delayed. An interrupted time-series analysis with a nonrandomized control population will allow assessment of the impact of the app on visit frequency.

RESULTS

Patient interviews identified several of the following needs for effective app and intervention design: the importance of a simple user interface facilitating rapid answering of PROs, the availability of condensed summary information with links to more in-depth answers to common questions regarding RA, and the need for clinicians to discuss the PRO data during visits with patients. Clinician interviews identified the following user needs: PRO data must be easy to view and use during the clinical workflow, and there should be reduced interval visits when PROs are trending worse. Some clinicians believed visits could be delayed for patients with stable PROs, whereas others raised concerns.

CONCLUSION

PRO apps may improve care efficiency in rheumatology. Formal evaluation of an integrated PRO RA app is forthcoming.

OUP accepted manuscript

Objective

We conducted a horizon scan to (1) identify challenges in patient-centered clinical decision support (PC CDS) and (2) identify future directions for PC CDS.

Materials and Methods

We engaged a technical expert panel, conducted a scoping literature review, and interviewed key informants. We qualitatively analyzed literature and interview transcripts, mapping findings to the 4 phases for translating evidence into PC CDS interventions (Prioritizing, Authoring, Implementing, and Measuring) and to external factors.

Results

We identified 12 challenges for PC CDS development. Lack of patient input was identified as a critical challenge. The key informants noted that patient input is critical to prioritizing topics for PC CDS and to ensuring that CDS aligns with patients’ routine behaviors. Lack of patient-centered terminology standards was viewed as a challenge in authoring PC CDS. We found a dearth of CDS studies that measured clinical outcomes, creating significant gaps in our understanding of PC CDS’ impact. Across all phases of CDS development, there is a lack of patient and provider trust and limited attention to patients’ and providers’ concerns.

Discussion

These challenges suggest opportunities for advancing PC CDS. There are opportunities to develop industry-wide practices and standards to increase transparency, standardize terminologies, and incorporate patient input. There is also opportunity to engage patients throughout the PC CDS research process to ensure that outcome measures are relevant to their needs.

Conclusion

Addressing these challenges and embracing these opportunities will help realize the promise of PC CDS—placing patients at the center of the healthcare system.