An Evaluation Service for Digital Public Health Interventions: User-Centered Design Approach

Using Civic Service Design Methods to Redevelop a Data Communication Website With a Health Literacy Lens

CONTEXT

Public health agencies routinely publish data in hopes that data influence public health policy and practice. However, data websites can often be difficult to use, posing barriers to people trying to access, understand, and use data. Working to make data websites easier to use can add value to public health data communication work.

PROGRAM

The New York City Department of Health and Mental Hygiene (DOHMH) redesigned its Environment and Health Data Portal, a website used to communicate environmental health data, with the goal of making data more accessible and understandable to a broader audience. The DOHMH used Civic Service Design methods to establish priorities and strategies for the redesign work, to build a data communication website that emphasizes a high level of usability, and content that explains data.

IMPLEMENTATION

By following a Civic Service Design process, the DOHMH synthesized findings from health communications, data visualization and communication, and web usability to create an easy-to-use website with explanations of data and findings alongside datasets. On the new site, automated dataset visualizations are supplemented with narrative content, explanatory content, and custom interactive applications designed to explain data and findings.

EVALUATION

Web analytics showed that, in its first year of operation, the site's web traffic grew substantially, with the last 12 weeks recording weekly page views 150% higher than the first 12 weeks of operation (7185 average weekly page views compared with 2866 average weekly page views). Two-thirds (66.3%) of page views include recorded user engagement. Additional evaluations to measure specific aspects of usability compared with the previous version of the site are planned.

DISCUSSION

By following a Civic Service Design process, the DOHMH redesigned a vital data communication platform to increase its usability and saw significant increase in engagement in its first year of operations. By designing data material with usability in mind, public health departments have the potential to improve public health data communication work.

Exploring the Impact of a Sleep App on Sleep Quality in a General Population Sample: Pilot Randomized Controlled Trial

BACKGROUND

A third of adults in Western countries have impaired sleep quality. A possible solution involves distributing sleep aids through smartphone apps, but most empirical studies are limited to small pilot trials in distinct populations (eg, soldiers) or individuals with clinical sleep disorders; therefore, general population data are required. Furthermore, recent research shows that sleep app users desire a personalized approach, offering an individually tailored choice of techniques. One such aid is Peak Sleep, a smartphone app based on scientifically validated principles for improving sleep quality, such as mindfulness meditation and cognitive behavioral therapy.

OBJECTIVE

We aimed to test the impact of the smartphone app Peak Sleep on sleep quality and collect user experience data to allow for future app development.

METHODS

This was a 2-arm pilot randomized controlled trial. Participants were general population adults in the United Kingdom (aged ≥18 years) who were interested in improving their sleep quality and were not undergoing clinical treatment for sleep disorder or using sleep medication ≥1 per week. Participants were individually randomized to receive the intervention (3 months of app use) versus a no-treatment control. The intervention involved free access to Peak Sleep, an app that offered a choice of behavioral techniques to support better sleep (mindfulness, cognitive behavioral therapy, and acceptance commitment therapy). The primary outcome was sleep quality assessed using the Insomnia Severity Index at baseline and 1-, 2-, and 3-month follow-ups. Assessments were remote using web-based questionnaires. Objective sleep data collection using the Oura Ring (Ōura Health Oy) was planned; however, because the COVID-19 pandemic lockdowns began just after recruitment started, this plan could not be realized. Participant engagement with the app was assessed using the Digital Behavior Change Intervention Engagement Scale and qualitative telephone interviews with a subsample.

RESULTS

A total of 101 participants were enrolled in the trial, and 21 (21%) were qualitatively interviewed. Sleep quality improved in both groups over time, with Insomnia Severity Index scores of the intervention group improving by a mean of 2.5 and the control group by a mean of 1.6 (between-group mean difference 0.9, 95% CI -2.0 to 3.8), with was no significant effect of group (P=.91). App users' engagement was mixed, with qualitative interviews supporting the view of a polarized sample who either strongly liked or disliked the app.

CONCLUSIONS

In this trial, self-reported sleep improved over time in both intervention and control arms, with no impact by group, suggesting no effect of the sleep app. Qualitative data suggested polarized views on liking or not liking the app, features that people engaged with, and areas for improvement. Future work could involve developing the app features and then testing the app using objective measures of sleep in a larger sample.

TRIAL REGISTRATION

ClinicalTrials.gov NCT04487483; https://www.clinicaltrials.gov/study/NCT04487483.

Methods, Indicators, and End-User Involvement in the Evaluation of Digital Health Interventions for the Public: Scoping Review

BACKGROUND

Digital health interventions (DHIs) have the potential to enable public end users, such as citizens and patients, to manage and improve their health. Although the number of available DHIs is increasing, examples of successfully established DHIs in public health systems are limited. To counteract the nonuse of DHIs, they should be comprehensively evaluated while integrating end users. Unfortunately, there is a wide variability and heterogeneity according to the approaches of evaluation, which creates a methodological challenge.

OBJECTIVE

This scoping review aims to provide an overview of the current established processes for evaluating DHIs, including methods, indicators, and end-user involvement. The review is not limited to a specific medical field or type of DHI but offers a holistic overview.

METHODS

This scoping review was conducted following the JBI methodology for scoping reviews based on the framework by Arksey & O'Malley and complies with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines. Three scientific databases (PubMed, Scopus, and Science Direct) were searched in April 2023. English and German studies between 2008 and 2023 were considered when evaluating DHIs that explicitly address public end users. The process of study selection was carried out by several researchers to avoid reviewer bias.

RESULTS

The search strategy identified 9618 publications, of which 160 were included. Among these included articles, 200 evaluations were derived and analyzed. The results showed that there is neither a consensus on the methods to evaluate DHIs nor a commonly agreed definition or usage of the evaluated indicators, which results in a broad variety of evaluation practices. This aligns with observations of the existing literature. It was found that there is a lack of references to existing frameworks for the evaluation of DHIs. The majority of the included studies referred to user-centered approaches and involved end users in the evaluation process. As assistance for people developing and evaluating DHIs and as a basis for thinking about appropriate ways to evaluate DHIs, a results matrix was created where the findings were combined per DHI cluster. Additionally, general recommendations for the evaluators of DHIs were formulated.

CONCLUSIONS

The findings of this scoping review offer a holistic overview of the variety and heterogeneity according to the approaches of evaluation of DHIs for public end users. Evaluators of these DHIs should be encouraged to reference established frameworks or measurements for justification. This would ease the transferability of the results among similar evaluation studies within the digital health sector, thereby enhancing the coherence and comparability of research in this area.

Digital intervention for public health: searching for implementing characteristics, concepts and recommendations: scoping review

Studying the impact of digital interventions on public health can help ensure that the offered services produce the desired results. In order to address these factors, the subsequent study uses a scope review to evaluate the state of the field while concentrating on ideas and suggestions that represent factors that have been crucial in the management of digital intervention for public health. To shed light on the traits, ideas and suggestions related to public health digital intervention, a scoping review was carried out. Five electronic databases were used to locate pertinent research that were published before February 2022. All texts were examined, and study abstracts were scrutinized to determine their eligibility. The last analysis of this study included fifteen publications; five reviews, four qualitative studies, two quantitative studies, one viewpoint study, one mixed-method study, one perspective study, and one interventional study. The key ideas for digital interventions in population management and health studies are presented in this overview. Many concepts, implementation characteristics and recommendations have been raised which highlight the future role of these interventions to enhance public engagement and health equity.

The Use of Evaluation Panels During the Development of a Digital Intervention for Veterans Based on Cognitive Behavioral Therapy for Insomnia: Qualitative Evaluation Study

BACKGROUND

Individuals enrolling in the Veterans Health Administration frequently report symptoms consistent with insomnia disorder. Cognitive behavioral therapy for insomnia (CBT-I) is a gold standard treatment for insomnia disorder. While the Veterans Health Administration has successfully implemented a large dissemination effort to train providers in CBT-I, the limited number of trained CBT-I providers continues to restrict the number of individuals who can receive CBT-I. Digital mental health intervention adaptations of CBT-I have been found to have similar efficacy as traditional CBT-I. To help address the unmet need for insomnia disorder treatment, the VA commissioned the creation of a freely available, internet-delivered digital mental health intervention adaptation of CBT-I known as Path to Better Sleep (PTBS).

OBJECTIVE

We aimed to describe the use of evaluation panels composed of veterans and spouses of veterans during the development of PTBS. Specifically, we report on the methods used to conduct the panels, the feedback they provided on elements of the course relevant to user engagement, and how their feedback influenced the design and content of PTBS.

METHODS

A communications firm was contracted to recruit 3 veteran (n=27) and 2 spouse of veteran (n=18) panels and convene them for three 1-hour meetings. Members of the VA team identified key questions for the panels, and the communications firm prepared facilitator guides to elicit feedback on these key questions. The guides provided a script for facilitators to follow while convening the panels. The panels were telephonically conducted, with visual content displayed via remote presentation software. The communications firm prepared reports summarizing the panelists' feedback during each panel meeting. The qualitative feedback described in these reports served as the raw material for this study.

RESULTS

The panel members provided markedly consistent feedback on several elements of PTBS, including recommendations to emphasize the efficacy of CBT-I techniques; clarify and simplify written content as much as possible; and ensure that content is consistent with the lived experiences of veterans. Their feedback was congruent with previous studies on the factors influencing user engagement with digital mental health interventions. Panelist feedback influenced multiple course design decisions, including reducing the effort required to use the course's sleep diary function, making written content more concise, and selecting veteran testimonial videos that emphasized the benefits of treating chronic insomnia symptoms.

CONCLUSIONS

The veteran and spouse evaluation panels provided useful feedback during the design of PTBS. This feedback was used to make concrete revisions and design decisions consistent with existing research on improving user engagement with digital mental health interventions. We believe that many of the key feedback messages provided by these evaluation panels could prove useful to other digital mental health intervention designers.

Development of the key performance indicators for digital health interventions: A scoping review

Background

Digital health interventions offer new methods for delivering healthcare, with the potential to innovate healthcare services. Key performance indicators play a role in the evaluation, measurement, and improvement in healthcare quality and service performance. The aim of this scoping review was to identify current knowledge and evidence surrounding the development of key performance indicators for digital health interventions.

Methods

A literature search was conducted across ten key databases: AMED - The Allied and Complementary Medicine Database, CINAHL - Complete, Health Source: Nursing/Academic Edition, MEDLINE, APA PsycINFO, EMBASE, EBM Reviews - Cochrane Database of Systematic Reviews, EBM Reviews - Database of Abstracts of Reviews of Effects, EBM Reviews - Health Technology Assessment, and IEEE Xplore.

Results

Five references were eligible for the review. Two were articles on original research studies of a specific digital health intervention, and two were overviews of methods for developing digital health interventions (not specific to a single digital health intervention). All the included reports discussed the involvement of stakeholders in developing key performance indicators for digital health interventions. The step of identifying and defining the key performance indicators was completed using various methodologies, but all centred on a form of stakeholder involvement. Potential options for stakeholder involvement for key performance indicator identification include the use of an elicitation framework, a factorial survey approach, or a Delphi study.

Conclusions

Few articles were identified, highlighting a significant gap in evidence-based knowledge in this domain. All the included articles discussed the involvement of stakeholders in developing key performance indicators for digital health interventions, which were performed using various methodologies. The articles acknowledged a lack of literature related to key performance indicator development for digital health interventions. To allow comparability between key performance indicator initiatives and facilitate work in the field, further research would be beneficial to develop a common methodology for key performance indicators development for digital health interventions.

Evaluation of Digital Interventions for Physical Activity Promotion: Scoping Review

Background

Digital interventions are interventions supported by digital tools or technologies, such as mobile apps, wearables, or web-based software. Digital interventions in the context of public health are specifically designed to promote and improve health. Recent reviews have shown that many digital interventions target physical activity promotion; however, it is unclear how such digital interventions are evaluated.

Objective

We aimed to investigate evaluation strategies in the context of digital interventions for physical activity promotion using a scoping review of published reviews. We focused on the target (ie, user outcomes or tool performance), methods (ie, tool data or self-reported data), and theoretical frameworks of the evaluation strategies.

Methods

A protocol for this study was preregistered and published. From among 300 reviews published up to March 19, 2021 in Medline, PsycINFO, and CINAHL databases, 40 reviews (1 rapid, 9 scoping, and 30 systematic) were included in this scoping review. Two authors independently performed study selection and data coding. Consensus was reached by discussion. If applicable, data were coded quantitatively into predefined categories or qualitatively using definitions or author statements from the included reviews. Data were analyzed using either descriptive statistics, for quantitative data (relative frequencies out of all studies), or narrative synthesis focusing on common themes, for qualitative data.

Results

Most reviews that were included in our scoping review were published in the period from 2019 to 2021 and originated from Europe or Australia. Most primary studies cited in the reviews included adult populations in clinical or nonclinical settings, and focused on mobile apps or wearables for physical activity promotion. The evaluation target was a user outcome (efficacy, acceptability, usability, feasibility, or engagement) in 38 of the 40 reviews or tool performance in 24 of the 40 reviews. Evaluation methods relied upon objective tool data (in 35/40 reviews) or other data from self-reports or assessments (in 28/40 reviews). Evaluation frameworks based on behavior change theory, including goal setting, self-monitoring, feedback on behavior, and educational or motivational content, were mentioned in 22 out of 40 reviews. Behavior change theory was included in the development phases of digital interventions according to the findings of 20 out of 22 reviews.

Conclusions

The evaluation of digital interventions is a high priority according to the reviews included in this scoping review. Evaluations of digital interventions, including mobile apps or wearables for physical activity promotion, typically target user outcomes and rely upon objective tool data. Behavior change theory may provide useful guidance not only for development of digital interventions but also for the evaluation of user outcomes in the context of physical activity promotion. Future research should investigate factors that could improve the efficacy of digital interventions and the standardization of terminology and reporting in this field.

International Registered Report Identifier (IRRID)

RR2-10.2196/35332

Benefit Assessment and Reimbursement of Digital Health Applications: Concepts for Setting Up a New System for Public Coverage

In Germany, some digital health applications (DiHA) became reimbursable through the statutory health insurance system with the adoption of the Digital Healthcare Act in 2019. Approaches and concepts for the German care context were developed in an iterative process, based on existing concepts from international experience. A DiHA categorization was developed that could be used as a basis to enable the creation of a reimbursed DiHA repository, and to derive evidence requirements for coverage and reimbursement for each DiHA. The results provide an overview of a possible classification of DiHA as well as approaches to assessment and evaluation. The structure of remuneration and pricing in connection with the formation of groups is demonstrated.

A Digital Health Service for Elderly People with Balance Disorders and Risk of Falling: A Design Science Approach

In this study, a design science research methodology was used aiming at designing, implementing and evaluating a digital health service to complement the provision of healthcare for elderly people with balance disorders and risk of falling. An explanatory sequential mixed methods study allowed to identify and explore the dissatisfaction with electronic medical records and the opportunity for using digital health solutions. The suggested recommendations helped to elaborate and develop “BALANCE”, a digital service implemented on the METHIS platform, which was recently validated for remote monitoring of chronic patients in primary healthcare. “BALANCE” provides clinical and interactive data, questionnaire pre and post-balance rehabilitation, tutorial videos with balance exercises and patient-recorded videos of the exercises. This digital service was demonstrated, including five elderly patients with clinical recommendations for balance rehabilitation at home. Finally, the authors conducted two focus groups with the participants and their caregivers as well as with physicians. The focus groups aimed at exploring their satisfaction level, needs of adjustment in the “BALANCE” service and strategies for applicability. The digital healthcare service evaluation revealed a significant potential for clinical applicability of this digital solution for elderly people with balance disorders and risk of falling.