[Development, Evaluation and Implementation of Digital Health Interventions (Part 1) - Discussion Paper of the Digital Health Working Group of the German Network for Health Services Research (DNVF)]

Methods, Indicators, and End-User Involvement in the Evaluation of Digital Health Interventions for the Public: Scoping Review

BACKGROUND

Digital health interventions (DHIs) have the potential to enable public end users, such as citizens and patients, to manage and improve their health. Although the number of available DHIs is increasing, examples of successfully established DHIs in public health systems are limited. To counteract the nonuse of DHIs, they should be comprehensively evaluated while integrating end users. Unfortunately, there is a wide variability and heterogeneity according to the approaches of evaluation, which creates a methodological challenge.

OBJECTIVE

This scoping review aims to provide an overview of the current established processes for evaluating DHIs, including methods, indicators, and end-user involvement. The review is not limited to a specific medical field or type of DHI but offers a holistic overview.

METHODS

This scoping review was conducted following the JBI methodology for scoping reviews based on the framework by Arksey & O'Malley and complies with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines. Three scientific databases (PubMed, Scopus, and Science Direct) were searched in April 2023. English and German studies between 2008 and 2023 were considered when evaluating DHIs that explicitly address public end users. The process of study selection was carried out by several researchers to avoid reviewer bias.

RESULTS

The search strategy identified 9618 publications, of which 160 were included. Among these included articles, 200 evaluations were derived and analyzed. The results showed that there is neither a consensus on the methods to evaluate DHIs nor a commonly agreed definition or usage of the evaluated indicators, which results in a broad variety of evaluation practices. This aligns with observations of the existing literature. It was found that there is a lack of references to existing frameworks for the evaluation of DHIs. The majority of the included studies referred to user-centered approaches and involved end users in the evaluation process. As assistance for people developing and evaluating DHIs and as a basis for thinking about appropriate ways to evaluate DHIs, a results matrix was created where the findings were combined per DHI cluster. Additionally, general recommendations for the evaluators of DHIs were formulated.

CONCLUSIONS

The findings of this scoping review offer a holistic overview of the variety and heterogeneity according to the approaches of evaluation of DHIs for public end users. Evaluators of these DHIs should be encouraged to reference established frameworks or measurements for justification. This would ease the transferability of the results among similar evaluation studies within the digital health sector, thereby enhancing the coherence and comparability of research in this area.

Virtual reality roleplays for patients with depression: A user experience evaluation

Background

Virtual reality (VR) has been used successfully and effectively in psychotherapy for a variety of disorders. In the field of depression, there are only a few VR interventions and approaches. Although simple social interactions have been successfully modeled in VR for several mental disorders, there has been no transfer to the field of depression therapy. VR may be employed for psychodynamic psychotherapy to work on interpersonal conflict patterns. In this study, we developed and evaluated a VR intervention for the simulation of roleplay situations in the context of supportive-expressive therapy.

Methods

We conducted a clinical user experience (UX) study at a psychotherapeutic clinic in Düsseldorf, Germany. Eight inpatients with depression and four therapists were included. Semi-structured interviews and qualitative content analysis were used to identify UX issues of the developed VR intervention. Usability questionnaires and technical usage data were also considered. The VR intervention consisted of two therapist-controlled roleplay scenarios designed to support work on the core conflictual relationship theme by allowing patients to interact in typical problematic social situations. Recorded VR roleplays allow for therapeutic debriefing with a change of perspective. Therapists were given the option of using the roleplay in multiple sessions.

Results

All therapists conducted one session per patient with the VR intervention. From the patient interviews, 26 UX issues were extracted, of which one technical malfunction and two unclarities in the interaction with the VR agent were rated as major problems. From the therapist interviews, 14 UX issues were extracted, of which five were rated as major problems related to the interface in the dialog control or the complex system setup.

Conclusion

The main problem was designing a dialog structure that allows both complex conversational flows and a clear control interface. In principle, VR roleplays could be integrated well and safely into therapy. The VR intervention shows promise for providing an emotional experience of interpersonal conflict patterns in the context of psychotherapy. Additionally, other roleplay situations involving various social problem areas must be created and evaluated in terms of the fit to the patients' core conflictual relationship themes.

[The effectiveness of digital care applications in people with need of home care: A systematic review]

The effectiveness of digital care applications in people with need of home care: A systematic review Abstract: Background: The growing need for long-term care poses challenges for healthcare systems. In June 2021, digital care applications (DiPA) were introduced as a new service in the statutory long-term care insurance in Germany. Their aim is to counteract the increased care need and/or support independence in the home. Aim: This project systematically identified and reported on national and international evidence on DiPA effectiveness, as described by the Federal Institute for Drugs and Medical Devices (BfArM). Methods: The systematic review was conducted in accordance with the PRISMA statement by means of systematic searches in the databases Medline and Cinahl, and supplemented by an extensive manual search (08/2022). Study quality was assessed using the Risk of Bias Tool (RoB2). The results were synthesized narratively. Results: Eight randomized controlled studies were included; improvements in cognition, activities of daily living, and mobility in home care patients, were reported. Study quality was low to moderate, due to mostly unblinded study designs and low case numbers. Conclusions: The currently available evidence fails to provide a reliable basis for assessing the benefits of DiPA, to neither DiPA manufacturers, nor to responsible parties in the healthcare system. Further research on DiPA effectiveness, particularly with high-quality studies, are necessary to assess its potential in the German care sector.

[Digital Healthcare: Requirements for a Successful Transformation]

Digital transformation of healthcare is the dominating discussion topic for all healthcare stakeholders. Digital transformation encompasses all areas of healthcare and is far more than digital healthcare applications (DiGA), digital care applications (DiPA), telemedicine applications, telematics infrastructure activities, and applications from the fields of eHealth, mHealth, or Telehealth. All existing care processes and structures in the healthcare system are undergoing an inventory in order to transfer analog components of care into a digital context. The digital transformation is not taking place exclusively in economic sectors such as healthcare, but is a process of change throughout society in the collection, use, provision, linking and evaluation of information (=data). For the healthcare sector, it is clear that different technical concepts are used, while digital healthcare takes place in different places and at different times, may include different user (groups) and retains, expands or changes the healthcare context. Established healthcare functions such as diagnostics, therapy, documentation and the management of healthcare services are retained and transferred to a digital context. In addition, new application areas will emerge, such as the overarching access to health data by different actors, real-time-driven monitoring systems of holistic health data, (clinical) decision systems, or the provision of data for health services. Even if the majority of the digital transformation has not yet taken place, it is assumed that these functions and application areas of healthcare will serve to sustainably improve the quality of care and benefit the well-being of all (future) patients. For the transfer of existing processes into a digital context and the establishment of new application areas, there are prerequisites for healthcare institutions and the healthcare system itself.