Methods Used to Evaluate mHealth Applications for Cardiovascular Disease: A Quasi-Systematic Scoping Review

Challenges and Alternatives to Evaluation Methods and Regulation Approaches for Medical Apps as Mobile Medical Devices: International and Multidisciplinary Focus Group Discussion

BACKGROUND

The rapid proliferation of medical apps has transformed the health care landscape by giving patients and health care providers unprecedented access to personalized health information and services. However, concerns regarding the effectiveness and safety of medical apps have raised questions regarding the efficacy of randomized controlled trials (RCTs) in the evaluation of such apps and as a requirement for their regulation as mobile medical devices.

OBJECTIVE

This study aims to address this issue by investigating alternative methods, apart from RCTs, for evaluating and regulating medical apps.

METHODS

Using a qualitative approach, a focus group study with 46 international and multidisciplinary public health experts was conducted at the 17th World Congress on Public Health in May 2023 in Rome, Italy. The group was split into 3 subgroups to gather in-depth insights into alternative approaches for evaluating and regulating medical apps. We conducted a policy analysis on the current regulation of medical apps as mobile medical devices for the 4 most represented countries in the workshop: Italy, Germany, Canada, and Australia. We developed a logic model that combines the evaluation and regulation domains on the basis of these findings.

RESULTS

The focus group discussions explored the strengths and limitations of the current evaluation and regulation methods and identified potential alternatives that could enhance the quality and safety of medical apps. Although RCTs were only explicitly mentioned in the German regulatory system as one of many options, an analysis of chosen evaluation methods for German apps on prescription pointed toward a "scientific reflex" where RCTs are always the chosen evaluation method. However, this method has substantial limitations when used to evaluate digital interventions such as medical apps. Comparable results were observed during the focus group discussions, where participants expressed similar experiences with their own evaluation approaches. In addition, the participants highlighted numerous alternatives to RCTs. These alternatives can be used at different points during the life cycle of a digital intervention to assess its efficacy and potential harm to users.

CONCLUSIONS

It is crucial to recognize that unlike analog tools, digital interventions constantly evolve, posing challenges to inflexible evaluation methods such as RCTs. Potential risks include high dropout rates, decreased adherence, and nonsignificant results. However, existing regulations do not explicitly advocate for other evaluation methodologies. Our research highlighted the necessity of overcoming the gap between regulatory demands to demonstrate safety and efficacy of medical apps and evolving scientific practices, ensuring that digital health innovation is evaluated and regulated in a way that considers the unique characteristics of mobile medical devices.

Methods, Indicators, and End-User Involvement in the Evaluation of Digital Health Interventions for the Public: Scoping Review

BACKGROUND

Digital health interventions (DHIs) have the potential to enable public end users, such as citizens and patients, to manage and improve their health. Although the number of available DHIs is increasing, examples of successfully established DHIs in public health systems are limited. To counteract the nonuse of DHIs, they should be comprehensively evaluated while integrating end users. Unfortunately, there is a wide variability and heterogeneity according to the approaches of evaluation, which creates a methodological challenge.

OBJECTIVE

This scoping review aims to provide an overview of the current established processes for evaluating DHIs, including methods, indicators, and end-user involvement. The review is not limited to a specific medical field or type of DHI but offers a holistic overview.

METHODS

This scoping review was conducted following the JBI methodology for scoping reviews based on the framework by Arksey & O'Malley and complies with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines. Three scientific databases (PubMed, Scopus, and Science Direct) were searched in April 2023. English and German studies between 2008 and 2023 were considered when evaluating DHIs that explicitly address public end users. The process of study selection was carried out by several researchers to avoid reviewer bias.

RESULTS

The search strategy identified 9618 publications, of which 160 were included. Among these included articles, 200 evaluations were derived and analyzed. The results showed that there is neither a consensus on the methods to evaluate DHIs nor a commonly agreed definition or usage of the evaluated indicators, which results in a broad variety of evaluation practices. This aligns with observations of the existing literature. It was found that there is a lack of references to existing frameworks for the evaluation of DHIs. The majority of the included studies referred to user-centered approaches and involved end users in the evaluation process. As assistance for people developing and evaluating DHIs and as a basis for thinking about appropriate ways to evaluate DHIs, a results matrix was created where the findings were combined per DHI cluster. Additionally, general recommendations for the evaluators of DHIs were formulated.

CONCLUSIONS

The findings of this scoping review offer a holistic overview of the variety and heterogeneity according to the approaches of evaluation of DHIs for public end users. Evaluators of these DHIs should be encouraged to reference established frameworks or measurements for justification. This would ease the transferability of the results among similar evaluation studies within the digital health sector, thereby enhancing the coherence and comparability of research in this area.

Mobile Apps for COVID-19: A Systematic Review of Reviews

BACKGROUND

One measure national governments took to react to the acute respiratory syndrome coronavirus type 2 (SARS-CoV-2) pandemic was mobile applications (apps). This study aims to provide a high-level overview of published reviews of mobile apps used in association with coronavirus disease 19 (COVID-19), examine factors that contributed to the success of these apps, and provide data for further research into this topic.

METHODS

We conducted a systematic review of reviews (also referred to as an umbrella review) and searched two databases, Medline and Embase, for peer-reviewed reviews of COVID-19 mobile apps that were written in English and published between January 1st 2020 and April 25th 2022.

RESULTS

Out of the initial 17,611 studies, 24 studies were eligible for the analysis. Publication dates ranged from May 2020 to January 2022. In total, 54% (n = 13) of the studies were published in 2021, and 33% (n = 8) were published in 2020. Most reviews included in our review of reviews analyzed apps from the USA, the UK, and India. Apps from most of the African and Middle and South American countries were not analyzed in the reviews included in our study. Categorization resulted in four clusters (app overview, privacy and security, MARS rating, and miscellaneous).

CONCLUSIONS

Our study provides a high-level overview of 24 reviews of apps for COVID-19, identifies factors that contributed to the success of these apps, and identifies a gap in the current literature. The study provides data for further analyses and further research.

Examining standardized tools used for the evaluation of mobile health applications for cardiovascular disease

Cardiovascular disease is one of the leading causes of death worldwide. Scarce resources and rising costs are pushing healthcare systems to their limits. There is an urgency to develop, optimize and evaluate technologies that provide more effective care for patients. Modern technologies, such as mobile health (mHealth) applications, can provide relief as a key strategy. To integrate digital interventions into care structures, a detailed impact assessment of all professional mHealth applications is needed. The aim of this study is to analyze the standardized tools used in the field of cardiovascular disease. The results show that questionnaires, usage logs, and key indicators are predominantly used. Although the identified mHealth interventions are specific to cardiovascular disease and thus require particular questions to evaluate apps, the user readiness, usability, or quality of life criteria are non-specific. Therefore, the results contribute to understanding how different mHealth interventions can be assessed, categorized, evaluated, and accepted.

Mobile Access to Medical Records in Heart Transplantation Aftercare: Mixed-Methods Study Assessing Usability, Feasibility and Effects of a Mobile Application

BACKGROUND

Patient access to medical records can improve quality of care. The phellow application (app) was developed to provide patients access to selected content of their medical record. It was tested at a heart transplantation (HTx) outpatient clinic. The aims of this study were (1) to assess usability of phellow, (2) to determine feasibility of implementation in routine care, and (3) to study the effects app use had on patients' self-management.

METHODS

Usability was measured quantitatively through the System Usability Scale (SUS). Furthermore, usability, feasibility, and effects on self-management were qualitatively assessed through interviews with users, non-users, and health care providers.

RESULTS

The SUS rating (n = 31) was 79.9, indicating good usability. Twenty-three interviews were conducted. Although appreciation and willingness-to-use were high, usability problems such as incompleteness of record, technical issues, and complex registration procedures were reported. Improved technical support infrastructure, clearly defined responsibilities, and app-specific trainings were suggested for further implementation. Patients described positive effects on their self-management.

CONCLUSIONS

To be feasible for implementation in routine care, usability problems should be addressed. Feedback on the effect of app use was encouraging. Accompanying research is crucial to monitor usability improvements and to further assess effects of app use on patients.

Evaluation Methods Applied to Digital Health Interventions: What Is Being Used beyond Randomised Controlled Trials?-A Scoping Review

Despite the potential of digital health interventions (DHIs), evaluations of their effectiveness face challenges. DHIs are complex interventions and currently established evaluation methods, e.g., the randomised controlled trial (RCT), are limited in their application. This study aimed at identifying alternatives to RCTs as potentially more appropriate evaluation approaches. A scoping review was conducted to provide an overview of existing evaluation methods of DHIs beyond the RCT. Cochrane Central Register of Controlled Trials, MEDLINE, Web of Science, and EMBASE were screened in May 2021 to identify relevant publications, while using defined inclusion and exclusion criteria. Eight studies were extracted for a synthesis comprising four alternative evaluation designs. Factorial designs were mostly used to evaluate DHIs followed by stepped-wedge designs, sequential multiple assignment randomised trials (SMARTs), and micro randomised trials (MRTs). Some of these methods allow for the adaptation of interventions (e.g., SMART or MRT) and the evaluation of specific components of interventions (e.g., factorial designs). Thus, they are appropriate for addressing some specific needs in the evaluation of DHIs. However, it remains unsolved how to establish these alternative evaluation designs in research practice and how to deal with the limitations of the designs.