Scoping review: Development and assessment of evaluation frameworks of mobile health apps for recommendations to consumers

Adolescent-Centered mHealth Applications in a Collaborative Care Model: A Virtual Focus Group Study With Audiologists

PURPOSE

Technology-enabled care, including the use of mobile health (mHealth), is emerging as a viable hearing health care delivery method. While the integration of mHealth with adult populations currently supports a wide array of hearing services, a better understanding of the implementation across the lifespan is needed. Literature surrounding the unique population of adolescent hearing aid users is currently lacking. Research is needed to highlight factors important to the use and clinical integration of mHealth hearing aid applications (apps) with adolescents. This study explored two primary objectives: (a) audiologists' perceptions around the use of mHealth apps to enable collaborative, child-inclusive hearing aid personalization, and (b) person-centered ideation around potential app design components to benefit users aged 12 to 17 years.

METHOD

Twelve audiologists participated in virtual synchronous focus groups, across three group sessions using Cisco Webex. Sessions were recorded, transcribed, and analyzed using an inductive, codebook thematic analysis approach.

RESULTS

Six main themes resulted from group discussion analyses: (a) client candidacy: characteristics impacting suitability for mHealth use; (b) clinical implementation: organizational, professional, or patient-level strategies for mHealth adoption; (c) collaboration: the use of two or more individuals working together; (d) empowerment: process of acquiring and using knowledge, skills, and strategies; (e) remote technology: technologies enabling remote hearing aid personalization; and (f) application functionality and design: features and characteristics important to an adolescent-focused app.

CONCLUSIONS

Findings identified the potential for clinical integration of hearing aid apps with adolescents in a collaborative care model, including consideration of child-specific use patterns, outcomes, and key design and technology components to support real-world implementation and use. Results may guide development and tailoring efforts around existing and future hearing aid apps for use with adolescent populations.

An evaluation and refinement of the "Hep B Story" app, tailored to meet the community's cultural needs

BACKGROUND

Hepatitis B is endemic amongst the Australian Aboriginal population in the Northern Territory. A participatory action research project identified the lack of culturally appropriate education tools and led to the development of the "Hep B Story" app in the Aboriginal language Yolŋu Matha. This paper describes a formal evaluation of the app's first version, which informed improvements and translation into a further ten Aboriginal languages.

METHODS

The evaluation employed Participatory Action Research (PAR) principles to work within Indigenous research methodologies and prioritise Indigenous knowledge to improve the app iteratively. Semi-structured interviews and focus groups were conducted across the Northern Territory with 11 different language groups. Local Community Based Researchers and Aboriginal Research team members coordinated sessions. The recorded, translated conversations were transcribed verbatim and thematically analysed using an inductive and deductive approach.

RESULTS

Between November 2018 and September 2020, 94 individuals from 11 language groups participated in 25 semi-structured interviews and 10 focus groups. All participants identified as Aboriginal. Most participants felt the app would be culturally appropriate for Aboriginal communities in the Northern Territory and improve knowledge surrounding hepatitis B. The information gathered from these interviews allowed for identifying five main themes: support for app, relationships, concept versus language, shame, and perceptions of images, along with errors that required modification.

CONCLUSIONS

A "real-life" evaluation of the app was comprehensively completed using a PAR approach blended with Indigenous research methods. This evaluation allowed us to develop an updated and enhanced version of the app before creating the additional ten language versions. An iterative approach alongside strong community engagement was pivotal in ensuring the app's cultural safety and appropriateness. We recommend avoiding the use of knowledge-based evaluations in an Aboriginal setting to ensure relevant and culturally appropriate feedback is obtained.

An assessment of ten popular pregnancy applications (Apps) available for women in Australia

OBJECTIVE

Given the growing popularity of health Apps, this study aimed to evaluate popular pregnancy Apps among Australian women.

DESIGN

Ten popular pregnancy mobile device Apps accessible within Australia were assessed using the Deakin Health E-technologies Assessment Lab (HEAL) framework, the Australian Privacy Principles (APP) and other context-specific criteria.

RESULTS

Most Apps were robust in use and user-friendly in terms of design. Based on the HEAL assessment, all the Apps scored between 70 and 89 out of 120. Only two of the ten Apps were locally hosted and operated. Only one App could search local health services using Victorian postcodes. All ten Apps were aligned with various privacy policy guidelines, but only one was reported to be fully aligned with the APP as it is locally hosted and operated. None of the Apps had accessibility features such as for visual or hearing difficulties. Only one App could be assessed as able to interface with the electronic medical record system in Australia due to a lack of publicly available information on this capability.

CONCLUSIONS

Due to privacy concerns, pregnancy Apps accessible in Australia should be used with caution unless care is taken to select an App that fully complies with local requirements or international equivalents. All Apps were subjectively rated as moderate in quality, and more attention to accessibility and privacy features by App developers is recommended, along with the focus on integration with Australian digital health services.

Don't judge a book or health app by its cover: User ratings and downloads are not linked to quality

OBJECTIVE

To analyse the relationship between health app quality with user ratings and the number of downloads of corresponding health apps.

MATERIALS AND METHODS

Utilising a dataset of 881 Android-based health apps, assessed via the 300-point objective Organisation for the Review of Care and Health Applications (ORCHA) assessment tool, we explored whether subjective user-level indicators of quality (user ratings and downloads) correlate with objective quality scores in the domains of user experience, data privacy and professional/clinical assurance. For this purpose, we applied spearman correlation and multiple linear regression models.

RESULTS

For user experience, professional/clinical assurance and data privacy scores, all models had very low adjusted R squared values (< .02). Suggesting that there is no meaningful link between subjective user ratings or the number of health app downloads and objective quality measures. Spearman correlations suggested that prior downloads only had a very weak positive correlation with user experience scores (Spearman = .084, p = .012) and data privacy scores (Spearman = .088, p = .009). There was a very weak negative correlation between downloads and professional/clinical assurance score (Spearman = -.081, p = .016). Additionally, user ratings demonstrated a very weak correlation with no statistically significant correlations observed between user ratings and the scores (all p > 0.05). For ORCHA scores multiple linear regression had adjusted R-squared = -.002.

CONCLUSION

This study highlights that widely available proxies which users may perceive to signify the quality of health apps, namely user ratings and downloads, are inaccurate predictors for estimating quality. This indicates the need for wider use of quality assurance methodologies which can accurately determine the quality, safety, and compliance of health apps. Findings suggest more should be done to enable users to recognise high-quality health apps, including digital health literacy training and the provision of nationally endorsed "libraries".

Old age or cognitive decline? Examining the usability of a mobile health app for older Australians

There is a growing literature on the role of mobile health applications (mHealth apps) in supporting older adults and the self-management of personal health. The purpose of this pilot study is to assess the usability of a government-funded mobile health app amongst older Australians and to evaluate whether cognitive function and demographic characteristics (i.e. age, gender, education) are associated with usability. A total of 28 older adults living in a regional city in Australia took part in the study. The participants were recruited using purposive sampling. Data collection instruments consisted of validated cognitive tests, task-based usability tests, and a questionnaire. The data was analyzed using non-parametric strategies. The findings of this study demonstrated that a government-funded, mHealth app was usable by older adults. Users were able to perform basic tasks in an effective and efficient manner. The hypothesis that elderly age would be significantly associated with performance on cognitive tests, as well as usability, was not supported. Performance on some cognitive tests was significantly and positively related to usability. Education and gender were not related to usability. The results suggest that traditional stereotypes surrounding aging and cognitive decline need to be reexamined.

Heuristics used for evaluating the usability of mobile health applications: A systematic literature review

Objective

Mobile health applications hold immense potential for enhancing health outcomes. Usability is one of the main factors for the adoption and use of mobile health applications. However, despite the growing importance of mHealth applications, clear standards for their evaluation remain elusive. The present study aimed to determine heuristics for the usability evaluation of health-related applications.

Methods

We systematically searched multiple databases for relevant papers published between January 2008 and April 2021. Articles were reviewed, and data were extracted and categorized from those meeting inclusion criteria by two authors independently. Heuristics were identified based on statements, words, and concepts expressed in the studies. These heuristics were first mapped to Nielsen's heuristics based on their differences or similarities. The remaining heuristics that were very important for mobile applications were categorized into new heuristics.

Results

Seventeen studies met the eligibility criteria. Seventy-nine heuristics were extracted from the papers. After combining the items with the same concepts and removing irrelevant items based on the exclusion criteria, 20 heuristics remained. Common heuristics such as "Visibility of system status" and "Flexibility and efficiency of use" were categorized into 10 previously established heuristics and new heuristics like "Navigation" and "User engagement" were recognized as new ones.

Conclusions

In our study, we have meticulously identified 20 heuristics that hold promise for evaluating and designing mHealth applications. These heuristics can be used by the researchers for the development of robust tools for heuristic evaluation. These tools, when adapted or tailored for health domain applications, have the potential to significantly enhance the quality of mHealth applications. Ultimately, this improvement in quality translates to enhanced patient safety.

Protocol Registration

(10.17605/OSF.IO/PZJ7H).

Quality of Digital Health Interventions Across Different Health Care Domains: Secondary Data Analysis Study

BACKGROUND

There are more than 350,000 digital health interventions (DHIs) in the app stores. To ensure that they are effective and safe to use, they should be assessed for compliance with best practice standards.

OBJECTIVE

The objective of this paper was to examine and compare the compliance of DHIs with best practice standards and adherence to user experience (UX), professional and clinical assurance (PCA), and data privacy (DP).

METHODS

We collected assessment data from 1574 DHIs using the Organisation for the Review of Care and Health Apps Baseline Review (OBR) assessment tool. As part of the assessment, each DHI received a score out of 100 for each of the abovementioned areas (ie, UX, PCA, and DP). These 3 OBR scores are combined to make up the overall ORCHA score (a proxy for quality). Inferential statistics, probability distributions, Kruskal-Wallis, Wilcoxon rank sum test, Cliff delta, and Dunn tests were used to conduct the data analysis.

RESULTS

We found that 57.3% (902/1574) of the DHIs had an Organisation for the Review of Care and Health Apps (ORCHA) score below the threshold of 65. The overall median OBR score (ORCHA score) for all DHIs was 61.5 (IQR 51.0-73.0) out of 100. A total of 46.2% (12/26) of DHI's health care domains had a median equal to or above the ORCHA threshold score of 65. For the 3 assessment areas (UX, DP, and PCA), DHIs scored the highest for the UX assessment 75.2 (IQR 70.0-79.6), followed by DP 65.1 (IQR 55.0-73.4) and PCA 49.6 (IQR 31.9-76.1). UX scores had the least variance (SD 13.9), while PCA scores had the most (SD 24.8). Respiratory and urology DHIs were consistently highly ranked in the National Institute for Health and Care Excellence Evidence Standards Framework tiers B and C based on their ORCHA score.

CONCLUSIONS

There is a high level of variability in the ORCHA scores of DHIs across different health care domains. This suggests that there is an urgent need to improve compliance with best practices in some health care areas. Possible explanations for the observed differences might include varied market maturity and commercial interests within the different health care domains. More investment to support the development of higher-quality DHIs in areas such as ophthalmology, allergy, women's health, sexual health, and dental care may be needed.

International practices in health technology assessment and public financing of digital health technologies: recommendations for Hungary

Background

Evaluating and integrating digital health technologies is a critical component of a national healthcare ecosystem in the 2020s and is expected to even increase in significance.

Design

The paper gives an overview of international practices on public financing and health technology assessment of digital health technologies (DHTs) in five European Union (EU) countries and outlines recommendations for country-level action that relevant stakeholders can consider in order to support uptake of digital health solutions in Hungary. A scoping review was carried out to identify and gather country-specific classifications and international practices on the financing DHTs in five pioneering EU countries: Germany, France, Belgium, the United Kingdom and Finland.

Results

Several frameworks have been developed for DHTs, however there is no single, unified framework or method for classification, evaluation, and financing of digital health technologies in European context. European countries apply different taxonomy, use different assessment domains and regulations for the reimbursement of DHTs. The Working Group of the Hungarian Health Economic Society recommends eight specific points for stakeholders, importantly taking active role in shaping common clinical evidence standards and technical quality criteria across in order for common standards to be developed in the European Union single market.

Conclusion

Specificities of national healthcare contexts must be taken into account in decisions to allocate public funds to certain therapies rather than others.

Rigorous and rapid evidence assessment in digital health with the evidence DEFINED framework

Dozens of frameworks have been proposed to assess evidence for digital health interventions (DHIs), but existing frameworks may not facilitate DHI evidence reviews that meet the needs of stakeholder organizations including payers, health systems, trade organizations, and others. These organizations may benefit from a DHI assessment framework that is both rigorous and rapid. Here we propose a framework to assess Evidence in Digital health for EFfectiveness of INterventions with Evaluative Depth (Evidence DEFINED). Designed for real-world use, the Evidence DEFINED Quick Start Guide may help streamline DHI assessment. A checklist is provided summarizing high-priority evidence considerations in digital health. Evidence-to-recommendation guidelines are proposed, specifying degrees of adoption that may be appropriate for a range of evidence quality levels. Evidence DEFINED differs from prior frameworks in its inclusion of unique elements designed for rigor and speed. Rigor is increased by addressing three gaps in prior frameworks. First, prior frameworks are not adapted adequately to address evidence considerations that are unique to digital health. Second, prior frameworks do not specify evidence quality criteria requiring increased vigilance for DHIs in the current regulatory context. Third, extant frameworks rarely leverage established, robust methodologies that were developed for non-digital interventions. Speed is achieved in the Evidence DEFINED Framework through screening optimization and deprioritization of steps that may have limited value. The primary goals of Evidence DEFINED are to a) facilitate standardized, rapid, rigorous DHI evidence assessment in organizations and b) guide digital health solutions providers who wish to generate evidence that drives DHI adoption.

Methods for Navigating the Mobile Mental Health App Landscape for Clinical Use

There are over 10,000 mental health and wellness apps on the market. Apps offer the opportunity to increase access to mental health care. However, with many apps to choose from and an app landscape that is largely unregulated, it can be difficult to incorporate this technology into clinical practice. The first step towards achieving this goal is to identify clinically relevant and appropriate apps. The purpose of this review is to discuss app evaluation, raise awareness of considerations involved in implementing mental health apps into clinical care, and provide an example of how apps can be used effectively in the clinical space. We discuss the current regulatory environment for health apps, how to evaluate apps, and implement them into clinical practice. We also showcase a digital clinic where apps are integrated into the clinical workflow and discuss barriers to app implementation. Mental health apps have the potential to increase access to care if they are clinically effective, easy to use, and protect patient privacy. Learning how to find, evaluate, and implement quality apps into practice is key in harnessing this technology for the benefit of patients.

Framework for Guiding the Development of High-Quality Conversational Agents in Healthcare

Evaluating conversational agents (CAs) that are supposed to be applied in healthcare settings and ensuring their quality is essential to avoid patient harm and ensure efficacy of the CA-delivered intervention. However, a guideline for a standardized quality assessment of health CAs is still missing. The objective of this work is to describe a framework that provides guidance for development and evaluation of health CAs. In previous work, consensus on categories for evaluating health CAs has been found. In this work, we identify concrete metrics, heuristics, and checklists for these evaluation categories to form a framework. We focus on a specific type of health CA, namely rule-based systems that are based on written input and output, have a simple personality without any kind of embodiment. First, we identified relevant metrics, heuristics, and checklists to be linked to the evaluation categories through a literature search. Second, five experts judged the metrics regarding their relevance to be considered within evaluation and development of health CAs. The final framework considers nine aspects from a general perspective, five aspects from a response understanding perspective, one aspect from a response generation perspective, and three aspects from an aesthetics perspective. Existing tools and heuristics specifically designed for evaluating CAs were linked to these evaluation aspects (e.g., Bot usability scale, design heuristics for CAs); tools related to mHealth evaluation were adapted when necessary (e.g., aspects from the ISO technical specification for mHealth Apps). The resulting framework comprises aspects to be considered not only as part of a system evaluation, but already during the development. In particular, aspects related to accessibility or security have to be addressed in the design phase (e.g., which input and output options are provided to ensure accessibility?) and have to be verified after the implementation phase. As a next step, transfer of the framework to other types of health CAs has to be studied. The framework has to be validated by applying it during health CA design and development.

Clinical Considerations for Digital Resources in Care for Patients With Suicidal Ideation

Smartphone apps offer accessible new tools that may help prevent suicide and that offer support for individuals with active suicidal ideation. Numerous smartphone apps for mental health conditions exist; however, their functionality is limited, and evidence is nascent. A new generation of apps using smartphone sensors and integrating real-time data on evolving risk offers the potential of more personalized support, but these apps present ethical risks and currently remain more in the research domain than in the clinical domain. Nevertheless, clinicians can use apps to benefit patients. This article outlines practical strategies to select safe and effective apps for the creation of a digital toolkit that can augment suicide prevention and safety plans. By creating a unique digital toolkit for each patient, clinicians can help ensure that the apps selected will be most relevant, engaging, and effective.

A Web-Based Stratified Stepped Care Platform for Mental Well-being (TourHeart+): User-Centered Research and Design

BACKGROUND

Internet-based mental health interventions have been demonstrated to be effective in alleviating psychological distress and promoting mental well-being. However, real-world uptake and engagement of such interventions have been low. Rather than being stand-alone interventions, situating internet-based interventions under a stratified stepped care system can support users to continue with mental health practice and monitor their mental health status for timely services that are commensurate with their needs. A user-centered approach should be used in the development of such web-based platforms to understand the facilitators and barriers in user engagement to enhance platform uptake, usability, and adherence so it can support the users' continued adoption and practice of self-care for their mental health.

OBJECTIVE

The aim of this study was to describe the design process taken to develop a web-based stratified stepped care mental health platform, TourHeart+, using a user-centered approach that gathers target users' perceptions on mental self-care and feedback on the platform design and incorporates them into the design.

METHODS

The process involved a design workshop with the interdisciplinary development team, user interviews, and 2 usability testing sessions on the flow of registration and mental health assessment and the web-based self-help interventions of the platform. The data collected were summarized as descriptive statistics if appropriate and insights are extracted inductively. Qualitative data were extracted using a thematic coding approach.

RESULTS

In the design workshop, the team generated empathy maps and point-of-view statements related to the possible mental health needs of target users. Four user personas and related processes in the mental health self-care journey were developed based on user interviews. Design considerations were derived based on the insights drawn from the personas and mental health self-care journey. Survey results from 104 users during usability testing showed that the overall experience during registration and mental health assessment was friendly, and they felt cared for, although no statistically significant differences on preference ratings were found between using a web-based questionnaire tool and through an interactive chatbot, except that chatbot format was deemed more interesting. Facilitators of and barriers to registering the platform and completing the mental health assessment were identified through user feedback during simulation with mock-ups. In the usability testing for guided self-help interventions, users expressed pain points in course adherence, and corresponding amendments were made in the flow and design of the web-based courses.

CONCLUSIONS

The design process and findings presented in the study are important in developing a user-centric platform to optimize users' acceptance and usability of a web-based stratified stepped care platform with guided self-help interventions for mental well-being. Accounting for users' perceptions and needs toward mental health self-care and their experiences in the design process can enhance the usability of an evidence-based mental health platform on the web.

Adaptation and Utilization of a Postmarket Evaluation Model for Digital Contact Tracing Mobile Health Tools in the United States: Observational Cross-sectional Study

BACKGROUND

Case investigation and contact tracing are core public health activities used to interrupt disease transmission. These activities are traditionally conducted manually. During periods of high COVID-19 incidence, US health departments were unable to scale up case management staff to deliver effective and timely contact-tracing services. In response, digital contact tracing (DCT) apps for mobile phones were introduced to automate these activities. DCT apps detect when other DCT users are close enough to transmit COVID-19 and enable alerts to notify users of potential disease exposure. These apps were deployed quickly during the pandemic without an opportunity to conduct experiments to determine effectiveness. However, it is unclear whether these apps can effectively supplement understaffed manual contact tracers.

OBJECTIVE

The aims of this study were to (1) evaluate the effectiveness of COVID-19 DCT apps deployed in the United States during the COVID-19 pandemic and (2) determine if there is sufficient DCT adoption and interest in adoption to meet a minimum population use rate to be effective (56%). To assess uptake, interest and safe use covariates were derived from evaluating DCTs using the American Psychological Association App Evaluation Model (AEM) framework.

METHODS

We analyzed data from a nationally representative survey of US adults about their COVID-19-related behaviors and experiences. Survey respondents were divided into three segments: those who adopted a DCT app, those who are interested but did not adopt, and those not interested. Descriptive statistics were used to characterize factors of the three groups. Multivariable logistic regression models were used to analyze the characteristics of segments adopting and interested in DCT apps against AEM framework covariates.

RESULTS

An insufficient percentage of the population adopted or was interested in DCTs to achieve our minimum national target effectiveness rate (56%). A total of 17.4% (n=490) of the study population reported adopting a DCT app, 24.7% (n=697) reported interest, and 58.0% (n=1637) were not interested. Younger, high-income, and uninsured individuals were more likely to adopt a DCT app. In contrast, people in fair to poor health were interested in DCT apps but did not adopt them. App adoption was positively associated with visiting friends and family outside the home (odds ratio [OR] 1.63, 95% CI 1.28-2.09), not wearing masks (OR 0.52, 95% CI 0.38-0.71), and adopters thinking they have or had COVID-19 (OR 1.60, 95% CI 1.21-2.12).

CONCLUSIONS

Overall, a small percentage of the population adopted DCT apps. These apps may not be effective in protecting adopters' friends and family from their maskless contacts outside the home given low adoption rates. The public health community should account for safe use behavioral factors in future public health contact-tracing app design. The AEM framework was useful in developing a study design to evaluate DCT effectiveness and safety.

Safety considerations for assessing the quality of apps used during pregnancy: A scoping review

Objective

Pregnant women are increasingly turning to apps targeting knowledge and behaviour change for supporting healthy lifestyles and managing medical conditions. Yet, there is growing concern over the credibility and safety of content within mobile health (mHealth) apps. This scoping review aimed to systematically and thematically consolidate safety considerations described in reviews evaluating pregnancy-specific apps.

Methods

PubMed, Ovid MEDLINE® and EPub, CINAHL, Web of Science, Cochrane Libraries, and SCOPUS were systematically searched to identify reviews that assessed apps targeting pregnant women. Data related to safety were extracted and thematically analysed to establish a set of relevant safety considerations.

Results

Sixteen reviews met the inclusion criteria. The included reviews assessed an average of 27 apps each and targeted pregnancy topics, such as nutrition and physical activity. Five major and 20 minor themes were identified, including information, transparency, credibility, privacy and security, and app tailoring. Information, transparency, and credibility relate to the evidence base of information within the app, privacy and security of apps relate to the protection of personal information and data, and app tailoring relates to the consideration of contextual factors, such as local guidelines and digital health literacy.

Conclusions

Results present possible safety considerations when evaluating pregnancy-specific apps and emphasise a clear need for consumer guidance on how to make informed decisions around engagement and use of mHealth apps during pregnancy.

Current challenges for evaluating mobile health applications

As mobile health applications continue to proliferate without clear regulation, the need for app evaluation frameworks to offer guidance to patients and clinicians also expands. However, this expanding number of app evaluation frameworks itself can be a source of confusion and often contradictory recommendations. In pursuit of better frameworks that offer innovation for app evaluation, we present 4 challenges that app evaluation frameworks must overcome as well as examples from our own experience toward overcoming them. The recommendations are applicable to all health apps from any field of medicine, although we use examples from mental health as they are illustrative.

Digital health applications and the fast-track pathway to public health coverage in Germany: challenges and opportunities based on first results

Objectives

Evidence-based decision-making is the sine qua non for safe and effective patient care and the long-term functioning of health systems. Since 2020 Digital Health Applications (DiHA) in Germany have been undergoing a systematic pathway to be reimbursed by statutory health insurance (SHI) which is attracting attention in other European countries. We therefore investigate coverage decisions on DiHA and the underlying evidence on health care effects, which legally include both medical outcomes and patient-centred structural and procedural outcomes.

Methods

Based on publicly available data of the Institute for Medicines and Medical Devices searched between 08/2021 and 02/2022, all DiHA listed in the corresponding registry and thus reimbursable by the SHI were systematically investigated and presented descriptively on the basis of predefined criteria, such as clinical condition, and costs. The clinical trials on DiHA permanently included in the registry were reviewed with regard to their study design, endpoints investigated, the survey instruments used, and whether an intention-to-treat analysis was performed. Risk of bias was assessed using the ROB II tool.

Results

By February 2022, 30 DiHA had been included in the DiHA registry, one third of them permanently and two thirds conditionally. Most DiHA were therapeutic applications for mental illness based on cognitive behavioural therapy. For all permanently included DiHA, randomised controlled trials were conducted to demonstrate the impact on health care effects. While medical outcomes were investigated for all of these DiHA, patient-centred structural and procedural outcomes were rarely investigated. The majority of clinical trials showed a high risk of bias, mainly due to insufficient reporting quality. Overall, the prices for DiHA covered by SHI are on average around € 150 per month (min. € 40; max. € 248).

Conclusions

Evidence-based decision-making on coverage of DiHA leaves room for improvements both in terms of reporting-quality and the use of patient-centred structural and procedural outcomes in addition to medical outcomes. With appropriate evidence, DiHA can offer an opportunity as an adjunct to existing therapy while currently the high risk of bias of the trials raises doubts about the justification of its high costs.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-022-08500-6.

Consumers’ Willingness to Pay for eHealth and Its Influencing Factors: Systematic Review and Meta-analysis

Background

Despite the great potential of eHealth, substantial costs are involved in its implementation, and it is essential to know whether these costs can be justified by its benefits. Such needs have led to an increased interest in measuring the benefits of eHealth, especially using the willingness to pay (WTP) metric as an accurate proxy for consumers’ perceived benefits of eHealth. This offered us an opportunity to systematically review and synthesize evidence from the literature to better understand WTP for eHealth and its influencing factors.

Objective

This study aimed to provide a systematic review of WTP for eHealth and its influencing factors.

Methods

This study was performed and reported as per the Cochrane Collaboration and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. PubMed, CINAHL Plus, Cochrane Library, EconLit, and PsycINFO databases were searched from their inception to April 19, 2022. We conducted random-effects meta-analyses to calculate WTP values for eHealth (at 2021 US dollar rates) and meta-regression analyses to examine the factors affecting WTP.

Results

A total of 30 articles representing 35 studies were included in the review. We found that WTP for eHealth varied across studies; when expressed as a 1-time payment, it ranged from US $0.88 to US $191.84, and when expressed as a monthly payment, it ranged from US $5.25 to US $45.64. Meta-regression analyses showed that WTP for eHealth was negatively associated with the percentages of women (β=−.76; P<.001) and positively associated with the percentages of college-educated respondents (β=.63; P<.001) and a country’s gross domestic product per capita (multiples of US $1000; β=.03; P<.001). Compared with eHealth provided through websites, people reported a lower WTP for eHealth provided through asynchronous communication (β=−1.43; P<.001) and a higher WTP for eHealth provided through medical devices (β=.66; P<.001), health apps (β=.25; P=.01), and synchronous communication (β=.58; P<.001). As for the methods used to measure WTP, single-bounded dichotomous choice (β=2.13; P<.001), double-bounded dichotomous choice (β=2.20; P<.001), and payment scale (β=1.11; P<.001) were shown to obtain higher WTP values than the open-ended format. Compared with ex ante evaluations, ex post evaluations were shown to obtain lower WTP values (β=−.37; P<.001).

Conclusions

WTP for eHealth varied significantly depending on the study population, modality used to provide eHealth, and methods used to measure it. WTP for eHealth was lower among certain population segments, suggesting that these segments may be at a disadvantage in terms of accessing and benefiting from eHealth. We also identified the modalities of eHealth that were highly valued by consumers and offered suggestions for the design of eHealth interventions. In addition, we found that different methods of measuring WTP led to significantly different WTP estimates, highlighting the need to undertake further methodological explorations of approaches to elicit WTP values.

Recommendations for developing a lifecycle, multidimensional assessment framework for mobile medical apps

Digital health and mobile medical apps (MMAs) have shown great promise in transforming health care, but their adoption in clinical care has been unsatisfactory, and regulatory guidance and coverage decisions have been lacking or incomplete. A multidimensional assessment framework for regulatory, policymaking, health technology assessment, and coverage purposes based on the MMA lifecycle is needed. A targeted review of relevant policy documents from international sources was conducted to map current MMA assessment frameworks, to formulate 10 recommendations, subsequently shared amongst an expert panel of key stakeholders. Recommendations go beyond economic dimensions such as cost and economic evaluation and also include MMA development and update, classification and evidentiary requirements, performance and maintenance monitoring, usability testing, clinical evidence requirements, safety and security, equity considerations, organizational assessment, and additional outcome domains (patient empowerment and environmental impact). The COVID-19 pandemic greatly expanded the use of MMAs, but temporary policies governing their use and oversight need consolidation through well-developed frameworks to support decision-makers, producers and introduction into clinical care processes, especially in light of the strong international, cross-border character of MMAs, the new EU medical device and health technology assessment regulations, and the Next Generation EU funding earmarked for health digitalization.

Appsolutely secure? Psychometric properties of the German version of an app information privacy concerns measure during COVID-19

Introduction

Privacy concerns are an important barrier to adoption and continued use of digital technologies, particularly in the health sector. With the introduction of mobile health applications (mHealth apps), the construct of app information privacy concerns has received increased attention. However, few validated measures exist to capture said concerns in population samples, although they can help to improve public health efforts.

Methods

Using a cross-sectional survey of German adults (mean age = 35.62; 63.5% female), this study examined psychometric properties of the app information privacy concerns scale (AIPC). Analyses comprised confirmatory factor analysis, factorial validity (exploratory factor analysis), internal consistency, convergent validity (i.e., correlations with privacy victimhood, and app privacy concerns), and discriminant validity (i.e., daily app use, adoption intentions, and attitudes toward COVID-19 contact tracing app use).

Results

The analysis did not support the proposed three-factor structure of the AIPC (i.e., anxiety, personal attitude, and requirements). Instead, a four-factor model was preferable that differentiated requirements regarding disclosure policies, and personal control. In addition, factors mirroring anxiety and personal attitude were extracted, but shared a significant overlap. However, these factors showed good reliability, convergent and discriminant validity.

Discussion

The findings underline the role of app information privacy concerns as a significant barrier to mHealth app use. In this context, anxiety and personal attitudes seemed particularly relevant, which has implications for health communication. Moreover, the observed differentiation of external (disclosure) and internal (control) requirements aligns with health behavior change models and thus is a promising area for future research.

Benefit Assessment and Reimbursement of Digital Health Applications: Concepts for Setting Up a New System for Public Coverage

In Germany, some digital health applications (DiHA) became reimbursable through the statutory health insurance system with the adoption of the Digital Healthcare Act in 2019. Approaches and concepts for the German care context were developed in an iterative process, based on existing concepts from international experience. A DiHA categorization was developed that could be used as a basis to enable the creation of a reimbursed DiHA repository, and to derive evidence requirements for coverage and reimbursement for each DiHA. The results provide an overview of a possible classification of DiHA as well as approaches to assessment and evaluation. The structure of remuneration and pricing in connection with the formation of groups is demonstrated.

Designing Supportive e-Interventions for Partners of Men With Prostate Cancer Using Female Partners’ Experiences: Qualitative Exploration Study

Background

Partners of men living with prostate cancer (PCa) can experience a variety of unmet needs that are largely unaddressed by health care professionals. There is limited evidence to suggest which approach may be most effective in supporting partners’ unmet needs and further research is required to determine how to provide support to caregivers and how technology solutions can be designed.

Objective

This study aims to explore the experience of partners of men living with PCa and their perceptions of the potential role of information technology in supporting their needs.

Methods

A qualitative descriptive methodology using focus groups and phone interviews was used. Purposive sampling was used to recruit people attending a national conference supported by a national PCa organization. Interview guides were adapted from an existing evidence-based smartphone app for caregivers of people with colorectal cancer. Sessions were audio recorded and transcribed verbatim. A coding framework was developed, and transcripts were coded line by line into the framework. Codes within the framework were grouped into descriptive categories that were then developed into analytical themes.

Results

A total of 17 female partners participated in the study, with an average age of 64 (SD 8.5) years. The following two main themes emerged: In the first theme, that is, How technology can be shaped to support female partners of prostate cancer survivors, the content and design of the smartphone app was discussed in addressing female partners’ needs. The following four subthemes were developed: getting support from social networks and resources, the lack of relevant information, demystifying future care expectations during and following a PCa diagnosis, and delivering the smartphone app—to whom and from whom. In the second theme, that is, The benefits and barriers of technology, the suitability of smartphone apps as a supportive modality for female partners was described. This included three subthemes: the smartphone app as an appropriate modality for supporting female partners, the future anticipated benefits of using the smartphone app, and concerns for storing and accessing information on the internet.

Conclusions

A smartphone app may be a suitable modality for providing information and peer support to female partners of men living with PCa. There is a need to provide peer support for female partners in future interventions to ensure that female partners’ intimacy and daily practical needs are met.

Spanish adaptation and validation of the User Version of the Mobile Application Rating Scale (uMARS)

OBJECTIVE

While the professional version of the Mobile App Rating Scale (MARS) has already been translated, and validated into the Spanish language, its user-centered counterpart has not yet been adapted. Furthermore, no other similar tools exist in the Spanish language. The aim of this paper is to adapt and validate User Version of the MARS (uMARS) into the Spanish language.

MATERIALS AND METHODS

Cross-cultural adaptation, translation, and metric evaluation. The internal consistency and test-retest reliability of the Spanish version of the uMARS were evaluated using the RadarCovid app. Two hundred and sixteen participants rated the app using the translated scale. The app was then rated again 2 weeks later by 21 of these participants to measure test-retest reliability.

RESULTS

No major differences were observed between the uMARS original and the Spanish version. Discrimination indices (item-scale correlation) obtained appropriate results for both raters. The Spanish uMARS presented with excellent internal consistency, α = .89 and .67 for objective and subjective quality, respectively, and temporal stability (r > 0.82 for all items and subscales).

DISCUSSION

The Spanish uMARS is a useful tool for health professionals to recommend high-quality mobile apps to their patients based on the user's perspective and for researchers and app developers to use end-user feedback and evaluation, to help them identify highly appraised and valued components, as well as areas for further development, to continue ensuring the increasing quality and prominence of the area of mHealth.

CONCLUSION

uMARS Spanish version is an instrument with adequate metric properties to assess the quality of health apps from the user perspective.

Evaluation Metrics for Health Chatbots: A Delphi Study

BACKGROUND

 In recent years, an increasing number of health chatbots has been published in app stores and described in research literature. Given the sensitive data they are processing and the care settings for which they are developed, evaluation is essential to avoid harm to users. However, evaluations of those systems are reported inconsistently and without using a standardized set of evaluation metrics. Missing standards in health chatbot evaluation prevent comparisons of systems, and this may hamper acceptability since their reliability is unclear.

OBJECTIVES

 The objective of this paper is to make an important step toward developing a health-specific chatbot evaluation framework by finding consensus on relevant metrics.

METHODS

 We used an adapted Delphi study design to verify and select potential metrics that we retrieved initially from a scoping review. We invited researchers, health professionals, and health informaticians to score each metric for inclusion in the final evaluation framework, over three survey rounds. We distinguished metrics scored relevant with high, moderate, and low consensus. The initial set of metrics comprised 26 metrics (categorized as global metrics, metrics related to response generation, response understanding and aesthetics).

RESULTS

 Twenty-eight experts joined the first round and 22 (75%) persisted to the third round. Twenty-four metrics achieved high consensus and three metrics achieved moderate consensus. The core set for our framework comprises mainly global metrics (e.g., ease of use, security content accuracy), metrics related to response generation (e.g., appropriateness of responses), and related to response understanding. Metrics on aesthetics (font type and size, color) are less well agreed upon-only moderate or low consensus was achieved for those metrics.

CONCLUSION

 The results indicate that experts largely agree on metrics and that the consensus set is broad. This implies that health chatbot evaluation must be multifaceted to ensure acceptability.

Accessibility and Digital Mental Health: Considerations for More Accessible and Equitable Mental Health Apps

Digital mental health is often touted as a solution to issues of access to mental health care. However, there has been little research done to understand the accessibility of digital mental health, especially for those with disabilities. In this piece, we define accessibility as it relates to mental health apps, describe the current state of accessibility in the digital world broadly and in mental health apps more specifically, outline why accessibility matters in mental health apps, and identify future steps to better incorporate accessibility into research and development of mental health apps.