SUITABILITY OF CURRENT EVALUATION FRAMEWORKS FOR USE IN THE HEALTH TECHNOLOGY ASSESSMENT OF MOBILE MEDICAL APPLICATIONS: A SYSTEMATIC REVIEW

Clinical Validation of Digital Healthcare Solutions: State of the Art, Challenges and Opportunities

Digital health technologies (DHTs) at the intersection of health, medical informatics, and business aim to enhance patient care through personalised digital approaches. Ensuring the efficacy and reliability of these innovations demands rigorous clinical validation. A PubMed literature review (January 2006 to July 2023) identified 1250 papers, highlighting growing academic interest. A focused narrative review (January 2018 to July 2023) delved into challenges, highlighting issues such as diverse regulatory landscapes, adoption issues in complex healthcare systems, and a plethora of evaluation frameworks lacking pragmatic guidance. Existing frameworks often omit crucial criteria, neglect empirical evidence, and clinical effectiveness is rarely included as a criterion for DHT quality. The paper underscores the urgency of addressing challenges in accreditation, adoption, business models, and integration to safeguard the quality, efficacy, and safety of DHTs. A pivotal illustration of collaborative efforts to address these challenges is exemplified by the Digital Health Validation Center, dedicated to generating clinical evidence of innovative healthcare technologies and facilitating seamless technology transfer. In conclusion, it is necessary to harmonise evaluation approaches and frameworks, improve regulatory clarity, and commit to collaboration to integrate rigorous clinical validation and empirical evidence throughout the DHT life cycle.

A mixed methods analysis of existing assessment and evaluation tools (AETs) for mental health applications

Introduction

Mental health Applications (MH Apps) can potentially improve access to high-quality mental health care. However, the recent rapid expansion of MH Apps has created growing concern regarding their safety and effectiveness, leading to the development of AETs (Assessment and Evaluation Tools) to help guide users. This article provides a critical, mixed methods analysis of existing AETs for MH Apps by reviewing the criteria used to evaluate MH Apps and assessing their effectiveness as evaluation tools.

Methods

To identify relevant AETs, gray and scholarly literature were located through stakeholder consultation, Internet searching via Google and a literature search of bibliographic databases Medline, APA PsycInfo, and LISTA. Materials in English that provided a tool or method to evaluate MH Apps and were published from January 1, 2000, to January 26, 2021 were considered for inclusion.

Results

Thirteen relevant AETs targeted for MH Apps met the inclusion criteria. The qualitative analysis of AETs and their evaluation criteria revealed that despite purporting to focus on MH Apps, the included AETs did not contain criteria that made them more specific to MH Apps than general health applications. There appeared to be very little agreed-upon terminology in this field, and the focus of selection criteria in AETs is often IT-related, with a lesser focus on clinical issues, equity, and scientific evidence. The quality of AETs was quantitatively assessed using the AGREE II, a standardized tool for evaluating assessment guidelines. Three out of 13 AETs were deemed 'recommended' using the AGREE II.

Discussion

There is a need for further improvements to existing AETs. To realize the full potential of MH Apps and reduce stakeholders' concerns, AETs must be developed within the current laws and governmental health policies, be specific to mental health, be feasible to implement and be supported by rigorous research methodology, medical education, and public awareness.

Methodological Frameworks and Dimensions to Be Considered in Digital Health Technology Assessment: Scoping Review and Thematic Analysis

BACKGROUND

Digital health technologies (dHTs) offer a unique opportunity to address some of the major challenges facing health care systems worldwide. However, the implementation of dHTs raises some concerns, such as the limited understanding of their real impact on health systems and people's well-being or the potential risks derived from their use. In this context, health technology assessment (HTA) is 1 of the main tools that health systems can use to appraise evidence and determine the value of a given dHT. Nevertheless, due to the nature of dHTs, experts highlight the need to reconsider the frameworks used in traditional HTA.

OBJECTIVE

This scoping review (ScR) aimed to identify the methodological frameworks used worldwide for digital health technology assessment (dHTA); determine what domains are being considered; and generate, through a thematic analysis, a proposal for a methodological framework based on the most frequently described domains in the literature.

METHODS

The ScR was performed in accordance with the guidelines established in the PRISMA-ScR guidelines. We searched 7 databases for peer reviews and gray literature published between January 2011 and December 2021. The retrieved studies were screened using Rayyan in a single-blind manner by 2 independent authors, and data were extracted using ATLAS.ti software. The same software was used for thematic analysis.

RESULTS

The systematic search retrieved 3061 studies (n=2238, 73.1%, unique), of which 26 (0.8%) studies were included. From these, we identified 102 methodological frameworks designed for dHTA. These frameworks revealed great heterogeneity between them due to their different structures, approaches, and items to be considered in dHTA. In addition, we identified different wording used to refer to similar concepts. Through thematic analysis, we reduced this heterogeneity. In the first phase of the analysis, 176 provisional codes related to different assessment items emerged. In the second phase, these codes were clustered into 86 descriptive themes, which, in turn, were grouped in the third phase into 61 analytical themes and organized through a vertical hierarchy of 3 levels: level 1 formed by 13 domains, level 2 formed by 38 dimensions, and level 3 formed by 11 subdimensions. From these 61 analytical themes, we developed a proposal for a methodological framework for dHTA.

CONCLUSIONS

There is a need to adapt the existing frameworks used for dHTA or create new ones to more comprehensively assess different kinds of dHTs. Through this ScR, we identified 26 studies including 102 methodological frameworks and tools for dHTA. The thematic analysis of those 26 studies led to the definition of 12 domains, 38 dimensions, and 11 subdimensions that should be considered in dHTA.

Pragmatic Approach to the Assessment and Use of Digital Mental Health Interventions for Health Workers

Symptoms of poor mental health among working people, especially health workers, are on the rise in the United States, contributing to a burgeoning market of thousands of mental health technology products, few of which have undergone rigorous evaluation. Most research on these products focuses on deploying digital mental health interventions as ancillary support in clinical practice and community settings. Little is known of the effectiveness of these tools when employers offer them. We describe the landscape of digital mental health interventions, providing an overview of mental health conditions that are addressed with technology-based solutions in the workplace and the products and services available. We argue for employers to apply a methodical approach to evaluating and selecting technologies for their organizations, and we review relevant frameworks for evaluation. Considering the rapidly evolving landscape of digital mental health interventions, we offer evidence-informed recommendations to organizations and decision-makers seeking to support workplace mental health and well-being, and we advocate the development of products that help organizations assess how they can mitigate workplace conditions that may contribute to poorer mental health. (Am J Public Health. 2024;114(S2):S171-S179. https://doi.org/10.2105/AJPH.2023.307505).

Quality of Digital Health Interventions Across Different Health Care Domains: Secondary Data Analysis Study

BACKGROUND

There are more than 350,000 digital health interventions (DHIs) in the app stores. To ensure that they are effective and safe to use, they should be assessed for compliance with best practice standards.

OBJECTIVE

The objective of this paper was to examine and compare the compliance of DHIs with best practice standards and adherence to user experience (UX), professional and clinical assurance (PCA), and data privacy (DP).

METHODS

We collected assessment data from 1574 DHIs using the Organisation for the Review of Care and Health Apps Baseline Review (OBR) assessment tool. As part of the assessment, each DHI received a score out of 100 for each of the abovementioned areas (ie, UX, PCA, and DP). These 3 OBR scores are combined to make up the overall ORCHA score (a proxy for quality). Inferential statistics, probability distributions, Kruskal-Wallis, Wilcoxon rank sum test, Cliff delta, and Dunn tests were used to conduct the data analysis.

RESULTS

We found that 57.3% (902/1574) of the DHIs had an Organisation for the Review of Care and Health Apps (ORCHA) score below the threshold of 65. The overall median OBR score (ORCHA score) for all DHIs was 61.5 (IQR 51.0-73.0) out of 100. A total of 46.2% (12/26) of DHI's health care domains had a median equal to or above the ORCHA threshold score of 65. For the 3 assessment areas (UX, DP, and PCA), DHIs scored the highest for the UX assessment 75.2 (IQR 70.0-79.6), followed by DP 65.1 (IQR 55.0-73.4) and PCA 49.6 (IQR 31.9-76.1). UX scores had the least variance (SD 13.9), while PCA scores had the most (SD 24.8). Respiratory and urology DHIs were consistently highly ranked in the National Institute for Health and Care Excellence Evidence Standards Framework tiers B and C based on their ORCHA score.

CONCLUSIONS

There is a high level of variability in the ORCHA scores of DHIs across different health care domains. This suggests that there is an urgent need to improve compliance with best practices in some health care areas. Possible explanations for the observed differences might include varied market maturity and commercial interests within the different health care domains. More investment to support the development of higher-quality DHIs in areas such as ophthalmology, allergy, women's health, sexual health, and dental care may be needed.

Stakeholders' views and opinions on existing guidelines on "How to Choose Mental Health Apps"

Background

Mental health Applications (Mhealth Apps) can change how healthcare is delivered. However, very little is known about the efficacy of Mhealth Apps. Currently, only minimum guidance is available in Assessment and Evaluation Tools (AETs). Therefore, this project aims to understand AET developers' perspectives and end users' experiences and opinions on "how to choose a Mhealth App".

Objective

The primary objectives were: (1) obtaining stakeholder's opinions and experiences of development and use of AETs for Mhealth Apps, their weaknesses and strengths, and barriers in their implementation of Mhealth Apps; (2) the experiences of App users, their analyzation and, obstacles in the use of apps; and (3) to quantify themes related to choosing a Mhealth App.

Methods

This qualitative study, used a sampling method to recruit six stakeholders (one App developer, two AET developers, an individual with lived experience of mental health illness, and two physicians) who were interviewed using a topic guide. These were examined by researchers (CT, WK, & FN) using thematic content analysis. Additionally, an anonymous online survey of 107 individuals was conducted.

Findings

Our analyses revealed six main themes: (a) needs and opportunities; (b) views on Mhealth apps; (c) views & opinions on AETs; (d) implementation barriers; (e) system of evaluation and; (f) future directions. The first key concept was, all stakeholders agreed that Apps could significantly impact mental health and that end-users were unaware of mental health AETs and Apps. Secondly, due to commercial interests, end-users reliability of App evaluations requires clear conflict-free guidelines. Thirdly, AETs should be evaluated and developed through a rigorous methodology. Finally, stakeholders shared insights into future developments for AETs and Mhealth Apps. Additionally, online survey respondents chose a "health professional" as their preferred source of guidance in selecting a Mhealth app (84%) and best suited to develop guidelines (70%).

Conclusion

The interviews and survey highlight the need for Mhealth Apps to be regulated and the importance of health professionals' engagement in the implementation process. Similarly, without well-defined roles for App evaluations within the health care system, it is unlikely that AETs will have wider spread use and impact without risk.

Dynamic HTA for digital health solutions: opportunities and challenges for patient-centered evaluation

OBJECTIVES

Germany's 2019 Digital Healthcare Act (Digitale-Versorgung-Gesetz, or DVG) created a number of opportunities for the digital transformation of the healthcare delivery system. Key among these was the creation of a reimbursement pathway for patient-centered digital health applications (digitale Gesundheitsanwendungen, or DiGA). Worldwide, this is the first structured pathway for "prescribable" health applications at scale. As of October 10, 2023, 49 DiGA were listed in the official directory maintained by Germany's Federal Institute for Drugs and Medical Devices (BfArM); these are prescribable by physicians and psychotherapists and reimbursed by the German statutory health insurance system for all its 73 million beneficiaries. Looking ahead, a major challenge facing DiGA manufacturers will be the generation of the evidence required for ongoing price negotiations and reimbursement. Current health technology assessment (HTA) methods will need to be adapted for DiGA.

METHODS

We describe the core issues that distinguish HTA in this setting: (i) explicit allowance for more flexible research designs, (ii) the nature of initial evidence generation, which can be delivered (in its final form) up to one year after becoming reimbursable, and (iii) the dynamic nature of both product development and product evaluation. We present the digital health applications in the German DiGA scheme as a case study and highlight the role of RWE in the successful evaluation of DiGA on an ongoing basis.

RESULTS

When a DiGA is likely to be updated and assessed regularly, full-scale RCTs are infeasible; we therefore make the case for using real-world data and real-world evidence (RWE) for dynamic HTAs.

CONCLUSIONS

Continous evaluation using RWD is a regulatory innovation that can help improve the quality of DiGAs on the market.

International practices in health technology assessment and public financing of digital health technologies: recommendations for Hungary

Background

Evaluating and integrating digital health technologies is a critical component of a national healthcare ecosystem in the 2020s and is expected to even increase in significance.

Design

The paper gives an overview of international practices on public financing and health technology assessment of digital health technologies (DHTs) in five European Union (EU) countries and outlines recommendations for country-level action that relevant stakeholders can consider in order to support uptake of digital health solutions in Hungary. A scoping review was carried out to identify and gather country-specific classifications and international practices on the financing DHTs in five pioneering EU countries: Germany, France, Belgium, the United Kingdom and Finland.

Results

Several frameworks have been developed for DHTs, however there is no single, unified framework or method for classification, evaluation, and financing of digital health technologies in European context. European countries apply different taxonomy, use different assessment domains and regulations for the reimbursement of DHTs. The Working Group of the Hungarian Health Economic Society recommends eight specific points for stakeholders, importantly taking active role in shaping common clinical evidence standards and technical quality criteria across in order for common standards to be developed in the European Union single market.

Conclusion

Specificities of national healthcare contexts must be taken into account in decisions to allocate public funds to certain therapies rather than others.

Stress Management Apps: Systematic Search and Multidimensional Assessment of Quality and Characteristics

BACKGROUND

Chronic stress poses risks for physical and mental well-being. Stress management interventions have been shown to be effective, and stress management apps (SMAs) might help to transfer strategies into everyday life.

OBJECTIVE

This review aims to provide a comprehensive overview of the quality and characteristics of SMAs to give potential users or health professionals a guideline when searching for SMAs in common app stores.

METHODS

SMAs were identified with a systematic search in the European Google Play Store and Apple App Store. SMAs were screened and checked according to the inclusion criteria. General characteristics and quality were assessed by 2 independent raters using the German Mobile Application Rating Scale (MARS-G). The MARS-G assesses quality (range 1 to 5) on the following four dimensions: (1) engagement, (2) functionality, (3) esthetics, and (4) information. In addition, the theory-based stress management strategies, evidence base, long-term availability, and common characteristics of the 5 top-rated SMAs were assessed and derived.

RESULTS

Of 2044 identified apps, 121 SMAs were included. Frequently implemented strategies (also in the 5 top-rated SMAs) were psychoeducation, breathing, and mindfulness, as well as the use of monitoring and reminder functions. Of the 121 SMAs, 111 (91.7%) provided a privacy policy, but only 44 (36.4%) required an active confirmation of informed consent. Data sharing with third parties was disclosed in only 14.0% (17/121) of the SMAs. The average quality of the included apps was above the cutoff score of 3.5 (mean 3.59, SD 0.50). The MARS-G dimensions yielded values above this cutoff score (functionality: mean 4.14, SD 0.47; esthetics: mean 3.76, SD 0.73) and below this score (information: mean 3.42, SD 0.46; engagement: mean 3.05, SD 0.78). Most theory-based stress management strategies were regenerative stress management strategies. The evidence base for 9.1% (11/121) of the SMAs could be identified, indicating significant group differences in several variables (eg, stress or depressive symptoms) in favor of SMAs. Moreover, 38.0% (46/121) of the SMAs were no longer available after a 2-year period.

CONCLUSIONS

The moderate information quality, scarce evidence base, constraints in data privacy and security features, and high volatility of SMAs pose challenges for users, health professionals, and researchers. However, owing to the scalability of SMAs and the few but promising results regarding their effectiveness, they have a high potential to reach and help a broad audience. For a holistic stress management approach, SMAs could benefit from a broader repertoire of strategies, such as more instrumental and mental stress management strategies. The common characteristics of SMAs with top-rated quality can be used as guidance for potential users and health professionals, but owing to the high volatility of SMAs, enhanced evaluation frameworks are needed.

Conducting a systematic review and evaluation of commercially available mobile applications (apps) on a health-related topic: the TECH approach and a step-by-step methodological guide

OBJECTIVES

To provide an overview of the methodological considerations for conducting commercial smartphone health app reviews (mHealth reviews), with the aim of systematising the process and supporting high-quality evaluations of mHealth apps.

DESIGN

Synthesis of our research team's experiences of conducting and publishing various reviews of mHealth apps available on app stores and hand-searching the top medical informatics journals (eg, The Lancet Digital Health, npj Digital Medicine, Journal of Biomedical Informatics and the Journal of the American Medical Informatics Association) over the last five years (2018-2022) to identify other app reviews to contribute to the discussion of this method and supporting framework for developing a research (review) question and determining the eligibility criteria.

RESULTS

We present seven steps to support rigour in conducting reviews of health apps available on the app market: (1) writing a research question or aims, (2) conducting scoping searches and developing the protocol, (3) determining the eligibility criteria using the TECH framework, (4) conducting the final search and screening of health apps, (5) data extraction, (6) quality, functionality and other assessments and (7) analysis and synthesis of findings. We introduce the novel TECH approach to developing review questions and the eligibility criteria, which considers the Target user, Evaluation focus, Connectedness and the Health domain. Patient and public involvement and engagement opportunities are acknowledged, including co-developing the protocol and undertaking quality or usability assessments.

CONCLUSION

Commercial mHealth app reviews can provide important insights into the health app market, including the availability of apps and their quality and functionality. We have outlined seven key steps for conducting rigorous health app reviews in addition to the TECH acronym, which can support researchers in writing research questions and determining the eligibility criteria. Future work will include a collaborative effort to develop reporting guidelines and a quality appraisal tool to ensure transparency and quality in systematic app reviews.

Rigorous and rapid evidence assessment in digital health with the evidence DEFINED framework

Dozens of frameworks have been proposed to assess evidence for digital health interventions (DHIs), but existing frameworks may not facilitate DHI evidence reviews that meet the needs of stakeholder organizations including payers, health systems, trade organizations, and others. These organizations may benefit from a DHI assessment framework that is both rigorous and rapid. Here we propose a framework to assess Evidence in Digital health for EFfectiveness of INterventions with Evaluative Depth (Evidence DEFINED). Designed for real-world use, the Evidence DEFINED Quick Start Guide may help streamline DHI assessment. A checklist is provided summarizing high-priority evidence considerations in digital health. Evidence-to-recommendation guidelines are proposed, specifying degrees of adoption that may be appropriate for a range of evidence quality levels. Evidence DEFINED differs from prior frameworks in its inclusion of unique elements designed for rigor and speed. Rigor is increased by addressing three gaps in prior frameworks. First, prior frameworks are not adapted adequately to address evidence considerations that are unique to digital health. Second, prior frameworks do not specify evidence quality criteria requiring increased vigilance for DHIs in the current regulatory context. Third, extant frameworks rarely leverage established, robust methodologies that were developed for non-digital interventions. Speed is achieved in the Evidence DEFINED Framework through screening optimization and deprioritization of steps that may have limited value. The primary goals of Evidence DEFINED are to a) facilitate standardized, rapid, rigorous DHI evidence assessment in organizations and b) guide digital health solutions providers who wish to generate evidence that drives DHI adoption.

Adaptation and Utilization of a Postmarket Evaluation Model for Digital Contact Tracing Mobile Health Tools in the United States: Observational Cross-sectional Study

BACKGROUND

Case investigation and contact tracing are core public health activities used to interrupt disease transmission. These activities are traditionally conducted manually. During periods of high COVID-19 incidence, US health departments were unable to scale up case management staff to deliver effective and timely contact-tracing services. In response, digital contact tracing (DCT) apps for mobile phones were introduced to automate these activities. DCT apps detect when other DCT users are close enough to transmit COVID-19 and enable alerts to notify users of potential disease exposure. These apps were deployed quickly during the pandemic without an opportunity to conduct experiments to determine effectiveness. However, it is unclear whether these apps can effectively supplement understaffed manual contact tracers.

OBJECTIVE

The aims of this study were to (1) evaluate the effectiveness of COVID-19 DCT apps deployed in the United States during the COVID-19 pandemic and (2) determine if there is sufficient DCT adoption and interest in adoption to meet a minimum population use rate to be effective (56%). To assess uptake, interest and safe use covariates were derived from evaluating DCTs using the American Psychological Association App Evaluation Model (AEM) framework.

METHODS

We analyzed data from a nationally representative survey of US adults about their COVID-19-related behaviors and experiences. Survey respondents were divided into three segments: those who adopted a DCT app, those who are interested but did not adopt, and those not interested. Descriptive statistics were used to characterize factors of the three groups. Multivariable logistic regression models were used to analyze the characteristics of segments adopting and interested in DCT apps against AEM framework covariates.

RESULTS

An insufficient percentage of the population adopted or was interested in DCTs to achieve our minimum national target effectiveness rate (56%). A total of 17.4% (n=490) of the study population reported adopting a DCT app, 24.7% (n=697) reported interest, and 58.0% (n=1637) were not interested. Younger, high-income, and uninsured individuals were more likely to adopt a DCT app. In contrast, people in fair to poor health were interested in DCT apps but did not adopt them. App adoption was positively associated with visiting friends and family outside the home (odds ratio [OR] 1.63, 95% CI 1.28-2.09), not wearing masks (OR 0.52, 95% CI 0.38-0.71), and adopters thinking they have or had COVID-19 (OR 1.60, 95% CI 1.21-2.12).

CONCLUSIONS

Overall, a small percentage of the population adopted DCT apps. These apps may not be effective in protecting adopters' friends and family from their maskless contacts outside the home given low adoption rates. The public health community should account for safe use behavioral factors in future public health contact-tracing app design. The AEM framework was useful in developing a study design to evaluate DCT effectiveness and safety.

Safety considerations for assessing the quality of apps used during pregnancy: A scoping review

Objective

Pregnant women are increasingly turning to apps targeting knowledge and behaviour change for supporting healthy lifestyles and managing medical conditions. Yet, there is growing concern over the credibility and safety of content within mobile health (mHealth) apps. This scoping review aimed to systematically and thematically consolidate safety considerations described in reviews evaluating pregnancy-specific apps.

Methods

PubMed, Ovid MEDLINE® and EPub, CINAHL, Web of Science, Cochrane Libraries, and SCOPUS were systematically searched to identify reviews that assessed apps targeting pregnant women. Data related to safety were extracted and thematically analysed to establish a set of relevant safety considerations.

Results

Sixteen reviews met the inclusion criteria. The included reviews assessed an average of 27 apps each and targeted pregnancy topics, such as nutrition and physical activity. Five major and 20 minor themes were identified, including information, transparency, credibility, privacy and security, and app tailoring. Information, transparency, and credibility relate to the evidence base of information within the app, privacy and security of apps relate to the protection of personal information and data, and app tailoring relates to the consideration of contextual factors, such as local guidelines and digital health literacy.

Conclusions

Results present possible safety considerations when evaluating pregnancy-specific apps and emphasise a clear need for consumer guidance on how to make informed decisions around engagement and use of mHealth apps during pregnancy.

Evaluating User Preferences, Comprehension, and Trust in Apps for Environmental Health Hazards: Qualitative Case Study

BACKGROUND

Climate change is projected to increase environmental health hazard risks through fire-related air pollution and increased airborne pollen levels. To protect vulnerable populations, it is imperative that evidence-based and accessible interventions are available. The environmental health app, AirRater, was developed in 2015 in Australia to provide information on multiple atmospheric health hazards in near real time. The app allows users to view local environmental conditions, and input and track their personal symptoms to enable behaviors that protect health in response to environmental hazards.

OBJECTIVE

This study aimed to develop insights into users' perceptions of engagement, comprehension, and trust in AirRater to inform the future development of environmental health apps. Specifically, this study explored which AirRater features users engaged with, what additional features or functionality needs users felt they required, users' self-perception of understanding app information, and their level of trust in the information provided.

METHODS

A total of 42 adult AirRater users were recruited from 3 locations in Australia to participate in semistructured interviews to capture location- or context-specific experiences. Participants were notified of the recruitment opportunity through multiple avenues including newsletter articles and social media. Informed consent was obtained before participation, and the participants were remunerated for their time and perspectives. A preinterview questionnaire collected data including age range, any preexisting conditions, and location (postcode). All participant data were deidentified. Interviews were recorded, transcribed, and analyzed using thematic analysis in NVivo 12 (QSR International).

RESULTS

Participants discussed app features and functionality, as well as their understanding of, and trust in, the information provided by the app. Most (26/42, 62%) participants used and valued visual environmental hazard features, especially maps, location settings, and hazard alerts. Most (33/42, 78%) found information in the app easy to understand and support their needs, irrespective of their self-reported literacy levels. Many (21/42, 50%) users reported that they did not question the accuracy of the data presented in the app. Suggested enhancements include the provision of meteorological information (eg, wind speed or direction, air pressure, UV rating, and humidity), functionality enhancements (eg, forecasting, additional alerts, and the inclusion of health advice), and clarification of existing information (eg, symptom triggers), including the capacity to download personal summary data for a specified period.

CONCLUSIONS

Participants' perspectives can inform the future development of environmental health apps. Specifically, participants' insights support the identification of key elements for the optimal development of environmental health app design, including streamlining, capacity for users to customize, use of real time data, visual cues, credibility, and accuracy of data. The results also suggest that, in the future, iterative collaboration between developers, environmental agencies, and users will likely promote better functional design, user trust in the data, and ultimately better population health outcomes.

WtsWrng Interim Comparative Effectiveness Evaluation and Description of the Challenges to Develop, Assess, and Introduce This Novel Digital Application in a Traditional Health System

Science and technology have evolved quickly during the two decades of the 21st century, but healthcare systems are grounded in last century's structure and processes. Changes in the way health care is provided are demanded; digital transformation is a key driver making healthcare systems more accessible, agile, efficient, and citizen-centered. Nevertheless, the way healthcare systems function challenges the development (Innovation + Development and regulatory requirements), assessment (methodological guidance weaknesses), and adoption of digital applications (DAs). WtsWrng (WW), an innovative DA which uses images to interact with citizens for symptom triage and monitoring, is used as an example to show the challenges faced in its development and clinical validation and how these are being overcome. To prove WW's value from inception, novel approaches for evidence generation that allows for an agile and patient-centered development have been applied. Early scientific advice from NICE (UK) was sought for study design, an iterative development and interim analysis was performed, and different statistical parameters (Kappa, B statistic) were explored to face development and assessment challenges. WW triage accuracy at cutoff time ranged from 0.62 to 0.94 for the most frequent symptoms attending the Emergency Department (ED), with the observed concordance for the 12 most frequent diagnostics at hospital discharge fluctuating between 0.4 to 0.97; 8 of the diagnostics had a concordance greater than 0.8. This experience should provoke reflective thinking for DA developers, digital health scientists, regulators, health technology assessors, and payers.

Methodological Frameworks and Dimensions to Be Taken Into Consideration in Digital Health Technology Assessment: Protocol for a Scoping Review

Background

Health technology assessment (HTA) is one of the main tools that health systems have to appraise evidence and determine the value of a given health technology. Although the existing HTA frameworks are useful tools for the evaluation of a wide range of health technologies, more and more experts, organizations across the world, and HTA agencies are highlighting the need to update or develop specific methodological frameworks for the evaluation of digital health technologies in order to take into account additional domains that cover these technologies’ intrinsic characteristics.

Objective

The purpose of our scoping review is to identify the methodological frameworks that are used worldwide for the assessment of digital health technologies; determine what dimensions and aspects are being considered; and generate, through a thematic analysis, a proposal for a methodological framework that is based on the most frequently described dimensions in the literature.

Methods

The scoping review will be performed in accordance with the guidelines established in the updated statement of the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews). We will search for peer-reviewed and grey literature published between 2011 and the date of the search execution. The retrieved references will be reviewed in a single-blind manner by 2 independent authors, and their quality will be assessed by using the Critical Appraisal Skills Program tool. The ATLAS.ti software (Scientific Software Development GmbH) will be used for data extraction and to perform the thematic analysis.

Results

The scoping review is currently (May 2022) in progress. It is expected to be completed in October 2022, and the final results of the research will be presented and published by November 2022.

Conclusions

To our knowledge, no studies have been published to date that identify the existing methodological frameworks for digital HTA, determine which dimensions must be evaluated for correct decision-making, and serve as a basis for the development of a methodological framework of reference that health care systems can use to carry out this kind of assessment. This work is intended to address this knowledge gap of key relevance for the field of HTA.

International Registered Report Identifier (IRRID)

DERR1-10.2196/39905

Digital health applications and the fast-track pathway to public health coverage in Germany: challenges and opportunities based on first results

Objectives

Evidence-based decision-making is the sine qua non for safe and effective patient care and the long-term functioning of health systems. Since 2020 Digital Health Applications (DiHA) in Germany have been undergoing a systematic pathway to be reimbursed by statutory health insurance (SHI) which is attracting attention in other European countries. We therefore investigate coverage decisions on DiHA and the underlying evidence on health care effects, which legally include both medical outcomes and patient-centred structural and procedural outcomes.

Methods

Based on publicly available data of the Institute for Medicines and Medical Devices searched between 08/2021 and 02/2022, all DiHA listed in the corresponding registry and thus reimbursable by the SHI were systematically investigated and presented descriptively on the basis of predefined criteria, such as clinical condition, and costs. The clinical trials on DiHA permanently included in the registry were reviewed with regard to their study design, endpoints investigated, the survey instruments used, and whether an intention-to-treat analysis was performed. Risk of bias was assessed using the ROB II tool.

Results

By February 2022, 30 DiHA had been included in the DiHA registry, one third of them permanently and two thirds conditionally. Most DiHA were therapeutic applications for mental illness based on cognitive behavioural therapy. For all permanently included DiHA, randomised controlled trials were conducted to demonstrate the impact on health care effects. While medical outcomes were investigated for all of these DiHA, patient-centred structural and procedural outcomes were rarely investigated. The majority of clinical trials showed a high risk of bias, mainly due to insufficient reporting quality. Overall, the prices for DiHA covered by SHI are on average around € 150 per month (min. € 40; max. € 248).

Conclusions

Evidence-based decision-making on coverage of DiHA leaves room for improvements both in terms of reporting-quality and the use of patient-centred structural and procedural outcomes in addition to medical outcomes. With appropriate evidence, DiHA can offer an opportunity as an adjunct to existing therapy while currently the high risk of bias of the trials raises doubts about the justification of its high costs.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-022-08500-6.

Recommendations for developing a lifecycle, multidimensional assessment framework for mobile medical apps

Digital health and mobile medical apps (MMAs) have shown great promise in transforming health care, but their adoption in clinical care has been unsatisfactory, and regulatory guidance and coverage decisions have been lacking or incomplete. A multidimensional assessment framework for regulatory, policymaking, health technology assessment, and coverage purposes based on the MMA lifecycle is needed. A targeted review of relevant policy documents from international sources was conducted to map current MMA assessment frameworks, to formulate 10 recommendations, subsequently shared amongst an expert panel of key stakeholders. Recommendations go beyond economic dimensions such as cost and economic evaluation and also include MMA development and update, classification and evidentiary requirements, performance and maintenance monitoring, usability testing, clinical evidence requirements, safety and security, equity considerations, organizational assessment, and additional outcome domains (patient empowerment and environmental impact). The COVID-19 pandemic greatly expanded the use of MMAs, but temporary policies governing their use and oversight need consolidation through well-developed frameworks to support decision-makers, producers and introduction into clinical care processes, especially in light of the strong international, cross-border character of MMAs, the new EU medical device and health technology assessment regulations, and the Next Generation EU funding earmarked for health digitalization.

Implementation of a new Digi-HTA process for digital health technologies in Finland

OBJECTIVES

There has been a lack of health technology assessment (HTA) methods for novel digital health technologies (DHTs) such as mHealth, artificial intelligence, and robotics in Finland. The Digi-HTA method has been developed for this purpose. The aim of this study is to determine whether it would be possible to use Digi-HTA recommendations to support healthcare decision-makers. Secondly, from the perspective of companies offering different types of DHT products, this study assesses the suitability of using the Digi-HTA framework to perform HTAs for their products.

METHODS

Feedback about Digi-HTA recommendations was collected from healthcare professionals. DHT companies provided input about the Digi-HTA framework. Data were collected via a web-based survey and were analyzed using qualitative methods.

RESULTS

Of the twenty-four healthcare professional respondents, twenty said that the Digi-HTA recommendations contained all the necessary information, and twenty-one found them useful for their work. Respondents hoped that the Digi-HTA recommendations would be better integrated into the decision-making processes and healthcare professionals would be more informed about this new HTA process. The questions of the Digi-HTA framework were applicable for different DHT products based on the responses from DHT companies (n = 8).

CONCLUSIONS

According to the study participants, although the Digi-HTA recommendations include clear and beneficial information, their integration into healthcare decision-making processes should be improved. Responses from DHT companies indicate that the Digi-HTA framework would be an appropriate tool for performing assessments for their products. To generalize the findings of this study, more comprehensive studies will be needed.

Reimbursed Medication Adherence Enhancing Interventions in European Countries: Results of the EUREcA Study

Introduction: Current literature lacks detailed understanding of the reimbursement framework of medication adherence enhancing interventions (MAEIs). As part of the ENABLE COST Action, the EUREcA (“EUropen REimbursement strategies for interventions targeting medication Adherence”) study aimed to provide an in-depth overview of reimbursed MAEIs currently available in European countries at national and regional levels and to pave the way for further MAEIs to be implemented in the future.

Methods: A web-based, cross-sectional survey was performed across 38 European countries and Israel. The survey questionnaire was developed as a result of an iterative process of discussion informed by a desk review. The survey was performed among invited ENABLE collaborators from June to July 2021. Besides descriptive analysis, association between country income and health care expenditure, and the availability of reimbursed MAEIs were also assessed.

Results: The survey identified 13 reimbursed MAEIs in nine countries: multi-dose drug dispensing (n = 5), medication review (n = 4), smart device (n = 2), mobile application (n = 1), and patient education (n = 1). The median GDP per capita of countries having ≥1 reimbursed MAEI was significantly higher compared to countries having no reimbursed adherence intervention (33,888 EUR vs 16,620 EUR, respectively; p = 0.05).

Conclusions: Our findings highlight that to date only a small number of MAEIs have been reimbursed in European countries. Comprehensive health technology assessment recommendations and multi-stakeholder collaboration could help removing barriers related to the implementation and reimbursement of MAEIs.

Evaluation of Digital Interventions for Physical Activity Promotion: Protocol for a Scoping Review

Background

Digital interventions (DIs) could support physical activity (PA) promotion, according to recent reviews. However, it remains unclear if and how DIs for PA promotion are evaluated; thus, it is unclear if they support behavior change in real-world settings. A mapping of evidence from published reviews is required to focus on the evaluation of DIs for PA promotion.

Objective

The aim of our study is to investigate evaluation strategies for any outcome in the context of DIs for PA promotion by conducting a scoping review of published reviews.

Methods

Our scoping review adheres to the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines. The information sources include bibliographic databases (MEDLINE, PsycINFO, and CINAHL) and the bibliographies of the selected studies. The electronic search strategy was developed and conducted in collaboration with an experienced database specialist. The electronic search was conducted in English with no limits up to March 19, 2021, for sources with the terms digital intervention AND evaluation AND physical activity in titles or abstracts. After deduplication, 300 reviews selected from 4912 search results were assessed for eligibility by 2 authors working independently. The inclusion criteria were (1) healthy or clinical samples (population), (2) DIs for PA promotion (intervention), (3) comparisons to any other intervention or no intervention (comparison), (4) evaluation strategies (methods, results, or frameworks) for any outcome in the context of DIs for PA promotion (outcome), and (5) any published review (study type). According to the consensus reached during a discussion, 40 reviews met the inclusion criteria—36 from the electronic search and 4 from the manual search of the bibliographies of the 36 reviews. All reviews reported the evaluation strategies for any outcomes in the context of DIs for PA promotion in healthy or clinical samples. Data coding and the quality appraisal of systematic reviews are currently being performed independently by 2 authors.

Results

Our scoping review includes data from 40 published reviews (1 rapid review, 9 scoping reviews, and 30 systematic reviews). The focus of data coding is on evaluation strategies in the context of DIs for PA promotion and on the critical appraisal of the included systematic reviews. The final consensus regarding all data is expected in early 2022.

Conclusions

Interventions for PA promotion that are supported by digital technologies require evaluation to ensure their efficacy in real-world settings. Our scoping review is needed because it addresses novel objectives that focus on such evaluations and are not answered in the published reviews identified in our search. The evaluation strategies addressing DIs for PA promotion will be mapped to synthesize the results that have been reported in published reviews so far.

International Registered Report Identifier (IRRID)

DERR1-10.2196/35332

Considerations of diversity, equity, and inclusion in mental health apps: A scoping review of evaluation frameworks

Mental health applications or apps are a promising approach to reduce the pervasive unmet mental health need observed among marginalized groups. However, these groups face unique challenges to engage in and benefit from these interventions, and apps that consider issues of diversity, equity, and inclusion (DEI) may be better positioned to reach them. App evaluation frameworks emerge as crucial tools for researchers, clinicians, and users to select evidence-based apps. Nevertheless, it is unclear the extent to which existing assessment tools capture DEI factors. The present scoping review identified 68 studies that employed one or more assessment tools to evaluate a mental health app, leading to 44 unique app evaluation frameworks. Results showed that most frameworks were developed after 2015, and only 58% of them considered at least one DEI criterion. Frameworks that performed exceptionally well were those that combined multiple standardized and validated measures. This lack of consideration for DEI variables may limit the ability of app-based interventions to serve marginalized communities, or even worse, create new disparities. Therefore, we provide recommendations to improve current app evaluation frameworks' cultural robustness and clinical utility, maximizing their effectiveness when working with individuals from marginalized communities.

Identification and Evaluation of Methodologies to Assess the Quality of Mobile Health Apps in High-, Low-, and Middle-Income Countries: Rapid Review

BACKGROUND

In recent years, there has been rapid growth in the availability and use of mobile health (mHealth) apps around the world. A consensus regarding an accepted standard to assess the quality of such apps has yet to be reached. A factor that exacerbates the challenge of mHealth app quality assessment is variations in the interpretation of quality and its subdimensions. Consequently, it has become increasingly difficult for health care professionals worldwide to distinguish apps of high quality from those of lower quality. This exposes both patients and health care professionals to unnecessary risks. Despite progress, limited understanding of the contributions of researchers in low- and middle-income countries (LMICs) exists on this topic. Furthermore, the applicability of quality assessment methodologies in LMIC settings remains relatively unexplored.

OBJECTIVE

This rapid review aims to identify current methodologies in the literature to assess the quality of mHealth apps, understand what aspects of quality these methodologies address, determine what input has been made by authors from LMICs, and examine the applicability of such methodologies in LMICs.

METHODS

This review was registered with PROSPERO (International Prospective Register of Systematic Reviews). A search of PubMed, EMBASE, Web of Science, and Scopus was performed for papers related to mHealth app quality assessment methodologies, which were published in English between 2005 and 2020. By taking a rapid review approach, a thematic and descriptive analysis of the papers was performed.

RESULTS

Electronic database searches identified 841 papers. After the screening process, 52 papers remained for inclusion. Of the 52 papers, 5 (10%) proposed novel methodologies that could be used to evaluate mHealth apps of diverse medical areas of interest, 8 (15%) proposed methodologies that could be used to assess apps concerned with a specific medical focus, and 39 (75%) used methodologies developed by other published authors to evaluate the quality of various groups of mHealth apps. The authors in 6% (3/52) of papers were solely affiliated to institutes in LMICs. A further 15% (8/52) of papers had at least one coauthor affiliated to an institute in an LMIC.

CONCLUSIONS

Quality assessment of mHealth apps is complex in nature and at times subjective. Despite growing research on this topic, to date, an all-encompassing appropriate means for evaluating the quality of mHealth apps does not exist. There has been engagement with authors affiliated to institutes across LMICs; however, limited consideration of current generic methodologies for application in LMIC settings has been identified.

TRIAL REGISTRATION

PROSPERO CRD42020205149; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=205149.

The NICE Evidence Standards Framework for digital health and care technologies - Developing and maintaining an innovative evidence framework with global impact

Objective

In 2018, the UK National Institute for Health and Care Excellence (NICE), in partnership with Public Health England, NHS England, NHS Improvement and others, developed an evidence standards framework (ESF) for digital health and care technologies (DHTs). The ESF was designed to provide a standardised approach to guide developers and commissioners on the levels of evidence needed for the clinical and economic evaluation of DHTs by health and care systems.

Methods

The framework was developed using an agile and iterative methodology that included a literature review of existing initiatives and comparison of these against the requirements set by NHS England; iterative consultation with stakeholders through an expert working group and workshops; and questionnaire-based stakeholder input on a publicly available draft document.

Results

The evidence standards framework has been well-received and to date the ESF has been viewed online over 55,000 times and downloaded over 19,000 times.

Conclusions

In April 2021 we published an update to the ESF. Here, we summarise the process through which the ESF was developed, reflect on its global impact to date, and describe NICE’s ongoing work to maintain and improve the framework in the context for a fast moving, innovative field.

Health technology assessment for digital technologies that manage chronic disease: a systematic review

OBJECTIVE

A growing number of evaluation frameworks have emerged over recent years addressing the unique benefits and risk profiles of new classes of digital health technologies (DHTs). This systematic review aims to identify relevant frameworks and synthesize their recommendations into DHT-specific content to be considered when performing Health Technology Assessments (HTAs) for DHTs that manage chronic noncommunicable disease at home.

METHODS

Searches were undertaken of Medline, Embase, Econlit, CINAHL, and The Cochrane Library (January 2015 to March 2020), and relevant gray literature (January 2015 to August 2020) using keywords related to HTA, evaluation frameworks, and DHTs. Included framework reference lists were searched from 2010 until 2015. The EUNetHTA HTA Core Model version 3.0 was selected as a scaffold for content evaluation.

RESULTS

Forty-four frameworks were identified, mainly covering clinical effectiveness (n = 30) and safety (n = 23) issues. DHT-specific content recommended by framework authors fell within 28 of the 145 HTA Core Model issues. A further twenty-two DHT-specific issues not currently in the HTA Core Model were recommended.

CONCLUSIONS

Current HTA frameworks are unlikely to be sufficient for assessing DHTs. The development of DHT-specific content for HTA frameworks is hampered by DHTs having varied benefit and risk profiles. By focusing on DHTs that actively monitor/treat chronic noncommunicable diseases at home, we have extended DHT-specific content to all nine HTA Core Model domains. We plan to develop a supplementary evaluation framework for designing research studies, undertaking HTAs, and appraising the completeness of HTAs for DHTs.

Scoping review: Development and assessment of evaluation frameworks of mobile health apps for recommendations to consumers

Objective

The study sought to review the different assessment items that have been used within existing health app evaluation frameworks aimed at individual, clinician, or organizational users, and to analyze the scoring and evaluation methods used in these frameworks.

Materials and Methods

We searched multiple bibliographic databases and conducted backward searches of reference lists, using search terms that were synonyms of “health apps,” “evaluation,” and “frameworks.” The review covered publications from 2011 to April 2020. Studies on health app evaluation frameworks and studies that elaborated on the scaling and scoring mechanisms applied in such frameworks were included.

Results

Ten common domains were identified across general health app evaluation frameworks. A list of 430 assessment criteria was compiled across 97 identified studies. The most frequently used scaling mechanism was a 5-point Likert scale. Most studies have adopted summary statistics to generate the total scoring of each app, and the most popular approach taken was the calculation of mean or average scores. Other frameworks did not use any scaling or scoring mechanism and adopted criteria-based, pictorial, or descriptive approaches, or “threshold” filter.

Discussion

There is wide variance in the approaches to evaluating health apps within published frameworks, and this variance leads to ongoing uncertainty in how to evaluate health apps.

Conclusions

A new evaluation framework is needed that can integrate the full range of evaluative criteria within one structure, and provide summative guidance on health app rating, to support individual app users, clinicians, and health organizations in choosing or recommending the best health app.

Evaluating evaluation frameworks: a scoping review of frameworks for assessing health apps

OBJECTIVES

Despite an estimated 300 000 mobile health apps on the market, there remains no consensus around helping patients and clinicians select safe and effective apps. In 2018, our team drew on existing evaluation frameworks to identify salient categories and create a new framework endorsed by the American Psychiatric Association (APA). We have since created a more expanded and operational framework Mhealth Index and Navigation Database (MIND) that aligns with the APA categories but includes objective and auditable questions (105). We sought to survey the existing space, conducting a review of all mobile health app evaluation frameworks published since 2018, and demonstrate the comprehensiveness of this new model by comparing it to existing and emerging frameworks.

DESIGN

We conducted a scoping review of mobile health app evaluation frameworks.

DATA SOURCES

References were identified through searches of PubMed, EMBASE and PsychINFO with publication date between January 2018 and October 2020.

ELIGIBILITY CRITERIA

Papers were selected for inclusion if they meet the predetermined eligibility criteria-presenting an evaluation framework for mobile health apps with patient, clinician or end user-facing questions.

DATA EXTRACTION AND SYNTHESIS

Two reviewers screened the literature separately and applied the inclusion criteria. The data extracted from the papers included: author and dates of publication, source affiliation, country of origin, name of framework, study design, description of framework, intended audience/user and framework scoring system. We then compiled a collection of more than 1701 questions across 79 frameworks. We compared and grouped these questions using the MIND framework as a reference. We sought to identify the most common domains of evaluation while assessing the comprehensiveness and flexibility-as well as any potential gaps-of MIND.

RESULTS

New app evaluation frameworks continue to emerge and expand. Since our 2019 review of the app evaluation framework space, more frameworks include questions around privacy (43) and clinical foundation (57), reflecting an increased focus on issues of app security and evidence base. The majority of mapped frameworks overlapped with at least half of the MIND categories. The results of this search have informed a database (apps.digitalpsych.org) that users can access today.

CONCLUSION

As the number of app evaluation frameworks continues to rise, it is becoming difficult for users to select both an appropriate evaluation tool and to find an appropriate health app. This review provides a comparison of what different app evaluation frameworks are offering, where the field is converging and new priorities for improving clinical guidance.

Rapid Design and Delivery of an Experience-Based Co-designed Mobile App to Support the Mental Health Needs of Health Care Workers Affected by the COVID-19 Pandemic: Impact Evaluation Protocol

BACKGROUND

The COVID-19 pandemic has highlighted the importance of health care workers' mental health and well-being for the successful function of the health care system. Few targeted digital tools exist to support the mental health of hospital-based health care workers, and none of them appear to have been led and co-designed by health care workers.

OBJECTIVE

RMHive is being led and developed by health care workers using experience-based co-design (EBCD) processes as a mobile app to support the mental health challenges posed by the COVID-19 pandemic to health care workers. We present a protocol for the impact evaluation for the rapid design and delivery of the RMHive mobile app.

METHODS

The impact evaluation will adopt a mixed methods design. Qualitative data from photo interviews undertaken with up to 30 health care workers and semistructured interviews conducted with up to 30 governance stakeholders will be integrated with qualitative and quantitative user analytics data and user-generated demographic and mental health data entered into the app. Analyses will address three evaluation questions related to engagement with the mobile app, implementation and integration of the app, and the impact of the app on individual mental health outcomes. The design and development will be described using the Mobile Health Evidence Reporting and Assessment guidelines. Implementation of the app will be evaluated using normalization process theory to analyze qualitative data from interviews combined with text and video analysis from the semistructured interviews. Mental health impacts will be assessed using the total score of the 4-item Patient Health Questionnaire (PHQ4) and subscale scores for the 2-item Patient Health Questionnaire for depression and the 2-item Generalized Anxiety Scale for anxiety. The PHQ4 will be completed at baseline and at 14 and 28 days.

RESULTS

The anticipated average use period of the app is 30 days. The rapid design will occur over four months using EBCD to collect qualitative data and develop app content. The impact evaluation will monitor outcome data for up to 12 weeks following hospital-wide release of the minimal viable product release. The study received funding and ethics approvals in June 2020. Outcome data is expected to be available in March 2021, and the impact evaluation is expected to be published mid-2021.

CONCLUSIONS

The impact evaluation will examine the rapid design, development, and implementation of the RMHive app and its impact on mental health outcomes for health care workers. Findings from the impact evaluation will provide guidance for the integration of EBCD in rapid design and implementation processes. The evaluation will also inform future development and rollout of the app to support the mental health needs of hospital-based health care workers more widely.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/26168.

Distinguishing features in the assessment of mHealth apps

Introduction: The unparalleled surge in digital health adoption during the COVID-19 pandemic has emphasized the potential of mHealth apps. However, the quality of available evidence is generally low, and regulatory frameworks have focused on apps with medical purposes only, overlooking apps with significant interactions with patients that may require stronger oversight.Areas covered: To support this expanded evidence generation process, we identified the reasons that distinguish mHealth apps compared to medical devices at large and that should differentially feature their assessment. mHealth apps are characterized by the iterative nature of the corresponding interventions, frequent user interactions with a non-linear relationship between technology usage, engagement and outcomes, significant organizational implications, as well as challenges associated with genericization, their broad diagnostic potential, and price setting.Expert Opinion: The renewed reliance experienced during the pandemic and the unprecedented injection of resources through recovery instruments can further boost the development of apps. Only robust evidence of the benefits of mHealth apps will persuade health-care professionals and beneficiaries to systematically deploy them. Regulatory bodies will need to question their current approaches by adopting comprehensive evaluation processes that adequately consider the specific features of mHealth apps.

Evaluation of two mobile health apps for patients with breast cancer using the Mobile Application Rating Scale

BACKGROUND

Breast cancer is one of the most frequently diagnosed cancers worldwide. Screening, education about signs and symptoms, and improved access to treatment has helped reduce mortality. An understanding of the informational needs of women with breast cancer can help identify areas where mobile apps can further improve the experience of this patient group.

METHODS

Personas are a commonly used tools in user centred design to help represent particular user archetypes. Knowledge of existing informational needs and prior research using personas in breast cancer app design were used to create two different personas through which to source apps for evaluation. The Mobile Application Rating Scale, a common evaluation framework, was used to evaluate the mobile apps across several important domains.

RESULTS

Becca and OWise, two apps for breast cancer, were found through a discovery process in line with the personas described. Overall, both apps scored highly on the Mobile Application Rating Scale. Both apps had limited or no research to support their use in this patient group, and had issues related to data privacy. Becca scored particularly highly in domains related to accessibility while OWise's extensive range of features scored highly for functionality.

CONCLUSIONS

Both apps demonstrate the ability to fill an informational needs gap as evidenced in the existing literature. As with many mobile health apps, more clinical evidence and improved data handling would help support the widespread recommendation of their use in women who are undergoing or have completed treatment for breast cancer.

Here's something I prepared earlier: a review of the time to publication of cross-sectional reviews of smartphone health apps

OBJECTIVES

Across a range of health conditions, apps are increasingly valued as tools for supporting the delivery and coordination of healthcare. Research-led cross-sectional reviews of apps are a potential resource to inform app selection in face of uncertainties around content quality, safety and privacy. However, these peer-reviewed publications only capture a snapshot of highly dynamic app stores and marketplaces. To determine the extent to which marketplace dynamics might impact the interpretation of app reviews, the current study sought to quantify the lag between the reported time of app assessment and publication of the results of these studies.

DESIGN

Searches were conducted on MEDLINE, Embase and PsycINFO to identify published cross-sectional reviews of health, fitness or wellness apps. Publication timeline metadata were extracted, allowing the primary outcome measure, the delay between app store search and manuscript publication, to be calculated. A secondary measure, the time between search and manuscript submission, was also calculated where possible.

RESULTS

After screening, 136 relevant cross-sectional app review studies were analysed. The median time to publication was 431 days (approximately 14 months, range: 42-1054 days). The median time to submission was 269 days (approximately 9 months, range: 5-874 days). Studies which downloaded apps typically took longer to publish (p=0.010), however the number of apps reviewed did not impact the time to publication (p=0.964). Studies which recommended specific apps were not published more rapidly (p=0.998).

CONCLUSIONS

Most health app reviews present data that are at least a year out-of-date at the time of publication. Given the high rate of turnover of health apps in public marketplaces, it may not be appropriate, therefore, for these reviews to be presented as a resource concerning specific products for commissioners, clinicians and the public. Alternative sources of information may be better calibrated to the dynamics of the app marketplace.

Treating Psychological Trauma in the Midst of COVID-19: The Role of Smartphone Apps

With the COVID-19 pandemic confronting health systems worldwide, medical practitioners are treating a myriad of physical symptoms that have, sadly, killed many thousands of people. There are signs that the public is also experiencing psychological trauma as they attempt to navigate their way through the COVID-19 restrictions impinging on many aspects of society. With unprecedented demand for health professionals' time, people who are unable to access face-to-face assistance are turning to smartphone apps to help them deal with symptoms of trauma. However, the evidence for smartphone apps to treat trauma is limited, and clinicians need to be aware of the limitations and unresolved issues involved in using mental health apps.

Actionable health app evaluation: translating expert frameworks into objective metrics

As use and availability of mobile health apps have increased, so too has the need for a thorough, accessible framework for app evaluation. The American Psychiatric Association's app evaluation model has emerged as a way to critically assess an app by considering accessibility, privacy and security, clinical foundation, engagement, and interoperability; however, there is no centralized database where users can view how various health apps perform when assessed via the APA model. In this perspective, we propose and outline our effort to translate the APA's model for the evaluation of health apps into a set of objective metrics that can be published online, making the framework actionable and accessible to a broad audience. The questions from the APA model were operationalized into 105 objective questions that are either binary or numeric. These questions serve as the foundation of an online database, where app evaluation consists of answering these 105 questions and can be crowdsourced. While the database has yet to be published and crowdsourced, initial internal testing demonstrated excellent interrater reliability. The database proposed here introduces a public and interactive approach to data collection that is guided by the APA model. The published product enables users to sort through the many mobile health apps and filter them according to individual preferences and priorities, making the ever-growing health app market more navigable.

Apps With Maps-Anxiety and Depression Mobile Apps With Evidence-Based Frameworks: Systematic Search of Major App Stores

BACKGROUND

Mobile mental health apps have become ubiquitous tools to assist people in managing symptoms of anxiety and depression. However, due to the lack of research and expert input that has accompanied the development of most apps, concerns have been raised by clinicians, researchers, and government authorities about their efficacy.

OBJECTIVE

This review aimed to estimate the proportion of mental health apps offering comprehensive therapeutic treatments for anxiety and/or depression available in the app stores that have been developed using evidence-based frameworks. It also aimed to estimate the proportions of specific frameworks being used in an effort to understand which frameworks are having the most influence on app developers in this area.

METHODS

A systematic review of the Apple App Store and Google Play store was performed to identify apps offering comprehensive therapeutic interventions that targeted anxiety and/or depression. The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist was adapted to guide this approach.

RESULTS

Of the 293 apps shortlisted as offering a therapeutic treatment for anxiety and/or depression, 162 (55.3%) mentioned an evidence-based framework in their app store descriptions. Of the 293 apps, 88 (30.0%) claimed to use cognitive behavioral therapy techniques, 46 (15.7%) claimed to use mindfulness, 27 (9.2%) claimed to use positive psychology, 10 (3.4%) claimed to use dialectical behavior therapy, 5 (1.7%) claimed to use acceptance and commitment therapy, and 20 (6.8%) claimed to use other techniques. Of the 162 apps that claimed to use a theoretical framework, only 10 (6.2%) had published evidence for their efficacy.

CONCLUSIONS

The current proportion of apps developed using evidence-based frameworks is unacceptably low, and those without tested frameworks may be ineffective, or worse, pose a risk of harm to users. Future research should establish what other factors work in conjunction with evidence-based frameworks to produce efficacious mental health apps.

Development of a health technology assessment module for evaluating mobile medical applications

OBJECTIVE

The aim of this study was to develop a module which could be used to facilitate the assessment of mobile medical applications (MMA) for regulatory and reimbursement purposes.

METHODS

In-depth interviews were conducted with policymakers, healthcare practitioners, and application developers to determine possible pathways and impediments to MMA reimbursement. These findings were integrated with our previous research on MMA reimbursement and regulation to create a module that could be used with existing health technology assessment (HTA) methodological frameworks to guide the evaluation of MMAs.

RESULTS

Stakeholders indicated that they trust how traditional medical devices are currently appraised for reimbursement. They were concerned that there was a lack of clarity regarding which entity in the health system was responsible for determining app quality. They were also concerned about the digital health literacy of medical practitioners and patients. Concepts emerging from our previous research were reinforced by the interview findings, including that the connectivity and cybersecurity of apps need to be considered, along with an assessment of software reliability. It is also critical that the credibility of the information presented in apps is assessed as it could potentially mislead patients and clinicians.

CONCLUSION

An MMA evaluation module was created that would enable an existing HTA process to be adapted for the assessment of MMA technology. These adaptations include making provisions for an assessment of app cybersecurity, the impact on MMA clinical utility of software updates, and compatibility issues. Items to address concerns around practitioner responsibility and app misinformation were also incorporated into the module.

The Development of a Mobile Health App for Breast Cancer Self-Management Support in Taiwan: Design Thinking Approach

Background

Evidence has shown that breast cancer self-management support from mobile health (mHealth) apps can improve the quality of life of survivors. Although many breast cancer self-management support apps exist, few papers have documented the procedure for the development of a user-friendly app from the patient’s perspective.

Objective

This study aimed to investigate the information needs of Taiwanese women with breast cancer to inform the development of a self-management support mHealth app.

Methods

A 5-step design thinking approach, comprising empathy, define, ideate, prototype, and test steps, was used in the focus groups and individual interviews conducted to collect information on the requirements and expectations of Taiwanese women with breast cancer with respect to the app. A thematic analysis was used to identify information needs.

Results

A total of 8 major themes including treatment, physical activity, diet, emotional support, health records, social resources, experience sharing, and expert consultation were identified. Minor themes included the desire to use the app under professional supervision and a trustworthy app manager to ensure the credibility of information.

Conclusions

The strengths of the design thinking approach were user-centered design and cultural sensitivity. The results retrieved from each step contributed to the development of the app and reduction of the gap between end users and developers. An mHealth app that addresses these 8 main themes can facilitate disease self-management for Taiwanese women with breast cancer.

By the numbers: ratings and utilization of behavioral health mobile applications

Although >10,000 behavioral health applications ("apps") are currently available on the Apple and Google Play marketplaces, they have been minimally evaluated or regulated and little is known about "real world" usage patterns. This investigation combined data from online behavioral health app rating frameworks and a mobile health market research firm to identify the most downloaded apps as well as determine rating and ranking concordance between frameworks. Findings demonstrated that the most commonly downloaded apps focus on relaxation, mindfulness, and meditation skills and that they often have notably discordant reviews across rating frameworks. Our results suggest that there is a growing need for: (1) standardized behavioral health app quality and effectiveness measures, (2) up-to-date behavioral health app guidance for clinicians and consumers, and (3) evidence-based apps that incorporate revealed consumer preferences.