1
|
Gomis-Pastor M, Berdún J, Borrás-Santos A, De Dios López A, Fernández-Montells Rama B, García-Esquirol Ó, Gratacòs M, Ontiveros Rodríguez GD, Pelegrín Cruz R, Real J, Bachs i Ferrer J, Comella A. Clinical Validation of Digital Healthcare Solutions: State of the Art, Challenges and Opportunities. Healthcare (Basel) 2024; 12:1057. [PMID: 38891132 PMCID: PMC11171879 DOI: 10.3390/healthcare12111057] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2024] [Revised: 05/13/2024] [Accepted: 05/20/2024] [Indexed: 06/21/2024] Open
Abstract
Digital health technologies (DHTs) at the intersection of health, medical informatics, and business aim to enhance patient care through personalised digital approaches. Ensuring the efficacy and reliability of these innovations demands rigorous clinical validation. A PubMed literature review (January 2006 to July 2023) identified 1250 papers, highlighting growing academic interest. A focused narrative review (January 2018 to July 2023) delved into challenges, highlighting issues such as diverse regulatory landscapes, adoption issues in complex healthcare systems, and a plethora of evaluation frameworks lacking pragmatic guidance. Existing frameworks often omit crucial criteria, neglect empirical evidence, and clinical effectiveness is rarely included as a criterion for DHT quality. The paper underscores the urgency of addressing challenges in accreditation, adoption, business models, and integration to safeguard the quality, efficacy, and safety of DHTs. A pivotal illustration of collaborative efforts to address these challenges is exemplified by the Digital Health Validation Center, dedicated to generating clinical evidence of innovative healthcare technologies and facilitating seamless technology transfer. In conclusion, it is necessary to harmonise evaluation approaches and frameworks, improve regulatory clarity, and commit to collaboration to integrate rigorous clinical validation and empirical evidence throughout the DHT life cycle.
Collapse
Affiliation(s)
- Mar Gomis-Pastor
- Digital Health Validation Center, Hospital de la Santa Creu i Sant Pau, Sant Pau Campus Salut Barcelona, 08041 Barcelona, Spain; (J.B.); (A.B.-S.); (A.D.D.L.); (B.F.-M.R.); (G.D.O.R.); (R.P.C.); (J.R.); (A.C.)
- Institut de Recerca Sant Pau (IR SANT PAU), Sant Quintí 77 79, 08041 Barcelona, Spain
| | - Jesús Berdún
- Digital Health Validation Center, Hospital de la Santa Creu i Sant Pau, Sant Pau Campus Salut Barcelona, 08041 Barcelona, Spain; (J.B.); (A.B.-S.); (A.D.D.L.); (B.F.-M.R.); (G.D.O.R.); (R.P.C.); (J.R.); (A.C.)
- Institut de Recerca Sant Pau (IR SANT PAU), Sant Quintí 77 79, 08041 Barcelona, Spain
| | - Alicia Borrás-Santos
- Digital Health Validation Center, Hospital de la Santa Creu i Sant Pau, Sant Pau Campus Salut Barcelona, 08041 Barcelona, Spain; (J.B.); (A.B.-S.); (A.D.D.L.); (B.F.-M.R.); (G.D.O.R.); (R.P.C.); (J.R.); (A.C.)
- Institut de Recerca Sant Pau (IR SANT PAU), Sant Quintí 77 79, 08041 Barcelona, Spain
| | - Anna De Dios López
- Digital Health Validation Center, Hospital de la Santa Creu i Sant Pau, Sant Pau Campus Salut Barcelona, 08041 Barcelona, Spain; (J.B.); (A.B.-S.); (A.D.D.L.); (B.F.-M.R.); (G.D.O.R.); (R.P.C.); (J.R.); (A.C.)
- Institut de Recerca Sant Pau (IR SANT PAU), Sant Quintí 77 79, 08041 Barcelona, Spain
- Pharmacy Department, Hospital de la Santa Creu i Sant Pau, IIB Sant Pau, 08041 Barcelona, Spain
| | - Beatriz Fernández-Montells Rama
- Digital Health Validation Center, Hospital de la Santa Creu i Sant Pau, Sant Pau Campus Salut Barcelona, 08041 Barcelona, Spain; (J.B.); (A.B.-S.); (A.D.D.L.); (B.F.-M.R.); (G.D.O.R.); (R.P.C.); (J.R.); (A.C.)
- Institut de Recerca Sant Pau (IR SANT PAU), Sant Quintí 77 79, 08041 Barcelona, Spain
| | | | - Mònica Gratacòs
- DAP-Cat Group, Unitat de Suport a la Recerca Barcelona, Fundació Institut Universitari Per a la Recerca a l’Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol), 08028 Barcelona, Spain;
| | - Gerardo D. Ontiveros Rodríguez
- Digital Health Validation Center, Hospital de la Santa Creu i Sant Pau, Sant Pau Campus Salut Barcelona, 08041 Barcelona, Spain; (J.B.); (A.B.-S.); (A.D.D.L.); (B.F.-M.R.); (G.D.O.R.); (R.P.C.); (J.R.); (A.C.)
- Institut de Recerca Sant Pau (IR SANT PAU), Sant Quintí 77 79, 08041 Barcelona, Spain
| | - Rebeca Pelegrín Cruz
- Digital Health Validation Center, Hospital de la Santa Creu i Sant Pau, Sant Pau Campus Salut Barcelona, 08041 Barcelona, Spain; (J.B.); (A.B.-S.); (A.D.D.L.); (B.F.-M.R.); (G.D.O.R.); (R.P.C.); (J.R.); (A.C.)
- Institut de Recerca Sant Pau (IR SANT PAU), Sant Quintí 77 79, 08041 Barcelona, Spain
- Pharmacy Department, Hospital de la Santa Creu i Sant Pau, IIB Sant Pau, 08041 Barcelona, Spain
| | - Jordi Real
- Digital Health Validation Center, Hospital de la Santa Creu i Sant Pau, Sant Pau Campus Salut Barcelona, 08041 Barcelona, Spain; (J.B.); (A.B.-S.); (A.D.D.L.); (B.F.-M.R.); (G.D.O.R.); (R.P.C.); (J.R.); (A.C.)
- Institut de Recerca Sant Pau (IR SANT PAU), Sant Quintí 77 79, 08041 Barcelona, Spain
| | - Jordi Bachs i Ferrer
- Departament d’Economia i Organització d’Empreses, Universitat de Barcelona (UB), 08036 Barcelona, Spain;
| | - Adrià Comella
- Digital Health Validation Center, Hospital de la Santa Creu i Sant Pau, Sant Pau Campus Salut Barcelona, 08041 Barcelona, Spain; (J.B.); (A.B.-S.); (A.D.D.L.); (B.F.-M.R.); (G.D.O.R.); (R.P.C.); (J.R.); (A.C.)
| |
Collapse
|
2
|
Ahmed S, Trimmer C, Khan W, Tuck A, Rodak T, Agic B, Kavic K, Wadhawan S, Abbott M, Husain MO, Husain MI, McKenzie K, Quintana Y, Naeem F. A mixed methods analysis of existing assessment and evaluation tools (AETs) for mental health applications. Front Public Health 2024; 12:1196491. [PMID: 38774052 PMCID: PMC11106355 DOI: 10.3389/fpubh.2024.1196491] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2023] [Accepted: 04/17/2024] [Indexed: 05/24/2024] Open
Abstract
Introduction Mental health Applications (MH Apps) can potentially improve access to high-quality mental health care. However, the recent rapid expansion of MH Apps has created growing concern regarding their safety and effectiveness, leading to the development of AETs (Assessment and Evaluation Tools) to help guide users. This article provides a critical, mixed methods analysis of existing AETs for MH Apps by reviewing the criteria used to evaluate MH Apps and assessing their effectiveness as evaluation tools. Methods To identify relevant AETs, gray and scholarly literature were located through stakeholder consultation, Internet searching via Google and a literature search of bibliographic databases Medline, APA PsycInfo, and LISTA. Materials in English that provided a tool or method to evaluate MH Apps and were published from January 1, 2000, to January 26, 2021 were considered for inclusion. Results Thirteen relevant AETs targeted for MH Apps met the inclusion criteria. The qualitative analysis of AETs and their evaluation criteria revealed that despite purporting to focus on MH Apps, the included AETs did not contain criteria that made them more specific to MH Apps than general health applications. There appeared to be very little agreed-upon terminology in this field, and the focus of selection criteria in AETs is often IT-related, with a lesser focus on clinical issues, equity, and scientific evidence. The quality of AETs was quantitatively assessed using the AGREE II, a standardized tool for evaluating assessment guidelines. Three out of 13 AETs were deemed 'recommended' using the AGREE II. Discussion There is a need for further improvements to existing AETs. To realize the full potential of MH Apps and reduce stakeholders' concerns, AETs must be developed within the current laws and governmental health policies, be specific to mental health, be feasible to implement and be supported by rigorous research methodology, medical education, and public awareness.
Collapse
Affiliation(s)
- Sarah Ahmed
- Centre for Addiction and Mental Health (CAMH), Toronto, ON, Canada
| | - Chris Trimmer
- Centre for Addiction and Mental Health (CAMH), Toronto, ON, Canada
| | - Wishah Khan
- Centre for Addiction and Mental Health (CAMH), Toronto, ON, Canada
| | - Andrew Tuck
- Centre for Addiction and Mental Health (CAMH), Toronto, ON, Canada
| | - Terri Rodak
- Centre for Addiction and Mental Health (CAMH), Toronto, ON, Canada
| | - Branka Agic
- Centre for Addiction and Mental Health (CAMH), Toronto, ON, Canada
- Department of Psychiatry, University of Toronto, Toronto, ON, Canada
| | - Kelsey Kavic
- Mental Health Commission of Canada, Ottawa, ON, Canada
| | | | | | - M. Omair Husain
- Centre for Addiction and Mental Health (CAMH), Toronto, ON, Canada
- Department of Psychiatry, University of Toronto, Toronto, ON, Canada
| | - M. Ishrat Husain
- Centre for Addiction and Mental Health (CAMH), Toronto, ON, Canada
- Department of Psychiatry, University of Toronto, Toronto, ON, Canada
| | - Kwame McKenzie
- Centre for Addiction and Mental Health (CAMH), Toronto, ON, Canada
- Department of Psychiatry, University of Toronto, Toronto, ON, Canada
| | - Yuri Quintana
- Harvard Medical School, Harvard University, Boston, MA, United States
- Division of Clinical Informatics, Beth Israel Deaconess Medical Center, Boston, MA, United States
| | - Farooq Naeem
- Centre for Addiction and Mental Health (CAMH), Toronto, ON, Canada
- Department of Psychiatry, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
3
|
Segur-Ferrer J, Moltó-Puigmartí C, Pastells-Peiró R, Vivanco-Hidalgo RM. Methodological Frameworks and Dimensions to Be Considered in Digital Health Technology Assessment: Scoping Review and Thematic Analysis. J Med Internet Res 2024; 26:e48694. [PMID: 38598288 PMCID: PMC11043933 DOI: 10.2196/48694] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/03/2023] [Revised: 12/01/2023] [Accepted: 02/20/2024] [Indexed: 04/11/2024] Open
Abstract
BACKGROUND Digital health technologies (dHTs) offer a unique opportunity to address some of the major challenges facing health care systems worldwide. However, the implementation of dHTs raises some concerns, such as the limited understanding of their real impact on health systems and people's well-being or the potential risks derived from their use. In this context, health technology assessment (HTA) is 1 of the main tools that health systems can use to appraise evidence and determine the value of a given dHT. Nevertheless, due to the nature of dHTs, experts highlight the need to reconsider the frameworks used in traditional HTA. OBJECTIVE This scoping review (ScR) aimed to identify the methodological frameworks used worldwide for digital health technology assessment (dHTA); determine what domains are being considered; and generate, through a thematic analysis, a proposal for a methodological framework based on the most frequently described domains in the literature. METHODS The ScR was performed in accordance with the guidelines established in the PRISMA-ScR guidelines. We searched 7 databases for peer reviews and gray literature published between January 2011 and December 2021. The retrieved studies were screened using Rayyan in a single-blind manner by 2 independent authors, and data were extracted using ATLAS.ti software. The same software was used for thematic analysis. RESULTS The systematic search retrieved 3061 studies (n=2238, 73.1%, unique), of which 26 (0.8%) studies were included. From these, we identified 102 methodological frameworks designed for dHTA. These frameworks revealed great heterogeneity between them due to their different structures, approaches, and items to be considered in dHTA. In addition, we identified different wording used to refer to similar concepts. Through thematic analysis, we reduced this heterogeneity. In the first phase of the analysis, 176 provisional codes related to different assessment items emerged. In the second phase, these codes were clustered into 86 descriptive themes, which, in turn, were grouped in the third phase into 61 analytical themes and organized through a vertical hierarchy of 3 levels: level 1 formed by 13 domains, level 2 formed by 38 dimensions, and level 3 formed by 11 subdimensions. From these 61 analytical themes, we developed a proposal for a methodological framework for dHTA. CONCLUSIONS There is a need to adapt the existing frameworks used for dHTA or create new ones to more comprehensively assess different kinds of dHTs. Through this ScR, we identified 26 studies including 102 methodological frameworks and tools for dHTA. The thematic analysis of those 26 studies led to the definition of 12 domains, 38 dimensions, and 11 subdimensions that should be considered in dHTA.
Collapse
Affiliation(s)
- Joan Segur-Ferrer
- Agency for Health Quality and Assessment of Catalonia, Barcelona, Spain
| | | | | | | |
Collapse
|
4
|
Krisher L, Boeldt DL, Sigmon CAN, Rimel SE, Newman LS. Pragmatic Approach to the Assessment and Use of Digital Mental Health Interventions for Health Workers. Am J Public Health 2024; 114:171-179. [PMID: 38354345 PMCID: PMC10916727 DOI: 10.2105/ajph.2023.307505] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 10/14/2023] [Indexed: 02/16/2024]
Abstract
Symptoms of poor mental health among working people, especially health workers, are on the rise in the United States, contributing to a burgeoning market of thousands of mental health technology products, few of which have undergone rigorous evaluation. Most research on these products focuses on deploying digital mental health interventions as ancillary support in clinical practice and community settings. Little is known of the effectiveness of these tools when employers offer them. We describe the landscape of digital mental health interventions, providing an overview of mental health conditions that are addressed with technology-based solutions in the workplace and the products and services available. We argue for employers to apply a methodical approach to evaluating and selecting technologies for their organizations, and we review relevant frameworks for evaluation. Considering the rapidly evolving landscape of digital mental health interventions, we offer evidence-informed recommendations to organizations and decision-makers seeking to support workplace mental health and well-being, and we advocate the development of products that help organizations assess how they can mitigate workplace conditions that may contribute to poorer mental health. (Am J Public Health. 2024;114(S2):S171-S179. https://doi.org/10.2105/AJPH.2023.307505).
Collapse
Affiliation(s)
- Lyndsay Krisher
- Lyndsay Krisher is with the Center for Health, Work & Environment and the Department of Environmental & Occupational Health, Colorado School of Public Health, University of Colorado Anschutz Medical Campus, Aurora. Debra L. Boeldt, Chloe A. Nicksic Sigmon, and Sarah E. Rimel are with the National Mental Health Innovation Center, University of Colorado Anschutz Medical Campus. Lee S. Newman is with the Center for Health, Work & Environment and the Department of Environmental & Occupational Health, Colorado School of Public Health, Aurora
| | - Debra L Boeldt
- Lyndsay Krisher is with the Center for Health, Work & Environment and the Department of Environmental & Occupational Health, Colorado School of Public Health, University of Colorado Anschutz Medical Campus, Aurora. Debra L. Boeldt, Chloe A. Nicksic Sigmon, and Sarah E. Rimel are with the National Mental Health Innovation Center, University of Colorado Anschutz Medical Campus. Lee S. Newman is with the Center for Health, Work & Environment and the Department of Environmental & Occupational Health, Colorado School of Public Health, Aurora
| | - Chloe A Nicksic Sigmon
- Lyndsay Krisher is with the Center for Health, Work & Environment and the Department of Environmental & Occupational Health, Colorado School of Public Health, University of Colorado Anschutz Medical Campus, Aurora. Debra L. Boeldt, Chloe A. Nicksic Sigmon, and Sarah E. Rimel are with the National Mental Health Innovation Center, University of Colorado Anschutz Medical Campus. Lee S. Newman is with the Center for Health, Work & Environment and the Department of Environmental & Occupational Health, Colorado School of Public Health, Aurora
| | - Sarah E Rimel
- Lyndsay Krisher is with the Center for Health, Work & Environment and the Department of Environmental & Occupational Health, Colorado School of Public Health, University of Colorado Anschutz Medical Campus, Aurora. Debra L. Boeldt, Chloe A. Nicksic Sigmon, and Sarah E. Rimel are with the National Mental Health Innovation Center, University of Colorado Anschutz Medical Campus. Lee S. Newman is with the Center for Health, Work & Environment and the Department of Environmental & Occupational Health, Colorado School of Public Health, Aurora
| | - Lee S Newman
- Lyndsay Krisher is with the Center for Health, Work & Environment and the Department of Environmental & Occupational Health, Colorado School of Public Health, University of Colorado Anschutz Medical Campus, Aurora. Debra L. Boeldt, Chloe A. Nicksic Sigmon, and Sarah E. Rimel are with the National Mental Health Innovation Center, University of Colorado Anschutz Medical Campus. Lee S. Newman is with the Center for Health, Work & Environment and the Department of Environmental & Occupational Health, Colorado School of Public Health, Aurora
| |
Collapse
|
5
|
Hyzy M, Bond R, Mulvenna M, Bai L, Dix A, Daly R, Frey AL, Leigh S. Quality of Digital Health Interventions Across Different Health Care Domains: Secondary Data Analysis Study. JMIR Mhealth Uhealth 2023; 11:e47043. [PMID: 37995121 PMCID: PMC10704310 DOI: 10.2196/47043] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/06/2023] [Revised: 07/06/2023] [Accepted: 10/18/2023] [Indexed: 11/24/2023] Open
Abstract
BACKGROUND There are more than 350,000 digital health interventions (DHIs) in the app stores. To ensure that they are effective and safe to use, they should be assessed for compliance with best practice standards. OBJECTIVE The objective of this paper was to examine and compare the compliance of DHIs with best practice standards and adherence to user experience (UX), professional and clinical assurance (PCA), and data privacy (DP). METHODS We collected assessment data from 1574 DHIs using the Organisation for the Review of Care and Health Apps Baseline Review (OBR) assessment tool. As part of the assessment, each DHI received a score out of 100 for each of the abovementioned areas (ie, UX, PCA, and DP). These 3 OBR scores are combined to make up the overall ORCHA score (a proxy for quality). Inferential statistics, probability distributions, Kruskal-Wallis, Wilcoxon rank sum test, Cliff delta, and Dunn tests were used to conduct the data analysis. RESULTS We found that 57.3% (902/1574) of the DHIs had an Organisation for the Review of Care and Health Apps (ORCHA) score below the threshold of 65. The overall median OBR score (ORCHA score) for all DHIs was 61.5 (IQR 51.0-73.0) out of 100. A total of 46.2% (12/26) of DHI's health care domains had a median equal to or above the ORCHA threshold score of 65. For the 3 assessment areas (UX, DP, and PCA), DHIs scored the highest for the UX assessment 75.2 (IQR 70.0-79.6), followed by DP 65.1 (IQR 55.0-73.4) and PCA 49.6 (IQR 31.9-76.1). UX scores had the least variance (SD 13.9), while PCA scores had the most (SD 24.8). Respiratory and urology DHIs were consistently highly ranked in the National Institute for Health and Care Excellence Evidence Standards Framework tiers B and C based on their ORCHA score. CONCLUSIONS There is a high level of variability in the ORCHA scores of DHIs across different health care domains. This suggests that there is an urgent need to improve compliance with best practices in some health care areas. Possible explanations for the observed differences might include varied market maturity and commercial interests within the different health care domains. More investment to support the development of higher-quality DHIs in areas such as ophthalmology, allergy, women's health, sexual health, and dental care may be needed.
Collapse
Affiliation(s)
- Maciej Hyzy
- School of Computing, Ulster University, Belfast, United Kingdom
| | - Raymond Bond
- School of Computing, Ulster University, Belfast, United Kingdom
| | | | - Lu Bai
- School of Electronics, Electrical Engineering and Computer Science, Queen's University Belfast, Belfast, United Kingdom
| | - Alan Dix
- Swansea University, Swansea, United Kingdom
| | - Robert Daly
- Organisation for the Review of Care and Health Apps (ORCHA), Daresbury, United Kingdom
| | - Anna-Lena Frey
- Organisation for the Review of Care and Health Apps (ORCHA), Daresbury, United Kingdom
| | - Simon Leigh
- Organisation for the Review of Care and Health Apps (ORCHA), Daresbury, United Kingdom
| |
Collapse
|
6
|
Khan W, Jebanesan B, Ahmed S, Trimmer C, Agic B, Safa F, Ashraf A, Tuck A, Kavic K, Wadhawan S, Abbott M, Husain O, Husain I, Akhter Hamid M, McKenzie K, Quintana Y, Naeem F. Stakeholders' views and opinions on existing guidelines on "How to Choose Mental Health Apps". Front Public Health 2023; 11:1251050. [PMID: 38074730 PMCID: PMC10703154 DOI: 10.3389/fpubh.2023.1251050] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2023] [Accepted: 10/30/2023] [Indexed: 12/18/2023] Open
Abstract
Background Mental health Applications (Mhealth Apps) can change how healthcare is delivered. However, very little is known about the efficacy of Mhealth Apps. Currently, only minimum guidance is available in Assessment and Evaluation Tools (AETs). Therefore, this project aims to understand AET developers' perspectives and end users' experiences and opinions on "how to choose a Mhealth App". Objective The primary objectives were: (1) obtaining stakeholder's opinions and experiences of development and use of AETs for Mhealth Apps, their weaknesses and strengths, and barriers in their implementation of Mhealth Apps; (2) the experiences of App users, their analyzation and, obstacles in the use of apps; and (3) to quantify themes related to choosing a Mhealth App. Methods This qualitative study, used a sampling method to recruit six stakeholders (one App developer, two AET developers, an individual with lived experience of mental health illness, and two physicians) who were interviewed using a topic guide. These were examined by researchers (CT, WK, & FN) using thematic content analysis. Additionally, an anonymous online survey of 107 individuals was conducted. Findings Our analyses revealed six main themes: (a) needs and opportunities; (b) views on Mhealth apps; (c) views & opinions on AETs; (d) implementation barriers; (e) system of evaluation and; (f) future directions. The first key concept was, all stakeholders agreed that Apps could significantly impact mental health and that end-users were unaware of mental health AETs and Apps. Secondly, due to commercial interests, end-users reliability of App evaluations requires clear conflict-free guidelines. Thirdly, AETs should be evaluated and developed through a rigorous methodology. Finally, stakeholders shared insights into future developments for AETs and Mhealth Apps. Additionally, online survey respondents chose a "health professional" as their preferred source of guidance in selecting a Mhealth app (84%) and best suited to develop guidelines (70%). Conclusion The interviews and survey highlight the need for Mhealth Apps to be regulated and the importance of health professionals' engagement in the implementation process. Similarly, without well-defined roles for App evaluations within the health care system, it is unlikely that AETs will have wider spread use and impact without risk.
Collapse
Affiliation(s)
- Wishah Khan
- Centre for Addiction and Mental Health (CAMH), Toronto, ON, Canada
| | | | - Sarah Ahmed
- Centre for Addiction and Mental Health (CAMH), Toronto, ON, Canada
| | - Chris Trimmer
- Centre for Addiction and Mental Health (CAMH), Toronto, ON, Canada
| | - Branka Agic
- Centre for Addiction and Mental Health (CAMH), Toronto, ON, Canada
- Department of Psychiatry, University of Toronto, Toronto, ON, Canada
| | - Farhana Safa
- Centre for Addiction and Mental Health (CAMH), Toronto, ON, Canada
| | - Aamna Ashraf
- Centre for Addiction and Mental Health (CAMH), Toronto, ON, Canada
| | - Andrew Tuck
- Centre for Addiction and Mental Health (CAMH), Toronto, ON, Canada
| | - Kelsey Kavic
- Mental Health Commission of Canada, Ottawa, ON, Canada
| | | | | | - Omair Husain
- Centre for Addiction and Mental Health (CAMH), Toronto, ON, Canada
- Department of Psychiatry, University of Toronto, Toronto, ON, Canada
| | - Ishrat Husain
- Centre for Addiction and Mental Health (CAMH), Toronto, ON, Canada
- Department of Psychiatry, University of Toronto, Toronto, ON, Canada
| | - Muhammad Akhter Hamid
- Department of Paediatrics, University of Toronto, Toronto, ON, Canada
- Scarborough Health Network, Scarborough, ON, Canada
| | - Kwame McKenzie
- Centre for Addiction and Mental Health (CAMH), Toronto, ON, Canada
- Department of Psychiatry, University of Toronto, Toronto, ON, Canada
| | - Yuri Quintana
- Department of Medicine, Harvard University, Boston, MA, United States
- Beth Israel Deaconess Medical Center, Boston, MA, United States
| | - Farooq Naeem
- Centre for Addiction and Mental Health (CAMH), Toronto, ON, Canada
- Department of Psychiatry, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
7
|
Brönneke JB, Herr A, Reif S, Stern AD. Dynamic HTA for digital health solutions: opportunities and challenges for patient-centered evaluation. Int J Technol Assess Health Care 2023; 39:e72. [PMID: 37973549 DOI: 10.1017/s0266462323002726] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2023]
Abstract
OBJECTIVES Germany's 2019 Digital Healthcare Act (Digitale-Versorgung-Gesetz, or DVG) created a number of opportunities for the digital transformation of the healthcare delivery system. Key among these was the creation of a reimbursement pathway for patient-centered digital health applications (digitale Gesundheitsanwendungen, or DiGA). Worldwide, this is the first structured pathway for "prescribable" health applications at scale. As of October 10, 2023, 49 DiGA were listed in the official directory maintained by Germany's Federal Institute for Drugs and Medical Devices (BfArM); these are prescribable by physicians and psychotherapists and reimbursed by the German statutory health insurance system for all its 73 million beneficiaries. Looking ahead, a major challenge facing DiGA manufacturers will be the generation of the evidence required for ongoing price negotiations and reimbursement. Current health technology assessment (HTA) methods will need to be adapted for DiGA. METHODS We describe the core issues that distinguish HTA in this setting: (i) explicit allowance for more flexible research designs, (ii) the nature of initial evidence generation, which can be delivered (in its final form) up to one year after becoming reimbursable, and (iii) the dynamic nature of both product development and product evaluation. We present the digital health applications in the German DiGA scheme as a case study and highlight the role of RWE in the successful evaluation of DiGA on an ongoing basis. RESULTS When a DiGA is likely to be updated and assessed regularly, full-scale RCTs are infeasible; we therefore make the case for using real-world data and real-world evidence (RWE) for dynamic HTAs. CONCLUSIONS Continous evaluation using RWD is a regulatory innovation that can help improve the quality of DiGAs on the market.
Collapse
Affiliation(s)
- Jan B Brönneke
- Health Innovation Hub of the German Federal Ministry of Health, Berlin, Germany
| | - Annika Herr
- Leibniz University Hannover, Hannover, Germany
| | - Simon Reif
- ZEW - Leibniz Centre for European Economic Research, Mannheim, Germany
- University of Erlangen-Nürnberg, Nuremberg, Germany
| | - Ariel D Stern
- Harvard Business School, Health Innovation Hub of the German Federal Ministry of Health, Hasso Plattner Institute, Digital Health Cluster, Harvard-MIT Center for Regulatory Science, Boston, Massachusetts, USA
| |
Collapse
|
8
|
Mezei F, Horváth K, Pálfi M, Lovas K, Ádám I, Túri G. International practices in health technology assessment and public financing of digital health technologies: recommendations for Hungary. Front Public Health 2023; 11:1197949. [PMID: 37719722 PMCID: PMC10501404 DOI: 10.3389/fpubh.2023.1197949] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/31/2023] [Accepted: 08/18/2023] [Indexed: 09/19/2023] Open
Abstract
Background Evaluating and integrating digital health technologies is a critical component of a national healthcare ecosystem in the 2020s and is expected to even increase in significance. Design The paper gives an overview of international practices on public financing and health technology assessment of digital health technologies (DHTs) in five European Union (EU) countries and outlines recommendations for country-level action that relevant stakeholders can consider in order to support uptake of digital health solutions in Hungary. A scoping review was carried out to identify and gather country-specific classifications and international practices on the financing DHTs in five pioneering EU countries: Germany, France, Belgium, the United Kingdom and Finland. Results Several frameworks have been developed for DHTs, however there is no single, unified framework or method for classification, evaluation, and financing of digital health technologies in European context. European countries apply different taxonomy, use different assessment domains and regulations for the reimbursement of DHTs. The Working Group of the Hungarian Health Economic Society recommends eight specific points for stakeholders, importantly taking active role in shaping common clinical evidence standards and technical quality criteria across in order for common standards to be developed in the European Union single market. Conclusion Specificities of national healthcare contexts must be taken into account in decisions to allocate public funds to certain therapies rather than others.
Collapse
Affiliation(s)
- Fruzsina Mezei
- Data-Driven Health Division of National Laboratory for Health Security, Health Services Management Training Centre, Semmelweis University, Budapest, Hungary
- EIT Health France, Paris, France
| | - Krisztián Horváth
- Department of Public Health, Semmelweis University, Budapest, Hungary
| | - Máté Pálfi
- Center for Health Technology Assessment, Semmelweis University, Budapest, Hungary
| | - Kornélia Lovas
- CE Certiso Ltd, Budakeszi, Hungary
- Department of Clinical Pharmacy, University of Szeged, Szeged, Hungary
| | - Ildikó Ádám
- Center for Health Technology Assessment, Semmelweis University, Budapest, Hungary
| | - Gergő Túri
- Epidemiology and Surveillance Centre, Semmelweis University, Budapest, Hungary
- Synthesis Health Research Foundation, Budapest, Hungary
| |
Collapse
|
9
|
Paganini S, Meier E, Terhorst Y, Wurst R, Hohberg V, Schultchen D, Strahler J, Wursthorn M, Baumeister H, Messner EM. Stress Management Apps: Systematic Search and Multidimensional Assessment of Quality and Characteristics. JMIR Mhealth Uhealth 2023; 11:e42415. [PMID: 37642999 PMCID: PMC10498318 DOI: 10.2196/42415] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/02/2022] [Accepted: 06/06/2023] [Indexed: 08/31/2023] Open
Abstract
BACKGROUND Chronic stress poses risks for physical and mental well-being. Stress management interventions have been shown to be effective, and stress management apps (SMAs) might help to transfer strategies into everyday life. OBJECTIVE This review aims to provide a comprehensive overview of the quality and characteristics of SMAs to give potential users or health professionals a guideline when searching for SMAs in common app stores. METHODS SMAs were identified with a systematic search in the European Google Play Store and Apple App Store. SMAs were screened and checked according to the inclusion criteria. General characteristics and quality were assessed by 2 independent raters using the German Mobile Application Rating Scale (MARS-G). The MARS-G assesses quality (range 1 to 5) on the following four dimensions: (1) engagement, (2) functionality, (3) esthetics, and (4) information. In addition, the theory-based stress management strategies, evidence base, long-term availability, and common characteristics of the 5 top-rated SMAs were assessed and derived. RESULTS Of 2044 identified apps, 121 SMAs were included. Frequently implemented strategies (also in the 5 top-rated SMAs) were psychoeducation, breathing, and mindfulness, as well as the use of monitoring and reminder functions. Of the 121 SMAs, 111 (91.7%) provided a privacy policy, but only 44 (36.4%) required an active confirmation of informed consent. Data sharing with third parties was disclosed in only 14.0% (17/121) of the SMAs. The average quality of the included apps was above the cutoff score of 3.5 (mean 3.59, SD 0.50). The MARS-G dimensions yielded values above this cutoff score (functionality: mean 4.14, SD 0.47; esthetics: mean 3.76, SD 0.73) and below this score (information: mean 3.42, SD 0.46; engagement: mean 3.05, SD 0.78). Most theory-based stress management strategies were regenerative stress management strategies. The evidence base for 9.1% (11/121) of the SMAs could be identified, indicating significant group differences in several variables (eg, stress or depressive symptoms) in favor of SMAs. Moreover, 38.0% (46/121) of the SMAs were no longer available after a 2-year period. CONCLUSIONS The moderate information quality, scarce evidence base, constraints in data privacy and security features, and high volatility of SMAs pose challenges for users, health professionals, and researchers. However, owing to the scalability of SMAs and the few but promising results regarding their effectiveness, they have a high potential to reach and help a broad audience. For a holistic stress management approach, SMAs could benefit from a broader repertoire of strategies, such as more instrumental and mental stress management strategies. The common characteristics of SMAs with top-rated quality can be used as guidance for potential users and health professionals, but owing to the high volatility of SMAs, enhanced evaluation frameworks are needed.
Collapse
Affiliation(s)
- Sarah Paganini
- Department of Sport Psychology, Institute of Sports and Sport Science, University of Freiburg, Freiburg, Germany
| | - Evelyn Meier
- University of Education Freiburg, Freiburg, Germany
| | - Yannik Terhorst
- Clinical Psychology and Psychotherapy, Ulm University, Ulm, Germany
| | - Ramona Wurst
- Department of Sport Psychology, Institute of Sports and Sport Science, University of Freiburg, Freiburg, Germany
| | - Vivien Hohberg
- Department of Sport, Exercise and Health, Faculty of Medicine, University of Basel, Basel, Switzerland
| | - Dana Schultchen
- Clinical and Health Psychology, Ulm University, Ulm, Germany
| | - Jana Strahler
- Department of Sport Psychology, Institute of Sports and Sport Science, University of Freiburg, Freiburg, Germany
| | - Max Wursthorn
- Department of Public and Nonprofit Management, University of Freiburg, Freiburg, Germany
| | | | | |
Collapse
|
10
|
Gasteiger N, Dowding D, Norman G, McGarrigle L, Eost-Telling C, Jones D, Vercell A, Ali SM, O'Connor S. Conducting a systematic review and evaluation of commercially available mobile applications (apps) on a health-related topic: the TECH approach and a step-by-step methodological guide. BMJ Open 2023; 13:e073283. [PMID: 37308269 PMCID: PMC10277147 DOI: 10.1136/bmjopen-2023-073283] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/01/2023] [Accepted: 05/25/2023] [Indexed: 06/14/2023] Open
Abstract
OBJECTIVES To provide an overview of the methodological considerations for conducting commercial smartphone health app reviews (mHealth reviews), with the aim of systematising the process and supporting high-quality evaluations of mHealth apps. DESIGN Synthesis of our research team's experiences of conducting and publishing various reviews of mHealth apps available on app stores and hand-searching the top medical informatics journals (eg, The Lancet Digital Health, npj Digital Medicine, Journal of Biomedical Informatics and the Journal of the American Medical Informatics Association) over the last five years (2018-2022) to identify other app reviews to contribute to the discussion of this method and supporting framework for developing a research (review) question and determining the eligibility criteria. RESULTS We present seven steps to support rigour in conducting reviews of health apps available on the app market: (1) writing a research question or aims, (2) conducting scoping searches and developing the protocol, (3) determining the eligibility criteria using the TECH framework, (4) conducting the final search and screening of health apps, (5) data extraction, (6) quality, functionality and other assessments and (7) analysis and synthesis of findings. We introduce the novel TECH approach to developing review questions and the eligibility criteria, which considers the Target user, Evaluation focus, Connectedness and the Health domain. Patient and public involvement and engagement opportunities are acknowledged, including co-developing the protocol and undertaking quality or usability assessments. CONCLUSION Commercial mHealth app reviews can provide important insights into the health app market, including the availability of apps and their quality and functionality. We have outlined seven key steps for conducting rigorous health app reviews in addition to the TECH acronym, which can support researchers in writing research questions and determining the eligibility criteria. Future work will include a collaborative effort to develop reporting guidelines and a quality appraisal tool to ensure transparency and quality in systematic app reviews.
Collapse
Affiliation(s)
- Norina Gasteiger
- Division of Nursing, Midwifery and Social Work, The University of Manchester, Manchester, UK
- Division of Informatics, Imaging and Data Sciences, The University of Manchester, Manchester, UK
| | - Dawn Dowding
- Division of Nursing, Midwifery and Social Work, The University of Manchester, Manchester, UK
| | - Gill Norman
- Division of Nursing, Midwifery and Social Work, The University of Manchester, Manchester, UK
| | - Lisa McGarrigle
- Division of Nursing, Midwifery and Social Work, The University of Manchester, Manchester, UK
- Manchester Academic Health Science Centre, Manchester, UK
| | - Charlotte Eost-Telling
- Division of Nursing, Midwifery and Social Work, The University of Manchester, Manchester, UK
| | - Debra Jones
- Division of Nursing, Midwifery and Social Work, The University of Manchester, Manchester, UK
| | - Amy Vercell
- Division of Nursing, Midwifery and Social Work, The University of Manchester, Manchester, UK
- The Christie NHS Foundation Trust, Manchester, UK
| | - Syed Mustafa Ali
- Division of Informatics, Imaging and Data Sciences, The University of Manchester, Manchester, UK
| | - Siobhan O'Connor
- Division of Nursing, Midwifery and Social Work, The University of Manchester, Manchester, UK
| |
Collapse
|
11
|
Silberman J, Wicks P, Patel S, Sarlati S, Park S, Korolev IO, Carl JR, Owusu JT, Mishra V, Kaur M, Willey VJ, Sucala ML, Campellone TR, Geoghegan C, Rodriguez-Chavez IR, Vandendriessche B, Goldsack JC. Rigorous and rapid evidence assessment in digital health with the evidence DEFINED framework. NPJ Digit Med 2023; 6:101. [PMID: 37258851 DOI: 10.1038/s41746-023-00836-5] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/29/2022] [Accepted: 05/05/2023] [Indexed: 06/02/2023] Open
Abstract
Dozens of frameworks have been proposed to assess evidence for digital health interventions (DHIs), but existing frameworks may not facilitate DHI evidence reviews that meet the needs of stakeholder organizations including payers, health systems, trade organizations, and others. These organizations may benefit from a DHI assessment framework that is both rigorous and rapid. Here we propose a framework to assess Evidence in Digital health for EFfectiveness of INterventions with Evaluative Depth (Evidence DEFINED). Designed for real-world use, the Evidence DEFINED Quick Start Guide may help streamline DHI assessment. A checklist is provided summarizing high-priority evidence considerations in digital health. Evidence-to-recommendation guidelines are proposed, specifying degrees of adoption that may be appropriate for a range of evidence quality levels. Evidence DEFINED differs from prior frameworks in its inclusion of unique elements designed for rigor and speed. Rigor is increased by addressing three gaps in prior frameworks. First, prior frameworks are not adapted adequately to address evidence considerations that are unique to digital health. Second, prior frameworks do not specify evidence quality criteria requiring increased vigilance for DHIs in the current regulatory context. Third, extant frameworks rarely leverage established, robust methodologies that were developed for non-digital interventions. Speed is achieved in the Evidence DEFINED Framework through screening optimization and deprioritization of steps that may have limited value. The primary goals of Evidence DEFINED are to a) facilitate standardized, rapid, rigorous DHI evidence assessment in organizations and b) guide digital health solutions providers who wish to generate evidence that drives DHI adoption.
Collapse
Affiliation(s)
- Jordan Silberman
- Office of Medical Policy and Technology Assessment, Elevance Health, Palo Alto, CA, USA.
| | | | - Smit Patel
- Digital Medicine Society, Boston, MA, USA
| | - Siavash Sarlati
- Office of Medical Policy and Technology Assessment, Elevance Health, Palo Alto, CA, USA
- Department of Emergency Medicine, School of Medicine, University of California, San Francisco, CA, USA
| | - Siyeon Park
- Geisinger Health System, Danville, PA, USA
- Pharmesol, Inc., Newton, MA, USA
| | | | | | | | - Vimal Mishra
- Department of Medicine and Health Administration, Virginia Commonwealth University, Richmond, VA, USA
- UC Davis Health, Sacramento, CA, USA
| | - Manpreet Kaur
- Office of Medical Policy and Technology Assessment, Elevance Health, Palo Alto, CA, USA
| | | | | | | | - Cindy Geoghegan
- Digital Medicine Society, Boston, MA, USA
- Patients and Partners, LLC, Madison, CT, USA
| | | | - Benjamin Vandendriessche
- Department of Electrical, Computer and Systems Engineering, Case Western Reserve University, Cleveland, OH, USA
- Byteflies, Antwerp, Belgium
| | | |
Collapse
|
12
|
Cevasco KE, Roess AA. Adaptation and Utilization of a Postmarket Evaluation Model for Digital Contact Tracing Mobile Health Tools in the United States: Observational Cross-sectional Study. JMIR Public Health Surveill 2023; 9:e38633. [PMID: 36947135 PMCID: PMC10036112 DOI: 10.2196/38633] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/10/2022] [Revised: 01/13/2023] [Accepted: 01/19/2023] [Indexed: 03/23/2023] Open
Abstract
BACKGROUND Case investigation and contact tracing are core public health activities used to interrupt disease transmission. These activities are traditionally conducted manually. During periods of high COVID-19 incidence, US health departments were unable to scale up case management staff to deliver effective and timely contact-tracing services. In response, digital contact tracing (DCT) apps for mobile phones were introduced to automate these activities. DCT apps detect when other DCT users are close enough to transmit COVID-19 and enable alerts to notify users of potential disease exposure. These apps were deployed quickly during the pandemic without an opportunity to conduct experiments to determine effectiveness. However, it is unclear whether these apps can effectively supplement understaffed manual contact tracers. OBJECTIVE The aims of this study were to (1) evaluate the effectiveness of COVID-19 DCT apps deployed in the United States during the COVID-19 pandemic and (2) determine if there is sufficient DCT adoption and interest in adoption to meet a minimum population use rate to be effective (56%). To assess uptake, interest and safe use covariates were derived from evaluating DCTs using the American Psychological Association App Evaluation Model (AEM) framework. METHODS We analyzed data from a nationally representative survey of US adults about their COVID-19-related behaviors and experiences. Survey respondents were divided into three segments: those who adopted a DCT app, those who are interested but did not adopt, and those not interested. Descriptive statistics were used to characterize factors of the three groups. Multivariable logistic regression models were used to analyze the characteristics of segments adopting and interested in DCT apps against AEM framework covariates. RESULTS An insufficient percentage of the population adopted or was interested in DCTs to achieve our minimum national target effectiveness rate (56%). A total of 17.4% (n=490) of the study population reported adopting a DCT app, 24.7% (n=697) reported interest, and 58.0% (n=1637) were not interested. Younger, high-income, and uninsured individuals were more likely to adopt a DCT app. In contrast, people in fair to poor health were interested in DCT apps but did not adopt them. App adoption was positively associated with visiting friends and family outside the home (odds ratio [OR] 1.63, 95% CI 1.28-2.09), not wearing masks (OR 0.52, 95% CI 0.38-0.71), and adopters thinking they have or had COVID-19 (OR 1.60, 95% CI 1.21-2.12). CONCLUSIONS Overall, a small percentage of the population adopted DCT apps. These apps may not be effective in protecting adopters' friends and family from their maskless contacts outside the home given low adoption rates. The public health community should account for safe use behavioral factors in future public health contact-tracing app design. The AEM framework was useful in developing a study design to evaluate DCT effectiveness and safety.
Collapse
Affiliation(s)
- Kevin E Cevasco
- College of Health and Human Services, George Mason University, Fairfax, VA, United States
| | - Amira A Roess
- College of Health and Human Services, George Mason University, Fairfax, VA, United States
| |
Collapse
|
13
|
Carrandi A, Hayman M, Harrison CL. Safety considerations for assessing the quality of apps used during pregnancy: A scoping review. Digit Health 2023; 9:20552076231198683. [PMID: 37675058 PMCID: PMC10478559 DOI: 10.1177/20552076231198683] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2023] [Accepted: 08/15/2023] [Indexed: 09/08/2023] Open
Abstract
Objective Pregnant women are increasingly turning to apps targeting knowledge and behaviour change for supporting healthy lifestyles and managing medical conditions. Yet, there is growing concern over the credibility and safety of content within mobile health (mHealth) apps. This scoping review aimed to systematically and thematically consolidate safety considerations described in reviews evaluating pregnancy-specific apps. Methods PubMed, Ovid MEDLINE® and EPub, CINAHL, Web of Science, Cochrane Libraries, and SCOPUS were systematically searched to identify reviews that assessed apps targeting pregnant women. Data related to safety were extracted and thematically analysed to establish a set of relevant safety considerations. Results Sixteen reviews met the inclusion criteria. The included reviews assessed an average of 27 apps each and targeted pregnancy topics, such as nutrition and physical activity. Five major and 20 minor themes were identified, including information, transparency, credibility, privacy and security, and app tailoring. Information, transparency, and credibility relate to the evidence base of information within the app, privacy and security of apps relate to the protection of personal information and data, and app tailoring relates to the consideration of contextual factors, such as local guidelines and digital health literacy. Conclusions Results present possible safety considerations when evaluating pregnancy-specific apps and emphasise a clear need for consumer guidance on how to make informed decisions around engagement and use of mHealth apps during pregnancy.
Collapse
Affiliation(s)
- Alayna Carrandi
- Monash Centre for Health Research and Implementation, Monash University, Australia
| | | | - Cheryce L Harrison
- Monash Centre for Health Research and Implementation, Monash University, Australia
- Diabetes and Vascular Medicine, Monash Health, Australia
| |
Collapse
|
14
|
Mavragani A, Johnston FH, Campbell SL, Williamson GJ, Lucani C, Bowman DMJS, Cooling N, Jones PJ. Evaluating User Preferences, Comprehension, and Trust in Apps for Environmental Health Hazards: Qualitative Case Study. JMIR Form Res 2022; 6:e38471. [PMID: 36548030 PMCID: PMC9816954 DOI: 10.2196/38471] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2022] [Revised: 11/21/2022] [Accepted: 11/22/2022] [Indexed: 12/24/2022] Open
Abstract
BACKGROUND Climate change is projected to increase environmental health hazard risks through fire-related air pollution and increased airborne pollen levels. To protect vulnerable populations, it is imperative that evidence-based and accessible interventions are available. The environmental health app, AirRater, was developed in 2015 in Australia to provide information on multiple atmospheric health hazards in near real time. The app allows users to view local environmental conditions, and input and track their personal symptoms to enable behaviors that protect health in response to environmental hazards. OBJECTIVE This study aimed to develop insights into users' perceptions of engagement, comprehension, and trust in AirRater to inform the future development of environmental health apps. Specifically, this study explored which AirRater features users engaged with, what additional features or functionality needs users felt they required, users' self-perception of understanding app information, and their level of trust in the information provided. METHODS A total of 42 adult AirRater users were recruited from 3 locations in Australia to participate in semistructured interviews to capture location- or context-specific experiences. Participants were notified of the recruitment opportunity through multiple avenues including newsletter articles and social media. Informed consent was obtained before participation, and the participants were remunerated for their time and perspectives. A preinterview questionnaire collected data including age range, any preexisting conditions, and location (postcode). All participant data were deidentified. Interviews were recorded, transcribed, and analyzed using thematic analysis in NVivo 12 (QSR International). RESULTS Participants discussed app features and functionality, as well as their understanding of, and trust in, the information provided by the app. Most (26/42, 62%) participants used and valued visual environmental hazard features, especially maps, location settings, and hazard alerts. Most (33/42, 78%) found information in the app easy to understand and support their needs, irrespective of their self-reported literacy levels. Many (21/42, 50%) users reported that they did not question the accuracy of the data presented in the app. Suggested enhancements include the provision of meteorological information (eg, wind speed or direction, air pressure, UV rating, and humidity), functionality enhancements (eg, forecasting, additional alerts, and the inclusion of health advice), and clarification of existing information (eg, symptom triggers), including the capacity to download personal summary data for a specified period. CONCLUSIONS Participants' perspectives can inform the future development of environmental health apps. Specifically, participants' insights support the identification of key elements for the optimal development of environmental health app design, including streamlining, capacity for users to customize, use of real time data, visual cues, credibility, and accuracy of data. The results also suggest that, in the future, iterative collaboration between developers, environmental agencies, and users will likely promote better functional design, user trust in the data, and ultimately better population health outcomes.
Collapse
Affiliation(s)
| | - Fay H Johnston
- Menzies Institute for Medical Research, University of Tasmania, Hobart, Australia.,Public Health Services, Tasmanian Department of Health, Hobart, Australia
| | - Sharon L Campbell
- Menzies Institute for Medical Research, University of Tasmania, Hobart, Australia.,Public Health Services, Tasmanian Department of Health, Hobart, Australia
| | | | - Chris Lucani
- School of Natural Sciences, University of Tasmania, Hobart, Australia
| | | | - Nick Cooling
- School of Medicine, University of Tasmania, Hobart, Australia
| | - Penelope J Jones
- Menzies Institute for Medical Research, University of Tasmania, Hobart, Australia
| |
Collapse
|
15
|
Sampietro-Colom L, Fernandez-Barcelo C, Abbas I, Valdasquin B, Rabasseda N, García-Lorenzo B, Sanchez M, Sans M, Garcia N, Granados A. WtsWrng Interim Comparative Effectiveness Evaluation and Description of the Challenges to Develop, Assess, and Introduce This Novel Digital Application in a Traditional Health System. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2022; 19:13873. [PMID: 36360756 PMCID: PMC9654177 DOI: 10.3390/ijerph192113873] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/12/2022] [Revised: 10/21/2022] [Accepted: 10/22/2022] [Indexed: 06/16/2023]
Abstract
Science and technology have evolved quickly during the two decades of the 21st century, but healthcare systems are grounded in last century's structure and processes. Changes in the way health care is provided are demanded; digital transformation is a key driver making healthcare systems more accessible, agile, efficient, and citizen-centered. Nevertheless, the way healthcare systems function challenges the development (Innovation + Development and regulatory requirements), assessment (methodological guidance weaknesses), and adoption of digital applications (DAs). WtsWrng (WW), an innovative DA which uses images to interact with citizens for symptom triage and monitoring, is used as an example to show the challenges faced in its development and clinical validation and how these are being overcome. To prove WW's value from inception, novel approaches for evidence generation that allows for an agile and patient-centered development have been applied. Early scientific advice from NICE (UK) was sought for study design, an iterative development and interim analysis was performed, and different statistical parameters (Kappa, B statistic) were explored to face development and assessment challenges. WW triage accuracy at cutoff time ranged from 0.62 to 0.94 for the most frequent symptoms attending the Emergency Department (ED), with the observed concordance for the 12 most frequent diagnostics at hospital discharge fluctuating between 0.4 to 0.97; 8 of the diagnostics had a concordance greater than 0.8. This experience should provoke reflective thinking for DA developers, digital health scientists, regulators, health technology assessors, and payers.
Collapse
Affiliation(s)
- Laura Sampietro-Colom
- Assessment of Innovations and New Technologies Unit, Research and Innovation Directorate, Clínic Barcelona University Hospital, 08036 Barcelona, Spain
- Mangrana Ventures S.L., 08006 Barcelona, Spain
| | - Carla Fernandez-Barcelo
- Assessment of Innovations and New Technologies Unit, Research and Innovation Directorate, Clínic Barcelona University Hospital, 08036 Barcelona, Spain
| | - Ismail Abbas
- Assessment of Innovations and New Technologies Unit, Research and Innovation Directorate, Clínic Barcelona University Hospital, 08036 Barcelona, Spain
| | - Blanca Valdasquin
- Assessment of Innovations and New Technologies Unit, Research and Innovation Directorate, Clínic Barcelona University Hospital, 08036 Barcelona, Spain
| | | | - Borja García-Lorenzo
- Assessment of Innovations and New Technologies Unit, Research and Innovation Directorate, Clínic Barcelona University Hospital, 08036 Barcelona, Spain
- Kronikgune Institute for Health Sciences Research, 48902 Barakaldo, Spain
| | - Miquel Sanchez
- Emergency Department, Clínic Barcelona University Hospital, 08036 Barcelona, Spain
| | - Mireia Sans
- CAP Comte Borrell, Consorci Atenció Primaria Salut Barcelona Esquerra—CAPSBE, 08029 Barcelona, Spain
- Health 2.0 Section of the Col·Legi Oficial de Metges de Barcelona, 08017 Barcelona, Spain
| | - Noemi Garcia
- CAP Comte Borrell, Consorci Atenció Primaria Salut Barcelona Esquerra—CAPSBE, 08029 Barcelona, Spain
| | | |
Collapse
|
16
|
Segur-Ferrer J, Moltó-Puigmartí C, Pastells-Peiró R, Vivanco-Hidalgo RM. Methodological Frameworks and Dimensions to Be Taken Into Consideration in Digital Health Technology Assessment: Protocol for a Scoping Review. JMIR Res Protoc 2022; 11:e39905. [PMID: 36222788 PMCID: PMC9597420 DOI: 10.2196/39905] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/27/2022] [Revised: 07/15/2022] [Accepted: 07/31/2022] [Indexed: 12/02/2022] Open
Abstract
Background Health technology assessment (HTA) is one of the main tools that health systems have to appraise evidence and determine the value of a given health technology. Although the existing HTA frameworks are useful tools for the evaluation of a wide range of health technologies, more and more experts, organizations across the world, and HTA agencies are highlighting the need to update or develop specific methodological frameworks for the evaluation of digital health technologies in order to take into account additional domains that cover these technologies’ intrinsic characteristics. Objective The purpose of our scoping review is to identify the methodological frameworks that are used worldwide for the assessment of digital health technologies; determine what dimensions and aspects are being considered; and generate, through a thematic analysis, a proposal for a methodological framework that is based on the most frequently described dimensions in the literature. Methods The scoping review will be performed in accordance with the guidelines established in the updated statement of the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews). We will search for peer-reviewed and grey literature published between 2011 and the date of the search execution. The retrieved references will be reviewed in a single-blind manner by 2 independent authors, and their quality will be assessed by using the Critical Appraisal Skills Program tool. The ATLAS.ti software (Scientific Software Development GmbH) will be used for data extraction and to perform the thematic analysis. Results The scoping review is currently (May 2022) in progress. It is expected to be completed in October 2022, and the final results of the research will be presented and published by November 2022. Conclusions To our knowledge, no studies have been published to date that identify the existing methodological frameworks for digital HTA, determine which dimensions must be evaluated for correct decision-making, and serve as a basis for the development of a methodological framework of reference that health care systems can use to carry out this kind of assessment. This work is intended to address this knowledge gap of key relevance for the field of HTA. International Registered Report Identifier (IRRID) DERR1-10.2196/39905
Collapse
Affiliation(s)
- Joan Segur-Ferrer
- Health Technology Assessment and Quality of Care Area, Agency for Health Quality and Assessment of Catalonia, Barcelona, Spain
| | - Carolina Moltó-Puigmartí
- Health Technology Assessment and Quality of Care Area, Agency for Health Quality and Assessment of Catalonia, Barcelona, Spain
| | - Roland Pastells-Peiró
- Health Technology Assessment and Quality of Care Area, Agency for Health Quality and Assessment of Catalonia, Barcelona, Spain
| | - Rosa Maria Vivanco-Hidalgo
- Health Technology Assessment and Quality of Care Area, Agency for Health Quality and Assessment of Catalonia, Barcelona, Spain
| |
Collapse
|
17
|
Lantzsch H, Eckhardt H, Campione A, Busse R, Henschke C. Digital health applications and the fast-track pathway to public health coverage in Germany: challenges and opportunities based on first results. BMC Health Serv Res 2022; 22:1182. [PMID: 36131288 PMCID: PMC9490912 DOI: 10.1186/s12913-022-08500-6] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/23/2022] [Accepted: 08/23/2022] [Indexed: 11/29/2022] Open
Abstract
Objectives Evidence-based decision-making is the sine qua non for safe and effective patient care and the long-term functioning of health systems. Since 2020 Digital Health Applications (DiHA) in Germany have been undergoing a systematic pathway to be reimbursed by statutory health insurance (SHI) which is attracting attention in other European countries. We therefore investigate coverage decisions on DiHA and the underlying evidence on health care effects, which legally include both medical outcomes and patient-centred structural and procedural outcomes. Methods Based on publicly available data of the Institute for Medicines and Medical Devices searched between 08/2021 and 02/2022, all DiHA listed in the corresponding registry and thus reimbursable by the SHI were systematically investigated and presented descriptively on the basis of predefined criteria, such as clinical condition, and costs. The clinical trials on DiHA permanently included in the registry were reviewed with regard to their study design, endpoints investigated, the survey instruments used, and whether an intention-to-treat analysis was performed. Risk of bias was assessed using the ROB II tool. Results By February 2022, 30 DiHA had been included in the DiHA registry, one third of them permanently and two thirds conditionally. Most DiHA were therapeutic applications for mental illness based on cognitive behavioural therapy. For all permanently included DiHA, randomised controlled trials were conducted to demonstrate the impact on health care effects. While medical outcomes were investigated for all of these DiHA, patient-centred structural and procedural outcomes were rarely investigated. The majority of clinical trials showed a high risk of bias, mainly due to insufficient reporting quality. Overall, the prices for DiHA covered by SHI are on average around € 150 per month (min. € 40; max. € 248). Conclusions Evidence-based decision-making on coverage of DiHA leaves room for improvements both in terms of reporting-quality and the use of patient-centred structural and procedural outcomes in addition to medical outcomes. With appropriate evidence, DiHA can offer an opportunity as an adjunct to existing therapy while currently the high risk of bias of the trials raises doubts about the justification of its high costs. Supplementary Information The online version contains supplementary material available at 10.1186/s12913-022-08500-6.
Collapse
Affiliation(s)
- Hendrikje Lantzsch
- Department of Health Care Management, Technische Universität Berlin, Straße des 17. Juni 135, 10623, Berlin, Germany.
| | - Helene Eckhardt
- Department of Health Care Management, Technische Universität Berlin, Straße des 17. Juni 135, 10623, Berlin, Germany
| | - Alessandro Campione
- Department of Health Care Management, Technische Universität Berlin, Straße des 17. Juni 135, 10623, Berlin, Germany
| | - Reinhard Busse
- Department of Health Care Management, Technische Universität Berlin, Straße des 17. Juni 135, 10623, Berlin, Germany.,Technische Universität Berlin, Berlin Centre of Health Economics Research, Berlin, Germany
| | - Cornelia Henschke
- Department of Health Care Management, Technische Universität Berlin, Straße des 17. Juni 135, 10623, Berlin, Germany.,Technische Universität Berlin, Berlin Centre of Health Economics Research, Berlin, Germany
| |
Collapse
|
18
|
Tarricone R, Petracca F, Cucciniello M, Ciani O. Recommendations for developing a lifecycle, multidimensional assessment framework for mobile medical apps. HEALTH ECONOMICS 2022; 31 Suppl 1:73-97. [PMID: 35388585 PMCID: PMC9545972 DOI: 10.1002/hec.4505] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Received: 08/06/2021] [Revised: 02/18/2022] [Accepted: 03/02/2022] [Indexed: 06/14/2023]
Abstract
Digital health and mobile medical apps (MMAs) have shown great promise in transforming health care, but their adoption in clinical care has been unsatisfactory, and regulatory guidance and coverage decisions have been lacking or incomplete. A multidimensional assessment framework for regulatory, policymaking, health technology assessment, and coverage purposes based on the MMA lifecycle is needed. A targeted review of relevant policy documents from international sources was conducted to map current MMA assessment frameworks, to formulate 10 recommendations, subsequently shared amongst an expert panel of key stakeholders. Recommendations go beyond economic dimensions such as cost and economic evaluation and also include MMA development and update, classification and evidentiary requirements, performance and maintenance monitoring, usability testing, clinical evidence requirements, safety and security, equity considerations, organizational assessment, and additional outcome domains (patient empowerment and environmental impact). The COVID-19 pandemic greatly expanded the use of MMAs, but temporary policies governing their use and oversight need consolidation through well-developed frameworks to support decision-makers, producers and introduction into clinical care processes, especially in light of the strong international, cross-border character of MMAs, the new EU medical device and health technology assessment regulations, and the Next Generation EU funding earmarked for health digitalization.
Collapse
Affiliation(s)
- Rosanna Tarricone
- Centre for Research in Health and Social Care Management (CERGAS)Government, Health and Non Profit DivisionSDA BocconiMilanItaly
- Department of Social and Political SciencesBocconi UniversityMilanItaly
| | - Francesco Petracca
- Centre for Research in Health and Social Care Management (CERGAS)Government, Health and Non Profit DivisionSDA BocconiMilanItaly
| | - Maria Cucciniello
- Department of Social and Political SciencesBocconi UniversityMilanItaly
- University of Edinburgh Business SchoolScotlandUK
| | - Oriana Ciani
- Centre for Research in Health and Social Care Management (CERGAS)Government, Health and Non Profit DivisionSDA BocconiMilanItaly
- Institute of Health ResearchUniversity of Exeter Medical SchoolExeterUK
| |
Collapse
|
19
|
Implementation of a new Digi-HTA process for digital health technologies in Finland. Int J Technol Assess Health Care 2022; 38:e68. [PMID: 35983625 DOI: 10.1017/s0266462322000502] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
OBJECTIVES There has been a lack of health technology assessment (HTA) methods for novel digital health technologies (DHTs) such as mHealth, artificial intelligence, and robotics in Finland. The Digi-HTA method has been developed for this purpose. The aim of this study is to determine whether it would be possible to use Digi-HTA recommendations to support healthcare decision-makers. Secondly, from the perspective of companies offering different types of DHT products, this study assesses the suitability of using the Digi-HTA framework to perform HTAs for their products. METHODS Feedback about Digi-HTA recommendations was collected from healthcare professionals. DHT companies provided input about the Digi-HTA framework. Data were collected via a web-based survey and were analyzed using qualitative methods. RESULTS Of the twenty-four healthcare professional respondents, twenty said that the Digi-HTA recommendations contained all the necessary information, and twenty-one found them useful for their work. Respondents hoped that the Digi-HTA recommendations would be better integrated into the decision-making processes and healthcare professionals would be more informed about this new HTA process. The questions of the Digi-HTA framework were applicable for different DHT products based on the responses from DHT companies (n = 8). CONCLUSIONS According to the study participants, although the Digi-HTA recommendations include clear and beneficial information, their integration into healthcare decision-making processes should be improved. Responses from DHT companies indicate that the Digi-HTA framework would be an appropriate tool for performing assessments for their products. To generalize the findings of this study, more comprehensive studies will be needed.
Collapse
|
20
|
Ágh T, Hadžiabdić MO, Garuoliene K, Granas AG, Aarnio E, Menditto E, Gregório J, Barnestein-Fonseca P, Mevsim V, Kardas P. Reimbursed Medication Adherence Enhancing Interventions in European Countries: Results of the EUREcA Study. Front Pharmacol 2022; 13:892240. [PMID: 35784711 PMCID: PMC9247400 DOI: 10.3389/fphar.2022.892240] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/08/2022] [Accepted: 05/12/2022] [Indexed: 11/13/2022] Open
Abstract
Introduction: Current literature lacks detailed understanding of the reimbursement framework of medication adherence enhancing interventions (MAEIs). As part of the ENABLE COST Action, the EUREcA (“EUropen REimbursement strategies for interventions targeting medication Adherence”) study aimed to provide an in-depth overview of reimbursed MAEIs currently available in European countries at national and regional levels and to pave the way for further MAEIs to be implemented in the future. Methods: A web-based, cross-sectional survey was performed across 38 European countries and Israel. The survey questionnaire was developed as a result of an iterative process of discussion informed by a desk review. The survey was performed among invited ENABLE collaborators from June to July 2021. Besides descriptive analysis, association between country income and health care expenditure, and the availability of reimbursed MAEIs were also assessed. Results: The survey identified 13 reimbursed MAEIs in nine countries: multi-dose drug dispensing (n = 5), medication review (n = 4), smart device (n = 2), mobile application (n = 1), and patient education (n = 1). The median GDP per capita of countries having ≥1 reimbursed MAEI was significantly higher compared to countries having no reimbursed adherence intervention (33,888 EUR vs 16,620 EUR, respectively; p = 0.05). Conclusions: Our findings highlight that to date only a small number of MAEIs have been reimbursed in European countries. Comprehensive health technology assessment recommendations and multi-stakeholder collaboration could help removing barriers related to the implementation and reimbursement of MAEIs.
Collapse
Affiliation(s)
- Tamás Ágh
- Syreon Research Institute, Budapest, Hungary
- Center for Health Technology Assessment and Pharmacoeconomic Research, University of Pécs, Pécs, Hungary
- *Correspondence: Tamás Ágh,
| | - Maja Ortner Hadžiabdić
- Centre for Applied Pharmacy, Faculty of Pharmacy and Biochemistry, University of Zagreb, Zagreb, Croatia
| | - Kristina Garuoliene
- Pharmacy Center, Institute of Biomedical Science, Faculty of Medicine, Vilnius University, Vilnius, Lithuania
| | - Anne Gerd Granas
- Section for Pharmaceutics and Social Pharmacy, Department of Pharmacy, University of Oslo, Oslo, Norway
- Norwegian Centre for E-health Research, University Hospital of North Norway, Tromsø, Norway
| | - Emma Aarnio
- School of Pharmacy, University of Eastern Finland, Kuopio, Finland
| | - Enrica Menditto
- CIRFF, Center of Pharmacoeconomics and Drug Utilization Research, Department of Pharmacy University of Naples Federico II, Naples, Italy
| | - João Gregório
- CBIOS, Universidade Lusófona’s Research Center for Biosciences & Health Technologies, Lisboa, Portugal
| | - Pilar Barnestein-Fonseca
- CUDECA Institute for Training and Research in Palliative Care, CUDECA Hospice Foundation, Málaga, Spain
| | - Vildan Mevsim
- Department of Family Medicine, Faculty of Medicine, Dokuz Eylul University, Izmir, Turkey
| | - Przemysław Kardas
- Medication Adherence Research Centre, Department of Family Medicine, Medical University of Lodz, Lodz, Poland
| | | |
Collapse
|
21
|
Maaß L, Freye M, Pan CC, Dassow HH, Niess J, Jahnel T. Definitions of Health Apps & Medical Apps in the Perspectives of Public Health and Law: Qualitative Analysis of an Interdisciplinary Literature Overview (Preprint). JMIR Mhealth Uhealth 2022; 10:e37980. [DOI: 10.2196/37980] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/14/2022] [Revised: 05/30/2022] [Accepted: 06/30/2022] [Indexed: 11/13/2022] Open
|
22
|
De Santis KK, Jahnel T, Mergenthal L, Zeeb H, Matthias K. Evaluation of Digital Interventions for Physical Activity Promotion: Protocol for a Scoping Review. JMIR Res Protoc 2022; 11:e35332. [PMID: 35238321 PMCID: PMC8931641 DOI: 10.2196/35332] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/01/2021] [Revised: 01/12/2022] [Accepted: 02/03/2022] [Indexed: 12/24/2022] Open
Abstract
Background Digital interventions (DIs) could support physical activity (PA) promotion, according to recent reviews. However, it remains unclear if and how DIs for PA promotion are evaluated; thus, it is unclear if they support behavior change in real-world settings. A mapping of evidence from published reviews is required to focus on the evaluation of DIs for PA promotion. Objective The aim of our study is to investigate evaluation strategies for any outcome in the context of DIs for PA promotion by conducting a scoping review of published reviews. Methods Our scoping review adheres to the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines. The information sources include bibliographic databases (MEDLINE, PsycINFO, and CINAHL) and the bibliographies of the selected studies. The electronic search strategy was developed and conducted in collaboration with an experienced database specialist. The electronic search was conducted in English with no limits up to March 19, 2021, for sources with the terms digital intervention AND evaluation AND physical activity in titles or abstracts. After deduplication, 300 reviews selected from 4912 search results were assessed for eligibility by 2 authors working independently. The inclusion criteria were (1) healthy or clinical samples (population), (2) DIs for PA promotion (intervention), (3) comparisons to any other intervention or no intervention (comparison), (4) evaluation strategies (methods, results, or frameworks) for any outcome in the context of DIs for PA promotion (outcome), and (5) any published review (study type). According to the consensus reached during a discussion, 40 reviews met the inclusion criteria—36 from the electronic search and 4 from the manual search of the bibliographies of the 36 reviews. All reviews reported the evaluation strategies for any outcomes in the context of DIs for PA promotion in healthy or clinical samples. Data coding and the quality appraisal of systematic reviews are currently being performed independently by 2 authors. Results Our scoping review includes data from 40 published reviews (1 rapid review, 9 scoping reviews, and 30 systematic reviews). The focus of data coding is on evaluation strategies in the context of DIs for PA promotion and on the critical appraisal of the included systematic reviews. The final consensus regarding all data is expected in early 2022. Conclusions Interventions for PA promotion that are supported by digital technologies require evaluation to ensure their efficacy in real-world settings. Our scoping review is needed because it addresses novel objectives that focus on such evaluations and are not answered in the published reviews identified in our search. The evaluation strategies addressing DIs for PA promotion will be mapped to synthesize the results that have been reported in published reviews so far. International Registered Report Identifier (IRRID) DERR1-10.2196/35332
Collapse
Affiliation(s)
- Karina Karolina De Santis
- Department of Prevention and Evaluation, Leibniz Institute for Prevention Research and Epidemiology - BIPS, Bremen, Germany
| | - Tina Jahnel
- Department of Prevention and Evaluation, Leibniz Institute for Prevention Research and Epidemiology - BIPS, Bremen, Germany.,Faculty 11 Human and Health Sciences, University of Bremen, Bremen, Germany
| | - Lea Mergenthal
- Department of Prevention and Evaluation, Leibniz Institute for Prevention Research and Epidemiology - BIPS, Bremen, Germany
| | - Hajo Zeeb
- Department of Prevention and Evaluation, Leibniz Institute for Prevention Research and Epidemiology - BIPS, Bremen, Germany
| | - Katja Matthias
- Faculty of Electrical Engineering and Computer Science, University of Applied Science Stralsund, Stralsund, Germany
| |
Collapse
|
23
|
Ramos G, Ponting C, Labao JP, Sobowale K. Considerations of diversity, equity, and inclusion in mental health apps: A scoping review of evaluation frameworks. Behav Res Ther 2021; 147:103990. [PMID: 34715396 DOI: 10.1016/j.brat.2021.103990] [Citation(s) in RCA: 41] [Impact Index Per Article: 13.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/16/2021] [Revised: 09/26/2021] [Accepted: 10/12/2021] [Indexed: 01/02/2023]
Abstract
Mental health applications or apps are a promising approach to reduce the pervasive unmet mental health need observed among marginalized groups. However, these groups face unique challenges to engage in and benefit from these interventions, and apps that consider issues of diversity, equity, and inclusion (DEI) may be better positioned to reach them. App evaluation frameworks emerge as crucial tools for researchers, clinicians, and users to select evidence-based apps. Nevertheless, it is unclear the extent to which existing assessment tools capture DEI factors. The present scoping review identified 68 studies that employed one or more assessment tools to evaluate a mental health app, leading to 44 unique app evaluation frameworks. Results showed that most frameworks were developed after 2015, and only 58% of them considered at least one DEI criterion. Frameworks that performed exceptionally well were those that combined multiple standardized and validated measures. This lack of consideration for DEI variables may limit the ability of app-based interventions to serve marginalized communities, or even worse, create new disparities. Therefore, we provide recommendations to improve current app evaluation frameworks' cultural robustness and clinical utility, maximizing their effectiveness when working with individuals from marginalized communities.
Collapse
Affiliation(s)
- Giovanni Ramos
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA
| | - Carolyn Ponting
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA
| | - Jerome P Labao
- University of California, Los Angeles, Los Angeles, CA, USA
| | - Kunmi Sobowale
- University of California, Los Angeles-Semel Institute for Neuroscience and Human Behavior, Los Angeles, CA, USA.
| |
Collapse
|
24
|
Woulfe F, Fadahunsi KP, Smith S, Chirambo GB, Larsson E, Henn P, Mawkin M, O' Donoghue J. Identification and Evaluation of Methodologies to Assess the Quality of Mobile Health Apps in High-, Low-, and Middle-Income Countries: Rapid Review. JMIR Mhealth Uhealth 2021; 9:e28384. [PMID: 34636737 PMCID: PMC8548973 DOI: 10.2196/28384] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/03/2021] [Revised: 06/22/2021] [Accepted: 07/21/2021] [Indexed: 12/24/2022] Open
Abstract
BACKGROUND In recent years, there has been rapid growth in the availability and use of mobile health (mHealth) apps around the world. A consensus regarding an accepted standard to assess the quality of such apps has yet to be reached. A factor that exacerbates the challenge of mHealth app quality assessment is variations in the interpretation of quality and its subdimensions. Consequently, it has become increasingly difficult for health care professionals worldwide to distinguish apps of high quality from those of lower quality. This exposes both patients and health care professionals to unnecessary risks. Despite progress, limited understanding of the contributions of researchers in low- and middle-income countries (LMICs) exists on this topic. Furthermore, the applicability of quality assessment methodologies in LMIC settings remains relatively unexplored. OBJECTIVE This rapid review aims to identify current methodologies in the literature to assess the quality of mHealth apps, understand what aspects of quality these methodologies address, determine what input has been made by authors from LMICs, and examine the applicability of such methodologies in LMICs. METHODS This review was registered with PROSPERO (International Prospective Register of Systematic Reviews). A search of PubMed, EMBASE, Web of Science, and Scopus was performed for papers related to mHealth app quality assessment methodologies, which were published in English between 2005 and 2020. By taking a rapid review approach, a thematic and descriptive analysis of the papers was performed. RESULTS Electronic database searches identified 841 papers. After the screening process, 52 papers remained for inclusion. Of the 52 papers, 5 (10%) proposed novel methodologies that could be used to evaluate mHealth apps of diverse medical areas of interest, 8 (15%) proposed methodologies that could be used to assess apps concerned with a specific medical focus, and 39 (75%) used methodologies developed by other published authors to evaluate the quality of various groups of mHealth apps. The authors in 6% (3/52) of papers were solely affiliated to institutes in LMICs. A further 15% (8/52) of papers had at least one coauthor affiliated to an institute in an LMIC. CONCLUSIONS Quality assessment of mHealth apps is complex in nature and at times subjective. Despite growing research on this topic, to date, an all-encompassing appropriate means for evaluating the quality of mHealth apps does not exist. There has been engagement with authors affiliated to institutes across LMICs; however, limited consideration of current generic methodologies for application in LMIC settings has been identified. TRIAL REGISTRATION PROSPERO CRD42020205149; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=205149.
Collapse
Affiliation(s)
- Fionn Woulfe
- School of Medicine, University College Cork, Cork, Ireland
| | - Kayode Philip Fadahunsi
- Department of Primary Care and Public Health, Imperial College London, London, United Kingdom
| | - Simon Smith
- School of Medicine, University College Cork, Cork, Ireland
| | | | - Emma Larsson
- Ashford and St Peter's Hospitals NHS Trust, Chertsey, United Kingdom
| | - Patrick Henn
- Assert Centre, College of Medicine & Health, University College Cork, Cork, Ireland
| | - Mala Mawkin
- School of Medicine, Imperial College London, London, United Kingdom
| | - John O' Donoghue
- Malawi eHealth Research Centre, University College Cork, College Road, Cork, Ireland
| |
Collapse
|
25
|
Unsworth H, Dillon B, Collinson L, Powell H, Salmon M, Oladapo T, Ayiku L, Shield G, Holden J, Patel N, Campbell M, Greaves F, Joshi I, Powell J, Tonnel A. The NICE Evidence Standards Framework for digital health and care technologies - Developing and maintaining an innovative evidence framework with global impact. Digit Health 2021; 7:20552076211018617. [PMID: 34249371 PMCID: PMC8236783 DOI: 10.1177/20552076211018617] [Citation(s) in RCA: 32] [Impact Index Per Article: 10.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2020] [Accepted: 04/24/2021] [Indexed: 11/29/2022] Open
Abstract
Objective In 2018, the UK National Institute for Health and Care Excellence (NICE), in partnership with Public Health England, NHS England, NHS Improvement and others, developed an evidence standards framework (ESF) for digital health and care technologies (DHTs). The ESF was designed to provide a standardised approach to guide developers and commissioners on the levels of evidence needed for the clinical and economic evaluation of DHTs by health and care systems. Methods The framework was developed using an agile and iterative methodology that included a literature review of existing initiatives and comparison of these against the requirements set by NHS England; iterative consultation with stakeholders through an expert working group and workshops; and questionnaire-based stakeholder input on a publicly available draft document. Results The evidence standards framework has been well-received and to date the ESF has been viewed online over 55,000 times and downloaded over 19,000 times. Conclusions In April 2021 we published an update to the ESF. Here, we summarise the process through which the ESF was developed, reflect on its global impact to date, and describe NICE’s ongoing work to maintain and improve the framework in the context for a fast moving, innovative field.
Collapse
Affiliation(s)
| | - Bernice Dillon
- Centre for Health Technology Evaluation, NICE, Manchester, UK
| | - Lucie Collinson
- Centre for Health Technology Evaluation, NICE, Manchester, UK
| | - Helen Powell
- Centre for Health Technology Evaluation, NICE, Manchester, UK
| | - Mark Salmon
- Digital, Information and Technology Directorate, NICE, Manchester, UK
| | - Tosin Oladapo
- Centre for Health Technology Evaluation, NICE, Manchester, UK
| | - Lynda Ayiku
- Digital, Information and Technology Directorate, NICE, Manchester, UK
| | - Gary Shield
- Health and Social Care Directorate, NICE, Manchester, UK
| | - Joanne Holden
- Centre for Health Technology Evaluation, NICE, Manchester, UK
| | | | - Mark Campbell
- Centre for Health Technology Evaluation, NICE, Manchester, UK
| | - Felix Greaves
- Science, Evidence and Analytics Directorate, NICE, Manchester, UK
| | | | - John Powell
- Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK
| | - Alexia Tonnel
- Digital, Information and Technology Directorate, NICE, Manchester, UK
| |
Collapse
|
26
|
Health technology assessment for digital technologies that manage chronic disease: a systematic review. Int J Technol Assess Health Care 2021; 37:e66. [PMID: 34034851 DOI: 10.1017/s0266462321000362] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
OBJECTIVE A growing number of evaluation frameworks have emerged over recent years addressing the unique benefits and risk profiles of new classes of digital health technologies (DHTs). This systematic review aims to identify relevant frameworks and synthesize their recommendations into DHT-specific content to be considered when performing Health Technology Assessments (HTAs) for DHTs that manage chronic noncommunicable disease at home. METHODS Searches were undertaken of Medline, Embase, Econlit, CINAHL, and The Cochrane Library (January 2015 to March 2020), and relevant gray literature (January 2015 to August 2020) using keywords related to HTA, evaluation frameworks, and DHTs. Included framework reference lists were searched from 2010 until 2015. The EUNetHTA HTA Core Model version 3.0 was selected as a scaffold for content evaluation. RESULTS Forty-four frameworks were identified, mainly covering clinical effectiveness (n = 30) and safety (n = 23) issues. DHT-specific content recommended by framework authors fell within 28 of the 145 HTA Core Model issues. A further twenty-two DHT-specific issues not currently in the HTA Core Model were recommended. CONCLUSIONS Current HTA frameworks are unlikely to be sufficient for assessing DHTs. The development of DHT-specific content for HTA frameworks is hampered by DHTs having varied benefit and risk profiles. By focusing on DHTs that actively monitor/treat chronic noncommunicable diseases at home, we have extended DHT-specific content to all nine HTA Core Model domains. We plan to develop a supplementary evaluation framework for designing research studies, undertaking HTAs, and appraising the completeness of HTAs for DHTs.
Collapse
|
27
|
Hensher M, Cooper P, Dona SWA, Angeles MR, Nguyen D, Heynsbergh N, Chatterton ML, Peeters A. Scoping review: Development and assessment of evaluation frameworks of mobile health apps for recommendations to consumers. J Am Med Inform Assoc 2021; 28:1318-1329. [PMID: 33787894 PMCID: PMC8263081 DOI: 10.1093/jamia/ocab041] [Citation(s) in RCA: 27] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2020] [Revised: 02/12/2021] [Accepted: 03/03/2021] [Indexed: 02/06/2023] Open
Abstract
Objective The study sought to review the different assessment items that have been used within existing health app evaluation frameworks aimed at individual, clinician, or organizational users, and to analyze the scoring and evaluation methods used in these frameworks. Materials and Methods We searched multiple bibliographic databases and conducted backward searches of reference lists, using search terms that were synonyms of “health apps,” “evaluation,” and “frameworks.” The review covered publications from 2011 to April 2020. Studies on health app evaluation frameworks and studies that elaborated on the scaling and scoring mechanisms applied in such frameworks were included. Results Ten common domains were identified across general health app evaluation frameworks. A list of 430 assessment criteria was compiled across 97 identified studies. The most frequently used scaling mechanism was a 5-point Likert scale. Most studies have adopted summary statistics to generate the total scoring of each app, and the most popular approach taken was the calculation of mean or average scores. Other frameworks did not use any scaling or scoring mechanism and adopted criteria-based, pictorial, or descriptive approaches, or “threshold” filter. Discussion There is wide variance in the approaches to evaluating health apps within published frameworks, and this variance leads to ongoing uncertainty in how to evaluate health apps. Conclusions A new evaluation framework is needed that can integrate the full range of evaluative criteria within one structure, and provide summative guidance on health app rating, to support individual app users, clinicians, and health organizations in choosing or recommending the best health app.
Collapse
Affiliation(s)
- Martin Hensher
- Institute for Health Transformation, Faculty of Health, Deakin University, Geelong, Victoria, Australia.,Deakin Health Economics, School of Health and Social Development, Faculty of Health, Deakin University, Burwood, Victoria, Australia
| | - Paul Cooper
- Institute for Health Transformation, Faculty of Health, Deakin University, Geelong, Victoria, Australia.,Deakin Health Economics, School of Health and Social Development, Faculty of Health, Deakin University, Burwood, Victoria, Australia
| | - Sithara Wanni Arachchige Dona
- Institute for Health Transformation, Faculty of Health, Deakin University, Geelong, Victoria, Australia.,Deakin Health Economics, School of Health and Social Development, Faculty of Health, Deakin University, Burwood, Victoria, Australia
| | - Mary Rose Angeles
- Institute for Health Transformation, Faculty of Health, Deakin University, Geelong, Victoria, Australia.,Deakin Health Economics, School of Health and Social Development, Faculty of Health, Deakin University, Burwood, Victoria, Australia
| | - Dieu Nguyen
- Institute for Health Transformation, Faculty of Health, Deakin University, Geelong, Victoria, Australia.,Deakin Health Economics, School of Health and Social Development, Faculty of Health, Deakin University, Burwood, Victoria, Australia
| | - Natalie Heynsbergh
- Institute for Health Transformation, Faculty of Health, Deakin University, Geelong, Victoria, Australia.,School of Nursing and Midwifery, Faculty of Health, Deakin University, Geelong, Victoria, Australia
| | - Mary Lou Chatterton
- Institute for Health Transformation, Faculty of Health, Deakin University, Geelong, Victoria, Australia.,Deakin Health Economics, School of Health and Social Development, Faculty of Health, Deakin University, Burwood, Victoria, Australia
| | - Anna Peeters
- Institute for Health Transformation, Faculty of Health, Deakin University, Geelong, Victoria, Australia
| |
Collapse
|
28
|
Lagan S, Sandler L, Torous J. Evaluating evaluation frameworks: a scoping review of frameworks for assessing health apps. BMJ Open 2021; 11:e047001. [PMID: 33741674 PMCID: PMC7986656 DOI: 10.1136/bmjopen-2020-047001] [Citation(s) in RCA: 41] [Impact Index Per Article: 13.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/17/2020] [Revised: 03/06/2021] [Accepted: 03/10/2021] [Indexed: 12/25/2022] Open
Abstract
OBJECTIVES Despite an estimated 300 000 mobile health apps on the market, there remains no consensus around helping patients and clinicians select safe and effective apps. In 2018, our team drew on existing evaluation frameworks to identify salient categories and create a new framework endorsed by the American Psychiatric Association (APA). We have since created a more expanded and operational framework Mhealth Index and Navigation Database (MIND) that aligns with the APA categories but includes objective and auditable questions (105). We sought to survey the existing space, conducting a review of all mobile health app evaluation frameworks published since 2018, and demonstrate the comprehensiveness of this new model by comparing it to existing and emerging frameworks. DESIGN We conducted a scoping review of mobile health app evaluation frameworks. DATA SOURCES References were identified through searches of PubMed, EMBASE and PsychINFO with publication date between January 2018 and October 2020. ELIGIBILITY CRITERIA Papers were selected for inclusion if they meet the predetermined eligibility criteria-presenting an evaluation framework for mobile health apps with patient, clinician or end user-facing questions. DATA EXTRACTION AND SYNTHESIS Two reviewers screened the literature separately and applied the inclusion criteria. The data extracted from the papers included: author and dates of publication, source affiliation, country of origin, name of framework, study design, description of framework, intended audience/user and framework scoring system. We then compiled a collection of more than 1701 questions across 79 frameworks. We compared and grouped these questions using the MIND framework as a reference. We sought to identify the most common domains of evaluation while assessing the comprehensiveness and flexibility-as well as any potential gaps-of MIND. RESULTS New app evaluation frameworks continue to emerge and expand. Since our 2019 review of the app evaluation framework space, more frameworks include questions around privacy (43) and clinical foundation (57), reflecting an increased focus on issues of app security and evidence base. The majority of mapped frameworks overlapped with at least half of the MIND categories. The results of this search have informed a database (apps.digitalpsych.org) that users can access today. CONCLUSION As the number of app evaluation frameworks continues to rise, it is becoming difficult for users to select both an appropriate evaluation tool and to find an appropriate health app. This review provides a comparison of what different app evaluation frameworks are offering, where the field is converging and new priorities for improving clinical guidance.
Collapse
Affiliation(s)
- Sarah Lagan
- Division of DIgital Psychaitry, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts, USA
| | - Lev Sandler
- Division of DIgital Psychaitry, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts, USA
| | - John Torous
- Division of DIgital Psychaitry, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts, USA
| |
Collapse
|
29
|
Lewis M, Palmer VJ, Kotevski A, Densley K, O'Donnell ML, Johnson C, Wohlgezogen F, Gray K, Robins-Browne K, Burchill L. Rapid Design and Delivery of an Experience-Based Co-designed Mobile App to Support the Mental Health Needs of Health Care Workers Affected by the COVID-19 Pandemic: Impact Evaluation Protocol. JMIR Res Protoc 2021; 10:e26168. [PMID: 33635823 PMCID: PMC7945974 DOI: 10.2196/26168] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/02/2020] [Revised: 01/10/2021] [Accepted: 02/25/2021] [Indexed: 01/19/2023] Open
Abstract
BACKGROUND The COVID-19 pandemic has highlighted the importance of health care workers' mental health and well-being for the successful function of the health care system. Few targeted digital tools exist to support the mental health of hospital-based health care workers, and none of them appear to have been led and co-designed by health care workers. OBJECTIVE RMHive is being led and developed by health care workers using experience-based co-design (EBCD) processes as a mobile app to support the mental health challenges posed by the COVID-19 pandemic to health care workers. We present a protocol for the impact evaluation for the rapid design and delivery of the RMHive mobile app. METHODS The impact evaluation will adopt a mixed methods design. Qualitative data from photo interviews undertaken with up to 30 health care workers and semistructured interviews conducted with up to 30 governance stakeholders will be integrated with qualitative and quantitative user analytics data and user-generated demographic and mental health data entered into the app. Analyses will address three evaluation questions related to engagement with the mobile app, implementation and integration of the app, and the impact of the app on individual mental health outcomes. The design and development will be described using the Mobile Health Evidence Reporting and Assessment guidelines. Implementation of the app will be evaluated using normalization process theory to analyze qualitative data from interviews combined with text and video analysis from the semistructured interviews. Mental health impacts will be assessed using the total score of the 4-item Patient Health Questionnaire (PHQ4) and subscale scores for the 2-item Patient Health Questionnaire for depression and the 2-item Generalized Anxiety Scale for anxiety. The PHQ4 will be completed at baseline and at 14 and 28 days. RESULTS The anticipated average use period of the app is 30 days. The rapid design will occur over four months using EBCD to collect qualitative data and develop app content. The impact evaluation will monitor outcome data for up to 12 weeks following hospital-wide release of the minimal viable product release. The study received funding and ethics approvals in June 2020. Outcome data is expected to be available in March 2021, and the impact evaluation is expected to be published mid-2021. CONCLUSIONS The impact evaluation will examine the rapid design, development, and implementation of the RMHive app and its impact on mental health outcomes for health care workers. Findings from the impact evaluation will provide guidance for the integration of EBCD in rapid design and implementation processes. The evaluation will also inform future development and rollout of the app to support the mental health needs of hospital-based health care workers more widely. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID) DERR1-10.2196/26168.
Collapse
Affiliation(s)
- Matthew Lewis
- Integrated Mental Health Research Program, Department of General Practice, Melbourne Medical School, The University of Melbourne, Melbourne, Australia
| | - Victoria J Palmer
- Integrated Mental Health Research Program, Department of General Practice, Melbourne Medical School, The University of Melbourne, Melbourne, Australia.,Centre for Digital Transformation of Health, The University of Melbourne, Melbourne, Australia
| | - Aneta Kotevski
- Department of Medicine, The University of Melbourne, Melbourne, Australia
| | - Konstancja Densley
- Integrated Mental Health Research Program, Department of General Practice, Melbourne Medical School, The University of Melbourne, Melbourne, Australia
| | - Meaghan L O'Donnell
- Phoenix Australia, Department of Psychiatry, The University of Melbourne, Melbourne, Australia
| | - Caroline Johnson
- Integrated Mental Health Research Program, Department of General Practice, Melbourne Medical School, The University of Melbourne, Melbourne, Australia
| | - Franz Wohlgezogen
- Centre for Workplace Leadership, Faculty of Business and Economics, The University of Melbourne, Melbourne, Australia
| | - Kathleen Gray
- Centre for Digital Transformation of Health, The University of Melbourne, Melbourne, Australia
| | - Kate Robins-Browne
- Integrated Mental Health Research Program, Department of General Practice, Melbourne Medical School, The University of Melbourne, Melbourne, Australia
| | - Luke Burchill
- Department of Medicine, The University of Melbourne, Parkville, Australia.,Department of Cardiology, Royal Melbourne Hospital, Parkville, Australia
| |
Collapse
|
30
|
Tarricone R, Petracca F, Ciani O, Cucciniello M. Distinguishing features in the assessment of mHealth apps. Expert Rev Pharmacoecon Outcomes Res 2021; 21:521-526. [PMID: 33663324 DOI: 10.1080/14737167.2021.1891883] [Citation(s) in RCA: 22] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/07/2023]
Abstract
Introduction: The unparalleled surge in digital health adoption during the COVID-19 pandemic has emphasized the potential of mHealth apps. However, the quality of available evidence is generally low, and regulatory frameworks have focused on apps with medical purposes only, overlooking apps with significant interactions with patients that may require stronger oversight.Areas covered: To support this expanded evidence generation process, we identified the reasons that distinguish mHealth apps compared to medical devices at large and that should differentially feature their assessment. mHealth apps are characterized by the iterative nature of the corresponding interventions, frequent user interactions with a non-linear relationship between technology usage, engagement and outcomes, significant organizational implications, as well as challenges associated with genericization, their broad diagnostic potential, and price setting.Expert Opinion: The renewed reliance experienced during the pandemic and the unprecedented injection of resources through recovery instruments can further boost the development of apps. Only robust evidence of the benefits of mHealth apps will persuade health-care professionals and beneficiaries to systematically deploy them. Regulatory bodies will need to question their current approaches by adopting comprehensive evaluation processes that adequately consider the specific features of mHealth apps.
Collapse
Affiliation(s)
- Rosanna Tarricone
- Centre for Research on Health and Social Care Management (CERGAS), SDA Bocconi School of Management, Milan, Italy.,Department of Social and Political Sciences, Bocconi University, Milano, Italy
| | - Francesco Petracca
- Centre for Research on Health and Social Care Management (CERGAS), SDA Bocconi School of Management, Milan, Italy
| | - Oriana Ciani
- Centre for Research on Health and Social Care Management (CERGAS), SDA Bocconi School of Management, Milan, Italy.,Institute of Health Research, University of Exeter Medical School, South Cloisters, St Luke's Campus, Exeter, UK
| | - Maria Cucciniello
- Strategy Department, University of Edinburgh Business School, Edinburgh, Scotland, UK
| |
Collapse
|
31
|
Wright A. Evaluation of two mobile health apps for patients with breast cancer using the Mobile Application Rating Scale. Mhealth 2021; 7:60. [PMID: 34805391 PMCID: PMC8572758 DOI: 10.21037/mhealth-20-161] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/06/2020] [Accepted: 03/26/2021] [Indexed: 01/10/2023] Open
Abstract
BACKGROUND Breast cancer is one of the most frequently diagnosed cancers worldwide. Screening, education about signs and symptoms, and improved access to treatment has helped reduce mortality. An understanding of the informational needs of women with breast cancer can help identify areas where mobile apps can further improve the experience of this patient group. METHODS Personas are a commonly used tools in user centred design to help represent particular user archetypes. Knowledge of existing informational needs and prior research using personas in breast cancer app design were used to create two different personas through which to source apps for evaluation. The Mobile Application Rating Scale, a common evaluation framework, was used to evaluate the mobile apps across several important domains. RESULTS Becca and OWise, two apps for breast cancer, were found through a discovery process in line with the personas described. Overall, both apps scored highly on the Mobile Application Rating Scale. Both apps had limited or no research to support their use in this patient group, and had issues related to data privacy. Becca scored particularly highly in domains related to accessibility while OWise's extensive range of features scored highly for functionality. CONCLUSIONS Both apps demonstrate the ability to fill an informational needs gap as evidenced in the existing literature. As with many mobile health apps, more clinical evidence and improved data handling would help support the widespread recommendation of their use in women who are undergoing or have completed treatment for breast cancer.
Collapse
|
32
|
Larsen M, Nicholas J, Han J, Lemon C, Okun K, Torok M, Wong D, Wong I, Wong Q, Huckvale K. Here's something I prepared earlier: a review of the time to publication of cross-sectional reviews of smartphone health apps. BMJ Open 2020; 10:e039817. [PMID: 33268414 PMCID: PMC7713229 DOI: 10.1136/bmjopen-2020-039817] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/27/2020] [Revised: 11/09/2020] [Accepted: 11/17/2020] [Indexed: 01/07/2023] Open
Abstract
OBJECTIVES Across a range of health conditions, apps are increasingly valued as tools for supporting the delivery and coordination of healthcare. Research-led cross-sectional reviews of apps are a potential resource to inform app selection in face of uncertainties around content quality, safety and privacy. However, these peer-reviewed publications only capture a snapshot of highly dynamic app stores and marketplaces. To determine the extent to which marketplace dynamics might impact the interpretation of app reviews, the current study sought to quantify the lag between the reported time of app assessment and publication of the results of these studies. DESIGN Searches were conducted on MEDLINE, Embase and PsycINFO to identify published cross-sectional reviews of health, fitness or wellness apps. Publication timeline metadata were extracted, allowing the primary outcome measure, the delay between app store search and manuscript publication, to be calculated. A secondary measure, the time between search and manuscript submission, was also calculated where possible. RESULTS After screening, 136 relevant cross-sectional app review studies were analysed. The median time to publication was 431 days (approximately 14 months, range: 42-1054 days). The median time to submission was 269 days (approximately 9 months, range: 5-874 days). Studies which downloaded apps typically took longer to publish (p=0.010), however the number of apps reviewed did not impact the time to publication (p=0.964). Studies which recommended specific apps were not published more rapidly (p=0.998). CONCLUSIONS Most health app reviews present data that are at least a year out-of-date at the time of publication. Given the high rate of turnover of health apps in public marketplaces, it may not be appropriate, therefore, for these reviews to be presented as a resource concerning specific products for commissioners, clinicians and the public. Alternative sources of information may be better calibrated to the dynamics of the app marketplace.
Collapse
Affiliation(s)
- Mark Larsen
- Black Dog Institute, UNSW Sydney, Randwick, New South Wales, Australia
| | - Jennifer Nicholas
- Orygen, Parkville, Victoria, Australia
- Centre for Youth Mental Health, University of Melbourne, Parkville, Victoria, Australia
| | - Jin Han
- Black Dog Institute, UNSW Sydney, Randwick, New South Wales, Australia
| | - Christopher Lemon
- St Vincent's Hospital, Sydney, New South Wales, Australia
- Faculty of Medicine, UNSW Sydney, Sydney, New South Wales, Australia
| | - Kelsi Okun
- Stanford University School of Humanities and Science, Stanford, California, USA
| | - Michelle Torok
- Black Dog Institute, UNSW Sydney, Randwick, New South Wales, Australia
| | - David Wong
- Centre for Health Informatics, The University of Manchester, Manchester, UK
| | - Iana Wong
- Black Dog Institute, UNSW Sydney, Randwick, New South Wales, Australia
| | - Quincy Wong
- School of Psychology, Western Sydney University, Sydney, New South Wales, Australia
| | - Kit Huckvale
- Black Dog Institute, UNSW Sydney, Randwick, New South Wales, Australia
| |
Collapse
|
33
|
Marshall JM, Dunstan DA, Bartik W. Treating Psychological Trauma in the Midst of COVID-19: The Role of Smartphone Apps. Front Public Health 2020; 8:402. [PMID: 33014955 PMCID: PMC7461952 DOI: 10.3389/fpubh.2020.00402] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/06/2020] [Accepted: 07/08/2020] [Indexed: 01/18/2023] Open
Abstract
With the COVID-19 pandemic confronting health systems worldwide, medical practitioners are treating a myriad of physical symptoms that have, sadly, killed many thousands of people. There are signs that the public is also experiencing psychological trauma as they attempt to navigate their way through the COVID-19 restrictions impinging on many aspects of society. With unprecedented demand for health professionals' time, people who are unable to access face-to-face assistance are turning to smartphone apps to help them deal with symptoms of trauma. However, the evidence for smartphone apps to treat trauma is limited, and clinicians need to be aware of the limitations and unresolved issues involved in using mental health apps.
Collapse
Affiliation(s)
- Jamie M Marshall
- Faculty of Medicine and Health, School of Psychology, University of New England, Armidale, NSW, Australia
| | - Debra A Dunstan
- Faculty of Medicine and Health, School of Psychology, University of New England, Armidale, NSW, Australia
| | - Warren Bartik
- Faculty of Medicine and Health, School of Psychology, University of New England, Armidale, NSW, Australia
| |
Collapse
|
34
|
Lagan S, Aquino P, Emerson MR, Fortuna K, Walker R, Torous J. Actionable health app evaluation: translating expert frameworks into objective metrics. NPJ Digit Med 2020; 3:100. [PMID: 32821855 PMCID: PMC7393366 DOI: 10.1038/s41746-020-00312-4] [Citation(s) in RCA: 67] [Impact Index Per Article: 16.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/20/2020] [Accepted: 07/06/2020] [Indexed: 01/21/2023] Open
Abstract
As use and availability of mobile health apps have increased, so too has the need for a thorough, accessible framework for app evaluation. The American Psychiatric Association's app evaluation model has emerged as a way to critically assess an app by considering accessibility, privacy and security, clinical foundation, engagement, and interoperability; however, there is no centralized database where users can view how various health apps perform when assessed via the APA model. In this perspective, we propose and outline our effort to translate the APA's model for the evaluation of health apps into a set of objective metrics that can be published online, making the framework actionable and accessible to a broad audience. The questions from the APA model were operationalized into 105 objective questions that are either binary or numeric. These questions serve as the foundation of an online database, where app evaluation consists of answering these 105 questions and can be crowdsourced. While the database has yet to be published and crowdsourced, initial internal testing demonstrated excellent interrater reliability. The database proposed here introduces a public and interactive approach to data collection that is guided by the APA model. The published product enables users to sort through the many mobile health apps and filter them according to individual preferences and priorities, making the ever-growing health app market more navigable.
Collapse
Affiliation(s)
- Sarah Lagan
- Department of Psychiatry, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA 02115 USA
| | - Patrick Aquino
- Department of Psychiatry, Lahey Hospital and Medical Center, Boston, MA 01805 USA
| | - Margaret R. Emerson
- College of Nursing, University of Nebraska Medical Center, Omaha, NE 68198 USA
| | - Karen Fortuna
- Department of Psychiatry, Geisel School of Medicine, Dartmouth College, Hanover, NH 03755 USA
| | - Robert Walker
- Department of Mental Health, Office of Recovery and Empowerment, Boston, MA 02114 USA
| | - John Torous
- Department of Psychiatry, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA 02115 USA
| |
Collapse
|
35
|
Marshall JM, Dunstan DA, Bartik W. Apps With Maps-Anxiety and Depression Mobile Apps With Evidence-Based Frameworks: Systematic Search of Major App Stores. JMIR Ment Health 2020; 7:e16525. [PMID: 32579127 PMCID: PMC7381031 DOI: 10.2196/16525] [Citation(s) in RCA: 24] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/06/2019] [Revised: 03/27/2020] [Accepted: 04/12/2020] [Indexed: 01/03/2023] Open
Abstract
BACKGROUND Mobile mental health apps have become ubiquitous tools to assist people in managing symptoms of anxiety and depression. However, due to the lack of research and expert input that has accompanied the development of most apps, concerns have been raised by clinicians, researchers, and government authorities about their efficacy. OBJECTIVE This review aimed to estimate the proportion of mental health apps offering comprehensive therapeutic treatments for anxiety and/or depression available in the app stores that have been developed using evidence-based frameworks. It also aimed to estimate the proportions of specific frameworks being used in an effort to understand which frameworks are having the most influence on app developers in this area. METHODS A systematic review of the Apple App Store and Google Play store was performed to identify apps offering comprehensive therapeutic interventions that targeted anxiety and/or depression. The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist was adapted to guide this approach. RESULTS Of the 293 apps shortlisted as offering a therapeutic treatment for anxiety and/or depression, 162 (55.3%) mentioned an evidence-based framework in their app store descriptions. Of the 293 apps, 88 (30.0%) claimed to use cognitive behavioral therapy techniques, 46 (15.7%) claimed to use mindfulness, 27 (9.2%) claimed to use positive psychology, 10 (3.4%) claimed to use dialectical behavior therapy, 5 (1.7%) claimed to use acceptance and commitment therapy, and 20 (6.8%) claimed to use other techniques. Of the 162 apps that claimed to use a theoretical framework, only 10 (6.2%) had published evidence for their efficacy. CONCLUSIONS The current proportion of apps developed using evidence-based frameworks is unacceptably low, and those without tested frameworks may be ineffective, or worse, pose a risk of harm to users. Future research should establish what other factors work in conjunction with evidence-based frameworks to produce efficacious mental health apps.
Collapse
Affiliation(s)
- Jamie M Marshall
- School of Psychology, Faculty of Medicine and Health, University of New England, Armidale, Australia
| | - Debra A Dunstan
- School of Psychology, Faculty of Medicine and Health, University of New England, Armidale, Australia
| | - Warren Bartik
- School of Psychology, Faculty of Medicine and Health, University of New England, Armidale, Australia
| |
Collapse
|
36
|
Development of a health technology assessment module for evaluating mobile medical applications. Int J Technol Assess Health Care 2020; 36:252-261. [PMID: 32419676 DOI: 10.1017/s0266462320000288] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
OBJECTIVE The aim of this study was to develop a module which could be used to facilitate the assessment of mobile medical applications (MMA) for regulatory and reimbursement purposes. METHODS In-depth interviews were conducted with policymakers, healthcare practitioners, and application developers to determine possible pathways and impediments to MMA reimbursement. These findings were integrated with our previous research on MMA reimbursement and regulation to create a module that could be used with existing health technology assessment (HTA) methodological frameworks to guide the evaluation of MMAs. RESULTS Stakeholders indicated that they trust how traditional medical devices are currently appraised for reimbursement. They were concerned that there was a lack of clarity regarding which entity in the health system was responsible for determining app quality. They were also concerned about the digital health literacy of medical practitioners and patients. Concepts emerging from our previous research were reinforced by the interview findings, including that the connectivity and cybersecurity of apps need to be considered, along with an assessment of software reliability. It is also critical that the credibility of the information presented in apps is assessed as it could potentially mislead patients and clinicians. CONCLUSION An MMA evaluation module was created that would enable an existing HTA process to be adapted for the assessment of MMA technology. These adaptations include making provisions for an assessment of app cybersecurity, the impact on MMA clinical utility of software updates, and compatibility issues. Items to address concerns around practitioner responsibility and app misinformation were also incorporated into the module.
Collapse
|
37
|
Unsworth C. Writing systematic, scoping, and app reviews: summarising and appraising best evidence. Br J Occup Ther 2020. [DOI: 10.1177/0308022620914464] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Affiliation(s)
- Carolyn Unsworth
- School of Health, Medical and Applied Sciences, Central Queensland University, Melbourne, Australia
| |
Collapse
|
38
|
Hou IC, Lan MF, Shen SH, Tsai PY, Chang KJ, Tai HC, Tsai AJ, Chang P, Wang TF, Sheu SJ, Dykes PC. The Development of a Mobile Health App for Breast Cancer Self-Management Support in Taiwan: Design Thinking Approach. JMIR Mhealth Uhealth 2020; 8:e15780. [PMID: 32352390 PMCID: PMC7226037 DOI: 10.2196/15780] [Citation(s) in RCA: 20] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/07/2019] [Revised: 12/16/2019] [Accepted: 12/21/2019] [Indexed: 12/20/2022] Open
Abstract
Background Evidence has shown that breast cancer self-management support from mobile health (mHealth) apps can improve the quality of life of survivors. Although many breast cancer self-management support apps exist, few papers have documented the procedure for the development of a user-friendly app from the patient’s perspective. Objective This study aimed to investigate the information needs of Taiwanese women with breast cancer to inform the development of a self-management support mHealth app. Methods A 5-step design thinking approach, comprising empathy, define, ideate, prototype, and test steps, was used in the focus groups and individual interviews conducted to collect information on the requirements and expectations of Taiwanese women with breast cancer with respect to the app. A thematic analysis was used to identify information needs. Results A total of 8 major themes including treatment, physical activity, diet, emotional support, health records, social resources, experience sharing, and expert consultation were identified. Minor themes included the desire to use the app under professional supervision and a trustworthy app manager to ensure the credibility of information. Conclusions The strengths of the design thinking approach were user-centered design and cultural sensitivity. The results retrieved from each step contributed to the development of the app and reduction of the gap between end users and developers. An mHealth app that addresses these 8 main themes can facilitate disease self-management for Taiwanese women with breast cancer.
Collapse
Affiliation(s)
- I-Ching Hou
- School of Nursing, National Yang-Ming University, Taipei, Taiwan
| | - Min-Fang Lan
- Institute of Biomedical Informatics, National Yang-Ming University, Taipei, Taiwan.,Department of Nursing, National Taiwan University Hospital, Taipei, Taiwan
| | - Shan-Hsiang Shen
- Department of Computer Science and Information Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan
| | - Pei Yu Tsai
- School of Nursing, National Yang-Ming University, Taipei, Taiwan.,Department of Nursing, National Taiwan University Hospital, Taipei, Taiwan
| | | | - Hao-Chih Tai
- Division of Plastic Surgery, Department of Surgery, National Taiwan University Hospital, Taipei, Taiwan.,Department of Surgery, College of Medicine, National Taiwan University, Taipei, Taiwan
| | - Ay-Jen Tsai
- Taiwan Breast Cancer Foundation, Taipei, Taiwan
| | - Polun Chang
- Institute of Biomedical Informatics, National Yang-Ming University, Taipei, Taiwan
| | - Tze-Fang Wang
- School of Nursing, National Yang-Ming University, Taipei, Taiwan
| | - Shuh-Jen Sheu
- School of Nursing, National Yang-Ming University, Taipei, Taiwan
| | - Patricia C Dykes
- Center for Patient Safety Research and Practice, Brigham and Women's Hospital, Boston, MA, United States.,Department of Medicine, Harvard Medical School, Boston, MA, United States
| |
Collapse
|
39
|
Carlo AD, Hosseini Ghomi R, Renn BN, Areán PA. By the numbers: ratings and utilization of behavioral health mobile applications. NPJ Digit Med 2019; 2:54. [PMID: 31304400 PMCID: PMC6572775 DOI: 10.1038/s41746-019-0129-6] [Citation(s) in RCA: 87] [Impact Index Per Article: 17.4] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/22/2019] [Accepted: 05/24/2019] [Indexed: 01/10/2023] Open
Abstract
Although >10,000 behavioral health applications ("apps") are currently available on the Apple and Google Play marketplaces, they have been minimally evaluated or regulated and little is known about "real world" usage patterns. This investigation combined data from online behavioral health app rating frameworks and a mobile health market research firm to identify the most downloaded apps as well as determine rating and ranking concordance between frameworks. Findings demonstrated that the most commonly downloaded apps focus on relaxation, mindfulness, and meditation skills and that they often have notably discordant reviews across rating frameworks. Our results suggest that there is a growing need for: (1) standardized behavioral health app quality and effectiveness measures, (2) up-to-date behavioral health app guidance for clinicians and consumers, and (3) evidence-based apps that incorporate revealed consumer preferences.
Collapse
Affiliation(s)
- Andrew D. Carlo
- Department of Psychiatry & Behavioral Sciences, University of Washington, Seattle, WA USA
| | | | - Brenna N. Renn
- Department of Psychiatry & Behavioral Sciences, University of Washington, Seattle, WA USA
| | - Patricia A. Areán
- Department of Psychiatry & Behavioral Sciences, University of Washington, Seattle, WA USA
| |
Collapse
|
40
|
Henson P, David G, Albright K, Torous J. Deriving a practical framework for the evaluation of health apps. LANCET DIGITAL HEALTH 2019; 1:e52-e54. [PMID: 33323229 DOI: 10.1016/s2589-7500(19)30013-5] [Citation(s) in RCA: 79] [Impact Index Per Article: 15.8] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/12/2019] [Revised: 04/03/2019] [Accepted: 04/05/2019] [Indexed: 10/26/2022]
Affiliation(s)
- Philip Henson
- Department of Psychiatry, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA
| | - Gary David
- Association for Applied and Clinical Sociology, Bentley University, Waltham, MA, USA
| | - Karen Albright
- Adult and Child Consortium for Health Outcomes Research and Delivery Science, University of Colorado Anschutz Medical Campus and Children's Hospital Colorado, Aurora, CO, USA; Department of Community and Behavioral Health, Colorado School of Public Health, Aurora, CO, USA; Department of Sociology and Criminology, University of Denver, Denver, CO, USA
| | - John Torous
- Departments of Psychiatry and Clinical Informatics, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, 02115, USA.
| |
Collapse
|