Mobile health app usability and quality rating scales: a systematic review

Clinical Validation of Digital Healthcare Solutions: State of the Art, Challenges and Opportunities

Digital health technologies (DHTs) at the intersection of health, medical informatics, and business aim to enhance patient care through personalised digital approaches. Ensuring the efficacy and reliability of these innovations demands rigorous clinical validation. A PubMed literature review (January 2006 to July 2023) identified 1250 papers, highlighting growing academic interest. A focused narrative review (January 2018 to July 2023) delved into challenges, highlighting issues such as diverse regulatory landscapes, adoption issues in complex healthcare systems, and a plethora of evaluation frameworks lacking pragmatic guidance. Existing frameworks often omit crucial criteria, neglect empirical evidence, and clinical effectiveness is rarely included as a criterion for DHT quality. The paper underscores the urgency of addressing challenges in accreditation, adoption, business models, and integration to safeguard the quality, efficacy, and safety of DHTs. A pivotal illustration of collaborative efforts to address these challenges is exemplified by the Digital Health Validation Center, dedicated to generating clinical evidence of innovative healthcare technologies and facilitating seamless technology transfer. In conclusion, it is necessary to harmonise evaluation approaches and frameworks, improve regulatory clarity, and commit to collaboration to integrate rigorous clinical validation and empirical evidence throughout the DHT life cycle.

Quality assessment of mHealth apps: a scoping review

Introduction

The number of mHealth apps has increased rapidly during recent years. Literature suggests a number of problems and barriers to the adoption of mHealth apps, including issues such as validity, usability, as well as data privacy and security. Continuous quality assessment and assurance systems might help to overcome these barriers. Aim of this scoping review was to collate literature on quality assessment tools and quality assurance systems for mHealth apps, compile the components of the tools, and derive overarching quality dimensions, which are potentially relevant for the continuous quality assessment of mHealth apps.

Methods

Literature searches were performed in Medline, EMBASE and PsycInfo. Articles in English or German language were included if they contained information on development, application, or validation of generic concepts of quality assessment or quality assurance of mHealth apps. Screening and extraction were carried out by two researchers independently. Identified quality criteria and aspects were extracted and clustered into quality dimensions.

Results

A total of 70 publications met inclusion criteria. Included publications contain information on five quality assurance systems and further 24 quality assessment tools for mHealth apps. Of these 29 systems/tools, 8 were developed for the assessment of mHealth apps for specific diseases, 16 for assessing mHealth apps for all fields of health and another five are not restricted to health apps. Identified quality criteria and aspects were extracted and grouped into a total of 14 quality dimensions, namely "information and transparency", "validity and (added) value", "(medical) safety", "interoperability and compatibility", "actuality", "engagement", "data privacy and data security", "usability and design", "technology", "organizational aspects", "social aspects", "legal aspects", "equity and equality", and "cost(-effectiveness)".

Discussion

This scoping review provides a broad overview of existing quality assessment and assurance systems. Many of the tools included cover only a few dimensions and aspects and therefore do not allow for a comprehensive quality assessment or quality assurance. Our findings can contribute to the development of continuous quality assessment and assurance systems for mHealth apps.

Systematic Review Registration

https://www.researchprotocols.org/2022/7/e36974/, International Registered Report Identifier, IRRID (DERR1-10.2196/36974).

Attributes, Quality, and Downloads of Dementia-Related Mobile Apps for Patients With Dementia and Their Caregivers: App Review and Evaluation Study

BACKGROUND

The adoption of mobile health (mHealth) apps among older adults (>65 years) is rapidly increasing. However, use of such apps has not been fully effective in supporting people with dementia and their caregivers in their daily lives. This is mainly attributed to the heterogeneous quality of mHealth apps, highlighting the need for improved app quality in the development of dementia-related mHealth apps.

OBJECTIVE

The aims of this study were (1) to assess the quality and content of mobile apps for dementia management and (2) to investigate the relationship between app quality and download numbers.

METHODS

We reviewed dementia-related mHealth apps available in the Google Play Store and Apple App Store in Taiwan. The identified mobile apps were stratified according to a random sampling approach and evaluated by five independent reviewers with sufficient training and proficiency in the field of mHealth and the related health care sector. App quality was scored according to the user version of the Mobile Application Rating Scale. A correlation analysis was then performed between the app quality score and number of app downloads.

RESULTS

Among the 17 apps that were evaluated, only one was specifically designed to provide dementia-related education. The mean score for the overall app quality was 3.35 (SD 0.56), with the engagement (mean 3.04, SD 0.82) and information (mean 3.14, SD 0.88) sections of the scale receiving the lowest ratings. Our analyses showed clear differences between the top three- and bottom three-rated apps, particularly in the entertainment and interest subsections of the engagement category where the ratings ranged from 1.4 to 5. The top three apps had a common feature in their interface, which included memory, attention, focus, calculation, and speed-training games, whereas the apps that received lower ratings were found to be deficient in providing adequate information. Although there was a correlation between the number of downloads (5000 or more) and app quality (t15=4.087, P<.001), this may not be a significant determinant of the app's perceived impact.

CONCLUSIONS

The quality of dementia-related mHealth apps is highly variable. In particular, our results show that the top three quality apps performed well in terms of engagement and information, and they all received more than 5000 downloads. The findings of this study are limited due to the small sample size and possibility of disregarding exceptional occurrences. Publicly available expert ratings of mobile apps could help people with dementia and their caregivers choose a quality mHealth app.

Problems and Barriers Related to the Use of mHealth Apps From the Perspective of Patients: Focus Group and Interview Study

BACKGROUND

Since fall 2020, mobile health (mHealth) apps have become an integral part of the German health care system. The belief that mHealth apps have the potential to make the health care system more efficient, close gaps in care, and improve the economic outcomes related to health is unwavering and already partially confirmed. Nevertheless, problems and barriers in the context of mHealth apps usually remain unconsidered.

OBJECTIVE

The focus groups and interviews conducted in this study aim to shed light on problems and barriers in the context of mHealth apps from the perspective of patients.

METHODS

Guided focus groups and individual interviews were conducted with patients with a disease for which an approved mHealth app was available at the time of the interviews. Participants were recruited via self-help groups. The interviews were recorded, transcribed, and subjected to a qualitative content analysis. The content analysis was based on 10 problem categories ("validity," "usability," "technology," "use and adherence," "data privacy and security," "patient-physician relationship," "knowledge and skills," "individuality," "implementation," and "costs") identified in a previously conducted scoping review. Participants were asked to fill out an additional questionnaire about their sociodemographic data and about their use of technology.

RESULTS

A total of 38 patients were interviewed in 5 focus groups (3 onsite and 2 web-based) and 5 individual web-based interviews. The additional questionnaire was completed by 32 of the participants. Patients presented with a variety of different diseases, such as arthrosis, tinnitus, depression, or lung cancer. Overall, 16% (5/32) of the participants had already been prescribed an app. During the interviews, all 10 problem categories were discussed and considered important by patients. A myriad of problem manifestations could be identified for each category. This study shows that there are relevant problems and barriers in the context of mHealth apps from the perspective of patients, which warrant further attention.

CONCLUSIONS

There are essentially 3 different areas of problems in the context of mHealth apps that could be addressed to improve care: quality of the respective mHealth app, its integration into health care, and the expandable digital literacy of patients.

Attributes, Methods, and Frameworks Used to Evaluate Wearables and Their Companion mHealth Apps: Scoping Review

BACKGROUND

Wearable devices, mobile technologies, and their combination have been accepted into clinical use to better assess the physical fitness and quality of life of patients and as preventive measures. Usability is pivotal for overcoming constraints and gaining users' acceptance of technology such as wearables and their companion mobile health (mHealth) apps. However, owing to limitations in design and evaluation, interactive wearables and mHealth apps have often been restricted from their full potential.

OBJECTIVE

This study aims to identify studies that have incorporated wearable devices and determine their frequency of use in conjunction with mHealth apps or their combination. Specifically, this study aims to understand the attributes and evaluation techniques used to evaluate usability in the health care domain for these technologies and their combinations.

METHODS

We conducted an extensive search across 4 electronic databases, spanning the last 30 years up to December 2021. Studies including the keywords "wearable devices," "mobile apps," "mHealth apps," "physiological data," "usability," "user experience," and "user evaluation" were considered for inclusion. A team of 5 reviewers screened the collected publications and charted the features based on the research questions. Subsequently, we categorized these characteristics following existing usability and wearable taxonomies. We applied a methodological framework for scoping reviews and the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklist.

RESULTS

A total of 382 reports were identified from the search strategy, and 68 articles were included. Most of the studies (57/68, 84%) involved the simultaneous use of wearables and connected mobile apps. Wrist-worn commercial consumer devices such as wristbands were the most prevalent, accounting for 66% (45/68) of the wearables identified in our review. Approximately half of the data from the medical domain (32/68, 47%) focused on studies involving participants with chronic illnesses or disorders. Overall, 29 usability attributes were identified, and 5 attributes were frequently used for evaluation: satisfaction (34/68, 50%), ease of use (27/68, 40%), user experience (16/68, 24%), perceived usefulness (18/68, 26%), and effectiveness (15/68, 22%). Only 10% (7/68) of the studies used a user- or human-centered design paradigm for usability evaluation.

CONCLUSIONS

Our scoping review identified the types and categories of wearable devices and mHealth apps, their frequency of use in studies, and their implementation in the medical context. In addition, we examined the usability evaluation of these technologies: methods, attributes, and frameworks. Within the array of available wearables and mHealth apps, health care providers encounter the challenge of selecting devices and companion apps that are effective, user-friendly, and compatible with user interactions. The current gap in usability and user experience in health care research limits our understanding of the strengths and limitations of wearable technologies and their companion apps. Additional research is necessary to overcome these limitations.

mHealth App Usability Questionnaire for Stand-Alone mHealth Apps Used by Health Care Providers: Canadian French Translation, Cross-Cultural Adaptation, and Validation (Part 1)

BACKGROUND

An increasing number of health care professionals are using mobile apps. The mHealth App Usability Questionnaire (MAUQ) was designed to evaluate the usability of mobile health apps by patients and providers. However, this questionnaire is not available in French.

OBJECTIVE

This study aims to translate (from English to Canadian French), cross-culturally adapt, and initiate the validation of the original version of MAUQ for stand-alone mobile health apps used by French-speaking health care providers.

METHODS

A cross-cultural research study using a well-established method was conducted to translate MAUQ to Canadian French by certified translators and subsequently review it with a translation committee. It was then back translated to English. The back translations were compared with the original by the members of the committee to reach consensus regarding the prefinal version. A pilot test of the prefinal version was conducted with a sample of 49 potential users and 10 experts for content validation.

RESULTS

The statements are considered clear, with interrater agreement of 99.14% among potential users and 90% among experts. Of 21 statements, 5 (24%) did not exceed the 80% interrater agreement of the experts regarding clarity. Following the revisions, interrater agreement exceeded 80%. The content validity index of the items varied from 0.90 to 1, and the overall content validity index was 0.981. Individual Fleiss multirater κ of each item was between 0.89 and 1, showing excellent agreement and increasing confidence in the questionnaire's content validity.

CONCLUSIONS

This process of translation and cultural adaptation produced a new version of MAUQ that was validated for later use among the Canadian French-speaking population. An upcoming separate study will investigate the psychometric properties of the adapted questionnaire.

Improving the design of patient-generated health data visualizations: design considerations from a Fitbit sleep study

Interactive data visualization can be a viable way to discover patterns in patient-generated health data and enable health behavior changes. However, very few studies have investigated the design and usability of such data visualization. The present study aimed to (1) explore user experiences with sleep data visualizations in the Fitbit app, and (2) focus on end users' perspectives to identify areas of improvement and potential solutions. The study recruited eighteen pre-medicine college students, who wore Fitbit watches for a two-week sleep data collection period and participated in an exit semi-structured interview to share their experience. A focus group was conducted subsequently to ideate potential solutions. The qualitative analysis identified six pain points (PPs) from the interview data using affinity mapping. Four design solutions were proposed by the focus group to address these PPs and illustrated by a set of mock-ups. The study findings informed four design considerations: (1) usability, (2) transparency and explainability, (3) understandability and actionability, and (4) individualized benchmarking. Further research is needed to examine the design guidelines and best practices of sleep data visualization, to create well-designed visualizations for the general population that enables health behavior changes.

Heuristics used for evaluating the usability of mobile health applications: A systematic literature review

Objective

Mobile health applications hold immense potential for enhancing health outcomes. Usability is one of the main factors for the adoption and use of mobile health applications. However, despite the growing importance of mHealth applications, clear standards for their evaluation remain elusive. The present study aimed to determine heuristics for the usability evaluation of health-related applications.

Methods

We systematically searched multiple databases for relevant papers published between January 2008 and April 2021. Articles were reviewed, and data were extracted and categorized from those meeting inclusion criteria by two authors independently. Heuristics were identified based on statements, words, and concepts expressed in the studies. These heuristics were first mapped to Nielsen's heuristics based on their differences or similarities. The remaining heuristics that were very important for mobile applications were categorized into new heuristics.

Results

Seventeen studies met the eligibility criteria. Seventy-nine heuristics were extracted from the papers. After combining the items with the same concepts and removing irrelevant items based on the exclusion criteria, 20 heuristics remained. Common heuristics such as "Visibility of system status" and "Flexibility and efficiency of use" were categorized into 10 previously established heuristics and new heuristics like "Navigation" and "User engagement" were recognized as new ones.

Conclusions

In our study, we have meticulously identified 20 heuristics that hold promise for evaluating and designing mHealth applications. These heuristics can be used by the researchers for the development of robust tools for heuristic evaluation. These tools, when adapted or tailored for health domain applications, have the potential to significantly enhance the quality of mHealth applications. Ultimately, this improvement in quality translates to enhanced patient safety.

Protocol Registration

(10.17605/OSF.IO/PZJ7H).

Mobile-Based Application Interventions to Enhance Cancer Control and Care in Low- and Middle-Income Countries: A Systematic Review

Objective: To identify and appraise mobile-based application (mAPP) interventions that have been used to support cancer control and care in low- and middle-income countries (LMICs). Methods: Four electronic databases were systematically searched for studies that reported primary research findings related to mAPP interventions applied in oncology settings in LMICs. A narrative synthesis was performed using the Mhealth Index and Navigation Database as an analytical framework. Results: Twenty studies reporting 18 cancer control and care mAPPs were included in this review. Among these mAPPs, ten focused on prevention, screening and early detection of cancer, five provided information to optimise supportive and palliative care, two provided support to assist treatment-shared decision-making and one covered information for follow-up and survivorship care. Conclusion: Cancer mAPP interventions are gradually gaining attention in LMICs as they provide unique resources for empowering and strengthening the role of people with cancer in their own care. To enhance cancer control, a focus on prevention and early detection is important; however, more mAPP interventions related to cancer treatment, follow-up and survivorship are also needed to enable more cost-effective cancer care.

Quality of Digital Health Interventions Across Different Health Care Domains: Secondary Data Analysis Study

BACKGROUND

There are more than 350,000 digital health interventions (DHIs) in the app stores. To ensure that they are effective and safe to use, they should be assessed for compliance with best practice standards.

OBJECTIVE

The objective of this paper was to examine and compare the compliance of DHIs with best practice standards and adherence to user experience (UX), professional and clinical assurance (PCA), and data privacy (DP).

METHODS

We collected assessment data from 1574 DHIs using the Organisation for the Review of Care and Health Apps Baseline Review (OBR) assessment tool. As part of the assessment, each DHI received a score out of 100 for each of the abovementioned areas (ie, UX, PCA, and DP). These 3 OBR scores are combined to make up the overall ORCHA score (a proxy for quality). Inferential statistics, probability distributions, Kruskal-Wallis, Wilcoxon rank sum test, Cliff delta, and Dunn tests were used to conduct the data analysis.

RESULTS

We found that 57.3% (902/1574) of the DHIs had an Organisation for the Review of Care and Health Apps (ORCHA) score below the threshold of 65. The overall median OBR score (ORCHA score) for all DHIs was 61.5 (IQR 51.0-73.0) out of 100. A total of 46.2% (12/26) of DHI's health care domains had a median equal to or above the ORCHA threshold score of 65. For the 3 assessment areas (UX, DP, and PCA), DHIs scored the highest for the UX assessment 75.2 (IQR 70.0-79.6), followed by DP 65.1 (IQR 55.0-73.4) and PCA 49.6 (IQR 31.9-76.1). UX scores had the least variance (SD 13.9), while PCA scores had the most (SD 24.8). Respiratory and urology DHIs were consistently highly ranked in the National Institute for Health and Care Excellence Evidence Standards Framework tiers B and C based on their ORCHA score.

CONCLUSIONS

There is a high level of variability in the ORCHA scores of DHIs across different health care domains. This suggests that there is an urgent need to improve compliance with best practices in some health care areas. Possible explanations for the observed differences might include varied market maturity and commercial interests within the different health care domains. More investment to support the development of higher-quality DHIs in areas such as ophthalmology, allergy, women's health, sexual health, and dental care may be needed.

Cultural Adaptation of the Actionable Health App Evaluation in Japan: Protocol for a Web-Based Modified Delphi Expert Consensus Study

BACKGROUND

With an increase in both the number of mental health disorders people are experiencing and the difficulty in accessing mental health care, the demand for accessible mental health care services has increased. The use of mobile devices has allowed people to receive care in their daily lives without restrictions on time or location. However, the majority of publicly available mobile health apps are not evidence-based, and the top-rated apps are not always safe or user-friendly and may not offer clinically beneficial results.

OBJECTIVE

This study aims to create a cultural adaptation of the American Psychiatric Association's comprehensive app evaluation framework in Japan using a web-based modified Delphi expert consensus.

METHODS

A web-based modified Delphi study includes developing the Japanese version of the comprehensive app evaluation framework and 3 Delphi rounds. In the first round, our working group sends a questionnaire to the panelists, who then complete it. In the second and third rounds, the working group sends a questionnaire and a summary of the panelists' answers based on each of the previous rounds. The panelists answer the questionnaires based on this summary. The summarization procedure is automated to help reduce the biases that can be generated when panelists' answers are summarized and when the panelists receive them. The working group sends only the result of the summarization with the next round's questionnaire. All interactions between the working group and the panelists will be conducted on Qualtrics (Qualtrics Japan LLC), a questionnaire platform. To culturally validate the comprehensive mental health app evaluation framework, participants from the following three categories will be recruited in Japan: (1) researchers, (2) practitioners, and (3) app developers.

RESULTS

This study received funding from a crowdfunding campaign in Japan (April 2023). The Delphi study began in January 2023 and will be completed in December 2023. We had already completed the translation of the 105 original app evaluation item questions by December 2022.

CONCLUSIONS

While the need for treatment using mental health apps is increasing, no framework that can be used to develop a centralized database for health apps is available or accessible, and no consensus has been reached among stakeholders in Japan about an appropriate framework. The results of the web-based modified Delphi method presented in this paper may provide direction for the development and use of mental health apps in the future among the relevant stakeholders. Furthermore, this study will enhance recognition of the framework among researchers, clinicians, mental health app developers, and users, in addition to devising new instruments to help users or practitioners efficiently choose the right app for their situations.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/44469.

Methods used to evaluate usability of mobile clinical decision support systems for healthcare emergencies: a systematic review and qualitative synthesis

Objective

The aim of this study was to determine the methods and metrics used to evaluate the usability of mobile application Clinical Decision Support Systems (CDSSs) used in healthcare emergencies. Secondary aims were to describe the characteristics and usability of evaluated CDSSs.

Materials and Methods

A systematic literature review was conducted using Pubmed/Medline, Embase, Scopus, and IEEE Xplore databases. Quantitative data were descriptively analyzed, and qualitative data were described and synthesized using inductive thematic analysis.

Results

Twenty-three studies were included in the analysis. The usability metrics most frequently evaluated were efficiency and usefulness, followed by user errors, satisfaction, learnability, effectiveness, and memorability. Methods used to assess usability included questionnaires in 20 (87%) studies, user trials in 17 (74%), interviews in 6 (26%), and heuristic evaluations in 3 (13%). Most CDSS inputs consisted of manual input (18, 78%) rather than automatic input (2, 9%). Most CDSS outputs comprised a recommendation (18, 78%), with a minority advising a specific treatment (6, 26%), or a score, risk level or likelihood of diagnosis (6, 26%). Interviews and heuristic evaluations identified more usability-related barriers and facilitators to adoption than did questionnaires and user testing studies.

Discussion

A wide range of metrics and methods are used to evaluate the usability of mobile CDSS in medical emergencies. Input of information into CDSS was predominantly manual, impeding usability. Studies employing both qualitative and quantitative methods to evaluate usability yielded more thorough results.

Conclusion

When planning CDSS projects, developers should consider multiple methods to comprehensively evaluate usability.

Healthcare professionals' and patients' assessments of listed mobile health apps in China: a qualitative study

Background

In recent years, mobile health (mHealth) has gradually developed in China, and intelligent medicine has become an important research topic. However, there are still significant problems in mHealth applications (apps). Although healthcare professionals and patients are the main users, few studies have focused on their perceptions of the quality of mHealth apps.

Objective

This study aimed to (1) understand the respective perceptions of healthcare professionals and patients regarding mHealth apps, (2) assess what barriers exist that influence the user experience, and (3) explore how to improve the quality of mHealth apps and the development of the mHealth market in China. The study aims to promote the standardization of mHealth apps and provide effective information for the improvement and development of mHealth apps in the future.

Methods

Semistructured interviews with 9 patients and 14 healthcare professionals were conducted from January 2022 to April 2022 in the Affiliated Hospital of Xuzhou Medical University. The participants used mHealth apps for more than 3 months, including the "Good Mood" and "Peace and Safe Doctors" apps and apps developed by the hospital that were popular in China. Interview transcripts were analysed using thematic analysis.

Results

The following five themes were extracted: different concerns, hidden medical dangers, distance and insecurity, barriers for older people, and having positive perceptions of mHealth apps. Healthcare professionals prioritized simplicity in regard to mHealth apps, whereas patients rated effectiveness as the most crucial factor. The study also revealed several problems with mHealth apps, including insufficient information about physician qualifications, inaccurate medical content, nonstandard treatment processes, and unclear accountability, which led to a sense of distance and insecurity among participants. Older individuals faced additional obstacles when using mHealth apps. Despite these issues, the participants remained optimistic about the future of mHealth app development.

Conclusion

The utilization, advantages, and obstacles of mHealth applications for healthcare professionals and patients were explored through semistructured interviews. Despite the promising prospects for mHealth apps in China, numerous issues still need to be addressed. Enhancing the safety monitoring system and developing user-friendly mHealth apps for older adult patients are essential steps to bridge the gap between healthcare providers and patients.

Utilization of patient portals: a cross-sectional study investigating associations with mobile app quality

BACKGROUND

Mobile apps facilitate patients' access to portals and interaction with their healthcare providers. The COVID-19 pandemic accelerated this trend globally, but little evidence exists on patient portal usage in the Middle East, where internet access and digital literacy are limited. Our study aimed to explore how users utilize a patient portal through its related mobile app (MyChart by EPIC).

METHODS

We conducted a cross-sectional survey of MyChart users, recruited from a tertiary care center in Lebanon. We collected MyChart usage patterns, perceived outcomes, and app quality, based on the Mobile Application Rating Scale (user version, uMARS), and sociodemographic factors. We examined associations between app usage, app quality, and sociodemographic factors using Pearson's correlations, Chi-square, ANOVA, and t-tests.

RESULTS

428 users completed the survey; they were primarily female (63%), aged 41.3 ± 15.6 years, with a higher education level (87%) and a relatively high crowding index of 1.4 ± 0.6. Most of the sample was in good and very good health (78%) and had no chronic illnesses (67%), and accessed the portal through MyChart once a month or less (76%). The most frequently used features were accessing health records (98%), scheduling appointments (67%), and messaging physicians (56%). According to uMARS completers (n = 200), the objective quality score was 3.8 ± 0.5, and the subjective quality was 3.6 ± 0.7. No significant association was found between overall app usage and the mobile app quality measured via uMARS. Moreover, app use frequency was negatively associated with education, socioeconomic status, and perceived health status. On the other hand, app use was positively related to having chronic conditions, the number of physician visits and subjective app quality.

CONCLUSION

The patient portal usage was not associated with app quality but with some of the participants' demographic factors. The app offers a user-friendly, good-quality interface to patient health records and physicians, appreciated chiefly by users with relatively low socioeconomic status and education. While this is encouraging, more research is needed to capture the usage patterns and perceptions of male patients and those with even lower education and socioeconomic status, to make patient portals more inclusive.

Covid-19 triage in the emergency department 2.0: how analytics and AI transform a human-made algorithm for the prediction of clinical pathways

The Covid-19 pandemic has pushed many hospitals to their capacity limits. Therefore, a triage of patients has been discussed controversially primarily through an ethical perspective. The term triage contains many aspects such as urgency of treatment, severity of the disease and pre-existing conditions, access to critical care, or the classification of patients regarding subsequent clinical pathways starting from the emergency department. The determination of the pathways is important not only for patient care, but also for capacity planning in hospitals. We examine the performance of a human-made triage algorithm for clinical pathways which is considered a guideline for emergency departments in Germany based on a large multicenter dataset with over 4,000 European Covid-19 patients from the LEOSS registry. We find an accuracy of 28 percent and approximately 15 percent sensitivity for the ward class. The results serve as a benchmark for our extensions including an additional category of palliative care as a new label, analytics, AI, XAI, and interactive techniques. We find significant potential of analytics and AI in Covid-19 triage regarding accuracy, sensitivity, and other performance metrics whilst our interactive human-AI algorithm shows superior performance with approximately 73 percent accuracy and up to 76 percent sensitivity. The results are independent of the data preparation process regarding the imputation of missing values or grouping of comorbidities. In addition, we find that the consideration of an additional label palliative care does not improve the results.

Development, dissemination and evaluation of a smartphone-based app for pressure ulcer/injury prevention and treatment for use at the bedside

OBJECTIVE

After launching the 2019 International Pressure Ulcer/Injury Guideline, the National Pressure Injury Advisory Panel (NPIAP), the European Pressure Ulcer Advisory Panel (EPUAP) and the Pan Pacific Pressure Injury Alliance (PPPIA) collaborated with Sensorydata Corp., US, to develop a guideline app (InterPIP App). The app was designed to: present evidence-based guideline recommendations; incorporate search capacities and functionality to facilitate easy access to clinical guidance; provide accessibility in multiple languages; and to be available worldwide at a reasonable price, including opportunities for free access in low-resource countries. This paper describes the development, dissemination and formative evaluation of a mobile app providing evidence-based recommendations for pressure injury prevention, assessment/classification, and treatment at the point of care.

METHOD

An evaluation tool was designed based on a framework developed by Nouri et al. and made available to all app subscribers.

RESULTS

The InterPIP App is currently available in 11 languages and had been downloaded 3616 times by February 2022 in 78 countries. A total of 62 individuals responded to the survey of end-users. In this formal evaluation of user experiences, the app was rated positively on criteria of: information/content; usability; design; functionality; ethics; and security/privacy (median=4 on a 1-5 Likert scale). Overall perceived value was ranked lower with a median of three. Users provided suggestions for ongoing app enhancement.

CONCLUSION

The InterPIP App offers a unique opportunity to bring evidence-based guidance to the point of care. Formal evaluation of end-user experiences identified opportunities for quality improvement, and informed plans for future development and evaluation.

Mobile health applications for pregnancy in Iran: Current state investigation

BACKGROUND

The expansion of pregnancy-related mobile applications has led to an increase in the usage of these applications by Iranian women, which has the potential to have a long-term impact on women's pregnancy health and, as a result, community health. The goal of this research was to investigate the state of pregnancy-related mobile applications in Iran.

MATERIALS AND METHODS

This was a descriptive-applied research approach which was conducted in 2020. One hundred and fifty-seven pregnancy apps from different Iranian mobile app markets were installed and were examined in terms of the resources used and their validity, subject distribution, participation of obstetricians in the app development, apps volume, rank acquired by users and distribution of apps according to the developers. The relationship between the number of resources used and user rankings was also tested.

RESULTS

A total of 157 apps were categorized into 12 subject categories based on their content. Only 1.3% of the apps were developed with the participation of obstetricians. Only 5% of the apps had used reliable information resources. Furthermore, the number of resources referenced in the studied apps did not have a significant relationship with star ratings (r = 0.03, P = 0.64), according to the Pearson correlation test.

CONCLUSION

Iranian mobile applications in the field of pregnancy have grown significantly in terms of quantity, subject diversity, and user acceptance; however, the credibility of the information in these apps cannot be validated. This serves as a reminder to users to exercise caution while using such apps.

User Interface (UI) Design and User Experience Questionnaire (UEQ) Evaluation of a To-Do List Mobile Application to Support Day-To-Day Life of Older Adults

Because of the spread of smartphones, older adults enjoy the assistance of smartphones. However, fewer mobile applications are designed for older adults. Smartphone user interface (UI) serves as an external brain to capture information, and older adults may have memory complaints that affect self-confidence and lead to memory decline. Non-declarative memory requires more effort. Therefore, this study aims to design and evaluate a to-do list application to help older adults encode, store, and retrieve non-declarative memory, such as tasks they plan to do. We recruited 15 participants (5 men and 10 women) aged 60 to 75 years old (SD = 5.32). They were asked to complete nine usability tasks, and to answer a user experience questionnaire (UEQ) and a few interview questions. Sixty percent of users completed with only one or two attempts (median = 2.80, SD = 1.63). We found three usability issues and proposed an iteration plan. The application has attractiveness, efficiency, dependability, stimulation, novelty, and good perspicuity for older adults. The product was rated excellent except for perspicuity, which met the users’ expectations. This indicates that the user is satisfied with the application prototype. The results of this measurement can be utilized as a benchmark for the next model for developing mobile to-do list applications on user experience.

Exploring Saudi Individuals' Perspectives and Needs to Design a Hypertension Management Mobile Technology Solution: Qualitative Study

Hypertension is a chronic condition caused by a poor lifestyle that affects patients' lives. Adherence to self-management programs increases hypertension self-monitoring, and allows greater prevention and disease management. Patient compliance with hypertension self-management is low in general; therefore, mobile health applications (mHealth-Apps) are becoming a daily necessity and provide opportunities to improve the prevention and treatment of chronic diseases, including hypertension. This research aims to explore Saudi individuals' perspectives and needs regarding designing a hypertension management mobile app to be used by hypertension patients to better manage their illnesses. Semi-structured interviews were conducted with 21 Saudi participants to explore their perspectives and views about the needs and requirements in designing a hypertension mobile technology solution, as well as usability and culture in the Saudi context. The study used NVivo to analyze data and divided the themes into four main themes: the app's perceived health benefits, features and usability, suggestions for the app's content, and security and privacy. The results showed that there are many suggestions for improvements in mobile health apps that developers should take into consideration when designing apps. The mobile health apps should include physical activity tracking, related diet information, and reminders, which are promising, and could increase adherence to healthy lifestyles and consequently improve the self-management of hypertension patients. Mobile health apps provide opportunities to improve hypertension patients' self-management and self-monitoring. However, this study asserts that mobile health apps should not share users' data, and that adequate privacy disclosures should be implemented.

[Mobile technologies in healthcare: reflections on development, application, legal aspects, and ethicsTecnologías sanitarias móviles: reflexiones sobre desarrollo, aplicación, legislación y ética]

The association between fast-growing mobile technologies and increasingly more mobile devices has allowed the introduction of virtual environments into daily activities. That includes the health care domain, where concepts such telemedicine, telehealth, eHealth, and mHealth have emerged. In addition to presenting these new concepts, this article aims to discuss the advancements and challenges of mobile health technologies stemming from considerations regarding development, application, legal aspects, and ethics. Because of their innovative nature, mobile health technologies entail the engagement of many actors in the journey to reach end users, covering conception, technical development, sanitary regulations, and design of clinical guidelines, having raised a great deal or interest in terms of monitoring and care across a variety of clinical conditions. However, assessment of the effectiveness and safety of mobile health technologies does not seem to involve the same methodological rigor imposed for clinical trials of drugs and other health products; still, the enthusiasm produced by this innovation counters some of the regulatory and ethics concerns relating to data protection, privacy, access to mobile devices, and technological or social inequality. Despite possible limitations, mobile technologies, as well as other telehealth resources, have produced promising results. Digital healthcare has great potential for expansion and represents an opportunity for the review of traditional practices with selection of mobile technologies for incorporation into the health care system whenever evidence-based benefits are verified.

PlanTEA: Supporting Planning and Anticipation for Children with ASD Attending Medical Appointments

In people with Autism Spectrum Disorder (ASD), skills related to anticipation and mental flexibility are often impaired, so their thinking tends to be very rigid and their behavior is based on establishing routines. For this reason, children with ASD may show disruptive behaviors when faced with disturbing but necessary activities, such as going to a doctor’s appointment. Therefore, it is very convenient and necessary for their families to prepare in advance for the visit and to explain the details of the procedure to be performed at the consultation. The use of anticipation boards in these situations allows to prepare such situations and to reduce stress for both the ASD child and their families or caregivers. In this context, the use of technology can provide great benefits for anticipating a new event, or whatever risks the control of their routines, as well as enhancing developmental skills such as communication, autonomy, social interaction, etc. This article describes a software tool, for mobile devices such as tablets, that allows the planning of the attendance of children with ASD to the necessary medical appointments throughout their childhood and adolescence, as well as communication with specialists. This app, named PlanTEA, has undergone a preliminary evaluation that has yielded very positive results. Most participants found the app useful in helping to anticipate (94.1%) and improve communication (94.2%) for people with ASD in medical contexts, considered it easy to use, with no technical support needed to use it (almost 100%), and would recommend its use (94.2%). This first evaluation has also allowed us to define the next steps to be taken to improve and enhance this tool and thus reach a wider population within the autistic disorder. As a result of the evaluation carried out and the comments received, it is proposed to extend its use to adult users and those with high-functioning autism, which supposes expanding and extending the functionalities of the current version of PlanTEA.

Assessing a new pre-screening score for the simplified evaluation of the clinical quality and relevance of e-health applications. (Preprint)

Background

In 2020, more than 250 eHealth solutions were added to app stores each day, or 90,000 in the year; however, the vast majority of these solutions have not undergone clinical validation, their quality is unknown, and the user does not know if they are effective and safe. We sought to develop a simple prescreening scoring method that would assess the quality and clinical relevance of each app. We designed this tool with 3 health care stakeholder groups in mind: eHealth solution designers seeking to evaluate a potential competitor or their own tool, investors considering a fundraising candidate, and a hospital clinician or IT department wishing to evaluate a current or potential eHealth solution.

Objective

We built and tested a novel prescreening scoring tool (the Medical Digital Solution scoring tool). The tool, which consists of 26 questions that enable the quick assessment and comparison of the clinical relevance and quality of eHealth apps, was tested on 68 eHealth solutions.

Methods

The Medical Digital Solution scoring tool is based on the 2021 evaluation criteria of the French National Health Authority, the 2022 European Society of Medical Oncology recommendations, and other provided scores. We built the scoring tool with patient association and eHealth experts and submitted it to eHealth app creators, who evaluated their apps via the web-based form in January 2022. After completing the evaluation criteria, their apps obtained an overall score and 4 categories of subscores. These criteria evaluated the type of solution and domain, the solution’s targeted population size, the level of clinical assessment, and information about the provider.

Results

In total, 68 eHealth solutions were evaluated with the scoring tool. Oncology apps (22%, 20/90) and general health solutions (23%, 21/90) were the most represented. Of the 68 apps, 32 (47%) were involved in remote monitoring by health professionals. Regarding clinical outcomes, 5% (9/169) of the apps assessed overall survival. Randomized studies had been conducted for 21% (23/110) of the apps to assess their benefit. Of the 68 providers, 38 (56%) declared the objective of obtaining reimbursement, and 7 (18%) out of the 38 solutions seeking reimbursement were assessed as having a high probability of reimbursement. The median global score was 11.2 (range 4.7-17.4) out of 20 and the distribution of the scores followed a normal distribution pattern (Shapiro-Wilk test: P=.33).

Conclusions

This multidomain prescreening scoring tool is simple, fast, and can be deployed on a large scale to initiate an assessment of the clinical relevance and quality of a clinical eHealth app. This simple tool can help a decision-maker determine which aspects of the app require further analysis and improvement.

Alberta Rating Index for Apps: Study of Reliability and Validity

Background. The number of mobile health applications is rapidly increasing, yet no reliable tool exists for occupational therapists and their clients to rate the quality of these apps. Purpose. To develop the Alberta Rating Index for Apps (ARIA). Methods. Through a sequential design in three phases, we developed a rating index for mobile health applications and examined its reliability and validity with 10 participants. Findings. The coefficients of reliability were 0.95 for occupational therapists, 0.60 for older adults, and 0.88 for adults with a mental health condition. ARIA's correlation with another scale used in app review studies, U-MARS, was low to moderate. Implications. ARIA showed a high inter-rater reliability in two of the three user groups. ARIA is comprehensive and includes criteria not captured by U-MARS, such as privacy and security. Further studies are warranted with diverse raters and health apps.

Technology Acceptance and Usability of a Mobile Application to Support the Workflow of Health Care Aides who Provide Services to older adults residing in a care facility: A Pilot Mixed Methods Study (Preprint)

Background

Health care aides are unlicensed support personnel who provide direct care, personal assistance, and support to people with health conditions. The shortage of health care aides has been attributed to recruitment challenges, high turnover, an aging population, the COVID-19 pandemic, and low retention rates. Mobile apps are among the many information communication technologies that are paving the way for eHealth solutions to help address this workforce shortage by enhancing the workflow of health care aides. In collaboration with Clinisys EMR Inc, we developed a mobile app (Mobile Smart Care System [mSCS]) to support the workflow of health care aides who provide services to older adult residents of a long-term care facility.

Objective

The purpose of this study was to investigate the technology acceptance and usability of a mobile app in a real-world environment, while it is used by health care aides who provide services to older adults.

Methods

This pilot study used a mixed methods design: sequential mixed methods (QUANTITATIVE, qualitative). Our study included a pre– and post–paper-based questionnaire with no control group (QUAN). Toward the end of the study, 2 focus groups were conducted with a subsample of health care aides (qual, qualitative description design). Technology acceptance and usability questionnaires used a 5-point Likert scale ranging from disagree (1) to agree (5). The items included in the questionnaires were validated in earlier research as having high levels of internal consistency for the Unified Theory of Acceptance and Use of Technology constructs. A total of 60 health care aides who provided services to older adults as part of their routine caseloads used the mobile app for 1 month. Comparisons of the Unified Theory of Acceptance and Use of Technology constructs’ summative scores at pretest and posttest were calculated using a paired t test (2-tailed). We used the partial least squares structural regression model to determine the factors influencing mobile app acceptance and usability for health care aides. The α level of significance for all tests was set at P≤.05 (2-tailed).

Results

We found that acceptance of the mSCS was high among health care aides, performance expectancy construct was the strongest predictor of intention to use the mSCS, intention to use the mSCS predicted usage behavior. The qualitative data support the quantitative findings and showed health care aides’ strong belief that the mSCS was useful, portable, and reliable, although there were still opportunities for improvement, especially with regard to the mSCS user interface.

Conclusions

Overall, these results support the assertion that mSCS technology acceptance and usability are high among health care aides. In other words, health care aides perceived that the mSCS assisted them in addressing their workflow issues.

Parents' Experience With a Mobile Health Intervention to Influence Human Papillomavirus Vaccination Decision Making: Mixed Methods Study

BACKGROUND

Human papillomavirus (HPV)-attributed cancers are preventable, yet HPV vaccination rates severely lag behind other adolescent vaccinations. HPVcancerFree (HPVCF) is a mobile health (mHealth) intervention developed to influence parental HPV vaccination decision making by raising awareness of HPV, reducing HPV vaccination barriers, and enabling HPV vaccination scheduling and reminders through a smartphone app. Evaluating the user experience of mHealth interventions is a vital component in assessing their quality and success but tends to be underreported in mHealth intervention evaluation.

OBJECTIVE

We aimed to evaluate the user experience of HPVCF, an HPV cancer prevention app designed for a pediatric clinic network, using mixed methods data collected from log files, survey measures, and qualitative feedback.

METHODS

Study data were evaluated from parents in a large US pediatric clinic network using HPVCF in the treatment study condition of a group randomized controlled trial. Log data captured HPVCF retention and use. Postintervention rating scales and items assessed HPVCF utility, usefulness, understandability, appeal, credibility, and perceived impact. Overall quality was evaluated using the user version of the Mobile Application Rating Scale (uMars). Open-ended responses assessed parent recommendations for HPVCF enhancement.

RESULTS

The 98 parents were mainly female (n=94, 96%), 41 (5.67) years of age, college educated (n=55, 56%), and White and non-Hispanic (n=55, 56%) and had private health insurance for their children (n=75, 77%). Parents used HPVCF 197 times, with the average visit duration approximating 3.5 minutes. The uMARS app quality score was positively skewed (4.2/5.0). Mean ratings were highest for information (4.46 [SD 0.53]) and lowest for engagement (3.74 [SD 0.69]). In addition, of 95 parents, 45 (47%) rated HPVCF as helpful in HPV vaccination decision making and 16 (17%) attributed HPV vaccine initiation to HPVCF. Parents reported that HPVCF increased their awareness (84/95, 88%), knowledge (84/95, 88%), and HPV vaccination intentions (64/95, 67%). Most of the 98 parents rated the 4 HPVCF components as useful (72-92 [73%-94%]). Parents also agreed that HPVCF is clear (86/95, 91%), accurate (86/95, 91%), and more helpful than other HPV vaccine information they had received (89/95, 94%) and that they would recommend it to others (81/95, 85%). In addition, parents suggested ways to increase awareness and engagement with the app, along with opportunities to enhance the content and functionality.

CONCLUSIONS

HPVCF was well received by parents and performed well on indicators of quality, usefulness, utility, credibility, and perceived impact. This study contributes a multimethod and multimeasure evaluation to the growing body of literature focused on assessing the user experience of patient-focused technology-mediated applications for HPV education.

Quality Assessment of Digital Health Applications: A Scoping Review Protocol (Preprint)

Background

All over the world, development and usage of mobile health (mHealth) apps is increasing. While apps offer numerous opportunities to improve health care, there are associated problems that differ significantly from those of traditional health care services. Further investigations on the quality of mHealth apps are needed to address these problems.

Objective

This study aims to identify and map research on quality assessment and quality assurance of mHealth apps and their transferability to continuous quality assurance of mHealth apps.

Methods

The scoping review will follow published methodological frameworks for scoping studies as well as Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews criteria. Electronic databases (Medline, EMBASE, and PsycINFO), reference lists of relevant articles, and websites of relevant institutions will be searched. Two reviewers will independently assess eligibility of articles. Therefore, a 2-stage (title and abstract, followed by full text) screening process was conducted. Quality management systems and quality assessment tools will be analyzed and included in our review. Particular focus is placed on quality dimensions.

Results

This scoping review provides an overview of the available evidence and identifies research gaps regarding continuous quality assessment of mHealth apps. Thereby, relevant quality dimensions and criteria can be identified and their eligibility and relevance for the development of a continuous quality assurance system of mHealth apps can be determined. Our results are planned to be submitted to an indexed, peer-reviewed journal in the second half of 2022.

Conclusions

This is the first review in the context of continuous quality assurance of mHealth apps. Our results will be used within the research “Continuous quality assurance of Digital Health Applications” (“QuaSiApps”) project funded by the German Federal Joint Committee.

International Registered Report Identifier (IRRID)

DERR1-10.2196/36974

Prioritization of Quality Principles for Health Apps Using the Kano Model: Survey Study

Background

Health apps are often used without adequately taking aspects related to their quality under consideration. This may partially be due to inadequate awareness about necessary criteria and how to prioritize them when evaluating an app.

Objective

The aim of this study was to introduce a method for prioritizing quality attributes in the mobile health context. To this end, physicians were asked about their assessment of nine app quality principles relevant in health contexts and their responses were used as a basis for designing a method for app prioritization. Ultimately, the goal was to aid in making better use of limited resources (eg, time) by assisting with the decision as to the specific quality principles that deserve priority in everyday medical practice and those that can be given lower priority, even in cases where the overall principles are rated similarly.

Methods

A total of 9503 members of two German professional societies in the field of orthopedics were invited by email to participate in an anonymous online survey over a 1-month period. Participants were asked to rate a set of nine app quality principles using a Kano survey with functional and dysfunctional (ie, positively and negatively worded) questions. The evaluation was based on the work of Kano (baseline), supplemented by a self-designed approach.

Results

Among the 9503 invited members, 382 completed relevant parts of the survey (return rate of 4.02%). These participants were equally and randomly assigned to two groups (test group and validation group, n=191 each). Demographic characteristics did not significantly differ between groups (all P>.05). Participants were predominantly male (328/382, 85.9%) and older than 40 years (290/382, 75.9%). Given similar ratings, common evaluation strategies for Kano surveys did not allow for conclusive prioritization of the principles, and the same was true when using the more elaborate approach of satisfaction and dissatisfaction indices following the work of Timko. Therefore, an extended, so-called “in-line-of-sight” method was developed and applied for this evaluation. Modified from the Timko method, this approach is based on a “point of view” (POV) metric, which generates a ranking coefficient. Although the principles were previously almost exclusively rated as must-be (with the exception of resource efficiency), which was not conducive to their prioritization, the new method applied from the must-be POV resulted in identical rankings for the test and validation groups: (1) legal conformity, (2) content validity, (3) risk adequacy, (4) practicality, (5) ethical soundness, (6) usability, (7) transparency, (8) technical adequacy, and (9) resource efficiency.

Conclusions

Established survey methodologies based on the work of Kano predominantly seek to categorize the attributes to be evaluated. The methodology presented here is an interesting option for prioritization, and enables focusing on the most important criteria, thus saving valuable time when reviewing apps for use in the medical field, even with otherwise largely similar categorization results. The extent to which this approach is applicable beyond the scenario presented herein requires further investigation.

Towards Understanding the Usability Attributes of AI-Enabled eHealth Mobile Applications

Mobile application (app) use is increasingly becoming an essential part of our daily lives. Due to their significant usefulness, people rely on them to perform multiple tasks seamlessly in almost all aspects of everyday life. Similarly, there has been immense progress in artificial intelligence (AI) technology, especially deep learning, computer vision, natural language processing, and robotics. These technologies are now actively being implemented in smartphone apps and healthcare, providing multiple healthcare services. However, several factors affect the usefulness of mobile healthcare apps, and usability is an important one. There are various healthcare apps developed for each specific task, and the success of these apps depends on their performance. This study presents a systematic review of the existing apps and discusses their usability attributes. It highlights the usability models, outlines, and guidelines proposed in previous research for designing apps with improved usability characteristics. Thirty-nine research articles were reviewed and examined to identify the usability attributes, framework, and app design conducted. The results showed that satisfaction, efficiency, and learnability are the most important usability attributes to consider when designing eHealth mobile apps. Surprisingly, other significant attributes for healthcare apps, such as privacy and security, were not among the most indicated attributes in the studies.

Identifying adoption and usability factors of locator devices for persons living with dementia

A growing number of Canadians live with dementia. Strategies to reduce the risks of getting lost include physical barriers, restraints and medications. However, these strategies can restrict one’s participation in meaningful activities and reduce quality of life. Locator devices can be used to manage safety risks while also supporting engagement and independence among persons living with dementia. As more locator devices become available on the market, adoption rates would be affected by certain factors. There is no clear, standardized approach to identify the factors that have an influence on the acceptance and usability of locator devices for persons with dementia and their care partners. This project aimed to identify factors related to acceptance and usability of locator devices that are important to individuals with dementia, their care partners, service providers and technology developers. Qualitative description and conventional content analysis guided our approach. We conducted 5 focus groups with 21 participants. Trustworthiness strategies included multiple data sources, data verification for accuracy and peer debrief. Five overarching factors emerged as critical aspects in the acceptance and usability of locator devices. These factors were inclusivity, simplicity, features, physical properties and ethics. Participants thought that locator devices do not adequately consider privacy and stigma. Therefore, the acceptance and usability of locator devices could be enhanced if privacy and stigma are addressed. The factors identified will inform the creation of an acceptance and usability scale for locator devices used by persons living with dementia, their care partners and service providers.

ACCU3RATE: A mobile health application rating scale based on user reviews

Background

Over the last decade, mobile health applications (mHealth App) have evolved exponentially to assess and support our health and well-being.

Objective

This paper presents an Artificial Intelligence (AI)-enabled mHealth app rating tool, called ACCU³RATE, which takes multidimensional measures such as user star rating, user review and features declared by the developer to generate the rating of an app. However, currently, there is very little conceptual understanding on how user reviews affect app rating from a multi-dimensional perspective. This study applies AI-based text mining technique to develop more comprehensive understanding of user feedback based on several important factors, determining the mHealth app ratings.

Method

Based on the literature, six variables were identified that influence the mHealth app rating scale. These factors are user star rating, user text review, user interface (UI) design, functionality, security and privacy, and clinical approval. Natural Language Toolkit package is used for interpreting text and to identify the App users’ sentiment. Additional considerations were accessibility, protection and privacy, UI design for people living with physical disability. Moreover, the details of clinical approval, if exists, were taken from the developer’s statement. Finally, we fused all the inputs using fuzzy logic to calculate the new app rating score.

Results and conclusions

ACCU³RATE concentrates on heart related Apps found in the play store and App gallery. The findings indicate the efficacy of the proposed method as opposed to the current device scale. This study has implications for both App developers and consumers who are using mHealth Apps to monitor and track their health. The performance evaluation shows that the proposed mHealth scale has shown excellent reliability as well as internal consistency of the scale, and high inter-rater reliability index. It has also been noticed that the fuzzy based rating scale, as in ACCU³RATE, matches more closely to the rating performed by experts.

A Review of the Quality and Impact of Mobile Health Apps

Mobile health applications (apps) have transformed the possibilities for health promotion and disease self-management; however, their promise is not fully realized owing to their reliance on commercial ecosystems for development and distribution. This review provides an overview of the types of mobile health apps and describes key stakeholders in terms of how apps are used, developed, and regulated. I outline key challenges facing consumers, public health professionals, and policy makers in evaluating the quality of health apps and summarize what is known about the impact of apps on health outcomes and health equity. I suggest that factors within the wider mobile ecosystem largely define the impact of health apps and, most notably, practices around the collection and commercialization of user data. Finally, I suggest that upstream public health strategies, grounded in an understanding of corporate influences on health, are necessary to promote healthy digital environments in which mobile health app innovation can flourish. Expected final online publication date for the Annual Review of Public Health, Volume 43 is April 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Mobile Health Apps That Act as Surgical Preparatory Guides: App Store Search and Quality Evaluation

BACKGROUND

Mobile health (mHealth) apps are becoming increasingly common in surgical practices for training, education, and communication. Factors leading to increased delays, morbidity, and mortality in surgery include inadequate preoperative patient preparation due to a failure to identify patients and procedure details, and missing instruments and equipment required for the procedure. Many apps are available for supporting preoperative, intraoperative, and postoperative care. However, there is a lack of studies that assess the quality of apps that act as surgical preparatory guides.

OBJECTIVE

The aim of this study is to evaluate the quality of apps that act as surgical preparatory guides for operating room personnel through an in-house quality assessment tool.

METHODS

The quality assessment tool comprises 35 questions categorized into 5 sections: (1) engagement (customization, interactivity, target audience; 19 points), (2) functionality (performance, ease of use, navigation; 12 points), (3) aesthetics (layout, visual appeal; 6 points), (4) information (quality and quantity of information, visual information, credibility; 29 points), and (5) privacy and security (4 points). An app search was conducted in the Australian Apple and Google Play stores using the following keywords: "surgical apps", "surgical preferences", "surgeon preferences", "operating room", and "perioperative procedures". The overall total scores and scores for each section were reported as medians and IQRs, expressed as raw scores and percentages.

RESULTS

A total of 5 unique apps were evaluated on both iOS and Android platforms. The median overall score across all apps was 35/70 (50%; IQR 38.6%-64.3%). ScrubUp (48/70, 69%) and MySurgeon (42/70, 60%) had the highest overall scores, followed by PrefCard (35/70, 50%) and Scrubnote (28/70, 40%). The lowest scoring app was BrainPadd (26/70, 37%). The sections with the highest median scores, in decreasing order, were privacy and security (4/4, 100%; IQR 75%-100%), aesthetics (5/6, 83%; IQR 75%-91.7%), engagement (15/19, 79%; IQR 57.9%-86.8%), functionality (7/12, 58%; IQR 29.2%-75%), and information (5/29, 17%; IQR 15.5%-34.5%). Most apps scored well (4/4, 100%) on privacy and security, except for Scrubnote (2/4, 50%). ScrubUp received a perfect score for aesthetics (6/6, 100%). MySurgeon (17/19, 90%) had the highest engagement score, while ScrubUp and MySurgeon had the highest functionality scores (9/12, 75% each). All apps scored below 50% for the information section, with ScrubUp having the highest score of 13/29 (45%).

CONCLUSIONS

ScrubUp and MySurgeon had the highest quality scores and can be used as adjuncts to hospital protocols by operating room personnel for their surgical preparation. Developers are encouraged to develop appropriate apps for surgical preparation based on relevant guidelines and standards, as well as the quality evaluation criteria in our tool. Operating room personnel can also use this tool as a guide to select and assess their preferred apps in their practices.

Spanish adaptation and validation of the User Version of the Mobile Application Rating Scale (uMARS)

OBJECTIVE

While the professional version of the Mobile App Rating Scale (MARS) has already been translated, and validated into the Spanish language, its user-centered counterpart has not yet been adapted. Furthermore, no other similar tools exist in the Spanish language. The aim of this paper is to adapt and validate User Version of the MARS (uMARS) into the Spanish language.

MATERIALS AND METHODS

Cross-cultural adaptation, translation, and metric evaluation. The internal consistency and test-retest reliability of the Spanish version of the uMARS were evaluated using the RadarCovid app. Two hundred and sixteen participants rated the app using the translated scale. The app was then rated again 2 weeks later by 21 of these participants to measure test-retest reliability.

RESULTS

No major differences were observed between the uMARS original and the Spanish version. Discrimination indices (item-scale correlation) obtained appropriate results for both raters. The Spanish uMARS presented with excellent internal consistency, α = .89 and .67 for objective and subjective quality, respectively, and temporal stability (r > 0.82 for all items and subscales).

DISCUSSION

The Spanish uMARS is a useful tool for health professionals to recommend high-quality mobile apps to their patients based on the user's perspective and for researchers and app developers to use end-user feedback and evaluation, to help them identify highly appraised and valued components, as well as areas for further development, to continue ensuring the increasing quality and prominence of the area of mHealth.

CONCLUSION

uMARS Spanish version is an instrument with adequate metric properties to assess the quality of health apps from the user perspective.

Is the quality of mobile health applications for burns being adequately assessed?

High quality mobile health applications (mhealth apps) have the potential to enhance the prevention, diagnosis and treatment of burns. The primary aim of this study was to evaluate whether the quality of mhealth apps for burns care is being adequately assessed. The secondary aim was to determine whether these apps meet regulatory standards in the UK. We searched AMED, BNI, CINAHL, Cochrane library, Embase, Emcare, Medline and PsychInfo to identify studies assessing mhealth app quality for burns. The PRISMA reporting guideline was adhered to. Two independent reviewers screened abstracts to identify relevant studies. The quality of identified studies was assessed according to the framework proposed by Nouri et al, including design, information/content, usability, functionality, ethical issues, security/privacy and user-perceived value. Of the 28 included studies, none assessed all seven domains of quality. Design was assessed in 4/28 studies; information/content in 26/28 studies; usability in 12/28 studies; functionality in 10/28 studies; ethical issues were never assessed in any studies; security/privacy was not assessed; subjective assessment was made in 9/28 studies. 17/28 studies included apps that met the definition of 'medical device' according to MHRA guidance, yet only one app was appropriately certified with the UK Conformity Assessed (UKCA) mark. The quality of mHealth apps for burns are not being adequately assessed. The majority of apps should be considered medical devices according to UK standards, yet only one was appropriately certified. Regulatory bodies should support mhealth app developers, so as to improve quality control whilst simultaneously fostering innovation.

Characteristics of Acute Childhood Illness Apps for Parents: Environmental Scan

Background

Providing parents with resources that aid in the identification and management of acute childhood illnesses helps those parents feel better equipped to assess their children’s health and significantly changes parental health-seeking behaviors. Some of these resources are limited by accessibility and scalability. Remote locations and staffing limitations create challenges for parents aiming to access their child’s health information. Mobile health apps offer a scalable, accessible solution for improving health literacy by enabling access to health information through mobile devices. 

Objective

The aim of our study is to create an inventory of acute childhood illness apps that are available to North American parents and caregivers, assess their quality, and identify the areas in which future apps can be improved.

Methods

We conducted an environmental scan to identify and summarize app information for parents and digital health researchers. The Google and Apple app marketplaces were used as search platforms. We built a list of search terms and searched the platforms for apps targeted at parents and related to acute pediatric illnesses in the United States and Canada. We assessed apps meeting the inclusion criteria using the Mobile App Rating Scale (MARS), a validated tool for assessing the quality of health apps. The MARS examines apps on 5 subscales: engagement, functionality, aesthetics, information quality, and subjective quality. Data were analyzed by MARS subscale averages and individual item scores.

Results

Overall, 650 unique apps were screened, and 53 (8.2%) were included. On a scale of 1-5, apps had an average engagement score of 2.82/5 (SD 0.86), functionality score of 3.98/5 (SD 0.72), aesthetics score of 3.09/5 (SD 0.87), information quality score of 2.73/5 (SD 1.32), and subjective quality score of 2.20/5 (SD 0.79). On the same scale of 1-5, app scores ranged from 2.2/5 to 4.5/5 (mean 3.2, SD 0.6). The top 3 MARS-scored apps were Baby and Child First Aid (4.5/5), Ada (4.5/5), and HANDi Paediatric (4.2/5). Taken together, the top 3 apps covered topics of emergency pediatric first aid, identification of (and appropriate response to) common childhood illnesses, a means of checking symptoms, and a means of responding to emergency situations. There was a lack of Canadian-based app content available to parents in both marketplaces; this space was filled with content originating primarily in the United Kingdom and the United States. In addition, published evidence of the impact of the included apps was poor: of 53 apps, only 5 (9%) had an evidence base showing that the app had been trialed for usability or efficacy.

Conclusions

There is a need for evidence-based acute childhood illness apps of Canadian origin. This environmental scan offers a comprehensive picture of the health app landscape by examining trends in acute childhood illness apps that are readily available to parents and by identifying gaps in app design.

Scoping review: the empowerment of Alzheimer's Disease caregivers with mHealth applications

Alzheimer's Disease (AD) is one of the most prevalent neurodegenerative chronic diseases. As it progresses, patients become increasingly dependent, and their caregivers are burdened with the increasing demand for managing their care. Mobile health (mHealth) technology, such as smartphone applications, can support the need of these caregivers. This paper examines the published academic literature of mHealth applications that support the caregivers of AD patients. Following the PRISMA for scoping reviews, we searched published literature in five electronic databases between January 2014 and January 2021. Twelve articles were included in the final review. Six themes emerged based on the functionalities provided by the reviewed applications for caregivers. They are tracking, task management, monitoring, caregiver mental support, education, and caregiver communication platform. The review revealed that mHealth applications for AD patients' caregivers are inadequate. There is an opportunity for industry, government, and academia to fill the unmet need of these caregiver.

Promoting Health via mHealth Applications Using a French Version of the Mobile App Rating Scale: Adaptation and Validation Study

Background

In the recent decades, the number of apps promoting health behaviors and health-related strategies and interventions has increased alongside the number of smartphone users. Nevertheless, the validity process for measuring and reporting app quality remains unsatisfactory for health professionals and end users and represents a public health concern. The Mobile Application Rating Scale (MARS) is a tool validated and widely used in the scientific literature to evaluate and compare mHealth app functionalities. However, MARS is not adapted to the French culture nor to the language.

Objective

This study aims to translate, adapt, and validate the equivalent French version of MARS (ie, MARS-F).

Methods

The original MARS was first translated to French by two independent bilingual scientists, and their common version was blind back-translated twice by two native English speakers, culminating in a final well-established MARS-F. Its comprehensibility was then evaluated by 6 individuals (3 researchers and 3 nonacademics), and the final MARS-F version was created. Two bilingual raters independently completed the evaluation of 63 apps using MARS and MARS-F. Interrater reliability was assessed using intraclass correlation coefficients. In addition, internal consistency and validity of both scales were assessed. Mokken scale analysis was used to investigate the scalability of both MARS and MARS-F.

Results

MARS-F had a good alignment with the original MARS, with properties comparable between the two scales. The correlation coefficients (r) between the corresponding dimensions of MARS and MARS-F ranged from 0.97 to 0.99. The internal consistencies of the MARS-F dimensions engagement (ω=0.79), functionality (ω=0.79), esthetics (ω=0.78), and information quality (ω=0.61) were acceptable and that for the overall MARS score (ω=0.86) was good. Mokken scale analysis revealed a strong scalability for MARS (Loevinger H=0.37) and a good scalability for MARS-F (H=0.35).

Conclusions

MARS-F is a valid tool, and it would serve as a crucial aid for researchers, health care professionals, public health authorities, and interested third parties, to assess the quality of mHealth apps in French-speaking countries.

Scoping review: Development and assessment of evaluation frameworks of mobile health apps for recommendations to consumers

Objective

The study sought to review the different assessment items that have been used within existing health app evaluation frameworks aimed at individual, clinician, or organizational users, and to analyze the scoring and evaluation methods used in these frameworks.

Materials and Methods

We searched multiple bibliographic databases and conducted backward searches of reference lists, using search terms that were synonyms of “health apps,” “evaluation,” and “frameworks.” The review covered publications from 2011 to April 2020. Studies on health app evaluation frameworks and studies that elaborated on the scaling and scoring mechanisms applied in such frameworks were included.

Results

Ten common domains were identified across general health app evaluation frameworks. A list of 430 assessment criteria was compiled across 97 identified studies. The most frequently used scaling mechanism was a 5-point Likert scale. Most studies have adopted summary statistics to generate the total scoring of each app, and the most popular approach taken was the calculation of mean or average scores. Other frameworks did not use any scaling or scoring mechanism and adopted criteria-based, pictorial, or descriptive approaches, or “threshold” filter.

Discussion

There is wide variance in the approaches to evaluating health apps within published frameworks, and this variance leads to ongoing uncertainty in how to evaluate health apps.

Conclusions

A new evaluation framework is needed that can integrate the full range of evaluative criteria within one structure, and provide summative guidance on health app rating, to support individual app users, clinicians, and health organizations in choosing or recommending the best health app.

Procedures of User-Centered Usability Assessment for Digital Solutions: Scoping Review of Reviews Reporting on Digital Solutions Relevant for Older Adults

BACKGROUND

The assessment of usability is a complex process that involves several steps and procedures. It is important to standardize the evaluation and reporting of usability procedures across studies to guide researchers, facilitate comparisons across studies, and promote high-quality usability studies. The first step to standardizing is to have an overview of how usability study procedures are reported across the literature.

OBJECTIVE

This scoping review of reviews aims to synthesize the procedures reported for the different steps of the process of conducting a user-centered usability assessment of digital solutions relevant for older adults and to identify potential gaps in the present reporting of procedures. The secondary aim is to identify any principles or frameworks guiding this assessment in view of a standardized approach.

METHODS

This is a scoping review of reviews. A 5-stage scoping review methodology was used to identify and describe relevant literature published between 2009 and 2020 as follows: identify the research question, identify relevant studies, select studies for review, chart data from selected literature, and summarize and report results. The research was conducted on 5 electronic databases: PubMed, ACM Digital Library, IEEE, Scopus, and Web of Science. Reviews that met the inclusion criteria (reporting on user-centered usability evaluation procedures for any digital solution that could be relevant for older adults and were published in English) were identified, and data were extracted for further analysis regarding study evaluators, study participants, methods and techniques, tasks, and test environment.

RESULTS

A total of 3958 articles were identified. After a detailed screening, 20 reviews matched the eligibility criteria. The characteristics of the study evaluators and participants and task procedures were only briefly and differently reported. The methods and techniques used for the assessment of usability are the topics that were most commonly and comprehensively reported in the reviews, whereas the test environment was seldom and poorly characterized.

CONCLUSIONS

A lack of a detailed description of several steps of the process of assessing usability and no evidence on good practices of performing it suggests that there is a need for a consensus framework on the assessment of user-centered usability evaluation. Such a consensus would inform researchers and allow standardization of procedures, which are likely to result in improved study quality and reporting, increased sensitivity of the usability assessment, and improved comparability across studies and digital solutions. Our findings also highlight the need to investigate whether different ways of assessing usability are more sensitive than others. These findings need to be considered in light of review limitations.

Standards, Processes and Tools Used to Evaluate Health Information Systems Quality: A Systematic Literature Review (Preprint)

Background

Evaluating health information system (HIS) quality is strategically advantageous for improving the quality of patient care. Nevertheless, few systematic studies have reported what methods, such as standards, processes, and tools, were proposed to evaluate HIS quality.

Objective

This study aimed to identify and discuss the existing literature that describes standards, processes, and tools used to evaluate HIS quality.

Methods

We conducted a systematic literature review using review guidelines focused on software and systems. We examined seven electronic databases—Scopus, ACM (Association for Computing Machinery), ScienceDirect, Google Scholar, IEEE Xplore, Web of Science, and PubMed—to search for and select primary studies.

Results

Out of 782 papers, we identified 17 (2.2%) primary studies. We found that most of the primary studies addressed quality evaluation from a management perspective. On the other hand, there was little explicit and pragmatic evidence on the processes and tools that allowed for the evaluation of HIS quality.

Conclusions

To promote quality evaluation of HISs, it is necessary to define mechanisms and methods that operationalize the standards in HISs. Additionally, it is necessary to create metrics that measure the quality of the most critical components and processes of HISs.

A Smartphone App With a Digital Care Pathway for Patients Undergoing Spine Surgery: Development and Feasibility Study

BACKGROUND

There is a great unmet clinical need to provide patients undergoing spinal surgery and their caregivers with ongoing, high-quality care before and after surgery in an efficiency-focused health care environment.

OBJECTIVE

The objective of this study is to design, develop, and evaluate the acceptability and feasibility of a novel planning-, outcomes-, and analytics-based smartphone app called ManageMySurgery (MMS) in patients undergoing elective spine surgery (MMS-Spine).

METHODS

The development process of the MMS app was conducted over 2 sequential stages: (1) an evidence-based intervention design with refinement from surgeon and patient feedback and (2) feasibility testing in a clinical pilot study. We developed a novel, mobile-based, Health Insurance Portability and Accountability Act-compliant platform for interventional and surgical procedures. It is a patient-centric mobile health app that streamlines patients' interactions with their care team. MMS divides the patient journey into phases, making it feasible to provide customized care pathways that meet patients' unique needs. Patient-reported outcomes are easily collected and conform to the National Institutes of Health Patient-Reported Outcomes Measurement Information System (PROMIS) standard.

RESULTS

We tested the feasibility of the MMS-Spine app with patients undergoing elective spine surgery at a large academic health system. A total of 47 patients undergoing elective spine surgery (26 cervical spine and 21 lumbar spine surgeries) downloaded and used MMS-Spine to navigate their surgical journey, quantify their baseline characteristics and postoperative outcomes, and provide feedback on the utility of the app in preparing for and recovering from their spinal surgery. The median age was 59.0 (range 33-77) years, 22 of the 47 patients (47%) were women, and 26 patients (55%) had commercial insurance. Of the 47 patients, a total of 33 (70%) logged in on an iOS device, 11 (23%) on an Android device, and 3 (6%) on a computer or tablet. A total of 17 of the 47 patients (36%) added a caregiver, of which 7 (41%) logged in. The median number of sign-ins was 2. A total of 38 of 47 patients (81%) completed their baseline preoperative PROMIS-29 outcomes, and 14 patients (30%) completed at least one PROMIS-29 survey during the postoperative period. Of the 24 patients who completed the MMS survey, 21 (88%) said it was helpful during preparation for their procedure, 16 (67%) said it was helpful during the postoperative period, and 23 (96%) said that they would recommend MMS to a friend or family member.

CONCLUSIONS

We used a patient-centered approach based on proven behavior change techniques to develop a comprehensive smartphone app for patients undergoing elective spine surgery. The optimized version of the app is ready for formal testing in a larger randomized clinical study to establish its cost-effectiveness and effect on patients' self-management skills and long-term outcomes.

Rating of Pelvic Floor Muscle Training Mobile Applications for Treatment of Urinary Incontinence in Women

OBJECTIVE

To identify pelvic floor muscle therapy mobile health applications (apps) targeting women with urinary incontinence (UI), and evaluate them in a standardized fashion.

METHODS

A systematic search of English language apps on the Canadian App Store (iOS) and Google Play (Android) Store was performed. Eligible apps were evaluated independently by 5 reviewers using the validated Mobile App Rating Scale (MARS) tool. Descriptive characteristics were summarized and MARS subscale and overall quality scores werereported.

RESULTS

Of 139 mobile health apps identified, 20 unique apps were included for full review, of which there were 7 iOS only apps, 6 Android only apps, and 7 apps available in both stores. At the time of analysis, most apps had been updated within the last year (60%). Only 1 app had been trialed and verified by evidence in scientific literature. The majority of apps were free to download (80%). The median (interquartile range) MARS overall quality score was 3.7 (0.8) on a 0-5 scale, ranging from 2.7 to 4.1. The highest-rated subscale was "functionality" with a median score of 4.1 (0.6); the lowest-rated was "information" with a median score of 3.4 (0.6). The median MARS subjective quality score was 2.9 (1.0).

CONCLUSION

There are both free and paid apps available on-line that deliver pelvic floor muscle therapy programs. Evaluation using the MARS tool identified that many apps are not of high quality, and only 1 was evidence-based and has been trialed clinically. This knowledge is relevant to the choice of apps by both patients and caregivers.

A Smartphone App With a Digital Care Pathway for Patients Undergoing Spine Surgery: Development and Feasibility Study (Preprint).. [DOI: 10.2196/preprints.21138] [Citation(s) in RCA: 0] [Impact Index Per Article: 0]

BACKGROUND

There is a great unmet clinical need to provide patients undergoing spinal surgery and their caregivers with ongoing, high-quality care before and after surgery in an efficiency-focused health care environment.

OBJECTIVE

The objective of this study is to design, develop, and evaluate the acceptability and feasibility of a novel planning-, outcomes-, and analytics-based smartphone app called ManageMySurgery (MMS) in patients undergoing elective spine surgery (MMS-Spine).

METHODS

The development process of the MMS app was conducted over 2 sequential stages: (1) an evidence-based intervention design with refinement from surgeon and patient feedback and (2) feasibility testing in a clinical pilot study. We developed a novel, mobile-based, Health Insurance Portability and Accountability Act–compliant platform for interventional and surgical procedures. It is a patient-centric mobile health app that streamlines patients’ interactions with their care team. MMS divides the patient journey into phases, making it feasible to provide customized care pathways that meet patients’ unique needs. Patient-reported outcomes are easily collected and conform to the National Institutes of Health Patient-Reported Outcomes Measurement Information System (PROMIS) standard.

RESULTS

We tested the feasibility of the MMS-Spine app with patients undergoing elective spine surgery at a large academic health system. A total of 47 patients undergoing elective spine surgery (26 cervical spine and 21 lumbar spine surgeries) downloaded and used MMS-Spine to navigate their surgical journey, quantify their baseline characteristics and postoperative outcomes, and provide feedback on the utility of the app in preparing for and recovering from their spinal surgery. The median age was 59.0 (range 33-77) years, 22 of the 47 patients (47%) were women, and 26 patients (55%) had commercial insurance. Of the 47 patients, a total of 33 (70%) logged in on an iOS device, 11 (23%) on an Android device, and 3 (6%) on a computer or tablet. A total of 17 of the 47 patients (36%) added a caregiver, of which 7 (41%) logged in. The median number of sign-ins was 2. A total of 38 of 47 patients (81%) completed their baseline preoperative PROMIS-29 outcomes, and 14 patients (30%) completed at least one PROMIS-29 survey during the postoperative period. Of the 24 patients who completed the MMS survey, 21 (88%) said it was helpful during preparation for their procedure, 16 (67%) said it was helpful during the postoperative period, and 23 (96%) said that they would recommend MMS to a friend or family member.

CONCLUSIONS

We used a patient-centered approach based on proven behavior change techniques to develop a comprehensive smartphone app for patients undergoing elective spine surgery. The optimized version of the app is ready for formal testing in a larger randomized clinical study to establish its cost-effectiveness and effect on patients’ self-management skills and long-term outcomes.