Technology Evaluation and Assessment Criteria for Health Apps (TEACH-Apps): Pilot Study

Mobile applications in nursing science education: A scoping review with snowballing method

OBJECTIVES

To review the available evidence on the use and effectiveness of mobile applications to assist nursing students in comprehending, utilising, and applying specialised language and knowledge terminologies when learning the language of biosciences.

DESIGN

A scoping review.

DATA SOURCES

The databases CINAHL Complete, ERIC, EMCare, MEDLINE, PubMed, the OVID scholarly interface and the web search engine Google Scholar were searched.

REVIEW METHODS

Peer-reviewed literature published in English during the period 2010-2023 was reviewed. Snowballing methods saw the reference lists of all included articles searched, and a secondary search of the Scopus ranked top ten nursing journals. Articles were included if they reported on any app or digital resource used when teaching undergraduate nursing students biosciences/science language skills, concepts, or terminology. Studies were excluded if the participants were non-nursing student cohorts or content did not meet the inclusion criteria.

RESULTS

Mobile applications generally contribute positively to nursing students' education. These applications are deemed valuable tools, offering structured content in easily digestible formats. Some applications also foster teamwork and collaboration during clinical placements, promoting peer learning, and a sense of community. User internet access and preparation for learning were the only noted barriers. A range of science-based concepts were taught using applications, including diabetes mellitus, medical terminology, asthma, and cardiac conditions. Despite the promise shown by using mobile applications to teach nurse sciences, few are dedicated to bioscience language and scientific terminology. Recognising the challenging nature of teaching these concepts, developing specialised applications could substantially improve the educational experience for nursing students.

CONCLUSION

Nurse educators are encouraged to teach with applications given their reported effectiveness in knowledge gains for students learning science concepts. Rigorous interventional study designs are warranted to extend suggestions that using applications enhances student understanding of challenging scientific concepts and support quality in clinical learning.

The Digital Navigator: Standardizing Human Technology Support in App-Integrated Clinical Care

Background: Mental health apps offer scalable care, yet clinical adoption is hindered by low user engagement and integration challenges into clinic workflows. Human support staff called digital navigators, trained in mental health technology, could enhance care access and patient adherence and remove workflow burdens from clinicians. While the potential of this role is clear, training staff to become digital navigators and assessing their impact are primary challenges. Methods: We present a detailed manual/framework for implementation of the Digital Navigator within a short-term, cognitive-behavioral therapy-focused hybrid clinic. We analyze patient engagement, satisfaction, and digital phenotyping data quality outcomes. Data from 83 patients, for the period spanning September 2022 to September 2023, included Digital Navigator satisfaction, correlated with demographics, mindLAMP app satisfaction, engagement, and passive data quality. Additionally, average passive data across 33 clinic patients from November 2023 to January 2024 were assessed for missingness. Results: Digital Navigator satisfaction averaged 18.8/20. Satisfaction was not influenced by sex, race, gender, or education. Average passive data quality across 33 clinic patients was 0.82 at the time this article was written. Digital Navigator satisfaction scores had significant positive correlation with both clinic app engagement and perception of that app. Conclusions: Results demonstrate preliminary support and patient endorsement for the Digital Navigator role and positive outcomes around digital engagement and digital phenotyping data quality. Through sharing training resources and standardizing the role, we aim to enable clinicians and researchers to adapt and utilize the Digital Navigator for their own needs.

Quality assessment of mHealth apps: a scoping review

Introduction

The number of mHealth apps has increased rapidly during recent years. Literature suggests a number of problems and barriers to the adoption of mHealth apps, including issues such as validity, usability, as well as data privacy and security. Continuous quality assessment and assurance systems might help to overcome these barriers. Aim of this scoping review was to collate literature on quality assessment tools and quality assurance systems for mHealth apps, compile the components of the tools, and derive overarching quality dimensions, which are potentially relevant for the continuous quality assessment of mHealth apps.

Methods

Literature searches were performed in Medline, EMBASE and PsycInfo. Articles in English or German language were included if they contained information on development, application, or validation of generic concepts of quality assessment or quality assurance of mHealth apps. Screening and extraction were carried out by two researchers independently. Identified quality criteria and aspects were extracted and clustered into quality dimensions.

Results

A total of 70 publications met inclusion criteria. Included publications contain information on five quality assurance systems and further 24 quality assessment tools for mHealth apps. Of these 29 systems/tools, 8 were developed for the assessment of mHealth apps for specific diseases, 16 for assessing mHealth apps for all fields of health and another five are not restricted to health apps. Identified quality criteria and aspects were extracted and grouped into a total of 14 quality dimensions, namely "information and transparency", "validity and (added) value", "(medical) safety", "interoperability and compatibility", "actuality", "engagement", "data privacy and data security", "usability and design", "technology", "organizational aspects", "social aspects", "legal aspects", "equity and equality", and "cost(-effectiveness)".

Discussion

This scoping review provides a broad overview of existing quality assessment and assurance systems. Many of the tools included cover only a few dimensions and aspects and therefore do not allow for a comprehensive quality assessment or quality assurance. Our findings can contribute to the development of continuous quality assessment and assurance systems for mHealth apps.

Systematic Review Registration

https://www.researchprotocols.org/2022/7/e36974/, International Registered Report Identifier, IRRID (DERR1-10.2196/36974).

Quality of Digital Health Interventions Across Different Health Care Domains: Secondary Data Analysis Study

BACKGROUND

There are more than 350,000 digital health interventions (DHIs) in the app stores. To ensure that they are effective and safe to use, they should be assessed for compliance with best practice standards.

OBJECTIVE

The objective of this paper was to examine and compare the compliance of DHIs with best practice standards and adherence to user experience (UX), professional and clinical assurance (PCA), and data privacy (DP).

METHODS

We collected assessment data from 1574 DHIs using the Organisation for the Review of Care and Health Apps Baseline Review (OBR) assessment tool. As part of the assessment, each DHI received a score out of 100 for each of the abovementioned areas (ie, UX, PCA, and DP). These 3 OBR scores are combined to make up the overall ORCHA score (a proxy for quality). Inferential statistics, probability distributions, Kruskal-Wallis, Wilcoxon rank sum test, Cliff delta, and Dunn tests were used to conduct the data analysis.

RESULTS

We found that 57.3% (902/1574) of the DHIs had an Organisation for the Review of Care and Health Apps (ORCHA) score below the threshold of 65. The overall median OBR score (ORCHA score) for all DHIs was 61.5 (IQR 51.0-73.0) out of 100. A total of 46.2% (12/26) of DHI's health care domains had a median equal to or above the ORCHA threshold score of 65. For the 3 assessment areas (UX, DP, and PCA), DHIs scored the highest for the UX assessment 75.2 (IQR 70.0-79.6), followed by DP 65.1 (IQR 55.0-73.4) and PCA 49.6 (IQR 31.9-76.1). UX scores had the least variance (SD 13.9), while PCA scores had the most (SD 24.8). Respiratory and urology DHIs were consistently highly ranked in the National Institute for Health and Care Excellence Evidence Standards Framework tiers B and C based on their ORCHA score.

CONCLUSIONS

There is a high level of variability in the ORCHA scores of DHIs across different health care domains. This suggests that there is an urgent need to improve compliance with best practices in some health care areas. Possible explanations for the observed differences might include varied market maturity and commercial interests within the different health care domains. More investment to support the development of higher-quality DHIs in areas such as ophthalmology, allergy, women's health, sexual health, and dental care may be needed.

Assessing the Quality and Impact of eHealth Tools: Systematic Literature Review and Narrative Synthesis

BACKGROUND

Technological advancements have opened the path for many technology providers to easily develop and introduce eHealth tools to the public. The use of these tools is increasingly recognized as a critical quality driver in health care; however, choosing a quality tool from the myriad of tools available for a specific health need does not come without challenges.

OBJECTIVE

This review aimed to systematically investigate the literature to understand the different approaches and criteria used to assess the quality and impact of eHealth tools by considering sociotechnical factors (from technical, social, and organizational perspectives).

METHODS

A structured search was completed following the participants, intervention, comparators, and outcomes framework. We searched the PubMed, Cochrane, Web of Science, Scopus, and ProQuest databases for studies published between January 2012 and January 2022 in English, which yielded 675 results, of which 40 (5.9%) studies met the inclusion criteria. The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines and the Cochrane Handbook for Systematic Reviews of Interventions were followed to ensure a systematic process. Extracted data were analyzed using NVivo (QSR International), with a thematic analysis and narrative synthesis of emergent themes.

RESULTS

Similar measures from the different papers, frameworks, and initiatives were aggregated into 36 unique criteria grouped into 13 clusters. Using the sociotechnical approach, we classified the relevant criteria into technical, social, and organizational assessment criteria. Technical assessment criteria were grouped into 5 clusters: technical aspects, functionality, content, data management, and design. Social assessment criteria were grouped into 4 clusters: human centricity, health outcomes, visible popularity metrics, and social aspects. Organizational assessment criteria were grouped into 4 clusters: sustainability and scalability, health care organization, health care context, and developer.

CONCLUSIONS

This review builds on the growing body of research that investigates the criteria used to assess the quality and impact of eHealth tools and highlights the complexity and challenges facing these initiatives. It demonstrates that there is no single framework that is used uniformly to assess the quality and impact of eHealth tools. It also highlights the need for a more comprehensive approach that balances the social, organizational, and technical assessment criteria in a way that reflects the complexity and interdependence of the health care ecosystem and is aligned with the factors affecting users' adoption to ensure uptake and adherence in the long term.

Features and characteristics of publicly available mHealth apps for self-management in chronic obstructive pulmonary disease

Mobile health applications (mHealth apps) may be able to support people living with chronic obstructive pulmonary disease (COPD) to develop the appropriate skills and routines for adequate self-management. Given the wide variety of publicly available mHealth apps, it is important to be aware of their characteristics to optimize their use and mitigate potential harms.

Objective

To report the characteristics and features of publicly available apps for COPD self-management.

Methods

MHealth apps designed for patients' COPD self-management were searched in the Google Play and Apple app stores. Two reviewers trialed and assessed the eligible apps using the MHealth Index and Navigation Database framework to describe the characteristics, qualities, and features of mHealth apps across five domains.

Results

From the Google Play and Apple stores, thirteen apps were identified and eligible for further evaluation. All thirteen apps were available for Android devices, but only seven were available for Apple devices. Most apps were developed by for-profit organizations (8/13), non-profit organizations (2/13), and unknown developers (3/13). Many apps had privacy policies (9/13), but only three apps described their security systems and two mentioned compliance with local health information and data usage laws. Education was the common app feature; additional features were medication reminders, symptom tracking, journaling, and action planning. None provided clinical evidence to support their use.

Conclusions

Publicly available COPD apps vary in their designs, features, and overall quality. These apps lack evidence to support their clinical use and cannot be recommended at this time.

Evaluation and comparison of smartphone application tracing, web based artificial intelligence tracing and conventional hand tracing methods

AIM

The aim of this study was to compare and evaluate the reliability of three different cephalometric assessment methods: Smartphone Application Tracing Method CephNinja (SATM), Web Based Artificial Intelligence (AI) Driven Tracing Method WebCeph (WATM) and Conventional Hand Tracing Method (CHTM).

METHODS

110 lateral cephalometric radiographs were enrolled in the study and 4 linear and 7 angular parameters were traced and measured by one examiner using CephNinja, WebCeph and conventional hand tracing methods. Independent-samples Kolmogorov-Smirnov and Shapiro Wilks tests were used to compare the mean values of intra-examiner differences. Both intra-method and inter-method correlations were evaluated.

RESULTS

There were statistically significant differences between the methods in terms of SNA (p:0.003; p < 0.05); SNB measurements (p:0.001; p < 0.05); SN-MP angle (p:0.001; p < 0.05); U1-SN angle (p:0.001; p < 0.05); L1-NB(mm) (p:0.007; p < 0.05) and E Line-Upper Lip(mm) measurements (p:0.013; p < 0.05). All intra-method correlation coefficients are 80% and above. In terms of inter-method coefficients the lowest coefficient of agreement is 0.170 and it is the coefficient of agreement between CHTM and SATM for measurement of U1-NA. The highest coefficient of agreement is 0.884 which is between WBTM and SATM for SNB measurement.

CONCLUSIONS

The null hypothesis was rejected. There were statistically and clinically significant differences in various measurements among groups.

Benefit Assessment and Reimbursement of Digital Health Applications: Concepts for Setting Up a New System for Public Coverage

In Germany, some digital health applications (DiHA) became reimbursable through the statutory health insurance system with the adoption of the Digital Healthcare Act in 2019. Approaches and concepts for the German care context were developed in an iterative process, based on existing concepts from international experience. A DiHA categorization was developed that could be used as a basis to enable the creation of a reimbursed DiHA repository, and to derive evidence requirements for coverage and reimbursement for each DiHA. The results provide an overview of a possible classification of DiHA as well as approaches to assessment and evaluation. The structure of remuneration and pricing in connection with the formation of groups is demonstrated.

Curating the Digital Mental Health Landscape With a Guide to Behavioral Health Apps: A County-Driven Resource

With more than 10,000 mental health apps available, consumers and clinicians who want to adopt such tools can be overwhelmed by the multitude of options and lack of clear evaluative standards. Despite the increasing prevalence of curated lists, or app guides, challenges remain. Organizations providing mental health services to consumers have an opportunity to address these challenges by producing guides that meet relevant standards of quality and are tailored to local needs. This column summarizes an example of the collaborative process of app guide development in a publicly funded mental health service context and highlights opportunities and barriers identified through the process.

Mental Health App Evaluation: Updating the American Psychiatric Association's Framework Through a Stakeholder-Engaged Workshop

The app evaluation framework of the American Psychiatric Association (APA) has emerged as an adaptable tool for those seeking to navigate the ever-growing space of mental health apps. The authors describe a meeting convened in December 2019 to refine the APA framework. The expert panel comprised 16 individuals across health care fields, with representation from psychiatry, psychology, social work, nursing, clinical informatics, peer support specialists, and individuals with lived mental health experience. This meeting resulted in an update to the APA framework with the addition of clearer question descriptions and the release of an eight-question screener that may be useful in clinical settings.

Assessing Patient Proficiency with Internet-Connected Technology and Their Preferences for E-Health in Cirrhosis

There is a rapidly evolving need for e-health to support chronic disease self-management and connect patients with their healthcare teams. Patients with cirrhosis have a high symptom burden, significant comorbidities, and a range of psychological and cognitive issues. Patients with cirrhosis were assessed for their readiness and interest in e-health. Adults attending one of two outpatient cirrhosis clinics in Alberta were recruited. Eligible participants were not required to own or have experience with digital technologies or the Internet. Medical history, socioeconomic status, and attitudes regarding e-health, the Computer Proficiency Questionnaire, and the Mobile Device Proficiency Questionnaire were used to describe participants' knowledge and skills. Of the 117 recruited patients, 68.4% owned a computer and 84.6% owned a mobile device. Patients had mean proficiency scores of 72.8% (SD 25.9%) and 69.3% (SD 26.4%) for these devices, respectively. In multiple regression analyses, significant predictors of device proficiency were age, education, and household income. Most patients (78.7%) were confident they could participate in videoconferencing after training and most (61.5%) were interested in an online personalized health management program. This diverse group of patients with cirrhosis had technology ownership, proficiency, and online behaviours similar to the general population. Moreover, the patients were very receptive to e-health if training was provided. This promising data is timely given the unique demands of COVID-19 and its influence on self-management and healthcare delivery to a vulnerable population.

Enhancing attention and memory of individuals at clinical high risk for psychosis with mHealth technology

BACKGROUND

Cognitive impairment has adverse impact on the social and role functions of those at clinical high risk for psychosis and it has become an important target for intervention. Mobile health applications are user-friendly, real-time, personalized and portable in administering cognitive training and have promising application prospects in the field of mental health.

METHODS

Eighty CHR subjects were randomized into an intervention group and a control group. CHR subjects of the intervention group performed attention and memory training via a Specific Memory Attention Resource and Training (SMART) application in their smart phones for 10 min per day, five days per week for three months. Both groups were followed up for three months. At baseline and follow-up phases, cognitive function was measured using the MATRICS Consensus Cognitive Battery (MCCB). In the follow-up, the intervention group completed the Mobile Application Rating Scale (MARS) to provide feedback to improve SMART.

RESULTS

There is a significant group by time interaction effect in the Attention/Vigilance domain, which is significantly better in the intervention group than in the control group at 3- month follow-up. The improvement in Attention/Vigilance in the intervention group is significantly related to the amount of cognitive training time. Global Assessment of Function (GAF) reduction rate at baseline could predict the improvement of Attention/Vigilance. MARS results indicate that CHR subjects were receptive of SMART.

CONCLUSION

Mobile technology can be applied to improve cognitive function of CHR individuals, especially in the Attention/Vigilance domain.

Psychiatric rehabilitation through teaching smartphone skills to improve functional outcomes in serious mental illness

This study measured the impact of a digital competencies and skills course on participants with serious mental illness. Close to 75% of participants reported an improvement in a smartphone related skill, and the majority of participants that reported improvement in one skill reported improvement in at least one other. Qualitative feedback from participants suggests how digital competencies acquired were used to immediately support functional outcomes.

OBJECTIVE

To improve functional outcomes in patients with serious mental illness through a multi-session curriculum designed to improve smartphone skills and engage participants in group learning and problem solving, targeting negative and cognitive symptoms of illness.

METHODS

An eight-week smartphone digital competencies and skills course was offered to two distinct groups of youth with serious mental illness. Pre and post self-report measurements were captured for each participant for each session.

RESULTS

Group participation varied by session, but overall 28 unique patients attended. From survey results, 75% reported improvement in smartphone related skills because of the groups. Qualitative feedback suggests how skills acquired by patients were immediately utilized to gain insight into health and support functional outcomes.

CONCLUSIONS AND IMPLICATIONS FOR PRACTICE

Smartphone skills groups are a means to provide practical psychiatric rehabilitation that may enable some patients to compensate for cognitive and social deficits due to illness. While ensuring groups are responsive to patients with varying degrees of skills remains a challenge, adapting lesson structures and mediums, as well as creating new measurement tools, offers a means to modify the course with the clinical need.