Conceptual Framework for Development of Comprehensive e-Health Evaluation Tool

Developing and Evaluating Digital Public Health Interventions Using the Digital Public Health Framework DigiPHrame: A Framework Development Study

BACKGROUND

Digital public health (DiPH) interventions may help us tackle substantial public health challenges and reach historically underserved populations, in addition to presenting valuable opportunities to improve and complement existing services. However, DiPH interventions are often triggered through technological advancements and opportunities rather than public health needs. To develop and evaluate interventions designed to serve public health needs, a comprehensive framework is needed that systematically covers all aspects with relevance for public health. This includes considering the complexity of the technology, the context in which the technology is supposed to operate, its implementation, and its effects on public health, including ethical, legal, and social aspects.

OBJECTIVE

We aimed to develop such a DiPH framework with a comprehensive list of core principles to be considered throughout the development and evaluation process of any DiPH intervention.

METHODS

The resulting digital public health framework (DigiPHrame) was based on a scoping review of existing digital health and public health frameworks. After extracting all assessment criteria from these frameworks, we clustered the criteria. During a series of multidisciplinary meetings with experts from the Leibniz ScienceCampus Digital Public Health, we restructured each domain to represent the complexity of DiPH. In this paper, we used a COVID-19 contact-tracing app as a use case to illustrate how DigiPHrame may be applied to assess DiPH interventions.

RESULTS

The current version of DigiPHrame consists of 182 questions nested under 12 domains. Domain 1 describes the current status of health needs and existing interventions; domains 2 and 3, the DiPH technology under assessment and aspects related to human-computer interaction, respectively; domains 4 and 5, structural and process aspects, respectively; and domains 6-12, contextual conditions and the outcomes of the DiPH intervention from broad perspectives. In the CWA use case, a number of questions relevant during its development but also important for assessors once the CWA was available were highlighted.

CONCLUSIONS

DigiPHrame is a comprehensive framework for the development and assessment of digital technologies designed for public health purposes. It is a living framework and will, therefore, be updated regularly and as new public health needs and technological advancements emerge.

Preparation and validation of the instrument "QualiAPS digital-Brazil" for assessing digital health care in primary health care: a required tool

Introduction

The use of Information and Communication Technologies in the field of health is increasing across the world, demarcating the field of digital health. The goal of this study is to formulate and validate a matrix of indicators, design assessment scripts and indicate data collection techniques for assessing the quality of digital health care in Brazilian Primary Health Care (PHC).

Methodology

This is a validation study divided into three phases: preparation of the instrument, analysis of validity and pilot study. The instrument was prepared based on the PHC assessment model from a literature review; the analysis of validity used the Delphi technique associated with the nominal group and the evidence from the literature reference. In the pilot study, audio-recorded interviews were conducted with strategic primary care actors.

Results

The matrix of indicators "QualiAPS Digital-Brazil" introduces a set of 37 indicators, distributed into three distinct components and their respective dimensions. The component "Structure" includes the dimension "Resources"; the component "Processes" includes the dimensions "Technical," "Organizational" and "Relational"; and the component "Results" includes the dimensions "Short-Term Results" and "Medium-Term Results." The general values obtained for CVI and IRR were 0.89 and 1.00; respectively. Therefore, it was possible to design assessment scripts and indicate qualitative data collection techniques for assessing digital health in Brazilian PHC.

Conclusion

The instrument presented was validated regarding its relevance, content and theoretical support to evaluate the quality of digital health care, supporting decision-making by managers and health professionals in the search for improving remote primary care provided to the population.

Methodological Frameworks and Dimensions to Be Considered in Digital Health Technology Assessment: Scoping Review and Thematic Analysis

BACKGROUND

Digital health technologies (dHTs) offer a unique opportunity to address some of the major challenges facing health care systems worldwide. However, the implementation of dHTs raises some concerns, such as the limited understanding of their real impact on health systems and people's well-being or the potential risks derived from their use. In this context, health technology assessment (HTA) is 1 of the main tools that health systems can use to appraise evidence and determine the value of a given dHT. Nevertheless, due to the nature of dHTs, experts highlight the need to reconsider the frameworks used in traditional HTA.

OBJECTIVE

This scoping review (ScR) aimed to identify the methodological frameworks used worldwide for digital health technology assessment (dHTA); determine what domains are being considered; and generate, through a thematic analysis, a proposal for a methodological framework based on the most frequently described domains in the literature.

METHODS

The ScR was performed in accordance with the guidelines established in the PRISMA-ScR guidelines. We searched 7 databases for peer reviews and gray literature published between January 2011 and December 2021. The retrieved studies were screened using Rayyan in a single-blind manner by 2 independent authors, and data were extracted using ATLAS.ti software. The same software was used for thematic analysis.

RESULTS

The systematic search retrieved 3061 studies (n=2238, 73.1%, unique), of which 26 (0.8%) studies were included. From these, we identified 102 methodological frameworks designed for dHTA. These frameworks revealed great heterogeneity between them due to their different structures, approaches, and items to be considered in dHTA. In addition, we identified different wording used to refer to similar concepts. Through thematic analysis, we reduced this heterogeneity. In the first phase of the analysis, 176 provisional codes related to different assessment items emerged. In the second phase, these codes were clustered into 86 descriptive themes, which, in turn, were grouped in the third phase into 61 analytical themes and organized through a vertical hierarchy of 3 levels: level 1 formed by 13 domains, level 2 formed by 38 dimensions, and level 3 formed by 11 subdimensions. From these 61 analytical themes, we developed a proposal for a methodological framework for dHTA.

CONCLUSIONS

There is a need to adapt the existing frameworks used for dHTA or create new ones to more comprehensively assess different kinds of dHTs. Through this ScR, we identified 26 studies including 102 methodological frameworks and tools for dHTA. The thematic analysis of those 26 studies led to the definition of 12 domains, 38 dimensions, and 11 subdimensions that should be considered in dHTA.

Quality assessment of mobile applications on postpartum hemorrhage management

OBJECTIVE

To assess mobile application quality on the management of postpartum hemorrhage available in the digital stores of the main operating systems.

METHOD

A descriptive evaluative study, carried out from January to February 2023 on the App Store® and Google Play Store®. The Mobile Application Rating Scale was used to assess quality (engagement, functionality, aesthetics, information and subjective quality). Information extraction and assessment on postpartum hemorrhage was carried out using a table with information based on official documents, containing stratification, prevention, diagnosis and treatment.

RESULTS

Seven applications were included; of these, three were in English, six had an Android operating system. The quality mean was 3.88. The highest means were for functionality, reaching 5.0 (n = 6), and the lowest were for engagement, less than 3.0 (n = 4). The majority of applications presented less than 50% of the information on postpartum hemorrhage management.

CONCLUSION

The applications assessed achieved an acceptable quality mean and, according to health organizations' current protocols, did not contain the necessary information for complete postpartum hemorrhage management.

Presenting a Framework for Telebehavioral Health Implementation

PURPOSE OF REVIEW

This paper presents OPTIC as a framework to guide the conceptualization and implementation of telebehavioral health (TBH) in a comprehensive, structured, and accessible manner.

RECENT FINDINGS

There is a need for comprehensive frameworks for TBH implementation, yet current models and frameworks described in the literature have limitations. Many studies highlight favorable outcomes of TBH during COVID-19, along with increased adoption. However, despite the plethora of publications on general telehealth implementation, knowledge is disparate, inconsistent, not comprehensive, and not TBH-specific. The framework incorporates five components: Originating site, Patient population, Teleclinician, Information and communication technologies, and Cultural and regulatory context. These components, abbreviated using the acronym OPTIC, are discussed, with examples of implementation considerations under each component throughout the project cycle. The value and larger implications of OPTIC are discussed as a foundation for stakeholders involved with TBH, in addition to key performance indicators, and considerations for quality enhancement.

Assessment of healthcare application quality: Development of a standardized methods for healthcare professionals

The advancement of the mobile app market is reshaping healthcare, emphasizing the imperative for quality and efficacy in health applications. This methodology has been devised to assess mobile health applications, aiming to assist healthcare professionals in selecting apps for e-healthcare consumers. Key facets of this approach are: •A stringent selection process within mobile app stores•A standardized assessment using the Mobile App Rating Scale to achieve consistent and replicable evaluations, systematically organizing app evaluations•A comprehensive framework guiding healthcare practitioners in determining which apps to integrate into clinical practice and which to exclude Central to this method is the emphasis on distinguishing apps that enhance clinical practice from those that fall short in important areas such as the effectiveness of proposed health features, data accuracy, adherence to clinical guidelines, data security, and user privacy. With heightened attention to usability and accessibility, the methodology also addresses the common risks inherent in mHealth implementation, ensuring that selected apps not only meet technical criteria but also align with the broader healthcare ecosystem's needs and challenges.

Development of a novel instrument to characterize telemedicine programs in primary care

BACKGROUND

Given the rapid deployment of telemedicine at the onset of the COVID - 19 pandemic, updated assessment methods are needed to study and characterize telemedicine programs. We developed a novel semi - structured survey instrument to systematically describe the characteristics and implementation processes of telemedicine programs in primary care.

METHODS

In the context of a larger study aiming to describe telemedicine programs in primary care, a survey was developed in 3 iterative steps: 1) literature review to obtain a list of telemedicine features, facilitators, and barriers; 2) application of three evaluation frameworks; and 3) stakeholder engagement through a 2-stage feedback process. During survey refinement, items were tested against the evaluation frameworks while ensuring it could be completed within 20-25 min. Data reduction techniques were applied to explore opportunity for condensed variables/items.

RESULTS

Sixty initially identified telemedicine features were reduced to 32 items / questions after stakeholder feedback. Per the life cycle framework, respondents are asked to report a month in which their telemedicine program reached a steady state, i.e., "maturation". Subsequent questions on telemedicine features are then stratified by telemedicine services offered at the pandemic onset and the reported point of maturation. Several open - ended questions allow for additional telemedicine experiences to be captured. Data reduction techniques revealed no indication for data reduction.

CONCLUSION

This 32-item semi-structured survey standardizes the description of primary care telemedicine programs in terms of features as well as maturation process. This tool will facilitate evaluation of and comparisons between telemedicine programs across the United States, particularly those that were deployed at the pandemic onset.

Testing the Efficacy of a Scalable Telephone-Delivered Guided Imagery Tobacco Cessation Treatment: Protocol for a Randomized Clinical Trial

BACKGROUND

Tobacco use continues to be a leading preventable cause of death and disease in the United States, accounting for >480,000 deaths each year. Although treatments for tobacco use are effective for many, there is substantial variability in outcomes, and these approaches are not effective for all individuals seeking to quit smoking cigarettes. New, effective therapeutic approaches are needed to meet the preferences of people who want to stop smoking. Guided imagery (GI) is a mind-body technique that involves the guided visualization of specific mental images, which is enhanced with other sensory modalities and emotions. Preliminary evidence provides initial support for the use of GI as a treatment for cigarette smoking. Meta-analyses have shown that standard treatment for cigarette smoking delivered over the telephone via quitlines is effective. A telephone-based intervention that uses GI might provide another effective treatment option and increase the reach and effectiveness of quitlines.

OBJECTIVE

This study aims to test the efficacy of Be Smoke Free, a telephone-delivered GI treatment for smoking cessation.

METHODS

This multisite randomized clinical trial (RCT) will compare a novel telephone-delivered GI tobacco cessation treatment with a standard evidence-based behavioral treatment. The study will be conducted over 5 years. In phase 1, we refined protocols and procedures for the New York State and West Virginia sites for use in the RCT. During phase 2, we will conduct an RCT with 1200 participants: 600 (50%) recruited via quitlines and 600 (50%) recruited via population-based methods. Participants will be randomly assigned to either the GI condition or the behavioral condition; both treatments will be delivered by trained study coaches located at the University of Arizona. Assessments will be conducted at baseline and 3 and 6 months after enrollment by University of Arizona research staff. The primary outcome will be self-reported 30-day point prevalence abstinence 6 months after enrollment. Secondary outcomes include biochemically verified 7-day point prevalence abstinence 6 months after enrollment.

RESULTS

Recruitment in West Virginia and New York began in October 2022. As of March 31, 2023, a total of 242 participants had been enrolled. Follow-up assessments began in November 2022. As of March 31, 2023, of the 118 eligible participants, 97 (82.2%) had completed the 3-month assessment, and 93% (26/28) of eligible participants had completed the 6-month assessment. Biochemical verification and qualitative interviews began in April 2023. Recruitment will continue through 2025 and follow-up assessments through 2026. Primary results are expected to be published in 2027.

CONCLUSIONS

The Be Smoke Free study is a first-of-its-kind RCT that incorporates GI into telephone-based tobacco cessation treatment. If successful, Be Smoke Free will have substantial benefits for the long-term health of people who use tobacco across the United States.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05277831; https://clinicaltrials.gov/ct2/show/NCT05277831.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/48898.

eHealth Promoting Stoma Self-care for People With an Elimination Ostomy: Focus Group Study

BACKGROUND

The construction of an elimination stoma has a physical, psychological, and social impact on the person. The development of stoma self-care competence contributes to the adaptation to a new health condition and improvement of quality of life. eHealth refers to everything associated with information and communication technology and health care, including telemedicine, mobile health, and health informatics. The use of eHealth platforms by the person with an ostomy, as a digital application that includes websites and mobile phone apps, can bring scientific knowledge and well-informed practices to individuals, families, and communities. It also allows functionalities that enable the person to describe and identify early signs and symptoms and precursors of complications and to be guided to an adequate health response for their problems.

OBJECTIVE

This study aimed to define the most relevant content and features to promote ostomy self-care integrated into an eHealth platform as a digital app or website to be used by patients for self-management of stoma care.

METHODS

We developed a descriptive, exploratory study with a qualitative approach using the focus group methodology, which was oriented to reach a consensus of at least 80%. A convenience sample of 7 participants consisting of stomatherapy nurses was used. The focus group discussion was recorded, and field notes were taken. The focus group meeting was fully transcribed, and a qualitative analysis was performed. The research question was: Which content and features for ostomy self-care promotion should be integrated into an eHealth platform as a digital app or website?

RESULTS

An eHealth platform, which can be a smartphone app or website, for people with ostomy should provide content aimed at promoting self-care, namely in the field of knowledge and self-monitoring, as well as the possibility of interacting with a stomatherapy care nurse.

CONCLUSIONS

The stomatherapy nurse has a decisive role in promoting adaptation to life with a stoma, namely through the promotion of stoma self-care. Technological evolution has emerged as a useful tool to enhance nursing interventions and promote self-care competence. The development of an eHealth platform aimed at promoting ostomy self-care should include the capabilities for telehealth and help with decision-making regarding self-monitoring and seeking differentiated care.

Fact or Fiction? The Development and Evaluation of a Tobacco Virtual Health Tool

The virtual setting is an important setting for health promotion as individuals increasingly go online for health information and support. Yet, users can have difficulty finding valid, trustworthy, and user-friendly health information online. In 2022, we launched an interactive Fact or Fiction Tobacco virtual health tool. The virtual health tool uses evidence-informed tailored content to engage users and refer them to local tobacco cessation resources. The present paper describes the development, user testing, and evaluation of this tool. The Fact or Fiction virtual health tool was designed by tobacco cessation and health marketing experts and informed by health behaviour theories of change. The tool captures data on who is seeking health information, the user's stage of readiness to quit tobacco products, and whether they act by accessing referred resources. In 2021, we conducted two phases of user testing prior to marketing the tool publicly. After 7 weeks of marketing, we collected data on user interactions with the tool and evaluated the reach of the tool. Results from user testing found the tool to be engaging, easy to use, and quick to complete. Adaptations were made to simplify and condense text and include additional animations. During the first seven weeks of the tool being live, it reached 2306 users, and 38.7% of those users were current or occasional tobacco users. Users were classified based on their intention to quit. Bivariate analysis found that the tool was successful in driving tobacco users towards action as 21.2% tobacco users who were looking to quit and 8.8% of tobacco users who were not looking to quit clicked on local tobacco cessation resources. This virtual health tool is reaching the targeted population and providing tailored information needed at each stage of the continuum of health behaviour change. Among tobacco users looking to quit, this virtual health tool acts as a quick referral to local tobacco cessation resources.

Development, dissemination and evaluation of a smartphone-based app for pressure ulcer/injury prevention and treatment for use at the bedside

OBJECTIVE

After launching the 2019 International Pressure Ulcer/Injury Guideline, the National Pressure Injury Advisory Panel (NPIAP), the European Pressure Ulcer Advisory Panel (EPUAP) and the Pan Pacific Pressure Injury Alliance (PPPIA) collaborated with Sensorydata Corp., US, to develop a guideline app (InterPIP App). The app was designed to: present evidence-based guideline recommendations; incorporate search capacities and functionality to facilitate easy access to clinical guidance; provide accessibility in multiple languages; and to be available worldwide at a reasonable price, including opportunities for free access in low-resource countries. This paper describes the development, dissemination and formative evaluation of a mobile app providing evidence-based recommendations for pressure injury prevention, assessment/classification, and treatment at the point of care.

METHOD

An evaluation tool was designed based on a framework developed by Nouri et al. and made available to all app subscribers.

RESULTS

The InterPIP App is currently available in 11 languages and had been downloaded 3616 times by February 2022 in 78 countries. A total of 62 individuals responded to the survey of end-users. In this formal evaluation of user experiences, the app was rated positively on criteria of: information/content; usability; design; functionality; ethics; and security/privacy (median=4 on a 1-5 Likert scale). Overall perceived value was ranked lower with a median of three. Users provided suggestions for ongoing app enhancement.

CONCLUSION

The InterPIP App offers a unique opportunity to bring evidence-based guidance to the point of care. Formal evaluation of end-user experiences identified opportunities for quality improvement, and informed plans for future development and evaluation.

A scoping review of applications of the Consolidated Framework for Implementation Research (CFIR) to telehealth service implementation initiatives

BACKGROUND

The Consolidated Framework for Implementation Research (CFIR), introduced in 2009, has the potential to provide a comprehensive understanding of the determinants of implementation-effectiveness of health service innovations. Although the CFIR has been increasingly used in recent years to examine factors influencing telehealth implementation, no comprehensive reviews currently exist on the scope of knowledge gained exclusively from applications of the CFIR to telehealth implementation initiatives. This review sought to address this gap.

METHODS

PRISMA-ScR criteria were used to inform a scoping review of the literature. Five academic databases (PUBMED, PROQUEST, SCIDIRECT, CINAHL, and WoS) were searched for eligible sources of evidence from 01.01.2010 through 12.31.2021. The initial search yielded a total of 18,388 records, of which, 64 peer-reviewed articles met the inclusion criteria for the review. Included articles were reviewed in full to extract data, and data collected were synthesized to address the review questions.

RESULTS

Most included articles were published during or after 2020 (64%), and a majority (77%) were qualitative or mixed-method studies seeking to understand barriers or facilitators to telehealth implementation using the CFIR. There were few comparative- or implementation-effectiveness studies containing outcome measures (5%). The database search however, revealed a growing number of protocols for implementation-effectiveness studies published since 2020. Most articles (91%) reported the CFIR Inner Setting domain (e.g., leadership engagement) to have a predominant influence over telehealth implementation success. By comparison, few articles (14%) reported the CFIR Outer Setting domain (e.g., telehealth policies) to have notable influence. While more (63%) telehealth initiatives were focused on specialty (vs primary) care, a vast majority (78%) were focused on clinical practice over medical education, healthcare administration, or population health.

CONCLUSIONS

Organized provider groups have historically paid considerable attention to advocating for telehealth policy (Outer Setting) reform. However, results suggest that for effective telehealth implementation, provider groups need to refocus their efforts on educating individual providers on the complex inter-relationships between Inner Setting constructs and telehealth implementation-effectiveness. On a separate note, the growth in implementation-effectiveness study protocols since 2020, suggests that additional outcome measures may soon be available, to provide a more nuanced understanding of the determinants of effective telehealth implementation based on the CFIR domains and constructs.

Beyond monitoring functionality to results evaluation of eHealth interventions: Development and validation of an eHealth evaluation framework

Evidencing eHealth interventions, benefits generates data as a basis for assessing whether observed changes in behavior, processes or healthcare outcomes can be attributed to the eHealth interventions. Generating such evidence requires the use of frameworks or some other type of organizing schemes to help in guiding the process and making sense of eHealth systems and the findings. The frameworks available in literature do not clearly guide on how to monitor eHealth implementation and evaluate eHealth implementation results. This study aimed to develop and validate an eHealth evaluation framework to guide the process of monitoring eHealth implementations and evaluation of eHealth results in terms of outcomes and impact on healthcare in developing countries. The Design Science Research Methodology was followed to conduct this study. Recommendations from an eHealth evaluation exploratory study in Uganda and other eHealth evaluation literature formed key inputs into the design and development of the framework. The framework consists of a generic reference model with eHealth monitoring and evaluation dimensions, performance indicators, and guidelines on how to conduct eHealth monitoring and evaluation. The eHealth evaluation framework received high acceptance (>80%) as regards its fitness for purpose during its validation.

Standardization of the assessment process within telerehabilitation in chronic diseases: a scoping meta-review

Background

Telerehabilitation (TR) interventions are receiving increasing attention. They have been evaluated in various scientific areas through systematic reviews. However, there is a lack of data on how to standardize assessment and report on their domains to guide researchers across studies and bring together the best evidence to assess TR for chronic diseases.

Aims and objectives

The aim of this study was to identify domains of assessment in TR and to qualitatively and quantitatively analyze how and when they are examined to gain an overview of assessment in chronic disease.

Methods

A scoping meta-review was carried out on 9 databases and gray literature from 2009 to 2019. The keyword search strategy was based on "telerehabilitation", “evaluation", “chronic disease" and their synonyms. All articles were subjected to qualitative analysis using the Health Technology Assessment (HTA) Core Model prior to further analysis and narrative synthesis.

Results

Among the 7412 identified articles, 80 studies met the inclusion criteria and addressed at least one of the noncommunicable diseases (NCD) categories of cardiovascular disease (cardiovascular accidents), cancer, chronic respiratory disease, diabetes, and obesity. Regarding the domains of assessment, the most frequently occurring were “social aspect” (n = 63, 79%) (e.g., effects on behavioral changes) and “clinical efficacy” (n = 53, 66%), and the least frequently occurring was “safety aspects” (n = 2, 3%). We also identified the phases of TR in which the assessment was conducted and found that it most commonly occurred in the pilot study and randomized trial phases and least commonly occurred in the design, pretest, and post-implementation phases.

Conclusions

Through the HTA model, this scoping meta-review highlighted 10 assessment domains which have not been studied with the same degree of interest in the recent literature. We showed that each of these assessment domains could appear at different phases of TR development and proposed a new cross-disciplinary and comprehensive method for assessing TR interventions. Future studies will benefit from approaches that leverage the best evidence regarding the assessment of TR, and it will be interesting to extend this assessment framework to other chronic diseases.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-022-08370-y.

Successes of and Lessons From the First Joint eHealth Program of the Dutch University Hospitals: Evaluation Study

BACKGROUND

A total of 8 Dutch university hospitals are at the forefront of contributing meaningfully to a future-proof health care system. To stimulate nationwide collaboration and knowledge-sharing on the topic of evidence-based eHealth, the Dutch university hospitals joined forces from 2016 to 2019 with the first Citrien Fund (CF) program eHealth; 29 eHealth projects with various subjects and themes were selected, supported, and evaluated. To determine the accomplishment of the 10 deliverables for the CF program eHealth and to contribute to the theory and practice of formative evaluation of eHealth in general, a comprehensive evaluation was deemed essential.

OBJECTIVE

The first aim of this study is to evaluate whether the 10 deliverables of the CF program eHealth were accomplished. The second aim is to evaluate the progress of the 29 eHealth projects to determine the barriers to and facilitators of the development of the CF program eHealth projects.

METHODS

To achieve the first aim of this study, an evaluation study was carried out using an adapted version of the Commonwealth Scientific and Industrial Research Organization framework. A mixed methods study, consisting of a 2-part questionnaire and semistructured interviews, was conducted to analyze the second aim of the study.

RESULTS

The 10 deliverables of the CF program eHealth were successfully achieved. The program yielded 22 tangible eHealth solutions, and significant knowledge on the development and use of eHealth solutions. We have learned that the patient is enthusiastic about accessing and downloading their own medical data but the physicians are more cautious. It was not always possible to implement the Dutch set of standards for interoperability, owing to a lack of information technology (IT) capacities. In addition, more attention needed to be paid to patients with low eHealth skills, and education in such cases is important. The eHealth projects' progress aspects such as planning, IT services, and legal played an important role in the success of the 29 projects. The in-depth interviews illustrated that a novel eHealth solution should fulfill a need, that partners already having the knowledge and means to accelerate development should be involved, that clear communication with IT developers and other stakeholders is crucial, and that having a dedicated project leader with sufficient time is of utmost importance for the success of a project.

CONCLUSIONS

The 8 Dutch university hospitals were able to collaborate successfully and stimulate through a bottom-up approach, nationwide eHealth development and knowledge-sharing. In total, 22 tangible eHealth solutions were developed, and significant eHealth knowledge about their development and use was shared. The eHealth projects' progress aspects such as planning, IT services, and legal played an important role in the successful progress of the projects and should therefore be closely monitored when developing novel eHealth solutions.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.1016/j.ceh.2020.12.002.

Orthodontic apps: an assessment of quality (using the Mobile App Rating Scale (MARS)) and behaviour change techniques (BCTs)

BACKGROUND

Apps have been shown to be an effective tool in changing patients' behaviours in orthodontics and can be used to improve their compliance with treatment. The Behaviour Change Techniques (BCTs) and quality (using MARS) within these apps have previously not been published.

OBJECTIVES

1. To evaluate the quality of these apps aiming to change behaviour. 2. To assess BCTs used in patient focused orthodontic apps.

METHODS

The UK Google Play and Apple App Stores were searched to identify all orthodontic apps and 305 apps were identified. All 305 apps were assessed for the presence of BCTs using an accepted taxonomy of BCTs (Behaviour Change Wheel (BCW)), widely utilised in healthcare. Of those containing BCTs, the quality was assessed using the Mobile App Rating Scale (MARS), a validated and multi-dimensional tool which rates apps according to 19 objective criteria. Data collection was carried out by two calibrated, independent assessors and repeated after 6 weeks for 25% of the apps by both assessors.

RESULTS

BCTs were found in 31 apps, although only 18 of them were analysed for quality and 13 apps were excluded. Six different BCTs were identified: these were most commonly 'prompts/cues', and 'information about health consequences'. All apps were shown to be of moderate quality (range 3.1-3.7/5). Inter-rater and intra-rater reliability for BCT and quality assessment were excellent.

CONCLUSIONS

The current availability of orthodontic apps of sufficient quality to recommend to patients is very limited. There is therefore a need for high-quality orthodontic apps with appropriate BCTs to be created, which may be utilised to improve patients' compliance with treatment.

Use of Patient-reported Outcome Measures and Patient-reported Experience Measures within Evaluation Studies of Telemedicine Applications: Systematic Review

Background

With the rise of digital health technologies and telemedicine, the need for evidence-based evaluation is growing. Patient-reported outcome measures (PROMs) and patient-reported experience measures (PREMs) are recommended as an essential part of the evaluation of telemedicine. For the first time, a systematic review has been conducted to investigate the use of PROMs and PREMs in the evaluation studies of telemedicine covering all application types and medical purposes.

Objective

This study investigates the following research questions: in which scenarios are PROMs and PREMs collected for evaluation purposes, which PROM and PREM outcome domains have been covered and how often, which outcome measurement instruments have been used and how often, does the selection and quantity of PROMs and PREMs differ between study types and application types, and has the use of PROMs and PREMs changed over time.

Methods

We conducted a systematic literature search of the MEDLINE and Embase databases and included studies published from inception until April 2, 2020. We included studies evaluating telemedicine with patients as the main users; these studies reported PROMs and PREMs within randomized controlled trials, controlled trials, noncontrolled trials, and feasibility trials in English and German.

Results

Of the identified 2671 studies, 303 (11.34%) were included; of the 303 studies, 67 (22.1%) were feasibility studies, 70 (23.1%) were noncontrolled trials, 20 (6.6%) were controlled trials, and 146 (48.2%) were randomized controlled trials. Health-related quality of life (n=310; mean 1.02, SD 1.05), emotional function (n=244; mean 0.81, SD 1.18), and adherence (n=103; mean 0.34, SD 0.53) were the most frequently assessed outcome domains. Self-developed PROMs were used in 21.4% (65/303) of the studies, and self-developed PREMs were used in 22.3% (68/303). PROMs (n=884) were assessed more frequently than PREMs (n=234). As the evidence level of the studies increased, the number of PROMs also increased (τ=−0.45), and the number of PREMs decreased (τ=0.35). Since 2000, not only has the number of studies using PROMs and PREMs increased, but the level of evidence and the number of outcome measurement instruments used have also increased, with the number of PREMs permanently remaining at a lower level.

Conclusions

There have been increasingly more studies, particularly high-evidence studies, which use PROMs and PREMs to evaluate telemedicine. PROMs have been used more frequently than PREMs. With the increasing maturity stage of telemedicine applications and higher evidence level, the use of PROMs increased in line with the recommendations of evaluation guidelines. Health-related quality of life and emotional function were measured in almost all the studies. Simultaneously, health literacy as a precondition for using the application adequately, alongside proper training and guidance, has rarely been reported. Further efforts should be pursued to standardize PROM and PREM collection in evaluation studies of telemedicine.

Evaluation frameworks for digital nursing technologies: analysis, assessment, and guidance. An overview of the literature

Background

The evaluation of digital nursing technologies (DNT) plays a major role in gaining knowledge about certain aspects of a technology such as acceptance, effectiveness, or efficiency. Evaluation frameworks can help to classify the success or failure of a DNT or to further develop the technology. In general, there are many different evaluation frameworks in the literature that provide overviews of a wide variety of aspects, which makes this a highly diverse field and raises the question how to select a suitable framework. The aim of this article is to provide orientation in the field of comprehensive evaluation frameworks that can be applied to the field of DNT and to conduct a detailed analysis and assessment of these frameworks to guide field researchers.

Methods

This overview was conducted using a three-component search process to identify relevant frameworks. These components were (1) a systematized literature search in PubMed; (2) a narrative review and (3) expert consultations. Data relating to the frameworks’ evaluation areas, purpose, perspectives, and success definitions were extracted. Quality criteria were developed in an expert workshop and a strength and weakness assessment was carried out.

Results

Eighteen relevant comprehensive evaluation frameworks for DNT were identified. Nine overarching evaluation areas, seven categories of purposes, five evaluation perspectives and three categories of success definitions could be identified. Eleven quality criteria for the strengths and weaknesses of DNT-related evaluation frameworks were developed and the included frameworks were assessed against them.

Conclusion

Evaluators can use the concise information and quality criteria of this article as a starting point to select and apply appropriate DNT evaluation frameworks for their research projects or to assess the quality of an evaluation framework for DNT, as well as a basis for exploring the questions raised in this article. Future research could address gaps and weaknesses in existing evaluation frameworks, which could improve the quality of future DNT evaluations.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12912-021-00654-8.

Health technology assessment for digital technologies that manage chronic disease: a systematic review

OBJECTIVE

A growing number of evaluation frameworks have emerged over recent years addressing the unique benefits and risk profiles of new classes of digital health technologies (DHTs). This systematic review aims to identify relevant frameworks and synthesize their recommendations into DHT-specific content to be considered when performing Health Technology Assessments (HTAs) for DHTs that manage chronic noncommunicable disease at home.

METHODS

Searches were undertaken of Medline, Embase, Econlit, CINAHL, and The Cochrane Library (January 2015 to March 2020), and relevant gray literature (January 2015 to August 2020) using keywords related to HTA, evaluation frameworks, and DHTs. Included framework reference lists were searched from 2010 until 2015. The EUNetHTA HTA Core Model version 3.0 was selected as a scaffold for content evaluation.

RESULTS

Forty-four frameworks were identified, mainly covering clinical effectiveness (n = 30) and safety (n = 23) issues. DHT-specific content recommended by framework authors fell within 28 of the 145 HTA Core Model issues. A further twenty-two DHT-specific issues not currently in the HTA Core Model were recommended.

CONCLUSIONS

Current HTA frameworks are unlikely to be sufficient for assessing DHTs. The development of DHT-specific content for HTA frameworks is hampered by DHTs having varied benefit and risk profiles. By focusing on DHTs that actively monitor/treat chronic noncommunicable diseases at home, we have extended DHT-specific content to all nine HTA Core Model domains. We plan to develop a supplementary evaluation framework for designing research studies, undertaking HTAs, and appraising the completeness of HTAs for DHTs.

A Narrative Review of Factors Historically Influencing Telehealth Use across Six Medical Specialties in the United States

Prior to the COVID-19 pandemic, studies in the US have identified wide variations in telehealth use across medical specialties. This is an intriguing problem, because the US has historically lacked a standardized set of telehealth coverage and reimbursement policies, which has posed a barrier to telehealth use across all specialties. Although all medical specialties in the US have been affected by these macro (policy-level) barriers, some specialties have been able to integrate telehealth use into mainstream practice, while others are just gaining momentum with telehealth during COVID-19. Although the temporary removal of policy (coverage) restrictions during the pandemic has accelerated telehealth use, uncertainties remain regarding future telehealth sustainability. Since macro (policy-level) factors by themselves do not serve to explain the variation in telehealth use across specialties, it would be important to examine meso (organizational-level) and micro (individual-level) factors historically influencing telehealth use across specialties, to understand underlying reasons for variation and identify implications for widespread sustainability. This paper draws upon the existing literature to develop a conceptual framework on macro-meso-micro factors influencing telehealth use within a medical specialty. The framework is then used to guide a narrative review of the telehealth literature across six medical specialties, including three specialties with lower telehealth use (allergy-immunology, family medicine, gastroenterology) and three with higher telehealth use (psychiatry, cardiology, radiology) in the US, in order to synthesize themes and gain insights into barriers and facilitators to telehealth use. In doing so, this review addresses a gap in the literature and provides a foundation for future research. Importantly, it helps to identify implications for ensuring widespread sustainability of telehealth use in the post-pandemic future.

Planning a holistic summative eHealth evaluation in an interdisciplinary and multi-national setting: a case study and propositions for guideline development

Background

Summative eHealth evaluations frequently lack quality, which affects the generalizability of the evidence, and its use in practice and further research. To guarantee quality, a number of activities are recommended in the guidelines for evaluation planning. This study aimed to examine a case of an eHealth evaluation planning in a multi-national and interdisciplinary setting and to provide recommendations for eHealth evaluation planning guidelines.

Methods

An empirical eHealth evaluation process was developed through a case study. The empirical process was compared with selected guidelines for eHealth evaluation planning using a pattern-matching technique.

Results

Planning in the interdisciplinary and multi-national team demanded extensive negotiation and alignment to support the future use of the evidence created. The evaluation planning guidelines did not provide specific strategies for different set-ups of the evaluation teams. Further, they did not address important aspects of quality evaluation, such as feasibility analysis of the outcome measures and data collection, monitoring of data quality, and consideration of the methods and measures employed in similar evaluations.

Conclusions

Activities to prevent quality problems need to be incorporated in the guidelines for evaluation planning. Additionally, evaluators could benefit from guidance in evaluation planning related to the different set-ups of the evaluation teams.

Going digital: a narrative overview of the effects, quality and utility of mobile apps in chronic disease self-management

Objective Smartphone health applications (apps) are being increasingly used to assist patients in chronic disease self-management. The effects of such apps on patient outcomes are uncertain, as are design features that maximise usability and efficacy, and the best methods for evaluating app quality and utility. Methods In assessing efficacy, PubMed, Cochrane Library and EMBASE were searched for systematic reviews (and single studies if no systematic review was available) published between January 2007 and January 2018 using search terms (and synonyms) of 'smartphone' and 'mobile applications', and terms for each of 11 chronic diseases: asthma, chronic obstructive lung disease (COPD), diabetes, chronic pain, serious mental health disorders, alcohol and substance addiction, heart failure, ischaemic heart disease, cancer, cognitive impairment, chronic kidney disease (CKD). With regard to design features and evaluation methods, additional reviews were sought using search terms 'design', 'quality,' 'usability', 'functionality,' 'adherence', 'evaluation' and related synonyms. Results Of 13 reviews and six single studies assessing efficacy, consistent evidence of benefit was seen only with apps for diabetes, as measured by decreased glycosylated haemoglobin levels (HbA1c). Some, but not all, studies showed benefit in asthma, low back pain, alcohol addiction, heart failure, ischaemic heart disease and cancer. There was no evidence of benefit in COPD, cognitive impairment or CKD. In all studies, benefits were clinically marginal and none related to morbid events or hospitalisation. Twelve design features were identified as enhancing usability. An evaluation framework comprising 32 items was formulated. Conclusion Evidence of clinical benefit of most available apps is very limited. Design features that enhance usability and maximise efficacy were identified. A provisional 'first-pass' evaluation framework is proposed that can help decide which apps should be endorsed by government agencies following more detailed technical assessments and which could then be recommended with confidence by clinicians to their patients. What is known about the topic? Smartphone health apps have attracted considerable interest from patients and health managers as a means of promoting more effective self-management of chronic diseases, which leads to better health outcomes. However, most commercially available apps have never been evaluated for benefits or harms in clinical trials, and there are currently no agreed quality criteria, standards or regulations to ensure health apps are user-friendly, accurate in content, evidence based or efficacious. What does this paper add? This paper presents a comprehensive review of evidence relating to the efficacy, usability and evaluation of apps for 11 common diseases aimed at assisting patients in self-management. Consistent evidence of benefit was only seen for diabetes apps; there was absent or conflicting evidence of benefit for apps for the remaining 10 diseases. Benefits that were detected were of marginal clinical importance, with no reporting of hard clinical end-points, such as mortality or hospitalisations. Only a minority of studies explicitly reported using behaviour change theories to underpin the app intervention. Many apps lacked design features that the literature identified as enhancing usability and potential to confer benefit. Despite a plethora of published evaluation tools, there is no universal framework that covers all relevant clinical and technical attributes. An inclusive list of evaluation criteria is proposed that may overcome this shortcoming. What are the implications for practitioners? The number of smartphone apps will continue to grow, as will the appetite for patients and clinicians to use them in chronic disease self-management. However, the evidence to date of clinical benefit of most apps already available is very limited. Design features that enhance usability and clinical efficacy need to be considered. In making decisions about which apps should be endorsed by government agencies and recommended with confidence by clinicians to their patients, a comprehensive but workable evaluation framework needs to be used by bodies assuming the roles of setting and applying standards.

Three decades of telemedicine in Brazil: Mapping the regulatory framework from 1990 to 2018

This study characterized the evolution of Brazilian public telemedicine policy in the Brazilian Unified Health System for 30 years from 1988 to 2019 by analyzing its legal framework. We identified 79 telemedicine-related legislations from the federal government (laws, decrees, and ordinances) and 31 regulations of federal councils of health professionals. Three historical phases were established according to the public policy cycle, and material was classified according to the purpose of the normative documents. The content analysis was based on the advocacy coalition framework model. Of the federal legislations, 8.9% were for the Formulation/Decision-Making phase, 43% for the Organization/Implementation phase, and 48.1% for the Expansion/Maturation phase of telemedicine policy in Brazil. The Federal Council of Medicine was the most active in standardizing telemedicine and was responsible for 21 (67.7%) regulations. The first legislations were passed in 2000; however, the coalitions discussed topics related to telemedicine and created their belief systems from the 1990's. The time cycle which included formulation and decision making for Brazilian telemedicine policy, extended until 2007 with the creation of several technical working groups. The expansion and maturation of telemedicine services began in 2011 with the decentralization of telemedicine policy actions across the country. Telemedicine centers which performed telediagnosis influenced the computerization of primary health care units. We conclude that Brazilian telemedicine field has greatly grown and changed in recent years. However, despite the proliferation of legislations and regulations in the period studied, there is still no fully consolidated process for setting up a wholly defined regulatory framework for telemedicine in Brazil.

Telemonitoring for Patients With COVID-19: Recommendations for Design and Implementation

Despite significant efforts, the COVID-19 pandemic has put enormous pressure on health care systems around the world, threatening the quality of patient care. Telemonitoring offers the opportunity to carefully monitor patients with a confirmed or suspected case of COVID-19 from home and allows for the timely identification of worsening symptoms. Additionally, it may decrease the number of hospital visits and admissions, thereby reducing the use of scarce resources, optimizing health care capacity, and minimizing the risk of viral transmission. In this paper, we present a COVID-19 telemonitoring care pathway developed at a tertiary care hospital in the Netherlands, which combined the monitoring of vital parameters with video consultations for adequate clinical assessment. Additionally, we report a series of medical, scientific, organizational, and ethical recommendations that may be used as a guide for the design and implementation of telemonitoring pathways for COVID-19 and other diseases worldwide.

Health technology assessment frameworks for eHealth: A systematic review

Objectives

Traditionally, health technology assessment (HTA) focuses on assessing the impact of pharmaceutical technologies on health and care. Resources are scarce and policy makers aim to achieve effective, accessible health care. eHealth innovations are increasingly more integrated in all healthcare domains. However, how eHealth is assessed prior to its implementation in care practices is unclear. To support evidence-informed policy making, this study aimed to identify frameworks and methods for assessing eHealth's impact on health care.

Methods

The scientific literature in five bibliographical databases was systematically reviewed. Articles were included if the study was conducted in a clinical setting, used an HTA framework and assessed an eHealth service. A systematic qualitative narrative approach was applied for analysis and reporting.

Results

Twenty-one HTA frameworks were identified in twenty-three articles. All frameworks addressed outcomes related to the technical performance and functionalities of eHealth service under assessment. The majority also addressed costs (n = 19), clinical outcomes (n = 14), organizational (n = 15) and system level aspects (n = 13). Most frameworks can be classified as dimensional (n = 13), followed by staged (n = 3), hybrid (n = 3), and business modeling frameworks (n = 2). Six frameworks specified assessment outcomes and methods.

Conclusions

HTA frameworks are available for a-priori impact assessment of eHealth services. The frameworks vary in assessment outcomes, methods, and specificity. Demonstrated applicability in practice is limited. Recommendations include standardization of: (i) reporting characteristics of eHealth services, and (ii) specifying assessment outcomes and methods following a stepped-approach tailored to the functional characteristics of eHealth services. Standardization might improve the quality and comparability of eHTA assessments.

The role of mobile health technologies in allergy care: An EAACI position paper

Mobile health (mHealth) uses mobile communication devices such as smartphones and tablet computers to support and improve health-related services, data and information flow, patient self-management, surveillance, and disease management from the moment of first diagnosis to an optimized treatment. The European Academy of Allergy and Clinical Immunology created a task force to assess the state of the art and future potential of mHealth in allergology. The task force endorsed the "Be He@lthy, Be Mobile" WHO initiative and debated the quality, usability, efficiency, advantages, limitations, and risks of mobile solutions for allergic diseases. The results are summarized in this position paper, analyzing also the regulatory background with regard to the "General Data Protection Regulation" and Medical Directives of the European Community. The task force assessed the design, user engagement, content, potential of inducing behavioral change, credibility/accountability, and privacy policies of mHealth products. The perspectives of healthcare professionals and allergic patients are discussed, underlining the need of thorough investigation for an effective design of mHealth technologies as auxiliary tools to improve quality of care. Within the context of precision medicine, these could facilitate the change in perspective from clinician- to patient-centered care. The current and future potential of mHealth is then examined for specific areas of allergology, including allergic rhinitis, aerobiology, allergen immunotherapy, asthma, dermatological diseases, food allergies, anaphylaxis, insect venom, and drug allergy. The impact of mobile technologies and associated big data sets are outlined. Facts and recommendations for future mHealth initiatives within EAACI are listed.

Evaluation of user satisfaction and usability of a mobile app for smoking cessation

BACKGROUND

Mobile apps have a great potential to support patients in healthcare, and to encourage healthy behavioral changes such as smoking cessation. Nevertheless, the user rejection levels are still high. A set of factors that has impact on the app effectiveness is related to the quality of those features that lead to positive user experiences when using the app. This work aims to evaluate the user experience, and more specifically the usability and the user satisfaction with a mobile application for smoking cessation. This will also provide a basis for future improvements.

METHODS

We provided a smoking cessation mobile Android app to two different user cohorts, the smokers as valid users and the experts, for three weeks. The app featured usual functionalities to help quit smoking, including an achieved benefits section, mini-games to distract during cravings, and supportive motivational messages. We collected information about user experience, through game playability and message satisfaction questionnaires, and the experts' opinions. We also considered usage of app sections, the duration of the mini-game sessions, and the user ratings for motivational messages.

RESULTS

We included 45 valid users and 25 experts in this study. The questionnaire indicated 80% satisfaction rate for the motivational messages. According to game questionnaires, over 69% of the participants agreed that the games have good usability features, however, for questions related to mobility and gameplay heuristics, agreements were below 67%. The most accessed app sections were achieved benefits and the one with motivational messages. The experts described issues that could help to improve the application.

CONCLUSIONS

The combination of questionnaires with expert reports allowed to identify several problems and possible corrections. Our study showed that motivational messages have a good satisfaction rate, although it is necessary to consider technical features of some mobile devices that may hinder message reception. Games have good usability and it's expected that the addition of difficulty levels and a better accessibility to the game menu could make them more attractive and increase its usage. Future development of mHealth apps based on gamification and motivational messages need to consider these factors for better user satisfaction and usability.

Standards as applied in reality: a case study on the translation of standards in eHealth evaluation practice

BACKGROUND

Application of standards is a way to increase quality in an evaluation study. However, standards are used insufficiently in eHealth evaluation, affecting the generalization of the knowledge generated. This study aimed to explore how standards are used in a practical setting of an eHealth evaluation, and to identify the factors that can hinder their use.

METHODS

The data were collected in a multi-national and interdisciplinary eHealth evaluation study targeted at the elderly people suffering from mild cognitive impairment and mild dementia. The study was carried out in four countries and funded by the European Union. The collected data included meeting minutes (n = 8) and e-mail correspondence (n = 261) between partners. The chronological sequence of events related to the use of standards was established. Subsequently, the hindering factors related to the use of standards were identified from the sequence.

RESULTS

The use of four standards was described, reflecting a variety of related processes or barriers that emerge during eHealth evaluation. The processes differed based on the type of the standard. Specifically, evaluation frameworks were found to be conceptual standards and they were easy to agree upon, while standardized metrics were more tangible and their use triggered negotiations. Further, the following factors hindered the use of standards in eHealth evaluations: (1) inadequacy of a standard to address a target population or a disease, (2) insufficient resources to use a standard, (3) lack of experience in using a standard, and (4) lack of validation of a standard in a particular location.

CONCLUSIONS

Standardization initiatives in eHealth evaluation provide a blueprint for evaluation, but their practical application is problematic. The practical circumstances of an evaluation study can cause deviations in the standards, thus producing heterogeneity in the evaluation methodologies.

A design and evaluation framework for digital health interventions

Digital health interventions (DHIs) have the potential to help the growing number of chronic disease patients better manage their everyday lives. However, guidelines for the systematic development of DHIs are still scarce. The current work has, therefore, the objective to propose a framework for the design and evaluation of DHIs (DEDHI). The DEDHI framework is meant to support both researchers and practitioners alike from early conceptual DHI models to large-scale implementations of DHIs in the healthcare market.

Depression care services and telehealth technology use for homebound elderly in the United States

Objectives: Despite the increasing evidence for the effectiveness of telehealth technology in screening and treating depression in older adults, they have been slowly adopted by Home Health Care (HHC) agencies. Therefore, this study was conducted to determine how HHC agencies perceive and use telehealth technology for depression care among homebound older adult patients. Methods: Five-hundred-and-sixteen staff from the National Association for Homecare & Hospice (NAHC) member home health care agencies completed the online survey. Questions were asked of HHC staff regarding performance expectancy, effort expectancy, social influences, facilitating conditions, telehealth use and intention to use/continue to use telehealth. Results: The majority had a neutral or positive perception towards telehealth. However, participants from agencies that have yet to use telehealth (mean: 3.25, SD: 1.56) reported a less intention to use the technology for depression care versus those from agencies that did (mean: 4.64, SD: 1.37). This may be partially explained by the finding that only 32% perceived themselves as having the knowledge and 25% as having resources to use telehealth. Additionally, facilitating conditions and social influences were significant predictors of intention to use/continue to use telehealth for depression care (p-values < .01). Conclusion: Overall, staff had a neutral or positive perception towards telehealth. Factors such as fewer years of experience in using telehealth and a small annual budget may explain a negative perception towards telehealth. Therefore, further education and resources are needed to support telehealth use. Future studies may consider comparing telehealth programs and identifying supporting policies.

Some Multidimensional Unintended Consequences of Telehealth Utilization: A Multi-Project Evaluation Synthesis

Background: Telehealth initiatives have bloomed around the globe, but their integration and diffusion remain challenging because of the complex issues they raise. Available evidence around telehealth usually deals with its expected effects and benefits, but its unintended consequences (UCs) and influencing factors are little documented. This study aims to explore, describe and analyze multidimensional UCs that have been associated with the use of telehealth.

Methods: We performed a secondary analysis of the evaluations of 10 telehealth projects conducted over a 22-year period in the province of Quebec (Canada). All material was subjected to a qualitative thematic-pragmatic content analysis with triangulation of methodologies and data sources. We used the conceptual model of the UCs of health information technologies proposed by Bloomrosen et al to structure our analysis.

Results: Four major findings emerged from our analysis. First, telehealth utilization requires many adjustments, changes and negotiations often underestimated in the planning and initial phases of the projects. Second, telehealth may result in the emergence of new services corridors that disturb existing ones and involve several adjustments for organizations, such as additional investments and resources, but also the risk of fragmentation of services and the need to balance between standardization of practices and local innovation. Third, telehealth may accentuate power relations between stakeholders. Fourth, it may lead to significant changes in the responsibilities of each actor in the supply chain of services. Finally, current legislative and regulatory frameworks appear ill-adapted to many of the new realities brought by telehealth.

Conclusion: This study provides a first attempt for an overview of the UCs associated with the use of telehealth. Future research-evaluation studies should be more sensitive to the multidimensional and interdependent factors that influence telehealth implementation and utilization as well as its impacts, intended or unintended, at all levels. Thus, a consideration of potential UCs should inform telehealth projects, from their planning until their scaling-up.

Optimising patient active role with a user-centred eHealth platform (CONCERTO+) in chronic diseases management: a study protocol for a pilot cluster randomised controlled trial

INTRODUCTION

Multimorbidity increases care needs and primary care use among people with chronic diseases. The Concerto Health Program (CHP) has been developed to optimise chronic disease management in primary care services. However, in its current version, the CHP primarily targets clinicians and does not aim to answer directly patients' and their informal caregivers' needs for chronic disease management. Various studies have shown that interventions that increase patient activation level are associated with better health outcomes. Furthermore, educational tools must be adapted to patients and caregivers in terms of health literacy and usability. This project aims to develop, implement and evaluate a user-centred, multifunctional and personalised eHealth platform (CONCERTO+) to promote a more active patient role in chronic disease management and decision-making.

METHODS AND ANALYSIS

This project uses a collaborative research approach, aiming at the personalisation of CHP through three phases: (1) the development of one module of an eHealth platform based on scientific evidence and user-centred design; (2) a feasibility study of CONCERTO+ through a pilot cluster randomised controlled trial where patients with chronic diseases from a primary healthcare practice will receive CONCERTO+ during 6 months and be compared to patients from a control practice receiving usual care and (3) an analysis of CONCERTO+ potential for scaling up. To do so, we will conduct two focus groups with patients and informal caregivers and individual interviews with health professionals at the two study sites, as well as health care managers, information officers and representatives of the Ministry of Health.

ETHICS AND DISSEMINATION

This study received ethical approval from Ethics Committee of Université Laval. The findings will be used to inform the effectiveness of CONCERTO+ to improve management care in chronic diseases. We will disseminate findings through presentations in scientific conferences and publication in peer-reviewed journals.

TRIAL REGISTRATION NUMBER

NCT03628963; Pre-results.

Client-Centered Mobile Health Care Applications: Using the Mobile Application Rating Scale Instrument for Evidence-Based Evaluation

The use of mobile devices and applications (apps) to monitor or assist in health behaviors is rapidly expanding in many areas of society. Clinicians desire evidence-based app recommendations for their clients to increase self-care and wellness management in such areas as mindfulness, weight loss and activity tracking, glycemic control, and consumer medication information. Given the constant influx of new apps into the major app repositories, clinicians need to be able to ensure the quality of information and interaction that occurs within the mobile health (mHealth) marketplace. The Mobile Application Rating Scale (MARS) and the user version of the scale are valid and reliable instruments used to examine the engagement, functionality, aesthetics, and quality of information in mHealth apps. MARS-rated apps can be readily available resources for busy clinicians to make app suggestions to assist clients on a variety of topics that promote improved outcomes. This article reviews the MARS instrument and utilization of the instrument by clinicians and summarizes several primary care and wellness apps that have been evaluated using this tool.

Development of the mHealth App Trustworthiness checklist

BACKGROUND

Mobile health applications (mHealth apps) currently lack a consensus on substantial quality and safety standards. As such, the number of individuals engaging with untrustworthy mHealth apps continues to grow at a steady pace.

OBJECTIVE

The purpose of this study was to investigate end-users' opinions on the features or actions necessary for trustworthy mHealth apps; and to convey this information to app developers via a succinct but informative checklist: the mHealth app trustworthiness checklist.

METHODS

The checklist was formulated in three stages: (a) a literature review of studies identified the desirable features of the most prolific mHealth apps (health and fitness apps); (b) four focus group sessions with past or current users of these apps (n = 20); and (c) expert feedback on whether the checklist items are conceivable in a real-life setting (n = 6).

RESULTS

Five major themes emerged from the focus group discussions: informational content, organizational attributes, societal influence, technology-related features, and user control factors. The mHealth app trustworthiness checklist was developed to incorporate these five themes and subsequently modified following expert consultation. In addition to the trustworthiness themes, we identified features that lie between trust and mistrust (limited digital literacy and indifference) as well as 10 features and actions that cause end-users to mistrust mHealth apps.

CONCLUSION

This study contributes to the evidence base on the attributes of trustworthy mHealth apps. The mHealth app trustworthiness checklist is a useful tool in advancing continued efforts to ensure that health technologies are trustworthy.

Systematic review to identify proxy indicators to quantify the impact of eHealth tools on maternal and neonatal health outcomes in low-income and middle-income countries including Delphi consensus

OBJECTIVE

To identify interventions that could serve as reliable proxy indicators to measure eHealth impact on maternal and neonatal outcomes.

DESIGN

Systematic review and Delphi study.

METHODS

We searched PubMed, Embase and Cochrane from January 1990 to May 2016 for studies and reviews that evaluated interventions aimed at improving maternal/neonatal health and reducing mortality. Interventions that are not low-income and middle-income context appropriate and that cannot currently be diagnosed, managed or impacted by eHealth (eg, via telemedicine distance diagnostic or e-learning) were excluded. We used the Cochrane risk of bias, Risk Of Bias In Non- randomised Studies - of Interventions and ROBIS tool to assess the risk of bias. A three-step modified Delphi method was added to identify additional proxy indicators and prioritise the results, involving a panel of 13 experts from different regions, representing obstetricians and neonatologists.

RESULTS

We included 44 studies and reviews, identifying 40 potential proxy indicators with a positive impact on maternal/neonatal outcomes. The Delphi experts completed and prioritised these, resulting in a list of 77 potential proxy indicators.

CONCLUSIONS

The proxy indicators propose relevant outcome measures to evaluate if eHealth tools directly affect maternal/neonatal outcomes. Some proxy indicators require mapping to the local context, practices and available resources. The local mapping facilitates the utilisation of the proxy indicators in various contexts while allowing the systematic collection of data from different projects and programmes. Based on the mapping, the same proxy indicator can be used for different contexts, allowing it to measure what is locally and temporally relevant, making the proxy indicator sustainable.

PROSPERO REGISTRATION NUMBER

CRD42015027351.

Key Intervention Characteristics in e-Health: Steps Towards Standardized Communication

PURPOSE

This paper reports expert opinion on e-health intervention characteristics that enable effective communication of characteristics across the diverse field of e-health interventions. The paper presents a visualization tool to support communication of the defining characteristics.

METHODS

An initial list of e-health intervention characteristics was developed through an iterative process of item generation and discussion among the 12 authors. The list was distributed to 123 experts in the field, who were emailed an invitation to assess and rank the items. Participants were asked to evaluate these characteristics in three separate ways.

RESULTS

A total of 50 responses were received for a response rate of 40.7%. Six respondents who reported having little or no expertise in e-health research were removed from the dataset. Our results suggest that 10 specific intervention characteristics were consistently supported as of central importance by the panel of 44 e-intervention experts. The weight and perceived relevance of individual items differed between experts; oftentimes, this difference is a result of the individual theoretical perspective and/or behavioral target of interest.

CONCLUSIONS

The first iteration of the visualization of salient characteristics represents an ambitious effort to develop a tool that will support communication of the defining characteristics for e-health interventions aimed to assist e-health developers and researchers to communicate the key characteristics of their interventions in a standardized manner that facilitates dialog.

Evaluation Criteria of Noninvasive Telemonitoring for Patients With Heart Failure: Systematic Review

BACKGROUND

Telemonitoring can improve heart failure (HF) management, but there is no standardized evaluation framework to comprehensively evaluate its impact.

OBJECTIVE

Our objectives were to list the criteria used in published evaluations of noninvasive HF telemonitoring projects, describe how they are used in the evaluation studies, and organize them into a consistent scheme.

METHODS

Articles published from January 1990 to August 2015 were obtained through MEDLINE, Web of Science, and EMBASE. Articles were eligible if they were original reports of a noninvasive HF telemonitoring evaluation study in the English language. Studies of implantable telemonitoring devices were excluded. Each selected article was screened to extract the description of the telemonitoring project and the evaluation process and criteria. A qualitative synthesis was performed.

RESULTS

We identified and reviewed 128 articles leading to 52 evaluation criteria classified into 6 dimensions: clinical, economic, user perspective, educational, organizational, and technical. The clinical and economic impacts were evaluated in more than 70% of studies, whereas the educational, organizational, and technical impacts were studied in fewer than 15%. User perspective was the most frequently covered dimension in the development phase of telemonitoring projects, whereas clinical and economic impacts were the focus of later phases.

CONCLUSIONS

Telemonitoring evaluation frameworks should cover all 6 dimensions appropriately distributed along the telemonitoring project lifecycle. Our next goal is to build such a comprehensive evaluation framework for telemonitoring and test it on an ongoing noninvasive HF telemonitoring project.

Reach, engagement, and effectiveness: a systematic review of evaluation methodologies used in health promotion via social networking sites

Issue addressed Social networking sites (SNS) are increasingly popular platforms for health promotion. Advancements in SNS health promotion require quality evidence; however, interventions are often not formally evaluated. This study aims to describe evaluation practices used in SNS health promotion. Methods A systematic review was undertaken of Medline, PsycINFO, Scopus, EMBASE, CINAHL Plus, Communication and Mass Media Complete, and Cochrane Library databases. Articles published between 2006 and 2013 describing any health promotion intervention delivered using SNS were included. Results Forty-seven studies were included. There were two main evaluation approaches: closed designs (n=23), which used traditional research designs and formal recruitment procedures; and open designs (n=19), which evaluated the intervention in a real-world setting, allowing unknown SNS users to interact with the content without enrolling in research. Closed designs were unable to assess reach and engagement beyond their research sample. Open designs often relied on weaker study designs with no use of objective outcome measures and yielded low response rates. Conclusions Barriers to evaluation included low participation rates, high attrition, unknown representativeness and lack of comparison groups. Acceptability was typically assessed among those engaged with the intervention, with limited population data available to accurately assess intervention reach. Few studies were able to assess uptake of the intervention in a real-life setting while simultaneously assessing effectiveness of interventions with research rigour. So what? Through use of quasi-experimental or well designed before-after evaluations, in combination with detailed engagement metrics, it is possible to balance assessment of effectiveness and reach to evaluate SNS health promotion.

Exploring factors associated with the uneven utilization of telemedicine in Norway: a mixed methods study

BACKGROUND

Norway has a long history of using telemedicine, especially for geographical reasons. Despite the availability of promising telemedicine applications and the implementation of national initiatives and policies, the sustainability and scaling-up of telemedicine in the health system is still far from accomplished. The main objective of this study was to explore and identify the multi-level (micro, meso and macro) factors affecting telemedicine utilization in Norway.

METHODS

We used a mixed methods approach. Data from a national registry were collected to analyze the use of outpatient visits and telemedicine contacts in Norway from 2009 to 2015. Interviews with key stakeholders at national, regional and local level helped complete and contextualize the data analysis and explore the main issues affecting the use of telemedicine by health authorities and hospitals. Relevant national documents were also used to support, contradict, contextualize or clarify information and data.

RESULTS

Telemedicine use in Norway from 2009 to 2015 remained very low, not exceeding 0.5% of total outpatient activity at regional level and 0.1% at national level. All four regions used telemedicine. Of the 29 hospitals, 24 used it at least once over the 7-year period. Telemedicine was not used regularly everywhere, with some hospitals using it sporadically. Telemedicine was mostly used in selected specialties, including rehabilitation, neurosurgery, skin and venereal diseases. Three major themes affecting implementation and utilization of telemedicine in Norway emerged: (i) governance and strategy; (ii) organizational and professional dimensions; (iii) economic and financial dimensions. For each theme, a number of factors and challenges faced at different health care levels were identified.

CONCLUSIONS

This study allowed shedding light on multi-level and interdependent factors affecting utilization of telemedicine in Norway. The identification of the main implementation and utilization challenges might support decision makers and practitioners in the successful scaling-up of telemedicine. This work provides a knowledge base useful to other countries which intend to implement telemedicine or other digital health services into their healthcare systems.

Effective? Engaging? Secure? Applying the ORCHA-24 framework to evaluate apps for chronic insomnia disorder

BACKGROUND

Mobile health offers many opportunities; however, the 'side-effects' of health apps are often unclear. With no guarantee health apps first do no harm, their role as a viable, safe and effective therapeutic option is limited.

OBJECTIVE

To assess the quality of apps for chronic insomnia disorder, available on the Android Google Play Store, and determine whether a novel approach to app assessment could identify high-quality and low-risk health apps in the absence of indicators such as National Health Service (NHS) approval.

METHODS

The Organisation for the Review of Care and Health Applications- 24 Question Assessment (ORCHA-24), 24 app assessment criteria concerning data privacy, clinical efficacy and user experience, answered on a 'yes' or 'no' and evidence-driven basis, was applied to assess 18 insomnia apps identified via the Android Google Play Store, in addition to the NHS-approved iOS app Sleepio.

FINDINGS

63.2% of apps (12/19) provided a privacy policy, with seven (36.8%) stating no user data would be shared without explicit consent. 10.5% (2/19) stated they had been shown to be of benefit to those with insomnia, with cognitive behavioural therapy apps outperforming hypnosis and meditation apps (p=0.046). Both the number of app downloads (p=0.29) and user-review scores (p=0.23) were unrelated to ORCHA-24 scores. The NHS-approved app Sleepio, consistently outperformed non-accredited apps across all domains of the ORCHA-24.

CONCLUSIONS

Apps for chronic insomnia disorder exhibit substantial variation in adherence to published data privacy, user experience and clinical efficacy standards, which are not clearly correlated with app downloads or user-review scores.

CLINICAL IMPLICATIONS

In absence of formal app accreditation, the ORCHA-24 could feasibly be used to highlight the risk-benefit profiles of health apps prior to downloading.

All that Glitters is not Gold: Six Steps Before Selecting and Prioritizing e-Health Services

Since the market for e-health applications is constantly growing, it is getting an ever more complex endeavor to select and prioritize the right service offering given a particular situation. In examining the extant literature, it was revealed that little emphasis is actually placed on how to analyze contextual or environmental factors prior to the selection and prioritization of e-health services. With this paper, we therefore propose a formative framework consisting of six fundamental yet very pragmatic steps that may support decision makers in identifying the most important contextual pre-requisites that e-health services need to fulfill in order to be considered as effective for their environment to be implemented.

A mixed methods systematic review of success factors of mhealth and telehealth for maternal health in Sub-Saharan Africa

Access to health care is still limited for many women in sub-Saharan Africa (SSA), while it remains an important determinant of maternal mortality and morbidity. Information and communication technologies (ICTs), such as mhealth and telehealth, can help to facilitate this access by acting on the various obstacles encountered by women, be they socio-cultural, economic, geographical or organizational. However, various factors contribute to the success of mhealth and telehealth implementation and use, and must be considered for these technologies to go beyond the pilot project stage. The objective of this systematic literature review is to synthesize the empirical knowledge on the success factors of the implementation and use of telehealth and mhealth to facilitate access to maternal care in SSA. The methodology used is based on that of the Cochrane Collaboration, including a documentary search using standardized language in six databases, selection of studies corresponding to the inclusion criteria, data extraction, evaluation of study quality, and synthesis of the results. A total of 93 articles were identified, which allowed the inclusion of seven studies, six of which were on mhealth. Based on the framework proposed by Broens et al., we synthesized success factors into five categories: (I) technology, such as technical support to maintain, troubleshoot and train users, good network coverage, existence of a source of energy and user friendliness; (II) user acceptance, which is facilitated by factors such as unrestricted use of the device, perceived usefulness to the worker, adequate literacy, or previous experience of use ; (III) short- and long-term funding; (IV) organizational factors, such as the existence of a well-organized health system and effective coordination of interventions; and (V) political or legislative aspects, in this case strong government support to deploy technology on a large scale. Telehealth and mhealth are promising solutions to reduce maternal morbidity and mortality in SSA, but knowledge on how these interventions can succeed and move to scale is limited. Success factors identified in this review can provide guidance on elements that should be considered in the design and implementation of telehealth and mhealth for maternal health in SSA.

Mobile app rating scale: a new tool for assessing the quality of health mobile apps

Background

The use of mobile apps for health and well being promotion has grown exponentially in recent years. Yet, there is currently no app-quality assessment tool beyond “star”-ratings.

Objective

The objective of this study was to develop a reliable, multidimensional measure for trialling, classifying, and rating the quality of mobile health apps.

Methods

A literature search was conducted to identify articles containing explicit Web or app quality rating criteria published between January 2000 and January 2013. Existing criteria for the assessment of app quality were categorized by an expert panel to develop the new Mobile App Rating Scale (MARS) subscales, items, descriptors, and anchors. There were sixty well being apps that were randomly selected using an iTunes search for MARS rating. There were ten that were used to pilot the rating procedure, and the remaining 50 provided data on interrater reliability.

Results

There were 372 explicit criteria for assessing Web or app quality that were extracted from 25 published papers, conference proceedings, and Internet resources. There were five broad categories of criteria that were identified including four objective quality scales: engagement, functionality, aesthetics, and information quality; and one subjective quality scale; which were refined into the 23-item MARS. The MARS demonstrated excellent internal consistency (alpha = .90) and interrater reliability intraclass correlation coefficient (ICC = .79).

Conclusions

The MARS is a simple, objective, and reliable tool for classifying and assessing the quality of mobile health apps. It can also be used to provide a checklist for the design and development of new high quality health apps.

"Real-world" practical evaluation strategies: a review of telehealth evaluation

Background

Currently, the increasing interest in telehealth and significant technological breakthroughs of the past decade create favorable conditions for the widespread adoption of telehealth services. Therefore, expectations are high that telehealth can help alleviate prevailing challenges in health care delivery. However, in order to translate current research to policy and facilitate adoption by patients and health care providers, there is need for compelling evidence of the effectiveness of telehealth interventions. Such evidence is gathered from rigorously designed research studies, which may not always be practical in many real-world settings.

Objective

Our aim was to summarize current telehealth evaluation strategies and challenges and to outline practical approaches to conduct evaluation in real-world settings using one of our previously reported telehealth initiatives, the Diabetes Connect program, as a case study.

Methods

We reviewed commonly used current evaluation frameworks and strategies, as well as best practices based on successful evaluative efforts to date to address commonly encountered challenges in telehealth evaluation. These challenges in telehealth evaluation and commonly used frameworks are described relevant to the evaluation of Diabetes Connect, a 12-month Web-based blood glucose monitoring program.

Results

Designers of telehealth evaluation frameworks must give careful consideration to the elements of planning, implementation, and impact assessment of interventions. Evaluating performance at each of these phases is critical to the overall success of an intervention. Although impact assessment occurs at the end of a program, our review shows that it should begin at the point of problem definition. Critical to the success of an evaluative strategy is early planning that involves all stakeholders to identify the overall goals of the program and key measures of success at each phase of the program life cycle. This strategy should enable selection of an appropriate evaluation strategy and measures to aid in the ongoing development and implementation of telehealth and provide better evidence of program impact.

Conclusions

We recommend a pragmatic, multi-method, multi-phase approach to telehealth evaluation that is flexible and can be adapted to the characteristics and challenges unique to each telehealth program.

Telehealth distance education course in Latin America: analysis of an experience involving 15 countries

BACKGROUND

Telehealth activities are already going on in many Latin American countries. This article aims to present and evaluate a distance learning telehealth training course in the region.

MATERIALS AND METHODS

This was a cross-sectional descriptive study. A coordinating committee was formed, composed of medical school faculty from 15 countries, which defined the course's syllabus, teaching model, and mentoring structure. A questionnaire was prepared, using a Likert scale, in order to verify if the parameters of gender, age, professional category, postgraduate degree, and experience in distance education indicated any difference in relation to the course evaluation. The responses were analyzed by chi-squared test, considering as significant a value of p<0.05.

RESULTS

Of the 353 enrolled participants, 251 (71.10%) did the basic modules, and 96 (43.91%) completed the full training. In relation to the overall course assessment, it was considered excellent or good by 80.92% of participants, the mentors received positive evaluations by 72.83% of students, the course content was evaluated as excellent or good by 87.4% of students, and 94.40% of participants would recommend it. As for the parameters assessed, only experience in distance education was statistically significant for the evaluation of the tutors.

CONCLUSIONS

The results presented indicate an important concern on the part of the Latin American countries participating on the course in relation to telehealth training activities. Regarding course assessment, high approval rates in relation to tutoring, educational model, course content, and goals were noted, corroborating literature data. The experience of conducting a Latin American shared telehealth training course was indeed positive, contributing to the development of telehealth actions.

Standards and Guidelines in Telemedicine and Telehealth

The development of guidelines and standards for telemedicine is an important and valuable process to help insure effective and safe delivery of quality healthcare. Some organizations, such as the American Telemedicine Association (ATA), have made the development of standards and guidelines a priority. The practice guidelines developed so far have been well received by the telemedicine community and are being adopted in numerous practices, as well as being used in research to support the practice and growth of telemedicine. Studies that utilize published guidelines not only help bring them into greater public awareness, but they also provide evidence needed to validate existing guidelines and guide the revision of future versions. Telemedicine will continue to grow and be adopted by more healthcare practitioners and patients in a wide variety of forms not just in the traditional clinical environments, and practice guidelines will be a key factor in fostering this growth. Creation of guidelines is important to payers and regulators as well as increasingly they are adopting and integrating them into regulations and policies. This paper will review some of the recent ATA efforts in developing telemedicine practice guidelines, review the role of research in guidelines development, review data regarding their use, and discuss some of areas where guidelines are still needed.