Health technology assessment frameworks for eHealth: A systematic review

Methodological Frameworks and Dimensions to Be Considered in Digital Health Technology Assessment: Scoping Review and Thematic Analysis

BACKGROUND

Digital health technologies (dHTs) offer a unique opportunity to address some of the major challenges facing health care systems worldwide. However, the implementation of dHTs raises some concerns, such as the limited understanding of their real impact on health systems and people's well-being or the potential risks derived from their use. In this context, health technology assessment (HTA) is 1 of the main tools that health systems can use to appraise evidence and determine the value of a given dHT. Nevertheless, due to the nature of dHTs, experts highlight the need to reconsider the frameworks used in traditional HTA.

OBJECTIVE

This scoping review (ScR) aimed to identify the methodological frameworks used worldwide for digital health technology assessment (dHTA); determine what domains are being considered; and generate, through a thematic analysis, a proposal for a methodological framework based on the most frequently described domains in the literature.

METHODS

The ScR was performed in accordance with the guidelines established in the PRISMA-ScR guidelines. We searched 7 databases for peer reviews and gray literature published between January 2011 and December 2021. The retrieved studies were screened using Rayyan in a single-blind manner by 2 independent authors, and data were extracted using ATLAS.ti software. The same software was used for thematic analysis.

RESULTS

The systematic search retrieved 3061 studies (n=2238, 73.1%, unique), of which 26 (0.8%) studies were included. From these, we identified 102 methodological frameworks designed for dHTA. These frameworks revealed great heterogeneity between them due to their different structures, approaches, and items to be considered in dHTA. In addition, we identified different wording used to refer to similar concepts. Through thematic analysis, we reduced this heterogeneity. In the first phase of the analysis, 176 provisional codes related to different assessment items emerged. In the second phase, these codes were clustered into 86 descriptive themes, which, in turn, were grouped in the third phase into 61 analytical themes and organized through a vertical hierarchy of 3 levels: level 1 formed by 13 domains, level 2 formed by 38 dimensions, and level 3 formed by 11 subdimensions. From these 61 analytical themes, we developed a proposal for a methodological framework for dHTA.

CONCLUSIONS

There is a need to adapt the existing frameworks used for dHTA or create new ones to more comprehensively assess different kinds of dHTs. Through this ScR, we identified 26 studies including 102 methodological frameworks and tools for dHTA. The thematic analysis of those 26 studies led to the definition of 12 domains, 38 dimensions, and 11 subdimensions that should be considered in dHTA.

[Technology assessment of digital health applications for reimbursement decisions]

BACKGROUND

For most digital health applications (DiGA) only limited evidence of benefit is available. Currently available assessment frameworks do not cover all domains of a full health technology assessment (HTA). Additionally, technology-specific aspects are required for the evaluation of DiGA. This work aimed to analyze the available assessment frameworks and design an evaluation process for DiGA.

METHODS

By a systematic literature search six assessment frameworks for DiGA were selected and analyzed. A hand search for strategies on DiGA of selected countries was conducted.

RESULTS

Of the analyzed assessment frameworks four described study designs. One assessment framework proposed a risk classification of DiGA. Aspects of artificial intelligence were assessed by one assessment framework. The analyzed countries have differing strategies for reimbursement of DiGA.

CONCLUSION

Assessment frameworks for DiGA are very heterogeneous. There are efforts to find regulations for DiGA on a national level. When evaluating DiGA, a staged approach considering risk classes with subsequent evaluation of relevant HTA aspects is recommended.

A sociotechnical framework to assess patient-facing eHealth tools: results of a modified Delphi process

Among the thousands of eHealth tools available, the vast majority do not get past pilot phases because they cannot prove value, and only a few have been systematically assessed. Although multiple eHealth assessment frameworks have been developed, these efforts face multiple challenges. This study aimed to address some of these challenges by validating and refining an initial list of 55 assessment criteria based on previous frameworks through a two-round modified Delphi process with in-between rounds of interviews. The expert panel (n = 57) included participants from 18 countries and 9 concerned parties. A consensus was reached on 46 criteria that were classified into foundational and contextual criteria. The 36 foundational criteria focus on evaluating the eHealth tool itself and were grouped into nine clusters: technical aspects, clinical utility and safety, usability and human centricity, functionality, content, data management, endorsement, maintenance, and developer. The 10 contextual criteria focus on evaluating the factors that vary depending on the context the tool is being evaluated for and were grouped into seven clusters: data-protection compliance, safety regulatory compliance, interoperability and data integration, cultural requirements, affordability, cost-benefit, and implementability. The classification of criteria into foundational and contextual helps us assess not only the quality of an isolated tool, but also its potential fit in a specific setting. Criteria subscales may be particularly relevant when determining the strengths and weaknesses of the tool being evaluated. This granularity enables different concerned parties to make informed decisions about which tools to consider according to their specific needs and priorities.

International practices in health technology assessment and public financing of digital health technologies: recommendations for Hungary

Background

Evaluating and integrating digital health technologies is a critical component of a national healthcare ecosystem in the 2020s and is expected to even increase in significance.

Design

The paper gives an overview of international practices on public financing and health technology assessment of digital health technologies (DHTs) in five European Union (EU) countries and outlines recommendations for country-level action that relevant stakeholders can consider in order to support uptake of digital health solutions in Hungary. A scoping review was carried out to identify and gather country-specific classifications and international practices on the financing DHTs in five pioneering EU countries: Germany, France, Belgium, the United Kingdom and Finland.

Results

Several frameworks have been developed for DHTs, however there is no single, unified framework or method for classification, evaluation, and financing of digital health technologies in European context. European countries apply different taxonomy, use different assessment domains and regulations for the reimbursement of DHTs. The Working Group of the Hungarian Health Economic Society recommends eight specific points for stakeholders, importantly taking active role in shaping common clinical evidence standards and technical quality criteria across in order for common standards to be developed in the European Union single market.

Conclusion

Specificities of national healthcare contexts must be taken into account in decisions to allocate public funds to certain therapies rather than others.

Strengthening Digital Transformation and Innovation in the Health Care System: Protocol for the Design and Implementation of a Multidisciplinary National Health Innovation Research School

BACKGROUND

Digital health technologies have the potential to transform health care services to be more cost-effective, coordinated, and accessible on equal terms for entire populations. In the future, people will be assisted by such technologies to monitor their health status, take preventive measures, and have more control of their health situation. An increase in digital supplementation or substitution of physical care visits can potentially add value to patients and care providers by increasing accessibility, safety, and quality of care. However, health care organizations struggle with the challenges of developing and implementing digital health technologies and services in practice. As a response to this, we have developed a national multidisciplinary research school to increase competence and capacity for research on the development, implementation, and dissemination of digital health technology solutions. The overall aim of the research school is to increase national competence and capacity for the development, implementation, and dissemination of digital health technology to increase the preparedness to support and facilitate the ongoing digital transformation in the health care system.

OBJECTIVE

The purpose of this paper is to outline the protocol for the development and implementation of a national multidisciplinary doctoral education program of health innovation supporting digital transformation in the health care system.

METHODS

A national multidisciplinary research school for health innovation was planned in collaboration between 7 Swedish universities and their partners from industry and the public sector. The research school will run over 6 years, of which 5 years are dedicated for the doctoral education program and 1 year for the project start-up and closing. In this paper, we outline the methodological approach of the research school; the combining of knowledge and expertise of the universities that are important to run the research school; the jointly formulated research-oriented and societally relevant research focus, goals, and objectives for the research school; the established and developed relationships with partners from industry and the public sector for joint research training projects; the forms of collaboration in the research school; and the format of the doctoral education process.

RESULTS

The research school was funded in December 2021 and started in March 2022. The research school starts with an initiation period from March 2022 to December 2022 where the infrastructure and the action plans to run the school are set up. The PhD projects start in January 2023, and these projects will be completed in 5 years. Additional activities within the research program are doctoral courses, networking activities, and dissemination of results.

CONCLUSIONS

The network of several partners from industry, public sector, and academia enables the research school to pose research questions that can contribute to solving relevant societal problems related to the development, evaluation, implementation, and dissemination of methods and processes assisted by digital technologies. Ultimately, this will promote innovation to improve health outcomes, quality of care, and prioritizations of resources.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/46595.

eHealth policy framework in Low and Lower Middle-Income Countries; a PRISMA systematic review and analysis

BACKGROUND

Low and lower middle-income countries suffer lack of healthcare providers and proper workforce education programs, a greater spread of illnesses, poor surveillance, efficient management, etc., which are addressable by a central policy framework implementation. Accordingly, an eHealth policy framework is required specifically for these countries to successfully implement eHealth solutions. This study explores existing frameworks and fills the gap by proposing an eHealth policy framework in the context of developing countries.

METHODS

This PRISMA-based (PRISMA Preferred Reporting Items For Systematic Reviews and Meta-Analyses) systematic review used Google Scholar, IEEE, Web of Science, and PubMed latest on 23^rd May 2022, explored 83 publications regarding eHealth policy frameworks, and extracted 11 publications scrutinizing eHealth policy frameworks in their title, abstract, or keywords. These publications were analyzed by using both expert opinion and Rstudio programming tools. They were explored based on their developing/developed countries' context, research approach, main contribution, constructs/dimensions of the framework, and related categories. In addition, by using cloudword and latent semantic space techniques, the most discussed concepts and targeted keywords were explored and a correlation test was conducted to depict the important concepts mentioned in the related literature and extract their relation with the targeted keywords in the interest of this study.

RESULTS

Most of these publications do not develop or synthesize new frameworks for eHealth policy implementation, but rather introduce eHealth implementation frameworks, explain policy dimensions, identify and extract relevant components of existing frameworks or point out legal or other relevant eHealth implementation issues.

CONCLUSION

After a thorough exploration of related literature, this study identified the main factors affecting an effective eHealth policy framework, found a gap in the context of developing countries, and proposed a four-step eHealth policy implementation guideline for successful implementation of eHealth in the context of developing. The limitation of this study is the lack of a proper amount of practically implemented eHealth policy framework cases in developing countries published in the literature for the review. Ultimately, this study is part of the BETTEReHEALTH (More information about the BETTEReHEALTH project at https://betterehealth.eu ) project funded by the European Union Horizon's 2020 under agreement number 101017450.

Evaluation of eHealth assistance in-hospital care for improved quality of life in patients

Health conditions contribute significantly to patients' quality of life. Healthcare infrastructure and healthcare services, including their accessibility, belong to objective factors influencing their perception of their health. The growing disparity between supply and demand for specialized inpatient facilities due to the aging population calls for new solutions, including eHealth technologies. Automatized activities could be taken over by eHealth technologies that do not require a constant presence of staff. We tested whether eHealth technical solutions reduce patients' health risks on a sample of 61 patients on the covid-19 unit in Tomas Bata hospital in Zlin. We have applied the randomized control trial to select patients for the treatment and the control groups. Moreover, we tested eHealth technologies and their help to staff in the hospital. Due to the severity of the covid-19 disease and its rapid course and the size of the sample in our research, we did not demonstrate a statistically significant impact of eHealth technologies on patient health. The evaluation results confirm that even the limited number of technologies deployed proves to be an effective help for staff in critical situations like the pandemic. The main issue is psychological support to staff in hospitals and relieving stressful work.

Assessing the Quality and Impact of eHealth Tools: Systematic Literature Review and Narrative Synthesis

BACKGROUND

Technological advancements have opened the path for many technology providers to easily develop and introduce eHealth tools to the public. The use of these tools is increasingly recognized as a critical quality driver in health care; however, choosing a quality tool from the myriad of tools available for a specific health need does not come without challenges.

OBJECTIVE

This review aimed to systematically investigate the literature to understand the different approaches and criteria used to assess the quality and impact of eHealth tools by considering sociotechnical factors (from technical, social, and organizational perspectives).

METHODS

A structured search was completed following the participants, intervention, comparators, and outcomes framework. We searched the PubMed, Cochrane, Web of Science, Scopus, and ProQuest databases for studies published between January 2012 and January 2022 in English, which yielded 675 results, of which 40 (5.9%) studies met the inclusion criteria. The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines and the Cochrane Handbook for Systematic Reviews of Interventions were followed to ensure a systematic process. Extracted data were analyzed using NVivo (QSR International), with a thematic analysis and narrative synthesis of emergent themes.

RESULTS

Similar measures from the different papers, frameworks, and initiatives were aggregated into 36 unique criteria grouped into 13 clusters. Using the sociotechnical approach, we classified the relevant criteria into technical, social, and organizational assessment criteria. Technical assessment criteria were grouped into 5 clusters: technical aspects, functionality, content, data management, and design. Social assessment criteria were grouped into 4 clusters: human centricity, health outcomes, visible popularity metrics, and social aspects. Organizational assessment criteria were grouped into 4 clusters: sustainability and scalability, health care organization, health care context, and developer.

CONCLUSIONS

This review builds on the growing body of research that investigates the criteria used to assess the quality and impact of eHealth tools and highlights the complexity and challenges facing these initiatives. It demonstrates that there is no single framework that is used uniformly to assess the quality and impact of eHealth tools. It also highlights the need for a more comprehensive approach that balances the social, organizational, and technical assessment criteria in a way that reflects the complexity and interdependence of the health care ecosystem and is aligned with the factors affecting users' adoption to ensure uptake and adherence in the long term.

WtsWrng Interim Comparative Effectiveness Evaluation and Description of the Challenges to Develop, Assess, and Introduce This Novel Digital Application in a Traditional Health System

Science and technology have evolved quickly during the two decades of the 21st century, but healthcare systems are grounded in last century's structure and processes. Changes in the way health care is provided are demanded; digital transformation is a key driver making healthcare systems more accessible, agile, efficient, and citizen-centered. Nevertheless, the way healthcare systems function challenges the development (Innovation + Development and regulatory requirements), assessment (methodological guidance weaknesses), and adoption of digital applications (DAs). WtsWrng (WW), an innovative DA which uses images to interact with citizens for symptom triage and monitoring, is used as an example to show the challenges faced in its development and clinical validation and how these are being overcome. To prove WW's value from inception, novel approaches for evidence generation that allows for an agile and patient-centered development have been applied. Early scientific advice from NICE (UK) was sought for study design, an iterative development and interim analysis was performed, and different statistical parameters (Kappa, B statistic) were explored to face development and assessment challenges. WW triage accuracy at cutoff time ranged from 0.62 to 0.94 for the most frequent symptoms attending the Emergency Department (ED), with the observed concordance for the 12 most frequent diagnostics at hospital discharge fluctuating between 0.4 to 0.97; 8 of the diagnostics had a concordance greater than 0.8. This experience should provoke reflective thinking for DA developers, digital health scientists, regulators, health technology assessors, and payers.

How Digital Therapeutics Are Urging the Need for a Paradigm Shift: From Evidence-Based Health Care to Evidence-Based Well-being

A scientific paradigm consists of a set of shared rules, beliefs, values, methods, and instruments for addressing scientific problems. Currently, health care embraces the paradigm of evidence-based health care (EBH). This paradigm prompts health care institutions to base decisions on the best available evidence, which is commonly generated in large-scale randomized controlled trials. We illustrate the application of EBH via the evaluation of drugs. We show how EBH is challenged when it is applied to the evaluation of digital therapeutics, which refers to technology and data to prevent, manage, or treat a medical disorder or disease. We conclude that amid the growing application of digital therapeutics, the paradigm of EBH is challenged in four domains: population, intervention, comparison, outcome. In the second part of this viewpoint, we argue for a paradigm shift in health care so we can optimally evaluate and implement digital therapeutics, and we sketch out the contours of this novel paradigm. We address the need for considering design in health care and evaluation processes, studying user values so that health care can move from a focus on health to well-being, focusing on individual experiences rather than the average, addressing the need for evaluation in authentic use contexts, and stressing the need for continuous evaluation of the dynamic relations between users, context, and digital therapeutics. We conclude that the transition from EBH toward evidence-based well-being would improve the successful implementation of digital technologies in health care.

Recommendations for developing a lifecycle, multidimensional assessment framework for mobile medical apps

Digital health and mobile medical apps (MMAs) have shown great promise in transforming health care, but their adoption in clinical care has been unsatisfactory, and regulatory guidance and coverage decisions have been lacking or incomplete. A multidimensional assessment framework for regulatory, policymaking, health technology assessment, and coverage purposes based on the MMA lifecycle is needed. A targeted review of relevant policy documents from international sources was conducted to map current MMA assessment frameworks, to formulate 10 recommendations, subsequently shared amongst an expert panel of key stakeholders. Recommendations go beyond economic dimensions such as cost and economic evaluation and also include MMA development and update, classification and evidentiary requirements, performance and maintenance monitoring, usability testing, clinical evidence requirements, safety and security, equity considerations, organizational assessment, and additional outcome domains (patient empowerment and environmental impact). The COVID-19 pandemic greatly expanded the use of MMAs, but temporary policies governing their use and oversight need consolidation through well-developed frameworks to support decision-makers, producers and introduction into clinical care processes, especially in light of the strong international, cross-border character of MMAs, the new EU medical device and health technology assessment regulations, and the Next Generation EU funding earmarked for health digitalization.

Implementation of a new Digi-HTA process for digital health technologies in Finland

OBJECTIVES

There has been a lack of health technology assessment (HTA) methods for novel digital health technologies (DHTs) such as mHealth, artificial intelligence, and robotics in Finland. The Digi-HTA method has been developed for this purpose. The aim of this study is to determine whether it would be possible to use Digi-HTA recommendations to support healthcare decision-makers. Secondly, from the perspective of companies offering different types of DHT products, this study assesses the suitability of using the Digi-HTA framework to perform HTAs for their products.

METHODS

Feedback about Digi-HTA recommendations was collected from healthcare professionals. DHT companies provided input about the Digi-HTA framework. Data were collected via a web-based survey and were analyzed using qualitative methods.

RESULTS

Of the twenty-four healthcare professional respondents, twenty said that the Digi-HTA recommendations contained all the necessary information, and twenty-one found them useful for their work. Respondents hoped that the Digi-HTA recommendations would be better integrated into the decision-making processes and healthcare professionals would be more informed about this new HTA process. The questions of the Digi-HTA framework were applicable for different DHT products based on the responses from DHT companies (n = 8).

CONCLUSIONS

According to the study participants, although the Digi-HTA recommendations include clear and beneficial information, their integration into healthcare decision-making processes should be improved. Responses from DHT companies indicate that the Digi-HTA framework would be an appropriate tool for performing assessments for their products. To generalize the findings of this study, more comprehensive studies will be needed.

Real-World Implementation of Precision Psychiatry: A Systematic Review of Barriers and Facilitators

Background: Despite significant research progress surrounding precision medicine in psychiatry, there has been little tangible impact upon real-world clinical care. Objective: To identify barriers and facilitators affecting the real-world implementation of precision psychiatry. Method: A PRISMA-compliant systematic literature search of primary research studies, conducted in the Web of Science, Cochrane Central Register of Controlled Trials, PsycINFO and OpenGrey databases. We included a qualitative data synthesis structured according to the ‘Consolidated Framework for Implementation Research’ (CFIR) key constructs. Results: Of 93,886 records screened, 28 studies were suitable for inclusion. The included studies reported 38 barriers and facilitators attributed to the CFIR constructs. Commonly reported barriers included: potential psychological harm to the service user (n = 11), cost and time investments (n = 9), potential economic and occupational harm to the service user (n = 8), poor accuracy and utility of the model (n = 8), and poor perceived competence in precision medicine amongst staff (n = 7). The most highly reported facilitator was the availability of adequate competence and skills training for staff (n = 7). Conclusions: Psychiatry faces widespread challenges in the implementation of precision medicine methods. Innovative solutions are required at the level of the individual and the wider system to fulfil the translational gap and impact real-world care.

Challenges to implementing artificial intelligence in healthcare: a qualitative interview study with healthcare leaders in Sweden

BACKGROUND

Artificial intelligence (AI) for healthcare presents potential solutions to some of the challenges faced by health systems around the world. However, it is well established in implementation and innovation research that novel technologies are often resisted by healthcare leaders, which contributes to their slow and variable uptake. Although research on various stakeholders' perspectives on AI implementation has been undertaken, very few studies have investigated leaders' perspectives on the issue of AI implementation in healthcare. It is essential to understand the perspectives of healthcare leaders, because they have a key role in the implementation process of new technologies in healthcare. The aim of this study was to explore challenges perceived by leaders in a regional Swedish healthcare setting concerning the implementation of AI in healthcare.

METHODS

The study takes an explorative qualitative approach. Individual, semi-structured interviews were conducted from October 2020 to May 2021 with 26 healthcare leaders. The analysis was performed using qualitative content analysis, with an inductive approach.

RESULTS

The analysis yielded three categories, representing three types of challenge perceived to be linked with the implementation of AI in healthcare: 1) Conditions external to the healthcare system; 2) Capacity for strategic change management; 3) Transformation of healthcare professions and healthcare practice.

CONCLUSIONS

In conclusion, healthcare leaders highlighted several implementation challenges in relation to AI within and beyond the healthcare system in general and their organisations in particular. The challenges comprised conditions external to the healthcare system, internal capacity for strategic change management, along with transformation of healthcare professions and healthcare practice. The results point to the need to develop implementation strategies across healthcare organisations to address challenges to AI-specific capacity building. Laws and policies are needed to regulate the design and execution of effective AI implementation strategies. There is a need to invest time and resources in implementation processes, with collaboration across healthcare, county councils, and industry partnerships.

Evaluation of Digital Interventions for Physical Activity Promotion: Scoping Review

Background

Digital interventions are interventions supported by digital tools or technologies, such as mobile apps, wearables, or web-based software. Digital interventions in the context of public health are specifically designed to promote and improve health. Recent reviews have shown that many digital interventions target physical activity promotion; however, it is unclear how such digital interventions are evaluated.

Objective

We aimed to investigate evaluation strategies in the context of digital interventions for physical activity promotion using a scoping review of published reviews. We focused on the target (ie, user outcomes or tool performance), methods (ie, tool data or self-reported data), and theoretical frameworks of the evaluation strategies.

Methods

A protocol for this study was preregistered and published. From among 300 reviews published up to March 19, 2021 in Medline, PsycINFO, and CINAHL databases, 40 reviews (1 rapid, 9 scoping, and 30 systematic) were included in this scoping review. Two authors independently performed study selection and data coding. Consensus was reached by discussion. If applicable, data were coded quantitatively into predefined categories or qualitatively using definitions or author statements from the included reviews. Data were analyzed using either descriptive statistics, for quantitative data (relative frequencies out of all studies), or narrative synthesis focusing on common themes, for qualitative data.

Results

Most reviews that were included in our scoping review were published in the period from 2019 to 2021 and originated from Europe or Australia. Most primary studies cited in the reviews included adult populations in clinical or nonclinical settings, and focused on mobile apps or wearables for physical activity promotion. The evaluation target was a user outcome (efficacy, acceptability, usability, feasibility, or engagement) in 38 of the 40 reviews or tool performance in 24 of the 40 reviews. Evaluation methods relied upon objective tool data (in 35/40 reviews) or other data from self-reports or assessments (in 28/40 reviews). Evaluation frameworks based on behavior change theory, including goal setting, self-monitoring, feedback on behavior, and educational or motivational content, were mentioned in 22 out of 40 reviews. Behavior change theory was included in the development phases of digital interventions according to the findings of 20 out of 22 reviews.

Conclusions

The evaluation of digital interventions is a high priority according to the reviews included in this scoping review. Evaluations of digital interventions, including mobile apps or wearables for physical activity promotion, typically target user outcomes and rely upon objective tool data. Behavior change theory may provide useful guidance not only for development of digital interventions but also for the evaluation of user outcomes in the context of physical activity promotion. Future research should investigate factors that could improve the efficacy of digital interventions and the standardization of terminology and reporting in this field.

International Registered Report Identifier (IRRID)

RR2-10.2196/35332

Evolving Optimised Convolutional Neural Networks for Lung Cancer Classification

Detecting pulmonary nodules early significantly contributes to the treatment success of lung cancer. Several deep learning models for medical image analysis have been developed to help classify pulmonary nodules. The design of convolutional neural network (CNN) architectures, however, is still heavily reliant on human domain knowledge. Manually designing CNN design solutions has been shown to limit the data’s utility by creating a co-dependency on the creator’s cognitive bias, which urges the development of smart CNN architecture design solutions. In this paper, an evolutionary algorithm is used to optimise the classification of pulmonary nodules with CNNs. The implementation of a genetic algorithm (GA) for CNN architectures design and hyperparameter optimisation is proposed, which approximates optimal solutions by implementing a range of bio-inspired mechanisms of natural selection and Darwinism. For comparison purposes, two manually designed deep learning models, FractalNet and Deep Local-Global Network, were trained. The results show an outstanding classification accuracy of the fittest GA-CNN (91.3%), which outperformed both manually designed models. The findings indicate that GAs pose advantageous solutions for diagnostic challenges, the development of which may to be fully automated in the future using GAs to design and optimise CNN architectures for various clinical applications.

Searching for Sustainability in Health Systems: Toward a Multidisciplinary Evaluation of Mobile Health Innovations

Mobile health (mHealth) innovations are considered by governments as game changers toward more sustainable health systems. The existing literature focuses on the clinical aspects of mHealth but lacks an integrated framework on its sustainability. The foundational idea for this paper is to include disciplinary complementarities into a multi-dimensional vision to evaluate the non-clinical aspects of mHealth innovations. We performed a targeted literature review to find how the sustainability of mHealth innovations was appraised in each discipline. We found that each discipline considers a different outcome of interest and adopts different time horizons and perspectives for the evaluation. This article reflects on how the sustainability of mHealth innovation can be assessed at both the level of the device itself as well as the level of the health system. We identify some of the challenges ahead of researchers working on mobile health innovations in contributing to shaping a more sustainable health system.

From Digital Health to Digital Well-being: Systematic Scoping Review

Background

Digital health refers to the proper use of technology for improving the health and well-being of people and enhancing the care of patients through the intelligent processing of clinical and genetic data. Despite increasing interest in well-being in both health care and technology, there is no clear understanding of what constitutes well-being, which leads to uncertainty in how to create well-being through digital health. In an effort to clarify this uncertainty, Brey developed a framework to define problems in technology for well-being using the following four categories: epistemological problem, scope problem, specification problem, and aggregation problem.

Objective

This systematic scoping review aims to gain insights into how to define and address well-being in digital health.

Methods

We followed the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklist. Papers were identified from 6 databases and included if they addressed the design or evaluation of digital health and reported the enhancement of patient well-being as their purpose. These papers were divided into design and evaluation papers. We studied how the 4 problems in technology for well-being are considered per paper.

Results

A total of 117 studies were eligible for analysis (n=46, 39.3% design papers and n=71, 60.7% evaluation papers). For the epistemological problem, the thematic analysis resulted in various definitions of well-being, which were grouped into the following seven values: healthy body, functional me, healthy mind, happy me, social me, self-managing me, and external conditions. Design papers mostly considered well-being as healthy body and self-managing me, whereas evaluation papers considered the values of healthy mind and happy me. Users were rarely involved in defining well-being. For the scope problem, patients with chronic care needs were commonly considered as the main users. Design papers also regularly involved other users, such as caregivers and relatives. These users were often not involved in evaluation papers. For the specification problem, most design and evaluation papers focused on the provision of care support through a digital platform. Design papers used numerous design methods, whereas evaluation papers mostly considered pre-post measurements and randomized controlled trials. For the aggregation problem, value conflicts were rarely described.

Conclusions

Current practice has found pragmatic ways of circumventing or dealing with the problems of digital health for well-being. Major differences exist between the design and evaluation of digital health, particularly regarding their conceptualization of well-being and the types of users studied. In addition, we found that current methodologies for designing and evaluating digital health can be improved. For optimal digital health for well-being, multidisciplinary collaborations that move beyond the common dichotomy of design and evaluation are needed.

Evaluation in Life Cycle of Information Technology (ELICIT) framework: Supporting the innovation life cycle from business case assessment to summative evaluation

OBJECTIVE

Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation.

METHODS

The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies.

RESULTS

The resulting Evaluation in Life Cycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: (1) business case assessment; (2) stakeholder requirements gathering; (3) technical requirements gathering; (4) technical acceptability assessment; (5) user acceptability assessment; (6) social acceptability assessment; (7) social implementation assessment; (8) initial user satisfaction assessment; (9) technical implementation assessment; (10) technical portability assessment; (11) long-term user satisfaction assessment; and (12) social outcomes assessment.

DISCUSSION

Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation.

CONCLUSION

As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.

Perspectives of Policy Makers and Service Users Concerning the Implementation of eHealth in Sweden: Interview Study

Background

Increasing life spans of populations and a growing demand for more advanced care make effective and cost-efficient provision of health care necessary. eHealth technology is often proposed, although research on barriers to and facilitators of the implementation of eHealth technology is still scarce and fragmented.

Objective

The aim of this study is to explore the perceptions concerning barriers to and facilitators of the implementation of eHealth among policy makers and service users and explore the ways in which their perceptions converge and differ.

Methods

This study used interview data from policy makers at different levels of health care (n=7) and service users enrolled in eHealth interventions (n=25). The analysis included separate qualitative content analyses for the 2 groups and then a second qualitative content analysis to explore differences and commonalities.

Results

Implementation barriers perceived by policy makers were that not all service users benefit from eHealth and that there is uncertainty about the impact of eHealth on the work of health care professionals. Policy makers also perceived political decision-making as complex; this included problems related to provision of technical infrastructure and lack of extra resources for health care digitalization. Facilitators were policy makers’ conviction that eHealth is what citizens want, their belief in eHealth solutions as beneficial for health care practice, and their belief in the importance of health care digitalization. Barriers for service users comprised capability limitations and varied preferences of service users and a mismatch of technology with user needs, lack of data protection, and their perception of eHealth as being more time consuming. Facilitators for service users were eHealth technology design and match with their skill set, personal feedback and staff support, a sense of privacy, a credible sender, and flexible use of time.There were several commonalities between the 2 stakeholder groups. Facilitators for both groups were the strong impetus toward technology adoption in society and expectations of time flexibility. Both groups perceived barriers in the difficulties of tailoring eHealth, and both groups expressed uncertainty about the care burden distribution. There were also differences: policy makers perceived that their decision-making was very complex and that resources for implementation were limited. Service users highlighted their need to feel that their digital data were protected and that they needed to trust the eHealth sender.

Conclusions

Perceptions about barriers to and facilitators of eHealth implementation varied among stakeholders in different parts of the health care system. The study points to the need to reach an enhanced mutual understanding of priorities and overcome challenges at both the micro and macro levels of the health care system. More well-balanced decisions at the policy-maker level may lead to more effective and sustainable development and future implementation of eHealth.

Implementation Frameworks for Artificial Intelligence Translation Into Health Care Practice: Scoping Review

Background

Significant efforts have been made to develop artificial intelligence (AI) solutions for health care improvement. Despite the enthusiasm, health care professionals still struggle to implement AI in their daily practice.

Objective

This paper aims to identify the implementation frameworks used to understand the application of AI in health care practice.

Methods

A scoping review was conducted using the Cochrane, Evidence Based Medicine Reviews, Embase, MEDLINE, and PsycINFO databases to identify publications that reported frameworks, models, and theories concerning AI implementation in health care. This review focused on studies published in English and investigating AI implementation in health care since 2000. A total of 2541 unique publications were retrieved from the databases and screened on titles and abstracts by 2 independent reviewers. Selected articles were thematically analyzed against the Nilsen taxonomy of implementation frameworks, and the Greenhalgh framework for the nonadoption, abandonment, scale-up, spread, and sustainability (NASSS) of health care technologies.

Results

In total, 7 articles met all eligibility criteria for inclusion in the review, and 2 articles included formal frameworks that directly addressed AI implementation, whereas the other articles provided limited descriptions of elements influencing implementation. Collectively, the 7 articles identified elements that aligned with all the NASSS domains, but no single article comprehensively considered the factors known to influence technology implementation. New domains were identified, including dependency on data input and existing processes, shared decision-making, the role of human oversight, and ethics of population impact and inequality, suggesting that existing frameworks do not fully consider the unique needs of AI implementation.

Conclusions

This literature review demonstrates that understanding how to implement AI in health care practice is still in its early stages of development. Our findings suggest that further research is needed to provide the knowledge necessary to develop implementation frameworks to guide the future implementation of AI in clinical practice and highlight the opportunity to draw on existing knowledge from the field of implementation science.

A review of audiovisual telemedicine utilization and satisfaction assessment during the COVID-19 pandemic

INTRODUCTION

The use of telemedicine has broadened as technology that both restores continuity of care during disruptions in healthcare delivery and routinely provides primary care alone or in combination with in-person care. During the Covid-19 outbreak, the use of telemedicine as a routine care modality further accelerated.

METHODS

A review of scientific studies that used telemedicine to provide care from December 2019 to December 2020 is presented. From an initial set of 2,191 articles, 36 studies are analyzed. Evidence is organized and evaluated according to the country of study, the clinical specialty, the technology platform used, and satisfaction and utilization outcomes.

RESULTS

Thirty-one studies reported high patient satisfaction scores. Eight studies reported satisfaction from both providers and patients with no uniformly accepted assessment instrument. Eight studies conducted a descriptive analysis of telemedicine use and patient adoption patterns. Less than one-third of studies were controlled before/after studies. Most studies were conducted in the USA followed by Europe.

CONCLUSIONS

Reported satisfaction rates are high, consistent with previously documented research, whereas utilization rates increased significantly compared with the prepandemic period. Future work in developing standardized uniform assessment instruments, embedded with each telemedicine system, would increase versatility and agility in the assessment, boosting statistical power and the interpretation of results.

Use of E-Health in Dutch General Practice during the COVID-19 Pandemic

The COVID-19 pandemic has forced general practices to search for possibilities to provide healthcare remotely (e.g., e-health). In this study, the impact of the pandemic on the use of e-health in general practices in the Netherlands was investigated. In addition, the intention of practices to continue using e-health more intensively and differences in the use of e-health between practice types were investigated. For this purpose, web surveys were sent to general practices in April and July 2020. Descriptive data analysis was performed and differences in the use of e-health between practice types were tested using one-way ANOVA. Response rates were 34% (n = 1433) in April and 17% (n = 719) in July. The pandemic invoked an increased use of several (new) e-health applications. A minority of practices indicated the intention to maintain this increased use. In addition, small differences in the use of e-health between the different practice types were found. This study showed that although there was an increased uptake of e-health in Dutch general practice during the COVID-19 pandemic, only a minority of practices intends to maintain this increased use in the future. This may point towards a temporary uptake of digital healthcare delivery rather than accelerated implementation of digital processes.

Towards successful implementation of artificial intelligence in healthcare practice: A research program (Preprint)

Background

Objective

Methods

Results

Conclusions

International Registered Report Identifier (IRRID)

Evaluation framework to guide implementation of AI systems into healthcare settings

Objectives

To date, many artificial intelligence (AI) systems have been developed in healthcare, but adoption has been limited. This may be due to inappropriate or incomplete evaluation and a lack of internationally recognised AI standards on evaluation. To have confidence in the generalisability of AI systems in healthcare and to enable their integration into workflows, there is a need for a practical yet comprehensive instrument to assess the translational aspects of the available AI systems. Currently available evaluation frameworks for AI in healthcare focus on the reporting and regulatory aspects but have little guidance regarding assessment of the translational aspects of the AI systems like the functional, utility and ethical components.

Methods

To address this gap and create a framework that assesses real-world systems, an international team has developed a translationally focused evaluation framework termed ‘Translational Evaluation of Healthcare AI (TEHAI)’. A critical review of literature assessed existing evaluation and reporting frameworks and gaps. Next, using health technology evaluation and translational principles, reporting components were identified for consideration. These were independently reviewed for consensus inclusion in a final framework by an international panel of eight expert.

Results

TEHAI includes three main components: capability, utility and adoption. The emphasis on translational and ethical features of the model development and deployment distinguishes TEHAI from other evaluation instruments. In specific, the evaluation components can be applied at any stage of the development and deployment of the AI system.

Discussion

One major limitation of existing reporting or evaluation frameworks is their narrow focus. TEHAI, because of its strong foundation in translation research models and an emphasis on safety, translational value and generalisability, not only has a theoretical basis but also practical application to assessing real-world systems.

Conclusion

The translational research theoretic approach used to develop TEHAI should see it having application not just for evaluation of clinical AI in research settings, but more broadly to guide evaluation of working clinical systems.

Health Technology Assessment for Cardiovascular Digital Health Technologies and Artificial Intelligence: Why Is It Different?

Innovations in health care are growing exponentially, resulting in improved quality of and access to care, as well as rising societal costs of care and variable reimbursement. In recent years, digital health technologies and artificial intelligence have become of increasing interest in cardiovascular medicine owing to their unique ability to empower patients and to use increasing quantities of data for moving toward personalised and precision medicine. Health technology assessment agencies evaluate the money spent on a health care intervention or technology to attain a given clinical impact and make recommendations for reimbursement considerations. However, there is a scarcity of economic evaluations of cardiovascular digital health technologies and artificial intelligence. The current health technology assessment framework is not equipped to address the unique, dynamic, and unpredictable value considerations of these technologies and highlight the need to better approach the digital health technologies and artificial intelligence health technology assessment process. In this review, we compare digital health technologies and artificial intelligence with traditional health care technologies, review existing health technology assessment frameworks, and discuss challenges and opportunities related to cardiovascular digital health technologies and artificial intelligence health technology assessment. Specifically, we argue that health technology assessments for digital health technologies and artificial intelligence applications must allow for a much shorter device life cycle, given the rapid and even potentially continuously iterative nature of this technology, and thus an evidence base that maybe less mature, compared with traditional health technologies and interventions.

Evaluation frameworks for digital nursing technologies: analysis, assessment, and guidance. An overview of the literature

Background

The evaluation of digital nursing technologies (DNT) plays a major role in gaining knowledge about certain aspects of a technology such as acceptance, effectiveness, or efficiency. Evaluation frameworks can help to classify the success or failure of a DNT or to further develop the technology. In general, there are many different evaluation frameworks in the literature that provide overviews of a wide variety of aspects, which makes this a highly diverse field and raises the question how to select a suitable framework. The aim of this article is to provide orientation in the field of comprehensive evaluation frameworks that can be applied to the field of DNT and to conduct a detailed analysis and assessment of these frameworks to guide field researchers.

Methods

This overview was conducted using a three-component search process to identify relevant frameworks. These components were (1) a systematized literature search in PubMed; (2) a narrative review and (3) expert consultations. Data relating to the frameworks’ evaluation areas, purpose, perspectives, and success definitions were extracted. Quality criteria were developed in an expert workshop and a strength and weakness assessment was carried out.

Results

Eighteen relevant comprehensive evaluation frameworks for DNT were identified. Nine overarching evaluation areas, seven categories of purposes, five evaluation perspectives and three categories of success definitions could be identified. Eleven quality criteria for the strengths and weaknesses of DNT-related evaluation frameworks were developed and the included frameworks were assessed against them.

Conclusion

Evaluators can use the concise information and quality criteria of this article as a starting point to select and apply appropriate DNT evaluation frameworks for their research projects or to assess the quality of an evaluation framework for DNT, as well as a basis for exploring the questions raised in this article. Future research could address gaps and weaknesses in existing evaluation frameworks, which could improve the quality of future DNT evaluations.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12912-021-00654-8.

Technology assessment framework for precision health applications

OBJECTIVE

Established and emerging technologies-such as wearable sensors, smartphones, mobile apps, and artificial intelligence-are shaping positive healthcare models and patient outcomes. These technologies have the potential to become precision health (PH) innovations. However, not all innovations meet regulatory standards or have the required scientific evidence to be used for health applications. In response, an assessment framework was developed to facilitate and standardize the assessment of innovations deemed suitable for PH.

METHODS

A scoping literature review undertaken through PubMed and Google Scholar identified approximately 100 relevant articles. These were then shortlisted (n = 12) to those that included specific metrics, criteria, or frameworks for assessing technologies that could be applied to the PH context.

RESULTS

The proposed framework identified nine core criteria with subcriteria and grouped them into four categories for assessment: technical, clinical, human factors, and implementation. Guiding statements with response options and recommendations were used as metrics against each criterion.

CONCLUSION

The proposed framework supports health services, health technology innovators, and researchers in leveraging current and emerging technologies for PH innovations. It covers a comprehensive set of criteria as part of the assessment process of these technologies.