Evaluation Practices of a Major Canadian Telehealth Provider: Lessons and Future Directions for the Field

Methodological Frameworks and Dimensions to Be Considered in Digital Health Technology Assessment: Scoping Review and Thematic Analysis

BACKGROUND

Digital health technologies (dHTs) offer a unique opportunity to address some of the major challenges facing health care systems worldwide. However, the implementation of dHTs raises some concerns, such as the limited understanding of their real impact on health systems and people's well-being or the potential risks derived from their use. In this context, health technology assessment (HTA) is 1 of the main tools that health systems can use to appraise evidence and determine the value of a given dHT. Nevertheless, due to the nature of dHTs, experts highlight the need to reconsider the frameworks used in traditional HTA.

OBJECTIVE

This scoping review (ScR) aimed to identify the methodological frameworks used worldwide for digital health technology assessment (dHTA); determine what domains are being considered; and generate, through a thematic analysis, a proposal for a methodological framework based on the most frequently described domains in the literature.

METHODS

The ScR was performed in accordance with the guidelines established in the PRISMA-ScR guidelines. We searched 7 databases for peer reviews and gray literature published between January 2011 and December 2021. The retrieved studies were screened using Rayyan in a single-blind manner by 2 independent authors, and data were extracted using ATLAS.ti software. The same software was used for thematic analysis.

RESULTS

The systematic search retrieved 3061 studies (n=2238, 73.1%, unique), of which 26 (0.8%) studies were included. From these, we identified 102 methodological frameworks designed for dHTA. These frameworks revealed great heterogeneity between them due to their different structures, approaches, and items to be considered in dHTA. In addition, we identified different wording used to refer to similar concepts. Through thematic analysis, we reduced this heterogeneity. In the first phase of the analysis, 176 provisional codes related to different assessment items emerged. In the second phase, these codes were clustered into 86 descriptive themes, which, in turn, were grouped in the third phase into 61 analytical themes and organized through a vertical hierarchy of 3 levels: level 1 formed by 13 domains, level 2 formed by 38 dimensions, and level 3 formed by 11 subdimensions. From these 61 analytical themes, we developed a proposal for a methodological framework for dHTA.

CONCLUSIONS

There is a need to adapt the existing frameworks used for dHTA or create new ones to more comprehensively assess different kinds of dHTs. Through this ScR, we identified 26 studies including 102 methodological frameworks and tools for dHTA. The thematic analysis of those 26 studies led to the definition of 12 domains, 38 dimensions, and 11 subdimensions that should be considered in dHTA.

Health technology assessment frameworks for eHealth: A systematic review

Objectives

Traditionally, health technology assessment (HTA) focuses on assessing the impact of pharmaceutical technologies on health and care. Resources are scarce and policy makers aim to achieve effective, accessible health care. eHealth innovations are increasingly more integrated in all healthcare domains. However, how eHealth is assessed prior to its implementation in care practices is unclear. To support evidence-informed policy making, this study aimed to identify frameworks and methods for assessing eHealth's impact on health care.

Methods

The scientific literature in five bibliographical databases was systematically reviewed. Articles were included if the study was conducted in a clinical setting, used an HTA framework and assessed an eHealth service. A systematic qualitative narrative approach was applied for analysis and reporting.

Results

Twenty-one HTA frameworks were identified in twenty-three articles. All frameworks addressed outcomes related to the technical performance and functionalities of eHealth service under assessment. The majority also addressed costs (n = 19), clinical outcomes (n = 14), organizational (n = 15) and system level aspects (n = 13). Most frameworks can be classified as dimensional (n = 13), followed by staged (n = 3), hybrid (n = 3), and business modeling frameworks (n = 2). Six frameworks specified assessment outcomes and methods.

Conclusions

HTA frameworks are available for a-priori impact assessment of eHealth services. The frameworks vary in assessment outcomes, methods, and specificity. Demonstrated applicability in practice is limited. Recommendations include standardization of: (i) reporting characteristics of eHealth services, and (ii) specifying assessment outcomes and methods following a stepped-approach tailored to the functional characteristics of eHealth services. Standardization might improve the quality and comparability of eHTA assessments.