What Are Complex eHealth Innovations and How Do You Measure Them? Position Paper

Characterizing attributes of innovation of technologies for healthcare: a systematic review

OBJECTIVES

Characterizing and evaluating the holistic value of innovative healthcare technologies (e.g. treatments, services) constitutes a crucial goal to maximize limited resources. However, the characteristics of innovation have not been well identified. This review aims to describe the characteristics of healthcare innovation.

METHODS

We performed a comprehensive systematic search using PubMed, Embase, PsycINFO, and Econlit from inception to July 2022. Articles were included if they described innovation or the characteristics of innovation of the technologies in healthcare. Characteristics or definitions of innovation directly or indirectly described as innovation were extracted from the included articles. Two independent reviewers then conceptualized the identified characteristics of innovation to generate innovation attributes in healthcare.

RESULTS

In total, 103 articles were included in this review. Eight attributes describing innovation, i.e. novelty, step change, substantial benefits, an improvement over existing technologies, convenience and/or adherence, added value, acceptable cost, and uncounted benefits, were conceptualized. Most of the identified innovation attributes were based on the researchers' perspective.

CONCLUSIONS

This study conceptualized innovation attributes in healthcare based on the characteristics of healthcare innovation as defined in the literature. Further research is warranted to obtain a complete understanding of the perspectives of researchers and other stakeholders, including patients, healthcare providers, healthcare payers, and the pharmaceutical industry, on recognizing innovation in healthcare.KEY POINTSThis is the first systematic review to conceptualize attributes of healthcare innovation.We conceptualized eight attributes describing innovation, i.e. novelty, step change, substantial benefits, an improvement over existing technologies, convenience and/or adherence, added value, acceptable cost, and uncounted benefits based on the similar concept.In existing literature, patients' and caregivers' perspectives were less frequently found to describe the innovation attributes.Future research is needed to identify, measure, and value various stakeholders, including patients' and caregivers' perspectives on healthcare innovation.

The Effect of Innovation Capabilities of Health Care Organizations on the Quality of Health Information Technology: Model Development With Cross-sectional Data

Background

Large health organizations often struggle to build complex health information technology (HIT) solutions and are faced with ever-growing pressure to continuously innovate their information systems. Limited research has been conducted that explores the relationship between organizations’ innovative capabilities and HIT quality in the sense of achieving high-quality support for patient care processes.

Objective

The aim of this study is to explain how core constructs of organizational innovation capabilities are linked to HIT quality based on a conceptual sociotechnical model on innovation and quality of HIT, called the IQ_HIT model, to help determine how better information provision in health organizations can be achieved.

Methods

We designed a survey to assess various domains of HIT quality, innovation capabilities of health organizations, and context variables and administered it to hospital chief information officers across Austria, Germany, and Switzerland. Data from 232 hospitals were used to empirically fit the model using partial least squares structural equation modeling to reveal associations and mediating and moderating effects.

Results

The resulting empirical IQ_HIT model reveals several associations between the analyzed constructs, which can be summarized in 2 main insights. First, it illustrates the linkage between the constructs measuring HIT quality by showing that the professionalism of information management explains the degree of HIT workflow support (R²=0.56), which in turn explains the perceived HIT quality (R²=0.53). Second, the model shows that HIT quality was positively influenced by innovation capabilities related to the top management team, the information technology department, and the organization at large. The assessment of the model’s statistical quality criteria indicated valid model specifications, including sufficient convergent and discriminant validity for measuring the latent constructs that underlie the measures of HIT quality and innovation capabilities.

Conclusions

The proposed sociotechnical IQ_HIT model points to the key role of professional information management for HIT workflow support in patient care and perceived HIT quality from the viewpoint of hospital chief information officers. Furthermore, it highlights that organizational innovation capabilities, particularly with respect to the top management team, facilitate HIT quality and suggests that health organizations establish this link by applying professional information management practices. The model may serve to stimulate further scientific work in the field of HIT adoption and diffusion and to provide practical guidance to managers, policy makers, and educators on how to achieve better patient care using HIT.

360-degree Delphi: addressing sociotechnical challenges of healthcare IT

BACKGROUND

IT systems in the healthcare field can have a marked sociotechnical impact: they modify communication habits, alter clinical processes and may have serious ethical implications. The introduction of such systems involves very different groups of stakeholders because of the inherent multi-professionalism in medicine and the role of patients and their relatives that are often underrepresented. Each group contributes distinct perspectives and particular needs, which create specific requirements for IT systems and may strongly influence their acceptance and success. In the past, needs analysis, challenges and requirements for medical IT systems have often been addressed using consensus techniques such as the Delphi technique. Facing the heterogeneous spectrum of stakeholders there is a need to develop these techniques further to control the (strong) influence of the composition of the expert panel on the outcome and to deal systematically with potentially incompatible needs of stakeholder groups. This approach uses the strong advantages a Delphi study has, identifies the disadvantages of traditional Delphi techniques and aims to introduce and evaluate a modified approach called 360-Degree Delphi. Key aspects of 360-Degree Delphi are tested by applying the approach to the needs and requirements analysis of a system for managing patients' advance directives and living wills.

METHODS

360-Degree Delphi (short 360°D), as a modified Delphi process, is specified as a structured workflow with the optional use of stakeholder groups. The approach redefines the composition of the expert panel by setting up groups of different stakeholders. Consensus is created within individual stakeholder groups, but is also communicated between groups, while the iterative structure of the Delphi process remains unchanged. We hypothesize that (1) 360-Degree Delphi yields complementary statements from different stakeholders, which would be lost in classical Delphi; while (2) the variation of statements within individual stakeholder groups is lower than within the total collective. A user study is performed that addresses five stakeholder groups (patients, relatives, medical doctors, nurses and software developers) on the topic of living will communication in an emergency context. Qualitative open questions are used in a Delphi round 0. Answer texts are coded by independent raters who carry out systematic bottom-up qualitative text analysis. Inter-rater reliability is calculated and the resulting codes are used to test the hypotheses. Qualitative results are transferred into quantitative questions and then surveyed in round 1. The study took place in Germany.

RESULTS

About 25% of the invited experts (stakeholders) agreed to take part in the Delphi round 0 (three patients, two relatives, three medical doctors, two qualified nurses and three developers), forming a structured panel of the five stakeholder groups. Two raters created a bottom-up coding, and 238 thematic codes were identified by the qualitative text analysis. The inter-rater reliability showed that 44.95% of the codes were semantically similar and coded for the same parts of the raw textual replies. Based on a consented coding list, a quantitative online-questionnaire was developed and send to different stakeholder groups. With respect to the hypotheses, Delphi round 0 had the following results: (1) doctors had a completely different focus from all the other stakeholder groups on possible channels of communications with the patient; (2) the dispersion of codes within individual stakeholder groups and within the total collective - visualized by box plots - was approximately 28% higher in the total collective than in the sub-collectives, but without a marked effect size. With respect to the hypotheses, Delphi round 1 had the following results: different stakeholder groups had highly diverging opinions with respect to central questions on IT-development. For example, when asked to rate the importance of access control against high availability of data (likert scale, 1 meaning restrictive data access, 6 easy access to all data), patients (mean 4.862, Stdev +/- 1.866) and caregivers (mean 5.667, Stdev: +/- 0.816) highly favored data availability, while relatives would restrict data access (mean 2.778, stdev +/- 1.093). In comparison, the total group would not be representative of either of these individual stakeholder needs (mean 4.344, stdev +/- 1.870).

CONCLUSION

360-Degree Delphi is feasible and allows different stakeholder groups within an expert panel to reach agreement individually. Thus, it generates a more detailed consensus which pays more tribute to individual stakeholders needs. This has the potential to improve the time to consensus as well as to produce a more representative and precise needs and requirements analysis. However, the method may create new challenges for the IT development process, which will have to deal with complementary or even contradictory statements from different stakeholder groups.

Innovative Power of Health Care Organisations Affects IT Adoption: A bi-National Health IT Benchmark Comparing Austria and Germany

Multinational health IT benchmarks foster cross-country learning and have been employed at various levels, e.g. OECD and Nordic countries. A bi-national benchmark study conducted in 2007 revealed a significantly higher adoption of health IT in Austria compared to Germany, two countries with comparable healthcare systems. We now investigated whether these differences still persisted. We further studied whether these differences were associated with hospital intrinsic factors, i.e. the innovative power of the organisation and hospital demographics. We thus performed a survey to measure the "perceived IT availability" and the "innovative power of the hospital" of 464 German and 70 Austrian hospitals. The survey was based on a questionnaire with 52 items and was given to the directors of nursing in 2013/2014. Our findings confirmed a significantly greater IT availability in Austria than in Germany. This was visible in the aggregated IT adoption composite score "IT function" as well as in the IT adoption for the individual functions "nursing documentation" (OR = 5.98), "intensive care unit (ICU) documentation" (OR = 2.49), "medication administration documentation" (OR = 2.48), "electronic archive" (OR = 2.27) and "medication" (OR = 2.16). "Innovative power" was the strongest factor to explain the variance of the composite score "IT function". It was effective in hospitals of both countries but significantly more effective in Austria than in Germany. "Hospital size" and "hospital system affiliation" were also significantly associated with the composite score "IT function", but they did not differ between the countries. These findings can be partly associated with the national characteristics. Indicators point to a more favourable financial situation in Austrian hospitals; we thus argue that Austrian hospitals may possess a larger degree of financial freedom to be innovative and to act accordingly. This study is the first to empirically demonstrate the effect of "innovative power" in hospitals on health IT adoption in a bi-national health IT benchmark. We recommend directly including the financial situation into future regression models. On a political level, measures to stimulate the "innovative power" of hospitals should be considered to increase the digitalisation of healthcare.

How Does Learnability of Primary Care Resident Physicians Increase After Seven Months of Using an Electronic Health Record? A Longitudinal Study

Background

Electronic health records (EHRs) with poor usability present steep learning curves for new resident physicians, who are already overwhelmed in learning a new specialty. This may lead to error-prone use of EHRs in medical practice by new resident physicians.

Objective

The study goal was to determine learnability gaps between expert and novice primary care resident physician groups by comparing performance measures when using EHRs.

Methods

We compared performance measures after two rounds of learnability tests (November 12, 2013 to December 19, 2013; February 12, 2014 to April 22, 2014). In Rounds 1 and 2, 10 novice and 6 expert physicians, and 8 novice and 4 expert physicians participated, respectively. Laboratory-based learnability tests using video analyses were conducted to analyze learnability gaps between novice and expert physicians. Physicians completed 19 tasks, using a think-aloud strategy, based on an artificial but typical patient visit note. We used quantitative performance measures (percent task success, time-on-task, mouse activities), a system usability scale (SUS), and qualitative narrative feedback during the participant debriefing session.

Results

There was a 6-percentage-point increase in novice physicians’ task success rate (Round 1: 92%, 95% CI 87-99; Round 2: 98%, 95% CI 95-100) and a 7-percentage-point increase in expert physicians’ task success rate (Round 1: 90%, 95% CI 83-97; Round 2: 97%, 95% CI 93-100); a 10% decrease in novice physicians’ time-on-task (Round 1: 44s, 95% CI 32-62; Round 2: 40s, 95% CI 27-59) and 21% decrease in expert physicians’ time-on-task (Round 1: 39s, 95% CI 29-51; Round 2: 31s, 95% CI 22-42); a 20% decrease in novice physicians mouse clicks (Round 1: 8 clicks, 95% CI 6-13; Round 2: 7 clicks, 95% CI 4-12) and 39% decrease in expert physicians’ mouse clicks (Round 1: 8 clicks, 95% CI 5-11; Round 2: 3 clicks, 95% CI 1-10); a 14% increase in novice mouse movements (Round 1: 9247 pixels, 95% CI 6404-13,353; Round 2: 7991 pixels, 95% CI 5350-11,936) and 14% decrease in expert physicians’ mouse movements (Round 1: 7325 pixels, 95% CI 5237-10,247; Round 2: 6329 pixels, 95% CI 4299-9317). The SUS measure of overall usability demonstrated only minimal change in the novice group (Round 1: 69, high marginal; Round 2: 68, high marginal) and no change in the expert group (74; high marginal for both rounds).

Conclusions

This study found differences in novice and expert physicians’ performance, demonstrating that physicians’ proficiency increased with EHR experience. Our study may serve as a guideline to improve current EHR training programs. Future directions include identifying usability issues faced by physicians when using EHRs, through a more granular task analysis to recognize subtle usability issues that would otherwise be overlooked.