14 Years longitudinal evaluation of clinical information systems acceptance: The HEGP case

Replication studies in the clinical decision support literature-frequency, fidelity, and impact

OBJECTIVE

To assess the frequency, fidelity, and impact of replication studies in the clinical decision support system (CDSS) literature.

MATERIALS AND METHODS

A PRISMA-compliant review identified CDSS replications across 28 health and biomedical informatics journals. Included articles were assessed for fidelity to the original study using 5 categories: Identical, Substitutable, In-class, Augmented, and Out-of-class; and 7 IMPISCO domains: Investigators (I), Method (M), Population (P), Intervention (I), Setting (S), Comparator (C), and Outcome (O). A fidelity score and heat map were generated using the ratings.

RESULTS

From 4063 publications matching search criteria for CDSS research, only 12/4063 (0.3%) were ultimately identified as replications. Six articles replicated but could not reproduce the results of the Han et al (2005) CPOE study showing mortality increase and, over time, changed from truth testing to generalizing this result. Other replications successfully tested variants of CDSS technology (2/12) or validated measurement instruments (4/12).

DISCUSSION

A replication rate of 3 in a thousand studies is low even by the low rates in other disciplines. Several new reporting methods were developed for this study, including the IMPISCO framework, fidelity scores, and fidelity heat maps. A reporting structure for clearly identifying replication research is also proposed.

CONCLUSION

There is an urgent need to better characterize which core CDSS principles require replication, identify past replication data, and conduct missing replication studies. Attention to replication should improve the efficiency and effectiveness of CDSS research and avoiding potentially harmful trial and error technology deployment.

A Technology Acceptance Model for Deploying Masks to Combat the COVID-19 Pandemic in Taiwan (My Health Bank): Web-Based Cross-sectional Survey Study

BACKGROUND

The successful completion of medical practices often relies on information collection and analysis. Government agencies and medical institutions have encouraged people to use medical information technology (MIT) to manage their conditions and promote personal health. In 2014, Taiwan established the first electronic personal health record (PHR) platform, My Health Bank (MHB), which allows people to access and manage their PHRs at any time. In the face of the COVID-19 pandemic in 2020, Taiwan has used MIT to effectively prevent the spread of COVID-19 and undertaken various prevention measures before the onset of the outbreak. Using MHB to purchase masks in an efficient and orderly way and thoroughly implementing personal protection efforts is highly important to contain disease spread.

OBJECTIVE

This study aims to understand people's intention to use the electronic PHR platform MHB and to investigate the factors affecting their intention to use this platform.

METHODS

From March 31 to April 9, 2014, in a promotion via email and Facebook, participants were asked to fill out a structured questionnaire after watching an introductory video about MHB on YouTube. The questionnaire included seven dimensions: perceived usefulness, perceived ease of use, health literacy, privacy and security, computer self-efficacy, attitude toward use, and behavioral intention to use. Each question was measured on a 5-point Likert scale ranging from "strongly disagree" (1 point) to "strongly agree" (5 points). Descriptive statistics and structural equation analysis were performed using SPSS 21 and AMOS 21 software.

RESULTS

A total of 350 valid questionnaire responses were collected (female: 219/350, 62.6%; age: 21-30 years: 238/350, 68.0%; university-level education: 228/350, 65.1%; occupation as student: 195/350, 56.6%; average monthly income

CONCLUSIONS

From the perspective of the populace, this study explored the factors affecting the use of MHB and constructed an interpretation model with a strong goodness of fit. The results of our analysis are consistent with the technology acceptance model. Through the diverse value-added services of MHB, Taiwan's experience in pandemic prevention with smart technology can facilitate future responses to unknown, emerging infectious diseases.

Collapse

Key Words

• COVID-19
• electronic health record
• electronic medical record
• intention
• literacy
• model
• my health bank
• personal health record
• privacy
• protection
• security
• structural equation model
• survey
• technology acceptance model
• usage

Collapse

MESH Headings

• Adult
• COVID-19/epidemiology
• COVID-19/prevention & control
• COVID-19/transmission
• Cross-Sectional Studies
• Disease Transmission, Infectious/prevention & control
• Female
• Humans
• Internet
• Male
• Masks
• Pandemics
• SARS-CoV-2
• Surveys and Questionnaires
• Taiwan/epidemiology
• Technology
• Young Adult

Collapse

Grants Collapse

Affiliation(s)

Wen-Hsun Tsai Graduate Institute of Medical Sciences, National Defense Medical Center, Taipei, Taiwan Medical Administration Office, Beitou Branch, Tri-Service General Hospital, Taipei, Taiwan
Yi-Syuan Wu Graduate Institute of Life Sciences, National Defense Medical Center, Taipei, Taiwan
Chia-Shiang Cheng Graduate Institute of Life Sciences, National Defense Medical Center, Taipei, Taiwan
Ming-Hao Kuo Graduate Institute of Medical Sciences, National Defense Medical Center, Taipei, Taiwan
Yu-Tien Chang School of Public Health, National Defense Medical Center, Taipei, Taiwan
Fu-Kang Hu Graduate Institute of Medical Sciences, National Defense Medical Center, Taipei, Taiwan Medical Informatics Office, Tri-Service General Hospital, Taipei, Taiwan
Chien-An Sun Big Data Research Center, College of Medicine, Fu-Jen Catholic University, New Taipei City, Taiwan
Chi-Wen Chang School of Nursing, College of Medicine, Chang-Gung University, Taoyuan, Taiwan Division of Pediatric Endocrinology & Genetics, Department of Pediatrics, Chang-Gung Memorial Hospital, Taoyuan, Taiwan
Ta-Chien Chan Research Center for Humanities and Social Sciences, Academia Sinica, Taipei, Taiwan
Chao-Wen Chen Trauma and Critical Care Service Department of Surgery, Kaohsiung Medical University Hospital, Kaohsiung, Taiwan
Chia-Cheng Lee Medical Informatics Office, Tri-Service General Hospital, Taipei, Taiwan Division of Colorectal Surgery, Department of Surgery, Tri-Service General Hospital, Taipei, Taiwan
Chi-Ming Chu Graduate Institute of Medical Sciences, National Defense Medical Center, Taipei, Taiwan Graduate Institute of Life Sciences, National Defense Medical Center, Taipei, Taiwan School of Public Health, National Defense Medical Center, Taipei, Taiwan Big Data Research Center, College of Medicine, Fu-Jen Catholic University, New Taipei City, Taiwan Department of Public Health, China Medical University, Taichung, Taiwan Department of Public Health, College of Health Sciences, Kaohsiung Medical University, Kaohsiung, Taiwan Department of Healthcare Administration and Medical Informatics, College of Health Sciences, Kaohsiung Medical University, Kaohsiung, Taiwan Department of Medical Research, Kaohsiung Medical University Hospital, Kaohsiung, Taiwan

Collapse

External validation of the unified model of information systems continuance (UMISC): An international comparison

CONTEXT

The Unified Model of Information Systems Continuance (UMISC) is a metamodel for the evaluation of clinical information systems (CISs) that integrates constructs from five models that have previously been published in the literature. UMISC was developed at the Georges Pompidou University Hospital (HEGP) in Paris and was partially validated at the Saint Joseph Hospital Group (HPSJ), another acute care institution using the same CIS as HEGP.

OBJECTIVE

The aim of this replication study was twofold: (1) to perform an external validation of UMISC in two different hospitals and country contexts: the Italian Hospital of Buenos Aires (HIBA) in Argentina and the Hospital Sirio Libanes in Sao Paulo, Brazil (HSL); (2) to compare, using the same evaluation model, the determinants of satisfaction, use, and continuance intention observed at HIBA and HSL with those previously observed at HEGP and HPSJ.

METHODS

The UMISC evaluation questionnaires were translated from their original languages (English and French) to Brazilian Portuguese and Spanish following the translation/back-translation method. These questionnaires were then applied at each target site. The 21 UMISC-associated hypotheses were tested using structural equation modeling (SEM).

RESULTS

A total of 3020 users, 1079 at HIBA and 1941 at the HSL, were included in the analysis. The respondents included 1406 medical staff and 1001 nursing staff. The average profession-adjusted use, overall satisfaction and continuance intention were significantly higher at HIBA than at HSL in the medical and nursing groups. In SEM analysis, UMISC explained 23% and 11% of the CIS use dimension, 72% and 85% of health professionals' satisfaction, and 41% and 60% of continuance intention at HIBA and HSL, respectively. Twenty of the 21 UMISC-related hypotheses were validated in at least one of the four evaluation sites, and 16 were validated in two or more sites.

CONCLUSION

The UMISC evaluation metamodel appears to be a robust comparison and explanatory model of satisfaction, use and continuance intention for CISs in late post adoption situations.

Evaluation of the unified model of information systems continuance (UMISC) in two hospital environments

CONTEXT

The deployment and long-term acceptance of clinical information systems (CISs) are faced with multiple difficulties. They include insufficient quality of the systems in place and resistance to the multiple changes they induce in care processes. Permanent evaluation of deployed solutions is a prerequisite to their continuous improvement.

OBJECTIVE

The purpose of this study was twofold: (1) To validate the post-adoption unified model of information systems continuance (UMISC) progressively developed at the Georges Pompidou University Hospital (HEGP) in Paris (internal validation); and (2) To compare, using the same evaluation model, the results observed at HEGP with those of the Saint-Joseph Hospital Group (HPSJ), another Paris acute care institution (external validation).

METHODS

The UMISC post-adoption model is built around nine dimensions: end-user characteristics, social norm (SN), IS quality (ISQ), facilitating conditions (FC), perceived usefulness (PU), confirmation of expectations (CE), profession-adjusted use (PAU), satisfaction (SAT), and continuance intention (CI). Two semi-quantitative evaluation surveys were performed at HEGP in 2014 and 2015, and one at HPSJ in 2015. Statistical analysis included multiple regression analysis and structural equation modeling (SEM).

RESULTS

The analysis concerned 459 responders, 264 at HEGP and 195 at HPSJ. UMISC indicators, with the exception of SN, are superior at HEGP than at HPSJ, which had a shorter CIS anteriority than HEGP. In SEM analysis, the UMISC model explained 25% and 40% of the CIS use, 92% and 93% of health professionals' satisfaction, and 72% and 71% of continuance intention at HEGP and HPSJ, respectively. Seventeen of the 21 tested UMISC hypotheses were supported in at least one of the two sites.

CONCLUSION

The UMISC evaluation model can be used as a comparison and explanatory model of CIS use, satisfaction and continuance intention in post-CIS adoption situations that become prevalent in current electronic hospitals.

The Effectiveness of Health Care Information Technologies: Evaluation of Trust, Security Beliefs, and Privacy as Determinants of Health Care Outcomes

Background

The diffusion of health information technologies (HITs) within the health care sector continues to grow. However, there is no theory explaining how success of HITs influences patient care outcomes. With the increase in data breaches, HITs’ success now hinges on the effectiveness of data protection solutions. Still, empirical research has only addressed privacy concerns, with little regard for other factors of information assurance.

Objective

The objective of this study was to study the effectiveness of HITs using the DeLone and McLean Information Systems Success Model (DMISSM). We examined the role of information assurance constructs (ie, the role of information security beliefs, privacy concerns, and trust in health information) as measures of HIT effectiveness. We also investigated the relationships between information assurance and three aspects of system success: attitude toward health information exchange (HIE), patient access to health records, and perceived patient care quality.

Methods

Using structural equation modeling, we analyzed the data from a sample of 3677 cancer patients from a public dataset. We used R software (R Project for Statistical Computing) and the Lavaan package to test the hypothesized relationships.

Results

Our extension of the DMISSM to health care was supported. We found that increased privacy concerns reduce the frequency of patient access to health records use, positive attitudes toward HIE, and perceptions of patient care quality. Also, belief in the effectiveness of information security increases the frequency of patient access to health records and positive attitude toward HIE. Trust in health information had a positive association with attitudes toward HIE and perceived patient care quality. Trust in health information had no direct effect on patient access to health records; however, it had an indirect relationship through privacy concerns.

Conclusions

Trust in health information and belief in the effectiveness of information security safeguards increases perceptions of patient care quality. Privacy concerns reduce patients’ frequency of accessing health records, patients’ positive attitudes toward HIE exchange, and overall perceived patient care quality. Health care organizations are encouraged to implement security safeguards to increase trust, the frequency of health record use, and reduce privacy concerns, consequently increasing patient care quality.

Understanding optimisation processes of electronic health records (EHRs) in select leading hospitals: a qualitative study

Background

Little is known about optimisation of electronic health records (EHRs) systems in the hospital setting while adoption of EHR systems continues in the United States.

Objective

To understand optimisation processes of EHR systems undertaken in leading healthcare organisations in the United States.

Methods

Informed by a grounded theory approach, a qualitative study was undertaken that involved 11 in-depth interviews and a focus group with the EHR experts from the high performing healthcare organisations across the United States.

Results

The study describes EHR optimisation processes characterised by prioritising exponentially increasing requests with predominant focus on improving efficiency of EHR, building optimisation teams or advisory groups and standardisation. The study discusses 16 types of optimisation that interdependently produced 16 results along with identifying 11 barriers and 20 facilitators to optimisation.

Conclusions

The study describes overall experiences of optimising EHRs in select high performing healthcare organisations in the US. The findings highlight the importance of optimising the EHR after, and even before, go-live and dedicating resources exclusively for optimisation.

Clinical Information Systems as the Backbone of a Complex Information Logistics Process: Findings from the Clinical Information Systems Perspective for 2016

Objective: To summarize recent research and to propose a selection of best papers published in 2016 in the field of Clinical Information Systems (CIS). Method: The query used to retrieve the articles for the CIS section of the 2016 edition of the IMIA Yearbook of Medical Informatics was reused. It again aimed at identifying relevant publications in the field of CIS from PubMed and Web of Science and comprised search terms from the Medical Subject Headings (MeSH) catalog as well as additional free text search terms. The retrieved articles were categorized in a multi-pass review carried out by the two section editors. The final selection of candidate papers was then peer-reviewed by Yearbook editors and external reviewers. Based on the review results, the best papers were then chosen at the selection meeting with the IMIA Yearbook editorial board. Text mining, term co-occurrence mapping, and topic modelling techniques were used to get an overview on the content of the retrieved articles. Results: The query was carried out in mid-January 2017, yielding a consolidated result set of 2,190 articles published in 921 different journals. Out of them, 14 papers were nominated as candidate best papers and three of them were finally selected as the best papers of the CIS field. The content analysis of the articles revealed the broad spectrum of topics covered by CIS research. Conclusions: The CIS field is multi-dimensional and complex. It is hard to draw a well-defined outline between CIS and other domains or other sections of the IMIA Yearbook. The trends observed in the previous years are progressing. Clinical information systems are more than just sociotechnical systems for data collection, processing, exchange, presentation, and archiving. They are the backbone of a complex, trans-institutional information logistics process.

The Virtuous Circles of Clinical Information Systems: a Modern Utopia

CONTEXT

Clinical information systems (CIS) are developed with the aim of improving both the efficiency and the quality of care.

OBJECTIVE

This position paper is based on the hypothesis that such vision is partly a utopian view of the emerging eSociety.

METHODS

Examples are drawn from 15 years of experience with the fully integrated Georges Pompidou University Hospital (HEGP) CIS and temporal data series extracted from the data warehouses of Assistance Publique - Hôpitaux de Paris (AP-HP) acute care hospitals which share the same administrative organization as HEGP. Three main virtuous circles are considered: user satisfaction vs. system use, system use vs. cost efficiency, and system use vs quality of care.

RESULTS

In structural equation models (SEM), the positive bidirectional relationship between user satisfaction and use was only observed in the early HEGP CIS deployment phase (first four years) but disappeared in late post-adoption (≥8 years). From 2009 to 2013, financial efficiency of 20 AP-HP hospitals evaluated with stochastic frontier analysis (SFA) models diminished by 0.5% per year. The lower decrease of efficiency observed between the three hospitals equipped with a more mature CIS and the 17 other hospitals was of the same order of magnitude than the difference observed between pediatric and non-pediatric hospitals. Outcome quality benefits that would bring evidence to the system use vs. quality loop are unlikely to be obtained in a near future since they require integration with population-based outcome measures including mortality, morbidity, and quality of life that may not be easily available.

CONCLUSION

Barriers to making the transformation of the utopian part of the CIS virtuous circles happen should be overcome to actually benefit the emerging eSociety.

Information system end-user satisfaction and continuance intention: A unified modeling approach

CONTEXT

Permanent evaluation of end-user satisfaction and continuance intention is a critical issue at each phase of a clinical information system (CIS) project, but most validation studies are concerned with the pre- or early post-adoption phases.

OBJECTIVE

The purpose of this study was twofold: to validate at the Pompidou University Hospital (HEGP) an information technology late post-adoption model built from four validated models and to propose a unified metamodel of evaluation that could be adapted to each context or deployment phase of a CIS project.

METHODS

Five dimensions, i.e., CIS quality (CISQ), perceived usefulness (PU), confirmation of expectations (CE), user satisfaction (SAT), and continuance intention (CI) were selected to constitute the CI evaluation model. The validity of the model was tested using the combined answers to four surveys performed between 2011 and 2015, i.e., more than ten years after the opening of HEGP in July 2000. Structural equation modeling was used to test the eight model-associated hypotheses.

RESULTS

The multi-professional study group of 571 responders consisted of 158 doctors, 282 nurses, and 131 secretaries. The evaluation model accounted for 84% of variance of satisfaction and 53% of CI variance for the period 2011-2015 and for 92% and 69% for the period 2014-2015. In very late post adoption, CISQ appears to be the major determinant of satisfaction and CI. Combining the results obtained at various phases of CIS deployment, a Unified Model of Information System Continuance (UMISC) is proposed.

CONCLUSION

In a meaningful CIS use situation at HEGP, this study confirms the importance of CISQ in explaining satisfaction and CI. The proposed UMISC model that can be adapted to each phase of CIS deployment could facilitate the necessary efforts of permanent CIS acceptance and continuance evaluation.